diff --git a/architecture/deep-dive-dashboard.md b/architecture/deep-dive-dashboard.md index 9f8269cc5..532eaaab8 100644 --- a/architecture/deep-dive-dashboard.md +++ b/architecture/deep-dive-dashboard.md @@ -4,8 +4,10 @@ Back to [Architecture](README.md). See also the review index in [overview.md](overview.md) (§§3, 12): §3 is the birds-eye route topology, this file is the dashboard authority. -`rust/src/server/dashboard/` is the dashboard subsystem (17 files: 8 -top-level plus 9 under `render/`). Its `mod.rs` is a +`rust/src/server/dashboard/` is the dashboard subsystem (19 files: 9 +top-level plus 10 under `render/`). Its `admin.rs` owns bounded user, +profile, usage, audit, and personal-token page handlers; `render/admin.rs` +owns their pure HTML projections. Its `mod.rs` is a thin facade that preserves the route entry points consumed by `rust/src/server/mod.rs`. `routes.rs` owns page request parsing and bounded data gathering; `api.rs` owns dashboard JSON request parsing and projections; @@ -48,7 +50,7 @@ unconditionally (inference at `/v1/chat/completions`, `/v1/messages`, `/static/dashboard.js`, `/static/chart.js` (served from `chart.umd.min.js`), `/static/favicon.svg`, `/static/theme.css`). The remaining dashboard surface below is dashboard-gated: it is registered -only when `[dashboard].enabled`, for 14 pages + 8 JSON endpoints. +only when `[dashboard].enabled`, for 17 dashboard-gated pages + 8 JSON endpoints. When `[dashboard].enabled`, the router additionally serves `/`, `/accounts`, `/models`, `/models/{*model_id}`, `/latency`, `/events`, `/timeseries`, `/bandwidth`, @@ -98,6 +100,18 @@ responses, and rendering escapes operator/provider-controlled values. Runtime and generation views distinguish active and retiring ownership without exposing raw prompts, credentials, cache keys, or provider bodies. +The admin/member surface is owned by `dashboard/admin.rs` and the pure +`render/admin.rs` renderer. `/users`, `/profiles`, and `/admin/audit` are +registered only with the dashboard. `/my-account`, `/my-account/models`, and +`/usage` are browser-session routes; all remain private after identity +initialization even when `dashboard.public = true`. Admin handlers require an +active admin browser principal and every mutation checks same-origin plus the +session-bound CSRF token. Agent credentials cannot enter this surface. +Personal credential plaintext is rendered only in the creation response, +which is `no-store`; subsequent account reads contain only bounded labels and +metadata. The persistent audit projection stores actor, action, target kind, +target ID, and time, without request content or secrets. + ## Rendering and themes Pages share one layout (`render_dashboard_layout`) with period/theme @@ -118,11 +132,14 @@ the embedded `rust/assets/dashboard/themes/*.toml` set parsed into CSS variables. Static assets carry explicit cache headers; failures to read dashboard data never block inference. -Default auth posture: `[dashboard].public` defaults to `true`, so ordinary -pages and non-sensitive stats data render without an API key. Inference -(`/v1/*`), integration (`/api/integrations/*`), and runtime/update/status -endpoints never inherit that exemption and stay authenticated; -`eggpool dashboard public --off` restores key auth on ordinary pages and data. +Legacy auth posture: `[dashboard].public` defaults to `true`, so ordinary +pages and non-sensitive stats data in `legacy_unmigrated` deployments render +without an API key. After identity initialization, dashboard pages and every +`/api/*` route require a browser session regardless of this setting; agent +tokens remain limited to inference, model discovery, and the integration +profile. `eggpool dashboard public --off` retains its legacy behavior, while +`eggpool users enable-managed` requires `dashboard.public = false` before the +managed transition. ## Safety invariants @@ -199,12 +216,31 @@ remain unavailable rather than being inferred by page rendering. The milestone evidence and source-to-panel review are recorded in `plans/closure/dashboard/004-status.md`. +## On-prem admin and member browser qualification + +The frozen Python oracle in `scripts/dashboard_parity/` only ever implemented the +historical M001-M012 pages. It is therefore the reference for those routes alone +and is never reopened as evidence for the on-prem user-access pages. Those pages +are owned by EggPool and are qualified by `scripts/dashboard_parity/on_prem.py`, +which starts one managed instance, bootstraps an administrator through +`eggpool users bootstrap`, and drives a real browser through both roles against +a Rust-owned approved reference at +`tests/fixtures/dashboard-on-prem-browser/reference.json`. + +Browser sign-in is gated by `server::auth::same_origin`. A same-origin browser +form POST sends `Origin` and `Referer` together, so every provenance header that +is present must agree with the request `Host`; duplicated or userinfo-bearing +provenance still fails closed. Accepting only a lone `Origin` would have rejected +every real browser while still admitting a bare `Referer`. + ## Verification ```bash cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 cargo test --manifest-path rust/Cargo.toml --test status_command -- --test-threads=1 cargo test --manifest-path rust/Cargo.toml --lib server::dashboard::tests -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_admin_ui -- --test-threads=1 uv run python scripts/qualification_dashboard_parity.py --screenshots uv run python scripts/qualification_dashboard_parity.py --shutdown-restart +uv run python scripts/qualification_dashboard_parity.py --skip-build --on-prem ``` diff --git a/architecture/deep-dive-database.md b/architecture/deep-dive-database.md index 3e3cbd808..5d8dc61f0 100644 --- a/architecture/deep-dive-database.md +++ b/architecture/deep-dive-database.md @@ -18,7 +18,72 @@ Commit/rollback ambiguity fails closed: body failure maps to `Transaction`/`Sqli ## Migrations -Migrations are embedded from `rust/assets/db/migrations/` (zero-padded `0001_*.sql`…`0054_*.sql` = versions 1–54) with `checksums.json` (`files` map, 54 entries): v1–v54, 54 files, validated by `MigrationRunner::validate_embedded_checksums` before any database use (`canonical_inventory_is_complete_and_immutable` guards first = 1, last = 54, len = 54). `MigrationRunner::run` keeps the `_migrations` ledger, rejects unknown versions and ledger-name mismatches, refuses read-only migration, and applies all pending migrations in one atomic transaction. History is never rewritten or renumbered; restore validation additionally checks the staged ledger against this inventory. +Migrations are embedded from `rust/assets/db/migrations/` (zero-padded `0001_*.sql`…`0062_*.sql` = versions 1–62) with `checksums.json` (`files` map, 62 entries), validated by `MigrationRunner::validate_embedded_checksums` before any database use. Migration 55 adds downstream identity and credential storage without repurposing upstream `accounts`; migration 56 adds browser sessions and CSRF digests; migration 57 adds user/auth-epoch attribution to request rows and selector-parent linkage; migration 58 adds local model-policy profiles, exact rules, per-user overrides, instance denials, and a monotonic policy revision; migration 59 adds user budget rules, request holds, attempt usage, audit, and bounded user/model/time rollups; migration 60 adds bounded secret-free administrative audit events; migration 61 adds the hard/estimated enforcement mode, worst-case hold facts, and price provenance to the budget tables; migration 62 adds the instance-global budget scope (rules, request holds, per-model usage buckets, audit, policy-revision triggers) with no user reference. Schema history is never rewritten or renumbered. `MigrationRunner::run` keeps the `_migrations` ledger, rejects unknown versions and ledger-name mismatches, refuses read-only migration, and applies all pending migrations in one atomic transaction. Restore validation additionally checks the staged ledger against this inventory. + +`users`, `user_credentials`, and `user_access_state` hold usernames, password +hashes, downstream roles, token digests, scopes, and explicit transition state. +The initial admin and first personal token are committed atomically. Backups +therefore carry the identity records and non-recoverable credential digests; +plaintext passwords and API tokens are never stored. +On a brand-new schema, migration entrypoints record `fresh_uninitialized`; +onboarding can complete bootstrap after an interrupted setup, and a keyless +fresh listener refuses to start. Existing databases start in +`legacy_unmigrated` and retain their configured authentication until local +identity initialization. Migration 56 adds opaque `web_sessions` with +session-token and CSRF digests, expiry, idle-use, and revocation state; password +reset revokes sessions in the same transaction as the credential revocation. +Request rows keep a nullable historical `user_id` without a foreign key, so +disabling or removing a local user cannot erase request attribution. New +authenticated publications snapshot the user, credential/session ID, auth +epoch, and principal kind in the same publication transaction. Internal +semantic selector rows carry a parent proxy request ID and are excluded from +logical user history queries. `identity::recent_user_requests` uses the +`(user_id, started_at DESC, id DESC)` index, clamps reads to 500 rows, and +returns content-free request facts. + +`model_policy` owns migration-58 profile and exact-model policy access. Policy +mutations advance the singleton revision in the same SQLite transaction; +database triggers also advance it when user state, role, or auth epoch changes. +The bounded process cache checks that revision before serving a new request, +then retains an immutable policy snapshot for the admitted request. Exact +instance denial has highest precedence, followed by user override, assigned +profile rule, and the role-safe default. Policy state remains separate from +provider topology and provider-account quota. + +`user_budget` keeps downstream budget enforcement separate from upstream +`reservations`, `AccountQuota`, and provider scoring. Daily and monthly windows +use UTC calendar boundaries. Publication checks exact user/model and +user-aggregate rules and inserts an active hold in the same transaction as the +request/attempt/provider reservation; a concurrent last-slot request cannot +pass on a stale preflight. Retries extend that logical hold without +incrementing its request count. Rules carry an explicit `hard`/`estimated` +enforcement mode (migration 61; pre-existing rows are estimated, promotion is +an explicit admin operation): estimated holds keep the request-size token +projection and fallback microdollars, while hard holds reserve the admission +worst case (input bound plus caller output cap, cost converted at the +operator-configured `[providers..budget]` tariff) and record the bound +plus price provenance on the reservation row. Unbounded output, unknown tariff, +or arithmetic overflow denies with `BudgetNotEnforceable` before any upstream +submission. Attempt facts are idempotently retained, and +terminal finalization settles one daily and one monthly usage bucket in the +same transaction that terminalizes the request. Unknown usage consumes the +remaining estimate and increments explicit unknown counters; provider-reported +usage above a hard hold is settled visibly, never clamped. Instance-global +ceilings (`instance_budget`) reuse the same hold formulas and the M008 tariff +trust boundary in the same publication transaction: global rules aggregate +across per-model usage rows with bounded indexed sums (no full history scan +under the gate), retries extend without consuming a second request, and +settlement folds the shared attempt rows. Instance rows carry no user +reference, so user disable/delete preserves history; with no matching +instance rule, publication takes no instance hold and writes no instance rows. +Provider-reported, +locally estimated, and reservation-fallback microdollars remain distinct in the +user rollup. Rule changes retain an actor/version/mode audit row. `GET +/api/user-budget` returns only the authenticated user's data unless an active +admin asks for a target `user_id`; each rule reports its enforcement mode. +Migration 60 adds `user_admin_audit`, a +bounded, secret-free actor/action/target record written transactionally with +user, token, and profile administration mutations. ## Publication and finalization transactions @@ -67,6 +132,9 @@ Persistence M004 (`plans/closure/persistence/004-status.md`) collected 14 accept - Migrations immutable and checksum-pinned; unknown or mismatched ledgers fail closed. - `backup_to` is a primitive; orchestration, validation, and restore stay in `operations/backup.rs`. - Startup reconciliation repairs only covered states; ambiguity never resolves silently. + Covered budget states: active user/instance holds on interrupted requests + are released without folding usage; settled holds, usage buckets, and + attempt-usage facts are never mutated by reconciliation. ## Verification diff --git a/architecture/deep-dive-integrations.md b/architecture/deep-dive-integrations.md index ef6307372..0cd8a8262 100644 --- a/architecture/deep-dive-integrations.md +++ b/architecture/deep-dive-integrations.md @@ -229,6 +229,14 @@ must-revalidate`, `ETag: ""` with `If-None-Match` → `304`. Errors are bounded and generic (`401`/`403` auth, `503` unavailable with no internal body). +For a personal user principal, model rows and virtual aliases are filtered by +the same effective model policy used at inference admission. The portable body +revision remains its sanitized content hash; the HTTP ETag adds the persisted +policy revision (`-p`) so edits that do not +change the visible projection still invalidate desktop caches. Restricted +users with no assigned profile default-deny unknown models; administrators +without an explicit profile retain the historical unrestricted default. + ## OpenCode provider contracts Both variants render from the same conservative projection and never claim @@ -344,3 +352,15 @@ duplicates renderers: proxy/agent/daemon, or dashboard assets, so publishing a Windows helper never implies Windows proxy support. + +## Personal client credentials + +After local identity initialization, a user can issue a personal agent token +from `/my-account` or an active administrator can issue one for a selected +member. The one-time `epk1` plaintext is returned in a no-store browser +response; the account page displays only bounded credential metadata and +revocation controls. Desktop clients continue to read the user's token from +`EGGPOOL_API_KEY`; `configremote`, Codex, OpenCode, and `eggpool-connect` do +not embed a global administrator key. Agent tokens remain scoped to inference, +model discovery, the versioned integration profile, and the user's budget +view. They do not authorize dashboard or administration routes. diff --git a/architecture/deep-dive-observability.md b/architecture/deep-dive-observability.md index 4f9049e0b..00a427994 100644 --- a/architecture/deep-dive-observability.md +++ b/architecture/deep-dive-observability.md @@ -126,6 +126,12 @@ correctness-critical request/accounting lifecycle rows stay durable. Shutdown performs one deadline-bounded metrics flush before the database closes. +Downstream user ownership lives on durable request rows, separate from the +provider/account-keyed `usage_rollups` dimensions. User history is queried by +owner through a bounded indexed repository and excludes internal selector +children from logical-request views. Routine metrics, routing traces, and +diagnostics still omit user IDs, credential IDs, scopes, prompts, and tokens. + ## Verification ```bash diff --git a/architecture/deep-dive-request-lifecycle.md b/architecture/deep-dive-request-lifecycle.md index 85ec5697e..02401efbf 100644 --- a/architecture/deep-dive-request-lifecycle.md +++ b/architecture/deep-dive-request-lifecycle.md @@ -31,6 +31,16 @@ adapts the response, and converges durable state. - `authenticate`/`requires_auth` enforce constant-time `Bearer`/`x-api-key` auth; `/api/integrations/*`, `/api/stats/runtime`, `/api/stats/update`, `/api/status`, and all `/v1/*` stay authenticated. + For authenticated inference, middleware installs a typed + `AuthenticatedPrincipal`: personal agent tokens resolve user, credential, + scopes, and auth epoch from SQLite; browser sessions resolve the same user + and session identity; the staged shared-key path is explicitly marked + `legacy_shared_key` with no downstream user ID. Request headers never set + the principal. Managed user requests capture one immutable policy snapshot + before model resolution. Requested aliases, provider-qualified models, + selector models, and resolved concrete targets are checked against it before + concrete provider dispatch. Denial is a generic 403; policy reads fail + closed, and admitted work retains its snapshot. `finish_empty_transport_body` only drains provably bodyless requests. ## 2. Endpoint classification (one parse, one call) @@ -53,7 +63,10 @@ adapts the response, and converges durable state. the ingress `Bytes` handle via refcount; provider-qualified rewrite re-encodes only on an actual model-id change); virtual aliases resolve through `SemanticSelector` over the same `FiniteCoordinator` with recursion refusal - and affinity commit in the endpoints layer only. + and affinity commit in the endpoints layer only. The typed principal is + passed through virtual selection too; selector dispatches persist as + `internal_selector` children linked to the parent proxy request ID, so + they remain attributable without counting as another logical user request. - `EndpointError::status` maps admission errors; `endpoint_error_body` shapes OpenAI vs Messages envelopes truncated to 512 bytes. `new_proxy_request_id` mints the opaque proxy identity; `coordinator/attempt.rs::add_forwarded_headers` @@ -106,13 +119,59 @@ adapts the response, and converges durable state. - `PublicationService::publish` converts one routing claim into durable `requests`/`request_attempts`/`reservations`/`routing_decisions` rows plus a `FinalizationIdentity`; duplicates observe `AlreadyPublished`, conflicts fail - closed. The worker survives caller cancellation and compensates lost - delivery. + closed. The request row stores owner user ID, credential/session ID, auth + epoch, principal kind, request kind, and optional selector parent. The + publication transaction rechecks active user/credential state before + dispatch; the immutable owner fields travel in `FinalizationIdentity` and + must still match before terminalization. Null user IDs remain explicitly + legacy/unattributed history. The worker survives caller cancellation and + compensates lost delivery. - `FinalizationSupervisor::register`/`wait` retains terminal ownership; `DurableFinalizer::finalize_request` vs `finalize_failed_attempt` separate terminal from retry-cleanup convergence. `CrashReconciler::reconcile_once` converges crash leftovers to interrupted/released without replay or double-charge. +- For managed principals, publication checks aggregate and exact-model + daily/monthly user budget rules and inserts the logical-request hold in that + same SQLite transaction. The hold survives failed attempts and is extended + by retries; it is not released until terminal settlement. Concurrency counts + all live holds across UTC window boundaries. Finalization records each + transitioned attempt once and settles one logical request in the terminal + transaction. Unknown usage consumes the remaining estimate and is marked + unknown in the indexed rollup. Request and concurrency caps are strictly + enforceable in both modes. Token/cost rules carry an explicit `hard` or + `estimated` mode (migration 61, default estimated): estimated rules keep the + request-size token projection and fallback microdollars, which do not prove + exact output or self-hosted cost bounds; hard rules admit only requests with + a provable worst-case bound — admission-carried input bound plus the + caller's explicit compatible output cap (`max_tokens`/`max_completion_tokens` + on chat, `max_output_tokens` on Responses/compact, `max_tokens` on Anthropic + Messages) converted at the operator-configured provider tariff — and deny + with `BudgetNotEnforceable` (429, no upstream submission) when unbounded, + unpriced, or overflowing. The initial hard hold covers two attempt + equivalents (concrete attempt plus unattributed selector overhead); retries + extend by one equivalent after re-checking. Observed provider usage above + the hold is settled visibly, never clamped. +- An optional instance-global ceiling is intersected in the same publication + transaction for every admitted call, attributed or not: the request must + satisfy all applicable instance-global, instance-model, user-global, and + user-model rules before claim conversion, with one shared hold per scope so + concurrent users cannot both take the final slot and retries never + double-count. Global usage folds the same attempt rows at terminal + settlement; absent global rules, no global hold or row is written. +- Crash recovery converges budget holds without folding usage. A restart + interrupts every `pending` request, and the same reconciliation pass + releases each active user/instance hold on those requests (`active → + released`, never settled): the logical request can never settle + afterwards, so the release is compensation, not credit, and unknown + upstream usage stays unknown. Settled holds and folded buckets are + untouched, and a stale duplicate terminal command conflicts instead of + re-folding. A retry under the same proxy ID after interruption is refused + at the duplicate gate (non-pending requests never re-enter admission), so + no hold is resurrected on dead work. Cancel terminalizes through the + normal terminal path and settles the hold conservatively at the held + amounts. Backups capture active holds verbatim as committed rows; a + restored image reconciles exactly like a crashed live database. - Reload policy is referenced, never duplicated: only `config_reload_policy.rs::classify_transition` decides reload vs restart. diff --git a/architecture/deep-dive-routing.md b/architecture/deep-dive-routing.md index 66887a01c..8e8a4fc10 100644 --- a/architecture/deep-dive-routing.md +++ b/architecture/deep-dive-routing.md @@ -24,6 +24,16 @@ validates/compiles policy and `rust/src/model_router.rs` owns the Tokio affinity selector resolves a virtual model to one concrete model; it cannot pin an account, bypass health/quota, or reselect after submission. +Downstream per-user model authorization is a separate earlier gate in +`coordinator/endpoints.rs`. One immutable `ModelPolicySnapshot` is resolved for +the authenticated user and checks the requested ID, the virtual alias, the +selector model, and the resolved concrete route before ordinary account +selection. Exact `model/provider` rules are supported; instance denials win +over user overrides, which win over profile rules and defaults. An affinity +hit and selector repair use the same request snapshot. A policy denial returns +403 before the concrete provider attempt, while provider eligibility, +health, quota, and fairness remain owned by this routing module. + ## Eligibility gates `eligibility.rs::build_eligible_candidates` is the read-only gate. `EligibilityPolicy::from_config` @@ -121,6 +131,15 @@ left `Pending` after a failed release. `record_success` and `apply_failure_effec health transitions owned at that boundary; model-scoped failures never advance the account circuit. +User budgets are enforced later at the durable publication boundary, after the +principal and concrete model are bound. `coordinator/publication.rs` checks +every matching aggregate and exact-model UTC day/month rule and inserts a +SQLite hold atomically with the request, attempt, and provider reservation. +This is independent from account quota scoring and does not change the +load-based selection order. Token holds use a bounded request-size estimate; +microdollar holds use the existing estimated cost projection, including its +fallback. They are not exact output-token or invoice caps. + ## Invariants - Selection is deterministic and load-based; pricing/accounting never influences ranking. diff --git a/architecture/deep-dive-security.md b/architecture/deep-dive-security.md index 2ded43bb9..db58fe536 100644 --- a/architecture/deep-dive-security.md +++ b/architecture/deep-dive-security.md @@ -20,15 +20,65 @@ structured events. capitalization) or the `x-api-key` header, pads both sides to fixed 512-byte buffers, and folds the full pads before combining with the shape bits — so shape validity never short-circuits the compare. Provided and -expected values must both satisfy `valid_key_shape()`. `validate_server_key()` -fails startup closed when a non-loopback bind has no key or any key has a -bad shape; a loopback listener without a key is allowed by design but logs -a loud unauthenticated-loopback warning. `requires_auth()` exempts only -`/v1/healthz`, `/v1/readyz`, and `/static/*`; inference (`/v1/*`), -`/api/integrations/*`, `/api/stats/runtime`, `/api/stats/update`, and -`/api/status` are always authenticated, and dashboard pages plus remaining -`/api/*` routes require a key only once `[dashboard].public` is off. The -integration profile never inherits a dashboard-public exemption. +expected values must both satisfy `valid_key_shape()`. A shared server key is +required for a non-loopback `legacy_unmigrated` installation; malformed +configured keys fail startup. After local identity initialization, keyless +listeners are allowed because protected routes use a personal agent token or +browser session. Keyless loopback legacy mode retains its explicit +unauthenticated warning. Legacy route behavior remains unchanged: health, +readiness, and static assets are public; sensitive agent/operational routes +require the shared key; ordinary dashboard pages follow +`[dashboard].public`. + +After local identity initialization, the configured global key is not +accepted on any route. Personal credentials are checked against their +selector and stored digest on each agent request. A new empty schema is marked +`fresh_uninitialized`, and a keyless listener refuses to start until local +bootstrap. In `admin_initialized` and `managed`, the public-dashboard setting +does not expose dashboard or `/api/*` data. Browser login/account routes are +session-only; agent credentials cannot access them. Health/readiness and +static assets remain public. Unknown `/api/*` routes require an administrator +session before normal 404 handling. + +Downstream human identity is separate from provider `accounts`. Migration 55 +adds local users, credentials, and access state; migration 56 adds opaque +browser sessions with only token and CSRF digests stored. Passwords use +bounded Argon2id hashes, and personal `epk1` credentials and `eps1` sessions +store a SHA-256 digest plus a random selector. Password hashing runs in a +bounded blocking worker pool outside the SQLite gate. Personal credentials +authenticate only inference, `/v1/models`, and the scoped integration-profile +endpoint; sessions authenticate dashboard administration and account routes +but never inference. `eggpool users reset-password` revokes API credentials +and browser sessions. The configured shared key stops authenticating when the +first local identity is created. Upgraded schemas remain `legacy_unmigrated` +until that identity is initialized. + +Sessions have a 12-hour idle bound and 7-day absolute expiry. Cookies are +HttpOnly, SameSite=Lax, root-path scoped, and Secure when the accepted +connection is HTTPS. Unsafe browser forms require same-origin Origin/Referer +and a session-bound CSRF token. A same-origin browser form POST presents `Origin` +and `Referer` together, so `server::auth::same_origin` requires every provenance +header that is present to agree with the request `Host` rather than exactly one +of them; duplicated headers and userinfo-bearing authorities still fail closed. +Remote login requires HTTPS or a trusted proxy peer listed as a literal IP in +`security.trusted_proxies` that sends one `X-Forwarded-Proto: https` value. +For browser-login and failed personal-token throttling, one literal client IP +in `X-Forwarded-For` is used when received from that exact trusted peer; the +proxy must replace this header. Missing, duplicated, malformed, or chained +values fall back to the proxy peer. No other forwarded header establishes +trust. +`security.allow_insecure_web_auth` is an explicit development override and +emits a startup warning. +Session-authenticated responses carry `Cache-Control: no-store, max-age=0` so +account and administration HTML containing identity, budget, or CSRF data is +not retained by shared or browser caches. Privileged browser mutations also +revalidate the exact session, user role, auth epoch, and forced-password state +inside the SQLite transaction that commits the write; creating a user and +assigning its initial profile is one transaction. This closes the interval +between HTTP middleware authentication and database commit. +Failed personal-token lookups use a 256-entry, one-second negative cache keyed +by token digest. Successful token lookups are not cached, so revocation takes +effect on the next request. ## Headers, bodies, and redaction diff --git a/artifacts/qualification/m010-gate-observations.json b/artifacts/qualification/m010-gate-observations.json new file mode 100644 index 000000000..4658ce690 --- /dev/null +++ b/artifacts/qualification/m010-gate-observations.json @@ -0,0 +1,44 @@ +{ + "milestone": "on-prem-user-access M010 package D", + "purpose": "Paired ordinary-host SQLite gate observations for budget admission/settlement overhead (plan section 5D). Not physical-SBC capacity evidence; M012 owns that proof.", + "collected_at": "2026-10-14", + "host": { + "os": "macos", + "arch": "aarch64", + "kernel": "Darwin 25.6.0 RELEASE_ARM64_T6041", + "description": "ordinary Apple-Silicon laptop host, unloaded, single run" + }, + "build": { + "profile": "debug (cargo test)", + "binary": "rust test target on_prem_global_budgets::paired_gate_observations_across_identities_and_rules", + "schema": 62, + "sqlite": "WAL, synchronous NORMAL, busy_timeout 5000ms, autocheckpoint 1000 pages" + }, + "method": { + "workload": "sequential publish_attributed + terminal settle per distinct credential identity; no concurrency, no upstream I/O", + "modes": { + "baseline": "no budget rules, unattributed legacy-shared-key publication", + "user_estimated": "per-user estimated token+USD rules for every identity, estimator projection admission", + "user_hard_global_hard": "per-user rules plus one instance-global hard rule, worst-case proof admission" + }, + "bounds": "per-operation anti-hang bound 30s only; no performance assertion in the test" + }, + "cells": [ + {"mode":"baseline","identities":1,"publish_us_min":651,"publish_us_p50":651,"publish_us_max":651,"settle_us_min":383,"settle_us_p50":383,"settle_us_max":383,"db_bytes":4096,"wal_bytes":1149512,"shm_bytes":32768,"wal_bytes_after_checkpoint":0,"db_bytes_after_checkpoint":831488,"requests":1,"active_user_holds":0,"active_instance_holds":0,"user_usage_rows":0}, + {"mode":"user_estimated","identities":1,"publish_us_min":780,"publish_us_p50":780,"publish_us_max":780,"settle_us_min":733,"settle_us_p50":733,"settle_us_max":733,"db_bytes":4096,"wal_bytes":1219552,"shm_bytes":32768,"wal_bytes_after_checkpoint":0,"db_bytes_after_checkpoint":831488,"requests":1,"active_user_holds":0,"active_instance_holds":0,"user_usage_rows":2}, + {"mode":"user_hard_global_hard","identities":1,"publish_us_min":910,"publish_us_p50":910,"publish_us_max":910,"settle_us_min":684,"settle_us_p50":684,"settle_us_max":684,"db_bytes":4096,"wal_bytes":1285472,"shm_bytes":32768,"wal_bytes_after_checkpoint":0,"db_bytes_after_checkpoint":831488,"requests":1,"active_user_holds":0,"active_instance_holds":0,"user_usage_rows":2}, + {"mode":"baseline","identities":10,"publish_us_min":485,"publish_us_p50":571,"publish_us_max":733,"settle_us_min":272,"settle_us_p50":310,"settle_us_max":853,"db_bytes":4096,"wal_bytes":3246592,"shm_bytes":32768,"wal_bytes_after_checkpoint":0,"db_bytes_after_checkpoint":843776,"requests":10,"active_user_holds":0,"active_instance_holds":0,"user_usage_rows":0}, + {"mode":"user_estimated","identities":10,"publish_us_min":585,"publish_us_p50":673,"publish_us_max":843,"settle_us_min":397,"settle_us_p50":467,"settle_us_max":521,"db_bytes":4096,"wal_bytes":3946992,"shm_bytes":32768,"wal_bytes_after_checkpoint":0,"db_bytes_after_checkpoint":843776,"requests":10,"active_user_holds":0,"active_instance_holds":0,"user_usage_rows":20}, + {"mode":"user_hard_global_hard","identities":10,"publish_us_min":626,"publish_us_p50":722,"publish_us_max":2776,"settle_us_min":512,"settle_us_p50":598,"settle_us_max":702,"db_bytes":843776,"wal_bytes":4181832,"shm_bytes":32768,"wal_bytes_after_checkpoint":0,"db_bytes_after_checkpoint":843776,"requests":10,"active_user_holds":0,"active_instance_holds":0,"user_usage_rows":20}, + {"mode":"baseline","identities":49,"publish_us_min":345,"publish_us_p50":424,"publish_us_max":2034,"settle_us_min":186,"settle_us_p50":224,"settle_us_max":529,"db_bytes":884736,"wal_bytes":4206552,"shm_bytes":32768,"wal_bytes_after_checkpoint":0,"db_bytes_after_checkpoint":925696,"requests":49,"active_user_holds":0,"active_instance_holds":0,"user_usage_rows":0}, + {"mode":"user_estimated","identities":49,"publish_us_min":440,"publish_us_p50":498,"publish_us_max":876,"settle_us_min":332,"settle_us_p50":365,"settle_us_max":2545,"db_bytes":913408,"wal_bytes":4185952,"shm_bytes":32768,"wal_bytes_after_checkpoint":0,"db_bytes_after_checkpoint":950272,"requests":49,"active_user_holds":0,"active_instance_holds":0,"user_usage_rows":98}, + {"mode":"user_hard_global_hard","identities":49,"publish_us_min":495,"publish_us_p50":588,"publish_us_max":2791,"settle_us_min":457,"settle_us_p50":503,"settle_us_max":1644,"db_bytes":954368,"wal_bytes":4317792,"shm_bytes":32768,"wal_bytes_after_checkpoint":0,"db_bytes_after_checkpoint":958464,"requests":49,"active_user_holds":0,"active_instance_holds":0,"user_usage_rows":98} + ], + "process_rss_kb_after_suite": 11984, + "reading": { + "publish_p50_49_identities_us": {"baseline": 424, "user_estimated": 498, "user_hard_global_hard": 588}, + "settle_p50_49_identities_us": {"baseline": 224, "user_estimated": 365, "user_hard_global_hard": 503}, + "worst_outlier_us": 2791, + "verdict": "Budget rules add sub-millisecond per-operation overhead in a debug build (worst observed outlier 2.8ms, cold first-touch). No active-hold leakage at any scale. WAL peaks at ~4.3MB for 49 sequential cycles and truncates cleanly on checkpoint; main DB stays under 1MB. No evidence of unacceptable gate-tenure widening from the global rollups; no storage redesign warranted." + } +} diff --git a/artifacts/qualification/on-prem-concurrency-performance/functional.json b/artifacts/qualification/on-prem-concurrency-performance/functional.json new file mode 100644 index 000000000..2462d03bd --- /dev/null +++ b/artifacts/qualification/on-prem-concurrency-performance/functional.json @@ -0,0 +1,658 @@ +{ + "backup": { + "archive_sha256": "edb4b3731da4b884ae4a2411804fcbe4481e7947670012625d0d4d4dfa0c8e24", + "archive_size_bytes": 843223, + "elapsed_ms": 5329, + "member_count": 3 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "q005-qualified-aarch64-copy", + "build_profile": "release", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 5, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 4, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 6, + "id": "check-config", + "reason": "Configuration loaded successfully from /tmp/eggpool-q008-2ud79jxt/config.toml Server: 127.0.0.1:34043 Accounts: 1 Database: Content digest: e777ef0b152f5929215f466ff6dfa535216aff22675573bed5a8881e7be1e4fe", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 3108, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 355, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 5, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 9, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 9, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 9, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 5, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 8, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 5329, + "id": "backup", + "reason": "Wrote backup: /tmp/eggpool-q008-2ud79jxt/backups/eggpool-backup-20261015-015727.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 4658, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2583925)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 9749, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /tmp/eggpool-q008-2ud79jxt/recovery/config.toml db: /tmp/eggpool-q008-2ud79jxt/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 6096, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 366, + "id": "stop", + "reason": "Stopping server (PID 2584004)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 8, + "pending_requests": 0, + "request_statuses": { + "completed": 8 + }, + "requests": 8, + "reservations": 8 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_governor": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_requests": 8, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 40.8, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 7, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 6 + }, + { + "body_bytes": 309, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 213, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 145, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 480, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18518016, + 18518016, + 20000768, + 20103168, + 19394560 + ], + "samples": 5, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 7, + 4, + 4 + ], + "finite_ttft_ms": [ + 6, + 4, + 4 + ], + "second_workload_elapsed_ms": [ + 7, + 4 + ], + "second_workload_ttft_ms": [ + 7, + 4 + ], + "streaming_elapsed_ms": [ + 4, + 4, + 4 + ], + "streaming_ttft_ms": [ + 3, + 3, + 3 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18518016, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98308096, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18518016, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98308096, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20000768, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20103168, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 1693352, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 8, + "cpu_percent": 0.0, + "db_bytes": 827392, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "pending_requests": 0, + "request_statuses": { + "completed": 8 + }, + "requests": 8, + "reservations": 8, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19394560, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98480128, + "wal_bytes": 556232, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v1", + "startup_to_ready_ms": 351, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-concurrency-performance/matrix.json b/artifacts/qualification/on-prem-concurrency-performance/matrix.json new file mode 100644 index 000000000..923f7b74a --- /dev/null +++ b/artifacts/qualification/on-prem-concurrency-performance/matrix.json @@ -0,0 +1,177 @@ +{ + "schema_version": "on-prem-concurrency-matrix.v1", + "plan": "on-prem-concurrency-performance M001", + "candidate": { + "repository_commit_sha": "d2c0830acaeff90c45bffa393453cf0500c4fa64", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "binary_size_bytes": 34984648, + "reported_origin": "q005-qualified-aarch64-copy", + "provenance_note": "The runner default origin was not overridden on the six benchmark invocations. The same-hash binary was built on-device by the preceding recorded locked release-build command; this metadata mismatch is retained and disclosed rather than editing raw reports." + }, + "target": { + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "architecture": "aarch64", + "os": "Ubuntu 24.04.4 LTS", + "kernel": "6.8.0-1065-raspi", + "filesystem": "ext4", + "root_storage_device_class": "mmc", + "attestation": "Linux aarch64 plus device-tree board model" + }, + "thresholds": { + "source": "On-prem user-access M012 G3, preserved unchanged", + "g3_p95_ms": 1000, + "g3_max_ms": 3000 + }, + "workload": { + "config_fixture": "tests/tooling/fixtures/qualification/sbc-benchmark.toml", + "provider": "deterministic loopback fixture", + "wire": "native OpenAI Responses finite", + "request_count_per_run": 32, + "concurrency": 4, + "sample_count": 30, + "warmup_count": 5, + "configured_provider_accounts": 1, + "named_principal": "one synthetic administrator personal token", + "hard_user_or_global_budgets_enabled": false, + "other_bounded_benchmarks": "30 each of native finite, native streaming, and translated Messages streaming", + "convergence": "all requests completed; pending_requests=0; active_reservations=0" + }, + "scope_limits": { + "tested_concurrency_levels": [4], + "tested_named_user_count": 1, + "hard_user_or_global_budgets_enabled": false + }, + "conditions": { + "mmc": { + "database_storage": "runner temporary root under /tmp on root ext4/MMC", + "runs": [ + "mmc-run-1.json", + "mmc-run-2.json", + "mmc-run-3.json" + ], + "concurrent_native_responses_finite": [ + {"p50_ms": 10, "p95_ms": 7599, "max_ms": 7599, "batch_ms": 13220}, + {"p50_ms": 11, "p95_ms": 12058, "max_ms": 12058, "batch_ms": 18524}, + {"p50_ms": 11, "p95_ms": 8579, "max_ms": 8579, "batch_ms": 15132} + ], + "concurrent_medians": {"p50_ms": 11, "p95_ms": 8579, "max_ms": 8579, "batch_ms": 15132}, + "native_finite_medians": {"p50_ms": 4, "p95_ms": 6780, "max_ms": 8381} + }, + "tmpfs": { + "database_storage": "runner temporary root under /dev/shm (tmpfs); qualification output remains on MMC", + "runs": [ + "tmpfs-run-1.json", + "tmpfs-run-2.json", + "tmpfs-run-3.json" + ], + "concurrent_native_responses_finite": [ + {"p50_ms": 10, "p95_ms": 10, "max_ms": 11, "batch_ms": 80}, + {"p50_ms": 10, "p95_ms": 10, "max_ms": 10, "batch_ms": 81}, + {"p50_ms": 9, "p95_ms": 13, "max_ms": 13, "batch_ms": 82} + ], + "concurrent_medians": {"p50_ms": 10, "p95_ms": 10, "max_ms": 11, "batch_ms": 81}, + "native_finite_medians": {"p50_ms": 4, "p95_ms": 4, "max_ms": 4} + } + }, + "supplemental_runs": { + "persistent_nvme_single_run": { + "artifact": "ssd-c4-u5-user-global-run-1.json", + "repeat_count": 1, + "database_storage": { + "filesystem": "ext4", + "storage_medium_class": "non-rotational", + "device_class": "nvme" + }, + "workload": { + "concurrency": 4, + "request_count": 1000, + "sample_count": 30, + "principal_count": 5, + "budget_mode": "user-global", + "competing_io": false + }, + "concurrent_native_responses_finite": { + "p50_ms": 13, + "p95_ms": 30, + "p99_ms": 32, + "max_ms": 33, + "completed": 1000, + "failed": 0, + "batch_ms": 4444 + }, + "interpretation": "Exploratory out-of-scope media run; SSD/NVMe is not required by M001 and this is not a paired media comparison." + }, + "microsd_user_global_budget_smoke": { + "artifact": "microsd-c4-u5-user-global-minimal-run-1.json", + "repeat_count": 1, + "database_storage": { + "filesystem": "ext4", + "storage_medium_class": "non-rotational", + "device_class": "mmc" + }, + "workload": { + "concurrency": 4, + "request_count": 4, + "sample_count": 1, + "principal_count": 5, + "budget_mode": "user-global", + "competing_io": false + }, + "concurrent_native_responses_finite": { + "p50_ms": 10, + "p95_ms": 13, + "max_ms": 13, + "completed": 4, + "failed": 0, + "batch_ms": 16 + }, + "interpretation": "Functional budget-path smoke only; four latency observations are not suitable for percentile or capacity inference." + } + }, + "commands": [ + "CARGO_BUILD_JOBS=1 cargo build --manifest-path rust/Cargo.toml --locked --release", + "uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --output artifacts/qualification/on-prem-concurrency-performance/functional.json", + "uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/mmc-run-1.json", + "uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/mmc-run-2.json", + "uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/mmc-run-3.json", + "TMPDIR=/dev/shm uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-1.json", + "TMPDIR=/dev/shm uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-2.json", + "TMPDIR=/dev/shm uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-3.json", + "TMPDIR=/mnt/qualification-ssd/eggpool-m001-2026-10-14 uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --expected-sha256 8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538 --candidate-origin on-device-release-build --benchmark-samples 30 --concurrency-workers 4 --concurrency-requests 1000 --benchmark-principals 5 --benchmark-budget-mode user-global --output artifacts/qualification/on-prem-concurrency-performance/ssd-c4-u5-user-global-run-1.json", + "TMPDIR=/tmp/eggpool-m001-microsd-2026-10-14 uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --expected-sha256 8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538 --candidate-origin on-device-release-build --benchmark-samples 30 --concurrency-workers 4 --concurrency-requests 1000 --benchmark-principals 5 --benchmark-budget-mode user-global --output artifacts/qualification/on-prem-concurrency-performance/microsd-c4-u5-user-global-run-1.json", + "TMPDIR=/tmp/eggpool-m001-microsd-c4-2026-10-15 uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --expected-sha256 8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538 --candidate-origin on-device-release-build --benchmark-samples 1 --concurrency-workers 4 --concurrency-requests 4 --benchmark-principals 5 --benchmark-budget-mode user-global --output artifacts/qualification/on-prem-concurrency-performance/microsd-c4-u5-user-global-minimal-run-1.json" + ], + "limitations": [ + "The 1/2/8 concurrency levels and 5/49 named-principal cases were not run; this corpus isolates the already-reproduced four-way tail.", + "Hard user/global budget policies were disabled in this fixture and were not varied.", + "One supplemental persistent NVMe run was recorded as exploratory evidence only; SSD/NVMe is not required by M001. No controlled competing-I/O trial is available. Ambient host I/O was not controlled and varied during the MMC runs.", + "The original Q008 matrix emits p50/p95/max; the supplemental 1000-request run also emits p99, but no p99 acceptance threshold has been frozen.", + "tmpfs is a diagnostic control only and is not a durability recommendation." + ], + "root_storage_medium": "microSD (SD card on MMC interface; root /dev/mmcblk0p2, ext4)", + "attempted_runs": { + "microsd_c4_u5_user_global_aborted": { + "command": "TMPDIR=/tmp/eggpool-m001-microsd-2026-10-14 uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --expected-sha256 8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538 --candidate-origin on-device-release-build --benchmark-samples 30 --concurrency-workers 4 --concurrency-requests 1000 --benchmark-principals 5 --benchmark-budget-mode user-global --output artifacts/qualification/on-prem-concurrency-performance/microsd-c4-u5-user-global-run-1.json", + "status": "aborted_by_plan_stop_condition", + "target_storage": "microSD (SD card via MMC interface)", + "observed_io_pressure_percent": {"some_avg10": 76.31, "full_avg10": 72.21, "some_avg60": 59.67, "full_avg60": 56.82}, + "report_artifact": "not produced", + "result_interpretation": "The run was interrupted after sustained system I/O pressure crossed the plan stop condition; no performance result is claimed." + }, + "microsd_c4_u5_user_global_c64_aborted": { + "command": "TMPDIR=/tmp/eggpool-m001-microsd-c64-2026-10-14 uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --expected-sha256 8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538 --candidate-origin on-device-release-build --benchmark-samples 30 --concurrency-workers 4 --concurrency-requests 64 --benchmark-principals 5 --benchmark-budget-mode user-global --output artifacts/qualification/on-prem-concurrency-performance/microsd-c4-u5-user-global-c64-run-1.json", + "status": "aborted_by_plan_stop_condition", + "observed_io_pressure_percent": {"some_avg10": 76.62, "full_avg10": 71.59, "some_avg60": 59.67, "full_avg60": 50.80}, + "report_artifact": "not produced", + "result_interpretation": "The 64-request attempt was interrupted as sustained microSD I/O pressure approached the plan stop condition." + }, + "microsd_c4_u5_user_global_c16_aborted": { + "command": "TMPDIR=/tmp/eggpool-m001-microsd-c16-2026-10-14 uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --expected-sha256 8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538 --candidate-origin on-device-release-build --benchmark-samples 30 --concurrency-workers 4 --concurrency-requests 16 --benchmark-principals 5 --benchmark-budget-mode user-global --output artifacts/qualification/on-prem-concurrency-performance/microsd-c4-u5-user-global-c16-run-1.json", + "status": "aborted_by_plan_stop_condition", + "observed_io_pressure_percent": {"some_avg10": 73.78, "full_avg10": 70.56, "some_avg60": 47.94, "full_avg60": 45.70}, + "report_artifact": "not produced", + "result_interpretation": "The 16-request attempt was interrupted as sustained microSD I/O pressure approached the plan stop condition." + } + } +} diff --git a/artifacts/qualification/on-prem-concurrency-performance/microsd-c4-u5-user-global-minimal-run-1.json b/artifacts/qualification/on-prem-concurrency-performance/microsd-c4-u5-user-global-minimal-run-1.json new file mode 100644 index 000000000..ee839973a --- /dev/null +++ b/artifacts/qualification/on-prem-concurrency-performance/microsd-c4-u5-user-global-minimal-run-1.json @@ -0,0 +1,1024 @@ +{ + "backup": { + "archive_sha256": "72e7cda85d9cc8bb7786d13258c02e43bdab4989a664ccf2a826302ba84590b0", + "archive_size_bytes": 884323, + "elapsed_ms": 3214, + "member_count": 3 + }, + "benchmark": { + "budget_mode": "user-global", + "cadence": { + "backup_enabled": "false", + "backup_interval_s": "86400", + "database_busy_timeout_ms": "10000", + "database_synchronous": "\"NORMAL\"", + "database_wal": "true", + "database_worker_threads": "1", + "metrics_aggregate_only": "true", + "metrics_cleanup_interval_s": "86400", + "metrics_event_loop_lag_enabled": "false", + "metrics_flush_interval_s": "120", + "metrics_max_buffered_events": "250", + "metrics_write_mode": "\"low_wear\"", + "model_info_enabled": "false", + "model_info_startup_refresh": "false", + "models_refresh_interval_s": "300", + "models_startup_refresh": "true", + "routing_trace_mode": "\"off\"", + "routing_trace_sample_rate": "0.0", + "server_access_log": "false", + "server_threads": "1" + }, + "client_concurrency": 4, + "concurrency_batch_size": 4, + "config_fixture": "sbc-benchmark.toml", + "database_storage": { + "device_class": "mmc", + "filesystem": "ext4", + "storage_medium_class": "non-rotational" + }, + "interpretation": "descriptive loopback characterization; no performance threshold applied", + "principal_count": 5, + "resource_summary": { + "baseline_peak_rss_bytes": 20021248, + "baseline_rss_bytes": 20021248, + "final_active_reservations": 0, + "final_finalization_jobs": null, + "final_pending_requests": 0, + "final_rss_bytes": 40538112, + "peak_rss_bytes": 40538112 + }, + "runs": { + "concurrent_native_responses_finite": { + "batch_elapsed_ms": 16, + "client_concurrency": 4, + "competing_io": { + "bytes_written": 0, + "configured_mib_per_second": 0, + "elapsed_ms": 0, + "enabled": false, + "maximum_mib": 0, + "storage": { + "device_class": "mmc", + "filesystem": "ext4", + "storage_medium_class": "non-rotational" + } + }, + "completed_count": 4, + "cpu": { + "cpu_percent": 126.63, + "cpu_ticks": 2, + "cpu_time_ms": 20 + }, + "failed_count": 0, + "http_status_counts": { + "200": 4 + }, + "request_count": 4, + "requests_per_second": 253.258, + "status": "measured", + "timings": { + "maximum_elapsed_ms": 13, + "minimum_elapsed_ms": 9, + "p50_elapsed_ms": 10, + "p50_ttft_ms": 10, + "p95_elapsed_ms": 13, + "p95_ttft_ms": 13, + "sample_count": 4 + } + }, + "native_responses_finite": { + "cpu": { + "cpu_percent": null, + "cpu_ticks": null, + "cpu_time_ms": null + }, + "maximum_elapsed_ms": 4, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 4, + "p95_elapsed_ms": 4, + "p95_ttft_ms": 4, + "sample_count": 1, + "status": "measured", + "streaming": false, + "warmup_count": 5 + }, + "native_responses_streaming": { + "cpu": { + "cpu_percent": null, + "cpu_ticks": null, + "cpu_time_ms": null + }, + "maximum_elapsed_ms": 4, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 4, + "p95_ttft_ms": 3, + "sample_count": 1, + "status": "measured", + "streaming": true, + "warmup_count": 5 + }, + "translated_responses_to_messages_streaming": { + "cpu": { + "cpu_percent": null, + "cpu_ticks": null, + "cpu_time_ms": null + }, + "expected_upstream_path": "/messages", + "maximum_elapsed_ms": 4, + "minimum_elapsed_ms": 4, + "model": "q008-messages", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 4, + "p95_ttft_ms": 3, + "sample_count": 1, + "status": "measured", + "streaming": true, + "upstream_proof": "fixture Messages path observed", + "upstream_request_delta": 6, + "warmup_count": 5 + } + }, + "sample_count": 1 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "on-device-release-build", + "build_profile": "release", + "repository_commit_sha": "60bdeac5f90deaac6d2429f30f5371e129f3d746", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 4, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 4, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 6, + "id": "check-config", + "reason": "Configuration loaded successfully from /tmp/eggpool-m001-microsd-c4-2026-10-15/eggpool-q008-hdp92ruw/config.toml Server: 127.0.0.1:44349 Accounts: 1 Database: Content digest: e7683e7ad3b02a70c3d3c43c00a519b256198123623e12ada01e6db3318c0b06", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 2566, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 358, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 6, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 10, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 10, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 10, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 6, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 8, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 3214, + "id": "backup", + "reason": "Wrote backup: /tmp/eggpool-m001-microsd-c4-2026-10-15/eggpool-q008-hdp92ruw/backups/eggpool-backup-20261015-040037.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 9877, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2616671)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 7315, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /tmp/eggpool-m001-microsd-c4-2026-10-15/eggpool-q008-hdp92ruw/recovery/config.toml db: /tmp/eggpool-m001-microsd-c4-2026-10-15/eggpool-q008-hdp92ruw/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 6817, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 1336, + "id": "stop", + "reason": "Stopping server (PID 2616825)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 30, + "pending_requests": 0, + "request_statuses": { + "completed": 30 + }, + "requests": 30, + "reservations": 30 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_frequency_policy_end": "1500000", + "cpu_governor": "ondemand", + "cpu_governor_end": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_path_counts": { + "/chat/completions": 3, + "/messages": 8, + "/responses": 19 + }, + "loopback_provider_requests": 30, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 41.9, + "power_thermal_mode_celsius_end": 40.8, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 7, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 6 + }, + { + "body_bytes": 309, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 213, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 145, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 480, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18530304, + 18530304, + 20021248, + 20021248, + 20033536, + 20058112, + 20086784, + 40538112, + 40538112, + 40607744, + 19464192 + ], + "samples": 11, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 3, + 3, + 3, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 7, + 4, + 4 + ], + "finite_ttft_ms": [ + 6, + 4, + 4 + ], + "second_workload_elapsed_ms": [ + 7, + 5 + ], + "second_workload_ttft_ms": [ + 7, + 5 + ], + "streaming_elapsed_ms": [ + 4, + 4, + 4 + ], + "streaming_ttft_ms": [ + 3, + 3, + 3 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18530304, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18530304, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98324480, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18530304, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18530304, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98324480, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "peak_rss_bytes": 20021248, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20021248, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98492416, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "benchmark-baseline", + "open_fd_count": 14, + "peak_rss_bytes": 20021248, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20021248, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98492416, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 12, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-finite", + "open_fd_count": 14, + "peak_rss_bytes": 20033536, + "pending_requests": 0, + "request_statuses": { + "completed": 12 + }, + "requests": 12, + "reservations": 12, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20033536, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98492416, + "wal_bytes": 3201272, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 18, + "cpu_percent": 0.0, + "db_bytes": 856064, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20058112, + "pending_requests": 0, + "request_statuses": { + "completed": 18 + }, + "requests": 18, + "reservations": 18, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20058112, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98627584, + "wal_bytes": 4169472, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 24, + "cpu_percent": 0.0, + "db_bytes": 856064, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-translated-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20086784, + "pending_requests": 0, + "request_statuses": { + "completed": 24 + }, + "requests": 24, + "reservations": 24, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20086784, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98627584, + "wal_bytes": 4169472, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 28, + "cpu_percent": 0.0, + "db_bytes": 856064, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-concurrency-4", + "open_fd_count": 14, + "peak_rss_bytes": 40538112, + "pending_requests": 0, + "request_statuses": { + "completed": 28 + }, + "requests": 28, + "reservations": 28, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 40538112, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 3, + "virtual_bytes": 168206336, + "wal_bytes": 4169472, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 28, + "cpu_percent": 0.0, + "db_bytes": 856064, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-benchmark-stabilization", + "open_fd_count": 14, + "peak_rss_bytes": 40538112, + "pending_requests": 0, + "request_statuses": { + "completed": 28 + }, + "requests": 28, + "reservations": 28, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 40538112, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 3000, + "terminal_references": 0, + "thread_count": 3, + "virtual_bytes": 168206336, + "wal_bytes": 4169472, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 28, + "cpu_percent": 0.0, + "db_bytes": 856064, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "peak_rss_bytes": 40607744, + "pending_requests": 0, + "request_statuses": { + "completed": 28 + }, + "requests": 28, + "reservations": 28, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 40607744, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 3, + "virtual_bytes": 168206336, + "wal_bytes": 4169472, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 30, + "cpu_percent": 0.0, + "db_bytes": 868352, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19464192, + "pending_requests": 0, + "request_statuses": { + "completed": 30 + }, + "requests": 30, + "reservations": 30, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19464192, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98488320, + "wal_bytes": 630392, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v2", + "startup_to_ready_ms": 481, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-concurrency-performance/mmc-run-1.json b/artifacts/qualification/on-prem-concurrency-performance/mmc-run-1.json new file mode 100644 index 000000000..4bc87b210 --- /dev/null +++ b/artifacts/qualification/on-prem-concurrency-performance/mmc-run-1.json @@ -0,0 +1,1002 @@ +{ + "backup": { + "archive_sha256": "f13d9f1e102cb9a4028d8d4ba6f0148539c01f7842759c041c63074215401b78", + "archive_size_bytes": 1236439, + "elapsed_ms": 4004, + "member_count": 3 + }, + "benchmark": { + "cadence": { + "backup_enabled": "false", + "backup_interval_s": "86400", + "database_busy_timeout_ms": "10000", + "database_synchronous": "\"NORMAL\"", + "database_wal": "true", + "database_worker_threads": "1", + "metrics_aggregate_only": "true", + "metrics_cleanup_interval_s": "86400", + "metrics_event_loop_lag_enabled": "false", + "metrics_flush_interval_s": "120", + "metrics_max_buffered_events": "250", + "metrics_write_mode": "\"low_wear\"", + "model_info_enabled": "false", + "model_info_startup_refresh": "false", + "models_refresh_interval_s": "300", + "models_startup_refresh": "true", + "routing_trace_mode": "\"off\"", + "routing_trace_sample_rate": "0.0", + "server_access_log": "false", + "server_threads": "1" + }, + "client_concurrency": 4, + "concurrency_batch_size": 32, + "config_fixture": "sbc-benchmark.toml", + "interpretation": "descriptive loopback characterization; no performance threshold applied", + "resource_summary": { + "baseline_peak_rss_bytes": 20004864, + "baseline_rss_bytes": 20004864, + "final_active_reservations": 0, + "final_finalization_jobs": null, + "final_pending_requests": 0, + "final_rss_bytes": 20844544, + "peak_rss_bytes": 20844544 + }, + "runs": { + "concurrent_native_responses_finite": { + "batch_elapsed_ms": 13220, + "client_concurrency": 4, + "completed_count": 32, + "cpu": { + "cpu_percent": 0.91, + "cpu_ticks": 12, + "cpu_time_ms": 120 + }, + "failed_count": 0, + "request_count": 32, + "requests_per_second": 2.421, + "status": "measured", + "timings": { + "maximum_elapsed_ms": 7599, + "minimum_elapsed_ms": 9, + "p50_elapsed_ms": 10, + "p50_ttft_ms": 10, + "p95_elapsed_ms": 7599, + "p95_ttft_ms": 7599, + "sample_count": 32 + } + }, + "native_responses_finite": { + "cpu": { + "cpu_percent": 0.65, + "cpu_ticks": 10, + "cpu_time_ms": 100 + }, + "maximum_elapsed_ms": 7656, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 4, + "p95_elapsed_ms": 7644, + "p95_ttft_ms": 7644, + "sample_count": 30, + "status": "measured", + "streaming": false, + "warmup_count": 5 + }, + "native_responses_streaming": { + "cpu": { + "cpu_percent": 0.65, + "cpu_ticks": 12, + "cpu_time_ms": 120 + }, + "maximum_elapsed_ms": 11077, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 7125, + "p95_ttft_ms": 18, + "sample_count": 30, + "status": "measured", + "streaming": true, + "warmup_count": 5 + }, + "translated_responses_to_messages_streaming": { + "cpu": { + "cpu_percent": 0.56, + "cpu_ticks": 11, + "cpu_time_ms": 110 + }, + "expected_upstream_path": "/messages", + "maximum_elapsed_ms": 14255, + "minimum_elapsed_ms": 4, + "model": "q008-messages", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 5326, + "p95_ttft_ms": 4, + "sample_count": 30, + "status": "measured", + "streaming": true, + "upstream_proof": "fixture Messages path observed", + "upstream_request_delta": 35, + "warmup_count": 5 + } + }, + "sample_count": 30 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "q005-qualified-aarch64-copy", + "build_profile": "release", + "repository_commit_sha": "d2c0830acaeff90c45bffa393453cf0500c4fa64", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 4, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 4, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 6, + "id": "check-config", + "reason": "Configuration loaded successfully from /tmp/eggpool-q008-0p342vcv/config.toml Server: 127.0.0.1:39771 Accounts: 1 Database: Content digest: c78806b46bb975d49b0c5870d406a28d2702e27dd0c7758254821816c3e1de0d", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 3927, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 256, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 6, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 11, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 10, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 10, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 6, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 7, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 4004, + "id": "backup", + "reason": "Wrote backup: /tmp/eggpool-q008-0p342vcv/backups/eggpool-backup-20261015-015923.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 1849, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2584045)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 10546, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /tmp/eggpool-q008-0p342vcv/recovery/config.toml db: /tmp/eggpool-q008-0p342vcv/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 5306, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 1081, + "id": "stop", + "reason": "Stopping server (PID 2584451)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 145, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_frequency_policy_end": "1600000", + "cpu_governor": "ondemand", + "cpu_governor_end": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_path_counts": { + "/chat/completions": 3, + "/messages": 37, + "/responses": 105 + }, + "loopback_provider_requests": 145, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 41.9, + "power_thermal_mode_celsius_end": 40.8, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 6, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 6 + }, + { + "body_bytes": 309, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 213, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 145, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 480, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18513920, + 18513920, + 20004864, + 20004864, + 20148224, + 20267008, + 20422656, + 20844544, + 20844544, + 20910080, + 19603456 + ], + "samples": 11, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 6, + 4, + 4 + ], + "finite_ttft_ms": [ + 6, + 4, + 4 + ], + "second_workload_elapsed_ms": [ + 7, + 5 + ], + "second_workload_ttft_ms": [ + 6, + 5 + ], + "streaming_elapsed_ms": [ + 4, + 4, + 4 + ], + "streaming_ttft_ms": [ + 3, + 3, + 3 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18513920, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18513920, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98308096, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18513920, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18513920, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98308096, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "peak_rss_bytes": 20004864, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20004864, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "benchmark-baseline", + "open_fd_count": 14, + "peak_rss_bytes": 20004864, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20004864, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 41, + "cpu_percent": 0.0, + "db_bytes": 884736, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-finite", + "open_fd_count": 14, + "peak_rss_bytes": 20148224, + "pending_requests": 0, + "request_statuses": { + "completed": 41 + }, + "requests": 41, + "reservations": 41, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20148224, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98611200, + "wal_bytes": 4165352, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 76, + "cpu_percent": 0.0, + "db_bytes": 958464, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20267008, + "pending_requests": 0, + "request_statuses": { + "completed": 76 + }, + "requests": 76, + "reservations": 76, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20267008, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98611200, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 111, + "cpu_percent": 0.0, + "db_bytes": 1105920, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-translated-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20422656, + "pending_requests": 0, + "request_statuses": { + "completed": 111 + }, + "requests": 111, + "reservations": 111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20422656, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98611200, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-concurrency-4", + "open_fd_count": 14, + "peak_rss_bytes": 20844544, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20844544, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98885632, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-benchmark-stabilization", + "open_fd_count": 14, + "peak_rss_bytes": 20844544, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20844544, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 3000, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98885632, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "peak_rss_bytes": 20910080, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20910080, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98885632, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 145, + "cpu_percent": 0.0, + "db_bytes": 1216512, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19603456, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19603456, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98476032, + "wal_bytes": 638632, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v2", + "startup_to_ready_ms": 448, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-concurrency-performance/mmc-run-2.json b/artifacts/qualification/on-prem-concurrency-performance/mmc-run-2.json new file mode 100644 index 000000000..747e58af8 --- /dev/null +++ b/artifacts/qualification/on-prem-concurrency-performance/mmc-run-2.json @@ -0,0 +1,1002 @@ +{ + "backup": { + "archive_sha256": "a9009e0f65fc381747f9f8fa4a324bc973caada131d5272e4e35222999cd36e9", + "archive_size_bytes": 1236439, + "elapsed_ms": 4093, + "member_count": 3 + }, + "benchmark": { + "cadence": { + "backup_enabled": "false", + "backup_interval_s": "86400", + "database_busy_timeout_ms": "10000", + "database_synchronous": "\"NORMAL\"", + "database_wal": "true", + "database_worker_threads": "1", + "metrics_aggregate_only": "true", + "metrics_cleanup_interval_s": "86400", + "metrics_event_loop_lag_enabled": "false", + "metrics_flush_interval_s": "120", + "metrics_max_buffered_events": "250", + "metrics_write_mode": "\"low_wear\"", + "model_info_enabled": "false", + "model_info_startup_refresh": "false", + "models_refresh_interval_s": "300", + "models_startup_refresh": "true", + "routing_trace_mode": "\"off\"", + "routing_trace_sample_rate": "0.0", + "server_access_log": "false", + "server_threads": "1" + }, + "client_concurrency": 4, + "concurrency_batch_size": 32, + "config_fixture": "sbc-benchmark.toml", + "interpretation": "descriptive loopback characterization; no performance threshold applied", + "resource_summary": { + "baseline_peak_rss_bytes": 20004864, + "baseline_rss_bytes": 20004864, + "final_active_reservations": 0, + "final_finalization_jobs": null, + "final_pending_requests": 0, + "final_rss_bytes": 20852736, + "peak_rss_bytes": 20852736 + }, + "runs": { + "concurrent_native_responses_finite": { + "batch_elapsed_ms": 18524, + "client_concurrency": 4, + "completed_count": 32, + "cpu": { + "cpu_percent": 0.65, + "cpu_ticks": 12, + "cpu_time_ms": 120 + }, + "failed_count": 0, + "request_count": 32, + "requests_per_second": 1.728, + "status": "measured", + "timings": { + "maximum_elapsed_ms": 12058, + "minimum_elapsed_ms": 10, + "p50_elapsed_ms": 11, + "p50_ttft_ms": 11, + "p95_elapsed_ms": 12058, + "p95_ttft_ms": 12058, + "sample_count": 32 + } + }, + "native_responses_finite": { + "cpu": { + "cpu_percent": 0.63, + "cpu_ticks": 10, + "cpu_time_ms": 100 + }, + "maximum_elapsed_ms": 8942, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 4, + "p95_elapsed_ms": 6780, + "p95_ttft_ms": 6780, + "sample_count": 30, + "status": "measured", + "streaming": false, + "warmup_count": 5 + }, + "native_responses_streaming": { + "cpu": { + "cpu_percent": 0.8, + "cpu_ticks": 12, + "cpu_time_ms": 120 + }, + "maximum_elapsed_ms": 8780, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 6159, + "p95_ttft_ms": 31, + "sample_count": 30, + "status": "measured", + "streaming": true, + "warmup_count": 5 + }, + "translated_responses_to_messages_streaming": { + "cpu": { + "cpu_percent": 0.42, + "cpu_ticks": 11, + "cpu_time_ms": 110 + }, + "expected_upstream_path": "/messages", + "maximum_elapsed_ms": 13802, + "minimum_elapsed_ms": 4, + "model": "q008-messages", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 12162, + "p95_ttft_ms": 4, + "sample_count": 30, + "status": "measured", + "streaming": true, + "upstream_proof": "fixture Messages path observed", + "upstream_request_delta": 35, + "warmup_count": 5 + } + }, + "sample_count": 30 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "q005-qualified-aarch64-copy", + "build_profile": "release", + "repository_commit_sha": "d2c0830acaeff90c45bffa393453cf0500c4fa64", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 5, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 4, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 5, + "id": "check-config", + "reason": "Configuration loaded successfully from /tmp/eggpool-q008-pzcehr5k/config.toml Server: 127.0.0.1:33719 Accounts: 1 Database: Content digest: 34b2046194fc3578aa036e5d52bec46832f94f96c3e8f53747c008520145d31b", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 632, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 409, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 6, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 10, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 11, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 10, + "id": "transcoding-stats", + "reason": "{ \"total\": 6, \"native_count\": 6, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 6, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 7, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 4093, + "id": "backup", + "reason": "Wrote backup: /tmp/eggpool-q008-pzcehr5k/backups/eggpool-backup-20261015-020153.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 1390, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2584565)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 7345, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /tmp/eggpool-q008-pzcehr5k/recovery/config.toml db: /tmp/eggpool-q008-pzcehr5k/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 4008, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 519, + "id": "stop", + "reason": "Stopping server (PID 2584967)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 145, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_frequency_policy_end": "1600000", + "cpu_governor": "ondemand", + "cpu_governor_end": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_path_counts": { + "/chat/completions": 3, + "/messages": 37, + "/responses": 105 + }, + "loopback_provider_requests": 145, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 42.5, + "power_thermal_mode_celsius_end": 41.4, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 6, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 6 + }, + { + "body_bytes": 309, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 213, + "elapsed_ms": 5, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 5 + }, + { + "body_bytes": 145, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 5, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 4 + }, + { + "body_bytes": 480, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18509824, + 18513920, + 20004864, + 20004864, + 20156416, + 20267008, + 20430848, + 20852736, + 20852736, + 20918272, + 19599360 + ], + "samples": 11, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 6, + 4, + 5 + ], + "finite_ttft_ms": [ + 6, + 4, + 5 + ], + "second_workload_elapsed_ms": [ + 7, + 5 + ], + "second_workload_ttft_ms": [ + 7, + 5 + ], + "streaming_elapsed_ms": [ + 4, + 5, + 4 + ], + "streaming_ttft_ms": [ + 3, + 4, + 3 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18509824, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18509824, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98308096, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18513920, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18513920, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98308096, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "peak_rss_bytes": 20004864, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20004864, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "benchmark-baseline", + "open_fd_count": 14, + "peak_rss_bytes": 20004864, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20004864, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 41, + "cpu_percent": 0.0, + "db_bytes": 884736, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-finite", + "open_fd_count": 14, + "peak_rss_bytes": 20156416, + "pending_requests": 0, + "request_statuses": { + "completed": 41 + }, + "requests": 41, + "reservations": 41, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20156416, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98611200, + "wal_bytes": 4165352, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 76, + "cpu_percent": 0.0, + "db_bytes": 958464, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20267008, + "pending_requests": 0, + "request_statuses": { + "completed": 76 + }, + "requests": 76, + "reservations": 76, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20267008, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98611200, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 111, + "cpu_percent": 0.0, + "db_bytes": 1105920, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-translated-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20430848, + "pending_requests": 0, + "request_statuses": { + "completed": 111 + }, + "requests": 111, + "reservations": 111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20430848, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98611200, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-concurrency-4", + "open_fd_count": 14, + "peak_rss_bytes": 20852736, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20852736, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-benchmark-stabilization", + "open_fd_count": 14, + "peak_rss_bytes": 20852736, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20852736, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 3000, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "peak_rss_bytes": 20918272, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20918272, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 145, + "cpu_percent": 0.0, + "db_bytes": 1216512, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19599360, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19599360, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98480128, + "wal_bytes": 638632, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v2", + "startup_to_ready_ms": 161, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-concurrency-performance/mmc-run-3.json b/artifacts/qualification/on-prem-concurrency-performance/mmc-run-3.json new file mode 100644 index 000000000..34027434c --- /dev/null +++ b/artifacts/qualification/on-prem-concurrency-performance/mmc-run-3.json @@ -0,0 +1,1002 @@ +{ + "backup": { + "archive_sha256": "fb1a1a41c98ee6069f8f17510828a049a05f41644ae9a9167fc2863bfe34aae9", + "archive_size_bytes": 1236439, + "elapsed_ms": 697, + "member_count": 3 + }, + "benchmark": { + "cadence": { + "backup_enabled": "false", + "backup_interval_s": "86400", + "database_busy_timeout_ms": "10000", + "database_synchronous": "\"NORMAL\"", + "database_wal": "true", + "database_worker_threads": "1", + "metrics_aggregate_only": "true", + "metrics_cleanup_interval_s": "86400", + "metrics_event_loop_lag_enabled": "false", + "metrics_flush_interval_s": "120", + "metrics_max_buffered_events": "250", + "metrics_write_mode": "\"low_wear\"", + "model_info_enabled": "false", + "model_info_startup_refresh": "false", + "models_refresh_interval_s": "300", + "models_startup_refresh": "true", + "routing_trace_mode": "\"off\"", + "routing_trace_sample_rate": "0.0", + "server_access_log": "false", + "server_threads": "1" + }, + "client_concurrency": 4, + "concurrency_batch_size": 32, + "config_fixture": "sbc-benchmark.toml", + "interpretation": "descriptive loopback characterization; no performance threshold applied", + "resource_summary": { + "baseline_peak_rss_bytes": 20000768, + "baseline_rss_bytes": 20000768, + "final_active_reservations": 0, + "final_finalization_jobs": null, + "final_pending_requests": 0, + "final_rss_bytes": 20824064, + "peak_rss_bytes": 20824064 + }, + "runs": { + "concurrent_native_responses_finite": { + "batch_elapsed_ms": 15132, + "client_concurrency": 4, + "completed_count": 32, + "cpu": { + "cpu_percent": 0.93, + "cpu_ticks": 14, + "cpu_time_ms": 140 + }, + "failed_count": 0, + "request_count": 32, + "requests_per_second": 2.115, + "status": "measured", + "timings": { + "maximum_elapsed_ms": 8579, + "minimum_elapsed_ms": 10, + "p50_elapsed_ms": 11, + "p50_ttft_ms": 11, + "p95_elapsed_ms": 8579, + "p95_ttft_ms": 8579, + "sample_count": 32 + } + }, + "native_responses_finite": { + "cpu": { + "cpu_percent": 0.73, + "cpu_ticks": 11, + "cpu_time_ms": 110 + }, + "maximum_elapsed_ms": 8381, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 4, + "p95_elapsed_ms": 6580, + "p95_ttft_ms": 6580, + "sample_count": 30, + "status": "measured", + "streaming": false, + "warmup_count": 5 + }, + "native_responses_streaming": { + "cpu": { + "cpu_percent": 0.55, + "cpu_ticks": 11, + "cpu_time_ms": 110 + }, + "maximum_elapsed_ms": 10043, + "minimum_elapsed_ms": 3, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 9833, + "p95_ttft_ms": 20, + "sample_count": 30, + "status": "measured", + "streaming": true, + "warmup_count": 5 + }, + "translated_responses_to_messages_streaming": { + "cpu": { + "cpu_percent": 0.73, + "cpu_ticks": 11, + "cpu_time_ms": 110 + }, + "expected_upstream_path": "/messages", + "maximum_elapsed_ms": 10440, + "minimum_elapsed_ms": 4, + "model": "q008-messages", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 4506, + "p95_ttft_ms": 4, + "sample_count": 30, + "status": "measured", + "streaming": true, + "upstream_proof": "fixture Messages path observed", + "upstream_request_delta": 35, + "warmup_count": 5 + } + }, + "sample_count": 30 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "q005-qualified-aarch64-copy", + "build_profile": "release", + "repository_commit_sha": "d2c0830acaeff90c45bffa393453cf0500c4fa64", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 4, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 4, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 6, + "id": "check-config", + "reason": "Configuration loaded successfully from /tmp/eggpool-q008-ucalh83x/config.toml Server: 127.0.0.1:34513 Accounts: 1 Database: Content digest: d8b73c67c704ed2366c313b11ebe56bef52acdb6cd419cdd1fcfa29e6ed08180", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 896, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 629, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 6, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 10, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 10, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 11, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 6, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 7, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 697, + "id": "backup", + "reason": "Wrote backup: /tmp/eggpool-q008-ucalh83x/backups/eggpool-backup-20261015-020400.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 1849, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2585282)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 9447, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /tmp/eggpool-q008-ucalh83x/recovery/config.toml db: /tmp/eggpool-q008-ucalh83x/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 12629, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 1232, + "id": "stop", + "reason": "Stopping server (PID 2585708)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 145, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_frequency_policy_end": "1600000", + "cpu_governor": "ondemand", + "cpu_governor_end": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_path_counts": { + "/chat/completions": 3, + "/messages": 37, + "/responses": 105 + }, + "loopback_provider_requests": 145, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 42.5, + "power_thermal_mode_celsius_end": 40.8, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 6, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 6 + }, + { + "body_bytes": 309, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 213, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 145, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 480, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18513920, + 18513920, + 20000768, + 20000768, + 20144128, + 20275200, + 20430848, + 20824064, + 20824064, + 20889600, + 19611648 + ], + "samples": 11, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 6, + 4, + 4 + ], + "finite_ttft_ms": [ + 6, + 4, + 4 + ], + "second_workload_elapsed_ms": [ + 7, + 5 + ], + "second_workload_ttft_ms": [ + 7, + 5 + ], + "streaming_elapsed_ms": [ + 4, + 4, + 4 + ], + "streaming_ttft_ms": [ + 3, + 3, + 3 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18513920, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18513920, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98308096, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18513920, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18513920, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98308096, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "peak_rss_bytes": 20000768, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20000768, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "benchmark-baseline", + "open_fd_count": 14, + "peak_rss_bytes": 20000768, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20000768, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 41, + "cpu_percent": 0.0, + "db_bytes": 884736, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-finite", + "open_fd_count": 14, + "peak_rss_bytes": 20144128, + "pending_requests": 0, + "request_statuses": { + "completed": 41 + }, + "requests": 41, + "reservations": 41, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20144128, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4165352, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 76, + "cpu_percent": 0.0, + "db_bytes": 958464, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20275200, + "pending_requests": 0, + "request_statuses": { + "completed": 76 + }, + "requests": 76, + "reservations": 76, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20275200, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 111, + "cpu_percent": 0.0, + "db_bytes": 1105920, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-translated-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20430848, + "pending_requests": 0, + "request_statuses": { + "completed": 111 + }, + "requests": 111, + "reservations": 111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20430848, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-concurrency-4", + "open_fd_count": 14, + "peak_rss_bytes": 20824064, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20824064, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-benchmark-stabilization", + "open_fd_count": 14, + "peak_rss_bytes": 20824064, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20824064, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 3000, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "peak_rss_bytes": 20889600, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20889600, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 145, + "cpu_percent": 0.0, + "db_bytes": 1216512, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19611648, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19611648, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98476032, + "wal_bytes": 638632, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v2", + "startup_to_ready_ms": 267, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-concurrency-performance/ssd-c4-u5-user-global-run-1.json b/artifacts/qualification/on-prem-concurrency-performance/ssd-c4-u5-user-global-run-1.json new file mode 100644 index 000000000..998f4aa0d --- /dev/null +++ b/artifacts/qualification/on-prem-concurrency-performance/ssd-c4-u5-user-global-run-1.json @@ -0,0 +1,1026 @@ +{ + "backup": { + "archive_sha256": "eb71af1b6b74bfd94058383b80b952e303282e22158bc92bd2b197815df769a4", + "archive_size_bytes": 4636287, + "elapsed_ms": 456, + "member_count": 3 + }, + "benchmark": { + "budget_mode": "user-global", + "cadence": { + "backup_enabled": "false", + "backup_interval_s": "86400", + "database_busy_timeout_ms": "10000", + "database_synchronous": "\"NORMAL\"", + "database_wal": "true", + "database_worker_threads": "1", + "metrics_aggregate_only": "true", + "metrics_cleanup_interval_s": "86400", + "metrics_event_loop_lag_enabled": "false", + "metrics_flush_interval_s": "120", + "metrics_max_buffered_events": "250", + "metrics_write_mode": "\"low_wear\"", + "model_info_enabled": "false", + "model_info_startup_refresh": "false", + "models_refresh_interval_s": "300", + "models_startup_refresh": "true", + "routing_trace_mode": "\"off\"", + "routing_trace_sample_rate": "0.0", + "server_access_log": "false", + "server_threads": "1" + }, + "client_concurrency": 4, + "concurrency_batch_size": 1000, + "config_fixture": "sbc-benchmark.toml", + "database_storage": { + "device_class": "nvme", + "filesystem": "ext4", + "storage_medium_class": "non-rotational" + }, + "interpretation": "descriptive loopback characterization; no performance threshold applied", + "principal_count": 5, + "resource_summary": { + "baseline_peak_rss_bytes": 20041728, + "baseline_rss_bytes": 20041728, + "final_active_reservations": 0, + "final_finalization_jobs": null, + "final_pending_requests": 0, + "final_rss_bytes": 42721280, + "peak_rss_bytes": 42721280 + }, + "runs": { + "concurrent_native_responses_finite": { + "batch_elapsed_ms": 4444, + "client_concurrency": 4, + "competing_io": { + "bytes_written": 0, + "configured_mib_per_second": 0, + "elapsed_ms": 0, + "enabled": false, + "maximum_mib": 0, + "storage": { + "device_class": "nvme", + "filesystem": "ext4", + "storage_medium_class": "non-rotational" + } + }, + "completed_count": 1000, + "cpu": { + "cpu_percent": 97.67, + "cpu_ticks": 434, + "cpu_time_ms": 4340 + }, + "failed_count": 0, + "http_status_counts": { + "200": 1000 + }, + "request_count": 1000, + "requests_per_second": 225.038, + "status": "measured", + "timings": { + "maximum_elapsed_ms": 33, + "minimum_elapsed_ms": 9, + "p50_elapsed_ms": 13, + "p50_ttft_ms": 13, + "p95_elapsed_ms": 30, + "p95_ttft_ms": 30, + "p99_elapsed_ms": 32, + "p99_ttft_ms": 32, + "sample_count": 1000 + } + }, + "native_responses_finite": { + "cpu": { + "cpu_percent": null, + "cpu_ticks": null, + "cpu_time_ms": null + }, + "maximum_elapsed_ms": 43, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 4, + "p95_elapsed_ms": 18, + "p95_ttft_ms": 18, + "sample_count": 30, + "status": "measured", + "streaming": false, + "warmup_count": 5 + }, + "native_responses_streaming": { + "cpu": { + "cpu_percent": null, + "cpu_ticks": null, + "cpu_time_ms": null + }, + "maximum_elapsed_ms": 21, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 20, + "p95_ttft_ms": 4, + "sample_count": 30, + "status": "measured", + "streaming": true, + "warmup_count": 5 + }, + "translated_responses_to_messages_streaming": { + "cpu": { + "cpu_percent": null, + "cpu_ticks": null, + "cpu_time_ms": null + }, + "expected_upstream_path": "/messages", + "maximum_elapsed_ms": 20, + "minimum_elapsed_ms": 4, + "model": "q008-messages", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 19, + "p95_ttft_ms": 4, + "sample_count": 30, + "status": "measured", + "streaming": true, + "upstream_proof": "fixture Messages path observed", + "upstream_request_delta": 35, + "warmup_count": 5 + } + }, + "sample_count": 30 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "on-device-release-build", + "build_profile": "release", + "repository_commit_sha": "b4d15635c18615684e977c8c6dc3f396e705c6f2", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 4, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 4, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 6, + "id": "check-config", + "reason": "Configuration loaded successfully from /mnt/qualification-ssd/eggpool-m001-2026-10-14/eggpool-q008-74f3bg5w/config.toml Server: 127.0.0.1:46997 Accounts: 1 Database: Content digest: c3cea7dee0c1823cca58b6bd648fb41af82246bba3070c28c83234216978bc90", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 148, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 93, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 6, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 10, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 10, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 10, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 6, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 8, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 456, + "id": "backup", + "reason": "Wrote backup: /mnt/qualification-ssd/eggpool-m001-2026-10-14/eggpool-q008-74f3bg5w/backups/eggpool-backup-20261015-031002.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 58, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2607699)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 183, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /mnt/qualification-ssd/eggpool-m001-2026-10-14/eggpool-q008-74f3bg5w/recovery/config.toml db: /mnt/qualification-ssd/eggpool-m001-2026-10-14/eggpool-q008-74f3bg5w/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 101, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 58, + "id": "stop", + "reason": "Stopping server (PID 2608880)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 1113, + "pending_requests": 0, + "request_statuses": { + "completed": 1113 + }, + "requests": 1113, + "reservations": 1113 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_frequency_policy_end": "1600000", + "cpu_governor": "ondemand", + "cpu_governor_end": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_path_counts": { + "/chat/completions": 3, + "/messages": 37, + "/responses": 1073 + }, + "loopback_provider_requests": 1113, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 43.5, + "power_thermal_mode_celsius_end": 41.4, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 6, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 6 + }, + { + "body_bytes": 309, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 213, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 145, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 5, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 480, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18546688, + 18546688, + 20041728, + 20041728, + 20152320, + 20279296, + 20447232, + 42721280, + 42721280, + 42786816, + 20307968 + ], + "samples": 11, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 3, + 3, + 3, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 6, + 4, + 4 + ], + "finite_ttft_ms": [ + 6, + 4, + 4 + ], + "second_workload_elapsed_ms": [ + 9, + 6 + ], + "second_workload_ttft_ms": [ + 9, + 6 + ], + "streaming_elapsed_ms": [ + 4, + 5, + 4 + ], + "streaming_ttft_ms": [ + 3, + 3, + 3 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18546688, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18546688, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98324480, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18546688, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18546688, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98324480, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "peak_rss_bytes": 20041728, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20041728, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98496512, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "benchmark-baseline", + "open_fd_count": 14, + "peak_rss_bytes": 20041728, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20041728, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98496512, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 41, + "cpu_percent": 0.0, + "db_bytes": 884736, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-finite", + "open_fd_count": 14, + "peak_rss_bytes": 20152320, + "pending_requests": 0, + "request_statuses": { + "completed": 41 + }, + "requests": 41, + "reservations": 41, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20152320, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98631680, + "wal_bytes": 4144752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 76, + "cpu_percent": 0.0, + "db_bytes": 958464, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20279296, + "pending_requests": 0, + "request_statuses": { + "completed": 76 + }, + "requests": 76, + "reservations": 76, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20279296, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98631680, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 111, + "cpu_percent": 0.0, + "db_bytes": 1105920, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-translated-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20447232, + "pending_requests": 0, + "request_statuses": { + "completed": 111 + }, + "requests": 111, + "reservations": 111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20447232, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98631680, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 1111, + "cpu_percent": 0.0, + "db_bytes": 4632576, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-concurrency-4", + "open_fd_count": 14, + "peak_rss_bytes": 42721280, + "pending_requests": 0, + "request_statuses": { + "completed": 1111 + }, + "requests": 1111, + "reservations": 1111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 42721280, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 3, + "virtual_bytes": 168484864, + "wal_bytes": 4387832, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 1111, + "cpu_percent": 0.0, + "db_bytes": 4632576, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-benchmark-stabilization", + "open_fd_count": 14, + "peak_rss_bytes": 42721280, + "pending_requests": 0, + "request_statuses": { + "completed": 1111 + }, + "requests": 1111, + "reservations": 1111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 42721280, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 3000, + "terminal_references": 0, + "thread_count": 3, + "virtual_bytes": 168484864, + "wal_bytes": 4387832, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 1111, + "cpu_percent": 0.0, + "db_bytes": 4632576, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "peak_rss_bytes": 42786816, + "pending_requests": 0, + "request_statuses": { + "completed": 1111 + }, + "requests": 1111, + "reservations": 1111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 42786816, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 3, + "virtual_bytes": 168484864, + "wal_bytes": 4387832, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 1113, + "cpu_percent": 0.0, + "db_bytes": 4464640, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "peak_rss_bytes": 20307968, + "pending_requests": 0, + "request_statuses": { + "completed": 1113 + }, + "requests": 1113, + "reservations": 1113, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20307968, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98492416, + "wal_bytes": 894072, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v2", + "startup_to_ready_ms": 40, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-1.json b/artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-1.json new file mode 100644 index 000000000..d5597d631 --- /dev/null +++ b/artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-1.json @@ -0,0 +1,1002 @@ +{ + "backup": { + "archive_sha256": "7046751383e13d7e197a7cc689458315660644bd1402a20b7ff068cbb9ab1954", + "archive_size_bytes": 1236455, + "elapsed_ms": 122, + "member_count": 3 + }, + "benchmark": { + "cadence": { + "backup_enabled": "false", + "backup_interval_s": "86400", + "database_busy_timeout_ms": "10000", + "database_synchronous": "\"NORMAL\"", + "database_wal": "true", + "database_worker_threads": "1", + "metrics_aggregate_only": "true", + "metrics_cleanup_interval_s": "86400", + "metrics_event_loop_lag_enabled": "false", + "metrics_flush_interval_s": "120", + "metrics_max_buffered_events": "250", + "metrics_write_mode": "\"low_wear\"", + "model_info_enabled": "false", + "model_info_startup_refresh": "false", + "models_refresh_interval_s": "300", + "models_startup_refresh": "true", + "routing_trace_mode": "\"off\"", + "routing_trace_sample_rate": "0.0", + "server_access_log": "false", + "server_threads": "1" + }, + "client_concurrency": 4, + "concurrency_batch_size": 32, + "config_fixture": "sbc-benchmark.toml", + "interpretation": "descriptive loopback characterization; no performance threshold applied", + "resource_summary": { + "baseline_peak_rss_bytes": 20008960, + "baseline_rss_bytes": 20008960, + "final_active_reservations": 0, + "final_finalization_jobs": null, + "final_pending_requests": 0, + "final_rss_bytes": 20840448, + "peak_rss_bytes": 20840448 + }, + "runs": { + "concurrent_native_responses_finite": { + "batch_elapsed_ms": 80, + "client_concurrency": 4, + "completed_count": 32, + "cpu": { + "cpu_percent": 124.34, + "cpu_ticks": 10, + "cpu_time_ms": 100 + }, + "failed_count": 0, + "request_count": 32, + "requests_per_second": 397.884, + "status": "measured", + "timings": { + "maximum_elapsed_ms": 11, + "minimum_elapsed_ms": 7, + "p50_elapsed_ms": 10, + "p50_ttft_ms": 9, + "p95_elapsed_ms": 10, + "p95_ttft_ms": 10, + "sample_count": 32 + } + }, + "native_responses_finite": { + "cpu": { + "cpu_percent": 77.67, + "cpu_ticks": 9, + "cpu_time_ms": 90 + }, + "maximum_elapsed_ms": 4, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 4, + "p95_elapsed_ms": 4, + "p95_ttft_ms": 4, + "sample_count": 30, + "status": "measured", + "streaming": false, + "warmup_count": 5 + }, + "native_responses_streaming": { + "cpu": { + "cpu_percent": 77.85, + "cpu_ticks": 9, + "cpu_time_ms": 90 + }, + "maximum_elapsed_ms": 4, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 4, + "p95_ttft_ms": 3, + "sample_count": 30, + "status": "measured", + "streaming": true, + "warmup_count": 5 + }, + "translated_responses_to_messages_streaming": { + "cpu": { + "cpu_percent": 82.12, + "cpu_ticks": 10, + "cpu_time_ms": 100 + }, + "expected_upstream_path": "/messages", + "maximum_elapsed_ms": 5, + "minimum_elapsed_ms": 3, + "model": "q008-messages", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 5, + "p95_ttft_ms": 4, + "sample_count": 30, + "status": "measured", + "streaming": true, + "upstream_proof": "fixture Messages path observed", + "upstream_request_delta": 35, + "warmup_count": 5 + } + }, + "sample_count": 30 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "q005-qualified-aarch64-copy", + "build_profile": "release", + "repository_commit_sha": "d2c0830acaeff90c45bffa393453cf0500c4fa64", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 4, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 5, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 6, + "id": "check-config", + "reason": "Configuration loaded successfully from /dev/shm/eggpool-q008-75m9pmu9/config.toml Server: 127.0.0.1:55371 Accounts: 1 Database: Content digest: 9f9288f0a85427211e80df437e1156021cf25e1d4572f5c28ea5a0f92144b91c", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 104, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 79, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 6, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 10, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 10, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 10, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 6, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 7, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 122, + "id": "backup", + "reason": "Wrote backup: /dev/shm/eggpool-q008-75m9pmu9/backups/eggpool-backup-20261015-020231.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 59, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2585051)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 40, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /dev/shm/eggpool-q008-75m9pmu9/recovery/config.toml db: /dev/shm/eggpool-q008-75m9pmu9/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 31, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 58, + "id": "stop", + "reason": "Stopping server (PID 2585233)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 145, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_frequency_policy_end": "1500000", + "cpu_governor": "ondemand", + "cpu_governor_end": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_path_counts": { + "/chat/completions": 3, + "/messages": 37, + "/responses": 105 + }, + "loopback_provider_requests": 145, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 41.9, + "power_thermal_mode_celsius_end": 40.8, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 6, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 6 + }, + { + "body_bytes": 309, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 213, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 145, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 480, + "elapsed_ms": 5, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 4 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18518016, + 18518016, + 20008960, + 20008960, + 20127744, + 20254720, + 20418560, + 20840448, + 20840448, + 20905984, + 19615744 + ], + "samples": 11, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 6, + 4, + 4 + ], + "finite_ttft_ms": [ + 6, + 4, + 4 + ], + "second_workload_elapsed_ms": [ + 7, + 5 + ], + "second_workload_ttft_ms": [ + 7, + 5 + ], + "streaming_elapsed_ms": [ + 4, + 4, + 5 + ], + "streaming_ttft_ms": [ + 3, + 3, + 4 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18518016, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18518016, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98312192, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18518016, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18518016, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98312192, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "peak_rss_bytes": 20008960, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20008960, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98480128, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "benchmark-baseline", + "open_fd_count": 14, + "peak_rss_bytes": 20008960, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20008960, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98480128, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 41, + "cpu_percent": 0.0, + "db_bytes": 884736, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-finite", + "open_fd_count": 14, + "peak_rss_bytes": 20127744, + "pending_requests": 0, + "request_statuses": { + "completed": 41 + }, + "requests": 41, + "reservations": 41, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20127744, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98615296, + "wal_bytes": 4144752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 76, + "cpu_percent": 0.0, + "db_bytes": 958464, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20254720, + "pending_requests": 0, + "request_statuses": { + "completed": 76 + }, + "requests": 76, + "reservations": 76, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20254720, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98615296, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 111, + "cpu_percent": 0.0, + "db_bytes": 1105920, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-translated-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20418560, + "pending_requests": 0, + "request_statuses": { + "completed": 111 + }, + "requests": 111, + "reservations": 111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20418560, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98615296, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-concurrency-4", + "open_fd_count": 14, + "peak_rss_bytes": 20840448, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20840448, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98889728, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.33, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-benchmark-stabilization", + "open_fd_count": 14, + "peak_rss_bytes": 20840448, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20840448, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 3000, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98889728, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "peak_rss_bytes": 20905984, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20905984, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98889728, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 145, + "cpu_percent": 0.0, + "db_bytes": 1216512, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19615744, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19615744, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98476032, + "wal_bytes": 638632, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v2", + "startup_to_ready_ms": 41, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-2.json b/artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-2.json new file mode 100644 index 000000000..a3a53544d --- /dev/null +++ b/artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-2.json @@ -0,0 +1,1002 @@ +{ + "backup": { + "archive_sha256": "8a461555e6c4304fff291be3381ab173496d0a0474f06c3fe4e7e1ff01748961", + "archive_size_bytes": 1236455, + "elapsed_ms": 121, + "member_count": 3 + }, + "benchmark": { + "cadence": { + "backup_enabled": "false", + "backup_interval_s": "86400", + "database_busy_timeout_ms": "10000", + "database_synchronous": "\"NORMAL\"", + "database_wal": "true", + "database_worker_threads": "1", + "metrics_aggregate_only": "true", + "metrics_cleanup_interval_s": "86400", + "metrics_event_loop_lag_enabled": "false", + "metrics_flush_interval_s": "120", + "metrics_max_buffered_events": "250", + "metrics_write_mode": "\"low_wear\"", + "model_info_enabled": "false", + "model_info_startup_refresh": "false", + "models_refresh_interval_s": "300", + "models_startup_refresh": "true", + "routing_trace_mode": "\"off\"", + "routing_trace_sample_rate": "0.0", + "server_access_log": "false", + "server_threads": "1" + }, + "client_concurrency": 4, + "concurrency_batch_size": 32, + "config_fixture": "sbc-benchmark.toml", + "interpretation": "descriptive loopback characterization; no performance threshold applied", + "resource_summary": { + "baseline_peak_rss_bytes": 20054016, + "baseline_rss_bytes": 20004864, + "final_active_reservations": 0, + "final_finalization_jobs": null, + "final_pending_requests": 0, + "final_rss_bytes": 20828160, + "peak_rss_bytes": 20828160 + }, + "runs": { + "concurrent_native_responses_finite": { + "batch_elapsed_ms": 81, + "client_concurrency": 4, + "completed_count": 32, + "cpu": { + "cpu_percent": 135.94, + "cpu_ticks": 11, + "cpu_time_ms": 110 + }, + "failed_count": 0, + "request_count": 32, + "requests_per_second": 395.454, + "status": "measured", + "timings": { + "maximum_elapsed_ms": 10, + "minimum_elapsed_ms": 7, + "p50_elapsed_ms": 10, + "p50_ttft_ms": 9, + "p95_elapsed_ms": 10, + "p95_ttft_ms": 10, + "sample_count": 32 + } + }, + "native_responses_finite": { + "cpu": { + "cpu_percent": 78.25, + "cpu_ticks": 9, + "cpu_time_ms": 90 + }, + "maximum_elapsed_ms": 4, + "minimum_elapsed_ms": 3, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 4, + "p95_elapsed_ms": 4, + "p95_ttft_ms": 4, + "sample_count": 30, + "status": "measured", + "streaming": false, + "warmup_count": 5 + }, + "native_responses_streaming": { + "cpu": { + "cpu_percent": 87.18, + "cpu_ticks": 10, + "cpu_time_ms": 100 + }, + "maximum_elapsed_ms": 4, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 4, + "p95_ttft_ms": 3, + "sample_count": 30, + "status": "measured", + "streaming": true, + "warmup_count": 5 + }, + "translated_responses_to_messages_streaming": { + "cpu": { + "cpu_percent": 74.8, + "cpu_ticks": 9, + "cpu_time_ms": 90 + }, + "expected_upstream_path": "/messages", + "maximum_elapsed_ms": 4, + "minimum_elapsed_ms": 4, + "model": "q008-messages", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 4, + "p95_ttft_ms": 3, + "sample_count": 30, + "status": "measured", + "streaming": true, + "upstream_proof": "fixture Messages path observed", + "upstream_request_delta": 35, + "warmup_count": 5 + } + }, + "sample_count": 30 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "q005-qualified-aarch64-copy", + "build_profile": "release", + "repository_commit_sha": "d2c0830acaeff90c45bffa393453cf0500c4fa64", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 5, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 5, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 6, + "id": "check-config", + "reason": "Configuration loaded successfully from /dev/shm/eggpool-q008-l4a37d6q/config.toml Server: 127.0.0.1:42519 Accounts: 1 Database: Content digest: 8aa8143fbb01421c8b2b0c23143b779aef4393467c5355736cdb774f9121fc93", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 104, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 81, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 6, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 11, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 10, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 10, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 6, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 7, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 121, + "id": "backup", + "reason": "Wrote backup: /dev/shm/eggpool-q008-l4a37d6q/backups/eggpool-backup-20261015-020439.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 58, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2585748)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 40, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /dev/shm/eggpool-q008-l4a37d6q/recovery/config.toml db: /dev/shm/eggpool-q008-l4a37d6q/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 31, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 59, + "id": "stop", + "reason": "Stopping server (PID 2585929)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 145, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_frequency_policy_end": "1500000", + "cpu_governor": "ondemand", + "cpu_governor_end": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_path_counts": { + "/chat/completions": 3, + "/messages": 37, + "/responses": 105 + }, + "loopback_provider_requests": 145, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 42.5, + "power_thermal_mode_celsius_end": 40.8, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 6, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 6 + }, + { + "body_bytes": 309, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 213, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 145, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 480, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18513920, + 18513920, + 20004864, + 20004864, + 20119552, + 20242432, + 20418560, + 20828160, + 20828160, + 20897792, + 19611648 + ], + "samples": 11, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 6, + 4, + 4 + ], + "finite_ttft_ms": [ + 6, + 4, + 4 + ], + "second_workload_elapsed_ms": [ + 7, + 4 + ], + "second_workload_ttft_ms": [ + 6, + 4 + ], + "streaming_elapsed_ms": [ + 4, + 4, + 4 + ], + "streaming_ttft_ms": [ + 3, + 3, + 3 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18513920, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18513920, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98312192, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18513920, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18513920, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98312192, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "peak_rss_bytes": 20054016, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20004864, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98480128, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "benchmark-baseline", + "open_fd_count": 14, + "peak_rss_bytes": 20054016, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20004864, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98480128, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 41, + "cpu_percent": 0.0, + "db_bytes": 884736, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-finite", + "open_fd_count": 14, + "peak_rss_bytes": 20119552, + "pending_requests": 0, + "request_statuses": { + "completed": 41 + }, + "requests": 41, + "reservations": 41, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20119552, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98615296, + "wal_bytes": 4144752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 76, + "cpu_percent": 0.0, + "db_bytes": 958464, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20242432, + "pending_requests": 0, + "request_statuses": { + "completed": 76 + }, + "requests": 76, + "reservations": 76, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20242432, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98615296, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 111, + "cpu_percent": 0.0, + "db_bytes": 1105920, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-translated-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20418560, + "pending_requests": 0, + "request_statuses": { + "completed": 111 + }, + "requests": 111, + "reservations": 111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20418560, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98615296, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-concurrency-4", + "open_fd_count": 14, + "peak_rss_bytes": 20828160, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20828160, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98889728, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-benchmark-stabilization", + "open_fd_count": 14, + "peak_rss_bytes": 20828160, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20828160, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 3000, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98889728, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "peak_rss_bytes": 20897792, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20897792, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98889728, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 145, + "cpu_percent": 0.0, + "db_bytes": 1216512, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19611648, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19611648, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 638632, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v2", + "startup_to_ready_ms": 38, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-3.json b/artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-3.json new file mode 100644 index 000000000..304f066be --- /dev/null +++ b/artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-3.json @@ -0,0 +1,1002 @@ +{ + "backup": { + "archive_sha256": "6764a51bd7f92e36087652a8dd8b17e9457c5fcf3981f3c2ab08537f90d0f927", + "archive_size_bytes": 1232359, + "elapsed_ms": 122, + "member_count": 3 + }, + "benchmark": { + "cadence": { + "backup_enabled": "false", + "backup_interval_s": "86400", + "database_busy_timeout_ms": "10000", + "database_synchronous": "\"NORMAL\"", + "database_wal": "true", + "database_worker_threads": "1", + "metrics_aggregate_only": "true", + "metrics_cleanup_interval_s": "86400", + "metrics_event_loop_lag_enabled": "false", + "metrics_flush_interval_s": "120", + "metrics_max_buffered_events": "250", + "metrics_write_mode": "\"low_wear\"", + "model_info_enabled": "false", + "model_info_startup_refresh": "false", + "models_refresh_interval_s": "300", + "models_startup_refresh": "true", + "routing_trace_mode": "\"off\"", + "routing_trace_sample_rate": "0.0", + "server_access_log": "false", + "server_threads": "1" + }, + "client_concurrency": 4, + "concurrency_batch_size": 32, + "config_fixture": "sbc-benchmark.toml", + "interpretation": "descriptive loopback characterization; no performance threshold applied", + "resource_summary": { + "baseline_peak_rss_bytes": 20008960, + "baseline_rss_bytes": 20008960, + "final_active_reservations": 0, + "final_finalization_jobs": null, + "final_pending_requests": 0, + "final_rss_bytes": 20844544, + "peak_rss_bytes": 20844544 + }, + "runs": { + "concurrent_native_responses_finite": { + "batch_elapsed_ms": 82, + "client_concurrency": 4, + "completed_count": 32, + "cpu": { + "cpu_percent": 121.88, + "cpu_ticks": 10, + "cpu_time_ms": 100 + }, + "failed_count": 0, + "request_count": 32, + "requests_per_second": 390.01, + "status": "measured", + "timings": { + "maximum_elapsed_ms": 13, + "minimum_elapsed_ms": 7, + "p50_elapsed_ms": 9, + "p50_ttft_ms": 9, + "p95_elapsed_ms": 13, + "p95_ttft_ms": 12, + "sample_count": 32 + } + }, + "native_responses_finite": { + "cpu": { + "cpu_percent": 76.46, + "cpu_ticks": 9, + "cpu_time_ms": 90 + }, + "maximum_elapsed_ms": 4, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 4, + "p95_elapsed_ms": 4, + "p95_ttft_ms": 4, + "sample_count": 30, + "status": "measured", + "streaming": false, + "warmup_count": 5 + }, + "native_responses_streaming": { + "cpu": { + "cpu_percent": 77.08, + "cpu_ticks": 9, + "cpu_time_ms": 90 + }, + "maximum_elapsed_ms": 4, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 4, + "p95_ttft_ms": 3, + "sample_count": 30, + "status": "measured", + "streaming": true, + "warmup_count": 5 + }, + "translated_responses_to_messages_streaming": { + "cpu": { + "cpu_percent": 74.45, + "cpu_ticks": 9, + "cpu_time_ms": 90 + }, + "expected_upstream_path": "/messages", + "maximum_elapsed_ms": 5, + "minimum_elapsed_ms": 4, + "model": "q008-messages", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 4, + "p95_ttft_ms": 3, + "sample_count": 30, + "status": "measured", + "streaming": true, + "upstream_proof": "fixture Messages path observed", + "upstream_request_delta": 35, + "warmup_count": 5 + } + }, + "sample_count": 30 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "q005-qualified-aarch64-copy", + "build_profile": "release", + "repository_commit_sha": "d2c0830acaeff90c45bffa393453cf0500c4fa64", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 5, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 5, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 8, + "id": "check-config", + "reason": "Configuration loaded successfully from /dev/shm/eggpool-q008-a9f1kj9a/config.toml Server: 127.0.0.1:41009 Accounts: 1 Database: Content digest: 91637c54d30b739d44c7eb769779680299f17731beaa6205207dc8f0bc257036", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 104, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 80, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 6, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 10, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 10, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 11, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 7, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 7, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 122, + "id": "backup", + "reason": "Wrote backup: /dev/shm/eggpool-q008-a9f1kj9a/backups/eggpool-backup-20261015-020450.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 57, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2585963)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 40, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /dev/shm/eggpool-q008-a9f1kj9a/recovery/config.toml db: /dev/shm/eggpool-q008-a9f1kj9a/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 28, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 59, + "id": "stop", + "reason": "Stopping server (PID 2586140)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 145, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_frequency_policy_end": "1500000", + "cpu_governor": "ondemand", + "cpu_governor_end": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_path_counts": { + "/chat/completions": 3, + "/messages": 37, + "/responses": 105 + }, + "loopback_provider_requests": 145, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 41.9, + "power_thermal_mode_celsius_end": 40.2, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 6, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 6 + }, + { + "body_bytes": 309, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 213, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 145, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 480, + "elapsed_ms": 5, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 4 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18518016, + 18518016, + 20008960, + 20008960, + 20123648, + 20254720, + 20418560, + 20844544, + 20844544, + 20910080, + 19615744 + ], + "samples": 11, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 6, + 4, + 4 + ], + "finite_ttft_ms": [ + 6, + 4, + 4 + ], + "second_workload_elapsed_ms": [ + 6, + 5 + ], + "second_workload_ttft_ms": [ + 6, + 5 + ], + "streaming_elapsed_ms": [ + 4, + 4, + 5 + ], + "streaming_ttft_ms": [ + 3, + 3, + 4 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18518016, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18518016, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98312192, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18518016, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18518016, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98312192, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "peak_rss_bytes": 20008960, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20008960, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98480128, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "benchmark-baseline", + "open_fd_count": 14, + "peak_rss_bytes": 20008960, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20008960, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98480128, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 41, + "cpu_percent": 0.0, + "db_bytes": 884736, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-finite", + "open_fd_count": 14, + "peak_rss_bytes": 20123648, + "pending_requests": 0, + "request_statuses": { + "completed": 41 + }, + "requests": 41, + "reservations": 41, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20123648, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98615296, + "wal_bytes": 4144752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 76, + "cpu_percent": 0.0, + "db_bytes": 958464, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20254720, + "pending_requests": 0, + "request_statuses": { + "completed": 76 + }, + "requests": 76, + "reservations": 76, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20254720, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98615296, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 111, + "cpu_percent": 0.0, + "db_bytes": 1105920, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-translated-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20418560, + "pending_requests": 0, + "request_statuses": { + "completed": 111 + }, + "requests": 111, + "reservations": 111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20418560, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98615296, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1228800, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-concurrency-4", + "open_fd_count": 14, + "peak_rss_bytes": 20844544, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20844544, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98889728, + "wal_bytes": 4260112, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1228800, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-benchmark-stabilization", + "open_fd_count": 14, + "peak_rss_bytes": 20844544, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20844544, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 3000, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98889728, + "wal_bytes": 4260112, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1228800, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "peak_rss_bytes": 20910080, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20910080, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98889728, + "wal_bytes": 4260112, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 145, + "cpu_percent": 0.0, + "db_bytes": 1216512, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19615744, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19615744, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98476032, + "wal_bytes": 638632, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v2", + "startup_to_ready_ms": 38, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-m012-pi5-benchmark-run-1.json b/artifacts/qualification/on-prem-m012-pi5-benchmark-run-1.json new file mode 100644 index 000000000..71b3b9ad3 --- /dev/null +++ b/artifacts/qualification/on-prem-m012-pi5-benchmark-run-1.json @@ -0,0 +1,1002 @@ +{ + "backup": { + "archive_sha256": "f12078435f9e79b5e4b37790c771712bc6ae87da1cf5bb6036ba4d4249265d8d", + "archive_size_bytes": 1236439, + "elapsed_ms": 3596, + "member_count": 3 + }, + "benchmark": { + "cadence": { + "backup_enabled": "false", + "backup_interval_s": "86400", + "database_busy_timeout_ms": "10000", + "database_synchronous": "\"NORMAL\"", + "database_wal": "true", + "database_worker_threads": "1", + "metrics_aggregate_only": "true", + "metrics_cleanup_interval_s": "86400", + "metrics_event_loop_lag_enabled": "false", + "metrics_flush_interval_s": "120", + "metrics_max_buffered_events": "250", + "metrics_write_mode": "\"low_wear\"", + "model_info_enabled": "false", + "model_info_startup_refresh": "false", + "models_refresh_interval_s": "300", + "models_startup_refresh": "true", + "routing_trace_mode": "\"off\"", + "routing_trace_sample_rate": "0.0", + "server_access_log": "false", + "server_threads": "1" + }, + "client_concurrency": 4, + "concurrency_batch_size": 32, + "config_fixture": "sbc-benchmark.toml", + "interpretation": "descriptive loopback characterization; no performance threshold applied", + "resource_summary": { + "baseline_peak_rss_bytes": 20004864, + "baseline_rss_bytes": 20004864, + "final_active_reservations": 0, + "final_finalization_jobs": null, + "final_pending_requests": 0, + "final_rss_bytes": 20836352, + "peak_rss_bytes": 20836352 + }, + "runs": { + "concurrent_native_responses_finite": { + "batch_elapsed_ms": 17124, + "client_concurrency": 4, + "completed_count": 32, + "cpu": { + "cpu_percent": 0.82, + "cpu_ticks": 14, + "cpu_time_ms": 140 + }, + "failed_count": 0, + "request_count": 32, + "requests_per_second": 1.869, + "status": "measured", + "timings": { + "maximum_elapsed_ms": 10148, + "minimum_elapsed_ms": 12, + "p50_elapsed_ms": 13, + "p50_ttft_ms": 12, + "p95_elapsed_ms": 10148, + "p95_ttft_ms": 10148, + "sample_count": 32 + } + }, + "native_responses_finite": { + "cpu": { + "cpu_percent": 1.06, + "cpu_ticks": 11, + "cpu_time_ms": 110 + }, + "maximum_elapsed_ms": 8239, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 5, + "p50_ttft_ms": 4, + "p95_elapsed_ms": 2011, + "p95_ttft_ms": 2010, + "sample_count": 30, + "status": "measured", + "streaming": false, + "warmup_count": 5 + }, + "native_responses_streaming": { + "cpu": { + "cpu_percent": 1.44, + "cpu_ticks": 12, + "cpu_time_ms": 120 + }, + "maximum_elapsed_ms": 7996, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 5, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 223, + "p95_ttft_ms": 19, + "sample_count": 30, + "status": "measured", + "streaming": true, + "warmup_count": 5 + }, + "translated_responses_to_messages_streaming": { + "cpu": { + "cpu_percent": 0.82, + "cpu_ticks": 12, + "cpu_time_ms": 120 + }, + "expected_upstream_path": "/messages", + "maximum_elapsed_ms": 7373, + "minimum_elapsed_ms": 4, + "model": "q008-messages", + "p50_elapsed_ms": 5, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 7066, + "p95_ttft_ms": 4, + "sample_count": 30, + "status": "measured", + "streaming": true, + "upstream_proof": "fixture Messages path observed", + "upstream_request_delta": 35, + "warmup_count": 5 + } + }, + "sample_count": 30 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "on-device-release-build", + "build_profile": "release", + "repository_commit_sha": "161ba3ef5114902a7af1cff556491f7727bd9bd9", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 6, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 5, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 5, + "id": "check-config", + "reason": "Configuration loaded successfully from /tmp/eggpool-q008-q82qthqu/config.toml Server: 127.0.0.1:33775 Accounts: 1 Database: Content digest: b26b0baa7296a7974e58b08c5735690bf8c8676bd3bc2933421d631a22c05808", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 315, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 172, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 6, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 10, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 10, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 10, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 6, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 6, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 3596, + "id": "backup", + "reason": "Wrote backup: /tmp/eggpool-q008-q82qthqu/backups/eggpool-backup-20261014-211419.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 1286, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2539661)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 7110, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /tmp/eggpool-q008-q82qthqu/recovery/config.toml db: /tmp/eggpool-q008-q82qthqu/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 8102, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 1850, + "id": "stop", + "reason": "Stopping server (PID 2539978)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 145, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_frequency_policy_end": "1900000", + "cpu_governor": "ondemand", + "cpu_governor_end": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_path_counts": { + "/chat/completions": 3, + "/messages": 37, + "/responses": 105 + }, + "loopback_provider_requests": 145, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 50.1, + "power_thermal_mode_celsius_end": 46.3, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 6, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 6 + }, + { + "body_bytes": 309, + "elapsed_ms": 5, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 5 + }, + { + "body_bytes": 213, + "elapsed_ms": 5, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 5 + }, + { + "body_bytes": 145, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 480, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18505728, + 18509824, + 20004864, + 20004864, + 20140032, + 20262912, + 20414464, + 20836352, + 20836352, + 20901888, + 19599360 + ], + "samples": 11, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 6, + 5, + 5 + ], + "finite_ttft_ms": [ + 6, + 5, + 5 + ], + "second_workload_elapsed_ms": [ + 6, + 5 + ], + "second_workload_ttft_ms": [ + 6, + 5 + ], + "streaming_elapsed_ms": [ + 4, + 4, + 4 + ], + "streaming_ttft_ms": [ + 3, + 3, + 3 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18505728, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18505728, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98308096, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18509824, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18509824, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98308096, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "peak_rss_bytes": 20004864, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20004864, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "benchmark-baseline", + "open_fd_count": 14, + "peak_rss_bytes": 20004864, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20004864, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 41, + "cpu_percent": 0.0, + "db_bytes": 884736, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-finite", + "open_fd_count": 14, + "peak_rss_bytes": 20140032, + "pending_requests": 0, + "request_statuses": { + "completed": 41 + }, + "requests": 41, + "reservations": 41, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20140032, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4165352, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 76, + "cpu_percent": 0.0, + "db_bytes": 958464, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20262912, + "pending_requests": 0, + "request_statuses": { + "completed": 76 + }, + "requests": 76, + "reservations": 76, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20262912, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 111, + "cpu_percent": 0.0, + "db_bytes": 1105920, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-translated-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20414464, + "pending_requests": 0, + "request_statuses": { + "completed": 111 + }, + "requests": 111, + "reservations": 111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20414464, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-concurrency-4", + "open_fd_count": 14, + "peak_rss_bytes": 20836352, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20836352, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-benchmark-stabilization", + "open_fd_count": 14, + "peak_rss_bytes": 20836352, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20836352, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 3000, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "peak_rss_bytes": 20901888, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20901888, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 145, + "cpu_percent": 0.0, + "db_bytes": 1216512, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19599360, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19599360, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98476032, + "wal_bytes": 638632, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v2", + "startup_to_ready_ms": 46, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-m012-pi5-benchmark-run-2.json b/artifacts/qualification/on-prem-m012-pi5-benchmark-run-2.json new file mode 100644 index 000000000..321e6f6b0 --- /dev/null +++ b/artifacts/qualification/on-prem-m012-pi5-benchmark-run-2.json @@ -0,0 +1,1002 @@ +{ + "backup": { + "archive_sha256": "e85634e1722f79a74471c27f3c82dafe832f0a5362031ded0c1af4b6a4df2139", + "archive_size_bytes": 1236439, + "elapsed_ms": 4693, + "member_count": 3 + }, + "benchmark": { + "cadence": { + "backup_enabled": "false", + "backup_interval_s": "86400", + "database_busy_timeout_ms": "10000", + "database_synchronous": "\"NORMAL\"", + "database_wal": "true", + "database_worker_threads": "1", + "metrics_aggregate_only": "true", + "metrics_cleanup_interval_s": "86400", + "metrics_event_loop_lag_enabled": "false", + "metrics_flush_interval_s": "120", + "metrics_max_buffered_events": "250", + "metrics_write_mode": "\"low_wear\"", + "model_info_enabled": "false", + "model_info_startup_refresh": "false", + "models_refresh_interval_s": "300", + "models_startup_refresh": "true", + "routing_trace_mode": "\"off\"", + "routing_trace_sample_rate": "0.0", + "server_access_log": "false", + "server_threads": "1" + }, + "client_concurrency": 4, + "concurrency_batch_size": 32, + "config_fixture": "sbc-benchmark.toml", + "interpretation": "descriptive loopback characterization; no performance threshold applied", + "resource_summary": { + "baseline_peak_rss_bytes": 19992576, + "baseline_rss_bytes": 19992576, + "final_active_reservations": 0, + "final_finalization_jobs": null, + "final_pending_requests": 0, + "final_rss_bytes": 20844544, + "peak_rss_bytes": 20844544 + }, + "runs": { + "concurrent_native_responses_finite": { + "batch_elapsed_ms": 18630, + "client_concurrency": 4, + "completed_count": 32, + "cpu": { + "cpu_percent": 0.75, + "cpu_ticks": 14, + "cpu_time_ms": 140 + }, + "failed_count": 0, + "request_count": 32, + "requests_per_second": 1.718, + "status": "measured", + "timings": { + "maximum_elapsed_ms": 13899, + "minimum_elapsed_ms": 11, + "p50_elapsed_ms": 14, + "p50_ttft_ms": 14, + "p95_elapsed_ms": 13897, + "p95_ttft_ms": 13897, + "sample_count": 32 + } + }, + "native_responses_finite": { + "cpu": { + "cpu_percent": 1.16, + "cpu_ticks": 11, + "cpu_time_ms": 110 + }, + "maximum_elapsed_ms": 8940, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 5, + "p50_ttft_ms": 5, + "p95_elapsed_ms": 402, + "p95_ttft_ms": 402, + "sample_count": 30, + "status": "measured", + "streaming": false, + "warmup_count": 5 + }, + "native_responses_streaming": { + "cpu": { + "cpu_percent": 0.7, + "cpu_ticks": 11, + "cpu_time_ms": 110 + }, + "maximum_elapsed_ms": 11697, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 4, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 3993, + "p95_ttft_ms": 10, + "sample_count": 30, + "status": "measured", + "streaming": true, + "warmup_count": 5 + }, + "translated_responses_to_messages_streaming": { + "cpu": { + "cpu_percent": 0.75, + "cpu_ticks": 13, + "cpu_time_ms": 130 + }, + "expected_upstream_path": "/messages", + "maximum_elapsed_ms": 9380, + "minimum_elapsed_ms": 4, + "model": "q008-messages", + "p50_elapsed_ms": 5, + "p50_ttft_ms": 4, + "p95_elapsed_ms": 7870, + "p95_ttft_ms": 6, + "sample_count": 30, + "status": "measured", + "streaming": true, + "upstream_proof": "fixture Messages path observed", + "upstream_request_delta": 35, + "warmup_count": 5 + } + }, + "sample_count": 30 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "on-device-release-build", + "build_profile": "release", + "repository_commit_sha": "161ba3ef5114902a7af1cff556491f7727bd9bd9", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 4, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 5, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 6, + "id": "check-config", + "reason": "Configuration loaded successfully from /tmp/eggpool-q008-w91fwl8e/config.toml Server: 127.0.0.1:42541 Accounts: 1 Database: Content digest: 927358f005f5955034daf39a46f63c0204dd67d651c5b37f14df628d3177e35a", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 3591, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 395, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 6, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 10, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 10, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 10, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 6, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 7, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 4693, + "id": "backup", + "reason": "Wrote backup: /tmp/eggpool-q008-w91fwl8e/backups/eggpool-backup-20261014-211602.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 1130, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2540017)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 11937, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /tmp/eggpool-q008-w91fwl8e/recovery/config.toml db: /tmp/eggpool-q008-w91fwl8e/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 6147, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 3023, + "id": "stop", + "reason": "Stopping server (PID 2540407)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 145, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_frequency_policy_end": "1900000", + "cpu_governor": "ondemand", + "cpu_governor_end": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_path_counts": { + "/chat/completions": 3, + "/messages": 37, + "/responses": 105 + }, + "loopback_provider_requests": 145, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 48.0, + "power_thermal_mode_celsius_end": 48.0, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 5, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 5 + }, + { + "body_bytes": 309, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 213, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 145, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 5, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 480, + "elapsed_ms": 5, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 4 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18505728, + 18505728, + 19992576, + 19992576, + 20119552, + 20250624, + 20410368, + 20844544, + 20844544, + 20910080, + 19599360 + ], + "samples": 11, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 5, + 4, + 4 + ], + "finite_ttft_ms": [ + 5, + 4, + 4 + ], + "second_workload_elapsed_ms": [ + 7, + 5 + ], + "second_workload_ttft_ms": [ + 7, + 5 + ], + "streaming_elapsed_ms": [ + 4, + 5, + 5 + ], + "streaming_ttft_ms": [ + 3, + 3, + 4 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18505728, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18505728, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98304000, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18505728, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18505728, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98304000, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19992576, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19992576, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "benchmark-baseline", + "open_fd_count": 14, + "peak_rss_bytes": 19992576, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19992576, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98471936, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 41, + "cpu_percent": 0.0, + "db_bytes": 884736, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-finite", + "open_fd_count": 14, + "peak_rss_bytes": 20119552, + "pending_requests": 0, + "request_statuses": { + "completed": 41 + }, + "requests": 41, + "reservations": 41, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20119552, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4144752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 76, + "cpu_percent": 0.0, + "db_bytes": 958464, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20250624, + "pending_requests": 0, + "request_statuses": { + "completed": 76 + }, + "requests": 76, + "reservations": 76, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20250624, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 111, + "cpu_percent": 0.0, + "db_bytes": 1105920, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-translated-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20410368, + "pending_requests": 0, + "request_statuses": { + "completed": 111 + }, + "requests": 111, + "reservations": 111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20410368, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-concurrency-4", + "open_fd_count": 14, + "peak_rss_bytes": 20844544, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20844544, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98877440, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-benchmark-stabilization", + "open_fd_count": 14, + "peak_rss_bytes": 20844544, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20844544, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 3000, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98877440, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "peak_rss_bytes": 20910080, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20910080, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98877440, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 145, + "cpu_percent": 0.0, + "db_bytes": 1216512, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19599360, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19599360, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98476032, + "wal_bytes": 638632, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v2", + "startup_to_ready_ms": 335, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-m012-pi5-benchmark-run-3.json b/artifacts/qualification/on-prem-m012-pi5-benchmark-run-3.json new file mode 100644 index 000000000..f803b9ac1 --- /dev/null +++ b/artifacts/qualification/on-prem-m012-pi5-benchmark-run-3.json @@ -0,0 +1,1002 @@ +{ + "backup": { + "archive_sha256": "909e6be0311600e34a5e3af05477cc4605ebe8d99b8bc589f00dfbf66f475a65", + "archive_size_bytes": 1236439, + "elapsed_ms": 8287, + "member_count": 3 + }, + "benchmark": { + "cadence": { + "backup_enabled": "false", + "backup_interval_s": "86400", + "database_busy_timeout_ms": "10000", + "database_synchronous": "\"NORMAL\"", + "database_wal": "true", + "database_worker_threads": "1", + "metrics_aggregate_only": "true", + "metrics_cleanup_interval_s": "86400", + "metrics_event_loop_lag_enabled": "false", + "metrics_flush_interval_s": "120", + "metrics_max_buffered_events": "250", + "metrics_write_mode": "\"low_wear\"", + "model_info_enabled": "false", + "model_info_startup_refresh": "false", + "models_refresh_interval_s": "300", + "models_startup_refresh": "true", + "routing_trace_mode": "\"off\"", + "routing_trace_sample_rate": "0.0", + "server_access_log": "false", + "server_threads": "1" + }, + "client_concurrency": 4, + "concurrency_batch_size": 32, + "config_fixture": "sbc-benchmark.toml", + "interpretation": "descriptive loopback characterization; no performance threshold applied", + "resource_summary": { + "baseline_peak_rss_bytes": 19996672, + "baseline_rss_bytes": 19996672, + "final_active_reservations": 0, + "final_finalization_jobs": null, + "final_pending_requests": 0, + "final_rss_bytes": 20856832, + "peak_rss_bytes": 20856832 + }, + "runs": { + "concurrent_native_responses_finite": { + "batch_elapsed_ms": 20747, + "client_concurrency": 4, + "completed_count": 32, + "cpu": { + "cpu_percent": 0.72, + "cpu_ticks": 15, + "cpu_time_ms": 150 + }, + "failed_count": 0, + "request_count": 32, + "requests_per_second": 1.542, + "status": "measured", + "timings": { + "maximum_elapsed_ms": 13172, + "minimum_elapsed_ms": 8, + "p50_elapsed_ms": 15, + "p50_ttft_ms": 14, + "p95_elapsed_ms": 13168, + "p95_ttft_ms": 13168, + "sample_count": 32 + } + }, + "native_responses_finite": { + "cpu": { + "cpu_percent": 0.79, + "cpu_ticks": 12, + "cpu_time_ms": 120 + }, + "maximum_elapsed_ms": 8117, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 5, + "p50_ttft_ms": 5, + "p95_elapsed_ms": 6820, + "p95_ttft_ms": 6819, + "sample_count": 30, + "status": "measured", + "streaming": false, + "warmup_count": 5 + }, + "native_responses_streaming": { + "cpu": { + "cpu_percent": 1.14, + "cpu_ticks": 13, + "cpu_time_ms": 130 + }, + "maximum_elapsed_ms": 8788, + "minimum_elapsed_ms": 4, + "model": "q008-responses", + "p50_elapsed_ms": 5, + "p50_ttft_ms": 4, + "p95_elapsed_ms": 2519, + "p95_ttft_ms": 10, + "sample_count": 30, + "status": "measured", + "streaming": true, + "warmup_count": 5 + }, + "translated_responses_to_messages_streaming": { + "cpu": { + "cpu_percent": 0.96, + "cpu_ticks": 13, + "cpu_time_ms": 130 + }, + "expected_upstream_path": "/messages", + "maximum_elapsed_ms": 7150, + "minimum_elapsed_ms": 4, + "model": "q008-messages", + "p50_elapsed_ms": 5, + "p50_ttft_ms": 3, + "p95_elapsed_ms": 6267, + "p95_ttft_ms": 8, + "sample_count": 30, + "status": "measured", + "streaming": true, + "upstream_proof": "fixture Messages path observed", + "upstream_request_delta": 35, + "warmup_count": 5 + } + }, + "sample_count": 30 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_mode": "on-device-release-build", + "build_profile": "release", + "repository_commit_sha": "161ba3ef5114902a7af1cff556491f7727bd9bd9", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 4, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 5, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 6, + "id": "check-config", + "reason": "Configuration loaded successfully from /tmp/eggpool-q008-0o91wg7n/config.toml Server: 127.0.0.1:41919 Accounts: 1 Database: Content digest: 3508cc8a2fe6bb86679201271ac1504901137358478e75e37af679eed02b0d42", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 3440, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 421, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 6, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 10, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 10, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 12, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 6, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 6, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"commit\", \"exit_code\": 0, \"generation\": 1, \"changed_sections\": [], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": false, \"message\": \"no changes\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 8287, + "id": "backup", + "reason": "Wrote backup: /tmp/eggpool-q008-0o91wg7n/backups/eggpool-backup-20261014-211756.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 465, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2540447)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 7159, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /tmp/eggpool-q008-0o91wg7n/recovery/config.toml db: /tmp/eggpool-q008-0o91wg7n/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 2815, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 823, + "id": "stop", + "reason": "Stopping server (PID 2540828)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 145, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_frequency_policy_end": "2000000", + "cpu_governor": "ondemand", + "cpu_governor_end": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_path_counts": { + "/chat/completions": 3, + "/messages": 37, + "/responses": 105 + }, + "loopback_provider_requests": 145, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 50.1, + "power_thermal_mode_celsius_end": 46.3, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 6, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 6 + }, + { + "body_bytes": 309, + "elapsed_ms": 5, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 5 + }, + { + "body_bytes": 213, + "elapsed_ms": 5, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 5 + }, + { + "body_bytes": 145, + "elapsed_ms": 5, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 4 + }, + { + "body_bytes": 270, + "elapsed_ms": 5, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 4 + }, + { + "body_bytes": 480, + "elapsed_ms": 5, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 4 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18505728, + 18505728, + 19996672, + 19996672, + 20140032, + 20254720, + 20418560, + 20856832, + 20856832, + 20922368, + 19595264 + ], + "samples": 11, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 6, + 5, + 5 + ], + "finite_ttft_ms": [ + 6, + 5, + 5 + ], + "second_workload_elapsed_ms": [ + 9, + 5 + ], + "second_workload_ttft_ms": [ + 9, + 5 + ], + "streaming_elapsed_ms": [ + 5, + 5, + 5 + ], + "streaming_ttft_ms": [ + 4, + 4, + 4 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18505728, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18505728, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98304000, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "peak_rss_bytes": 18505728, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18505728, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98304000, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19996672, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19996672, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98467840, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "benchmark-baseline", + "open_fd_count": 14, + "peak_rss_bytes": 19996672, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19996672, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98467840, + "wal_bytes": 1672752, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 41, + "cpu_percent": 0.0, + "db_bytes": 884736, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-finite", + "open_fd_count": 14, + "peak_rss_bytes": 20140032, + "pending_requests": 0, + "request_statuses": { + "completed": 41 + }, + "requests": 41, + "reservations": 41, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20140032, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4165352, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 76, + "cpu_percent": 0.0, + "db_bytes": 958464, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-native-responses-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20254720, + "pending_requests": 0, + "request_statuses": { + "completed": 76 + }, + "requests": 76, + "reservations": 76, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20254720, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4185952, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 111, + "cpu_percent": 0.0, + "db_bytes": 1105920, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-translated-streaming", + "open_fd_count": 14, + "peak_rss_bytes": 20418560, + "pending_requests": 0, + "request_statuses": { + "completed": 111 + }, + "requests": 111, + "reservations": 111, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20418560, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98607104, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-concurrency-4", + "open_fd_count": 14, + "peak_rss_bytes": 20856832, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20856832, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-benchmark-stabilization", + "open_fd_count": 14, + "peak_rss_bytes": 20856832, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20856832, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 3000, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 143, + "cpu_percent": 0.0, + "db_bytes": 1232896, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 14, + "peak_rss_bytes": 20922368, + "pending_requests": 0, + "request_statuses": { + "completed": 143 + }, + "requests": 143, + "reservations": 143, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 20922368, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98881536, + "wal_bytes": 4297192, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 145, + "cpu_percent": 0.0, + "db_bytes": 1216512, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "peak_rss_bytes": 19595264, + "pending_requests": 0, + "request_statuses": { + "completed": 145 + }, + "requests": 145, + "reservations": 145, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19595264, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98476032, + "wal_bytes": 638632, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v2", + "startup_to_ready_ms": 347, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-m012-pi5-functional.json b/artifacts/qualification/on-prem-m012-pi5-functional.json new file mode 100644 index 000000000..d358d175c --- /dev/null +++ b/artifacts/qualification/on-prem-m012-pi5-functional.json @@ -0,0 +1,659 @@ +{ + "backup": { + "archive_sha256": "bf84e6ec1487173dcad74479b3252e60ce69dbbd75ea6204592b0e5cbb44a2df", + "archive_size_bytes": 842130, + "elapsed_ms": 8983, + "member_count": 3 + }, + "candidate": { + "binary_size_bytes": 34984648, + "build_elapsed_ms": 1521999, + "build_mode": "on-device-release-build", + "build_profile": "release", + "rust_toolchain": "rustc 1.99.0 (b940084d7 2026-09-28)", + "sha256": "8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538" + }, + "commands": [ + { + "command": [ + "", + "--config", + "/config.toml", + "version" + ], + "duration_ms": 6, + "id": "version", + "reason": "0.8.2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "--help" + ], + "duration_ms": 4, + "id": "help", + "reason": "EggPool - aggregate OpenCode Go subscriptions. Usage: eggpool [OPTIONS] [COMMAND] Commands: serve connect logout check-config edit getkey newkey configsetup configremote deploy accounts dashboard db models modelinfo stats users onboard croncheck ensure-running migrate stop restart init-config help recover uninstall update set rehash runtime-status status backup version Options: --config Path to the TOML configuration file. Falls back to $EGGPOOL_CONFIG, then ~/.config/eggpool/config.toml, then ./config.toml. -h, --help Print help -V, --version Print version", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "check-config" + ], + "duration_ms": 6, + "id": "check-config", + "reason": "Configuration loaded successfully from /tmp/eggpool-q008-2l3zr0jw/config.toml Server: 127.0.0.1:37919 Accounts: 1 Database: Content digest: 241a6e7f7229d228cddac9e3370bc55775ff39ce786a257a38fac6e4c57ce2f2", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "migrate" + ], + "duration_ms": 4352, + "id": "migrate", + "reason": "Migrations completed successfully Applied migrations: 62 Current schema: 62", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "users", + "bootstrap" + ], + "duration_ms": 331, + "id": "bootstrap-admin", + "reason": "synthetic administrator bootstrap complete", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "accounts", + "list" + ], + "duration_ms": 5, + "id": "accounts-list", + "reason": "Configured accounts: fixture/q008-account Total: 1 accounts", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "modelinfo", + "list" + ], + "duration_ms": 10, + "id": "modelinfo-list", + "reason": "No model-info rows found.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stats", + "explain-dashboard", + "--json" + ], + "duration_ms": 11, + "id": "operator-stats", + "reason": "{ \"period\": \"24h\", \"bucket\": \"hour\", \"group_by\": \"provider_model\", \"queries\": [ { \"name\": \"fetch_timeseries\", \"plan\": [ \"SEARCH r USING COVERING INDEX idx_requests_started (started_at>? AND started_at? AND started_at? AND started_at? AND started_at", + "--config", + "/config.toml", + "stats", + "transcoding", + "--json" + ], + "duration_ms": 10, + "id": "transcoding-stats", + "reason": "{ \"total\": 0, \"native_count\": 0, \"transcoded_count\": 0, \"per_direction\": {}, \"top_loss_warnings\": [], \"period\": \"24h\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "runtime-status", + "--json" + ], + "duration_ms": 6, + "id": "runtime-status", + "reason": "runtime-status authentication failed (HTTP 401)", + "returncode": 1, + "status": "fail", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "rehash", + "--json" + ], + "duration_ms": 8, + "id": "rehash", + "reason": "{ \"ok\": true, \"stage\": \"activation\", \"exit_code\": 0, \"generation\": 2, \"changed_sections\": [ \"metrics\" ], \"warnings\": [], \"restart_required\": [], \"retirement_pending\": true, \"message\": \"reload applied\" }", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "backup", + "--output-dir", + "/backups" + ], + "duration_ms": 8983, + "id": "backup", + "reason": "Wrote backup: /tmp/eggpool-q008-2l3zr0jw/backups/eggpool-backup-20261014-211225.zip included: config.toml included: usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 15028, + "id": "stop-before-recovery", + "reason": "Stopping server (PID 2539379)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/recovery/config.toml", + "recover", + "/recovery/recovered.zip" + ], + "duration_ms": 4452, + "id": "recover", + "reason": "Overwrite current configuration and database? [y/N] Restore complete. Restart the server to load the new config. config: /tmp/eggpool-q008-2l3zr0jw/recovery/config.toml db: /tmp/eggpool-q008-2l3zr0jw/recovery/usage.sqlite3", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "db", + "vacuum" + ], + "duration_ms": 3608, + "id": "vacuum", + "reason": "Database vacuum completed successfully", + "returncode": 0, + "status": "pass", + "timed_out": false + }, + { + "command": [ + "", + "--config", + "/config.toml", + "stop", + "--timeout", + "30" + ], + "duration_ms": 2359, + "id": "stop", + "reason": "Stopping server (PID 2539504)... Server stopped.", + "returncode": 0, + "status": "pass", + "timed_out": false + } + ], + "durable": { + "active_reservations": 0, + "attempts": 8, + "pending_requests": 0, + "request_statuses": { + "completed": 8 + }, + "requests": 8, + "reservations": 8 + }, + "environment": { + "architecture": "aarch64", + "attestation": "Linux aarch64 plus device-tree board model", + "board_model": "Raspberry Pi 5 Model B Rev 1.0", + "cpu_frequency_policy": "2400000", + "cpu_governor": "ondemand", + "filesystem": "ext4", + "kernel": "6.8.0-1065-raspi", + "loopback_provider_requests": 8, + "os": "Ubuntu 24.04.4 LTS", + "os_version_id": "24.04", + "power_thermal_mode_celsius": 52.4, + "ram_bytes": 8322752512, + "root_storage_device_class": "mmc", + "soc_cpu_core_count": 4, + "storage_medium_class": "non-rotational" + }, + "findings": [], + "functional": [ + { + "body_bytes": 15, + "http_status": 200, + "id": "startup-readyz", + "status": "pass" + }, + { + "body_bytes": 272, + "http_status": 200, + "id": "model-listing", + "status": "pass" + }, + { + "body_bytes": 225, + "elapsed_ms": 4, + "http_status": 200, + "id": "chat_completions-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 309, + "elapsed_ms": 4, + "http_status": 200, + "id": "responses-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 213, + "elapsed_ms": 4, + "http_status": 200, + "id": "messages-finite", + "status": "pass", + "ttft_ms": 4 + }, + { + "body_bytes": 145, + "elapsed_ms": 5, + "http_status": 200, + "id": "chat_completions-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 3 + }, + { + "body_bytes": 270, + "elapsed_ms": 5, + "http_status": 200, + "id": "responses-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 4 + }, + { + "body_bytes": 480, + "elapsed_ms": 5, + "http_status": 200, + "id": "messages-stream", + "status": "pass", + "terminal_evidence": true, + "ttft_ms": 4 + }, + { + "id": "accounts-list", + "status": "pass" + }, + { + "id": "modelinfo-list", + "status": "pass" + }, + { + "id": "operator-stats", + "status": "pass" + }, + { + "id": "transcoding-stats", + "status": "pass" + }, + { + "id": "runtime-status", + "note": "managed-fencing: server-key rejected, browser session required", + "status": "pass" + }, + { + "body_bytes": 29, + "http_status": 401, + "id": "dashboard-page", + "note": "managed-fencing: dashboard shell requires browser session", + "status": "pass" + }, + { + "body_bytes": 56004, + "http_status": 200, + "id": "dashboard-static-dashboard.css", + "status": "pass" + }, + { + "body_bytes": 49083, + "http_status": 200, + "id": "dashboard-static-dashboard.js", + "status": "pass" + }, + { + "body_bytes": 131072, + "http_status": 200, + "id": "dashboard-static-chart.js", + "status": "pass" + }, + { + "body_bytes": 1203, + "http_status": 200, + "id": "dashboard-static-favicon.svg", + "status": "pass" + }, + { + "body_bytes": 1264, + "http_status": 200, + "id": "dashboard-static-theme.css?theme=Nord", + "status": "pass" + }, + { + "id": "rehash-runtime-status", + "status": "pass" + }, + { + "id": "graceful-stop-before-recovery", + "status": "pass" + }, + { + "id": "backup-recover-isolated", + "recovered_database": true, + "status": "pass" + }, + { + "id": "bounded-maintenance", + "status": "pass" + }, + { + "id": "restart-reconcile", + "status": "pass" + }, + { + "id": "graceful-shutdown", + "status": "pass" + } + ], + "isolated_temporary_root": true, + "manifest": "runtime-q001.v1", + "plan": "SBC qualification", + "python_reference": { + "reason": "optional same-board comparison was not supplied", + "status": "not-run" + }, + "repeated_run_stability": { + "fd_counts": [ + 14, + 14, + 14, + 17, + 14 + ], + "interpretation": "bounded samples for release characterization; no fixed performance threshold applied", + "logical_leaks": false, + "rss_bytes": [ + 18505728, + 21630976, + 22675456, + 23691264, + 19390464 + ], + "samples": 5, + "status": "pass", + "thread_counts": [ + 2, + 2, + 2, + 3, + 2 + ] + }, + "request_workload_timings": { + "elapsed_ms": { + "finite_elapsed_ms": [ + 4, + 4, + 4 + ], + "finite_ttft_ms": [ + 4, + 4, + 4 + ], + "second_workload_elapsed_ms": [ + 6, + 5 + ], + "second_workload_ttft_ms": [ + 6, + 5 + ], + "streaming_elapsed_ms": [ + 5, + 5, + 5 + ], + "streaming_ttft_ms": [ + 3, + 4, + 4 + ] + }, + "interpretation": "bounded client-observed timings; first-byte values are diagnostic characterization, not an SLA", + "sample_count": 16 + }, + "resource_samples": [ + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "before-warmup", + "open_fd_count": 14, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 18505728, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98308096, + "wal_bytes": 103032, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 0, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-warmup", + "open_fd_count": 14, + "pending_requests": 0, + "request_statuses": {}, + "requests": 0, + "reservations": 0, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 21630976, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 167682048, + "wal_bytes": 160712, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-first-workload", + "open_fd_count": 14, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 22675456, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 168005632, + "wal_bytes": 1730432, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 6, + "cpu_percent": 0.0, + "db_bytes": 831488, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-rehash-backup", + "open_fd_count": 17, + "pending_requests": 0, + "request_statuses": { + "completed": 6 + }, + "requests": 6, + "reservations": 6, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 23691264, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 3, + "virtual_bytes": 168058880, + "wal_bytes": 1911712, + "wire_flights": null + }, + { + "active_generation_id": null, + "active_leases": null, + "active_reservations": 0, + "attempts": 8, + "cpu_percent": 0.0, + "db_bytes": 827392, + "finalization_capacity": null, + "finalization_jobs": null, + "generation_count": null, + "label": "after-second-workload", + "open_fd_count": 14, + "pending_requests": 0, + "request_statuses": { + "completed": 8 + }, + "requests": 8, + "reservations": 8, + "retiring_generations": null, + "retiring_leases": 0, + "rss_bytes": 19390464, + "running_tasks": 0, + "runtime_read_error": false, + "runtime_tasks": 0, + "sample_window_ms": 200, + "terminal_references": 0, + "thread_count": 2, + "virtual_bytes": 98484224, + "wal_bytes": 556232, + "wire_flights": null + } + ], + "schema_version": "runtime-q008.v1", + "startup_to_ready_ms": 136, + "status": "pass" +} diff --git a/artifacts/qualification/on-prem-user-access-m007-local.json b/artifacts/qualification/on-prem-user-access-m007-local.json new file mode 100644 index 000000000..0231b4a45 --- /dev/null +++ b/artifacts/qualification/on-prem-user-access-m007-local.json @@ -0,0 +1,557 @@ +{ + "budget": { + "automatic_retry_policy": "EggPool ordinary bounded policy only", + "max_output_tokens": 16, + "maximum_requests": 8, + "planned_requests": 7 + }, + "candidate_stderr": "", + "candidate_stdout": "2026-10-14T01:42:37.467605Z INFO Rust development server listening address=\"127.0.0.1:51264\" 2026-10-14T01:42:37.585248Z WARN staged candidate dropped without acceptance; restored live wire preferences", + "catalog": { + "expected_models": [ + "mimo-v2.5", + "minimax-m3", + "muse-spark-1.2-contributor" + ], + "resolved_models": [ + "mimo-v2.5", + "minimax-m3", + "muse-spark-1.2-contributor" + ], + "status": "pass" + }, + "cells": [ + { + "attempt_error_class": null, + "client_surface": "responses", + "content_type": "application/json", + "cross_surface": false, + "durable_status": "completed", + "expected_upstream_path": "/responses", + "expected_upstream_surface": "openai_responses", + "http_status": 200, + "id": "responses-finite", + "input_tokens": 1, + "model_id": "muse-spark-1.2-contributor", + "output_tokens": 1, + "provider_id": "opencode-go", + "raw_response_body_retained": false, + "request_id_present": true, + "response": { + "body_bytes": 244, + "normalized_response": true, + "raw_body_retained": false, + "usage": { + "input_tokens": 1, + "output_tokens": 1, + "reported": true, + "total_tokens": 2 + } + }, + "status": "pass", + "streaming": false, + "upstream_observation": { + "auth_shape": "bearer", + "credential_valid": true, + "header_names": [ + "accept", + "authorization", + "content-length", + "content-type", + "host", + "user-agent", + "x-opencode-session", + "x-request-id" + ], + "input_kind": "str", + "input_length": 21, + "max_output_tokens": 16, + "method": "POST", + "model": "muse-spark-1.2-contributor", + "path": "/responses", + "request_keys": [ + "input", + "max_output_tokens", + "model", + "store", + "stream" + ], + "session_header_present": true, + "store": false, + "streaming": false, + "surface": "openai_responses" + }, + "upstream_status": 200 + }, + { + "attempt_error_class": null, + "client_surface": "chat_completions", + "content_type": "application/json", + "cross_surface": false, + "durable_status": "completed", + "expected_upstream_path": "/chat/completions", + "expected_upstream_surface": "openai_chat_completions", + "http_status": 200, + "id": "chat-finite", + "input_tokens": 1, + "model_id": "mimo-v2.5", + "output_tokens": 1, + "provider_id": "opencode-go", + "raw_response_body_retained": false, + "request_id_present": true, + "response": { + "body_bytes": 212, + "normalized_response": true, + "raw_body_retained": false, + "usage": { + "completion_tokens": 1, + "prompt_tokens": 1, + "reported": true, + "total_tokens": 2 + } + }, + "status": "pass", + "streaming": false, + "upstream_observation": { + "auth_shape": "bearer", + "credential_valid": true, + "header_names": [ + "accept", + "authorization", + "content-length", + "content-type", + "host", + "user-agent", + "x-opencode-session", + "x-request-id" + ], + "input_kind": "NoneType", + "input_length": null, + "max_output_tokens": null, + "method": "POST", + "model": "mimo-v2.5", + "path": "/chat/completions", + "request_keys": [ + "max_tokens", + "messages", + "model", + "stream" + ], + "session_header_present": true, + "store": null, + "streaming": false, + "surface": "openai_chat_completions" + }, + "upstream_status": 200 + }, + { + "attempt_error_class": null, + "client_surface": "messages", + "content_type": "application/json", + "cross_surface": false, + "durable_status": "completed", + "expected_upstream_path": "/messages", + "expected_upstream_surface": "anthropic_messages", + "http_status": 200, + "id": "messages-finite", + "input_tokens": 1, + "model_id": "minimax-m3", + "output_tokens": 1, + "provider_id": "opencode-go", + "raw_response_body_retained": false, + "request_id_present": true, + "response": { + "body_bytes": 173, + "normalized_response": true, + "raw_body_retained": false, + "usage": { + "input_tokens": 1, + "output_tokens": 1, + "reported": true + } + }, + "status": "pass", + "streaming": false, + "upstream_observation": { + "auth_shape": "api_key", + "credential_valid": true, + "header_names": [ + "accept", + "content-length", + "content-type", + "host", + "user-agent", + "x-api-key", + "x-opencode-session", + "x-request-id" + ], + "input_kind": "NoneType", + "input_length": null, + "max_output_tokens": null, + "method": "POST", + "model": "minimax-m3", + "path": "/messages", + "request_keys": [ + "max_tokens", + "messages", + "model", + "stream" + ], + "session_header_present": true, + "store": null, + "streaming": false, + "surface": "anthropic_messages" + }, + "upstream_status": 200 + }, + { + "attempt_error_class": null, + "client_surface": "messages", + "content_type": "application/json", + "cross_surface": true, + "durable_status": "completed", + "expected_upstream_path": "/responses", + "expected_upstream_surface": "openai_responses", + "http_status": 200, + "id": "messages-to-responses-finite", + "input_tokens": 1, + "model_id": "muse-spark-1.2-contributor", + "output_tokens": 1, + "provider_id": "opencode-go", + "raw_response_body_retained": false, + "request_id_present": true, + "response": { + "body_bytes": 196, + "normalized_response": true, + "raw_body_retained": false, + "usage": { + "input_tokens": 1, + "output_tokens": 1, + "reported": true + } + }, + "status": "pass", + "streaming": false, + "upstream_observation": { + "auth_shape": "bearer", + "credential_valid": true, + "header_names": [ + "accept", + "authorization", + "content-length", + "content-type", + "host", + "user-agent", + "x-opencode-session", + "x-request-id" + ], + "input_kind": "list", + "input_length": null, + "max_output_tokens": 16, + "method": "POST", + "model": "muse-spark-1.2-contributor", + "path": "/responses", + "request_keys": [ + "input", + "max_output_tokens", + "model", + "store", + "stream" + ], + "session_header_present": true, + "store": false, + "streaming": false, + "surface": "openai_responses" + }, + "upstream_status": 200 + }, + { + "attempt_error_class": null, + "client_surface": "responses", + "content_type": "text/event-stream", + "cross_surface": false, + "durable_status": "completed", + "expected_upstream_path": "/responses", + "expected_upstream_surface": "openai_responses", + "http_status": 200, + "id": "responses-stream", + "input_tokens": 1, + "model_id": "muse-spark-1.2-contributor", + "output_tokens": 1, + "provider_id": "opencode-go", + "raw_response_body_retained": false, + "request_id_present": true, + "response": { + "body_bytes": 268, + "incremental_events": 2, + "raw_body_retained": false, + "terminal_evidence": true, + "usage": { + "reported": true + } + }, + "status": "pass", + "streaming": true, + "upstream_observation": { + "auth_shape": "bearer", + "credential_valid": true, + "header_names": [ + "accept", + "authorization", + "content-length", + "content-type", + "host", + "user-agent", + "x-opencode-session", + "x-request-id" + ], + "input_kind": "str", + "input_length": 21, + "max_output_tokens": 16, + "method": "POST", + "model": "muse-spark-1.2-contributor", + "path": "/responses", + "request_keys": [ + "input", + "max_output_tokens", + "model", + "store", + "stream" + ], + "session_header_present": true, + "store": false, + "streaming": true, + "surface": "openai_responses" + }, + "upstream_status": 200 + }, + { + "attempt_error_class": null, + "client_surface": "chat_completions", + "content_type": "text/event-stream", + "cross_surface": false, + "durable_status": "completed", + "expected_upstream_path": "/chat/completions", + "expected_upstream_surface": "openai_chat_completions", + "http_status": 200, + "id": "chat-stream", + "input_tokens": 1, + "model_id": "mimo-v2.5", + "output_tokens": 1, + "provider_id": "opencode-go", + "raw_response_body_retained": false, + "request_id_present": true, + "response": { + "body_bytes": 377, + "incremental_events": 3, + "raw_body_retained": false, + "terminal_evidence": true, + "usage": { + "reported": true + } + }, + "status": "pass", + "streaming": true, + "upstream_observation": { + "auth_shape": "bearer", + "credential_valid": true, + "header_names": [ + "accept", + "authorization", + "content-length", + "content-type", + "host", + "user-agent", + "x-opencode-session", + "x-request-id" + ], + "input_kind": "NoneType", + "input_length": null, + "max_output_tokens": null, + "method": "POST", + "model": "mimo-v2.5", + "path": "/chat/completions", + "request_keys": [ + "max_tokens", + "messages", + "model", + "stream" + ], + "session_header_present": true, + "store": null, + "streaming": true, + "surface": "openai_chat_completions" + }, + "upstream_status": 200 + }, + { + "attempt_error_class": null, + "client_surface": "messages", + "content_type": "text/event-stream", + "cross_surface": false, + "durable_status": "completed", + "expected_upstream_path": "/messages", + "expected_upstream_surface": "anthropic_messages", + "http_status": 200, + "id": "messages-stream", + "input_tokens": 1, + "model_id": "minimax-m3", + "output_tokens": 1, + "provider_id": "opencode-go", + "raw_response_body_retained": false, + "request_id_present": true, + "response": { + "body_bytes": 480, + "incremental_events": 5, + "raw_body_retained": false, + "terminal_evidence": true, + "usage": { + "reported": true + } + }, + "status": "pass", + "streaming": true, + "upstream_observation": { + "auth_shape": "api_key", + "credential_valid": true, + "header_names": [ + "accept", + "content-length", + "content-type", + "host", + "user-agent", + "x-api-key", + "x-opencode-session", + "x-request-id" + ], + "input_kind": "NoneType", + "input_length": null, + "max_output_tokens": null, + "method": "POST", + "model": "minimax-m3", + "path": "/messages", + "request_keys": [ + "max_tokens", + "messages", + "model", + "stream" + ], + "session_header_present": true, + "store": null, + "streaming": true, + "surface": "anthropic_messages" + }, + "upstream_status": 200 + } + ], + "credentials": { + "names": [ + "EGGPOOL_E2E_OPENCODE_GO_API_KEY" + ], + "source": "environment variable", + "values_written_to_evidence": false + }, + "durable": { + "account_backoffs": 0, + "active_reservations": 0, + "attempts": 7, + "attempts_with_upstream_request_id": 7, + "completed_requests": 7, + "database_exists": true, + "input_tokens": 7, + "output_tokens": 7, + "pending_requests": 0, + "requests": 7 + }, + "duration_ms": 4832, + "environment": { + "architecture": "x86_64", + "candidate_sha256": "479c02eac9b9300767e5199f164dc3855536b984b9a616cf0d638443a2691216", + "os": "macOS-26.6.2-x86_64-i386-64bit", + "provider": "loopback fake provider", + "python": "3.11.9", + "rust": "rustc 1.98.1 (48a229cea 2026-09-01) (Homebrew)", + "system": "Darwin", + "transport": "direct" + }, + "manifest": "runtime-q001.v1", + "offline_fake": true, + "plan": "live-provider qualification", + "planned_matrix": [ + { + "client_surface": "responses", + "cross_surface": false, + "id": "responses-finite", + "model_id": "muse-spark-1.2-contributor", + "provider_id": "opencode-go", + "streaming": false, + "upstream_surface": "openai_responses" + }, + { + "client_surface": "chat_completions", + "cross_surface": false, + "id": "chat-finite", + "model_id": "mimo-v2.5", + "provider_id": "opencode-go", + "streaming": false, + "upstream_surface": "openai_chat_completions" + }, + { + "client_surface": "messages", + "cross_surface": false, + "id": "messages-finite", + "model_id": "minimax-m3", + "provider_id": "opencode-go", + "streaming": false, + "upstream_surface": "anthropic_messages" + }, + { + "client_surface": "messages", + "cross_surface": true, + "id": "messages-to-responses-finite", + "model_id": "muse-spark-1.2-contributor", + "provider_id": "opencode-go", + "streaming": false, + "upstream_surface": "openai_responses" + }, + { + "client_surface": "responses", + "cross_surface": false, + "id": "responses-stream", + "model_id": "muse-spark-1.2-contributor", + "provider_id": "opencode-go", + "streaming": true, + "upstream_surface": "openai_responses" + }, + { + "client_surface": "chat_completions", + "cross_surface": false, + "id": "chat-stream", + "model_id": "mimo-v2.5", + "provider_id": "opencode-go", + "streaming": true, + "upstream_surface": "openai_chat_completions" + }, + { + "client_surface": "messages", + "cross_surface": false, + "id": "messages-stream", + "model_id": "minimax-m3", + "provider_id": "opencode-go", + "streaming": true, + "upstream_surface": "anthropic_messages" + } + ], + "proxy": { + "reason": "no Q001 live proxy cell", + "status": "not-applicable" + }, + "redaction_review": { + "credentials_written_to_evidence": false, + "proxy_credentials_written_to_evidence": false, + "raw_response_bodies_persisted": false + }, + "runtime_status_http": 401, + "schema": "runtime-q007.v1", + "status": "pass" +} diff --git a/docs/agent-configuration.md b/docs/agent-configuration.md index 339140df2..9546fe1b4 100644 --- a/docs/agent-configuration.md +++ b/docs/agent-configuration.md @@ -87,7 +87,12 @@ Precedence: explicit `--base-url` wins, configured advertisement is next, and a `configremote` is read-only: it never creates/rotates server keys, mutates config/transcoding, refreshes catalogs, or requires the service to be running. If the server key is missing it reports the prerequisite instead of manufacturing one. Output is secret-free (`EGGPOOL_API_KEY` reference only); `--format token` prints the raw `epc1` token and `--format json` prints the stable `eggpool.configremote/v1` object. Default human output prints version-pinned desktop bootstrap commands (see below); `--shell posix|powershell|all` selects the rendered shell (`auto` prints both), and `--no-bootstrap` prints the profile only. -The token references `GET /api/integrations/v1/profile` (authenticated, revisioned) rather than embedding a model inventory, so it stays reusable as models change. +The token references `GET /api/integrations/v1/profile` (authenticated, +revisioned) rather than embedding a model inventory, so it stays reusable as +models change. Managed users see only models allowed by their policy in the +profile and `/v1/models`; dispatch enforces the same policy for concrete +models, virtual aliases, and provider-qualified targets. The profile ETag +includes the policy revision, so clients can refresh after an access change. ## Desktop Bootstrap @@ -202,12 +207,15 @@ wire_api = "responses" supports_websockets = false ``` -Set `EGGPOOL_API_KEY` to EggPool's server key in the environment used to run Codex. Retrieve the current key without putting it in the TOML with: +Set `EGGPOOL_API_KEY` to the user's personal token in the environment used to run Codex. Issue one locally and copy its one-time output with: ```bash -export EGGPOOL_API_KEY="$(eggpool getkey)" +eggpool users token USER ``` +`eggpool getkey` remains for existing legacy installations; avoid sharing that +global key with new users. + The default server port is `11300`; use `--base-url` when the server is configured differently. Generate a model-specific snippet with: ```bash @@ -228,9 +236,13 @@ EggPool relies on Codex local compaction by default and does not advertise remot Image/tool capability is never claimed across a heterogeneous alias route. Neither variant embeds the resolved server key, so OpenCode output prints by default like Codex. Set the variable in the environment that launches OpenCode: ```bash -export EGGPOOL_API_KEY="$(eggpool getkey)" +eggpool users token USER ``` +Copy the one-time personal token into the environment that launches OpenCode. +Existing single-owner installations may continue using `eggpool getkey` until +they explicitly migrate away from the legacy shared server key. + Managed OpenCode installation touches only the owned `eggpool` entry (plus its narrowly scoped parent key when EggPool creates it), preserving JSONC comments, trailing commas, indentation, ordering, unrelated providers, and nested settings byte-for-byte. A pre-existing `eggpool` entry is captured on first apply and restored exactly on `--remove`; an emptied parent key is removed only when it holds no comments. Global config lives at `~/.config/opencode/opencode.json` (`OPENCODE_CONFIG` overrides; Windows uses `%APPDATA%\opencode\opencode.json`). Ambiguous documents (both `provider` and `providers` present) and invalid JSONC fail closed with a bounded line/column location instead of a lossy rewrite. Provider-scoped model IDs are used when `models.collapse_models = false` (the default), so OpenCode can disambiguate providers serving the same upstream model. diff --git a/docs/api-reference.md b/docs/api-reference.md index bbf91bb9d..65ea78076 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -6,7 +6,8 @@ EggPool exposes OpenAI Chat Completions- and Anthropic Messages-compatible paths | Method | Path | Description | |--------|------|-------------| -| `GET` | `/v1/models` | List available models | +| `GET` | `/v1/models` | List models visible to the authenticated principal; managed users see only models allowed by their effective policy | +| `GET` | `/api/user-budget` | Authenticated user rules and bounded usage; admins may add `?user_id=` to inspect one user | | `POST` | `/v1/chat/completions` | OpenAI Chat Completions-compatible requests | | `POST` | `/v1/responses` | Stateless OpenAI Responses-compatible requests; canonical adaptation is allowed to eligible upstream surfaces | | `POST` | `/v1/responses/compact` | Bounded remote-compaction operation returning replacement history/checkpoint material; native compact-capable upstreams only, otherwise rejected before submission | @@ -101,13 +102,49 @@ Request ingestion is bounded by `[server].max_request_body_bytes` (default 10 Mi ## Authentication -Inference endpoints (`/v1/*`), the integration profile -(`/api/integrations/*`), and runtime/update/status endpoints -(`/api/stats/runtime`, `/api/stats/update`, `/api/status`) always require the -server API key whenever one is configured. Ordinary dashboard pages and their -non-sensitive JSON data are public while `[dashboard].public = true` (the -default) and require the key when it is `false`. The key is sent as -`Authorization: Bearer ` for OpenAI-compatible endpoints or -`x-api-key: ` for Anthropic-compatible endpoints. Health, readiness, and -static assets are unauthenticated, and a loopback-only install without a -configured key remains open for local development. +In `legacy_unmigrated`, the shared server key keeps its legacy behavior. After +local identity initialization, inference uses per-user agent tokens and +browser sessions; the legacy key is retired. A personal token grants only its +inference/discovery scopes and does not grant dashboard or account access. +Browser sessions authenticate the web surface, with CSRF proof required for +state-changing requests. In managed mode, `[dashboard].public` does not make +dashboard or `/api/*` data public. Health, readiness, login, and static assets +are unauthenticated. See [deployment](deployment.md) and [firewall](firewall.md) +for operator-managed HTTPS termination, trusted proxy, and loopback guidance. +Managed users receive policy-filtered model discovery, and runtime dispatch +enforces the same rules even when a client submits a hidden model manually. +The integration profile's ETag includes the policy revision so client caches +refresh after an administrator changes model access. + +`GET /api/user-budget` is available to personal tokens and browser sessions. +Members can read only their own rules and rollups; an active admin may select a +target with `user_id`. The response identifies UTC calendar windows and reports +integer microdollars as modeled/estimated usage, with separate +`provider_reported_microdollars`, `local_estimated_microdollars`, and +`reservation_fallback_microdollars` totals. Unknown usage consumes the +reserved estimate and increments an explicit unknown counter. Each rule carries +an `enforcement` mode: `hard` rules enforce a provable worst-case bound (token +rules require an explicit caller output cap; USD rules require an +operator-configured provider tariff) and deny otherwise, while `estimated` +rules use approximate holds and are never a guaranteed cap. This API does +not represent invoices or exact charges for flat-rate provider accounts. +Requests denied because a hard rule cannot be bounded return HTTP 429 with a +stable machine-readable message +(`User budget hard limit is not enforceable for this request`) before any +upstream submission. Administrator responses additionally include +`instance_rules` and `instance_usage`: the single instance-wide ceiling +intersected with every per-user rule. Members never receive instance fields, +so one member cannot observe the shared ceiling or infer other users' +consumption from it. + +With `[dashboard].enabled = true`, the dashboard also serves browser-session +pages for `/users`, `/profiles`, `/usage`, `/admin/audit`, `/my-account`, and +`/my-account/models`. Mutations use same-origin, CSRF-protected form posts +under `/admin/users/*`, `/admin/profiles/*`, and `/account/tokens*`. +Administrator pages and writes require an active admin browser session; +members can read only their own usage, model access, and credentials. Agent +tokens never authenticate these pages. A newly issued token is returned once +in a `no-store` response and omitted from the account page thereafter. +Profile edits use a version check so a stale form cannot overwrite a newer +edit. The audit view contains actor, action, target type, target ID, and time; +it does not record passwords, token values, prompts, or request bodies. diff --git a/docs/backup-restore.md b/docs/backup-restore.md index 24284578e..68d746cf1 100644 --- a/docs/backup-restore.md +++ b/docs/backup-restore.md @@ -48,6 +48,11 @@ EggPool ships with `eggpool backup` that bundles `config.toml`, `.env`, and the SQLite database (plus `-wal`/`-shm` if present) into a single timestamped `.zip` archive. +The database snapshot includes local users, budget rules and audit records, +active budget holds, and settled user/model usage. Restoring the snapshot +preserves the remaining budget state; crash reconciliation settles unknown +in-flight use conservatively before new requests resume. + ```bash # Default location: ~/backups/eggpool/ eggpool backup @@ -181,6 +186,30 @@ sudo systemctl start eggpool Migrations are idempotent and safe to run multiple times. +Schema migrations 55–62 add local downstream identity, browser sessions, +policy profiles, user budgets with enforceable hold/settlement state +(migration 61), instance-global budgets (migration 62), and administrative +audit tables. Backups made after those migrations include usernames, password hashes, personal-token +digests, session/CSRF digests, policy and budget state, bounded usage records, +and secret-free audit facts. They do not contain plaintext passwords, API +tokens, session cookies, or CSRF tokens. A backup taken while requests are in +flight captures their pending rows and active budget holds exactly as +committed; restoring such an image reconciles on next startup (pending +requests interrupt, active holds release without folding usage) exactly like +a crashed live database. After restoring an older backup, the +migrations are applied by `eggpool migrate` or normal server startup; existing +provider accounts and keys remain owned by TOML/environment as before. +A schema-54 image restores and migrates forward through 55–62 with legacy +requests, usage history, quarantine, and catalog state preserved; historical +shared-key requests stay unattributed (`legacy_shared_key`, never converted +to a fabricated human identity), and the upgraded database reports +`legacy_unmigrated` until an administrator bootstraps. +SQLite migrations are forward-only: an older executable may reject a database +that has reached schema 62. Roll back by restoring a protected backup made +before the upgrade together with the matching older binary; do not point an +older executable at the upgraded database. Capture the protected backup +before running `migrate`, not after. + ## Uninstall To remove EggPool entirely, run: diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 671719f21..17d32bc7a 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -12,7 +12,7 @@ Command list verified against `eggpool help` (0.8.2). | `eggpool stop` | Stop the running server | | `eggpool restart` | Fully restart the server (stop then start) | | `eggpool rehash` | Apply supported config changes live without restart (`--json` for structured output) | -| `eggpool onboard` | Interactive onboarding wizard | +| `eggpool onboard` | Interactive onboarding wizard; a fresh database prompts for an initial administrator | | `eggpool ensure-running` | Ensure server is running (no full runtime snapshot) | | `eggpool croncheck` | Fast-path cron watchdog check (no server connection) | @@ -54,9 +54,34 @@ Command list verified against `eggpool help` (0.8.2). | `eggpool edit` | Edit config in $EDITOR | | `eggpool getkey` | Print the server API key | | `eggpool newkey` | Generate and write a new server API key (`--show-old`, `--show-secrets`) | +| `eggpool users bootstrap` | Create the initial local administrator (`--password-stdin` for unattended input) | +| `eggpool users list` | List local usernames, role and status | +| `eggpool users token USER` | Issue a personal inference/discovery token (`--label` optional; `--show-secret` allows non-TTY output) | +| `eggpool users revoke USER` | Revoke all personal API tokens for a user | +| `eggpool users reset-password USER` | Reset a local password and revoke that user's API tokens and browser sessions (`--password-stdin` supported) | +| `eggpool users enable-managed` | Retire the legacy key path after clearing `[server].api_key`/`api_key_env` and setting `dashboard.public = false` | +| `eggpool users policy create ID NAME allow\|deny` | Create an exact-rule model profile | +| `eggpool users policy rule PROFILE MODEL allow\|deny` | Add or replace one exact model or `model/provider` rule | +| `eggpool users policy assign USER PROFILE` | Assign a profile to a local user | +| `eggpool users policy prefer PROFILE [MODEL]` | Set or clear a profile's allowed client recommendation | +| `eggpool users budget set USER --window day\|month` | Set daily/monthly request, token, microdollar, and concurrency limits (`--model-id` scopes an exact model; `--enforcement hard\|estimated`, default `estimated`) | +| `eggpool users budget remove USER --window day\|month` | Remove the matching user/model budget rule | +| `eggpool users global-budget set --window day\|month` | Set an instance-wide ceiling intersected with every per-user rule (`--model-id` scopes an exact model; `--enforcement hard\|estimated`, default `estimated`) | +| `eggpool users global-budget remove --window day\|month` | Remove the matching instance-wide ceiling | | `eggpool init-config` | Initialize config from template | | `eggpool dashboard public --on\|--off` | Toggle dashboard key requirement (default public) | +Policy commands remain available as a local operator setup path alongside the +admin dashboard. They write SQLite policy state directly; model IDs are exact +and wildcards are rejected. `--actor ADMIN` records the active administrator +responsible for each change. Restricted profiles should use `deny` as their +default and add explicit allowed models. + +Budget commands are local operator controls. At least one positive limit is +required; each rule is keyed by user, optional exact model, and UTC calendar +window. Cost limits apply to estimated microdollars, not provider invoices. +Rule changes are written to the SQLite audit table. + ## Maintenance | Command | Description | @@ -78,6 +103,15 @@ Command list verified against `eggpool help` (0.8.2). Full deploy commands reference: [deployment.md](deployment.md#deploy-commands-reference) +Personal tokens are stored as digests and can authenticate inference, +`/v1/models`, and the integration profile; they do not grant dashboard or +administrative access. The shared `[server].api_key` authenticates only legacy +`legacy_unmigrated` installations and stops working after local identity +initialization. Browser sessions use HttpOnly cookies and protect logout and +password changes with same-origin checks plus a session-bound CSRF token. +Passwords are never accepted as command-line arguments. Local CLI access is +the recovery authority; password reset revokes API credentials and sessions. + ## Agents & updates | Command | Description | diff --git a/docs/deployment.md b/docs/deployment.md index 1f5955b6f..2ede58d41 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -46,7 +46,47 @@ command is committed and verified, without overwriting an existing file, and prints the resolved config path. After install, `eggpool onboard` walks the operator through provider -connections, configuration validation, and an optional server start. +connections, configuration validation, and an optional server start. On a +new database it also creates the initial local administrator with a +non-default password and prints one personal API token once. For unattended +bootstrap, pass `--password-stdin` and provide the password on stdin; the +password is never accepted in argv. Secret output is suppressed when stdout +is not a terminal unless `--show-secret` is explicitly passed. Existing databases keep their legacy +authentication behavior until an operator explicitly initializes local user +access. + +Manage local identities from the machine that owns the data directory with +`eggpool users list`, `eggpool users token USER`, `eggpool users revoke USER`, +and `eggpool users reset-password USER`. Personal tokens are for inference, +model discovery, and the integration profile; they do not authorize dashboard +or admin pages. The configured server API key remains a legacy shared +inference credential during the staged migration. Do not share it with new +users; issue an individual token instead. + +With `[dashboard].enabled = true`, an administrator can manage users, profiles, +budgets, tokens, and audit history in the embedded dashboard. New members +receive a temporary password and must change it at first login. Members can +manage their personal tokens and view only their own usage and accessible +models. A browser-issued token is shown once in a no-store response; set it as +`EGGPOOL_API_KEY` in the environment that launches the desktop client. +Remote browser login must use the operator's TLS proxy and trusted-proxy setup +below: LAN reachability alone does not encrypt passwords or cookies. + +Budget rules are `estimated` by default. Promoting a rule to `hard` requires +proof the proxy can actually bound: hard token rules need clients to send an +explicit output cap, and hard USD rules need a conservative operator tariff on +every provider those users can reach, configured as +`[providers..budget] max_microdollars_per_million_tokens` (microdollars +per million tokens, covering the most expensive model on that provider; +omit for unknown price, `0` for verified free). Without a bound, hard-mode +requests are denied before upstream dispatch — hard mode is unavailable rather +than approximate. See [model limits](model-limits.md#local-user-budgets). + +An optional instance-wide ceiling (`eggpool users global-budget set` or the +dashboard instance section) caps total consumption across all users and is +intersected with every per-user rule. It covers unattributed legacy-key +requests as well, survives user disable/delete, and needs no second database +or writer. See [model limits](model-limits.md#instance-budget-ceilings). ### Updating EggPool @@ -390,11 +430,29 @@ single-event-loop default (`threads = 1`). `eggpool onboard` keeps any explicit operator bind address and never rewrites it to loopback; newly initialized configs carry the canonical `0.0.0.0` LAN -bind. The server API key generated during onboarding still guards inference -(`/v1/*`), integration (`/api/integrations/*`), and runtime/update/status -endpoints even though the human-facing dashboard is public by default. Use -`eggpool dashboard public --off` to require the API key on ordinary dashboard -pages and data as well. +bind. Before local identity initialization, the configured server API key +retains legacy behavior and the public read-only dashboard setting applies. +After the first local administrator is created, the shared key is no longer +accepted; personal tokens authenticate agent surfaces, while dashboard and +admin routes require a browser session regardless of `dashboard.public`. + +For managed operation, remove `[server].api_key` and unset its configured +environment variable, then set `dashboard.public = false` and run +`eggpool users enable-managed`. That command refuses the transition while the +shared key or public-dashboard setting remains. Non-loopback listeners may +then run without a shared server key because protected routes require a +personal token or browser session. + +Browser password login is allowed over loopback HTTP. For remote browser use, +terminate TLS at an operator-managed proxy and add the proxy's literal peer IP +to `[security].trusted_proxies`; EggPool accepts only a single +`X-Forwarded-Proto: https` from that peer. Browser sign-in and failed personal- +token throttling use one literal client IP in `X-Forwarded-For` from the +trusted peer; the proxy must replace this header, and missing, duplicate, or +chained values use the proxy peer as the throttle identity. The proxy must preserve the public +Host header and the firewall must prevent direct access around the proxy. The +optional `[security].allow_insecure_web_auth = true` switch is for isolated +development only and emits a startup warning. Optional background features that remain explicit: set `[backup].enabled = true` for in-process archives or `[update_checker].enabled = true` for the periodic @@ -562,7 +620,7 @@ sudo nano /etc/eggpool/env # # [server] # host = "0.0.0.0" # canonical LAN bind (set "127.0.0.1" for loopback-only) -# port = 11300 # a server API key is required for non-loopback binds +# port = 11300 # legacy installs need a key; managed installs use per-user credentials # # [database] # path = "/var/lib/eggpool/usage.sqlite3" @@ -837,14 +895,17 @@ sudo journalctl -u eggpool -n 50 --no-pager ### Cannot connect from LAN 1. Verify that the explicit config selected `server.host = "0.0.0.0"` (or - another non-loopback address) and configured `[server].api_key` or - `[server].api_key_env`. + another non-loopback address). Legacy installs need `[server].api_key` or + `[server].api_key_env`; managed installs authenticate with personal tokens + and browser sessions. 2. Check firewall: `ss -tlnp | grep 11300` 3. See `docs/firewall.md` -The copyable SBC profile stays on loopback by default. Shared startup and -`check-config` validation reject LAN or wildcard binding without the existing -simple server API key; loopback-only development may remain unauthenticated. +The copyable SBC profile stays on loopback by default. A non-loopback legacy +install requires the shared server key. After identity initialization, a +keyless listener is allowed, but protected routes still require personal +credentials or a browser session; unauthenticated loopback behavior is limited +to the explicit legacy state. ### Headless server + desktop workgroup diff --git a/docs/firewall.md b/docs/firewall.md index fee18a29d..aab8a9303 100644 --- a/docs/firewall.md +++ b/docs/firewall.md @@ -67,6 +67,27 @@ host = "127.0.0.1" No firewall rules are needed in this case. +## Browser administration + +Legacy inference clients use the shared server key only while the database is +`legacy_unmigrated`. After local identity initialization, EggPool retires that +key and requires personal API tokens for inference and browser sessions for +dashboard administration. Remote browser password login must reach EggPool +through operator-managed HTTPS termination. Add the proxy's exact peer IP to +`[security].trusted_proxies`; only that peer's single +`X-Forwarded-Proto: https` header is trusted. Browser sign-in and failed +personal-token throttling use one literal `X-Forwarded-For` client IP only +from that trusted peer; the proxy must replace the incoming header with the +client's IP. Missing, duplicated, or chained values fall back to the proxy peer. The proxy should +preserve the public `Host` header, and the firewall should allow port 11300 +only from that proxy. Do not expose the backend port directly to clients when +trusting a proxy. + +Loopback HTTP sign-in is supported for local administration. The optional +`[security].allow_insecure_web_auth = true` setting permits remote HTTP only +for isolated development and logs a warning because passwords and session +cookies then travel without TLS. + ## Verify ```bash diff --git a/docs/model-limits.md b/docs/model-limits.md index 841aff0cb..8a5c054a8 100644 --- a/docs/model-limits.md +++ b/docs/model-limits.md @@ -116,3 +116,60 @@ focused on routing availability, usage, cost, and pricing exactness so the default Models view stays scannable. Pricing exactness follows the same request-cost labels used elsewhere in EggPool: `provider_reported`, `exact`, `derived`, `partial`, `estimated`, and `unknown`. + +## Local User Budgets + +Per-user request, token, microdollar, and concurrent request budgets are +separate from these model context/output limits. Operators manage rules with +`eggpool users budget set` (`--enforcement hard|estimated`, default +`estimated`); daily and monthly periods use UTC calendar windows. A hold is +reserved atomically before provider dispatch, and retries extend the same +logical request hold. User usage distinguishes provider-reported cost, local +estimates, and conservative reservation fallback. + +Each rule is explicitly `hard` or `estimated`, visible in the dashboard, +`GET /api/user-budget`, and the stored audit history. Pre-existing rules are +`estimated` and stay that way until an administrator explicitly promotes them; +nothing is silently upgraded to a guarantee. + +- `estimated` (default): token holds use a bounded request-size estimate and + microdollar holds use EggPool's estimated pricing projection, including + fallback rates when no model price is known. These are operational + approximations, not an exact charge or invoice for flat-rate providers, and + not a hard ceiling. +- `hard`: token rules admit only requests carrying an explicit caller output + cap compatible with the request surface, and reserve a conservative + worst-case bound (admission input bound plus that cap, covering the concrete + attempt plus selector overhead; retries extend per attempt after + re-checking). USD rules additionally require an operator-configured + conservative tariff (`[providers..budget] + max_microdollars_per_million_tokens`); unpriced providers fail closed, an + explicit zero tariff means verified free, and estimator fallbacks are never + a hard price. Requests that cannot be bounded — unbounded output, unknown + tariff, arithmetic overflow — are denied with a stable 429 + (`User budget hard limit is not enforceable for this request`) before any + upstream submission. Observed provider usage above a hard hold is settled + visibly, never clamped. + +See the [API reference](api-reference.md#authentication) for the +authenticated `/api/user-budget` view. + +## Instance Budget Ceilings + +One optional instance-wide ceiling is intersected with every per-user rule: +an admitted call must satisfy all applicable instance-global, instance-model, +user-global, and user-model limits. Manage ceilings with +`eggpool users global-budget set` (same windows, modes, and `--model-id` +scoping; a blank model covers all models) or the admin dashboard's instance +section. Absent ceilings mean no additional cap, and single-admin behavior is +otherwise unchanged. + +The shared ceiling is enforced atomically: concurrent requests from different +users cannot both take the final global slot, and no retry, alternate model, +identity switch, or legacy-key request bypasses it — unattributed requests +count against it too. Each logical request counts once across failover and +retries; hard/estimated semantics and the operator-tariff trust boundary match +the per-user contract above. Global counters never depend on a user row, so +disabling or deleting a user preserves history and remaining headroom. Members +see only their own rules and usage; instance ceilings and aggregate usage are +admin-visible only. diff --git a/docs/model-routing.md b/docs/model-routing.md index 52760b9b1..bafce8dc6 100644 --- a/docs/model-routing.md +++ b/docs/model-routing.md @@ -136,6 +136,28 @@ process restart. A safe rehash preserves entries only when the router's semantic fingerprint is unchanged; changing the selector, default, route target/description, or affinity policy causes the next request to reclassify. +## Per-user model authorization + +Managed identities are checked against a versioned SQLite policy snapshot +before provider/account routing. Policies use exact canonical model IDs and +exact provider-qualified IDs in `model/provider` form. Instance-wide hard +denials always win, then per-user overrides, profile rules, and the profile +default. Restricted profiles default-deny models without an explicit allow; +an administrator without an assigned profile retains unrestricted access. + +For a virtual alias, EggPool checks the alias, selector model, and resolved +concrete route against the same snapshot. A selector, repair result, sticky +affinity entry, provider qualifier, or compact Responses call cannot bypass a +concrete-model denial. A denied request returns a generic 403 before the +concrete model is submitted. + +`GET /v1/models` and `GET /api/integrations/v1/profile` return the catalog +allowed for the authenticated user. The integration profile's HTTP ETag also +contains the effective policy revision, so policy edits invalidate caches +even when the visible model list stays the same. A stored preferred model is +setup metadata. Inference authorizes the model a client explicitly requests +and does not silently rewrite that request. + ## Operations and troubleshooting The complete `[model_routers.*]` mapping is live-reloadable as one atomic field. diff --git a/docs/raspberry-pi.md b/docs/raspberry-pi.md index 6e56b48de..77be7dd11 100644 --- a/docs/raspberry-pi.md +++ b/docs/raspberry-pi.md @@ -234,6 +234,44 @@ scalars, and record unavailable hardware/tmpfs dimensions as `not measured`. This mode is diagnostic-only and does not justify a production durability or publication change by itself. +### On-prem small-team envelope (M012, Pi 5 managed) + +On-prem milestone M012 qualified one managed Pi 5 as a solo/small-team +envelope (closure `plans/closure/on-prem-user-access/012-status.md`). Target +frozen at measurement: Raspberry Pi 5 Model B Rev 1.0, 4 cores, 8 GB RAM, +Ubuntu 24.04.4, kernel 6.8.0-1065-raspi, ext4 on MMC (`mmcblk0p2`), release +`0.8.2` built on-device (SHA-256 `8609119a…1538`, 34.9 MB, schema 62, +WAL/NORMAL, one database worker, `threads = 1`). Inference and catalog use a +synthetic administrator personal token; the shared server key is rejected for +those surfaces (managed fencing), and `runtime-status` CLI plus the dashboard +shell require a browser session (expected; full runtime projection remains +available there per M011). + +Managed functional plus three fresh-root benchmark runs (30 samples, +concurrency 4, 32-request batch) all pass with secret-free aggregates in +`artifacts/qualification/on-prem-m012-pi5-functional.json` and +`on-prem-m012-pi5-benchmark-run-{1,2,3}.json`: 25 functional cells, 145 +requests per benchmark run with zero pending/active reservations, backup, +recovery, restart, and shutdown green. Sequential p50 is about 5 ms +(finite/streaming) and 13–15 ms concurrent; streaming TTFT p95 is 4–19 ms. +MMC tails reach seconds (concurrent p95 up to about 14 s, maxima similar) with +no OOM, swap, drops, or WAL divergence; RSS stays about 20 MB baseline to +20.8 MB peak. Argon2id logins (19 MiB, t=2, p=1, two slots) measure about +51 ms p50 one-way idle and 71–73 ms two-way, unchanged under gentle inference +load. Provisioning reaches 49 named users (one admin plus 48 members) at about +51 ms p50 each with all spot logins passing; five personal tokens issue +cleanly. Bounded concurrency 1/2/4/8 completes (1–2 at p50 5–6 ms with +second-scale tails; 8 saturates at p50 about 4.1 s). + +Deploy accordingly: solo personal use is practical in one process with no +extra services; up to 2–4 concurrent requests work with degraded tails; 8 +concurrent exceeds this Pi/MMC class. Forty-nine named users are inexpensive +metadata, not 49 simultaneous generations — no 50-concurrent claim is made. +Le Potato and a separate ordinary-Linux reference host were not measured here +(docs promise Pi 4/Pi 5 only; paired same-Pi controls ground the solo +deltas); 16-way concurrency was not attempted because 8-way already +saturates. + ## Verify from LAN 1. Find Pi IP: `hostname -I` diff --git a/docs/rust-dashboard-qualification.md b/docs/rust-dashboard-qualification.md index e975d01da..e9d589a28 100644 --- a/docs/rust-dashboard-qualification.md +++ b/docs/rust-dashboard-qualification.md @@ -17,6 +17,41 @@ static bytes/content types, theme inventory, and referenced assets. Use `--skip-build` when `rust/target/debug/eggpool` already represents the source under review. +## Split oracle contract + +Dashboard qualification has two references, and they are not interchangeable. + +The historical M001-M012 pages are compared against the frozen Python oracle +pinned at `ORACLE_COMMIT`. That oracle only ever implemented those pages, so it +can never serve as the reference for the on-prem admin and member pages that +were added afterwards. + +The on-prem pages are therefore owned by EggPool itself. `--on-prem` starts one +managed instance with a bootstrapped administrator, drives a real browser +through both roles, and compares the result against a Rust-owned approved +reference at `tests/fixtures/dashboard-on-prem-browser/reference.json`. + +```text +uv run python scripts/qualification_dashboard_parity.py --skip-build --on-prem +``` + +That gate covers sign-in with and without JavaScript, administrator profile and +user creation, suspend and reactivate, model override and budget editing, +one-time personal token reveal, forced password change at first member login, +member-only pages, and the negative matrix for anonymous, member, and personal +agent credentials. Every admin and member page is checked at desktop, narrow +mobile, and HiDPI viewports, and across every bundled theme. Each observation +fails on duplicate IDs, horizontal overflow, an unbounded table, an unlabelled +control, a missing viewport declaration, or any planted credential marker +reaching the DOM. + +Re-freeze the reference only after reviewing the diff: + +```text +uv run python scripts/qualification_dashboard_parity.py --skip-build --on-prem \ + --write-on-prem-reference +``` + ## Browser review procedure Use an existing local Chrome, Firefox, or equivalent browser against the diff --git a/plans/000-long-term-specification.md b/plans/000-long-term-specification.md index 8d1506901..983465980 100644 --- a/plans/000-long-term-specification.md +++ b/plans/000-long-term-specification.md @@ -29,6 +29,17 @@ including `/v1/responses/compact`), and Anthropic Messages executable plus metadata/assets only. Repo-root `pyproject.toml` is tooling-only. +EggPool's canonical operating profile is a single on-prem instance on SBC or ordinary +local hardware, administered by one owner. An optional local-users mode serves +small teams (target: fewer than approximately 50 named users) sharing +operator-controlled routing to authorized provider APIs/subscriptions and +self-hosted models. This is not a SaaS, enterprise IAM, or multi-organization +hosting product; a 50-user capacity claim requires measured target evidence. +Provider topology, upstream credentials, and routing configuration remain in +operator-owned TOML/environment files; human identity, access profiles, +sessions, individual credentials, user quotas, and usage statistics belong in +SQLite. See accepted `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md`. + ## 2. End-state invariants (normative) 1. **Generation-owned execution.** All inference work runs under a @@ -70,6 +81,16 @@ tooling-only. test; it keeps direct/non-SSH proxy paths and rejects SSH proxy config as `TransportError::ProxyConfiguration` before dialing. +11. **Single-instance local authority.** One installation has one administrative + domain. A first-run local admin bootstrap must not expose a shared default + password or allow remote creation of the first administrator; existing + installations require an explicit compatibility/migration path. +12. **Identity-bound policy and accounting.** Inference authorization attaches a + trustworthy user principal at ingress and preserves it through model + aliases, concrete selection, retries, streaming, publication, and finalization. + User budget enforcement is separate from provider-account quota scoring; + access denial is never expressed as upstream provider unhealthiness. + ## 3. Ownership boundaries (normative) - `runtime.rs` adapts CLI to operations; reusable lifecycle lives in @@ -84,6 +105,11 @@ tooling-only. compatibility surfaces; compact execution may use a private single-owner admission representation internally. +- Upstream `accounts` and account quotas describe provider credentials/capacity, + never downstream people. Durable human identities, policy versions, browser + sessions, client credentials, and user usage are separate database-owned facts. + Provider TOML and local admin access must not be writable by ordinary users. + ## 4. Protocol and compatibility EggPool serves OpenAI-compatible Chat Completions, Responses (finite, @@ -101,6 +127,9 @@ narrowly scoped change. ## 6. Non-goals +No hosted organization/tenant service, public self-registration, built-in VPN, +built-in internet exposure, remote IAM/SSO requirement, or billing service. +Off-LAN access is an operator-owned TLS reverse-proxy/VPN concern. No second HTTP abstraction or retry owner; no new restart/reload key lists; no buffering of arbitrary native streams; no EggPool SSH executor fallback (the exact-pinned Eggress outbound crate owns proxy transport); no Windows proxy support diff --git a/plans/001-terminology-and-domain-model.md b/plans/001-terminology-and-domain-model.md index e11e64cb9..d488aea84 100644 --- a/plans/001-terminology-and-domain-model.md +++ b/plans/001-terminology-and-domain-model.md @@ -10,6 +10,25 @@ this document is normative. ## Identity and lifecycle terms +- **Instance** — one on-prem EggPool process/database/control domain with one + operator-administered provider configuration, not a hosted SaaS tenant. +- **User** — a downstream human identity using EggPool; never equivalent to an + upstream provider account. The initial user is the local administrator. +- **Provider account** — an upstream configured provider credential and its + health, scoring, and provider-quota state (`accounts` table). +- **Principal** — authenticated downstream request identity (user ID and + optional client credential ID); never trusted from caller-supplied identity headers. +- **Client credential** — revocable per-user secret for agent/API access, distinct + from a human password, browser session, and upstream provider secret. +- **Profile** — named reusable downstream model-authorization, preferred-model, + and usage-budget policy assigned to a user; not an upstream provider profile. +- **Effective user policy** — versioned intersection of instance restrictions, + user overrides, and the assigned profile. It governs concrete/virtual models + and user budgets; it is distinct from provider quota and health state. +- **User budget reservation** — durable pre-dispatch hold against an enforceable + user/model allowance, settled or compensated once for a logical request. + + - **RuntimeGeneration** — an immutable, atomically published snapshot of all runtime-owned services for one configuration epoch (`rust/src/runtime_lifecycle/`). Published by `RuntimeManager`; retired diff --git a/plans/002-long-term-roadmap.md b/plans/002-long-term-roadmap.md index 7eec4bcef..341c96a2f 100644 --- a/plans/002-long-term-roadmap.md +++ b/plans/002-long-term-roadmap.md @@ -78,6 +78,31 @@ changes. IO stays in `operations/integrations.rs`; helper never implies Windows proxy support. -Phases 2–4 create NO automatic milestones. Subsystem roadmaps are written on +## Phase 5 — Optional on-prem small-team access and budgeting + +Objective: add identity-bound use of a single locally administered EggPool +instance without regressing the single-owner SBC and LAN deployment. +Authorized clients may route to operator-configured hosted or self-hosted LLMs; +all provider secrets/topology remain under TOML/environment ownership. + +- Authority: `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` + and `plans/subsystems/on-prem-user-access-roadmap.md`. +- Order: secure administrator bootstrap and user credentials; browser-session + and route authorization; principal propagation/attribution; model-policy + profiles; durable user budgets; web administration/client projection; small-team + physical hardware and migration qualification. +- Integration edges: preserve generation leases, publication/finalization + ownership, provider quota eligibility, single SQLite gate/WAL/NORMAL, and + compatible Chat/Responses/Messages client authentication. +- Non-goals: multi-organization/tenant hosted management, SaaS billing, + mandatory cloud/SSO, native public internet access, and unqualified claims + about 50 concurrent generations. +- Completion: legacy upgrade preserves data and access; fresh bootstrap is + fail closed without defaults; user policy cannot be bypassed via virtual + routing or alternate surfaces; per-user budgets recover across cancellation/ + restart; measured SBC and ordinary-hardware evidence scopes supported + team size and concurrent workload honestly. + +Phases 2–5 create NO automatic milestones. Subsystem roadmaps are written on demand when a slice is ready to be reasoned about, with typed dependencies (hard / interface / soft / operational) and explicit non-goals. diff --git a/plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md b/plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md new file mode 100644 index 000000000..81bc86836 --- /dev/null +++ b/plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md @@ -0,0 +1,201 @@ +# ADR-0002: On-prem user identity, policy, and ownership boundaries + +Status: accepted + +Date: 2026-10-13 + +Decision owners: project maintainers (explicit product direction, 2026-10-13) + +Related specification sections: + +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/000-long-term-specification.md#3-ownership-boundaries-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Affected subsystem roadmaps: + +- `plans/subsystems/on-prem-user-access-roadmap.md` + +## Context + +EggPool is a native Rust LAN provider router, not hosted SaaS. The canonical +operator is one owner on SBC/local hardware, with several TOML-configured +upstream provider accounts and optional operator-hosted private LLMs. The +desired extension is a low-friction single-instance installation supporting a +small team (roughly fewer than 50 named humans, **not** 50 simultaneous +inferences without evidence). Administrator-created users need browser login, +revocable per-person agent tokens, model access rules, reusable named profiles, +and per-user/per-model consumption limits. + +Today `rust/src/server/middleware.rs` verifies one `server.api_key` or +`x-api-key`, `DashboardConfig.public` defaults true, `/v1/models` and +`/api/integrations/v1/profile` project global models, `accounts` represent +upstream credentials, and `quota/ + routing/` score provider-account capacity. +`coordinator/publication.rs` and `finalization.rs` persist and settle +provider-bound requests without downstream user identity. A login page and +dashboard-side usage counters would not establish tenant isolation or hard +usage limits. + +## Decision drivers + +- Keep the one-user SBC/LAN onboarding and inference path canonical and small. +- Avoid public registration, multi-organization infrastructure, cloud + dependencies, enterprise IAM, built-in VPN, and billing. +- Respect local operators' provider choices, including their own privacy- + sensitive self-hosted LLMs; do not require catalog cloud calls for identity. +- Keep upstream provider accounts/keys/topology and routing configuration in + TOML/environment; keep human/user state, policies, sessions, audit and usage + in the SQLite database. +- Preserve client wire, provider health/fairness, generation leases, SQLite + single gate/worker, streaming, compensation, and existing deployments. +- Treat licensing/terms of upstream consumer subscriptions as operator + responsibility; no claim that personal plans permit team sharing. + +## Considered options + +### A — Separate team-mode daemon or external identity service + +Extra deployable components and distributed configuration add maintenance, +upgrade, credential handling, and SBC overhead. Rejected. + +### B — Keep one global ingress key, add usernames only to the dashboard + +A stolen/shared key has no accountable caller identity. Aliases, retries, +model discovery, and budgets cannot be enforced per person. Rejected. + +### C — One instance, mandatory initial local administrator, optional users + +Use one identity/authorization engine: one admin is the simple base case, +additional users and profiles enable managed small-team behavior. Separate +human browser authentication from per-user agent/API tokens. Selected. + +## Decision + +1. **Administrative domain and storage.** One instance/database and one + operator-owned TOML provider/routing authority. The database owns local + `users`, password hashes, browser sessions, client credential digests, + profiles, overrides, user budget reservations, usage and audit. Never + repurpose `accounts` or `account_models` for downstream people. No + per-organization `tenant_id`, service mesh, hosted control plane or + separate database per person. +2. **Bootstrap.** New interactive onboarding creates one administrator from a + username and non-default password via local CLI/TTY; unattended setup uses + protected stdin/FD rather than argv/environment. There is **no** + `admin/admin`, network bootstrap, silent fallback, or unauthenticated admin + endpoint. Initialization is atomic and protected by local filesystem + ownership. The CLI provides controlled local recovery. +3. **Compatibility.** Legacy installations and pre-existing server keys must + have an explicit upgrade path; never silently lock out unattended systemd + users. New auth cutover may associate a legacy key with the initialized + admin for inference-only compatibility, but may not grant web-admin access + or attribute multi-user traffic under that shared key. Before adding a + second user, retire or disable the shared legacy key. Managed mode never + accepts unauthenticated loopback inference or `dashboard.public` + exemptions for private/admin data. +4. **Two credential classes.** Local password+opaque, revocable server-side + browser session is for human UI. Per-user, revocable high-entropy bearer/ + `x-api-key` tokens are for inference/model discovery/profile, with explicit + scopes. Human password and cookie are not inference credentials; agent + credentials are never administrative browser sessions. Existing OpenAI + compatible wire and integration env-key conventions continue. +5. **Authentication and authorization.** Ingress resolves a typed principal + from verified credentials; never accept a caller-supplied user ID as + authority. All private/admin endpoints require role/scope checks and web + mutation CSRF protection. Effective model policies intersect instance hard + restrictions with a profile and optional admin-set per-user overrides. + Explicit requested and eventual concrete provider-qualified targets, + virtual routes, selector calls, compact and every inference surface are + covered. Unauthorized models are not exposed in per-user discovery. +6. **Budgets.** Provider capacity/quota scoring remains provider-owned. A + separate user/model budget authority has atomic pre-dispatch admission, + durable reservations, exactly-once settlement/compensation semantics and + request attribution across retries/streaming/recovery. Local model use can + be metered in tokens/requests independently of USD. Missing pricing + cannot silently count as zero under a hard USD limit. +7. **Administration.** Only admins create/disable users, modify profiles, + grants and budgets. A member may manage their password and their own + scoped tokens and view only authorized catalogs/own usage. Provider + credentials and TOML routing remain local operator functions, not mutable + member web UI state. Admin UI extends embedded Rust dashboard assets; + no required SPA, JavaScript framework, external auth/DB server. +8. **Network boundary.** No native internet access stack. Loopback is the + lowest-friction local case. Multi-user LAN browser credentials should use + HTTPS through an operator-managed reverse proxy or VPN; plain HTTP LAN + credentials are risky and require an explicit, documented security posture. + No implicit trust of spoofable forwarding/identity headers. +9. **Performance target.** Design/test for approximately 1–49 named users + on reasonable local hardware, keeping the single-user path fast. Bound + caches, hash verification concurrency, DB write amplification, idle RSS + and login cost on SBCs. Publish no concurrency capacity claim without + measured representative traffic on the actual target class. + +## Consequences + +### Positive + +- Minimal deployment unit; no external authentication or database dependency. +- Accountable per-user access and differentiated frontier-model consumption. +- Existing on-prem provider routing, including private LLMs, remains intact. +- Shared admin workflow without provider credentials crossing into the user DB. + +### Negative + +- Security-critical ingress, dashboard and lifecycle changes require coordinated + migrations, backup semantics, failure recovery and extensive negative tests. +- Strong user budget enforcement increases admission/settlement SQLite work. +- HTTPS on LAN requires operator reverse-proxy configuration unless separate + local-only access is chosen; plaintext LAN use is not silently safe. + +### Neutral or deferred + +- No SSO, invite/email workflow, cloud discovery, chargeback/billing, complex + nested roles, multi-instance synchronization or native TLS frontend. +- Fine-grained organization hierarchies and cluster leases are out of scope. +- Named users are a capacity target, not a limit enforced by fiat. + +## Compatibility and migration + +Append checksum-verified SQLite migrations, preserve source-schema fixtures and +historical rows, and explicitly distinguish legacy-unattributed usage from +new authenticated requests. Preserve existing `[server].api_key` behavior +until an operator-controlled upgrade cutover. Separate the ability to migrate +old daemon installs unattended from permission to add users. Do not move +`providers.*.accounts` or upstream credentials into the database. +`configsetup`, `configremote`, `eggpool-connect`, `newkey/getkey`, and +`/api/integrations/v1/profile` require scoped compatibility tests and a +deprecation path for globally shared keys. + +## Security and reliability implications + +Require Argon2id password hashes calibrated for constrained SBCs, strict +bounds on simultaneous memory-hard authentication work, high-entropy random +tokens with non-recoverable at-rest digests, session rotation, CSRF protection, +constant-time comparisons, rate limiting, SameSite/HttpOnly/Secure cookies as +applicable, restrictive DB/backup modes and redacted observability. Auth +failure is closed, not the existing unauthenticated-loopback success path. +Explicitly define principal/policy snapshot semantics across reload, account +disable and mid-stream revocation. Reserve and compensate user budgets in +the coordinator's durable ownership lifecycle; never hold the DB gate during +provider I/O. Avoid adding separate producer DB writers or a per-request +external identity dependency. + +## Verification + +- Fresh admin bootstrap, one-user local use, headless stdin, and legacy + systemd/backup upgrade paths; no default/remote bootstrap credential. +- Browser sessions, cookies, CSRF, role and cross-user endpoint attacks. +- Shared-key retirement and revocation race, token hashing/no exfiltration. +- Unauthorized concrete/provider-qualified/virtual/semantic/compact models + cannot be dispatched; all client surface and discovery projections match. +- Concurrent user/model hard caps, retries, cancellation, streaming terminal, + double finalization, crash/restart and stale-policy behavior. +- Pi/Le Potato-class and ordinary-hardware measured single-user overhead, + 1/5/10/25/49 named-account workloads, and bounded concurrent requests. +- Full default and no-default serial test/lint matrix, migrations/checksums, + backup/restore, restart/rehash and docs/ops qualification. + +## Supersession + +None. diff --git a/plans/closure/on-prem-concurrency-performance/001-status.md b/plans/closure/on-prem-concurrency-performance/001-status.md new file mode 100644 index 000000000..2672baa2a --- /dev/null +++ b/plans/closure/on-prem-concurrency-performance/001-status.md @@ -0,0 +1,149 @@ +# On-Prem Concurrency Performance M001 — Closure Status + +Status: conditionally closed — durable database storage on the Pi 5 microSD/MMC path is the dominant measured tail contributor; remaining workload and external-I/O controls are named below + +Source implementation plan: + +- `plans/implementation/on-prem-concurrency-performance/001-pi5-mmc-tail-latency-attribution.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-concurrency-performance-roadmap.md#milestone-001--pi-5mmc-concurrent-tail-attribution` + +Repository baseline reviewed: `d2c0830acaeff90c45bffa393453cf0500c4fa64` + +Implementation commits or pull requests: + +- `03317052` — paired physical MMC/tmpfs qualification corpus and artifact guard, plus M001 closure/registry reconciliation; no Rust runtime or database-policy changes. + +## 1. Executive finding + +M001 has a reproducible physical attribution for the existing Pi 5 four-way finite-request tail. Across three fresh-root runs on the same Pi 5, release candidate, fixture, and 32-request/four-client workload, root ext4/MMC produced a median concurrent p95/max of **8,579 ms**. Moving the qualification temporary root, including its SQLite database, to RAM-backed `/dev/shm` reduced the corresponding median p95 to **10 ms** and median maximum to **11 ms**. Every run completed 32/32 requests and converged with zero pending requests and active reservations. + +This agrees with the independent Plan 237 direct-provider and finite-phase evidence: provider service remained 0–1 ms, slow requests were pre-provider dominated, and the 60-request tmpfs control reached 4 ms maximum versus 1,785–3,473 ms on MMC. Plan 239 independently correlated the multi-second request with foreground publication `COMMIT` and WAL checkpoint-sequence changes. Taken together, the tested tail is owned by durable publication/SQLite/storage work on this Pi 5/MMC path, not provider service. + +Plan 238 independently moved only the SQLite database/WAL/SHM to its temporary filesystem: its three MMC diagnostics had 3,331–4,944 ms maxima while all three database-only tmpfs controls had 4 ms maximum, with background-task contamination ruled out. This isolates the database storage path while leaving the qualification root on MMC. + +The target is the Raspberry Pi 5 microSD boot device, not NVMe. The original six-run corpus used `/dev/mmcblk0p2` on the Pi's SD device; a read-only host check reports `/sys/block/mmcblk0/device/type` as `SD`. This is the intended low-cost SBC storage class and is the device on which the paired MMC/tmpfs attribution was measured. SSD/NVMe is not a closure requirement. An additional microSD qualification smoke run completed with four concurrent requests, five managed principals, and hard user/global request budgets enabled; all four requests returned HTTP 200 and durable state converged. Its four timing observations are functional smoke evidence only, not a latency estimate or capacity claim. A separate NVMe run remains exploratory and out of scope. M001's remaining gaps are controlled competing-I/O and the broader concurrency/principal/budget matrix; these limit broader workload claims but do not create a requirement to test alternate persistent media. No new capacity claim, durable-tmpfs recommendation, or SQLite topology/journal/checkpoint/concurrency-policy change is authorized. Existing restricted Pi guidance remains authoritative. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Freeze physical target, candidate, workload, and G3 gates | `artifacts/qualification/on-prem-concurrency-performance/matrix.json`; six raw Q008 reports | pass | Raspberry Pi 5 Model B Rev 1.0, Ubuntu 24.04.4, Linux/aarch64, kernel 6.8.0-1065-raspi; root is ext4 on `/dev/mmcblk0p2`, the MMC device identifies as SD; candidate SHA-256 `8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538`; preserved G3 p95 ≤1,000 ms and max ≤3,000 ms. | +| Ordinary physical qualification before benchmarks | `functional.json` | pass | Q008 v1, fresh temporary root, all functional checks passed. | +| Three paired MMC and tmpfs runs | `mmc-run-1..3.json`, `tmpfs-run-1..3.json`; matrix guard | pass | Same release binary, fixture, 30-sample sequential phases, 32-request four-way batch, and device. The tmpfs variable changes the temporary root and DB placement; output artifacts stay on MMC. | +| Reproduce the MMC tail without relabeling a failure | MMC artifacts and `test_m001_storage_control_reproduces_and_removes_the_g3_tail` | pass — G3 fails | All three four-way MMC runs exceed both G3 thresholds; p95/max are 7,599/7,599 ms, 12,058/12,058 ms, and 8,579/8,579 ms. The reports are qualification `pass` because functional/convergence checks pass; their latency gate outcome remains fail. | +| Falsify provider as the dominant contributor | `artifacts/qualification/237-sbc-finite-tail-diagnostic.json`; `artifacts/qualification/237-sbc-finite-tail-run-1..3.json` | pass | Direct-provider maximum 1–2 ms; slowest request pre-provider phase 1,778–3,473 ms, provider-service maximum 0–1 ms. | +| Falsify durable MMC storage as the dominant contributor | Paired MMC/tmpfs corpus; Plan 237, 238, and 239 physical artifacts | pass | New four-way p95 median falls from 8,579 ms to 10 ms; new max median falls from 8,579 ms to 11 ms. Plan 237's sequential diagnostic falls from 1,785–3,473 ms maximum to 4 ms on tmpfs. Plan 238 moved only the SQLite files: MMC max 3,331–4,944 ms vs. tmpfs max 4 ms, with no task contamination. Plan 239's slowest requests correlate with foreground publication commit/checkpoint-sequence changes. | +| Capture p50/p95/max and bounded resource/convergence facts | Six Q008 v2 reports and matrix guard | pass | The original 30-sample reports do not support an independent p99 estimate. All 192 concurrent requests completed and every report converged. | +| Exercise configurable concurrency, principal, budget, p99, and competing-I/O reporting | Runner implementation; six Q008 reports; microSD smoke artifact; NVMe exploratory artifact | partial | The runner exposes these controls. The microSD smoke covers four-way concurrency, five principals, and user/global budgets with four successful observations; this verifies the loopback path but is too small for latency inference. Competing-I/O was not enabled, and the broader matrix is incomplete. Alternate persistent media are optional and not required by M001. | +| Vary 1/2/4/8 concurrency and 1/5/49 managed principals, budget policy, and controlled host I/O | M012 closure/artifacts plus the M001 microSD corpus and matrix manifest | partial | The repeated attribution corpus holds four-way concurrency and one synthetic admin principal with hard budgets disabled. The new microSD smoke adds five principals and user/global budgets but only four concurrent requests. Controlled competing-I/O and concurrency 1/2/8 plus 49-principal budget cases remain unqualified. | +| Keep production, schema, and operator policy unchanged absent causal design review | `git diff --name-only -- rust/` empty; no migration/config/API change | pass | Existing single SQLite gate, WAL/NORMAL and the documented restricted envelope remain unchanged. | +| Machine-check the committed M001 qualification corpus | `tests/tooling/test_on_prem_mmc_tail_attribution.py` | pass | The guard covers all six required Q008 microSD/tmpfs reports. The separately labeled NVMe report is exploratory and outside the M001 corpus. | + +## 3. Production implementation evidence + +No production Rust, SQLite schema, config, endpoint, auth, or persistence-policy change was justified or made. The production owners remain `rust/src/coordinator/publication.rs`, `rust/src/coordinator/finalization.rs`, and `rust/src/db/connection.rs`; the independent phase evidence attributes the tested tail to the current durable SQLite publication/checkpoint path. Changing WAL mode, transaction topology, checkpoint policy, worker count, or database placement would require a separate reviewed plan and, where ownership/journal semantics change, an accepted ADR. + +Implementation additions include the sanitized physical reports, a self-describing experiment manifest, a tooling regression guard, and benchmark-only controls for concurrency/request count, synthetic managed-principal count, hard budget mode, p99 at 1,000+ concurrent observations, mount-class reporting, and capped same-filesystem competing writes. No production runtime or database-policy change was made. + +## 4. Verification executed + +### Commands run + +```bash +CARGO_BUILD_JOBS=1 cargo build --manifest-path rust/Cargo.toml --locked --release +uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --output artifacts/qualification/on-prem-concurrency-performance/functional.json +uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/mmc-run-1.json +uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/mmc-run-2.json +uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/mmc-run-3.json +TMPDIR=/dev/shm uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-1.json +TMPDIR=/dev/shm uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-2.json +TMPDIR=/dev/shm uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-concurrency-performance/tmpfs-run-3.json +TMPDIR=/mnt/qualification-ssd/eggpool-m001-2026-10-14 uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --expected-sha256 8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538 --candidate-origin on-device-release-build --benchmark-samples 30 --concurrency-workers 4 --concurrency-requests 1000 --benchmark-principals 5 --benchmark-budget-mode user-global --output artifacts/qualification/on-prem-concurrency-performance/ssd-c4-u5-user-global-run-1.json +TMPDIR=/tmp/eggpool-m001-microsd-2026-10-14 uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --expected-sha256 8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538 --candidate-origin on-device-release-build --benchmark-samples 30 --concurrency-workers 4 --concurrency-requests 1000 --benchmark-principals 5 --benchmark-budget-mode user-global --output artifacts/qualification/on-prem-concurrency-performance/microsd-c4-u5-user-global-run-1.json +TMPDIR=/tmp/eggpool-m001-microsd-c4-2026-10-15 uv run python scripts/qualification_sbc.py --binary rust/target/release/eggpool --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --expected-sha256 8609119a7f0494c8541b7a47ddbbe86be45ce3f84096b5ee87c228c71513a538 --candidate-origin on-device-release-build --benchmark-samples 1 --concurrency-workers 4 --concurrency-requests 4 --benchmark-principals 5 --benchmark-budget-mode user-global --output artifacts/qualification/on-prem-concurrency-performance/microsd-c4-u5-user-global-minimal-run-1.json +findmnt -no SOURCE,FSTYPE / +cat /sys/block/mmcblk0/device/type +uv run pytest tests/tooling/test_on_prem_mmc_tail_attribution.py -q +uv run pytest tests/tooling/test_qualification_sbc.py -q +uv run pytest tests/tooling/ -q --tb=short --maxfail=1 +uv run ruff format --check tests/tooling/test_on_prem_mmc_tail_attribution.py +uv run ruff check tests/tooling/test_on_prem_mmc_tail_attribution.py +uv run ruff format --check scripts/ tests/tooling/ +uv run ruff check scripts/ tests/tooling/ +uv run pyright scripts/ +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +git diff --check +``` + +The attempted debug refresh was `CARGO_BUILD_JOBS=1 cargo build --manifest-path rust/Cargo.toml --locked`; it was interrupted due to sustained shared-host I/O pressure. + +### Results + +| Command | Result | +|---|---| +| Locked release build | pass; release binary SHA-256 matches all six physical reports. | +| Ordinary Q008 run | pass on physical Pi 5; functional fixture only. | +| Three MMC + three tmpfs Q008 benchmark runs | pass on physical Pi 5; every 32-request batch completed 32/32 and converged. G3 latency thresholds fail on MMC and pass on tmpfs. | +| Supplemental NVMe benchmark | Exploratory and outside M001 target requirements; 1,000/1,000 HTTP 200, p50/p95/p99/max 13/30/32/33 ms, five principals, user/global request budgets, and zero pending requests or active reservations. Competing I/O was disabled. | +| Supplemental microSD budget smoke | pass; four-way/four-request batch, five principals, user/global budget mode, 4/4 HTTP 200, zero pending requests and reservations. Four timing observations only; percentiles are not used for performance inference. | +| Supplemental microSD 1,000-request attempt | stopped under the resource limit; I/O pressure reached `full avg10=72.21%`, `full avg60=56.82%`; no report produced. | +| Supplemental microSD 64-request and 16-request attempts | both stopped before longer-window I/O pressure reached the prior limit; peak observed `full avg10` was 71.59% and 70.56%, respectively. No reports produced. | +| Read-only target-media attestation | pass on the physical Pi 5: root `/dev/mmcblk0p2` is ext4 and the MMC device type is `SD`. This confirms the repeated MMC corpus targeted microSD storage. | +| `uv sync --frozen` | pass; 9 packages checked. | +| `uv run pytest tests/tooling/test_on_prem_mmc_tail_attribution.py -q` | pass; 4 tests. | +| `uv run pytest tests/tooling/test_qualification_sbc.py -q` | pass; 41 tests. | +| Full tooling suite | first attempt stopped after 81 passed / 3 skipped at the known stale-debug-binary Q007 test (`rust/target/debug/eggpool` lacks the current `users` command). | +| Current debug build | interrupted after the build drove shared-host I/O pressure above 90% over 60 seconds; no build result. This respects the plan's stop condition. | +| Ruff format/check on the new file | attempted after the saturation stop; `uv run` did not return, so the command was interrupted with no result. The file was formatted with Ruff before the focused test. | +| Full tooling rerun, Pyright, Rust fmt/Clippy/no-default/full serial tests | not run after the saturation stop; no result is claimed. | +| Hosted CI | not run from this local closure pass. | + +The six original reports' runner-origin field says `q005-qualified-aarch64-copy` because the CLI default was not overridden. The release binary was built on this Pi immediately before that corpus and the measured SHA-256 is consistent across the reports; the manifest and this closure retain and disclose the provenance-label mismatch instead of rewriting raw reports. The supplemental NVMe run explicitly passed `--candidate-origin on-device-release-build`. + +## 5. Invariant review + +- Single SQLite worker/gate, WAL/NORMAL, generation ownership, retries/finalization, quota policy, and profile gates were not changed. +- No second connection, writer, checkpointer, runtime worker, service, API bypass, or provider-wire alteration was introduced. +- The physical G3 failure remains explicit. No result was relabeled as a performance pass. +- Reports contain bounded timing/resource facts and no credentials, prompts, bodies, user tokens, or provider keys. +- tmpfs is only a diagnostic control. It is not proposed for durable production database placement. + +## 6. Failure and recovery review + +All six original benchmark reports completed the 32-request batch without request failures and converged to zero pending requests and active reservations. The supplemental NVMe run completed its 1,000-request batch, and the minimal microSD budget smoke completed 4/4 requests; both converged to zero pending requests and active reservations. MicroSD runs at 1,000, 64, and 16 measured requests were interrupted under the plan resource stop condition before producing reports; their isolated servers exited. No result is claimed from those interrupted attempts. The completed qualification runs include isolated backup/recovery, restart reconciliation, and graceful shutdown. No production-data operation occurred. Ambient host load and the absence of a controlled competing-I/O trial limit causal claims beyond the MMC-versus-tmpfs storage result. + +## 7. Migration and compatibility review + +No schema, protocol, config, CLI, or client compatibility change was made. Migrations remain unchanged. Existing M012 G3 and capacity outcomes are preserved as historical evidence. + +## 8. Security review + +Qualification used a loopback-only synthetic provider and the runner's disposable managed bootstrap. The report guard rejects bootstrap password/token material. No real provider/user credentials or production database were used. + +## 9. Documentation and operations + +`docs/raspberry-pi.md` was not changed: the evidence does not justify a new durable-storage requirement or a safe operator mitigation. The existing restricted Pi 5 envelope remains accurate. The closure and matrix manifest link the six original raw physical reports and the supplemental NVMe report. No physical-SBC claim is made from a hosted or non-aarch64 environment. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| medium | G3 p95/max fail on microSD/MMC in every repeated four-way run; tmpfs removes the tail. Controlled competing-I/O has not been isolated | Existing small-team envelope remains restricted; no new concurrency SLA or safe storage workaround can be claimed | Keep Pi guidance restricted. Before any optimization or wider claim, a bounded physical plan must compare controlled idle/competing-I/O conditions while preserving WAL/NORMAL and one-writer safety. SSD/NVMe comparison is not required. | +| low | The out-of-scope NVMe run covers four-way concurrency, five principals, and user/global hard budgets; the matching microSD attempt stopped under the plan's I/O-pressure limit. Repeated 1/2/8 concurrency and 49-principal cases also remain unqualified | No broad account-count or budget-overhead inference can be drawn | Use existing M012 results only for their recorded scope; if these factors affect an implementation decision, include them in a registered follow-up with three paired runs per condition. | +| low | p99 is available from the supplemental 1,000-sample run only; no p99 acceptance threshold is frozen | One sample's p99 does not define a stable gate or tail guarantee | Keep the reported p99 diagnostic-only; define a repeated sampling and acceptance method before using it for a decision. | +| low | The six original benchmark reports retain the runner's default `q005-qualified-aarch64-copy` origin label although the same-hash binary was built locally immediately before qualification | Raw candidate-origin metadata is inaccurate; the binary hash, repository revision, and preceding build command remain consistent | Preserve raw evidence and this disclosure; the supplemental run records `on-device-release-build`, and any repeat should pass that option. | + +## 11. Roadmap disposition + +M001 is **conditionally closed** with the named competing-I/O and workload-matrix conditions above. Its core attribution is supported by three paired physical runs per microSD/MMC and tmpfs condition plus independent phase evidence. Alternate persistent media, including SSD/NVMe, are not required to close this diagnostic plan; the supplemental NVMe cell is out-of-scope evidence. No optimization is authorized: changing SQLite ownership or checkpoint policy would need separate architectural review and its own bounded acceptance gates. The subsystem roadmap remains active because future storage or concurrency performance work is not yet registered. + +## 12. Registry updates + +The registry removes M001 from dependency-ready work and records this conditional closure. A search of `plans/implementation/`, the subsystem dependency graph, and the dependency-ready, active, and blocked registry tables found no plan declaring M001 as a hard, interface, soft, or operational dependency. No plan is promoted or unblocked. On-prem user-access M016 remains blocked on M014 and M015; this optional workstream has no dependency edge to that security/merge sequence. Future controlled-I/O and budget-matrix research requires a new bounded plan and does not reopen this immutable closure. diff --git a/plans/closure/on-prem-user-access/001-status.md b/plans/closure/on-prem-user-access/001-status.md new file mode 100644 index 000000000..4f7b4092a --- /dev/null +++ b/plans/closure/on-prem-user-access/001-status.md @@ -0,0 +1,179 @@ +# On-Prem User Access Milestone 001 — Closure Status + +Status: conditionally closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/001-admin-bootstrap-and-credentials.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md#m001--secure-admin-bootstrap-and-client-credentials` + +Repository baseline reviewed: `0d6ebcd9` (task branch baseline); roadmap/ADR authority `81beec4e` + +Implementation commits: + +- `ecdc313` — add local admin bootstrap, per-user credentials, staged auth migration, tests, and docs. +- `1a2d329` — move M001 into closure after implementation landed. +- `b4f769b` — activate M001 and move it from the blocked registry into active work. + +## 1. Executive finding + +M001's schema, local administrator bootstrap/recovery, personal agent tokens, +legacy compatibility gate, and documentation are implemented. The initial +admin and token are committed atomically; passwords are Argon2id-hashed +outside SQLite's gate with fixed bounded parameters and at most two concurrent +hash workers. The legacy global server key authenticates only while the +database remains `legacy_unmigrated`; after identity bootstrap, agent routes +require a personal token. M001 does not provide browser login or multi-user +web authorization. + +This milestone is conditionally closed because the physical Le Potato/Pi +password-hashing latency and memory qualification required by the source plan +could not be performed on the available Darwin x86_64 host. This limits target +hardware sizing claims; it does not block M002's credential/schema interface. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Append-only identity schema and checksum | Migration `0055_user_identity.sql`; migration runner and database compatibility suite; SHA-256 `352224c8e2505509efa8fad0017c9aaac74ef8338771b35388ef22cf77cd8a62` | pass | Prior migration files/checksums remain unchanged; runtime manifest now records 55 migrations. | +| Atomic first-admin creation and race safety | `rust/tests/on_prem_identity.rs`: atomic bootstrap and concurrent first-run tests | pass | Exactly one initial administrator is committed. | +| Password storage, reset and token revocation | Identity integration tests; passwords are never stored in plaintext; reset revokes personal credentials; explicit revoke is covered | pass | Argon2id v19, m=19456 KiB, t=2, p=1; maximum two concurrent hash workers. | +| High-entropy personal agent token and one-way storage | Bootstrap/token integration tests; selector plus SHA-256 digest stored; no `Debug` implementation for plaintext token | pass | Token is emitted once and is withheld by default when stdout is not a TTY. | +| Legacy key is staged and cannot cross cutover | Middleware lifecycle test; full server transport suite | pass | Configured global key works only in `legacy_unmigrated`; personal tokens are limited to agent surfaces. | +| Fresh headless CLI flow and secret-output posture | Disposable release-binary smoke: `users bootstrap --password-stdin`, then `users list` | pass | Random password supplied through stdin; non-TTY output contained no `epk1_` token. | +| Existing DB/config and restore compatibility | `database_compatibility` suite and full serial workspace suite | pass | Upgrade is forward-only and transactional; no upstream account/provider topology was repurposed. | +| Strict build/test parity and dependency audit | Commands below | pass | All checks are local; no hosted CI result is claimed. | +| Physical Le Potato/Pi Argon2 qualification | Not available on this host | partial | Carry exact qualification into M007; do not infer SBC sizing from desktop results. | + +## 3. Production implementation evidence + +Migration 55 adds downstream `users`, `user_credentials`, and singleton +`user_access_state` tables, including a unique active-admin constraint and +`legacy_unmigrated`, `fresh_uninitialized`, `admin_initialized`, and `managed` +states. The schema does not modify upstream `accounts` or provider config. + +`rust/src/identity.rs` owns password hashing, credential issuance/revocation, +token verification, and access-state operations. The local CLI supports +bootstrap, list, password reset, token issue, and token revocation. Bootstrap +accepts confirmed TTY passwords or one bounded stdin line; no password CLI +argument exists. Plaintext token output requires a TTY or explicit +`--show-secret`. Fresh keyless server startup is rejected until bootstrap. + +The ingress middleware keeps the shared configured key within the staged +legacy state and limits personal credentials to inference/model discovery and +the integration profile. No browser session or web-admin authority was added. + +## 4. Verification executed + +### Commands run + +```bash +rtk cargo fmt --manifest-path rust/Cargo.toml --all -- --check +rtk cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +rtk cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +rtk cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +rtk cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test on_prem_identity -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --lib server::middleware::tests::global_server_key_is_valid_only_before_identity_cutover -- --exact +rtk cargo build --manifest-path rust/Cargo.toml --locked --release +rtk cargo deny --manifest-path rust/Cargo.toml check +rtk cargo tree --manifest-path rust/Cargo.toml -e features +rtk uv run python scripts/validate_release_docs.py +rtk uv run python scripts/validate_runtime_package_boundary.py +rtk git diff --check +``` + +### Results + +- Final full serial workspace run: 939 passed, 1 ignored, 73 suites. +- Focused identity run: 3 passed; legacy-key lifecycle unit test: 1 passed. +- Strict default Clippy: pass. No-default check and strict Clippy: pass. +- Locked release build: pass. Release binary: 31,811,040 bytes; SHA-256 `5b448d86506388a95680f51ffa76fed9717203b2e2f3d7a341aec4f89c0882e1`. +- `cargo deny`: advisories, bans, licenses, and sources passed; existing duplicate-version warnings were reported. +- Release-doc and runtime-package-boundary validators: pass. `git diff --check`: pass. +- Disposable release-binary bootstrap/list smoke: pass; token plaintext was withheld on captured stdout. +- One earlier full run reported `operations_o008::verified_replacement_self_checks_and_preserves_other_files` as `SelfCheckFailed`. The isolated full `operations_o008` target then passed all 10 tests, and the final full workspace run passed. No change was made to update self-check behavior. +- Local resource observation: `rtk /usr/bin/time -l cargo test --manifest-path rust/Cargo.toml --test on_prem_identity -- --test-threads=1` completed with 13.76 s wall and 72,011,776-byte maximum RSS for the Cargo/test process tree (three tests, including concurrent password hashing). Host: Darwin 25.6.0 x86_64. This aggregate desktop measurement is not SBC or per-hash calibration evidence. + +## 5. Invariant review + +- One SQLite worker/gate remains the only database writer; password hashing + occurs before the database transaction and uses bounded blocking workers. +- Migration history is append-only and old migration checksums are unchanged. +- Upstream provider accounts/configuration and inference wire shapes remain + owned by their existing modules. +- Personal tokens do not grant dashboard/admin access; the legacy global key + is rejected after identity cutover. Database-state errors do not enable a + keyless bypass. +- Secrets are excluded from ordinary diagnostics and password/token plaintext + is not persisted. +- `--no-default-features` compilation and strict Clippy pass. + +## 6. Failure and recovery review + +The admin row and initial credential share one transaction; concurrent +bootstrap produces one winner. Password hashing failure occurs before the +transaction and cannot leave a partial administrator. Password reset and +credential revocation are transactional. Credential verification is +fail-closed on database errors and revocation has no positive token cache. +Failed token digests are bounded to 256 entries with a one-second TTL. + +Recovery remains local through `eggpool users reset-password`; it never +reveals the old password and revokes that user's existing API credentials. +Before downgrading to a binary that predates schema 55, retain a protected +pre-upgrade DB backup: migration is forward-only, and older binaries may reject +the newer migration ledger. Restore through the existing staged backup/restore +validation path. + +## 7. Migration and compatibility review + +Fresh databases begin `fresh_uninitialized`; existing databases receive the +additive migration seeded as `legacy_unmigrated`. Existing shared-key and +dashboard behavior therefore continues until local identity bootstrap. The +bootstrap transition commits the admin and initial token together and changes +the state to `admin_initialized`; after that, the configured global key no +longer authenticates requests. The migration does not update request history, +provider credentials, or upstream account rows. There is no down migration. + +## 8. Security review + +Passwords require 12–1024 bytes and are hashed with Argon2id using a fresh +OS-random salt. Parameters are fixed and PHC parameters are checked before +verification; hashing concurrency is capped at two. Personal tokens contain a +high-entropy random secret, are stored only as selector plus digest, and are +compared at fixed digest length. Token plaintext is one-time output only. +This is an infrastructure milestone: cookie sessions, login throttling, CSRF, +and browser route authorization belong to M002. + +## 9. Documentation and operations + +Updated `architecture/deep-dive-security.md`, +`architecture/deep-dive-database.md`, `docs/deployment.md`, +`docs/cli-reference.md`, `docs/backup-restore.md`, and +`docs/agent-configuration.md`. Release-doc link/phrase validation passed. +The disposable CLI smoke used an isolated `HOME`, config, and data directory; +it did not start a listener or contact provider services. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| medium | No physical Le Potato/Pi Argon2 latency, RSS, or concurrent-login measurement was available. | The fixed 19 MiB / two-worker bound has no demonstrated memory/latency margin on the supported SBC class. | In M007, run the documented qualification on physical Le Potato/Pi hardware: single and two concurrent password hashes, wall time and peak RSS, under representative service load; record model, OS, storage, command, repetitions, p50/p95, peak RSS, and pass/fail against safe memory headroom. Adjust parameters only through a reviewed corrective if the bound fails. | + +## 11. Roadmap disposition + +M001 is conditionally closed. Production/schema/bootstrap and credential +contracts are complete and M002 can proceed against them. SBC hash calibration +remains an explicit M007 operational condition; no SBC performance claim is +made here. + +## 12. Registry updates + +The same closure commit marks implementation plan M001 `implemented`, updates +the roadmap M001 to `conditionally closed`, marks M002 `ready`, promotes M002 +to dependency-ready, removes it from blocked work, and records this closure in +the registry. M003–M007 remain blocked on their named hard predecessors; +physical qualification remains assigned to M007. diff --git a/plans/closure/on-prem-user-access/002-status.md b/plans/closure/on-prem-user-access/002-status.md new file mode 100644 index 000000000..07f0376c4 --- /dev/null +++ b/plans/closure/on-prem-user-access/002-status.md @@ -0,0 +1,176 @@ +# On-Prem User Access Milestone 002 — Closure Status + +Status: closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/002-browser-sessions-and-route-auth.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md#m002--browser-sessions-and-route-authorization` + +Repository baseline reviewed: `dcf184f` (M002 activation; M001 accepted) + +Implementation commits: + +- `087953a3` — implement browser sessions and route authorization. +- `f4a1861` — move M002 to closure after implementation and verification. + +## 1. Executive finding + +M002's browser authentication, session lifecycle, CSRF checks, managed-mode +authorization gates, and operator-managed HTTPS posture are implemented. +The session principal is resolved from a database-backed opaque cookie on each +request, so password reset, logout, expiry, and user disablement take effect +immediately for subsequent requests. Agent bearer credentials remain limited +to inference/discovery and cannot establish a browser session or reach admin +surfaces. `dashboard.public` does not grant public access after identity +initialization. + +M002 is closed. The physical SBC Argon2 measurement recorded as M001's +condition remains an explicit M007 qualification item; no M002 or SBC sizing +claim depends on a desktop estimate. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Opaque session and CSRF values, expiry, revocation | Migration 0056 and `rust/tests/on_prem_sessions.rs` digest-only/revoke, expiry, disabled-user and password-reset cases | pass | Session and CSRF secrets are stored as SHA-256 digests; session secret has 256 random bits. Absolute expiry is 7 days; idle expiry is 12 hours; activity timestamp updates at most every 15 minutes. | +| Login/logout/password change and forced first change | Managed loopback HTTP integration fixture | pass | Covers login, account page, forced change, re-login, logout, missing CSRF denial, and stale-session rejection with ordinary HTML forms and no JavaScript. | +| Generic login failure and bounded abuse controls | `server/auth.rs`, dummy Argon2 verification path, bounded username/IP throttle | pass | Same failure text is used for bad credentials; throttling state is capped at 2,048 hashed keys, with 10 failures per username and 30 per peer per 15-minute window. Argon2 settings and maximum two hash workers are inherited from M001. | +| Admin/member/agent/anonymous route separation | Managed HTTP fixture plus middleware classification | pass | Anonymous access is limited to health/readiness, static assets, and login. Browser admins access protected dashboard/admin routes; non-admin sessions are limited to account routes. Personal tokens are restricted to agent routes and the scoped integration profile. | +| Public-dashboard, shared-key and unknown API bypass prevention | Managed HTTP fixture (`dashboard.public=true` still denied) and middleware lifecycle rules | pass | The global key is accepted only in `legacy_unmigrated`; unknown `/api/*` paths do not inherit the public dashboard setting. Agent access to `/account` and `/api/status` is denied. | +| CSRF, same-origin and proxy transport rules | Auth module unit tests and HTTP unsafe-method fixture | pass | Unsafe browser methods require Origin/Referer same-origin validation and a session-bound CSRF token. Forwarded HTTPS is accepted only from an exact trusted peer IP. Remote cleartext login is rejected unless the explicit development override is enabled. | +| Migration, backup and config compatibility | Database compatibility, migration checksum, runtime asset manifest tests | pass | Migration 0056 is append-only; previous migration checksums remain unchanged. Trusted proxy entries must parse as literal IP addresses. The insecure override defaults off and is restart-required. | +| Documentation and release asset integrity | Release-doc validator and runtime manifest test | pass | Deployment/firewall docs describe operator-owned HTTPS/VPN; both config examples are hash-locked. | +| Physical SBC login latency/RSS | M001 condition and M007 target plan | carried | Not measured on this Darwin x86_64 host; M007 must record physical target results before any SBC performance claim. | + +## 3. Production implementation evidence + +Migration `0056_web_sessions.sql` adds the session selector, token digest, +CSRF digest, timestamps, user association, and lookup indexes. Its SHA-256 is +`b35541860dbc9c258de617f6f5a9c0111de6428be3128cd79d962bdda9634167`. +`rust/src/web_session.rs` owns session issuance, authentication, CSRF proof, +revocation, cookie attributes, and bounded login throttling. Password reset +invalidates both personal API credentials and browser sessions. + +`rust/src/server/auth.rs` provides `/login`, `/account`, password change, +logout, same-origin parsing, and trusted-proxy HTTPS validation. The server +middleware enforces role and credential-surface rules before handlers run. +`users enable-managed` performs the checked transition only after the shared +key is cleared and the public dashboard setting is off. The optional +`security.allow_insecure_web_auth` setting defaults to false and emits a +startup warning when enabled. + +## 4. Verification executed + +### Commands run + +```bash +rtk cargo fmt --manifest-path rust/Cargo.toml --all -- --check +rtk cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +rtk cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +rtk cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +rtk cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test on_prem_sessions -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test dashboard_filters -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test runtime_lifecycle_r005 -- --test-threads=1 +rtk cargo build --manifest-path rust/Cargo.toml --locked --release +rtk cargo deny --manifest-path rust/Cargo.toml check +rtk cargo tree --manifest-path rust/Cargo.toml -e features +rtk uv run python scripts/validate_release_docs.py +rtk git diff --check +``` + +### Results + +- Full serial workspace suite: **947 passed, 1 ignored across 74 suites**. +- Focused suites: `on_prem_sessions` 5 passed; `server_transport` 14 passed; + `dashboard_filters` 2 passed; `runtime_lifecycle_r005` 10 passed. +- Formatting, strict default Clippy, no-default-features check, and strict + no-default-features Clippy passed. +- Locked release build, `cargo tree -e features`, `cargo deny`, release-doc + validation, and `git diff --check` passed. Cargo deny reported existing + duplicate-version warnings; advisories, bans, licenses, and sources passed. +- Release binary: 32,205,208 bytes; SHA-256 + `942ca66e289f82a779b3184965b8391ab281d19fbef58f8544807bd2284bc25a`. +- One early suite attempt exposed runtime-manifest drift for the edited config + examples and a missing reload-schema disposition. Hashes and the canonical + reload-policy fixture were corrected; the focused contract test and final + full suite then passed. +- These are local results. No hosted CI or physical SBC result is claimed. + +## 5. Invariant review + +- Browser sessions and agent tokens remain separate credentials. Browser + cookies are not copied into inference requests. +- One SQLite worker/gate remains the session authority; no positive session + cache delays revocation. Database failure rejects protected routes. +- Argon2 runs under M001's bounded worker limit, outside SQLite transactions. +- TOML/environment continue to own providers and upstream credentials. No + web-based provider-secret editor or external identity service was added. +- The legacy shared key remains confined to `legacy_unmigrated`; managed + access has no public-dashboard or loopback privilege shortcut. +- The added security setting uses the existing `classify_transition` + authority and is restart-required; no separate reload list was introduced. + +## 6. Failure and recovery review + +Invalid, expired, revoked, reset, or disabled-user sessions fail closed. +Logout and password reset revoke database rows before returning success. +Database errors reject protected requests and do not fall back to anonymous +or shared-key access. Session issuance requires an active user inside the +database transaction. Malformed cookies and CSRF values are rejected. +Requests already executing may finish; revocation applies to subsequent +requests. Session state survives restart only while it remains unexpired and +unrevoked. + +## 7. Migration and compatibility review + +Migration 0056 is additive and forward-only. Fresh databases retain the +existing bootstrap flow; legacy databases retain their shared-key behavior +until identity initialization. Managed browser authorization supersedes the +legacy `dashboard.public` behavior without changing legacy-readonly dashboard +compatibility. Provider APIs, streaming behavior, and request wire formats +were not changed. Downgrade to binaries predating schema 56 requires restoring +a protected pre-upgrade database backup. + +## 8. Security review + +The browser session has a random selector and 256-bit secret; only its digest +and the CSRF digest are persisted. Cookies are HttpOnly, scoped to `/`, +SameSite Lax/Strict, and Secure when the connection is directly HTTPS or +HTTPS from a configured trusted proxy. Same-origin checks reject missing, +ambiguous, or cross-origin mutation metadata. The opt-in insecure setting is +explicitly labeled for development and warns at startup. Login throttling is +bounded in memory and keyed by digests, not raw usernames or addresses. + +## 9. Documentation and operations + +Updated the security, dashboard, and database architecture deep dives, backup +and restore, deployment, firewall, and CLI reference docs. Both example +configs document exact trusted proxy IPs and the insecure-auth override. +Operators must provide TLS termination or use a VPN for remote browser access; +EggPool does not provide a native TLS server or remote-access service. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| low | Physical Le Potato/Pi Argon2 login latency, RSS, and concurrency remain unmeasured. | No target-class login sizing claim is supported. | M007 carries M001's exact physical qualification; record model, OS, storage, repetitions, wall-time percentiles, peak RSS, and one/two-worker results. | + +## 11. Roadmap disposition + +M002 is closed. Its browser session and authorization interfaces are stable +for M003 principal propagation. M003 is promoted to ready. M004–M007 remain +blocked on their listed hard predecessors; M001's physical hash condition +remains with M007. + +## 12. Registry updates + +The closure commit marks this plan implemented and M002 closed, moves it from +the active table to recently closed, promotes M003 to the dependency-ready +table, removes M003 from blocked work, and updates the subsystem roadmap in +the same commit. diff --git a/plans/closure/on-prem-user-access/003-status.md b/plans/closure/on-prem-user-access/003-status.md new file mode 100644 index 000000000..2649beb0c --- /dev/null +++ b/plans/closure/on-prem-user-access/003-status.md @@ -0,0 +1,137 @@ +# On-Prem User Access Milestone 003 — Closure Status + +Status: conditionally closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/003-principal-propagation-and-attribution.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md#m003--request-principal-and-durable-user-attribution` + +Implementation commits: + +- `9af8787a` — propagate user identity through inference. +- `98f90b8f` — move M003 into closure. + +## 1. Executive finding + +M003 gives authenticated inference requests one typed, server-resolved +principal, propagates its stable owner through finite, streaming, compact, and +semantic-selector publication, and persists attribution atomically with the +request. Duplicate proxy request IDs require matching owner facts; publication +rechecks account/session state and auth epoch before dispatch. Terminalization +checks the same owner snapshot. Legacy shared-key requests remain explicitly +unattributed. M004 is ready to build model policy on this principal contract. + +M003 is conditionally closed. The complete integration target set and the +required compile/lint gates pass. The full all-targets run reaches unrelated +`operations::update` unit tests and fails there on this Darwin host. One of +those tests reproduces in isolation. M003 does not modify update code. Runtime +overhead and physical SBC resource measurements remain unqualified and are +assigned to M007; no sizing claim is made here. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Typed principal from trusted auth middleware | `identity::AuthenticatedPrincipal`; middleware resolves agent credentials and browser sessions from SQLite; legacy shared key is explicitly typed | pass | Client identity headers do not set principal fields. | +| Stable identity across finite, streaming, compact, retries, and selector dispatch | Attribution is carried through endpoint decision, finite coordinator, streaming coordinator, and semantic selection; `rust/tests/on_prem_attribution.rs`; `coordinator_c011.rs`; `codex_compaction_compat.rs` | pass | Selector publication is an internal child and retains the original user and credential. | +| Durable attribution and legacy history | Migration 0057, checksum/runtime manifest updates, `database_compatibility.rs`, and HTTP inference spoof-header fixture | pass | Existing rows remain null/unattributed; user attribution intentionally has no FK so deletion cannot erase history. | +| Cross-user replay and finalization owner equality | `coordinator_publication.rs`, `coordinator_finalization.rs`, coordinator boundary suites | pass | Mismatched duplicate attribution fails closed. | +| Revocation/auth-epoch check before publication | `on_prem_attribution.rs`, `coordinator_publication.rs` | pass | Password reset increments the epoch and revokes API credentials and browser sessions. | +| Bounded content-free user request history | `identity::recent_user_requests`; attribution test | pass | Query limit is clamped to 1–500; internal selector rows are excluded. | +| No public request-shape regression | Existing coordinator public request structures remain unchanged; coordinator boundary and compact compatibility tests | pass | Attribution uses additive private input and publication APIs. | +| Architecture and protocol documentation | Request lifecycle, database, observability, and API reference docs | pass | User IDs and credentials remain out of routine diagnostics and provider rollups. | + +## 3. Migration and implementation evidence + +Migration 0057 adds `users.auth_epoch` and nullable request owner, +credential/session, principal-kind, request-kind, and selector-parent fields. +The `(user_id, started_at DESC, id DESC)` index supports bounded history. +Historical request rows remain queryable with null ownership; user removal +does not cascade into request history. The embedded checksum inventory and +runtime manifest now include version 57. + +Publication validates current active user, auth epoch, and credential/session +state inside its transaction. Request owner facts are persisted in that same +transaction and copied into `FinalizationIdentity`. Duplicate publication and +terminalization compare owner facts before observing or settling an existing +row. Internal selector attempts use `request_kind='internal_selector'` and a +parent proxy request ID; logical history omits them. + +## 4. Verification executed + +### Commands run + +```bash +rtk cargo fmt --manifest-path rust/Cargo.toml --all -- --check +rtk cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +rtk cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +rtk cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +rtk cargo test --manifest-path rust/Cargo.toml --workspace --tests -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test on_prem_attribution -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test coordinator_publication -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test coordinator_boundaries -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test coordinator_c011 -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test coordinator_finalization -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test codex_compaction_compat -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --lib operations::update::tests::package_failure_after_mutation_restores_the_previous_exact_release -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --lib operations::update::tests::pip_transition_updates_metadata_and_runs_target_without_raw_replacement -- --test-threads=1 +rtk git diff --check +``` + +### Results + +- Formatting, strict default Clippy, no-default-features check, and strict + no-default-features Clippy passed. +- The full integration target set passed: **953 passed, 1 ignored across 75 + suites**. Focused attribution passed 3 tests; publication passed 10; + coordinator boundaries 6; `coordinator_c011` 18; finalization 11; compact + compatibility 13; server transport 14. +- The full `--workspace --all-targets` command did not pass. Its library unit + suite had 162 passed and 2 failed: `package_failure_after_mutation_restores_the_previous_exact_release` + and `pip_transition_updates_metadata_and_runs_target_without_raw_replacement`. + The first failed again in isolation with + `RollbackFailed { reason: "ownership_changed_unexpectedly" }`; the second + passed in isolation. Both belong to `operations::update`, outside M003. +- `git diff --check` passed before closure metadata edits. +- No dependency changes were made; release build, dependency audit, and + dependency feature-tree checks were not required. +- No request latency, memory, or physical SBC benchmark was run. Existing + M001 Argon2 and new attribution resource observations remain assigned to + M007. This closure makes no performance or capacity claim. + +## 5. Invariant and failure review + +- Middleware resolves identities from stored credentials/session state; + spoofed downstream-user headers do not affect attribution. +- Credential revocation and password reset are checked before publication. + Already published requests retain immutable owner facts through retries and + terminal settlement. +- DB errors fail closed; no path degrades managed inference to anonymous + ownership. Legacy shared-key use stays separately marked and userless. +- SQLite remains the single publication authority. No database transaction + spans provider I/O. Request bodies, prompts, bearer secrets, and cache keys + are not added to attribution or user-history results. +- Semantic selector executions remain attributable children and do not count + as additional logical user requests. + +## 6. Residual conditions + +| Severity | Condition | Impact | Required evidence | +|---|---|---|---| +| medium | Two unrelated `operations::update` library tests fail in the all-targets run on this host; one reproduces alone. | The repository-wide all-targets gate is not green in this environment. M003 integration targets and compile/lint gates are green. | Diagnose and close under the update/deployment owner before claiming a green all-targets workspace run. | +| low | Request-attribution overhead and physical target resource use are not measured. | No latency/RSS sizing assertion is supported. | M007 paired baseline and post-change measurements on named hardware. | + +## 7. Roadmap disposition + +M003 is conditionally closed: the principal and durable-ownership contracts +are stable, and M004 may proceed. M004 is promoted to ready. M005 and M006 +remain blocked on their hard predecessors. M007 remains blocked on M006 and +must carry M001 and M003 physical qualification evidence. + +The update unit-test failures are recorded as a repository gate condition; +they do not indicate an attribution failure or change M004's policy boundary. diff --git a/plans/closure/on-prem-user-access/004-status.md b/plans/closure/on-prem-user-access/004-status.md new file mode 100644 index 000000000..ef5cfd318 --- /dev/null +++ b/plans/closure/on-prem-user-access/004-status.md @@ -0,0 +1,172 @@ +# On-Prem User Access Milestone 004 — Closure Status + +Status: closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/004-policy-profiles-and-model-gates.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md#m004--profile-based-model-policy` + +Repository baseline reviewed: `1a4ae78` (M004 activated after M003 closure). + +Implementation commits: + +- `f537df4` — add per-user model policy profiles and enforce model gates. +- `c53bf4d` — test concurrent policy edit and snapshot refresh. + +## 1. Executive finding + +M004 delivers persisted, reusable model-policy profiles, exact model rules, +per-user overrides, instance-wide denials, preferred-model metadata, live +policy revisioning, concrete-dispatch checks, and principal-filtered model +discovery. The same coordinator admission path covers finite, streaming, +compact, protocol aliases and selector resolution. A request keeps its +immutable authorization snapshot after admission; new requests see a committed +policy or user-state change without a process restart. M004 is closed and M005 +is ready. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Persist profiles, exact rules, assignments, overrides, denials and revision | Migration 0058; migration checksum and runtime manifest; `database_compatibility.rs`, `build_manifest.rs` | pass | Tables and indexes are append-only; mutation and user-state triggers advance the revision. | +| Apply deterministic precedence and reject broad model keys | `model_policy::ModelPolicySnapshot`; `on_prem_model_policy.rs` | pass | Instance denial wins, then user override, profile rule and safe role default; model keys are exact and provider-qualified syntax is tested. | +| Authorize requested and concrete models before dispatch | `coordinator/endpoints.rs`; `on_prem_attribution.rs`; `server_transport.rs` | pass | The shared endpoint path validates requested model, selector model and resolved concrete candidate; denied requests return 403. | +| Preserve immutable admitted policy and refresh new requests | policy snapshot immutability, disabled-user invalidation, revision refresh and concurrent edit/read fixtures in `on_prem_model_policy.rs` | pass | Database transactions serialize mutations; snapshots remain fixed while later requests observe the newer revision. | +| Filter model list and integration profile, including aliases | `server/health.rs`; `on_prem_attribution.rs`; `dashboard_filters.rs`; `status_command.rs` | pass | Denied model and ineligible alias disappear; policy revision is included in the integration ETag. | +| Provide local setup surface and safe preferred model validation | `eggpool users policy` commands; `cli_contract.rs`; policy service tests | pass | Actor must be an active admin; preferred model must currently be allowed. Preference is stored metadata, not an implicit rewrite of explicit requests. | +| Keep provider configuration and quotas separate | `model_policy.rs`, migration 0058 and architecture/database documentation | pass | No upstream credential, account routing or provider quota ownership moved into user policy. | + +## 3. Production implementation evidence + +Migration 0058 adds `policy_state`, `policy_profiles`, +`profile_model_rules`, `user_profile_assignments`, +`user_policy_overrides`, and `instance_model_denials`. The checksum inventory +has 58 files; the migration checksum is +`dc98951394abed29a02f48d75d2b72554da74d98de49563fa87c2548dd757fe2`. + +`rust/src/model_policy.rs` owns the typed policy snapshot and storage service. +Its bounded cache is keyed by user and checked against the persisted monotonic +revision. Mutations are transactional, user state/role/auth-epoch triggers +invalidate snapshots, and an admitted snapshot is immutable. Exact instance +denials cannot be overridden. Members with no profile default-deny; an +unassigned admin retains the single-owner allow behavior. + +`InferenceState` receives the policy service in production. The coordinator +checks requested, selector and resolved concrete model IDs before dispatch. +The health projection filters `/v1/models` and the integration profile by the +authenticated principal, omits virtual aliases with no allowed route, and +adds the policy revision to the integration ETag. Local CLI commands support +profile creation, exact rule changes, assignment and preferred model setup. + +## 4. Verification executed + +### Commands run + +```bash +rtk cargo fmt --manifest-path rust/Cargo.toml --all -- --check +rtk cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +rtk cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +rtk cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +rtk cargo test --manifest-path rust/Cargo.toml --workspace --tests -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test on_prem_model_policy -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test on_prem_attribution -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test build_manifest -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test database_compatibility -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test cli_contract -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test status_command -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test dashboard_filters -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test model_router -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test codex_compaction_compat -- --test-threads=1 +rtk cargo test --manifest-path rust/Cargo.toml --test operations_o005 -- --test-threads=1 +rtk git diff --check +``` + +### Results + +- Formatting, strict default Clippy, no-default check and strict no-default + Clippy passed. +- The initial full integration run passed **956 tests, 1 ignored across 76 + suites**. After adding the concurrent policy fixture, the full + `--workspace --all-targets` run passed **957 tests, 1 ignored across 76 + suites** in 117.94 seconds. +- Focused policy tests passed 3; attribution 3; build manifest 4; database + compatibility 9; CLI contract 5; status command fixture 1; server transport + 14; dashboard filters 2; model router 4; compact compatibility 13; and + operations O005 14. +- `git diff --check` passed. No dependencies or Cargo features changed, so + dependency audit and release build were not required. + +## 5. Invariant review + +- Provider topology, credentials, account selection and provider quotas remain + operator-owned and separate from downstream user policy. +- Authorization precedes upstream dispatch and is repeated for the resolved + concrete candidate; selector aliases and provider-qualified IDs do not + bypass it. +- Policy snapshots are immutable per admitted request. New requests check the + current durable revision and disabled-user state. +- SQLite remains one connection/gate/worker. No transaction spans provider + I/O; policy storage and cache are bounded. +- Errors and discovery expose no credentials or other users' policy details. + +## 6. Failure and recovery review + +Policy reads and mutation errors fail closed. Profile/rule/assignment changes +commit atomically; revision triggers cover direct user state, role and +auth-epoch edits as well as policy table mutations. A stale admitted request +continues with its captured snapshot; later requests observe the new revision. +Service restart begins with an empty cache and loads current durable state. +There is no separate selector, retry or compact authorization implementation +that can silently omit the shared coordinator gate. + +## 7. Migration and compatibility review + +Migration 0058 is append-only and checksum-validated. Existing schema history +and upstream `accounts` are unchanged. Existing databases receive empty policy +tables and preserve unassigned-admin access; unassigned members default-deny. +The migration and manifest counts are 58. No public inference request shape or +provider TOML format changed. + +## 8. Security review + +The CLI policy mutation commands require an active administrator actor. +Exact model identifiers reject wildcards and noncanonical whitespace. A user +override cannot clear an instance denial. Disabled users fail policy snapshot +resolution. Model discovery is a convenience projection only; runtime checks +remain authoritative. Policy revision is not a credential or secret and is +the only new cache invalidation fact exposed in the ETag. + +## 9. Documentation and operations + +Updated the database, routing, request lifecycle and integration deep dives, +plus model routing, agent configuration, API and CLI reference docs. These +describe role defaults, exact policy precedence, no-restart invalidation, +filtered catalogs and preferred-model behavior. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| low | Physical SBC resource/performance measurements for the full user-access line are not available in this M004 environment. | No physical target capacity or latency claim is made. | M007 owns target hardware qualification and carries the earlier M001/M003 measurement conditions. | + +No M004 correctness or security findings remain open. + +## 11. Roadmap disposition + +M004 is closed. Its profile, policy revision and concrete-model authorization +contracts satisfy the hard predecessor for M005. Physical footprint evidence +remains assigned to M007; it does not block budget implementation. + +## 12. Registry updates + +The on-prem user-access M004 row moves to recently closed. M005 is promoted +from blocked to dependency-ready because M004's policy authorization is now +closed. M006 remains blocked on M005; M007 remains blocked on M006 and +operational SBC/ordinary-host evidence. The audit found no other registered +plan newly eligible under this dependency chain. diff --git a/plans/closure/on-prem-user-access/005-status.md b/plans/closure/on-prem-user-access/005-status.md new file mode 100644 index 000000000..73f569bab --- /dev/null +++ b/plans/closure/on-prem-user-access/005-status.md @@ -0,0 +1,154 @@ +# On-Prem User Access Milestone 005 — Closure Status + +Status: conditionally closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/005-durable-user-budgets.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md` + +Repository baseline reviewed: `03f6bbfe` (M005 activation) + +Implementation commits: + +- `49dc6ce` — durable budget rules, transactional holds, settlement, CLI and read API +- `0bf3e71` — move M005 into closure review + +## 1. Executive finding + +SQLite-backed user/model budget capability is implemented for logical request +count, request-size token estimates, estimated microdollars, and concurrency. +Publication owns the hold in the same transaction as request and provider +reservation state; terminal finalization owns idempotent settlement. UTC daily +and monthly usage is queryable through an authenticated bounded API. M005 is +conditionally closed because token and cost dimensions are estimates rather +than provable hard ceilings, crash/restore-specific budget fixtures and gate +overhead measurements were not collected, and global-instance budget rules +are not implemented. M006 may proceed against this durable contract, with +these conditions visible in its UI and the later qualification audit. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| UTC windows, integer units, bounded rule validation and change audit | `rust/assets/db/migrations/0059_user_budgets.sql`; `user_budget_rules_validate_admin_scope_and_usage_reads_are_bounded` | pass | Requests/tokens/microdollars/concurrency are positive bounded integers; day/month are UTC calendar periods. | +| Atomic aggregate and exact-model pre-dispatch holds | `rust/src/coordinator/publication.rs`; `concurrent_last_request_reservations_cannot_over_admit` | pass | Hold and durable publication commit together; concurrent final slot has one winner. | +| Concurrency release after terminal settlement | `concurrent_limit_releases_only_after_terminal_settlement` | pass | All active requests count across day/month boundaries; terminal settlement releases the slot. | +| Retry accounting, partial/unknown usage, duplicate finalization | `rust/src/coordinator/finalization.rs`; `on_prem_budgets` retry and unknown-cost cases | pass | One logical request is charged; transitioned attempts retain usage; unknown values consume reserved fallback; duplicate finalization is idempotent. | +| Authenticated bounded usage projection | `rust/src/server/health.rs`, `middleware.rs`; `on_prem_attribution` | pass | Members read only their own rules/usage; admins may select a target; query limit is capped. | +| CLI rule management | `rust/src/cli.rs`, `runtime.rs`, `rust/tests/cli_contract.rs` | pass | Admin scope and UTC period are explicit; no dashboard editor is added in M005. | +| Migration/checksum/runtime asset registration | migration 0059, `checksums.json`, `runtime-manifest.json`; full serial workspace suite | pass | Migration is append-only and backup includes the authoritative SQLite state. | +| Exact output-token cap for every supported protocol | `request/limits.rs` and publication reservation currently use a bounded request-size estimate | partial | The estimate does not reserve a guaranteed requested output maximum or reject every unbounded-output request. | +| Hard USD cap for unpriced models | routing estimator fallback and budget settlement fallback | partial | Unknown usage is not counted as zero, but a configured user microdollar rule can still rely on the generic estimator fallback; no explicit local-price requirement or fail-closed unknown-price gate exists. | +| Hard global instance budget intersection | no global rule scope in migration/service | not implemented | Profile budget rules were optional; global instance ceilings were required by the source plan. | +| Budget-specific process-crash/restore campaign and measured publish/settle overhead | not collected in this milestone | not run | Generic lifecycle/migration tests ran in the workspace suite; they do not substitute for budget-specific crash/restore fixtures or measurements. | + +## 3. Production implementation evidence + +Migration 0059 adds bounded user rules, active per-request holds, per-attempt +usage, audit history, and indexed user/model/day/month rollups. The publication +transaction validates current user/credential ownership, evaluates every +matching aggregate and exact-model rule, then inserts or extends the hold with +the durable request/attempt/provider reservation. Retry attempts do not add a +second logical request. Terminal finalization records each transitioned +attempt and settles the request once in the same SQLite transaction. Unknown +token/cost use consumes the held estimate and increments explicit unknown +counters. `/api/user-budget` provides authenticated member/admin reads, and +`eggpool users budget set|remove` manages rules locally. + +## 4. Verification executed + +### Commands run + +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +git diff --check +``` + +### Results + +- `on_prem_budgets`: 4 passed. +- Full local serial Rust workspace/all-target suite: 962 passed, 1 ignored, + 77 suites. +- Formatting, default strict Clippy, no-default check, no-default strict + Clippy, and whitespace checks passed. +- No CI run or physical SBC measurement was performed for this closure. + +## 5. Invariant review + +- User budgets remain separate from upstream provider quota scoring and do not + affect load-based account selection. +- Publication has one SQLite transaction owner; a user hold cannot commit + separately from the durable request/provider reservation. +- Holds survive nonterminal attempts and retries; terminal settlement and + duplicate finalization are idempotent. +- User usage reads are bounded and owner-fenced; diagnostics do not include + request bodies, prompts, or credentials. +- Unsupported exactness is documented in `docs/model-limits.md` and the + request/routing/database deep dives. Estimated USD is not labeled as an + invoice or exact cost. + +## 6. Failure and recovery review + +Concurrent last-slot publication and terminal concurrency release are covered. +Retry partial usage, unknown usage fallback, and duplicate terminal settlement +are covered. Database errors propagate through publication/finalization rather +than being treated as permission to dispatch. A budget-specific process kill +between publication and settlement, post-provider-response recovery, and +backup/restore round-trip were not directly injected; those remain evidence +conditions rather than inferred passes. + +## 7. Migration and compatibility review + +Migration 0059 is additive and checksummed; the full workspace migration and +database compatibility tests passed. Backups include the new tables as part of +the SQLite database, while provider credentials remain config/environment +owned. Older executables still require a matching pre-upgrade backup to roll +back across the schema version. + +## 8. Security review + +Rule mutation requires an active admin and records actor/action/version. Member +reads are scoped to the authenticated owner; admin target selection is +role-checked. Usage endpoints cap row counts. Cost, token, and request counters +are bounded integers; hold evaluation uses checked wide arithmetic. No secret +or body fields are included in usage projections. + +## 9. Documentation and operations + +Updated `architecture/deep-dive-database.md`, +`architecture/deep-dive-request-lifecycle.md`, +`architecture/deep-dive-routing.md`, `docs/api-reference.md`, +`docs/backup-restore.md`, `docs/cli-reference.md`, and `docs/model-limits.md`. +The API exposes UTC windows and separates provider-reported, local-estimated, +and reservation-fallback microdollars. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| Medium | Output-token reservations are request-size estimates and do not enforce a guaranteed maximum output for every protocol. | A token rule is not a mathematically hard spend ceiling when the provider can emit more output than the admitted estimate. | Add protocol-aware maximum-output admission and fail closed when a hard token rule cannot be bounded. | +| Medium | USD rules can use generic fallback pricing for an unpriced model. | A configured USD limit is an estimate and cannot prove a hard cap for self-hosted/unpriced usage. | Require a documented operator-configured conservative local price or reject hard-cost admission when price is unknown. | +| Medium | No global-instance budget scope/intersection exists. | Administrators cannot enforce the source plan's instance-wide hard ceiling independently of user rules. | Add global rule scope and intersection tests before claiming full M005 hard-cap completion. | +| Medium | M005-specific crash/restart/backup-restore tests and publish/settle overhead measurements were not run. | Recovery correctness and SQLite gate cost for new budget writes are not directly quantified. | Add restart/restore fault fixtures and reproducible overhead measurement during qualification. | + +## 11. Roadmap disposition + +M005 is conditionally closed. The durable user-level request/usage capability +is available for M006 integration, while the four findings above remain +explicit conditions. M006 is promoted to ready because its UI and API +integration can consume the implemented rule/usage interfaces; M007 remains +blocked on M006 closure and physical target qualification. + +## 12. Registry updates + +The registry and roadmap mark M005 conditionally closed, promote M006 to ready, +and retain M007 as blocked on M006 plus operational hardware evidence. diff --git a/plans/closure/on-prem-user-access/006-status.md b/plans/closure/on-prem-user-access/006-status.md new file mode 100644 index 000000000..cbb473651 --- /dev/null +++ b/plans/closure/on-prem-user-access/006-status.md @@ -0,0 +1,171 @@ +# On-Prem User Access Milestone 006 — Closure Status + +Status: conditionally closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/006-web-admin-and-personal-integrations.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md` + +Repository baseline reviewed: `5bf0cfe4` (M005 conditional closure) + +Implementation commits or pull requests: + +- `a3f2d00a` — add on-prem user administration and qualification updates +- `1aeb892c` — begin M006 closure evidence collection + +## 1. Executive finding + +The embedded browser admin/member capability is implemented and exercised by a +real two-user loopback journey. Administrators can manage users, profile rules, +model overrides, budgets, credentials, and audit history; members can change +their initial password, inspect their own models and usage, and issue/revoke +personal credentials. Credential delivery is one-time and no-store. M006 is +conditionally closed because matched dashboard parity captures for the new +admin/member pages were not available in this environment. The underlying M005 +estimate limits remain accurately labeled in the UI and are carried to M007. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Admin/member route authorization and CSRF | `rust/src/server/middleware.rs`; `rust/tests/on_prem_admin_ui.rs` | pass | Agent token receives 401 for `/users`; member browser receives 403 for `/users` and an admin create post; missing CSRF is rejected. | +| User lifecycle and last-admin safety | `identity::create_member`, `set_user_enabled`, `reset_user_password`; `on_prem_admin_ui` | pass | Temporary password forces first-login change; disabling/recovery revokes credentials and sessions; last active admin cannot be disabled. | +| Profile CRUD, rules, and stale writes | `model_policy.rs`; profile CAS/audit test | pass | Version compare-and-swap accepts one edit and rejects stale edit; rule removal and audit facts verified. | +| Member-scoped account, models, and usage | `/my-account`, `/my-account/models`, `/usage`; two-user browser test | pass | Member pages do not expose the admin username or admin summaries. | +| One-time personal credentials and revocation | `on_prem_admin_ui`; `/account/tokens*` | pass | Token appears once, has `no-store`, authenticates discovery, cannot enter admin pages, and stops authenticating after revocation. Refresh omits plaintext. | +| Budget and audit administration | `/users`, `/usage`, `/admin/audit`; API and UI renderers | pass | Existing M005 estimate dimensions are labeled as estimates. Audit rows contain actor/action/target/time only. | +| Migrations and manifest | migration 60, checksum/runtime manifests, migration inventory and compatibility tests | pass | Append-only migration adds secret-free administrative audit rows; older schema fixtures upgrade through v60. | +| Personal client profile and desktop paths | `on_prem_client_profiles`, `operations_o005`, `eggpool-connect`, client-config tests in workspace suite | pass | Existing Codex/OpenCode profile flow consumes per-user authorization facts and the user's `EGGPOOL_API_KEY`; no server-side provider editor was added. | +| Dashboard browser parity captures for new pages | Frozen oracle/browser interaction harness | not run | The pinned oracle worktree and its required `EGGPOOL_DASHBOARD_ORACLE_*` paths were not available. New pages use ordinary server-rendered forms. | +| Physical SBC sizing | M007 | deferred | This closure makes no hardware capacity or named-user performance claim. | + +## 3. Production implementation evidence + +Migration 60 adds indexed `user_admin_audit` storage and is checksum-pinned in +the migration and runtime manifests. Identity and policy services now provide +transactional user creation, enabled-state changes, forced password reset, +bounded credential and profile listing, profile compare-and-swap, profile and +rule management, and secret-free audit projection. Audit rows are committed +with the corresponding mutation. + +The existing embedded dashboard now has admin pages for users, profiles, +usage, and audit, plus member pages for their own account, model access, and +usage. Unsafe form posts require same-origin evidence and the session-bound +CSRF token; admin operations check the active browser principal's current role. +Agent tokens never provide dashboard access. The one-time token response is +`no-store`, `no-referrer`, and `nosniff`. The local provider configuration +remains operator-owned. + +The offline live-provider qualification fixture now bootstraps a synthetic +administrator in its isolated temporary database, uses the resulting personal +token for discovery/inference, and redacts it from candidate diagnostics. This +keeps the fixture aligned with fresh-database authentication after legacy-key +retirement. + +## 4. Verification executed + +All results below are local runs, not hosted CI results. + +### Commands run + +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_admin_ui -- --test-threads=1 +uv sync --frozen +uv run pytest tests/tooling/ -q --tb=short --maxfail=1 +uv run ruff format --check scripts/ tests/tooling/ +uv run ruff check scripts/ tests/tooling/ +uv run pyright scripts/ +uv run python scripts/validate_release_docs.py +uv run python scripts/validate_runtime_package_boundary.py +git diff --check +``` + +### Results + +- Rust workspace: 964 passed, 1 ignored across 78 suites (serial). +- Focused admin journey: 2 passed after the final route and token-revocation assertions. +- Default and no-default strict Clippy: pass; no-default workspace check: pass; formatting check: pass. +- Tooling suite: 166 passed, 1 skipped. The skipped test requires a separately built or provisioned qualification candidate; the focused offline-provider test passed after rebuilding `cargo build --manifest-path rust/Cargo.toml --locked`. +- Ruff: 59 files already formatted; all checks passed. Pyright: 0 errors, 0 warnings, 0 informations. +- Release documentation validator: pass, 7 docs checked. Runtime package-boundary validator: pass. +- `git diff --check`: pass. +- No dependency or feature graph changed; no dependency audit or release build was required. + +## 5. Invariant review + +- Provider topology/configuration and upstream credentials remain outside the web admin. +- User/profile mutations use SQLite transactions and policy revision invalidation; profile edits reject stale versions. +- Dashboard mutations use session-bound CSRF and database-backed role checks; personal tokens are not browser sessions. +- Token/password values, prompts, request bodies, and raw credentials are absent from audit and admin projections. +- Member account, model, and usage views are scoped to the logged-in user; query/list projections are bounded. +- Existing Codex/OpenCode portable configuration and helper ownership remains unchanged; `EGGPOOL_API_KEY` receives a per-user token. + +## 6. Failure and recovery review + +User creation commits the forced password-change state, managed-mode transition, +and audit record atomically. Profile edits use compare-and-swap; stale updates +return a conflict. Password reset, disable, and token revocation invalidate +credentials/sessions before subsequent requests. A user created without an +assigned profile remains under the member default-deny policy while assignment +completes. Migration 60 is append-only and audit writes share mutation +transactions. Cross-version binary rollback still requires restoring a +compatible pre-upgrade database backup; this is qualified in M007. + +## 7. Migration and compatibility review + +Migration 60 is additive and checksum-locked. Migration inventory, runtime +asset manifest, fresh-database counts, and historical schema upgrade tests now +expect versions 1–60. `/v1/models` and the versioned integration profile retain +their existing response contracts while applying user-specific visibility. +Codex/OpenCode configurations continue to consume the same versioned profile +and environment-key contract. + +## 8. Security review + +The two-user HTTP journey confirms admin/member separation, forced initial +password change, same-origin/CSRF enforcement, one-time token display, +non-admin denial, and immediate token revocation. Admin browser routes require +a browser session even when the legacy dashboard is public; agent tokens cannot +authenticate them. Token-bearing response headers prevent caching/referrer +leaks. The new audit table stores only actor/action/target identifiers/time. + +## 9. Documentation and operations + +Updated the dashboard, database, integrations, CLI, deployment, agent +configuration, and API reference documents. They describe per-user token +setup, member/admin route scope, the UI's estimated budget dimensions, and the +operator-managed TLS proxy requirement for remote browser login. No capacity or +SBC performance statement was added. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| Medium | Matched dashboard parity screenshots/interaction captures for the new admin/member screens were not run because the pinned dashboard oracle worktree was not available. | Visual and interaction parity evidence is incomplete; runtime route/security tests pass. | Run the dashboard parity and interaction harness against the pinned oracle and current build, or record exact intentional differences in M007 closure. | +| Medium | M005's estimated output-token and unpriced-USD limits, instance-wide ceiling, crash/restore qualification, and overhead measurements remain open. | UI does not claim exact enforcement; some budget dimensions remain estimate-based and are not a global instance cap. | Preserve labels and carry the existing M005 closure conditions into M007; create a separate corrective plan if exact caps are required. | +| Low | Physical SBC Argon2/RSS and small-team capacity evidence remain unavailable on this macOS x86_64 host. | No SBC sizing or 49-named-user operating claim is supported. | Collect paired physical SBC/ordinary-host evidence under M007 when provisioned hardware is available. | + +## 11. Roadmap disposition + +M006 is conditionally closed. The capability is implemented and local Rust/tooling +verification is green. Browser parity evidence is explicitly outstanding, and +M005 budget limitations remain visible and carried forward. M007 is promoted to +ready: its hard M006 dependency is satisfied; its physical hardware requirement +is an operational evidence gate for qualification/claims, not a reason to block +preparatory implementation work. + +## 12. Registry updates + +`plans/registry.md` and the source roadmap record M006 as conditionally closed, +link this closure, and promote M007 from blocked to ready. The roadmap remains +active while M007 qualification proceeds. The M006 unblock audit found no +other registered dependent plan in the subsystem or dependency tables. diff --git a/plans/closure/on-prem-user-access/007-status.md b/plans/closure/on-prem-user-access/007-status.md new file mode 100644 index 000000000..59af8fd93 --- /dev/null +++ b/plans/closure/on-prem-user-access/007-status.md @@ -0,0 +1,179 @@ +# On-Prem User Access Milestone 007 — Closure Status + +Status: conditionally closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/007-small-team-qualification-and-migration.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md` + +Repository baseline reviewed: `e035b9a` (M006 conditional closure) + +Implementation commits or pull requests: + +- `a3f2d00a` — M006 admin/member implementation and qualification alignment +- `e035b9a` — conditionally close M006 and promote M007 +- `29c4e2b6`, `235d717` — activate M007 and synchronize its registry status +- `64f0062` — begin M007 closure evidence collection + +## 1. Executive finding + +Local migration, auth, lifecycle, backup, installer, and synthetic loopback +qualification pass. The final local fake-provider run completed its bounded +seven-request matrix, recorded all seven attempts/completions, and left no +pending requests or active reservations. It ran on macOS x86_64 and is not SBC +capacity evidence. No representative Pi/Le Potato hardware was available, so +paired 1/5/10/25/49-user load, SBC Argon2/RSS costs, TTFT/latency percentiles, +WAL behavior under team load, and single-user versus team overhead were not +measured. M007 is therefore conditionally closed: the available local work is +complete, while the physical capacity claims in its plan remain unqualified. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Fresh bootstrap and personal-token operation | `on_prem_identity`, `on_prem_sessions`, `on_prem_admin_ui`, synthetic Q007 loopback run | pass | Fresh local admin bootstrap and personal token authenticate discovery/inference; shared legacy key is not used after managed initialization. | +| Two-user policy, account, and admin isolation | `on_prem_model_policy`, `on_prem_admin_ui`, `on_prem_client_profiles` workspace tests | pass | Distinct browser roles, profile policy, member data fences, token revocation and scoped discovery are exercised. | +| Legacy schema migration | `database_compatibility` (9 passed), migration inventory/build manifest tests | partial | A historical schema-11 fixture migrates through v60; a schema-54 semantic seed is exercised after current migrations. A restored schema-54 snapshot was not used as a source database. | +| Fresh/legacy install ownership and service transitions | tooling tests `test_installer.py`, `test_service_transition.py` | pass | Disposable/synthetic test coverage; no live installation was modified. | +| Backup and restore safety | `cargo test --lib operations::backup -- --test-threads=1` | partial | One backup module test passed; a cross-version schema-54 identity/policy restore was not performed. | +| Server transport and restart/lifecycle | `server_transport` (14), `runtime_lifecycle_r009` (6) | pass | Local loopback and lifecycle tests. | +| Durable terminalization and recovery | `coordinator_finalization` (11), serial workspace suite | pass | Existing cancellation/finalization invariants remain green. | +| Synthetic provider interoperation | `scripts/qualification_live_provider.py --offline-fake` | pass | 7/7 planned finite/streaming surface cases; catalog resolved all 3 fixture models; 7 completions, 0 pending requests, 0 active reservations. This is a short loopback correctness run, not load qualification. | +| Installer/SBC/stability/service tooling | focused tooling selection | pass | 51 passed; fixtures were disposable and did not target a real install or hardware. | +| 1/5/10/25/49 named-user capacity and explicit concurrency levels | physical qualification | not measured | No Pi 5/Le Potato target and no paired ordinary-Linux performance run were available. No capacity claim is made. | +| M006 browser parity screenshots/interaction manifest | frozen dashboard oracle/browser harness | not run | Oracle worktree and required environment paths were unavailable. See M006 closure. | +| Hosted CI | GitHub Actions runs `38497388228`, `38497388225`, `38497388244` | pass | Main CI, Linux Bash/macOS zsh installer qualification, and cargo-deny dependency audit completed successfully after push. | + +## 3. Production and qualification artifact manifest + +- `artifacts/qualification/on-prem-user-access-m007-local.json` — sanitized local Q007 fake-provider evidence; SHA-256 `7942776b031523494afdc1c2f316a8bf99fedcae590b7f5358f0ea544cc25c04`. +- Candidate used by that run: `rust/target/debug/eggpool`; SHA-256 `479c02eac9b9300767e5199f164dc3855536b984b9a616cf0d638443a2691216`. +- Local environment in the artifact: macOS 26.6.2, x86_64 process, Rust 1.98.1, direct transport, loopback fake provider. Host memory was 24 GiB and the workspace volume reported 13 GiB available at inspection time. These facts describe the development host only. +- Physical SBC OS/RAM/disk/build hashes: **not measured**. No physical SBC artifact exists. +- Named-user/concurrency corpus beyond the separate two-user M006 correctness journey: **not run**. +- Performance output (RSS, CPU, Argon2 duration/memory contention, auth cache, SQLite wait/transaction, WAL/checkpoint deltas, TTFT, throughput, p50/p95/p99): **not measured**. + +The saved JSON contains only bounded aggregate facts and synthetic fixture +labels. A review found no personal token, provider key, server key, password, +request body, or prompt in the artifact. + +## 4. Verification executed + +Commands below ran locally. Hosted workflow results are recorded separately. + +### Commands run + +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test database_compatibility -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test runtime_lifecycle_r009 -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_finalization -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --lib operations::backup -- --test-threads=1 +cargo build --manifest-path rust/Cargo.toml --locked +uv sync --frozen +uv run pytest tests/tooling/ -q --tb=short --maxfail=1 +uv run pytest tests/tooling/test_qualification_sbc.py tests/tooling/test_qualification_stability.py tests/tooling/test_installer.py tests/tooling/test_service_transition.py -q --tb=short +uv run python scripts/qualification_live_provider.py --binary rust/target/debug/eggpool --offline-fake --output artifacts/qualification/on-prem-user-access-m007-local.json +uv run ruff format --check scripts/ tests/tooling/ +uv run ruff check scripts/ tests/tooling/ +uv run pyright scripts/ +uv run python scripts/validate_release_docs.py +uv run python scripts/validate_runtime_package_boundary.py +git diff --check +``` + +### Results + +- Rust workspace: 964 passed, 1 ignored across 78 serial suites. +- Focused Rust targets: server transport 14 passed; database compatibility 9; runtime lifecycle R009 6; coordinator finalization 11; backup module 1. +- Strict Clippy default/no-default and no-default workspace check passed; formatting check passed; locked debug build passed. +- Full tooling suite: 166 passed, 1 skipped. Focused SBC/stability/installer/service tests: 51 passed. +- Offline Q007 run: pass, 7/7 requests, 3/3 catalog models; 7 durable completions, no pending request, no active reservation; 4.832 seconds wall duration on the local host. +- Ruff: 59 files formatted, checks passed. Pyright: 0 errors, 0 warnings, 0 informations. +- Release documentation validator: pass, 7 docs checked. Runtime package-boundary validator: pass. `git diff --check`: pass. +- The full Rust suite passed before the final test-only member-role denial assertion; the focused M006 browser target passed again after that assertion. The pushed head then passed hosted CI run `38497388228` in 7m33s, including format, default/no-default Clippy/check, serial workspace tests, Ruff, Pyright, and the full tooling suite. Hosted installer qualification `38497388225` passed on Linux Bash and macOS zsh; dependency audit `38497388244` passed. No-default tests were not run because CI requires no-default check/Clippy only. + +## 5. Invariant review + +- Qualification used isolated temporary config/database/runtime paths and a fake loopback provider; it used no real provider credential or contributor deployment. +- The Q007 synthetic admin was created with stdin-only password input. Its personal token was used only for inference/discovery and is redacted from diagnostic output. +- User identity and policy remain downstream of upstream provider accounts; provider credentials remain in environment/config ownership. +- Database ownership remains one connection/gate/worker with WAL/NORMAL; this closure added no runtime concurrency or storage architecture change. +- No hosted/SaaS, enterprise identity, 50-concurrent-generation, or 49-user capacity claim was introduced. + +## 6. Failure, restart, and recovery review + +The two-user M006 loopback covers first-login password change, admin/member +role denials, CSRF, token issue/revoke, model visibility and last-admin safety. +The synthetic provider run confirms all bounded cases terminalized without +leftover reservations. Database compatibility, lifecycle, finalization, +server transport, installer, and service-transition suites pass. A true +restore from a schema-54 backup into this build, power-loss test on SBC, and +paired cross-version backup rollback were not performed; they remain explicit +conditions rather than inferred guarantees. + +## 7. Migration and compatibility review + +Migrations are append-only through v60, checksum-locked, and runtime-manifest +registered. The v11 historical fixture exercises migrations through v60; a +schema-54 routing seed validates existing semantic state after migration. The +backup guide now identifies user, policy, budget, usage, and audit data in the +SQLite snapshot and states that downgrade requires restoring a matching +pre-upgrade backup. Exact schema-54 snapshot restoration remains unverified. + +## 8. Security review + +Synthetic qualifications used no real secrets. Browser-role/CSRF/token revocation +coverage is in the M006 two-user journey; member tokens are denied admin pages. +The local Q007 artifact was checked for token/key/password/body leakage. The +plan's larger selector-bypass, quota-last-slot, process-kill, database +corruption/read-only, and target-specific power-loss campaign has no new +physical result here; existing Rust policy, budget, finalization, and recovery +tests passed within the serial suite. + +## 9. Documentation and operations + +`docs/backup-restore.md` now identifies schema migrations 55–60, included +identity/policy/budget/usage/audit state, and forward-only rollback behavior. +The deployment, API, CLI, agent configuration, dashboard, integrations, +database, and security docs describe verified M006 behavior. Raspberry Pi and +release docs continue to make no user-count or concurrency claim. Browser +passwords on remote LAN access still require an operator-managed TLS proxy and +trusted-proxy configuration. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| Medium | No physical Pi 5/Le Potato SBC or paired ordinary-Linux target was available. | M001 Argon2 latency/RSS and M007 resource/capacity operating points are unqualified; 49 named users and any concurrent completion target cannot be claimed. | Run the frozen corpus on explicitly provisioned hardware with OS/RAM/disk/build hashes, 1/5/10/25/49 named users, bounded concurrency, paired single-user baseline, and all required CPU/RSS/auth/SQLite/WAL/TTFT/percentile observations. | +| Medium | A restored schema-54 backup and cross-version identity/policy restore were not exercised. | Existing v11-to-v60 migration coverage and backup tests do not prove the exact schema-54 restore path. | Restore an actual protected schema-54 fixture into the target build; verify user IDs, credential revocation semantics, policy, budget holds/rollups, audit, and documented rollback to the matching old binary. | +| Medium | M005 exact output-token and unknown-price USD caps, global instance ceilings, crash/restore qualification, and overhead measurements remain open. | Some user limits are estimates and there is no instance-wide aggregate cap; M006 UI labels those limits as estimates. | Preserve the M005 conditions; create a separate corrective plan if exact enforceable limits are required. | +| Medium | M006 browser parity captures for newly added admin/member pages were not run. | Route/security evidence passes, but matched screenshot/interaction evidence is absent. | Run the pinned dashboard oracle/browser parity harness and record matched pages or intentional differences. | + +## 11. Roadmap disposition + +M007 is conditionally closed for the local code and synthetic qualification work. +The subsystem roadmap has no remaining implementation plan in this line and is +closed for current engineering work. Physical SBC capacity, exact schema-54 +restore, dashboard parity, and inherited M005 limits remain conditions; they +do not qualify the 49-named-user target. Any later capacity claim or corrective +implementation needs an explicitly bounded follow-up with the required hardware +and evidence. + +## 12. Registry updates + +`plans/registry.md` and the source roadmap record M007 as conditionally closed, +link this closure, mark the subsystem roadmap closed, and remove M007 from the +active/ready/blocked tables. The M007 unblock audit found no registered +downstream plan to promote and no successor in this line. All M001–M007 plans +now have closure records; M001/M003/M005/M006/M007 retain their explicitly +named conditional evidence. diff --git a/plans/closure/on-prem-user-access/008-status.md b/plans/closure/on-prem-user-access/008-status.md new file mode 100644 index 000000000..dc289a469 --- /dev/null +++ b/plans/closure/on-prem-user-access/008-status.md @@ -0,0 +1,164 @@ +# On-Prem User Access Milestone 008 — Closure Status + +Status: closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/008-hard-token-cost-admission-corrective.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md#m008--enforceable-output-token-and-price-bound-admission-corrective` + +Repository baseline reviewed: `ca77c547` (M008 proof-carriage coverage) + +Implementation commits or pull requests: + +- `0149a6c8` — Implement on-prem user access M008 hard token/cost admission corrective +- `ca77c547` — Cover M008 proof carriage from admission through claim snapshot +- `49a0f845` — Activate on-prem user access M008 (planning status only) + +## 1. Executive finding + +Enforceable token/cost admission is implemented as a bounded corrective to the +conditionally closed M005/M006/M007 records, which remain immutable. Every +budget rule now carries an explicit `hard` or `estimated` enforcement mode +(pre-existing rows are `estimated`; promotion is an explicit admin +operation). Hard token rules admit only requests with a provable worst-case +bound and hard USD rules additionally require an operator-configured provider +tariff; anything unprovable is denied inside the publication transaction with +`BudgetNotEnforceable` before claim conversion, so zero upstream submissions +occur. Estimated rules behave exactly as M005 left them. The capability +boundary is complete on local evidence; no medium-or-higher finding remains. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Explicit hard vs estimated enforcement semantics (§5A) | `rust/src/user_budget.rs::BudgetEnforcement`; migration 0061; `UserBudgetRule.enforcement` in `/api/user-budget`, dashboard, CLI | pass | Request/concurrency caps stay strict in both modes; old rows map to estimated. | +| Typed per-surface output-capability contract (§5B) | `request/admission.rs::budget_admission_facts` + `BudgetAdmissionFacts`; chat `max_tokens`/`max_completion_tokens`, Responses/compact `max_output_tokens`, Messages `max_tokens`; `admission_extracts_per_surface_output_bounds` | pass | Caller fields are read, never mutated; absent/invalid caps degrade to unbounded and fail closed under hard rules. | +| Proof-or-reject before provider dispatch (§5B) | `check_initial`/`check_retry` return `NotEnforceable`; `PublicationError::BudgetNotEnforceable` → `EndpointError::BudgetNotEnforceable` (429, stable message); zero-row assertions in `hard_token_rule_denies_unbounded_request_without_upstream_io`, `hard_token_rule_denies_missing_proof_without_upstream_io` | pass | Denial happens inside the publication transaction before claim conversion. | +| Pricing trust boundary (§5C) | `ProviderConfig.budget.max_microdollars_per_million_tokens` + validation; `hard_usd_rule_denies_unknown_price_without_upstream_io`, `hard_usd_rule_admits_with_operator_tariff_hold`, `zero_cost_tariff_admits_hard_usd_without_hold`, `provider_tariff_overflow_is_rejected_by_config_validation` | pass | `pricing.fallback=generic_estimate` is never a hard price; `None` = unknown (deny), `Some(0)` = verified free; overflow fails closed, never saturates. | +| Retry/selector cost policy (§5A) | Initial hard hold covers two attempt-equivalents (concrete + unattributed selector overhead); retries extend by one equivalent after re-check (`hold_for_admission`/`hold_for_retry`); `hard_retry_denies_extension_beyond_remaining_budget`, `hard_retry_extends_within_remaining_budget` | pass | Retry path re-derives hard flags from current rules, so post-admission promotion still gates retries. | +| Transaction integration, no check-then-dispatch gap (§5D) | Proof assembled by coordinators from immutable claim-snapshot facts + generation tariff snapshot (`budget_proof_for_claim` in finite + streaming paths); atomic check+hold in `publish_transaction`; `hard_proof_flows_from_admission_through_claim_snapshot` | pass | In-flight holds keep their generation's tariff; tariff edits affect new requests only, matching rule-edit semantics. | +| UI/API/CLI truthful projection (§5D) | Dashboard budget form mode select + member mode column; `/api/user-budget` rule `enforcement`; CLI `--enforcement hard\|estimated` (default estimated); `on_prem_admin_ui` hard-form round trip + invalid-mode 400; `cli_contract` default/boundary cases | pass | Old dashboard forms without the field post as estimated. | +| Compatibility migration (§9) | Additive migration 0061 (defaults `estimated`, NULL price source); `rules_validate_admin_scope_and_usage_reads_are_bounded` asserts estimated default; historical fixture upgrades through 61 in `database_compatibility` | pass | No silent upgrade; downgrade still needs a pre-upgrade backup. | +| Settlement honesty, no hidden overruns (§5D/§8) | `hard_settlement_records_provider_overrun_without_hiding` (400 observed vs 300 held charged visibly, unknown counters zero) | pass | Unknown usage still consumes the conservative hold. | +| Docs (§12) | `docs/model-limits.md`, `docs/api-reference.md`, `docs/cli-reference.md`, `docs/deployment.md`, `architecture/deep-dive-request-lifecycle.md`, `architecture/deep-dive-database.md`, `rust/config.example.toml` tariff example | pass | `scripts/validate_release_docs.py` passes. | +| Non-goals respected (§6) | No billing reconciliation, no cost-based routing, no second SQLite writer, no provider-response mutation, no wire-format change | pass | Tariff is provider-wide; per-model refinement explicitly deferred. | + +## 3. Production implementation evidence + +Landed ownership/storage/protocol/runtime/operator changes: + +- Storage: migration `0061_user_budget_enforcement.sql` (rule/audit `enforcement`, reservation `enforcement`/`bound_tokens`/`bound_microdollars`/`price_source`, enforcement index); `checksums.json` + `runtime-manifest.json` (count 61, refreshed asset hashes); inventory guards updated (`migrations.rs`, `build_manifest.rs`, `database_compatibility.rs`). +- Admission: `BudgetAdmissionFacts` computed in `admit_parsed_request`/`admit_compact_parsed_request` from the parsed body before it moves; carried via `RoutingRequestFacts` → `SelectionSnapshot` → `SelectionClaim` getters. +- Enforcement: `user_budget.rs` hard/estimated rule evaluation with worst-case holds (`single = input_worst + output`, initial ×2, retry ×1, tariff conversion with checked i128 arithmetic); `BudgetCheck::NotEnforceable`; reservation rows record mode/bounds/provenance. +- Coordinators: finite + streaming attach `budget_proof_for_claim` (snapshot facts + selected-provider tariff) to every `PublicationInput`; publication enforces atomically on initial and retry paths. +- Endpoint: `EndpointError::BudgetNotEnforceable` → 429 with stable message, mapped from both coordinators. +- Operator: `ProviderConfig.budget` tariff with `i64` range validation (live-reloadable under the existing `providers.` disposition; per-request snapshot semantics documented); CLI `--enforcement`; dashboard form + mode display; API rule mode. +- Tests: `on_prem_budgets` 4 → 19 tests (15 new hard/compat/carriage cases); `on_prem_admin_ui` hard-form round trip + mode display; `cli_contract` enforcement default/boundary. + +Planned but absent: nothing in the source plan's scope is unimplemented. Per-model +tariffs and an HTTP-level hard-denial test remain explicit non-goals/follow-ups +(see §10). + +## 4. Verification executed + +### Commands run + +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test wire_extraction_contract -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test wire_kernel_boundary -- --test-threads=1 +git diff --check +uv run python scripts/validate_release_docs.py +``` + +Focused targets from the source plan §10 were run explicitly before the full +suite: `on_prem_budgets`, `coordinator_publication`, `coordinator_finalization`, +`coordinator_c011`, `on_prem_model_policy`, `on_prem_admin_ui`, +`codex_compaction_compat`, `wire_runtime`, `database_compatibility`, +`cli_contract`, plus `on_prem_attribution` and `server_transport` for the +touched coordinator/endpoint paths. New coverage is concentrated in +`on_prem_budgets`/`on_prem_admin_ui`/`cli_contract`; the remaining targets +verify no regression. No Cargo dependency/feature changed, so `cargo deny`, +the feature tree, and the locked release build were not re-run (nothing they +guard changed). No Python tooling files changed, so the ruff/pyright/pytest +tooling matrix was not re-run. + +### Results + +Local (this machine, `current_thread`/serial as required): + +- Full default workspace/all-targets serial suite: 978 passed, 0 failed + (77 suites; includes 19 `on_prem_budgets`, 11 `coordinator_publication`, + 10 `coordinator_finalization`, 18 `coordinator_c011`, 9 + `database_compatibility`, 13 `codex_compaction_compat`). +- Full no-default workspace/all-targets serial suite: 979 passed, 0 failed. +- Strict Clippy default + no-default, `fmt --check`, `git diff --check`: + pass. Wire seam guards (`wire_extraction_contract` 15, + `wire_kernel_boundary` 4): pass. Release-docs validator: pass. +- Hosted CI: not run from here; PR #7 checks execute on push. Physical + hardware: not required by this plan; SBC/small-team measurement belongs to + M012. No test was claimed without running it. + +## 5. Invariant review + +- One local/SBC instance, optional <50 named users; no SaaS/tenancy/billing/IAM/WAN additions. Upstream keys stay in TOML/environment; no user-table or admin-UI key surface added. +- Generation leases, coordinator publication/finalization ownership, single SQLite gate/worker with WAL/NORMAL, canonical wire, transparent streaming, bounded hot-path memory, provider fairness: unchanged (no topology, runtime, or streaming change; proof facts are bounded scalars). +- Budget refusal never marks provider unhealthiness: denials roll back the routing claim (`rollback_after_error`) without health effects. +- Published Chat/Responses/compact/Messages behavior, auth headers, client profiles, dashboard themes, legacy upgrade: unchanged; admission never mutates caller output fields. +- Identity/cost metadata redaction preserved; no secrets/prompts/bodies in logs, HTML, or artifacts; DB failures and unprovable bounds fail closed. +- Routing stays deterministic and load-based; selector decisions still precede provider/account routing with no account pinning, and selector spend is covered by the documented upfront equivalent rather than a separate hold. + +## 6. Failure and recovery review + +- Unbounded output, missing proof, unknown tariff, non-positive or overflowing bounds all deny as `NotEnforceable` inside the publication transaction; zero-row assertions prove no partial request/reservation survives. +- Retry re-checks current rules and extends transactionally; over-budget retries deny as `Exceeded` without touching the surviving hold. +- Rule/tariff edits are atomic and new-request-only; in-flight requests keep their permit and held amounts. +- Cancellation/duplicate-finalization paths are unchanged from M005/M007; settlement stays idempotent, records observed usage visibly, and preserves holds until terminal settlement. A dedicated budget crash/restore fault campaign is M010's scope, not this plan's. + +## 7. Migration and compatibility review + +Migration 0061 is additive with conservative defaults (`estimated`, zero bounds, NULL price source); fresh databases migrate to 61 and the pre-61 historical fixture upgrades through 61 with repositories round-tripping. Stored user rules, provider accounts, historic consumption, and API schemas remain readable; old dashboard forms and CLI invocations without the new mode field behave as estimated. Wire bytes are untouched. Downgrade still requires a matching pre-upgrade backup. + +## 8. Security review + +Rule promotion requires an active admin (CLI actor, CSRF + admin session, audit row with mode). Member reads stay owner-scoped; hard-mode denial messages are stable, secret-free, and machine-readable without leaking limits, usage, tariffs, or identity. Tariff validation rejects out-of-range values; budget arithmetic uses checked i128 and fails closed on overflow. No password/token/bearer material enters the new columns, logs, or snapshots. The tariff is security-relevant operator configuration: it is covered by the existing `providers.` reload disposition with per-request snapshot semantics, so a live tariff change cannot retroactively weaken in-flight holds. + +## 9. Documentation and operations + +Updated `architecture/deep-dive-request-lifecycle.md` (admission/enforcement semantics), `architecture/deep-dive-database.md` (migration 61, hold facts, API mode), `docs/model-limits.md` (hard/estimated contract, tariff, denial code), `docs/api-reference.md` (rule mode, 429 semantics), `docs/cli-reference.md` (`--enforcement`), `docs/deployment.md` (tariff configuration), and `rust/config.example.toml` (commented tariff). Formal accounting assumptions (byte-length input bound, per-surface output fields, ×2 initial / ×1 retry equivalents, tariff-only hard USD, overflow fail-closed, per-model tariff non-goal) are stated identically in code docs, user docs, and §2 above. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| Low | Hard-mode holds over-reserve by design (×2 upfront); unknown-usage settlement charges the conservative hold. | Usage accounting reads high until observed usage settles; visible via unknown counters and observed columns. | None — conservative by design; refine only under a future bounded plan with measured selector/usage evidence. | +| Low | Tariff is provider-wide and must cover the provider's most expensive model. | Operators with mixed free/paid models on one provider must set the paid ceiling or split providers. | Documented; per-model tariffs need a separately approved bounded plan. | +| Low | No HTTP-level hard-denial test through `execute_endpoint`; denial is proven at the publication gate plus endpoint mapping. | A future endpoint refactor could only break this via the covered gate/mapping, both tested. | Optional follow-up test under a later milestone; not a correctness gap. | + +No critical, high, or medium findings. No physical-capacity or hard-spend claim is made beyond the tested worst-case reservation contract. + +## 11. Roadmap disposition + +M008 is closed. Its hard predecessor requirement on M009 is satisfied: the +enforcement-mode and conservative reservation contract M009 must consume is +landed and verified, so M009 is promoted to ready in the same commit. M010 +remains blocked on M008+M009 closures (still needs M009). M011 stays ready and +independent. M012 remains blocked on M008–M011 closures plus operational +hardware. No predecessor closure record was altered. + +## 12. Registry updates + +Applied in the same commit: M008 plan `active` → `implemented` with this +closure recorded as `closed`; M009 plan `blocked` → `ready`; roadmap M008 +`active` → `closed`, M009 `blocked` → `ready`; M008 row moved from active to +recently-closed; blocked-work table retains M010/M012 with unchanged reasons. +Historical M005/M006/M007 records remain immutable. diff --git a/plans/closure/on-prem-user-access/009-status.md b/plans/closure/on-prem-user-access/009-status.md new file mode 100644 index 000000000..4fde27dec --- /dev/null +++ b/plans/closure/on-prem-user-access/009-status.md @@ -0,0 +1,170 @@ +# On-Prem User Access Milestone 009 — Closure Status + +Status: closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/009-global-budget-intersection-corrective.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md#m009--instance-global-budget-and-scope-intersection-corrective` + +Repository baseline reviewed: `04264a18` (M009 implementation) + +Implementation commits or pull requests: + +- `04264a18` — Implement on-prem user access M009 global budget intersection corrective +- `372f0e1d` — Activate on-prem user access M009 (planning status only) + +## 1. Executive finding + +The instance-global budget ceiling is implemented as a new corrective to the +conditionally closed M005 record, which remains immutable. Exactly one +instance-wide scope exists (all-model rules plus optional exact-model rules, +no tenant identifier, no second database): every admitted call — attributed or +legacy-unattributed — must satisfy all applicable instance and user rules in +the same publication SQLite transaction, or it is denied before claim +conversion with zero upstream submission. The M008 enforcement-mode and +conservative reservation contract is reused unchanged, including hard-mode +proof-or-reject and the operator-tariff trust boundary. Concurrent requests +from distinct users cannot both take the final global slot (proven for 2, 10, +and 49 identities), retries never double-count, and global counters never +depend on a user row. The capability boundary is complete on local evidence; +no medium-or-higher finding remains. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| One instance-owned budget namespace, no tenant hierarchy (§5A) | Migration 0062 (`instance_budget_rules` UNIQUE `(ifnull(model,''), window)`, no user column); single-instance scope documented in `instance_budget.rs` header | pass | Day/month UTC windows, exact-model or all-model scope, malformed/overflow rejection, admin-only mutation. | +| Intersection: all applicable instance/user rules must allow (§5A) | Publication checks user scope then instance scope in one transaction; `global_and_user_rules_intersect_without_bypass` (global-spent denies despite user headroom; second user still capped) | pass | Absent instance rules mean no additional cap and no instance rows. | +| M008 hard/estimated semantics per dimension (§5A) | Shared `hold_for_admission`/`hold_for_retry`, tariff-only hard USD, `NotEnforceable` propagation; `global_hard_token_and_usd_deny_unprovable_without_upstream_io` (unbounded + unpriced deny with zero rows; proven request holds 300 tokens / 900 micros with tariff provenance) | pass | Instance hard flags re-derive from current rules on every path. | +| Atomic combined reservation, no double last-slot (§5B) | Same-txn check+insert for both scopes; `global_request_last_slot_admits_exactly_one_across_users` (1 winner + 1 denial, 1 request + 1 hold); 10- and 49-identity races with exactly one winner each | pass | Single SQLite gate serializes the race; see gate observations below. | +| One logical request across failover/selector/retry (§5B) | Retry re-check consumes no new request slot (`new_request=false` in both evaluators); `global_retry_and_settlement_count_one_logical_request` (retry extends, terminal folds 57 tokens / 25 micros across two attempts into request_count 1 per bucket) | pass | Fixed during implementation: the first draft consumed a request per retry and the new test caught it. | +| Bounded aggregates, no full scans, one gate (§5B) | Indexed window/window_start/scope sums in `rules_exceeded`; per-model usage rows with cross-model SUM only for all-model rules; no second writer; `settle` folds shared attempt rows | pass | 10-identity race 5.4 ms, 49-identity race 15.8 ms on local host (observation only, not a capacity claim). | +| History independent of user rows (§5B) | No user FK on any instance table; `disabled_user_does_not_erase_global_history` (usage intact after disable, remaining headroom usable) | pass | `user_budget_usage` keeps its RESTRICT semantics; instance history never joins `users`. | +| Admin CRUD + observable state, member fencing (§5C) | CLI `users global-budget set/remove`; dashboard instance section with set/remove; `/api/user-budget` `instance_rules`/`instance_usage` for admins only; `on_prem_admin_ui` set/remove round trip + member 403 + invalid-mode rejection; `global_rules_validate_admin_scope_and_reject_malformed` | pass | Members keep their existing owner-scoped view (their effective user-side cap); global aggregates are admin-visible only, so members cannot infer others' consumption. | +| Rollover with active holds (§8) | Window-CASE hold projection shared with the user scope; `global_window_projection_ignores_crossed_utc_boundaries` (backdated day+month holds free today's/month's slots while concurrency still counts live holds; third request denied) | pass | True clock travel is not simulable in-process; the projection SQL is the reviewed shared pattern. | +| No-config single-admin parity (§9/§13) | `no_global_rules_preserves_single_admin_parity` (attributed + legacy requests take no hold, write no rows); rule-free focused targets green unchanged | pass | Measured overhead of the rule-free path is the two extra indexed rule lookups; see §10. | +| Legacy/identity-switch bypass closed (§13) | Instance checks run for unattributed requests too; `legacy_unattributed_requests_count_against_global_ceiling` + `legacy_unattributed_settlement_folds_instance_usage` | pass | Required an `record_attempt_usage` refactor so attempt facts exist without a user reservation. | +| Docs (§12) | `docs/model-limits.md` (new instance section), `docs/api-reference.md` (admin fields, member fencing), `docs/cli-reference.md` (global-budget rows), `docs/deployment.md` (ceiling paragraph), dashboard help text, `architecture/deep-dive-database.md` + `deep-dive-request-lifecycle.md` | pass | Provider quotas documented as unrelated quantities. | +| Non-goals respected (§6) | No multi-org/external-billing/sync, no per-provider credential quotas, no substitution after refusal, no second pool/writer | pass | Tariff stays provider-wide per M008 (documented non-goal). | + +## 3. Production implementation evidence + +Landed ownership/storage/protocol/runtime/operator changes: + +- Storage: migration `0062_instance_budgets.sql` (rules, reservations without user FK, per-model usage buckets, audit, policy-revision triggers); `checksums.json` + `runtime-manifest.json` (count 62, refreshed hashes); inventory guards updated (`migrations.rs`, `build_manifest.rs`, `database_compatibility.rs` incl. historical upgrade through 62). +- Service: `rust/src/instance_budget.rs` owns `InstanceBudgetService` (admin-gated CRUD, bounded reads), the pub(crate) check/insert/extend/settlement contract reusing M008 hold math, and per-scope bounded evaluation. Shared M008 helpers (`hold_for_admission`, `hold_for_retry`, tariff/worst-case math, `exceeds`, `BudgetPriceSource::as_str`) are now `pub(crate)`. +- Coordinator: `publication.rs` evaluates the instance intersection on initial (all requests) and retry (hold-bearing requests only) paths in the same transaction, inserting/extending the shared hold atomically; `finalization.rs` folds instance settlement in the same terminal transaction. `user_budget::record_attempt_usage` was split out so attempt facts exist for unattributed requests. +- Operator: CLI `users global-budget set/remove`; dashboard `/users` instance section with set/remove forms; API instance fields for admins, unchanged member shape. +- Tests: new `rust/tests/on_prem_global_budgets.rs` (13 tests: cross-user last-slot, 10/49-identity races with timings, intersection, model sharing, retry counting, concurrency release, rollover projection, legacy coverage + settlement, disable-durability, hard proof-or-reject, validation/fencing, parity); `on_prem_admin_ui` instance round trip + member 403; `cli_contract` global-budget parse cases. + +Planned but absent: nothing in the source plan's scope is unimplemented. The +dedicated crash/restore fault campaign is M010's scope; physical SBC/small-team +measurement is M012's. + +## 4. Verification executed + +### Commands run + +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_global_budgets global_last_slot_race -- --test-threads=1 --nocapture +git diff --check +uv run python scripts/validate_release_docs.py +``` + +Focused targets from the source plan §10 were run explicitly before the full +suite: `on_prem_budgets` (19, M008 regression over the touched +publication/finalization paths), `on_prem_global_budgets` (13 new), +`on_prem_admin_ui` (2), `database_compatibility` (9), +`coordinator_publication` (11), `coordinator_finalization` (10), +`server_transport` (14), `cli_contract` (7). No Cargo dependency/feature +changed, so `cargo deny`, the feature tree, and the locked release build were +not re-run. No Python tooling files changed, so the ruff/pyright/pytest +tooling matrix was not re-run. + +### Results + +Local (this machine, serial as required): + +- Full default workspace/all-targets serial suite: 992 passed, 0 failed + (79 suites; 978 carried from M008 + 13 global + 1 CLI contract). +- Full no-default workspace/all-targets serial suite: 993 passed, 0 failed. +- Strict Clippy default + no-default, `fmt --check`, `git diff --check`: + pass. Release-docs validator: pass. +- DB gate observations (ordinary local host, in-memory SQLite, NOT SBC or + capacity evidence): global last-slot race with 10 identities took + ~5.4 ms, with 49 identities ~15.8 ms wall time including claim setup; + exactly one winner each. Recorded here as the plan's controlled-concurrency + observation; no SBC performance is asserted. +- Hosted CI: not run from here; PR #7 checks execute on push. Physical + hardware: not required by this plan; belongs to M012. No test was claimed + without running it. + +## 5. Invariant review + +- One local/SBC instance, optional <50 named users; no SaaS/tenancy/billing/IAM/WAN additions. No upstream keys in user/instance tables or admin UI. +- Generation leases, coordinator publication/finalization ownership, single SQLite gate/worker with WAL/NORMAL, canonical wire, transparent streaming, bounded hot-path memory, provider fairness: unchanged. Instance work adds indexed lookups inside the existing publication/finalization transactions only. +- Downstream budget refusal (user or global) never marks provider unhealthiness: denials roll back the routing claim without health effects. +- Published wire behavior, auth headers, client profiles, dashboard themes, legacy upgrade: unchanged. +- Identity/cost metadata redaction preserved; member fencing verified (403 on instance mutation, no instance fields in member API, no cross-user usage on member pages); DB failures and unprovable hard bounds fail closed. +- Routing stays deterministic and load-based; budgets never influence provider/account selection. + +## 6. Failure and recovery review + +- Cross-user last-slot races converge to exactly one winner at 2, 10, and 49 identities; losers get `BudgetExceeded` with no partial rows. +- Retry under a spent global slot denies without consuming a second request; retry within budget extends transactionally. +- Global hard rules deny unprovable requests as `NotEnforceable` with zero rows; instance hard flags re-derive from current rules on every path. +- Rule edits are atomic and new-request-only; in-flight holds keep their permit and amounts. +- Settlement is idempotent per scope (settled/released short-circuit in both evaluators); user and instance folds share one terminal transaction, so crash leftovers converge together. The dedicated budget crash/restore fault matrix is M010's scope. +- Malformed/overflowed rules are rejected at the service boundary; member mutation attempts fail closed at service and HTTP layers. + +## 7. Migration and compatibility review + +Migration 0062 is additive with conservative defaults; fresh databases migrate to 62 and the pre-62 historical fixture upgrades through 62 with repositories round-tripping. M005/M008 user rows, provider accounts, historic consumption, and API schemas remain readable; member API responses are byte-compatible with M008 (new fields are admin-only additions). Absent global rules, publication takes no instance hold and writes no instance rows — the one-admin workload is identical except the two bounded rule lookups. Downgrade still requires a matching pre-upgrade backup. + +## 8. Security review + +Instance rule mutation requires an active admin at every layer (CLI actor, CSRF + admin session, audit row with actor/version/mode); member and personal-token credentials get 403 and cannot select another scope. Instance reads are admin-gated in API and dashboard; usage aggregates never join user identities into member-visible output. No secret, prompt, body, or credential enters instance tables, logs, snapshots, or artifacts. Integer validation and checked worst-case arithmetic match the M008 contract; global denial messages carry no limits, usage, tariffs, or identity. + +## 9. Documentation and operations + +Updated `architecture/deep-dive-database.md` (migration 62, scope tables, bounded aggregates, admin-only visibility), `architecture/deep-dive-request-lifecycle.md` (same-transaction intersection, no-double-count, rule-free no-op), `docs/model-limits.md` (instance-ceilings section: scope, atomicity, no-bypass list, durability, fencing), `docs/api-reference.md`, `docs/cli-reference.md`, `docs/deployment.md`, and dashboard help text. Formal assumptions (one instance scope, per-scope independent holds, shared M008 hard contract, bounded indexed evaluation, no-bypass including legacy keys, admin-only global visibility) are stated identically in code docs, user docs, and §2 above. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| Low | A request under both a user-hard and a global-hard token cap holds the worst case twice (once per scope). | Conservative over-reservation when both scopes are hard; each hold settles independently against its own scope. | None — intersection semantics require independent per-scope holds; refine only under a future bounded plan with cross-scope evidence. | +| Low | Rule-free requests still execute two bounded indexed rule lookups per publication. | Sub-millisecond local overhead (see §4 timings); no row writes. | None — measured and bounded; SBC-side quantification belongs to M012. | +| Low | UTC-boundary behavior is proven via backdated holds, not clock travel. | The window-projection SQL is covered but a real midnight rollover was not observed in-process. | Optional: observe a live UTC reset during M012 physical qualification; not a correctness gap. | + +No critical, high, or medium findings. No physical-capacity or hard-spend claim is made beyond the tested reservation contract. + +## 11. Roadmap disposition + +M009 is closed. Unblock audit (same commit): M009's hard M008 dependency was +already satisfied by the accepted M008 closure. M010's hard dependency (M008 +and M009 accepted closures with a stable global user-budget state machine) is +now fully satisfied — the instance scope, intersection, and settlement it must +consume are landed — so M010 is promoted to ready. M011 stays ready and +independent (parallel browser work, unaffected by this closure). M012 remains +blocked: it additionally needs M011 closure plus provisioned physical hardware. +No predecessor closure record was altered. + +## 12. Registry updates + +Applied in the same commit: M009 plan `active` → `implemented` with this +closure recorded as `closed`; M010 plan `blocked` → `ready`; roadmap M009 +`active` → `closed` with this closure linked and roadmap M010 `blocked` → +`ready`; M009 row moved from active to recently-closed; blocked-work table +retains M012 with its unchanged reason. Historical M005/M007/M008 records +remain immutable. diff --git a/plans/closure/on-prem-user-access/010-status.md b/plans/closure/on-prem-user-access/010-status.md new file mode 100644 index 000000000..ad2d6d739 --- /dev/null +++ b/plans/closure/on-prem-user-access/010-status.md @@ -0,0 +1,194 @@ +# On-Prem User Access Milestone 010 — Closure Status + +Status: closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/010-budget-crash-recovery-and-schema54-restore.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md` + +Repository baseline reviewed: `be6248fc` (M010 implementation) + +Implementation commits or pull requests: + +- `542a2f28` — Activate on-prem user access M010 (planning status only; records actual baseline `1218423b`/schema 62, superseding the plan text's stale `c18dcc20`/schema 60) +- `be6248fc` — Implement on-prem user access M010 budget crash recovery and schema-54 restore + +## 1. Executive finding + +The crash-state audit proved one real defect and three non-defects. The +defect: startup reconciliation terminalized `pending` requests but never +touched user/instance budget holds, and no expiry sweeper exists, so any +crash between publication commit and terminal settlement leaked a phantom +`active` hold that permanently occupied quota (hard caps could wedge; the +single-concurrency-slot case wedged the whole instance). The fix is +additive and narrow: `fix_bounded` releases active budget holds on the +requests it interrupts, in the same transaction, without folding usage. +Double-counting is provably impossible (idempotent attempt facts, +settle-once guards, no accounting writes in reconciliation), dispatched +attempts always carry their durable hold (check+hold commits strictly +before `submit_once`), and committed holds cannot vanish (single-writer +transactions, WAL durability, retention spares nonterminal rows). A frozen +byte-valid schema-54 image restores and migrates forward 55→62 with legacy +history intact and no fabricated identity; ordinary-host gate observations +show sub-millisecond budget overhead with no gate-tenure concern. The M003 +updater-test condition is a stale historical failure: both flagged tests +pass in isolation and in the full serial lib run at this SHA. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Crash-state transition audit (§A) | §3 matrix below, built from branch code with file:line references | pass | One real defect (phantom holds); dispatch ordering, double-count, and hold-loss proved safe. | +| Additive budget recovery, no core behavior change (§A) | `rust/src/coordinator/reconciliation.rs::fix_bounded` gains two conditional bounded `active→released` updates in the existing transaction; `classify_bounded` gains two counts; report/aggregate gain four fields; no publication/finalization/settlement line changed | pass | Reconciliation still inserts nothing and mutates no usage bucket or attempt fact. | +| Fault injection before/after commit, acceptance, mid-stream, finalizer, duplicate terminal, cancel, backup-during-hold (§B) | 7 new `on_prem_budgets` tests + 2 new `on_prem_global_budgets` tests + backup round-trip test; distinct durable shapes covered (pending+user hold, pending+user+instance holds, compensated+retried+extended hold, settled, cancelled, duplicate terminal, backup image) | pass | Assertions converge on rows/reconciler reports, no sleeps. Mid-stream and post-acceptance crashes share the pending+active-hold durable shape, covered by construction. | +| Independent schema-54 restore + forward upgrade (§C) | Frozen `tests/fixtures/database/schema54-pre-upgrade.sql` (SHA-256 `a76d43047986282d271ce99527f5827c743f50860625c9ffd44963a306fd42af`); 2 new `database_compatibility` tests | pass | Ledger 1..54, checksums, integrity, upgrade 55..62, bootstrap/login/tokens/profiles/rules, restart, backup/restore round trip. | +| Storage-pressure observations 1/10/49 (§D) | `paired_gate_observations_across_identities_and_rules` + `artifacts/qualification/m010-gate-observations.json` | pass | Sub-ms overhead, worst outlier 2.8ms, WAL truncates cleanly; not SBC capacity evidence (M012 owns that). | +| M003 updater condition disposition (§E) | Both flagged tests pass isolated + full serial lib (164/164) at this SHA | pass | Stale historical failure; no separate corrective opened; M003 history untouched. | +| Docs (§12) | `architecture/deep-dive-request-lifecycle.md`, `architecture/deep-dive-database.md`, `docs/backup-restore.md` | pass | `docs/deployment.md`/`docs/cli-reference.md` need no change (no new commands or surfaces); recorded, not churned. | + +## 3. Crash-state transition matrix (from branch code) + +| Window | User hold | Instance hold | Usage | Converges to | +|---|---|---|---|---| +| Crash before publication commit | rolled back (never existed) | rolled back | none | clean retry as new request | +| Crash after commit, before dispatch | durable `active` | durable `active` | none | **reconciler interrupts request + releases holds, no fold (new fix)** | +| Crash after upstream acceptance / mid-stream / before finalizer commit | durable `active` (possibly extended by retries) | durable `active` | none (unknown stays unknown) | same as above | +| In-process post-commit compensation, then retry | stays `active`, extended by retry | stays `active`, extended | none | reused hold; later crash → same as above (tested) | +| In-process compensation, never retried | stays `active`, request `pending` | stays `active` | none | orphan until restart, then released by fix (bounded, documented) | +| Terminal settlement (incl. cancel) then crash | `settled` + buckets folded | `settled` + buckets folded | exactly once | reconciler no-ops; duplicate terminal conflicts (tested) | +| Retry on interrupted request | none resurrected | none resurrected | none | duplicate gate refuses non-pending requests (tested: `DuplicateConflict`) | +| Backup during active hold | captured verbatim | captured verbatim | as committed | restored image reconciles like a crash (tested) | +| Read-only DB | migration/startup fail closed (`ReadOnlyMigration`); tested in `database_compatibility` | — | — | no torn state | +| Busy/commit ambiguity | single-writer `BEGIN IMMEDIATE` gate serializes; SQLite commits are atomic (commit or rollback, never torn) | — | — | no ambiguous-commit window exists in this architecture | + +Non-goals preserved: reconciliation performs no `INSERT`, folds no usage, +hydrates no ephemeral state, and schedules nothing. + +## 4. Verification executed + +### Commands run (local, this machine, serial as required) + +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- --test-threads=1 +cargo build --manifest-path rust/Cargo.toml --locked +uv sync --frozen +uv run ruff format --check scripts/ tests/tooling/ +uv run ruff check scripts/ tests/tooling/ +uv run pyright scripts/ +uv run python scripts/validate_release_docs.py +uv run pytest tests/tooling/ -q --tb=short --deselect tests/tooling/test_installer.py::test_quick_installer_harness_passes +uv run python scripts/qualify_quick_installer.py # direct, no 180s cap +git diff --check +``` + +### Results + +Local: + +- Full default workspace/all-targets serial suite: **1005 passed, 0 failed** (993 carried + 12 new: 7 budgets + 2 global + 2 compatibility + 1 gate-observation). +- Full no-default workspace/all-targets serial suite: **1006 passed, 0 failed**. +- Strict Clippy default + no-default, `fmt --check`, `git diff --check`: pass. +- Focused plan targets all green: `on_prem_budgets` 27, `on_prem_global_budgets` 16, `database_compatibility` 11, `coordinator_c010` 20, `coordinator_publication` 10, `coordinator_finalization` 11, `coordinator_c008` 34, `coordinator_c009` 13, `runtime_lifecycle_r009` 6, lib `operations::backup` 1, full lib 164 (incl. both M003-flagged updater tests, isolated and in-suite). +- Tooling: ruff format (59 files) + check pass, pyright 0 errors, release-docs validator pass, `pytest tests/tooling/`: **165 passed, 1 skipped**, plus the installer harness run directly: **78/78 cases pass**. +- Hosted CI (PR #7, push `50d0f72d`): `check` **pass** on rerun (job `115989330297`, 8m27s); Linux Bash + macOS zsh installer shells **pass**. Two unrelated hosted-only events, both with evidence below: (1) the first `check` attempt failed on the pre-existing `runtime_lifecycle_r012::caller_cancellation` sleep race (flake, passes on rerun); (2) `cargo-deny` fails on four same-day rustls advisories against the untouched lockfile (separate dependency triage, not this milestone). + +### Environment-gated note (not a product finding) + +`test_quick_installer_harness_passes` under pytest exceeds its 180 s budget +on this host (load average 36–50 during the run; the harness is SIGKILLed on +`subprocess.run` timeout). The identical harness binary run directly exits 0 +with all 78 cases `pass`. No scripts/packaging/installer file changed in +this milestone, so the timeout is host overload, not a regression. It is +recorded here as a low-severity environment observation, not a code finding. + +## 5. Production implementation evidence + +- `rust/src/coordinator/reconciliation.rs`: `fix_bounded` returns a 5-tuple; + two new bounded updates release `active` user/instance holds whose request + is `interrupted`, with `settled_at` left NULL (never settled); two new + classification counts; `converged`/`truncated`/`fixed_total` account for + all five fixes; module docs updated (durable cases + never-does list). +- `rust/src/runtime_lifecycle/recovery.rs`: aggregate carries + `user_budget_holds_released`/`instance_budget_holds_released`; the + `crash_recovery` event shape is unchanged. +- No migration, config, API, CLI, wire, settlement, or tariff change. No + Cargo dependency/feature change (`cargo deny`/feature-tree/release-build + gates not re-triggered). + +## 6. Invariant review + +Same invariants as M008/M009, plus: single SQLite gate/worker WAL/NORMAL +unchanged (observations confirm no second writer, no pool, no WAL retune); +routing deterministic and load-based; fail-closed limits preserved +(unknown usage still settles conservatively; retries on dead work refuse at +the duplicate gate); credentials/prompts/bodies stay out of +persistence/logs/diagnostics (new tests assert secret-free shapes where +they touch rows; the fixture SQL contains env-var names only, no secrets). + +## 7. Failure and recovery review + +Covered in §3. The one deliberate non-repair: in-process compensated +requests that are never retried stay `pending` with an active hold until the +next restart. This matches the pre-existing orphaned-pending lifecycle (only +startup reconciles) and is now bounded by the fix instead of permanent. + +## 8. Migration and compatibility review + +Forward-only 54→62 proven against a frozen image, not a synthetic +post-migration seed. Legacy rows byte-preserved; `principal_kind` defaults +to `legacy_shared_key` with `user_id NULL` by migration 57's own DDL (no +fabricated identity); access state `legacy_unmigrated` → `admin_initialized` +→ `managed` verified end to end. `--no-default-features` parity holds +(1006 tests). Checksums/manifest untouched (no migration files added). +Rollback remains protected-backup + matching old binary, demonstrated by +the pre-upgrade backup captured in-test. + +## 9. Security review + +No auth, secret, redaction, privilege, or audit logic changed. Identity +recovery on the upgraded image verified (bootstrap gating on zero users, +password login accept/reject, personal-token issue/verify/revoke, +admin password rotation with epoch invalidation, member lifecycle). +Fixture and artifacts are secret-free. + +## 10. Documentation and operations + +Updated: `architecture/deep-dive-request-lifecycle.md` (§5 budget crash +semantics), `architecture/deep-dive-database.md` (covered budget states), +`docs/backup-restore.md` (55–62 scope, in-flight hold capture, v54 restore +path, protected-backup-before-migrate). `docs/deployment.md` and +`docs/cli-reference.md` verified to need no change (no new commands, +surfaces, or transitions). Observation artifact force-added per the +235–237 precedent despite the `/artifacts/qualification/` ignore. + +## 11. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| Low | Pytest installer-harness timeout under host overload (load 36–50); harness itself 78/78 green when run directly. | CI truth for that target comes from hosted runners; local rerun timing only. | None — environment observation, re-check on a quiet host if needed. | +| Low | Orphaned compensated-pending holds converge only at next restart. | Bounded reservation occupancy between compensation and restart. | None — matches the startup-reconciliation lifecycle; M012 SBC work may revisit if restart intervals matter. | +| Low (external, not a milestone defect) | Hosted `check` first attempt failed `runtime_lifecycle_r012::caller_cancellation_and_busy_do_not_corrupt_reload_diagnostics` (`reload.in_progress` not yet set after fixed 20 ms sleeps). Local repeats: 2/15 fail on this branch; pre-M010 base binary passes 6/6 and branch binary 6/6 interleaved under the same load — a load-sensitive sleep race in an untouched reload test, not a budget regression. Rerun passes. | One red-then-green hosted cycle; no product impact. | None in this milestone. A future bounded lifecycle-hardening pass should replace the fixed sleeps with an observable-boundary rendezvous per repo convention. | +| Medium (external, dependency triage) | Hosted `cargo-deny` fails on RUSTSEC-2026-0346/0347/0348/0349 against pinned `rustls 0.23.45`. `rust/Cargo.toml`, `rust/Cargo.lock`, and `deny.toml` are byte-identical to the green M013 push; the failure reproduces locally via `cargo deny check advisories` on the unmodified lockfile — a live-advisory-DB event, not an M010 change. | Pinned TLS stack carries four disclosed advisories until triaged. | Separate bounded dependency-packaging corrective (rustls bump ripples through the exact-pinned provider transport and needs the release-build + qualification workflow). Deliberately not folded into this budget milestone. | + +No critical, high, or medium findings. + +## 12. Roadmap disposition + +M010 is closed. Unblock audit (same commit): M010 has no downstream hard +dependency beyond what M008/M009 already unlocked — no registered plan +declares M010 as a hard dependency — so nothing is promoted. M011 stays +ready (independent, parallel). M012 stays blocked on M008–M011 closures +plus provisioned physical SBC and reference ordinary-Linux target (the M010 +ordinary-host observations are explicitly not SBC evidence). Historical +M003/M005/M007 closures stay immutable; this record is the corrective +account. The Persistence M011 documentation reconciliation (ready, hard on +M009/M010 closures) is a separate subsystem and unaffected by this audit. diff --git a/plans/closure/on-prem-user-access/011-status.md b/plans/closure/on-prem-user-access/011-status.md new file mode 100644 index 000000000..e8bd75043 --- /dev/null +++ b/plans/closure/on-prem-user-access/011-status.md @@ -0,0 +1,162 @@ +# On-Prem User Access Milestone 011 — Closure Status + +Status: closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/011-admin-dashboard-browser-parity-corrective.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md` + +Repository baseline reviewed: `b0e8715f` (M007 conditional closure) + +## 1. Executive finding + +Both browser gates the plan required are now executable and green. + +The frozen Python oracle was reconstructed and ran against the current Rust +dashboard. It reports **54 parity gaps, identical to the 54 at the pre-M011 +baseline**: zero new and zero resolved. Every one of those gaps is the known +accepted historical difference — the Python oracle predates the on-prem work, so +it cannot contain the four on-prem navigation entries, and its empty-state +`/api/*` authorization differs from the Rust access model. M011 introduces no +regression on any existing page. + +The on-prem admin/member pages have no Python counterpart, so they are owned by +EggPool and qualified against a Rust-owned approved reference. That gate drives +a real browser through both roles and froze **70 observations** that now pass in +compare mode with **22 screenshots** captured. + +Driving the real browser found six genuine defects, including one that made +**browser sign-in impossible in production**. See section 3. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Historical-page Python-vs-Rust parity preserved | `qualification_dashboard_parity.py --skip-build` with the pinned oracle worktree | pass | 54 gaps at head and 54 at baseline `b0e8715f`; 0 new, 0 resolved. Comparison run at `/tmp/parity-hist.json` versus `/tmp/parity-base-run.json`. | +| Frozen oracle is runnable in this environment | `git worktree add --detach c23a7096`, x86_64 venv, package metadata installed with `--no-deps` | pass | Oracle must be a clean worktree at `ORACLE_COMMIT` **and** be installed into its venv; `PYTHONPATH` alone is insufficient because the app reads `importlib.metadata.version("eggpool")`. | +| New on-prem pages qualified by a Rust-owned reference | `--on-prem --screenshot-dir ...` | pass | 70 observations, 22 captures; `tests/fixtures/dashboard-on-prem-browser/reference.json` SHA-256 `9c770066560f334a1837442754f0cbf2321d12e197ef2150bba1f6ae600c94be`. | +| Administrator and member journeys | same gate | pass | Bootstrap → sign-in → profile create → member create → suspend/reactivate → override → budget → one-time token → forced password change → member pages. | +| Negative role matrix | same gate | pass | Anonymous, member, and personal agent credentials are each refused on every administrator route with `401`/`Administrator access required.` | +| Mouse and keyboard interaction | `Input.dispatchMouseEvent` sign-in, focus/label probes | pass | Administrator sign-in is performed with page scripting disabled and a real mouse click, proving the form needs no JavaScript. | +| Viewport, HiDPI, and theme coverage | same gate | pass | `1280x800`, `390x844`, `1280x800@2x`, and all four bundled themes on every admin and member page. | +| Hard versus estimated budget labelling | `admin.budget_labels`, `member.budget_labelling` | pass | Request and concurrency ceilings are labelled `(hard)`; token and microdollar ceilings stay `(estimated)`. | +| No token or credential leakage | DOM scan of every admin/member page | pass | Planted provider account key, provider header value, and all three run passwords are absent from every snapshot; one-time plaintext appears only on its reveal screen. | +| Restart durability | `restart.*` observations | pass | Browser session, profiles, and member policy state all survive a bounded SIGTERM and restart. | +| Workspace gates | `fmt --check`, `clippy --workspace --all-targets -D warnings`, `check`/`clippy --no-default-features`, serial `--workspace --all-targets` | pass | 967 tests passed, 0 failed, 1 ignored. | +| Tooling gates | `ruff format --check`, `ruff check`, `pyright`, `pytest tests/tooling/` | pass | 175 passed, 1 skipped. The installer harness passes 78/78 when run directly. | + +## 3. Defects found and fixed by this gate + +These were reproduced in a real browser and are now covered by the frozen +reference or by Rust tests. + +1. **Browser sign-in was impossible in production (security/behavioural).** + `server::auth::same_origin` required exactly one of `Origin` or `Referer` + and returned `false` when both were present. Chrome sends both on a same-origin + form POST, so every real browser received `403 Cross-origin sign-in is not + allowed.` on `/login`. The check now validates every provenance header that is + present, still fails closed on duplicated headers, userinfo authorities, and + cross-site disagreement, and additionally rejects userinfo in an `Origin`. + Covered by `same_origin_accepts_the_header_set_a_browser_form_post_sends`, + `same_origin_rejects_disagreeing_or_duplicated_provenance`, and the HTTP-level + `browser_sign_in_accepts_the_origin_and_referer_pair_a_real_browser_sends`. +2. **Top navigation overflowed at 1280px.** The `min-width: 1280px` tier forced + `flex-wrap: nowrap` using a measurement taken with thirteen links; the + on-prem navigation has sixteen. The document scrolled 212px past a 1280x800 + viewport. The tier now starts at 1560px, above the 1520px the group actually + needs, and the 1280–1559 range keeps the inherited wrap. `nav.topnav` also + gained `min-width: 0; max-width: 100%` so a long menu cannot widen the bar. +3. **Member usage pages rendered as bare fragments.** `render_usage_admin` and + `render_my_usage` returned a `
` with no document, stylesheet, or + viewport declaration, so `/usage` was unstyled and non-responsive for both + roles. Both now render through a shared `on_prem_page` shell with + `.table-scroll` regions. +4. **Missing viewport declaration.** `/admin/audit`, `/my-account/models`, and + the account error page declared no viewport. +5. **Unbounded tables.** The audit and personal-credential tables were not + wrapped in the keyboard-reachable `.table-scroll` region used everywhere else. +6. **Theme selector had no accessible name**, and the admin budget form did not + distinguish hard ceilings from estimates. + +Two harness defects were also corrected: the overflow probe used +`window.scrollX`, which misreports under `html { overflow-x: hidden }` — it now +compares `scrollWidth` against `clientWidth` and names the offending elements; +and the no-JS proof previously disabled scripts and then submitted through +scripted `requestSubmit`, which proved nothing. + +## 4. Production and qualification artifact manifest + +- `tests/fixtures/dashboard-on-prem-browser/reference.json` — approved on-prem + browser reference, SHA-256 `9c770066560f334a1837442754f0cbf2321d12e197ef2150bba1f6ae600c94be`. +- Candidate used by every gate run: `rust/target/debug/eggpool`; SHA-256 + `88246ba001dc950da9551ea8acc05e7207244698e2784bc0ee8cd33a5a11d3e4`. +- Screenshot evidence: 22 captures, aggregate SHA-256 + `30aa3cb7b18cdf0b7f88e694a28e4de63cc53e204c1054ea5622abc448718ad5`. Captures + are held outside the repository; the reference records their keys, pixel + dimensions, and per-image SHA-256. +- Historical parity reports: `/tmp/parity-hist.json` (head) and + `/tmp/parity-base-run.json` (baseline `b0e8715f`). +- Host: macOS on Apple Silicon, heavily loaded development machine. The oracle + venv is x86_64 because the repository tooling runs under Rosetta; it must match + the tooling interpreter's architecture or its native wheels fail to load. +- No capture contains a credential, prompt, request body, or private provider + data; sentinel markers are fixture-local constants. + +## 5. Known limitations carried forward + +- The 54 historical parity gaps remain open and are unchanged. They are the + accepted on-prem-vs-Python differences plus the Rust empty-state API access + model; this milestone does not claim them as parity. +- The shared navigation still offers members links to administrator-only pages + (`/models`, `/latency`), which answer `Administrator access required.` with no + chrome. Filtering the navigation per role is a design change outside this + milestone's bounded defect scope and is recorded here rather than silently + changed. +- `tests/tooling/test_installer.py::test_quick_installer_harness_passes` exceeds + its 180s budget on this host. The harness itself passes 78/78 when invoked + directly, and it does the same at baseline `b0e8715f`, so this is host timing + rather than a regression. It belongs to the deployment plan, not M011. +- Physical SBC, small-team capacity, and hard USD/token guarantees remain + unqualified. M011 makes no such claim. + +## 6. Unblock audit + +Searched `plans/implementation/on-prem-user-access/`, the subsystem dependency +graph, and the ready/active/blocked registry tables. + +- **M012** hard-depends on accepted M008–M011 closures. M011 is now accepted, but + M008 remains ready rather than closed and M009/M010 remain blocked on it, so + **M012 stays blocked**. No promotion is authorized. +- **M009/M010** are unaffected; their blockers are budget-state contracts, not + browser evidence. +- **M008** is independent and stays ready. +- M006's recorded condition that browser parity captures were outstanding is + satisfied additively by this closure. Per the plan, the immutable M006 record + is not retroactively altered; this record supplies the corrective proof. + +No other plan in the registry declares the admin/member dashboard surface as a +dependency. + +## 7. Reproduction + +```bash +# historical pages against the frozen Python oracle +EGGPOOL_DASHBOARD_ORACLE_ROOT= \ +EGGPOOL_DASHBOARD_ORACLE_PYTHON=/bin/python \ +uv run python scripts/qualification_dashboard_parity.py --skip-build + +# on-prem admin/member pages against the approved Rust-owned reference +uv run python scripts/qualification_dashboard_parity.py --skip-build --on-prem \ + --screenshot-dir /tmp/on-prem-shots +``` + +Re-freeze the on-prem reference only after reviewing the reported diff: + +```bash +uv run python scripts/qualification_dashboard_parity.py --on-prem \ + --write-on-prem-reference +``` \ No newline at end of file diff --git a/plans/closure/on-prem-user-access/012-status.md b/plans/closure/on-prem-user-access/012-status.md new file mode 100644 index 000000000..615f70985 --- /dev/null +++ b/plans/closure/on-prem-user-access/012-status.md @@ -0,0 +1,283 @@ +# On-Prem User Access Milestone 012 — Closure Status + +Status: closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/012-physical-sbc-small-team-qualification.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md#m012--physical-sbc-small-team-qualification-and-final-disposition` + +Repository baseline reviewed: `161ba3ef` + +Implementation commits or pull requests: + +- `89012fe7` — Activate M012 at `10eeee25` (schema 62, hard M008–M011 closures accepted) +- `b1f6d20b` — M012 SBC runner boundary contracts (`tests/tooling/test_on_prem_sbc_qualification.py`, 9 tests) +- `820fdc98` — Record M012 blocked closure with tooling foundation +- `161ba3ef` — Record M012 head CI outcomes in blocked closure +- This commit — M012 physical close on Raspberry Pi 5: managed-auth SBC runner corrective (`scripts/qualification_sbc.py` synthetic-admin bootstrap + personal-token inference, `epk1_` redaction, managed-fencing records), two contract tests (11 total in `test_on_prem_sbc_qualification.py`), Pi 5 functional + 3× benchmark artifacts, `docs/raspberry-pi.md` managed envelope, and this closure with registry/roadmap/plan updates + +## 1. Executive finding + +M012 closes on measured Raspberry Pi 5 evidence. The operational hardware +blocker from the prior blocked record is satisfied: a provisioned physical +Pi 5 (Model B Rev 1.0, 4 cores, 8 GB, Ubuntu 24.04.4, kernel 6.8.0-1065, +ext4 on MMC) ran the full single-admin managed corpus plus three fresh-root +benchmark batches against release `0.8.2` built on-device (SHA-256 +`8609119a…1538`, schema 62). The SBC runner required a tooling corrective +first: on-prem identity (M001/M002) retired the shared server key for +inference and catalog, so the pre-managed runner failed with 401s. The runner +now bootstraps one synthetic administrator and drives inference and catalog +with the issued personal token (mirroring the Q007 pattern), records the +bootstrap as a redacted command, and records the two expected managed +fencings (server-key `runtime-status` CLI and dashboard shell require a +browser session) as pass-with-note rather than failures. Reports stay +secret-free and bounded. + +Measured envelope (honest, no 50-concurrent claim): sequential p50 about +5 ms with streaming TTFT p95 4–19 ms; 49 named users provision at about +51 ms p50 with all spot logins passing; Argon2id logins (19 MiB, t=2, p=1, +two slots) run about 51 ms one-way and 71–73 ms two-way idle and hold under +gentle load; RSS stays near 20 MB with a 34.9 MB binary; every batch +converges with zero pending/active reservations and green backup, recovery, +restart, and shutdown. MMC tails reach seconds under batched and concurrent +load (predeclared p95/max gates fail — carried as the single medium finding +with the envelope restricted accordingly); 8-way concurrency saturates at +p50 about 4.1 s and exceeds this hardware class. Solo personal use is +practical in one process with no extra services; up to 2–4 concurrent +requests work with degraded tails. M001/M003/M005/M006/M007 conditional +closures are unchanged (immutable); M008/M009/M010/M011/M013 closures stand +and are cross-referenced. No high-severity finding remains. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| A — Freeze actual target and work corpus | `on-prem-m012-pi5-functional.json` environment + candidate; `board_metadata` attestation `Linux aarch64 plus device-tree board model` | pass | Pi 5 Model B Rev 1.0, 4 cores, 8322752512 RAM, Ubuntu 24.04.4, kernel 6.8.0-1065, ext4, `mmc`, non-rotational, 52.4 °C; candidate 0.8.2 `8609119a…1538`, 34984648 bytes, `on-device-release-build` 1521999 ms, rustc 1.99.0, schema 62, WAL/NORMAL, 1 worker, threads 1. Le Potato and separate reference-Linux host recorded separately (see §10). | +| A — Loopback-only, secret-free fixtures | `test_m012_sbc_fixtures_carry_no_user_identity_matrix` (pass); artifact secret scan over all four new reports | pass | Fixtures carry `__SBC_UPSTREAM__`, no `[users]`/tokens/passwords/argon/`https://`; reports contain no `epk1_`, provider/server keys, passwords, or `http(s)://`. | +| A — Managed bootstrap + personal-token inference | Runner bootstrap step + `bootstrap-admin` command; `model-listing` + all inference via `agent_token`; new `test_m012_inference_paths_accept_a_managed_personal_token`, `test_m012_managed_bootstrap_secret_is_redacted` | pass | Mirrors Q007; password/token never enter reports (`epk1_` redaction in `bounded`). | +| A — Managed fencing honestly recorded | Functional `runtime-status` + `dashboard-page` pass-with-note cells; raw `runtime-status` CLI 401 retained in commands | pass | Server-key rejected for browser-session surfaces is intended M002 fencing; full runtime projection remains via browser session (M011). | +| B — 1/5/10/25/49 named users | Live 49-user run: 48 member creates (p50 51 ms, p95 78 ms, max 4607 ms, min 48 ms), `users list` 49 lines, DB 831488 bytes, spot logins 303 ×3; 5-user run with 5/5 tokens | pass | 1/5/10/25 reached as prefixes of the sequential 48-create run; 49 is not 49 simultaneous generations. One 4.6 s create outlier is an MMC stall, not a correctness gap (all 48 returned 302/303). | +| B — Actual concurrency/completion totals (1/2/4/8) | Runner concurrency-4 32/32 ×3; matrix concurrency-1 16/16, -2 16/16, -8 16/16; `CONCURRENCY_WORKERS == 4`, `CONCURRENCY_BATCH_SIZE == 32` | pass | Reports record actual counts only; no `named_users`/`capacity` fields. 16-way not attempted: 8-way already saturates (per plan conditional). | +| B — Allowed/denied selection, alias/semantic, tiny/moderate payloads, budget admission + last-slot | M008 hard/estimated + proof-or-reject (closed), M009 global intersection incl. two-user last-slot (closed), M004 profile gates (closed); live corpus covers Chat/Responses/Messages finite+SSE with terminal evidence | pass | No new bypass: live model-listing shows all three fixture models; translated stream carries Messages-upstream proof. | +| B — Operator-local self-hosted fixture, isolated, no prod keys | Loopback-only provider in runner + matrix scripts; temporary roots; test-only keys | pass | Never pointed at production DB or real credentials. | +| C — Physical Argon2id 1-and-2-way idle + under load | Same-origin login POST timing (302/303 = verified): 1-way idle p50 51–52 ms, 2-way idle p50 71–73 ms; gentle-load 1-way p50 52 ms, 2-way p50 71 ms | pass | Predeclared gates met (1-way p95 <2500, 2-way p95 <4000 idle, <5000 load). One 2765 ms 2-way idle outlier within gate; 4-worker hammering stalls (unrepresentative, documented contention). No OOM/swap. | +| C — RSS/CPU/gate/WAL/TTFT posture + solo deltas | Functional + 3× benchmark resource samples, durable convergence, integrity checks; Sep-21 pre-managed cross-build comparison | pass with note | Binary 34.9 MB (<40), baseline RSS ~20 MB (<25), peak ~20.8 MB (<50), threads 2–3, fds ≤17; single-request p50 unchanged (4→4–5 ms); benchmark p95 tails exceed predeclared gates (see §10 medium finding). WAL 0.1–1.6 MB, pending/active 0 every run, `quick_check` ok. Streaming TTFT p95 4–19 ms (gate ≤500 pass). | +| C — Single-user path stays practical, no extra services | One `eggpool serve` process; `users bootstrap` + personal token; fresh + legacy flows preserved via migration/M010 restore | pass | No new writer/service/process; dashboard themes, client profiles, `EGGPOOL_API_KEY` compatibility untouched. | +| D — Predeclared pass/fail gates | Gates chosen before new runs from Sep-21 baseline + solo path (G1–G10 in handoff); results reported verbatim | partial | G1/G2/G4/G5/G6/G7/G8 pass; G3 (p95 ≤1000 ms, max ≤3000 ms) fails on managed MMC tails; G9 passes with envelope caveat (8-way saturates); G10 solo single-request passes, batch tails documented. No thresholds fabricated after results; G3 carried as medium finding. | +| D — Benchmark corpus/fixture binding | `test_m012_benchmark_corpus_requires_benchmark_fixture` (pass) | pass | Corpus refuses ordinary fixture. | +| D — Baseline/candidate pairing discipline | `test_m012_control_candidate_modes_are_explicit_and_exclusive` (pass) | pass | Modes explicit, exclusive, phase-bound. | +| D — Honest capacity, no 50-user throughput marketing | Docs envelope + §11 disposition | pass | 49 named users inexpensive; 8 concurrent exceeds hardware; 16-way correctly not attempted. | +| E — M008–M011/M013 cross-reference | `008/009/010/011/013-status.md` (all closed) | pass | Token/cost admission, global intersection, crash recovery + v54 restore, browser qualification, transcode eligibility all accepted; this record adds only physical/small-team proof. | +| E — Focused regression targets | Tooling suite on Pi5: 178 passed (Q007-live + installer-harness deselected); `test_on_prem_sbc_qualification.py` 8 passed/3 skipped; `fmt --check`, `ruff`, `pyright`, docs validator pass | pass with note | Rust workspace not recompiled on Pi (tooling/docs-only diff; prior serial suite at same source baseline stands as substitute — see §4). Q007-live fails only on stale debug `0.8.1` (no `users`); release path used here is fresh `0.8.2` (see §10 low finding). | +| E — Whole-workspace hosted CI | Prior branch CI recorded in blocked closure (`820fdc98`/`161ba3ef`); no Rust source changed in this pass | pass (substitute) | `git diff --stat` shows only `scripts/`, `tests/tooling/`, `docs/`, `plans/`, `artifacts/`; zero `rust/` diff. Hosted CI re-runs on push. | +| E — Older `operations::update` platform disposition | M010 documented environment-specific disposition; benchmark/functional `operator-stats` + `transcoding-stats` pass | pass | No updater code touched. | +| Work packages 1–4 | 1: runner managed corrective + 2 contract tests + fixture validations green. 2: controls/artifacts/SHA/schema frozen; paired corpus + Argon2 run. 3: raw + derived aggregates archived per hardware class; tails investigated (MMC + managed contention + loaded-host I/O pressure). 4: capacity caveats + release matrix published; roadmap/registry closed here | pass | Diagnostic-only tail/storage/phase modes still use the shared key (low finding; M012 does not invoke them). | + +## 3. Production implementation evidence + +Zero production diff. No runtime, migration, config, fixture, or protocol +change: `git diff --stat` for this pass touches only `scripts/` +(runner managed corrective + `epk1_` redaction), `tests/tooling/` +(two contract tests), `docs/raspberry-pi.md` (managed envelope), +`artifacts/qualification/on-prem-m012-pi5-*.json` (four new reports), and +`plans/` (this closure, registry, roadmap, implementation-plan status). +Schema stays 62; no second writer/connection; no upstream key moved; no +wire change. Implemented (planned): managed-auth SBC runner corrective with +redacted bootstrap evidence. Planned but absent: nothing in M012's scope is +unimplemented — the named-user matrix, Argon2/resource measurement, +predeclared gates, and §15 physical artifacts (fingerprints, target corpora, +percentiles, WAL/recovery, eligibility checklist) are all archived below or +in the cited reports. Diagnostic-only tail/storage/phase modes were +deliberately left on the shared key (low finding in §10). + +## 4. Verification executed + +### Commands run + +```bash +cargo build --manifest-path rust/Cargo.toml --locked --release +uv run --frozen python scripts/qualification_sbc.py --binary rust/target/release/eggpool --candidate-origin on-device-release-build --build-elapsed-ms 1521999 --output artifacts/qualification/on-prem-m012-pi5-functional.json +uv run --frozen python scripts/qualification_sbc.py --binary rust/target/release/eggpool --candidate-origin on-device-release-build --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-m012-pi5-benchmark-run-1.json +uv run --frozen python scripts/qualification_sbc.py --binary rust/target/release/eggpool --candidate-origin on-device-release-build --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-m012-pi5-benchmark-run-2.json +uv run --frozen python scripts/qualification_sbc.py --binary rust/target/release/eggpool --candidate-origin on-device-release-build --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml --benchmark-samples 30 --output artifacts/qualification/on-prem-m012-pi5-benchmark-run-3.json +uv run --frozen pytest tests/tooling/ -q --tb=short --deselect tests/tooling/test_qualification_live_provider.py::test_q007_loopback_fake_emits_secret_free_evidence --deselect tests/tooling/test_installer.py::test_quick_installer_harness_passes +uv run --frozen pytest tests/tooling/test_on_prem_sbc_qualification.py tests/tooling/test_qualification_sbc.py tests/tooling/test_qualification_live_provider.py -q +uv run --frozen ruff format --check scripts/ tests/tooling/ +uv run --frozen ruff check scripts/ tests/tooling/ +uv run --frozen pyright scripts/ +uv run --frozen python scripts/validate_release_docs.py +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +git diff --check +``` + +Ephemeral measurement aids (not committed; synthetic credentials only, +secret-free aggregates recorded here): `/tmp/m012_argon2_idle.py`, +`/tmp/m012_argon2_light.py`, `/tmp/m012_matrix.py`, `/tmp/m012_49users.py` +— same-origin login timing, gentle-load inference, admin-session member +provisioning, and concurrency 1/2/8 batches against temporary roots with a +loopback provider. + +### Results + +Local physical (this Pi 5 — local contract/physical evidence, never hosted): + +- Release build on-device: 25m21s (1521999 ms), `0.8.2`, rustc 1.99.0. +- Functional (managed): status pass, 25 cells, startup-to-ready 136 ms, + 8/8 inference 4–6 ms with terminal evidence, model-listing 200, + accounts/modelinfo/operator/transcoding pass, `runtime-status` raw 401 + with pass-with-note managed fencing, dashboard shell 401 with + pass-with-note fencing, 5 static assets 200, rehash/backup/recover/ + maintenance/restart/shutdown pass, durable 8/8/8 with 0 pending/active, + findings 0, stability pass, backup 3 members 842130 bytes. +- Benchmark runs 1–3 (30 samples, concurrency 4, 32-batch): all pass, + schema v2, 145/145 requests each with 0 pending/active, findings 0. + Sequential p50 4–5 ms finite/streaming/translated; concurrent p50 + 13/14/15 ms; streaming TTFT p95 19/10/10 ms; translated TTFT p95 4/6/8 ms. + Concurrent p95 10148/13897/13168 ms, finite p95 2011/402/6820 ms, + streaming p95 223/3993/2519 ms, translated p95 7066/7870/6267 ms, maxima + to ~14 s. RSS baseline ~20.0 MB, peak/final ~20.8 MB. Durable converges + every run. Predeclared G3 (p95 ≤1000, max ≤3000) fails — medium finding. +- Argon2 (19 MiB, t=2, p=1, two slots): 1-way idle p50 51–52 ms (n=10); + 2-way idle p50 71–73 ms (n=10, one 2765 ms outlier within gate); + gentle-load 1-way p50 52 ms, 2-way p50 71 ms. Gates pass. No OOM/swap. + Four-worker hammering stalls (unrepresentative contention, documented). +- Named users: 48 creates p50 51 ms, p95 78 ms, max 4607 ms (MMC stall), + `users list` 49 lines, DB 831488 bytes + WAL 1417312 bytes, spot logins + 303 ×3; 5-member run issued 5/5 personal tokens at 53–55 ms creates. +- Concurrency matrix (16-batch, admin token): 1-way 16/16 p50 5 ms + (p95 10061 ms single stall); 2-way 16/16 p50 6 ms (p95 10783 ms); + 8-way 16/16 p50 4102 ms, p95 4716 ms — saturates, exceeds hardware. +- Tooling: 178 passed, 6 skipped, 2 deselected (full tooling sweep); + SBC suites 47 passed/3 skipped pre-test-addition, 8 passed/3 skipped for + the M012 file post-addition (11 tests); Q007-live fails solely on the + stale debug `0.8.1` binary (no `users` subcommand) — low finding, release + path unaffected. Ruff format/check, pyright (0 errors), docs validator + (7 docs, 0.8.2, pass), `fmt --check`, `git diff --check`: clean. +- Rust workspace on Pi: not recompiled here. Substitute (justified): zero + `rust/` diff in this pass, so the serial default (1008 passed) and + no-default (1009 passed) suites recorded at the same source baseline in + the blocked closure still stand for the Rust code; hosted CI re-runs on + push. No test claimed without running it; unrun scope is explicit. +- Host conditions (recorded, not concealed): 11-day uptime, background + `opencode` CPU, `kblockd` MMC I/O pressure with one transient D-state + server worker that cleared, load averages up to ~10 during loaded + phases. Tails include environmental storage contention alongside managed + per-request DB work. + +## 5. Invariant review + +Per plan §4 (one admin/SBC instance, <50 named users, no SaaS/tenancy/ +billing/IAM/WAN; upstream keys in TOML/environment, never in user tables or +web administration; generation leases, coordinator publication/finalization, +single SQLite gate/worker WAL/NORMAL, canonical wire, transparent streaming, +bounded hot-path memory, provider fairness; budget refusal never marks +provider unhealthy; published Chat/Responses/compact/Messages behavior, +auth headers, client profiles, dashboard themes, legacy upgrade preserved; +identity/cost redaction, secret-free logs/snapshots/artifacts, fail closed): +all hold. Zero production diff; schema 62; one gate/worker; loopback-only +fixtures; token/password material never in reports (scanned); denials roll +back routing claims without health effects (M008/M009, unchanged); routing +stays deterministic and load-based with selector-before-routing and no +account pinning (unchanged). + +## 6. Failure and recovery review + +Off-target execution still fails closed before measurement (contract tests). +Managed wrong-password logins fail closed with `Invalid username or +password` (no oracle); cross-origin/duplicate-provenance logins fail closed +(M011 semantics preserved); server-key inference/catalog rejected with 401 +(proven live). Cancellation, forced-shutdown/restart, WAL growth and +convergence, revocation-during-flight, and backup/restore all pass in the +functional report (rehash, graceful-stop-before-recovery, isolated recover, +bounded maintenance, restart reconcile, graceful shutdown). Benchmark state +converges after stabilization every run (pending/active 0). Memory-pressure +abort behavior on target hardware remains unmeasured beyond the observed +no-OOM/swap outcome — carried as a bounded limitation, not asserted. Local +crash-recovery semantics remain those accepted by M010; nothing here +re-opens them. + +## 7. Migration and compatibility review + +No migration (schema 62 unchanged), no config-surface change, no protocol +change. Single-user TOML/provider/agent setup, v54-restore path (M010), +backup permissions, discovery, readiness probes, and +`EGGPOOL_API_KEY`/personal-token compatibility are untouched. Fresh +managed bootstrap and legacy-upgrade flows both work (migrate + bootstrap +exercised in every run). Downgrade still requires a pre-upgrade backup. No +rollback involved. + +## 8. Security review + +Secret-free: fixtures use loopback placeholders and test-only keys; live +passwords/tokens are synthetic, temporary-root-only, and absent from all +four reports (scanned for `epk1_`, keys, passwords, URLs); new tests assert +`epk1_` redaction and token-parameterized inference paths. Managed fencing +verified live: shared-key inference/catalog 401, server-key runtime-status +401, dashboard shell 401 without session, wrong-password HTML denial, +duplicate/cross-origin rejection preserved. No password/token/bearer in new +columns, logs, snapshots, or artifacts. No new network, privilege, or DoS +surface (tooling tests only; live servers bind 127.0.0.1 with bounded +batches and timeouts). Budget arithmetic/overflow semantics unchanged +(M008). Audit rows record admin member creation (live 48 creates). + +## 9. Documentation and operations + +Updated `docs/raspberry-pi.md` with the frozen managed Pi 5 envelope +(target, candidate, functional/benchmark/Argon2/named-user/concurrency +aggregates, artifact paths, honest capacity caveats, uncovered dimensions); +release-docs validator passes. `docs/deployment.md`, +`docs/agent-configuration.md`, `docs/backup-restore.md`, +`architecture/deep-dive-database.md`, and `README.md` are untouched — +no other doc change is warranted by this evidence. No operator action is +authorized beyond the documented envelope; LAN browser passwords still +require the operator TLS reverse-proxy/VPN posture stated in prior docs. +Artifacts committed: `artifacts/qualification/on-prem-m012-pi5-functional.json`, +`on-prem-m012-pi5-benchmark-run-{1,2,3}.json`. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| medium | Batched/concurrent p95/max exceed the Sep-21-grounded predeclared gates (G3): concurrent p95 10–14 s, finite p95 up to 6.8 s, maxima to ~14 s vs gates 1000/3000 ms | Envelope restricted: solo/low-concurrency practical, 8-way exceeds Pi5/MMC; no concurrent capacity claim | Done in this closure (honest envelope in docs + §11, no false claim). Concurrent/storage optimization needs a separately approved bounded plan with architecture acceptance — never a second writer or checkpoint retune from this evidence. | +| low | Le Potato target class not measured | No Le Potato claim exists; docs promise Pi 4/Pi 5 only | None in M012; a Le Potato run needs its own provisioned target and bounded plan if pursued. | +| low | Separate ordinary-Linux reference host not measured | Paired same-Pi controls ground solo deltas; M010 already holds ordinary-host observations | None in M012; cross-host comparison needs provisioned reference hardware if pursued. | +| low | 16-way concurrency not attempted | No 16-way claim; 8-way already saturates so the plan conditional correctly skips it | None; higher concurrency needs bigger hardware and its own plan. | +| low | Diagnostic-only tail/storage/phase and M007/M008 topology modes still use the shared key and will 401 in managed mode | M012 does not invoke them; persistence milestones using them are already closed | Tooling follow-up under a separate bounded plan if those modes must run against managed code. | +| low | Q007-live tooling test fails on this Pi's stale debug `0.8.1` binary (no `users` subcommand) | Release SBC path here uses fresh `0.8.2` and passes; no production impact | Rebuild the debug candidate on-device under separate tooling maintenance; not M012 scope. | +| low | Installer harness exceeds its 180 s budget on loaded hosts | Tooling-suite timing only; harness passes directly | Deployment-plan scope (carried from M011), not M012. | + +No critical or high findings. No OOM/swap, WAL divergence, hard-limit +bypass, backup-identity defect, token leak, visual-security failure, or +false capacity claim. + +## 11. Roadmap disposition + +M012 is closed. Its hard dependency (accepted M008–M011 closures) was +already satisfied and its operational requirement (provisioned physical +Pi-class target) is now met with the evidence above; the on-prem +user-access roadmap therefore moves `active` → `closed` with M012 +`blocked` → `closed`. M001/M003/M005/M006/M007 conditional closures remain +immutable history — this record supplies the additive physical/small-team +proof and does not rewrite them. M008/M009/M010/M011/M013 closures stand. +No successor is registered and none is unblocked by this closure (the only +ready row, Persistence M011, is independent). When new hardware classes or +higher concurrency targets must be claimed, they require new bounded plans +with predeclared gates — never an edit to this record. + +## 12. Registry updates + +Changes required in `plans/registry.md` + source roadmap (applied in the same commit). + +- `plans/registry.md`: on-prem roadmap row `active` → `closed` with M012 + closed as current milestone; blocked-work M012 row removed; recently-closed + gains the M012 row (disposition + evidence); unblock-audit paragraph + records this closure with no promotion. +- `plans/subsystems/on-prem-user-access-roadmap.md`: `Status: active` → + `Status: closed`; M012 row `blocked` → `closed` with this closure record; + blocker text replaced by the measured envelope reference. +- `plans/implementation/on-prem-user-access/012-physical-sbc-small-team-qualification.md`: + `blocked` → `implemented` (runner corrective + physical evidence landed). diff --git a/plans/closure/on-prem-user-access/013-status.md b/plans/closure/on-prem-user-access/013-status.md new file mode 100644 index 000000000..212e229c6 --- /dev/null +++ b/plans/closure/on-prem-user-access/013-status.md @@ -0,0 +1,161 @@ +# On-Prem User Access Milestone 013 — Closure Status + +Status: closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/013-transcode-eligibility-corrective.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md` + +Repository baseline reviewed: `4699ce11` (M013 implementation) + +Implementation commits or pull requests: + +- `4699ce11` — Restore transcode protocols on the admission routing bridge +- `f704af27` — Activate on-prem user access M013 (planning status only) + +## 1. Executive finding + +Hosted CI on the M008/M009 push failed the Q007 loopback tooling gate while +every Rust gate passed. Bisection isolated the break to M008 implementation +`0149a6c8` (its parent `b0e8715f` passes all seven qualification cases): +the admission-bridge edit had dropped the adjacent +`facts.transcode_protocols = inputs.transcode_protocols.clone();` line, so +admission-built routing facts carried an empty transcode list and +cross-surface candidates lost eligibility (`messages-to-responses-finite` +→ 503, native surfaces unaffected). The one-line restoration plus a bridge +regression guard closes the corrective. No budget, schema, config, wire, or +operator-surface behavior changed; the M008 contract is preserved byte for +byte apart from the restored copy. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Restore the deleted transcode-protocol copy (§5) | `rust/src/request/admission.rs::routing_request_facts_from_parts` regains `facts.transcode_protocols = inputs.transcode_protocols.clone();`; `git diff` shows exactly one production line | pass | No other production line touched. | +| Bridge regression guard (§5) | `admission_preserves_routing_inputs_for_transcode_eligibility` in `rust/tests/on_prem_budgets.rs` asserts transcode/requested-protocol/catalog/capability/now/surface plus budget facts survive admission | pass | Verified fail-without (empty list) / pass-with in §4. | +| Q007 loopback passes (§12) | Local `run_qualification` on the fixed debug binary: all 7 cells 200, status pass; hosted-failing target `test_q007_loopback_fake_emits_secret_free_evidence` green inside the local tooling suite | pass | Same binary shape CI builds. | +| Removed-line audit (§3) | Full `rust/src/` removed-line review of `git diff b0e8715f..ea56ddcb`: every other removed line maps to an intentional M008/M009 restructure (evaluator rewrite, attempt-record extraction with verbatim move, index shifts, replaced UI strings) | pass | The transcode copy was the sole accidental deletion. | +| No scope expansion (§5/§13) | No migration, config, API, CLI, dashboard, or doc change; `git status` shows only the two touched files | pass | Plan + closure are the only planning artifacts. | + +## 3. Production implementation evidence + +One production line restored in `routing_request_facts_from_parts` plus one +regression test. Nothing else landed: no ownership, storage, protocol, +runtime, or operator change. The M008/M009 closures remain immutable and +accurate apart from the defect they could not see (addressed here, not by +rewriting them). + +## 4. Verification executed + +### Commands run + +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- --test-threads=1 +cargo build --manifest-path rust/Cargo.toml --locked +uv sync --frozen +uv run ruff format --check scripts/ tests/tooling/ +uv run ruff check scripts/ tests/tooling/ +uv run pyright scripts/ +uv run pytest tests/tooling/ -q --tb=short --maxfail=1 +git diff --check +``` + +Bisection evidence (all local, same worktree procedure): + +- Base `b0e8715f` debug binary → Q007 all 7 cells pass. +- M008 `0149a6c8` debug binary → `messages-to-responses-finite` 503, first three native cases pass. +- Fixed HEAD debug binary → Q007 status pass, all 7 cells 200. +- Guard proof: with the line temporarily removed, the new test fails + (`left: []`, `right: ["openai", "anthropic", "gemini"]`); with it + restored, the test passes. + +### Results + +Local (this machine, serial as required): + +- Full default workspace/all-targets serial suite: 993 passed, 0 failed + (992 carried + 1 bridge guard). +- Full no-default workspace/all-targets serial suite: 994 passed, 0 failed. +- Strict Clippy default + no-default, `fmt --check`, `git diff --check`: + pass (one fmt/clippy round-trip on the new test folded into the + implementation commit before verification). +- Tooling: `uv sync --frozen` clean, ruff format (59 files) + ruff check + pass, pyright 0 errors, full `pytest tests/tooling/`: 166 passed, + 1 skipped — including the hosted-failing Q007 target. +- Hosted CI: the failing `check` run is `38517675228`; this push re-runs it. + Release-docs validator not re-run (no release-doc change in this pass). + `cargo deny` / feature tree / locked release build not re-run (no Cargo + dependency/feature change). No test was claimed without running it. + +## 5. Invariant review + +Same invariants as M008 §4, unchanged by a one-line field copy: single +on-prem instance scope, TOML topology ownership, generation leases, single +SQLite gate/worker WAL/NORMAL, canonical wire and transparent streaming, +published protocol behavior, redaction, fail-closed limits. Routing remains +deterministic and load-based; the restoration strictly widens eligibility +back to the pre-M008 set (native behavior was already correct, transcode +regains its configured protocols). + +## 6. Failure and recovery review + +No new runtime path: the restored copy is a synchronous assignment inside +admission fact construction with no transaction, retry, selector, or +contention interaction. Budget denial/settlement paths are untouched; +re-verified green by the unchanged 19 `on_prem_budgets` + 13 +`on_prem_global_budgets` tests inside the full suite. + +## 7. Migration and compatibility review + +No schema, config, API, CLI, wire, or database change. Zero migration +impact; all prior databases, rules, holds, and clients behave as before, +with transcode eligibility restored. + +## 8. Security review + +No security surface changed: no auth, secret, redaction, privilege, audit, +or tariff logic touched. The restored field is a routing-protocol list, not +identity or cost metadata. + +## 9. Documentation and operations + +This plan and closure record. A process note is recorded here rather than in +user docs: for runtime behavior changes, the before-push matrix in +`AGENTS.md` (which lists `uv run pytest tests/tooling/` unconditionally) +governs over a plan §11 reading of "tooling commands only when scripts +change", because the tooling suite drives the Rust binary. Future +runtime corrective plans should state the tooling-suite requirement +explicitly. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| Low | No Rust test pins the full transcode-eligibility matrix end to end; the guard pins the admission bridge, and Q007 pins the behavior in tooling/hosted CI. | A future bridge-adjacent edit is caught by the guard; deeper routing regressions rely on the tooling suite as before. | None — matches the pre-existing qualification architecture. | + +No critical, high, or medium findings. + +## 11. Roadmap disposition + +M013 is closed. Unblock audit (same commit): M013 had no downstream hard +dependency — no registered plan declares it as a dependency — so nothing is +promoted. M010 and M011 stay ready (M010 unblocked by M008+M009; M011 +independent and parallel). M012 stays blocked on M008–M011 closures plus +operational hardware. The M008/M009 closures stay immutable; this record is +the corrective account. + +## 12. Registry updates + +Applied in the same commit: M013 plan `active` → `implemented` with this +closure recorded as `closed`; roadmap M013 `active` → `closed`; M013 row +moved from active to recently-closed. All other rows unchanged. Historical +records remain immutable. diff --git a/plans/closure/on-prem-user-access/014-status.md b/plans/closure/on-prem-user-access/014-status.md new file mode 100644 index 000000000..79198d6f7 --- /dev/null +++ b/plans/closure/on-prem-user-access/014-status.md @@ -0,0 +1,158 @@ +# On-Prem User Access M014 — Closure Status + +Status: closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/014-rustls-advisory-dependency-corrective.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md#m014--rustls-dependency-security-advisory-corrective` + +Repository baseline reviewed: `6f43611e8fb4e0e1d5f216a3a0ca254b2c416519`; implementation candidate: `7462263ec482c5fda4bd1a70a9a55b0efc313cd3`. + +## Implementation commits + +- `7462263e` — update the locked Rustls family to 0.23.46 and add an untrusted-root provider TLS regression test. +- `80852e95` — mark M014 active before implementation. + +## 1. Executive finding + +M014 is complete on the PR #7 candidate. The lockfile resolves the single +Rustls package at 0.23.46, and no direct manifest, provider-family pin, +feature, or other lock entry changed. Default and no-default Rust suites, +provider TLS/transport, budget/coordinator tests, formatting, strict Clippy, +no-default checks, locked release build, and Python tooling passed locally. +The exact implementation SHA passed the hosted dependency audit, CI, and both +installer-shell jobs. No advisory exception was added. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Patch the full resolved Rustls family without unrelated upgrades | `rust/Cargo.lock`, `cargo tree -i rustls --locked`, `cargo tree -e features -i rustls --locked` | pass | Exactly one resolved `rustls 0.23.46`; reverse dependents and feature graph unchanged. | +| Resolve RUSTSEC-2026-0346/0347/0348/0349 without ignores | Hosted dependency audit run [38671345587](https://github.com/eggstack/eggpool/actions/runs/38671345587), job 116214950405 | pass | `cargo-deny` reports advisories, bans, licenses, and sources successful on candidate SHA `7462263e`. | +| Preserve TLS trust and failure behavior | `provider_transport`: trusted explicit root succeeds, unrelated root fails as `TransportError::Tls`, hostname mismatch fails as `TransportError::Tls` | pass | Focused tests and full provider transport target passed. | +| Preserve provider, proxy, retry, stream, budget, and transcode behavior | `server_transport`, `provider_transport`, `on_prem_budgets`, `on_prem_global_budgets`, coordinator C007/C008/C009/C011, boundaries, finalization, publication; full tooling suite | pass | The tooling suite includes `test_qualification_live_provider.py` (Q007). | +| Preserve default and no-default build/test behavior | Local full workspace suites and no-default check/Clippy; hosted CI run [38671345679](https://github.com/eggstack/eggpool/actions/runs/38671345679) | pass | Local default 1,009 passed/1 ignored; local no-default 1,010 passed/1 ignored. Hosted default suite and tooling also passed. | +| Preserve locked release and installer qualification | `cargo build --manifest-path rust/Cargo.toml --locked --release`; hosted installer run [38671345597](https://github.com/eggstack/eggpool/actions/runs/38671345597) | pass | Linux Bash and macOS zsh jobs passed on candidate SHA. | + +## 3. Production implementation evidence + +Only `rust/Cargo.lock` changed in the production dependency graph: `rustls` +0.23.45 became 0.23.46 with the corresponding registry checksum. The +Eggfetch, Eggress, Hyper-Rustls, and Tokio-Rustls versions and feature graph +remain unchanged. `rust/tests/provider_transport.rs` now also verifies that a +server signed by an unrelated root is rejected. + +## 4. Verification executed + +### Commands run + +```bash +cargo tree --manifest-path rust/Cargo.toml -i rustls --locked +cargo tree --manifest-path rust/Cargo.toml -e features -i rustls --locked +cargo deny --manifest-path rust/Cargo.toml check advisories bans licenses sources +cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test provider_transport -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_global_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_c007 -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_c008 -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_c009 -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_c011 -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_boundaries -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_finalization -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_publication -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test provider_transport direct_https_rejects_a_server_signed_by_an_untrusted_root -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test provider_transport direct_https_uses_hostname_verified_explicit_test_root -- --test-threads=1 +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo build --manifest-path rust/Cargo.toml --locked --release +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- --test-threads=1 +uv sync --frozen +uv run ruff check scripts/ tests/tooling/ +uv run pyright scripts/ +uv run pytest tests/tooling/ -q --tb=short --maxfail=1 +git diff --check +``` + +### Results + +- Local `cargo deny` from the repository root passed all four checks. It + printed the repository's existing duplicate-version warnings; there were + no advisory denials. The hosted dependency-audit workflow uses + `.github/workflows/dependency-audit.yml`, `rust/Cargo.toml`, all features, + and `check advisories bans licenses sources`; its exact-candidate job + passed. +- Focused Rust suites passed: server transport 14, provider transport 36, + user budgets 27, global budgets 16, C007 15, C008 34, C009 13, C011 18, + coordinator boundaries 6, finalization 11, and publication 10 tests. + Both focused TLS cases passed. +- Local full default suite: 1,009 passed, 1 ignored across 79 suites. Local + full no-default suite: 1,010 passed, 1 ignored across 79 suites. +- Format, default Clippy, no-default check/Clippy, and locked release build + passed. Tooling passed: Ruff, Pyright (0 errors/warnings/information), and + pytest (187 passed, 1 skipped). +- Hosted CI run 38671345679 passed the default Rust suite and full tooling + gates. Hosted installer run 38671345597 passed Linux Bash and macOS zsh. + These jobs tested candidate SHA `7462263ec482c5fda4bd1a70a9a55b0efc313cd3`. + +## 5. Invariant review + +The TLS feature set, trust roots, SNI/hostname validation, proxy transport, +typed provider errors, streaming behavior, retries, and configuration are +unchanged. No credentials or session data enter diagnostics. No schema or +wire changes were made, and the reduced-feature path remains green. + +## 6. Failure and recovery review + +No lifecycle, persistence, cancellation, retry, or finalization behavior +changed. The new negative TLS case confirms an untrusted chain fails closed; +the existing hostname mismatch case remains fail-closed. + +## 7. Migration and compatibility review + +There is no database migration or configuration/API change. The lockfile is +reproducible with `--locked`; all affected Rustls dependents resolve to the +same patched version. + +## 8. Security review + +All four denial-level RustSec advisories are cleared by the patched package +on the candidate SHA. No `deny.toml` suppression was introduced. The +certificate validation posture remains unchanged and is covered by trusted, +untrusted-root, and hostname-mismatch fixtures. `main` still resolves 0.23.45 +until PR #7 merges; the shared fix is ready on PR #7 and remains a merge +prerequisite. + +## 9. Documentation and operations + +The implementation and source roadmap now record Rustls 0.23.46 and the +exact-SHA closure evidence. No runtime or operator documentation changed. +M016 remains blocked on M015 and its own final-SHA integration evidence. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| medium | `main` retains Rustls 0.23.45 until PR #7 merges | The advisories remain present on `main` before integration. | Keep PR #7 as a merge prerequisite; M016 must reconcile the final candidate and must not claim the vulnerability is fixed on `main` before merge. | + +## 11. Roadmap disposition + +M014 is closed on PR #7's exact audited candidate. The only successor +dependency is M016, which remains blocked on M015's accepted closure and +final candidate integration checks. The separate Pi 5/MMC performance plan +has no dependency edge to M014–M016 and remains untouched. + +## 12. Registry updates + +`plans/registry.md` removes M014 from the ready/active tables, records this +closure, promotes M015 (whose hard predecessors are satisfied) as ready, and +retains M016 as blocked. The on-prem roadmap records M014 closed. The +on-prem concurrency-performance M001 status and evidence were not changed by +this closure. diff --git a/plans/closure/on-prem-user-access/015-status.md b/plans/closure/on-prem-user-access/015-status.md new file mode 100644 index 000000000..7b621653e --- /dev/null +++ b/plans/closure/on-prem-user-access/015-status.md @@ -0,0 +1,196 @@ +# On-Prem User Access M015 — Closure Status + +Status: closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/015-authz-threat-review-corrective.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md#m015--independent-authentication-and-authorization-threat-review` + +Repository baseline reviewed: PR #7 candidate `3c7def1e35871209e68563b2ee88f36033aff76e`; final reviewed implementation SHA `d8fd9c2a5cbcd4cda317a215e17e9a188699cb71`. + +## Implementation commits + +- `a526980c` — harden on-prem authentication and authorization boundaries. +- `deb1622f` — address independent review findings on hard-budget reservation and browser budget authorization. +- `d8fd9c2a` — require conservative token proof for hard-cost reservations and allow browser-session authentication for `/api/user-budget` while preserving role and target checks. +- `80852e95` — record M015 active status before implementation. + +## 1. Executive finding + +M015 is closed on the exact PR #7 candidate SHA above. The independent +adversarial review found two substantive defects: hard USD reservations used +estimated token counts instead of the conservative proof bound, and browser +sessions could not reach `/api/user-budget` because personal-credential +middleware rejected the request before session authentication. A subsequent +review also found that the configured shared-key early rejection had the same +effect for cookie-only sessions. All three paths were corrected with +regression coverage. The independent reviewer confirmed the final patch +closed its findings, and the completed scoped Codex Security diff scan found +no additional reportable issues. + +The full local default/no-default Rust suites and exact-SHA hosted CI, +installer and dependency-audit jobs pass. M016 is now dependency-ready; it +owns final PR #7 integration, metadata and merge recommendation. PR #7 remains +open and has not been merged. The separate Pi MMC latency plan was excluded +and untouched. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Review identity, browser-session, personal-token and shared-key boundaries | Independent review of PR #7 auth middleware and session paths; `rust/tests/on_prem_sessions.rs`; scoped security scan `52bd79cc-9e8b-4aef-a405-4ec2486f8e65` | pass | Cookie sessions reach `/api/user-budget`; managed shared-key presence does not block session auth; legacy shared-key behavior remains enforced. | +| Preserve user and role isolation for budget reads | `browser_sessions_can_read_scoped_user_budget_api` in `rust/tests/on_prem_sessions.rs` | pass | Member sees self only; other-member targeting is forbidden; admin can select a member; instance rules stay admin-only; response uses `no-store`. | +| Prove hard-cost monetary holds from a conservative token bound | `cost_hard_holds_require_and_use_conservative_proof_tokens` in `rust/src/user_budget.rs` | pass | Admission and retry use `single_attempt_worst_tokens(proof)`; missing output cap fails closed; an underestimated estimate cannot reduce the hard-cost hold. | +| Review broader route, CSRF, proxy, model-policy, recovery, secret and redaction invariants | M015 route/principal and authz review; existing focused on-prem/session/model/budget/transport/recovery tests; no schema or auth contract change | pass | No new privilege, model-policy, CSRF, trusted-proxy, migration, or secret-redaction violation remains open. | +| Correct findings with bounded changes and regression tests | `deb1622f`, `d8fd9c2a`; focused integration/unit tests | pass | The fixes remain within existing route and budget ownership. | +| Complete final local verification at the reviewed source | Commands and results below | pass | Default and no-default serial suites both pass. | +| Confirm hosted checks at the exact reviewed SHA | CI `38720507683`, installer `38720509034`, dependency audit `38720507720` | pass | All three completed successfully with head SHA `d8fd9c2a5cbcd4cda317a215e17e9a188699cb71`. | + +## 3. Production implementation evidence + +`rust/src/user_budget.rs` computes hard-cost admission and retry amounts from +the single-attempt worst-case token proof, multiplied by the existing bounded +attempt-equivalent factors. It fails closed when a usable proof is absent. +This prevents a low or resource-capped estimate from justifying an +under-reserved hard USD ceiling. + +`rust/src/server/middleware.rs` exempts `/api/user-budget` from the two +personal-credential-only rejection branches so the existing browser-session +authentication and handler authorization can run. This does not grant access +to the shared server key: managed users still resolve through their session, +members remain self-scoped, and legacy mode continues to require its shared +key. The integration test covers configured shared-key behavior, member and +admin scope, cross-member denial, instance-rule visibility, and cache policy. + +No database migration, public API shape, provider routing behavior, or +authentication contract was added or changed. No Pi performance plan or +physical-Pi artifact was changed. + +## 4. Verification executed + +### Commands run + +```bash +cargo test --manifest-path rust/Cargo.toml --lib cost_hard_holds_require_and_use_conservative_proof_tokens -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_sessions browser_sessions_can_read_scoped_user_budget_api -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_sessions -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_global_budgets -- --test-threads=1 +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- --test-threads=1 +git diff --check +``` + +### Results + +- New conservative hard-cost unit test passed. The browser budget integration + test passed, including configured server-key managed mode, member + self-scope, cross-member 403, admin target access, instance-rule visibility, + and `no-store`. +- Focused suites passed: `on_prem_sessions` 10, `on_prem_budgets` 27, + `on_prem_global_budgets` 16. +- Full default workspace suite: 1,024 passed, 1 ignored across 79 suites. +- Full no-default workspace suite: 1,025 passed, 1 ignored across 79 suites. +- Formatting, strict default Clippy, no-default check, and strict no-default + Clippy passed. `git diff --check` passed before the implementation commit. +- Hosted CI [38720507683](https://github.com/eggstack/eggpool/actions/runs/38720507683), + installer qualification [38720509034](https://github.com/eggstack/eggpool/actions/runs/38720509034), + and dependency audit [38720507720](https://github.com/eggstack/eggpool/actions/runs/38720507720) + all passed on exact head `d8fd9c2a5cbcd4cda317a215e17e9a188699cb71`. +- The prior M015 local verification also passed `cargo deny`, locked release + build, Python tooling, release-doc validation, and browser parity/Q007 + qualification. Those results were run on the immediately preceding M015 + implementation before the final narrow Rust correction; the exact final + hosted CI/installer/audit jobs above cover `d8fd9c2a`. + +## 5. Invariant review + +Authentication remains fail-closed. A missing/invalid managed credential +does not fall back to anonymous or legacy access; the budget route reaches +the existing session resolver and role-scoped handler. Cross-user target +selection remains denied to members. Hard USD holds now require a conservative +token proof for both admission and retry. No provider health, quota, request +publication, retry ownership, streaming finalization, secret persistence, or +redaction boundary changed. + +The first scoped security scan (`776c75d7-a6f4-4210-8ba1-d3828989d889`) +reported no findings for its corrective diff, but its bounded review did not +identify the two pre-existing gaps later found by the independent reviewer. +The reviewer’s findings were therefore fixed and separately re-reviewed; the +follow-up scan `52bd79cc-9e8b-4aef-a405-4ec2486f8e65` reviewed the exact +`deb1622f..d8fd9c2a` corrective diff and completed with no reportable +findings. Neither bounded diff scan is represented as an exhaustive +repository-wide security certification. + +## 6. Failure and recovery review + +Missing output proof fails closed for hard-cost admission and retry. Missing +or invalid browser credentials proceed only to the existing session check and +are rejected when no valid session exists. Member target escalation remains +403. The changes add no persistent state and therefore require no recovery or +restart procedure. + +## 7. Migration and compatibility review + +No schema or migration changed. Existing personal agent credentials, browser +session cookies, CSRF validation, legacy shared-key mode, and managed-mode +roles retain their contracts. Hard-cost reservations become more +conservative and may reject when no output proof exists; that is the +fail-closed behavior required by the existing hard-budget contract. + +## 8. Security review + +The initial independent review identified and ranked: + +| Severity | Finding | Disposition | +|---|---|---| +| High | Hard USD reservations derived from estimates rather than conservative token proof, allowing under-reservation | Fixed in `deb1622f` and strengthened in `d8fd9c2a`; unit regression test proves estimate cannot lower hold and missing proof fails closed. | +| Medium | Browser sessions could not access `/api/user-budget` through personal-credential middleware | Fixed in `deb1622f`; middleware now permits session authentication, with a role and target-isolation integration test. | +| Medium | Configured shared-key early rejection still blocked cookie-only session access in managed mode | Fixed in `d8fd9c2a`; same integration test uses a configured server key and confirms managed cookie-session authorization. | + +The independent reviewer confirmed the final corrections close all reported +findings. The exact corrective-diff security scan found no additional +reportable issue. There are no unresolved high or medium authorization or +privacy findings. + +## 9. Documentation and operations + +No product documentation change was required; the fixes restore existing +authorization and hard-budget contracts rather than introducing new behavior. +The closure, roadmap, plan status and registry now record current state. +M016 must reconcile stale PR #7 description and perform the final merge +readiness review. PR #7 has not been merged. The physical Pi 5 restricted +operating envelope in M012 remains the only qualified performance statement; +the separate Pi MMC attribution plan is not a dependency and was not touched. + +## 10. Unresolved findings + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| None | No unresolved M015 security-correctness finding | No known M015 release blocker remains. | M016 performs final candidate and PR reconciliation; do not merge automatically. | + +## 11. Roadmap disposition + +M015 is closed; M016's hard dependencies M014 and M015 are accepted. The +registry audit found M016 is the only registered successor blocked on this +closure, so it is promoted to `ready`. It can now begin final-SHA integration +and PR #7 reconciliation. M016 remains an explicit no-merge plan. The +separate on-prem concurrency-performance M001 Pi 5/MMC plan is neither a +dependency nor modified by this closure. + +## 12. Registry updates + +In the same closure commit, `plans/implementation/on-prem-user-access/015-authz-threat-review-corrective.md` +is marked `implemented`; the roadmap marks M015 `closed` and M016 `ready`; +`plans/registry.md` removes M015 from active work, records M015 in recent +closures, and promotes M016 from blocked to ready. The stale registry audit +claim that M015 “remains ready” is replaced with the accepted closure and +successor-promotion evidence. No Pi plan, closure, or artifact was changed. diff --git a/plans/closure/on-prem-user-access/016-status.md b/plans/closure/on-prem-user-access/016-status.md new file mode 100644 index 000000000..7d7569568 --- /dev/null +++ b/plans/closure/on-prem-user-access/016-status.md @@ -0,0 +1,207 @@ +# On-Prem User Access M016 — Closure Status + +Status: closed + +Source implementation plan: + +- `plans/implementation/on-prem-user-access/016-final-pr7-merge-reconciliation.md` + +Source subsystem roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md#m016--final-integration-and-merge-readiness-reconciliation` + +Repository baseline reviewed: PR #7 base `main` at `9a0a903c9944b69a6c1ae9fa969bb8603d5c65d9`; runtime candidate `d8fd9c2a5cbcd4cda317a215e17e9a188699cb71`; M016 reconciliation head `3d68c715fdf5342cc36b46bef52e73dc1b9f5ba0` (planning-only commits over the same runtime tree). + +## Implementation commits + +- `7462263e` — M014 Rustls advisory corrective. +- `a526980c`, `deb1622f`, `d8fd9c2a` — M015 authz implementation and independent-review corrections. +- `d5683b1a` — close M015 and promote M016 to ready. +- `3d68c715` — activate M016 and reconcile PR #7 title/description. + +## 1. Executive finding + +M016 completed the final pre-merge reconciliation for PR #7 and recommends +**merge held pending maintainer review**. M014 and M015 are closed, local +default/no-default and tooling/browser/release checks pass, and hosted CI, +installer and dependency-audit checks pass on the exact M016 reconciliation +head `3d68c715fdf5342cc36b46bef52e73dc1b9f5ba0`. The runtime code at that +head is unchanged from the exact tested implementation candidate +`d8fd9c2a5cbcd4cda317a215e17e9a188699cb71`. + +The hold is procedural: GitHub reports no submitted PR reviews +(`reviews: []`, `reviewDecision` empty). The independent M015 reviewer +confirmed its corrective findings were closed, but no maintainer has +submitted a formal PR review. PR #7 remains open and unmerged. This closure +records the completed recommendation; it does not approve or merge the PR. +The overall on-prem user-access roadmap remains active until a maintainer +decides whether to merge. + +## 2. Requirement-to-evidence matrix + +| Requirement | Evidence | Result | Notes | +|---|---|---|---| +| Freeze base/head and inspect the complete PR diff | PR #7; `gh pr view`; `gh api repos/eggstack/eggpool/compare/main...plans/on-prem-small-team-user-access` | pass | Base `main` at `9a0a903c`; tested head `3d68c715`; 78 commits ahead, 0 behind; 154 changed files, 40,350 additions and 331 deletions. | +| Reconcile append-only migrations from schema 54 | `rust/assets/db/migrations/0055_user_identity.sql` through `0062_instance_budgets.sql`; `checksums.json`; M010/M012 closure records | pass | Eight append-only migrations advance schema 54 through 62. M010 records a frozen schema-54 restore/forward migration to 62 with legacy history preserved; M012's physical artifact used schema 62. No M014–M016 migration was added. | +| Verify M014 advisory remediation and hosted audit | `plans/closure/on-prem-user-access/014-status.md`; audit [38721601318](https://github.com/eggstack/eggpool/actions/runs/38721601318) | pass | Rustls 0.23.46; no advisory ignore; exact M016-head audit passes. `main` remains on its old resolution until PR merge. | +| Verify M015 security findings and corrective behavior | `plans/closure/on-prem-user-access/015-status.md`; independent reviewer; scans `776c75d7` and `52bd79cc` | pass | Hard-cost proof reservation and managed browser-session access findings corrected and independently re-reviewed; no unresolved M015 security-correctness finding. | +| Run exact-head CI and tooling | CI [38721601336](https://github.com/eggstack/eggpool/actions/runs/38721601336) | pass | Exact PR head `3d68c715`; serial Rust suite, formatting, strict Clippy, no-default checks, Ruff, Pyright and tooling pytest all pass. | +| Run Linux/macOS installer shell qualification | Installer [38721601305](https://github.com/eggstack/eggpool/actions/runs/38721601305) | pass | Linux Bash and macOS zsh pass on exact PR head `3d68c715`. | +| Requalify dashboard and live-provider paths | Dashboard parity command and Q007 command below | pass | On-prem page reference: 70 observations / 22 screenshots; Q007 loopback fake: 7 passed. Historical-page parity remains the accepted M011 frozen-oracle result; no dashboard HTML changed in M014–M016. | +| Validate release docs and installed release executable | `scripts/validate_release_docs.py`; locked release build and isolated installed-binary execution | pass | Docs matrix validates release 0.8.2. Locally built release binary reports 0.8.2 and runs `--help`; SHA-256 recorded below. The quick-install script correctly rejects this Mac x86_64 host as unsupported for raw release installs. | +| Preserve measured Pi envelope and avoid unsupported claims | `docs/raspberry-pi.md` lines 237–272; M012 closure and artifacts | pass | Docs continue to distinguish 49 named user records from inference concurrency; 2–4 concurrent requests are the limited operating envelope, and 8-way saturates. No Pi performance plan or artifact was changed. | +| Reconcile PR metadata and report merge decision | PR #7 title/body; review API and `gh pr view` | pass | Title/body now reflect M014/M015 closure, current candidate and Pi envelope. Recommendation is merge held pending a maintainer PR review; PR remains open. | + +## 3. Production implementation evidence + +M016 made no runtime, dependency, schema, protocol or operator-config change. +The tested runtime candidate is `d8fd9c2a`; M016's subsequent commits only +update plans and PR metadata. The current PR head `3d68c715` has the same +runtime tree. The complete PR diff includes the on-prem feature, M014's +lockfile-only Rustls patch and M015's bounded security corrections. + +Migration inventory is append-only: 0055 user identity, 0056 web sessions, +0057 request attribution, 0058 model policy, 0059 user budgets, 0060 admin +audit, 0061 budget enforcement and 0062 instance budgets. The migration +checksum manifest includes them. M010's schema-54 restore test is retained +as prior closure evidence; no downgrade migration is promised, and operators +must restore the matching pre-upgrade database backup to roll back. + +The PR title is now “Implement on-prem small-team access — merge held for +maintainer review”. Its description says the work is on this branch +and not yet in `main`, links M014/M015 evidence, updates the measured Pi +envelope, and distinguishes 49 named users from concurrent inference. + +## 4. Verification executed + +### Commands run + +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- --test-threads=1 +cargo build --manifest-path rust/Cargo.toml --locked --release +uv run pytest tests/tooling/test_qualification_live_provider.py -q +uv run python scripts/validate_release_docs.py +uv run python scripts/qualification_dashboard_parity.py --skip-build --on-prem --screenshot-dir /tmp/m016-onprem-shots +git diff --check +``` + +### Results + +- At runtime candidate `d8fd9c2a`: default workspace suite 1,024 passed / 1 + ignored; no-default workspace suite 1,025 passed / 1 ignored (79 suites). + Focused auth/budget suites passed: sessions 10, user budgets 27, global + budgets 16. The added hard-cost proof unit test and browser-session budget + API integration test passed. +- Format, strict default Clippy, no-default check and strict no-default + Clippy passed locally. Exact M016 head CI [38721601336] passed the full + hosted Rust/tooling workflow, including serial default Rust tests, Ruff, + Pyright and the full tooling suite. +- Exact M016 head installer run [38721601305] passed Linux Bash and macOS + zsh. Exact M016 head dependency audit [38721601318] passed cargo-deny. + The audit workflow invokes `.github/workflows/dependency-audit.yml` against + `rust/Cargo.toml` with all features and checks advisories, bans, licenses + and sources; no advisory exception was introduced. +- `uv run pytest tests/tooling/test_qualification_live_provider.py -q`: + 7 passed. `scripts/validate_release_docs.py`: pass for published 0.8.2 and + Linux x86_64, Linux aarch64 and macOS arm64 targets. On-prem dashboard + qualification: 70 observations and 22 screenshots passed against + `tests/fixtures/dashboard-on-prem-browser/reference.json`. +- `cargo build --manifest-path rust/Cargo.toml --locked --release` passed. + The release executable was copied to an isolated temporary install path; + `eggpool version` returned `0.8.2` and `eggpool --help` completed. Binary + SHA-256: `bc576809c7cf190393b6face4afbfde4fb25051bf3863f7777db606675b0e7e7`. + The repository's raw installer was also invoked from the checkout and + refused before install with `unsupported platform Darwin/x86_64`, as this + host is outside the supported raw-release target matrix. The binary itself + was independently installed and exercised in the temporary path. +- No release was packaged or published. Existing M014 locked release build, + M011 historical Python-oracle parity (54 unchanged historical gaps), M010 + schema-54 restore evidence and M012 physical artifacts are referenced from + their immutable closure records. + +## 5. Invariant review + +The base is current and the PR has no merge-base conflict (`behind_by: 0`). +Rustls advisory remediation is on the PR candidate and remains absent from +`main` until merge. M015 confirmed managed/legacy auth separation, member +budget isolation and proof-bounded hard-cost holds. The default and +no-default suites pass; no known route, model, quota, recovery, TLS, +transcode, or redaction regression remains. The user-access boundary remains +single-instance and local; there is no external IAM or provider-secret web +editor. + +## 6. Failure, recovery and contention review + +The M010 closure covers startup recovery of abandoned user/global budget +holds and no double-charge behavior. M015's integration matrix covers +configured-key managed browser sessions, member self-scope, cross-member +denial and admin instance-rule visibility. M014 retains wrong-root and +hostname-mismatch TLS rejection. M013 retains the Messages-to-Responses +eligibility regression guard. M012 and `docs/raspberry-pi.md` keep MMC tail +behavior explicit; no WAL or SQLite writer topology change is proposed. + +## 7. Migration and compatibility review + +Schema 54 to 62 uses immutable append-only migrations 0055–0062 and the +checksum manifest; restored-schema compatibility and legacy history are +covered by M010. M016 adds no migration. Current personal tokens, browser +sessions, legacy-key migration behavior, provider TOML, local model choices, +solo SBC install, client wire surfaces and explicit backup-based downgrade +procedure remain unchanged. The locked dependency graph passes hosted +cargo-deny and release build. + +## 8. Security and reviewer disposition + +M014 clears the four RustSec advisories by upgrading the resolved Rustls +family to 0.23.46 without suppression. M015's independent reviewer found and +closed the hard-cost reservation and browser-session authorization defects; +the final scoped security scan of `deb1622f..d8fd9c2a` completed with zero +reportable findings. Its earlier bounded scan did not find those issues, so +the independent reviewer evidence and corrective re-review are retained +explicitly in the M015 closure rather than overstating scan coverage. + +GitHub currently reports `reviews: []` and an empty `reviewDecision`. No +maintainer PR review or approval has been submitted. This is the exact reason +for the **merge held** recommendation. The automated checks are green; no +code finding requires another corrective plan. A maintainer must review the +reconciled PR and decide whether to merge. + +## 9. Documentation and operations + +The PR body is reconciled with current implementation and M014–M015 status. +README, deployment, model-limit, backup and security architecture claims +required no source change: the on-prem implementation and M012 envelope +remain accurately described. The Pi documentation still states seconds-scale +MMC tails, limits practical concurrency to 2–4 requests, and says 49 named +records are not 49 simultaneous generations. The separate Pi latency plan +was not opened, edited or used as a blocker. + +## 10. Unresolved findings and hold + +| Severity | Finding | Impact | Required action | +|---|---|---|---| +| Operational hold | No maintainer review has been submitted on PR #7 (`reviews: []`, `reviewDecision` empty). | The code and hosted checks are ready, but the requested final reviewer/maintainer gate is not satisfied; do not merge automatically. | Maintainer reviews PR #7 and independently decides whether to merge. No further implementation plan is required unless that review finds a new bounded defect. | + +## 11. Roadmap disposition and unblock audit + +M016 is closed as a completed pre-merge reconciliation with recommendation +**merge held pending maintainer review**. The on-prem user-access roadmap +remains active because PR #7 is open and its runtime changes are not in +`main`. A registry and dependency-graph search found no registered plan +blocked on M016; no future implementation plan can be promoted from this +closure. Persistence M011 remains independently ready. On-prem concurrency +performance M001 remains a separate Pi workstream and was explicitly excluded +from this work. + +## 12. Registry updates + +The registry records M014 and M015 closed, M016 implemented/closed with an +explicit merge hold, and no blocked on-prem successor. The on-prem roadmap +retains active status until the maintainer decision and PR integration. No +M001–M015 historical closure was rewritten, no Pi plan or artifact was +modified, and no merge was performed. diff --git a/plans/implementation/on-prem-concurrency-performance/001-pi5-mmc-tail-latency-attribution.md b/plans/implementation/on-prem-concurrency-performance/001-pi5-mmc-tail-latency-attribution.md new file mode 100644 index 000000000..c50bcba40 --- /dev/null +++ b/plans/implementation/on-prem-concurrency-performance/001-pi5-mmc-tail-latency-attribution.md @@ -0,0 +1,255 @@ +# On-Prem Concurrency Performance M001 — Pi 5/MMC tail-latency attribution + +Status: implemented + +Repository baseline: `3c7def1e35871209e68563b2ee88f36033aff76e` (Pi 5 M012 closure and release 0.8.2 candidate). + +Source roadmap: +- `plans/subsystems/on-prem-concurrency-performance-roadmap.md#m001--pi-5mmc-concurrent-tail-attribution` + +Long-term requirements: +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) +- Existing persisted WAL/single-worker architecture; a new accepted ADR is + **required** before modifying journal/concurrency topology. + +Primary class: polish + +## 1. Objective + +Determine why the existing Pi 5/MMC qualification reports seconds-scale +p95/max while solo p50 remains milliseconds, without changing any +production SQLite/storage/identity behavior until reproducible causal +evidence exists. This plan is **nonblocking to PR #7 merging** given +the existing restricted concurrency envelope and passing correctness. + +## 2. Why ready + +Hard: accepted M012 closure and frozen artifacts exist. +Interface: existing SBC/diagnostic instrumentation runners and +immutable one SQLite gate contract. Operational: a physical Pi 5 +with its microSD/MMC boot storage is the required target. +Alternate storage is an optional diagnostic control, not a +qualification prerequisite. Local code/tooling instrumentation +can start now, but no metric may be invented without physical +target evidence. + +## 3. Current evidence and why prior work missed attribution + +M012 Pi 5 8 GB, Ubuntu 24.04, ext4/MMC, release schema 62: +solo p50 ~4–5 ms, streaming TTFT p95 ~4–19 ms, +4-way benchmark concurrent p50 ~13–15 ms with p95 +10–14 s and maxima similar; 8-way p50 ~4.1 s. +Measured p95/max fail predeclared G3 gate (1/3 s). +RSS stays ~20 MB, WAL converges, 49 named users +provision but that is account count, not throughput. +Target had background `opencode` CPU, `kblockd` MMC +pressure and an observed transient D-state worker; +so neither workload contention nor an EggPool-specific +SQLite bottleneck has been isolated. `plans/closure/on-prem-user-access/012-status.md` +keeps a medium finding with restricted envelope. +Previous persistence M007/M008 Pi/MMC checkpoint experiments +failed their own tail/WAL gates, so a blind WAL reset is unsafe. + +Relevant owners: +`scripts/qualification_sbc.py`, +`scripts/qualification_stability.py`, +`tests/tooling/test_qualification_sbc.py`, +`rust/src/db/connection.rs`, +`rust/src/coordinator/publication.rs`, +`rust/src/coordinator/finalization.rs`, +`rust/src/user_budget.rs`, `rust/src/instance_budget.rs`. +Historical baseline evidence lives in +`artifacts/qualification/on-prem-m012-pi5-*.json` +and persistence Pi artifacts. + +## 4. Invariants that must not regress + +- Single worker/gate, WAL/NORMAL, generation and retry/finalization, + deterministic quota enforcement and profile gates remain unchanged. +- No extra services, cloud usage, producer DB connection, + global thread explosion, authenticated API bypass, + protocol body rewrite or dynamic checkpoint retune. +- Operator provider TOML/keys and downstream SQLite identity + retain current ownership; no sensitive data in diagnostics. +- Preserve real observed poor tail results; no relabeling + as pass and no fabricated Pi capacity statement. + +## 5. Scope + +### In scope + +Controlled replay of the M012 fake-provider corpus +on Pi 5 microSD/MMC with 1,2,4,8 in-flight requests across +configured 1/5/49 named accounts, comparing (a) idle clean +device, (b) deliberate bounded competing disk I/O, +(c) microSD/MMC against tmpfs and, only if useful and available, +optional alternate storage, +(d) managed auth with/without hard user/global budgets, +(e) matched solo pre-multitenancy candidate control. +Correlate disk service/queue/major faults/scheduler, +SQLite gate wait vs transaction tenure, WAL bytes/ +checkpoint/fsync, Argon2 login activity, latency and TTFT. +Do phase instrumentation only where needed and behind a +qualifier/test feature with zero production default overhead. + +### Out of scope + +No guaranteed optimizations, permanent extra tracing +service, SQL schema migration, adoption of PERSIST/EXTRA, +second writer/checkpointer, changing provider transport +or model client wire, retry algorithm rewrite, benchmark +using production provider keys, WAN load generator, or +unverified 50-concurrent operating claim. + +## 6. Required production changes + +**None by default**. Add bounded local qualification +instrumentation or opt-in counters only if the current +runner cannot isolate a phase. Prefer existing spans/SQLite +gate metrics and OS block I/O counters before writing Rust +hot-path code. If a production defect is actually isolated, +do not implement it inside this diagnostic: author the +next narrowly scoped corrective with regression/test +and architecture review (ADR if persistence topology changes). + +For a diagnostic-only mode that now 401s due to shared +legacy key (M012 low finding), update only the disposable +qualification runner to use local `users bootstrap` personal +token for managed mode and verify secrets never enter JSON. +Do not alter server auth to make an old script pass. + +## 7. Ordered work packages + +A. Freeze source/hardware/control artifact, write hypotheses, + exact target environment/competing I/O controls and + predeclared p95/p99/MAX/WAL/RSS acceptance and abort gates. + +B. Add or reuse minimal opt-in instrumentation for + publish/finalize SQL gate queue/wait/hold vs provider/response; + record zero-production-change baseline where possible. + +C. Run paired/repeated target trials (prefer >=3 per + condition) with bounded login and concurrency; record + I/O queue/fsync, CPU/background process load, same fake + provider/wire corpus and isolated data dirs. + +D. Compare distribution shifts and falsify causes + (e.g. remove competing microSD activity, move isolated + DB to tmpfs or optional alternate storage, disable budget rules while keeping model + metadata fixed). Publish cause confidence, operator + mitigation and exact next-plan/no-code decision. + +## 8. Failure, cancellation, restart and contention + +Abort before memory/swap/OOM or sustained disk +saturation damages a shared SBC. No process kill +outside test-managed PID group. All burst runs +must settle pending/active user/instance holds +to zero, recover on restart, preserve WAL +quick-check and show no last-slot oversubscription. +Record test fixture's competing I/O details and +monitor WAL nonconvergence; stop immediately +on any correctness or encryption regression. + +## 9. Compatibility and migration + +No product DB migration, TOML change or API schema +change permitted. New output artifacts must be +versioned, redacted, self-describing and comparable +with original M012 artifacts. Preserve M012 +correctness and existing constrained capacity docs. +If method changes make old gate percentiles +incomparable, label the difference rather than +claim performance gain. + +## 10. Required tests + +Extend `tests/tooling/test_qualification_sbc.py` +and add narrow `test_on_prem_mmc_tail_attribution.py` +if necessary; use `scripts/qualification_sbc.py --help` +to resolve **real** CLI flags. Validate 49 named +users versus 1/2/4/8 active, secret redaction, +target attestation, WAL and pending/active convergence, +bounded output, abort controls and paired candidate +identity. If Rust diagnostic spans are added, +write opt-in zero-runtime-default overhead tests. +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_global_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_finalization -- --test-threads=1 +uv run pytest tests/tooling/test_qualification_sbc.py -q +``` + +## 11. Required verification commands + +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +uv sync --frozen +uv run pytest tests/tooling/ -q --tb=short --maxfail=1 +git diff --check +``` +Any additional script/feature change requires its +corresponding existing targeted tooling/build gate. +Physical data on a real Pi/MMC is mandatory for closure; +report exact commands and actual results only. + +## 12. Documentation updates + +Update `docs/raspberry-pi.md` only when a +**reproducible** environment mitigation or +resource requirement changes practical operator +guidance. Link qualified artifacts under +`artifacts/qualification/`; do not retroactively +edit M012 accepted closure evidence. + +## 13. Acceptance criteria + +Each candidate tail source has a controlled +falsification attempt and measured contribution. +Reproduction and negative/control runs include +physical device attestations, repeatable work +and the actual concurrency, with p50/p95/p99/max, +gate hold/wait, WAL/checkpoint/IO telemetry and +zero leaked authentication material. Emit a +supported cause/uncertain/no-code conclusion and +either an operator mitigation or a precise new +bounded corrective proposal. No current PR #7 +release gate depends on this work. + +## 14. Stop conditions + +Absent provisioned target or clean control, +nonconverging WAL, orphan budget holds, +unbounded I/O pressure, raw secret artifacts, +nonreproducible claimed improvement, or proposal +to change journal/multiwriter topology without ADR. +Mark the diagnostic blocked/conditional and +preserve the documented Pi envelope rather +than declaring a fix. + +## 15. Closure evidence required + +`plans/closure/on-prem-concurrency-performance/001-status.md` +must list hardware/media/OS/build, frozen thresholds, +hypotheses and attempted falsifications, paired +3x raw artifacts, variance and confounders, +service wait/IO/WAL latency attribution, serial +tests and hosted result (if source changes), +specific operator/next-plan decision and registry +unblock audit. + +## 16. Handoff notes + +Separate parallel/nonblocking workstream. +No claim of fixed concurrency performance before +a later measured implementation closure. diff --git a/plans/implementation/on-prem-user-access/001-admin-bootstrap-and-credentials.md b/plans/implementation/on-prem-user-access/001-admin-bootstrap-and-credentials.md new file mode 100644 index 000000000..2511a8a02 --- /dev/null +++ b/plans/implementation/on-prem-user-access/001-admin-bootstrap-and-credentials.md @@ -0,0 +1,189 @@ +# On-Prem User Access Milestone 001 — Admin bootstrap and client credentials + +Status: implemented + +Repository baseline: `main` at `9a0a903c9944b69a6c1ae9fa969bb8603d5c65d9` (architecture/roadmap authority committed on `plans/on-prem-small-team-user-access` at `81beec4e215e8991880d407ed2752d01ff88d58a`) + +Source roadmap: +- `plans/subsystems/on-prem-user-access-roadmap.md#m001--secure-admin-bootstrap-and-client-credentials` + +Long-term requirements: +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADRs: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) + +Primary class: infrastructure + +## 1. Objective and boundary + +Provide an atomic, local, non-default administrator bootstrap and distinct +per-user API credentials on **fresh** installations, plus safe staged +compatibility for **existing** ones. This milestone is an infrastructure +foundation, not a claim that interactive multi-user web management works. + +No hard predecessor beyond ADR-0002 and existing native lifecycle/storage +contracts. This phase must not depend on internet access or a web login. + +## 2. Baseline evidence + +`rust/src/cli.rs` exposes `Onboard`, `Newkey`, `Getkey`; +`rust/src/runtime.rs` implements their workflows. +`rust/src/server/middleware.rs` checks one constant-time configured key, +and keyless loopback is accepted. `rust/src/config.rs` defaults LAN bind and +public dashboard. `rust/src/db/migrations.rs` embeds checksum-validated +migrations through 0054 and `rust/src/db/connection.rs` is the one worker/gate; +`rust/src/server/mod.rs` builds Axum/EggServe routes. Upstream `accounts` +in `0001_initial.sql` must not become human accounts. +Existing `rust/tests/cli_contract.rs`, `server_transport.rs`, +`database_compatibility.rs` and `operations_o005.rs` are regression owners. + +### Always-preserved invariants +- A single on-prem instance; TOML/environment retains upstream provider topology/credentials, model routing, and operator-owned settings. SQLite owns only downstream users, sessions, grants, profiles, budget reservations, consumption and audit. +- No SaaS organizations, hosted signup, embedded reverse proxy/VPN, enterprise identity dependency, separate service/database, provider credential web editor, or assertion of legal authority to share personal subscription seats. +- All legacy inference wire paths and return shapes (Chat Completions, Messages, Responses, Responses compact), streaming/terminal semantics, redaction, configuration lifecycle, and operator CLI remain compatible unless an explicit migration contract states otherwise. +- Respect `GenerationLease`, coordinator publication/finalization compensation, one SQLite gate/worker with WAL/NORMAL, `config_reload_policy.rs` as reload authority, and `--no-default-features` compilation. +- In-flight resource ownership remains deterministic. No database locks span provider I/O or expensive password hashing. No raw prompt, token, password, session, provider key, or cache key reaches diagnostics or ordinary request storage. + +### Shared repository verification +Run on implementation and report actual results in the closure record, not the plan: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +``` +For dependency changes also run locked release build, `cargo deny` and +`cargo tree -e features`; for user-facing release-doc edits run +`uv run python scripts/validate_release_docs.py`. Ensure `git diff --check`. + +## 3. Required production changes + +### A — Immutable user identity and migration +Create `users` with stable opaque user IDs, unique normalized usernames, +explicit administrator role, active/disabled state, first-login/password +change requirement and password-hash metadata. Add `user_credentials` +(per-user label, scope, irreversible HMAC/digest, lookup selector, +created/last-used/expiry/revoked state); reserve naming space for later +`web_sessions`, policies, and audit. Prefer Argon2id +with per-password salt, independently reviewed bounded parameters calibrated +for Le Potato/Pi class, an explicit hash-algorithm/parameter version, and +a strict parallelism/rate limit. A zero-byte/plain password is forbidden. +Use OS cryptographic RNG for high-entropy, prefix-identifiable client secrets; +display only at creation and hash at rest. Do not use upstream `accounts`. + +Add checksum-verified, append-only `0055_*.sql` migration and update +`rust/assets/db/migrations/checksums.json` via the existing build process +(not hand-rewriting old entries). Extend backups, restores, migration and +permissions qualifications; schema upgrades are all-or-nothing. + +### B — Local administrator bootstrap +Extend `eggpool onboard` and the local operations/CLI boundary to create +one admin using interactive TTY username/password+confirmation; provide +non-echo stdin or FD input for unattended installation, no argv password, +default `admin` username allowed but **no default password**. Initial +setup must be idempotent and race-safe: only one initial admin transaction, +no remote bootstrap endpoint, no bootstrap flag in GET requests, no +network-unlocked empty DB transition. Add a local `eggpool users` CLI +entry for inspection and protected password reset/recovery, preserving +local config/data-dir ownership and symlink refusal. A recovery reset +invalidates credentials/sessions according to an explicit documented +security choice; it must not reveal existing passwords. + +### C — Runtime personal API key authentication +Implement the credential validator at the existing ingress middleware, +accepting `Authorization: Bearer` / `x-api-key`, supporting constant-time +verification, bounded lookup caches and negative-rate controls. API keys +are inference/read-only integration credentials, **never** web-admin grants. +Create a token for the first admin through protected CLI output/TTY; keep +`EGGPOOL_API_KEY` environment semantics (a per-user token fits). +No password-to-inference fallback. Maintain current configured global server +key as a **scoped legacy inference credential only** during explicit +migration. No extra auth server or HTTP stack. + +### D — Upgrade staging and feature state +Define a tiny explicit auth lifecycle (`legacy_unmigrated`, +`admin_initialized`, later `managed`) with startup checks. Pre-existing +configs/databases must start with their existing credentials and dashboard +semantics until a documented local migration command is executed; avoid +unattended systemd lockout. Fresh installations must not start in +unauthenticated loopback or expose an admin/default credential. Until M002 +adds browser sessions, no extra users or mutating web-admin routes are +enabled. Do not turn `dashboard.public=true` into a write privilege. +`getkey/newkey` retain legacy semantics only while legacy mode remains +valid, with guided replacement and explicit no-global-key behavior after +cutover. Do not automatically rotate unrelated upstream provider secrets. + +## 4. Ordered work packages + +1. Finalize credentials/roles/scheme and append migrations; prove zero + destructive changes on fixtures and restored backups. +2. Add local bootstrap/recovery operations and CLI; prove fresh vs upgrade + behavior, stdin secrecy and concurrent first-run exclusivity. +3. Add personal token auth + legacy scoped key verification; prove agent + surface compatibility, no server-admin privilege or secret logging. +4. Wire single-user fresh installation and migration diagnostics into + onboard/init, installer qualification, operations/docs; keep dormant user + API/browser features for M002. + +## 5. Failure, cancellation, restart and contention + +Bootstrap commits identity and password hash together; interrupted stdin +cannot leave an empty-password admin. Password hashing happens outside DB +critical sections with bounded memory concurrency. Concurrent bootstrap +attempts return one committed admin. Token creation and revocation are atomic; +cache entries cannot outlive revocation once managed in later milestones +(design invalidation contract now). Failure to read identity state in +managed/fresh configuration fails closed, not keyless loopback. +No locks held during provider I/O. A daemon restart retains token hashes +but never resurrects revoked keys. + +## 6. Compatibility and migration + +No changes to provider TOML or upstream `accounts`. +Legacy inference key remains usable only before supervised cutover and cannot +gain admin powers; legacy user identity/usage separation is specified but +attribution lands in M003. Document upgrade rollback limits for auth schema, +protected DB backup, recovery from lost administrator password, and how +existing `configremote` desktop profiles obtain per-user tokens. + +## 7. Required tests and verification + +Create `rust/tests/on_prem_identity.rs`. Extend +`rust/tests/cli_contract.rs`, `server_transport.rs`, +`database_compatibility.rs`, `operations_o005.rs`, and installer tests. +Cover no password defaults, password equality/entropy rules, Argon2 verification, +bounded malicious-login concurrency, header precedence/shape/constant-time +checks, token expiration/revocation, restored legacy keys, keyless-loopback +fresh rejection, ephemeral `HOME` with non-interactive stdin, +simultaneous bootstraps, interrupted writes and no sensitive diagnostics. +Run focused: +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_identity -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test database_compatibility -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test cli_contract -- --test-threads=1 +``` +Also run the shared matrix above. New password hashing dependency must +qualify Rust binary size/RSS and `--no-default-features`; no network services. + +## 8. Documentation, acceptance, stop and closure + +Update `architecture/deep-dive-security.md`, +`architecture/deep-dive-database.md`, `docs/deployment.md`, +`docs/cli-reference.md`, `docs/backup-restore.md`, and +`docs/agent-configuration.md`. Do not imply browser login exists. + +Accept only when fresh admin setup, agent key, legacy upgrade, +password reset, restart and schema restore work without a default +credential or external dependency. Stop if migration would lock out an +unattended existing installation or alter provider auth/topology. +Closure file `plans/closure/on-prem-user-access/001-status.md` must include +commits, exact migration/checksum fixture evidence, actual commands/results, +local adversarial tests, SBC password-hashing measurements, legacy upgrade, +rollback/recovery notes, and registry unblock audit. Never call M001 +multi-user web authorization. + diff --git a/plans/implementation/on-prem-user-access/002-browser-sessions-and-route-auth.md b/plans/implementation/on-prem-user-access/002-browser-sessions-and-route-auth.md new file mode 100644 index 000000000..12e5a6bba --- /dev/null +++ b/plans/implementation/on-prem-user-access/002-browser-sessions-and-route-auth.md @@ -0,0 +1,159 @@ +# On-Prem User Access Milestone 002 — Browser sessions and route authorization + +Status: implemented + +Repository baseline: `main` at `9a0a903c9944b69a6c1ae9fa969bb8603d5c65d9` (architecture/roadmap authority committed on `plans/on-prem-small-team-user-access` at `81beec4e215e8991880d407ed2752d01ff88d58a`) + +Source roadmap: +- `plans/subsystems/on-prem-user-access-roadmap.md#m002--browser-sessions-and-route-authorization` + +Long-term requirements: +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADRs: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) + +Primary class: capability + +## 1. Objective and boundary + +Enable password-authenticated browser sessions and explicit admin/member route +authorization, without mixing agent bearer keys with human sessions. +**Hard blocker:** M001 must have an accepted closure; do not mark ready +before the schema/credential/bootstrap contracts are verified. + +## 2. Baseline and contracts + +`rust/src/server/mod.rs::build_router` installs global +`authenticate` / `admit_inference_body` layers; `requires_auth` has +special public dashboard cases, including `/api/*`. +`DashboardConfig.public` defaults true. Dashboard rendering lives in +`rust/src/server/dashboard/`; there is currently no session/CSRF web +authority. M001 supplies `users`, credential handling and auth-state +transition. Do not replace EggServe or add OAuth/OIDC. + +### Always-preserved invariants +- A single on-prem instance; TOML/environment retains upstream provider topology/credentials, model routing, and operator-owned settings. SQLite owns only downstream users, sessions, grants, profiles, budget reservations, consumption and audit. +- No SaaS organizations, hosted signup, embedded reverse proxy/VPN, enterprise identity dependency, separate service/database, provider credential web editor, or assertion of legal authority to share personal subscription seats. +- All legacy inference wire paths and return shapes (Chat Completions, Messages, Responses, Responses compact), streaming/terminal semantics, redaction, configuration lifecycle, and operator CLI remain compatible unless an explicit migration contract states otherwise. +- Respect `GenerationLease`, coordinator publication/finalization compensation, one SQLite gate/worker with WAL/NORMAL, `config_reload_policy.rs` as reload authority, and `--no-default-features` compilation. +- In-flight resource ownership remains deterministic. No database locks span provider I/O or expensive password hashing. No raw prompt, token, password, session, provider key, or cache key reaches diagnostics or ordinary request storage. + +### Shared repository verification +Run on implementation and report actual results in the closure record, not the plan: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +``` +For dependency changes also run locked release build, `cargo deny` and +`cargo tree -e features`; for user-facing release-doc edits run +`uv run python scripts/validate_release_docs.py`. Ensure `git diff --check`. + +## 3. Required production changes + +### A — Browser authentication surface +Add dedicated `GET /login`, `POST /login`, `POST /logout`, +`GET /account`, `POST /account/password` (or equivalent versioned +auth API), with typed request validation, generic failure messages, +login throttling and no username enumeration. Support first-login +password change. Session IDs are independent high-entropy opaque tokens; +store only digests plus user association, issue/expiry/revocation, inactivity +limits and rotating tokens as appropriate in `web_sessions`. Bind +cookies `HttpOnly`, `SameSite` appropriately, `Secure` on HTTPS, +narrow `Path`, and server-side expiry. Ensure logouts/password resets +revoke active sessions with a bounded invalidation cache. + +### B — Explicit route/access matrix +Replace catch-all public/private boolean behavior with tested classifications: +anonymous: only health/readiness, needed static assets and login; +member: own account/settings/usage when that surface exists; +admin: full dashboard diagnostics and protected config-adjacent views; +agent token: inference, `/v1/models`, scoped integration profile only. +`/api/stats/runtime`, `/api/status`, admin mutations and +`/api/integrations/*` receive explicit rules. Unknown `/api/*` +paths default private/404, never inherit `dashboard.public`. No agent +credential may create web sessions/admin privilege. Preserve legacy-readonly +dashboard compatibility only in legacy mode with explicit migration guard. + +### C — CSRF and transport posture +For cookie-authenticated unsafe methods, require CSRF proof (same-origin +Origin/Referer validation plus per-session token or synchronizer mechanism); +reject cross-site GET-to-mutation, CORS credential surprises, and spoofed +forwarding headers. Ensure state-changing methods never accept GET. +Local loopback HTTP can be explicitly supported; nonloopback password login +requires operator-configured HTTPS termination with *trusted-proxy-only* +forwarded-proto validation, or a consciously configured documented insecure +development override with warnings and limited blast radius. A LAN address +is not a trust boundary; no hidden integrated TLS server or proxy. + +### D — Managed transition +Add a checked transition from `admin_initialized` to managed. Require +explicit retirement of shared global legacy key and disabling of +unauthenticated loopback and public private-data exemptions **before** +granting second-user access. Fresh initial single-admin experience remains +a basic login and ordinary dashboard, not a compulsory setup wizard. +Ensure change-password, session logout and re-authentication flows are +usable without frontend dependencies. + +## 4. Ordered work packages + +1. Schema/session repository with expiry/revocation and Argon2 bounds; tests. +2. Strict route classification and role/scope principal extraction; + negative tests across every existing endpoint. +3. Login/logout/change-password pages, cookies, CSRF and transport checks; + browser/HTTP fixture coverage. +4. Managed-mode transition and legacy key retirement + documentation. + +## 5. Failure, cancellation, restart and contention + +Server-side session revocation must be observed for **new** requests +immediately; in-progress HTML render may finish. Expired/disabled accounts +cannot renew sessions. DB failure rejects protected endpoints. Throttle +login attempts and cap Argon2 concurrent workers on SBC hardware without +locking the SQLite gate while hashing. Sessions survive a normal restart +only if deliberately unexpired; password reset/global revocation invalidates +them. Cookies must never be forwarded upstream. + +## 6. Compatibility and migration + +Preserve API bearer semantics for agents, guest health/readiness and +legacy dashboard in explicitly unmigrated deployments. Do not silently +turn public dashboards into write/admin access. `dashboard.public` +in managed mode must not expose user/admin data; define a safe precedence +and a startup diagnostic. CLI admin recovery bypasses HTTP only via +OS-owned local access. + +## 7. Tests and verification + +New `rust/tests/on_prem_sessions.rs` and extended +`server_transport.rs`/`dashboard_filters.rs`/`cli_contract.rs`. +Fixtures: all endpoints by anonymous/member/admin/agent, stale cookie, +stolen/revoked session, password reset, CSRF origin and token missing, +remote HTTP vs trusted TLS proxy vs spoofed headers, limited login +concurrency, first-login forced change, browser-disabled/Javascript-off +login and initial single-admin use. Focused: +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_sessions -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test dashboard_filters -- --test-threads=1 +``` +Run shared matrix and browser parity baseline where dashboard shell changes. + +## 8. Docs, acceptance and stop conditions + +Update `architecture/deep-dive-security.md`, +`architecture/deep-dive-dashboard.md`, `docs/deployment.md` and +`docs/firewall.md`; document operator-managed HTTPS/VPN rather than +promising a built-in remote feature. Accept only when anonymous/agent/member +cannot reach privileged dashboard/admin data, no CSRF bypass exists, +and local admin login/logout/change works. Stop on unresolved trust-proxy +or unknown-route ambiguity. Closure +`plans/closure/on-prem-user-access/002-status.md`: exact route matrix, +browser negative tests, protocol compatibility and login cost, migration +evidence, CI results and unblock audit. + diff --git a/plans/implementation/on-prem-user-access/003-principal-propagation-and-attribution.md b/plans/implementation/on-prem-user-access/003-principal-propagation-and-attribution.md new file mode 100644 index 000000000..655451215 --- /dev/null +++ b/plans/implementation/on-prem-user-access/003-principal-propagation-and-attribution.md @@ -0,0 +1,162 @@ +# On-Prem User Access Milestone 003 — Principal propagation and durable attribution + +Status: implemented + +Repository baseline: `main` at `9a0a903c9944b69a6c1ae9fa969bb8603d5c65d9` (architecture/roadmap authority committed on `plans/on-prem-small-team-user-access` at `81beec4e215e8991880d407ed2752d01ff88d58a`) + +Source roadmap: +- `plans/subsystems/on-prem-user-access-roadmap.md#m003--request-principal-and-durable-user-attribution` + +Long-term requirements: +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADRs: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) + +Primary class: infrastructure + +## 1. Objective and boundary + +Bind downstream user ID to every logical inference execution, durable request +and terminal settlement without changing upstream provider-account identity. +**Hard blocker:** M002 closure and a stable route principal/scope contract. + +## 2. Baseline + +`server/inference.rs` extracts headers but no principal; in +`coordinator/endpoints.rs` `execute_endpoint` parses, resolves aliases +and selects finite/stream. `coordinator/finite.rs::FiniteRequest` and +`coordinator/streaming/types.rs::StreamRequest` carry inference state. +`coordinator/publication.rs::PublicationInput` inserts `requests` +indexed by provider-account `account_id`; `FinalizationIdentity` +persists the selected account/model. `finalization.rs` and +`reconciliation.rs` settle/recover. `server/health.rs::models_api` +and `integration_profile` currently have no downstream user filter. + +### Always-preserved invariants +- A single on-prem instance; TOML/environment retains upstream provider topology/credentials, model routing, and operator-owned settings. SQLite owns only downstream users, sessions, grants, profiles, budget reservations, consumption and audit. +- No SaaS organizations, hosted signup, embedded reverse proxy/VPN, enterprise identity dependency, separate service/database, provider credential web editor, or assertion of legal authority to share personal subscription seats. +- All legacy inference wire paths and return shapes (Chat Completions, Messages, Responses, Responses compact), streaming/terminal semantics, redaction, configuration lifecycle, and operator CLI remain compatible unless an explicit migration contract states otherwise. +- Respect `GenerationLease`, coordinator publication/finalization compensation, one SQLite gate/worker with WAL/NORMAL, `config_reload_policy.rs` as reload authority, and `--no-default-features` compilation. +- In-flight resource ownership remains deterministic. No database locks span provider I/O or expensive password hashing. No raw prompt, token, password, session, provider key, or cache key reaches diagnostics or ordinary request storage. + +### Shared repository verification +Run on implementation and report actual results in the closure record, not the plan: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +``` +For dependency changes also run locked release build, `cargo deny` and +`cargo tree -e features`; for user-facing release-doc edits run +`uv run python scripts/validate_release_docs.py`. Ensure `git diff --check`. + +## 3. Required production changes + +### A — One ingress principal +Create typed `AuthenticatedPrincipal {user_id, credential_id, role/scopes, +auth_epoch}` (or narrower equivalent). Server middleware resolves and +sets it as request extension; do not pass mutable user IDs in untrusted +headers or provider payload. Route handlers consume it and use same +principal for all `/v1/*`, including `/v1/responses/compact`. +Leave generation lease handling untouched and avoid public-request-struct +breakage where compatibility demands private wrappers. + +### B — Request ownership and propagation +Thread principal through the one parsed request endpoint decision into +finite, streaming, compact, internal semantic selector and all retries. +Do not allow selector calls to accidentally acquire a second operator +identity or escape all accounting; explicitly record internal request +parent/child linkage, or define an auditable attribution strategy without +duplicating user logical request counts. In future M004 ensure any +unsupported identity propagation path fails before provider dispatch. +Prefer stable `user_id` across redirects, retries and failovers, never +derived from provider account. + +### C — Durable attribution +Append a migration with nullable `requests.user_id` and credential ID +where genuinely necessary (avoid storing full bearer secret). Add appropriate +indexes for user/recent queries and FK semantics so disabling/removing +accounts preserves historical attribution. New managed requests must always +have non-null user ID; existing rows explicitly legacy/unattributed. +Carry owner identity into `FinalizationIdentity`; idempotent request +republication and terminalization require identity equality before +resuming an existing row, preventing cross-user request-ID collisions. +Retries must not overwrite user identity, even when provider account +changes. Add bounded user-filtered query helpers, not full-table scan or +user dimensions on every existing provider rollup by default. + +### D — Invalidation contract and discovery seam +Resolve current user active status and scope on **new** request. Snapshot +the relevant authorization epoch per request; decide explicitly that +already-dispatched requests may finish after revocation while no new +request can start. Provide typed authority for M004 model-filtered +`/v1/models` and `/api/integrations/v1/profile` with revision/ETag +invalidation when policy changes, without yet implementing model rules. + +## 4. Ordered work packages + +1. Define principal, lifetime and request-extension scope, cross-surface tests. +2. Propagate principal through endpoint/finite/stream/compact/selector + while preserving source-native no-rewrite and existing wire fixtures. +3. Migrate request attribution and publication/finalization invariants, + including crash recovery and replay/duplicate semantics. +4. Add filtered data-access seams, versioned discovery contract and + memory/latency evidence. + +## 5. Failure/cancellation/restart/contention + +Identity persists before dispatch as part of the same publication +transaction and survives cancellation/finalization supervision; do not +misattribute a retried account. A stale/disabled principal at admission +fails closed, but an already-started stream's owner stays stable for +settlement. Old process-generated requests retain lease-scoped immutable +identity on config reload. DB/read outage must not degrade into anonymous +managed inference. No second SQLite writer/connection. + +## 6. Compatibility/migration + +Keep old `requests` rows queryable; never overwrite historical caller +identity as initial admin. Ensure old analytics, provider account view, +`model_routers`, response bodies and client configurations remain +unchanged. Explicitly identify old shared-key inference as a legacy +principal until removed. Schema checksum integrity and backup restore are +mandatory. + +## 7. Focused verification + +New `rust/tests/on_prem_attribution.rs` and extend +`coordinator_publication.rs`, `coordinator_finalization.rs`, +`coordinator_boundaries.rs`, `coordinator_c008.rs`, +`coordinator_c009.rs`, `coordinator_c011.rs`, +`database_compatibility.rs`, `codex_compaction_compat.rs`. +Negative: spoofed downstream user header, cross-user duplicate +`proxy_request_id`, alternate provider retry, stream abrupt EOF, +compact native dispatch, semantic internal selector, password revocation +mid-stream and foreign-key attribution after user disable. +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_attribution -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_publication -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_finalization -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_c009 -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test codex_compaction_compat -- --test-threads=1 +``` +Also run shared suite. + +## 8. Acceptance, docs and closure + +Update `architecture/deep-dive-request-lifecycle.md`, +`architecture/deep-dive-database.md`, +`architecture/deep-dive-observability.md` and protocol docs where needed. +Accept when every managed inference request is stably and durably owned, +including retries, compact and aborted streams, and existing provider +metrics stay stable. Stop if public coordinator ABI would regress or +selection/compensation owner is ambiguous. Closure +`plans/closure/on-prem-user-access/003-status.md` needs per-surface +and fault-injection matrix, migration/legacy fixtures, measured overhead, +commands/results and unblock audit. + diff --git a/plans/implementation/on-prem-user-access/004-policy-profiles-and-model-gates.md b/plans/implementation/on-prem-user-access/004-policy-profiles-and-model-gates.md new file mode 100644 index 000000000..045b586ed --- /dev/null +++ b/plans/implementation/on-prem-user-access/004-policy-profiles-and-model-gates.md @@ -0,0 +1,216 @@ +# On-Prem User Access Milestone 004 — Profile-based model authorization + +Status: implemented + +Repository baseline: `main` at `9a0a903c9944b69a6c1ae9fa969bb8603d5c65d9` (accepted new architecture on `plans/on-prem-small-team-user-access`). + +Source roadmap: `plans/subsystems/on-prem-user-access-roadmap.md`. + +Long-term requirements: +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted). + +Primary class: capability + +## 1. Objective + +Create reusable named local profiles and per-user overrides controlling model +access and default/recommended model selection, with authority evaluated at +the concrete-dispatch boundary. Explicit model denials and global hard limits +cannot be bypassed via provider-qualified names, virtual selectors, compact, +or protocol conversion. **Hard dependency:** accepted M003 closure establishing +stable principal propagation and persistent user identity. No model-access +browser editor or consumption budget is delivered here. + +## 2. Readiness + +M001 and M002 are closed. M003 is conditionally closed with stable typed +principal propagation and durable request ownership. Its unrelated update +unit-test failures and deferred physical resource evidence are recorded in +the M003 closure; they do not weaken M004's auth boundary. +`plans/subsystems/on-prem-user-access-roadmap.md` defines scope and typed +dependency chain. + +## 3. Current implementation evidence + +`rust/src/coordinator/endpoints.rs` calls `resolve_concrete` and +`resolve_virtual` before routing; `rust/src/model_router.rs` and +`rust/src/coordinator/semantic.rs` implement affinity/selector selection; +`rust/src/request/admission.rs` creates canonical request facts, +`rust/src/routing/eligibility.rs` filters upstream candidates, and +`rust/src/server/health.rs` exposes a global `/v1/models`. +`rust/src/config.rs` defines TOML provider/model overrides; it must remain +the operator policy authority and never be reinterpreted as per-user grants. + +## 4. Non-regressing invariants + +- This is one LAN/on-prem instance, not hosted SaaS. The first-admin SBC flow remains canonical; no mandatory external service. +- TOML/environment retain provider topology, provider credentials, outbound routing, model overrides and self-hosted LLM connection details. SQLite holds downstream users, policies, budgets, sessions and usage. Never make the web dashboard a general provider configuration editor. +- Existing Chat Completions/Responses/Messages/compact wire, streaming termination, auth-header compatibility, metadata redaction, recovery, provider scoring and health remain intact. +- Single SQLite gate/worker (WAL/NORMAL), `GenerationLease` and immutable provider topology remain authoritative. No secret/body logging, second retry/finalization owner or unbounded in-memory cache. +- A denied user cannot transform a permitted virtual model or provider qualifier into an unauthorized underlying dispatch. + +## 5. In-scope production changes + +### A — SQLite profile and assignment state + +Add versioned, normalized `policy_profiles`, +`profile_model_rules`, `user_profile_assignments` (or one profile FK +on `users`), and `user_policy_overrides` with creation/update identity, +auditable versions and strict constraints. Profiles control model permissions +(allow/deny), a preferred default model and a reference to quota policy +to be added in M005. Start with **one assigned profile per user**; no +nested profiles, organization trees, groups or role inheritance. Admin +profile preserves historical unrestricted single-admin access unless +configured otherwise. Restricted profiles default-deny unknown models. +A user's explicit allow cannot override global hard deny or inactive +provider. Keep instance/provider configuration in TOML only. + +### B — Effective policy compiler + +Build a typed, bounded effective policy from instance restrictions, user +overrides and assigned profile, with deterministic precedence: +instance hard deny/ceiling always wins; explicit per-user restriction/ +allowance within instance constraints; then profile; then default. +Use immutable cached policy with monotonic version/epoch and atomic +invalidation for **new** requests when admin edits or revokes privileges. +Changes must not require a process restart or TOML rehash; security +cache invalidation must not mistakenly resurrect stale permissions across +generation swap. Model identifiers use canonical IDs plus optional exact +provider-qualified rules. Avoid display-name policy keys and open-ended +wildcards unless a carefully bounded/fail-closed grammar is necessary. +Do not treat catalog absence as permission. + +### C — Authorize all model choices + +At ingress authorize explicitly requested model ID (including virtual +alias), then authorize every resolved *concrete* candidate before upstream +dispatch. A permitted virtual alias cannot gain access to denied concrete +models through stickiness, semantic selection, fallback/repair, or +provider-qualified syntax. Selector's own model requires permission (or an +explicit tightly scoped internal exception with attributable usage and +separate operator approval); reject rather than silently spending denied +models. Retain a single request policy snapshot so concurrent edits do not +cause inconsistent partial decisions inside one request. Native +`/v1/responses/compact` and all Chat/Responses/Messages surfaces share +identical checks; no alternate path around endpoint admission. +Return an unambiguous authorization denial without exposing secret +provider topology, credentials or other users' policies. + +### D — User-specific discovery and integrations + +Filter `GET /v1/models` and +`GET /api/integrations/v1/profile` to principal-allowed models and +aliases. Apply the same conservative capability/limit projection as +current `eggpool-client-config`, not a new client catalog codec. +Include the effective-policy version in profile revision/ETag to invalidate +desktop caches on permission change. A model absent from discovery must +also fail at runtime even when a client sends it manually. Clarify that +a preferred model is a client default/recommendation, never a silent +rewrite of an explicit request. + +## 6. Out of scope + +No token/USD budgets (M005), admin profile editor (M006), upstream credential +policies, per-user local LLM startup, nested groups, model-rewriting by a +policy engine, changes to provider health/fairness, browser auth work. + +## 7. Ordered work packages + +1. Append checksum-verified schema and profile repository APIs. Define + exact precedence with pure property tests. +2. Add effective-policy caching, version invalidation and negative + authorization tests, including disabled-user updates. +3. Insert model gates into the single endpoint/selector lifecycle, retaining + current parsed-body preservation and generation lease. +4. Apply filtered model discovery/integration profile revision, then + add CLI-only operator profile fixture/setup path until M006 adds the UI. + +## 8. Failure, cancellation, restart and contention + +DB/policy compilation failure fails closed for restricted users. Admin +policy edit is atomic: no partial profile visibility; new requests observe +the new policy version once committed. Already-admitted requests retain the +captured version and may complete; document this and ensure a disabled user +cannot start any new work. Selector/retry paths never shed their version. +No SQLite transaction may span a model inference, selector call or stream. +Bound cache count by known local users and active generations; invalidate +on delete/revoke and avoid resurrecting stale grants after restart. + +## 9. Compatibility and migration + +No provider TOML rewrite, no upstream secret move, no public model rename, +no old dashboard analytics regression. Preserve the historical default +single-owner all-model access with explicit admin/default policy. +Historical user records gain one default profile through a deterministic +migration. Expose limited context and capability metadata only after +authorization, including `model/provider` and aliases. + +## 10. Required tests + +Add `rust/tests/on_prem_model_policy.rs` with allowed/denied concrete, +provider-qualified, virtual aliases, semantic selector/repair, sticky +affinity, retry failover, native compact, Messages/Chat/Responses, +model-list/ETag projection and concurrent update/revoke cases. Extend +`rust/tests/model_router.rs`, `routing_domain.rs`, +`codex_responses_compat.rs`, `codex_compaction_compat.rs`, +`operations_o005.rs` and `server_transport.rs`. +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_model_policy -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test model_router -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test codex_compaction_compat -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test operations_o005 -- --test-threads=1 +``` +## 11. Required verification commands + +Run the following after focused tests; preserve exact results for closure: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +git diff --check +``` +If new dependencies/feature flags: `cargo deny --manifest-path rust/Cargo.toml check`, +`cargo tree --manifest-path rust/Cargo.toml -e features`, +`cargo build --manifest-path rust/Cargo.toml --locked --release`. +No test may be declared passed unless actually run. + +## 12. Documentation + +Update `architecture/deep-dive-routing.md`, +`architecture/deep-dive-request-lifecycle.md`, +`architecture/deep-dive-integrations.md`, +`docs/model-routing.md`, `docs/agent-configuration.md` and +`docs/api-reference.md` with exact authorization/discovery semantics. + +## 13. Acceptance criteria + +A restricted member cannot dispatch, discover, or request a forbidden +concrete model through **any** supported API, qualifier, alias, selector +or retry, while an authorized admin retains normal routing and data. +Profile edits apply to new requests without restart and test fixtures prove +immutable request policy ownership and redacted errors. + +## 14. Stop conditions + +Stop if public request/wire contracts would break, model resolution occurs +on an unguarded path, or a policy update would bypass versioning/fencing. +Do not implement budgets or UI to conceal missing enforcement. + +## 15. Closure evidence required + +`plans/closure/on-prem-user-access/004-status.md`: commit IDs, +spec-to-test matrix, all alternate-model bypass negatives, concurrency and +restart observations, discovery/ETag fixtures, migrations and commands +actually run, known residual risks and registry unblock audit. + +## 16. Handoff + +One implementation agent owns the policy/compiler and admission seam. +Keep provider-account health/quotas entirely separate; no generic SaaS +tenant abstraction. Confirm current branch/source before modification. diff --git a/plans/implementation/on-prem-user-access/005-durable-user-budgets.md b/plans/implementation/on-prem-user-access/005-durable-user-budgets.md new file mode 100644 index 000000000..0d0abb73d --- /dev/null +++ b/plans/implementation/on-prem-user-access/005-durable-user-budgets.md @@ -0,0 +1,236 @@ +# On-Prem User Access Milestone 005 — Durable user/model budgets + +Status: implemented + +Repository baseline: `main` at `9a0a903c9944b69a6c1ae9fa969bb8603d5c65d9` (accepted new architecture on `plans/on-prem-small-team-user-access`). + +Source roadmap: `plans/subsystems/on-prem-user-access-roadmap.md`. + +Long-term requirements: +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted). + +Primary class: capability + +## 1. Objective + +Enforce administratively configured per-user and per-model consumption +budgets independently from existing upstream provider quotas. A budget is +checked and durably reserved **before** provider dispatch and safely +settled/compensated across finite/streaming, retries and crash recovery. +Support request count, token, estimated cost and concurrent-request +dimensions within explicit enforceability limits; handle self-hosted +unpriced LLMs meaningfully. **Hard dependency:** accepted M004 policy +authorization closure. No dashboard editor in this milestone. + +## 2. Readiness + +M004 is closed with durable policy profiles, revision invalidation, and +concrete-model checks at coordinator admission. M003 established stable +principal ownership and request attribution. Reusing `QuotaPolicy` provider +scoring without an independent user admission authority remains out of scope. + +## 3. Baseline evidence + +`rust/src/quota/state.rs::AccountQuota` and +`routing/eligibility.rs::LocalQuotaMode` represent provider-account +capacity (default score-only). `coordinator/publication.rs` +creates durable request/attempt/provider reservations; +`coordinator/finalization.rs` settles terminal state; +`coordinator/reconciliation.rs` recovers pending requests; +`rust/assets/db/migrations/0032_usage_rollups.sql` is provider-model- +account aggregated. `db/connection.rs` serializes transactions and +preserves single-gate/WAL constraints. Those mechanisms are references for +ownership, not reusable user budget objects by renaming. + +## 4. Non-regressing invariants + +- This is one LAN/on-prem instance, not hosted SaaS. The first-admin SBC flow remains canonical; no mandatory external service. +- TOML/environment retain provider topology, provider credentials, outbound routing, model overrides and self-hosted LLM connection details. SQLite holds downstream users, policies, budgets, sessions and usage. Never make the web dashboard a general provider configuration editor. +- Existing Chat Completions/Responses/Messages/compact wire, streaming termination, auth-header compatibility, metadata redaction, recovery, provider scoring and health remain intact. +- Single SQLite gate/worker (WAL/NORMAL), `GenerationLease` and immutable provider topology remain authoritative. No secret/body logging, second retry/finalization owner or unbounded in-memory cache. +- A denied user cannot transform a permitted virtual model or provider qualifier into an unauthorized underlying dispatch. + +## 5. Required production changes + +### A — Budget definition and units + +Add bounded SQLite-managed `user_budget_rules` plus optionally +`profile_budget_rules`, per-user and per-model daily/monthly request, +token and microdollar limits, and a small explicit concurrency ceiling. +Choose and document calendar UTC vs rolling windows; prohibit ambiguous +time-zone reset rules, zero/negative limits and overflows. Distinguish +request **logical count** from upstream attempts; retry/failover may +incur actual upstream token/cost use without counting as another logical +request. Support hard global instance ceilings as intersection of limits +and profile/user rules. Store money as integers; differentiate modeled/ +reported/provider-local cost; do not call flat-rate subscription consumption +an actual incremental API charge. + +### B — Atomic pre-dispatch reservation + +Define `user_budget_reservations` with stable logical request/user/ +canonical model ID, applicable policy version and budget window, limits +and projected holds, status, expiry and idempotency token. Use the existing +SQLite transactional single-owner lifecycle to atomically check +`used + live_reserved + request_estimate <= limit` against *every* +applicable aggregate/model window. Concurrent clients cannot both +pass a final remaining slot. In admission, bind the resolved model and +user principal first; no separate model authorization bypass. +Integrate reservation transaction into durable publication or a fully +specified compensation handshake that cannot leave provider submission +without an owned user hold. Never hold locks during network requests. + +### C — Settlement, retries and recovery + +On terminal outcome, settle one logical request exactly once with +observed token/cost totals, release unused estimates and concurrency +slot, and preserve any actual upstream use due to failed attempts. +Avoid retries counting request quota twice or releasing user reservation +after a nonterminal attempt. Account for delayed streams, +partial usage, disconnect, process kill, finalization supervisor retry, +replayed publication and active provider reservation compensation. +Integrate crash/reconciliation cleanup with explicit cutoff and observed +vs unknown-use treatment. A terminal publish failure must not leave a +phantom hard hold indefinitely; recovery must not grant spent budget +prematurely. Use trusted durable state, not volatile dashboard counters, +for enforcement. + +### D — Exactness and unknown costs + +Before dispatch, enforce maximum output tokens/request-size bounds where +protocols support them; if a requested hard bound cannot be guaranteed, +reject or conservatively limit admission rather than claim mathematically +exact post-hoc spend enforcement. If provider prices are absent, +token/request budgets still operate; hard USD budgets fail closed for +unpriced usage unless an admin configures a documented conservative +local pricing estimate. No implicit `$0` cost for self-hosted models. +Report provisional vs provider-reported cost and whether a hard +cost cap can be proved for a particular model. Ensure selector/internal +calls are authorized and charged or explicitly blocked. + +### E — Usage projection + +Add bounded user/model/time usage rollups and authenticated user/admin +read APIs; do **not** inflate `0032_usage_rollups` composite keys +prematurely. Reuse existing retained metadata and cost correction semantics +without retrospectively misbilling user quotas; define what happens on +historical price recompute. Track token, request, cost estimate, and +reservation counters. Queries must use indexed range scans, not unbounded +request history. Include audit on admin quota changes. + +## 6. Out of scope + +No cloud billing service, charged invoices, provider routing-score changes, +automatic model downgrade when a limit hits, complete fiscal cost +reconciliation for personal subscriptions, user budget UI (M006) or +multi-instance quota coordination. + +## 7. Ordered work packages + +1. Specify exact budget windows/units and reservation transition table; + add checked SQL migration/indexes and model/profile rule merge. +2. Implement atomic user-hold admission co-owned by durable publication, + concurrent last-slot negative fixtures and idempotent replay. +3. Integrate finite, streaming, selector and compact settlement with + finalizer/recovery; fault inject at every commit/terminal boundary. +4. Add user rollups and bounded read APIs; perform single-owner baseline + admission/TTFT/SQLite-gate measurements before accepting. + +## 8. Failure, cancellation, restart and contention + +Define explicit transitions `pending -> active -> settled/released`, +with invariant that durable request admission and budget reservation +cannot disagree after recoverable committed stages. Keep an exactly-once +settlement identity and compensation idempotency even on duplicate +callbacks or server shutdown. If DB unavailable while protected budget +is evaluated, fail closed. An already-started stream is charged to its +snapshot user/profile; administrative limit decreases affect new work +immediately and cannot silently underflow existing holds. In-memory +fast-path counters may accelerate reads but cannot replace the durable +transaction at a hard-cap boundary. + +## 9. Compatibility and migration + +Do not re-interpret upstream `accounts`, `reservations` or provider +quota `score_only`. No change to API response streaming semantics, +provider health or model attribution. Existing database usage is historical +legacy/unattributed and must not be silently retroallocated. Backup and +restore conserve remaining budget and the authoritative state of holds. + +## 10. Tests + +Add `rust/tests/on_prem_budgets.rs`. Cover exact simultaneous last-slot +requests, aggregate+model caps, UTC window boundaries, depleted token cap, +inflight concurrency, cost unknown/overflow, private LLM unpriced usage, +pre-dispatch rejection, retry paid upstream attempts, stream cancellation, +EOF without usage, crash after publication commit, crash after provider +response but before finalization, duplicate finalization, DB busy/outage, +policy update midstream, restored database and recovery reconciliation. +Extend `rust/tests/quota.rs`, `coordinator_publication.rs`, +`coordinator_finalization.rs`, `coordinator_c008.rs`, +`coordinator_c009.rs`, `database_compatibility.rs` and +`rust/tests/persistence_projection_architecture.rs`. +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_publication -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_finalization -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test quota -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test database_compatibility -- --test-threads=1 +``` +## 11. Required verification commands + +Run the following after focused tests; preserve exact results for closure: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +git diff --check +``` +If new dependencies/feature flags: `cargo deny --manifest-path rust/Cargo.toml check`, +`cargo tree --manifest-path rust/Cargo.toml -e features`, +`cargo build --manifest-path rust/Cargo.toml --locked --release`. +No test may be declared passed unless actually run. + +## 12. Documentation + +Update `architecture/deep-dive-database.md`, +`architecture/deep-dive-request-lifecycle.md`, +`architecture/deep-dive-routing.md`, +`docs/model-limits.md`, `docs/api-reference.md` and +`docs/backup-restore.md` with exact units and guarantees. +Do not label estimated USD consumption as real billed charges. + +## 13. Acceptance criteria + +No user can over-admit a defined hard concurrent request/token hold in a +reproducible race, cross all allowed model budgets by alias/failover, +or free a hold twice. All recoverable crashes converge with correct +attribution and no leak of another user's usage. Unknown-price and +partial-usage status are explicit rather than successfully counted as +free. Latency/DB costs are measured and documented. + +## 14. Stop conditions + +Stop on unsafe cross-transaction gap between user reservation and provider +publication, inexact quotas presented as exact, unbounded SQLite gate +occupation, or any ambiguity concerning compensation authority. Split +ownership/failure research into corrective ADR/plan rather than widening +scope. + +## 15. Closure evidence required + +`plans/closure/on-prem-user-access/005-status.md`: state transition +matrix, adversarial overlap/crash fixtures, cost-metering examples, +SQL/index tests, backup/restart results, measured publish/settle overhead, +actual commands/results, severity-tagged residuals and registry audit. + +## 16. Handoff + +Single DB remains authoritative. Do not alter existing quota-scoring +semantics to enforce user budgets; keep no-default builds working. diff --git a/plans/implementation/on-prem-user-access/006-web-admin-and-personal-integrations.md b/plans/implementation/on-prem-user-access/006-web-admin-and-personal-integrations.md new file mode 100644 index 000000000..f9ad7f397 --- /dev/null +++ b/plans/implementation/on-prem-user-access/006-web-admin-and-personal-integrations.md @@ -0,0 +1,230 @@ +# On-Prem User Access Milestone 006 — Web administration and personal client integration + +Status: implemented + +Repository baseline: `main` at `9a0a903c9944b69a6c1ae9fa969bb8603d5c65d9` (accepted new architecture on `plans/on-prem-small-team-user-access`). + +Source roadmap: `plans/subsystems/on-prem-user-access-roadmap.md`. + +Long-term requirements: +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted). + +Primary class: capability + +## 1. Objective + +Expose secure small-team administration and member self-service through +the existing embedded Rust dashboard. Admins can create/disable users, +assign profiles, apply overrides and budget rules, inspect aggregate and +per-user usage, revoke tokens, reset passwords, and audit changes. Members +can log in, change passwords, manage their own scoped agent tokens and +view their accessible models/usage. A personal client profile works with +OpenCode, Codex and current Eggpool remote desktop helper. **Hard +dependency:** accepted M005 closure (all lower layers/quotas functional). + +## 2. Readiness and M005 conditions + +Admin mutation endpoints rely on M002 sessions/CSRF, M003 principal, M004 +authorization and M005 budget persistence. M005 is conditionally closed: +budget CRUD and bounded usage reads are available, while output-token and +unpriced-USD limits remain estimates and global instance ceilings are absent. +The M006 UI must label these dimensions accurately; do not present the current +estimated limits as exact hard caps. A static dashboard mock is not accepted +capability evidence. + +## 3. Baseline + +`rust/src/server/dashboard/routes.rs` serves read-only pages, +`dashboard/render/layout.rs` renders navigation and layout, +`dashboard/api.rs` reads aggregate stats, and bundled +`rust/assets/dashboard/static/dashboard.css` and `dashboard.js` +own frontend assets. `rust/src/operations/integrations.rs` generates +Codex/OpenCode snippets and remote profiles using one resolved server +key, while `rust/crates/eggpool-client-config/` and +`rust/crates/eggpool-connect/` own portable projection and desktop +configuration. Current `docs/agent-configuration.md` documents +`EGGPOOL_API_KEY` global key handling. + +## 4. Non-regressing invariants + +- This is one LAN/on-prem instance, not hosted SaaS. The first-admin SBC flow remains canonical; no mandatory external service. +- TOML/environment retain provider topology, provider credentials, outbound routing, model overrides and self-hosted LLM connection details. SQLite holds downstream users, policies, budgets, sessions and usage. Never make the web dashboard a general provider configuration editor. +- Existing Chat Completions/Responses/Messages/compact wire, streaming termination, auth-header compatibility, metadata redaction, recovery, provider scoring and health remain intact. +- Single SQLite gate/worker (WAL/NORMAL), `GenerationLease` and immutable provider topology remain authoritative. No secret/body logging, second retry/finalization owner or unbounded in-memory cache. +- A denied user cannot transform a permitted virtual model or provider qualifier into an unauthorized underlying dispatch. + +## 5. Required production changes + +### A — Authorized administrator endpoints + +Add minimal versioned web/API handlers for user create, disable/reactivate, +admin-initiated password reset, profile CRUD, profile assignment, user-level +model/budget overrides, token revocation and audit log browsing. Return +generic redacted failures; enforce actual DB-based admin role and CSRF on +every mutation, validate bounded form bodies and constrained input. Prevent +deleting/disabling the last usable admin, self-demotion bypass and stale +write overwrites (profile edit version or compare-and-swap). All policy +writes atomically bump effective-policy revision/invalidation. No +reconfiguration of provider TOML or upstream secrets via browser. + +### B — Embedded web UI and member scope + +Extend existing Rust-rendered pages with `Users`, `Profiles`, +`Usage` and `My Account` tabs, retaining Halloy themes, current +layout, keyboard focus/assistive labels, ordinary form semantics, +graceful no-JS operation where practical and the historical overview/ +routing/latency/account pages. Admins see user-status, token counts, +budget remaining and safe aggregate operational diagnostics. Members +see only their own identity, usage, remaining quotas, accessible +models and a password/token change interface; no exposure of other +users' usernames, emails, provider-account lists or diagnostic +summaries. Token plaintext appears once on creation and is omitted +from HTML on refresh/history; warn and permit safe regeneration if lost. + +### C — Personal desktop-client configuration + +`/v1/models` and `/api/integrations/v1/profile` are filtered by +the authenticated token owner. Adapt `configsetup`, `configremote`, +`eggpool-connect` flows to prompt for or consume the user's personal +token from `EGGPOOL_API_KEY`, an interactive terminal or +`--api-key-stdin`; never embed a global admin key in a shared +bootstrap URL, URL query, checked-in file or output by default. Keep +the existing `epc1` profile/revision contract or introduce explicit +backward-compatible version negotiation only if necessary, with +conservative model metadata and scopes. A profile/policy update +invalidates ETag/client catalogue; revoked credentials cannot refresh. +Do not silently allow client helpers to change server policy. + +### D — Onboarding and operator ergonomics + +A fresh administrator logs in with the bootstrap credentials and can +continue one-user work without seeing forced team setup; a clear +`Add user` step activates managed mode only after legacy key retirement. +Create users with one-time temporary passwords or local password-reset +mechanism requiring first login change, never email-delivered secrets or +`admin/admin`. Provide a `users list` local CLI summary but keep +primary team management in the web admin. Document no native remote +hosting, terms-sensitive pooling, VPN/reverse-proxy instructions and +self-hosted model examples controlled in TOML. + +## 6. Out of scope + +No provider TOML web editor, central SSO, registration/sign-up page, +email invitation or automated password email, tenant hierarchy, +billing/invoices, native TLS certificate manager or external frontend. + +## 7. Ordered work packages + +1. Define admin/member HTTP/CSRF, authorization and DTO contract with + an exhaustive route-method matrix; implement thin DB-backed handlers. +2. Add profile and user forms using current dashboard render/asset + system; ensure accessibility and legacy dashboard parity. +3. Integrate filtered per-user agent profiles and token delivery into + `configremote`/`eggpool-connect`, preserve atomic backup/rollback. +4. Add testable two-user onboarding journey, error handling, docs + and migration guidance with perf/size spot checks. + +## 8. Failure, cancellation, restart and contention + +All admin mutation operations must be atomic/consistent; password reset +invalidates existing sessions and follows a documented token policy. +Concurrent profile edits have one winner or structured conflict; users +never briefly gain more access than the committed policy. A user's +revoked API credential cannot reuse cached discovery or existing sessions +to start new inference. A rotating browser session retains correct CSRF +binding. A historical request remains visible to an admin even when +the owner is disabled; member queries cannot infer other users' data. +No blocking DB transaction spans page render/network calls. + +## 9. Compatibility and migration + +`/v1/models` keeps OpenAI schema, only filters visibility. +`GET /api/integrations/v1/profile` retains revision/ETag and +conservative limits, but scopes model selection to token owner. +All existing `epc1`, Codex TOML and OpenCode JSONC paths must +remain supported, including rollback and remove of managed entries; +replacing legacy server key may require user rerunning credential +bootstrap, not full client reinstall. Legacy single-owner dashboard +pages and PNG/JS/theme parity must not regress. + +## 10. Tests and qualification + +Add `rust/tests/on_prem_admin_ui.rs` and +`rust/tests/on_prem_client_profiles.rs`; extend +`rust/tests/dashboard_filters.rs`, `server_transport.rs`, +`operations_o005.rs`, `cli_contract.rs`, +`rust/tests/runtime_lifecycle_r005.rs` and +`rust/crates/eggpool-connect/tests/connect_transaction.rs`. +Test login-to-create-user flow, direct endpoint privilege escalation, +CSRF, password first-change, profile grant/deny, stale ETag, +two concurrent profile editors, last-admin protection, member's +view of a forbidden user, token reveal-once/revoke/refresh, +Codex/OpenCode live model subset, compact Responses client use, +connection helper rollback, no-credential logs and no-JS fallback. +Repeat dashboard browser parity snapshots/interaction harness where +shell/nav/layout differs. +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_admin_ui -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_client_profiles -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test dashboard_filters -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test operations_o005 -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml -p eggpool-connect -- --test-threads=1 +cargo test --manifest-path rust/crates/eggpool-client-config/Cargo.toml -- --test-threads=1 +``` +## 11. Required verification commands + +Run the following after focused tests; preserve exact results for closure: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +git diff --check +``` +If new dependencies/feature flags: `cargo deny --manifest-path rust/Cargo.toml check`, +`cargo tree --manifest-path rust/Cargo.toml -e features`, +`cargo build --manifest-path rust/Cargo.toml --locked --release`. +No test may be declared passed unless actually run. + +## 12. Documentation + +Update `architecture/deep-dive-dashboard.md`, +`architecture/deep-dive-integrations.md`, +`architecture/deep-dive-security.md`, +`docs/agent-configuration.md`, `docs/cli-reference.md`, +`docs/deployment.md`, `docs/api-reference.md` and dashboard +qualification manifests; keep operational docs accurate about +plaintext LAN password hazards. + +## 13. Acceptance criteria + +A one-user admin can log in and continue use without creating another +user. Admin can create two members with distinct roles/profiles/budgets, +and each agent sees only allowed models, receives correctly scoped 401/403 +on revocation/denial, and uses Codex/OpenCode without server-key sharing. +Non-admin cannot mutate any user or inspect others' usage. Existing +dashboard visual/functional parity and theme switching remain intact. + +## 14. Stop conditions + +Stop on incomplete lower-layer auth, CSRF, policy revocation, budget or +private-data query fencing. Never ship web forms as a facade over +unsafe admin operations. Do not convert this into an SPA/framework rewrite. + +## 15. Closure evidence required + +`plans/closure/on-prem-user-access/006-status.md`: exact screen/route +authorization matrix, two-user full journey, client integration and ETag +fixtures, browser parity/CSRF tests, security review, footprint and +commands actually run, known gaps and unblock audit. + +## 16. Handoff + +Keep the Rust dashboard read/render system and portable +`eggpool-client-config` ownership; server remains the source of truth +for identity and policy, provider TOML remains operator-owned. diff --git a/plans/implementation/on-prem-user-access/007-small-team-qualification-and-migration.md b/plans/implementation/on-prem-user-access/007-small-team-qualification-and-migration.md new file mode 100644 index 000000000..bd91708dc --- /dev/null +++ b/plans/implementation/on-prem-user-access/007-small-team-qualification-and-migration.md @@ -0,0 +1,237 @@ +# On-Prem User Access Milestone 007 — Small-team qualification, migration and release hardening + +Status: implemented + +Repository baseline: `main` at `9a0a903c9944b69a6c1ae9fa969bb8603d5c65d9` (accepted new architecture on `plans/on-prem-small-team-user-access`). + +Source roadmap: `plans/subsystems/on-prem-user-access-roadmap.md`. + +Long-term requirements: +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted). + +Primary class: polish + +## 1. Objective + +Qualify secure on-prem user operation on real SBC and reasonable local +hardware, including fresh/legacy/headless upgrades, read/write backup +restore, single-user non-regression and small-team controlled load with up +to 49 **named users**. Produce a supportable resource-sizing/deployment +guide rather than an invented 50-concurrent-request SLA. +**Hard dependency:** accepted M006 closure. Real target hardware is an +operational evidence requirement, not a paper substitute. + +## 2. Readiness and remaining qualification gate + +M006 is conditionally closed with the admin/member route journey and +user-scoped client behavior implemented. M006's dashboard parity and M005's +estimate/recovery findings are carried into this qualification pass. Physical +SBC capacity evidence is still unavailable on the current macOS x86_64 host; +run reproducible synthetic/local checks now, but make no SBC or 49-named-user +capacity claim without explicitly provisioned target hardware. CI and hosted +ARM timings cannot replace physical Pi/Le Potato measurements. + +## 3. Baseline evidence + +`scripts/qualification_sbc.py` is the physical-target runner, +`scripts/qualification_stability.py` checks long-lived load, +`rust/tests/server_transport.rs` and `runtime_lifecycle_r009.rs` +test serving lifecycle, `tests/tooling/test_installer.py` and release +qualification tools validate install ownership. `docs/raspberry-pi.md`, +`docs/deployment.md`, `docs/backup-restore.md`, +`docs/agent-configuration.md` and +`architecture/deep-dive-database.md` define existing operational +expectations; preserve their current source-backed claims. + +## 4. Non-regressing invariants + +- This is one LAN/on-prem instance, not hosted SaaS. The first-admin SBC flow remains canonical; no mandatory external service. +- TOML/environment retain provider topology, provider credentials, outbound routing, model overrides and self-hosted LLM connection details. SQLite holds downstream users, policies, budgets, sessions and usage. Never make the web dashboard a general provider configuration editor. +- Existing Chat Completions/Responses/Messages/compact wire, streaming termination, auth-header compatibility, metadata redaction, recovery, provider scoring and health remain intact. +- Single SQLite gate/worker (WAL/NORMAL), `GenerationLease` and immutable provider topology remain authoritative. No secret/body logging, second retry/finalization owner or unbounded in-memory cache. +- A denied user cannot transform a permitted virtual model or provider qualifier into an unauthorized underlying dispatch. + +## 5. Required qualification scope + +### A — Fresh and migration matrix + +Use disposable per-test `HOME`, config, runtime dir and DB. Exercise +binary-first quick-install and `eggpool onboard` in single-owner local +mode; noninteractive stdin-based bootstrap; restored older schema54 +database; upgraded legacy `server.api_key`; systemd owned files; local +password reset; single-user normal client operations; adding a second +user; retiring a shared key without accidental privilege extension; +restarting/reloading with changed TOML; backup/restore across versions; +partial/busy database; corrupted/missing state fail-closed; and rollback +instructions without implying older binaries can read newer SQLite. +No real credentials used. Keep paths shell-safe and ensure installed +`eggpool` command remains immediately usable per existing installer +closure. + +### B — Representative capacity profiles + +Measure single-user baseline with current release and new auth build on +the same hardware/config, then 1, 5, 10, 25, 49 locally created identities +with independently scoped credentials and profiles. For each named-user +count run bounded concurrency levels (for example 1, 2, 4, 8, 16 +simultaneous requests as hardware allows) with explicit provider stub +latency, body size, stream mix and quota conditions; do not equate +named-user count with simultaneous active completions. Use reproducible +loopback fake providers, local/self-hosted model where available, and +fixed request corpora, not unbounded paid live tokens. Capture idle and +peak RSS, binary size, CPU, login Argon2 hash duration and memory +contention, auth cache hit/miss, SQLite gate wait, transaction duration, +WAL growth/checkpoint behavior, request p50/p95/p99 and TTFT/throughput. +Separate provider latency from proxy overhead. Compare Le Potato/Pi 5 +when available plus ordinary Linux/macOS host; do not claim a target +without physical evidence. + +### C — Security/correctness fault campaign + +Re-run fresh/legacy shared token bypass matrix, member/admin/anon role +tests, first-login and CSRF, spoofed trusted proxy headers, unpriced local +model caps, selector/virtual-model bypass, quota simultaneous last slot, +retry/stream aborts, policy revoke midstream, process kill during +reserve/publication/finalization, power-loss/restart simulation, +DB corruption/read-only, and user-specific model discovery. +Ensure prompt/body/key privacy across diagnostics, traces, +`usage_rollups`, audit, backups and logs. Confirm data retention and +account deletion strategy preserves audit/usage without leaking +member-private details. + +### D — Deployment/release documentation + +Publish honest sizing and hardening table (LAN HTTP threat model, +trusted TLS proxy/VPN, firewall, operator's upstream terms, no SaaS +registration, private self-hosted upstream support, password/token +rotation, multi-user upgrade steps) with tested commands. Add +scripted regression guards to CI where reproducible and use physical +qualification artifacts for hardware claims. Avoid introducing remote +access feature or native HTTPS server. + +## 6. Out of scope + +No cloud load service, hosted billing or metering, cluster load sharing, +Kubernetes operator, 50 simultaneous generations guarantee, total +provider costs being inferred from flat-rate plans, WAN setup or +security compliance certification. + +## 7. Ordered work packages + +1. Freeze protocols/DB/CLI/browser compatibility matrix and test corpus, + establish physical target baseline and acceptance gate thresholds. +2. Execute fresh/legacy/headless upgrade/recovery/backup and negative + authentication regression harness. Repair any found bug through new + narrowly scoped corrective plans when cross-subsystem. +3. Run paired pre-/post-change SBC and ordinary-host measured load + at each named-user level with bounded concurrency; record raw/summary + evidence and repeat outliers. +4. Reconcile docs, qualification scripts, release/security instructions, + migrations and planning status; mark closure only with evidence. + +## 8. Failure, cancellation, restart and contention + +Any quota/policy bypass, cross-user data leak, lost-admin unrecoverable +state, or silent downgrade to keyless auth is release-blocking. +Missing hardware evidence leaves physical-capacity claims unqualified. +No config/database mutation of contributor's real deployment, no real +upstream account tokens in qualification artifacts. Shutdown and reload +must preserve recoverable user reservation and attribute active requests +correctly; on error, fail closed for protected endpoints. Detect +stale-SQLite/backup scripts and migration checksum drift. + +## 9. Compatibility/migration + +Backups include identity, credential hashes, session state and policy +while provider secrets remain config/env-owned; restore must preserve +token invalidation and stable user IDs. Old-client `EGGPOOL_API_KEY` +environment semantics continue with personal tokens. Migrations are +forward append-only and older executables may require restore of a +matching pre-upgrade backup to roll back. Maintain dashboard parity, +`newkey/getkey` legacy transition documentation and installer provenance. + +## 10. Focused validation + +Add tooling coverage (for example +`tests/tooling/test_on_prem_user_qualification.py`) with entirely +synthetic users/models/traffic; extend +`tests/tooling/test_qualification_sbc.py`, +`test_qualification_stability.py` if present or add a scoped fixture, +`test_installer.py`, `test_service_transition.py`, and +`rust/tests/database_compatibility.rs`, +`runtime_lifecycle_r009.rs`, `coordinator_finalization.rs`. +```bash +cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test database_compatibility -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test runtime_lifecycle_r009 -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_finalization -- --test-threads=1 +uv run pytest tests/tooling/ -q --tb=short --maxfail=1 +uv run python scripts/validate_release_docs.py +``` +## 11. Required verification commands + +Run the following after focused tests; preserve exact results for closure: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +git diff --check +``` +If new dependencies/feature flags: `cargo deny --manifest-path rust/Cargo.toml check`, +`cargo tree --manifest-path rust/Cargo.toml -e features`, +`cargo build --manifest-path rust/Cargo.toml --locked --release`. +No test may be declared passed unless actually run. + +CI is necessary but physical SBC results and resource observations +are additional acceptance evidence. + +## 12. Documentation + +Update `README.md`, `docs/raspberry-pi.md`, +`docs/deployment.md`, `docs/backup-restore.md`, +`docs/firewall.md`, `docs/agent-configuration.md`, +`docs/cli-reference.md`, `architecture/overview.md` and +deployment/security deep dives only to reflect verified behavior. +Use model/provider compatibility-specific caveats where needed. + +## 13. Acceptance criteria + +A fresh single-owner install is straightforward, an existing owner +upgrades with no data loss or surprise lockout, two users exhibit +independent permissions/quotas and correctly scoped client catalogs, +and all negative/security/restart/backup tests pass. Physical +measurements establish feasible **named-account** scaling and practical +concurrency boundaries with quantified regression against single-user +reference. Published documentation never calls Eggpool SaaS or +enterprise identity platform and never claims 50 simultaneous LLM +requests without evidence. + +## 14. Stop conditions + +Stop on uncorrected high-severity privacy/auth/quota defect; absent +representative target hardware; unknown backup/migration guarantees; +uncontrolled resource growth; unrelated provider transport regression +or license-sensitive default sharing claim. Record blocker or issue a +bounded follow-up, never waive security with an unsubstantiated metric. + +## 15. Closure evidence required + +`plans/closure/on-prem-user-access/007-status.md`: complete artifact +manifest, hardware OS/RAM/disk/build hashes, named users versus actual +concurrency, auth-cost measurement, TTFT/request/SQLite/WAL deltas, +all migration/restart/backup/security checks, docs diffs, actual CI +and serial test outcomes, severity findings, final capacity statement, +and downstream unblock/registry audit. Only this evidence can +qualify the 49-named-user operating target. + +## 16. Handoff + +Run physical tests only on explicitly provisioned isolated SBC and +ordinary target hosts; no real upstream credentials. Preserve stable +release bounds and existing routing/accounting semantics. diff --git a/plans/implementation/on-prem-user-access/008-hard-token-cost-admission-corrective.md b/plans/implementation/on-prem-user-access/008-hard-token-cost-admission-corrective.md new file mode 100644 index 000000000..93674fc47 --- /dev/null +++ b/plans/implementation/on-prem-user-access/008-hard-token-cost-admission-corrective.md @@ -0,0 +1,238 @@ +# On-Prem User Access M008 — Enforceable token and cost admission + +Status: implemented + +Repository baseline: `c18dcc2051ae4987a2cb95a7cd046b8aff2095ff` on `plans/on-prem-small-team-user-access` (PR #7), schema 60, M001–M007 closure records as of 2026-10-13. + +Source roadmap: +- `plans/subsystems/on-prem-user-access-roadmap.md#m008--enforceable-output-token-and-price-bound-admission-corrective` + +Long-term references: +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) + +Primary class: capability + +## 1. Objective + +Enforceable token and cost admission. This is a **new corrective** to the conditionally closed M005/M006/M007 records, not a rewrite of historical closure. + +## 2. Readiness and dependencies + +No hard predecessor beyond the accepted M004 model-policy contract and the M005 +durable reservation/settlement interface. Those are present; this plan +is ready. The protocol-level capability and pricing contract must be +settled **before** an irreversible change to admitted wire behavior. + +## 3. Evidence and why prior qualification missed it + +`plans/closure/on-prem-user-access/005-status.md` reports: +(1) output tokens are reserved from a request-size estimate, not a proven +protocol/model maximum; (2) unpriced models may consume a generic fallback +USD estimate, which cannot establish a hard cap. Existing concurrent +last-slot tests prove atomic **estimated** reservations, not an upper bound +on actual provider-reported or billable tokens/cost. The M006 HTML currently +labels these as estimates. In `rust/src/user_budget.rs`, +`check_initial` accepts `tokens`/`microdollars` estimates; +`rules_exceeded` compares them with durable used and active holds; +`record_attempt_and_maybe_settle` records observed/fallback usage. +The `rust/src/coordinator/endpoints.rs`, `finite.rs`, +`streaming/`, `wire/` and `rust/src/config.rs` (pricing fallback) +are relevant ownership boundaries. + +## 4. Non-regressing invariants + +- Canonical use is **one administrator on one local/SBC instance**, optional <50 named users. No SaaS, organization tenancy, billing platform, centralized IAM, new database writer/service, built-in WAN/VPN/reverse proxy, or mandatory cloud calls. +- Upstream provider topology, API keys, routing and self-hosted model endpoints remain in TOML/environment; SQLite owns users, personal credentials, profiles, policy, downstream budgets, accounting and audit. Do not place upstream keys in user tables or web administration. +- Maintain generation leases, existing coordinator publication/finalization, a **single SQLite connection/gate/worker with WAL/NORMAL**, canonical wire and transparent streaming, bounded hot-path memory and provider fairness. Do not turn downstream user budget refusal into upstream provider unhealthiness. +- Preserve published OpenAI Chat/Responses/compact and Anthropic Messages behavior, permitted client auth headers, existing client profiles, dashboard themes and local legacy upgrade behavior. +- Keep identity and cost metadata redacted, prohibit secret/password/bearer leakage in logs, HTML snapshots or qualification artifacts, and fail closed rather than silently weaken configured security limits. + +## 5. Scope and required production changes + +### A — Define enforceable versus measured limits + +Introduce explicit rule enforcement semantics: +- `hard`: never admit without a demonstrably conservative upper bound for + **every chargeable execution** (input, output, internal selector work, + bounded retries/failovers) and all simultaneously enforced windows. +- `estimated`: permits explicitly approximate token or USD budgets; + must never be advertised as hard, must be auditable at the API/CLI/UI. + +Request-count and concurrency limits can remain strictly enforceable; +do not broaden their semantics. Distinguish provider-reported tokens, +local token estimates, provider-reported price, locally configured tariff, +reservation fallback and unknowns in durable usage and operator projections. +Do not infer exact monetary spend from consumer subscription flat-rate quotas. +Decide policy for retry costs and selector calls explicitly: either reserve +their worst-case bounded additional spend up front and extend transactionally +before each attempt, or deny further submission when capacity cannot be +reserved. Do not suppress internal usage. + +### B — Protocol/model admissible maxima + +Implement a typed, per-surface bounded output-capability contract for Chat +(`max_tokens`/supported variants), Responses (including native compact and +`max_output_tokens`), and Anthropic Messages (`max_tokens`). Never +silently drop/mutate an explicit caller field in a native passthrough mode. +For a configured hard cap, either: +1. prove a safe worst-case bound on billable input+output across the admitted + context, supported models and all permitted attempts; constrain output + explicitly using the caller's compatible bound; or +2. reject the **request** with a stable, sanitized and machine-readable + "not enforceable" response **before provider dispatch**. + +Provider tokenizer discrepancies, prompt/tool overhead, native unknown fields +and post-hoc usage all defeat exact counting unless a formally conservative +bound exists. Do not present a field-level `max_output_tokens` alone as a +guarantee of total chargeable tokens. Preserve user-requested output bounds +and cross-surface adaptation notices. + +### C — Pricing trust boundary + +Hard USD policies require a reviewed operator-specified conservative +tariff/upper-bound price **in provider TOML** (including self-hosted LLMs), +or a verified provider-specific bound that cannot increase unnoticed during +the request. Do not accept `pricing.fallback=generic_estimate` as a hard +price. For no price or unsupported/dynamic billing, fail closed for hard +USD requests; estimated USD policies may continue with visible uncertainty. +Handle integer overflow, zero-cost versus unknown-price, cache changes, and +unit precision. No online price discovery requirement. + +### D — Transaction integration and UI truthful projection + +Wire admission type/estimate/worst-case permit into the existing +`coordinator/publication.rs` per-logical-request transaction, +`coordinator/semantic.rs` selector path, and retry pre-submission +`user_budget::check_retry`. Prevent budget-check-then-dispatch gaps or +tokens consumed after a failed extension. Maintain finalizer idempotence; +do not hide observed overruns if a provider violates limits. +Expose `hard`/`estimated` and `not_enforceable` consistently in +`/api/user-budget`, `rust/src/server/dashboard/admin.rs`, +CLI `users budget`, and docs without renaming old keys gratuitously. +Provide a compatibility migration/default: existing token/USD rules +remain explicitly `estimated` until the operator chooses hard mode; +**never** silently upgrade them to a falsely enforced guarantee. + +## 6. Explicit non-goals + +No built-in provider billing reconciliation, guaranteed spend prediction on +unpriced/dynamic models, new provider metadata service, routing based on +budget/cost, additional SQLite writers, or unrelated upstream quota changes. +If the supported wire cannot prove a bound, make hard mode unavailable rather +than modifying the provider's responses. + +## 7. Ordered work packages + +1. Document authoritative limit units/scope, price provenance, compatibility + defaults and the immutable admission maximum/unknown state contract. +2. Add append-only migration only if new durable enforcement mode or provenance + is required (version >60, manifest/checksums), plus parser/API roundtrips. +3. Implement protocol/model bound validation and retry/selector reservation + with proof-or-reject behavior before network I/O. +4. Preserve settlement/analytics and extend accurate hard-vs-estimated UI + presentation, docs and negative tests. + +## 8. Failure, cancellation, restart and contention + +No database transaction crosses provider I/O or expensive tokenization. +A hard-mode rule with missing tokenizer upper bound, unknown tariff or +non-conservative retry/selector estimate must deny admission without starting +an upstream attempt. A rule edit is atomic; in-flight requests retain the +authenticated immutable policy version and their durable held amount, and +new requests see the new enforcement policy. On transport EOF, cancellation, +ambiguous usage, or duplicate finalization, record lower-bound/unknown +usage honestly and preserve holds until an authorized bounded recovery path +can safely settle. DB failures fail closed. + +## 9. Compatibility and migration + +Existing M005 user-scoped stored rules, provider `accounts`, historic +consumption and source API schemas remain readable. Additive schema only; +old rows explicitly map to an estimated mode and can be upgraded by an admin +with an explicit operation. Preserve unchanged native Chat/Responses/Messages +bytes where possible and never inject output caps into an unsupported surface. +Self-hosted LLM pricing belongs only to operator TOML, not user DB. + +## 10. Focused and adversarial tests + +Extend `rust/tests/on_prem_budgets.rs`, +`coordinator_publication.rs`, `coordinator_finalization.rs`, +`coordinator_c011.rs`, `on_prem_model_policy.rs`, +`codex_compaction_compat.rs`, `wire_runtime.rs`, +`database_compatibility.rs`, `on_prem_admin_ui.rs`. +Cover Chat/Responses/compact/Messages, allowed/denied native fields, +bounded/unbounded output, omitted tokenizer, provider-reported use above +estimate, retry to different model/provider, selector child spend, +unpriced/private LLM, dynamic price, integer overflow, no-cost token-only +policy, concurrent last-slot, and explicit estimated-mode compatibility. +Assert **zero upstream submissions** on unprovable hard-bound cases. +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_publication -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_finalization -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test codex_compaction_compat -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test database_compatibility -- --test-threads=1 +``` +## 11. Required verification commands + +Run applicable focused targets in §10, then the full default/no-default gates: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +git diff --check +``` +Run `cargo deny --manifest-path rust/Cargo.toml check`, the feature tree, +and `cargo build --manifest-path rust/Cargo.toml --locked --release` when +Cargo dependencies/features change. Use the repository's tooling/test commands +when scripts are changed. Closure records must distinguish local vs hosted +vs physical hardware results; never claim unrun tests as passed. +## 12. Documentation updates + +Update `architecture/deep-dive-request-lifecycle.md`, +`architecture/deep-dive-database.md`, `docs/model-limits.md`, +`docs/api-reference.md`, `docs/cli-reference.md` and +`docs/deployment.md` with hard/estimated semantics and hard-mode +proof-or-reject. Keep examples with owned local models. + +## 13. Acceptance criteria + +Every admitted request under a **hard** rule carries a documented +conservative upper bound covering every possible accounted attempt and +selector, or is denied **without upstream I/O**. Under unknown-price or +unbounded-output cases no false hard USD/token promise can be configured. +Existing estimated rules continue to behave accurately as estimates; +previous exact request/concurrency limits still hold. UI, API and +persistence agree on enforcement mode; all supported wire formats +retain non-budget behavior. + +## 14. Stop conditions + +Stop if a strict upper bound is unprovable for a required model/wire: +record the unsupported strict mode rather than claiming it. Stop on +cross-attempt double spend, a publisher/finalizer ownership gap, +accidental provider TOML-to-user DB migration, any native protocol rewrite +with lost user intent, or a different SQLite writer design. + +## 15. Closure evidence required + +New `plans/closure/on-prem-user-access/008-status.md` must include: +formal accounting/maxima assumptions and explicit non-support cases; +admission/selector/retry and unknown-price negative matrix; migration +compatibility; source and test revisions; exact commands/CI outcomes; +security/reservation/finalization review and severity-tagged residuals. +Do **not** retrospectively edit `005-status.md`. + +## 16. Handoff + +One coherent budget-admission/wire interface pass; `server/*` stays +thin and coordinator remains the single ingress execution owner. Physical +latency/sizing and global hard ceilings are later separate milestones. diff --git a/plans/implementation/on-prem-user-access/009-global-budget-intersection-corrective.md b/plans/implementation/on-prem-user-access/009-global-budget-intersection-corrective.md new file mode 100644 index 000000000..7aac21c45 --- /dev/null +++ b/plans/implementation/on-prem-user-access/009-global-budget-intersection-corrective.md @@ -0,0 +1,192 @@ +# On-Prem User Access M009 — Global-instance budget ceiling and intersection + +Status: implemented + +Repository baseline: `c18dcc2051ae4987a2cb95a7cd046b8aff2095ff` on `plans/on-prem-small-team-user-access` (PR #7), schema 60, M001–M007 closure records as of 2026-10-13. + +Source roadmap: +- `plans/subsystems/on-prem-user-access-roadmap.md#m009--instance-global-budget-and-scope-intersection-corrective` + +Long-term references: +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) + +Primary class: capability + +## 1. Objective + +Global-instance budget ceiling and intersection. This is a **new corrective** to the conditionally closed M005/M006/M007 records, not a rewrite of historical closure. + +## 2. Readiness and dependencies + +**Hard:** accepted M008 closure. M005's durable accounting is present, +but global-instance scope must consume the final M008 enforcement-mode and +conservative reservation contract before implementation. No physical +hardware prerequisite; SQL contention qualifications are local first. + +## 3. Evidence and why prior tests missed it + +`plans/closure/on-prem-user-access/005-status.md` explicitly reports no +global-instance budget scope. Current +`rust/src/user_budget.rs::rules_exceeded` queries +`user_budget_rules WHERE user_id=?` and aggregates a user's usage +and active reservations. Those last-slot tests exercise **one user**, so +they do not prove a shared ceiling across two separately authenticated +accounts. Neither the prior M005 schema nor the M006 dashboard UI owns +an all-users hard-cap bound. + +## 4. Non-regressing invariants + +- Canonical use is **one administrator on one local/SBC instance**, optional <50 named users. No SaaS, organization tenancy, billing platform, centralized IAM, new database writer/service, built-in WAN/VPN/reverse proxy, or mandatory cloud calls. +- Upstream provider topology, API keys, routing and self-hosted model endpoints remain in TOML/environment; SQLite owns users, personal credentials, profiles, policy, downstream budgets, accounting and audit. Do not place upstream keys in user tables or web administration. +- Maintain generation leases, existing coordinator publication/finalization, a **single SQLite connection/gate/worker with WAL/NORMAL**, canonical wire and transparent streaming, bounded hot-path memory and provider fairness. Do not turn downstream user budget refusal into upstream provider unhealthiness. +- Preserve published OpenAI Chat/Responses/compact and Anthropic Messages behavior, permitted client auth headers, existing client profiles, dashboard themes and local legacy upgrade behavior. +- Keep identity and cost metadata redacted, prohibit secret/password/bearer leakage in logs, HTML snapshots or qualification artifacts, and fail closed rather than silently weaken configured security limits. + +## 5. Required production changes + +### A — One instance-owned budget namespace + +Append a checksum-pinned migration (>60) representing exactly one +**instance-wide** daily/monthly request, token, cost and active-concurrency +rule scope, plus optional exact model keys. Do not introduce an +organization/tenant identifier or second database. Define intersection: +an admitted call must satisfy all applicable instance-global, instance- +model, user-global and user-model rules. Retain M008 `hard` vs +`estimated` semantics per dimension. Absent instance rules mean no +additional cap. Reject malformed/overflowed rules. Ownership stays with +the local admin; ordinary member/API tokens never mutate instance rules. + +### B — Atomic combined reservation + +Check usage **plus active holds** for every matching instance/user/model +window and insert the hold under the same SQLite transaction that +publishes the logical request/provider reservation. Concurrent requests +from different users must not both take a final global slot. Count each +logical request once even if provider failover, internal selector attempt +and stream retry occur; charge attempt costs/tokens correctly per M008. +Global held and settled counters cannot depend on a user still existing +in the users table; disabling/deleting user must not erase historical +spend. Use bounded indexed aggregates or a dedicated aggregate rollup, +with no full request-history scans under the writer gate. + +### C — Admin management and observable state + +Add instance-rule CRUD to local `users budget` CLI (or similarly scoped +existing admin command), `/api/user-budget` admin projection and +server-rendered `dashboard/admin.rs`; member views may show only their +effective cap without disclosing others' usage. Honor existing CSRF, +admin session requirements, profile revision / cached policy invalidation, +and same-window concurrent update semantics. The UI must identify +instance-global vs per-user and hard vs estimated rules without treating +estimates as billed dollars. + +## 6. Non-goals + +No multi-organization budgeting, external billing, cloud synchronization, +new per-provider credential quota rules, dynamic model substitution after +quota refusal, or second SQLite pool/writer. + +## 7. Ordered work packages + +1. Declare global rule schema, intersection and exact window semantics; + extend `rust/assets/db/migrations/`, checksums, runtime manifest. +2. Implement transactional all-scope reservation + settlement/read model + including retry, selector and user-removal invariants. +3. Add local admin API/CLI/UI forms and model/instance rule conflict tests. +4. Benchmark DB gate tenure for 1/10/49 stored **identities** with + controlled concurrency on normal host; do not assert SBC performance. + +## 8. Failure, cancellation, restart and contention + +Budget-edit commit changes admission for **new** requests only; existing +authoritative holds retain source policy version. If global aggregation +fails, reject requests under a hard rule rather than fall back to +user-only enforcement. On cancellation, terminalization and restart, +recovery must release the same global hold exactly once; M010 supplies +the dedicated fault corpus. Guard races at day/month boundaries, including +requests active across UTC reset; active concurrency counts across windows. + +## 9. Compatibility and migration + +Existing M005 user rules stay intact and continue to apply by +intersection. Absence of global rules must leave a single-admin workload +identical except measured bounded overhead. Do not rewrite provider +`[routing]` or `[providers]`. Migration has schema version >60 +and tests automatic upgrade from 60 and restored older fixtures. A +pre-migration binary requires a matching pre-upgrade DB backup to downgrade. + +## 10. Tests + +Extend `rust/tests/on_prem_budgets.rs`, `on_prem_admin_ui.rs`, +`database_compatibility.rs`, `coordinator_publication.rs`, +`coordinator_finalization.rs` and `server_transport.rs`. +Include 2/10/49 distinct users simultaneously consuming the last +global request/concurrency slot, two models sharing a global budget, +selector/retry without double count, finalization failure, +day/month rollover with active holds, disable/revoke/delete owner, +global underflow/overflow, 403 cross-user mutation, and no-config +single-admin parity. All DB operations must use the one gate. +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_admin_ui -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test database_compatibility -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_publication -- --test-threads=1 +``` +## 11. Required verification commands + +Run applicable focused targets in §10, then the full default/no-default gates: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +git diff --check +``` +Run `cargo deny --manifest-path rust/Cargo.toml check`, the feature tree, +and `cargo build --manifest-path rust/Cargo.toml --locked --release` when +Cargo dependencies/features change. Use the repository's tooling/test commands +when scripts are changed. Closure records must distinguish local vs hosted +vs physical hardware results; never claim unrun tests as passed. +## 12. Documentation + +Update `architecture/deep-dive-database.md`, +`architecture/deep-dive-request-lifecycle.md`, `docs/model-limits.md`, +`docs/api-reference.md`, `docs/cli-reference.md` and UI help +text. Document specific ceilings, precedence and provider-side quotas +as unrelated quantities. + +## 13. Acceptance criteria + +With two disjoint authenticated users and profiles, an instance-wide +last slot admits **exactly one** logical request; no retry, alternate model +or user-switch bypass exists. All hard request/concurrency limits remain +strict, hard token/cost rules honor M008's enforceability contract, +and member-only browser/API credentials cannot edit global rules. +One-admin behavior and wire formats remain compatible. + +## 14. Stop conditions + +Stop if an all-users rule requires repeated full-scans on the foreground +hot path, introduces a second DB writer, makes the global ceiling +dependent on a user being present, or bypasses M008 proven-versus-estimated +typing. Keep scope within instance budget ownership. + +## 15. Closure evidence + +Write `plans/closure/on-prem-user-access/009-status.md` with +schema/upgrade evidence, concurrent distinct-user last-slot and rollover +tests, DB gate contention and memory observations, authorization/CSRF +matrix, exact commands and real CI, and unblock audit. Historical M005 +closure remains immutable. + +## 16. Handoff + +Implement only after M008 contract is accepted. M010 owns the +crash/restore campaign; M011 owns visual dashboard parity. diff --git a/plans/implementation/on-prem-user-access/010-budget-crash-recovery-and-schema54-restore.md b/plans/implementation/on-prem-user-access/010-budget-crash-recovery-and-schema54-restore.md new file mode 100644 index 000000000..cd1d68a98 --- /dev/null +++ b/plans/implementation/on-prem-user-access/010-budget-crash-recovery-and-schema54-restore.md @@ -0,0 +1,231 @@ +# On-Prem User Access M010 — Budget crash recovery and schema-54 restore + +Status: implemented + +Repository baseline: `1218423b` on `plans/on-prem-small-team-user-access` (PR #7), schema 62, M008/M009/M013 closed as of 2026-10-14. (Plan text cites `c18dcc20`/schema 60; superseded by the M008/M009/M013 landings — activation records the actual baseline.) + +Source roadmap: +- `plans/subsystems/on-prem-user-access-roadmap.md#m010--durable-crash-recovery-and-schema-54-restore-corrective` + +Long-term references: +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) + +Primary class: invariant + +## 1. Objective + +Budget crash recovery and schema-54 restore. This is a **new corrective** to the conditionally closed M005/M006/M007 records, not a rewrite of historical closure. + +## 2. Readiness and dependencies + +**Hard:** M008 and M009 accepted closure. Their new hold semantics and +instance-global scope must be frozen before fault injection. +**Interface:** existing `operations::backup`/`recover`, +`db::MigrationRunner` and coordinator reconciliation contracts are stable. +Runs on disposable local databases; does not require physical SBC hardware. + +## 3. Evidence and why prior tests missed it + +`plans/closure/on-prem-user-access/005-status.md` reports missing +budget-specific process-crash/restore testing and DB gate overhead. +`plans/closure/on-prem-user-access/007-status.md` reports the real +schema-54 snapshot restore was not exercised; migrating a synthetic +older fixture through v60 is **not** the same thing. Current +`rust/src/user_budget.rs` and `coordinator/publication.rs` add +budget holds in publication, with settlement in +`coordinator/finalization.rs`; `coordinator/reconciliation.rs` +handles interrupted legacy reservations. The source M007 acceptance +requirements also require authenticated restore/upgrade behavior. +`rust/src/db/migrations.rs` enforces checksum identities. + +## 4. Non-regressing invariants + +- Canonical use is **one administrator on one local/SBC instance**, optional <50 named users. No SaaS, organization tenancy, billing platform, centralized IAM, new database writer/service, built-in WAN/VPN/reverse proxy, or mandatory cloud calls. +- Upstream provider topology, API keys, routing and self-hosted model endpoints remain in TOML/environment; SQLite owns users, personal credentials, profiles, policy, downstream budgets, accounting and audit. Do not place upstream keys in user tables or web administration. +- Maintain generation leases, existing coordinator publication/finalization, a **single SQLite connection/gate/worker with WAL/NORMAL**, canonical wire and transparent streaming, bounded hot-path memory and provider fairness. Do not turn downstream user budget refusal into upstream provider unhealthiness. +- Preserve published OpenAI Chat/Responses/compact and Anthropic Messages behavior, permitted client auth headers, existing client profiles, dashboard themes and local legacy upgrade behavior. +- Keep identity and cost metadata redacted, prohibit secret/password/bearer leakage in logs, HTML snapshots or qualification artifacts, and fail closed rather than silently weaken configured security limits. + +## 5. Required correctness/evidence work + +### A — Crash-state transition audit + +Inventory the precise authority/transaction invariants across: +credential/policy resolution; selector pre-work; initial +budget/provider publication; retry hold extension; upstream acceptance; +attempt finalization; terminal response; reconciliation; backup; and +normal/forced shutdown. Record an explicit matrix for committed, +ambiguous, and rolled-back transactions. Ensure no dispatched provider +attempt can lack its durable user/global hold and no terminal request +retains a phantom active hold. Use actual branch code to decide whether +existing `reconciliation.rs` needs an additive budget recovery step. +Do not modify historical migration data or alter core +`reconciliation.rs` behavior without narrowly proved gap. + +### B — Deterministic fault injection + +Extend `rust/tests/on_prem_budgets.rs`, +`coordinator_publication.rs`, `coordinator_finalization.rs` +and new scoped process/test harness to pause at real observable +transaction/transport boundaries and terminate/restart the process +(or safely simulate abrupt failure against an isolated database). +Include crash **before/after** publication commit, selector child, +after upstream acceptance, mid-stream, before/after finalizer commit, +duplicate terminal command, cancel, backup during active reservation +and DB read-only/busy/commit ambiguity. On restart verify bounded +convergence, idempotent adjustment, no double cost, no leaked global +hold, and no bypass of next admitted request. Assertions must +converge on observable rows/awaited channels under bounded timeouts, +not fixed sleeps or arbitrary yield loops. + +### C — Historical schema-54 backup and migration + +Capture or construct a byte-valid **schema-54** backup independently of +current migration application, with audited legacy upstream-account, +request and token-usage history and appropriate SQLite WAL checkpoint. +Prove source identity/checksum/integrity before applying migration +55+; run current `eggpool migrate` against a disposable restore, +`users bootstrap --password-stdin`, login, personal token revoke, +profile editing, hard/estimated/global rule migration, forced restart, +backup/restore of v61+ state, and compare request ownership/account +quota/usage/audit. Legacy shared-key behavior before activation must +be distinct from fresh-init; never convert historical shared-key +inference to a fabricated human identity. Old binary rollback is via +protected matching old backup, not a destructive down-migration. + +### D — Quantify storage pressure + +Measure transaction/wait times and write counts for 1/10/49 +distinct credential identities, with/without budget rules on +ordinary local hardware; track WAL bytes, checkpoint behavior, +pending/finalized count, RSS, CPU and latency. Record paired baseline +configuration/build and fail if new global rollups unacceptably widen +SQLite gate tenure. This is not yet physical SBC capacity evidence: +M012 owns that proof. + +### E — Resolve M003 updater-unit-test condition accurately + +Reconcile `plans/closure/on-prem-user-access/003-status.md`'s two +local `operations::update` failures with current hosted CI at the +exact branch SHA. Attempt deterministic local isolated reproduction +(with home/ownership fixtures) and classify as existing platform +regression versus stale historical failure; if still reproducible, +open a **separate bounded deployment-packaging corrective** rather than +changing updater authority in this budget pass. Do not erase or edit +M003 closure history. + +## 6. Out of scope + +No storage redesign, projection second writer, DB pool, new WAL policy, +cloud persistence, production hardware qualification, UI screenshot +work, or rewriting historical closed records. + +## 7. Ordered work packages + +1. Baseline current source/closure, define exact identity/budget + transaction state machine and failpoints. +2. Add reproducible crash/cancel/finalizer restart tests; fix only + demonstrated budget/reconciliation defects and add regressions. +3. Exercise independent schema-54 restore and forward upgrade + + backup/restore of current state with sanitizer/manifest assertions. +4. Run paired ordinary-host SQLite/latency observations; report + update-test condition and either close evidence or register a + distinct deployment-owned follow-up. + +## 8. Failure, cancellation, restart and contention + +Committing a durable reservation and request must remain atomic. +After crash, recovery must preserve conservatively committed costs, +not forgive paid attempts when response usage is unknown; unknown +usage must remain marked as such. Holds may be compensated only when +the upstream submission state supports doing so without false credit. +Run concurrency tests under the same SQLite gate; never attempt +reconciliation with an independent writer or asynchronous worker pool. + +## 9. Compatibility + +Existing schema 54/55/60 fixtures and all legacy data survive the upgrade. +Verify `--no-default-features` and generated checksums/manifest. +No provider TOML changes or network dependency. Forward-only migration +is explicit; never promise a newer DB can be opened by an older binary. + +## 10. Test and focused verification commands + +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_publication -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_finalization -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_c008 -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_c009 -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test database_compatibility -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test runtime_lifecycle_r009 -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --lib operations::backup -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --lib operations::update -- --test-threads=1 +``` +Add separate documented test commands for the isolated child-process, +schema-54 image and paired performance fixture; preserve captured +SHA-256 and redacted artifact manifest. +## 11. Required verification commands + +Run applicable focused targets in §10, then the full default/no-default gates: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +git diff --check +``` +Run `cargo deny --manifest-path rust/Cargo.toml check`, the feature tree, +and `cargo build --manifest-path rust/Cargo.toml --locked --release` when +Cargo dependencies/features change. Use the repository's tooling/test commands +when scripts are changed. Closure records must distinguish local vs hosted +vs physical hardware results; never claim unrun tests as passed. +## 12. Documentation + +Update `architecture/deep-dive-database.md`, +`architecture/deep-dive-request-lifecycle.md`, +`docs/backup-restore.md`, `docs/deployment.md`, +`docs/cli-reference.md` and any test-owned qualification guide. +Use a reproducible artifact path under `artifacts/qualification/` +without sensitive home paths or credentials. + +## 13. Acceptance criteria + +Each supported failpoint converges after restart to the same +durable logical request owner, correct user/global holds and +idempotent finalization; no new request can over-admit and no +nonterminal orphan holds remain without documented bounded recovery. +An independent v54 fixture upgrades to current schema and recovers +permissions, usage, revocation and existing provider-account history. +Ordinary-host gate measurements are recorded, including worst-case +outliers. Updater regression receives an evidence-based disposition, +not a silent waiver. + +## 14. Stop conditions + +Stop if a committed paid attempt would be credited back without +authoritative evidence; crash matrix reveals unbounded orphan rows; +a historical schema-54 migration or checksum is inconsistent; +test runner touches live data; or DB architecture must change. +Write a separate architecture/owner plan for any unbounded fix. + +## 15. Closure evidence required + +`plans/closure/on-prem-user-access/010-status.md`: state graph and +failpoint matrix with actual results; v54 fixture identity/restore/ +upgrade checksums; DB overhead samples; security redaction; +exact CI/local commands and revisions; updater condition disposition; +severity findings, registry unblock audit. Never rewrite previous +conditional M003/M005/M007 closures. + +## 16. Handoff + +Bounded persistence and recovery owner. The physical target +qualification remains M012. diff --git a/plans/implementation/on-prem-user-access/011-admin-dashboard-browser-parity-corrective.md b/plans/implementation/on-prem-user-access/011-admin-dashboard-browser-parity-corrective.md new file mode 100644 index 000000000..97a7ae03a --- /dev/null +++ b/plans/implementation/on-prem-user-access/011-admin-dashboard-browser-parity-corrective.md @@ -0,0 +1,220 @@ +# On-Prem User Access M011 — Admin/member dashboard browser qualification + +Status: implemented + +Repository baseline: implemented as `d495d483` on `feat/auto-20261014-e526194d` (separate session; activation step skipped there — recorded here), integrated onto `plans/on-prem-small-team-user-access` (PR #7) post-M010. (Plan text cites `c18dcc20`/schema 60; superseded by later landings. Original baseline line retained below for traceability.) + +Repository baseline (original): `c18dcc2051ae4987a2cb95a7cd046b8aff2095ff` on `plans/on-prem-small-team-user-access` (PR #7), schema 60, M001–M007 closure records as of 2026-10-13. + +Source roadmap: +- `plans/subsystems/on-prem-user-access-roadmap.md#m011--adminmember-browser-qualification-corrective` + +Long-term references: +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) + +Primary class: polish + +## 1. Objective + +Admin/member dashboard browser qualification. This is a **new corrective** to the conditionally closed M005/M006/M007 records, not a rewrite of historical closure. + +## 2. Readiness and dependencies + +**Hard:** none; M006 browser UI, theme and role/session interfaces are +implemented and the corrective can begin now. **Operational:** a +render-capable browser runner is needed for closure; pin the historical +Python oracle only for **pre-existing pages**, because the new admin/ +member pages did not exist in the Python implementation. +M008/M009 may change budget wording later; capture CSS/layout and +access controls now, then rerun the affected budget labels at integration. + +## 3. Evidence and why prior tests missed it + +`plans/closure/on-prem-user-access/006-status.md` reports that matched +screenshots/browser interactions for the new admin/member pages were +not run because a pinned Python oracle worktree and +`EGGPOOL_DASHBOARD_ORACLE_*` environment were unavailable. +`plans/closure/on-prem-user-access/007-status.md` carries that +condition forward. Existing `rust/src/server/dashboard/admin.rs`, +`render/admin.rs`, `render/layout.rs`, `rust/src/server/auth.rs`, +`rust/tests/on_prem_admin_ui.rs` and +`scripts/dashboard_parity/` provide current sources. +The legacy oracle has **no equivalent new admin/member pages**; a +pixel-for-pixel matching requirement for those would be invalid. +Prior unit tests verified HTTP/HTML and authorization, not actual +responsive rendering or browser interaction across themes. + +## 4. Non-regressing invariants + +- Canonical use is **one administrator on one local/SBC instance**, optional <50 named users. No SaaS, organization tenancy, billing platform, centralized IAM, new database writer/service, built-in WAN/VPN/reverse proxy, or mandatory cloud calls. +- Upstream provider topology, API keys, routing and self-hosted model endpoints remain in TOML/environment; SQLite owns users, personal credentials, profiles, policy, downstream budgets, accounting and audit. Do not place upstream keys in user tables or web administration. +- Maintain generation leases, existing coordinator publication/finalization, a **single SQLite connection/gate/worker with WAL/NORMAL**, canonical wire and transparent streaming, bounded hot-path memory and provider fairness. Do not turn downstream user budget refusal into upstream provider unhealthiness. +- Preserve published OpenAI Chat/Responses/compact and Anthropic Messages behavior, permitted client auth headers, existing client profiles, dashboard themes and local legacy upgrade behavior. +- Keep identity and cost metadata redacted, prohibit secret/password/bearer leakage in logs, HTML snapshots or qualification artifacts, and fail closed rather than silently weaken configured security limits. + +## 5. Required qualification work + +### A — Correct the visual oracle boundary + +Split two independent contracts: +1. **Existing dashboard:** preserve previously accepted Python-vs-Rust + shell, navigation, charts, theme/CSS and interaction parity on old routes, + using the pinned frozen Python oracle/manifest and its explicitly accepted + historical differences; no reopening old M001–M012 dashboard closures. +2. **New pages:** freeze a sanitized Rust-owned approved screenshot/DOM/ + accessibility reference for `/users`, `/profiles`, + `/admin/audit`, `/my-account`, `/my-account/models`, + `/usage`, token reveal/reset/expiry states, and login/forced + password change. Compare these with a stable reference on subsequent runs. + Do not invent Python page equivalents. + +A pinned git revision/asset hash is essential for reproducibility. +Design a small browser manifest using present +`scripts/dashboard_parity/browser.py`, `runner.py`, +`projection.py`, `report.py` rather than embedding a second +visual harness. If the oracle is unavailable, historical-page gate is +explicitly unqualified; new-page independent browser tests can run. + +### B — Real browser interaction and negative cases + +Test admin and member accounts at 1280x800 and narrow mobile viewport +(and one standard HiDPI where supported), with mouse and keyboard: +session creation/expiry/logout; first-login forced password change; +CSRF errors; user create/disable; profile assign/edit and conflict; +member-only catalog/usage; budget hard-vs-estimated labels; +one-time personal token visibility and refresh; member/agent/ +anonymous denial; accessible focus order/forms; error banners; +no-JS ordinary form operation; dark/light and representative Halloy +themes; no horizontal overflow, clipped dialogs, unreadable badges, +unbounded table lengths, or accidental server-secret render. +Use synthetic provider/user labels, never live tokens. + +### C — Bounded implementation fixes and guards + +Only fix demonstrated local renderer/CSS/semantic HTML issues; +keep `dashboard/render/` pure, not a second session/policy +authority. Add deterministic browser assertions/manifest for +new pages and static guards that prohibit provider passwords/ +API tokens in HTML snapshots and require session/CSRF fences. +If defects require unrelated dashboard architecture rewrite, open +separate implementation plan; don't expand this pass. + +### D — Documentation / parity disposition + +Update current `architecture/deep-dive-dashboard.md`, +`docs/deployment.md`, browser qualification docs and M006 +superseding **additive** closure note (in new M011 closure). +Record source-backed intentional design differences rather than +rewriting accepted old oracle output. + +## 6. Explicit non-goals + +No UI framework migration, provider-TOML web editor, identity redesign, +new Rust stateful frontend, admin/API security work without a failing +test, or claiming Python parity for pages Python never had. + +## 7. Ordered work packages + +1. Freeze the old-page oracle contract and new-page browser fixture + manifest. Pin tool versions, reference hashes, accounts and viewports. +2. Run admin/member browser flows, all role negative cases and theme/ + accessibility checks; capture sanitized images and DOM facts. +3. Fix only observed renderer/interaction regressions; rerun legacy-page + comparison, Rust route/security suites and new-page browser suite. +4. Record explicit absent-oracle condition if old page comparison cannot + execute; never pretend a green unit suite proves visual parity. + +## 8. Failure, cancellation, restart and contention + +Browser-session expiry and password reset must immediately deny +new mutations while already-rendered static content remains safe. +Concurrent profile edits surface a conflict without stale last-write +success; token plaintext only appears once and is not cached. +No credentials or raw API prompts in browser traces/fixtures. +The generated reference manifest must be deterministic and bounded. +New browser tooling must use disposable DB/home and loopback fake +provider only; no live config mutation. + +## 9. Compatibility + +Retain existing themes and dashboard assets, without new CDN, +external JS or front-end service. Preserve the accepted historical +browser parity contract on all pre-existing routes. Make a new +independent golden for the admin/member pages and explain why +old Python pages cannot serve as their oracle. + +## 10. Tests and focused commands + +Add/extend `tests/tooling/test_dashboard_parity_modules.py`, +`test_dashboard_parity_projection.py`, a narrow new browser +qualification fixture, `rust/tests/on_prem_admin_ui.rs`, +`dashboard_filters.rs`, `on_prem_sessions.rs` and +`operations_o005.rs`. Browser runner entry: +`scripts/qualification_dashboard_parity.py` with existing validated +flags/env (inspect `--help`; do not invent invocation options). +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_admin_ui -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test dashboard_filters -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_sessions -- --test-threads=1 +uv run pytest tests/tooling/test_dashboard_parity_modules.py tests/tooling/test_dashboard_parity_projection.py -q +``` +## 11. Required verification commands + +Run applicable focused targets in §10, then the full default/no-default gates: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +git diff --check +``` +Run `cargo deny --manifest-path rust/Cargo.toml check`, the feature tree, +and `cargo build --manifest-path rust/Cargo.toml --locked --release` when +Cargo dependencies/features change. Use the repository's tooling/test commands +when scripts are changed. Closure records must distinguish local vs hosted +vs physical hardware results; never claim unrun tests as passed. +## 12. Documentation + +Update `architecture/deep-dive-dashboard.md`, +`docs/deployment.md` and the dashboard test/qualification documentation +with old-page oracle versus new-page Rust reference rules and traceability. + +## 13. Acceptance criteria + +Every new admin/member page has a reusable redacted +desktop/mobile browser baseline with documented approved appearance; +forms and role guards work interactively, including forced password +change and one-time token. Old pages show no unreviewed regression +against the frozen Python oracle. A screen without historical Python +equivalent is **not** judged by invented Python pixel parity. +UI accurately distinguishes hard versus estimated quotas after M008. +No leakage, missed role/CSRF fence or serious accessibility defect remains. + +## 14. Stop conditions + +Stop if the oracle dependency is unavailable at final closure; +mark conditional/blocked evidence and request the exact worktree or +runner artifact rather than bypassing regression checks. Stop +on newly revealed auth/privacy flaw (new security corrective owner), +massive frontend rewrite or raw secret fixture. + +## 15. Closure evidence required + +`plans/closure/on-prem-user-access/011-status.md`: +browser-manifest hashes, per-page/view/theme interaction and +negative-role matrix, historical-route parity results with accepted +differences, new Rust-owned reference screenshots, accessibility +checks, actual commands and CI results, residuals, registry audit. +If any mandatory browser gate cannot run, do not mark fully closed. + +## 16. Handoff + +Independent browser qualification can proceed in parallel with +budget correctness; rerun budget wording once M008/M009 land. diff --git a/plans/implementation/on-prem-user-access/012-physical-sbc-small-team-qualification.md b/plans/implementation/on-prem-user-access/012-physical-sbc-small-team-qualification.md new file mode 100644 index 000000000..10ef1523b --- /dev/null +++ b/plans/implementation/on-prem-user-access/012-physical-sbc-small-team-qualification.md @@ -0,0 +1,250 @@ +# On-Prem User Access M012 — Physical SBC and small-team qualification closure + +Status: implemented + +Repository baseline: `10eeee25` on `plans/on-prem-small-team-user-access` (PR #7), schema 62, M001–M007 closure records as of 2026-10-13 plus accepted M008/M009/M010/M011/M013 closures as of 2026-10-14. + +Tooling foundation landed (`b1f6d20b`); physical SBC and reference +Linux host evidence remain missing — see `plans/closure/on-prem-user-access/012-status.md`. + +Source roadmap: +- `plans/subsystems/on-prem-user-access-roadmap.md#m012--physical-sbc-small-team-qualification-and-final-disposition` + +Long-term references: +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) + +Primary class: polish + +## 1. Objective + +Physical SBC and small-team qualification closure. This is a **new corrective** to the conditionally closed M005/M006/M007 records, not a rewrite of historical closure. + +## 2. Readiness and dependencies + +**Hard:** accepted M008, M009, M010 and M011 closures, with +stable budget/storage/UI contracts. **Operational:** provisioned +physical Le Potato/Pi-class machine and ordinary Linux reference +host; macOS x86_64 synthetic fixtures are insufficient. This +qualification plan is registered but **not ready** until the +dependency and evidence gates are met. It is not a promise to +perform hardware work without those targets. + +## 3. Evidence and why prior tests missed it + +`plans/closure/on-prem-user-access/001-status.md` deferred Argon2id +19 MiB / two-worker memory and latency measurement. +`003-status.md` deferred principal/accounting overhead. +`007-status.md` records 7/7 local synthetic loopback +requests at schema 60 on macOS x86_64, no physical SBC +or 1/5/10/25/49-user/concurrency benchmarks, no paired +legacy single-owner control, no WAL/SQLite gate percentiles. +The existing `scripts/qualification_sbc.py` is the one +physical-target runner; `scripts/qualification_stability.py` +and `tests/tooling/test_qualification_sbc.py` provide +tooling fixtures. Previously measured Pi WAL checkpoint stalls +mean simply observing green average latency is not sufficient. + +## 4. Non-regressing invariants + +- Canonical use is **one administrator on one local/SBC instance**, optional <50 named users. No SaaS, organization tenancy, billing platform, centralized IAM, new database writer/service, built-in WAN/VPN/reverse proxy, or mandatory cloud calls. +- Upstream provider topology, API keys, routing and self-hosted model endpoints remain in TOML/environment; SQLite owns users, personal credentials, profiles, policy, downstream budgets, accounting and audit. Do not place upstream keys in user tables or web administration. +- Maintain generation leases, existing coordinator publication/finalization, a **single SQLite connection/gate/worker with WAL/NORMAL**, canonical wire and transparent streaming, bounded hot-path memory and provider fairness. Do not turn downstream user budget refusal into upstream provider unhealthiness. +- Preserve published OpenAI Chat/Responses/compact and Anthropic Messages behavior, permitted client auth headers, existing client profiles, dashboard themes and local legacy upgrade behavior. +- Keep identity and cost metadata redacted, prohibit secret/password/bearer leakage in logs, HTML snapshots or qualification artifacts, and fail closed rather than silently weaken configured security limits. + +## 5. Required qualification scope + +### A — Freeze actual target and work corpus + +Provision and record CPU type, core count, OS image, RAM, disk/MMC/ +filesystem, board model, power/network, Rust and binary SHA-256, +DB schema, persistent WAL flags, native feature set, config +and allowed provider stub. Verify representative lightweight +Le Potato and Raspberry Pi 5 where available; otherwise record +missing target separately, without silently substituting a laptop. +Run isolated synthetic users and fake upstreams; **never** +point test harness at production DB or real credentials. +Maintain both installed single-owner upgrade and newly provisioned +small-team startup flows. Identify and lock baselines before +candidate measurement. + +### B — Named users distinct from concurrency + +Measure **1/5/10/25/49 named users** (49 is not 49 simultaneous +LLM generations) under explicitly chosen bounded in-flight +concurrency levels 1/2/4/8 and optionally 16 when hardware and +memory headroom allow. For each compare same input corpora, +Chat/Responses/Messages, finite+SSE, allowed/denied model +selection, qualified alias/semantic routing, realistic tiny and +moderate payloads, user/global budget admission and last-slot +refusals. Include an operator-local self-hosted model fixture +(no cloud dependency); use fake provider responses and optional +local model only if isolated. Record actual concurrency and +completion totals, no hidden extra selector work. + +### C — Physical Argon2 and runtime resource posture + +Run 1 and 2 simultaneous Argon2id password verifications under +idle and representative inference load, measuring p50/p95 +wall-time, RSS/peak heap, swap/OOM events and CPU. +Under above matrix capture binary size, idle and peak RSS, +CPU, credential cache/DB auth read, request p50/p95/p99, +stream TTFT, SQLite gate wait/tenure, WAL growth/checkpoint +progress, open-file/thread counts and backup/restart behavior. +Compare legacy unmodified main release, current managed +single-admin without budgets, and candidate with budgets. +The existing single-user SBC path must remain practical and +cannot require extra services or a multi-step SaaS onboarding. + +### D — Qualification gates and release disposition + +Before testing, choose explicit pass/fail gates using +existing `scripts/qualification_sbc.py` baseline evidence and +user-facing solo path (e.g. allowable TTFT p95/SQLite gate and +RSS deltas, bounded tail latencies, no WAL divergence, +no OOM/swap, no sustained request drops). Don't fabricate +thresholds or measurements after results. Failing a gate +requires small measured optimization or new corrective plan; +never introduce a second DB writer/connection or retune +checkpoint policy without a separate architecture acceptance. +If 49 named users are inexpensive while 8 concurrent inference +requests exceed hardware, document actual supported load +rather than marketing "50 users" as a throughput guarantee. + +### E — Cross-cutting final release audit + +Cross-reference M008–M011 closures, original conditions in +M001/M003/M005/M006/M007, independent schema-54 restore and +whole-workspace hosted CI. Run older unrelated +`operations::update` tests on the target or confirm M010's +documented environment-specific disposition. Reconcile +`plans/registry.md` and roadmap to `closed` only **if** +all normative requirements and physical gates have closure +evidence; otherwise keep `active` with precise blocked/conditional +items. Do not back-edit historical closure records. +Before merging PR #7, verify docs/PR summary accurately reflect +actual runtime work and security caveats. + +## 6. Non-goals + +No remote SaaS load generator, cloud IAM/service, internet-facing +deployer, exact promise of 50 simultaneous inferences, complex +multi-node scaling or unsupported SBC target claim. + +## 7. Ordered work packages + +1. Reuse `qualification_sbc.py` / tooling harness with bounded, + repeatable user-identity/client/profile/quotas matrix; add + `tests/tooling/` fixture validations and sanitization. +2. Freeze controls, build artifacts, scripts, hardware and + acceptance thresholds; run paired target corpus and Argon2. +3. Record raw evidence/derived aggregates per hardware class; + investigate failures and repeat controlled measurements. +4. Publish honest capacity/budget caveats, final release matrix + and roadmap/registry closure or remaining blocker. + +## 8. Failure, cancellation, restart and contention + +Long-lived streams and canceled users must release correct +per-user/global holds; revocation rejects new inference. +Forced shutdown/restart, WAL growth and restoration remain +bounded. A fixture inducing physical memory pressure must +abort cleanly before kernel OOM rather than render the box +unresponsive. If hardware is unavailable, mark as +**blocked** and publish no target claims; successful macOS +CI never substitutes for physical results. + +## 9. Compatibility and migration + +Protect the original single-user TOML/provider/agent setup, +schema-54-into-current restored deployment, backup permissions, +API model discovery, anonymous readiness probes, self-hosted +provider config and `EGGPOOL_API_KEY` compatibility with +personal tokens. No rollback to older binary without +pre-upgrade DB backup. + +## 10. Tests and target commands + +New `tests/tooling/test_on_prem_sbc_qualification.py` and +updates to `tests/tooling/test_qualification_sbc.py` +should validate manifest, redaction, exact target binding, +named-users vs actual concurrency and baseline/candidate +pairing. Use actual existing +`scripts/qualification_sbc.py --help` to select arguments, +not hardcoded/assumed flags. Focused: +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_identity -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test runtime_lifecycle_r009 -- --test-threads=1 +uv run pytest tests/tooling/test_qualification_sbc.py -q +``` +## 11. Required verification commands + +Run applicable focused targets in §10, then the full default/no-default gates: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +git diff --check +``` +Run `cargo deny --manifest-path rust/Cargo.toml check`, the feature tree, +and `cargo build --manifest-path rust/Cargo.toml --locked --release` when +Cargo dependencies/features change. Use the repository's tooling/test commands +when scripts are changed. Closure records must distinguish local vs hosted +vs physical hardware results; never claim unrun tests as passed. +## 12. Documentation + +Update `docs/raspberry-pi.md`, `docs/deployment.md`, +`docs/agent-configuration.md`, `docs/backup-restore.md`, +`architecture/deep-dive-database.md` and `README.md` +with only qualified capacity/requirements and recommended +operator TLS reverse-proxy/VPN for LAN browser passwords. +Run release docs validator for relevant files. + +## 13. Acceptance criteria + +All qualifying physical targets and reference-host results are +archived with reproducible manifests, p50/p95/p99 and named +user/concurrency coverage, and resource usage is within +**predeclared** accepted gates. No unqualified assertion about +Le Potato/Pi performance or 49 simultaneous completions. +M001/M003/M005/M006/M007 conditional findings have explicit +evidence-based resolution, with approved exception only when +non-normative and transparently documented. One-admin fresh +install remains low-friction, no provider secret moved to DB, +and managed policy/budget cannot regress isolation. + +## 14. Stop conditions + +Block closure on missing physical hardware, target-class OOM, +WAL nonconvergence, unaddressed hard-limit bypass, bad backup +identity, leaked tokens, failed visual security checks or +false capacity claim. Open narrowly scoped corrective for +new defects rather than marking a condition as implicitly +satisfied. + +## 15. Closure evidence required + +`plans/closure/on-prem-user-access/012-status.md` must +include environment/bin/config fingerprints, direct target +and control artifacts, Argon2, named-user and concurrency +metrics, single-admin deltas, SQLite WAL and recovery, +long-running failures, eligibility checklist, +CI/migration/dashboard verification, known risks and +registry unblock audit. If hardware evidence is missing, +this milestone cannot close and on-prem roadmap stays +`active` with the blocker explicit. + +## 16. Handoff + +This is physical evidence / polish, **not** an authorization +to redesign persistence for an unsupported benchmark. diff --git a/plans/implementation/on-prem-user-access/013-transcode-eligibility-corrective.md b/plans/implementation/on-prem-user-access/013-transcode-eligibility-corrective.md new file mode 100644 index 000000000..3ab6ba644 --- /dev/null +++ b/plans/implementation/on-prem-user-access/013-transcode-eligibility-corrective.md @@ -0,0 +1,173 @@ +# On-Prem User Access M013 — Transcode-eligibility admission bridge corrective + +Status: implemented + +Repository baseline: `ea56ddcb` on `plans/on-prem-small-team-user-access` (PR #7), schema 62, M008/M009 closures accepted. + +Source roadmap: + +- `plans/subsystems/on-prem-user-access-roadmap.md` (corrective dependency graph and promotion) + +Long-term references: + +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: + +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) + +Primary class: capability + +## 1. Objective + +Restore cross-surface upstream eligibility broken by M008's admission edit. +This is a **new corrective** to the closed M008 record +(`plans/closure/on-prem-user-access/008-status.md`), not a rewrite of it. + +## 2. Readiness and dependencies + +No hard predecessor beyond the accepted M008/M009 closures, which supply the +admission bridge and the budget contract this pass preserves. Ready: the +defect, the missing line, and the failing hosted gate are all identified. +No physical hardware prerequisite. + +## 3. Evidence and why prior qualification missed it + +Hosted CI `check` run `38517675228` (PR #7) failed at +`uv run pytest tests/tooling/` with +`test_q007_loopback_fake_emits_secret_free_evidence` reporting +`messages-to-responses-finite returned HTTP 503` (`No eligible account was +available`). All Rust gates in the same run passed. Local bisection proved +the cause: M008 implementation `0149a6c8` fails the case while its parent +`b0e8715f` passes all seven. + +Root cause: when M008 extended `routing_request_facts_from_parts` +(`rust/src/request/admission.rs`) with the budget-fact copies, the edit +dropped the adjacent line +`facts.transcode_protocols = inputs.transcode_protocols.clone();`. +Admission-built routing facts therefore carried an empty transcode list, so +cross-surface candidates (Messages → Responses) lost eligibility while +native surfaces kept passing — exactly the observed 3-pass/1-fail shape. + +Why verification missed it: + +1. The dropped line sat adjacent to the added lines in one edit hunk; review + read the additions, not the surviving context. +2. No Rust test asserted the admission→facts bridge carries routing inputs: + existing tests build `RoutingRequestFacts` directly (bypassing admission) + or assert budget facts only. +3. The Python tooling suite was not run locally for M008/M009. The source + plans' §11 only requires tooling commands "when scripts are changed", and + no script changed — but the tooling suite drives the Rust binary, so a + Rust-only behavior break hid until hosted CI. The before-push matrix in + `AGENTS.md` lists `uv run pytest tests/tooling/` unconditionally; that + authority should have governed. + +A removed-line audit of the full M008+M009 `rust/src/` diff +(`git diff b0e8715f..ea56ddcb`) confirms this was the only accidental +deletion; every other removed line maps to an intentional restructure. + +## 4. Non-regressing invariants + +- Same invariants as M008 §4 (single-instance on-prem, TOML topology keys, + generation leases, single SQLite gate/worker WAL/NORMAL, canonical wire, + published protocol behavior, redaction, fail-closed limits). +- No budget, schema, config, wire-format, or operator-surface change in this + pass: the fix restores one deleted copy, nothing more. + +## 5. Scope and required production changes + +### In scope + +- Restore `facts.transcode_protocols = inputs.transcode_protocols.clone();` + in `routing_request_facts_from_parts`. +- One regression test proving admission-built facts preserve routing inputs + (`transcode_protocols`, `requested_protocol`, `catalog_stale_after_s`, + `capability_policy`, `now`, `request_surface`) alongside budget facts. + +### Explicitly out of scope + +- No budget semantics, migration, config, dashboard, CLI, API, or doc change + beyond this plan and its closure record. No new restart/reload keys. No + unrelated closed scope is reopened. + +## 6. Ordered work packages + +1. Restore the deleted copy (one line, `rust/src/request/admission.rs`). +2. Add the bridge regression test + (`admission_preserves_routing_inputs_for_transcode_eligibility` in + `rust/tests/on_prem_budgets.rs`); verify it fails with the line removed + and passes with it restored. +3. Run the full default/no-default gates **plus** the Python tooling suite + locally (`uv run pytest tests/tooling/ -q`) and the Q007 loopback + qualification to pass. + +## 7. Failure, cancellation, restart and contention + +No new runtime path: the restored copy is a synchronous field assignment +inside admission fact construction. No transaction, retry, selector, or +contention semantic changes. + +## 8. Compatibility and migration + +No schema, config, API, CLI, or wire change. Zero migration impact; existing +databases, rules, holds, and clients are unaffected. + +## 9. Focused and adversarial tests + +- `cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1` +- `uv run pytest tests/tooling/test_qualification_live_provider.py -q` (the + hosted-failing target, run locally against the debug binary) +- Then the full default/no-default gates per §11. + +## 10. Required verification commands + +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- --test-threads=1 +uv sync --frozen +uv run pytest tests/tooling/ -q --tb=short --maxfail=1 +git diff --check +``` + +`cargo deny`, the feature tree, and the locked release build run only on +Cargo dependency/feature changes (none here). + +## 11. Documentation updates + +This plan and its closure record. No user/architecture doc change: behavior +returns to the documented M008 contract. + +## 12. Acceptance criteria + +`messages-to-responses-finite` returns 200 in the Q007 loopback +qualification; the full local default/no-default Rust suites and the full +Python tooling suite pass; the bridge regression test fails without the fix +and passes with it. + +## 13. Stop conditions + +Stop if the restored line does not repair the Q007 case (the defect would be +elsewhere and this plan's premise wrong), if any gate regresses, or if scope +expands beyond the one-line restoration plus its guard. + +## 14. Closure evidence required + +New `plans/closure/on-prem-user-access/013-status.md` must include: the +bisected commits (base pass, M008 fail), the exact dropped line, the +removed-line audit method and result, the guard's fail-without/pass-with +proof, exact commands and local outcomes for every §10 gate including the +tooling suite, and the hosted CI outcome on push. Do **not** +retrospectively edit `008-status.md`. + +## 15. Handoff + +One-line restoration plus guard; `server/*` stays thin and coordinator +remains the single ingress execution owner. M010/M011/M012 are unaffected. diff --git a/plans/implementation/on-prem-user-access/014-rustls-advisory-dependency-corrective.md b/plans/implementation/on-prem-user-access/014-rustls-advisory-dependency-corrective.md new file mode 100644 index 000000000..0dbd9a20f --- /dev/null +++ b/plans/implementation/on-prem-user-access/014-rustls-advisory-dependency-corrective.md @@ -0,0 +1,241 @@ +# On-Prem User Access M014 — Rustls advisory remediation and transport qualification + +Status: implemented + +Repository baseline: `3c7def1e35871209e68563b2ee88f36033aff76e` on `plans/on-prem-small-team-user-access` (PR #7, open; 134-file implementation, schema 62). + +Source roadmap: +- `plans/subsystems/on-prem-user-access-roadmap.md#m014--rustls-dependency-security-advisory-corrective` + +Long-term requirements: +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted; no new ADR needed unless an auth or lease boundary must change). + +Primary class: invariant + +## 1. Objective + +Restore a clean mandatory dependency-security gate on the on-prem integration +branch by resolving all four newly reported RustSec advisories against +`rustls 0.23.45`, with the **smallest lockfile-compatible TLS patch bump** +and transport/feature regression proof. Avoid unrelated provider-family +upgrades or cargo-deny exceptions. This is a merge prerequisite, but +does not assert an unverified exploit path in EggPool. + +## 2. Why this milestone is ready + +Hard dependencies: none beyond accepted provider-transport and M001–M013 +contracts. `Cargo.toml` permits `rustls 0.23`; the resolved lockfile +is still 0.23.45 in both `main` and PR #7. The GitHub audit log itself +identifies the minimum fixed version (>=0.23.46). No hardware is needed +to make or test the patch; hosted security audit is the closure gate. +This issue also affects **main**; implement on PR #7 for merge +qualification and coordinate equivalent upstream/main resolution if it is +not merged promptly. + +## 3. Current implementation evidence and missed verification + +PR #7 head `3c7def1e35871209e68563b2ee88f36033aff76e`: GitHub Actions check and Linux/macOS installer +jobs pass, but `Dependency audit` run 38647245245/job 116137044959 +fails. Its `cargo-deny` output contains **four denial-level advisories** +for resolved `rustls v0.23.45`: +- `RUSTSEC-2026-0346` — key-bearing `Debug` output from TLS session values. +- `RUSTSEC-2026-0347` — ECH rejection buffered data handling. +- `RUSTSEC-2026-0348` — Brotli certificate decompression allocation risk. +- `RUSTSEC-2026-0349` — ServerHello downgrade-related behavior. + +The audit recommends `rustls >=0.23.46`. Separate duplicate-version +warnings are not what fails the gate. Direct `rust/Cargo.toml` +dependencies include `rustls`, `hyper-rustls`, `tokio-rustls`, +`eggfetch-core =0.2.2` and `eggress-outbound =1.0.12`; +transitives share the resolved Rustls family. `deny.toml` intentionally +rejects advisories without a blanket ignore. Earlier CI was green +**before** publication of the new advisories, so green Rust tests +never meant current security-audit clearance. + +## 4. Invariants that must not regress + +- EggPool remains a **single on-prem instance** for an owner or small team, + with provider topology, upstream credentials, TLS trust, private LLM + endpoints and routing configuration controlled by operator TOML/environment; + user identities, profiles, sessions, audit and downstream quotas stay in + SQLite. No SaaS, org/tenant control plane, embedded reverse proxy, external + IAM, secret-bearing HTML or browser provider editor. +- Preserve generation leases, provider health/eligibility, request identity, + retry and streaming finalization, single SQLite gate/worker and WAL/NORMAL, + live/restart config authority, all supported Chat/Responses/compact/Messages + client and cross-protocol routes, and the canonical solo SBC installation. +- Authorization and budget failure must not become provider unhealthiness + or anonymous fallback. No unauthenticated web mutation, shared + `admin/admin` credential, bypass of profile/global/model denies, or + moving operator secrets to SQLite. +- All qualification uses fake provider tokens, temporary roots and redacted + artifacts; never production data or real user passwords. + +## 5. Scope + +### In scope + +- Capture current lockfile resolution and resolved feature graph. +- Qualify a minimal security-fixed `rustls 0.23.x` patch in the existing + MSRV/toolchain constraints; preferred minimum 0.23.46 or a later safe + compatible patch (select from actual crates.io graph at execution). +- Regenerate only necessary `rust/Cargo.lock` entries; review and + justify every changed transitive (including `rustls-webpki` if forced). +- Verify TLS certificate validation, provider transports and feature matrix, + and the actual hosted cargo-deny result against the same final SHA. +- Reconcile branch/main common vulnerability without mixing unrelated + provider adoption. + +### Explicitly out of scope + +No upgrading Eggress 1.0.12/Eggfetch 0.2.2/EggServe 0.4.0 solely for +new features, no `deny.toml` advisory ignore/skip, no disabling TLS +verification, ECH flags or Brotli as a substitute for patching, no +new TLS architecture, no SQLite/identity changes, no unrelated lockfile +freshening, and no claim to security compliance certification. + +## 6. Required production changes + +Prefer lockfile-only resolution of the existing semver `rustls = "0.23"` +requirement. Inspect with `cargo tree --manifest-path rust/Cargo.toml -i +rustls --locked` (or explicitly select duplicate packages if needed) +and patch using: +```bash +cargo update --manifest-path rust/Cargo.toml -p rustls --precise 0.23.46 +``` +If 0.23.46 cannot resolve compatibly, choose the newest compatible +non-vulnerable 0.23.x patch and **record the reason and exact version**; +do not force an ABI/API-breaking major upgrade to make the gate green. +Run the audit against all features and the entire source lock graph, +not just default runtime. Ensure no second affected `rustls` copy or +known flagged `webpki` dependency remains. If feature unification alters +TLS client trust, SNI, TLS1.2/1.3, proxy or stream semantics, add the +narrowest fix and fixture for the regression, not another transport +backend. + +## 7. Ordered work packages + +A. Lock/advisory inventory — record all four IDs, original/fixed graph, + reverse dependents, deny policy and reproducible failing command. + +B. Minimal patch — update resolved Rustls, inspect lock diff, run + `cargo tree -e features` and ensure permitted licenses/sources/MSRV. + +C. Qualification — focused TLS and transcode routes, certificate + negative cases, proxy transport, streaming and cancellation; whole + default/no-default Rust plus Python tooling/installer matrix. Preserve + native Q007 messages-to-responses bridge regression (M013). + +D. Hosted verification — push candidate, require GitHub Actions CI, + both installer jobs, and `cargo-deny` green **on new SHA**. Record + exact advisory-free audit and a cross-branch note for the same issue + still present in `main` prior to merge. + +## 8. Failure, cancellation, restart and contention semantics + +Provider TLS handshake, certificate rejection, upstream proxy/connect, +and truncated-body failure must stay in their existing typed error classes. +Do not log Rustls session `Debug` fields or other key material. No +streaming framing/content or retries change. TLS client state remains +generation-scoped where currently owned. A failed dependency resolution +must leave the committed lock graph unchanged. No schema/DB migration. +If a patched version changes provider trust behavior or features, stop +and register a transport-owned corrective, not a silent security bypass. + +## 9. Compatibility and migration + +No TOML keys, downstream credential types, binary installer semantics, +API/wire/DB changes expected. Preserve release reproducibility with +`--locked`. Both `main` and PR #7 share the existing vulnerable +resolution; coordinate the permanent fix once and avoid a fresh +lockfile conflict at merge. Published upstream crate pins remain +unchanged unless dependency solving proves that impossible, in which +case stop for a separate authorization decision. + +## 10. Required tests + +Focused: +```bash +cargo tree --manifest-path rust/Cargo.toml -i rustls --locked +cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test provider_transport -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_global_budgets -- --test-threads=1 +uv run pytest tests/tooling/test_qualification_live_provider.py -q +``` +Inspect actual test-target inventory and substitute existing target +names if `provider_transport` is not a binary; do not claim a +nonexistent command passed. Add/extend TLS fixture tests for +valid CA, invalid/self-signed cert, hostname/SNI mismatch, and +proxy/transcode streaming EOF where missing. Include `cargo deny check +advisories` with actual repository root policy and lockfile selection. + +## 11. Required verification commands + +Record commands/results **actually executed** at the implementation SHA: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- --test-threads=1 +uv sync --frozen +uv run ruff check scripts/ tests/tooling/ +uv run pyright scripts/ +uv run pytest tests/tooling/ -q --tb=short --maxfail=1 +git diff --check +``` +Use `cargo deny check` with the repository's root `deny.toml` +and verify the command/working directory that the GitHub audit workflow +actually invokes. Run `cargo build --manifest-path rust/Cargo.toml --locked +--release` for production-affecting Cargo changes. The hosted CI, +installer-shell and dependency-audit jobs must be green on the final +head SHA; no earlier green check is a substitute. + +## 12. Documentation updates + +Update `plans/registry.md`, `plans/subsystems/on-prem-user-access-roadmap.md`, +new closure record, and `architecture/deep-dive-security.md` or +`architecture/deep-dive-provider-transport.md` **only** if dependency +qualification reveals documentation drift. No upstream provider setup +changes are expected. + +## 13. Acceptance criteria + +At final candidate SHA, `Cargo.lock` resolves a patched Rustls family +with none of RUSTSEC-2026-0346/0347/0348/0349 denied, and the **actual** +hosted Dependency audit passes. Default/no-default and TLS negative/positive +transport, Q007 transcode, provider-retry/stream/installer tests pass. +No blanket deny exception or changed certificate-validation posture; +reproducible locked release builds and preserved SBC resource posture +within existing documented bounds. + +## 14. Stop conditions + +Stop on resolver incompatibility, MSRV violation, a forced unrelated +family upgrade, unstable TLS feature graph, upstream verification +regression, lack of actual hosted audit proof, or any remaining +denial-level security alert. Register a separate minimal upstream +dependency fix instead of widening this pass. + +## 15. Closure evidence required + +`plans/closure/on-prem-user-access/014-status.md`: before/after +lockfile and reverse dependency tree, all four advisory IDs, exact +`cargo-deny` output, verified patched version, test/feature/locked +release result, hosted job URLs/SHA, negative TLS fixtures, separate +main-branch coordination status, risk review, and registry unblock +audit (promote M016 only after M014 + M015 accepted closure). + +## 16. Handoff notes + +This is a repository-wide security dependency issue observed during +on-prem PR review, **not** evidence the new auth code caused the +Rustls vulnerability. Leave provider family pins and owner TOML alone. diff --git a/plans/implementation/on-prem-user-access/015-authz-threat-review-corrective.md b/plans/implementation/on-prem-user-access/015-authz-threat-review-corrective.md new file mode 100644 index 000000000..20b7c5d62 --- /dev/null +++ b/plans/implementation/on-prem-user-access/015-authz-threat-review-corrective.md @@ -0,0 +1,275 @@ +# On-Prem User Access M015 — Final on-prem authentication and authorization threat review + +Status: implemented + +Repository baseline: `3c7def1e35871209e68563b2ee88f36033aff76e` on `plans/on-prem-small-team-user-access` (PR #7, open; 134-file implementation, schema 62). + +Source roadmap: +- `plans/subsystems/on-prem-user-access-roadmap.md#m015--independent-authentication-and-authorization-threat-review` + +Long-term requirements: +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted; no new ADR needed unless an auth or lease boundary must change). + +Primary class: invariant + +## 1. Objective + +Perform a targeted adversarial security review of the on-prem identity, +session, model policy, provider routing and budget boundaries newly +introduced by PR #7; strengthen tests or issue **separate bounded +correctives** for actual defects. This is a release-gating review +with explicit negative evidence, not a claim that green integration +tests constitute a complete security audit. + +## 2. Why this milestone is ready + +M001–M013 implementation and additive closure evidence exist at +`3c7def1e35871209e68563b2ee88f36033aff76e`. There is no hard dependency on M014's unrelated +TLS dependency patch for the **review** itself. Run this work in +parallel with M014. M016 final merge-readiness requires both their +closure records. An independent reviewer should inspect the +actual diff, not merely accept earlier closure prose. + +## 3. Current implementation evidence and blind spots + +PR #7 introduces identity/session tables (migrations 0055–0057), +`rust/src/identity.rs`, `web_session.rs`, `model_policy.rs`, +`user_budget.rs`, `instance_budget.rs`; ingress +`rust/src/server/middleware.rs`, `auth.rs`, coordinator +`endpoints.rs`, `publication.rs`, `finalization.rs`, +`reconciliation.rs`, and `dashboard/admin.rs`. M008/M009 hard/ +estimated rules, M010 phantom-hold crash fix, M011 sign-in/CSRF +fix, M012 physical testing and M013 transcode-bridge fix have +separate closure records. However, those tests are largely +implementation-owned and there has not yet been an **independent +diff-wide, adversarial security acceptance** of the entire +134-file on-prem feature integration. Earlier browser tests found +an unusable sign-in flow; admission changes broke Messages +transcode despite green Rust tests. These show why a coherent +boundary-level negative review remains useful after milestone +closure. + +## 4. Invariants that must not regress + +- EggPool remains a **single on-prem instance** for an owner or small team, + with provider topology, upstream credentials, TLS trust, private LLM + endpoints and routing configuration controlled by operator TOML/environment; + user identities, profiles, sessions, audit and downstream quotas stay in + SQLite. No SaaS, org/tenant control plane, embedded reverse proxy, external + IAM, secret-bearing HTML or browser provider editor. +- Preserve generation leases, provider health/eligibility, request identity, + retry and streaming finalization, single SQLite gate/worker and WAL/NORMAL, + live/restart config authority, all supported Chat/Responses/compact/Messages + client and cross-protocol routes, and the canonical solo SBC installation. +- Authorization and budget failure must not become provider unhealthiness + or anonymous fallback. No unauthenticated web mutation, shared + `admin/admin` credential, bypass of profile/global/model denies, or + moving operator secrets to SQLite. +- All qualification uses fake provider tokens, temporary roots and redacted + artifacts; never production data or real user passwords. + +## 5. Scope + +### In scope + +- External attack surface, route/method authorization and browser + provenance/transport trust on direct LAN and trusted proxy. +- User lifecycle, initial bootstrap, legacy-key cutover, personal + credential/scope, sessions, last-admin invariant, reset/revoke. +- Requested/virtual/concrete/qualified/selector model authorization, + metadata discovery/profile and cross-surface transcode. +- Atomic hard-vs-estimated user and global budget admission, + retry/selector cost holds, crash/reconcile/restore and account removal. +- Cross-user read isolation, audit/redaction and resource abuse + on constrained hardware; explicit privacy posture for local models. +- Minimal code/test changes for proven gaps; severity rubric and + explicit closure recommendation. + +### Explicitly out of scope + +No hosted SaaS, SSO, IAM enterprise RBAC, frontend rewrite, TLS dependency +upgrade (M014), provider TOML web editor, unmeasured Pi MMC tuning, +storage architecture redesign, or unlimited exploratory pen-testing. +No claim of formal certification or absolute absence of vulnerabilities. + +## 6. Required production/test changes + +A. Construct a full **route-method-credential matrix** from +`server/mod.rs`, `middleware.rs` and dashboard/API routers, +covering anonymous, legacy shared token, admin browser, member browser, +admin/member agent tokens, disabled/revoked user, invalid CSRF, and +path/method variants including normalized/encoded paths. Add negative +tests in `rust/tests/server_transport.rs`, +`on_prem_sessions.rs`, `on_prem_admin_ui.rs` for any uncovered +privilege surface. Unknown `/api/*` must never inherit public +dashboard access. + +B. Threat-model the boundary of `trusted_proxy_ips` and +`x-forwarded-proto`, duplicate `Host`/`Origin`/`Referer` +headers, real browser same-origin submissions, Secure/SameSite/ +HttpOnly cookies, login throttling and Argon2 worker memory cap +(19 MiB/two workers). Test HTTP LAN password rejection by default, +loopback allowed posture, and spoofed/proxied TLS acceptance. +Do not conflate LAN with authenticated confidentiality. + +C. Verify stable `user_id` and credential fencing across +provider account selection, fallback, alias, semantic selector, +compact/Responses/Messages/transcode, configuration reload, +revocation, password reset, request-ID collision and finalization. +A user must not obtain another member's usage/model list or +ever force a forbidden concrete model through routing. Include +`runtime-status`, `/v1/models`, `/api/user-budget`, +`/api/integrations/v1/profile` and admin password reset/token +reveal/usage/audit. + +D. Independently trace **one logical request** through all +budget scopes on concurrent last-slot, multi-user retry, +selector child, hard USD tariff provenance, estimated-mode +fallback, output-token bound mismatch, cancellation, process +death and restart. Assert no provider submission on unprovable +strict rule, no false hard cost from unknown self-hosted price, +no stranded budget hold, and no policy metadata leaks in 429/403. +The M010 recovery fix should fail under a deliberately reverted +two-line change and pass with it restored, if safe and supported +by the test harness; otherwise document the proof limitation. + +E. Secret and abuse review: password/token plaintext lifetime, +one-time display, database hashed storage, audit/session backups, +query-parameter and HTML/cache leakage, `Debug` logs, browser +snapshots, request body admission before password validation, +token selector brute-force, request/login rate limiting and +SQLite gate amplification. Non-redacted immutable artifacts are +a blocking finding. + +## 7. Ordered work packages + +1. Freeze PR diff/commit, inventory every HTTP endpoint and + principal/owner authority, enumerate high-value threat cases. +2. Run focused adversarial negative tests; record gaps with + severity and direct reproduction, not speculative assertions. +3. Fix only bounded in-scope defects with code+regression test; if + root cause crosses auth ADR or persistent trust boundary, stop + and write a new architecture/corrective plan. +4. Produce independent threat/evidence review, verify default/ + no-default and full tooling gates (Q007 as standard), preserve + historical closures and promote release integration M016 only + when both M014 and M015 are accepted. + +## 8. Failure, cancellation, restart and contention semantics + +Evaluate credential/user disable between middleware resolution and +publication; a **new** request must not start with stale privileges. +Already admitted streams may complete under their immutable policy +snapshot, with budget ownership retained. A contested last global +slot cannot admit two members. Forced process exit during publish +must recover without freeing actually incurred upstream spend. +Failed auth/database read rejects rather than falling back to +legacy/loopback. Hash throttling and session lookup cannot hold +the SQLite gate while Argon2 runs or create unbounded attacker- +controlled memory. + +## 9. Compatibility and migration + +No new auth contract is assumed; preserve historical one-user +`server.api_key` upgrade behavior only in explicit legacy state, +personal `EGGPOOL_API_KEY` tokens, browser-only sessions, +and source-owned provider TOML. If a security fix changes default +network posture, legacy key rights, CSRF behavior, token privileges +or API error shapes, stop for an ADR or scoped compatibility +decision before implementation. Schema changes require append-only +checksum-backed migration (>62) and an independent restored-schema +compatibility test; absent a finding, no migration is expected. + +## 10. Required tests + +Cross protocol/credential negatives: +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_identity -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_sessions -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_admin_ui -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_attribution -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_model_policy -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_global_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_finalization -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +uv run pytest tests/tooling/test_qualification_live_provider.py -q +``` +Review M011 real-browser artifact/manifest; re-run browser gate if +mutated HTML/session middleware changes, not merely DOM assertions. +Run at least one synthetic nonloopback reverse-proxy scenario and +a complete direct-local no-proxy scenario. Cover failed credential +brute-force and duplicate provenance headers with bounded synthetic +inputs. + +## 11. Required verification commands + +Record commands/results **actually executed** at the implementation SHA: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- --test-threads=1 +uv sync --frozen +uv run ruff check scripts/ tests/tooling/ +uv run pyright scripts/ +uv run pytest tests/tooling/ -q --tb=short --maxfail=1 +git diff --check +``` +Use `cargo deny check` with the repository's root `deny.toml` +and verify the command/working directory that the GitHub audit workflow +actually invokes. Run `cargo build --manifest-path rust/Cargo.toml --locked +--release` for production-affecting Cargo changes. The hosted CI, +installer-shell and dependency-audit jobs must be green on the final +head SHA; no earlier green check is a substitute. + +## 12. Documentation updates + +New security review closure, `architecture/deep-dive-security.md` +and `docs/deployment.md` only for actual changed behavior or threat +model gaps; release docs validator if touched. Preserve public claims +about Pi 5 storage tails from M012 and no remote access support. + +## 13. Acceptance criteria + +A reproducible route/principal matrix covers all existing public +and private entry points and no cross-role, cross-user, cross-model +or budget bypass remains. The negative/failure corpus demonstrates +managed-mode key separation, CSRF and trusted proxy checks, +authorization at concrete dispatch and stable user-budget settlement. +No high/critical findings, and no unaccepted medium **security +correctness** findings. Every defect is either fixed with regression +tests or has a **separate blocked merge** disposition; negative +coverage is independent of the implementation's previously chosen +happy paths. Full Rust/tooling gates green on the review head. + +## 14. Stop conditions + +Stop when an actual violation would require a new authorization ADR, +provider credential ownership shift, unsafely permissive default, +cross-transaction write, or architectural rewrite. Mark M015 +`corrective pass required` and open the narrow follow-up rather +than glossing over an unresolved security finding. + +## 15. Closure evidence required + +`plans/closure/on-prem-user-access/015-status.md`: source SHA, +reviewer-boundary matrix, threat assumptions, attack input/test +identity, reproducible fail/pass fixtures, source edits, exact serial +commands, local vs hosted CI results, leaked-secret scan, +severity-ranked open risks and M016 dependency promotion audit. +Do not back-edit M001–M013 closure records. + +## 16. Handoff notes + +This review may run **in parallel** with the M014 TLS patch. +Do not merge PR #7 based solely on green integration tests; +M016 owns the final SHA/merge gate once security and audit close. diff --git a/plans/implementation/on-prem-user-access/016-final-pr7-merge-reconciliation.md b/plans/implementation/on-prem-user-access/016-final-pr7-merge-reconciliation.md new file mode 100644 index 000000000..cdb8bd286 --- /dev/null +++ b/plans/implementation/on-prem-user-access/016-final-pr7-merge-reconciliation.md @@ -0,0 +1,255 @@ +# On-Prem User Access M016 — Final on-prem integration and PR #7 merge-readiness reconciliation + +Status: implemented + +Repository baseline: `3c7def1e35871209e68563b2ee88f36033aff76e` on `plans/on-prem-small-team-user-access` (PR #7, open; 134-file implementation, schema 62). + +Source roadmap: +- `plans/subsystems/on-prem-user-access-roadmap.md#m016--final-integration-and-merge-readiness-reconciliation` + +Long-term requirements: +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Applicable ADR: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted; no new ADR needed unless an auth or lease boundary must change). + +Primary class: polish + +## 1. Objective + +Produce one auditable pre-merge recommendation for PR #7 with its +current implementation, security review, fixed TLS dependency, hosted +CI status, accurate planning registry, release docs and documented +Pi 5 concurrency envelope. Do **not** merge automatically. This +is an integration/closure gate, not a duplicate Rust implementation +milestone or a new performance promise. + +## 2. Readiness and remaining operational gates + +The hard dependencies M014 (Rustls advisories resolved + hosted +audit green) and M015 (independent authz/threat review accepted) +are closed. All older on-prem M001–M013 work remains supported by +its historical closure records. This plan must use the exact PR +candidate's checks rather than inherit a stale green badge from a +prior SHA. **Operational:** final-candidate hosted +CI/installer/dependency audit, browser parity fixtures and release +artifact qualification remain part of M016 closure. The separate +optional Pi MMC tail-latency investigation does **not** block this +plan; docs must retain its measured restricted operating envelope. + +## 3. Current implementation evidence and why extra gate is needed + +As of `3c7def1e35871209e68563b2ee88f36033aff76e`, PR #7 is open, mergeable at a Git tree level, +but check state is `unstable` because the dependency audit +fails. It has 134 changed files / 58 commits and includes +new identity, policy, quota, dashboard, recovery and planning +code. M012 measured physical Pi 5 under MMC storage: solo +p50 ~4–5 ms, low idle RSS, 49 **named** identities provisioned, +but concurrent p95 up to ~14 s and 8-way saturating; do not +market 49 concurrent inference sessions. M011 restored browser +sign-in, new-page browser reference and unchanged historical +Python parity count; M013 fixed a routing eligibility line +lost in M008 that Rust unit tests missed. These are integration +risks to verify at the *final merged candidate*, not an +excuse to reopen correct historical closure records. + +## 4. Invariants that must not regress + +- EggPool remains a **single on-prem instance** for an owner or small team, + with provider topology, upstream credentials, TLS trust, private LLM + endpoints and routing configuration controlled by operator TOML/environment; + user identities, profiles, sessions, audit and downstream quotas stay in + SQLite. No SaaS, org/tenant control plane, embedded reverse proxy, external + IAM, secret-bearing HTML or browser provider editor. +- Preserve generation leases, provider health/eligibility, request identity, + retry and streaming finalization, single SQLite gate/worker and WAL/NORMAL, + live/restart config authority, all supported Chat/Responses/compact/Messages + client and cross-protocol routes, and the canonical solo SBC installation. +- Authorization and budget failure must not become provider unhealthiness + or anonymous fallback. No unauthenticated web mutation, shared + `admin/admin` credential, bypass of profile/global/model denies, or + moving operator secrets to SQLite. +- All qualification uses fake provider tokens, temporary roots and redacted + artifacts; never production data or real user passwords. + +## 5. Scope + +### In scope + +- Freeze PR head/base, inspect changed-file manifest and compare + all M001–M016 plan/closure claims to actual code and CI. +- Confirm M014 rustls security resolution and M015 threat review. +- Re-run the full combined auth/model/budget/transcode/UI/installer + compatibility matrix on the final locked dependency graph. +- Validate clean merge calculation, accurate release documentation, + operator migration and rollback, version/release provenance and + safe one-owner onboarding; update stale PR title/description. +- Final reviewer approval or explicit hold with exact failed gate. + +### Explicitly out of scope + +No automatic merge, forced history rewrite, packaging release, +SaaS roadmap, fresh whole-codebase performance refactor, auth +redesign, new enterprise or cluster features, or claim of +on-prem 49-way throughput. + +## 6. Required production/documentation changes + +No production change is expected if M014/M015 close cleanly. +If merge review finds a **new** release-blocking violation, +write an independently bounded corrective plan and keep M016 +blocked; do not apply opportunistic major rewrites. + +Fix PR #7 metadata: current title/description still describes +"M008–M012 registered" despite M008–M013 implemented and M012 +closed on hardware. Explain precisely that implementation is +on the branch and not yet in `main`, that audit/security +correctives supersede the old claim, and that 49 accounts +are distinct from active inference concurrency. Update docs +only when source-tested claims materially changed. Ensure +status markers in `plans/registry.md` and roadmap show +M014/M015 closed only after accepted closure evidence, then M016 +closed only after exact-head verification and its explicit merge +recommendation are recorded. + +## 7. Ordered work packages + +1. Merge preflight — list final base and head SHA, commit/change + diff, migration inventory 54→current and additive 61/62, + release/hardware artifact hashes and public API/CLI changes. +2. Verify closure dependencies — independent M014 hosted + audit + M015 security matrix, confirm no ignored RustSec + advisories or privilege/quotas findings. +3. Execute combined full Rust default/no-default, Python + tooling/Q007, UI/browser parity and installer-shell tests + on one immutable candidate; check merge conflict and test + actual merge result if branch base has moved. +4. Reconcile PR description/review checklist, source-backed + README/deployment/model-limits/Raspberry Pi claims, and + planning roadmap/registry. Produce an explicit + `merge recommended` or `merge held` disposition; only + maintainer decides to merge. + +## 8. Failure, cancellation, restart and contention semantics + +Prove combined latest code retains M010 startup recovery +release of abandoned **user and global** holds, no double-charge, +unattributed legacy control, disabled-user fencing, TLS negative +certificate failures, no shared-key browser privilege, and +cross-protocol transcode. The dashboard browser must still sign +in with real Origin+Referer pair; privilege/CSRF tests remain +green across different roles. No SQLite gate-held provider I/O; +outstanding physical Pi tail regressions stay clearly documented. +CI of the final head must be genuinely green; rerun flaky +jobs with documented first-failure cause, not a green-only +report. + +## 9. Compatibility and migration + +Validate old schema-54 restore/upgrade through schema 62, current +personal token/role/bootstrap, `--locked` release, and +agent integrations via `EGGPOOL_API_KEY` against both +loopback and trusted TLS-proxy posture. The shared +`server.api_key` remains migration-only as documented; +preserve provider TOML, private LLM choice and solo SBC +installation. Downgrading requires matching pre-upgrade +DB backup, not down-migration. No new schema in this plan. + +## 10. Required tests + +Focused combined tests: +```bash +cargo test --manifest-path rust/Cargo.toml --test on_prem_admin_ui -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_sessions -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_model_policy -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test on_prem_global_budgets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test database_compatibility -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test coordinator_finalization -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --test server_transport -- --test-threads=1 +uv run pytest tests/tooling/test_qualification_live_provider.py -q +uv run python scripts/validate_release_docs.py +``` +Use `scripts/qualification_dashboard_parity.py` with its +actual supported flags and pinned oracle/reference for +historical/new pages; confirm documented M011 reproducibility. +Include tests for installed release binary after dependency bump, +not only `cargo test` debug artifacts. + +## 11. Required verification commands + +Record commands/results **actually executed** at the implementation SHA: +```bash +cargo fmt --manifest-path rust/Cargo.toml --all -- --check +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets -- -D warnings +cargo check --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features +cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- -D warnings +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets -- --test-threads=1 +cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --no-default-features -- --test-threads=1 +uv sync --frozen +uv run ruff check scripts/ tests/tooling/ +uv run pyright scripts/ +uv run pytest tests/tooling/ -q --tb=short --maxfail=1 +git diff --check +``` +Use `cargo deny check` with the repository's root `deny.toml` +and verify the command/working directory that the GitHub audit workflow +actually invokes. Run `cargo build --manifest-path rust/Cargo.toml --locked +--release` for production-affecting Cargo changes. The hosted CI, +installer-shell and dependency-audit jobs must be green on the final +head SHA; no earlier green check is a substitute. +## 12. Documentation updates + +Update `README.md`, `docs/deployment.md`, +`docs/agent-configuration.md`, `docs/model-limits.md`, +`docs/raspberry-pi.md`, `docs/backup-restore.md`, +and `architecture/deep-dive-security.md` **only if necessary** +to reconcile actual implementation. Update PR #7 description +and merge checklist; never misstate 8-way Pi performance. + +## 13. Acceptance criteria + +At the exact PR SHA intended for merge, default/no-default +serial Rust and local tooling tests pass, hosted CI + Linux/ +macOS installer + cargo-deny all pass, security review has no +unresolved high/critical/medium authorization or privacy +defect, Rustls advisories are not ignored, dashboard role and +historical parity are qualified, migration/restore remain +sound, and no in-repo source claim exceeds measured Pi 5 +capacity. Planning links/closures align and PR description +matches the diff. **Merge recommendation is explicit, but the +plan never performs the merge**. + +## 14. Stop conditions + +Any failed dependency audit, missing M015 closure, unreviewed +new attack surface, unqualified schema rollback, red hosted +integration, stale docs claiming a 50-concurrent guarantee, +or unresolved merge conflict => hold PR, open scoped fix. +If the main branch advanced materially, rebase/merge-test +the actual intended integration base under maintainer +policy instead of assuming the old base. + +## 15. Closure evidence required + +`plans/closure/on-prem-user-access/016-status.md`: final +PR/base/head SHA, diff/changed-file and migration manifest, +evidence links to 014/015 closures, complete local/hosted +job URLs and results at final SHA, mergeability/approvals, +security matrix disposition, legacy/managed recovery, +browser/oracle, hardware envelope caveat, docs/PR metadata, +severity-tagged findings and registry unblock audit. +Only `closed` with evidence may restore the on-prem +subsystem roadmap to `closed`. + +## 16. Handoff notes + +Run only after M014/M015 closure. Do not merge or release +automatically, and do not mark M016 closed before the +full integration head checks pass. The independent MMC +tail-latency workstream is follow-up research, not a +backdoor release blocker or a license to violate WAL +architecture invariants. diff --git a/plans/registry.md b/plans/registry.md index b6f2a6cf7..34d496af0 100644 --- a/plans/registry.md +++ b/plans/registry.md @@ -14,9 +14,10 @@ Canonical direction: Legacy archive (pre-251, immutable, top level): `plans/001-*` through `plans/250-*` plus `python_hotpath_dispatch_compression_optimization.md`. -Most recently closed: Deployment and Packaging M005 (active PATH integration -detection corrective; 78-case harness, hosted Linux Bash + macOS arm64 zsh -passed; `plans/closure/deployment-packaging/005-status.md`). +Most recently closed: On-prem Concurrency Performance M001 (conditionally +closed after three paired physical Pi 5/MMC and tmpfs runs; durable +publication/SQLite/storage attribution; remaining controls in +`plans/closure/on-prem-concurrency-performance/001-status.md`). Legacy archive latest: Plan 250 (EggServe 0.3.0 direct-Tower migration, `7879cbf9`). Plans 244–245, 215–220, 241 remain historical per their own closure passes; the `146-*` duplicate @@ -49,6 +50,8 @@ pair is a known numbering accident. | Persistence | active | `plans/subsystems/persistence-roadmap.md` | M011 ready — post-M010 roadmap/documentation reconciliation | M009/M010 closures accepted. Documentation/tooling-only corrective; production remains WAL/NORMAL. No technical persistence successor is registered. | | Deployment and packaging | closed | `plans/subsystems/deployment-packaging-roadmap.md` | M005 closed — active PATH integration detection corrective | M001–M005 closed; no successor registered. | | Dashboard | closed | `plans/subsystems/dashboard-roadmap.md` | M012 closed — lifecycle closure and documentation polish | M001–M012 closed; no registered successor; future dashboard work requires a new bounded plan. | +| On-prem user access | active | `plans/subsystems/on-prem-user-access-roadmap.md` | M016 closed — merge held pending maintainer review | M014/M015 and exact M016 candidate checks are closed/passed; PR #7 remains open pending maintainer review. M001–M013 history preserved; physical Pi 5 restricted envelope remains qualified only for solo/light concurrency. | +| On-prem concurrency performance | active | `plans/subsystems/on-prem-concurrency-performance-roadmap.md` | M001 conditionally closed — Pi 5/microSD durable-storage tail attribution | Optional follow-up, **nonblocking** to PR #7 merge. Controlled competing-I/O and account/budget matrix evidence remain before broader claims or optimization. SSD/NVMe is optional, not a closure dependency. | ## Dependency-ready implementation plans @@ -56,8 +59,6 @@ pair is a known numbering accident. |---|---|---|---|---| | Persistence | M011 — post-M010 roadmap and documentation reconciliation | ready | `plans/implementation/persistence/011-post-m010-roadmap-documentation-reconciliation.md` | Hard: M009/M010 closures accepted. Documentation/tooling only; no runtime or physical target dependency. | - - ## Active implementation plans | Subsystem | Milestone | Status | Implementation plan | Handoff note | @@ -69,6 +70,23 @@ pair is a known numbering accident. | Subsystem | Milestone | Blocker | |---|---|---| + +Unblock audit (on-prem user access reopening after M012 closure, baseline +`3c7def1e`): M001–M013 closure records remain immutable; no +historical milestone was reopened. M014 is closed after Rustls +0.23.46 passed exact-SHA hosted cargo-deny, CI, and installer jobs. +M015 is closed on `d8fd9c2a`; its independent review findings were +fixed, re-reviewed, and covered by local default/no-default suites and +exact-SHA hosted CI, installer and dependency-audit jobs. Registry audit +found M016 is the only registered successor blocked on M015, so M016 is +promoted to ready. M016 still needs final merge-readiness reconciliation. +Independent +on-prem concurrency performance M001 is ready as nonblocking +diagnostic research: the Pi 5/MMC G3 tail-latency failure is +documented, not waived or treated as a release-blocking +throughput commitment. No unrelated ready/active milestone +is promoted or blocked by this registration. + Explicit user direction opens Shared Provider Profile Contract M001 at baseline `b73eea6bc4ecdb6ce6b8974769b7821cf5e7e036`. The milestone extracts the existing secret-free provider endpoint/wire/auth/discovery/verification metadata into a sibling-consumable sans-I/O crate, keeps all account/credential/routing/runtime ownership in EggPool, and adds reviewed non-fixed OpenCode Go model-wire hints. CodeGG is an operational downstream only and must consume an immutable closure revision; no sibling repository is treated as first-party provider authority. Persistence M008 closure audit: searched the registered Persistence plans and dependency tables; no future plan depends on M008 and the blocked-work table has no Persistence entry. Nothing was unblocked or promoted. A production storage split or further journal experiment requires a separately approved bounded plan. @@ -82,6 +100,20 @@ Historical M007 blocker assessment at baseline `7e241ad` (`plans/closure/persist | Subsystem / plan | Disposition | Evidence | |---|---|---| +| On-prem user access M016 — final PR #7 integration and merge-readiness reconciliation | closed — local Rust/tooling, Q007, release-doc, browser and installed release-binary evidence pass; exact PR head `3d68c715` hosted CI, Linux/macOS installer and cargo-deny pass; merge held because no maintainer PR review has been submitted; no future plan depends on M016 | `plans/closure/on-prem-user-access/016-status.md`, PR #7, base `9a0a903c`, head `3d68c715`, CI `38721601336`, installer `38721601305`, audit `38721601318` | +| On-prem user access M015 — independent authentication/authorization threat review | closed — independent review findings on hard-cost proof bounds and browser-session access corrected and re-reviewed; default 1,024 / no-default 1,025 tests pass (one ignored each); exact-SHA CI, installer and dependency audit pass on `d8fd9c2a`; no M015 security findings remain; M016 promoted to ready | `plans/closure/on-prem-user-access/015-status.md`, implementation `a526980c`, `deb1622f`, `d8fd9c2a`, security scan `52bd79cc`, CI `38720507683`, installer `38720509034`, audit `38720507720` | +| On-prem user access M014 — Rustls dependency security-advisory corrective | closed — lockfile-only Rustls 0.23.46 patch, unchanged transport feature graph; local default/no-default and 1,009/1,010-test workspace suites, provider TLS (trusted root, wrong-root and hostname rejection), budget/coordinator suites, release build and tooling pass; hosted cargo-deny, CI and both installers pass on `7462263ec482c5fda4bd1a70a9a55b0efc313cd3`; main remains pending PR merge | `plans/closure/on-prem-user-access/014-status.md`, implementation `7462263e`, audit `38671345587`, CI `38671345679`, installers `38671345597` | +| On-prem concurrency performance M001 — Pi 5/microSD concurrent tail attribution | conditionally closed — three paired physical microSD/tmpfs runs identify durable publication/SQLite/storage as the dominant tested four-way tail contributor; no production change or capacity claim; SSD/NVMe is not required | `plans/closure/on-prem-concurrency-performance/001-status.md`, six Q008 reports + manifest under `artifacts/qualification/on-prem-concurrency-performance/`, artifact guard `tests/tooling/test_on_prem_mmc_tail_attribution.py` | +| On-prem user access M012 — physical SBC small-team qualification and final disposition | closed — Pi 5 managed envelope measured (functional 25 cells + 3×30 benchmark batches all pass, 145/145 each, 0 pending/active; Argon2id 51 ms 1-way / 71–73 ms 2-way; 49 named users at 51 ms p50 with spot logins passing; concurrency 1/2/4/8 completes with 8-way saturating at p50 4.1 s); runner managed-auth corrective + 2 contract tests; solo practical, 49 users inexpensive metadata, no 50-concurrent claim; one medium (MMC tails exceed Sep-21 gates, envelope restricted) + six low findings, no high | `plans/closure/on-prem-user-access/012-status.md`, artifacts `artifacts/qualification/on-prem-m012-pi5-functional.json` + `on-prem-m012-pi5-benchmark-run-{1,2,3}.json`, docs `docs/raspberry-pi.md` envelope | +| On-prem user access M011 — admin/member dashboard browser qualification | closed — frozen Python oracle rebuilt and run against head with 54 parity gaps, identical to the pre-M011 baseline (0 new, 0 resolved); on-prem admin/member pages qualified against a Rust-owned approved reference (70 observations, 22 captures) covering both roles, the negative matrix, no-JS sign-in, HiDPI/mobile/themed viewports, and a DOM credential-leak guard; browser sign-in, top-nav overflow at 1280px, unstyled usage pages, missing viewport declarations, unbounded tables, and budget labelling all corrected; 967 serial Rust tests and 175 tooling tests pass at the M011 baseline (re-verified post-integration — see integration note) | `plans/closure/on-prem-user-access/011-status.md` | +| On-prem user access M010 — budget crash recovery and schema-54 restore | closed +| On-prem user access M007 — small-team qualification, migration, and release hardening | conditionally closed — local 964 serial Rust tests + 166 tooling tests; hosted CI `38497388228`, installer qualification `38497388225`, and dependency audit `38497388244` pass; seven synthetic loopback requests complete with no reservations left; physical Pi/Le Potato capacity and schema-54 restore remain unmeasured, so no 49-user or concurrency claim | `plans/closure/on-prem-user-access/007-status.md`, local artifact `artifacts/qualification/on-prem-user-access-m007-local.json` | +| On-prem user access M006 — web administration and personal client integration | conditionally closed — migration 60, session/role/CSRF-fenced admin and member screens, one-time/revocable personal credentials, bounded audit/list reads, 964 serial Rust tests + 166 tooling tests; browser parity captures and M005 estimate/physical evidence carried into M007 | `plans/closure/on-prem-user-access/006-status.md`, implementation `a3f2d00a` | +| On-prem user access M004 — profile-based model authorization | closed — migration 58, exact profile and user policy, concrete-dispatch checks, filtered model discovery and policy ETag; default/no-default compile gates and 957 all-target tests passed; M005 promoted to ready | `plans/closure/on-prem-user-access/004-status.md`, implementation `f537df4`, concurrency fixture `c53bf4d` | +| On-prem user access M005 — durable user/model budgets | conditionally closed — migration 59, transactional holds, retry/terminal settlement, bounded authenticated usage and 962 serial workspace tests; exact output/USD caps, global scope, crash/restore tests and overhead measurement remain explicit conditions; M006 promoted to ready | `plans/closure/on-prem-user-access/005-status.md`, implementation `49dc6ce` | +| On-prem user access M003 — principal propagation and attribution | conditionally closed — typed principal and durable logical request ownership across finite, stream, compact, retries, and selector child; 953 integration tests passed, 1 ignored; two unrelated update unit-test failures and physical measurement carried as explicit conditions; M004 promoted to ready | `plans/closure/on-prem-user-access/003-status.md`, implementation `9af8787a` | +| On-prem user access M002 — browser sessions and route authorization | closed — opaque sessions, CSRF, role-separated managed routes, trusted-proxy transport checks; 947 Rust tests passed, 1 ignored; M003 promoted to ready; SBC hash qualification remains with M007 | `plans/closure/on-prem-user-access/002-status.md`, implementation `087953a3` | +| On-prem user access M001 — admin bootstrap and client credentials | conditionally closed — migration 55, local admin and personal-token lifecycle, legacy-key cutover, 939 workspace tests passed; physical SBC Argon2 qualification carried to M007; M002 promoted to ready | `plans/closure/on-prem-user-access/001-status.md`, implementation `ecdc313` | | Deployment and packaging M005 — active PATH integration detection corrective | closed — 78-case installer harness (73 baseline + 5 new Gregg/array regressions), Linux Bash and macOS arm64 zsh hosted qualification, full CI; no dependent plan was unblocked | `plans/closure/deployment-packaging/005-status.md`, implementation `576b462d`, CI `38030165909`, shell run `38030165862` | | Deployment and packaging M004 — shell command discovery and activation corrective | closed — 73-case installer harness, Linux Bash and macOS arm64 zsh hosted qualification, full CI; no dependent plan was unblocked | `plans/closure/deployment-packaging/004-status.md`, implementation `57af5bc` + test hardening `45149d2`, CI `38026958264`, shell run `38026958268` | | Provider transport M006 — Eggress 1.0.12 dependency refresh and requalification | closed — all 16 registry-resolved Eggress crates at 1.0.12; provider/coordinator/wire and default/no-default workspace gates passed; release binary +520 bytes (+0.0017%); hosted CI and dependency audit green; no plan unblocked | `plans/closure/provider-transport/006-status.md`, implementation `d8be1599` | @@ -382,3 +414,100 @@ selection, runtime efficiency, shared provider profile contract, and dashboard remain closed with no successor. No deployment-packaging successor is registered; future hardening (attestations, system/root distribution, additional targets) requires new bounded plans. + +Unblock audit (on-prem user access M001 conditionally closed, implementation +`ecdc313`, `plans/closure/on-prem-user-access/001-status.md`): searched +`plans/implementation/on-prem-user-access/`, the subsystem dependency graph, +and the ready/active/blocked registry tables. M002's hard M001 credential and +schema dependency is satisfied by the accepted closure and was promoted to +ready. M003 remains blocked on M002 and its stable principal/scope contract; +M004 remains blocked on M003 attribution; M005 on M004 policy gates; M006 on +M005 durable budgets; M007 on M006 and physical qualification. No other +registered plan depends on M001. The M001 Argon2 SBC measurement condition is +carried into M007, so promotion of M002 does not assert physical sizing. + + +Unblock audit (on-prem user access M006 conditionally closed, implementation +`a3f2d00a`, `plans/closure/on-prem-user-access/006-status.md`): searched +`plans/implementation/on-prem-user-access/`, the roadmap dependency graph, and +the dependency-ready, active, and blocked registry tables. M007's hard M006 +dependency is satisfied and it is promoted to ready. Its physical SBC and +ordinary-host measurements remain operational evidence required for M007 +closure; they do not block preparatory compatibility and recovery work. No +other registered plan declares M006 as a dependency, so no other plan was +promoted. M005 estimate/global-scope/recovery findings and M001 Argon2 hardware +measurement remain carried conditions. + + +Unblock audit (on-prem user access M007 conditionally closed, implementation +`a3f2d00a`, `plans/closure/on-prem-user-access/007-status.md`): searched +`plans/implementation/on-prem-user-access/`, the roadmap dependency graph, and +the dependency-ready, active, and blocked registry tables. M001–M007 are +terminal and no successor or downstream plan declares this workstream as a +dependency. No plan was promoted. The physical SBC, exact schema-54 restore, +dashboard parity, M005 budget, and hosted-CI conditions remain visible in the +closure; no named-user capacity claim is supported by the local synthetic run. + + +Unblock audit (on-prem user access M011 closed, +`plans/closure/on-prem-user-access/011-status.md`): searched +`plans/implementation/on-prem-user-access/`, the roadmap dependency graph, and +the dependency-ready, active, and blocked registry tables. M011 left the +dependency-ready table. At M011's own baseline its hard-dependency note held +(M008 still ready, M009/M010 blocked on it); on integration into this branch, +M008/M009/M010/M013 are all accepted closures, so M012's hard dependency +(M008–M011 accepted closures) is now fully satisfied and M012 stays blocked +solely on its operational requirement (provisioned physical SBC and reference +ordinary Linux target). No plan was promoted: nothing else declares these +milestones as a dependency. M006's recorded browser-parity condition is +satisfied additively here; the immutable M006 record is not retroactively +altered. + +Integration note (M011 cherry-picked as `d495d483` from +`feat/auto-20261014-e526194d` onto this branch): the pick's registry and +roadmap edits were reconciled to current truth (M008/M009/M010/M013 already +closed here); the `render/admin.rs` budget-form conflict resolved in favour of +the mode-aware M008 form (the M011 labels predated enforcement modes) while +keeping M011's `on_prem_page`/`scroll_table` structural fixes; the M011 plan +file moved `ready` → `implemented` (activation was skipped in the separate +session — recorded here, not rewritten). + +Unblock audit (on-prem user access M012 blocked closure, +`plans/closure/on-prem-user-access/012-status.md`): searched +`plans/implementation/on-prem-user-access/`, the roadmap dependency graph, and +the dependency-ready, active, and blocked registry tables. M012's hard +dependency (accepted M008–M011 closures) is satisfied; the operational +requirement (provisioned physical SBC and reference ordinary Linux target) +is not, so M012 stays blocked and the roadmap stays `active`. The closure +pass lands only the tooling boundary-contract suite +(`tests/tooling/test_on_prem_sbc_qualification.py`, 9 tests) with zero +production diff; no plan is promoted and nothing else declares these +milestones as a dependency. No capacity, latency, Argon2, RSS, WAL, or +named-user claim is accepted. When hardware is provisioned, resume at plan +work packages 2–4 with predeclared gates. + +Unblock audit (on-prem user access M012 closed, +`plans/closure/on-prem-user-access/012-status.md`): searched +`plans/implementation/on-prem-user-access/`, the roadmap dependency graph, and +the dependency-ready, active, and blocked registry tables at this commit. +M012's hard dependency (accepted M008–M011 closures) was already satisfied; +the operational requirement is now met with Pi 5 managed evidence +(functional + 3× benchmark + Argon2 + 49 named users + 1/2/4/8 concurrency), +so M012 moves `blocked` → `implemented`/`closed` and the on-prem roadmap +moves `active` → `closed`. This pass lands the runner managed-auth corrective, +two contract tests (11 total), four Pi 5 artifacts, and the `raspberry-pi.md` +envelope with zero production diff. No plan is promoted: nothing else +declares M012 as a dependency, M013 is already closed, and the only ready row +(Persistence M011) is independent and unaffected. The blocked-work table is +now empty. Future hardware-class or higher-concurrency claims require new +bounded plans; this closure record is not to be rewritten. + +Unblock audit (on-prem concurrency performance M001 conditionally closed, +`plans/closure/on-prem-concurrency-performance/001-status.md`): searched +`plans/implementation/`, the subsystem dependency graph, and the dependency- +ready, active, and blocked registry tables. No registered plan declares M001 +as a hard, interface, soft, or operational dependency. No plan is promoted or +unblocked. On-prem user-access M016 is ready after M014/M015 closure and still +requires final-SHA gates; this optional diagnostic workstream has no edge +to that sequence. The concurrency-performance roadmap remains active for any +future separately scoped controlled-I/O/media/budget work; none is registered. diff --git a/plans/subsystems/on-prem-concurrency-performance-roadmap.md b/plans/subsystems/on-prem-concurrency-performance-roadmap.md new file mode 100644 index 000000000..fe819e101 --- /dev/null +++ b/plans/subsystems/on-prem-concurrency-performance-roadmap.md @@ -0,0 +1,198 @@ +# On-Prem Concurrency Performance Roadmap + +Status: active + +Long-term references: +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/000-long-term-specification.md#3-ownership-boundaries-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Related ADRs: +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) +- Existing persistence ADRs governing one SQLite worker and WAL/NORMAL + (do not revise without explicit new architecture decision). + +## 1. Purpose and ownership boundary + +Improve the *measured* concurrent latency envelope of one local/SBC +EggPool instance without regressing the fast single-user path or +introducing SaaS/cluster complexity. This is a **separate non-release- +blocking diagnostic workstream**, opened because the original on-prem +M012 Pi 5/MMC qualification exceeded predeclared tail-latency gates. +It does **not** retroactively reopen the M012 closure or block +on-prem M016 final security/merge-readiness reconciliation. It may +find a new bounded optimization opportunity; no such implementation +is authorized until evidence isolates cause and a plan is accepted. + +`scripts/qualification_sbc.py` and `scripts/qualification_stability.py` +own physical reproducible qualification. `rust/src/db/`, +`rust/src/coordinator/`, `rust/src/user_budget.rs`, +`rust/src/instance_budget.rs`, `rust/src/identity.rs` and +`rust/src/server/` own source evidence. Existing Persistence +roadmap and previously failed checkpoint experiments remain +the authority for storage policy; performance work here may not +move DB ownership or change WAL policy without a separate ADR. + +## 2. Work classification + +### Invariants + +- Single-server worker, one SQLite gate/worker WAL/NORMAL and + generation/finalization identity/fencing remain authoritative. +- Single-owner latency, requests, SSE/TTFT, recovery and model + authorization must not regress to improve concurrency averages. +- No unsupported capacity marketing: 49 named account rows do not + equal 49 simultaneous generations; measured Pi/MMC saturation + and tails remain documented. +- No real provider/user keys or live DB in test data. + +### Capabilities + +- None currently added. This is root-cause attribution, not a + concurrency/SLA guarantee. + +### Infrastructure + +- Bounded opt-in instrumentation of foreground DB gate, WAL and + external block-device contention for controlled qualification + only, with no steady-state hot-path allocation. + +### Polish + +- Precise tail-latency failure attribution and a supportable + recommendation (environment, operator guide, or next bounded + optimization/architecture plan). + +## 3. Non-goals + +No implicit second SQLite writer, checkpoint policy retune, async +persistence redesign, background worker migration, public WAN/TLS, +model/request-body rewrite, cluster capacity, web framework rewrite, +external DB/queue, forceful OOM testing on a shared Pi, or new +50-concurrent-request service claim. + +## 4. Current state + +The accepted `plans/closure/on-prem-user-access/012-status.md` +reports one Pi 5 8 GiB, ext4 on MMC, schema 62, 1 DB worker, +baseline RSS ≈20 MB and solo p50 ≈4–5 ms, but batch/concurrent +p95 10–14 s and 8-way p50 ≈4.1 s. G3 p95≤1 s, +max≤3 s was **failed**; no other severe correctness problem or +WAL convergence failure was reported. A loaded device had +background `opencode` CPU and `kblockd` / MMC disk pressure; +one transient D-state worker cleared. Earlier persistence M007/M008 +Pi/MMC WAL experiments already demonstrated why checkpoint +changes based on synthetic averages are unsafe. + +Potential causes (hypotheses, not findings): host/device I/O +contention, foreground serialized DB hold times, periodic +checkpoint or fsync, admission/budget query frequency, +login hashing, or external test workload. Physical evidence +must distinguish these before code is changed. + +## 5. Target architecture + +Keep current production layout. Use opt-in, low-overhead, +redacted phase and block-I/O instrumentation on a +**pre-provisioned isolated target** with paired baseline +and candidate, then reason quantitatively about p95/p99 +contributor/causal share. Only if a specific EggPool-owned +cost is isolated and the existing persistence safety gates +can be preserved should a new bounded implementation plan +be proposed (or a new ADR if ownership/journal changes). +Otherwise update hardware/storage/workload guidance without +claiming a runtime optimization. + +## 6. Dependency graph + +- Diagnostic M001: hard — none; input M012 closure accepted. +- Operational for M001 full closure: physical Pi 5-class + target using microSD/MMC boot storage, controlled host + activity and performance counters. SSD/external storage is + optional exploratory evidence, not a completion dependency. +- Soft/interface: ongoing Persistence roadmap owns any + architectural storage-policy experiment; a future M002 + must not be registered until M001 identifies measurable + cause and independently accepted constraints. +- **No dependency edge** from this subsystem to on-prem + M014/M015/M016 security or merge readiness. The current + documented small-team envelope is limited but truthful. + +## 7. Milestones + +### M001 — Pi 5/MMC concurrent tail attribution + +Class: polish. Conditionally closed with repeated physical microSD/MMC/tmpfs +attribution in `plans/closure/on-prem-concurrency-performance/001-status.md`. +The completed corpus repeats the existing four-way workload on the Pi's +microSD and tmpfs; Plan 238's database-only tmpfs comparison and Plans 237/239's +phase controls independently support the storage attribution. SSD/NVMe is not +required for M001; one supplemental NVMe cell is retained as exploratory +evidence only. Controlled competing I/O and repeated 1/2/4/8 plus named-user/ +budget cases remain conditions for future optimization or broader operating +claims. Source: +`plans/implementation/on-prem-concurrency-performance/001-pi5-mmc-tail-latency-attribution.md`. + +Future optimization milestone: **not registered**. Requires +separate scoped plan, architectural check and measurable +regression guard; not authorized by the diagnostic alone. + +## 8. Cross-cutting requirements + +- Same principal/model-policy and global/user budget scope, + hard/estimated mode, published request/response wire, + provider-account health and request-level idempotence. +- No production key exposure, no mutable tracing of + request bodies, identity, tokens, upstream URLs or cookies. +- Instrumentation opt-in, bounded, removable and disabled + by default; JSON artifacts redacted and independently + validated. +- Record CPU/scheduler activity, disk queue and fsync + contribution, SQLite gate wait/hold, WAL size/checkpoint, + request p50/p95/p99/max, stream TTFT and memory. +- Frozen thresholds **before** interpreting new runs; + preserve the original failed G3 result unedited. + +## 9. Verification strategy + +Use existing `scripts/qualification_sbc.py` with +on-prem personal-token bootstrap and Q007 managed-auth +compatibility, `tests/tooling/test_qualification_sbc.py`, +current M012 frozen artifacts, and exact target attestations. +Prefer 3 paired/repeated physical run groups with controlled +background conditions and bounded concurrency. Qualify any +added instrumentation with serial Rust default/no-default +and tooling tests; do not claim physical measurements from +a normal developer workstation. + +## 10. Risks and decision points + +- A shared development Pi with competing MMC writes + can dominate high percentiles; uncontrolled performance + changes risk false causal attribution. +- Earlier persistence checkpoint/split experiments were + rejected under WAL convergence/tail gates. Avoid + optimistic retuning that reopens those defects. +- Diagnostic-only shared-key topology modes still 401 + in managed context (M012 low finding); repair the + narrow synthetic token path if needed to get comparable + evidence without changing production auth. +- If evidence points to external block device rather than + EggPool, recommend faster storage/guidance, not a + speculative storage rewrite. + +## 11. Completion definition + +M001 accepted closure with reproducible physical artifacts +and an explicit causal/no-causal conclusion; scope can close +even when an environment-only tail remains, provided +the restricted Pi/MMC envelope continues to be documented. +No optimization promise or new 8-way SLA is inferred. + +## 12. Milestone status + +| Milestone | Status | Implementation plan | Closure record | Blockers | +|---|---|---|---|---| +| M001 | conditionally closed | `plans/implementation/on-prem-concurrency-performance/001-pi5-mmc-tail-latency-attribution.md` | `plans/closure/on-prem-concurrency-performance/001-status.md` | Four-way microSD/MMC tail attributed to durable storage; controlled competing-I/O and broader account/budget matrix remain conditions. SSD/NVMe is optional and not a closure dependency. No dependency edge to on-prem M016. | diff --git a/plans/subsystems/on-prem-user-access-roadmap.md b/plans/subsystems/on-prem-user-access-roadmap.md new file mode 100644 index 000000000..b65511614 --- /dev/null +++ b/plans/subsystems/on-prem-user-access-roadmap.md @@ -0,0 +1,392 @@ +# On-Prem User Access Roadmap + +Status: active + +Long-term references: + +- `plans/000-long-term-specification.md#1-product-definition` +- `plans/000-long-term-specification.md#2-end-state-invariants-normative` +- `plans/000-long-term-specification.md#3-ownership-boundaries-normative` +- `plans/001-terminology-and-domain-model.md#identity-and-lifecycle-terms` +- `plans/002-long-term-roadmap.md#phase-5--optional-on-prem-small-team-access-and-budgeting` + +Related ADRs: + +- `plans/adrs/ADR-0002-on-prem-user-identity-and-policy-boundaries.md` (accepted) + +## 1. Purpose and ownership boundary + +Provide optional accountable use of **one** on-prem EggPool instance by +small teams while preserving its one-user SBC/LAN workflow. `rust/src/server/` +owns HTTP credential extraction, browser sessions/CSRF and route-role gates; +new `rust/src/identity/` and `rust/src/user_policy/` (suggested boundaries) +own account and effective authorization policy services; `rust/src/db/` owns +user tables and transactional reservation state; `rust/src/coordinator/` +owns authenticated-request propagation, durable publication and settlement; +`rust/src/operations/` + `runtime.rs` own local admin CLI/onboarding; +`rust/src/server/dashboard/` owns thin operator/member projections; existing +`routing/`, `quota/`, `providers/`, and TOML `Config` continue owning +upstream accounts and routing. Do not create a provider-secret web editor. + +## 2. Work classification + +### Invariants + +- Single-instance on-prem; no SaaS organizations, public signup or upstream + credentials in user-owned SQLite tables. +- Every admitted request retains a trustworthy principal; denial precedes + upstream dispatch; aliases/selectors/cross-protocol requests cannot bypass + model restrictions. +- User budgets and upstream provider quotas remain independent. +- Preserve generation leases, one SQLite connection/gate/worker and + WAL/NORMAL, request publication/finalization ownership and redaction. +- No shared default admin password or exposed unauthenticated control path; + managed access never inherits public-dashboard/loopback shortcuts. +- One-user default has no mandatory external IAM or added server process. + +### Capabilities + +- First-run admin CLI bootstrap + local recovery; password login; revocable + per-user agent tokens; admin-created users; reusable model-policy profiles; + user/model budgets; members' own catalogs and usage; administration UI. +- Self-hosted provider endpoints configured in TOML remain first-class, with + usage counted even if no USD price exists. + +### Infrastructure + +- Identity/session/credential migrations; typed principal and immutable policy + snapshot; cached credential resolution; durable budget holds/settlement; + user-specific profile revisions and audit records. + +### Polish + +- Embedded dashboard UX, migration diagnostics, docs, single-user and + 1–49-user footprint/latency measurement on representative target classes. + +## 3. Non-goals + +No billing/chargeback, hosted control plane, tenant/organization segregation, +federated login/SSO, invite/email reset, native reverse-proxy/VPN/TLS manager, +remote account creation, public internet deployment guide, cloud model catalog +as authority, custom upstream per-user provider secrets, or a published +50-simultaneous-inference capacity promise. + +## 4. Current state + +M001–M007 were implemented on PR #7 from baseline +`c18dcc2051ae4987a2cb95a7cd046b8aff2095ff` and +each has a retained closure record. M002 and M004 are closed. +M001, M003, M005, M006 and M007 are **conditionally closed**, +not equivalent to release qualification. + +The current Rust implementation contains: +- Append-only SQLite identity/session/request-attribution/policy/budget/admin + migrations **0055–0060**; `rust/src/identity.rs`, `web_session.rs`, + `model_policy.rs`, `user_budget.rs`. +- `rust/src/server/middleware.rs` / `auth.rs`: personal agent tokens, + opaque browser sessions, CSRF and managed-route role fencing. +- `rust/src/coordinator/endpoints.rs`, `publication.rs`, + `finalization.rs`: per-request principal, concrete-model authorization + and downstream-user budget holds/usage settlement. +- `rust/src/server/dashboard/admin.rs` and `render/admin.rs`: + admin/member user, profile, budget, usage and audit views. +- `rust/tests/on_prem_*.rs` and successful local/hosted default CI; + the M007 synthetic provider artifact exercises seven loopback requests. + +Remaining verified gaps (see immutable closure records M001/M003/M005–M007): +- M005 token/USD hard-limit semantics are not proven from estimated holds; + unknown-price models can receive generic USD estimates; **global instance + ceilings are absent**. +- Budget-specific process-crash and actual schema-54 restored-backup + qualification, plus ordinary-host SQLite-gate overhead measurements, + were not performed. +- M006 new admin/member pages lack browser screenshot/interaction + qualification. The Python oracle applies to pre-existing pages only. +- Physical Pi/Le Potato Argon2/SQLite/WAL/TTFT and 1/5/10/25/49 **named** + user workloads are unmeasured. No concurrency/capacity claim is authorized. +- M003 records local `operations::update` failures; the latest hosted + branch CI is green, but reproducible platform classification is not yet + documented. + +Correctives M008–M012 were created by explicit user direction; +the predecessor closures remain immutable. This roadmap is **active** +until required corrective closure evidence exists. + +## 5. Target architecture + +One initial admin, more users only through admin-controlled creation. Local +CLI bootstraps/recover users. Browser password sessions and per-user API tokens +both resolve to an identity; credentials are not interchangeable. Layer +instance security > explicit user override > named profile > safe default +for effective policy, with hard deny/ceiling intersection. Expose +`/v1/models` and integration profile filtered by that identity. Freeze policy +for each authorized request and preserve stable user ID across retries and +terminalization; invalidate authorization for **new** requests immediately on +admin change, with documented in-flight behavior. Use a separate user-budget +reservation state machine over SQLite, durable compensations, and bounded user +analytics; never change provider score-only quota into user hard-cap logic. +Web UI uses existing embedded Rust HTML/CSS/JS; provider TOML remains local. + +## 6. Dependency graph + +- M001 (hard: accepted ADR only) -> M002 (hard: credential and session schema). +- M002 -> M003 (hard: principal/scope API stable). +- M003 -> M004 (hard: concrete-model attribution and principal propagation). +- M004 -> M005 (hard: effective policy semantics + atomic denial established). +- M005 -> M006 (hard: storage and quota API; soft: UI skeleton may be prepared). +- M006 -> M007 (hard: usable onboarding, admin/member web and client flow). +- Operational: physical SBC and normal-host evidence needed for M007 closure. +- Interface: existing published provider topology, generation leases, budget + finalization, scoped `eggpool-client-config` integration shape throughout. + +## 7. Milestones + +### M001 — Secure admin bootstrap and client credentials + +Class: infrastructure. No hard dependencies. Deliverable: schema, username+ +password first-run CLI/TTY or stdin, one admin, non-recoverable token storage, +legacy-compatible staged migration, local recovery; no browser login or +multi-user claim. Exit: no admin/admin, no remote bootstrap, old installations +still start, isolated credential tests and backups pass. Deferred: cookies, +principal routing, admin web. + +### M002 — Browser sessions and route authorization + +Class: capability. Hard: M001. Deliverable: login/logout/change-password, +opaque sessions, CSRF, authorized route classification and managed +public-dashboard/loopback restrictions. Exit: end-to-end browser auth, role +denials and unauthenticated/no-CSRF negative tests. Deferred: user model +policies, usage. + +### M003 — Request principal and durable user attribution + +Class: infrastructure. Hard: M002. Deliverable: typed principal through all +client surfaces, compact, selector plumbing, publication/finalization; user ID +on new requests, legacy-unattributed history explicit; user-filtered +read-only discovery plumbing. Exit: retry/restart/cancel/stream stable user +attribution; no client header identity spoofing. Deferred: grants/budgets. + +### M004 — Profile-based model policy + +Class: capability. Hard: M003. Deliverable: admin-assignable profiles, +per-model access and preferences, user overrides, consistent concrete, +qualified/virtual/semantic/compact admission; user-discovery filtered. +Exit: no bypass through selectors or variants; disabled users fail closed +and new policy version visible without process restart. Deferred: quotas/UI. + +### M005 — Transactional user and model budgets + +Class: capability. Hard: M004. Deliverable: per-user and per-model request, +token, cost and concurrency caps where enforceable; atomic holds, settlement, +exactly-once recovery, unknown-price behavior, bounded rollups. Exit: +simultaneous requests cannot over-admit within defined reservation bounds; +stream/retry/failure recovery and local-model token metering pass. Deferred: UI. + +### M006 — Administration and personal client integration + +Class: capability. Hard: M005. Deliverable: embedded admin users/profiles/ +budget/audit screens; member self-service password/tokens/usage; filtered +profile and `configremote`/`eggpool-connect` workflows, no member access to +provider config. Exit: end-to-end two-user clients with distinct catalogs, +limits and revocation; parity and no credential exposure. Deferred: +physical target qualification. + +### M007 — Small-team SBC qualification and compatibility closure + +Class: polish. Hard: M006. Deliverable: fresh/legacy/headless installation, +backup/restore, connection/profile compatibility and physical hardware +qualification, docs, truthful service sizing. Exit: 1,5,10,25,49 named users +with explicit concurrent load levels; single-user deltas; safe failure modes +and no unjustified performance/capacity claim. Deferred: SaaS/enterprise. + +### M008 — Enforceable output-token and price-bound admission corrective + +Class: capability. Ready (M005 published reservation contract exists). +Corrects M005's estimated-output/unknown-price hard-cap overclaim. +Deliverable: typed hard-versus-estimated enforcement, conservative +proof-or-reject across model/wire/selector/retries and explicit local-price +authority in operator TOML; preservation of existing estimated rules. +Exit: no unbounded/unpriced hard request reaches upstream; strict request/ +concurrency behavior unchanged. Source: +`plans/implementation/on-prem-user-access/008-hard-token-cost-admission-corrective.md`. + +### M009 — Instance-global budget and scope intersection corrective + +Class: capability. Hard: M008 closure. Adds one-instance global and +optional model ceilings intersected with user/model rules, with +atomic cross-user last-slot holds, no tenant hierarchy or second writer. +Exit: two separate users cannot over-admit a shared global remaining +slot; one-admin/no-global-rule compatibility remains. Source: +`plans/implementation/on-prem-user-access/009-global-budget-intersection-corrective.md`. + +### M010 — Durable crash recovery and schema-54 restore corrective + +Class: invariant. Hard: M008 and M009 closures. Independent +schema-54 backup->migration and modern backup/restore checks, +restart/finalization/cancellation fault matrix, paired ordinary-host +SQLite gate observations. Reclassifies M003's local updater-test issue +against the latest hosted-green baseline without altering updater +code by default. Exit: every recoverable budget hold converges +correctly; no silent historical identity/backfill or restore gap. Source: +`plans/implementation/on-prem-user-access/010-budget-crash-recovery-and-schema54-restore.md`. + +### M011 — Admin/member browser qualification corrective + +Class: polish. Ready independently of M008–M010. Frozen Python oracle +continues verifying *historical dashboard pages*; new admin/member +pages get separate Rust-owned reference screenshots and interactive +CSRF/accessibility/role tests. Operational browser/oracle artifacts +must exist for full closure. Exit: no unreviewed existing-page +regression; new pages have reproducible desktop/mobile browser +evidence and no token leakage. Source: +`plans/implementation/on-prem-user-access/011-admin-dashboard-browser-parity-corrective.md`. + +### M012 — Physical SBC small-team qualification and final disposition + +Class: polish. Hard: M008–M011 closures. Operational: actual Pi/Le Potato +plus representative ordinary hardware. Measure 1/5/10/25/49 **named** +user counts separately from simultaneous request levels, login +hashing, RSS, SQLite/WAL and tail latency against a paired solo baseline. +Exit: evidence supports precisely stated deployment envelope; no +50-concurrent-request marketing claim and no unresolved release- +blocking budget/security/restore defects. Source: +`plans/implementation/on-prem-user-access/012-physical-sbc-small-team-qualification.md`. + +### Corrective dependency graph and promotion + +- Hard: M008 -> M009 -> M010 -> M012. +- Parallel: M011 may run alongside M008–M010, but M012 needs M011 closure. +- Operational: browser/oracle reference for full M011 closure; provisioned + physical SBC and ordinary Linux host for M012. Unavailable evidence + must be reported as blocker, never fabricated or replaced by hosted CI. +- No predecessor conditional closure is retroactively altered; an additive + M008–M012 closure record provides the corrective proof. On each + milestone closure, audit the blocked registry and promote only + genuinely ready successors. + +### M014 — Rustls dependency security-advisory corrective + +Class: invariant. Closed. PR #7 updates the shared resolved Rustls +family from 0.23.45 to 0.23.46 with no provider-family or feature +changes. The exact implementation SHA passed hosted cargo-deny, CI, +Linux/macOS installer qualification, and local default/no-default, +transport, budget, and tooling checks. `main` retains 0.23.45 until +the PR is merged; this remains a merge prerequisite, not a separate +production change. Source: +`plans/implementation/on-prem-user-access/014-rustls-advisory-dependency-corrective.md`. + +### M015 — Independent authentication and authorization threat review + +Class: invariant. Closed on PR #7 candidate `d8fd9c2a`. Independent +review found and corrected hard-cost proof under-reservation and two +browser-session middleware rejection paths. Regression tests and full +default/no-default local suites pass; exact-SHA hosted CI, installer and +dependency audit pass. The security scan and independent reviewer found no +remaining M015 finding. Closure evidence: +`plans/closure/on-prem-user-access/015-status.md`. Source: +`plans/implementation/on-prem-user-access/015-authz-threat-review-corrective.md`. + +### M016 — Final integration and merge-readiness reconciliation + +Class: polish. Closed as a pre-merge reconciliation on exact +head `3d68c715`. Local, browser, release-doc, Q007 and exact-head +hosted CI/installer/**cargo-deny** evidence pass. Recommendation: +**merge held pending maintainer review** because GitHub has no +submitted PR review. PR #7 remains open; the subsystem roadmap stays +active until a maintainer decides whether to merge. No automatic merge +was performed and no 49-concurrent capacity claim is made. Closure: +`plans/closure/on-prem-user-access/016-status.md`. Source: +`plans/implementation/on-prem-user-access/016-final-pr7-merge-reconciliation.md`. + +### Post-M012 security/merge dependency graph + +- Hard: M014 closure + M015 closure -> M016; both are accepted. +- Operational for M016 closure: hosted CI, installer shells and + security audit must pass on the exact final candidate commit. +- Separate nonblocking performance roadmap: + `plans/subsystems/on-prem-concurrency-performance-roadmap.md`, + M001 Pi 5/MMC tail attribution, does **not** block M014–M016 + or revise M012's closed physical evidence. Any new storage + topology needs a separate accepted ADR. + +## 8. Cross-cutting requirements + +- Storage: append immutable numbered migrations and checksums; preserve DB + ownership and backups; no separate writer. Historical request `user_id` + nullable and identified as legacy/unattributed. +- Protocol: OpenAI/Anthropic inputs and SSE semantics unchanged; agent + `Bearer`/`x-api-key` compatibility with personal tokens; login cookies + restricted to web, not inference. +- Security: Argon2id with SBC-calibrated memory/CPU and throttled parallelism, + cryptographically strong tokens/digests, session CSRF/rotation, no password + in argv, operator-supplied LAN TLS and robust role gates. +- Concurrency: no database held during provider I/O; idempotent publication/ + finalization/claim compensation; policy cache invalidation bounded and + fail closed; reload/generation behavior explicit. +- Observability: per-user counters only under authorization; no prompt, + credentials, hashes or sensitive metadata in ordinary diagnostics. +- Performance: low idle RSS and minimal hot-path overhead in solo profile; + physical target evidence before claiming multi-user load support. +- Docs: status diagrams and CLI change descriptions accompany code, not plan. + +## 9. Verification strategy + +Use `rust/tests/server_transport.rs`, `cli_contract.rs`, +`database_compatibility.rs`, `coordinator_publication.rs`, +`coordinator_finalization.rs`, `coordinator_c008.rs`, +`coordinator_c009.rs`, `coordinator_c011.rs`, `coordinator_boundaries.rs`, +`model_router.rs`, `quota.rs`, `operations_o005.rs`, and new named +user-access tests. Include adversarial authorization, exact-surface model +bypass, credential race, multi-request budget, cancel/restart, backup, +migrations and current dashboard parity. Run Rust tests serial using +`--test-threads=1`, including no-default compile/Clippy. Add physical +loopback tests to `scripts/qualification_sbc.py` without touching user's +live keys or provider services. + +## 10. Risks and decision points + +- Password-bearing HTTP over LAN: require explicit safe transport posture + and bound/label any insecure-lab override; do not mistake LAN for TLS. +- Existing `dashboard.public`, keyless loopback, and `getkey/newkey` + compatibility cannot produce managed-mode privilege bypass. +- Unknown provider pricing and private-LLM cost are not zero by default. +- Selector calls and virtual aliases may spend hidden tokens: charge/authorize + them or explicitly deny, never silently hide bypass. +- Single DB gate may bound concurrent request settlement; optimize only with + measured gate-pressure and no WAL safety regression. +- Subscription/account sharing must follow each upstream's terms; no built-in + claim of consumer-seat pooling rights. + +## 11. Completion definition + +M001–M007 historical closure records remain linked and truthful; +M008–M013 provide the accepted additive corrections, including a +physically measured **restricted** Pi 5/MMC envelope. The prior +M012 roadmap closure is preserved as historical evidence. New +RUSTSEC advisories and missing integrated pre-merge security +review require M014/M015 closure and M016 final same-SHA +integration evidence before this roadmap may close **again**. +Physical MMC tail attribution is separately nonblocking; no +unsupported concurrency/service capacity claim is authorized. +A ready plan is not an implemented capability. + +## 12. Milestone status + +| Milestone | Status | Implementation plan | Closure record | Blockers | +|---|---|---|---|---| +| M001 | conditionally closed | `plans/implementation/on-prem-user-access/001-admin-bootstrap-and-credentials.md` | `plans/closure/on-prem-user-access/001-status.md` | Physical Le Potato/Pi Argon2 latency/RSS qualification carried to M007. | +| M002 | closed | `plans/implementation/on-prem-user-access/002-browser-sessions-and-route-auth.md` | `plans/closure/on-prem-user-access/002-status.md` | Browser sessions, managed route authorization and transport posture accepted; SBC hash qualification remains M007 evidence. | +| M003 | conditionally closed | `plans/implementation/on-prem-user-access/003-principal-propagation-and-attribution.md` | `plans/closure/on-prem-user-access/003-status.md` | Stable principal accepted; unrelated update unit-test failure and physical resource evidence are explicit conditions. | +| M004 | closed | `plans/implementation/on-prem-user-access/004-policy-profiles-and-model-gates.md` | `plans/closure/on-prem-user-access/004-status.md` | Profile rules, concrete-model gates and revision invalidation verified; M005 unblocked. | +| M005 | conditionally closed | `plans/implementation/on-prem-user-access/005-durable-user-budgets.md` | `plans/closure/on-prem-user-access/005-status.md` | Durable user holds and usage API verified; exact output-token and USD caps, global scope, crash/restore tests, and overhead measurements remain named conditions. | +| M006 | conditionally closed | `plans/implementation/on-prem-user-access/006-web-admin-and-personal-integrations.md` | `plans/closure/on-prem-user-access/006-status.md` | Browser parity captures remain outstanding; M005 estimate conditions and physical qualification carry into M007. | +| M007 | conditionally closed | `plans/implementation/on-prem-user-access/007-small-team-qualification-and-migration.md` | `plans/closure/on-prem-user-access/007-status.md` | Physical SBC capacity, exact schema-54 restore, dashboard parity, and inherited M005 conditions remain unqualified; no capacity claim. | +| M008 | closed | `plans/implementation/on-prem-user-access/008-hard-token-cost-admission-corrective.md` | `plans/closure/on-prem-user-access/008-status.md` | Hard/estimated enforcement, proof-or-reject admission, operator tariff trust boundary verified. | +| M009 | closed | `plans/implementation/on-prem-user-access/009-global-budget-intersection-corrective.md` | `plans/closure/on-prem-user-access/009-status.md` | Instance-global intersection, atomic cross-user holds, user-independent history verified. | +| M010 | closed | `plans/implementation/on-prem-user-access/010-budget-crash-recovery-and-schema54-restore.md` | `plans/closure/on-prem-user-access/010-status.md` | Reconciliation releases crash-orphaned budget holds without folding usage; frozen v54 image migrates 55–62 with legacy history intact; ordinary-host gate observations recorded (not SBC evidence). | +| M011 | closed | `plans/implementation/on-prem-user-access/011-admin-dashboard-browser-parity-corrective.md` | `plans/closure/on-prem-user-access/011-status.md` | Frozen Python oracle rebuilt and run with no new historical parity gaps; on-prem admin/member pages qualified against a Rust-owned approved reference. Browser sign-in and responsive/navigation defects corrected. | +| M012 | closed | `plans/implementation/on-prem-user-access/012-physical-sbc-small-team-qualification.md` | `plans/closure/on-prem-user-access/012-status.md` (closed) | Pi 5 managed envelope measured (functional + 3× benchmark, Argon2, 49 named users, 1/2/4/8 concurrency); 8-way saturates, Le Potato/reference-host/16-way recorded as low-severity uncovered dimensions. | +| M013 | closed | `plans/implementation/on-prem-user-access/013-transcode-eligibility-corrective.md` | `plans/closure/on-prem-user-access/013-status.md` | One-line transcode-protocol restoration plus admission bridge guard; Q007 green. | +| M014 | closed | `plans/implementation/on-prem-user-access/014-rustls-advisory-dependency-corrective.md` | `plans/closure/on-prem-user-access/014-status.md` | Rustls 0.23.46; exact-SHA hosted cargo-deny, CI, and installer jobs pass. Main receives the fix when PR #7 merges. | +| M015 | closed | `plans/implementation/on-prem-user-access/015-authz-threat-review-corrective.md` | `plans/closure/on-prem-user-access/015-status.md` | Exact candidate `d8fd9c2a`; findings fixed and independently re-reviewed; exact-SHA hosted gates pass. | +| M016 | closed | `plans/implementation/on-prem-user-access/016-final-pr7-merge-reconciliation.md` | `plans/closure/on-prem-user-access/016-status.md` | Exact candidate checks pass; merge held pending maintainer PR review. The roadmap remains active until the PR decision. | diff --git a/rust/Cargo.lock b/rust/Cargo.lock index 8ad309389..d0235fa94 100644 --- a/rust/Cargo.lock +++ b/rust/Cargo.lock @@ -166,6 +166,18 @@ dependencies = [ "rustversion", ] +[[package]] +name = "argon2" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c3610892ee6e0cbce8ae2700349fcf8f98adb0dbfbee85aec3c9179d29cc072" +dependencies = [ + "base64ct", + "blake2 0.10.6", + "cpufeatures 0.2.17", + "password-hash 0.5.0", +] + [[package]] name = "argon2" version = "0.6.0" @@ -173,9 +185,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "134c52ddac6d63c576bef8168db10c83c49c26444ecbc68060fef078925a901c" dependencies = [ "base64ct", - "blake2", + "blake2 0.11.0", "cpufeatures 0.3.1", - "password-hash", + "password-hash 0.6.1", ] [[package]] @@ -360,6 +372,15 @@ version = "2.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" +[[package]] +name = "blake2" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" +dependencies = [ + "digest 0.10.7", +] + [[package]] name = "blake2" version = "0.11.0" @@ -969,6 +990,7 @@ name = "eggpool" version = "0.8.2" dependencies = [ "arc-swap", + "argon2 0.5.3", "axum", "bytes", "clap", @@ -985,6 +1007,7 @@ dependencies = [ "eggress-protocol-trojan", "eggress-server", "eggress-uri", + "eggserve-primitives", "eggserve-server", "getrandom 0.3.4", "http", @@ -994,6 +1017,7 @@ dependencies = [ "hyper-util", "nix", "rcgen", + "rpassword", "rustls", "serde", "serde_json", @@ -2332,6 +2356,17 @@ dependencies = [ "windows-strings", ] +[[package]] +name = "password-hash" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "346f04948ba92c43e8469c1ee6736c7563d71012b17d40745260fe106aac2166" +dependencies = [ + "base64ct", + "rand_core 0.6.4", + "subtle", +] + [[package]] name = "password-hash" version = "0.6.1" @@ -2707,6 +2742,17 @@ dependencies = [ "windows-sys 0.52.0", ] +[[package]] +name = "rpassword" +version = "7.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "66d4c8b64f049c6721ec8ccec37ddfc3d641c4a7fca57e8f2a89de509c73df39" +dependencies = [ + "libc", + "rtoolbox", + "windows-sys 0.59.0", +] + [[package]] name = "rsa" version = "0.10.0-rc.18" @@ -2736,6 +2782,16 @@ dependencies = [ "thiserror", ] +[[package]] +name = "rtoolbox" +version = "0.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a1efe12a1469752d0e6ff5ebec0b6ef4924cc5c4c71046b0ec730040535819d" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + [[package]] name = "rusqlite" version = "0.40.2" @@ -2880,9 +2936,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.45" +version = "0.23.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634" +checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf" dependencies = [ "aws-lc-rs", "log", @@ -3260,7 +3316,7 @@ version = "0.7.0-rc.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f9a32fae177b74a22aa9c5b01bf7e68b33545be32d9e381e248058d2adc15ce3" dependencies = [ - "argon2", + "argon2 0.6.0", "bcrypt-pbkdf", "ctutils", "ed25519-dalek", @@ -3906,6 +3962,15 @@ dependencies = [ "windows-targets", ] +[[package]] +name = "windows-sys" +version = "0.59.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" +dependencies = [ + "windows-targets", +] + [[package]] name = "windows-sys" version = "0.61.2" diff --git a/rust/Cargo.toml b/rust/Cargo.toml index b570632e6..920dcefc9 100644 --- a/rust/Cargo.toml +++ b/rust/Cargo.toml @@ -55,6 +55,7 @@ ssh = ["eggress-outbound/ssh", "eggress-pproxy-compat/ssh"] [dependencies] arc-swap = "1.9" +argon2 = "0.5" axum = "0.8" bytes = "1" clap = { version = "4.5", features = ["derive"] } @@ -75,6 +76,7 @@ eggfetch-core = { version = "=0.2.2", default-features = false, features = [ "tls-rustls", ] } eggserve-server = { version = "=0.4.0", default-features = false, features = ["tower"] } +eggserve-primitives = "=0.2.2" getrandom = "0.3.4" eggpool-client-config = { version = "0.1.0", path = "crates/eggpool-client-config" } eggpool-model-routing = { version = "0.1.0", path = "crates/eggpool-model-routing" } @@ -96,6 +98,7 @@ hyper-util = { version = "0.1", default-features = false, features = [ ] } nix = { version = "0.31.3", default-features = false, features = ["signal", "term", "user"] } rustls = { version = "0.23", default-features = false, features = ["ring", "tls12"] } +rpassword = "7" serde = { version = "1", features = ["derive"] } serde_json = { version = "1", features = ["preserve_order"] } sha2 = "0.10" diff --git a/rust/assets/dashboard/manifest.json b/rust/assets/dashboard/manifest.json index 52f715c24..d9ecd2049 100644 --- a/rust/assets/dashboard/manifest.json +++ b/rust/assets/dashboard/manifest.json @@ -1,6 +1,6 @@ [ {"path":"static/chart.umd.min.js","sha256":"206b6e8bb00fc7bba2c7ee80ca41db3e9e05ba7be0aa35abeba9cfd5357f5d0e"}, - {"path":"static/dashboard.css","sha256":"179689eeed4366ad6dc1575a3c4f4b7dbb9b39693a43827f1e4192b0b6a2d5fe"}, + {"path":"static/dashboard.css","sha256":"33edc8fb7b48551e56fbe823f9ce37445359848593de8b193872e9e6df663a50"}, {"path":"static/dashboard.js","sha256":"e801eca08461c88250934ef04e87c6d94fed9c381e460808e5378bc76c156b0d"}, {"path":"static/favicon.svg","sha256":"593a7d0f464160e832f62399cf224a04dda68a55f3a6ce847dffb62016d37653"}, {"path":"themes/Booberry.toml","sha256":"ca0588275bd2c59de561633c2dc7e2b7f95d183a7fd918a10d00dfff5ca96871"}, diff --git a/rust/assets/dashboard/static/dashboard.css b/rust/assets/dashboard/static/dashboard.css index 871e706ec..c2604fafc 100644 --- a/rust/assets/dashboard/static/dashboard.css +++ b/rust/assets/dashboard/static/dashboard.css @@ -137,6 +137,12 @@ nav.topnav { flex-wrap: wrap; gap: 0.5rem; align-items: center; + /* The menu reflows internally, but as a flex item of the topbar this element + still defaults to a `min-width: auto` floor of its max-content width, so a + long navigation widens the whole bar instead of wrapping inside it. + Letting the item shrink keeps the topbar inside its own container. */ + min-width: 0; + max-width: 100%; } nav.topnav a { @@ -1180,8 +1186,8 @@ label.trace-limit { width: 100%; } - /* Thirteen links at a 44px tap height plus the theme picker stack to - ~694px, which is taller than a typical phone viewport (~660px). The + /* Sixteen links at a 44px tap height plus the theme picker stack to + ~810px, which is taller than a typical phone viewport (~660px). The picker sits last, so on a phone it landed below the fold and the one control the operator most wants to reach was the one they could not. Cap the dropdown to the space actually below the sticky topbar and let @@ -1274,7 +1280,7 @@ label.trace-limit { width: auto; margin-top: 0; /* A zero flex-basis is required here, not `auto`. With `flex: 1 1 auto` - the menu's hypothetical main size is its full content width (13 links + the menu's hypothetical main size is its full content width (16 links plus the theme selector), which exceeds the topbar, so `flex-wrap` broke the line *before* any shrinking happened — the menu took a line by itself and the refresh button wrapped onto a row of its own. With @@ -1292,8 +1298,8 @@ label.trace-limit { /* ---- Compact metrics for the inline (non-burger) topbar ---- - Thirteen uppercase nav links plus a theme `` plus the reload + button need roughly 1900px at the default metrics. At full size the group fell to three or four rows from 761px all the way to 1279px — measured at 768px the sticky topbar was 175px tall (brand / two nav rows / theme picker alone on a 325px row), permanently occupying 23% of a @@ -1338,16 +1344,20 @@ label.trace-limit { /* ---- Wide: one line for links, theme picker, and reload button ---- - `nowrap` is the last step, layered on the compact metrics above. Measured - against the real topbar the group needs ~1360px to truly fit one row, so - this is not the breakpoint where the bar becomes a single row — it is where - forcing one stops costing more height than wrapping does. Between 1280 and - 1359 the rows measured 88px with `nowrap` against 129px with the wrap - inherited from the 761px block, so `nowrap` starts at 1280 and the bar is a - single 57px row from ~1340px. The 761-1279 range keeps the wrap: thirteen - links plus the picker do not fit a tablet, but the compact metrics above - hold that range to 129px instead of the 175px it measured before. */ -@media (min-width: 1280px) { + `nowrap` is the last step, layered on the compact metrics above. It is not + the breakpoint where the bar becomes a single row — it is where forcing one + stops costing more height than wrapping does. + + The tier used to start at 1280px because thirteen links plus the picker needed + ~1360px to fit one row, and between 1280 and 1359 the bar measured 88px with + `nowrap` against 129px with the wrap inherited from the 761px block. The + on-prem user-access navigation then added Users, Profiles, Usage, and My + Account, so the group is sixteen links and no longer fits at 1280px: measured + on a 1280x800 viewport the document scrolled 212px past the viewport with + `nowrap`, which is the common laptop width this tier was sized for. The tier + now starts at 1560px, above the 1520px the group actually needs, and the + 1280-1559 range keeps the inherited wrap. */ +@media (min-width: 1560px) { nav.topnav { flex-wrap: nowrap; } @@ -1360,7 +1370,7 @@ label.trace-limit { /* Touch-sized topbar controls above the burger breakpoint. - The 760/761 split is a layout-fit decision: below it thirteen nav + The 760/761 split is a layout-fit decision: below it sixteen nav links plus a theme picker do not fit on a row, so the burger appears. It cannot also express *input capability*, so every viewport from 761px up inherits mouse-sized controls — and a 768x1024 iPad is diff --git a/rust/assets/db/migrations/0055_user_identity.sql b/rust/assets/db/migrations/0055_user_identity.sql new file mode 100644 index 000000000..9a1161839 --- /dev/null +++ b/rust/assets/db/migrations/0055_user_identity.sql @@ -0,0 +1,38 @@ +CREATE TABLE users ( + id TEXT PRIMARY KEY NOT NULL, + username TEXT NOT NULL, + normalized_username TEXT NOT NULL UNIQUE, + role TEXT NOT NULL CHECK (role IN ('admin', 'member')), + state TEXT NOT NULL DEFAULT 'active' CHECK (state IN ('active', 'disabled')), + password_hash TEXT NOT NULL, + must_change_password INTEGER NOT NULL DEFAULT 0 CHECK (must_change_password IN (0, 1)), + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE UNIQUE INDEX users_one_active_admin + ON users(role) WHERE role = 'admin' AND state = 'active'; + +CREATE TABLE user_credentials ( + id TEXT PRIMARY KEY NOT NULL, + user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + label TEXT NOT NULL, + selector TEXT NOT NULL UNIQUE, + digest TEXT NOT NULL, + scopes TEXT NOT NULL DEFAULT 'inference,discovery,integrations', + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + last_used_at TEXT, + expires_at TEXT, + revoked_at TEXT +); + +CREATE INDEX user_credentials_user_active + ON user_credentials(user_id, revoked_at, expires_at); + +CREATE TABLE user_access_state ( + singleton INTEGER PRIMARY KEY CHECK (singleton = 1), + state TEXT NOT NULL CHECK (state IN ('legacy_unmigrated', 'fresh_uninitialized', 'admin_initialized', 'managed')), + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +INSERT INTO user_access_state(singleton, state) VALUES (1, 'legacy_unmigrated'); diff --git a/rust/assets/db/migrations/0056_web_sessions.sql b/rust/assets/db/migrations/0056_web_sessions.sql new file mode 100644 index 000000000..debb4a8e0 --- /dev/null +++ b/rust/assets/db/migrations/0056_web_sessions.sql @@ -0,0 +1,17 @@ +CREATE TABLE web_sessions ( + id TEXT PRIMARY KEY NOT NULL, + user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + selector TEXT NOT NULL UNIQUE, + digest TEXT NOT NULL, + csrf_digest TEXT NOT NULL, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + last_used_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + expires_at TEXT NOT NULL, + revoked_at TEXT +); + +CREATE INDEX web_sessions_user_active + ON web_sessions(user_id, revoked_at, expires_at); + +CREATE INDEX web_sessions_expiry + ON web_sessions(expires_at) WHERE revoked_at IS NULL; diff --git a/rust/assets/db/migrations/0057_request_attribution.sql b/rust/assets/db/migrations/0057_request_attribution.sql new file mode 100644 index 000000000..c37b35928 --- /dev/null +++ b/rust/assets/db/migrations/0057_request_attribution.sql @@ -0,0 +1,23 @@ +ALTER TABLE users + ADD COLUMN auth_epoch INTEGER NOT NULL DEFAULT 1 CHECK (auth_epoch > 0); + +ALTER TABLE requests + ADD COLUMN user_id TEXT; +ALTER TABLE requests + ADD COLUMN credential_id TEXT; +ALTER TABLE requests + ADD COLUMN auth_epoch INTEGER; +ALTER TABLE requests + ADD COLUMN principal_kind TEXT NOT NULL DEFAULT 'legacy_shared_key' + CHECK (principal_kind IN ('agent_token', 'browser_session', 'legacy_shared_key')); +ALTER TABLE requests + ADD COLUMN request_kind TEXT NOT NULL DEFAULT 'logical' + CHECK (request_kind IN ('logical', 'internal_selector')); +ALTER TABLE requests + ADD COLUMN parent_proxy_request_id TEXT; + +CREATE INDEX idx_requests_user_recent + ON requests(user_id, started_at DESC, id DESC); +CREATE INDEX idx_requests_parent_proxy_request + ON requests(parent_proxy_request_id) + WHERE parent_proxy_request_id IS NOT NULL; diff --git a/rust/assets/db/migrations/0058_model_policy.sql b/rust/assets/db/migrations/0058_model_policy.sql new file mode 100644 index 000000000..ea512ea7e --- /dev/null +++ b/rust/assets/db/migrations/0058_model_policy.sql @@ -0,0 +1,90 @@ +CREATE TABLE policy_state ( + singleton INTEGER PRIMARY KEY CHECK (singleton = 1), + revision INTEGER NOT NULL DEFAULT 1 CHECK (revision > 0) +); + +INSERT INTO policy_state(singleton, revision) VALUES (1, 1); + +CREATE TABLE policy_profiles ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE CHECK (length(name) BETWEEN 1 AND 64), + default_effect TEXT NOT NULL CHECK (default_effect IN ('allow', 'deny')), + preferred_model_id TEXT, + quota_policy_id TEXT, + version INTEGER NOT NULL DEFAULT 1 CHECK (version > 0), + created_by TEXT, + updated_by TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE profile_model_rules ( + profile_id TEXT NOT NULL REFERENCES policy_profiles(id) ON DELETE CASCADE, + model_id TEXT NOT NULL CHECK (length(model_id) BETWEEN 1 AND 512), + effect TEXT NOT NULL CHECK (effect IN ('allow', 'deny')), + created_by TEXT, + updated_by TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (profile_id, model_id) +); + +CREATE TABLE user_profile_assignments ( + user_id TEXT PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE, + profile_id TEXT NOT NULL REFERENCES policy_profiles(id) ON DELETE RESTRICT, + version INTEGER NOT NULL DEFAULT 1 CHECK (version > 0), + assigned_by TEXT, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE user_policy_overrides ( + user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + model_id TEXT NOT NULL CHECK (length(model_id) BETWEEN 1 AND 512), + effect TEXT NOT NULL CHECK (effect IN ('allow', 'deny')), + updated_by TEXT, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (user_id, model_id) +); + +CREATE TABLE instance_model_denials ( + model_id TEXT PRIMARY KEY CHECK (length(model_id) BETWEEN 1 AND 512), + created_by TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX idx_profile_model_rules_profile ON profile_model_rules(profile_id, model_id); +CREATE INDEX idx_user_policy_overrides_user ON user_policy_overrides(user_id, model_id); + +CREATE TRIGGER users_policy_revision_update AFTER UPDATE OF state, role, auth_epoch ON users +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; + +CREATE TRIGGER policy_profiles_revision_insert AFTER INSERT ON policy_profiles +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER policy_profiles_revision_update AFTER UPDATE ON policy_profiles +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER policy_profiles_revision_delete AFTER DELETE ON policy_profiles +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER profile_model_rules_revision_insert AFTER INSERT ON profile_model_rules +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER profile_model_rules_revision_update AFTER UPDATE ON profile_model_rules +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER profile_model_rules_revision_delete AFTER DELETE ON profile_model_rules +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER user_profile_assignments_revision_insert AFTER INSERT ON user_profile_assignments +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER user_profile_assignments_revision_update AFTER UPDATE ON user_profile_assignments +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER user_profile_assignments_revision_delete AFTER DELETE ON user_profile_assignments +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER user_policy_overrides_revision_insert AFTER INSERT ON user_policy_overrides +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER user_policy_overrides_revision_update AFTER UPDATE ON user_policy_overrides +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER user_policy_overrides_revision_delete AFTER DELETE ON user_policy_overrides +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER instance_model_denials_revision_insert AFTER INSERT ON instance_model_denials +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER instance_model_denials_revision_update AFTER UPDATE ON instance_model_denials +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER instance_model_denials_revision_delete AFTER DELETE ON instance_model_denials +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; diff --git a/rust/assets/db/migrations/0059_user_budgets.sql b/rust/assets/db/migrations/0059_user_budgets.sql new file mode 100644 index 000000000..3b9318872 --- /dev/null +++ b/rust/assets/db/migrations/0059_user_budgets.sql @@ -0,0 +1,107 @@ +CREATE TABLE user_budget_rules ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + model_id TEXT, + window TEXT NOT NULL CHECK (window IN ('day', 'month')), + max_requests INTEGER CHECK (max_requests IS NULL OR max_requests > 0), + max_tokens INTEGER CHECK (max_tokens IS NULL OR max_tokens > 0), + max_microdollars INTEGER CHECK (max_microdollars IS NULL OR max_microdollars > 0), + max_concurrent INTEGER CHECK (max_concurrent IS NULL OR max_concurrent > 0), + version INTEGER NOT NULL DEFAULT 1 CHECK (version > 0), + updated_by TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + CHECK (max_requests IS NOT NULL OR max_tokens IS NOT NULL OR + max_microdollars IS NOT NULL OR max_concurrent IS NOT NULL) +); + +CREATE UNIQUE INDEX idx_user_budget_rules_scope + ON user_budget_rules(user_id, ifnull(model_id, ''), window); +CREATE INDEX idx_user_budget_rules_user ON user_budget_rules(user_id, window); + +CREATE TABLE user_budget_reservations ( + request_id INTEGER PRIMARY KEY REFERENCES requests(id) ON DELETE CASCADE, + user_id TEXT NOT NULL REFERENCES users(id) ON DELETE RESTRICT, + model_id TEXT NOT NULL, + policy_version INTEGER NOT NULL CHECK (policy_version > 0), + day_start TEXT NOT NULL, + month_start TEXT NOT NULL, + held_requests INTEGER NOT NULL DEFAULT 1 CHECK (held_requests = 1), + held_tokens INTEGER NOT NULL CHECK (held_tokens >= 0), + held_microdollars INTEGER NOT NULL CHECK (held_microdollars >= 0), + observed_tokens INTEGER NOT NULL DEFAULT 0 CHECK (observed_tokens >= 0), + observed_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (observed_microdollars >= 0), + token_usage_complete INTEGER NOT NULL DEFAULT 0 CHECK (token_usage_complete IN (0, 1)), + cost_usage_complete INTEGER NOT NULL DEFAULT 0 CHECK (cost_usage_complete IN (0, 1)), + status TEXT NOT NULL CHECK (status IN ('active', 'settled', 'released')), + expires_at TEXT NOT NULL, + settled_at TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX idx_user_budget_reservations_active + ON user_budget_reservations(user_id, status, model_id, day_start, month_start); +CREATE INDEX idx_user_budget_reservations_expiry + ON user_budget_reservations(status, expires_at); + +CREATE TABLE user_budget_attempt_usage ( + attempt_id INTEGER PRIMARY KEY REFERENCES request_attempts(id) ON DELETE CASCADE, + request_id INTEGER NOT NULL REFERENCES requests(id) ON DELETE CASCADE, + input_tokens INTEGER NOT NULL CHECK (input_tokens >= 0), + output_tokens INTEGER NOT NULL CHECK (output_tokens >= 0), + cost_microdollars INTEGER NOT NULL CHECK (cost_microdollars >= 0), + provider_reported_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (provider_reported_microdollars >= 0), + local_estimated_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (local_estimated_microdollars >= 0), + cost_source TEXT NOT NULL CHECK (cost_source IN ('provider_reported', 'local_estimated', 'unknown')), + tokens_known INTEGER NOT NULL CHECK (tokens_known IN (0, 1)), + cost_known INTEGER NOT NULL CHECK (cost_known IN (0, 1)), + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX idx_user_budget_attempt_usage_request + ON user_budget_attempt_usage(request_id, attempt_id); + +CREATE TABLE user_budget_usage ( + user_id TEXT NOT NULL REFERENCES users(id) ON DELETE RESTRICT, + model_id TEXT NOT NULL, + window TEXT NOT NULL CHECK (window IN ('day', 'month')), + window_start TEXT NOT NULL, + request_count INTEGER NOT NULL DEFAULT 0 CHECK (request_count >= 0), + token_count INTEGER NOT NULL DEFAULT 0 CHECK (token_count >= 0), + cost_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (cost_microdollars >= 0), + provider_reported_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (provider_reported_microdollars >= 0), + local_estimated_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (local_estimated_microdollars >= 0), + reservation_fallback_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (reservation_fallback_microdollars >= 0), + unknown_token_requests INTEGER NOT NULL DEFAULT 0 CHECK (unknown_token_requests >= 0), + unknown_cost_requests INTEGER NOT NULL DEFAULT 0 CHECK (unknown_cost_requests >= 0), + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (user_id, model_id, window, window_start) +); + +CREATE INDEX idx_user_budget_usage_window + ON user_budget_usage(user_id, window, window_start, model_id); + +CREATE TABLE user_budget_rule_audit ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + user_id TEXT NOT NULL, + actor_id TEXT NOT NULL, + model_id TEXT, + window TEXT NOT NULL CHECK (window IN ('day', 'month')), + action TEXT NOT NULL CHECK (action IN ('set', 'remove')), + max_requests INTEGER, + max_tokens INTEGER, + max_microdollars INTEGER, + max_concurrent INTEGER, + rule_version INTEGER NOT NULL CHECK (rule_version > 0), + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX idx_user_budget_rule_audit_user + ON user_budget_rule_audit(user_id, created_at DESC, id DESC); + +CREATE TRIGGER user_budget_rules_revision_insert AFTER INSERT ON user_budget_rules +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER user_budget_rules_revision_update AFTER UPDATE ON user_budget_rules +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER user_budget_rules_revision_delete AFTER DELETE ON user_budget_rules +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; diff --git a/rust/assets/db/migrations/0060_user_admin_audit.sql b/rust/assets/db/migrations/0060_user_admin_audit.sql new file mode 100644 index 000000000..6ed2e7487 --- /dev/null +++ b/rust/assets/db/migrations/0060_user_admin_audit.sql @@ -0,0 +1,13 @@ +CREATE TABLE user_admin_audit ( + id INTEGER PRIMARY KEY, + actor_id TEXT NOT NULL, + action TEXT NOT NULL CHECK (length(action) BETWEEN 1 AND 64), + target_kind TEXT NOT NULL CHECK (length(target_kind) BETWEEN 1 AND 32), + target_id TEXT NOT NULL CHECK (length(target_id) BETWEEN 1 AND 128), + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX idx_user_admin_audit_recent + ON user_admin_audit(created_at DESC, id DESC); +CREATE INDEX idx_user_admin_audit_target + ON user_admin_audit(target_kind, target_id, created_at DESC, id DESC); diff --git a/rust/assets/db/migrations/0061_user_budget_enforcement.sql b/rust/assets/db/migrations/0061_user_budget_enforcement.sql new file mode 100644 index 000000000..655ccfa55 --- /dev/null +++ b/rust/assets/db/migrations/0061_user_budget_enforcement.sql @@ -0,0 +1,29 @@ +-- M008 enforceable token/cost admission: per-rule enforcement mode plus +-- auditable worst-case reservation facts. Existing rows keep the estimated +-- mode explicitly; promotion to hard mode is an admin operation, never an +-- automatic upgrade. +ALTER TABLE user_budget_rules + ADD COLUMN enforcement TEXT NOT NULL DEFAULT 'estimated' + CHECK (enforcement IN ('hard', 'estimated')); + +ALTER TABLE user_budget_rule_audit + ADD COLUMN enforcement TEXT NOT NULL DEFAULT 'estimated' + CHECK (enforcement IN ('hard', 'estimated')); + +-- Worst-case permit facts recorded with each held reservation so hard-mode +-- holds stay auditable after settlement. Estimated-mode rows keep zero +-- bounds with a NULL price source. +ALTER TABLE user_budget_reservations + ADD COLUMN enforcement TEXT NOT NULL DEFAULT 'estimated' + CHECK (enforcement IN ('hard', 'estimated')); +ALTER TABLE user_budget_reservations + ADD COLUMN bound_tokens INTEGER NOT NULL DEFAULT 0 + CHECK (bound_tokens >= 0); +ALTER TABLE user_budget_reservations + ADD COLUMN bound_microdollars INTEGER NOT NULL DEFAULT 0 + CHECK (bound_microdollars >= 0); +ALTER TABLE user_budget_reservations + ADD COLUMN price_source TEXT CHECK (price_source IN ('operator_tariff', 'estimate')); + +CREATE INDEX idx_user_budget_rules_enforcement + ON user_budget_rules(user_id, enforcement); diff --git a/rust/assets/db/migrations/0062_instance_budgets.sql b/rust/assets/db/migrations/0062_instance_budgets.sql new file mode 100644 index 000000000..824b04136 --- /dev/null +++ b/rust/assets/db/migrations/0062_instance_budgets.sql @@ -0,0 +1,106 @@ +-- M009 instance-global budget ceilings intersected with per-user rules. +-- Exactly one instance-wide scope: model_id NULL means all models. No +-- user/tenant identifier and no user foreign key: global held and settled +-- counters survive user disable/delete and cover unattributed requests too. +CREATE TABLE instance_budget_rules ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + model_id TEXT, + window TEXT NOT NULL CHECK (window IN ('day', 'month')), + max_requests INTEGER CHECK (max_requests IS NULL OR max_requests > 0), + max_tokens INTEGER CHECK (max_tokens IS NULL OR max_tokens > 0), + max_microdollars INTEGER CHECK (max_microdollars IS NULL OR max_microdollars > 0), + max_concurrent INTEGER CHECK (max_concurrent IS NULL OR max_concurrent > 0), + enforcement TEXT NOT NULL DEFAULT 'estimated' + CHECK (enforcement IN ('hard', 'estimated')), + version INTEGER NOT NULL DEFAULT 1 CHECK (version > 0), + updated_by TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + CHECK (max_requests IS NOT NULL OR max_tokens IS NOT NULL OR + max_microdollars IS NOT NULL OR max_concurrent IS NOT NULL) +); + +CREATE UNIQUE INDEX idx_instance_budget_rules_scope + ON instance_budget_rules(ifnull(model_id, ''), window); +CREATE INDEX idx_instance_budget_rules_enforcement + ON instance_budget_rules(enforcement); + +-- One row per logical request that matched an instance rule at admission. +-- request_id references the global request row; intentionally no user +-- reference so history never depends on a user still existing. +CREATE TABLE instance_budget_reservations ( + request_id INTEGER PRIMARY KEY REFERENCES requests(id) ON DELETE CASCADE, + model_id TEXT NOT NULL, + policy_version INTEGER NOT NULL CHECK (policy_version > 0), + day_start TEXT NOT NULL, + month_start TEXT NOT NULL, + held_requests INTEGER NOT NULL DEFAULT 1 CHECK (held_requests = 1), + held_tokens INTEGER NOT NULL CHECK (held_tokens >= 0), + held_microdollars INTEGER NOT NULL CHECK (held_microdollars >= 0), + observed_tokens INTEGER NOT NULL DEFAULT 0 CHECK (observed_tokens >= 0), + observed_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (observed_microdollars >= 0), + token_usage_complete INTEGER NOT NULL DEFAULT 0 CHECK (token_usage_complete IN (0, 1)), + cost_usage_complete INTEGER NOT NULL DEFAULT 0 CHECK (cost_usage_complete IN (0, 1)), + enforcement TEXT NOT NULL DEFAULT 'estimated' + CHECK (enforcement IN ('hard', 'estimated')), + bound_tokens INTEGER NOT NULL DEFAULT 0 CHECK (bound_tokens >= 0), + bound_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (bound_microdollars >= 0), + price_source TEXT CHECK (price_source IN ('operator_tariff')), + status TEXT NOT NULL CHECK (status IN ('active', 'settled', 'released')), + expires_at TEXT NOT NULL, + settled_at TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX idx_instance_budget_reservations_active + ON instance_budget_reservations(status, model_id, day_start, month_start); +CREATE INDEX idx_instance_budget_reservations_expiry + ON instance_budget_reservations(status, expires_at); + +-- Per-model settled buckets. Global (model-agnostic) rules aggregate across +-- these rows at check time with a bounded indexed sum; no full history scan +-- runs under the writer gate. +CREATE TABLE instance_budget_usage ( + model_id TEXT NOT NULL, + window TEXT NOT NULL CHECK (window IN ('day', 'month')), + window_start TEXT NOT NULL, + request_count INTEGER NOT NULL DEFAULT 0 CHECK (request_count >= 0), + token_count INTEGER NOT NULL DEFAULT 0 CHECK (token_count >= 0), + cost_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (cost_microdollars >= 0), + provider_reported_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (provider_reported_microdollars >= 0), + local_estimated_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (local_estimated_microdollars >= 0), + reservation_fallback_microdollars INTEGER NOT NULL DEFAULT 0 CHECK (reservation_fallback_microdollars >= 0), + unknown_token_requests INTEGER NOT NULL DEFAULT 0 CHECK (unknown_token_requests >= 0), + unknown_cost_requests INTEGER NOT NULL DEFAULT 0 CHECK (unknown_cost_requests >= 0), + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (model_id, window, window_start) +); + +CREATE INDEX idx_instance_budget_usage_window + ON instance_budget_usage(window, window_start, model_id); + +CREATE TABLE instance_budget_rule_audit ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + actor_id TEXT NOT NULL, + model_id TEXT, + window TEXT NOT NULL CHECK (window IN ('day', 'month')), + action TEXT NOT NULL CHECK (action IN ('set', 'remove')), + max_requests INTEGER, + max_tokens INTEGER, + max_microdollars INTEGER, + max_concurrent INTEGER, + enforcement TEXT NOT NULL DEFAULT 'estimated' + CHECK (enforcement IN ('hard', 'estimated')), + rule_version INTEGER NOT NULL CHECK (rule_version > 0), + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX idx_instance_budget_rule_audit_recent + ON instance_budget_rule_audit(created_at DESC, id DESC); + +CREATE TRIGGER instance_budget_rules_revision_insert AFTER INSERT ON instance_budget_rules +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER instance_budget_rules_revision_update AFTER UPDATE ON instance_budget_rules +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; +CREATE TRIGGER instance_budget_rules_revision_delete AFTER DELETE ON instance_budget_rules +BEGIN UPDATE policy_state SET revision = revision + 1 WHERE singleton = 1; END; diff --git a/rust/assets/db/migrations/checksums.json b/rust/assets/db/migrations/checksums.json index b9f2ac827..c43ef5772 100644 --- a/rust/assets/db/migrations/checksums.json +++ b/rust/assets/db/migrations/checksums.json @@ -53,6 +53,14 @@ "0051_model_quarantine.sql": "e60128ee2006073c9dd54af80facb70f6b3a8edfaf69d6619d0b41e35180a940", "0052_catalog_refresh_state.sql": "3931b8b1ef87b2e6413f8bba2f16f26b1cee899ea4e962ac9e5ff50be590c99d", "0053_remove_attempt_status_analytics_index.sql": "0918c57f786dcb953fd5a32238d94060c471e768f245f40aeb37e20fec8ffe99", - "0054_model_quarantine_null_identity.sql": "f382fa78f09a55e6a381c225e376b04c31a9775d79ecf6a3f41aeaadf9b8019e" + "0054_model_quarantine_null_identity.sql": "f382fa78f09a55e6a381c225e376b04c31a9775d79ecf6a3f41aeaadf9b8019e", + "0055_user_identity.sql": "352224c8e2505509efa8fad0017c9aaac74ef8338771b35388ef22cf77cd8a62", + "0056_web_sessions.sql": "b35541860dbc9c258de617f6f5a9c0111de6428be3128cd79d962bdda9634167", + "0057_request_attribution.sql": "d9b66c87ee15c0b88abcb27599dc7b80213cc67df988fa1ace50cf511012e87b", + "0058_model_policy.sql": "dc98951394abed29a02f48d75d2b72554da74d98de49563fa87c2548dd757fe2", + "0059_user_budgets.sql": "6637f6242750639cf292359ab3f476ff1f43ee8d9cb7de99ca67a84088f49bc2", + "0060_user_admin_audit.sql": "fe15aebc5a344aa25e049aaf554e8d7eaac69bd16134abbb41e24c390df8891c", + "0061_user_budget_enforcement.sql": "d19229166fed11622a3c924642705d78563e95bd02605123b7b2b9483ecf1a3f", + "0062_instance_budgets.sql": "bcb0999e1be7944512f082164f763671bb5ce1b6e82c071470d3ff5f03e5ebac" } } diff --git a/rust/assets/runtime-manifest.json b/rust/assets/runtime-manifest.json index 2bc6aa4d8..353e53eaa 100644 --- a/rust/assets/runtime-manifest.json +++ b/rust/assets/runtime-manifest.json @@ -5,7 +5,7 @@ "path": "db/migrations/checksums.json", "category": "sqlite-migrations", "source": "rust/assets/db/migrations/checksums.json", - "sha256": "0b0ea98e626a51a561c3e2e2cbff09f8a5c7d05a2ef2e61ba7589347c929bc95", + "sha256": "6391e4ddd06466dfb899dd2baabb6669c7299b8562b27854224b0d8dfe415e00", "identity": "exact" }, { @@ -26,14 +26,14 @@ "path": "config/config.example.toml", "category": "configuration-template", "source": "rust/config.example.toml", - "sha256": "3ad588d51b0f0c3d2be087f3bc44b01e137783bc036d613789596142942940f3", + "sha256": "046355f5469c730125d3179a190dc605bcdde344a7b1f1ef7ab3ad304dfce7b4", "identity": "exact" }, { "path": "config/config.sbc.example.toml", "category": "configuration-template", "source": "rust/config.sbc.example.toml", - "sha256": "fb6f32e73514a7f013acb270467fcb87978830fe71cfc6d0b1b87dc3f304d218", + "sha256": "cae152978cd756cc795f22110a01b28751bfe2804e18f253b60430163a9f07dd", "identity": "exact" }, { @@ -47,12 +47,12 @@ "path": "dashboard/manifest.json", "category": "dashboard-assets", "source": "rust/assets/dashboard/manifest.json", - "sha256": "78edb15cc669d61ad099d2c5c98c6a1aa32186c29e12c22a7cc3e7b4626944c3", + "sha256": "13d628c41486e65bb38f402eb5f26c230aba617105b7f1a3a059278df290ed63", "identity": "exact" } ], - "schema_migration_count": 54, - "schema_change": false, + "schema_migration_count": 62, + "schema_change": true, "ownership": "Rust build/runtime embeds these assets; source paths are canonical repository inputs", "rollback": "restore the source tree from Git history" } diff --git a/rust/config.example.toml b/rust/config.example.toml index ecff00e84..d6dee9416 100644 --- a/rust/config.example.toml +++ b/rust/config.example.toml @@ -437,10 +437,15 @@ cleanup_max_rows_per_pass = 5000 [security] allowed_hosts = [] cors_origins = [] -# Honor X-Forwarded-For/X-Real-IP only when the immediate peer is listed here. -# Configure only proxies that overwrite or correctly append these headers; -# X-Forwarded-For uses the nearest untrusted hop in the chain. -# An empty list ignores forwarded client-attribution headers. +# Honor forwarded client-attribution headers only when the immediate peer is +# listed here. For browser login over proxy-terminated HTTPS, EggPool trusts +# one X-Forwarded-Proto: https from an exact listed peer IP. Browser sign-in +# and failed personal-token throttling use one literal X-Forwarded-For client +# IP from that peer; the proxy must replace the header, and malformed, +# duplicate, or chained values fall back to the proxy peer. Other attribution +# uses the nearest untrusted hop in the X-Forwarded-For chain. Entries must be +# literal IP addresses, not hostnames or CIDRs. An empty list ignores all +# forwarded headers for trust decisions. # # List both IPv4 and IPv6 loopback addresses explicitly when the proxy is # reached over IPv6 loopback (for example ``::1`` from a client on the @@ -448,6 +453,9 @@ cors_origins = [] # back to the peer address and ignore forwarded headers in that case. # trusted_proxies = ["127.0.0.1", "::1"] trusted_proxies = [] +# Set true only for a deliberately insecure development deployment. Remote +# browser authentication otherwise requires direct HTTPS or a trusted proxy. +allow_insecure_web_auth = false redact_headers = ["authorization", "x-api-key"] persist_redacted_error_detail = false @@ -504,6 +512,15 @@ enabled = false # # proxy_url = "ss://chacha20:password@proxy.example.com:8388" # # proxy_url_env = "GOROUTER_PERSONAL_PROXY_URL" # +# [providers.opencode-go.budget] +# Optional conservative price ceiling for hard USD budget rules, in +# microdollars per million tokens. Omitted means unknown price: hard USD +# rules fail closed for this provider. Zero is an explicit verified +# zero-cost declaration (free local model), distinct from unknown. The +# tariff must cover the most expensive model on this provider; estimator +# fallbacks are never a hard price. +# max_microdollars_per_million_tokens = 3000000 # $3/MTok ceiling +# # [providers.opencode-go.auth] # The selected wire surface supplies endpoint-specific auth. # mode = "bearer" diff --git a/rust/config.sbc.example.toml b/rust/config.sbc.example.toml index 24e53d5ee..e07bda0c6 100644 --- a/rust/config.sbc.example.toml +++ b/rust/config.sbc.example.toml @@ -74,9 +74,11 @@ cleanup_interval_s = 86400 sample_rate = 0.0 [security] -# Honor forwarded client-attribution headers only from these exact peers. -# An empty list ignores X-Forwarded-For/X-Real-IP. +# Honor forwarded client-attribution headers only from exact peer IPs listed +# here. Browser login accepts X-Forwarded-Proto: https only from these peers. +# Use literal IPs (not hostnames or CIDRs); an empty list ignores forwarding. trusted_proxies = [] +allow_insecure_web_auth = false [dashboard] enabled = true diff --git a/rust/src/cli.rs b/rust/src/cli.rs index 832483577..c5f0b706c 100644 --- a/rust/src/cli.rs +++ b/rust/src/cli.rs @@ -49,6 +49,8 @@ pub enum Command { Modelinfo(ModelInfoCommand), #[command(subcommand)] Stats(StatsCommand), + #[command(subcommand)] + Users(UsersCommand), Onboard(OnboardArgs), Croncheck, EnsureRunning, @@ -131,6 +133,194 @@ pub struct NewkeyArgs { pub struct OnboardArgs { #[arg(long, value_name = "PATH")] pub providers: Option, + #[arg(long, default_value = "admin")] + pub admin_username: String, + #[arg(long)] + pub password_stdin: bool, + #[arg(long)] + pub show_secret: bool, +} + +#[derive(Debug, Subcommand)] +pub enum UsersCommand { + /// Create the first administrator and personal inference credential. + Bootstrap(UserPasswordArgs), + /// Show local account names and status without secrets. + List, + /// Reset a user's password and revoke their API credentials. + ResetPassword(UserPasswordArgs), + /// Issue a new personal API credential. Plaintext is shown once. + Token(UserTokenArgs), + /// Revoke every API credential owned by one user. + Revoke(UserNameArgs), + /// Retire legacy access after public dashboard access is disabled. + EnableManaged, + /// Create and assign model-policy profiles during the CLI-only M004 phase. + #[command(subcommand)] + Policy(UsersPolicyCommand), + /// Set or remove durable per-user budget rules. + #[command(subcommand)] + Budget(UsersBudgetCommand), + /// Set or remove instance-wide budget ceilings intersected with every + /// per-user rule. Requires an active administrator. + #[command(subcommand)] + GlobalBudget(UsersGlobalBudgetCommand), +} + +#[derive(Debug, Subcommand)] +pub enum UsersBudgetCommand { + Set(UserBudgetSetArgs), + Remove(UserBudgetRemoveArgs), +} + +#[derive(Debug, Subcommand)] +pub enum UsersGlobalBudgetCommand { + Set(UserGlobalBudgetSetArgs), + Remove(UserGlobalBudgetRemoveArgs), +} + +#[derive(Debug, clap::Args)] +pub struct UserGlobalBudgetSetArgs { + #[arg(long)] + pub model_id: Option, + #[arg(long, value_parser = ["day", "month"])] + pub window: String, + #[arg(long)] + pub max_requests: Option, + #[arg(long)] + pub max_tokens: Option, + #[arg(long)] + pub max_microdollars: Option, + #[arg(long)] + pub max_concurrent: Option, + /// `estimated` keeps estimator projections (default, explicit); `hard` + /// requires a provable worst-case bound and fails closed when the request + /// or price cannot be bounded. + #[arg(long, default_value = "estimated", value_parser = ["hard", "estimated"])] + pub enforcement: String, + #[arg(long, default_value = "admin")] + pub actor: String, +} + +#[derive(Debug, clap::Args)] +pub struct UserGlobalBudgetRemoveArgs { + #[arg(long)] + pub model_id: Option, + #[arg(long, value_parser = ["day", "month"])] + pub window: String, + #[arg(long, default_value = "admin")] + pub actor: String, +} + +#[derive(Debug, clap::Args)] +pub struct UserBudgetSetArgs { + pub username: String, + #[arg(long)] + pub model_id: Option, + #[arg(long, value_parser = ["day", "month"])] + pub window: String, + #[arg(long)] + pub max_requests: Option, + #[arg(long)] + pub max_tokens: Option, + #[arg(long)] + pub max_microdollars: Option, + #[arg(long)] + pub max_concurrent: Option, + /// `estimated` keeps estimator projections (default, explicit); `hard` + /// requires a provable worst-case bound and fails closed when the request + /// or price cannot be bounded. Existing rules stay estimated until an + /// admin explicitly promotes them. + #[arg(long, default_value = "estimated", value_parser = ["hard", "estimated"])] + pub enforcement: String, + #[arg(long, default_value = "admin")] + pub actor: String, +} + +#[derive(Debug, clap::Args)] +pub struct UserBudgetRemoveArgs { + pub username: String, + #[arg(long)] + pub model_id: Option, + #[arg(long, value_parser = ["day", "month"])] + pub window: String, + #[arg(long, default_value = "admin")] + pub actor: String, +} + +#[derive(Debug, Subcommand)] +pub enum UsersPolicyCommand { + /// Create a named profile with an allow-all or default-deny base. + Create(PolicyProfileCreateArgs), + /// Add or replace one exact canonical or model/provider rule. + Rule(PolicyProfileRuleArgs), + /// Assign a profile to one local user. + Assign(PolicyProfileAssignArgs), + /// Set a permitted client recommendation, or clear it when omitted. + Prefer(PolicyProfilePreferArgs), +} + +#[derive(Debug, clap::Args)] +pub struct PolicyProfileCreateArgs { + pub id: String, + pub name: String, + #[arg(value_parser = ["allow", "deny"])] + pub default_effect: String, + #[arg(long, default_value = "admin")] + pub actor: String, +} + +#[derive(Debug, clap::Args)] +pub struct PolicyProfileRuleArgs { + pub profile_id: String, + pub model_id: String, + #[arg(value_parser = ["allow", "deny"])] + pub effect: String, + #[arg(long, default_value = "admin")] + pub actor: String, +} + +#[derive(Debug, clap::Args)] +pub struct PolicyProfileAssignArgs { + pub username: String, + pub profile_id: String, + #[arg(long, default_value = "admin")] + pub actor: String, +} + +#[derive(Debug, clap::Args)] +pub struct PolicyProfilePreferArgs { + pub profile_id: String, + pub model_id: Option, + #[arg(long, default_value = "admin")] + pub actor: String, +} + +#[derive(Debug, clap::Args)] +pub struct UserPasswordArgs { + #[arg(default_value = "admin")] + pub username: String, + /// Read one password line from stdin instead of prompting on a TTY. + #[arg(long)] + pub password_stdin: bool, + /// Allow one-time token output when stdout is not a TTY. + #[arg(long)] + pub show_secret: bool, +} + +#[derive(Debug, clap::Args)] +pub struct UserTokenArgs { + pub username: String, + #[arg(long, default_value = "personal client")] + pub label: String, + /// Allow one-time token output when stdout is not a TTY. + #[arg(long)] + pub show_secret: bool, +} + +#[derive(Debug, clap::Args)] +pub struct UserNameArgs { + pub username: String, } #[derive(Debug, clap::Args)] pub struct TimeoutArgs { @@ -450,6 +640,7 @@ impl Command { Self::Models(_) => "models", Self::Modelinfo(_) => "modelinfo", Self::Stats(_) => "stats", + Self::Users(_) => "users", Self::Onboard(_) => "onboard", Self::Croncheck => "croncheck", Self::EnsureRunning => "ensure-running", diff --git a/rust/src/config.rs b/rust/src/config.rs index 072bd1b6c..30f2c70ec 100644 --- a/rust/src/config.rs +++ b/rust/src/config.rs @@ -452,7 +452,10 @@ impl Default for DashboardConfig { #[serde(deny_unknown_fields, default)] pub struct SecurityConfig { pub allowed_hosts: Vec, + pub allow_insecure_web_auth: bool, pub cors_origins: Vec, + /// Literal proxy IPs whose single `X-Forwarded-Proto: https` value may + /// establish the external scheme for browser authentication. pub trusted_proxies: Vec, pub redact_headers: Vec, pub persist_redacted_error_detail: bool, @@ -463,6 +466,7 @@ impl Default for SecurityConfig { allowed_hosts: Vec::new(), cors_origins: Vec::new(), trusted_proxies: Vec::new(), + allow_insecure_web_auth: false, redact_headers: vec!["authorization".into(), "x-api-key".into()], persist_redacted_error_detail: false, } @@ -715,6 +719,7 @@ pub struct ProviderConfig { pub models_endpoint: Option, pub static_models: Vec, pub verify: ProviderVerifyConfig, + pub budget: ProviderBudgetConfig, } impl Default for ProviderConfig { fn default() -> Self { @@ -748,10 +753,28 @@ impl Default for ProviderConfig { models_endpoint: None, static_models: Vec::new(), verify: ProviderVerifyConfig::default(), + budget: ProviderBudgetConfig::default(), } } } +#[derive(Debug, Clone, Default, Deserialize, Serialize)] +#[serde(deny_unknown_fields, default)] +pub struct ProviderBudgetConfig { + /// Operator-specified conservative price ceiling for this provider in + /// microdollars per million tokens. This is the ONLY price source hard + /// USD budget rules trust: estimator fallbacks (EWMA, family, generic) + /// are never a hard price. `None` means unknown price, so hard USD + /// admission fails closed for this provider. `Some(0)` is an explicit + /// verified zero-cost declaration (free local model), distinct from + /// unknown. Applies to every model dispatched to this provider, so it + /// must cover the most expensive one; per-model refinement is a + /// non-goal. Live-reloadable like other provider fields; each admitted + /// request snapshots the tariff of its generation, and in-flight holds + /// keep their original reservation. + pub max_microdollars_per_million_tokens: Option, +} + #[derive(Debug, Clone, Deserialize, Serialize)] #[serde(deny_unknown_fields, default)] pub struct ModelLimitOverrideConfig { @@ -1548,6 +1571,13 @@ impl Config { } pub fn validate(&mut self) -> Result<(), ConfigError> { + for proxy in &self.security.trusted_proxies { + if proxy.parse::().is_err() { + return Err(ConfigError::validation( + "security.trusted_proxies entries must be literal IP addresses", + )); + } + } let wire_registry = crate::wire::WireProfileRegistry::embedded().map_err(|error| { ConfigError::validation(format!("invalid wire profile registry: {error}")) })?; @@ -1947,6 +1977,15 @@ fn validate_provider( )); } } + if provider + .budget + .max_microdollars_per_million_tokens + .is_some_and(|tariff| tariff > i64::MAX as u64) + { + return Err(ConfigError::validation( + "provider.budget.max_microdollars_per_million_tokens must fit in a signed 64-bit integer", + )); + } if provider .stream_timeouts .max_lifetime_s diff --git a/rust/src/config_reload_policy.rs b/rust/src/config_reload_policy.rs index 3da77228f..5990935c5 100644 --- a/rust/src/config_reload_policy.rs +++ b/rust/src/config_reload_policy.rs @@ -883,6 +883,10 @@ const FIELD_DISPOSITIONS: &[(&str, ReloadDisposition)] = &[ ReloadDisposition::Live, ), ("security.allowed_hosts", ReloadDisposition::RestartRequired), + ( + "security.allow_insecure_web_auth", + ReloadDisposition::RestartRequired, + ), ("security.cors_origins", ReloadDisposition::RestartRequired), ( "security.persist_redacted_error_detail", diff --git a/rust/src/coordinator/endpoints.rs b/rust/src/coordinator/endpoints.rs index dfc5d3934..1eac8ec17 100644 --- a/rust/src/coordinator/endpoints.rs +++ b/rust/src/coordinator/endpoints.rs @@ -113,6 +113,14 @@ pub enum EndpointError { BodyTooLarge, #[error("Duplicate request identity")] PublicationConflict, + #[error("Authentication changed")] + PrincipalChanged, + #[error("Model is not authorized")] + ModelForbidden, + #[error("User budget limit reached")] + BudgetExceeded, + #[error("User budget hard limit is not enforceable for this request")] + BudgetNotEnforceable, #[error("Routing claim failed")] Claim, #[error("Provider attempt failed")] @@ -175,6 +183,9 @@ impl EndpointError { | Self::InvalidStream | Self::Admission => StatusCode::BAD_REQUEST, Self::BodyTooLarge => StatusCode::PAYLOAD_TOO_LARGE, + Self::PrincipalChanged => StatusCode::UNAUTHORIZED, + Self::ModelForbidden => StatusCode::FORBIDDEN, + Self::BudgetExceeded | Self::BudgetNotEnforceable => StatusCode::TOO_MANY_REQUESTS, Self::NoEligibleRoute | Self::Attempt => StatusCode::SERVICE_UNAVAILABLE, Self::Claim | Self::Finalization => StatusCode::INTERNAL_SERVER_ERROR, Self::MissingProvider { .. } | Self::MissingWireProfile { .. } | Self::Internal => { @@ -284,6 +295,7 @@ pub struct InferenceState { max_body_bytes: usize, router: RoutingRouter, catalog_service: Option>, + model_policy: Option, } impl std::fmt::Debug for InferenceState { @@ -329,6 +341,31 @@ impl InferenceState { max_body_bytes: usize, router: RoutingRouter, catalog_service: Option>, + ) -> Self { + Self::from_parts_with_model_policy( + finite, + streaming, + registry, + affinity, + known_providers, + max_body_bytes, + router, + catalog_service, + None, + ) + } + + #[allow(clippy::too_many_arguments)] + pub fn from_parts_with_model_policy( + finite: FiniteCoordinator, + streaming: StreamingCoordinator, + registry: ModelRouterRegistry, + affinity: Arc, + known_providers: BTreeSet, + max_body_bytes: usize, + router: RoutingRouter, + catalog_service: Option>, + model_policy: Option, ) -> Self { let routing_inputs = [ static_routing_facts(&known_providers, ClientSurface::ChatCompletions), @@ -345,6 +382,7 @@ impl InferenceState { max_body_bytes: max_body_bytes.max(1), router, catalog_service, + model_policy, } } @@ -392,6 +430,53 @@ impl InferenceState { self.router.catalog_model_ids() } + pub fn discovery_model_ids(&self) -> Vec { + let mut models = self.catalog_model_ids(); + models.extend(self.registry.virtual_model_ids().map(str::to_owned)); + models.sort(); + models.dedup(); + models + } + + pub fn authorized_discovery_model_ids( + &self, + policy: &crate::model_policy::ModelPolicySnapshot, + ) -> Vec { + let mut models = self + .catalog_model_ids() + .into_iter() + .filter(|model_id| policy.allows_catalog_id(model_id)) + .collect::>(); + models.extend(self.authorized_virtual_model_ids(policy)); + models.sort(); + models.dedup(); + models + } + + pub fn authorized_virtual_model_ids( + &self, + policy: &crate::model_policy::ModelPolicySnapshot, + ) -> Vec { + self.registry + .virtual_model_ids() + .filter(|model_id| { + let Some(router) = self.registry.get(model_id) else { + return false; + }; + policy.allows(model_id, None) + && policy_allows_model_reference( + policy, + &router.selector_model, + &self.known_providers, + ) + && router.routes.iter().any(|route| { + policy_allows_model_reference(policy, &route.model, &self.known_providers) + }) + }) + .map(str::to_owned) + .collect() + } + pub fn max_body_bytes(&self) -> usize { self.max_body_bytes } @@ -407,6 +492,10 @@ impl InferenceState { pub fn catalog_service(&self) -> Option> { self.catalog_service.clone() } + + pub fn model_policy_service(&self) -> Option { + self.model_policy.clone() + } } /// Virtual-router resolution facts (secret-free). @@ -486,6 +575,9 @@ fn map_stream_error(error: super::StreamingCoordinatorError) -> EndpointError { fn map_publication_error(error: super::PublicationError) -> EndpointError { match error { super::PublicationError::DuplicateConflict { .. } => EndpointError::PublicationConflict, + super::PublicationError::PrincipalChanged => EndpointError::PrincipalChanged, + super::PublicationError::BudgetExceeded => EndpointError::BudgetExceeded, + super::PublicationError::BudgetNotEnforceable => EndpointError::BudgetNotEnforceable, super::PublicationError::InvalidInput(_) => EndpointError::Admission, super::PublicationError::ClaimIdentity(_) | super::PublicationError::Claim(_) @@ -584,12 +676,38 @@ async fn resolve_concrete( mut parsed: ParsedRequestBody, session_header: Option<&str>, proxy_request_id: &str, + attribution: Option<&crate::identity::RequestAttribution>, ) -> Result { let model_value = model_field(parsed.object()?)?; + let policy = match attribution.and_then(|principal| principal.user_id.as_deref()) { + Some(user_id) => Some( + state + .model_policy + .as_ref() + .ok_or(EndpointError::Internal)? + .snapshot(user_id) + .await + .map_err(|error| match error { + crate::model_policy::ModelPolicyError::UserUnavailable => { + EndpointError::PrincipalChanged + } + crate::model_policy::ModelPolicyError::InvalidPolicyInput + | crate::model_policy::ModelPolicyError::PreferredModelDenied + | crate::model_policy::ModelPolicyError::Database(_) => EndpointError::Internal, + })?, + ), + None => None, + }; + let (requested_model, requested_provider) = + parse_provider_qualified_model(&model_value, &state.known_providers); + ensure_model_allowed( + policy.as_deref(), + &requested_model, + requested_provider.as_deref(), + )?; // Exact virtual-alias match comes before concrete provider parsing. let Some(router) = state.registry.get(model_value.trim()) else { - let (model_id, provider_id) = - parse_provider_qualified_model(&model_value, &state.known_providers); + let (model_id, provider_id) = (requested_model, requested_provider); // Strip any provider qualifier for the dispatched body: the upstream // does not understand EggPool's namespace. The qualifier survives as // a routing pin, matching the Python `provider_id` context field. @@ -624,10 +742,42 @@ async fn resolve_concrete( &router, session_header, proxy_request_id, + attribution, + policy.as_deref(), ) .await } +fn ensure_model_allowed( + policy: Option<&crate::model_policy::ModelPolicySnapshot>, + model_id: &str, + provider_id: Option<&str>, +) -> Result<(), EndpointError> { + if policy.is_some_and(|policy| !policy.allows(model_id, provider_id)) { + return Err(EndpointError::ModelForbidden); + } + Ok(()) +} + +fn ensure_concrete_model_allowed( + policy: Option<&crate::model_policy::ModelPolicySnapshot>, + model_name: &str, + known_providers: &BTreeSet, +) -> Result<(), EndpointError> { + let (model_id, provider_id) = parse_provider_qualified_model(model_name, known_providers); + ensure_model_allowed(policy, &model_id, provider_id.as_deref()) +} + +fn policy_allows_model_reference( + policy: &crate::model_policy::ModelPolicySnapshot, + model_name: &str, + known_providers: &BTreeSet, +) -> bool { + let (model_id, provider_id) = parse_provider_qualified_model(model_name, known_providers); + policy.allows(&model_id, provider_id.as_deref()) +} + +#[allow(clippy::too_many_arguments)] // Keeps immutable request, identity, and policy facts explicit. async fn resolve_virtual( state: &InferenceState, surface: ClientSurface, @@ -635,6 +785,8 @@ async fn resolve_virtual( router: &CompiledModelRouter, session_header: Option<&str>, proxy_request_id: &str, + attribution: Option<&crate::identity::RequestAttribution>, + policy: Option<&crate::model_policy::ModelPolicySnapshot>, ) -> Result { // Build the early canonical view for the semantic prompt and affinity // identity. Admission here is bounded by the same body ceiling. @@ -646,7 +798,9 @@ async fn resolve_virtual( let known = state.known_providers.clone(); let registry_virtual = state.registry.clone(); let selector = SemanticSelector::new(state.finite.clone(), known) - .with_virtual_check(move |model| registry_virtual.is_virtual(model)); + .with_virtual_check(move |model| registry_virtual.is_virtual(model)) + .with_attribution(attribution.map(|principal| principal.selector_child(proxy_request_id))); + ensure_concrete_model_allowed(policy, &router.selector_model, &state.known_providers)?; if router.sticky && let Some(identity) = identity { @@ -666,7 +820,7 @@ async fn resolve_virtual( crate::model_router::AffinityDecisionSource::Selector => SelectionSource::Selector, crate::model_router::AffinityDecisionSource::Default => SelectionSource::Default, }; - return finish_virtual_resolution( + let resolved = finish_virtual_resolution( state, surface, parsed, @@ -677,11 +831,13 @@ async fn resolve_virtual( source, resolution.cache_hit, resolution.decision.selector_attempts_for_metrics(), - ); + )?; + ensure_concrete_model_allowed(policy, &resolved.concrete_model, &state.known_providers)?; + return Ok(resolved); } // Non-sticky, no session identity, or sticky=false bypass: direct select. let selection = selector.select(router, &canonical, surface).await; - finish_virtual_resolution( + let resolved = finish_virtual_resolution( state, surface, parsed, @@ -692,11 +848,11 @@ async fn resolve_virtual( selection.source, false, selection.selector_attempts, - ) - .map(|mut resolved| { - resolved.selector_latency_ms = selection.selector_latency_ms; - resolved - }) + )?; + ensure_concrete_model_allowed(policy, &resolved.concrete_model, &state.known_providers)?; + let mut resolved = resolved; + resolved.selector_latency_ms = selection.selector_latency_ms; + Ok(resolved) } async fn selector_affinity_selection( @@ -805,6 +961,27 @@ pub async fn execute_endpoint( incoming_headers: HeaderMap, session_header: Option, proxy_request_id: String, +) -> Result { + execute_endpoint_attributed( + state, + surface, + raw_body, + incoming_headers, + session_header, + proxy_request_id, + None, + ) + .await +} + +pub(crate) async fn execute_endpoint_attributed( + state: &InferenceState, + surface: ClientSurface, + raw_body: Bytes, + incoming_headers: HeaderMap, + session_header: Option, + proxy_request_id: String, + attribution: Option, ) -> Result { let parsed = parse_request_body(raw_body, state.max_body_bytes).map_err(|error| match error { @@ -820,6 +997,7 @@ pub async fn execute_endpoint( parsed, session_header.as_deref(), &proxy_request_id, + attribution.as_ref(), ) .await?; let admitted = admit_resolved( @@ -845,11 +1023,16 @@ pub async fn execute_endpoint( // `from_admitted` only fails with `InvalidFacts`, which the typed // mapper already classifies as `Admission`. .map_err(map_stream_error)?; - let execution = state - .streaming - .execute(request) - .await - .map_err(map_stream_error)?; + let execution = match attribution.clone() { + Some(attribution) => { + state + .streaming + .execute_attributed(request, attribution) + .await + } + None => state.streaming.execute(request).await, + } + .map_err(map_stream_error)?; Ok(EndpointExecution::Stream( Box::new(execution), resolved.virtual_resolution, @@ -866,11 +1049,11 @@ pub async fn execute_endpoint( // `from_admitted` only fails with `InvalidFacts`, which the typed // mapper already classifies as `Admission`. .map_err(map_finite_error)?; - let execution = state - .finite - .execute(request) - .await - .map_err(map_finite_error)?; + let execution = match attribution { + Some(attribution) => state.finite.execute_attributed(request, attribution).await, + None => state.finite.execute(request).await, + } + .map_err(map_finite_error)?; Ok(EndpointExecution::Finite( Box::new(execution), resolved.virtual_resolution, @@ -896,6 +1079,25 @@ pub async fn execute_compact_finite( incoming_headers: HeaderMap, session_header: Option, proxy_request_id: String, +) -> Result<(FiniteExecution, Option), EndpointError> { + execute_compact_finite_attributed( + state, + raw_body, + incoming_headers, + session_header, + proxy_request_id, + None, + ) + .await +} + +pub(crate) async fn execute_compact_finite_attributed( + state: &InferenceState, + raw_body: Bytes, + incoming_headers: HeaderMap, + session_header: Option, + proxy_request_id: String, + attribution: Option, ) -> Result<(FiniteExecution, Option), EndpointError> { const SURFACE: ClientSurface = ClientSurface::Responses; let parsed = @@ -920,6 +1122,7 @@ pub async fn execute_compact_finite( parsed, session_header.as_deref(), &proxy_request_id, + attribution.as_ref(), ) .await?; let compact = admit_compact_parsed_request( @@ -947,6 +1150,7 @@ pub async fn execute_compact_finite( incoming_headers, compact, routing_facts, + attribution, ) .await .map_err(map_finite_error)?; @@ -1200,7 +1404,7 @@ pub(crate) async fn build_inference_state_with_shared_and_accounts( retry_policy, ); let known_providers: BTreeSet = config.providers.keys().cloned().collect(); - Ok(InferenceState::from_parts_with_catalog_service( + Ok(InferenceState::from_parts_with_model_policy( finite, streaming, model_registry, @@ -1209,6 +1413,9 @@ pub(crate) async fn build_inference_state_with_shared_and_accounts( config.server.max_request_body_bytes as usize, router, Some(catalog_service), + Some(crate::model_policy::ModelPolicyService::new( + database.clone(), + )), )) } diff --git a/rust/src/coordinator/finalization.rs b/rust/src/coordinator/finalization.rs index bd6e2db2e..649b116d7 100644 --- a/rust/src/coordinator/finalization.rs +++ b/rust/src/coordinator/finalization.rs @@ -328,7 +328,8 @@ impl DurableFinalizer { move |connection| { let request = connection .query_row( - "SELECT account_id, model_id, provider_id, protocol, streamed, status + "SELECT account_id, model_id, provider_id, protocol, streamed, status, + user_id, credential_id, auth_epoch, principal_kind, request_kind, parent_proxy_request_id FROM requests WHERE id = ?1", [identity.db_request_id], |row| { @@ -339,11 +340,30 @@ impl DurableFinalizer { row.get::<_, String>(3)?, row.get::<_, i64>(4)?, row.get::<_, String>(5)?, + row.get::<_, Option>(6)?, + row.get::<_, Option>(7)?, + row.get::<_, Option>(8)?, + row.get::<_, String>(9)?, + row.get::<_, String>(10)?, + row.get::<_, Option>(11)?, )) }, ) .optional()?; - let Some((account_id, model_id, provider_id, protocol, _streamed, current)) = request + let Some(( + account_id, + model_id, + provider_id, + protocol, + _streamed, + current, + user_id, + credential_id, + auth_epoch, + principal_kind, + request_kind, + parent_proxy_request_id, + )) = request else { return Ok(TxnResult::Invariant { entity: "request", @@ -353,6 +373,12 @@ impl DurableFinalizer { }; let request_identity_matches = model_id == identity.model_id && protocol == identity.client_protocol + && user_id == identity.user_id + && credential_id == identity.credential_id + && auth_epoch == identity.auth_epoch + && principal_kind == identity.principal_kind.as_str() + && request_kind == identity.request_kind.as_str() + && parent_proxy_request_id == identity.parent_proxy_request_id && (!terminalize_request || (account_id == identity.account_id && provider_id == identity.provider_id)); @@ -570,6 +596,53 @@ impl DurableFinalizer { return Ok(TxnResult::Invariant { entity: "reservation", id: identity.reservation_id, reason: "zero-row reservation transition did not converge".into() }); } } + crate::user_budget::record_attempt_and_maybe_settle( + connection, + crate::user_budget::BudgetAttemptSettlement { + request_id: identity.db_request_id, + attempt_id: identity.attempt_id, + attempt_transitioned: attempt_changed == 1, + terminalize_request, + input_tokens: data.input_tokens, + output_tokens: data.output_tokens, + cost_microdollars: data.cost_microdollars, + provider_reported_microdollars: data.provider_cost_microdollars, + local_estimated_microdollars: data.local_cost_microdollars, + tokens_known: data.input_tokens_reported.is_some() + || data.output_tokens_reported.is_some() + || data.total_tokens_reported.is_some() + || data.input_tokens > 0 + || data.output_tokens > 0, + cost_known: data.provider_cost_microdollars.is_some() + || data.local_cost_microdollars.is_some() + || data.cost_microdollars > 0, + }, + )?; + // Instance-global settlement folds the same attempt facts and is + // a no-op when the request never took an instance hold, so + // rule-free and unattributed requests keep their existing shape. + crate::instance_budget::record_attempt_and_maybe_settle( + connection, + crate::user_budget::BudgetAttemptSettlement { + request_id: identity.db_request_id, + attempt_id: identity.attempt_id, + attempt_transitioned: attempt_changed == 1, + terminalize_request, + input_tokens: data.input_tokens, + output_tokens: data.output_tokens, + cost_microdollars: data.cost_microdollars, + provider_reported_microdollars: data.provider_cost_microdollars, + local_estimated_microdollars: data.local_cost_microdollars, + tokens_known: data.input_tokens_reported.is_some() + || data.output_tokens_reported.is_some() + || data.total_tokens_reported.is_some() + || data.input_tokens > 0 + || data.output_tokens > 0, + cost_known: data.provider_cost_microdollars.is_some() + || data.local_cost_microdollars.is_some() + || data.cost_microdollars > 0, + }, + )?; Ok(TxnResult::Success { request_terminal: terminalize_request, request_transitioned, diff --git a/rust/src/coordinator/finite.rs b/rust/src/coordinator/finite.rs index 564da81b3..5b13d4832 100644 --- a/rust/src/coordinator/finite.rs +++ b/rust/src/coordinator/finite.rs @@ -36,7 +36,8 @@ use super::{ FinalizationError, FinalizationIdentity, FinalizationOutcome, FinalizationResult, FinalizationSupervisor, PostCommitInterruption, PublicationError, PublicationInput, PublicationOutcome, PublicationService, PublishedAttempt, RetryPolicy, - RuntimePublicationReceipt, WireResolver, release_claim, retry_after_from_headers, + RuntimePublicationReceipt, WireResolver, budget_proof_for_claim, release_claim, + retry_after_from_headers, }; const MAX_CLIENT_ERROR_BYTES: usize = 512; @@ -264,6 +265,7 @@ struct FiniteExecutionInput { routing_facts: RoutingRequestFacts, operation: super::endpoints::InferenceOperation, admission: FiniteAdmission, + attribution: Option, } impl FiniteExecutionInput { @@ -296,6 +298,7 @@ impl FiniteExecutionInput { routing_facts, operation, admission, + attribution: None, } } @@ -316,6 +319,7 @@ impl FiniteExecutionInput { routing_facts, operation: super::endpoints::InferenceOperation::Compact, admission: FiniteAdmission::Compact(Some(compact)), + attribution: None, } } @@ -388,6 +392,7 @@ impl FiniteRequest { raw_body_bytes: compact.raw_body_bytes, reservation_tokens: compact.reservation_tokens, context_tokens: compact.context_tokens, + budget_facts: compact.budget_facts, }; Ok(Self { proxy_request_id: proxy_request_id.into(), @@ -431,6 +436,7 @@ impl FiniteRequest { raw_body_bytes: compact.raw_body_bytes, reservation_tokens: compact.reservation_tokens, context_tokens: compact.context_tokens, + budget_facts: compact.budget_facts, }; Ok(Self { proxy_request_id: proxy_request_id.into(), @@ -558,10 +564,22 @@ impl FiniteCoordinator { &self, request: FiniteRequest, ) -> Result { - self.execute_input(FiniteExecutionInput::from_public(request)) + self.execute_input(FiniteExecutionInput::from_public(request), None) .await } + pub(crate) async fn execute_attributed( + &self, + request: FiniteRequest, + attribution: crate::identity::RequestAttribution, + ) -> Result { + self.execute_input( + FiniteExecutionInput::from_public(request), + Some(attribution), + ) + .await + } + /// Execute a production compact request without constructing the public /// compatibility `FiniteRequest` view. The compact admission remains the /// sole owner of the preserved source-native JSON tree for this path. @@ -572,21 +590,26 @@ impl FiniteCoordinator { incoming_headers: HeaderMap, compact: CompactAdmittedRequest, routing_facts: RoutingRequestFacts, + attribution: Option, ) -> Result { - self.execute_input(FiniteExecutionInput::compact( + let mut input = FiniteExecutionInput::compact( proxy_request_id, raw_body, incoming_headers, compact, routing_facts, - )) - .await + ); + input.attribution = attribution; + let attribution = input.attribution.clone(); + self.execute_input(input, attribution).await } async fn execute_input( &self, - request: FiniteExecutionInput, + mut request: FiniteExecutionInput, + attribution: Option, ) -> Result { + request.attribution = attribution; let Some(canonical) = request.canonical() else { return Err(FiniteCoordinatorError::InvalidFacts); }; @@ -768,14 +791,29 @@ impl FiniteCoordinator { .iter() .any(|candidate| !attempted.contains(&candidate.surface())); let upstream_protocol = protocol_for_surface(candidate.surface()); - let publication_input = PublicationInput::new( + let mut publication_input = PublicationInput::new( request.proxy_request_id.clone(), request.client_surface.protocol(), upstream_protocol, false, i64::from(attempt_number), ); - let published = match self.publication.publish(claim, publication_input).await { + // Worst-case budget proof travels with the publication so the + // atomic hold decision sees the admission output bound and the + // operator tariff for the selected provider. The tariff snapshot + // belongs to this generation; in-flight holds keep theirs. + publication_input.budget_proof = Some(budget_proof_for_claim( + &claim, + provider + .budget + .max_microdollars_per_million_tokens + .and_then(|tariff| i64::try_from(tariff).ok()), + )); + let published = match self + .publication + .publish_attributed(claim, publication_input, request.attribution.clone()) + .await + { Ok(PublicationOutcome::Published(value)) => value, Ok(PublicationOutcome::AlreadyPublished(_)) => { return Err(FiniteCoordinatorError::Publication( diff --git a/rust/src/coordinator/mod.rs b/rust/src/coordinator/mod.rs index 832ab5fd8..2b31c38dc 100644 --- a/rust/src/coordinator/mod.rs +++ b/rust/src/coordinator/mod.rs @@ -39,7 +39,7 @@ pub use finite::{ pub use publication::{ FinalizationIdentity, PostCommitInterruption, PublicationError, PublicationFaultInjector, PublicationInput, PublicationOutcome, PublicationService, PublicationStage, PublishedAttempt, - RuntimePublicationReceipt, + RuntimePublicationReceipt, budget_proof_for_claim, }; pub use reconciliation::{ @@ -66,7 +66,7 @@ pub use endpoints::{ }; pub(crate) use endpoints::{ build_inference_state_with_shared, build_inference_state_with_shared_and_accounts, - compile_provider_profiles, + compile_provider_profiles, execute_compact_finite_attributed, execute_endpoint_attributed, }; pub(crate) use finalization::release_claim; pub use semantic::{ diff --git a/rust/src/coordinator/publication.rs b/rust/src/coordinator/publication.rs index 367a83843..40adbda13 100644 --- a/rust/src/coordinator/publication.rs +++ b/rust/src/coordinator/publication.rs @@ -35,6 +35,12 @@ pub struct FinalizationIdentity { pub client_protocol: String, pub upstream_protocol: String, pub attempt_number: i64, + pub user_id: Option, + pub credential_id: Option, + pub auth_epoch: Option, + pub principal_kind: crate::identity::PrincipalKind, + pub request_kind: crate::identity::AttributedRequestKind, + pub parent_proxy_request_id: Option, } /// The local ownership conversion record for one durable publication. @@ -68,6 +74,10 @@ pub struct PublicationInput { pub streamed: bool, pub attempt_number: i64, pub reservation_ttl_seconds: i64, + /// Worst-case budget proof assembled by the coordinator from admission + /// facts and the operator tariff snapshot. `None` preserves the + /// estimated-mode path; hard rules fail closed on `None`. + pub budget_proof: Option, } impl PublicationInput { @@ -85,6 +95,7 @@ impl PublicationInput { streamed, attempt_number, reservation_ttl_seconds: DEFAULT_RESERVATION_TTL_SECONDS, + budget_proof: None, } } } @@ -185,6 +196,12 @@ pub enum PublicationError { Injected { stage: PublicationStage }, #[error("duplicate publication conflicts with existing request identity {proxy_request_id:?}")] DuplicateConflict { proxy_request_id: String }, + #[error("authenticated request principal changed before durable publication")] + PrincipalChanged, + #[error("user budget limit reached")] + BudgetExceeded, + #[error("user budget hard limit is not enforceable for this request")] + BudgetNotEnforceable, #[error("replacement attempt cannot publish before the prior attempt converges")] PriorAttemptNotFinalized, #[error("claim transition failed: {0}")] @@ -227,7 +244,7 @@ pub enum PublicationOutcome { Published(Box), /// A duplicate invocation observed the already-published attempt. The /// incoming claim has been rolled back and must not be released again. - AlreadyPublished(FinalizationIdentity), + AlreadyPublished(Box), } #[derive(Debug)] @@ -238,8 +255,11 @@ enum TransactionOutcome { reservation_id: i64, routing_decision_id: i64, }, - Observed(FinalizationIdentity), + Observed(Box), Conflict, + PrincipalChanged, + BudgetExceeded, + BudgetNotEnforceable, PriorAttemptNotFinalized, } @@ -271,11 +291,20 @@ impl PublicationService { &self, claim: SelectionClaim, input: PublicationInput, + ) -> Result { + self.publish_attributed(claim, input, None).await + } + + pub async fn publish_attributed( + &self, + claim: SelectionClaim, + input: PublicationInput, + attribution: Option, ) -> Result { let service = self.clone(); let (sender, receiver) = oneshot::channel(); tokio::spawn(async move { - let outcome = service.publish_owned(claim, input).await; + let outcome = service.publish_owned(claim, input, attribution).await; if let Err(outcome) = sender.send(outcome) { service.compensate_lost_delivery(outcome).await; } @@ -291,7 +320,11 @@ impl PublicationService { &self, claim: SelectionClaim, input: PublicationInput, + attribution: Option, ) -> Result { + let attribution = attribution.unwrap_or_else(|| { + crate::identity::AuthenticatedPrincipal::legacy_shared_key().request_attribution() + }); if let Err(error) = self.validate(&claim, &input) { return self.rollback_after_error(claim, error); } @@ -306,7 +339,9 @@ impl PublicationService { Ok(prepared) => prepared, Err(error) => return self.rollback_after_error(claim, error), }; - let transaction = self.publish_transaction(&claim, &input, prepared).await; + let transaction = self + .publish_transaction(&claim, &input, &attribution, prepared) + .await; let transaction = match transaction { Ok(value) => value, Err(error) => return self.rollback_after_error(claim, error), @@ -314,6 +349,15 @@ impl PublicationService { if matches!(transaction, TransactionOutcome::PriorAttemptNotFinalized) { return self.rollback_after_error(claim, PublicationError::PriorAttemptNotFinalized); } + if matches!(transaction, TransactionOutcome::PrincipalChanged) { + return self.rollback_after_error(claim, PublicationError::PrincipalChanged); + } + if matches!(transaction, TransactionOutcome::BudgetExceeded) { + return self.rollback_after_error(claim, PublicationError::BudgetExceeded); + } + if matches!(transaction, TransactionOutcome::BudgetNotEnforceable) { + return self.rollback_after_error(claim, PublicationError::BudgetNotEnforceable); + } let TransactionOutcome::Created { request_id, @@ -338,6 +382,15 @@ impl PublicationService { proxy_request_id: input.proxy_request_id, }, ), + TransactionOutcome::PrincipalChanged => { + self.rollback_after_error(claim, PublicationError::PrincipalChanged) + } + TransactionOutcome::BudgetExceeded => { + self.rollback_after_error(claim, PublicationError::BudgetExceeded) + } + TransactionOutcome::BudgetNotEnforceable => { + self.rollback_after_error(claim, PublicationError::BudgetNotEnforceable) + } TransactionOutcome::Created { .. } => Err(PublicationError::InvalidInput( "unexpected created outcome after duplicate-publication branch".to_owned(), )), @@ -347,7 +400,14 @@ impl PublicationService { }; }; - let identity = self.identity(&claim, &input, request_id, attempt_id, reservation_id); + let identity = self.identity( + &claim, + &input, + &attribution, + request_id, + attempt_id, + reservation_id, + ); let receipt = RuntimePublicationReceipt { pending_request_added: true, pending_tokens_added: true, @@ -419,10 +479,12 @@ impl PublicationService { &self, claim: &SelectionClaim, input: &PublicationInput, + attribution: &crate::identity::RequestAttribution, prepared: PreparedRoutingDecisionRow, ) -> Result { let database = self.database.clone(); let input = input.clone(); + let attribution = attribution.clone(); let account_name = claim.account_name().to_owned(); let account_id = claim.account_id(); let provider_id = claim.provider_id().to_owned(); @@ -434,9 +496,47 @@ impl PublicationService { let transaction_injector = injector.clone(); let result = database .with_named_transaction(crate::db::TransactionKind::Publication, move |connection| { + if let Some(user_id) = attribution.user_id.as_deref() { + let current: Option<(String, i64)> = connection + .query_row( + "SELECT state, auth_epoch FROM users WHERE id = ?1", + [user_id], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .optional()?; + if !current.is_some_and(|(state, epoch)| { + state == "active" && Some(epoch) == attribution.auth_epoch + }) { + return Ok(TransactionOutcome::PrincipalChanged); + } + if let Some(credential_id) = attribution.credential_id.as_deref() { + let credential_active: bool = match attribution.principal_kind { + crate::identity::PrincipalKind::AgentToken => connection.query_row( + "SELECT EXISTS(SELECT 1 FROM user_credentials \ + WHERE id = ?1 AND user_id = ?2 AND revoked_at IS NULL \ + AND (expires_at IS NULL OR expires_at > CURRENT_TIMESTAMP))", + (credential_id, user_id), + |row| row.get(0), + )?, + crate::identity::PrincipalKind::BrowserSession => connection.query_row( + "SELECT EXISTS(SELECT 1 FROM web_sessions \ + WHERE id = ?1 AND user_id = ?2 AND revoked_at IS NULL \ + AND expires_at > CURRENT_TIMESTAMP \ + AND last_used_at > datetime('now', '-12 hours'))", + (credential_id, user_id), + |row| row.get(0), + )?, + crate::identity::PrincipalKind::LegacySharedKey => true, + }; + if !credential_active { + return Ok(TransactionOutcome::PrincipalChanged); + } + } + } let existing = connection .query_row( - "SELECT id, account_id, model_id, provider_id, protocol, streamed, status\n\ + "SELECT id, account_id, model_id, provider_id, protocol, streamed, status,\n\ + user_id, credential_id, auth_epoch, principal_kind, request_kind, parent_proxy_request_id\n\ FROM requests WHERE proxy_request_id = ?1", [&input.proxy_request_id], |row| { @@ -448,6 +548,12 @@ impl PublicationService { row.get::<_, String>(4)?, row.get::<_, i64>(5)?, row.get::<_, String>(6)?, + row.get::<_, Option>(7)?, + row.get::<_, Option>(8)?, + row.get::<_, Option>(9)?, + row.get::<_, String>(10)?, + row.get::<_, String>(11)?, + row.get::<_, Option>(12)?, )) }, ) @@ -461,11 +567,23 @@ impl PublicationService { existing_protocol, existing_streamed, existing_status, + existing_user_id, + existing_credential_id, + existing_auth_epoch, + existing_principal_kind, + existing_request_kind, + existing_parent_proxy_request_id, )) = existing { if existing_model_id != model_id || existing_protocol != input.client_protocol || existing_streamed != i64::from(input.streamed) + || existing_user_id != attribution.user_id + || existing_credential_id != attribution.credential_id + || existing_auth_epoch != attribution.auth_epoch + || existing_principal_kind != attribution.principal_kind.as_str() + || existing_request_kind != attribution.request_kind.as_str() + || existing_parent_proxy_request_id != attribution.parent_proxy_request_id { return Ok(TransactionOutcome::Conflict); } @@ -519,7 +637,7 @@ impl PublicationService { [existing_account_id], |row| row.get::<_, String>(0), )?; - return Ok(TransactionOutcome::Observed(FinalizationIdentity { + return Ok(TransactionOutcome::Observed(Box::new(FinalizationIdentity { proxy_request_id: input.proxy_request_id, db_request_id: request_id, attempt_id, @@ -532,7 +650,13 @@ impl PublicationService { client_protocol: input.client_protocol, upstream_protocol, attempt_number: input.attempt_number, - })); + user_id: attribution.user_id, + credential_id: attribution.credential_id, + auth_epoch: attribution.auth_epoch, + principal_kind: attribution.principal_kind, + request_kind: attribution.request_kind, + parent_proxy_request_id: attribution.parent_proxy_request_id, + }))); } if input.attempt_number > 1 { let prior_attempt = connection @@ -563,6 +687,60 @@ impl PublicationService { return Ok(TransactionOutcome::PriorAttemptNotFinalized); } } + if attribution.user_id.is_some() { + let (retry_check, retry_hold) = crate::user_budget::check_retry( + connection, + request_id, + projected_tokens, + projected_cost, + input.budget_proof.as_ref(), + )?; + match retry_check { + crate::user_budget::BudgetCheck::Exceeded => { + return Ok(TransactionOutcome::BudgetExceeded); + } + crate::user_budget::BudgetCheck::NotEnforceable => { + return Ok(TransactionOutcome::BudgetNotEnforceable); + } + crate::user_budget::BudgetCheck::Allowed => {} + } + crate::user_budget::extend_reservation( + connection, + request_id, + retry_hold.tokens, + retry_hold.microdollars, + )?; + } + // Instance-global intersection applies to every retry, + // attributed or not. Requests that never took an + // instance hold skip this path entirely. + { + let (instance_check, instance_hold) = + crate::instance_budget::check_retry( + connection, + request_id, + projected_tokens, + projected_cost, + input.budget_proof.as_ref(), + )?; + match instance_check { + crate::user_budget::BudgetCheck::Exceeded => { + return Ok(TransactionOutcome::BudgetExceeded); + } + crate::user_budget::BudgetCheck::NotEnforceable => { + return Ok(TransactionOutcome::BudgetNotEnforceable); + } + crate::user_budget::BudgetCheck::Allowed => {} + } + if let Some(hold) = instance_hold { + crate::instance_budget::extend_reservation( + connection, + request_id, + hold.tokens, + hold.microdollars, + )?; + } + } // A retry attempt on an existing pending request is valid // only after the previous attempt's durable and local // release boundary has converged. @@ -589,6 +767,52 @@ impl PublicationService { ); } + let budget_hold = if let Some(user_id) = attribution.user_id.as_deref() { + let (check, permit, hold) = crate::user_budget::check_initial( + connection, + user_id, + &model_id, + projected_tokens, + projected_cost, + input.budget_proof.as_ref(), + )?; + match check { + crate::user_budget::BudgetCheck::Exceeded => { + return Ok(TransactionOutcome::BudgetExceeded); + } + crate::user_budget::BudgetCheck::NotEnforceable => { + return Ok(TransactionOutcome::BudgetNotEnforceable); + } + crate::user_budget::BudgetCheck::Allowed => {} + } + Some((permit, hold)) + } else { + None + }; + // Instance-global intersection covers every admitted call, + // including unattributed legacy-key requests, so no identity + // switch can bypass a shared ceiling. With no matching + // instance rule this takes no hold and writes no rows. + let instance_hold = { + let (check, permit, hold) = crate::instance_budget::check_initial( + connection, + &model_id, + projected_tokens, + projected_cost, + input.budget_proof.as_ref(), + )?; + match check { + crate::user_budget::BudgetCheck::Exceeded => { + return Ok(TransactionOutcome::BudgetExceeded); + } + crate::user_budget::BudgetCheck::NotEnforceable => { + return Ok(TransactionOutcome::BudgetNotEnforceable); + } + crate::user_budget::BudgetCheck::Allowed => {} + } + permit.map(|permit| (permit, hold)) + }; + fail_sql( transaction_injector.as_ref(), PublicationStage::RequestInsert, @@ -596,9 +820,11 @@ impl PublicationService { connection.execute( "INSERT INTO requests\n\ (account_id, model_id, status, protocol, streamed,\n\ - reserved_microdollars, proxy_request_id, provider_id, first_attempt_at)\n\ + reserved_microdollars, proxy_request_id, provider_id, first_attempt_at,\n\ + user_id, credential_id, auth_epoch, principal_kind, request_kind, parent_proxy_request_id)\n\ VALUES (?1, ?2, 'pending', ?3, ?4, ?5, ?6, ?7,\n\ - CASE WHEN ?8 = 1 THEN CURRENT_TIMESTAMP ELSE NULL END)", + CASE WHEN ?8 = 1 THEN CURRENT_TIMESTAMP ELSE NULL END,\n\ + ?9, ?10, ?11, ?12, ?13, ?14)", params![ account_id, model_id, @@ -608,9 +834,46 @@ impl PublicationService { input.proxy_request_id, provider_id, i64::from(input.attempt_number == 1), + attribution.user_id, + attribution.credential_id, + attribution.auth_epoch, + attribution.principal_kind.as_str(), + attribution.request_kind.as_str(), + attribution.parent_proxy_request_id, ], )?; let request_id = connection.last_insert_rowid(); + if let (Some(user_id), Some((permit, hold))) = + (attribution.user_id.as_deref(), budget_hold.as_ref()) + { + crate::user_budget::insert_reservation( + connection, + crate::user_budget::BudgetReservationSpec { + request_id, + user_id, + model_id: &model_id, + tokens: hold.tokens, + microdollars: hold.microdollars, + ttl_seconds: input.reservation_ttl_seconds, + hold: *hold, + }, + permit, + )?; + } + if let Some((permit, hold)) = instance_hold.as_ref() { + crate::instance_budget::insert_reservation( + connection, + crate::instance_budget::InstanceReservationSpec { + request_id, + model_id: &model_id, + tokens: hold.tokens, + microdollars: hold.microdollars, + ttl_seconds: input.reservation_ttl_seconds, + hold: *hold, + }, + permit, + )?; + } insert_attempt_rows( connection, transaction_injector.as_ref(), @@ -747,6 +1010,7 @@ impl PublicationService { &self, claim: &SelectionClaim, input: &PublicationInput, + attribution: &crate::identity::RequestAttribution, request_id: i64, attempt_id: i64, reservation_id: i64, @@ -764,6 +1028,12 @@ impl PublicationService { client_protocol: input.client_protocol.clone(), upstream_protocol: input.upstream_protocol.clone(), attempt_number: input.attempt_number, + user_id: attribution.user_id.clone(), + credential_id: attribution.credential_id.clone(), + auth_epoch: attribution.auth_epoch, + principal_kind: attribution.principal_kind, + request_kind: attribution.request_kind, + parent_proxy_request_id: attribution.parent_proxy_request_id.clone(), } } @@ -829,9 +1099,25 @@ fn fail_sql( Ok(()) } +/// Assemble the worst-case budget proof for one selected claim from the +/// admission-carried facts on its snapshot and the operator tariff snapshot +/// for the selected provider. `None` tariff means unknown price: hard USD +/// rules fail closed downstream, estimated rules are unaffected. +pub fn budget_proof_for_claim( + claim: &SelectionClaim, + tariff_microdollars_per_million_tokens: Option, +) -> crate::user_budget::BudgetProof { + crate::user_budget::BudgetProof { + input_worst_case_tokens: claim.budget_input_worst_case_tokens(), + requested_output_tokens: claim.budget_requested_output_tokens(), + tariff_microdollars_per_million_tokens, + } +} + /// Deterministic routing-decision persistence facts prepared from a borrowed /// selection snapshot before the database gate is acquired (persistence /// M002). Only the bounded scalar/JSON facts that `insert_attempt_rows` +/// M002). Only the bounded scalar/JSON facts that `insert_attempt_rows` /// persists travel into the worker transaction; the full runtime snapshot /// never crosses that boundary. #[derive(Debug, Clone)] diff --git a/rust/src/coordinator/reconciliation.rs b/rust/src/coordinator/reconciliation.rs index e11ad8985..405985522 100644 --- a/rust/src/coordinator/reconciliation.rs +++ b/rust/src/coordinator/reconciliation.rs @@ -34,16 +34,24 @@ //! - failed-attempt cleanup pending while the request is still nonterminal //! → fail closed: the request is terminalized as `interrupted`, never //! left retryable and never replayed upstream; +//! - pending request with an active user or instance budget hold → the +//! request and its holds converge in the same pass: the request is +//! terminalized as `interrupted` and each active hold moves to +//! `released` without folding usage (the logical request can never +//! settle afterwards, so the reservation release is compensation, not +//! credit); //! - terminal request with already released/expired reservation → no-op; //! - stale duplicate terminal command evidence → no durable write; the //! retained supervisor already converges duplicates (C006/C014). //! //! ## What reconciliation never does //! -//! - No `INSERT` into requests, attempts, reservations, or routing -//! decisions: repeated passes cannot fan out rows. -//! - No cost/token/byte mutation: usage stays exactly as persisted, so -//! reconciliation can never double-charge. +//! - No `INSERT` into requests, attempts, reservations, budget holds, or +//! routing decisions: repeated passes cannot fan out rows. +//! - No cost/token/byte mutation: usage buckets and attempt-usage facts stay +//! exactly as persisted, so reconciliation can never double-charge. Moving +//! a budget hold `active → released` frees a reservation for a request the +//! same pass terminalizes; it folds no usage and grants no credit. //! - No M5 hydration: active counts, quota mirrors, and probes are owned //! by the live process. A fresh process starts at zero and stays there; //! reconciliation only repairs durable truth. @@ -256,6 +264,10 @@ pub struct ReconciliationClassification { pub open_attempts_with_terminal_parent: usize, /// Terminal attempts still holding an active reservation. pub terminal_attempts_with_active_reservation: usize, + /// Pending requests holding at least one active user budget hold. + pub pending_requests_with_active_user_hold: usize, + /// Pending requests holding an active instance budget hold. + pub pending_requests_with_active_instance_hold: usize, /// Terminal requests whose reservations already converged (no-op). pub converged_terminal_requests: usize, } @@ -269,6 +281,10 @@ pub struct ReconciliationReport { pub requests_interrupted: usize, /// Reservations moved `active → released` by this pass. pub reservations_released: usize, + /// User budget holds moved `active → released` by this pass. + pub user_budget_holds_released: usize, + /// Instance budget holds moved `active → released` by this pass. + pub instance_budget_holds_released: usize, /// Attempts completed with `process_interrupted` by this pass. pub attempts_terminalized: usize, /// True when every fix ran under the batch bound. @@ -283,7 +299,11 @@ pub struct ReconciliationReport { impl ReconciliationReport { /// Total durable rows terminalized or released by this pass. pub fn fixed_total(&self) -> usize { - self.requests_interrupted + self.reservations_released + self.attempts_terminalized + self.requests_interrupted + + self.reservations_released + + self.user_budget_holds_released + + self.instance_budget_holds_released + + self.attempts_terminalized } } @@ -337,7 +357,7 @@ impl CrashReconciler { /// Classification reads only ids and status labels through /// indexed/bounded queries (`requests(status, started_at)`, /// `reservations(status, …)`, ordered `id` scans with `LIMIT`); fixing - /// applies three conditional bounded updates that never insert rows + /// applies five conditional bounded updates that never insert rows /// and never touch cost/token/byte accounting. Concurrent passes are /// safe: every update re-guards on its nonterminal predicate, so a /// racing pass simply fixes zero rows. @@ -361,15 +381,19 @@ impl CrashReconciler { } let truncated = fixed.0 >= self.batch_limit || fixed.1 >= self.batch_limit - || fixed.2 >= self.batch_limit; + || fixed.2 >= self.batch_limit + || fixed.3 >= self.batch_limit + || fixed.4 >= self.batch_limit; let report = ReconciliationReport { classification, requests_interrupted: fixed.0, reservations_released: fixed.1, attempts_terminalized: fixed.2, + user_budget_holds_released: fixed.3, + instance_budget_holds_released: fixed.4, bounded: true, truncated, - converged: fixed.0 == 0 && fixed.1 == 0 && fixed.2 == 0, + converged: fixed.0 == 0 && fixed.1 == 0 && fixed.2 == 0 && fixed.3 == 0 && fixed.4 == 0, }; Ok(report) } @@ -434,6 +458,24 @@ impl CrashReconciler { [batch], |row| row.get(0), )?; + let pending_with_user_hold: i64 = connection.query_row( + "SELECT COUNT(*) FROM (SELECT r.id FROM requests r + WHERE r.status = 'pending' + AND EXISTS (SELECT 1 FROM user_budget_reservations h + WHERE h.request_id = r.id AND h.status = 'active') + ORDER BY r.id LIMIT ?1)", + [batch], + |row| row.get(0), + )?; + let pending_with_instance_hold: i64 = connection.query_row( + "SELECT COUNT(*) FROM (SELECT r.id FROM requests r + WHERE r.status = 'pending' + AND EXISTS (SELECT 1 FROM instance_budget_reservations h + WHERE h.request_id = r.id AND h.status = 'active') + ORDER BY r.id LIMIT ?1)", + [batch], + |row| row.get(0), + )?; Ok(ReconciliationClassification { pending_requests_without_attempt: pending_without_attempt.max(0) as usize, pending_requests_with_open_attempt: pending_with_open.max(0) as usize, @@ -441,6 +483,9 @@ impl CrashReconciler { open_attempts_with_terminal_parent: open_with_terminal_parent.max(0) as usize, terminal_attempts_with_active_reservation: terminal_with_reservation.max(0) as usize, + pending_requests_with_active_user_hold: pending_with_user_hold.max(0) as usize, + pending_requests_with_active_instance_hold: pending_with_instance_hold.max(0) + as usize, converged_terminal_requests: converged_terminal.max(0) as usize, }) }) @@ -448,7 +493,10 @@ impl CrashReconciler { .map_err(ReconciliationError::Database) } - async fn fix_bounded(&self, batch: i64) -> Result<(usize, usize, usize), ReconciliationError> { + async fn fix_bounded( + &self, + batch: i64, + ) -> Result<(usize, usize, usize, usize, usize), ReconciliationError> { self.database .with_transaction(move |connection| { // Frozen Python `_crash_recovery` policy: every pending @@ -457,7 +505,7 @@ impl CrashReconciler { let requests = connection.execute( "UPDATE requests SET status = 'interrupted', completed_at = CURRENT_TIMESTAMP WHERE id IN (SELECT id FROM requests WHERE status = 'pending' - ORDER BY id LIMIT ?1)", + ORDER BY id LIMIT ?1)", [batch], )?; // Every active reservation is released with the explicit @@ -467,7 +515,7 @@ impl CrashReconciler { "UPDATE reservations SET status = 'released', released_at = CURRENT_TIMESTAMP, release_reason = 'crash_recovery' WHERE id IN (SELECT id FROM reservations WHERE status = 'active' - ORDER BY id LIMIT ?1)", + ORDER BY id LIMIT ?1)", [batch], )?; // Every open attempt is terminalized with the explicit @@ -478,10 +526,33 @@ impl CrashReconciler { "UPDATE request_attempts SET completed_at = CURRENT_TIMESTAMP, error_class = 'process_interrupted' WHERE id IN (SELECT id FROM request_attempts WHERE completed_at IS NULL - ORDER BY id LIMIT ?1)", + ORDER BY id LIMIT ?1)", + [batch], + )?; + // Active budget holds on interrupted requests are + // compensation, not credit: the logical request can never + // settle (finalization conflicts on terminal rows and + // retries fail closed without an active hold), so the + // reservation must not keep occupying quota. Usage buckets + // and attempt-usage facts are untouched — unknown usage + // stays unknown. `settled_at` stays NULL: the hold never + // settled. Only interrupted requests are eligible, so a + // live pending request's hold is never touched here. + let user_holds = connection.execute( + "UPDATE user_budget_reservations SET status = 'released' + WHERE status = 'active' AND request_id IN + (SELECT id FROM requests WHERE status = 'interrupted' + ORDER BY id LIMIT ?1)", + [batch], + )?; + let instance_holds = connection.execute( + "UPDATE instance_budget_reservations SET status = 'released' + WHERE status = 'active' AND request_id IN + (SELECT id FROM requests WHERE status = 'interrupted' + ORDER BY id LIMIT ?1)", [batch], )?; - Ok((requests, reservations, attempts)) + Ok((requests, reservations, attempts, user_holds, instance_holds)) }) .await .map_err(ReconciliationError::Database) diff --git a/rust/src/coordinator/semantic.rs b/rust/src/coordinator/semantic.rs index 9e046997f..c382f8835 100644 --- a/rust/src/coordinator/semantic.rs +++ b/rust/src/coordinator/semantic.rs @@ -417,6 +417,7 @@ pub struct SemanticSelector { known_providers: BTreeSet, max_response_bytes: usize, is_virtual: std::sync::Arc bool + Send + Sync>, + attribution: Option, } impl std::fmt::Debug for SemanticSelector { @@ -425,6 +426,7 @@ impl std::fmt::Debug for SemanticSelector { .debug_struct("SemanticSelector") .field("known_providers", &self.known_providers) .field("max_response_bytes", &self.max_response_bytes) + .field("has_attribution", &self.attribution.is_some()) .finish() } } @@ -436,9 +438,18 @@ impl SemanticSelector { known_providers, max_response_bytes: SELECTOR_MAX_RESPONSE_BYTES, is_virtual: std::sync::Arc::new(|_| false), + attribution: None, } } + pub fn with_attribution( + mut self, + attribution: Option, + ) -> Self { + self.attribution = attribution; + self + } + pub fn with_virtual_check( mut self, check: impl Fn(&str) -> bool + Send + Sync + 'static, @@ -708,11 +719,15 @@ impl SemanticSelector { ) .map_err(|error| SelectorDispatchError::Admission(error.to_string()))?; request.routing_facts.provider_id = provider_pin; - let execution = self - .coordinator - .execute(request) - .await - .map_err(|error| SelectorDispatchError::Dispatch(error.to_string()))?; + let execution = match self.attribution.clone() { + Some(attribution) => { + self.coordinator + .execute_attributed(request, attribution) + .await + } + None => self.coordinator.execute(request).await, + } + .map_err(|error| SelectorDispatchError::Dispatch(error.to_string()))?; let status = execution.response.status.as_u16(); let body = execution.response.body.clone(); // Converge retained C006 ownership before reading the route ID so a diff --git a/rust/src/coordinator/streaming/coordinator.rs b/rust/src/coordinator/streaming/coordinator.rs index 31649a948..ff1bc73f0 100644 --- a/rust/src/coordinator/streaming/coordinator.rs +++ b/rust/src/coordinator/streaming/coordinator.rs @@ -39,7 +39,7 @@ use crate::coordinator::{ FinalizationData, FinalizationIdentity, FinalizationOutcome, FinalizationResult, FinalizationSupervisor, NextAction, PublicationError, PublicationInput, PublicationOutcome, PublicationService, PublishedAttempt, RetryPolicy, RetryScope, WireResolver, - filter_response_headers, retry_after_from_headers, + budget_proof_for_claim, filter_response_headers, retry_after_from_headers, }; /// End-to-end streaming coordinator for one immutable runtime generation. @@ -205,6 +205,22 @@ impl StreamingCoordinator { pub async fn execute( &self, request: StreamRequest, + ) -> Result { + self.execute_inner(request, None).await + } + + pub(crate) async fn execute_attributed( + &self, + request: StreamRequest, + attribution: crate::identity::RequestAttribution, + ) -> Result { + self.execute_inner(request, Some(attribution)).await + } + + async fn execute_inner( + &self, + request: StreamRequest, + attribution: Option, ) -> Result { if request.admitted.canonical.origin.client_surface() != Some(request.client_surface) || request.routing_facts.canonical_model_id != request.admitted.canonical.model @@ -393,14 +409,25 @@ impl StreamingCoordinator { .iter() .any(|candidate| !attempted.contains(&candidate.surface())); let upstream_protocol = protocol_for_surface(candidate.surface()); - let publication_input = PublicationInput::new( + let mut publication_input = PublicationInput::new( request.proxy_request_id.clone(), request.client_surface.protocol(), upstream_protocol, true, i64::from(attempt_number), ); - let published = match self.publication.publish(claim, publication_input).await? { + publication_input.budget_proof = Some(budget_proof_for_claim( + &claim, + provider + .budget + .max_microdollars_per_million_tokens + .and_then(|tariff| i64::try_from(tariff).ok()), + )); + let published = match self + .publication + .publish_attributed(claim, publication_input, attribution.clone()) + .await? + { PublicationOutcome::Published(value) => value, PublicationOutcome::AlreadyPublished(_) => { return Err(StreamingCoordinatorError::Publication( diff --git a/rust/src/db/connection.rs b/rust/src/db/connection.rs index ffb61a636..a66935bfa 100644 --- a/rust/src/db/connection.rs +++ b/rust/src/db/connection.rs @@ -631,8 +631,15 @@ impl Database { { let permit = self.acquire_permit().await?; self.inner.calls.fetch_add(1, Ordering::Relaxed); - let result = self.inner.connection.call(operation).await; - drop(permit); + let result = self + .inner + .connection + .call(move |connection| { + let result = operation(connection); + drop(permit); + result + }) + .await; result.map_err(|error| match error { AsyncSqliteError::ConnectionClosed => DatabaseError::Closed, AsyncSqliteError::Close((_, source)) | AsyncSqliteError::Error(source) => { @@ -697,6 +704,11 @@ impl Database { .inner .connection .call(move |connection| { + // tokio-rusqlite queues this closure independently of the + // caller's future. Keep the serialization gate with the + // queued work so canceling the caller cannot release the gate + // while its SQLite operation is still waiting to execute. + let _permit = permit; #[cfg(feature = "qualification-persist-journal")] let worker_write_bytes_before = if worker_io_attribution { match read_worker_write_bytes() { @@ -821,7 +833,6 @@ impl Database { } }) .await; - drop(permit); #[cfg(feature = "qualification-dedicated-checkpointer")] if matches!( &result, @@ -974,6 +985,16 @@ impl Database { /// uncommitted. Ordinary repository writes should continue to use /// `with_transaction`. pub async fn begin_transaction(&self) -> Result { + self.begin_transaction_after_begin(|| {}).await + } + + async fn begin_transaction_after_begin( + &self, + after_begin: F, + ) -> Result + where + F: FnOnce() + Send + 'static, + { if self.inner.config.read_only { return Err(DatabaseError::ReadOnly); } @@ -981,27 +1002,33 @@ impl Database { self.inner.calls.fetch_add(1, Ordering::Relaxed); self.inner.transactions.fetch_add(1, Ordering::Relaxed); let timeout = self.inner.config.busy_timeout_ms; - if let Err(error) = self + let database = self.clone(); + let runtime = tokio::runtime::Handle::current(); + match self .inner .connection .clone() - .call(|connection| connection.execute_batch("BEGIN IMMEDIATE")) + .call(move |connection| { + connection.execute_batch("BEGIN IMMEDIATE")?; + after_begin(); + Ok(DatabaseTransaction { + database, + permit: Some(permit), + runtime, + finished: false, + }) + }) .await { - drop(permit); - return Err(match error { + Ok(transaction) => Ok(transaction), + Err(error) => Err(match error { AsyncSqliteError::ConnectionClosed => DatabaseError::Closed, AsyncSqliteError::Close((_, source)) | AsyncSqliteError::Error(source) => { map_sqlite("begin transaction", timeout, source) } _ => DatabaseError::WorkerClosed, - }); + }), } - Ok(DatabaseTransaction { - database: self.clone(), - permit: Some(permit), - finished: false, - }) } pub async fn quick_check(&self) -> Result<(), DatabaseError> { @@ -1996,13 +2023,13 @@ fn delete_old( /// /// Dropping an unfinished holder bricks later writes (the next `BEGIN /// IMMEDIATE` fails), so every holder must be committed or rolled back. -/// `Drop` issues a best-effort async `ROLLBACK`; when no Tokio runtime is -/// present the rollback cannot be issued and the brick-risk stands — callers -/// must not drop an unfinished transaction outside a runtime. +/// `Drop` issues a best-effort async `ROLLBACK` through the runtime handle +/// captured when the transaction began. #[must_use = "an unfinished DatabaseTransaction bricks later writes on drop; commit or roll it back"] pub struct DatabaseTransaction { database: Database, permit: Option, + runtime: tokio::runtime::Handle, finished: bool, } @@ -2138,27 +2165,12 @@ impl Drop for DatabaseTransaction { // is still pending, which SQLite rejects and which could otherwise // leave the new transaction rolled back underneath it. let permit = self.permit.take(); - if let Ok(handle) = tokio::runtime::Handle::try_current() { - handle.spawn(async move { - let _ = connection - .call(|connection| connection.execute_batch("ROLLBACK")) - .await; - drop(permit); - }); - } else { - // No runtime: the ROLLBACK cannot be issued. The permit releases - // here, but the SQLite transaction may still be open, so the next - // BEGIN IMMEDIATE can fail. This path is a programming error — - // surface it loudly and catch it in debug builds. - tracing::error!( - "database transaction dropped outside a Tokio runtime; ROLLBACK skipped" - ); - debug_assert!( - false, - "DatabaseTransaction dropped without commit/rollback outside a runtime" - ); + self.runtime.spawn(async move { + let _ = connection + .call(|connection| connection.execute_batch("ROLLBACK")) + .await; drop(permit); - } + }); } } @@ -2496,6 +2508,114 @@ mod maintenance_tests { use super::*; use std::sync::atomic::AtomicU64; + async fn assert_cancelled_call_keeps_gate_until_worker_finishes( + database: &Database, + transactional: bool, + ) { + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let (finished_tx, finished_rx) = tokio::sync::oneshot::channel(); + let task_database = database.clone(); + let task = tokio::spawn(async move { + let operation = move |connection: &mut SqliteConnection| { + let _ = started_tx.send(()); + release_rx + .recv() + .expect("test releases the queued SQLite operation"); + let result = connection.execute_batch("SELECT 1"); + let _ = finished_tx.send(()); + result + }; + if transactional { + task_database.with_transaction(operation).await + } else { + task_database.call(operation).await + } + }); + started_rx + .await + .expect("SQLite worker reaches test barrier"); + task.abort(); + assert!( + task.await + .expect_err("caller task is canceled") + .is_cancelled() + ); + assert_eq!( + database.inner.gate.available_permits(), + 0, + "the SQLite gate stays with queued work after caller cancellation" + ); + release_tx.send(()).expect("SQLite worker is waiting"); + tokio::time::timeout(Duration::from_secs(2), finished_rx) + .await + .expect("queued SQLite work completes within a bounded interval") + .expect("worker completion is observed"); + tokio::time::timeout( + Duration::from_secs(2), + database.call(|connection| connection.execute_batch("SELECT 1")), + ) + .await + .expect("the gate is released after queued work completes") + .expect("follow-up SQLite call succeeds"); + assert_eq!(database.inner.gate.available_permits(), 1); + } + + #[tokio::test(flavor = "current_thread")] + async fn canceled_call_retains_gate_until_queued_operation_finishes() { + let database = Database::open(DatabaseConfig::default()) + .await + .expect("database opens"); + assert_cancelled_call_keeps_gate_until_worker_finishes(&database, false).await; + database.close().await.expect("database closes"); + } + + #[tokio::test(flavor = "current_thread")] + async fn canceled_transaction_retains_gate_until_queued_operation_finishes() { + let database = Database::open(DatabaseConfig::default()) + .await + .expect("database opens"); + assert_cancelled_call_keeps_gate_until_worker_finishes(&database, true).await; + database.close().await.expect("database closes"); + } + + #[tokio::test(flavor = "current_thread")] + async fn canceled_begin_transaction_rolls_back_before_releasing_gate() { + let database = Database::open(DatabaseConfig::default()) + .await + .expect("database opens"); + let (begun_tx, begun_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let task_database = database.clone(); + let task = tokio::spawn(async move { + task_database + .begin_transaction_after_begin(move || { + let _ = begun_tx.send(()); + release_rx + .recv() + .expect("test releases the begun transaction"); + }) + .await + }); + begun_rx + .await + .expect("BEGIN IMMEDIATE reached the synchronized boundary"); + task.abort(); + assert!(task.await.expect_err("caller is canceled").is_cancelled()); + assert_eq!(database.inner.gate.available_permits(), 0); + + release_tx.send(()).expect("test releases the worker"); + tokio::time::timeout( + Duration::from_secs(2), + database.call(|connection| connection.execute_batch("SELECT 1")), + ) + .await + .expect("gate is released after the orphaned begin is rolled back") + .expect("follow-up SQLite work succeeds"); + assert_eq!(database.inner.gate.available_permits(), 1); + database.close().await.expect("database closes"); + } + #[tokio::test(flavor = "current_thread")] async fn idle_tick_reports_not_due_without_sqlite_work() { let database = Database::open(DatabaseConfig::default()) diff --git a/rust/src/db/migrations.rs b/rust/src/db/migrations.rs index 1df02cd71..47c74cfd4 100644 --- a/rust/src/db/migrations.rs +++ b/rust/src/db/migrations.rs @@ -203,9 +203,9 @@ mod tests { #[test] fn canonical_inventory_is_complete_and_immutable() { MigrationRunner::validate_embedded_checksums().expect("canonical checksums match"); - assert_eq!(MigrationRunner::migrations().len(), 54); + assert_eq!(MigrationRunner::migrations().len(), 62); assert_eq!(MigrationRunner::migrations().first().unwrap().version, 1); - assert_eq!(MigrationRunner::migrations().last().unwrap().version, 54); + assert_eq!(MigrationRunner::migrations().last().unwrap().version, 62); } #[test] diff --git a/rust/src/identity.rs b/rust/src/identity.rs new file mode 100644 index 000000000..28f322129 --- /dev/null +++ b/rust/src/identity.rs @@ -0,0 +1,1435 @@ +//! Local downstream-user identity and credential storage. +//! +//! Hashing is deliberately performed before entering SQLite's single gate. + +use std::{ + collections::VecDeque, + sync::{Arc, Mutex, OnceLock}, + time::{Duration, Instant}, +}; + +use argon2::{ + Algorithm, Argon2, Params, PasswordHash, PasswordHasher, PasswordVerifier, Version, + password_hash::SaltString, +}; +use sha2::{Digest, Sha256}; +use thiserror::Error; +use tokio::sync::Semaphore; +use tokio_rusqlite::rusqlite::OptionalExtension; + +use crate::db::Database; + +const MAX_PASSWORD_BYTES: usize = 1024; +const MIN_PASSWORD_BYTES: usize = 12; +const HASH_CONCURRENCY: usize = 2; + +static HASH_SLOTS: OnceLock> = OnceLock::new(); +static FAILED_TOKEN_DIGESTS: OnceLock>> = OnceLock::new(); +static DUMMY_PASSWORD_HASH: tokio::sync::OnceCell = tokio::sync::OnceCell::const_new(); +const DUMMY_PASSWORD: &str = "eggpool-invalid-login-dummy-password"; +const FAILED_TOKEN_CAPACITY: usize = 256; +const FAILED_TOKEN_TTL: Duration = Duration::from_secs(1); + +#[derive(Debug, Error)] +pub enum IdentityError { + #[error("username must be 1 to 64 ASCII letters, digits, dots, underscores, or hyphens")] + InvalidUsername, + #[error("password must contain between 12 and 1024 bytes")] + InvalidPassword, + #[error("credential label must contain between 1 and 80 bytes")] + InvalidCredentialLabel, + #[error("an administrator is already initialized")] + AdminAlreadyInitialized, + #[error("identity access state cannot be moved to managed")] + InvalidAccessTransition, + #[error("user was not found")] + UserNotFound, + #[error("administrator privileges are required")] + AdministratorRequired, + #[error("the last active administrator cannot be disabled")] + LastAdminProtected, + #[error("username is already in use")] + UsernameConflict, + #[error("credential was not found")] + CredentialNotFound, + #[error("credential entropy unavailable")] + Entropy, + #[error("password hashing worker failed")] + HashWorker, + #[error("password hash is invalid")] + InvalidHash, + #[error(transparent)] + Database(#[from] crate::db::DatabaseError), +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct UserSummary { + pub id: String, + pub username: String, + pub role: String, + pub state: String, + pub must_change_password: bool, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CredentialSummary { + pub id: String, + pub label: String, + pub created_at: String, + pub last_used_at: Option, + pub expires_at: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct AdminAuditEntry { + pub actor_id: String, + pub action: String, + pub target_kind: String, + pub target_id: String, + pub created_at: String, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct UserRequestSummary { + pub proxy_request_id: Option, + pub model_id: String, + pub provider_id: String, + pub status: String, + pub started_at: String, + pub input_tokens: i64, + pub output_tokens: i64, + pub cost_microdollars: i64, +} + +/// Return a bounded, content-free request history for one user. Internal +/// selector children are excluded so downstream callers see logical requests. +pub async fn recent_user_requests( + database: &Database, + user_id: &str, + limit: usize, +) -> Result, IdentityError> { + let user_id = user_id.to_owned(); + let limit = limit.clamp(1, 500) as i64; + database + .call(move |connection| { + let mut statement = connection.prepare( + "SELECT proxy_request_id, model_id, provider_id, status, started_at, \ + COALESCE(input_tokens, 0), COALESCE(output_tokens, 0), \ + COALESCE(cost_microdollars, 0) \ + FROM requests WHERE user_id = ?1 AND request_kind = 'logical' \ + ORDER BY started_at DESC, id DESC LIMIT ?2", + )?; + let rows = statement.query_map((user_id, limit), |row| { + Ok(UserRequestSummary { + proxy_request_id: row.get(0)?, + model_id: row.get(1)?, + provider_id: row.get(2)?, + status: row.get(3)?, + started_at: row.get(4)?, + input_tokens: row.get(5)?, + output_tokens: row.get(6)?, + cost_microdollars: row.get(7)?, + }) + })?; + rows.collect::, _>>() + }) + .await + .map_err(IdentityError::Database) +} + +#[derive(Clone, PartialEq, Eq)] +pub struct IssuedCredential { + pub user_id: String, + pub token: String, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)] +pub enum PrincipalKind { + AgentToken, + BrowserSession, + LegacySharedKey, +} + +impl PrincipalKind { + pub fn as_str(self) -> &'static str { + match self { + Self::AgentToken => "agent_token", + Self::BrowserSession => "browser_session", + Self::LegacySharedKey => "legacy_shared_key", + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct AuthenticatedPrincipal { + pub user_id: Option, + pub credential_id: Option, + pub role: Option, + pub scopes: Vec, + pub auth_epoch: Option, + pub kind: PrincipalKind, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)] +pub enum AttributedRequestKind { + Logical, + InternalSelector, +} + +impl AttributedRequestKind { + pub fn as_str(self) -> &'static str { + match self { + Self::Logical => "logical", + Self::InternalSelector => "internal_selector", + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct RequestAttribution { + pub user_id: Option, + pub credential_id: Option, + pub auth_epoch: Option, + pub principal_kind: PrincipalKind, + pub request_kind: AttributedRequestKind, + pub parent_proxy_request_id: Option, +} + +impl RequestAttribution { + pub fn selector_child(&self, parent_proxy_request_id: &str) -> Self { + Self { + request_kind: AttributedRequestKind::InternalSelector, + parent_proxy_request_id: Some(parent_proxy_request_id.to_owned()), + ..self.clone() + } + } +} + +impl AuthenticatedPrincipal { + pub fn legacy_shared_key() -> Self { + Self { + user_id: None, + credential_id: None, + role: None, + scopes: vec!["inference".to_owned()], + auth_epoch: None, + kind: PrincipalKind::LegacySharedKey, + } + } + + pub fn is_user_principal(&self) -> bool { + self.user_id.is_some() + } + + pub fn request_attribution(&self) -> RequestAttribution { + RequestAttribution { + user_id: self.user_id.clone(), + credential_id: self.credential_id.clone(), + auth_epoch: self.auth_epoch, + principal_kind: self.kind, + request_kind: AttributedRequestKind::Logical, + parent_proxy_request_id: None, + } + } + + pub fn selector_attribution(&self, parent_proxy_request_id: &str) -> RequestAttribution { + RequestAttribution { + request_kind: AttributedRequestKind::InternalSelector, + parent_proxy_request_id: Some(parent_proxy_request_id.to_owned()), + ..self.request_attribution() + } + } +} + +/// Create the sole initial administrator and an inference-only personal key +/// in one transaction. The password hash is computed before acquiring SQLite. +pub async fn bootstrap_admin( + database: &Database, + username: &str, + password: &str, +) -> Result<(UserSummary, IssuedCredential), IdentityError> { + let username = validate_username(username)?; + let password_hash = hash_password(password).await?; + let id = random_hex(16)?; + let credential_id = random_hex(16)?; + let selector = random_hex(8)?; + let secret = random_hex(32)?; + let token = format!("epk1_{selector}_{secret}"); + let digest = digest(&token); + let username_for_db = username.clone(); + let id_for_db = id.clone(); + let selector_for_db = selector.clone(); + let password_hash_for_db = password_hash.clone(); + + let transaction_result = database + .with_transaction(move |connection| { + let user_count: i64 = connection.query_row("SELECT COUNT(*) FROM users", [], |row| row.get(0))?; + if user_count != 0 { + return Err(tokio_rusqlite::rusqlite::Error::QueryReturnedNoRows); + } + connection.execute( + "INSERT INTO users(id, username, normalized_username, role, password_hash) VALUES (?1, ?2, ?3, 'admin', ?4)", + (&id_for_db, &username_for_db, username_for_db.to_ascii_lowercase(), &password_hash_for_db), + )?; + connection.execute( + "INSERT INTO user_credentials(id, user_id, label, selector, digest, scopes) VALUES (?1, ?2, 'initial admin client', ?3, ?4, 'inference,discovery,integrations')", + (&credential_id, &id_for_db, &selector_for_db, &digest), + )?; + connection.execute( + "UPDATE user_access_state SET state = 'admin_initialized', updated_at = CURRENT_TIMESTAMP WHERE singleton = 1", + [], + )?; + Ok(()) + }) + .await; + if let Err(error) = transaction_result { + let initialized = database + .call(|connection| { + connection.query_row("SELECT EXISTS(SELECT 1 FROM users)", [], |row| { + row.get::<_, bool>(0) + }) + }) + .await + .unwrap_or(false); + if initialized { + return Err(IdentityError::AdminAlreadyInitialized); + } + return Err(IdentityError::Database(error)); + } + + Ok(( + UserSummary { + id: id.clone(), + username, + role: "admin".to_owned(), + state: "active".to_owned(), + must_change_password: false, + }, + IssuedCredential { user_id: id, token }, + )) +} + +pub async fn list_users(database: &Database) -> Result, IdentityError> { + database + .call(|connection| { + let mut statement = connection.prepare( + "SELECT id, username, role, state, must_change_password FROM users \ + ORDER BY username COLLATE NOCASE, id LIMIT 500", + )?; + let rows = statement.query_map([], |row| { + Ok(UserSummary { + id: row.get(0)?, + username: row.get(1)?, + role: row.get(2)?, + state: row.get(3)?, + must_change_password: row.get::<_, i64>(4)? != 0, + }) + })?; + rows.collect() + }) + .await + .map_err(IdentityError::Database) +} + +/// Create a member with a one-time temporary password. The forced password +/// change, managed-mode transition, and administrator audit row commit with +/// the new user. +pub async fn create_member( + database: &Database, + actor_id: &str, + username: &str, + temporary_password: &str, +) -> Result { + create_member_with_fence(database, actor_id, username, temporary_password, None, None).await +} + +pub async fn create_member_from_browser( + database: &Database, + principal: &crate::web_session::Principal, + username: &str, + temporary_password: &str, +) -> Result { + create_member_with_fence( + database, + &principal.user_id, + username, + temporary_password, + Some(crate::web_session::SessionFence::from_principal(principal)), + None, + ) + .await +} + +pub async fn create_member_with_profile_from_browser( + database: &Database, + principal: &crate::web_session::Principal, + username: &str, + temporary_password: &str, + profile_id: &str, +) -> Result { + create_member_with_fence( + database, + &principal.user_id, + username, + temporary_password, + Some(crate::web_session::SessionFence::from_principal(principal)), + Some(profile_id), + ) + .await +} + +async fn create_member_with_fence( + database: &Database, + actor_id: &str, + username: &str, + temporary_password: &str, + session_fence: Option, + profile_id: Option<&str>, +) -> Result { + let username = validate_username(username)?; + let password_hash = hash_password(temporary_password).await?; + let id = random_hex(16)?; + let actor_id = actor_id.to_owned(); + let normalized = username.to_ascii_lowercase(); + let session_fence = session_fence.clone(); + let profile_id = profile_id.map(str::to_owned); + let outcome = database + .with_transaction(move |connection| { + let actor_is_admin = if let Some(fence) = session_fence.as_ref() { + fence.validates_actor(connection, &actor_id, true)? + } else { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE id = ?1 AND role = 'admin' AND state = 'active')", + [&actor_id], + |row| row.get(0), + )? + }; + if !actor_is_admin { + return Ok(AdminMutation::Forbidden); + } + let taken: bool = connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE normalized_username = ?1)", + [&normalized], + |row| row.get(0), + )?; + if taken { + return Ok(AdminMutation::Conflict); + } + connection.execute( + "INSERT INTO users(id, username, normalized_username, role, password_hash, must_change_password) \ + VALUES (?1, ?2, ?3, 'member', ?4, 1)", + (&id, &username, &normalized, &password_hash), + )?; + connection.execute( + "UPDATE user_access_state SET state = 'managed', updated_at = CURRENT_TIMESTAMP \ + WHERE singleton = 1 AND state IN ('admin_initialized', 'managed')", + [], + )?; + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'create_user', 'user', ?2)", + (&actor_id, &id), + )?; + if let Some(profile_id) = profile_id.as_ref() { + connection.execute( + "INSERT INTO user_profile_assignments(user_id, profile_id, assigned_by) \ + VALUES (?1, ?2, ?3)", + (&id, profile_id, &actor_id), + )?; + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'assign_profile', 'user', ?2)", + (&actor_id, &id), + )?; + } + Ok(AdminMutation::User(UserSummary { + id, + username, + role: "member".to_owned(), + state: "active".to_owned(), + must_change_password: true, + })) + }) + .await?; + match outcome { + AdminMutation::User(user) => Ok(user), + AdminMutation::Forbidden => Err(IdentityError::AdministratorRequired), + AdminMutation::Conflict => Err(IdentityError::UsernameConflict), + _ => Err(IdentityError::UserNotFound), + } +} + +/// Disable or reactivate a downstream account. Disabling revokes sessions and +/// credentials and advances auth_epoch in the same transaction. +pub async fn set_user_enabled( + database: &Database, + actor_id: &str, + target_id: &str, + enabled: bool, +) -> Result<(), IdentityError> { + set_user_enabled_with_fence(database, actor_id, target_id, enabled, None).await +} + +pub async fn set_user_enabled_from_browser( + database: &Database, + principal: &crate::web_session::Principal, + target_id: &str, + enabled: bool, +) -> Result<(), IdentityError> { + set_user_enabled_with_fence( + database, + &principal.user_id, + target_id, + enabled, + Some(crate::web_session::SessionFence::from_principal(principal)), + ) + .await +} + +async fn set_user_enabled_with_fence( + database: &Database, + actor_id: &str, + target_id: &str, + enabled: bool, + session_fence: Option, +) -> Result<(), IdentityError> { + let actor_id = actor_id.to_owned(); + let target_id = target_id.to_owned(); + let session_fence = session_fence.clone(); + let outcome = database + .with_transaction(move |connection| { + let actor_is_admin = if let Some(fence) = session_fence.as_ref() { + fence.validates_actor(connection, &actor_id, true)? + } else { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE id = ?1 AND role = 'admin' AND state = 'active')", + [&actor_id], + |row| row.get(0), + )? + }; + if !actor_is_admin { + return Ok(AdminMutation::Forbidden); + } + let target: Option<(String, String)> = connection + .query_row( + "SELECT role, state FROM users WHERE id = ?1", + [&target_id], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .optional()?; + let Some((role, current_state)) = target else { + return Ok(AdminMutation::Missing); + }; + let next_state = if enabled { "active" } else { "disabled" }; + if current_state == next_state { + return Ok(AdminMutation::Done); + } + if !enabled && role == "admin" { + let active_admins: i64 = connection.query_row( + "SELECT COUNT(*) FROM users WHERE role = 'admin' AND state = 'active'", + [], + |row| row.get(0), + )?; + if active_admins <= 1 { + return Ok(AdminMutation::LastAdmin); + } + } + connection.execute( + "UPDATE users SET state = ?1, auth_epoch = auth_epoch + 1, updated_at = CURRENT_TIMESTAMP WHERE id = ?2", + (next_state, &target_id), + )?; + if !enabled { + connection.execute( + "UPDATE user_credentials SET revoked_at = CURRENT_TIMESTAMP WHERE user_id = ?1 AND revoked_at IS NULL", + [&target_id], + )?; + connection.execute( + "UPDATE web_sessions SET revoked_at = CURRENT_TIMESTAMP WHERE user_id = ?1 AND revoked_at IS NULL", + [&target_id], + )?; + } + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) VALUES (?1, ?2, 'user', ?3)", + (&actor_id, if enabled { "reactivate_user" } else { "disable_user" }, &target_id), + )?; + Ok(AdminMutation::Done) + }) + .await?; + match outcome { + AdminMutation::Done | AdminMutation::User(_) => Ok(()), + AdminMutation::Forbidden => Err(IdentityError::AdministratorRequired), + AdminMutation::LastAdmin => Err(IdentityError::LastAdminProtected), + AdminMutation::Missing => Err(IdentityError::UserNotFound), + _ => Err(IdentityError::UsernameConflict), + } +} + +/// Set a temporary password for an active user and revoke all existing +/// sessions/credentials so the next browser login must change it. +pub async fn admin_reset_password( + database: &Database, + actor_id: &str, + target_id: &str, + temporary_password: &str, +) -> Result<(), IdentityError> { + admin_reset_password_with_fence(database, actor_id, target_id, temporary_password, None).await +} + +pub async fn admin_reset_password_from_browser( + database: &Database, + principal: &crate::web_session::Principal, + target_id: &str, + temporary_password: &str, +) -> Result<(), IdentityError> { + admin_reset_password_with_fence( + database, + &principal.user_id, + target_id, + temporary_password, + Some(crate::web_session::SessionFence::from_principal(principal)), + ) + .await +} + +async fn admin_reset_password_with_fence( + database: &Database, + actor_id: &str, + target_id: &str, + temporary_password: &str, + session_fence: Option, +) -> Result<(), IdentityError> { + let password_hash = hash_password(temporary_password).await?; + let actor_id = actor_id.to_owned(); + let target_id = target_id.to_owned(); + let session_fence = session_fence.clone(); + let outcome = database + .with_transaction(move |connection| { + let actor_is_admin = if let Some(fence) = session_fence.as_ref() { + fence.validates_actor(connection, &actor_id, true)? + } else { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE id = ?1 AND role = 'admin' AND state = 'active')", + [&actor_id], + |row| row.get(0), + )? + }; + if !actor_is_admin { + return Ok(AdminMutation::Forbidden); + } + let changed = connection.execute( + "UPDATE users SET password_hash = ?1, must_change_password = 1, auth_epoch = auth_epoch + 1, updated_at = CURRENT_TIMESTAMP WHERE id = ?2 AND state = 'active'", + (&password_hash, &target_id), + )?; + if changed == 0 { + return Ok(AdminMutation::Missing); + } + connection.execute( + "UPDATE user_credentials SET revoked_at = CURRENT_TIMESTAMP WHERE user_id = ?1 AND revoked_at IS NULL", + [&target_id], + )?; + connection.execute( + "UPDATE web_sessions SET revoked_at = CURRENT_TIMESTAMP WHERE user_id = ?1 AND revoked_at IS NULL", + [&target_id], + )?; + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) VALUES (?1, 'reset_password', 'user', ?2)", + (&actor_id, &target_id), + )?; + Ok(AdminMutation::Done) + }) + .await?; + match outcome { + AdminMutation::Done | AdminMutation::User(_) => Ok(()), + AdminMutation::Forbidden => Err(IdentityError::AdministratorRequired), + AdminMutation::Missing => Err(IdentityError::UserNotFound), + _ => Err(IdentityError::UserNotFound), + } +} + +pub async fn list_user_credentials( + database: &Database, + user_id: &str, +) -> Result, IdentityError> { + let user_id = user_id.to_owned(); + database + .call(move |connection| { + let mut statement = connection.prepare( + "SELECT id, label, created_at, last_used_at, expires_at FROM user_credentials \ + WHERE user_id = ?1 AND revoked_at IS NULL ORDER BY created_at DESC, id DESC LIMIT 100", + )?; + statement + .query_map([user_id], |row| { + Ok(CredentialSummary { + id: row.get(0)?, + label: row.get(1)?, + created_at: row.get(2)?, + last_used_at: row.get(3)?, + expires_at: row.get(4)?, + }) + })? + .collect() + }) + .await + .map_err(IdentityError::Database) +} + +pub async fn revoke_user_credential( + database: &Database, + actor_id: &str, + user_id: &str, + credential_id: &str, +) -> Result<(), IdentityError> { + revoke_user_credential_with_fence(database, actor_id, user_id, credential_id, None).await +} + +pub async fn revoke_user_credential_from_browser( + database: &Database, + principal: &crate::web_session::Principal, + user_id: &str, + credential_id: &str, +) -> Result<(), IdentityError> { + revoke_user_credential_with_fence( + database, + &principal.user_id, + user_id, + credential_id, + Some(crate::web_session::SessionFence::from_principal(principal)), + ) + .await +} + +async fn revoke_user_credential_with_fence( + database: &Database, + actor_id: &str, + user_id: &str, + credential_id: &str, + session_fence: Option, +) -> Result<(), IdentityError> { + let actor_id = actor_id.to_owned(); + let user_id = user_id.to_owned(); + let credential_id = credential_id.to_owned(); + let session_fence = session_fence.clone(); + let outcome = database + .with_transaction(move |connection| { + let authorized = if let Some(fence) = session_fence.as_ref() { + fence.validates_actor(connection, &actor_id, actor_id != user_id)? + && connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE id = ?1 AND state = 'active')", + [&user_id], + |row| row.get(0), + )? + } else { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users actor JOIN users target \ + ON actor.id = ?1 AND actor.state = 'active' AND target.id = ?2 AND target.state = 'active' \ + WHERE actor.role = 'admin' OR actor.id = target.id)", + (&actor_id, &user_id), + |row| row.get(0), + )? + }; + if !authorized { + return Ok(AdminMutation::Forbidden); + } + let changed = connection.execute( + "UPDATE user_credentials SET revoked_at = CURRENT_TIMESTAMP \ + WHERE id = ?1 AND user_id = ?2 AND revoked_at IS NULL", + (&credential_id, &user_id), + )?; + if changed == 0 { + return Ok(AdminMutation::Missing); + } + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'revoke_token', 'credential', ?2)", + (&actor_id, &credential_id), + )?; + Ok(AdminMutation::Done) + }) + .await?; + match outcome { + AdminMutation::Done | AdminMutation::User(_) => Ok(()), + AdminMutation::Forbidden => Err(IdentityError::AdministratorRequired), + AdminMutation::Missing => Err(IdentityError::CredentialNotFound), + _ => Err(IdentityError::CredentialNotFound), + } +} + +#[derive(Debug)] +enum AdminMutation { + Done, + User(UserSummary), + TokenCount(u64), + Forbidden, + LastAdmin, + Missing, + Conflict, +} + +pub async fn admin_audit( + database: &Database, + limit: usize, +) -> Result, IdentityError> { + let limit = limit.clamp(1, 500) as i64; + database + .call(move |connection| { + let mut statement = connection.prepare( + "SELECT actor_id, action, target_kind, target_id, created_at FROM ( \ + SELECT actor_id, action, target_kind, target_id, created_at, id AS row_id \ + FROM user_admin_audit \ + UNION ALL \ + SELECT actor_id, action, 'budget_rule' AS target_kind, CAST(user_id AS TEXT), \ + created_at, id AS row_id FROM user_budget_rule_audit \ + ) ORDER BY created_at DESC, row_id DESC LIMIT ?1", + )?; + statement + .query_map([limit], |row| { + Ok(AdminAuditEntry { + actor_id: row.get(0)?, + action: row.get(1)?, + target_kind: row.get(2)?, + target_id: row.get(3)?, + created_at: row.get(4)?, + }) + })? + .collect() + }) + .await + .map_err(IdentityError::Database) +} + +async fn issue_api_token_as( + database: &Database, + actor_id: &str, + user_id: &str, + label: &str, + session_fence: Option, +) -> Result { + let label = label.trim().to_owned(); + if label.is_empty() || label.len() > 80 { + return Err(IdentityError::InvalidCredentialLabel); + } + let id = random_hex(16)?; + let selector = random_hex(8)?; + let secret = random_hex(32)?; + let token = format!("epk1_{selector}_{secret}"); + let digest = digest(&token); + let actor_id = actor_id.to_owned(); + let user_id = user_id.to_owned(); + let returned_user_id = user_id.clone(); + let session_fence = session_fence.clone(); + let outcome = database + .with_transaction(move |connection| { + let authorized = if let Some(fence) = session_fence.as_ref() { + fence.validates_actor(connection, &actor_id, actor_id != user_id)? + && connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE id = ?1 AND state = 'active')", + [&user_id], + |row| row.get(0), + )? + } else { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users actor JOIN users target \ + ON actor.id = ?1 AND actor.state = 'active' AND target.id = ?2 AND target.state = 'active' \ + WHERE actor.role = 'admin' OR actor.id = target.id)", + (&actor_id, &user_id), + |row| row.get(0), + )? + }; + if !authorized { + return Ok(AdminMutation::Forbidden); + } + connection.execute( + "INSERT INTO user_credentials(id, user_id, label, selector, digest, scopes) \ + VALUES (?1, ?2, ?3, ?4, ?5, 'inference,discovery,integrations')", + (&id, &user_id, &label, &selector, &digest), + )?; + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'issue_token', 'credential', ?2)", + (&actor_id, &id), + )?; + Ok(AdminMutation::Done) + }) + .await?; + match outcome { + AdminMutation::Done => Ok(IssuedCredential { + user_id: returned_user_id, + token, + }), + _ => Err(IdentityError::AdministratorRequired), + } +} + +pub async fn issue_personal_api_token( + database: &Database, + actor_id: &str, + user_id: &str, + label: &str, +) -> Result { + issue_api_token_as(database, actor_id, user_id, label, None).await +} + +pub async fn issue_personal_api_token_from_browser( + database: &Database, + principal: &crate::web_session::Principal, + user_id: &str, + label: &str, +) -> Result { + issue_api_token_as( + database, + &principal.user_id, + user_id, + label, + Some(crate::web_session::SessionFence::from_principal(principal)), + ) + .await +} + +pub async fn revoke_user_api_tokens( + database: &Database, + actor_id: &str, + user_id: &str, +) -> Result { + revoke_user_api_tokens_with_fence(database, actor_id, user_id, None).await +} + +pub async fn revoke_user_api_tokens_from_browser( + database: &Database, + principal: &crate::web_session::Principal, + user_id: &str, +) -> Result { + revoke_user_api_tokens_with_fence( + database, + &principal.user_id, + user_id, + Some(crate::web_session::SessionFence::from_principal(principal)), + ) + .await +} + +async fn revoke_user_api_tokens_with_fence( + database: &Database, + actor_id: &str, + user_id: &str, + session_fence: Option, +) -> Result { + let actor_id = actor_id.to_owned(); + let user_id = user_id.to_owned(); + let session_fence = session_fence.clone(); + let outcome = database + .with_transaction(move |connection| { + let authorized = if let Some(fence) = session_fence.as_ref() { + fence.validates_actor(connection, &actor_id, true)? + && connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE id = ?1 AND state = 'active')", + [&user_id], + |row| row.get(0), + )? + } else { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users actor JOIN users target \ + ON actor.id = ?1 AND actor.role = 'admin' AND actor.state = 'active' \ + AND target.id = ?2 AND target.state = 'active')", + (&actor_id, &user_id), + |row| row.get(0), + )? + }; + if !authorized { + return Ok(AdminMutation::Forbidden); + } + let revoked = connection.execute( + "UPDATE user_credentials SET revoked_at = CURRENT_TIMESTAMP \ + WHERE user_id = ?1 AND revoked_at IS NULL", + [&user_id], + )? as u64; + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'revoke_user_tokens', 'user', ?2)", + (&actor_id, &user_id), + )?; + Ok(AdminMutation::TokenCount(revoked)) + }) + .await?; + match outcome { + AdminMutation::TokenCount(count) => Ok(count), + AdminMutation::Forbidden => Err(IdentityError::AdministratorRequired), + _ => Err(IdentityError::UserNotFound), + } +} + +pub async fn access_state(database: &Database) -> Result { + database + .call(|connection| { + connection.query_row( + "SELECT state FROM user_access_state WHERE singleton = 1", + [], + |row| row.get(0), + ) + }) + .await + .map_err(IdentityError::Database) +} + +pub async fn enable_managed(database: &Database) -> Result<(), IdentityError> { + let changed = database + .with_transaction(|connection| { + connection.execute( + "UPDATE user_access_state SET state = 'managed', updated_at = CURRENT_TIMESTAMP \ + WHERE singleton = 1 AND state = 'admin_initialized' \ + AND EXISTS (SELECT 1 FROM users WHERE role = 'admin' AND state = 'active')", + [], + )?; + Ok(connection.changes()) + }) + .await?; + if changed == 1 || access_state(database).await?.as_str() == "managed" { + Ok(()) + } else { + Err(IdentityError::InvalidAccessTransition) + } +} + +pub async fn mark_fresh_uninitialized(database: &Database) -> Result<(), IdentityError> { + database + .with_transaction(|connection| { + connection.execute( + "UPDATE user_access_state SET state = 'fresh_uninitialized', updated_at = CURRENT_TIMESTAMP WHERE singleton = 1 AND state = 'legacy_unmigrated' AND NOT EXISTS (SELECT 1 FROM users)", + [], + )?; + Ok(()) + }) + .await + .map_err(IdentityError::Database) +} + +/// Replace a password and revoke all the user's client credentials. The +/// browser-session migration adds session revocation to this same boundary. +pub async fn reset_password( + database: &Database, + username: &str, + password: &str, +) -> Result<(), IdentityError> { + reset_password_with_fence(database, username, password, None).await +} + +pub async fn reset_password_from_browser( + database: &Database, + principal: &crate::web_session::Principal, + password: &str, +) -> Result<(), IdentityError> { + reset_password_with_fence( + database, + &principal.username, + password, + Some(crate::web_session::SessionFence::from_principal(principal)), + ) + .await +} + +async fn reset_password_with_fence( + database: &Database, + username: &str, + password: &str, + session_fence: Option, +) -> Result<(), IdentityError> { + let normalized = validate_username(username)?.to_ascii_lowercase(); + let password_hash = hash_password(password).await?; + let normalized_for_error = normalized.clone(); + let session_fence = session_fence.clone(); + let result = database + .with_transaction(move |connection| { + let changed = if let Some(fence) = session_fence.as_ref() { + if !fence.validates_current_session(connection, fence.user_id())? { + return Err(tokio_rusqlite::rusqlite::Error::QueryReturnedNoRows); + } + connection.execute( + "UPDATE users SET password_hash = ?1, must_change_password = 0, auth_epoch = auth_epoch + 1, updated_at = CURRENT_TIMESTAMP WHERE id = ?2 AND normalized_username = ?3 AND state = 'active'", + (&password_hash, fence.user_id(), &normalized), + )? + } else { + connection.execute( + "UPDATE users SET password_hash = ?1, must_change_password = 0, auth_epoch = auth_epoch + 1, updated_at = CURRENT_TIMESTAMP WHERE normalized_username = ?2 AND state = 'active'", + (&password_hash, &normalized), + )? + }; + if changed == 0 { + return Err(tokio_rusqlite::rusqlite::Error::QueryReturnedNoRows); + } + connection.execute( + "UPDATE user_credentials SET revoked_at = CURRENT_TIMESTAMP WHERE user_id = (SELECT id FROM users WHERE normalized_username = ?1) AND revoked_at IS NULL", + [&normalized], + )?; + connection.execute( + "UPDATE web_sessions SET revoked_at = CURRENT_TIMESTAMP WHERE user_id = (SELECT id FROM users WHERE normalized_username = ?1) AND revoked_at IS NULL", + [&normalized], + )?; + Ok(()) + }) + .await; + match result { + Ok(()) => Ok(()), + Err(error) => { + if active_user_exists(database, normalized_for_error).await? { + Err(IdentityError::Database(error)) + } else { + Err(IdentityError::UserNotFound) + } + } + } +} + +pub async fn issue_api_token( + database: &Database, + username: &str, + label: &str, +) -> Result { + let normalized = validate_username(username)?.to_ascii_lowercase(); + let label = label.trim().to_owned(); + if label.is_empty() || label.len() > 80 { + return Err(IdentityError::InvalidCredentialLabel); + } + let id = random_hex(16)?; + let selector = random_hex(8)?; + let secret = random_hex(32)?; + let token = format!("epk1_{selector}_{secret}"); + let digest = digest(&token); + let (id, user_id) = (id.clone(), normalized.clone()); + let normalized_for_error = normalized.clone(); + match database + .with_transaction(move |connection| { + let user_id: Option = connection + .query_row( + "SELECT id FROM users WHERE normalized_username = ?1 AND state = 'active'", + [&user_id], + |row| row.get(0), + ) + .optional()?; + let Some(user_id) = user_id else { + return Err(tokio_rusqlite::rusqlite::Error::QueryReturnedNoRows); + }; + connection.execute( + "INSERT INTO user_credentials(id, user_id, label, selector, digest, scopes) VALUES (?1, ?2, ?3, ?4, ?5, 'inference,discovery,integrations')", + (&id, &user_id, &label, &selector, &digest), + )?; + Ok(user_id) + }) + .await + { + Ok(user_id) => Ok(IssuedCredential { user_id, token }), + Err(error) => { + if active_user_exists(database, normalized_for_error).await? { + Err(IdentityError::Database(error)) + } else { + Err(IdentityError::UserNotFound) + } + } + } +} + +pub async fn revoke_api_tokens(database: &Database, username: &str) -> Result { + let normalized = validate_username(username)?.to_ascii_lowercase(); + let normalized_for_error = normalized.clone(); + match database + .with_transaction(move |connection| { + let exists: bool = connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE normalized_username = ?1)", + [&normalized], + |row| row.get(0), + )?; + if !exists { + return Err(tokio_rusqlite::rusqlite::Error::QueryReturnedNoRows); + } + let changed = connection.execute( + "UPDATE user_credentials SET revoked_at = CURRENT_TIMESTAMP WHERE user_id = (SELECT id FROM users WHERE normalized_username = ?1) AND revoked_at IS NULL", + [&normalized], + )?; + Ok(changed as u64) + }) + .await + { + Ok(count) => Ok(count), + Err(error) => { + if active_user_exists(database, normalized_for_error).await? { + Err(IdentityError::Database(error)) + } else { + Err(IdentityError::UserNotFound) + } + } + } +} + +async fn active_user_exists( + database: &Database, + normalized: String, +) -> Result { + database + .call(move |connection| { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE normalized_username = ?1 AND state = 'active')", + [&normalized], + |row| row.get(0), + ) + }) + .await + .map_err(IdentityError::Database) +} + +pub async fn verify_password( + database: &Database, + username: &str, + password: &str, +) -> Result, IdentityError> { + if password.len() > MAX_PASSWORD_BYTES { + return Ok(None); + } + let dummy_hash = DUMMY_PASSWORD_HASH + .get_or_try_init(|| async { hash_password(DUMMY_PASSWORD).await }) + .await? + .clone(); + let normalized = validate_username(username)?.to_ascii_lowercase(); + let account: Option<(String, UserSummary)> = database.call(move |connection| { + connection.query_row( + "SELECT id, username, role, state, must_change_password, password_hash FROM users WHERE normalized_username = ?1", + [&normalized], + |row| { + let summary = UserSummary { + id: row.get(0)?, username: row.get(1)?, role: row.get(2)?, state: row.get(3)?, + must_change_password: row.get::<_, i64>(4)? != 0, + }; + Ok((row.get(5)?, summary)) + }, + ).optional() + }).await.map_err(IdentityError::Database)?; + let (password_hash, user) = account + .map(|(password_hash, user)| (password_hash, Some(user))) + .unwrap_or((dummy_hash, None)); + let slots = HASH_SLOTS + .get_or_init(|| Arc::new(Semaphore::new(HASH_CONCURRENCY))) + .clone(); + let password = password.to_owned(); + let verified = run_hash_worker(slots, move || { + let parsed = PasswordHash::new(&password_hash).map_err(|_| IdentityError::InvalidHash)?; + if parsed.algorithm.as_str() != "argon2id" + || parsed.version != Some(19) + || parsed.params.get_decimal("m") != Some(19_456) + || parsed.params.get_decimal("t") != Some(2) + || parsed.params.get_decimal("p") != Some(1) + { + return Err(IdentityError::InvalidHash); + } + Ok(Argon2::default() + .verify_password(password.as_bytes(), &parsed) + .is_ok()) + }) + .await??; + Ok(verified + .then_some(user) + .flatten() + .filter(|user| user.state == "active")) +} + +/// Resolve a high-entropy personal token by its non-secret selector. Only the +/// digest is stored; the final digest check is fixed length and branchless. +pub async fn verify_api_token(database: &Database, token: &str) -> Result { + Ok(verify_api_token_principal(database, token).await?.is_some()) +} + +pub async fn verify_api_token_principal( + database: &Database, + token: &str, +) -> Result, IdentityError> { + let Some(rest) = token.strip_prefix("epk1_") else { + return Ok(None); + }; + let Some((selector, _secret)) = rest.split_once('_') else { + return Ok(None); + }; + if selector.len() != 16 + || !selector.bytes().all(|byte| byte.is_ascii_hexdigit()) + || token.len() > 128 + { + return Ok(None); + } + let failed_digest = digest(token); + if failed_token_recently_seen(&failed_digest) { + return Ok(None); + } + let selector = selector.to_owned(); + let expected = failed_digest.clone(); + database.call(move |connection| { + let stored: Option<(String, String, String, String, i64)> = connection.query_row( + "SELECT c.digest, c.id, c.user_id, c.scopes, u.auth_epoch FROM user_credentials c JOIN users u ON u.id = c.user_id WHERE c.selector = ?1 AND c.revoked_at IS NULL AND (c.expires_at IS NULL OR c.expires_at > CURRENT_TIMESTAMP) AND u.state = 'active'", + [&selector], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?, row.get(4)?)), + ).optional()?; + let Some((stored, credential_id, user_id, scopes, auth_epoch)) = stored else { + remember_failed_token(&failed_digest); + return Ok(None); + }; + let matches = constant_time_equal(stored.as_bytes(), expected.as_bytes()); + if matches { + connection.execute("UPDATE user_credentials SET last_used_at = CURRENT_TIMESTAMP WHERE selector = ?1 AND (last_used_at IS NULL OR last_used_at < datetime('now', '-15 minutes'))", [&selector])?; + let scopes = scopes + .split(',') + .map(str::trim) + .filter(|scope| !scope.is_empty()) + .map(str::to_owned) + .collect(); + Ok(Some(AuthenticatedPrincipal { + user_id: Some(user_id), + credential_id: Some(credential_id), + role: Some("agent".to_owned()), + scopes, + auth_epoch: Some(auth_epoch), + kind: PrincipalKind::AgentToken, + })) + } else { + remember_failed_token(&failed_digest); + Ok(None) + } + }).await.map_err(IdentityError::Database) +} + +fn failed_token_recently_seen(token_digest: &str) -> bool { + let now = Instant::now(); + let mut cache = FAILED_TOKEN_DIGESTS + .get_or_init(|| Mutex::new(VecDeque::new())) + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + cache.retain(|(_, expires)| *expires > now); + cache.iter().any(|(cached, _)| cached == token_digest) +} + +fn remember_failed_token(token_digest: &str) { + let now = Instant::now(); + let mut cache = FAILED_TOKEN_DIGESTS + .get_or_init(|| Mutex::new(VecDeque::new())) + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + cache.retain(|(_, expires)| *expires > now); + cache.retain(|(cached, _)| cached != token_digest); + if cache.len() == FAILED_TOKEN_CAPACITY { + cache.pop_front(); + } + cache.push_back((token_digest.to_owned(), now + FAILED_TOKEN_TTL)); +} + +pub fn validate_username(username: &str) -> Result { + let username = username.trim(); + if username.is_empty() + || username.len() > 64 + || !username + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || b"._-".contains(&byte)) + { + return Err(IdentityError::InvalidUsername); + } + Ok(username.to_owned()) +} + +async fn hash_password(password: &str) -> Result { + if !(MIN_PASSWORD_BYTES..=MAX_PASSWORD_BYTES).contains(&password.len()) { + return Err(IdentityError::InvalidPassword); + } + let slots = HASH_SLOTS + .get_or_init(|| Arc::new(Semaphore::new(HASH_CONCURRENCY))) + .clone(); + let password = password.to_owned(); + run_hash_worker(slots, move || { + let mut salt = [0_u8; 16]; + getrandom::fill(&mut salt).map_err(|_| IdentityError::Entropy)?; + let salt = SaltString::encode_b64(&salt).map_err(|_| IdentityError::Entropy)?; + Argon2::new( + Algorithm::Argon2id, + Version::V0x13, + Params::new(19_456, 2, 1, None).map_err(|_| IdentityError::InvalidHash)?, + ) + .hash_password(password.as_bytes(), &salt) + .map(|hash| hash.to_string()) + .map_err(|_| IdentityError::InvalidHash) + }) + .await? +} + +async fn run_hash_worker(slots: Arc, operation: F) -> Result +where + T: Send + 'static, + F: FnOnce() -> T + Send + 'static, +{ + let permit = slots + .acquire_owned() + .await + .map_err(|_| IdentityError::HashWorker)?; + tokio::task::spawn_blocking(move || { + // The blocking worker can outlive its async caller. Keep its bounded + // Argon2 slot until the actual CPU work exits, even on cancellation. + let _permit = permit; + operation() + }) + .await + .map_err(|_| IdentityError::HashWorker) +} + +fn random_hex(length: usize) -> Result { + let mut bytes = vec![0; length]; + getrandom::fill(&mut bytes).map_err(|_| IdentityError::Entropy)?; + Ok(bytes.iter().map(|byte| format!("{byte:02x}")).collect()) +} + +fn digest(value: &str) -> String { + Sha256::digest(value.as_bytes()) + .iter() + .map(|byte| format!("{byte:02x}")) + .collect() +} + +fn constant_time_equal(left: &[u8], right: &[u8]) -> bool { + let mut difference = (left.len() ^ right.len()) as u8; + for index in 0..left.len().max(right.len()) { + difference |= + left.get(index).copied().unwrap_or(0) ^ right.get(index).copied().unwrap_or(0); + } + difference == 0 +} + +#[cfg(test)] +mod hash_worker_tests { + use super::*; + + #[tokio::test(flavor = "current_thread")] + async fn canceled_hash_caller_keeps_worker_slot_until_hash_exits() { + let slots = Arc::new(Semaphore::new(1)); + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let (finished_tx, finished_rx) = tokio::sync::oneshot::channel(); + let worker_slots = Arc::clone(&slots); + let caller = tokio::spawn(async move { + run_hash_worker(worker_slots, move || { + let _ = started_tx.send(()); + release_rx + .recv() + .expect("test releases the bounded hash worker"); + let _ = finished_tx.send(()); + }) + .await + }); + started_rx.await.expect("hash worker reaches test barrier"); + caller.abort(); + assert!(caller.await.expect_err("caller is canceled").is_cancelled()); + assert_eq!(slots.available_permits(), 0); + + release_tx.send(()).expect("hash worker is waiting"); + tokio::time::timeout(Duration::from_secs(2), finished_rx) + .await + .expect("blocking hash work finishes within a bounded interval") + .expect("hash worker completion is observed"); + tokio::time::timeout( + Duration::from_secs(2), + run_hash_worker(Arc::clone(&slots), || 1_u8), + ) + .await + .expect("released hash slot admits bounded follow-up work") + .expect("follow-up hash worker succeeds"); + assert_eq!(slots.available_permits(), 1); + } +} diff --git a/rust/src/instance_budget.rs b/rust/src/instance_budget.rs new file mode 100644 index 000000000..58ff170c4 --- /dev/null +++ b/rust/src/instance_budget.rs @@ -0,0 +1,801 @@ +//! Instance-global budget ceilings intersected with per-user rules. +//! +//! Exactly one instance-wide scope: a rule with `model_id = None` caps all +//! models, a rule with `model_id = Some` caps that exact model. There is no +//! organization/tenant identifier and no second database writer: checks and +//! holds run in the same publication SQLite transaction as the user budget, +//! and settlement folds the same per-attempt usage rows. Instance tables +//! carry no user reference, so global counters survive user disable/delete +//! and cover unattributed (legacy-key) requests as well. Absent instance +//! rules, publication takes no instance hold and writes no instance rows, so +//! a single-admin workload is unchanged except measured gate overhead. +//! +//! Hard/estimated semantics, worst-case holds, and the operator-tariff trust +//! boundary are shared with `user_budget` (M008 contract); this module only +//! adds the global scope and its intersection. + +use thiserror::Error; +use tokio_rusqlite::rusqlite::{OptionalExtension, params}; + +use crate::{ + db::{Database, DatabaseError}, + user_budget::{ + BudgetAttemptSettlement, BudgetCheck, BudgetEnforcement, BudgetHold, BudgetLimits, + BudgetPriceSource, BudgetProof, BudgetWindow, exceeds, hold_for_admission, hold_for_retry, + }, +}; + +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +pub struct InstanceBudgetRule { + pub id: i64, + pub model_id: Option, + pub window: BudgetWindow, + pub max_requests: Option, + pub max_tokens: Option, + pub max_microdollars: Option, + pub max_concurrent: Option, + pub enforcement: BudgetEnforcement, + pub version: i64, +} + +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +pub struct InstanceBudgetUsage { + pub model_id: String, + pub window: BudgetWindow, + pub window_start: String, + pub request_count: i64, + pub token_count: i64, + pub cost_microdollars: i64, + pub provider_reported_microdollars: i64, + pub local_estimated_microdollars: i64, + pub reservation_fallback_microdollars: i64, + pub unknown_token_requests: i64, + pub unknown_cost_requests: i64, +} + +#[derive(Debug, Error)] +pub enum InstanceBudgetError { + #[error("invalid instance budget rule")] + InvalidRule, + #[error("an active administrator is required")] + ActorNotAdmin, + #[error("instance budget database operation failed: {0}")] + Database(#[from] DatabaseError), +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum MutationResult { + Updated, + Removed(bool), + ActorNotAdmin, +} + +#[derive(Debug, Clone)] +pub struct InstanceBudgetService { + database: Database, + browser_admin_session: Option, +} + +impl InstanceBudgetService { + pub fn new(database: Database) -> Self { + Self { + database, + browser_admin_session: None, + } + } + + pub fn with_browser_admin_session(mut self, principal: &crate::web_session::Principal) -> Self { + self.browser_admin_session = + Some(crate::web_session::SessionFence::from_principal(principal)); + self + } + + /// Create or replace the instance rule for one model scope and window. + /// Only active admins may mutate; ordinary member/API tokens cannot. + pub async fn set_rule( + &self, + actor_id: &str, + model_id: Option<&str>, + window: BudgetWindow, + limits: BudgetLimits, + ) -> Result<(), InstanceBudgetError> { + if model_id.is_some_and(|value| value.is_empty() || value.trim() != value) + || [ + limits.requests, + limits.tokens, + limits.microdollars, + limits.concurrent, + ] + .into_iter() + .flatten() + .any(|value| value <= 0) + || [ + limits.requests, + limits.tokens, + limits.microdollars, + limits.concurrent, + ] + .into_iter() + .all(|value| value.is_none()) + { + return Err(InstanceBudgetError::InvalidRule); + } + let actor_id = actor_id.to_owned(); + let model_id = model_id.map(str::to_owned); + let window = window.as_str().to_owned(); + let enforcement = limits.enforcement.as_str().to_owned(); + let browser_admin_session = self.browser_admin_session.clone(); + let outcome = self + .database + .with_transaction(move |connection| { + let actor = if let Some(fence) = browser_admin_session.as_ref() { + fence.validates_actor(connection, &actor_id, true)? + } else { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE id = ?1 AND role = 'admin' AND state = 'active')", + [&actor_id], + |row| row.get(0), + )? + }; + if !actor { + return Ok(MutationResult::ActorNotAdmin); + } + connection.execute( + "INSERT INTO instance_budget_rules( + model_id, window, max_requests, max_tokens, + max_microdollars, max_concurrent, enforcement, updated_by) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8) + ON CONFLICT(ifnull(model_id, ''), window) DO UPDATE SET + max_requests = excluded.max_requests, + max_tokens = excluded.max_tokens, + max_microdollars = excluded.max_microdollars, + max_concurrent = excluded.max_concurrent, + enforcement = excluded.enforcement, + version = instance_budget_rules.version + 1, + updated_by = excluded.updated_by, + updated_at = CURRENT_TIMESTAMP", + params![ + model_id, + window, + limits.requests, + limits.tokens, + limits.microdollars, + limits.concurrent, + enforcement, + actor_id, + ], + )?; + connection.execute( + "INSERT INTO instance_budget_rule_audit( + actor_id, model_id, window, action, max_requests, + max_tokens, max_microdollars, max_concurrent, enforcement, rule_version) + SELECT updated_by, model_id, window, 'set', max_requests, + max_tokens, max_microdollars, max_concurrent, enforcement, version + FROM instance_budget_rules WHERE model_id IS ?1 AND window = ?2", + params![model_id, window], + )?; + Ok(MutationResult::Updated) + }) + .await + .map_err(InstanceBudgetError::Database)?; + match outcome { + MutationResult::Updated => Ok(()), + MutationResult::ActorNotAdmin => Err(InstanceBudgetError::ActorNotAdmin), + MutationResult::Removed(_) => Err(InstanceBudgetError::InvalidRule), + } + } + + pub async fn remove_rule( + &self, + actor_id: &str, + model_id: Option<&str>, + window: BudgetWindow, + ) -> Result { + let actor_id = actor_id.to_owned(); + let model_id = model_id.map(str::to_owned); + let window = window.as_str().to_owned(); + let browser_admin_session = self.browser_admin_session.clone(); + let outcome = self + .database + .with_transaction(move |connection| { + let actor = if let Some(fence) = browser_admin_session.as_ref() { + fence.validates_actor(connection, &actor_id, true)? + } else { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE id = ?1 AND role = 'admin' AND state = 'active')", + [&actor_id], + |row| row.get(0), + )? + }; + if !actor { + return Ok(MutationResult::ActorNotAdmin); + } + let previous = connection + .query_row( + "SELECT max_requests, max_tokens, max_microdollars, max_concurrent, enforcement, version + FROM instance_budget_rules WHERE model_id IS ?1 AND window = ?2", + params![model_id, window], + |row| { + Ok(( + row.get::<_, Option>(0)?, + row.get::<_, Option>(1)?, + row.get::<_, Option>(2)?, + row.get::<_, Option>(3)?, + row.get::<_, String>(4)?, + row.get::<_, i64>(5)?, + )) + }, + ) + .optional()?; + let removed = connection.execute( + "DELETE FROM instance_budget_rules WHERE model_id IS ?1 AND window = ?2", + params![model_id, window], + )? == 1; + if let Some((requests, tokens, cost, concurrent, enforcement, version)) = previous { + connection.execute( + "INSERT INTO instance_budget_rule_audit( + actor_id, model_id, window, action, max_requests, + max_tokens, max_microdollars, max_concurrent, enforcement, rule_version) + VALUES (?1, ?2, ?3, 'remove', ?4, ?5, ?6, ?7, ?8, ?9)", + params![ + actor_id, + model_id, + window, + requests, + tokens, + cost, + concurrent, + enforcement, + version + 1, + ], + )?; + } + Ok(MutationResult::Removed(removed)) + }) + .await + .map_err(InstanceBudgetError::Database)?; + match outcome { + MutationResult::Removed(changed) => Ok(changed), + MutationResult::ActorNotAdmin => Err(InstanceBudgetError::ActorNotAdmin), + MutationResult::Updated => Err(InstanceBudgetError::InvalidRule), + } + } + + pub async fn rules(&self) -> Result, InstanceBudgetError> { + self.database + .call(|connection| { + let mut statement = connection.prepare( + "SELECT id, model_id, window, max_requests, max_tokens, + max_microdollars, max_concurrent, enforcement, version + FROM instance_budget_rules + ORDER BY model_id IS NOT NULL, model_id, window", + )?; + statement + .query_map([], |row| { + let window: String = row.get(2)?; + let enforcement: String = row.get(7)?; + Ok(InstanceBudgetRule { + id: row.get(0)?, + model_id: row.get(1)?, + window: parse_window(&window), + max_requests: row.get(3)?, + max_tokens: row.get(4)?, + max_microdollars: row.get(5)?, + max_concurrent: row.get(6)?, + enforcement: BudgetEnforcement::parse(&enforcement) + .unwrap_or(BudgetEnforcement::Estimated), + version: row.get(8)?, + }) + })? + .collect() + }) + .await + .map_err(InstanceBudgetError::Database) + } + + pub async fn usage( + &self, + since_utc: &str, + limit: u16, + ) -> Result, InstanceBudgetError> { + if since_utc.len() != 10 || limit == 0 || limit > 500 { + return Err(InstanceBudgetError::InvalidRule); + } + let since_utc = since_utc.to_owned(); + self.database + .call(move |connection| { + let mut statement = connection.prepare( + "SELECT model_id, window, window_start, request_count, token_count, + cost_microdollars, provider_reported_microdollars, + local_estimated_microdollars, reservation_fallback_microdollars, + unknown_token_requests, unknown_cost_requests + FROM instance_budget_usage WHERE window_start >= ?1 + ORDER BY window_start DESC, model_id LIMIT ?2", + )?; + statement + .query_map(params![since_utc, i64::from(limit)], |row| { + let window: String = row.get(1)?; + Ok(InstanceBudgetUsage { + model_id: row.get(0)?, + window: parse_window(&window), + window_start: row.get(2)?, + request_count: row.get(3)?, + token_count: row.get(4)?, + cost_microdollars: row.get(5)?, + provider_reported_microdollars: row.get(6)?, + local_estimated_microdollars: row.get(7)?, + reservation_fallback_microdollars: row.get(8)?, + unknown_token_requests: row.get(9)?, + unknown_cost_requests: row.get(10)?, + }) + })? + .collect() + }) + .await + .map_err(InstanceBudgetError::Database) + } +} + +fn parse_window(value: &str) -> BudgetWindow { + if value == "month" { + BudgetWindow::Month + } else { + BudgetWindow::Day + } +} + +#[derive(Debug, Clone)] +pub(crate) struct InstancePermit { + pub policy_version: i64, + pub day_start: String, + pub month_start: String, +} + +struct LoadedInstanceRule { + scope_model: Option, + window: String, + max_requests: Option, + max_tokens: Option, + max_cost: Option, + max_concurrent: Option, + enforcement: BudgetEnforcement, +} + +/// Evaluate the instance-global intersection for one new logical request. +/// Returns `None` (allowed, no hold) when no instance rule matches the model, +/// so rule-free operation takes no instance hold and writes no instance rows. +pub(crate) fn check_initial( + connection: &tokio_rusqlite::rusqlite::Connection, + model_id: &str, + tokens: i64, + microdollars: i64, + proof: Option<&BudgetProof>, +) -> Result<(BudgetCheck, Option, BudgetHold), tokio_rusqlite::rusqlite::Error> { + let (policy_version, day_start, month_start): (i64, String, String) = connection.query_row( + "SELECT revision, date('now'), strftime('%Y-%m-01', 'now') FROM policy_state WHERE singleton = 1", + [], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + )?; + let permit = InstancePermit { + policy_version, + day_start, + month_start, + }; + let rules = load_matching_rules(connection, model_id)?; + if rules.is_empty() { + return Ok((BudgetCheck::Allowed, None, BudgetHold::empty())); + } + let hard_token = rules + .iter() + .any(|rule| rule.enforcement == BudgetEnforcement::Hard && rule.max_tokens.is_some()); + let hard_cost = rules + .iter() + .any(|rule| rule.enforcement == BudgetEnforcement::Hard && rule.max_cost.is_some()); + let Some(hold) = hold_for_admission(tokens, microdollars, proof, hard_token, hard_cost) else { + return Ok((BudgetCheck::NotEnforceable, None, BudgetHold::empty())); + }; + let exceeded = rules_exceeded( + connection, + &permit, + &rules, + hold, + tokens, + microdollars, + true, + )?; + Ok(( + if exceeded { + BudgetCheck::Exceeded + } else { + BudgetCheck::Allowed + }, + Some(permit), + hold, + )) +} + +/// Re-evaluate the instance intersection before a retry attempt extends the +/// logical hold. Returns `None` when this request never took an instance +/// hold, so rule-free requests skip instance work on every attempt. +pub(crate) fn check_retry( + connection: &tokio_rusqlite::rusqlite::Connection, + request_id: i64, + tokens: i64, + microdollars: i64, + proof: Option<&BudgetProof>, +) -> Result<(BudgetCheck, Option), tokio_rusqlite::rusqlite::Error> { + let reservation = connection + .query_row( + "SELECT model_id, policy_version, day_start, month_start + FROM instance_budget_reservations WHERE request_id = ?1 AND status = 'active'", + [request_id], + |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, i64>(1)?, + row.get::<_, String>(2)?, + row.get::<_, String>(3)?, + )) + }, + ) + .optional()?; + let Some((model_id, policy_version, day_start, month_start)) = reservation else { + return Ok((BudgetCheck::Allowed, None)); + }; + let permit = InstancePermit { + policy_version, + day_start, + month_start, + }; + let rules = load_matching_rules(connection, &model_id)?; + let hard_token = rules + .iter() + .any(|rule| rule.enforcement == BudgetEnforcement::Hard && rule.max_tokens.is_some()); + let hard_cost = rules + .iter() + .any(|rule| rule.enforcement == BudgetEnforcement::Hard && rule.max_cost.is_some()); + let Some(hold) = hold_for_retry(tokens, microdollars, proof, hard_token, hard_cost) else { + return Ok((BudgetCheck::NotEnforceable, None)); + }; + let exceeded = rules_exceeded( + connection, + &permit, + &rules, + hold, + tokens, + microdollars, + false, + )?; + Ok(( + if exceeded { + BudgetCheck::Exceeded + } else { + BudgetCheck::Allowed + }, + Some(hold), + )) +} + +pub(crate) struct InstanceReservationSpec<'a> { + pub request_id: i64, + pub model_id: &'a str, + pub tokens: i64, + pub microdollars: i64, + pub ttl_seconds: i64, + pub hold: BudgetHold, +} + +pub(crate) fn insert_reservation( + connection: &tokio_rusqlite::rusqlite::Connection, + spec: InstanceReservationSpec<'_>, + permit: &InstancePermit, +) -> Result<(), tokio_rusqlite::rusqlite::Error> { + connection.execute( + "INSERT INTO instance_budget_reservations( + request_id, model_id, policy_version, day_start, month_start, + held_tokens, held_microdollars, status, expires_at, + enforcement, bound_tokens, bound_microdollars, price_source) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, 'active', + datetime('now', '+' || ?8 || ' seconds'), ?9, ?10, ?11, ?12)", + params![ + spec.request_id, + spec.model_id, + permit.policy_version, + permit.day_start, + permit.month_start, + spec.tokens, + spec.microdollars, + spec.ttl_seconds, + spec.hold.enforcement.as_str(), + spec.hold.bound_tokens, + spec.hold.bound_microdollars, + spec.hold.price_source.map(BudgetPriceSource::as_str), + ], + )?; + Ok(()) +} + +pub(crate) fn extend_reservation( + connection: &tokio_rusqlite::rusqlite::Connection, + request_id: i64, + tokens: i64, + microdollars: i64, +) -> Result<(), tokio_rusqlite::rusqlite::Error> { + let changed = connection.execute( + "UPDATE instance_budget_reservations SET held_tokens = held_tokens + ?1, + held_microdollars = held_microdollars + ?2 + WHERE request_id = ?3 AND status = 'active'", + params![tokens, microdollars, request_id], + )?; + if changed != 1 { + return Err(tokio_rusqlite::rusqlite::Error::InvalidParameterName( + "active instance budget reservation missing".to_owned(), + )); + } + Ok(()) +} + +fn load_matching_rules( + connection: &tokio_rusqlite::rusqlite::Connection, + model_id: &str, +) -> Result, tokio_rusqlite::rusqlite::Error> { + let mut statement = connection.prepare( + "SELECT model_id, window, max_requests, max_tokens, max_microdollars, + max_concurrent, enforcement + FROM instance_budget_rules WHERE model_id IS NULL OR model_id = ?1", + )?; + statement + .query_map(params![model_id], |row| { + let enforcement: String = row.get(6)?; + Ok(LoadedInstanceRule { + scope_model: row.get(0)?, + window: row.get(1)?, + max_requests: row.get(2)?, + max_tokens: row.get(3)?, + max_cost: row.get(4)?, + max_concurrent: row.get(5)?, + enforcement: BudgetEnforcement::parse(&enforcement) + .unwrap_or(BudgetEnforcement::Estimated), + }) + })? + .collect() +} + +/// Evaluate every matching instance rule against settled usage plus active +/// holds. Global (model-agnostic) rules sum across per-model usage rows with +/// a bounded indexed aggregate; exact-model rules read their own row scope. +/// Each logical request counts once: retries extend the one hold instead of +/// adding requests. +fn rules_exceeded( + connection: &tokio_rusqlite::rusqlite::Connection, + permit: &InstancePermit, + rules: &[LoadedInstanceRule], + hold: BudgetHold, + estimate_tokens: i64, + estimate_microdollars: i64, + new_request: bool, +) -> Result { + for rule in rules { + let window_start = if rule.window == "day" { + &permit.day_start + } else { + &permit.month_start + }; + let window = rule.window.as_str(); + let token_add = if rule.enforcement == BudgetEnforcement::Hard && rule.max_tokens.is_some() + { + hold.tokens + } else { + estimate_tokens.max(0) + }; + let cost_add = if rule.enforcement == BudgetEnforcement::Hard && rule.max_cost.is_some() { + hold.microdollars + } else { + estimate_microdollars.max(0) + }; + let (used_requests, used_tokens, used_cost): (i64, i64, i64) = + if let Some(model) = rule.scope_model.as_deref() { + connection.query_row( + "SELECT coalesce(sum(request_count), 0), coalesce(sum(token_count), 0), + coalesce(sum(cost_microdollars), 0) + FROM instance_budget_usage WHERE model_id = ?1 + AND window = ?2 AND window_start = ?3", + params![model, window, window_start], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + )? + } else { + connection.query_row( + "SELECT coalesce(sum(request_count), 0), coalesce(sum(token_count), 0), + coalesce(sum(cost_microdollars), 0) + FROM instance_budget_usage WHERE window = ?1 AND window_start = ?2", + params![window, window_start], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + )? + }; + // Active holds use the same window-case projection as the user scope + // so requests straddling a UTC reset still occupy their slot, while + // the concurrency count spans windows. All aggregates are bounded + // indexed lookups; no full history scan runs under the writer gate. + let (held_requests, held_tokens, held_cost, active_count): (i64, i64, i64, i64) = + if let Some(model) = rule.scope_model.as_deref() { + connection.query_row( + "SELECT coalesce(sum(CASE WHEN (?2 = 'day' AND day_start = ?3) + OR (?2 = 'month' AND month_start = ?3) + THEN held_requests ELSE 0 END), 0), + coalesce(sum(CASE WHEN (?2 = 'day' AND day_start = ?3) + OR (?2 = 'month' AND month_start = ?3) + THEN held_tokens ELSE 0 END), 0), + coalesce(sum(CASE WHEN (?2 = 'day' AND day_start = ?3) + OR (?2 = 'month' AND month_start = ?3) + THEN held_microdollars ELSE 0 END), 0), count(*) + FROM instance_budget_reservations WHERE model_id = ?1 + AND status = 'active'", + params![model, window, window_start], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?)), + )? + } else { + connection.query_row( + "SELECT coalesce(sum(CASE WHEN (?1 = 'day' AND day_start = ?2) + OR (?1 = 'month' AND month_start = ?2) + THEN held_requests ELSE 0 END), 0), + coalesce(sum(CASE WHEN (?1 = 'day' AND day_start = ?2) + OR (?1 = 'month' AND month_start = ?2) + THEN held_tokens ELSE 0 END), 0), + coalesce(sum(CASE WHEN (?1 = 'day' AND day_start = ?2) + OR (?1 = 'month' AND month_start = ?2) + THEN held_microdollars ELSE 0 END), 0), count(*) + FROM instance_budget_reservations WHERE status = 'active'", + params![window, window_start], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?)), + )? + }; + // Retries extend the one logical hold: like the user scope, they + // consume no additional request or concurrency slot. + let request_hold = i64::from(new_request); + if rule + .max_requests + .is_some_and(|limit| exceeds(limit, used_requests, held_requests, request_hold)) + || rule + .max_tokens + .is_some_and(|limit| exceeds(limit, used_tokens, held_tokens, token_add)) + || rule + .max_cost + .is_some_and(|limit| exceeds(limit, used_cost, held_cost, cost_add)) + || rule + .max_concurrent + .is_some_and(|limit| exceeds(limit, active_count, request_hold, 0)) + { + return Ok(true); + } + } + Ok(false) +} + +/// Fold one request's per-attempt usage into the instance buckets and settle +/// its instance hold. Attempt facts come from the shared attempt table, so +/// unattributed requests settle exactly like attributed ones. No-op when the +/// request never took an instance hold. +pub(crate) fn record_attempt_and_maybe_settle( + connection: &tokio_rusqlite::rusqlite::Connection, + settlement: BudgetAttemptSettlement, +) -> Result<(), tokio_rusqlite::rusqlite::Error> { + let BudgetAttemptSettlement { + request_id, + terminalize_request, + .. + } = settlement; + let reservation = connection + .query_row( + "SELECT model_id, day_start, month_start, held_tokens, + held_microdollars, status + FROM instance_budget_reservations WHERE request_id = ?1", + [request_id], + |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, String>(2)?, + row.get::<_, i64>(3)?, + row.get::<_, i64>(4)?, + row.get::<_, String>(5)?, + )) + }, + ) + .optional()?; + let Some((model_id, day_start, month_start, held_tokens, held_cost, status)) = reservation + else { + return Ok(()); + }; + if !terminalize_request || status == "settled" || status == "released" { + return Ok(()); + } + if status != "active" { + return Err(tokio_rusqlite::rusqlite::Error::InvalidParameterName( + "unknown instance budget reservation state".to_owned(), + )); + } + let (observed_tokens, observed_cost, provider_cost, local_cost, token_unknown, cost_unknown): ( + i64, + i64, + i64, + i64, + i64, + i64, + ) = connection.query_row( + "SELECT coalesce(sum(input_tokens + output_tokens), 0), + coalesce(sum(cost_microdollars), 0), + coalesce(sum(provider_reported_microdollars), 0), + coalesce(sum(local_estimated_microdollars), 0), + coalesce(sum(CASE WHEN tokens_known = 0 THEN 1 ELSE 0 END), 0), + coalesce(sum(CASE WHEN cost_known = 0 THEN 1 ELSE 0 END), 0) + FROM user_budget_attempt_usage WHERE request_id = ?1", + [request_id], + |row| { + Ok(( + row.get(0)?, + row.get(1)?, + row.get(2)?, + row.get(3)?, + row.get(4)?, + row.get(5)?, + )) + }, + )?; + let charged_tokens = if token_unknown > 0 { + held_tokens.max(observed_tokens) + } else { + observed_tokens + }; + let charged_cost = if cost_unknown > 0 { + held_cost.max(observed_cost) + } else { + observed_cost + }; + let fallback_cost = (charged_cost - provider_cost - local_cost).max(0); + for (window, start) in [("day", day_start.as_str()), ("month", month_start.as_str())] { + connection.execute( + "INSERT INTO instance_budget_usage( + model_id, window, window_start, request_count, token_count, + cost_microdollars, provider_reported_microdollars, + local_estimated_microdollars, reservation_fallback_microdollars, + unknown_token_requests, unknown_cost_requests) + VALUES (?1, ?2, ?3, 1, ?4, ?5, ?6, ?7, ?8, ?9, ?10) + ON CONFLICT(model_id, window, window_start) DO UPDATE SET + request_count = instance_budget_usage.request_count + 1, + token_count = instance_budget_usage.token_count + excluded.token_count, + cost_microdollars = instance_budget_usage.cost_microdollars + excluded.cost_microdollars, + provider_reported_microdollars = instance_budget_usage.provider_reported_microdollars + excluded.provider_reported_microdollars, + local_estimated_microdollars = instance_budget_usage.local_estimated_microdollars + excluded.local_estimated_microdollars, + reservation_fallback_microdollars = instance_budget_usage.reservation_fallback_microdollars + excluded.reservation_fallback_microdollars, + unknown_token_requests = instance_budget_usage.unknown_token_requests + excluded.unknown_token_requests, + unknown_cost_requests = instance_budget_usage.unknown_cost_requests + excluded.unknown_cost_requests, + updated_at = CURRENT_TIMESTAMP", + params![ + model_id, + window, + start, + charged_tokens, + charged_cost, + provider_cost, + local_cost, + fallback_cost, + i64::from(token_unknown > 0), + i64::from(cost_unknown > 0), + ], + )?; + } + connection.execute( + "UPDATE instance_budget_reservations SET status = 'settled', settled_at = CURRENT_TIMESTAMP, + observed_tokens = ?1, observed_microdollars = ?2, + token_usage_complete = ?3, cost_usage_complete = ?4 + WHERE request_id = ?5 AND status = 'active'", + params![ + observed_tokens, + observed_cost, + i64::from(token_unknown == 0), + i64::from(cost_unknown == 0), + request_id, + ], + )?; + Ok(()) +} diff --git a/rust/src/lib.rs b/rust/src/lib.rs index 58014a6ce..e3c43a518 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -14,6 +14,9 @@ pub mod coordinator; pub mod db; mod error; pub mod health; +pub mod identity; +pub mod instance_budget; +pub mod model_policy; pub mod model_router; pub mod operations; pub mod provider_profile; @@ -26,7 +29,9 @@ mod runtime; pub mod runtime_lifecycle; pub mod server; pub mod task_supervisor; +pub mod user_budget; pub mod version; +pub mod web_session; pub mod wire; pub use cli::{Cli, Command}; diff --git a/rust/src/model_policy.rs b/rust/src/model_policy.rs new file mode 100644 index 000000000..b223f25aa --- /dev/null +++ b/rust/src/model_policy.rs @@ -0,0 +1,767 @@ +//! Versioned, user-specific model authorization. +//! +//! Policy snapshots are immutable for one admitted request. SQLite revision +//! triggers invalidate the bounded process cache on the next request. + +use std::{ + collections::{BTreeMap, BTreeSet, HashMap}, + sync::{Arc, Mutex}, +}; + +use thiserror::Error; + +use crate::db::Database; + +const POLICY_CACHE_CAPACITY: usize = 256; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Effect { + Allow, + Deny, +} + +impl Effect { + fn parse(value: &str) -> Self { + if value == "allow" { + Self::Allow + } else { + Self::Deny + } + } +} + +#[derive(Debug, Error)] +pub enum ModelPolicyError { + #[error("user policy is unavailable")] + UserUnavailable, + #[error("model policy input is invalid")] + InvalidPolicyInput, + #[error("preferred model is not allowed by its profile")] + PreferredModelDenied, + #[error(transparent)] + Database(#[from] crate::db::DatabaseError), +} + +#[derive(Debug, Clone)] +pub struct ModelPolicySnapshot { + revision: i64, + role: String, + default_effect: Option, + preferred_model_id: Option, + profile_rules: BTreeMap, + user_overrides: BTreeMap, + instance_denials: BTreeSet, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PolicyProfileSummary { + pub id: String, + pub name: String, + pub default_effect: String, + pub preferred_model_id: Option, + pub version: i64, + pub assigned_users: i64, + pub model_rules: i64, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PolicyProfileRuleSummary { + pub profile_id: String, + pub model_id: String, + pub effect: String, +} + +impl ModelPolicySnapshot { + /// Monotonic persisted revision used by discovery ETags and diagnostics. + pub fn revision(&self) -> i64 { + self.revision + } + + pub fn preferred_model_id(&self) -> Option<&str> { + self.preferred_model_id.as_deref() + } + + /// Resolve exact provider-qualified rules before the canonical model rule. + /// A global instance denial is always checked first. + pub fn allows(&self, model_id: &str, provider_id: Option<&str>) -> bool { + let qualified = provider_id.map(|provider| format!("{model_id}/{provider}")); + if self.instance_denials.contains(model_id) + || qualified + .as_ref() + .is_some_and(|model| self.instance_denials.contains(model)) + { + return false; + } + if let Some(effect) = exact_effect(&self.user_overrides, model_id, qualified.as_deref()) { + return *effect == Effect::Allow; + } + if let Some(effect) = exact_effect(&self.profile_rules, model_id, qualified.as_deref()) { + return *effect == Effect::Allow; + } + self.default_effect.unwrap_or(if self.role == "admin" { + Effect::Allow + } else { + Effect::Deny + }) == Effect::Allow + } + + pub fn filter_models(&self, model_ids: impl IntoIterator) -> Vec { + model_ids + .into_iter() + .filter(|model_id| self.allows_catalog_id(model_id)) + .collect() + } + + /// Catalog and integration identifiers append the provider as a suffix. + pub fn allows_catalog_id(&self, public_id: &str) -> bool { + match public_id.rsplit_once('/') { + Some((model_id, provider_id)) => self.allows(model_id, Some(provider_id)), + None => self.allows(public_id, None), + } + } +} + +fn exact_effect<'a>( + rules: &'a BTreeMap, + model_id: &str, + qualified: Option<&str>, +) -> Option<&'a Effect> { + qualified + .and_then(|model| rules.get(model)) + .or_else(|| rules.get(model_id)) +} + +#[derive(Clone)] +pub struct ModelPolicyService { + database: Database, + cache: Arc>>>, + mutation_fence: Option, +} + +type AdminSessionFence = crate::web_session::SessionFence; + +fn validate_mutation_actor( + connection: &mut tokio_rusqlite::rusqlite::Connection, + actor_id: &str, + fence: Option<&AdminSessionFence>, +) -> Result<(), tokio_rusqlite::rusqlite::Error> { + let Some(fence) = fence else { + return Ok(()); + }; + if !fence.validates_actor(connection, actor_id, true)? { + return Err(tokio_rusqlite::rusqlite::Error::QueryReturnedNoRows); + } + Ok(()) +} + +impl std::fmt::Debug for ModelPolicyService { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter + .debug_struct("ModelPolicyService") + .field( + "cached_users", + &self.cache.lock().map(|cache| cache.len()).unwrap_or(0), + ) + .finish() + } +} + +impl ModelPolicyService { + pub fn new(database: Database) -> Self { + Self { + database, + cache: Arc::new(Mutex::new(HashMap::new())), + mutation_fence: None, + } + } + + /// Fence browser-admin mutations against session revocation or user + /// changes that occur after middleware admits the request. + pub fn with_browser_admin_session(mut self, principal: &crate::web_session::Principal) -> Self { + self.mutation_fence = Some(AdminSessionFence::from_principal(principal)); + self + } + + pub async fn create_profile( + &self, + id: &str, + name: &str, + default_effect: &str, + actor_id: &str, + ) -> Result<(), ModelPolicyError> { + let id = id.trim(); + let name = name.trim().to_owned(); + if id.is_empty() || id.len() > 64 || name.is_empty() || name.len() > 64 { + return Err(ModelPolicyError::InvalidPolicyInput); + } + let id = id.to_owned(); + let default_effect = validate_effect(default_effect)?; + let actor_id = actor_id.to_owned(); + let mutation_fence = self.mutation_fence.clone(); + self.database + .with_transaction(move |connection| { + validate_mutation_actor(connection, &actor_id, mutation_fence.as_ref())?; + connection.execute( + "INSERT INTO policy_profiles(id, name, default_effect, created_by, updated_by) \ + VALUES (?1, ?2, ?3, ?4, ?4)", + (&id, &name, default_effect, &actor_id), + )?; + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'create_profile', 'profile', ?2)", + (&actor_id, &id), + )?; + Ok(()) + }) + .await?; + Ok(()) + } + + pub async fn list_profiles(&self) -> Result, ModelPolicyError> { + self.database + .call(|connection| { + let mut statement = connection.prepare( + "SELECT p.id, p.name, p.default_effect, p.preferred_model_id, p.version, \ + (SELECT COUNT(*) FROM user_profile_assignments a WHERE a.profile_id = p.id), \ + (SELECT COUNT(*) FROM profile_model_rules r WHERE r.profile_id = p.id) \ + FROM policy_profiles p ORDER BY p.name COLLATE NOCASE, p.id LIMIT 200", + )?; + statement + .query_map([], |row| { + Ok(PolicyProfileSummary { + id: row.get(0)?, + name: row.get(1)?, + default_effect: row.get(2)?, + preferred_model_id: row.get(3)?, + version: row.get(4)?, + assigned_users: row.get(5)?, + model_rules: row.get(6)?, + }) + })? + .collect() + }) + .await + .map_err(ModelPolicyError::Database) + } + + pub async fn list_profile_rules( + &self, + ) -> Result, ModelPolicyError> { + self.database + .call(|connection| { + let mut statement = connection.prepare( + "SELECT profile_id, model_id, effect FROM profile_model_rules \ + ORDER BY profile_id, model_id LIMIT 2000", + )?; + statement + .query_map([], |row| { + Ok(PolicyProfileRuleSummary { + profile_id: row.get(0)?, + model_id: row.get(1)?, + effect: row.get(2)?, + }) + })? + .collect() + }) + .await + .map_err(ModelPolicyError::Database) + } + + pub async fn remove_profile_rule( + &self, + profile_id: &str, + model_id: &str, + actor_id: &str, + ) -> Result { + let model_id = validate_model_id(model_id)?.to_owned(); + let profile_id = profile_id.to_owned(); + let actor_id = actor_id.to_owned(); + let mutation_fence = self.mutation_fence.clone(); + self.database + .with_transaction(move |connection| { + validate_mutation_actor(connection, &actor_id, mutation_fence.as_ref())?; + let removed = connection.execute( + "DELETE FROM profile_model_rules WHERE profile_id = ?1 AND model_id = ?2", + (&profile_id, &model_id), + )?; + if removed == 1 { + connection.execute( + "UPDATE policy_profiles SET version = version + 1, updated_by = ?2, \ + updated_at = CURRENT_TIMESTAMP WHERE id = ?1", + (&profile_id, &actor_id), + )?; + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'remove_profile_rule', 'profile', ?2)", + (&actor_id, &profile_id), + )?; + } + Ok(removed == 1) + }) + .await + .map_err(ModelPolicyError::Database) + } + + /// Compare-and-swap profile metadata. A stale browser form gets a conflict + /// instead of silently overwriting a newer administrator edit. + pub async fn update_profile( + &self, + id: &str, + name: &str, + default_effect: &str, + expected_version: i64, + actor_id: &str, + ) -> Result { + let name = name.trim().to_owned(); + if name.is_empty() || name.len() > 64 || expected_version <= 0 { + return Err(ModelPolicyError::InvalidPolicyInput); + } + let effect = validate_effect(default_effect)?; + let id = id.to_owned(); + let actor_id = actor_id.to_owned(); + let mutation_fence = self.mutation_fence.clone(); + self.database + .with_transaction(move |connection| { + validate_mutation_actor(connection, &actor_id, mutation_fence.as_ref())?; + let changed = connection.execute( + "UPDATE policy_profiles SET name = ?1, default_effect = ?2, \ + version = version + 1, updated_by = ?3, updated_at = CURRENT_TIMESTAMP \ + WHERE id = ?4 AND version = ?5", + (&name, effect, &actor_id, &id, expected_version), + )?; + if changed == 1 { + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'update_profile', 'profile', ?2)", + (&actor_id, &id), + )?; + } + Ok(changed == 1) + }) + .await + .map_err(ModelPolicyError::Database) + } + + pub async fn delete_profile( + &self, + id: &str, + expected_version: i64, + actor_id: &str, + ) -> Result { + if expected_version <= 0 { + return Err(ModelPolicyError::InvalidPolicyInput); + } + let id = id.to_owned(); + let actor_id = actor_id.to_owned(); + let mutation_fence = self.mutation_fence.clone(); + self.database + .with_transaction(move |connection| { + validate_mutation_actor(connection, &actor_id, mutation_fence.as_ref())?; + let assigned: bool = connection.query_row( + "SELECT EXISTS(SELECT 1 FROM user_profile_assignments WHERE profile_id = ?1)", + [&id], + |row| row.get(0), + )?; + if assigned { + return Ok(false); + } + let changed = connection.execute( + "DELETE FROM policy_profiles WHERE id = ?1 AND version = ?2", + (&id, expected_version), + )?; + if changed == 1 { + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'delete_profile', 'profile', ?2)", + (&actor_id, &id), + )?; + } + Ok(changed == 1) + }) + .await + .map_err(ModelPolicyError::Database) + } + + pub async fn set_profile_rule( + &self, + profile_id: &str, + model_id: &str, + effect: &str, + actor_id: &str, + ) -> Result<(), ModelPolicyError> { + let model_id = validate_model_id(model_id)?.to_owned(); + let effect = validate_effect(effect)?; + let profile_id = profile_id.to_owned(); + let actor_id = actor_id.to_owned(); + let mutation_fence = self.mutation_fence.clone(); + self.database + .with_transaction(move |connection| { + validate_mutation_actor(connection, &actor_id, mutation_fence.as_ref())?; + connection.execute( + "INSERT INTO profile_model_rules(profile_id, model_id, effect, created_by, updated_by) \ + VALUES (?1, ?2, ?3, ?4, ?4) \ + ON CONFLICT(profile_id, model_id) DO UPDATE SET \ + effect = excluded.effect, updated_by = excluded.updated_by, \ + updated_at = CURRENT_TIMESTAMP", + (&profile_id, &model_id, effect, &actor_id), + )?; + connection.execute( + "UPDATE policy_profiles SET version = version + 1, updated_by = ?2, \ + updated_at = CURRENT_TIMESTAMP WHERE id = ?1", + (&profile_id, &actor_id), + )?; + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'set_profile_rule', 'profile', ?2)", + (&actor_id, &profile_id), + )?; + Ok(()) + }) + .await?; + Ok(()) + } + + pub async fn set_profile_preference( + &self, + profile_id: &str, + model_id: Option<&str>, + actor_id: &str, + ) -> Result<(), ModelPolicyError> { + let profile_id = profile_id.to_owned(); + let model_id = model_id + .map(validate_model_id) + .transpose()? + .map(str::to_owned); + let actor_id = actor_id.to_owned(); + let mutation_fence = self.mutation_fence.clone(); + self.database + .with_transaction(move |connection| { + validate_mutation_actor(connection, &actor_id, mutation_fence.as_ref())?; + let default_effect = connection.query_row( + "SELECT default_effect FROM policy_profiles WHERE id = ?1", + [&profile_id], + |row| row.get::<_, String>(0), + )?; + if let Some(model_id) = model_id.as_deref() { + let rule = connection + .query_row( + "SELECT effect FROM profile_model_rules \ + WHERE profile_id = ?1 AND model_id = ?2", + (&profile_id, model_id), + |row| row.get::<_, String>(0), + ) + .optional()?; + let effect = rule + .as_deref() + .map(Effect::parse) + .unwrap_or_else(|| Effect::parse(&default_effect)); + let instance_denied = connection.query_row( + "SELECT EXISTS(SELECT 1 FROM instance_model_denials WHERE model_id = ?1)", + [model_id], + |row| row.get::<_, bool>(0), + )?; + if effect == Effect::Deny || instance_denied { + return Err(tokio_rusqlite::rusqlite::Error::InvalidParameterName( + "preferred model is denied by policy".to_owned(), + )); + } + } + connection.execute( + "UPDATE policy_profiles SET preferred_model_id = ?2, \ + version = version + 1, updated_by = ?3, updated_at = CURRENT_TIMESTAMP \ + WHERE id = ?1", + (&profile_id, model_id, &actor_id), + )?; + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'set_profile_preference', 'profile', ?2)", + (&actor_id, &profile_id), + )?; + Ok(()) + }) + .await + .map_err(|error| { + let preferred_denied = match &error { + crate::db::DatabaseError::Transaction { source } + | crate::db::DatabaseError::Sqlite { source, .. } => matches!( + source.as_ref(), + tokio_rusqlite::rusqlite::Error::InvalidParameterName(_) + ), + _ => false, + }; + if preferred_denied { + ModelPolicyError::PreferredModelDenied + } else { + ModelPolicyError::Database(error) + } + })?; + Ok(()) + } + + pub async fn assign_profile( + &self, + user_id: &str, + profile_id: &str, + actor_id: &str, + ) -> Result<(), ModelPolicyError> { + let user_id = user_id.to_owned(); + let profile_id = profile_id.to_owned(); + let actor_id = actor_id.to_owned(); + let mutation_fence = self.mutation_fence.clone(); + self.database + .with_transaction(move |connection| { + validate_mutation_actor(connection, &actor_id, mutation_fence.as_ref())?; + connection.execute( + "INSERT INTO user_profile_assignments(user_id, profile_id, assigned_by) \ + VALUES (?1, ?2, ?3) \ + ON CONFLICT(user_id) DO UPDATE SET \ + profile_id = excluded.profile_id, \ + version = user_profile_assignments.version + 1, \ + assigned_by = excluded.assigned_by, updated_at = CURRENT_TIMESTAMP", + (&user_id, &profile_id, &actor_id), + )?; + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'assign_profile', 'user', ?2)", + (&actor_id, &user_id), + )?; + Ok(()) + }) + .await?; + Ok(()) + } + + pub async fn set_user_override( + &self, + user_id: &str, + model_id: &str, + effect: &str, + actor_id: &str, + ) -> Result<(), ModelPolicyError> { + let user_id = user_id.to_owned(); + let model_id = validate_model_id(model_id)?.to_owned(); + let effect = validate_effect(effect)?; + let actor_id = actor_id.to_owned(); + let mutation_fence = self.mutation_fence.clone(); + self.database + .with_transaction(move |connection| { + validate_mutation_actor(connection, &actor_id, mutation_fence.as_ref())?; + connection.execute( + "INSERT INTO user_policy_overrides(user_id, model_id, effect, updated_by) \ + VALUES (?1, ?2, ?3, ?4) \ + ON CONFLICT(user_id, model_id) DO UPDATE SET \ + effect = excluded.effect, updated_by = excluded.updated_by, \ + updated_at = CURRENT_TIMESTAMP", + (&user_id, &model_id, effect, &actor_id), + )?; + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'set_user_override', 'user', ?2)", + (&actor_id, &user_id), + )?; + Ok(()) + }) + .await?; + Ok(()) + } + + pub async fn remove_user_override( + &self, + user_id: &str, + model_id: &str, + actor_id: &str, + ) -> Result { + let user_id = user_id.to_owned(); + let model_id = validate_model_id(model_id)?.to_owned(); + let actor_id = actor_id.to_owned(); + let mutation_fence = self.mutation_fence.clone(); + self.database + .with_transaction(move |connection| { + validate_mutation_actor(connection, &actor_id, mutation_fence.as_ref())?; + let removed = connection.execute( + "DELETE FROM user_policy_overrides WHERE user_id = ?1 AND model_id = ?2", + (&user_id, &model_id), + )?; + if removed == 1 { + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, 'remove_user_override', 'user', ?2)", + (&actor_id, &user_id), + )?; + } + Ok(removed == 1) + }) + .await + .map_err(ModelPolicyError::Database) + } + + pub async fn set_instance_denial( + &self, + model_id: &str, + actor_id: &str, + denied: bool, + ) -> Result<(), ModelPolicyError> { + let model_id = validate_model_id(model_id)?.to_owned(); + let actor_id = actor_id.to_owned(); + let mutation_fence = self.mutation_fence.clone(); + self.database + .with_transaction(move |connection| { + validate_mutation_actor(connection, &actor_id, mutation_fence.as_ref())?; + if denied { + connection.execute( + "INSERT INTO instance_model_denials(model_id, created_by) \ + VALUES (?1, ?2) ON CONFLICT(model_id) DO NOTHING", + (&model_id, &actor_id), + )?; + } else { + connection.execute( + "DELETE FROM instance_model_denials WHERE model_id = ?1", + [&model_id], + )?; + } + connection.execute( + "INSERT INTO user_admin_audit(actor_id, action, target_kind, target_id) \ + VALUES (?1, ?2, 'model', ?3)", + ( + &actor_id, + if denied { "deny_model" } else { "allow_model" }, + &model_id, + ), + )?; + Ok(()) + }) + .await?; + Ok(()) + } + + pub async fn snapshot( + &self, + user_id: &str, + ) -> Result, ModelPolicyError> { + let key = user_id.to_owned(); + let revision = self + .database + .call(|connection| { + connection.query_row( + "SELECT revision FROM policy_state WHERE singleton = 1", + [], + |row| row.get::<_, i64>(0), + ) + }) + .await?; + if let Ok(cache) = self.cache.lock() + && let Some(snapshot) = cache.get(&key) + && snapshot.revision == revision + { + return Ok(snapshot.clone()); + } + + let user_id_for_db = key.clone(); + let snapshot = self + .database + .with_transaction(move |connection| { + let revision = connection.query_row( + "SELECT revision FROM policy_state WHERE singleton = 1", + [], + |row| row.get::<_, i64>(0), + )?; + let user = connection + .query_row( + "SELECT role FROM users WHERE id = ?1 AND state = 'active'", + [&user_id_for_db], + |row| row.get::<_, String>(0), + ) + .optional()?; + let Some(role) = user else { + return Ok(None); + }; + let profile = connection + .query_row( + "SELECT p.default_effect, p.preferred_model_id FROM user_profile_assignments a \ + JOIN policy_profiles p ON p.id = a.profile_id \ + WHERE a.user_id = ?1", + [&user_id_for_db], + |row| Ok((row.get::<_, String>(0)?, row.get::<_, Option>(1)?)), + ) + .optional()?; + let profile_rules = query_rules( + connection, + "SELECT r.model_id, r.effect FROM profile_model_rules r \ + JOIN user_profile_assignments a ON a.profile_id = r.profile_id \ + WHERE a.user_id = ?1", + &user_id_for_db, + )?; + let user_overrides = query_rules( + connection, + "SELECT model_id, effect FROM user_policy_overrides WHERE user_id = ?1", + &user_id_for_db, + )?; + let mut statement = + connection.prepare("SELECT model_id FROM instance_model_denials")?; + let instance_denials = statement + .query_map([], |row| row.get::<_, String>(0))? + .collect::, _>>()?; + Ok(Some(ModelPolicySnapshot { + revision, + role, + default_effect: profile.as_ref().map(|(effect, _)| Effect::parse(effect)), + preferred_model_id: profile.and_then(|(_, preferred)| preferred), + profile_rules, + user_overrides, + instance_denials, + })) + }) + .await? + .ok_or(ModelPolicyError::UserUnavailable)?; + let snapshot = Arc::new(snapshot); + if let Ok(mut cache) = self.cache.lock() { + if cache.len() >= POLICY_CACHE_CAPACITY + && !cache.contains_key(&key) + && let Some(eviction) = cache.keys().next().cloned() + { + cache.remove(&eviction); + } + cache.insert(key, snapshot.clone()); + } + Ok(snapshot) + } +} + +fn validate_effect(effect: &str) -> Result<&'static str, ModelPolicyError> { + match effect { + "allow" => Ok("allow"), + "deny" => Ok("deny"), + _ => Err(ModelPolicyError::InvalidPolicyInput), + } +} + +fn validate_model_id(model_id: &str) -> Result<&str, ModelPolicyError> { + if model_id.is_empty() + || model_id.len() > 512 + || model_id.trim() != model_id + || model_id.chars().any(char::is_whitespace) + || model_id.contains('*') + { + return Err(ModelPolicyError::InvalidPolicyInput); + } + Ok(model_id) +} + +fn query_rules( + connection: &tokio_rusqlite::rusqlite::Connection, + sql: &str, + user_id: &str, +) -> Result, tokio_rusqlite::rusqlite::Error> { + let mut statement = connection.prepare(sql)?; + statement + .query_map([user_id], |row| { + Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)) + })? + .map(|row| { + let (model_id, effect) = row?; + Ok((model_id, Effect::parse(&effect))) + }) + .collect() +} + +use tokio_rusqlite::rusqlite::OptionalExtension; diff --git a/rust/src/request/admission.rs b/rust/src/request/admission.rs index ddd6d3ba0..a537b973d 100644 --- a/rust/src/request/admission.rs +++ b/rust/src/request/admission.rs @@ -108,6 +108,49 @@ impl std::fmt::Debug for NativeRequestPreservation { } } +#[derive(Clone, Copy, PartialEq, Eq, Default, Debug)] +pub struct BudgetAdmissionFacts { + /// Byte-length upper bound on input tokens: every token encodes at least + /// one ingress byte, so this is conservative for any byte-level + /// tokenizer, never an average. Hard-mode admission builds its + /// worst-case bound from this plus the caller's explicit output cap. + pub input_worst_case_tokens: i64, + /// The caller's explicit compatible output cap for the request surface + /// (`max_tokens`/`max_completion_tokens` on chat, `max_output_tokens` + /// on Responses/compact, `max_tokens` on Anthropic Messages). + /// `None` means unbounded, absent, or unusable — hard token rules must + /// fail closed on `None`. Admission never mutates caller fields to + /// manufacture a bound. + pub requested_output_tokens: Option, +} + +/// Derive the budget proof facts for one parsed request body. Pure and +/// total: an unusable output field degrades to `None` (hard-mode denial), +/// never to an invented bound. +pub fn budget_admission_facts( + raw_body_len: usize, + value: &Value, + surface: ClientSurface, + reservation_tokens: u64, + context_tokens: u64, +) -> BudgetAdmissionFacts { + let byte_bound = raw_body_len.min(i64::MAX as usize) as i64; + let input_worst_case_tokens = byte_bound + .max(reservation_tokens.min(i64::MAX as u64) as i64) + .max(context_tokens.min(i64::MAX as u64) as i64) + .max(0); + let requested_output_tokens = + super::limits::requested_output_tokens(value, surface.protocol(), surface.as_str()) + .ok() + .flatten() + .and_then(|bound| i64::try_from(bound).ok()) + .filter(|bound| *bound > 0); + BudgetAdmissionFacts { + input_worst_case_tokens, + requested_output_tokens, + } +} + #[derive(Clone, PartialEq)] pub struct AdmittedRequest { pub canonical: CanonicalRequest, @@ -115,6 +158,7 @@ pub struct AdmittedRequest { pub raw_body_bytes: usize, pub reservation_tokens: u64, pub context_tokens: u64, + pub budget_facts: BudgetAdmissionFacts, } impl std::fmt::Debug for AdmittedRequest { @@ -188,6 +232,7 @@ pub struct CompactAdmittedRequest { pub raw_body_bytes: usize, pub reservation_tokens: u64, pub context_tokens: u64, + pub budget_facts: BudgetAdmissionFacts, } impl std::fmt::Debug for CompactAdmittedRequest { @@ -204,7 +249,12 @@ impl std::fmt::Debug for CompactAdmittedRequest { impl CompactAdmittedRequest { pub fn routing_facts(&self, inputs: &StaticRoutingFacts) -> RoutingRequestFacts { - routing_request_facts_from_parts(&self.canonical, self.reservation_tokens, inputs) + routing_request_facts_from_parts( + &self.canonical, + self.reservation_tokens, + inputs, + self.budget_facts, + ) } pub fn affinity_identity(&self, explicit_session: Option<&str>) -> AffinityIdentityInput { @@ -276,6 +326,15 @@ pub(crate) fn admit_parsed_request( } else { None }; + // The output bound is read from the parsed object before `value` moves + // into native preservation (Responses) or is dropped (other surfaces). + let budget_facts = budget_admission_facts( + raw_body.len(), + &value, + options.client_surface, + reservation_tokens, + context_tokens, + ); let native_preservation = native_summary.map(|summary| NativeRequestPreservation { source_surface: options.client_surface, parsed: value, @@ -287,6 +346,7 @@ pub(crate) fn admit_parsed_request( raw_body_bytes: raw_body.len(), reservation_tokens, context_tokens, + budget_facts, }) } @@ -396,6 +456,13 @@ pub(crate) fn admit_compact_parsed_request( let context_tokens = estimate_context_input_tokens(&raw_body, &value, options.extra_context_tokens); let summary = native_feature_summary(object); + let budget_facts = budget_admission_facts( + raw_body.len(), + &value, + ClientSurface::Responses, + reservation_tokens, + context_tokens, + ); Ok(CompactAdmittedRequest { canonical, native_preservation: NativeRequestPreservation { @@ -406,6 +473,7 @@ pub(crate) fn admit_compact_parsed_request( raw_body_bytes: raw_body.len(), reservation_tokens, context_tokens, + budget_facts, }) } @@ -413,13 +481,19 @@ pub fn routing_request_facts( admitted: &AdmittedRequest, inputs: &StaticRoutingFacts, ) -> RoutingRequestFacts { - routing_request_facts_from_parts(&admitted.canonical, admitted.reservation_tokens, inputs) + routing_request_facts_from_parts( + &admitted.canonical, + admitted.reservation_tokens, + inputs, + admitted.budget_facts, + ) } fn routing_request_facts_from_parts( canonical: &CanonicalRequest, reservation_tokens: u64, inputs: &StaticRoutingFacts, + budget_facts: BudgetAdmissionFacts, ) -> RoutingRequestFacts { let mut facts = RoutingRequestFacts::from_model_id(&canonical.model, &inputs.known_provider_ids); @@ -429,6 +503,8 @@ fn routing_request_facts_from_parts( facts.request_surface = client_surface.as_str().into(); facts.transcode_protocols = inputs.transcode_protocols.clone(); facts.projected_tokens = reservation_tokens.min(i64::MAX as u64) as i64; + facts.budget_input_worst_case_tokens = budget_facts.input_worst_case_tokens; + facts.budget_requested_output_tokens = budget_facts.requested_output_tokens; facts.catalog_stale_after_s = inputs.catalog_stale_after_s; facts.thinking = thinking_requirement_from_intent(&canonical.reasoning); facts.capability_policy = inputs.capability_policy.clone(); @@ -752,6 +828,7 @@ mod tests { raw_body_bytes: compact.raw_body_bytes, reservation_tokens: compact.reservation_tokens, context_tokens: compact.context_tokens, + budget_facts: compact.budget_facts, }; let inputs = StaticRoutingFacts::default(); assert_eq!( diff --git a/rust/src/request/mod.rs b/rust/src/request/mod.rs index bf19ac8dc..2c78a1243 100644 --- a/rust/src/request/mod.rs +++ b/rust/src/request/mod.rs @@ -9,10 +9,11 @@ pub mod limits; pub(crate) mod resource_budget; pub use admission::{ - AdmissionError, AdmissionOptions, AdmittedRequest, CompactAdmittedRequest, - NativeFeatureSummary, NativeRequestPreservation, StaticRoutingFacts, admit_compact_request, - admit_request, affinity_identity_input, has_compaction_trigger, routing_request_facts, - thinking_requirement_from_intent, validate_responses_stateless_policy, + AdmissionError, AdmissionOptions, AdmittedRequest, BudgetAdmissionFacts, + CompactAdmittedRequest, NativeFeatureSummary, NativeRequestPreservation, StaticRoutingFacts, + admit_compact_request, admit_request, affinity_identity_input, budget_admission_facts, + has_compaction_trigger, routing_request_facts, thinking_requirement_from_intent, + validate_responses_stateless_policy, }; pub(crate) use admission::{ diff --git a/rust/src/routing/claim.rs b/rust/src/routing/claim.rs index d8a172834..8c623a818 100644 --- a/rust/src/routing/claim.rs +++ b/rust/src/routing/claim.rs @@ -63,6 +63,10 @@ pub struct SelectionSnapshot { pub provider_id: Option, pub protocol: Option, pub request_surface: String, + /// Admission-carried worst-case input bound for hard budget enforcement. + pub budget_input_worst_case_tokens: i64, + /// Admission-carried caller output cap, if any. + pub budget_requested_output_tokens: Option, pub candidates: Vec, pub exclusions: Vec, pub eligible_candidate_count: usize, @@ -96,6 +100,8 @@ impl SelectionSnapshot { provider_id: facts.provider_id.clone(), protocol: facts.requested_protocol.clone(), request_surface: facts.request_surface.clone(), + budget_input_worst_case_tokens: facts.budget_input_worst_case_tokens, + budget_requested_output_tokens: facts.budget_requested_output_tokens, eligible_candidate_count: candidates.len(), top_account_name: top.map(|candidate| candidate.account_name.clone()), top_score: top.map(|candidate| candidate.score.clone()), @@ -122,6 +128,8 @@ impl SelectionSnapshot { provider_id: plan.requested_provider_id, protocol: plan.requested_protocol, request_surface: plan.request_surface, + budget_input_worst_case_tokens: 0, + budget_requested_output_tokens: None, eligible_candidate_count: plan.candidates.len(), top_account_name: top.map(|candidate| candidate.account_name.clone()), top_score: top.map(|candidate| candidate.score.clone()), @@ -257,6 +265,12 @@ impl SelectionClaim { pub fn projected_cost_microdollars(&self) -> i64 { self.projected_cost_microdollars } + pub fn budget_input_worst_case_tokens(&self) -> i64 { + self.snapshot.budget_input_worst_case_tokens + } + pub fn budget_requested_output_tokens(&self) -> Option { + self.snapshot.budget_requested_output_tokens + } pub fn owns_probe(&self) -> bool { self.owns_probe } diff --git a/rust/src/routing/eligibility.rs b/rust/src/routing/eligibility.rs index 9132faea5..dda29a70d 100644 --- a/rust/src/routing/eligibility.rs +++ b/rust/src/routing/eligibility.rs @@ -21,6 +21,13 @@ pub struct RoutingRequestFacts { pub request_surface: String, pub transcode_protocols: Vec, pub projected_tokens: i64, + /// Byte-length upper bound on input tokens carried from admission. Hard + /// user-budget rules build their worst-case hold from this plus + /// `budget_requested_output_tokens`; estimated rules ignore both. + pub budget_input_worst_case_tokens: i64, + /// Caller-supplied compatible output cap, if any. `None` means + /// unbounded or unusable and fails closed under hard token rules. + pub budget_requested_output_tokens: Option, pub catalog_stale_after_s: Option, pub thinking: Option, pub capability_policy: BTreeMap, @@ -51,6 +58,8 @@ impl RoutingRequestFacts { request_surface: "chat_completions".into(), transcode_protocols: Vec::new(), projected_tokens: 0, + budget_input_worst_case_tokens: 0, + budget_requested_output_tokens: None, catalog_stale_after_s: None, thinking: None, capability_policy: BTreeMap::new(), diff --git a/rust/src/runtime.rs b/rust/src/runtime.rs index d0fdd766e..35ab27fe2 100644 --- a/rust/src/runtime.rs +++ b/rust/src/runtime.rs @@ -1,6 +1,6 @@ use std::{ env, fs, - io::{self, Write}, + io::{self, BufRead, IsTerminal, Read, Write}, path::{Path, PathBuf}, process::Command as OsCommand, time::Duration, @@ -87,6 +87,7 @@ pub async fn run(cli: Cli) -> Result<(), BootstrapError> { .await? } Some(Command::Onboard(args)) => onboard(&config_path, args).await?, + Some(Command::Users(command)) => users(&config_path, command).await?, Some(Command::Configsetup(command)) => configsetup(&config_path, command).await?, Some(Command::Configremote(args)) => configremote(&config_path, args).await?, Some(Command::Migrate) => migrate(&config_path).await?, @@ -651,14 +652,337 @@ async fn open_operational_database( config: &config::Config, require_existing: bool, ) -> Result { + let path = config::Config::runtime_path(&config.database.path); + if path.exists() + && std::fs::symlink_metadata(&path) + .map(|metadata| metadata.file_type().is_symlink()) + .unwrap_or(true) + { + return Err(command_error( + EXIT_VALIDATION, + format!( + "refusing symbolic-link or ambiguous database path: {}", + path.display() + ), + )); + } let database = open_maintenance_database(config, require_existing).await?; - if let Err(error) = crate::db::MigrationRunner::new(&database).run().await { + let migration_state = match crate::db::MigrationRunner::new(&database).run().await { + Ok(state) => state, + Err(error) => { + let _ = database.close().await; + return Err(command_error(EXIT_VALIDATION, error.to_string())); + } + }; + if migration_state.applied_this_run.len() == crate::db::MigrationRunner::migrations().len() + && let Err(error) = crate::identity::mark_fresh_uninitialized(&database).await + { let _ = database.close().await; return Err(command_error(EXIT_VALIDATION, error.to_string())); } Ok(database) } +async fn users(path: &Path, command: crate::cli::UsersCommand) -> Result<(), BootstrapError> { + let config = config::Config::from_toml(path)?; + if matches!(&command, crate::cli::UsersCommand::EnableManaged) + && (config.resolved_server_api_key().is_some() || config.dashboard.public) + { + return Err(command_error( + EXIT_VALIDATION, + "retire server.api_key/api_key_env and set dashboard.public = false before enabling managed access", + )); + } + let database = open_operational_database(&config, false).await?; + match command { + crate::cli::UsersCommand::EnableManaged => { + crate::identity::enable_managed(&database) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!("Managed browser and personal-token access is enabled."); + } + crate::cli::UsersCommand::List => { + let users = crate::identity::list_users(&database) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + if users.is_empty() { + println!("No local users. Run `eggpool users bootstrap` on a new installation."); + } else { + for user in users { + println!( + "{}\t{}\t{}\tmust_change_password={}", + user.username, user.role, user.state, user.must_change_password + ); + } + } + } + crate::cli::UsersCommand::Bootstrap(args) => { + let password = read_user_password(args.password_stdin, true)?; + let (user, credential) = + crate::identity::bootstrap_admin(&database, &args.username, &password) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!("Administrator {} created.", user.username); + print_one_time_secret(&credential.token, args.show_secret); + } + crate::cli::UsersCommand::ResetPassword(args) => { + let password = read_user_password(args.password_stdin, true)?; + crate::identity::reset_password(&database, &args.username, &password) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!( + "Password reset for {}; existing API credentials were revoked.", + args.username + ); + } + crate::cli::UsersCommand::Token(args) => { + let credential = + crate::identity::issue_api_token(&database, &args.username, &args.label) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!("Personal API credential for {} created.", args.username); + print_one_time_secret(&credential.token, args.show_secret); + } + crate::cli::UsersCommand::Revoke(args) => { + let count = crate::identity::revoke_api_tokens(&database, &args.username) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!("Revoked {count} API credential(s) for {}.", args.username); + } + crate::cli::UsersCommand::Policy(command) => { + use crate::cli::UsersPolicyCommand as Policy; + match command { + Policy::Create(args) => { + let actor = policy_actor_id(&database, &args.actor).await?; + crate::model_policy::ModelPolicyService::new(database.clone()) + .create_profile(&args.id, &args.name, &args.default_effect, &actor) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!("Model policy profile {} created.", args.id); + } + Policy::Rule(args) => { + let actor = policy_actor_id(&database, &args.actor).await?; + crate::model_policy::ModelPolicyService::new(database.clone()) + .set_profile_rule(&args.profile_id, &args.model_id, &args.effect, &actor) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!("Model rule for {} updated.", args.profile_id); + } + Policy::Assign(args) => { + let actor = policy_actor_id(&database, &args.actor).await?; + let users = crate::identity::list_users(&database) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + let user = users + .iter() + .find(|user| user.username.eq_ignore_ascii_case(&args.username)) + .ok_or_else(|| command_error(EXIT_VALIDATION, "user was not found"))?; + crate::model_policy::ModelPolicyService::new(database.clone()) + .assign_profile(&user.id, &args.profile_id, &actor) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!("Model policy profile assigned to {}.", user.username); + } + Policy::Prefer(args) => { + let actor = policy_actor_id(&database, &args.actor).await?; + crate::model_policy::ModelPolicyService::new(database.clone()) + .set_profile_preference(&args.profile_id, args.model_id.as_deref(), &actor) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!("Preferred model updated for {}.", args.profile_id); + } + } + } + crate::cli::UsersCommand::Budget(command) => { + use crate::cli::UsersBudgetCommand as Budget; + match command { + Budget::Set(args) => { + let actor = policy_actor_id(&database, &args.actor).await?; + let users = crate::identity::list_users(&database) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + let user = users + .iter() + .find(|user| user.username.eq_ignore_ascii_case(&args.username)) + .ok_or_else(|| command_error(EXIT_VALIDATION, "user was not found"))?; + let window = match args.window.as_str() { + "day" => crate::user_budget::BudgetWindow::Day, + "month" => crate::user_budget::BudgetWindow::Month, + _ => return Err(command_error(EXIT_VALIDATION, "invalid budget window")), + }; + let enforcement = match args.enforcement.as_str() { + "hard" => crate::user_budget::BudgetEnforcement::Hard, + "estimated" => crate::user_budget::BudgetEnforcement::Estimated, + _ => { + return Err(command_error( + EXIT_VALIDATION, + "invalid budget enforcement", + )); + } + }; + crate::user_budget::UserBudgetService::new(database.clone()) + .set_rule( + &actor, + &user.id, + args.model_id.as_deref(), + window, + crate::user_budget::BudgetLimits { + requests: args.max_requests, + tokens: args.max_tokens, + microdollars: args.max_microdollars, + concurrent: args.max_concurrent, + enforcement, + }, + ) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!( + "Budget rule for {} saved ({}).", + user.username, + enforcement.as_str() + ); + } + Budget::Remove(args) => { + let actor = policy_actor_id(&database, &args.actor).await?; + let users = crate::identity::list_users(&database) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + let user = users + .iter() + .find(|user| user.username.eq_ignore_ascii_case(&args.username)) + .ok_or_else(|| command_error(EXIT_VALIDATION, "user was not found"))?; + let window = match args.window.as_str() { + "day" => crate::user_budget::BudgetWindow::Day, + "month" => crate::user_budget::BudgetWindow::Month, + _ => return Err(command_error(EXIT_VALIDATION, "invalid budget window")), + }; + let removed = crate::user_budget::UserBudgetService::new(database.clone()) + .remove_rule(&actor, &user.id, args.model_id.as_deref(), window) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!( + "Budget rule for {} {}.", + user.username, + if removed { "removed" } else { "was absent" } + ); + } + } + } + crate::cli::UsersCommand::GlobalBudget(command) => { + use crate::cli::UsersGlobalBudgetCommand as GlobalBudget; + let window_of = |window: &str| match window { + "day" => Ok(crate::user_budget::BudgetWindow::Day), + "month" => Ok(crate::user_budget::BudgetWindow::Month), + _ => Err(command_error(EXIT_VALIDATION, "invalid budget window")), + }; + let enforcement_of = |enforcement: &str| match enforcement { + "hard" => Ok(crate::user_budget::BudgetEnforcement::Hard), + "estimated" => Ok(crate::user_budget::BudgetEnforcement::Estimated), + _ => Err(command_error(EXIT_VALIDATION, "invalid budget enforcement")), + }; + match command { + GlobalBudget::Set(args) => { + let actor = policy_actor_id(&database, &args.actor).await?; + let window = window_of(args.window.as_str())?; + let enforcement = enforcement_of(args.enforcement.as_str())?; + crate::instance_budget::InstanceBudgetService::new(database.clone()) + .set_rule( + &actor, + args.model_id.as_deref(), + window, + crate::user_budget::BudgetLimits { + requests: args.max_requests, + tokens: args.max_tokens, + microdollars: args.max_microdollars, + concurrent: args.max_concurrent, + enforcement, + }, + ) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!("Instance budget ceiling saved ({}).", enforcement.as_str()); + } + GlobalBudget::Remove(args) => { + let actor = policy_actor_id(&database, &args.actor).await?; + let window = window_of(args.window.as_str())?; + let removed = + crate::instance_budget::InstanceBudgetService::new(database.clone()) + .remove_rule(&actor, args.model_id.as_deref(), window) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!( + "Instance budget ceiling {}.", + if removed { "removed" } else { "was absent" } + ); + } + } + } + } + database + .close() + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + Ok(()) +} + +async fn policy_actor_id( + database: &crate::db::Database, + username: &str, +) -> Result { + let users = crate::identity::list_users(database) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + users + .into_iter() + .find(|user| { + user.username.eq_ignore_ascii_case(username) + && user.role == "admin" + && user.state == "active" + }) + .map(|user| user.id) + .ok_or_else(|| command_error(EXIT_VALIDATION, "active administrator was not found")) +} + +fn read_user_password(from_stdin: bool, confirm: bool) -> Result { + let password = if from_stdin { + let mut value = String::new(); + io::stdin() + .lock() + .take(1026) + .read_line(&mut value) + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + value.trim_end_matches(['\r', '\n']).to_owned() + } else { + if !io::stdin().is_terminal() { + return Err(command_error( + EXIT_VALIDATION, + "password input requires a TTY or --password-stdin", + )); + } + let first = rpassword::prompt_password("Administrator password: ") + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + if confirm { + let second = rpassword::prompt_password("Confirm password: ") + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + if first != second { + return Err(command_error(EXIT_VALIDATION, "passwords do not match")); + } + } + first + }; + Ok(password) +} + +fn print_one_time_secret(secret: &str, explicitly_allowed: bool) { + if io::stdout().is_terminal() || explicitly_allowed { + println!("Save this personal API credential; it is shown once:\n{secret}"); + } else { + println!( + "Credential plaintext withheld because stdout is not a TTY. Reissue with `eggpool users token USER --show-secret` on a secure console." + ); + } +} + async fn accounts(path: &Path, command: crate::cli::AccountsCommand) -> Result<(), BootstrapError> { let config = config::Config::from_toml(path)?; match command { @@ -1137,6 +1461,13 @@ async fn migrate(path: &Path) -> Result<(), BootstrapError> { let config = config::Config::from_toml(path)?; let database = open_maintenance_database(&config, false).await?; let result = crate::db::MigrationRunner::new(&database).run().await; + if result.as_ref().is_ok_and(|state| { + state.applied_this_run.len() == crate::db::MigrationRunner::migrations().len() + }) && let Err(error) = crate::identity::mark_fresh_uninitialized(&database).await + { + let _ = database.close().await; + return Err(command_error(EXIT_VALIDATION, error.to_string())); + } let close = database.close().await; match (result, close) { (Ok(state), Ok(())) => { @@ -2084,6 +2415,30 @@ async fn onboard(path: &Path, args: crate::cli::OnboardArgs) -> Result<(), Boots println!(" Generated server API key"); } } + let preflight_config = config::Config::from_toml(path).map_err(BootstrapError::from)?; + let preflight_database_path = config::Config::runtime_path(&preflight_config.database.path); + let bootstrap_admin = if !preflight_database_path.exists() { + true + } else { + let database = open_operational_database(&preflight_config, true).await?; + let access_state = crate::identity::access_state(&database) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + let users = crate::identity::list_users(&database) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + database + .close() + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + access_state == "fresh_uninitialized" && users.is_empty() + }; + let admin_password = if bootstrap_admin { + println!("Create the local administrator for this new installation."); + Some(read_user_password(args.password_stdin, true)?) + } else { + None + }; // Newly initialized configs already carry the canonical `0.0.0.0` LAN // bind default; existing configs keep their explicit operator host. // Onboarding never rewrites the bind address. @@ -2111,10 +2466,26 @@ async fn onboard(path: &Path, args: crate::cli::OnboardArgs) -> Result<(), Boots break; } } - config::Config::from_toml(path) - .map_err(BootstrapError::from)? + let setup_config = config::Config::from_toml(path).map_err(BootstrapError::from)?; + setup_config .validate_account_credentials() .map_err(BootstrapError::from)?; + if bootstrap_admin { + let password = admin_password.ok_or_else(|| { + command_error(EXIT_VALIDATION, "administrator password was not collected") + })?; + let database = open_operational_database(&setup_config, false).await?; + let (user, credential) = + crate::identity::bootstrap_admin(&database, &args.admin_username, &password) + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + database + .close() + .await + .map_err(|error| command_error(EXIT_VALIDATION, error.to_string()))?; + println!("Administrator {} created.", user.username); + print_one_time_secret(&credential.token, args.show_secret); + } println!("\nConnected {connected} provider(s).\n"); let paths = RuntimePaths::resolve(); if paths.pid_file.exists() diff --git a/rust/src/runtime_lifecycle/recovery.rs b/rust/src/runtime_lifecycle/recovery.rs index f10f4ad0b..eb07fbe69 100644 --- a/rust/src/runtime_lifecycle/recovery.rs +++ b/rust/src/runtime_lifecycle/recovery.rs @@ -11,6 +11,8 @@ pub struct StartupRecoveryReport { pub requests_interrupted: usize, pub reservations_released: usize, pub attempts_terminalized: usize, + pub user_budget_holds_released: usize, + pub instance_budget_holds_released: usize, pub last_classification: crate::coordinator::ReconciliationClassification, pub converged: bool, } @@ -51,6 +53,8 @@ pub(crate) async fn reconcile( requests_interrupted: 0, reservations_released: 0, attempts_terminalized: 0, + user_budget_holds_released: 0, + instance_budget_holds_released: 0, last_classification: Default::default(), converged: false, }; @@ -66,6 +70,12 @@ pub(crate) async fn reconcile( aggregate.attempts_terminalized = aggregate .attempts_terminalized .saturating_add(report.attempts_terminalized); + aggregate.user_budget_holds_released = aggregate + .user_budget_holds_released + .saturating_add(report.user_budget_holds_released); + aggregate.instance_budget_holds_released = aggregate + .instance_budget_holds_released + .saturating_add(report.instance_budget_holds_released); aggregate.last_classification = report.classification; if !report.truncated && report.converged { aggregate.converged = true; diff --git a/rust/src/server/auth.rs b/rust/src/server/auth.rs new file mode 100644 index 000000000..38e8c480d --- /dev/null +++ b/rust/src/server/auth.rs @@ -0,0 +1,629 @@ +//! Browser login, sessions, CSRF forms, and forwarded-HTTPS validation. + +use super::*; +use axum::{ + Extension, Form, + http::{HeaderMap, HeaderValue, header}, + response::{Html, Redirect}, +}; +use serde::Deserialize; +use std::net::IpAddr; + +#[derive(Deserialize)] +pub(super) struct LoginForm { + username: String, + password: String, +} + +#[derive(Deserialize)] +pub(super) struct CsrfForm { + csrf: String, +} + +#[derive(Deserialize)] +pub(super) struct PasswordChangeForm { + current_password: String, + new_password: String, + csrf: String, +} + +pub(super) async fn login_page() -> Html { + Html(login_html(None)) +} + +pub(super) async fn login_submit( + State(state): State, + headers: HeaderMap, + peer: Option>, + Form(form): Form, +) -> Response { + let connection = peer.as_ref().map(|extension| &extension.0.0); + let secure = request_is_secure(connection, &headers, &state.server); + if !web_auth_transport_allowed(connection, &headers, &state.server) { + return ( + StatusCode::FORBIDDEN, + "Browser sign-in requires HTTPS or loopback.", + ) + .into_response(); + } + if !same_origin(&headers, secure) { + return ( + StatusCode::FORBIDDEN, + "Cross-origin sign-in is not allowed.", + ) + .into_response(); + } + let peer_ip = throttle_peer_ip(connection, &headers, &state.server); + if !crate::web_session::login_allowed(&form.username, peer_ip) { + return ( + StatusCode::TOO_MANY_REQUESTS, + Html(login_html(Some("Sign-in failed. Try again later."))), + ) + .into_response(); + } + let access_state = match crate::identity::access_state(&state.database).await { + Ok(access_state) => access_state, + Err(_) => return unavailable(), + }; + if !matches!(access_state.as_str(), "admin_initialized" | "managed") { + return (StatusCode::NOT_FOUND, "Browser sign-in is not enabled.").into_response(); + } + + let user = crate::identity::verify_password(&state.database, &form.username, &form.password) + .await + .ok() + .flatten(); + let Some(user) = user else { + crate::web_session::record_login_failure(&form.username, peer_ip); + return ( + StatusCode::UNAUTHORIZED, + Html(login_html(Some("Invalid username or password."))), + ) + .into_response(); + }; + let session = match crate::web_session::issue(&state.database, &user.id).await { + Ok(session) => session, + Err(_) => return unavailable(), + }; + let mut response = Redirect::to(if user.must_change_password { + "/account?password=required" + } else { + "/account" + }) + .into_response(); + set_cookie( + &mut response, + crate::web_session::session_set_cookie(&session.session_token, secure), + ); + set_cookie( + &mut response, + crate::web_session::csrf_set_cookie(&session.csrf_token, secure), + ); + response +} + +pub(super) async fn account_page( + Extension(principal): Extension, + headers: HeaderMap, +) -> Html { + let csrf = crate::web_session::csrf_cookie_value(&headers) + .filter(|token| crate::web_session::verify_csrf(&principal, token)); + let csrf = csrf.unwrap_or_default(); + let username = html_escape(&principal.username); + let forced = principal.must_change_password; + let notice = if forced { + "

Change your password to continue.

" + } else { + "" + }; + Html(format!( + "Account

Account

Signed in as {username}.

{notice}
" + )) +} + +pub(super) async fn password_change( + State(state): State, + Extension(principal): Extension, + headers: HeaderMap, + peer: Option>, + Form(form): Form, +) -> Response { + if !crate::web_session::verify_csrf(&principal, &form.csrf) + || !crate::web_session::csrf_cookie_value(&headers) + .is_some_and(|cookie| crate::web_session::verify_csrf(&principal, &cookie)) + { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + let authenticated = crate::identity::verify_password( + &state.database, + &principal.username, + &form.current_password, + ) + .await + .ok() + .flatten() + .is_some_and(|user| user.id == principal.user_id); + if !authenticated { + return ( + StatusCode::BAD_REQUEST, + Html(account_error_html("Current password is incorrect.")), + ) + .into_response(); + } + match crate::identity::reset_password_from_browser( + &state.database, + &principal, + &form.new_password, + ) + .await + { + Ok(()) => { + let mut response = Redirect::to("/login").into_response(); + clear_auth_cookies( + &mut response, + request_is_secure( + peer.as_ref().map(|extension| &extension.0.0), + &headers, + &state.server, + ), + ); + response + } + Err(crate::identity::IdentityError::InvalidPassword) => ( + StatusCode::BAD_REQUEST, + Html(account_error_html("New password must be 12 to 1024 bytes.")), + ) + .into_response(), + Err(_) => unavailable(), + } +} + +pub(super) async fn logout_submit( + State(state): State, + Extension(principal): Extension, + headers: HeaderMap, + peer: Option>, + Form(form): Form, +) -> Response { + if !crate::web_session::verify_csrf(&principal, &form.csrf) + || !crate::web_session::csrf_cookie_value(&headers) + .is_some_and(|cookie| crate::web_session::verify_csrf(&principal, &cookie)) + { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if let Some(token) = crate::web_session::session_cookie_value(&headers) + && crate::web_session::revoke(&state.database, &token) + .await + .is_err() + { + return unavailable(); + } + let secure = request_is_secure( + peer.as_ref().map(|extension| &extension.0.0), + &headers, + &state.server, + ); + let mut response = Redirect::to("/login").into_response(); + clear_auth_cookies(&mut response, secure); + response +} + +pub(super) fn request_is_secure( + connection: Option<&eggserve_primitives::connection_info::ConnectionInfo>, + headers: &HeaderMap, + server: &ServerState, +) -> bool { + connection.is_some_and(|connection| { + connection.scheme == eggserve_primitives::connection_info::Scheme::Https + }) || forwarded_https(connection, headers, server) +} + +pub(super) fn web_auth_transport_allowed( + connection: Option<&eggserve_primitives::connection_info::ConnectionInfo>, + headers: &HeaderMap, + server: &ServerState, +) -> bool { + if request_is_secure(connection, headers, server) || server.allow_insecure_web_auth { + return true; + } + connection + .and_then(|connection| connection.remote_addr) + .is_some_and(|address| address.ip().is_loopback()) +} + +fn forwarded_https( + connection: Option<&eggserve_primitives::connection_info::ConnectionInfo>, + headers: &HeaderMap, + server: &ServerState, +) -> bool { + let Some(peer) = connection + .and_then(|connection| connection.remote_addr) + .map(|address| address.ip()) + else { + return false; + }; + if !trusted_proxy(peer, server) { + return false; + } + let mut values = headers.get_all("x-forwarded-proto").iter(); + let Some(value) = values.next().and_then(|value| value.to_str().ok()) else { + return false; + }; + values.next().is_none() && value.trim().eq_ignore_ascii_case("https") +} + +pub(super) fn throttle_peer_ip( + connection: Option<&eggserve_primitives::connection_info::ConnectionInfo>, + headers: &HeaderMap, + server: &ServerState, +) -> Option { + let peer = connection + .and_then(|connection| connection.remote_addr) + .map(|address| address.ip())?; + if !trusted_proxy(peer, server) { + return Some(peer); + } + let mut values = headers.get_all("x-forwarded-for").iter(); + let Some(value) = values.next().and_then(|value| value.to_str().ok()) else { + return Some(peer); + }; + if values.next().is_some() { + return Some(peer); + } + Some(value.trim().parse().unwrap_or(peer)) +} + +fn trusted_proxy(peer: IpAddr, server: &ServerState) -> bool { + server + .trusted_proxy_ips + .iter() + .any(|proxy| proxy.parse::().ok() == Some(peer)) +} + +pub(super) fn same_origin(headers: &HeaderMap, secure: bool) -> bool { + let mut host_values = headers.get_all(header::HOST).iter(); + let Some(host) = host_values.next().and_then(|value| value.to_str().ok()) else { + return false; + }; + if host_values.next().is_some() { + return false; + } + let expected_scheme = if secure { "https" } else { "http" }; + let Ok(expected_authority) = host.parse::() else { + return false; + }; + // A same-origin browser form POST carries `Origin` and `Referer` together, so + // every provenance header that is present has to agree. Requiring exactly one + // of them rejected real browsers while still accepting a bare `Referer`. + let mut seen = false; + for name in [header::ORIGIN, header::REFERER] { + let referer = name == header::REFERER; + let mut values = headers.get_all(&name).iter(); + let Some(first) = values.next() else { continue }; + seen = true; + // A duplicated header is ambiguous provenance; fail closed rather than + // silently preferring one of the values. + if values.next().is_some() { + return false; + } + let Ok(value) = first.to_str() else { + return false; + }; + let allowed = if referer { + referer_authority(value) + } else { + origin_authority(value) + }; + let Some((scheme, authority)) = allowed else { + return false; + }; + if !scheme.eq_ignore_ascii_case(expected_scheme) { + return false; + } + let Ok(provenance_authority) = authority.parse::() else { + return false; + }; + if !expected_authority + .host() + .eq_ignore_ascii_case(provenance_authority.host()) + || effective_port(expected_authority.port_u16(), expected_scheme) + != effective_port(provenance_authority.port_u16(), expected_scheme) + { + return false; + } + } + seen +} + +fn origin_authority(value: &str) -> Option<(&str, &str)> { + let (scheme, remainder) = value.split_once("://")?; + if !matches!(scheme.to_ascii_lowercase().as_str(), "http" | "https") { + return None; + } + let authority = remainder.split(['/', '?', '#']).next()?; + if authority.is_empty() + || authority.contains('@') + || remainder[authority.len()..].contains(['?', '#']) + { + return None; + } + Some((scheme, authority)) +} + +/// Split a `Referer`, which unlike `Origin` may carry an ordinary path, query, +/// or fragment, into the scheme and authority that matter for the same-origin +/// comparison. +fn referer_authority(value: &str) -> Option<(&str, &str)> { + let (scheme, remainder) = value.split_once("://")?; + if !matches!(scheme.to_ascii_lowercase().as_str(), "http" | "https") { + return None; + } + let authority = remainder.split(['/', '?', '#']).next()?; + // Userinfo would let `https://expected.test@attacker.test` read as the + // expected host, so ambiguous provenance is rejected outright. + if authority.is_empty() || authority.contains('@') || remainder.contains("://") { + return None; + } + Some((scheme, authority)) +} + +fn effective_port(port: Option, scheme: &str) -> Option { + port.or(match scheme { + "https" => Some(443), + "http" => Some(80), + _ => None, + }) +} + +fn set_cookie(response: &mut Response, value: String) { + if let Ok(value) = HeaderValue::from_str(&value) { + response.headers_mut().append(header::SET_COOKIE, value); + } +} + +fn clear_auth_cookies(response: &mut Response, secure: bool) { + for value in crate::web_session::clear_cookies(secure) { + set_cookie(response, value); + } +} + +fn login_html(message: Option<&str>) -> String { + let message = message.map_or(String::new(), |message| { + format!("

{message}

") + }); + format!( + "Sign in

Sign in

{message}
" + ) +} + +fn account_error_html(message: &str) -> String { + format!( + "Account

Account

{}

Return to account

", + html_escape(message) + ) +} + +fn html_escape(value: &str) -> String { + value + .replace('&', "&") + .replace('<', "<") + .replace('>', ">") + .replace('"', """) + .replace('\'', "'") +} + +fn unavailable() -> Response { + ( + StatusCode::SERVICE_UNAVAILABLE, + "Authentication service unavailable.", + ) + .into_response() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn same_origin_requires_matching_authority_and_scheme() { + let mut headers = HeaderMap::new(); + headers.insert(header::HOST, HeaderValue::from_static("example.test")); + headers.insert( + header::ORIGIN, + HeaderValue::from_static("https://example.test"), + ); + assert!(same_origin(&headers, true)); + assert!(!same_origin(&headers, false)); + headers.insert( + header::ORIGIN, + HeaderValue::from_static("https://attacker.test"), + ); + assert!(!same_origin(&headers, true)); + } + + #[test] + fn same_origin_accepts_the_header_set_a_browser_form_post_sends() { + let mut headers = HeaderMap::new(); + headers.insert(header::HOST, HeaderValue::from_static("127.0.0.1:11300")); + headers.insert( + header::ORIGIN, + HeaderValue::from_static("http://127.0.0.1:11300"), + ); + headers.insert( + header::REFERER, + HeaderValue::from_static("http://127.0.0.1:11300/login"), + ); + assert!(same_origin(&headers, false)); + assert!(!same_origin(&headers, true)); + // A real browser `Referer` may carry a query or fragment. + headers.insert( + header::REFERER, + HeaderValue::from_static("http://127.0.0.1:11300/login?next=%2Fusers"), + ); + assert!(same_origin(&headers, false)); + } + + #[test] + fn same_origin_rejects_disagreeing_or_duplicated_provenance() { + let mut headers = HeaderMap::new(); + headers.insert(header::HOST, HeaderValue::from_static("example.test")); + headers.insert( + header::ORIGIN, + HeaderValue::from_static("https://example.test"), + ); + headers.insert( + header::REFERER, + HeaderValue::from_static("https://attacker.test/login"), + ); + assert!(!same_origin(&headers, true)); + headers.insert( + header::REFERER, + HeaderValue::from_static("https://example.test/login"), + ); + assert!(same_origin(&headers, true)); + headers.append( + header::ORIGIN, + HeaderValue::from_static("https://example.test"), + ); + assert!(!same_origin(&headers, true)); + headers = HeaderMap::new(); + headers.insert(header::HOST, HeaderValue::from_static("example.test")); + assert!(!same_origin(&headers, true)); + } + + #[test] + fn origin_parser_rejects_opaque_and_ambiguous_values() { + assert_eq!( + origin_authority("https://example.test/path"), + Some(("https", "example.test")) + ); + assert_eq!(origin_authority("null"), None); + assert_eq!(origin_authority("https://example.test/?x=1"), None); + assert_eq!( + referer_authority("https://example.test/login?x=1#frag"), + Some(("https", "example.test")) + ); + assert_eq!(referer_authority("https://user@host.example.test/x"), None); + assert_eq!(referer_authority("null"), None); + } + + #[test] + fn forwarded_https_is_accepted_only_from_the_configured_proxy_socket() { + let mut server = ServerState { + api_key: None, + dashboard_enabled: true, + dashboard_public: false, + dashboard_theme: "default".to_owned(), + dashboard_refresh_interval_s: 60, + configured_server_threads: 1, + database_path: ":memory:".to_owned(), + trusted_proxy_ips: vec!["192.0.2.10".to_owned()], + allow_insecure_web_auth: false, + started_at: Instant::now(), + }; + let mut headers = HeaderMap::new(); + headers.insert("x-forwarded-proto", HeaderValue::from_static("https")); + let make_connection = |remote_addr| { + eggserve_primitives::connection_info::ConnectionInfo::new( + None, + Some(remote_addr), + eggserve_primitives::connection_info::Scheme::Http, + None, + ) + }; + let connection = make_connection("192.0.2.10:443".parse().unwrap()); + assert!(forwarded_https(Some(&connection), &headers, &server)); + let untrusted_connection = make_connection("198.51.100.20:443".parse().unwrap()); + assert!(!forwarded_https( + Some(&untrusted_connection), + &headers, + &server + )); + headers.append("x-forwarded-proto", HeaderValue::from_static("http")); + assert!(!forwarded_https(Some(&connection), &headers, &server)); + + server.trusted_proxy_ips.clear(); + assert!(!web_auth_transport_allowed( + Some(&connection), + &headers, + &server + )); + } + + #[test] + fn login_throttle_uses_forwarded_client_ip_only_from_a_trusted_proxy() { + let server = ServerState { + api_key: None, + dashboard_enabled: true, + dashboard_public: false, + dashboard_theme: "default".to_owned(), + dashboard_refresh_interval_s: 60, + configured_server_threads: 1, + database_path: ":memory:".to_owned(), + trusted_proxy_ips: vec!["192.0.2.10".to_owned()], + allow_insecure_web_auth: false, + started_at: Instant::now(), + }; + let connection = |remote_addr| { + eggserve_primitives::connection_info::ConnectionInfo::new( + None, + Some(remote_addr), + eggserve_primitives::connection_info::Scheme::Http, + None, + ) + }; + let mut headers = HeaderMap::new(); + headers.insert("x-forwarded-for", HeaderValue::from_static("198.51.100.24")); + + let trusted_peer = connection("192.0.2.10:443".parse().unwrap()); + assert_eq!( + throttle_peer_ip(Some(&trusted_peer), &headers, &server), + Some("198.51.100.24".parse().unwrap()) + ); + + let untrusted_peer = connection("198.51.100.30:443".parse().unwrap()); + assert_eq!( + throttle_peer_ip(Some(&untrusted_peer), &headers, &server), + Some("198.51.100.30".parse().unwrap()) + ); + } + + #[test] + fn login_throttle_rejects_ambiguous_forwarded_client_ip() { + let server = ServerState { + api_key: None, + dashboard_enabled: true, + dashboard_public: false, + dashboard_theme: "default".to_owned(), + dashboard_refresh_interval_s: 60, + configured_server_threads: 1, + database_path: ":memory:".to_owned(), + trusted_proxy_ips: vec!["192.0.2.10".to_owned()], + allow_insecure_web_auth: false, + started_at: Instant::now(), + }; + let trusted_peer = eggserve_primitives::connection_info::ConnectionInfo::new( + None, + Some("192.0.2.10:443".parse().unwrap()), + eggserve_primitives::connection_info::Scheme::Http, + None, + ); + let mut headers = HeaderMap::new(); + headers.append("x-forwarded-for", HeaderValue::from_static("198.51.100.24")); + headers.append("x-forwarded-for", HeaderValue::from_static("198.51.100.25")); + assert_eq!( + throttle_peer_ip(Some(&trusted_peer), &headers, &server), + Some("192.0.2.10".parse().unwrap()) + ); + + headers.remove("x-forwarded-for"); + headers.insert( + "x-forwarded-for", + HeaderValue::from_static("198.51.100.24, 198.51.100.25"), + ); + assert_eq!( + throttle_peer_ip(Some(&trusted_peer), &headers, &server), + Some("192.0.2.10".parse().unwrap()) + ); + } +} diff --git a/rust/src/server/dashboard/admin.rs b/rust/src/server/dashboard/admin.rs new file mode 100644 index 000000000..8232c0efa --- /dev/null +++ b/rust/src/server/dashboard/admin.rs @@ -0,0 +1,968 @@ +use super::*; +use axum::{Extension, Form, extract::State, http::HeaderMap, response::Redirect}; +use serde::Deserialize; + +#[derive(Deserialize)] +pub(in crate::server) struct CreateUserForm { + username: String, + password: String, + profile_id: Option, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct UserStateForm { + user_id: String, + enabled: bool, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct AdminPasswordForm { + user_id: String, + password: String, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct UserProfileForm { + user_id: String, + profile_id: String, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct ProfileCreateForm { + profile_id: String, + name: String, + default_effect: String, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct ProfileUpdateForm { + profile_id: String, + name: String, + default_effect: String, + version: i64, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct ProfileRuleForm { + profile_id: String, + model_id: String, + effect: String, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct ProfilePreferForm { + profile_id: String, + model_id: String, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct ProfileDeleteForm { + profile_id: String, + version: i64, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct ProfileRuleDeleteForm { + profile_id: String, + model_id: String, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct UserOverrideForm { + user_id: String, + model_id: String, + effect: String, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct UserOverrideRemoveForm { + user_id: String, + model_id: String, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct UserBudgetForm { + user_id: String, + model_id: String, + window: String, + max_requests: String, + max_tokens: String, + max_microdollars: String, + max_concurrent: String, + /// Optional so older posted forms keep working; absent means estimated. + enforcement: Option, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct InstanceBudgetForm { + model_id: String, + window: String, + max_requests: String, + max_tokens: String, + max_microdollars: String, + max_concurrent: String, + /// Optional so older posted forms keep working; absent means estimated. + enforcement: Option, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct TokenCreateForm { + label: String, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct TokenRevokeForm { + credential_id: String, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct AdminTokenForm { + user_id: String, + label: String, + csrf: String, +} + +#[derive(Deserialize)] +pub(in crate::server) struct AdminRevokeTokensForm { + user_id: String, + csrf: String, +} + +fn csrf_valid( + session: &crate::web_session::Principal, + headers: &HeaderMap, + supplied: &str, +) -> bool { + crate::web_session::verify_csrf(session, supplied) + && crate::web_session::csrf_cookie_value(headers) + .is_some_and(|cookie| crate::web_session::verify_csrf(session, &cookie)) +} + +fn admin_only(session: &crate::web_session::Principal) -> bool { + session.role == "admin" && !session.must_change_password +} + +fn bad_request(message: &'static str) -> Response { + (StatusCode::BAD_REQUEST, message).into_response() +} + +fn mutation_redirect(path: &'static str) -> Response { + Redirect::to(path).into_response() +} + +fn mutation_error(error: impl std::fmt::Display, fallback: &'static str) -> Response { + tracing::warn!(error = %error, "user administration mutation failed"); + (StatusCode::BAD_REQUEST, fallback).into_response() +} + +fn one_time_token_page(token: &str, return_path: &str, owner_message: &str) -> Response { + let mut response = html_response(format!( + "Save personal token

Save this credential now

It will not be shown again. {owner_message}

Store it in the environment that launches the client as EGGPOOL_API_KEY. The return screen will never contain the credential.

{}

Return

", + html_escape(token), + html_escape(return_path) + )); + response.headers_mut().insert( + header::CACHE_CONTROL, + HeaderValue::from_static("no-store, max-age=0"), + ); + response.headers_mut().insert( + http::HeaderName::from_static("referrer-policy"), + HeaderValue::from_static("no-referrer"), + ); + response.headers_mut().insert( + http::HeaderName::from_static("x-content-type-options"), + HeaderValue::from_static("nosniff"), + ); + response +} + +pub(in crate::server) async fn users_page( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, +) -> Response { + let csrf = crate::web_session::csrf_cookie_value(&headers) + .filter(|token| crate::web_session::verify_csrf(&session, token)) + .unwrap_or_default(); + let users = match crate::identity::list_users(&state.database).await { + Ok(users) => users, + Err(_) => return degraded("user administration unavailable"), + }; + let policy = crate::model_policy::ModelPolicyService::new(state.database.clone()); + let profiles = match policy.list_profiles().await { + Ok(profiles) => profiles, + Err(_) => return degraded("user administration unavailable"), + }; + let instance_rules = + match crate::instance_budget::InstanceBudgetService::new(state.database.clone()) + .rules() + .await + { + Ok(rules) => rules, + Err(_) => return degraded("user administration unavailable"), + }; + let body = render_users_admin(&users, &profiles, &instance_rules, &csrf); + dashboard_page_with_body( + &state.server.dashboard_theme, + "Users", + "users", + Some("24h".to_owned()), + None, + state.server.dashboard_refresh_interval_s, + body, + ) +} + +pub(in crate::server) async fn profiles_page( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, +) -> Response { + let csrf = crate::web_session::csrf_cookie_value(&headers) + .filter(|token| crate::web_session::verify_csrf(&session, token)) + .unwrap_or_default(); + let profiles = match crate::model_policy::ModelPolicyService::new(state.database.clone()) + .list_profiles() + .await + { + Ok(profiles) => profiles, + Err(_) => return degraded("profile administration unavailable"), + }; + let rules = match crate::model_policy::ModelPolicyService::new(state.database.clone()) + .list_profile_rules() + .await + { + Ok(rules) => rules, + Err(_) => return degraded("profile administration unavailable"), + }; + let body = render_profiles_admin(&profiles, &rules, &csrf); + dashboard_page_with_body( + &state.server.dashboard_theme, + "Profiles", + "profiles", + Some("24h".to_owned()), + None, + state.server.dashboard_refresh_interval_s, + body, + ) +} + +pub(in crate::server) async fn usage_page( + State(state): State, + Extension(session): Extension, + Extension(principal): Extension, +) -> Response { + let Some(user_id) = principal.user_id.as_deref() else { + return (StatusCode::FORBIDDEN, "User account required.").into_response(); + }; + if session.role != "admin" { + let rules = match crate::user_budget::UserBudgetService::new(state.database.clone()) + .rules(user_id) + .await + { + Ok(rules) => rules, + Err(_) => return degraded("usage unavailable"), + }; + let rows = match crate::user_budget::UserBudgetService::new(state.database.clone()) + .usage(user_id, "0000-01-01", 500) + .await + { + Ok(rows) => rows, + Err(_) => return degraded("usage unavailable"), + }; + return html_response(render_my_usage(&session.username, &rules, &rows)); + } + let rows = match state + .database + .call(|connection| { + let mut statement = connection.prepare( + "SELECT u.username, COALESCE(b.model_id, '(all models)'), \ + SUM(b.request_count), SUM(b.token_count), SUM(b.cost_microdollars) \ + FROM user_budget_usage b JOIN users u ON u.id = b.user_id \ + GROUP BY u.id, b.model_id ORDER BY u.username COLLATE NOCASE, b.model_id LIMIT 500", + )?; + statement + .query_map([], |row| { + Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?, row.get(4)?)) + })? + .collect::, _>>() + }) + .await + { + Ok(rows) => rows, + Err(_) => return degraded("usage unavailable"), + }; + html_response(render_usage_admin(&rows, "")) +} + +pub(in crate::server) async fn audit_page(State(state): State) -> Response { + let entries = match crate::identity::admin_audit(&state.database, 200).await { + Ok(entries) => entries, + Err(_) => return degraded("audit history unavailable"), + }; + let rows = entries + .iter() + .map(|entry| { + format!( + "{}{}{}{}{}", + html_escape(&entry.created_at), + html_escape(&entry.actor_id), + html_escape(&entry.action), + html_escape(&entry.target_kind), + html_escape(&entry.target_id), + ) + }) + .collect::(); + html_response(format!( + "Admin audit

Administrative audit

{rows}
UTC timeActor IDActionTarget typeTarget ID

Users

" + )) +} + +pub(in crate::server) async fn my_account_page( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, +) -> Response { + let csrf = crate::web_session::csrf_cookie_value(&headers) + .filter(|token| crate::web_session::verify_csrf(&session, token)) + .unwrap_or_default(); + let credentials = + match crate::identity::list_user_credentials(&state.database, &session.user_id).await { + Ok(credentials) => credentials, + Err(_) => return degraded("account details unavailable"), + }; + let rows = credentials.iter().map(|credential| format!( + "{}{}{}
", + html_escape(&credential.label), html_escape(&credential.created_at), + html_escape(credential.last_used_at.as_deref().unwrap_or("Never")), html_escape(&csrf), html_escape(&credential.id) + )).collect::(); + let token_flash = crate::web_session::csrf_cookie_value(&headers) + .map(|_| "

New credentials are displayed only once when created. Store them in your client environment as EGGPOOL_API_KEY.

") + .unwrap_or_default(); + html_response(format!( + "My account

My account

Signed in as {}

My usage · My accessible models · Change password

Personal client credentials

{token_flash}
{rows}
LabelCreatedLast usedAction
", + html_escape(&session.username), + html_escape(&csrf), + html_escape(&csrf) + )) +} + +pub(in crate::server) async fn my_models_page( + State(state): State, + Extension(session): Extension, +) -> Response { + let lease = match state.runtime.acquire().await { + Ok(lease) => lease, + Err(_) => return degraded("model list unavailable"), + }; + let inference = lease.generation().inference(); + let Some(policy) = inference.model_policy_service() else { + return degraded("model list unavailable"); + }; + let snapshot = match policy.snapshot(&session.user_id).await { + Ok(snapshot) => snapshot, + Err(_) => return degraded("model list unavailable"), + }; + let models = inference + .authorized_discovery_model_ids(&snapshot) + .into_iter() + .map(|model| format!("
  • {}
  • ", html_escape(&model))) + .collect::(); + html_response(format!( + "My models

    My accessible models

    This list is filtered by your current model policy.

      {models}

    My account

    " + )) +} + +pub(in crate::server) async fn create_user( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + let profile_id = form.profile_id.as_deref().filter(|id| !id.is_empty()); + let created = if let Some(profile_id) = profile_id { + crate::identity::create_member_with_profile_from_browser( + &state.database, + &session, + &form.username, + &form.password, + profile_id, + ) + .await + } else { + crate::identity::create_member_from_browser( + &state.database, + &session, + &form.username, + &form.password, + ) + .await + }; + let _user = match created { + Ok(user) => user, + Err(error) => return mutation_error(error, "Could not create user."), + }; + mutation_redirect("/users") +} + +pub(in crate::server) async fn set_user_state( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::identity::set_user_enabled_from_browser( + &state.database, + &session, + &form.user_id, + form.enabled, + ) + .await + { + Ok(()) => mutation_redirect("/users"), + Err(error) => mutation_error(error, "Could not change user state."), + } +} + +pub(in crate::server) async fn reset_user_password( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::identity::admin_reset_password_from_browser( + &state.database, + &session, + &form.user_id, + &form.password, + ) + .await + { + Ok(()) => mutation_redirect("/users"), + Err(error) => mutation_error(error, "Could not reset user password."), + } +} + +pub(in crate::server) async fn assign_user_profile( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::model_policy::ModelPolicyService::new(state.database.clone()) + .with_browser_admin_session(&session) + .assign_profile(&form.user_id, &form.profile_id, &session.user_id) + .await + { + Ok(()) => mutation_redirect("/users"), + Err(error) => mutation_error(error, "Could not assign profile."), + } +} + +pub(in crate::server) async fn create_profile( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::model_policy::ModelPolicyService::new(state.database.clone()) + .with_browser_admin_session(&session) + .create_profile( + &form.profile_id, + &form.name, + &form.default_effect, + &session.user_id, + ) + .await + { + Ok(()) => mutation_redirect("/profiles"), + Err(error) => mutation_error(error, "Could not create profile."), + } +} + +pub(in crate::server) async fn update_profile( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::model_policy::ModelPolicyService::new(state.database.clone()) + .with_browser_admin_session(&session) + .update_profile( + &form.profile_id, + &form.name, + &form.default_effect, + form.version, + &session.user_id, + ) + .await + { + Ok(true) => mutation_redirect("/profiles"), + Ok(false) => ( + StatusCode::CONFLICT, + "Profile changed since this page loaded.", + ) + .into_response(), + Err(error) => mutation_error(error, "Could not update profile."), + } +} + +pub(in crate::server) async fn delete_profile( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::model_policy::ModelPolicyService::new(state.database.clone()) + .with_browser_admin_session(&session) + .delete_profile(&form.profile_id, form.version, &session.user_id) + .await + { + Ok(true) => mutation_redirect("/profiles"), + Ok(false) => ( + StatusCode::CONFLICT, + "Profile is assigned or changed since this page loaded.", + ) + .into_response(), + Err(error) => mutation_error(error, "Could not delete profile."), + } +} + +pub(in crate::server) async fn set_profile_rule( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::model_policy::ModelPolicyService::new(state.database.clone()) + .with_browser_admin_session(&session) + .set_profile_rule( + &form.profile_id, + &form.model_id, + &form.effect, + &session.user_id, + ) + .await + { + Ok(()) => mutation_redirect("/profiles"), + Err(error) => mutation_error(error, "Could not save model rule."), + } +} + +pub(in crate::server) async fn remove_profile_rule( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::model_policy::ModelPolicyService::new(state.database.clone()) + .with_browser_admin_session(&session) + .remove_profile_rule(&form.profile_id, &form.model_id, &session.user_id) + .await + { + Ok(_) => mutation_redirect("/profiles"), + Err(error) => mutation_error(error, "Could not remove profile rule."), + } +} + +pub(in crate::server) async fn set_profile_preference( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + let model = (!form.model_id.trim().is_empty()).then_some(form.model_id.as_str()); + match crate::model_policy::ModelPolicyService::new(state.database.clone()) + .with_browser_admin_session(&session) + .set_profile_preference(&form.profile_id, model, &session.user_id) + .await + { + Ok(()) => mutation_redirect("/profiles"), + Err(error) => mutation_error(error, "Could not save model recommendation."), + } +} + +pub(in crate::server) async fn set_user_override( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::model_policy::ModelPolicyService::new(state.database.clone()) + .with_browser_admin_session(&session) + .set_user_override( + &form.user_id, + &form.model_id, + &form.effect, + &session.user_id, + ) + .await + { + Ok(()) => mutation_redirect("/users"), + Err(error) => mutation_error(error, "Could not save user model override."), + } +} + +pub(in crate::server) async fn remove_user_override( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::model_policy::ModelPolicyService::new(state.database.clone()) + .with_browser_admin_session(&session) + .remove_user_override(&form.user_id, &form.model_id, &session.user_id) + .await + { + Ok(_) => mutation_redirect("/users"), + Err(error) => mutation_error(error, "Could not remove user override."), + } +} + +fn parse_optional_limit(value: &str) -> Result, ()> { + if value.trim().is_empty() { + Ok(None) + } else { + value.parse::().map(Some).map_err(|_| ()) + } +} + +pub(in crate::server) async fn set_user_budget( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + let limits = match ( + parse_optional_limit(&form.max_requests), + parse_optional_limit(&form.max_tokens), + parse_optional_limit(&form.max_microdollars), + parse_optional_limit(&form.max_concurrent), + ) { + (Ok(requests), Ok(tokens), Ok(microdollars), Ok(concurrent)) => { + crate::user_budget::BudgetLimits { + requests, + tokens, + microdollars, + concurrent, + enforcement: crate::user_budget::BudgetEnforcement::Estimated, + } + } + _ => return bad_request("Budget limits must be positive integers."), + }; + let enforcement = match form.enforcement.as_deref().map(str::trim) { + None | Some("") | Some("estimated") => crate::user_budget::BudgetEnforcement::Estimated, + Some("hard") => crate::user_budget::BudgetEnforcement::Hard, + Some(_) => return bad_request("Invalid budget enforcement mode."), + }; + let limits = crate::user_budget::BudgetLimits { + enforcement, + ..limits + }; + let window = match form.window.as_str() { + "day" => crate::user_budget::BudgetWindow::Day, + "month" => crate::user_budget::BudgetWindow::Month, + _ => return bad_request("Invalid budget window."), + }; + let model_id = (!form.model_id.trim().is_empty()).then_some(form.model_id.as_str()); + match crate::user_budget::UserBudgetService::new(state.database.clone()) + .with_browser_admin_session(&session) + .set_rule(&session.user_id, &form.user_id, model_id, window, limits) + .await + { + Ok(()) => mutation_redirect("/users"), + Err(error) => mutation_error(error, "Could not save budget rule."), + } +} + +fn parse_instance_limits( + form: &InstanceBudgetForm, +) -> Result { + let limits = match ( + parse_optional_limit(&form.max_requests), + parse_optional_limit(&form.max_tokens), + parse_optional_limit(&form.max_microdollars), + parse_optional_limit(&form.max_concurrent), + ) { + (Ok(requests), Ok(tokens), Ok(microdollars), Ok(concurrent)) => { + crate::user_budget::BudgetLimits { + requests, + tokens, + microdollars, + concurrent, + enforcement: crate::user_budget::BudgetEnforcement::Estimated, + } + } + _ => return Err("Budget limits must be positive integers."), + }; + let enforcement = match form.enforcement.as_deref().map(str::trim) { + None | Some("") | Some("estimated") => crate::user_budget::BudgetEnforcement::Estimated, + Some("hard") => crate::user_budget::BudgetEnforcement::Hard, + Some(_) => return Err("Invalid budget enforcement mode."), + }; + Ok(crate::user_budget::BudgetLimits { + enforcement, + ..limits + }) +} + +fn parse_instance_window(window: &str) -> Result { + match window { + "day" => Ok(crate::user_budget::BudgetWindow::Day), + "month" => Ok(crate::user_budget::BudgetWindow::Month), + _ => Err("Invalid budget window."), + } +} + +pub(in crate::server) async fn set_instance_budget( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + let limits = match parse_instance_limits(&form) { + Ok(limits) => limits, + Err(message) => return bad_request(message), + }; + let window = match parse_instance_window(form.window.as_str()) { + Ok(window) => window, + Err(message) => return bad_request(message), + }; + let model_id = (!form.model_id.trim().is_empty()).then_some(form.model_id.as_str()); + match crate::instance_budget::InstanceBudgetService::new(state.database.clone()) + .with_browser_admin_session(&session) + .set_rule(&session.user_id, model_id, window, limits) + .await + { + Ok(()) => mutation_redirect("/users"), + Err(error) => mutation_error(error, "Could not save instance budget rule."), + } +} + +pub(in crate::server) async fn remove_instance_budget( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + let window = match parse_instance_window(form.window.as_str()) { + Ok(window) => window, + Err(message) => return bad_request(message), + }; + let model_id = (!form.model_id.trim().is_empty()).then_some(form.model_id.as_str()); + match crate::instance_budget::InstanceBudgetService::new(state.database.clone()) + .with_browser_admin_session(&session) + .remove_rule(&session.user_id, model_id, window) + .await + { + Ok(_) => mutation_redirect("/users"), + Err(error) => mutation_error(error, "Could not remove instance budget rule."), + } +} + +pub(in crate::server) async fn create_personal_token( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + match crate::identity::issue_personal_api_token_from_browser( + &state.database, + &session, + &session.user_id, + &form.label, + ) + .await + { + Ok(credential) => one_time_token_page( + &credential.token, + "/my-account", + "This is your personal credential.", + ), + Err(error) => mutation_error(error, "Could not create credential."), + } +} + +pub(in crate::server) async fn revoke_personal_token( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + match crate::identity::revoke_user_credential_from_browser( + &state.database, + &session, + &session.user_id, + &form.credential_id, + ) + .await + { + Ok(()) => mutation_redirect("/my-account"), + Err(error) => mutation_error(error, "Could not revoke credential."), + } +} + +pub(in crate::server) async fn issue_admin_token( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::identity::issue_personal_api_token_from_browser( + &state.database, + &session, + &form.user_id, + &form.label, + ) + .await + { + Ok(credential) => one_time_token_page( + &credential.token, + "/users", + "The administrator issued this for the selected user.", + ), + Err(error) => mutation_error(error, "Could not create credential."), + } +} + +pub(in crate::server) async fn revoke_user_tokens( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + if !csrf_valid(&session, &headers, &form.csrf) { + return (StatusCode::FORBIDDEN, "Invalid request token.").into_response(); + } + if !admin_only(&session) { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + match crate::identity::revoke_user_api_tokens_from_browser( + &state.database, + &session, + &form.user_id, + ) + .await + { + Ok(_) => mutation_redirect("/users"), + Err(error) => mutation_error(error, "Could not revoke credentials."), + } +} diff --git a/rust/src/server/dashboard/mod.rs b/rust/src/server/dashboard/mod.rs index 56f27010c..b35f107fa 100644 --- a/rust/src/server/dashboard/mod.rs +++ b/rust/src/server/dashboard/mod.rs @@ -111,6 +111,7 @@ pub(super) struct OverviewPage<'a> { health_snapshots: &'a [crate::health::AccountHealthSnapshot], } +mod admin; mod api; mod assets; mod format; @@ -127,8 +128,8 @@ use api::{ }; use format::*; use render::{ - dashboard_page_with_body, render_dashboard_page_body, render_model_detail, render_overview, - summary_json, + dashboard_page_with_body, render_dashboard_page_body, render_model_detail, render_my_usage, + render_overview, render_profiles_admin, render_usage_admin, render_users_admin, summary_json, }; #[cfg(test)] use render::{ @@ -139,6 +140,14 @@ use render::{ use response::{html_response, static_response}; use theme::*; +pub(super) use admin::{ + assign_user_profile, audit_page, create_personal_token, create_profile, create_user, + delete_profile, issue_admin_token, my_account_page, my_models_page, profiles_page, + remove_instance_budget, remove_profile_rule, remove_user_override, reset_user_password, + revoke_personal_token, revoke_user_tokens, set_instance_budget, set_profile_preference, + set_profile_rule, set_user_budget, set_user_override, set_user_state, update_profile, + usage_page, users_page, +}; pub(super) use api::{ grouped_timeseries_api, stats_cache_observability, stats_cache_stability, stats_request_segmentation, stats_request_shaping, stats_transcoding, summary, timeseries_api, diff --git a/rust/src/server/dashboard/render/admin.rs b/rust/src/server/dashboard/render/admin.rs new file mode 100644 index 000000000..255867c15 --- /dev/null +++ b/rust/src/server/dashboard/render/admin.rs @@ -0,0 +1,234 @@ +use super::*; + +pub(in crate::server::dashboard) fn render_users_admin( + users: &[crate::identity::UserSummary], + profiles: &[crate::model_policy::PolicyProfileSummary], + instance_rules: &[crate::instance_budget::InstanceBudgetRule], + csrf: &str, +) -> String { + let options = profiles + .iter() + .map(|profile| { + format!( + "", + html_escape(&profile.id), + html_escape(&profile.name) + ) + }) + .collect::(); + let rows = users + .iter() + .map(|user| { + let active = user.state == "active"; + let state_label = if active { "Disable" } else { "Reactivate" }; + let state_form = format!( + "
    ", + html_escape(csrf), html_escape(&user.id), !active, state_label + ); + let password_form = format!( + "
    ", + html_escape(csrf), html_escape(&user.id) + ); + let profile_form = format!( + "
    ", + html_escape(&user.username), options, html_escape(csrf), html_escape(&user.id) + ); + let override_form = format!( + "
    ", + html_escape(csrf), html_escape(&user.id) + ); + let remove_override_form = format!( + "
    ", + html_escape(csrf), html_escape(&user.id) + ); + let budget_form = format!( + "
    Set budget

    Hard token rules admit only requests with an explicit caller output cap and reserve a worst-case bound; hard USD rules additionally require an operator tariff in provider configuration. Estimated rules use approximate holds and are never a hard ceiling.

    ", + html_escape(csrf), html_escape(&user.id) + ); + let token_form = format!( + "
    ", + html_escape(csrf), html_escape(&user.id), html_escape(csrf), html_escape(&user.id) + ); + let actions = format!( + "{state_form}{password_form}{profile_form}{override_form}{remove_override_form}{budget_form}{token_form}" + ); + format!( + "{}{}{}{}{}", + html_escape(&user.username), + html_escape(&user.role), + html_escape(&user.state), + if user.must_change_password { "Must change password" } else { "—" }, + actions + ) + }) + .collect::(); + let instance_rows = instance_rules + .iter() + .map(|rule| { + format!( + "{}{}{}{}{}{}{}
    ", + html_escape(rule.model_id.as_deref().unwrap_or("All models")), + html_escape(rule.window.as_str()), + html_escape(rule.enforcement.as_str()), + rule.max_requests + .map_or_else(|| "—".to_owned(), |value| value.to_string()), + rule.max_tokens + .map_or_else(|| "—".to_owned(), |value| value.to_string()), + rule.max_microdollars + .map_or_else(|| "—".to_owned(), |value| value.to_string()), + rule.max_concurrent + .map_or_else(|| "—".to_owned(), |value| value.to_string()), + html_escape(csrf), + html_escape(rule.model_id.as_deref().unwrap_or("")), + html_escape(rule.window.as_str()), + ) + }) + .collect::(); + let instance_section = format!( + "

    Instance budget ceilings

    One instance-wide ceiling intersected with every per-user rule: an admitted call must satisfy all applicable instance and user limits. Absent rules mean no additional cap. Members never see other users' usage here. Hard token rules need an explicit caller output cap; hard USD rules need an operator tariff in provider configuration.

    {instance_rows}
    ModelWindowModeMax requestsMax tokensMax microdollarsMax concurrentActions
    Set instance ceiling
    ", + html_escape(csrf) + ); + format!( + "{instance_section}

    Users

    Create local member accounts and assign policy profiles. Temporary passwords are shown only to the administrator who enters them; users must change them at first login.

    Local accounts

    {rows}
    UsernameRoleStatePasswordActions
    ", + html_escape(csrf) + ) +} + +pub(in crate::server::dashboard) fn render_profiles_admin( + profiles: &[crate::model_policy::PolicyProfileSummary], + rules: &[crate::model_policy::PolicyProfileRuleSummary], + csrf: &str, +) -> String { + let rows = profiles + .iter() + .map(|profile| { + let profile_rules = rules + .iter() + .filter(|rule| rule.profile_id == profile.id) + .map(|rule| { + format!( + "
  • {}: {}
  • ", + html_escape(&rule.model_id), html_escape(&rule.effect), html_escape(csrf), + html_escape(&profile.id), html_escape(&rule.model_id) + ) + }) + .collect::(); + format!( + "

    {}

    ID: {} · default: {} · version: {} · assigned users: {} · exact model rules: {}

      {profile_rules}
    ", + html_escape(&profile.name), html_escape(&profile.id), html_escape(&profile.default_effect), + profile.version, profile.assigned_users, profile.model_rules, + html_escape(csrf), html_escape(&profile.id), profile.version, html_escape(&profile.name), + if profile.default_effect == "allow" { " selected" } else { "" }, + if profile.default_effect == "deny" { " selected" } else { "" }, + html_escape(csrf), html_escape(&profile.id), + html_escape(csrf), html_escape(&profile.id), html_escape(profile.preferred_model_id.as_deref().unwrap_or("")), + html_escape(csrf), html_escape(&profile.id), profile.version, + ) + }) + .collect::(); + format!( + "

    Profiles

    Profiles control model access and client recommendations. Provider topology and upstream secrets remain in operator-owned configuration.

    {rows}", + html_escape(csrf) + ) +} + +/// Standalone document shell for on-prem pages that are not rendered through +/// the shared dashboard layout. The usage pages are role-specific reads rather +/// than dashboard panels, so without a shell they would ship a bare `
    ` +/// fragment: no stylesheet, no viewport declaration, and tables with no way to +/// scroll on a narrow screen. +fn on_prem_page(title: &str, body: &str) -> String { + format!( + "{}
    {}
    ", + html_escape(title), + body + ) +} + +/// Wrap one table in the shared keyboard-reachable scroll region. +fn scroll_table(label: &str, table: &str) -> String { + format!( + "
    {table}
    ", + html_escape(label) + ) +} + +pub(in crate::server::dashboard) fn render_usage_admin( + rows: &[(String, String, i64, i64, i64)], + csrf: &str, +) -> String { + let content = rows.iter().map(|(user, model, requests, tokens, cost)| { + format!("{}{}{requests}{tokens}{cost}", html_escape(user), html_escape(model)) + }).collect::(); + let limits = scroll_table( + "Usage by user and model", + &format!( + "{content}
    UserModelRequestsTokensEstimated microdollars
    " + ), + ); + on_prem_page( + "Usage", + &format!( + "

    Usage

    Token and microdollar figures are usage estimates; estimated USD is not an invoice or a hard ceiling for unpriced models.

    {limits}
    ", + html_escape(csrf) + ), + ) +} + +pub(in crate::server::dashboard) fn render_my_usage( + user: &str, + rules: &[crate::user_budget::UserBudgetRule], + rows: &[crate::user_budget::UserBudgetUsage], +) -> String { + let rule_rows = rules + .iter() + .map(|rule| { + format!( + "{}{}{}{}{}{}{}", + html_escape(rule.model_id.as_deref().unwrap_or("All models")), + html_escape(rule.window.as_str()), + html_escape(rule.enforcement.as_str()), + rule.max_requests + .map_or_else(|| "—".to_owned(), |value| value.to_string()), + rule.max_tokens + .map_or_else(|| "—".to_owned(), |value| value.to_string()), + rule.max_microdollars + .map_or_else(|| "—".to_owned(), |value| value.to_string()), + rule.max_concurrent + .map_or_else(|| "—".to_owned(), |value| value.to_string()), + ) + }) + .collect::(); + let usage_rows = rows + .iter() + .map(|row| { + format!( + "{}{}{}{}{}", + html_escape(&row.model_id), + html_escape(&row.window_start), + row.request_count, + row.token_count, + row.cost_microdollars + ) + }) + .collect::(); + let limit_table = scroll_table( + "Configured usage limits", + &format!( + "{rule_rows}
    ModelWindowModeMax requestsMax tokensMax microdollarsMax concurrent
    " + ), + ); + let usage_table = scroll_table( + "Settled usage", + &format!( + "{usage_rows}
    ModelWindow start (UTC)RequestsTokensEstimated microdollars
    " + ), + ); + on_prem_page( + "My usage", + &format!( + "

    My usage

    Account: {}

    Daily/monthly periods use UTC. Rules marked hard enforce a provable worst-case bound (token rules need an explicit caller output cap; USD rules need an operator tariff) and deny otherwise; rules marked estimated use approximate holds and are not a guaranteed cap. Active requests hold budget before dispatch and may reduce admission headroom.

    Configured limits

    {limit_table}

    Settled usage

    {usage_table}

    My account

    ", + html_escape(user) + ), + ) +} diff --git a/rust/src/server/dashboard/render/layout.rs b/rust/src/server/dashboard/render/layout.rs index f7bd1efca..1cd0b2ab4 100644 --- a/rust/src/server/dashboard/render/layout.rs +++ b/rust/src/server/dashboard/render/layout.rs @@ -88,6 +88,10 @@ pub(in crate::server::dashboard) fn render_dashboard_layout( ("reliability", "/reliability", "Reliability"), ("routing", "/routing", "Routing"), ("cache", "/cache", "Cache"), + ("users", "/users", "Users"), + ("profiles", "/profiles", "Profiles"), + ("usage", "/usage", "Usage"), + ("my-account", "/my-account", "My Account"), ("accounts", "/accounts", "Accounts"), ("models", "/models", "Models"), ("latency", "/latency", "Latency"), @@ -138,7 +142,7 @@ pub(in crate::server::dashboard) fn render_dashboard_layout( String::new() }; let navigation_markup = format!( - "
    {}
    ", + "
    {}
    ", navigation, theme_options, html_escape(period) diff --git a/rust/src/server/dashboard/render/mod.rs b/rust/src/server/dashboard/render/mod.rs index d0330e265..e42ca5299 100644 --- a/rust/src/server/dashboard/render/mod.rs +++ b/rust/src/server/dashboard/render/mod.rs @@ -1,6 +1,7 @@ use super::*; mod accounts; +mod admin; mod cache; mod diagnostics; mod layout; @@ -10,6 +11,7 @@ mod runtime; mod telemetry; pub(super) use accounts::*; +pub(super) use admin::*; pub(super) use cache::*; pub(super) use diagnostics::*; pub(super) use layout::*; diff --git a/rust/src/server/health.rs b/rust/src/server/health.rs index c4e293a12..07ce95f94 100644 --- a/rust/src/server/health.rs +++ b/rust/src/server/health.rs @@ -6,15 +6,27 @@ pub(super) async fn healthz() -> Response { json_response(StatusCode::OK, json!({"status": "ok"})) } -pub(super) async fn models_api(State(state): State) -> Response { +pub(super) async fn models_api( + State(state): State, + Extension(principal): Extension, +) -> Response { let lease = match state.runtime.acquire().await { Ok(lease) => lease, Err(_) => return degraded("runtime unavailable"), }; - let data = lease - .generation() - .inference() - .catalog_model_ids() + let inference = lease.generation().inference(); + let mut model_ids = inference.discovery_model_ids(); + if let Some(user_id) = principal.user_id.as_deref() { + let Some(policy) = lease.generation().inference().model_policy_service() else { + return degraded("model policy unavailable"); + }; + let snapshot = match policy.snapshot(user_id).await { + Ok(snapshot) => snapshot, + Err(_) => return degraded("model policy unavailable"), + }; + model_ids = inference.authorized_discovery_model_ids(&snapshot); + } + let data = model_ids .into_iter() .map(|model_id| { json!({ @@ -37,6 +49,7 @@ pub(super) async fn models_api(State(state): State) -> Response { pub(super) async fn integration_profile( State(state): State, headers: HeaderMap, + Extension(principal): Extension, ) -> Response { let lease = match state.runtime.acquire().await { Ok(lease) => lease, @@ -58,7 +71,7 @@ pub(super) async fn integration_profile( ); } }; - let profile = match crate::operations::integrations::build_integration_profile( + let mut profile = match crate::operations::integrations::build_integration_profile( &config, &state.database, &base_url, @@ -73,6 +86,57 @@ pub(super) async fn integration_profile( ); } }; + let mut policy_revision = None; + if let Some(user_id) = principal.user_id.as_deref() { + let Some(policy) = lease.generation().inference().model_policy_service() else { + return json_response( + StatusCode::SERVICE_UNAVAILABLE, + json!({"detail": "integration profile unavailable"}), + ); + }; + let snapshot = match policy.snapshot(user_id).await { + Ok(snapshot) => snapshot, + Err(_) => { + return json_response( + StatusCode::SERVICE_UNAVAILABLE, + json!({"detail": "integration profile unavailable"}), + ); + } + }; + let inference = lease.generation().inference(); + let virtual_ids: std::collections::BTreeSet = inference + .authorized_virtual_model_ids(&snapshot) + .into_iter() + .collect(); + profile.models.extend(virtual_ids.iter().map(|model_id| { + eggpool_client_config::AgentModelProjection { + public_id: model_id.to_owned(), + display_name: model_id.to_owned(), + capabilities: eggpool_client_config::AgentModelCapabilities::conservative(), + } + })); + profile.models.retain(|model| { + if virtual_ids.contains(&model.public_id) { + snapshot.allows(&model.public_id, None) + } else { + snapshot.allows_catalog_id(&model.public_id) + } + }); + profile = match eggpool_client_config::AgentIntegrationProfileV1::new( + &profile.base_url, + profile.models, + profile.capabilities, + ) { + Ok(profile) => profile, + Err(_) => { + return json_response( + StatusCode::SERVICE_UNAVAILABLE, + json!({"detail": "integration profile unavailable"}), + ); + } + }; + policy_revision = Some(snapshot.revision()); + } let body = match profile.canonical_json() { Ok(body) => body, Err(_) => { @@ -88,14 +152,19 @@ pub(super) async fn integration_profile( json!({"detail": "integration profile unavailable"}), ); } - let etag = format!("\"{}\"", profile.revision); + // The portable body's revision stays a content hash. HTTP caches also + // fence on policy revision, including same-content policy edits. + let etag_revision = policy_revision + .map(|revision| format!("{}-p{revision}", profile.revision)) + .unwrap_or_else(|| profile.revision.clone()); + let etag = format!("\"{etag_revision}\""); if let Some(if_none) = headers .get(header::IF_NONE_MATCH) .and_then(|v| v.to_str().ok()) { let matched = if_none.split(',').any(|candidate| { let candidate = candidate.trim().trim_start_matches("W/").trim_matches('"'); - candidate == profile.revision + candidate == etag_revision }); if matched { return ( @@ -266,6 +335,68 @@ pub(super) fn parent_pid() -> serde_json::Value { } } +#[derive(Debug, Deserialize)] +pub(super) struct UserBudgetQuery { + user_id: Option, +} + +pub(super) async fn user_budget_api( + State(state): State, + Extension(principal): Extension, + Query(query): Query, +) -> Response { + let Some(owner_id) = principal.user_id.as_deref() else { + return (StatusCode::UNAUTHORIZED, "Personal user access required.").into_response(); + }; + let target_id = query.user_id.as_deref().unwrap_or(owner_id); + if target_id != owner_id && principal.role.as_deref() != Some("admin") { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + let budgets = crate::user_budget::UserBudgetService::new(state.database.clone()); + let rules = match budgets.rules(target_id).await { + Ok(value) => value, + Err(_) => return degraded("user budget unavailable"), + }; + let usage = match budgets.usage(target_id, "0000-01-01", 500).await { + Ok(value) => value, + Err(_) => return degraded("user budget unavailable"), + }; + // Instance ceilings are admin-visible only: member responses keep their + // existing shape so one member can never observe the shared ceiling or + // infer other users' consumption from it. + if principal.role.as_deref() != Some("admin") { + return json_response( + StatusCode::OK, + json!({ + "window_timezone": "UTC", + "cost_basis": "estimated_microdollars", + "rules": rules, + "usage": usage, + }), + ); + } + let instance = crate::instance_budget::InstanceBudgetService::new(state.database.clone()); + let instance_rules = match instance.rules().await { + Ok(value) => value, + Err(_) => return degraded("user budget unavailable"), + }; + let instance_usage = match instance.usage("0000-01-01", 500).await { + Ok(value) => value, + Err(_) => return degraded("user budget unavailable"), + }; + json_response( + StatusCode::OK, + json!({ + "window_timezone": "UTC", + "cost_basis": "estimated_microdollars", + "rules": rules, + "usage": usage, + "instance_rules": instance_rules, + "instance_usage": instance_usage, + }), + ) +} + pub(super) async fn readyz(State(state): State) -> Response { let snapshot = readiness_snapshot(&state).await; if snapshot.ready { diff --git a/rust/src/server/inference.rs b/rust/src/server/inference.rs index 2c2c34324..1d150500b 100644 --- a/rust/src/server/inference.rs +++ b/rust/src/server/inference.rs @@ -6,10 +6,19 @@ pub(super) async fn chat_completions( Extension(_raw_body_reservation): Extension< Arc, >, + Extension(principal): Extension, headers: HeaderMap, body: Bytes, ) -> Response { - handle_inference(state, ClientSurface::ChatCompletions, headers, body, lease).await + handle_inference( + state, + ClientSurface::ChatCompletions, + headers, + body, + lease, + principal, + ) + .await } pub(super) async fn messages( @@ -18,10 +27,19 @@ pub(super) async fn messages( Extension(_raw_body_reservation): Extension< Arc, >, + Extension(principal): Extension, headers: HeaderMap, body: Bytes, ) -> Response { - handle_inference(state, ClientSurface::Messages, headers, body, lease).await + handle_inference( + state, + ClientSurface::Messages, + headers, + body, + lease, + principal, + ) + .await } pub(super) async fn responses( @@ -30,10 +48,19 @@ pub(super) async fn responses( Extension(_raw_body_reservation): Extension< Arc, >, + Extension(principal): Extension, headers: HeaderMap, body: Bytes, ) -> Response { - handle_inference(state, ClientSurface::Responses, headers, body, lease).await + handle_inference( + state, + ClientSurface::Responses, + headers, + body, + lease, + principal, + ) + .await } /// Historical remote-compaction endpoint (`POST /v1/responses/compact`). @@ -49,6 +76,7 @@ pub(super) async fn responses_compact( Extension(_raw_body_reservation): Extension< Arc, >, + Extension(principal): Extension, headers: HeaderMap, body: Bytes, ) -> Response { @@ -57,7 +85,16 @@ pub(super) async fn responses_compact( .and_then(|value| value.to_str().ok()) .map(str::to_owned); let proxy_request_id = crate::coordinator::new_proxy_request_id(); - handle_finite_compact(state, headers, body, session, proxy_request_id, lease).await + handle_finite_compact( + state, + headers, + body, + session, + proxy_request_id, + lease, + principal, + ) + .await } pub(super) async fn handle_finite_compact( @@ -67,15 +104,17 @@ pub(super) async fn handle_finite_compact( session: Option, proxy_request_id: String, lease: Arc, + principal: crate::identity::AuthenticatedPrincipal, ) -> Response { let surface = ClientSurface::Responses; let incoming = filtered_incoming_headers(&headers); - match crate::coordinator::execute_compact_finite( + match crate::coordinator::execute_compact_finite_attributed( lease.generation().inference(), body, incoming, session, proxy_request_id.clone(), + Some(principal.request_attribution()), ) .await { @@ -125,6 +164,7 @@ pub(super) async fn handle_inference( headers: HeaderMap, body: Bytes, lease: Arc, + principal: crate::identity::AuthenticatedPrincipal, ) -> Response { let session = headers .get("x-eggpool-route-session") @@ -132,13 +172,14 @@ pub(super) async fn handle_inference( .map(str::to_owned); let proxy_request_id = crate::coordinator::new_proxy_request_id(); let incoming = filtered_incoming_headers(&headers); - match crate::coordinator::execute_endpoint( + match crate::coordinator::execute_endpoint_attributed( lease.generation().inference(), surface, body, incoming, session, proxy_request_id, + Some(principal.request_attribution()), ) .await { diff --git a/rust/src/server/middleware.rs b/rust/src/server/middleware.rs index 1b03f7288..76d605876 100644 --- a/rust/src/server/middleware.rs +++ b/rust/src/server/middleware.rs @@ -7,30 +7,223 @@ pub(super) async fn authenticate( request: axum::http::Request, next: Next, ) -> Response { - if !requires_auth(request.uri().path(), &state.server) { + let path = request.uri().path().to_owned(); + if is_unconditionally_public(&path) { return next.run(request).await; } - if state - .server - .api_key - .as_deref() - .is_none_or(|key| verify_api_key(request.headers(), key)) + let supplied_key = provided_api_key(request.headers()); + let token_candidate = supplied_key + .filter(|_| personal_credential_allowed(&path)) + .filter(|_| { + !state + .server + .api_key + .as_deref() + .is_some_and(|server_key| verify_api_key(request.headers(), server_key)) + }); + let connection = request + .extensions() + .get::() + .map(|extension| &extension.0); + let token_peer = super::auth::throttle_peer_ip(connection, request.headers(), &state.server); + if personal_credential_allowed(&path) + && path != "/api/user-budget" + && state.server.api_key.is_some() + && supplied_key.is_none() { - return next.run(request).await; + // With the configured shared key present, an absent key cannot be + // valid in either legacy-key or managed mode. Reject before the + // access-state SQLite lookup. + return unauthorized("Invalid or missing API key"); + } + if token_candidate.is_some() && !crate::web_session::personal_token_attempt_allowed(token_peer) + { + return unauthorized("Invalid or missing personal API credential"); + } + let access_state = match crate::identity::access_state(&state.database).await { + Ok(access_state) => access_state, + Err(_) => return auth_unavailable(), + }; + if legacy_server_key_allowed(Some(&access_state)) { + if requires_browser_session(&path) { + return unauthorized("Browser session required"); + } + if !requires_auth(&path, request.method(), &state.server) + || state.server.api_key.is_none() + || state + .server + .api_key + .as_deref() + .is_some_and(|key| verify_api_key(request.headers(), key)) + { + let mut request = request; + if personal_credential_allowed(&path) { + request + .extensions_mut() + .insert(crate::identity::AuthenticatedPrincipal::legacy_shared_key()); + } + return next.run(request).await; + } + return unauthorized("Invalid or missing API key"); + } + if personal_credential_allowed(&path) + && let Some(token) = token_candidate + { + match crate::identity::verify_api_token_principal(&state.database, token).await { + Ok(Some(principal)) => { + let mut request = request; + request.extensions_mut().insert(principal); + return next.run(request).await; + } + Ok(None) => crate::web_session::record_personal_token_failure(token_peer), + Err(_) => return auth_unavailable(), + } + } + if personal_credential_allowed(&path) && path != "/api/user-budget" { + return unauthorized("Invalid or missing personal API credential"); + } + + let Some(session_token) = crate::web_session::session_cookie_value(request.headers()) else { + return unauthorized("Sign in required"); + }; + let principal = match crate::web_session::authenticate(&state.database, &session_token).await { + Ok(Some(principal)) => principal, + Ok(None) => return unauthorized("Sign in required"), + Err(_) => return auth_unavailable(), + }; + let account_surface = matches!( + path.as_str(), + "/account" + | "/account/password" + | "/logout" + | "/api/user-budget" + | "/my-account" + | "/my-account/models" + | "/usage" + ); + let account_surface = account_surface || path.starts_with("/account/tokens"); + if principal.role != "admin" && !account_surface { + return (StatusCode::FORBIDDEN, "Administrator access required.").into_response(); + } + if principal.must_change_password + && !matches!(path.as_str(), "/account" | "/account/password" | "/logout") + { + return axum::response::Redirect::to("/account?password=required").into_response(); + } + let connection = request + .extensions() + .get::(); + let secure = super::auth::request_is_secure( + connection.map(|extension| &extension.0), + request.headers(), + &state.server, + ); + if !super::auth::web_auth_transport_allowed( + connection.map(|extension| &extension.0), + request.headers(), + &state.server, + ) || (is_unsafe_method(request.method()) + && !super::auth::same_origin(request.headers(), secure)) + { + return ( + StatusCode::FORBIDDEN, + "Cross-origin or insecure request rejected.", + ) + .into_response(); } + let mut request = request; + request + .extensions_mut() + .insert(crate::identity::AuthenticatedPrincipal { + user_id: Some(principal.user_id.clone()), + credential_id: Some(principal.session_id.clone()), + role: Some(principal.role.clone()), + scopes: vec!["inference".to_owned(), "discovery".to_owned()], + auth_epoch: Some(principal.auth_epoch), + kind: crate::identity::PrincipalKind::BrowserSession, + }); + request.extensions_mut().insert(principal); + let mut response = next.run(request).await; + response.headers_mut().insert( + axum::http::header::CACHE_CONTROL, + axum::http::HeaderValue::from_static("no-store, max-age=0"), + ); + response +} + +fn legacy_server_key_allowed(access_state: Option<&str>) -> bool { + access_state == Some("legacy_unmigrated") +} + +fn personal_credential_allowed(path: &str) -> bool { + (path.starts_with("/v1/") && path != "/v1/healthz" && path != "/v1/readyz") + || path == "/api/integrations/v1/profile" + || path == "/api/user-budget" +} + +fn requires_browser_session(path: &str) -> bool { + matches!( + path, + "/account" + | "/account/password" + | "/logout" + | "/my-account" + | "/my-account/models" + | "/usage" + | "/users" + | "/profiles" + ) || path.starts_with("/account/tokens") + || path.starts_with("/admin/") +} + +fn is_unconditionally_public(path: &str) -> bool { + path == "/v1/healthz" + || path == "/v1/readyz" + || path.starts_with("/static/") + || path == "/login" +} + +fn is_unsafe_method(method: &http::Method) -> bool { + matches!( + *method, + http::Method::POST | http::Method::PUT | http::Method::PATCH | http::Method::DELETE + ) +} + +fn unauthorized(detail: &str) -> Response { ( StatusCode::UNAUTHORIZED, - axum::Json(json!({ - "detail": "Invalid or missing API key" - })), + axum::Json(json!({ "detail": detail })), + ) + .into_response() +} + +fn auth_unavailable() -> Response { + ( + StatusCode::SERVICE_UNAVAILABLE, + "Authentication service unavailable.", ) .into_response() } -pub(super) fn requires_auth(path: &str, server: &ServerState) -> bool { - if path == "/v1/healthz" || path == "/v1/readyz" || path.starts_with("/static/") { +pub(super) fn requires_auth(path: &str, method: &http::Method, server: &ServerState) -> bool { + if is_unconditionally_public(path) { return false; } + if matches!( + path, + "/account" + | "/account/password" + | "/logout" + | "/my-account" + | "/my-account/models" + | "/usage" + ) || path.starts_with("/account/tokens") + || path.starts_with("/admin/") + || matches!(path, "/users" | "/profiles") + { + return true; + } // EggPool-specific integration profile is always authenticated. It must // never inherit a dashboard/public exemption even when the dashboard is // public; the returned object is sanitized but the endpoint is not public. @@ -46,6 +239,15 @@ pub(super) fn requires_auth(path: &str, server: &ServerState) -> bool { if path == "/api/status" { return true; } + if path == "/api/user-budget" { + return true; + } + if path.starts_with("/api/") || encoded_api_path(path) { + return !(server.dashboard_enabled + && server.dashboard_public + && *method == http::Method::GET + && public_dashboard_api_path(path)); + } if path.starts_with("/v1/") { return true; } @@ -68,7 +270,53 @@ pub(super) fn requires_auth(path: &str, server: &ServerState) -> bool { || path.starts_with("/models/"); server.dashboard_enabled && !server.dashboard_public - && (dashboard_page || path.starts_with("/api/")) + && *method == http::Method::GET + && dashboard_page +} + +fn public_dashboard_api_path(path: &str) -> bool { + matches!( + path, + "/api/stats/summary" + | "/api/stats/transcoding" + | "/api/stats/cache-observability" + | "/api/stats/canonical-request-segmentation" + | "/api/stats/cache-stability" + | "/api/stats/request-shaping" + | "/api/timeseries" + | "/api/timeseries/grouped" + ) +} + +fn encoded_api_path(path: &str) -> bool { + let Some(first_segment) = path.strip_prefix('/') else { + return false; + }; + let first_segment = first_segment.split('/').next().unwrap_or_default(); + let bytes = first_segment.as_bytes(); + let mut decoded = Vec::with_capacity(bytes.len()); + let mut index = 0; + while index < bytes.len() { + if bytes[index] == b'%' { + let Some(pair) = bytes.get(index + 1..index + 3) else { + return false; + }; + let high = (pair[0] as char).to_digit(16); + let low = (pair[1] as char).to_digit(16); + let (Some(high), Some(low)) = (high, low) else { + return false; + }; + decoded.push(((high << 4) | low) as u8); + index += 3; + } else { + decoded.push(bytes[index]); + index += 1; + } + } + let Ok(decoded) = std::str::from_utf8(&decoded) else { + return false; + }; + decoded == "api" || decoded.starts_with("api/") } /// Acquire the active generation before collecting an inference body. Axum's @@ -212,10 +460,27 @@ fn backpressure_response(surface: crate::wire::ir::ClientSurface, detail: Vec Result<(), ServerError> { let key = config.resolved_server_api_key(); - if !is_loopback_host(&config.server.host) && key.is_none() { + if key.as_deref().is_some_and(|value| !valid_key_shape(value)) { return Err(ServerError::InvalidApiKey); } - if key.as_deref().is_some_and(|value| !valid_key_shape(value)) { + Ok(()) +} + +pub(super) fn validate_server_access( + config: &Config, + access_state: &str, +) -> Result<(), ServerError> { + validate_server_key(config)?; + if access_state == "fresh_uninitialized" && config.resolved_server_api_key().is_none() { + return Err(ServerError::StartupConflict { + detail: "fresh database requires local administrator bootstrap before serving" + .to_owned(), + }); + } + if access_state == "legacy_unmigrated" + && !is_loopback_host(&config.server.host) + && config.resolved_server_api_key().is_none() + { return Err(ServerError::InvalidApiKey); } Ok(()) @@ -239,6 +504,22 @@ pub(super) fn verify_api_key(headers: &HeaderMap, expected: &str) -> bool { // Non-UTF8 header values map to `""` (fail-closed, same as absent). // There is deliberately no per-request log or counter here: this runs on // every request, and the outcome is already observable as a 401. + let provided = provided_api_key(headers).unwrap_or(""); + let left = fixed_key(provided); + let right = fixed_key(expected); + // Always fold over the full 512-byte pads before combining with the + // shape bits, so shape validity does not short-circuit the compare. + let different = left + .iter() + .zip(right.iter()) + .fold(0u8, |accumulator, (a, b)| accumulator | (a ^ b)); + let provided_ok = u8::from(valid_key_shape(provided)); + let expected_ok = u8::from(valid_key_shape(expected)); + let match_ok = u8::from(different == 0); + (provided_ok & expected_ok & match_ok) == 1 +} + +fn provided_api_key(headers: &HeaderMap) -> Option<&str> { let authorization = headers .get(header::AUTHORIZATION) .and_then(|value| value.to_str().ok()) @@ -257,18 +538,7 @@ pub(super) fn verify_api_key(headers: &HeaderMap, expected: &str) -> bool { .filter(|value| !value.is_empty()) }) .unwrap_or(""); - let left = fixed_key(provided); - let right = fixed_key(expected); - // Always fold over the full 512-byte pads before combining with the - // shape bits, so shape validity does not short-circuit the compare. - let different = left - .iter() - .zip(right.iter()) - .fold(0u8, |accumulator, (a, b)| accumulator | (a ^ b)); - let provided_ok = u8::from(valid_key_shape(provided)); - let expected_ok = u8::from(valid_key_shape(expected)); - let match_ok = u8::from(different == 0); - (provided_ok & expected_ok & match_ok) == 1 + (!provided.is_empty()).then_some(provided) } pub(super) fn fixed_key(value: &str) -> [u8; 512] { @@ -284,3 +554,88 @@ pub(super) fn fixed_key(value: &str) -> [u8; 512] { pub(super) fn is_loopback_host(host: &str) -> bool { crate::config::is_loopback_host(host) } + +#[cfg(test)] +mod tests { + use super::{legacy_server_key_allowed, personal_credential_allowed, requires_auth}; + use crate::server::ServerState; + use std::time::Instant; + + #[test] + fn global_server_key_is_valid_only_before_identity_cutover() { + assert!(legacy_server_key_allowed(Some("legacy_unmigrated"))); + assert!(!legacy_server_key_allowed(Some("fresh_uninitialized"))); + assert!(!legacy_server_key_allowed(Some("admin_initialized"))); + assert!(!legacy_server_key_allowed(Some("managed"))); + assert!(!legacy_server_key_allowed(None)); + } + + #[test] + fn personal_credentials_are_limited_to_agent_surfaces() { + for path in [ + "/v1/chat/completions", + "/v1/messages", + "/v1/responses", + "/v1/responses/compact", + "/v1/models", + "/api/integrations/v1/profile", + ] { + assert!(personal_credential_allowed(path), "{path}"); + } + for path in ["/", "/api/status", "/api/stats/runtime", "/api/users"] { + assert!(!personal_credential_allowed(path), "{path}"); + } + for path in [ + "/api/%73tats/summary", + "/api//stats/summary", + "/api/../admin/users", + "/api/stats/accounts", + ] { + assert!(!personal_credential_allowed(path), "{path}"); + } + } + + #[test] + fn unknown_and_encoded_api_paths_never_inherit_public_dashboard_access() { + let public = ServerState { + api_key: Some("test-server-key-12345678".to_owned()), + dashboard_enabled: true, + dashboard_public: true, + dashboard_theme: "default".to_owned(), + dashboard_refresh_interval_s: 60, + configured_server_threads: 1, + database_path: ":memory:".to_owned(), + trusted_proxy_ips: Vec::new(), + allow_insecure_web_auth: false, + started_at: Instant::now(), + }; + for path in [ + "/api/unknown", + "/api/stats/accounts", + "/api/%73tats/summary", + "/api//stats/summary", + "/api/../admin/users", + "/%61pi/stats/summary", + ] { + assert!( + requires_auth(path, &http::Method::GET, &public), + "{path} must fail closed" + ); + } + for path in [ + "/api/stats/summary", + "/api/stats/transcoding", + "/api/timeseries", + ] { + assert!( + !requires_auth(path, &http::Method::GET, &public), + "{path} is an explicit public API" + ); + } + assert!(requires_auth( + "/api/stats/summary", + &http::Method::POST, + &public + )); + } +} diff --git a/rust/src/server/mod.rs b/rust/src/server/mod.rs index 986d5e689..39c4356b4 100644 --- a/rust/src/server/mod.rs +++ b/rust/src/server/mod.rs @@ -6,33 +6,42 @@ //! Handlers invoke one coordinator entry point; routing/retry/finalization //! live in the coordinator, not here. +mod auth; mod dashboard; mod health; mod inference; mod middleware; +use auth::{account_page, login_page, login_submit, logout_submit, password_change}; use dashboard::{ - accounts_page, bandwidth_page, cache_page, events_page, grouped_timeseries_api, latency_page, - model_detail_page, models_page, overview, pings_page, reliability_page, routing_page, - runtime_page, static_chart_js, static_css, static_favicon, static_js, - stats_cache_observability, stats_cache_stability, stats_request_segmentation, - stats_request_shaping, stats_transcoding, summary, sync_accounts, theme_css, timeseries_api, - timeseries_page, traces_page, + accounts_page, assign_user_profile, audit_page, bandwidth_page, cache_page, + create_personal_token, create_profile, create_user, delete_profile, events_page, + grouped_timeseries_api, issue_admin_token, latency_page, model_detail_page, models_page, + my_account_page, my_models_page, overview, pings_page, profiles_page, reliability_page, + remove_instance_budget, remove_profile_rule, remove_user_override, reset_user_password, + revoke_personal_token, revoke_user_tokens, routing_page, runtime_page, set_instance_budget, + set_profile_preference, set_profile_rule, set_user_budget, set_user_override, set_user_state, + static_chart_js, static_css, static_favicon, static_js, stats_cache_observability, + stats_cache_stability, stats_request_segmentation, stats_request_shaping, stats_transcoding, + summary, sync_accounts, theme_css, timeseries_api, timeseries_page, traces_page, + update_profile, usage_page, users_page, }; use health::{ healthz, integration_profile, models_api, readyz, runtime_status, status_api, update_status, + user_budget_api, }; use inference::{chat_completions, messages, responses, responses_compact}; use middleware::{ - admit_inference_body, authenticate, is_loopback_host, map_generation_error, validate_server_key, + admit_inference_body, authenticate, is_loopback_host, map_generation_error, + validate_server_access, validate_server_key, }; -/// Loud startup warning when running without an API key (loopback dev mode -/// only). Non-loopback without a key is already a startup error via -/// `validate_server_key`; loopback without a key is wide open by design and -/// must be operator-visible. See `docs/deployment.md`. -fn warn_if_unauthenticated_loopback(config: &Config) { - if config.resolved_server_api_key().is_none() && is_loopback_host(&config.server.host) { +/// Loud startup warning while a legacy loopback instance remains keyless. +fn warn_if_unauthenticated_loopback(config: &Config, access_state: &str) { + if access_state == "legacy_unmigrated" + && config.resolved_server_api_key().is_none() + && is_loopback_host(&config.server.host) + { tracing::warn!( host = config.server.host.as_str(), "server.api_key is not configured: loopback listener is UNAUTHENTICATED (dev convenience). \ @@ -42,6 +51,14 @@ fn warn_if_unauthenticated_loopback(config: &Config) { } } +fn warn_if_insecure_web_auth(config: &Config) { + if config.security.allow_insecure_web_auth { + tracing::warn!( + "security.allow_insecure_web_auth is enabled: remote browser passwords and session cookies may travel over plaintext HTTP" + ); + } +} + use axum::{ Router, body::{Body, Bytes}, @@ -79,6 +96,8 @@ use std::{path::Path, sync::Arc, time::Instant}; pub enum ServerError { #[error("{0}")] Database(#[from] db::DatabaseError), + #[error("{0}")] + Identity(#[from] crate::identity::IdentityError), #[error("cannot bind listener: {0}")] Bind(#[from] std::io::Error), #[error("invalid server API key configuration")] @@ -653,6 +672,8 @@ pub struct ServerState { dashboard_refresh_interval_s: u64, configured_server_threads: u32, database_path: String, + trusted_proxy_ips: Vec, + allow_insecure_web_auth: bool, started_at: Instant, } @@ -668,6 +689,8 @@ impl ServerState { database_path: Config::runtime_path(&config.database.path) .to_string_lossy() .into_owned(), + trusted_proxy_ips: config.security.trusted_proxies.clone(), + allow_insecure_web_auth: config.security.allow_insecure_web_auth, started_at: Instant::now(), } } @@ -769,7 +792,6 @@ pub async fn run_with_digest( config_path: Option, ) -> Result<(), ServerError> { validate_server_key(&config)?; - warn_if_unauthenticated_loopback(&config); ensure_start_state(&config).await?; if config.server.threads != 1 { tracing::warn!( @@ -786,10 +808,29 @@ pub async fn run_with_digest( .map_err(crate::operations::process::ProcessError::Path)?; database_config.path = database_path.to_string_lossy().into_owned(); let database = db::Database::open(database_config).await?; - if let Err(error) = db::MigrationRunner::new(&database).run().await { + let migration_state = match db::MigrationRunner::new(&database).run().await { + Ok(state) => state, + Err(error) => { + let _ = database.close().await; + return Err(error.into()); + } + }; + if migration_state.applied_this_run.len() == db::MigrationRunner::migrations().len() + && let Err(error) = crate::identity::mark_fresh_uninitialized(&database).await + { let _ = database.close().await; return Err(error.into()); } + let access_state = crate::identity::access_state(&database).await?; + if config.resolved_server_api_key().is_none() && access_state == "fresh_uninitialized" { + let _ = database.close().await; + return Err(ServerError::StartupConflict { + detail: "fresh database requires `eggpool onboard` or `eggpool users bootstrap` before serving".to_owned(), + }); + } + validate_server_access(&config, &access_state)?; + warn_if_unauthenticated_loopback(&config, &access_state); + warn_if_insecure_web_auth(&config); if let Err(error) = sync_accounts(&config, &database).await { let _ = database.close().await; return Err(error); @@ -942,7 +983,6 @@ pub async fn serve_listener( listener: TcpListener, ) -> Result<(), ServerError> { validate_server_key(&config)?; - warn_if_unauthenticated_loopback(&config); let process = match ProcessRuntime::new_with_config(database.clone(), &config) { Ok(process) => process, Err(error) => { @@ -950,10 +990,27 @@ pub async fn serve_listener( return Err(error.into()); } }; - if let Err(error) = db::MigrationRunner::new(&database).run().await { + let migration_state = match db::MigrationRunner::new(&database).run().await { + Ok(state) => state, + Err(error) => { + let _ = database.close().await; + return Err(error.into()); + } + }; + if migration_state.applied_this_run.len() == db::MigrationRunner::migrations().len() { + crate::identity::mark_fresh_uninitialized(&database).await?; + } + let access_state = crate::identity::access_state(&database).await?; + if config.resolved_server_api_key().is_none() && access_state == "fresh_uninitialized" { let _ = database.close().await; - return Err(error.into()); + return Err(ServerError::StartupConflict { + detail: "fresh database requires local administrator bootstrap before serving" + .to_owned(), + }); } + validate_server_access(&config, &access_state)?; + warn_if_unauthenticated_loopback(&config, &access_state); + warn_if_insecure_web_auth(&config); if let Err(error) = sync_accounts(&config, &database).await { let _ = database.close().await; return Err(error); @@ -1010,7 +1067,10 @@ pub async fn serve_listener_with_inference( listener: TcpListener, ) -> Result<(), ServerError> { validate_server_key(&config)?; - warn_if_unauthenticated_loopback(&config); + let access_state = crate::identity::access_state(&database).await?; + validate_server_access(&config, &access_state)?; + warn_if_unauthenticated_loopback(&config, &access_state); + warn_if_insecure_web_auth(&config); let process = match ProcessRuntime::new_with_config(database.clone(), &config) { Ok(process) => process, Err(error) => { @@ -1052,6 +1112,7 @@ pub fn build_router(state: AppState) -> Router { .route("/api/stats/runtime", get(runtime_status)) .route("/api/stats/update", get(update_status)) .route("/api/status", get(status_api)) + .route("/api/user-budget", get(user_budget_api)) .route("/v1/chat/completions", post(chat_completions)) .route("/v1/messages", post(messages)) .route("/v1/responses", post(responses)) @@ -1062,6 +1123,17 @@ pub fn build_router(state: AppState) -> Router { .route("/static/favicon.svg", get(static_favicon)) .route("/static/theme.css", get(theme_css)); + router = router + .route("/login", get(login_page).post(login_submit)) + .route("/account", get(account_page)) + .route("/account/password", post(password_change)) + .route("/my-account", get(my_account_page)) + .route("/my-account/models", get(my_models_page)) + .route("/account/tokens", post(create_personal_token)) + .route("/account/tokens/revoke", post(revoke_personal_token)) + .route("/usage", get(usage_page)) + .route("/logout", post(logout_submit)); + if dashboard { router = router .route("/", get(overview)) @@ -1078,6 +1150,29 @@ pub fn build_router(state: AppState) -> Router { .route("/traces", get(traces_page)) .route("/runtime", get(runtime_page)) .route("/cache", get(cache_page)) + .route("/users", get(users_page)) + .route("/profiles", get(profiles_page)) + .route("/admin/audit", get(audit_page)) + .route("/admin/users/create", post(create_user)) + .route("/admin/users/state", post(set_user_state)) + .route("/admin/users/password", post(reset_user_password)) + .route("/admin/users/token", post(issue_admin_token)) + .route("/admin/users/revoke-tokens", post(revoke_user_tokens)) + .route("/admin/users/profile", post(assign_user_profile)) + .route("/admin/users/override", post(set_user_override)) + .route("/admin/users/override/remove", post(remove_user_override)) + .route("/admin/users/budget", post(set_user_budget)) + .route("/admin/budget/instance", post(set_instance_budget)) + .route( + "/admin/budget/instance/remove", + post(remove_instance_budget), + ) + .route("/admin/profiles/create", post(create_profile)) + .route("/admin/profiles/update", post(update_profile)) + .route("/admin/profiles/delete", post(delete_profile)) + .route("/admin/profiles/rule", post(set_profile_rule)) + .route("/admin/profiles/rule/remove", post(remove_profile_rule)) + .route("/admin/profiles/prefer", post(set_profile_preference)) .route("/api/stats/summary", get(summary)) .route("/api/stats/transcoding", get(stats_transcoding)) .route( @@ -1160,6 +1255,8 @@ mod tests { dashboard_refresh_interval_s: 60, configured_server_threads: 1, database_path: ":memory:".to_owned(), + trusted_proxy_ips: Vec::new(), + allow_insecure_web_auth: false, started_at: Instant::now(), } } @@ -1174,12 +1271,15 @@ mod tests { "/latency", "/runtime", "/api/stats/summary", - "/api/stats/accounts", + "/api/stats/transcoding", "/static/dashboard.css", "/v1/healthz", "/v1/readyz", ] { - assert!(!requires_auth(path, &public), "{path} is public"); + assert!( + !requires_auth(path, &http::Method::GET, &public), + "{path} is public" + ); } for path in [ "/v1/models", @@ -1188,11 +1288,15 @@ mod tests { "/v1/responses", "/v1/responses/compact", "/api/integrations/v1/profile", + "/api/user-budget", "/api/stats/runtime", "/api/stats/update", "/api/status", ] { - assert!(requires_auth(path, &public), "{path} stays authenticated"); + assert!( + requires_auth(path, &http::Method::GET, &public), + "{path} stays authenticated" + ); } } @@ -1204,21 +1308,31 @@ mod tests { "/accounts", "/models", "/api/stats/summary", - "/api/stats/accounts", + "/api/stats/transcoding", ] { - assert!(requires_auth(path, &private), "{path} requires a key"); + assert!( + requires_auth(path, &http::Method::GET, &private), + "{path} requires a key" + ); } for path in ["/static/dashboard.css", "/v1/healthz", "/v1/readyz"] { - assert!(!requires_auth(path, &private), "{path} stays exempt"); + assert!( + !requires_auth(path, &http::Method::GET, &private), + "{path} stays exempt" + ); } for path in [ "/v1/models", "/api/integrations/v1/profile", + "/api/user-budget", "/api/stats/runtime", "/api/stats/update", "/api/status", ] { - assert!(requires_auth(path, &private), "{path} stays authenticated"); + assert!( + requires_auth(path, &http::Method::GET, &private), + "{path} stays authenticated" + ); } } diff --git a/rust/src/user_budget.rs b/rust/src/user_budget.rs new file mode 100644 index 000000000..2dfeed709 --- /dev/null +++ b/rust/src/user_budget.rs @@ -0,0 +1,1222 @@ +//! Durable downstream-user budgets, reservations, and bounded usage reads. + +use thiserror::Error; +use tokio_rusqlite::rusqlite::{OptionalExtension, params}; + +use crate::db::{Database, DatabaseError}; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)] +#[serde(rename_all = "snake_case")] +pub enum BudgetWindow { + Day, + Month, +} + +impl BudgetWindow { + pub const fn as_str(self) -> &'static str { + match self { + Self::Day => "day", + Self::Month => "month", + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, serde::Serialize, serde::Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum BudgetEnforcement { + Hard, + #[default] + Estimated, +} + +impl BudgetEnforcement { + pub const fn as_str(self) -> &'static str { + match self { + Self::Hard => "hard", + Self::Estimated => "estimated", + } + } + + pub fn parse(value: &str) -> Option { + match value { + "hard" => Some(Self::Hard), + "estimated" => Some(Self::Estimated), + _ => None, + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +pub struct UserBudgetRule { + pub id: i64, + pub model_id: Option, + pub window: BudgetWindow, + pub max_requests: Option, + pub max_tokens: Option, + pub max_microdollars: Option, + pub max_concurrent: Option, + pub enforcement: BudgetEnforcement, + pub version: i64, +} + +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +pub struct UserBudgetUsage { + pub model_id: String, + pub window: BudgetWindow, + pub window_start: String, + pub request_count: i64, + pub token_count: i64, + pub cost_microdollars: i64, + pub provider_reported_microdollars: i64, + pub local_estimated_microdollars: i64, + pub reservation_fallback_microdollars: i64, + pub unknown_token_requests: i64, + pub unknown_cost_requests: i64, +} + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct BudgetLimits { + pub requests: Option, + pub tokens: Option, + pub microdollars: Option, + pub concurrent: Option, + pub enforcement: BudgetEnforcement, +} + +#[derive(Debug, Error)] +pub enum UserBudgetError { + #[error("invalid user budget rule")] + InvalidRule, + #[error("an active administrator is required")] + ActorNotAdmin, + #[error("user does not exist or is unavailable")] + UserUnavailable, + #[error("user budget database operation failed: {0}")] + Database(#[from] DatabaseError), +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum MutationResult { + Updated, + Removed(bool), + ActorNotAdmin, + UserUnavailable, +} + +#[derive(Debug, Clone)] +pub struct UserBudgetService { + database: Database, + browser_admin_session: Option, +} + +impl UserBudgetService { + pub fn new(database: Database) -> Self { + Self { + database, + browser_admin_session: None, + } + } + + pub fn with_browser_admin_session(mut self, principal: &crate::web_session::Principal) -> Self { + self.browser_admin_session = + Some(crate::web_session::SessionFence::from_principal(principal)); + self + } + + pub async fn set_rule( + &self, + actor_id: &str, + user_id: &str, + model_id: Option<&str>, + window: BudgetWindow, + limits: BudgetLimits, + ) -> Result<(), UserBudgetError> { + if model_id.is_some_and(|value| value.is_empty() || value.trim() != value) + || [ + limits.requests, + limits.tokens, + limits.microdollars, + limits.concurrent, + ] + .into_iter() + .flatten() + .any(|value| value <= 0) + || [ + limits.requests, + limits.tokens, + limits.microdollars, + limits.concurrent, + ] + .into_iter() + .all(|value| value.is_none()) + { + return Err(UserBudgetError::InvalidRule); + } + let actor_id = actor_id.to_owned(); + let user_id = user_id.to_owned(); + let model_id = model_id.map(str::to_owned); + let window = window.as_str().to_owned(); + let enforcement = limits.enforcement.as_str().to_owned(); + let browser_admin_session = self.browser_admin_session.clone(); + let outcome = self.database + .with_transaction(move |connection| { + let actor = if let Some(fence) = browser_admin_session.as_ref() { + fence.validates_actor(connection, &actor_id, true)? + } else { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE id = ?1 AND role = 'admin' AND state = 'active')", + [&actor_id], + |row| row.get(0), + )? + }; + if !actor { + return Ok(MutationResult::ActorNotAdmin); + } + let user_exists: bool = connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE id = ?1 AND state = 'active')", + [&user_id], + |row| row.get(0), + )?; + if !user_exists { + return Ok(MutationResult::UserUnavailable); + } + connection.execute( + "INSERT INTO user_budget_rules( + user_id, model_id, window, max_requests, max_tokens, + max_microdollars, max_concurrent, enforcement, updated_by) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9) + ON CONFLICT(user_id, ifnull(model_id, ''), window) DO UPDATE SET + max_requests = excluded.max_requests, + max_tokens = excluded.max_tokens, + max_microdollars = excluded.max_microdollars, + max_concurrent = excluded.max_concurrent, + enforcement = excluded.enforcement, + version = user_budget_rules.version + 1, + updated_by = excluded.updated_by, + updated_at = CURRENT_TIMESTAMP", + params![ + user_id, + model_id, + window, + limits.requests, + limits.tokens, + limits.microdollars, + limits.concurrent, + enforcement, + actor_id, + ], + )?; + connection.execute( + "INSERT INTO user_budget_rule_audit( + user_id, actor_id, model_id, window, action, max_requests, + max_tokens, max_microdollars, max_concurrent, enforcement, rule_version) + SELECT user_id, updated_by, model_id, window, 'set', max_requests, + max_tokens, max_microdollars, max_concurrent, enforcement, version + FROM user_budget_rules WHERE user_id = ?1 AND model_id IS ?2 AND window = ?3", + params![user_id, model_id, window], + )?; + Ok(MutationResult::Updated) + }) + .await + .map_err(UserBudgetError::Database)?; + match outcome { + MutationResult::Updated => Ok(()), + MutationResult::ActorNotAdmin => Err(UserBudgetError::ActorNotAdmin), + MutationResult::UserUnavailable => Err(UserBudgetError::UserUnavailable), + MutationResult::Removed(_) => Err(UserBudgetError::InvalidRule), + } + } + + pub async fn remove_rule( + &self, + actor_id: &str, + user_id: &str, + model_id: Option<&str>, + window: BudgetWindow, + ) -> Result { + let actor_id = actor_id.to_owned(); + let user_id = user_id.to_owned(); + let model_id = model_id.map(str::to_owned); + let window = window.as_str().to_owned(); + let browser_admin_session = self.browser_admin_session.clone(); + let outcome = self.database + .with_transaction(move |connection| { + let actor = if let Some(fence) = browser_admin_session.as_ref() { + fence.validates_actor(connection, &actor_id, true)? + } else { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM users WHERE id = ?1 AND role = 'admin' AND state = 'active')", + [&actor_id], + |row| row.get(0), + )? + }; + if !actor { + return Ok(MutationResult::ActorNotAdmin); + } + let previous = connection + .query_row( + "SELECT max_requests, max_tokens, max_microdollars, max_concurrent, enforcement, version + FROM user_budget_rules WHERE user_id = ?1 AND model_id IS ?2 AND window = ?3", + params![user_id, model_id, window], + |row| { + Ok(( + row.get::<_, Option>(0)?, + row.get::<_, Option>(1)?, + row.get::<_, Option>(2)?, + row.get::<_, Option>(3)?, + row.get::<_, String>(4)?, + row.get::<_, i64>(5)?, + )) + }, + ) + .optional()?; + let removed = connection.execute( + "DELETE FROM user_budget_rules WHERE user_id = ?1 AND model_id IS ?2 AND window = ?3", + params![user_id, model_id, window], + )? == 1; + if let Some((requests, tokens, cost, concurrent, enforcement, version)) = previous { + connection.execute( + "INSERT INTO user_budget_rule_audit( + user_id, actor_id, model_id, window, action, max_requests, + max_tokens, max_microdollars, max_concurrent, enforcement, rule_version) + VALUES (?1, ?2, ?3, ?4, 'remove', ?5, ?6, ?7, ?8, ?9, ?10)", + params![ + user_id, + actor_id, + model_id, + window, + requests, + tokens, + cost, + concurrent, + enforcement, + version + 1, + ], + )?; + } + Ok(MutationResult::Removed(removed)) + }) + .await + .map_err(UserBudgetError::Database)?; + match outcome { + MutationResult::Removed(changed) => Ok(changed), + MutationResult::ActorNotAdmin => Err(UserBudgetError::ActorNotAdmin), + MutationResult::Updated | MutationResult::UserUnavailable => { + Err(UserBudgetError::InvalidRule) + } + } + } + + pub async fn rules(&self, user_id: &str) -> Result, UserBudgetError> { + let user_id = user_id.to_owned(); + self.database + .call(move |connection| { + let mut statement = connection.prepare( + "SELECT id, model_id, window, max_requests, max_tokens, + max_microdollars, max_concurrent, enforcement, version + FROM user_budget_rules WHERE user_id = ?1 + ORDER BY model_id IS NOT NULL, model_id, window", + )?; + statement + .query_map([user_id], |row| { + let window: String = row.get(2)?; + let enforcement: String = row.get(7)?; + Ok(UserBudgetRule { + id: row.get(0)?, + model_id: row.get(1)?, + window: parse_window(&window), + max_requests: row.get(3)?, + max_tokens: row.get(4)?, + max_microdollars: row.get(5)?, + max_concurrent: row.get(6)?, + enforcement: BudgetEnforcement::parse(&enforcement) + .unwrap_or(BudgetEnforcement::Estimated), + version: row.get(8)?, + }) + })? + .collect() + }) + .await + .map_err(UserBudgetError::Database) + } + + pub async fn usage( + &self, + user_id: &str, + since_utc: &str, + limit: u16, + ) -> Result, UserBudgetError> { + if since_utc.len() != 10 || limit == 0 || limit > 500 { + return Err(UserBudgetError::InvalidRule); + } + let user_id = user_id.to_owned(); + let since_utc = since_utc.to_owned(); + self.database + .call(move |connection| { + let mut statement = connection.prepare( + "SELECT model_id, window, window_start, request_count, token_count, + cost_microdollars, provider_reported_microdollars, + local_estimated_microdollars, reservation_fallback_microdollars, + unknown_token_requests, unknown_cost_requests + FROM user_budget_usage WHERE user_id = ?1 AND window_start >= ?2 + ORDER BY window_start DESC, model_id LIMIT ?3", + )?; + statement + .query_map(params![user_id, since_utc, i64::from(limit)], |row| { + let window: String = row.get(1)?; + Ok(UserBudgetUsage { + model_id: row.get(0)?, + window: parse_window(&window), + window_start: row.get(2)?, + request_count: row.get(3)?, + token_count: row.get(4)?, + cost_microdollars: row.get(5)?, + provider_reported_microdollars: row.get(6)?, + local_estimated_microdollars: row.get(7)?, + reservation_fallback_microdollars: row.get(8)?, + unknown_token_requests: row.get(9)?, + unknown_cost_requests: row.get(10)?, + }) + })? + .collect() + }) + .await + .map_err(UserBudgetError::Database) + } +} + +#[derive(Debug, Clone)] +pub(crate) struct BudgetPermit { + pub policy_version: i64, + pub day_start: String, + pub month_start: String, +} + +/// Conservative price provenance recorded with each held reservation. +/// Only the operator tariff is ever recorded; estimated holds keep a NULL +/// price source so estimates can never be mistaken for a hard price. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum BudgetPriceSource { + OperatorTariff, +} + +impl BudgetPriceSource { + pub(crate) const fn as_str(self) -> &'static str { + match self { + Self::OperatorTariff => "operator_tariff", + } + } +} + +/// Per-request worst-case proof assembled by the coordinator from admission +/// facts (input/output bounds), the operator tariff snapshot for the selected +/// provider, and nothing else. Every value is an immutable request property; +/// the publication transaction only decides against it, so there is no +/// check-then-dispatch gap. +/// +/// Formal accounting assumptions (M008 corrective): +/// - `input_worst_case_tokens` is a byte-length upper bound on input tokens +/// (every token encodes at least one ingress byte), never an average. +/// - `requested_output_tokens` is the caller's explicit compatible output cap +/// for the request surface; `None` means unbounded or unusable. +/// - One attempt can bill at most `input + output`. The initial hold covers +/// two attempt-equivalents (the concrete attempt plus one selector/overhead +/// equivalent, because selector dispatch is unattributed by design); each +/// retry extends by one equivalent after re-checking. +/// - Hard USD converts worst-case tokens through the operator-configured +/// provider tariff only. Estimator fallbacks (EWMA/family/global generic) +/// are never a hard price. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct BudgetProof { + pub input_worst_case_tokens: i64, + pub requested_output_tokens: Option, + pub tariff_microdollars_per_million_tokens: Option, +} + +/// Initial hard-mode hold covers two single-attempt equivalents; each retry +/// extends by one equivalent after re-checking limits. +pub(crate) const HARD_HOLD_INITIAL_EQUIVALENTS: i128 = 2; +pub(crate) const HARD_HOLD_RETRY_EQUIVALENTS: i128 = 1; +const MICRODOLLARS_PER_MILLION_TOKENS: i128 = 1_000_000; + +/// Token/cost amounts the publication transaction must hold, plus the +/// auditable worst-case facts recorded with the reservation row. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct BudgetHold { + pub tokens: i64, + pub microdollars: i64, + pub enforcement: BudgetEnforcement, + pub bound_tokens: i64, + pub bound_microdollars: i64, + pub price_source: Option, +} + +/// Worst-case tokens billable by one attempt under a hard rule, or `None` +/// when no conservative bound exists (unbounded output, negative input, +/// non-positive or overflowing total). `None` always fails closed. +pub(crate) fn single_attempt_worst_tokens(proof: &BudgetProof) -> Option { + let output = proof.requested_output_tokens.filter(|value| *value > 0)?; + if proof.input_worst_case_tokens < 0 { + return None; + } + let single = i128::from(proof.input_worst_case_tokens) + i128::from(output); + if single <= 0 || single > i128::from(i64::MAX) { + return None; + } + Some(single as i64) +} + +/// Worst-case microdollars for `tokens` at the operator tariff, or `None` +/// when the tariff is unknown/negative or the conversion overflows. +pub(crate) fn tariff_cost_microdollars( + tokens: i64, + tariff_per_million: Option, +) -> Option { + let tariff = tariff_per_million.filter(|value| *value >= 0)?; + if tokens < 0 { + return None; + } + let numerator = i128::from(tokens) * i128::from(tariff); + let cost = numerator / MICRODOLLARS_PER_MILLION_TOKENS + + i128::from(numerator % MICRODOLLARS_PER_MILLION_TOKENS != 0); + if cost > i128::from(i64::MAX) { + return None; + } + Some(cost as i64) +} + +pub(crate) struct BudgetReservationSpec<'a> { + pub request_id: i64, + pub user_id: &'a str, + pub model_id: &'a str, + pub tokens: i64, + pub microdollars: i64, + pub ttl_seconds: i64, + pub hold: BudgetHold, +} + +struct BudgetCheckRequest<'a> { + user_id: &'a str, + permit: &'a BudgetPermit, + new_request: bool, +} + +struct LoadedBudgetRule { + scope_model: Option, + window: String, + max_requests: Option, + max_tokens: Option, + max_cost: Option, + max_concurrent: Option, + enforcement: BudgetEnforcement, +} + +pub(crate) struct BudgetAttemptSettlement { + pub request_id: i64, + pub attempt_id: i64, + pub attempt_transitioned: bool, + pub terminalize_request: bool, + pub input_tokens: i64, + pub output_tokens: i64, + pub cost_microdollars: i64, + pub provider_reported_microdollars: Option, + pub local_estimated_microdollars: Option, + pub tokens_known: bool, + pub cost_known: bool, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum BudgetCheck { + Allowed, + Exceeded, + /// A hard rule applies but its worst case is unprovable (unbounded + /// output, unknown tariff, overflow). Callers must deny before any + /// upstream submission; this is never coerced into an estimate. + NotEnforceable, +} + +pub(crate) fn check_initial( + connection: &tokio_rusqlite::rusqlite::Connection, + user_id: &str, + model_id: &str, + tokens: i64, + microdollars: i64, + proof: Option<&BudgetProof>, +) -> Result<(BudgetCheck, BudgetPermit, BudgetHold), tokio_rusqlite::rusqlite::Error> { + let (policy_version, day_start, month_start): (i64, String, String) = connection.query_row( + "SELECT revision, date('now'), strftime('%Y-%m-01', 'now') FROM policy_state WHERE singleton = 1", + [], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + )?; + let permit = BudgetPermit { + policy_version, + day_start, + month_start, + }; + let rules = load_matching_rules(connection, user_id, model_id)?; + let hard_token = rules + .iter() + .any(|rule| rule.enforcement == BudgetEnforcement::Hard && rule.max_tokens.is_some()); + let hard_cost = rules + .iter() + .any(|rule| rule.enforcement == BudgetEnforcement::Hard && rule.max_cost.is_some()); + let Some(hold) = hold_for_admission(tokens, microdollars, proof, hard_token, hard_cost) else { + return Ok((BudgetCheck::NotEnforceable, permit, BudgetHold::empty())); + }; + let exceeded = rules_exceeded( + connection, + BudgetCheckRequest { + user_id, + permit: &permit, + new_request: true, + }, + &rules, + hold, + tokens, + microdollars, + )?; + Ok(( + if exceeded { + BudgetCheck::Exceeded + } else { + BudgetCheck::Allowed + }, + permit, + hold, + )) +} + +/// Resolve the amounts a new logical request must hold. Hard dimensions use +/// the proof's worst case; estimated dimensions keep the estimator's +/// projection. Returns `None` when a hard dimension is unprovable. +pub(crate) fn hold_for_admission( + estimate_tokens: i64, + estimate_microdollars: i64, + proof: Option<&BudgetProof>, + hard_token: bool, + hard_cost: bool, +) -> Option { + let single = proof.and_then(single_attempt_worst_tokens); + let hold_tokens = if hard_token { + let single = single?; + checked_mul(single, HARD_HOLD_INITIAL_EQUIVALENTS)? + } else { + estimate_tokens.max(0) + }; + let hold_microdollars = if hard_cost { + let tariff = proof.and_then(|proof| proof.tariff_microdollars_per_million_tokens); + // A hard cost cap still needs a conservative token proof when token + // count itself is estimated. Estimates can be below the raw-body + // bound or capped for resource reasons, so they cannot justify a hard + // monetary reservation. + let cost_tokens = checked_mul(single?, HARD_HOLD_INITIAL_EQUIVALENTS)?; + tariff_cost_microdollars(cost_tokens, tariff)? + } else { + estimate_microdollars.max(0) + }; + let enforcement = if hard_token || hard_cost { + BudgetEnforcement::Hard + } else { + BudgetEnforcement::Estimated + }; + Some(BudgetHold { + tokens: hold_tokens, + microdollars: hold_microdollars, + enforcement, + bound_tokens: if hard_token { hold_tokens } else { 0 }, + bound_microdollars: if hard_cost { hold_microdollars } else { 0 }, + price_source: if hard_cost { + Some(BudgetPriceSource::OperatorTariff) + } else { + None + }, + }) +} + +/// Resolve the additional amounts one retry attempt must extend. The initial +/// hold already covers two equivalents, so each retry adds one single-attempt +/// equivalent under hard rules after re-checking limits. +pub(crate) fn hold_for_retry( + estimate_tokens: i64, + estimate_microdollars: i64, + proof: Option<&BudgetProof>, + hard_token: bool, + hard_cost: bool, +) -> Option { + let single = proof.and_then(single_attempt_worst_tokens); + let extra_tokens = if hard_token { + let single = single?; + checked_mul(single, HARD_HOLD_RETRY_EQUIVALENTS)? + } else { + estimate_tokens.max(0) + }; + let extra_microdollars = if hard_cost { + let tariff = proof.and_then(|proof| proof.tariff_microdollars_per_million_tokens); + let cost_tokens = checked_mul(single?, HARD_HOLD_RETRY_EQUIVALENTS)?; + tariff_cost_microdollars(cost_tokens, tariff)? + } else { + estimate_microdollars.max(0) + }; + let enforcement = if hard_token || hard_cost { + BudgetEnforcement::Hard + } else { + BudgetEnforcement::Estimated + }; + Some(BudgetHold { + tokens: extra_tokens, + microdollars: extra_microdollars, + enforcement, + bound_tokens: if hard_token { extra_tokens } else { 0 }, + bound_microdollars: if hard_cost { extra_microdollars } else { 0 }, + price_source: if hard_cost { + Some(BudgetPriceSource::OperatorTariff) + } else { + None + }, + }) +} + +pub(crate) fn checked_mul(value: i64, factor: i128) -> Option { + let product = i128::from(value) * factor; + if product < 0 || product > i128::from(i64::MAX) { + return None; + } + Some(product as i64) +} + +impl BudgetHold { + pub(crate) fn empty() -> Self { + Self { + tokens: 0, + microdollars: 0, + enforcement: BudgetEnforcement::Estimated, + bound_tokens: 0, + bound_microdollars: 0, + price_source: None, + } + } +} + +pub(crate) fn check_retry( + connection: &tokio_rusqlite::rusqlite::Connection, + request_id: i64, + tokens: i64, + microdollars: i64, + proof: Option<&BudgetProof>, +) -> Result<(BudgetCheck, BudgetHold), tokio_rusqlite::rusqlite::Error> { + let reservation = connection + .query_row( + "SELECT user_id, model_id, policy_version, day_start, month_start + FROM user_budget_reservations WHERE request_id = ?1 AND status = 'active'", + [request_id], + |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, i64>(2)?, + row.get::<_, String>(3)?, + row.get::<_, String>(4)?, + )) + }, + ) + .optional()?; + let Some((user_id, model_id, policy_version, day_start, month_start)) = reservation else { + return Ok((BudgetCheck::Exceeded, BudgetHold::empty())); + }; + let permit = BudgetPermit { + policy_version, + day_start, + month_start, + }; + // Hard flags come from the current rules, not the original permit: a + // rule promoted to hard after admission must still gate the retry. + let rules = load_matching_rules(connection, &user_id, &model_id)?; + let hard_token = rules + .iter() + .any(|rule| rule.enforcement == BudgetEnforcement::Hard && rule.max_tokens.is_some()); + let hard_cost = rules + .iter() + .any(|rule| rule.enforcement == BudgetEnforcement::Hard && rule.max_cost.is_some()); + let Some(hold) = hold_for_retry(tokens, microdollars, proof, hard_token, hard_cost) else { + return Ok((BudgetCheck::NotEnforceable, BudgetHold::empty())); + }; + let exceeded = rules_exceeded( + connection, + BudgetCheckRequest { + user_id: &user_id, + permit: &permit, + new_request: false, + }, + &rules, + hold, + tokens, + microdollars, + )?; + Ok(( + if exceeded { + BudgetCheck::Exceeded + } else { + BudgetCheck::Allowed + }, + hold, + )) +} + +pub(crate) fn insert_reservation( + connection: &tokio_rusqlite::rusqlite::Connection, + spec: BudgetReservationSpec<'_>, + permit: &BudgetPermit, +) -> Result<(), tokio_rusqlite::rusqlite::Error> { + connection.execute( + "INSERT INTO user_budget_reservations( + request_id, user_id, model_id, policy_version, day_start, month_start, + held_tokens, held_microdollars, status, expires_at, + enforcement, bound_tokens, bound_microdollars, price_source) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, 'active', + datetime('now', '+' || ?9 || ' seconds'), ?10, ?11, ?12, ?13)", + params![ + spec.request_id, + spec.user_id, + spec.model_id, + permit.policy_version, + permit.day_start, + permit.month_start, + spec.tokens, + spec.microdollars, + spec.ttl_seconds, + spec.hold.enforcement.as_str(), + spec.hold.bound_tokens, + spec.hold.bound_microdollars, + spec.hold.price_source.map(BudgetPriceSource::as_str), + ], + )?; + Ok(()) +} + +pub(crate) fn extend_reservation( + connection: &tokio_rusqlite::rusqlite::Connection, + request_id: i64, + tokens: i64, + microdollars: i64, +) -> Result<(), tokio_rusqlite::rusqlite::Error> { + let changed = connection.execute( + "UPDATE user_budget_reservations SET held_tokens = held_tokens + ?1, + held_microdollars = held_microdollars + ?2 + WHERE request_id = ?3 AND status = 'active'", + params![tokens, microdollars, request_id], + )?; + if changed != 1 { + return Err(tokio_rusqlite::rusqlite::Error::InvalidParameterName( + "active user budget reservation missing".to_owned(), + )); + } + Ok(()) +} + +fn load_matching_rules( + connection: &tokio_rusqlite::rusqlite::Connection, + user_id: &str, + model_id: &str, +) -> Result, tokio_rusqlite::rusqlite::Error> { + let mut statement = connection.prepare( + "SELECT model_id, window, max_requests, max_tokens, max_microdollars, + max_concurrent, enforcement + FROM user_budget_rules WHERE user_id = ?1 AND (model_id IS NULL OR model_id = ?2)", + )?; + statement + .query_map(params![user_id, model_id], |row| { + let enforcement: String = row.get(6)?; + Ok(LoadedBudgetRule { + scope_model: row.get(0)?, + window: row.get(1)?, + max_requests: row.get(2)?, + max_tokens: row.get(3)?, + max_cost: row.get(4)?, + max_concurrent: row.get(5)?, + enforcement: BudgetEnforcement::parse(&enforcement) + .unwrap_or(BudgetEnforcement::Estimated), + }) + })? + .collect() +} + +fn rules_exceeded( + connection: &tokio_rusqlite::rusqlite::Connection, + check: BudgetCheckRequest<'_>, + rules: &[LoadedBudgetRule], + hold: BudgetHold, + estimate_tokens: i64, + estimate_microdollars: i64, +) -> Result { + for rule in rules { + let window_start = if rule.window == "day" { + &check.permit.day_start + } else { + &check.permit.month_start + }; + let window = rule.window.as_str(); + // A hard row enforces its token/cost caps against the worst-case + // hold; an estimated row keeps the estimator projection even when a + // sibling hard row reserves worst-case amounts. Request and + // concurrency caps are strictly enforceable in both modes. + let token_add = if rule.enforcement == BudgetEnforcement::Hard && rule.max_tokens.is_some() + { + hold.tokens + } else { + estimate_tokens.max(0) + }; + let cost_add = if rule.enforcement == BudgetEnforcement::Hard && rule.max_cost.is_some() { + hold.microdollars + } else { + estimate_microdollars.max(0) + }; + let model_filter = rule.scope_model.as_deref(); + let (used_requests, used_tokens, used_cost): (i64, i64, i64) = + if let Some(model) = model_filter { + connection.query_row( + "SELECT coalesce(sum(request_count), 0), coalesce(sum(token_count), 0), + coalesce(sum(cost_microdollars), 0) + FROM user_budget_usage WHERE user_id = ?1 AND model_id = ?2 + AND window = ?3 AND window_start = ?4", + params![check.user_id, model, window, window_start], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + )? + } else { + connection.query_row( + "SELECT coalesce(sum(request_count), 0), coalesce(sum(token_count), 0), + coalesce(sum(cost_microdollars), 0) + FROM user_budget_usage WHERE user_id = ?1 AND window = ?2 AND window_start = ?3", + params![check.user_id, window, window_start], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + )? + }; + let (held_requests, held_tokens, held_cost, active_count): (i64, i64, i64, i64) = + if let Some(model) = model_filter { + connection.query_row( + "SELECT coalesce(sum(CASE WHEN (?3 = 'day' AND day_start = ?4) + OR (?3 = 'month' AND month_start = ?4) + THEN held_requests ELSE 0 END), 0), + coalesce(sum(CASE WHEN (?3 = 'day' AND day_start = ?4) + OR (?3 = 'month' AND month_start = ?4) + THEN held_tokens ELSE 0 END), 0), + coalesce(sum(CASE WHEN (?3 = 'day' AND day_start = ?4) + OR (?3 = 'month' AND month_start = ?4) + THEN held_microdollars ELSE 0 END), 0), count(*) + FROM user_budget_reservations WHERE user_id = ?1 AND model_id = ?2 + AND status = 'active'", + params![check.user_id, model, window, window_start], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?)), + )? + } else { + connection.query_row( + "SELECT coalesce(sum(CASE WHEN (?2 = 'day' AND day_start = ?3) + OR (?2 = 'month' AND month_start = ?3) + THEN held_requests ELSE 0 END), 0), + coalesce(sum(CASE WHEN (?2 = 'day' AND day_start = ?3) + OR (?2 = 'month' AND month_start = ?3) + THEN held_tokens ELSE 0 END), 0), + coalesce(sum(CASE WHEN (?2 = 'day' AND day_start = ?3) + OR (?2 = 'month' AND month_start = ?3) + THEN held_microdollars ELSE 0 END), 0), count(*) + FROM user_budget_reservations WHERE user_id = ?1 AND status = 'active'", + params![check.user_id, window, window_start], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?)), + )? + }; + let request_hold = i64::from(check.new_request); + if rule + .max_requests + .is_some_and(|limit| exceeds(limit, used_requests, held_requests, request_hold)) + || rule + .max_tokens + .is_some_and(|limit| exceeds(limit, used_tokens, held_tokens, token_add)) + || rule + .max_cost + .is_some_and(|limit| exceeds(limit, used_cost, held_cost, cost_add)) + || rule + .max_concurrent + .is_some_and(|limit| exceeds(limit, active_count, i64::from(check.new_request), 0)) + { + return Ok(true); + } + } + Ok(false) +} + +/// Record one transitioned attempt's usage facts idempotently. The attempt +/// table is keyed by attempt/request only, so both user and instance-global +/// settlement fold the same rows. +#[allow(clippy::too_many_arguments)] +pub(crate) fn record_attempt_usage( + connection: &tokio_rusqlite::rusqlite::Connection, + request_id: i64, + attempt_id: i64, + input_tokens: i64, + output_tokens: i64, + cost_microdollars: i64, + provider_reported_microdollars: Option, + local_estimated_microdollars: Option, + tokens_known: bool, + cost_known: bool, +) -> Result<(), tokio_rusqlite::rusqlite::Error> { + let provider_cost = provider_reported_microdollars.unwrap_or(0).max(0); + let local_cost = local_estimated_microdollars + .or_else(|| { + (provider_reported_microdollars.is_none() && cost_microdollars > 0) + .then_some(cost_microdollars) + }) + .unwrap_or(0) + .max(0); + let cost_source = if provider_reported_microdollars.is_some() { + "provider_reported" + } else if local_cost > 0 { + "local_estimated" + } else { + "unknown" + }; + connection.execute( + "INSERT OR IGNORE INTO user_budget_attempt_usage( + attempt_id, request_id, input_tokens, output_tokens, cost_microdollars, + provider_reported_microdollars, local_estimated_microdollars, + cost_source, tokens_known, cost_known) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10)", + params![ + attempt_id, + request_id, + input_tokens.max(0), + output_tokens.max(0), + cost_microdollars.max(0), + provider_cost, + local_cost, + cost_source, + i64::from(tokens_known), + i64::from(cost_known), + ], + )?; + Ok(()) +} + +pub(crate) fn record_attempt_and_maybe_settle( + connection: &tokio_rusqlite::rusqlite::Connection, + settlement: BudgetAttemptSettlement, +) -> Result<(), tokio_rusqlite::rusqlite::Error> { + let BudgetAttemptSettlement { + request_id, + attempt_id, + attempt_transitioned, + terminalize_request, + input_tokens, + output_tokens, + cost_microdollars, + provider_reported_microdollars, + local_estimated_microdollars, + tokens_known, + cost_known, + } = settlement; + // Per-attempt usage facts are request-scoped, not user-scoped: they are + // recorded unconditionally so instance-global settlement (M009) can fold + // the same attempts for unattributed requests. Only the usage-bucket + // fold below is gated on a user reservation. + if attempt_transitioned { + record_attempt_usage( + connection, + request_id, + attempt_id, + input_tokens, + output_tokens, + cost_microdollars, + provider_reported_microdollars, + local_estimated_microdollars, + tokens_known, + cost_known, + )?; + } + let reservation = connection + .query_row( + "SELECT user_id, model_id, day_start, month_start, held_tokens, + held_microdollars, status + FROM user_budget_reservations WHERE request_id = ?1", + [request_id], + |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, String>(2)?, + row.get::<_, String>(3)?, + row.get::<_, i64>(4)?, + row.get::<_, i64>(5)?, + row.get::<_, String>(6)?, + )) + }, + ) + .optional()?; + let Some((user_id, model_id, day_start, month_start, held_tokens, held_cost, status)) = + reservation + else { + return Ok(()); + }; + if !terminalize_request || status == "settled" || status == "released" { + return Ok(()); + } + if status != "active" { + return Err(tokio_rusqlite::rusqlite::Error::InvalidParameterName( + "unknown user budget reservation state".to_owned(), + )); + } + let (observed_tokens, observed_cost, provider_cost, local_cost, token_unknown, cost_unknown): ( + i64, + i64, + i64, + i64, + i64, + i64, + ) = connection.query_row( + "SELECT coalesce(sum(input_tokens + output_tokens), 0), + coalesce(sum(cost_microdollars), 0), + coalesce(sum(provider_reported_microdollars), 0), + coalesce(sum(local_estimated_microdollars), 0), + coalesce(sum(CASE WHEN tokens_known = 0 THEN 1 ELSE 0 END), 0), + coalesce(sum(CASE WHEN cost_known = 0 THEN 1 ELSE 0 END), 0) + FROM user_budget_attempt_usage WHERE request_id = ?1", + [request_id], + |row| { + Ok(( + row.get(0)?, + row.get(1)?, + row.get(2)?, + row.get(3)?, + row.get(4)?, + row.get(5)?, + )) + }, + )?; + let charged_tokens = if token_unknown > 0 { + held_tokens.max(observed_tokens) + } else { + observed_tokens + }; + let charged_cost = if cost_unknown > 0 { + held_cost.max(observed_cost) + } else { + observed_cost + }; + let fallback_cost = (charged_cost - provider_cost - local_cost).max(0); + for (window, start) in [("day", day_start.as_str()), ("month", month_start.as_str())] { + connection.execute( + "INSERT INTO user_budget_usage( + user_id, model_id, window, window_start, request_count, token_count, + cost_microdollars, provider_reported_microdollars, + local_estimated_microdollars, reservation_fallback_microdollars, + unknown_token_requests, unknown_cost_requests) + VALUES (?1, ?2, ?3, ?4, 1, ?5, ?6, ?7, ?8, ?9, ?10, ?11) + ON CONFLICT(user_id, model_id, window, window_start) DO UPDATE SET + request_count = user_budget_usage.request_count + 1, + token_count = user_budget_usage.token_count + excluded.token_count, + cost_microdollars = user_budget_usage.cost_microdollars + excluded.cost_microdollars, + provider_reported_microdollars = user_budget_usage.provider_reported_microdollars + excluded.provider_reported_microdollars, + local_estimated_microdollars = user_budget_usage.local_estimated_microdollars + excluded.local_estimated_microdollars, + reservation_fallback_microdollars = user_budget_usage.reservation_fallback_microdollars + excluded.reservation_fallback_microdollars, + unknown_token_requests = user_budget_usage.unknown_token_requests + excluded.unknown_token_requests, + unknown_cost_requests = user_budget_usage.unknown_cost_requests + excluded.unknown_cost_requests, + updated_at = CURRENT_TIMESTAMP", + params![ + user_id, + model_id, + window, + start, + charged_tokens, + charged_cost, + provider_cost, + local_cost, + fallback_cost, + i64::from(token_unknown > 0), + i64::from(cost_unknown > 0), + ], + )?; + } + connection.execute( + "UPDATE user_budget_reservations SET status = 'settled', settled_at = CURRENT_TIMESTAMP, + observed_tokens = ?1, observed_microdollars = ?2, + token_usage_complete = ?3, cost_usage_complete = ?4 + WHERE request_id = ?5 AND status = 'active'", + params![ + observed_tokens, + observed_cost, + i64::from(token_unknown == 0), + i64::from(cost_unknown == 0), + request_id, + ], + )?; + Ok(()) +} + +pub(crate) fn exceeds(limit: i64, first: i64, second: i64, third: i64) -> bool { + i128::from(first) + i128::from(second) + i128::from(third) > i128::from(limit) +} + +fn parse_window(value: &str) -> BudgetWindow { + if value == "month" { + BudgetWindow::Month + } else { + BudgetWindow::Day + } +} + +#[cfg(test)] +mod hard_cost_tests { + use super::*; + + #[test] + fn hard_cost_holds_round_up_for_initial_and_retry_attempts() { + let proof = BudgetProof { + input_worst_case_tokens: 0, + requested_output_tokens: Some(1), + tariff_microdollars_per_million_tokens: Some(1), + }; + + assert_eq!(tariff_cost_microdollars(1, Some(1)), Some(1)); + assert_eq!(tariff_cost_microdollars(0, Some(1)), Some(0)); + let initial = hold_for_admission(1, 0, Some(&proof), false, true) + .expect("known hard tariff is provable"); + assert_eq!(initial.microdollars, 1); + assert_eq!(initial.bound_microdollars, 1); + + let retry = hold_for_retry(1, 0, Some(&proof), false, true) + .expect("known retry tariff is provable"); + assert_eq!(retry.microdollars, 1); + assert_eq!(retry.bound_microdollars, 1); + + let combined = hold_for_admission(1, 0, Some(&proof), true, true) + .expect("hard token and cost bounds are provable"); + assert_eq!(combined.tokens, 2); + assert_eq!(combined.microdollars, 1); + } + + #[test] + fn cost_hard_holds_require_and_use_conservative_proof_tokens() { + let proof = BudgetProof { + input_worst_case_tokens: 100_000, + requested_output_tokens: Some(100_000), + tariff_microdollars_per_million_tokens: Some(1_000_000), + }; + + let initial = hold_for_admission(1, 0, Some(&proof), false, true) + .expect("hard cost requires a usable token proof"); + assert_eq!(initial.tokens, 1); + assert_eq!(initial.microdollars, 400_000); + assert_eq!(initial.bound_microdollars, 400_000); + + let retry = hold_for_retry(1, 0, Some(&proof), false, true) + .expect("retry cost uses the single-attempt proof bound"); + assert_eq!(retry.tokens, 1); + assert_eq!(retry.microdollars, 200_000); + assert_eq!(retry.bound_microdollars, 200_000); + + let unbounded = BudgetProof { + requested_output_tokens: None, + ..proof + }; + assert!(hold_for_admission(1, 0, Some(&unbounded), false, true).is_none()); + assert!(hold_for_retry(1, 0, Some(&unbounded), false, true).is_none()); + } +} diff --git a/rust/src/web_session.rs b/rust/src/web_session.rs new file mode 100644 index 000000000..f57c423e0 --- /dev/null +++ b/rust/src/web_session.rs @@ -0,0 +1,457 @@ +//! Opaque browser sessions and bounded local login throttling. + +use std::{ + collections::HashMap, + net::IpAddr, + sync::OnceLock, + time::{Duration, Instant}, +}; + +use sha2::{Digest, Sha256}; +use thiserror::Error; +use tokio_rusqlite::rusqlite::OptionalExtension; + +use crate::db::Database; + +const SESSION_PREFIX: &str = "eps1_"; +const LOGIN_WINDOW: Duration = Duration::from_secs(15 * 60); +const LOGIN_USERNAME_LIMIT: u8 = 10; +const LOGIN_IP_LIMIT: u8 = 30; +const PERSONAL_TOKEN_IP_LIMIT: u8 = 30; +const LOGIN_BUCKET_LIMIT: usize = 2048; + +static LOGIN_FAILURES: OnceLock>> = OnceLock::new(); + +type SessionRow = (String, String, String, String, i64, i64, String, String); + +#[derive(Debug, Error)] +pub enum WebSessionError { + #[error("session entropy unavailable")] + Entropy, + #[error("session worker failed")] + Worker, + #[error(transparent)] + Database(#[from] crate::db::DatabaseError), +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Principal { + pub session_id: String, + pub user_id: String, + pub username: String, + pub role: String, + pub must_change_password: bool, + pub auth_epoch: i64, + csrf_digest: String, +} + +/// Transaction-time fence for browser-originated mutations. It binds a +/// mutation to the exact session and privilege snapshot admitted by +/// middleware, so revocation, password reset, role change, or disablement +/// before commit makes the write fail closed. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct SessionFence { + session_id: String, + user_id: String, + role: String, + auth_epoch: i64, + must_change_password: bool, +} + +impl SessionFence { + pub(crate) fn from_principal(principal: &Principal) -> Self { + Self { + session_id: principal.session_id.clone(), + user_id: principal.user_id.clone(), + role: principal.role.clone(), + auth_epoch: principal.auth_epoch, + must_change_password: principal.must_change_password, + } + } + + pub(crate) fn validates_actor( + &self, + connection: &tokio_rusqlite::rusqlite::Connection, + actor_id: &str, + require_admin: bool, + ) -> Result { + if self.must_change_password + || actor_id != self.user_id + || (require_admin && self.role != "admin") + { + return Ok(false); + } + self.validates_current_session(connection, actor_id) + } + + pub(crate) fn validates_current_session( + &self, + connection: &tokio_rusqlite::rusqlite::Connection, + actor_id: &str, + ) -> Result { + if actor_id != self.user_id { + return Ok(false); + } + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM web_sessions s \ + JOIN users u ON u.id = s.user_id \ + WHERE s.id = ?1 AND s.user_id = ?2 AND s.revoked_at IS NULL \ + AND s.expires_at > CURRENT_TIMESTAMP \ + AND s.last_used_at > datetime('now', '-12 hours') \ + AND u.state = 'active' AND u.role = ?3 \ + AND u.must_change_password = ?4 AND u.auth_epoch = ?5)", + ( + &self.session_id, + &self.user_id, + &self.role, + i64::from(self.must_change_password), + self.auth_epoch, + ), + |row| row.get(0), + ) + } + + pub(crate) fn user_id(&self) -> &str { + &self.user_id + } +} + +pub struct IssuedSession { + pub session_token: String, + pub csrf_token: String, +} + +#[derive(Clone, Copy)] +struct LoginBucket { + failures: u8, + started: Instant, +} + +pub async fn issue(database: &Database, user_id: &str) -> Result { + let id = random_hex(16)?; + let selector = random_hex(8)?; + let secret = random_hex(32)?; + let session_token = format!("{SESSION_PREFIX}{selector}_{secret}"); + let token_digest = digest(&session_token); + let csrf_token = random_hex(32)?; + let csrf_digest = digest(&csrf_token); + let id_for_db = id.clone(); + let selector_for_db = selector.clone(); + let user_id = user_id.to_owned(); + database + .with_transaction(move |connection| { + connection.execute( + "INSERT INTO web_sessions(id, user_id, selector, digest, csrf_digest, expires_at) \ + SELECT ?1, id, ?2, ?3, ?4, datetime('now', '+7 days') FROM users WHERE id = ?5 AND state = 'active'", + (&id_for_db, &selector_for_db, &token_digest, &csrf_digest, &user_id), + )?; + if connection.changes() != 1 { + return Err(tokio_rusqlite::rusqlite::Error::QueryReturnedNoRows); + } + Ok(()) + }) + .await?; + Ok(IssuedSession { + session_token, + csrf_token, + }) +} + +pub async fn authenticate( + database: &Database, + session_token: &str, +) -> Result, WebSessionError> { + let Some((selector, _secret)) = parse_session_token(session_token) else { + return Ok(None); + }; + let selector = selector.to_owned(); + let expected = digest(session_token); + database + .with_transaction(move |connection| { + let row: Option = connection.query_row( + "SELECT s.id, u.id, u.username, u.role, u.must_change_password, u.auth_epoch, s.digest, s.csrf_digest \ + FROM web_sessions s JOIN users u ON u.id = s.user_id \ + WHERE s.selector = ?1 AND s.revoked_at IS NULL \ + AND s.expires_at > CURRENT_TIMESTAMP \ + AND s.last_used_at > datetime('now', '-12 hours') AND u.state = 'active'", + [&selector], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?, row.get(4)?, row.get(5)?, row.get(6)?, row.get(7)?)), + ).optional()?; + let Some((session_id, user_id, username, role, must_change_password, auth_epoch, stored, csrf_digest)) = row else { + return Ok(None); + }; + if !constant_time_equal(stored.as_bytes(), expected.as_bytes()) { + return Ok(None); + } + connection.execute( + "UPDATE web_sessions SET last_used_at = CURRENT_TIMESTAMP \ + WHERE selector = ?1 AND last_used_at < datetime('now', '-15 minutes')", + [&selector], + )?; + Ok(Some(Principal { + session_id, + user_id, + username, + role, + must_change_password: must_change_password != 0, + auth_epoch, + csrf_digest, + })) + }) + .await + .map_err(WebSessionError::Database) +} + +pub fn verify_csrf(principal: &Principal, csrf_token: &str) -> bool { + let supplied = digest(csrf_token); + constant_time_equal(principal.csrf_digest.as_bytes(), supplied.as_bytes()) +} + +pub async fn revoke(database: &Database, session_token: &str) -> Result<(), WebSessionError> { + let Some((selector, _secret)) = parse_session_token(session_token) else { + return Ok(()); + }; + let selector = selector.to_owned(); + let expected = digest(session_token); + database + .with_transaction(move |connection| { + let stored: Option = connection.query_row( + "SELECT digest FROM web_sessions WHERE selector = ?1 AND revoked_at IS NULL", + [&selector], + |row| row.get(0), + ).optional()?; + if stored.as_deref().is_some_and(|stored| constant_time_equal(stored.as_bytes(), expected.as_bytes())) { + connection.execute( + "UPDATE web_sessions SET revoked_at = CURRENT_TIMESTAMP WHERE selector = ?1 AND revoked_at IS NULL", + [&selector], + )?; + } + Ok(()) + }) + .await?; + Ok(()) +} + +pub async fn revoke_user_sessions( + database: &Database, + user_id: &str, +) -> Result<(), WebSessionError> { + let user_id = user_id.to_owned(); + database + .with_transaction(move |connection| { + connection.execute( + "UPDATE web_sessions SET revoked_at = CURRENT_TIMESTAMP WHERE user_id = ?1 AND revoked_at IS NULL", + [&user_id], + )?; + Ok(()) + }) + .await?; + Ok(()) +} + +pub fn session_cookie_value(headers: &axum::http::HeaderMap) -> Option { + cookie_value(headers, "eggpool_session") +} + +pub fn csrf_cookie_value(headers: &axum::http::HeaderMap) -> Option { + cookie_value(headers, "eggpool_csrf") +} + +fn cookie_value(headers: &axum::http::HeaderMap, name: &str) -> Option { + headers + .get(axum::http::header::COOKIE)? + .to_str() + .ok()? + .split(';') + .filter_map(|part| part.trim().split_once('=')) + .find_map(|(key, value)| (key == name).then(|| value.to_owned())) +} + +pub fn session_set_cookie(token: &str, secure: bool) -> String { + format!( + "eggpool_session={token}; Path=/; HttpOnly; SameSite=Lax; Max-Age=604800{}", + if secure { "; Secure" } else { "" } + ) +} + +pub fn csrf_set_cookie(token: &str, secure: bool) -> String { + format!( + "eggpool_csrf={token}; Path=/; HttpOnly; SameSite=Strict; Max-Age=604800{}", + if secure { "; Secure" } else { "" } + ) +} + +pub fn clear_cookies(secure: bool) -> [String; 2] { + let secure = if secure { "; Secure" } else { "" }; + [ + format!("eggpool_session=; Path=/; HttpOnly; SameSite=Lax; Max-Age=0{secure}"), + format!("eggpool_csrf=; Path=/; HttpOnly; SameSite=Strict; Max-Age=0{secure}"), + ] +} + +pub fn request_peer_ip(request: &axum::http::Request) -> Option { + request + .extensions() + .get::() + .and_then(|info| info.0.remote_addr) + .map(|address| address.ip()) +} + +pub fn login_allowed(username: &str, peer: Option) -> bool { + let username_key = username_throttle_key(username, peer); + let ip_key = throttle_key( + "ip", + &peer.map_or_else(|| "unknown".to_owned(), |ip| ip.to_string()), + ); + let now = Instant::now(); + let mut failures = LOGIN_FAILURES + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + failures.retain(|_, bucket| now.duration_since(bucket.started) < LOGIN_WINDOW); + failures + .get(&username_key) + .is_none_or(|bucket| bucket.failures < LOGIN_USERNAME_LIMIT) + && failures + .get(&ip_key) + .is_none_or(|bucket| bucket.failures < LOGIN_IP_LIMIT) +} + +pub fn record_login_failure(username: &str, peer: Option) { + let keys = [ + (username_throttle_key(username, peer), LOGIN_USERNAME_LIMIT), + ( + throttle_key( + "ip", + &peer.map_or_else(|| "unknown".to_owned(), |ip| ip.to_string()), + ), + LOGIN_IP_LIMIT, + ), + ]; + let now = Instant::now(); + let mut failures = LOGIN_FAILURES + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + failures.retain(|_, bucket| now.duration_since(bucket.started) < LOGIN_WINDOW); + for (key, limit) in keys { + if let Some(bucket) = failures.get_mut(&key) { + bucket.failures = bucket.failures.saturating_add(1).min(limit); + } else { + if failures.len() >= LOGIN_BUCKET_LIMIT + && let Some(oldest) = failures + .iter() + .min_by_key(|(_, bucket)| bucket.started) + .map(|(key, _)| key.clone()) + { + failures.remove(&oldest); + } + failures.insert( + key, + LoginBucket { + failures: 1, + started: now, + }, + ); + } + } +} + +pub fn personal_token_attempt_allowed(peer: Option) -> bool { + let key = throttle_key( + "personal-token-ip", + &peer.map_or_else(|| "unknown".to_owned(), |ip| ip.to_string()), + ); + let now = Instant::now(); + let mut failures = LOGIN_FAILURES + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + failures.retain(|_, bucket| now.duration_since(bucket.started) < LOGIN_WINDOW); + failures + .get(&key) + .is_none_or(|bucket| bucket.failures < PERSONAL_TOKEN_IP_LIMIT) +} + +pub fn record_personal_token_failure(peer: Option) { + let key = throttle_key( + "personal-token-ip", + &peer.map_or_else(|| "unknown".to_owned(), |ip| ip.to_string()), + ); + let now = Instant::now(); + let mut failures = LOGIN_FAILURES + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + failures.retain(|_, bucket| now.duration_since(bucket.started) < LOGIN_WINDOW); + if let Some(bucket) = failures.get_mut(&key) { + bucket.failures = bucket + .failures + .saturating_add(1) + .min(PERSONAL_TOKEN_IP_LIMIT); + return; + } + if failures.len() >= LOGIN_BUCKET_LIMIT + && let Some(oldest) = failures + .iter() + .min_by_key(|(_, bucket)| bucket.started) + .map(|(key, _)| key.clone()) + { + failures.remove(&oldest); + } + failures.insert( + key, + LoginBucket { + failures: 1, + started: now, + }, + ); +} + +fn username_throttle_key(username: &str, peer: Option) -> String { + let peer = peer.map_or_else(|| "unknown".to_owned(), |ip| ip.to_string()); + throttle_key( + "username", + &format!("{}:{peer}", username.trim().to_ascii_lowercase()), + ) +} + +fn throttle_key(kind: &str, value: &str) -> String { + digest(&format!("{kind}:{value}")) +} + +fn parse_session_token(token: &str) -> Option<(&str, &str)> { + let rest = token.strip_prefix(SESSION_PREFIX)?; + let (selector, secret) = rest.split_once('_')?; + if selector.len() != 16 + || !selector.bytes().all(|byte| byte.is_ascii_hexdigit()) + || secret.len() != 64 + || !secret.bytes().all(|byte| byte.is_ascii_hexdigit()) + || token.len() > 128 + { + return None; + } + Some((selector, secret)) +} + +fn random_hex(length: usize) -> Result { + let mut bytes = vec![0; length]; + getrandom::fill(&mut bytes).map_err(|_| WebSessionError::Entropy)?; + Ok(bytes.iter().map(|byte| format!("{byte:02x}")).collect()) +} + +fn digest(value: &str) -> String { + Sha256::digest(value.as_bytes()) + .iter() + .map(|byte| format!("{byte:02x}")) + .collect() +} + +fn constant_time_equal(left: &[u8], right: &[u8]) -> bool { + let mut difference = (left.len() ^ right.len()) as u8; + for index in 0..left.len().max(right.len()) { + difference |= + left.get(index).copied().unwrap_or(0) ^ right.get(index).copied().unwrap_or(0); + } + difference == 0 +} diff --git a/rust/src/wire/runtime.rs b/rust/src/wire/runtime.rs index 5ad51972f..8d6c9d1eb 100644 --- a/rust/src/wire/runtime.rs +++ b/rust/src/wire/runtime.rs @@ -765,6 +765,7 @@ impl WireRuntime { raw_body_bytes: admission.raw_body_bytes, reservation_tokens: admission.reservation_tokens, context_tokens: admission.context_tokens, + budget_facts: admission.budget_facts, }; Ok(PreparedRequest { identity, diff --git a/rust/tests/build_manifest.rs b/rust/tests/build_manifest.rs index a0da89836..87a44019c 100644 --- a/rust/tests/build_manifest.rs +++ b/rust/tests/build_manifest.rs @@ -28,7 +28,7 @@ fn semantically_equivalent_reformatted_manifest_is_accepted() { let reformatted = serde_json::to_string(&value).expect("compact JSON"); let parsed = build_support::parse_checksums(&reformatted); - assert_eq!(parsed.len(), 54); + assert_eq!(parsed.len(), 62); assert_eq!( parsed.get("0001_initial.sql").map(String::as_str), Some("2aa5800147fc3ef8cc35591c44564244dba138b4dbc92fc1779e4f88813912ad") @@ -55,8 +55,8 @@ fn rust_owned_runtime_assets_are_complete_and_hash_locked() { serde_json::from_str(include_str!("../assets/runtime-manifest.json")) .expect("runtime asset manifest is valid JSON"); assert_eq!(manifest.manifest_version, "runtime-assets.v1"); - assert_eq!(manifest.schema_migration_count, 54); - assert!(!manifest.schema_change); + assert_eq!(manifest.schema_migration_count, 62); + assert!(manifest.schema_change); let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")); let mut categories = BTreeSet::new(); diff --git a/rust/tests/cli_contract.rs b/rust/tests/cli_contract.rs index 2f562091b..47a939418 100644 --- a/rust/tests/cli_contract.rs +++ b/rust/tests/cli_contract.rs @@ -60,7 +60,7 @@ fn rust_command_options(root: &mut Command, path: &str) -> Vec { #[test] fn rust_parser_has_one_entry_for_every_frozen_python_command() { let expected = fixture_commands(); - assert_eq!(expected.len(), 65); + assert_eq!(expected.len(), 72); let mut root = Cli::command(); for (path, options) in expected { assert_eq!(rust_command_options(&mut root, &path), options, "{path}"); @@ -128,3 +128,198 @@ fn configremote_parses_targets_and_options() { // Missing target fails closed. assert!(Cli::try_parse_from(["eggpool", "configremote"]).is_err()); } + +#[test] +fn local_model_policy_profile_commands_parse_with_exact_effects() { + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "policy", + "create", + "restricted", + "Restricted", + "deny", + ]) + .is_ok() + ); + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "policy", + "rule", + "restricted", + "model-a/provider-a", + "allow", + ]) + .is_ok() + ); + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "policy", + "assign", + "member-a", + "restricted", + ]) + .is_ok() + ); + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "policy", + "rule", + "restricted", + "model-a", + "maybe", + ]) + .is_err() + ); +} + +#[test] +fn local_user_budget_commands_parse_with_bounded_windows() { + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "budget", + "set", + "member-a", + "--window", + "day", + "--max-requests", + "25", + "--max-tokens", + "100000", + "--max-concurrent", + "2", + ]) + .is_ok() + ); + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "budget", + "remove", + "member-a", + "--model-id", + "model-a", + "--window", + "month", + ]) + .is_ok() + ); + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "budget", + "set", + "member-a", + "--window", + "rolling", + "--max-requests", + "25", + ]) + .is_err() + ); + // M008: enforcement mode is explicit and bounded; the default stays + // estimated so existing rules are never silently promoted to hard. + let estimated = Cli::try_parse_from([ + "eggpool", + "users", + "budget", + "set", + "member-a", + "--window", + "day", + "--max-tokens", + "100000", + ]) + .expect("default enforcement parses"); + let rendered = format!("{:?}", estimated.command); + assert!( + rendered.contains("enforcement: \"estimated\""), + "budget set defaults to estimated enforcement: {rendered}" + ); + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "budget", + "set", + "member-a", + "--window", + "day", + "--max-tokens", + "100000", + "--enforcement", + "hard", + ]) + .is_ok() + ); + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "budget", + "set", + "member-a", + "--window", + "day", + "--max-tokens", + "100000", + "--enforcement", + "guaranteed", + ]) + .is_err() + ); + // M009: instance ceilings share the window/enforcement contract without + // a user argument. + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "global-budget", + "set", + "--window", + "month", + "--max-requests", + "1000", + "--enforcement", + "hard", + ]) + .is_ok() + ); + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "global-budget", + "remove", + "--window", + "month", + ]) + .is_ok() + ); + assert!( + Cli::try_parse_from([ + "eggpool", + "users", + "global-budget", + "set", + "--window", + "month", + "--max-requests", + "1000", + "--enforcement", + "guaranteed", + ]) + .is_err() + ); +} diff --git a/rust/tests/coordinator_boundaries.rs b/rust/tests/coordinator_boundaries.rs index ef9ed515c..301bcafd1 100644 --- a/rust/tests/coordinator_boundaries.rs +++ b/rust/tests/coordinator_boundaries.rs @@ -301,6 +301,12 @@ fn attempt_preparation_expands_path_and_never_debugs_credentials() { client_protocol: "openai".into(), upstream_protocol: "openai".into(), attempt_number: 1, + user_id: None, + credential_id: None, + auth_epoch: None, + principal_kind: eggpool::identity::PrincipalKind::LegacySharedKey, + request_kind: eggpool::identity::AttributedRequestKind::Logical, + parent_proxy_request_id: None, }; let attempt = builder .prepare(AttemptInput { diff --git a/rust/tests/coordinator_c011.rs b/rust/tests/coordinator_c011.rs index 929e5e748..2e4655bb3 100644 --- a/rust/tests/coordinator_c011.rs +++ b/rust/tests/coordinator_c011.rs @@ -2822,6 +2822,12 @@ async fn c011_security_dependency_resource_audit() { client_protocol: "openai".into(), upstream_protocol: "openai".into(), attempt_number: 1, + user_id: None, + credential_id: None, + auth_epoch: None, + principal_kind: eggpool::identity::PrincipalKind::LegacySharedKey, + request_kind: eggpool::identity::AttributedRequestKind::Logical, + parent_proxy_request_id: None, }, provider, account_api_key: Some("account-secret-xyz".into()), diff --git a/rust/tests/coordinator_c013.rs b/rust/tests/coordinator_c013.rs index e3c4d63ca..92e80eeaf 100644 --- a/rust/tests/coordinator_c013.rs +++ b/rust/tests/coordinator_c013.rs @@ -514,6 +514,12 @@ async fn c013_attempt_submission_observes_native_alias_and_one_request() { client_protocol: "openai".into(), upstream_protocol: "openai".into(), attempt_number: 1, + user_id: None, + credential_id: None, + auth_epoch: None, + principal_kind: eggpool::identity::PrincipalKind::LegacySharedKey, + request_kind: eggpool::identity::AttributedRequestKind::Logical, + parent_proxy_request_id: None, }; let mut incoming = http::HeaderMap::new(); incoming.insert("x-allowed", "forwarded".parse().unwrap()); diff --git a/rust/tests/coordinator_publication.rs b/rust/tests/coordinator_publication.rs index 557367e21..fabb97015 100644 --- a/rust/tests/coordinator_publication.rs +++ b/rust/tests/coordinator_publication.rs @@ -17,6 +17,7 @@ use eggpool::{ PublicationStage, }, db::{Account, Database, DatabaseConfig, MigrationRunner}, + identity::{AttributedRequestKind, AuthenticatedPrincipal, PrincipalKind}, quota::{AccountQuota, QuotaEstimator}, routing::{EligibilityPolicy, RoutingRequestFacts, RoutingRouter}, }; @@ -176,6 +177,248 @@ async fn publication_commits_all_rows_and_converts_the_claim_once() { fixture.database.close().await.expect("database closes"); } +#[tokio::test] +async fn publication_persists_owner_and_rejects_cross_user_request_id_replay() { + let fixture = fixture().await; + fixture + .database + .call(|connection| { + for (user_id, username, credential_id) in [ + ("user-a", "user-a", "credential-a"), + ("user-b", "user-b", "credential-b"), + ] { + connection.execute( + "INSERT INTO users\ + (id, username, normalized_username, role, password_hash)\ + VALUES (?1, ?2, ?2, 'member', 'test-hash')", + (user_id, username), + )?; + connection.execute( + "INSERT INTO user_credentials\ + (id, user_id, label, selector, digest)\ + VALUES (?1, ?2, 'test', ?1, 'test-digest')", + (credential_id, user_id), + )?; + } + Ok(()) + }) + .await + .unwrap(); + let service = PublicationService::new(fixture.database.clone()); + let attribution = AuthenticatedPrincipal { + user_id: Some("user-a".into()), + credential_id: Some("credential-a".into()), + role: Some("agent".into()), + scopes: vec!["inference".into()], + auth_epoch: Some(1), + kind: PrincipalKind::AgentToken, + } + .request_attribution(); + let outcome = service + .publish_attributed(claim(&fixture).await, input(1), Some(attribution)) + .await + .expect("attributed publication succeeds"); + let PublicationOutcome::Published(published) = outcome else { + panic!("expected a new publication"); + }; + assert_eq!(published.identity.user_id.as_deref(), Some("user-a")); + assert_eq!( + published.identity.credential_id.as_deref(), + Some("credential-a") + ); + assert_eq!(published.identity.auth_epoch, Some(1)); + assert_eq!( + published.identity.request_kind, + AttributedRequestKind::Logical + ); + + let persisted = fixture + .database + .call(|connection| { + connection.query_row( + "SELECT user_id, credential_id, auth_epoch, principal_kind, request_kind \ + FROM requests WHERE proxy_request_id = 'proxy-c002'", + [], + |row| { + Ok(( + row.get::<_, Option>(0)?, + row.get::<_, Option>(1)?, + row.get::<_, Option>(2)?, + row.get::<_, String>(3)?, + row.get::<_, String>(4)?, + )) + }, + ) + }) + .await + .unwrap(); + assert_eq!( + persisted, + ( + Some("user-a".into()), + Some("credential-a".into()), + Some(1), + "agent_token".into(), + "logical".into() + ) + ); + + let other_user = AuthenticatedPrincipal { + user_id: Some("user-b".into()), + credential_id: Some("credential-b".into()), + role: Some("agent".into()), + scopes: vec!["inference".into()], + auth_epoch: Some(1), + kind: PrincipalKind::AgentToken, + } + .request_attribution(); + let replay = service + .publish_attributed(claim(&fixture).await, input(1), Some(other_user)) + .await; + assert!(matches!( + replay, + Err(PublicationError::DuplicateConflict { .. }) + )); + + published + .claim + .release_active_claim() + .expect("claim release succeeds"); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn selector_publication_keeps_user_owner_and_parent_link() { + let fixture = fixture().await; + fixture + .database + .call(|connection| { + connection.execute( + "INSERT INTO users\ + (id, username, normalized_username, role, password_hash)\ + VALUES ('user-a', 'user-a', 'user-a', 'member', 'test-hash')", + [], + )?; + connection.execute( + "INSERT INTO user_credentials\ + (id, user_id, label, selector, digest)\ + VALUES ('credential-a', 'user-a', 'test', 'selector-a', 'test-digest')", + [], + )?; + Ok(()) + }) + .await + .unwrap(); + let attribution = AuthenticatedPrincipal { + user_id: Some("user-a".into()), + credential_id: Some("credential-a".into()), + role: Some("agent".into()), + scopes: vec!["inference".into()], + auth_epoch: Some(1), + kind: PrincipalKind::AgentToken, + } + .selector_attribution("parent-proxy-id"); + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed(claim(&fixture).await, input(1), Some(attribution)) + .await + .expect("selector publication succeeds"); + let PublicationOutcome::Published(published) = outcome else { + panic!("expected a new publication"); + }; + assert_eq!(published.identity.user_id.as_deref(), Some("user-a")); + assert_eq!( + published.identity.request_kind, + AttributedRequestKind::InternalSelector + ); + assert_eq!( + published.identity.parent_proxy_request_id.as_deref(), + Some("parent-proxy-id") + ); + let persisted = fixture + .database + .call(|connection| { + connection.query_row( + "SELECT user_id, request_kind, parent_proxy_request_id \ + FROM requests WHERE proxy_request_id = 'proxy-c002'", + [], + |row| { + Ok(( + row.get::<_, Option>(0)?, + row.get::<_, String>(1)?, + row.get::<_, Option>(2)?, + )) + }, + ) + }) + .await + .unwrap(); + assert_eq!( + persisted, + ( + Some("user-a".into()), + "internal_selector".into(), + Some("parent-proxy-id".into()) + ) + ); + published + .claim + .release_active_claim() + .expect("claim release"); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn principal_epoch_or_credential_revocation_before_publication_fails_closed() { + let fixture = fixture().await; + fixture + .database + .call(|connection| { + connection.execute( + "INSERT INTO users\ + (id, username, normalized_username, role, password_hash)\ + VALUES ('user-a', 'user-a', 'user-a', 'member', 'test-hash')", + [], + )?; + connection.execute( + "INSERT INTO user_credentials\ + (id, user_id, label, selector, digest)\ + VALUES ('credential-a', 'user-a', 'test', 'selector-a', 'test-digest')", + [], + )?; + Ok(()) + }) + .await + .unwrap(); + let attribution = AuthenticatedPrincipal { + user_id: Some("user-a".into()), + credential_id: Some("credential-a".into()), + role: Some("agent".into()), + scopes: vec!["inference".into()], + auth_epoch: Some(1), + kind: PrincipalKind::AgentToken, + } + .request_attribution(); + let claim = claim(&fixture).await; + fixture + .database + .call(|connection| { + connection.execute( + "UPDATE user_credentials SET revoked_at = CURRENT_TIMESTAMP \ + WHERE id = 'credential-a'", + [], + )?; + Ok(()) + }) + .await + .unwrap(); + let result = PublicationService::new(fixture.database.clone()) + .publish_attributed(claim, input(1), Some(attribution)) + .await; + assert!(matches!(result, Err(PublicationError::PrincipalChanged))); + assert_eq!(row_counts(&fixture.database).await, (0, 0, 0, 0)); + fixture.database.close().await.unwrap(); +} + #[tokio::test] async fn every_precommit_failure_rolls_back_the_complete_publication() { for stage in [ @@ -227,7 +470,7 @@ async fn duplicate_attempt_observes_existing_identity_without_row_fanout() { let PublicationOutcome::AlreadyPublished(identity) = second else { panic!("second publication must observe the first"); }; - assert_eq!(identity, first.identity); + assert_eq!(*identity, first.identity); assert_eq!(row_counts(&fixture.database).await, (1, 1, 1, 1)); assert_eq!(fixture.router.active_request_count("account-a"), 0); fixture.database.close().await.expect("database closes"); diff --git a/rust/tests/database_compatibility.rs b/rust/tests/database_compatibility.rs index b3dbe5aac..09b80bfe5 100644 --- a/rust/tests/database_compatibility.rs +++ b/rust/tests/database_compatibility.rs @@ -1,4 +1,5 @@ use std::{ + collections::BTreeSet, fs, path::PathBuf, process::Command, @@ -39,8 +40,8 @@ async fn fresh_database_migrates_idempotently_and_preserves_pragmas() { .run() .await .expect("migrations apply"); - assert_eq!(first.applied_versions.len(), 54); - assert_eq!(first.applied_this_run.len(), 54); + assert_eq!(first.applied_versions.len(), 62); + assert_eq!(first.applied_this_run.len(), 62); database .quick_check() .await @@ -258,7 +259,7 @@ async fn python_historical_fixture_upgrades_and_repositories_round_trip() { .await .expect("current migrations apply"); assert_eq!(state.applied_this_run.first(), Some(&12)); - assert_eq!(state.applied_this_run.last(), Some(&54)); + assert_eq!(state.applied_this_run.last(), Some(&62)); let account = AccountRepository::new(&database) .get_by_name("historical-account") @@ -504,3 +505,524 @@ async fn current_database_can_be_opened_read_only_without_writes() { read_only.close().await.expect("read-only database closes"); fs::remove_file(&path).expect("temporary database removed"); } + +// --------------------------------------------------------------------------- +// M010: an independent byte-valid schema-54 image restores, migrates +// forward, and recovers identity/policy/budgets without fabricating +// history. The frozen image is `tests/fixtures/database/schema54- +// pre-upgrade.sql` (SHA-256 recorded in the M010 closure); tests never run +// migrations 1..54 themselves — only the production 55..62 upgrade path. +// --------------------------------------------------------------------------- + +use eggpool::{ + Config, + accounts::{AccountRegistry, CredentialStore}, + catalog::{ModelCatalogCache, ModelInput, ProtocolResolutionStatus}, + coordinator::{ + CrashReconciler, DurableFinalizer, FinalizationData, FinalizationOutcome, PublicationInput, + PublicationOutcome, PublicationService, + }, + db::Account, + identity::{ + AuthenticatedPrincipal, PrincipalKind, access_state, admin_reset_password, bootstrap_admin, + create_member, issue_personal_api_token, revoke_user_api_tokens, verify_api_token, + verify_password, + }, + instance_budget::InstanceBudgetService, + model_policy::ModelPolicyService, + operations::backup::BackupService, + quota::{AccountQuota, QuotaEstimator}, + routing::{EligibilityPolicy, RoutingRequestFacts, RoutingRouter}, + user_budget::{BudgetLimits, BudgetWindow, UserBudgetService}, +}; + +async fn load_schema54_image(path: &std::path::Path) -> Database { + let database = open_database(path).await; + database + .call(|connection| { + connection.execute_batch(include_str!( + "../../tests/fixtures/database/schema54-pre-upgrade.sql" + )) + }) + .await + .expect("frozen schema-54 image loads"); + database +} + +async fn table_count(database: &Database, table: &str) -> i64 { + let query = format!("SELECT COUNT(*) FROM {table}"); + database + .call(move |connection| connection.query_row(&query, [], |row| row.get::<_, i64>(0))) + .await + .expect("count reads") +} + +async fn prove_schema54_source(database: &Database) { + // Source identity: the ledger names exactly migrations 1..54, the + // embedded checksums validate, and the image is internally consistent. + let ledger: Vec<(i64, String)> = database + .call(|connection| { + connection + .prepare("SELECT version, name FROM _migrations ORDER BY version")? + .query_map([], |row| Ok((row.get(0)?, row.get(1)?)))? + .collect::, _>>() + }) + .await + .expect("ledger reads"); + assert_eq!(ledger.len(), 54); + assert_eq!(ledger.first().map(|row| row.0), Some(1)); + assert_eq!(ledger.last().map(|row| row.0), Some(54)); + assert!( + ledger + .iter() + .all(|row| { row.1.starts_with(&format!("{:04}_", row.0)) && row.1.ends_with(".sql") }), + "every ledger entry carries its historical migration filename" + ); + MigrationRunner::validate_embedded_checksums().expect("canonical checksums validate"); + let integrity: String = database + .call(|connection| connection.query_row("PRAGMA integrity_check", [], |row| row.get(0))) + .await + .expect("integrity_check runs"); + assert_eq!(integrity, "ok"); + // Audited legacy content: upstream accounts, request/token-usage + // history, and quarantine state — and no post-54 user/policy tables. + assert_eq!(table_count(database, "accounts").await, 3); + assert_eq!(table_count(database, "requests").await, 2); + assert_eq!(table_count(database, "reservations").await, 2); + assert_eq!(table_count(database, "model_quarantine").await, 2); + assert_eq!(table_count(database, "catalog_refresh_state").await, 2); + let modern: Vec = database + .call(|connection| { + connection + .prepare( + "SELECT name FROM sqlite_master WHERE type = 'table' AND name IN \ + ('users', 'user_budget_rules', 'instance_budget_rules', 'policy_profiles')", + )? + .query_map([], |row| row.get(0))? + .collect::, _>>() + }) + .await + .expect("modern tables absent"); + assert!(modern.is_empty(), "pre-user image has no identity tables"); +} + +#[tokio::test(flavor = "current_thread")] +async fn schema54_image_upgrades_with_legacy_state_preserved() { + let path = temporary_database_path("schema54-upgrade"); + let database = load_schema54_image(&path).await; + prove_schema54_source(&database).await; + + // The protected matching backup is captured before the upgrade: old + // binary rollback uses this archive, never a down-migration. + let backup_root = tempfile::tempdir().expect("backup root"); + let config_path = backup_root.path().join("config.toml"); + let mut config = Config::default(); + config.database.path = path.to_string_lossy().into_owned(); + fs::write(&config_path, toml::to_string(&config).expect("config TOML")).expect("config writes"); + let backup = BackupService::from_config(&config_path, &config) + .with_output_dir(backup_root.path().join("backups")); + let archive = backup + .create(&database) + .await + .expect("pre-upgrade backup") + .archive; + BackupService::validate_archive(&archive).expect("pre-upgrade archive validates"); + + // Forward-only production upgrade: exactly migrations 55..62. + let state = MigrationRunner::new(&database) + .run() + .await + .expect("production upgrade applies"); + assert_eq!(state.applied_this_run.first(), Some(&55)); + assert_eq!(state.applied_this_run.last(), Some(&62)); + assert_eq!(state.applied_this_run.len(), 8); + database + .quick_check() + .await + .expect("upgraded image is valid"); + + // Legacy history survives byte-identical: upstream accounts, the + // completed request with its token/cost usage, quarantine, catalog. + let legacy = database + .call(|connection| { + Ok(( + connection.query_row( + "SELECT status, input_tokens, output_tokens, cost_microdollars \ + FROM requests WHERE id = 1", + [], + |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, i64>(1)?, + row.get::<_, i64>(2)?, + row.get::<_, i64>(3)?, + )) + }, + )?, + connection.query_row( + "SELECT enabled FROM accounts WHERE name = 'account-a'", + [], + |row| row.get::<_, i64>(0), + )?, + connection.query_row("SELECT COUNT(*) FROM model_quarantine", [], |row| { + row.get::<_, i64>(0) + })?, + connection.query_row("SELECT COUNT(*) FROM catalog_refresh_state", [], |row| { + row.get::<_, i64>(0) + })?, + )) + }) + .await + .expect("legacy rows read"); + assert_eq!(legacy.0, ("success".to_owned(), 100, 50, 700)); + assert_eq!(legacy.1, 1); + assert_eq!(legacy.2, 2); + assert_eq!(legacy.3, 2); + + // No fabricated human identity: historical shared-key inference stays + // unattributed under the legacy principal kind. + let attribution: (Option, String) = database + .call(|connection| { + connection.query_row( + "SELECT user_id, principal_kind FROM requests WHERE id = 1", + [], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + }) + .await + .expect("legacy attribution reads"); + assert_eq!(attribution, (None, "legacy_shared_key".to_owned())); + // Legacy shared-key state is distinct from fresh init: the upgraded + // image reports legacy_unmigrated with zero users. + assert_eq!( + access_state(&database).await.expect("state reads"), + "legacy_unmigrated" + ); + assert_eq!(table_count(&database, "users").await, 0); + + database.close().await.expect("database closes"); + fs::remove_file(&path).expect("temporary database removed"); +} + +#[tokio::test(flavor = "current_thread")] +async fn upgraded_schema54_database_recovers_identity_policy_and_budgets() { + // The database lives in a private directory: restore hardens its + // parent directory permissions, which a shared temp root must never + // receive. + let work_root = tempfile::tempdir().expect("work root"); + let path = work_root.path().join("schema54-recover.sqlite3"); + let database = load_schema54_image(&path).await; + prove_schema54_source(&database).await; + MigrationRunner::new(&database) + .run() + .await + .expect("production upgrade applies"); + + // Identity recovery: bootstrap, login, member, personal-token + // issue/revoke, password reset — all against the upgraded schema. + let (admin, admin_credential) = bootstrap_admin(&database, "legacy-admin", "correct-horse-47") + .await + .expect("admin bootstraps on upgraded schema"); + assert_eq!( + access_state(&database).await.expect("state reads"), + "admin_initialized" + ); + assert!( + verify_password(&database, "legacy-admin", "correct-horse-47") + .await + .expect("login verifies") + .is_some(), + "bootstrap password logs in" + ); + assert!( + verify_password(&database, "legacy-admin", "wrong-password") + .await + .expect("login rejects") + .is_none(), + "wrong password does not log in" + ); + let member = create_member(&database, &admin.id, "legacy-member", "temporary-99") + .await + .expect("member is created"); + let personal = issue_personal_api_token(&database, &admin.id, &member.id, "laptop") + .await + .expect("personal token issues"); + assert!( + verify_api_token(&database, &personal.token) + .await + .expect("token verifies") + ); + let revoked = revoke_user_api_tokens(&database, &admin.id, &member.id) + .await + .expect("personal tokens revoke"); + assert_eq!(revoked, 1); + assert!( + !verify_api_token(&database, &personal.token) + .await + .expect("revoked token rejects") + ); + admin_reset_password(&database, &admin.id, &member.id, "rotated-1000") + .await + .expect("admin resets member password"); + assert!( + verify_password(&database, "legacy-member", "rotated-1000") + .await + .expect("rotated password logs in") + .is_some() + ); + assert_eq!(admin_credential.user_id, admin.id); + + // Profile editing: create, rule, assign, edit, enforce. + let policies = ModelPolicyService::new(database.clone()); + policies + .create_profile("legacy-profile", "Legacy Profile", "allow", &admin.id) + .await + .expect("profile is created"); + policies + .set_profile_rule("legacy-profile", "shared-model", "allow", &admin.id) + .await + .expect("profile rule is set"); + policies + .assign_profile(&member.id, "legacy-profile", &admin.id) + .await + .expect("profile is assigned"); + // create (v1) plus the rule write (v2): the edit must target v2. + assert!( + policies + .update_profile("legacy-profile", "Legacy Profile v2", "deny", 2, &admin.id) + .await + .expect("profile is edited") + ); + let profiles = policies.list_profiles().await.expect("profiles list"); + assert_eq!(profiles.len(), 1); + assert_eq!(profiles[0].name, "Legacy Profile v2"); + + // Budget rules land on the upgraded schema and enforce for real: an + // estimated member request publishes and settles, and a global rule + // intersects it. + database + .call(|connection| { + connection.execute( + "INSERT INTO accounts (id, name, api_key_env, enabled, provider_id) + VALUES (10, 'budget-account', 'UNUSED', 1, 'provider-a')", + [], + )?; + connection.execute( + "INSERT INTO models (model_id, protocol, provider_id, resolution_status) + VALUES ('model-a', 'openai', 'provider-a', 'resolved')", + [], + )?; + Ok(()) + }) + .await + .expect("budget fixture rows insert"); + let user_budgets = UserBudgetService::new(database.clone()); + user_budgets + .set_rule( + &admin.id, + &member.id, + None, + BudgetWindow::Day, + BudgetLimits { + tokens: Some(1_000_000), + ..Default::default() + }, + ) + .await + .expect("estimated user rule is set"); + let global_budgets = InstanceBudgetService::new(database.clone()); + global_budgets + .set_rule( + &admin.id, + None, + BudgetWindow::Day, + BudgetLimits { + requests: Some(100), + ..Default::default() + }, + ) + .await + .expect("global rule is set"); + let mut provider_config = eggpool::config::ProviderConfig { + id: "provider-a".into(), + base_url: "https://provider.invalid/v1".into(), + protocols: vec!["openai".into()], + auth: eggpool::config::ProviderAuthConfig { + mode: "none".into(), + ..Default::default() + }, + ..Default::default() + }; + provider_config + .accounts + .push(eggpool::config::AccountConfig { + name: "budget-account".into(), + ..Default::default() + }); + let mut config = Config::default(); + config + .providers + .insert("provider-a".into(), provider_config); + config.validate().expect("fixture config validates"); + let registry = AccountRegistry::from_config( + &config, + &[Account { + id: 10, + name: "budget-account".into(), + api_key_env: "UNUSED".into(), + enabled: true, + weight: 1.0, + provider_id: "provider-a".into(), + }], + &CredentialStore::default(), + ) + .expect("registry builds"); + let mut catalog = ModelCatalogCache::default(); + catalog.set_account_provider("budget-account", "provider-a"); + let mut model = ModelInput::new("model-a"); + model.protocol = Some("openai".into()); + model.protocol_source = Some("fixture".into()); + model.resolution_status = ProtocolResolutionStatus::Resolved; + catalog + .update_from_account("budget-account", "provider-a", &[model], true, true) + .expect("catalog model"); + let router = RoutingRouter::new( + registry, + catalog, + QuotaEstimator::new([AccountQuota::new("budget-account")]), + None, + EligibilityPolicy::default(), + ); + let mut facts = RoutingRequestFacts::new("model-a"); + facts.requested_protocol = Some("openai".into()); + facts.client_protocol = Some("openai".into()); + facts.projected_tokens = 42; + let claim = router + .select_and_claim(&facts, &BTreeSet::new()) + .await + .expect("claim succeeds") + .expect("fixture has a candidate"); + // Resolve the live principal epoch exactly as the request edge does: + // the password rotation above invalidated older epochs. + let member_epoch: i64 = database + .call({ + let member_id = member.id.clone(); + move |connection| { + connection.query_row( + "SELECT auth_epoch FROM users WHERE id = ?1", + [member_id], + |row| row.get(0), + ) + } + }) + .await + .expect("member epoch reads"); + let attribution = AuthenticatedPrincipal { + user_id: Some(member.id.clone()), + credential_id: None, + role: Some("member".into()), + scopes: vec!["inference".into()], + auth_epoch: Some(member_epoch), + kind: PrincipalKind::AgentToken, + } + .request_attribution(); + let outcome = PublicationService::new(database.clone()) + .publish_attributed( + claim, + PublicationInput::new("legacy-upgrade-request", "openai", "openai", false, 1), + Some(attribution), + ) + .await + .expect("estimated request publishes under migrated rules"); + let PublicationOutcome::Published(published) = outcome else { + panic!("upgraded request should publish"); + }; + DurableFinalizer::new(database.clone()) + .finalize_request( + &published.identity, + FinalizationData { + outcome: FinalizationOutcome::Completed, + ..FinalizationData::default() + }, + Some(published.claim), + ) + .await + .expect("upgraded request settles"); + let settled_user: i64 = database + .call(|connection| { + connection.query_row( + "SELECT COUNT(*) FROM user_budget_reservations WHERE status = 'settled'", + [], + |row| row.get(0), + ) + }) + .await + .expect("user settlement reads"); + assert_eq!(settled_user, 1); + + // Forced restart over the upgraded image: the legacy pending request + // interrupts with its legacy reservation, the settled budget request + // is untouched, and startup reconverges. + database.close().await.expect("generation closes"); + let restarted = open_database(&path).await; + let rerun = MigrationRunner::new(&restarted) + .run() + .await + .expect("migrations stay idempotent across restart"); + assert!(rerun.applied_this_run.is_empty()); + let report = CrashReconciler::new(restarted.clone()) + .reconcile_once() + .await + .expect("restart reconciliation succeeds"); + assert_eq!(report.requests_interrupted, 1); + assert_eq!(report.reservations_released, 1); + assert_eq!(report.user_budget_holds_released, 0); + assert_eq!(report.instance_budget_holds_released, 0); + let second = CrashReconciler::new(restarted.clone()) + .reconcile_once() + .await + .expect("second pass succeeds"); + assert!(second.converged); + + // Backup/restore of the current (post-upgrade) state preserves users, + // rules, usage, and audit across the round trip. + let backup_root = tempfile::tempdir().expect("backup root"); + let config_path = backup_root.path().join("config.toml"); + let mut backup_config = Config::default(); + backup_config.database.path = path.to_string_lossy().into_owned(); + fs::write( + &config_path, + toml::to_string(&backup_config).expect("config TOML"), + ) + .expect("config writes"); + let backup = BackupService::from_config(&config_path, &backup_config) + .with_output_dir(backup_root.path().join("backups")); + let archive = backup + .create(&restarted) + .await + .expect("current-state backup") + .archive; + BackupService::validate_archive(&archive).expect("current archive validates"); + restarted + .call(|connection| { + connection.query_row("PRAGMA wal_checkpoint(TRUNCATE)", [], |_| Ok(()))?; + Ok(()) + }) + .await + .expect("wal checkpoints"); + restarted.close().await.expect("database closes"); + backup.recover(&archive).expect("current-state restore"); + let restored = open_database(&path).await; + assert_eq!(table_count(&restored, "users").await, 2); + assert_eq!(table_count(&restored, "user_budget_rules").await, 1); + assert_eq!(table_count(&restored, "instance_budget_rules").await, 1); + assert_eq!(table_count(&restored, "user_budget_usage").await, 2); + // Creating the first member moved the instance to managed, and the + // round trip preserves that activation state. + assert_eq!( + access_state(&restored).await.expect("state reads"), + "managed" + ); + restored.close().await.expect("restored database closes"); + fs::remove_file(&path).expect("temporary database removed"); +} diff --git a/rust/tests/on_prem_admin_ui.rs b/rust/tests/on_prem_admin_ui.rs new file mode 100644 index 000000000..941b9f15b --- /dev/null +++ b/rust/tests/on_prem_admin_ui.rs @@ -0,0 +1,742 @@ +use std::{sync::Arc, time::Duration}; + +use eggpool::{ + Config, + db::{Database, DatabaseConfig, MigrationRunner}, + identity, + model_policy::ModelPolicyService, + runtime_lifecycle::{ProcessRuntime, RuntimeGenerationFactory, RuntimeManager}, + server::{ServerRuntime, ServerRuntimeHandle, ShutdownReason}, +}; +use tokio::{ + io::{AsyncReadExt, AsyncWriteExt}, + net::{TcpListener, TcpStream}, +}; + +async fn database() -> (tempfile::TempDir, Database) { + let directory = tempfile::tempdir().expect("temporary admin directory"); + let database = Database::open(DatabaseConfig { + path: directory + .path() + .join("eggpool.sqlite3") + .to_string_lossy() + .into_owned(), + ..DatabaseConfig::default() + }) + .await + .expect("database opens"); + MigrationRunner::new(&database) + .run() + .await + .expect("schema migrates"); + (directory, database) +} + +fn cookie(response: &str, name: &str) -> String { + response + .lines() + .find_map(|line| { + let (header, value) = line.split_once(':')?; + if !header.eq_ignore_ascii_case("set-cookie") { + return None; + } + value + .trim() + .split(';') + .next()? + .strip_prefix(&format!("{name}=")) + .map(str::to_owned) + }) + .expect("cookie is set") +} + +async fn http_request( + address: std::net::SocketAddr, + method: &str, + path: &str, + body: &str, + cookies: Option<&str>, + origin: Option<&str>, + bearer: Option<&str>, +) -> String { + request_with_provenance(address, method, path, body, cookies, origin, None, bearer).await +} + +/// A same-origin browser form POST sends `Origin` and `Referer` together, which +/// is the header set the raw helper above never produces. +#[allow(clippy::too_many_arguments)] +async fn browser_form_post( + address: std::net::SocketAddr, + path: &str, + body: &str, + cookies: Option<&str>, + origin: &str, + referer: &str, +) -> String { + request_with_provenance( + address, + "POST", + path, + body, + cookies, + Some(origin), + Some(referer), + None, + ) + .await +} + +#[allow(clippy::too_many_arguments)] +async fn request_with_provenance( + address: std::net::SocketAddr, + method: &str, + path: &str, + body: &str, + cookies: Option<&str>, + origin: Option<&str>, + referer: Option<&str>, + bearer: Option<&str>, +) -> String { + let mut stream = TcpStream::connect(address).await.expect("connect"); + let mut request = + format!("{method} {path} HTTP/1.1\r\nHost: {address}\r\nConnection: close\r\n"); + if !body.is_empty() { + request.push_str(&format!( + "Content-Type: application/x-www-form-urlencoded\r\nContent-Length: {}\r\n", + body.len() + )); + } + if let Some(cookies) = cookies { + request.push_str(&format!("Cookie: {cookies}\r\n")); + } + if let Some(origin) = origin { + request.push_str(&format!("Origin: {origin}\r\n")); + } + if let Some(referer) = referer { + request.push_str(&format!("Referer: {referer}\r\n")); + } + if let Some(token) = bearer { + request.push_str(&format!("Authorization: Bearer {token}\r\n")); + } + request.push_str("\r\n"); + request.push_str(body); + stream + .write_all(request.as_bytes()) + .await + .expect("write request"); + let mut response = String::new(); + stream + .read_to_string(&mut response) + .await + .expect("read response"); + response +} + +fn session_pair(response: &str) -> (String, String, String) { + let session = cookie(response, "eggpool_session"); + let csrf = cookie(response, "eggpool_csrf"); + let header = format!("eggpool_session={session}; eggpool_csrf={csrf}"); + (session, csrf, header) +} + +#[tokio::test] +async fn profile_edits_use_compare_and_swap_and_are_audited() { + let (_directory, database) = database().await; + identity::mark_fresh_uninitialized(&database).await.unwrap(); + let (admin, _) = + identity::bootstrap_admin(&database, "admin", "M006 strong admin password passphrase") + .await + .unwrap(); + let policy = ModelPolicyService::new(database.clone()); + policy + .create_profile("restricted", "Restricted", "deny", &admin.id) + .await + .unwrap(); + let profile = policy.list_profiles().await.unwrap().remove(0); + assert_eq!(profile.version, 1); + assert!( + policy + .update_profile( + "restricted", + "Restricted v2", + "deny", + profile.version, + &admin.id, + ) + .await + .unwrap() + ); + assert!( + !policy + .update_profile( + "restricted", + "Stale edit", + "allow", + profile.version, + &admin.id, + ) + .await + .unwrap() + ); + policy + .set_profile_rule("restricted", "model-a", "allow", &admin.id) + .await + .unwrap(); + assert_eq!(policy.list_profile_rules().await.unwrap().len(), 1); + assert!( + policy + .remove_profile_rule("restricted", "model-a", &admin.id) + .await + .unwrap() + ); + let audit = identity::admin_audit(&database, 20).await.unwrap(); + assert!(audit.iter().any(|entry| entry.action == "create_profile")); + assert!(audit.iter().any(|entry| entry.action == "update_profile")); + assert!( + audit + .iter() + .any(|entry| entry.action == "remove_profile_rule") + ); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn two_user_browser_journey_fences_admin_data_and_personal_credentials() { + let (_directory, database) = database().await; + identity::mark_fresh_uninitialized(&database).await.unwrap(); + let (admin, admin_token) = + identity::bootstrap_admin(&database, "admin", "M006 strong admin password passphrase") + .await + .unwrap(); + identity::enable_managed(&database).await.unwrap(); + + let process = ProcessRuntime::new(database.clone()).expect("process runtime"); + let mut config = Config::default(); + config.server.host = "127.0.0.1".to_owned(); + config.server.api_key = None; + config.dashboard.enabled = true; + config.dashboard.public = true; + let candidate = + RuntimeGenerationFactory::prepare(&process, config.clone(), "admin-ui".into(), 1) + .await + .expect("runtime generation"); + let manager = Arc::new(RuntimeManager::new( + candidate.transfer().expect("generation transfer"), + )); + process + .install_initial_tasks((*manager).clone(), &config) + .await + .expect("tasks install"); + let runtime = ServerRuntime::new(process, manager, config); + let handle = runtime.handle(); + let listener = TcpListener::bind(("127.0.0.1", 0)).await.unwrap(); + let address = listener.local_addr().unwrap(); + let task = tokio::spawn(async move { runtime.serve_listener(listener).await }); + let origin = format!("http://{address}"); + + let login_form = "username=admin&password=M006+strong+admin+password+passphrase"; + let admin_login = http_request( + address, + "POST", + "/login", + login_form, + None, + Some(&origin), + None, + ) + .await; + assert!(admin_login.starts_with("HTTP/1.1 303"), "{admin_login}"); + let (_, csrf, admin_cookies) = session_pair(&admin_login); + let personal_admin_token_to_users = http_request( + address, + "GET", + "/users", + "", + None, + None, + Some(&admin_token.token), + ) + .await; + assert!( + personal_admin_token_to_users.starts_with("HTTP/1.1 401"), + "{personal_admin_token_to_users}" + ); + + let profile_form = format!("profile_id=team&name=Team&default_effect=deny&csrf={csrf}"); + let profile_created = http_request( + address, + "POST", + "/admin/profiles/create", + &profile_form, + Some(&admin_cookies), + Some(&origin), + None, + ) + .await; + assert!( + profile_created.starts_with("HTTP/1.1 303"), + "{profile_created}" + ); + + let create_form = format!( + "username=member&password=M006+temporary+member+password&profile_id=team&csrf={csrf}" + ); + let created = http_request( + address, + "POST", + "/admin/users/create", + &create_form, + Some(&admin_cookies), + Some(&origin), + None, + ) + .await; + assert!(created.starts_with("HTTP/1.1 303"), "{created}"); + let user_page = http_request( + address, + "GET", + "/users", + "", + Some(&admin_cookies), + None, + None, + ) + .await; + assert!(user_page.starts_with("HTTP/1.1 200"), "{user_page}"); + assert!(user_page.contains("member"), "{user_page}"); + // M008: the budget form distinguishes hard versus estimated enforcement + // instead of labeling every dimension an estimate. + assert!(user_page.contains("name=\"enforcement\""), "{user_page}"); + assert!( + user_page.contains("Hard (prove worst-case bound or deny)"), + "{user_page}" + ); + assert!( + user_page.contains("Estimated (approximate hold)"), + "{user_page}" + ); + + // M008: hard-mode budget rules are manageable through the admin form, + // invalid modes are rejected, and members see the mode on their page. + let budget_member_id = identity::list_users(&database) + .await + .unwrap() + .into_iter() + .find(|user| user.username == "member") + .expect("created member is listed") + .id; + let hard_budget_form = format!( + "user_id={budget_member_id}&model_id=&window=day&max_requests=&max_tokens=5000&max_microdollars=&max_concurrent=&enforcement=hard&csrf={csrf}" + ); + let hard_budget_saved = http_request( + address, + "POST", + "/admin/users/budget", + &hard_budget_form, + Some(&admin_cookies), + Some(&origin), + None, + ) + .await; + assert!( + hard_budget_saved.starts_with("HTTP/1.1 303"), + "{hard_budget_saved}" + ); + let invalid_budget_form = format!( + "user_id={budget_member_id}&model_id=&window=day&max_requests=&max_tokens=5000&max_microdollars=&max_concurrent=&enforcement=guaranteed&csrf={csrf}" + ); + let invalid_budget_rejected = http_request( + address, + "POST", + "/admin/users/budget", + &invalid_budget_form, + Some(&admin_cookies), + Some(&origin), + None, + ) + .await; + assert!( + invalid_budget_rejected.starts_with("HTTP/1.1 400"), + "{invalid_budget_rejected}" + ); + + // M009: instance ceilings are managed on the same admin page, visible in + // the global section, and fenced from member credentials. + assert!( + user_page.contains("Instance budget ceilings"), + "{user_page}" + ); + let instance_form = format!( + "model_id=&window=day&max_requests=100&max_tokens=&max_microdollars=&max_concurrent=&enforcement=estimated&csrf={csrf}" + ); + let instance_saved = http_request( + address, + "POST", + "/admin/budget/instance", + &instance_form, + Some(&admin_cookies), + Some(&origin), + None, + ) + .await; + assert!( + instance_saved.starts_with("HTTP/1.1 303"), + "{instance_saved}" + ); + let users_after_instance = http_request( + address, + "GET", + "/users", + "", + Some(&admin_cookies), + None, + None, + ) + .await; + assert!( + users_after_instance.contains("All models"), + "{users_after_instance}" + ); + let instance_remove_form = format!( + "model_id=&window=day&max_requests=&max_tokens=&max_microdollars=&max_concurrent=&csrf={csrf}" + ); + let instance_removed = http_request( + address, + "POST", + "/admin/budget/instance/remove", + &instance_remove_form, + Some(&admin_cookies), + Some(&origin), + None, + ) + .await; + assert!( + instance_removed.starts_with("HTTP/1.1 303"), + "{instance_removed}" + ); + + let missing_csrf = + "username=intruder&password=M006+temporary+member+password&profile_id=&csrf="; + let csrf_rejected = http_request( + address, + "POST", + "/admin/users/create", + missing_csrf, + Some(&admin_cookies), + Some(&origin), + None, + ) + .await; + assert!(csrf_rejected.starts_with("HTTP/1.1 403"), "{csrf_rejected}"); + + let member_login_form = "username=member&password=M006+temporary+member+password"; + let member_login = http_request( + address, + "POST", + "/login", + member_login_form, + None, + Some(&origin), + None, + ) + .await; + assert!(member_login.starts_with("HTTP/1.1 303"), "{member_login}"); + assert!( + member_login.contains("/account?password=required"), + "{member_login}" + ); + let (_, member_csrf, member_cookies) = session_pair(&member_login); + let forbidden_users = http_request( + address, + "GET", + "/users", + "", + Some(&member_cookies), + None, + None, + ) + .await; + assert!( + forbidden_users.starts_with("HTTP/1.1 403"), + "{forbidden_users}" + ); + let member_admin_post = format!( + "username=intruder&password=M006+temporary+member+password&profile_id=&csrf={member_csrf}" + ); + let member_create_forbidden = http_request( + address, + "POST", + "/admin/users/create", + &member_admin_post, + Some(&member_cookies), + Some(&origin), + None, + ) + .await; + assert!( + member_create_forbidden.starts_with("HTTP/1.1 403"), + "{member_create_forbidden}" + ); + let member_instance_post = format!( + "model_id=&window=day&max_requests=1&max_tokens=&max_microdollars=&max_concurrent=&enforcement=estimated&csrf={member_csrf}" + ); + let member_instance_forbidden = http_request( + address, + "POST", + "/admin/budget/instance", + &member_instance_post, + Some(&member_cookies), + Some(&origin), + None, + ) + .await; + assert!( + member_instance_forbidden.starts_with("HTTP/1.1 403"), + "{member_instance_forbidden}" + ); + + let change_password = format!( + "current_password=M006+temporary+member+password&new_password=M006+new+member+password+passphrase&csrf={member_csrf}" + ); + let changed = http_request( + address, + "POST", + "/account/password", + &change_password, + Some(&member_cookies), + Some(&origin), + None, + ) + .await; + assert!(changed.starts_with("HTTP/1.1 303"), "{changed}"); + let member_login_form = "username=member&password=M006+new+member+password+passphrase"; + let member_login = http_request( + address, + "POST", + "/login", + member_login_form, + None, + Some(&origin), + None, + ) + .await; + assert!(member_login.starts_with("HTTP/1.1 303"), "{member_login}"); + let (_, member_csrf, member_cookies) = session_pair(&member_login); + + let my_account = http_request( + address, + "GET", + "/my-account", + "", + Some(&member_cookies), + None, + None, + ) + .await; + assert!(my_account.starts_with("HTTP/1.1 200"), "{my_account}"); + assert!(my_account.contains("My accessible models"), "{my_account}"); + assert!(!my_account.contains(&admin.username), "{my_account}"); + + let member_usage = http_request( + address, + "GET", + "/usage", + "", + Some(&member_cookies), + None, + None, + ) + .await; + assert!(member_usage.starts_with("HTTP/1.1 200"), "{member_usage}"); + assert!(!member_usage.contains(&admin.username), "{member_usage}"); + assert!( + member_usage.contains("hard"), + "member usage page must show the hard enforcement mode: {member_usage}" + ); + + let token_form = format!("label=member+laptop&csrf={member_csrf}"); + let token_created = http_request( + address, + "POST", + "/account/tokens", + &token_form, + Some(&member_cookies), + Some(&origin), + None, + ) + .await; + assert!(token_created.starts_with("HTTP/1.1 200"), "{token_created}"); + assert!( + token_created + .to_ascii_lowercase() + .contains("cache-control: no-store") + ); + let token_start = token_created + .find("epk1_") + .expect("one-time token appears once in issue response"); + let token = token_created[token_start..] + .split('<') + .next() + .expect("token text is present") + .to_owned(); + assert!(token.starts_with("epk1_")); + assert_eq!(token_created.matches(&token).count(), 1); + let member_models = + http_request(address, "GET", "/v1/models", "", None, None, Some(&token)).await; + assert!(member_models.starts_with("HTTP/1.1 200"), "{member_models}"); + let token_to_admin = http_request(address, "GET", "/users", "", None, None, Some(&token)).await; + assert!( + token_to_admin.starts_with("HTTP/1.1 401"), + "{token_to_admin}" + ); + let member_id = identity::list_users(&database) + .await + .unwrap() + .into_iter() + .find(|user| user.username == "member") + .expect("created member is listed") + .id; + let member_credential = identity::list_user_credentials(&database, &member_id) + .await + .unwrap() + .remove(0); + let revoke_form = format!("credential_id={}&csrf={member_csrf}", member_credential.id); + let revoked = http_request( + address, + "POST", + "/account/tokens/revoke", + &revoke_form, + Some(&member_cookies), + Some(&origin), + None, + ) + .await; + assert!(revoked.starts_with("HTTP/1.1 303"), "{revoked}"); + let revoked_token = + http_request(address, "GET", "/v1/models", "", None, None, Some(&token)).await; + assert!(revoked_token.starts_with("HTTP/1.1 401"), "{revoked_token}"); + let refreshed_account = http_request( + address, + "GET", + "/my-account", + "", + Some(&member_cookies), + None, + None, + ) + .await; + assert!(!refreshed_account.contains(&token), "{refreshed_account}"); + + let audit = http_request( + address, + "GET", + "/admin/audit", + "", + Some(&admin_cookies), + None, + None, + ) + .await; + assert!(audit.starts_with("HTTP/1.1 200"), "{audit}"); + assert!(audit.contains("create_user"), "{audit}"); + assert!(audit.contains("assign_profile"), "{audit}"); + assert!(audit.contains("issue_token"), "{audit}"); + + assert!(matches!( + identity::set_user_enabled(&database, &admin.id, &admin.id, false).await, + Err(identity::IdentityError::LastAdminProtected) + )); + handle.request_shutdown(ShutdownReason::Requested); + tokio::time::timeout(Duration::from_secs(10), task) + .await + .expect("server shutdown completes") + .unwrap() + .unwrap(); + database.close().await.unwrap(); +} + +/// Start the managed dashboard on an ephemeral loopback port. +async fn managed_server( + database: &Database, +) -> ( + std::net::SocketAddr, + ServerRuntimeHandle, + tokio::task::JoinHandle<()>, +) { + let process = ProcessRuntime::new(database.clone()).expect("process runtime"); + let mut config = Config::default(); + config.server.host = "127.0.0.1".to_owned(); + config.server.api_key = None; + config.dashboard.enabled = true; + config.dashboard.public = true; + let candidate = + RuntimeGenerationFactory::prepare(&process, config.clone(), "browser-signin".into(), 1) + .await + .expect("runtime generation"); + let manager = Arc::new(RuntimeManager::new( + candidate.transfer().expect("generation transfer"), + )); + process + .install_initial_tasks((*manager).clone(), &config) + .await + .expect("tasks install"); + let runtime = ServerRuntime::new(process, manager, config); + let handle = runtime.handle(); + let listener = TcpListener::bind(("127.0.0.1", 0)).await.unwrap(); + let address = listener.local_addr().unwrap(); + let task = tokio::spawn(async move { + let _ = runtime.serve_listener(listener).await; + }); + (address, handle, task) +} + +#[tokio::test] +async fn browser_sign_in_accepts_the_origin_and_referer_pair_a_real_browser_sends() { + let (_directory, database) = database().await; + identity::mark_fresh_uninitialized(&database).await.unwrap(); + identity::bootstrap_admin(&database, "admin", "M011 strong admin password passphrase") + .await + .unwrap(); + identity::enable_managed(&database).await.unwrap(); + let (address, handle, task) = managed_server(&database).await; + let origin = format!("http://{address}"); + let form = "username=admin&password=M011+strong+admin+password+passphrase"; + + // Chrome sends both headers on a same-origin form POST; rejecting that pair + // would make the admin and member sign-in screens unusable in any browser. + let signed_in = browser_form_post( + address, + "/login", + form, + None, + &origin, + &format!("{origin}/login"), + ) + .await; + assert!(signed_in.starts_with("HTTP/1.1 303"), "{signed_in}"); + assert!(signed_in.contains("eggpool_session="), "{signed_in}"); + let (_, _csrf, cookies) = session_pair(&signed_in); + + let users = http_request(address, "GET", "/users", "", Some(&cookies), None, None).await; + assert!(users.starts_with("HTTP/1.1 200"), "{users}"); + + // The same pair must still fail closed when the referer is cross-site. + let refused = browser_form_post( + address, + "/login", + form, + None, + &origin, + "https://attacker.test/login", + ) + .await; + assert!(refused.starts_with("HTTP/1.1 403"), "{refused}"); + assert!(!refused.contains("eggpool_session="), "{refused}"); + + handle.request_shutdown(ShutdownReason::Requested); + tokio::time::timeout(Duration::from_secs(10), task) + .await + .expect("server shutdown completes") + .expect("server task joins"); + database.close().await.unwrap(); +} diff --git a/rust/tests/on_prem_attribution.rs b/rust/tests/on_prem_attribution.rs new file mode 100644 index 000000000..9a7d7f57c --- /dev/null +++ b/rust/tests/on_prem_attribution.rs @@ -0,0 +1,414 @@ +use eggpool::{ + Config, + config::{AccountConfig, ProviderAuthConfig, ProviderConfig, ProviderStaticModelConfig}, + db::{ + AccountConfig as DbAccountConfig, AccountRepository, Database, DatabaseConfig, + MigrationRunner, + }, + identity, + model_policy::ModelPolicyService, + runtime_lifecycle::{ProcessRuntime, RuntimeGenerationFactory, RuntimeManager}, + server::{ServerRuntime, ShutdownReason}, + web_session, +}; +use std::sync::Arc; +use tokio::{ + io::{AsyncReadExt, AsyncWriteExt}, + net::{TcpListener, TcpStream}, +}; + +async fn database() -> (tempfile::TempDir, Database) { + let directory = tempfile::tempdir().expect("temporary attribution directory"); + let database = Database::open(DatabaseConfig { + path: directory + .path() + .join("eggpool.sqlite3") + .to_string_lossy() + .into_owned(), + ..DatabaseConfig::default() + }) + .await + .expect("database opens"); + MigrationRunner::new(&database) + .run() + .await + .expect("schema migrates"); + (directory, database) +} + +#[tokio::test] +async fn credential_and_session_resolve_the_same_user_epoch() { + let (_directory, database) = database().await; + identity::mark_fresh_uninitialized(&database).await.unwrap(); + let (admin, credential) = identity::bootstrap_admin( + &database, + "attribution-admin", + "M003 attribution test password", + ) + .await + .unwrap(); + let agent = identity::verify_api_token_principal(&database, &credential.token) + .await + .unwrap() + .expect("agent token resolves to a typed user principal"); + assert_eq!(agent.user_id.as_deref(), Some(admin.id.as_str())); + assert_eq!(agent.role.as_deref(), Some("agent")); + assert!(agent.credential_id.is_some()); + assert!(agent.scopes.contains(&"inference".to_owned())); + assert_eq!(agent.auth_epoch, Some(1)); + + let issued = web_session::issue(&database, &admin.id).await.unwrap(); + let browser = web_session::authenticate(&database, &issued.session_token) + .await + .unwrap() + .unwrap(); + assert_eq!(browser.user_id, admin.id); + assert_eq!(browser.auth_epoch, 1); + + identity::reset_password(&database, "attribution-admin", "M003 replacement password") + .await + .unwrap(); + assert!( + identity::verify_api_token_principal(&database, &credential.token) + .await + .unwrap() + .is_none(), + "password reset revokes the old credential" + ); + assert!( + web_session::authenticate(&database, &issued.session_token) + .await + .unwrap() + .is_none(), + "password reset revokes the old session" + ); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn user_request_history_is_bounded_and_excludes_selector_children() { + let (_directory, database) = database().await; + database + .call(|connection| { + connection.execute( + "INSERT INTO accounts (id, name, api_key_env, enabled, provider_id)\ + VALUES (1, 'attribution-account', 'UNUSED', 1, 'provider-a')", + [], + )?; + connection.execute( + "INSERT INTO models (model_id, protocol, provider_id, resolution_status)\ + VALUES ('model-a', 'openai', 'provider-a', 'resolved')", + [], + )?; + for (proxy_id, user_id, request_kind) in [ + ("logical-a", "user-a", "logical"), + ("selector-a", "user-a", "internal_selector"), + ("logical-b", "user-b", "logical"), + ] { + connection.execute( + "INSERT INTO requests\ + (account_id, model_id, provider_id, proxy_request_id, user_id, request_kind)\ + VALUES (1, 'model-a', 'provider-a', ?1, ?2, ?3)", + (proxy_id, user_id, request_kind), + )?; + } + Ok(()) + }) + .await + .unwrap(); + + let user_a = identity::recent_user_requests(&database, "user-a", 50) + .await + .unwrap(); + assert_eq!(user_a.len(), 1); + assert_eq!(user_a[0].proxy_request_id.as_deref(), Some("logical-a")); + assert_eq!(user_a[0].model_id, "model-a"); + assert!( + identity::recent_user_requests(&database, "user-b", 0) + .await + .unwrap() + .iter() + .all(|row| row.proxy_request_id.as_deref() == Some("logical-b")) + ); + database.close().await.unwrap(); +} + +async fn provider_fixture(listener: TcpListener) { + let (mut stream, _) = listener.accept().await.unwrap(); + let mut request = Vec::new(); + let header_end = loop { + if let Some(index) = request.windows(4).position(|window| window == b"\r\n\r\n") { + break index + 4; + } + let mut buffer = [0_u8; 2048]; + let read = stream.read(&mut buffer).await.unwrap(); + assert_ne!(read, 0, "provider request closes before headers"); + request.extend_from_slice(&buffer[..read]); + }; + let headers = String::from_utf8_lossy(&request[..header_end]); + let content_length = headers + .lines() + .find_map(|line| { + let (name, value) = line.split_once(':')?; + name.eq_ignore_ascii_case("content-length") + .then(|| value.trim().parse::().ok()) + .flatten() + }) + .unwrap_or(0); + while request.len() < header_end + content_length { + let mut buffer = [0_u8; 2048]; + let read = stream.read(&mut buffer).await.unwrap(); + assert_ne!(read, 0, "provider request body ends early"); + request.extend_from_slice(&buffer[..read]); + } + let body = br#"{"id":"chatcmpl-attribution","object":"chat.completion","model":"attribution-model","choices":[{"index":0,"message":{"role":"assistant","content":"owned"},"finish_reason":"stop"}],"usage":{"prompt_tokens":1,"completion_tokens":1,"total_tokens":2}}"#; + let head = format!( + "HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n", + body.len() + ); + stream.write_all(head.as_bytes()).await.unwrap(); + stream.write_all(body).await.unwrap(); +} + +async fn inference_request(address: std::net::SocketAddr, token: &str) -> String { + inference_request_for_model(address, token, "attribution-model").await +} + +async fn inference_request_for_model( + address: std::net::SocketAddr, + token: &str, + model: &str, +) -> String { + let body = serde_json::json!({ + "model": model, + "messages": [{"role": "user", "content": "hello"}], + "stream": false, + }) + .to_string(); + let request = format!( + "POST /v1/chat/completions HTTP/1.1\r\nHost: {address}\r\nAuthorization: Bearer {token}\r\nx-eggpool-user-id: spoofed-user\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + body.len(), + body + ); + let mut stream = TcpStream::connect(address).await.unwrap(); + stream.write_all(request.as_bytes()).await.unwrap(); + let mut response = Vec::new(); + tokio::time::timeout( + std::time::Duration::from_secs(15), + stream.read_to_end(&mut response), + ) + .await + .expect("downstream response completes") + .unwrap(); + String::from_utf8_lossy(&response).into_owned() +} + +async fn get_request(address: std::net::SocketAddr, path: &str, token: &str) -> String { + let request = format!( + "GET {path} HTTP/1.1\r\nHost: {address}\r\nAuthorization: Bearer {token}\r\nConnection: close\r\n\r\n" + ); + let mut stream = TcpStream::connect(address).await.unwrap(); + stream.write_all(request.as_bytes()).await.unwrap(); + let mut response = Vec::new(); + tokio::time::timeout( + std::time::Duration::from_secs(15), + stream.read_to_end(&mut response), + ) + .await + .expect("discovery response completes") + .unwrap(); + String::from_utf8_lossy(&response).into_owned() +} + +#[tokio::test] +async fn managed_inference_ignores_spoofed_user_headers_and_persists_token_owner() { + let (_directory, database) = database().await; + identity::mark_fresh_uninitialized(&database).await.unwrap(); + let (admin, credential) = identity::bootstrap_admin( + &database, + "request-owner", + "M003 request attribution password", + ) + .await + .unwrap(); + database + .call(|connection| { + connection.execute( + "INSERT INTO users(id, username, normalized_username, role, password_hash) + VALUES ('request-member', 'request-member', 'request-member', 'member', 'test-hash')", + [], + )?; + Ok(()) + }) + .await + .unwrap(); + let member_credential = identity::issue_api_token(&database, "request-member", "budget-test") + .await + .unwrap(); + eggpool::user_budget::UserBudgetService::new(database.clone()) + .set_rule( + &admin.id, + &admin.id, + None, + eggpool::user_budget::BudgetWindow::Day, + eggpool::user_budget::BudgetLimits { + requests: Some(10), + concurrent: Some(2), + ..Default::default() + }, + ) + .await + .unwrap(); + + let provider_listener = TcpListener::bind(("127.0.0.1", 0)).await.unwrap(); + let provider_address = provider_listener.local_addr().unwrap(); + let provider_task = tokio::spawn(provider_fixture(provider_listener)); + let mut config = Config::default(); + config.server.host = "127.0.0.1".to_owned(); + config.server.api_key = None; + config.integrations.advertise_base_url = Some("https://pool.example/v1".to_owned()); + config.dashboard.public = false; + config.models.startup_refresh = false; + let provider = ProviderConfig { + id: "fixture".to_owned(), + base_url: format!("http://{provider_address}"), + protocols: vec!["openai".to_owned()], + auth: ProviderAuthConfig { + mode: "none".to_owned(), + ..ProviderAuthConfig::default() + }, + accounts: vec![AccountConfig { + name: "fixture-account".to_owned(), + ..AccountConfig::default() + }], + static_models: vec![ProviderStaticModelConfig { + id: "attribution-model".to_owned(), + protocol: Some("openai".to_owned()), + ..ProviderStaticModelConfig::default() + }], + ..ProviderConfig::default() + }; + config.providers.insert("fixture".to_owned(), provider); + AccountRepository::new(&database) + .sync_from_config(vec![DbAccountConfig { + name: "fixture-account".to_owned(), + api_key_env: String::new(), + enabled: true, + weight: 1.0, + provider_id: "fixture".to_owned(), + }]) + .await + .unwrap(); + database + .call(|connection| { + connection.execute( + "INSERT INTO models (model_id, protocol, provider_id, resolution_status)\ + VALUES ('attribution-model', 'openai', 'fixture', 'resolved')", + [], + )?; + Ok(()) + }) + .await + .unwrap(); + + let process = ProcessRuntime::new(database.clone()).unwrap(); + let candidate = RuntimeGenerationFactory::prepare( + &process, + config.clone(), + "on-prem-attribution".to_owned(), + 1, + ) + .await + .unwrap(); + let manager = Arc::new(RuntimeManager::new(candidate.transfer().unwrap())); + process + .install_initial_tasks((*manager).clone(), &config) + .await + .unwrap(); + let runtime = ServerRuntime::new(process, manager, config); + let handle = runtime.handle(); + let listener = TcpListener::bind(("127.0.0.1", 0)).await.unwrap(); + let address = listener.local_addr().unwrap(); + let server_task = tokio::spawn(async move { runtime.serve_listener(listener).await }); + + let response = inference_request(address, &credential.token).await; + assert!(response.starts_with("HTTP/1.1 200"), "{response}"); + ModelPolicyService::new(database.clone()) + .set_instance_denial("attribution-model", &admin.id, true) + .await + .unwrap(); + let denied = inference_request(address, &credential.token).await; + assert!(denied.starts_with("HTTP/1.1 403"), "{denied}"); + let qualified = + inference_request_for_model(address, &credential.token, "attribution-model/fixture").await; + assert!(qualified.starts_with("HTTP/1.1 403"), "{qualified}"); + let models = get_request(address, "/v1/models", &credential.token).await; + assert!(models.starts_with("HTTP/1.1 200"), "{models}"); + assert!(!models.contains("attribution-model"), "{models}"); + let profile = get_request(address, "/api/integrations/v1/profile", &credential.token).await; + assert!(profile.starts_with("HTTP/1.1 200"), "{profile}"); + assert!( + profile.to_ascii_lowercase().contains("etag: \""), + "{profile}" + ); + assert!( + profile.contains("-p"), + "policy revision is in the ETag: {profile}" + ); + assert!(!profile.contains("attribution-model"), "{profile}"); + let budget = get_request(address, "/api/user-budget", &credential.token).await; + assert!(budget.starts_with("HTTP/1.1 200"), "{budget}"); + assert!(budget.contains("estimated_microdollars"), "{budget}"); + assert!(budget.contains("max_requests"), "{budget}"); + let member_budget = get_request(address, "/api/user-budget", &member_credential.token).await; + assert!(member_budget.starts_with("HTTP/1.1 200"), "{member_budget}"); + let member_other_budget = get_request( + address, + &format!("/api/user-budget?user_id={}", admin.id), + &member_credential.token, + ) + .await; + assert!( + member_other_budget.starts_with("HTTP/1.1 403"), + "{member_other_budget}" + ); + let persisted = database + .call(|connection| { + connection.query_row( + "SELECT user_id, credential_id, auth_epoch, principal_kind, request_kind \ + FROM requests ORDER BY id DESC LIMIT 1", + [], + |row| { + Ok(( + row.get::<_, Option>(0)?, + row.get::<_, Option>(1)?, + row.get::<_, Option>(2)?, + row.get::<_, String>(3)?, + row.get::<_, String>(4)?, + )) + }, + ) + }) + .await + .unwrap(); + let credential_id = database + .call(|connection| { + connection.query_row( + "SELECT id FROM user_credentials ORDER BY created_at LIMIT 1", + [], + |row| row.get::<_, String>(0), + ) + }) + .await + .unwrap(); + assert_eq!(persisted.0.as_deref(), Some(admin.id.as_str())); + assert_eq!(persisted.1.as_deref(), Some(credential_id.as_str())); + assert_eq!(persisted.2, Some(1)); + assert_eq!(persisted.3, "agent_token"); + assert_eq!(persisted.4, "logical"); + + handle.request_shutdown(ShutdownReason::Requested); + server_task.await.unwrap().unwrap(); + provider_task.await.unwrap(); + database.close().await.unwrap(); +} diff --git a/rust/tests/on_prem_budgets.rs b/rust/tests/on_prem_budgets.rs new file mode 100644 index 000000000..022f8bd18 --- /dev/null +++ b/rust/tests/on_prem_budgets.rs @@ -0,0 +1,1667 @@ +use std::collections::BTreeSet; + +use eggpool::{ + Config, + accounts::{AccountRegistry, CredentialStore}, + catalog::{ModelCatalogCache, ModelInput, ProtocolResolutionStatus}, + coordinator::{ + CrashReconciler, DurableFinalizer, EndpointError, FinalizationData, FinalizationError, + FinalizationOutcome, PublicationError, PublicationFaultInjector, PublicationInput, + PublicationOutcome, PublicationService, PublicationStage, endpoint_error_body, + }, + db::{Account, Database, DatabaseConfig, MigrationRunner}, + identity::{AuthenticatedPrincipal, PrincipalKind}, + operations::backup::BackupService, + quota::{AccountQuota, QuotaEstimator}, + routing::{EligibilityPolicy, RoutingRequestFacts, RoutingRouter}, + user_budget::{ + BudgetEnforcement, BudgetLimits, BudgetProof, BudgetWindow, UserBudgetError, + UserBudgetService, + }, + wire::ir::ClientSurface, +}; + +struct Fixture { + database: Database, + router: RoutingRouter, +} + +async fn fixture() -> Fixture { + fixture_on(&DatabaseConfig::default()).await +} + +async fn fixture_on(config: &DatabaseConfig) -> Fixture { + let database = Database::open(config.clone()) + .await + .expect("database opens"); + MigrationRunner::new(&database) + .run() + .await + .expect("schema migrates"); + database + .call(|connection| { + connection.execute( + "INSERT INTO accounts (id, name, api_key_env, enabled, provider_id) + VALUES (1, 'budget-account', 'UNUSED', 1, 'provider-a')", + [], + )?; + connection.execute( + "INSERT INTO models (model_id, protocol, provider_id, resolution_status) + VALUES ('model-a', 'openai', 'provider-a', 'resolved')", + [], + )?; + connection.execute( + "INSERT INTO users(id, username, normalized_username, role, password_hash) + VALUES ('budget-admin', 'budget-admin', 'budget-admin', 'admin', 'test-hash')", + [], + )?; + connection.execute( + "INSERT INTO users(id, username, normalized_username, role, password_hash) + VALUES ('budget-member', 'budget-member', 'budget-member', 'member', 'test-hash')", + [], + )?; + Ok(()) + }) + .await + .expect("fixture rows insert"); + + let mut config = Config::default(); + let mut provider = eggpool::config::ProviderConfig { + id: "provider-a".into(), + base_url: "https://provider.invalid/v1".into(), + protocols: vec!["openai".into()], + auth: eggpool::config::ProviderAuthConfig { + mode: "none".into(), + ..Default::default() + }, + ..Default::default() + }; + provider.accounts.push(eggpool::config::AccountConfig { + name: "budget-account".into(), + ..Default::default() + }); + config.providers.insert("provider-a".into(), provider); + config.validate().expect("fixture config validates"); + let registry = AccountRegistry::from_config( + &config, + &[Account { + id: 1, + name: "budget-account".into(), + api_key_env: "UNUSED".into(), + enabled: true, + weight: 1.0, + provider_id: "provider-a".into(), + }], + &CredentialStore::default(), + ) + .expect("registry builds"); + let mut catalog = ModelCatalogCache::default(); + catalog.set_account_provider("budget-account", "provider-a"); + let mut model = ModelInput::new("model-a"); + model.protocol = Some("openai".into()); + model.protocol_source = Some("fixture".into()); + model.resolution_status = ProtocolResolutionStatus::Resolved; + catalog + .update_from_account("budget-account", "provider-a", &[model], true, true) + .expect("catalog model"); + let router = RoutingRouter::new( + registry, + catalog, + QuotaEstimator::new([AccountQuota::new("budget-account")]), + None, + EligibilityPolicy::default(), + ); + Fixture { database, router } +} + +async fn claim(fixture: &Fixture) -> eggpool::routing::SelectionClaim { + let mut facts = RoutingRequestFacts::new("model-a"); + facts.requested_protocol = Some("openai".into()); + facts.client_protocol = Some("openai".into()); + facts.projected_tokens = 42; + fixture + .router + .select_and_claim(&facts, &BTreeSet::new()) + .await + .unwrap() + .unwrap() +} + +fn attribution() -> eggpool::identity::RequestAttribution { + AuthenticatedPrincipal { + user_id: Some("budget-member".into()), + credential_id: None, + role: Some("member".into()), + scopes: vec!["inference".into()], + auth_epoch: Some(1), + kind: PrincipalKind::AgentToken, + } + .request_attribution() +} + +#[tokio::test] +async fn concurrent_last_request_reservations_cannot_over_admit() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + budgets + .set_rule( + "budget-admin", + "budget-member", + None, + BudgetWindow::Day, + BudgetLimits { + requests: Some(1), + tokens: Some(100), + concurrent: Some(1), + ..Default::default() + }, + ) + .await + .expect("admin sets request and concurrency ceilings"); + + let service = PublicationService::new(fixture.database.clone()); + let first_claim = claim(&fixture).await; + let second_claim = claim(&fixture).await; + let first = service.publish_attributed( + first_claim, + PublicationInput::new("budget-race-a", "openai", "openai", false, 1), + Some(attribution()), + ); + let second = service.publish_attributed( + second_claim, + PublicationInput::new("budget-race-b", "openai", "openai", false, 1), + Some(attribution()), + ); + let (first, second) = tokio::join!(first, second); + let mut published = None; + let mut denied = 0; + for result in [first, second] { + match result { + Ok(PublicationOutcome::Published(attempt)) => { + assert!(published.replace(attempt).is_none()); + } + Err(PublicationError::BudgetExceeded) => denied += 1, + other => panic!("unexpected concurrent budget result: {other:?}"), + } + } + assert_eq!(denied, 1); + assert_eq!( + fixture + .database + .call(|connection| { + Ok(( + connection.query_row("SELECT count(*) FROM requests", [], |row| { + row.get::<_, i64>(0) + })?, + connection.query_row( + "SELECT count(*) FROM user_budget_reservations WHERE status = 'active'", + [], + |row| row.get::<_, i64>(0), + )?, + )) + }) + .await + .unwrap(), + (1, 1) + ); + let published = published.expect("one request was published"); + let finalizer = DurableFinalizer::new(fixture.database.clone()); + let proxy_request_id = published.identity.proxy_request_id.clone(); + finalizer + .finalize_failed_attempt( + &published.identity, + FinalizationData { + outcome: FinalizationOutcome::UpstreamError, + status_code: Some(502), + input_tokens: 10, + output_tokens: 5, + input_tokens_reported: Some(10), + output_tokens_reported: Some(5), + cost_microdollars: 4, + provider_cost_microdollars: Some(4), + ..FinalizationData::default() + }, + Some(published.claim), + ) + .await + .expect("failed upstream attempt keeps the logical user hold"); + let retry = service + .publish_attributed( + claim(&fixture).await, + PublicationInput::new(proxy_request_id, "openai", "openai", false, 2), + Some(attribution()), + ) + .await + .expect("retry reuses the same logical reservation"); + let PublicationOutcome::Published(retry) = retry else { + panic!("retry should publish a new attempt"); + }; + let final_data = FinalizationData { + outcome: FinalizationOutcome::Completed, + input_tokens: 30, + output_tokens: 12, + input_tokens_reported: Some(30), + output_tokens_reported: Some(12), + cost_microdollars: 21, + provider_cost_microdollars: Some(21), + ..FinalizationData::default() + }; + finalizer + .finalize_request(&retry.identity, final_data.clone(), Some(retry.claim)) + .await + .expect("durable completion settles the user budget"); + finalizer + .finalize_request(&retry.identity, final_data, None) + .await + .expect("duplicate finalization converges idempotently"); + let usage = budgets + .usage("budget-member", "2026-01-01", 10) + .await + .expect("bounded usage query succeeds"); + assert_eq!(usage.len(), 2, "one daily and one monthly bucket are kept"); + assert!(usage.iter().all(|bucket| { + bucket.request_count == 1 + && bucket.token_count == 57 + && bucket.cost_microdollars == 25 + && bucket.provider_reported_microdollars == 25 + && bucket.local_estimated_microdollars == 0 + && bucket.reservation_fallback_microdollars == 0 + })); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn rules_validate_admin_scope_and_usage_reads_are_bounded() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + assert!(matches!( + budgets + .set_rule( + "budget-member", + "budget-member", + None, + BudgetWindow::Day, + BudgetLimits { + requests: Some(1), + ..Default::default() + }, + ) + .await, + Err(UserBudgetError::ActorNotAdmin) + )); + assert!(matches!( + budgets + .set_rule( + "budget-admin", + "budget-member", + None, + BudgetWindow::Day, + BudgetLimits { + requests: Some(0), + ..Default::default() + }, + ) + .await, + Err(UserBudgetError::InvalidRule) + )); + budgets + .set_rule( + "budget-admin", + "budget-member", + Some("model-a"), + BudgetWindow::Day, + BudgetLimits { + requests: Some(20), + tokens: Some(50_000), + microdollars: Some(10_000), + concurrent: Some(1), + ..Default::default() + }, + ) + .await + .expect("admin rule is recorded"); + let recorded = budgets.rules("budget-member").await.unwrap(); + assert_eq!(recorded.len(), 1); + // Compatibility default: rules created without an explicit mode are + // estimated, never silently hard. + assert_eq!( + recorded[0].enforcement, + eggpool::user_budget::BudgetEnforcement::Estimated + ); + assert!( + budgets + .remove_rule( + "budget-admin", + "budget-member", + Some("model-a"), + BudgetWindow::Day, + ) + .await + .unwrap() + ); + assert!(budgets.rules("budget-member").await.unwrap().is_empty()); + assert_eq!( + fixture + .database + .call(|connection| { + connection.query_row("SELECT count(*) FROM user_budget_rule_audit", [], |row| { + row.get::<_, i64>(0) + }) + }) + .await + .unwrap(), + 2 + ); + assert!( + budgets + .usage("budget-member", "2026-01-01", 501) + .await + .is_err() + ); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn unknown_cost_settlement_consumes_the_reserved_estimate() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + budgets + .set_rule( + "budget-admin", + "budget-member", + None, + BudgetWindow::Day, + BudgetLimits { + microdollars: Some(10_000), + ..Default::default() + }, + ) + .await + .unwrap(); + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + PublicationInput::new("budget-unknown-cost", "openai", "openai", false, 1), + Some(attribution()), + ) + .await + .expect("conservative estimate fits configured budget"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + let request_id = published.identity.db_request_id; + let held = fixture + .database + .call(move |connection| { + connection.query_row( + "SELECT held_microdollars FROM user_budget_reservations WHERE request_id = ?1", + [request_id], + |row| row.get::<_, i64>(0), + ) + }) + .await + .unwrap(); + assert!(held > 0, "unpriced use still carries a nonzero hold"); + DurableFinalizer::new(fixture.database.clone()) + .finalize_request( + &published.identity, + FinalizationData::default(), + Some(published.claim), + ) + .await + .expect("unknown usage settles conservatively"); + let usage = budgets + .usage("budget-member", "2026-01-01", 10) + .await + .unwrap(); + assert!(usage.iter().all(|bucket| { + bucket.cost_microdollars == held + && bucket.reservation_fallback_microdollars == held + && bucket.unknown_cost_requests == 1 + && bucket.unknown_token_requests == 1 + })); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn concurrent_limit_releases_only_after_terminal_settlement() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + budgets + .set_rule( + "budget-admin", + "budget-member", + None, + BudgetWindow::Day, + BudgetLimits { + concurrent: Some(1), + ..Default::default() + }, + ) + .await + .unwrap(); + let service = PublicationService::new(fixture.database.clone()); + let first = service + .publish_attributed( + claim(&fixture).await, + PublicationInput::new("budget-concurrent-a", "openai", "openai", false, 1), + Some(attribution()), + ) + .await + .unwrap(); + let PublicationOutcome::Published(first) = first else { + panic!("first concurrent request should publish"); + }; + assert!(matches!( + service + .publish_attributed( + claim(&fixture).await, + PublicationInput::new("budget-concurrent-b", "openai", "openai", false, 1), + Some(attribution()), + ) + .await, + Err(PublicationError::BudgetExceeded) + )); + DurableFinalizer::new(fixture.database.clone()) + .finalize_request( + &first.identity, + FinalizationData { + outcome: FinalizationOutcome::Completed, + input_tokens: 42, + input_tokens_reported: Some(42), + ..FinalizationData::default() + }, + Some(first.claim), + ) + .await + .unwrap(); + let after_settlement = service + .publish_attributed( + claim(&fixture).await, + PublicationInput::new("budget-concurrent-c", "openai", "openai", false, 1), + Some(attribution()), + ) + .await + .unwrap(); + let PublicationOutcome::Published(after_settlement) = after_settlement else { + panic!("terminal settlement should release the concurrency slot"); + }; + after_settlement + .claim + .release_all() + .expect("test claim is released"); + fixture.database.close().await.unwrap(); +} + +// M008 hard-mode helpers: the proof carries admission output bounds plus the +// operator tariff snapshot, mirroring what the coordinators assemble from +// the claim snapshot and provider configuration. +fn hard_input( + proxy_id: &str, + attempt: i64, + input_worst: i64, + output: Option, + tariff: Option, +) -> PublicationInput { + let mut input = PublicationInput::new(proxy_id, "openai", "openai", false, attempt); + input.budget_proof = Some(BudgetProof { + input_worst_case_tokens: input_worst, + requested_output_tokens: output, + tariff_microdollars_per_million_tokens: tariff, + }); + input +} + +async fn hard_rule(budgets: &UserBudgetService, model_id: Option<&str>, limits: BudgetLimits) { + budgets + .set_rule( + "budget-admin", + "budget-member", + model_id, + BudgetWindow::Day, + limits, + ) + .await + .expect("admin sets hard budget rule"); +} + +fn hard_limits(tokens: Option, microdollars: Option) -> BudgetLimits { + BudgetLimits { + tokens, + microdollars, + enforcement: BudgetEnforcement::Hard, + ..Default::default() + } +} + +async fn table_counts(database: &Database) -> (i64, i64) { + database + .call(|connection| { + Ok(( + connection.query_row("SELECT count(*) FROM requests", [], |row| { + row.get::<_, i64>(0) + })?, + connection.query_row( + "SELECT count(*) FROM user_budget_reservations", + [], + |row| row.get::<_, i64>(0), + )?, + )) + }) + .await + .unwrap() +} + +async fn reservation_hold( + database: &Database, + request_id: i64, +) -> (i64, i64, String, i64, i64, Option) { + database + .call(move |connection| { + connection.query_row( + "SELECT held_tokens, held_microdollars, enforcement, + bound_tokens, bound_microdollars, price_source + FROM user_budget_reservations WHERE request_id = ?1", + [request_id], + |row| { + Ok(( + row.get(0)?, + row.get(1)?, + row.get(2)?, + row.get(3)?, + row.get(4)?, + row.get(5)?, + )) + }, + ) + }) + .await + .unwrap() +} + +#[tokio::test] +async fn hard_token_rule_admits_bounded_request_with_worst_case_hold() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(1_000), None)).await; + // Single-attempt worst case is 100 + 50 = 150; the initial hold covers + // two attempt-equivalents (concrete attempt plus selector overhead). + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-hard-token", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("bounded hard-token request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("bounded request should publish"); + }; + let request_id = published.identity.db_request_id; + let (held_tokens, held_micros, enforcement, bound_tokens, bound_micros, price) = + reservation_hold(&fixture.database, request_id).await; + assert_eq!(held_tokens, 300); + assert_eq!(enforcement, "hard"); + assert_eq!(bound_tokens, 300); + // Token-hard leaves the cost dimension on the estimator projection: no + // hard price is claimed. + assert!(held_micros > 0, "estimated cost projection is preserved"); + assert_eq!(bound_micros, 0); + assert_eq!(price, None); + assert_eq!(table_counts(&fixture.database).await, (1, 1)); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn hard_token_rule_denies_unbounded_request_without_upstream_io() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(1_000_000), None)).await; + let result = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-hard-unbounded", 1, 100, None, None), + Some(attribution()), + ) + .await; + assert!( + matches!(result, Err(PublicationError::BudgetNotEnforceable)), + "unbounded hard-token request must be denied as not enforceable, got {result:?}" + ); + // Denial happens inside the publication transaction before claim + // conversion, so no request row exists for later dispatch. + assert_eq!(table_counts(&fixture.database).await, (0, 0)); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn hard_token_rule_denies_missing_proof_without_upstream_io() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(1_000_000), None)).await; + let result = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + PublicationInput::new("budget-hard-noproof", "openai", "openai", false, 1), + Some(attribution()), + ) + .await; + assert!(matches!( + result, + Err(PublicationError::BudgetNotEnforceable) + )); + assert_eq!(table_counts(&fixture.database).await, (0, 0)); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn hard_usd_rule_denies_unknown_price_without_upstream_io() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(None, Some(1_000_000))).await; + // A bounded token proof with no operator tariff still cannot prove a + // hard USD ceiling: estimator fallbacks are never a hard price. + let result = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-hard-noprice", 1, 100, Some(50), None), + Some(attribution()), + ) + .await; + assert!(matches!( + result, + Err(PublicationError::BudgetNotEnforceable) + )); + assert_eq!(table_counts(&fixture.database).await, (0, 0)); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn hard_usd_rule_admits_with_operator_tariff_hold() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(None, Some(1_000_000))).await; + // This positive bound rounds up instead of becoming zero at a tiny tariff. + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-hard-tariff", 1, 100, Some(50), Some(1)), + Some(attribution()), + ) + .await + .expect("tariff-backed hard-USD request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + assert_eq!( + reservation_hold(&fixture.database, published.identity.db_request_id).await, + ( + 42, + 1, + "hard".to_owned(), + 0, + 1, + Some("operator_tariff".to_owned()) + ) + ); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn hard_token_and_usd_rule_admits_with_combined_worst_case_hold() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(100_000), Some(1_000_000))).await; + // Worst case 150 tokens times two equivalents; USD converts the same + // 300-token bound at $3/MTok. + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-hard-combined", 1, 100, Some(50), Some(3_000_000)), + Some(attribution()), + ) + .await + .expect("combined hard rule publishes within bounds"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + assert_eq!( + reservation_hold(&fixture.database, published.identity.db_request_id).await, + ( + 300, + 900, + "hard".to_owned(), + 300, + 900, + Some("operator_tariff".to_owned()) + ) + ); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn hard_retry_denies_extension_beyond_remaining_budget() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(400), None)).await; + let service = PublicationService::new(fixture.database.clone()); + let first = service + .publish_attributed( + claim(&fixture).await, + hard_input("budget-hard-retry", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("initial 300-token hold fits the 400-token cap"); + let PublicationOutcome::Published(first) = first else { + panic!("first attempt should publish"); + }; + DurableFinalizer::new(fixture.database.clone()) + .finalize_failed_attempt( + &first.identity, + FinalizationData { + outcome: FinalizationOutcome::UpstreamError, + status_code: Some(502), + ..FinalizationData::default() + }, + Some(first.claim), + ) + .await + .expect("failed attempt keeps the logical hold"); + // Retry would extend by one 150-token equivalent: 300 + 150 > 400. + let retry = service + .publish_attributed( + claim(&fixture).await, + hard_input("budget-hard-retry", 2, 100, Some(50), None), + Some(attribution()), + ) + .await; + assert!( + matches!(retry, Err(PublicationError::BudgetExceeded)), + "retry beyond remaining hard budget must be denied, got {retry:?}" + ); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn hard_retry_extends_within_remaining_budget() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(1_000), None)).await; + let service = PublicationService::new(fixture.database.clone()); + let first = service + .publish_attributed( + claim(&fixture).await, + hard_input("budget-hard-retry-ok", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("initial hold publishes"); + let PublicationOutcome::Published(first) = first else { + panic!("first attempt should publish"); + }; + let proxy_request_id = first.identity.proxy_request_id.clone(); + DurableFinalizer::new(fixture.database.clone()) + .finalize_failed_attempt( + &first.identity, + FinalizationData { + outcome: FinalizationOutcome::UpstreamError, + status_code: Some(502), + ..FinalizationData::default() + }, + Some(first.claim), + ) + .await + .expect("failed attempt keeps the logical hold"); + let retry = service + .publish_attributed( + claim(&fixture).await, + hard_input(&proxy_request_id, 2, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("retry within remaining hard budget publishes"); + let PublicationOutcome::Published(retry) = retry else { + panic!("retry should publish"); + }; + // 300 initial + 150 retry extension, all inside the 1000-token cap. + let (held, _, _, _, _, _) = + reservation_hold(&fixture.database, retry.identity.db_request_id).await; + assert_eq!(held, 450); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn hard_overflow_and_nonpositive_output_fail_closed() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(i64::MAX), None)).await; + let service = PublicationService::new(fixture.database.clone()); + for (label, output) in [ + ("overflow", Some(i64::MAX)), + ("zero", Some(0)), + ("negative", Some(-5)), + ] { + let result = service + .publish_attributed( + claim(&fixture).await, + hard_input("budget-hard-overflow", 1, 100, output, None), + Some(attribution()), + ) + .await; + assert!( + matches!(result, Err(PublicationError::BudgetNotEnforceable)), + "{label} output bound must fail closed, got {result:?}" + ); + } + assert_eq!(table_counts(&fixture.database).await, (0, 0)); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn hard_settlement_records_provider_overrun_without_hiding() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(100_000), None)).await; + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-hard-overrun", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("bounded request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + // The provider reports 400 tokens against a 300-token worst-case hold. + // Settlement must charge what was observed, not silently clamp to held. + DurableFinalizer::new(fixture.database.clone()) + .finalize_request( + &published.identity, + FinalizationData { + outcome: FinalizationOutcome::Completed, + input_tokens: 200, + output_tokens: 200, + input_tokens_reported: Some(200), + output_tokens_reported: Some(200), + ..FinalizationData::default() + }, + Some(published.claim), + ) + .await + .expect("terminal settlement records observed usage"); + let usage = budgets + .usage("budget-member", "2026-01-01", 10) + .await + .expect("usage reads succeed"); + assert!( + usage + .iter() + .all(|bucket| bucket.token_count == 400 && bucket.unknown_token_requests == 0), + "observed overrun must be charged visibly, got {usage:?}" + ); + let observed = fixture + .database + .call({ + let request_id = published.identity.db_request_id; + move |connection| { + connection.query_row( + "SELECT observed_tokens FROM user_budget_reservations WHERE request_id = ?1", + [request_id], + |row| row.get::<_, i64>(0), + ) + } + }) + .await + .unwrap(); + assert_eq!(observed, 400); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn zero_cost_tariff_admits_hard_usd_without_hold() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(None, Some(1_000_000))).await; + // An explicit zero tariff is a verified free declaration, distinct from + // unknown price: hard USD admits with a zero hold. + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-hard-zerocost", 1, 100, Some(50), Some(0)), + Some(attribution()), + ) + .await + .expect("verified zero-cost request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + assert_eq!( + reservation_hold(&fixture.database, published.identity.db_request_id).await, + ( + 42, + 0, + "hard".to_owned(), + 0, + 0, + Some("operator_tariff".to_owned()) + ) + ); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn endpoint_error_maps_not_enforceable_to_stable_429() { + let error = EndpointError::BudgetNotEnforceable; + assert_eq!(error.status().as_u16(), 429); + assert_eq!( + error.to_string(), + "User budget hard limit is not enforceable for this request" + ); + let body = endpoint_error_body(ClientSurface::ChatCompletions, &error.to_string()); + let parsed: serde_json::Value = serde_json::from_slice(&body).expect("error body is JSON"); + assert_eq!( + parsed["error"]["message"].as_str(), + Some("User budget hard limit is not enforceable for this request") + ); +} + +#[tokio::test] +async fn admission_extracts_per_surface_output_bounds() { + use eggpool::request::{AdmissionOptions, admit_compact_request, admit_request}; + + let chat_capped = + br#"{"model":"m","messages":[{"role":"user","content":"hi"}],"max_tokens":128}"#; + let admitted = admit_request( + chat_capped, + AdmissionOptions { + client_surface: ClientSurface::ChatCompletions, + ..AdmissionOptions::default() + }, + ) + .expect("chat admission succeeds"); + assert_eq!(admitted.budget_facts.requested_output_tokens, Some(128)); + assert!(admitted.budget_facts.input_worst_case_tokens >= chat_capped.len() as i64); + + let chat_uncapped = br#"{"model":"m","messages":[{"role":"user","content":"hi"}]}"#; + let admitted = admit_request( + chat_uncapped, + AdmissionOptions { + client_surface: ClientSurface::ChatCompletions, + ..AdmissionOptions::default() + }, + ) + .expect("uncapped chat admission succeeds"); + assert_eq!(admitted.budget_facts.requested_output_tokens, None); + + let chat_zero = br#"{"model":"m","messages":[{"role":"user","content":"hi"}],"max_tokens":0}"#; + let admitted = admit_request( + chat_zero, + AdmissionOptions { + client_surface: ClientSurface::ChatCompletions, + ..AdmissionOptions::default() + }, + ) + .expect("zero-cap chat admission succeeds structurally"); + // An unusable cap degrades to unbounded: hard rules fail closed on it. + assert_eq!(admitted.budget_facts.requested_output_tokens, None); + + let responses_capped = br#"{"model":"m","input":"hi","max_output_tokens":64}"#; + let admitted = admit_request( + responses_capped, + AdmissionOptions { + client_surface: ClientSurface::Responses, + ..AdmissionOptions::default() + }, + ) + .expect("responses admission succeeds"); + assert_eq!(admitted.budget_facts.requested_output_tokens, Some(64)); + + let messages_capped = + br#"{"model":"m","max_tokens":32,"messages":[{"role":"user","content":"hi"}]}"#; + let admitted = admit_request( + messages_capped, + AdmissionOptions { + client_surface: ClientSurface::Messages, + ..AdmissionOptions::default() + }, + ) + .expect("messages admission succeeds"); + assert_eq!(admitted.budget_facts.requested_output_tokens, Some(32)); + + let compact_capped = br#"{"model":"m","input":"history","max_output_tokens":16}"#; + let compact = admit_compact_request( + compact_capped, + AdmissionOptions { + client_surface: ClientSurface::Responses, + ..AdmissionOptions::default() + }, + ) + .expect("compact admission succeeds"); + assert_eq!(compact.budget_facts.requested_output_tokens, Some(16)); +} + +#[tokio::test] +async fn admission_preserves_routing_inputs_for_transcode_eligibility() { + use eggpool::request::{AdmissionOptions, StaticRoutingFacts, admit_request}; + + // Regression guard: routing facts built from admission must carry the + // transcode protocols (and other routing inputs) untouched, or + // cross-surface requests lose upstream eligibility with no admission + // error. M008 dropped this copy and broke messages-to-responses + // transcoding while native surfaces kept passing. + let body = br#"{"model":"m","max_tokens":32,"messages":[{"role":"user","content":"hi"}]}"#; + let admitted = admit_request( + body.as_slice(), + AdmissionOptions { + client_surface: ClientSurface::Messages, + ..AdmissionOptions::default() + }, + ) + .expect("messages admission succeeds"); + let mut inputs = StaticRoutingFacts { + requested_protocol: Some("anthropic".into()), + transcode_protocols: vec!["openai".into(), "anthropic".into(), "gemini".into()], + catalog_stale_after_s: Some(60), + now: 1_700_000_000, + ..StaticRoutingFacts::default() + }; + inputs.capability_policy.insert("k".into(), "v".into()); + let facts = admitted.routing_facts(&inputs); + assert_eq!( + facts.transcode_protocols, + vec![ + "openai".to_owned(), + "anthropic".to_owned(), + "gemini".to_owned() + ] + ); + assert_eq!(facts.requested_protocol.as_deref(), Some("anthropic")); + assert_eq!(facts.catalog_stale_after_s, Some(60)); + assert_eq!(facts.now, 1_700_000_000); + assert_eq!(facts.request_surface, "messages"); + assert_eq!(facts.budget_requested_output_tokens, Some(32)); +} + +#[tokio::test] +async fn provider_tariff_overflow_is_rejected_by_config_validation() { + let mut config = Config::default(); + let mut provider = eggpool::config::ProviderConfig { + id: "provider-a".into(), + base_url: "https://provider.invalid/v1".into(), + protocols: vec!["openai".into()], + auth: eggpool::config::ProviderAuthConfig { + mode: "none".into(), + ..Default::default() + }, + ..Default::default() + }; + provider.accounts.push(eggpool::config::AccountConfig { + name: "budget-account".into(), + ..Default::default() + }); + provider.budget.max_microdollars_per_million_tokens = Some(u64::MAX); + config.providers.insert("provider-a".into(), provider); + assert!( + config.validate().is_err(), + "tariffs beyond i64 must be rejected" + ); + + let mut config = Config::default(); + let mut provider = eggpool::config::ProviderConfig { + id: "provider-a".into(), + base_url: "https://provider.invalid/v1".into(), + protocols: vec!["openai".into()], + auth: eggpool::config::ProviderAuthConfig { + mode: "none".into(), + ..Default::default() + }, + ..Default::default() + }; + provider.accounts.push(eggpool::config::AccountConfig { + name: "budget-account".into(), + ..Default::default() + }); + provider.budget.max_microdollars_per_million_tokens = Some(3_000_000); + config.providers.insert("provider-a".into(), provider); + config.validate().expect("in-range tariff validates"); +} + +#[tokio::test] +async fn hard_proof_flows_from_admission_through_claim_snapshot() { + use eggpool::coordinator::budget_proof_for_claim; + use eggpool::request::{AdmissionOptions, StaticRoutingFacts, admit_request}; + + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule( + &budgets, + None, + hard_limits(Some(100_000_000), Some(1_000_000_000)), + ) + .await; + + // Admission extracts the caller output cap and the byte-length input + // bound; routing facts and the claim snapshot carry them untouched. + let body = + br#"{"model":"model-a","messages":[{"role":"user","content":"hi"}],"max_tokens":64}"#; + let admitted = admit_request( + body.as_slice(), + AdmissionOptions { + client_surface: ClientSurface::ChatCompletions, + ..AdmissionOptions::default() + }, + ) + .expect("chat admission succeeds"); + let mut inputs = StaticRoutingFacts::default(); + inputs.known_provider_ids.insert("provider-a".into()); + inputs.requested_protocol = Some("openai".into()); + let facts = admitted.routing_facts(&inputs); + assert_eq!(facts.budget_requested_output_tokens, Some(64)); + let claim = fixture + .router + .select_and_claim(&facts, &BTreeSet::new()) + .await + .unwrap() + .expect("fixture route is selectable"); + assert_eq!(claim.budget_requested_output_tokens(), Some(64)); + assert!(claim.budget_input_worst_case_tokens() >= body.len() as i64); + + // The coordinator assembles the proof exactly as production does: + // snapshot facts plus the selected provider's tariff snapshot. + let worst = claim.budget_input_worst_case_tokens() + 64; + let mut input = PublicationInput::new("budget-carriage", "openai", "openai", false, 1); + input.budget_proof = Some(budget_proof_for_claim(&claim, Some(3_000_000))); + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed(claim, input, Some(attribution())) + .await + .expect("carried proof publishes under hard rules"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + let (held_tokens, held_micros, enforcement, bound_tokens, bound_micros, price) = + reservation_hold(&fixture.database, published.identity.db_request_id).await; + assert_eq!(held_tokens, worst * 2); + assert_eq!(held_micros, worst * 2 * 3); + assert_eq!(enforcement, "hard"); + assert_eq!(bound_tokens, worst * 2); + assert_eq!(bound_micros, worst * 2 * 3); + assert_eq!(price, Some("operator_tariff".to_owned())); + fixture.database.close().await.unwrap(); +} + +// --------------------------------------------------------------------------- +// M010 crash recovery: budget holds converge on restart without folding +// usage, and a released hold never blocks the next admitted request. +// --------------------------------------------------------------------------- + +async fn hold_status(database: &Database, request_id: i64) -> (String, Option) { + database + .call(move |connection| { + connection.query_row( + "SELECT status, settled_at FROM user_budget_reservations WHERE request_id = ?1", + [request_id], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + }) + .await + .unwrap() +} + +async fn budget_row_counts(database: &Database) -> (i64, i64, i64) { + database + .call(|connection| { + Ok(( + connection.query_row( + "SELECT count(*) FROM user_budget_reservations WHERE status = 'active'", + [], + |row| row.get::<_, i64>(0), + )?, + connection.query_row( + "SELECT count(*) FROM user_budget_attempt_usage", + [], + |row| row.get::<_, i64>(0), + )?, + connection.query_row("SELECT count(*) FROM user_budget_usage", [], |row| { + row.get::<_, i64>(0) + })?, + )) + }) + .await + .unwrap() +} + +async fn concurrent_hard_rule(budgets: &UserBudgetService) { + budgets + .set_rule( + "budget-admin", + "budget-member", + None, + BudgetWindow::Day, + BudgetLimits { + tokens: Some(1_000_000), + concurrent: Some(1), + enforcement: BudgetEnforcement::Hard, + ..Default::default() + }, + ) + .await + .expect("admin sets concurrent hard budget rule"); +} + +#[tokio::test] +async fn crash_after_publication_releases_user_hold_without_folding_usage() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + concurrent_hard_rule(&budgets).await; + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-crash-hold", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("bounded hard-token request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + let request_id = published.identity.db_request_id; + // The crash shape: committed publication, dead process. The hold is + // durably active and no usage has been folded. + assert_eq!(hold_status(&fixture.database, request_id).await.0, "active"); + assert_eq!(budget_row_counts(&fixture.database).await, (1, 0, 0)); + drop(published.claim); + + let report = CrashReconciler::new(fixture.database.clone()) + .reconcile_once() + .await + .expect("restart reconciliation succeeds"); + assert_eq!(report.requests_interrupted, 1); + assert_eq!(report.user_budget_holds_released, 1); + assert_eq!(report.instance_budget_holds_released, 0); + assert!(report.classification.pending_requests_with_active_user_hold >= 1); + + // The hold is compensation, not credit: released, never settled, and no + // usage bucket or attempt fact was folded. + assert_eq!( + hold_status(&fixture.database, request_id).await, + ("released".to_owned(), None) + ); + assert_eq!(budget_row_counts(&fixture.database).await, (0, 0, 0)); + let request_status: String = fixture + .database + .call(move |connection| { + connection.query_row( + "SELECT status FROM requests WHERE id = ?1", + [request_id], + |row| row.get(0), + ) + }) + .await + .unwrap(); + assert_eq!(request_status, "interrupted"); + + // A second pass is a no-op: convergence is idempotent. + let second = CrashReconciler::new(fixture.database.clone()) + .reconcile_once() + .await + .expect("second pass succeeds"); + assert!(second.converged); + assert_eq!(second.user_budget_holds_released, 0); + + // The released hold frees the single concurrency slot: the next + // admitted request publishes instead of wedging on a phantom hold. + let next = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-crash-next", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("next request admits after crash release"); + let PublicationOutcome::Published(next) = next else { + panic!("next request should publish"); + }; + assert_eq!( + hold_status(&fixture.database, next.identity.db_request_id) + .await + .0, + "active" + ); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn retry_after_crash_release_fails_closed_without_new_hold() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(1_000_000), None)).await; + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-crash-retry", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("bounded hard-token request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + drop(published.claim); + CrashReconciler::new(fixture.database.clone()) + .reconcile_once() + .await + .expect("restart reconciliation succeeds"); + // The interrupted request can never settle, so a retry under the same + // proxy id must deny instead of resurrecting a hold on dead work. The + // duplicate gate refuses non-pending requests before any budget or + // dispatch work happens. + let retry = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-crash-retry", 2, 100, Some(50), None), + Some(attribution()), + ) + .await; + assert!( + matches!(retry, Err(PublicationError::DuplicateConflict { .. })), + "retry on an interrupted request must fail closed, got {retry:?}" + ); + assert_eq!(budget_row_counts(&fixture.database).await, (0, 0, 0)); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn compensated_publication_retry_reuses_hold_then_crash_releases_it() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(1_000_000), None)).await; + let injector = PublicationFaultInjector::fail_once_at(PublicationStage::AfterCommit); + let service = PublicationService::new(fixture.database.clone()).with_fault_injector(injector); + let error = service + .publish_attributed( + claim(&fixture).await, + hard_input("budget-crash-compensate", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect_err("post-commit interruption is surfaced"); + let PublicationError::PostCommit { mut interruption } = error else { + panic!("expected retained post-commit identity"); + }; + let request_id = interruption.identity.db_request_id; + service + .compensate_post_commit(&mut interruption) + .await + .expect("compensation converges"); + // In-process compensation releases the legacy reservation but keeps the + // request pending with its budget hold: a retry must reuse the same + // hold rather than leak a second one. + assert_eq!(hold_status(&fixture.database, request_id).await.0, "active"); + let retry = service + .publish_attributed( + claim(&fixture).await, + hard_input("budget-crash-compensate", 2, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("retry reuses the compensated hold"); + let PublicationOutcome::Published(retried) = retry else { + panic!("retry should publish a second attempt"); + }; + assert_eq!(retried.identity.db_request_id, request_id); + assert_eq!(budget_row_counts(&fixture.database).await, (1, 0, 0)); + // Crash before the retry dispatches: the extended hold converges on + // restart with no usage folded. + drop(retried.claim); + let report = CrashReconciler::new(fixture.database.clone()) + .reconcile_once() + .await + .expect("restart reconciliation succeeds"); + assert_eq!(report.user_budget_holds_released, 1); + assert_eq!( + hold_status(&fixture.database, request_id).await, + ("released".to_owned(), None) + ); + assert_eq!(budget_row_counts(&fixture.database).await, (0, 0, 0)); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn settled_request_survives_restart_without_double_fold() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(1_000_000), None)).await; + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-crash-settled", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("bounded hard-token request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + let request_id = published.identity.db_request_id; + DurableFinalizer::new(fixture.database.clone()) + .finalize_request( + &published.identity, + FinalizationData::default(), + Some(published.claim), + ) + .await + .expect("terminal request settles"); + assert_eq!( + hold_status(&fixture.database, request_id).await.0, + "settled" + ); + assert_eq!(budget_row_counts(&fixture.database).await, (0, 1, 2)); + // A restart after settlement must not touch the settled hold or fold + // usage a second time. + let report = CrashReconciler::new(fixture.database.clone()) + .reconcile_once() + .await + .expect("restart reconciliation succeeds"); + assert_eq!(report.user_budget_holds_released, 0); + assert_eq!(report.instance_budget_holds_released, 0); + assert!(report.converged); + assert_eq!( + hold_status(&fixture.database, request_id).await.0, + "settled" + ); + assert_eq!(budget_row_counts(&fixture.database).await, (0, 1, 2)); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn duplicate_terminal_after_crash_release_cannot_double_fold() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(1_000_000), None)).await; + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-crash-duplicate", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("bounded hard-token request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + let identity = published.identity.clone(); + drop(published.claim); + CrashReconciler::new(fixture.database.clone()) + .reconcile_once() + .await + .expect("restart reconciliation succeeds"); + // A stale duplicate terminal command for the interrupted request + // conflicts instead of folding usage onto a released hold. + let duplicate = DurableFinalizer::new(fixture.database.clone()) + .finalize_request(&identity, FinalizationData::default(), None) + .await; + assert!( + matches!(duplicate, Err(FinalizationError::TerminalConflict { .. })), + "duplicate terminal must conflict, got {duplicate:?}" + ); + assert_eq!(budget_row_counts(&fixture.database).await, (0, 0, 0)); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn cancel_terminalizes_and_settles_user_hold_conservatively() { + let fixture = fixture().await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(1_000_000), None)).await; + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-crash-cancel", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("bounded hard-token request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + let request_id = published.identity.db_request_id; + let held: i64 = fixture + .database + .call(move |connection| { + connection.query_row( + "SELECT held_microdollars FROM user_budget_reservations WHERE request_id = ?1", + [request_id], + |row| row.get(0), + ) + }) + .await + .unwrap(); + DurableFinalizer::new(fixture.database.clone()) + .finalize_request( + &published.identity, + FinalizationData { + outcome: FinalizationOutcome::ClientCancelled, + ..FinalizationData::default() + }, + Some(published.claim), + ) + .await + .expect("cancel terminalizes"); + // Cancel is terminal: the hold settles at the conservative unknown-usage + // charge and a restart leaves it untouched. Day and month buckets each + // carry the held charge exactly once. + assert_eq!( + hold_status(&fixture.database, request_id).await.0, + "settled" + ); + let bucket_costs: Vec = fixture + .database + .call(|connection| { + let mut statement = connection + .prepare("SELECT cost_microdollars FROM user_budget_usage ORDER BY window")?; + let rows = statement.query_map([], |row| row.get(0))?; + rows.collect::, _>>() + }) + .await + .unwrap(); + assert_eq!(bucket_costs, vec![held, held]); + let report = CrashReconciler::new(fixture.database.clone()) + .reconcile_once() + .await + .expect("restart reconciliation succeeds"); + assert!(report.converged); + assert_eq!(report.user_budget_holds_released, 0); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn backup_during_active_reservation_captures_hold_verbatim() { + let directory = tempfile::tempdir().expect("temp root"); + let config_path = directory.path().join("config.toml"); + let db_path = directory.path().join("usage.sqlite3"); + let mut config = Config::default(); + config.database.path = db_path.display().to_string(); + std::fs::write(&config_path, toml::to_string(&config).expect("config TOML")) + .expect("config writes"); + let fixture = fixture_on(&DatabaseConfig { + path: db_path.display().to_string(), + ..Default::default() + }) + .await; + let budgets = UserBudgetService::new(fixture.database.clone()); + hard_rule(&budgets, None, hard_limits(Some(1_000_000), None)).await; + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim(&fixture).await, + hard_input("budget-backup-hold", 1, 100, Some(50), None), + Some(attribution()), + ) + .await + .expect("bounded hard-token request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + let request_id = published.identity.db_request_id; + + // Snapshot while the hold is active: the online backup must carry the + // pending request and its active hold exactly as committed. + let service = BackupService::from_config(&config_path, &config) + .with_output_dir(directory.path().join("backups")); + let archive = service + .create(&fixture.database) + .await + .expect("backup during active hold") + .archive; + BackupService::validate_archive(&archive).expect("archive validates"); + + // Restore over the live paths and prove the hold survived the round + // trip with its active status and held amounts intact. The WAL is + // checkpointed before close so no stale sidecar can shadow the + // restored image on reopen. + drop(published.claim); + fixture + .database + .call(|connection| { + connection.query_row("PRAGMA wal_checkpoint(TRUNCATE)", [], |_| Ok(()))?; + Ok(()) + }) + .await + .expect("wal checkpoints"); + fixture + .database + .close() + .await + .expect("live database closes"); + let restored = service.recover(&archive).expect("restore"); + assert_eq!(restored.database, db_path); + let reopened = Database::open(DatabaseConfig { + path: db_path.display().to_string(), + ..Default::default() + }) + .await + .expect("restored database opens"); + let request_status: String = reopened + .call(move |connection| { + connection.query_row( + "SELECT status FROM requests WHERE id = ?1", + [request_id], + |row| row.get(0), + ) + }) + .await + .unwrap(); + assert_eq!(request_status, "pending"); + assert_eq!(hold_status(&reopened, request_id).await.0, "active"); + // The restored copy reconciles exactly like a crashed live database: + // the request interrupts and the hold releases with no usage folded. + let report = CrashReconciler::new(reopened.clone()) + .reconcile_once() + .await + .expect("restored database reconciles"); + assert_eq!(report.requests_interrupted, 1); + assert_eq!(report.user_budget_holds_released, 1); + assert_eq!(hold_status(&reopened, request_id).await.0, "released"); + reopened.close().await.expect("restored database closes"); +} diff --git a/rust/tests/on_prem_global_budgets.rs b/rust/tests/on_prem_global_budgets.rs new file mode 100644 index 000000000..74e23c35b --- /dev/null +++ b/rust/tests/on_prem_global_budgets.rs @@ -0,0 +1,1383 @@ +use std::collections::BTreeSet; +use std::time::Instant; + +use eggpool::{ + Config, + accounts::{AccountRegistry, CredentialStore}, + catalog::{ModelCatalogCache, ModelInput, ProtocolResolutionStatus}, + coordinator::{ + CrashReconciler, DurableFinalizer, FinalizationData, FinalizationOutcome, PublicationError, + PublicationInput, PublicationOutcome, PublicationService, + }, + db::{Account, Database, DatabaseConfig, MigrationRunner}, + identity::{AuthenticatedPrincipal, PrincipalKind}, + instance_budget::{InstanceBudgetError, InstanceBudgetService}, + quota::{AccountQuota, QuotaEstimator}, + routing::{EligibilityPolicy, RoutingRequestFacts, RoutingRouter}, + user_budget::{BudgetEnforcement, BudgetLimits, BudgetProof, BudgetWindow, UserBudgetService}, +}; + +struct Fixture { + database: Database, + router: RoutingRouter, +} + +async fn fixture() -> Fixture { + fixture_on(&DatabaseConfig::default()).await +} + +async fn fixture_on(config: &DatabaseConfig) -> Fixture { + let database = Database::open(config.clone()) + .await + .expect("database opens"); + MigrationRunner::new(&database) + .run() + .await + .expect("schema migrates"); + database + .call(|connection| { + connection.execute( + "INSERT INTO accounts (id, name, api_key_env, enabled, provider_id) + VALUES (1, 'budget-account', 'UNUSED', 1, 'provider-a')", + [], + )?; + for model in ["model-a", "model-b"] { + connection.execute( + "INSERT INTO models (model_id, protocol, provider_id, resolution_status) + VALUES (?1, 'openai', 'provider-a', 'resolved')", + [model], + )?; + } + connection.execute( + "INSERT INTO users(id, username, normalized_username, role, password_hash) + VALUES ('budget-admin', 'budget-admin', 'budget-admin', 'admin', 'test-hash')", + [], + )?; + for user in ["budget-member", "budget-member-2"] { + connection.execute( + "INSERT INTO users(id, username, normalized_username, role, password_hash) + VALUES (?1, ?1, ?1, 'member', 'test-hash')", + [user], + )?; + } + Ok(()) + }) + .await + .expect("fixture rows insert"); + + let mut config = Config::default(); + let mut provider = eggpool::config::ProviderConfig { + id: "provider-a".into(), + base_url: "https://provider.invalid/v1".into(), + protocols: vec!["openai".into()], + auth: eggpool::config::ProviderAuthConfig { + mode: "none".into(), + ..Default::default() + }, + ..Default::default() + }; + provider.accounts.push(eggpool::config::AccountConfig { + name: "budget-account".into(), + ..Default::default() + }); + config.providers.insert("provider-a".into(), provider); + config.validate().expect("fixture config validates"); + let registry = AccountRegistry::from_config( + &config, + &[Account { + id: 1, + name: "budget-account".into(), + api_key_env: "UNUSED".into(), + enabled: true, + weight: 1.0, + provider_id: "provider-a".into(), + }], + &CredentialStore::default(), + ) + .expect("registry builds"); + let mut catalog = ModelCatalogCache::default(); + catalog.set_account_provider("budget-account", "provider-a"); + let models: Vec = ["model-a", "model-b"] + .into_iter() + .map(|model_id| { + let mut model = ModelInput::new(model_id); + model.protocol = Some("openai".into()); + model.protocol_source = Some("fixture".into()); + model.resolution_status = ProtocolResolutionStatus::Resolved; + model + }) + .collect(); + catalog + .update_from_account("budget-account", "provider-a", &models, true, true) + .expect("catalog models"); + let router = RoutingRouter::new( + registry, + catalog, + QuotaEstimator::new([AccountQuota::new("budget-account")]), + None, + EligibilityPolicy::default(), + ); + Fixture { database, router } +} + +async fn extra_member(database: &Database, username: &str) { + database + .call({ + let username = username.to_owned(); + move |connection| { + connection.execute( + "INSERT INTO users(id, username, normalized_username, role, password_hash) + VALUES (?1, ?1, ?1, 'member', 'test-hash')", + [&username], + )?; + Ok(()) + } + }) + .await + .expect("extra member inserts"); +} + +async fn claim_for(fixture: &Fixture, model: &str) -> eggpool::routing::SelectionClaim { + let mut facts = RoutingRequestFacts::new(model); + facts.requested_protocol = Some("openai".into()); + facts.client_protocol = Some("openai".into()); + facts.projected_tokens = 42; + fixture + .router + .select_and_claim(&facts, &BTreeSet::new()) + .await + .unwrap() + .unwrap() +} + +fn attribution_for(user: &str) -> eggpool::identity::RequestAttribution { + AuthenticatedPrincipal { + user_id: Some(user.into()), + credential_id: None, + role: Some("member".into()), + scopes: vec!["inference".into()], + auth_epoch: Some(1), + kind: PrincipalKind::AgentToken, + } + .request_attribution() +} + +fn estimated_input(proxy_id: &str, attempt: i64) -> PublicationInput { + PublicationInput::new(proxy_id, "openai", "openai", false, attempt) +} + +fn hard_input( + proxy_id: &str, + attempt: i64, + output: Option, + tariff: Option, +) -> PublicationInput { + let mut input = PublicationInput::new(proxy_id, "openai", "openai", false, attempt); + input.budget_proof = Some(BudgetProof { + input_worst_case_tokens: 100, + requested_output_tokens: output, + tariff_microdollars_per_million_tokens: tariff, + }); + input +} + +fn estimated_limits( + requests: Option, + tokens: Option, + microdollars: Option, + concurrent: Option, +) -> BudgetLimits { + BudgetLimits { + requests, + tokens, + microdollars, + concurrent, + ..Default::default() + } +} + +fn hard_limits(tokens: Option, microdollars: Option) -> BudgetLimits { + BudgetLimits { + tokens, + microdollars, + enforcement: BudgetEnforcement::Hard, + ..Default::default() + } +} + +async fn set_global( + budgets: &InstanceBudgetService, + actor: &str, + model_id: Option<&str>, + limits: BudgetLimits, +) { + budgets + .set_rule(actor, model_id, BudgetWindow::Day, limits) + .await + .expect("admin sets global rule"); +} + +async fn instance_counts(database: &Database) -> (i64, i64, i64) { + database + .call(|connection| { + Ok(( + connection.query_row("SELECT count(*) FROM requests", [], |row| { + row.get::<_, i64>(0) + })?, + connection.query_row( + "SELECT count(*) FROM instance_budget_reservations WHERE status = 'active'", + [], + |row| row.get::<_, i64>(0), + )?, + connection.query_row("SELECT count(*) FROM instance_budget_usage", [], |row| { + row.get::<_, i64>(0) + })?, + )) + }) + .await + .unwrap() +} + +async fn instance_usage_requests(database: &Database) -> i64 { + database + .call(|connection| { + connection.query_row( + "SELECT coalesce(sum(request_count), 0) FROM instance_budget_usage", + [], + |row| row.get::<_, i64>(0), + ) + }) + .await + .unwrap() +} + +#[tokio::test] +async fn global_request_last_slot_admits_exactly_one_across_users() { + let fixture = fixture().await; + let budgets = InstanceBudgetService::new(fixture.database.clone()); + set_global( + &budgets, + "budget-admin", + None, + estimated_limits(Some(1), None, None, None), + ) + .await; + + let service = PublicationService::new(fixture.database.clone()); + let first = service.publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-race-a", 1), + Some(attribution_for("budget-member")), + ); + let second = service.publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-race-b", 1), + Some(attribution_for("budget-member-2")), + ); + let (first, second) = tokio::join!(first, second); + let mut winners = 0; + let mut denied = 0; + for result in [first, second] { + match result { + Ok(PublicationOutcome::Published(_)) => winners += 1, + Err(PublicationError::BudgetExceeded) => denied += 1, + other => panic!("unexpected global race result: {other:?}"), + } + } + assert_eq!((winners, denied), (1, 1)); + // One logical request, one shared hold: two separately authenticated + // accounts cannot both take the final global slot. + assert_eq!(instance_counts(&fixture.database).await, (1, 1, 0)); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn global_last_slot_race_scales_to_10_and_49_identities() { + for size in [10_usize, 49] { + let fixture = fixture().await; + for index in 0..size { + extra_member(&fixture.database, &format!("race-user-{index:02}")).await; + } + let budgets = InstanceBudgetService::new(fixture.database.clone()); + set_global( + &budgets, + "budget-admin", + None, + estimated_limits(Some(1), None, None, None), + ) + .await; + + // Claims are built sequentially; only the durable publication races, + // which is the atomicity boundary under test. + let mut claims = Vec::new(); + for _ in 0..size { + claims.push(claim_for(&fixture, "model-a").await); + } + let started = Instant::now(); + let mut racers = tokio::task::JoinSet::new(); + for (index, claim) in claims.into_iter().enumerate() { + let service = PublicationService::new(fixture.database.clone()); + let user = format!("race-user-{index:02}"); + racers.spawn(async move { + service + .publish_attributed( + claim, + estimated_input(&format!("global-race-{index:02}"), 1), + Some(attribution_for(&user)), + ) + .await + }); + } + let mut winners = 0; + let mut denied = 0; + while let Some(outcome) = racers.join_next().await { + match outcome.expect("racer completes") { + Ok(PublicationOutcome::Published(_)) => winners += 1, + Err(PublicationError::BudgetExceeded) => denied += 1, + other => panic!("unexpected {size}-way race result: {other:?}"), + } + } + let elapsed = started.elapsed(); + eprintln!("global last-slot race with {size} identities took {elapsed:?}"); + assert_eq!(winners, 1, "{size} racers must leave exactly one winner"); + assert_eq!(denied, size - 1); + assert_eq!(instance_counts(&fixture.database).await, (1, 1, 0)); + fixture.database.close().await.unwrap(); + } +} + +#[tokio::test] +async fn global_and_user_rules_intersect_without_bypass() { + let fixture = fixture().await; + let user_budgets = UserBudgetService::new(fixture.database.clone()); + user_budgets + .set_rule( + "budget-admin", + "budget-member", + None, + BudgetWindow::Day, + estimated_limits(Some(10), None, None, None), + ) + .await + .unwrap(); + let global = InstanceBudgetService::new(fixture.database.clone()); + set_global( + &global, + "budget-admin", + None, + estimated_limits(Some(1), None, None, None), + ) + .await; + + let service = PublicationService::new(fixture.database.clone()); + service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("intersect-a", 1), + Some(attribution_for("budget-member")), + ) + .await + .expect("first request satisfies both scopes"); + // The user scope still allows nine more, but the global scope is spent. + let denied = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("intersect-b", 1), + Some(attribution_for("budget-member")), + ) + .await; + assert!( + matches!(denied, Err(PublicationError::BudgetExceeded)), + "global exhaustion must deny despite user headroom, got {denied:?}" + ); + // A different user with a fresh user budget is still globally capped. + let other = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("intersect-c", 1), + Some(attribution_for("budget-member-2")), + ) + .await; + assert!(matches!(other, Err(PublicationError::BudgetExceeded))); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn model_scoped_global_shares_one_budget_across_two_models() { + let fixture = fixture().await; + let global = InstanceBudgetService::new(fixture.database.clone()); + set_global( + &global, + "budget-admin", + None, + estimated_limits(Some(2), None, None, None), + ) + .await; + // Instance-model ceiling on model-a only. + global + .set_rule( + "budget-admin", + Some("model-a"), + BudgetWindow::Day, + estimated_limits(Some(1), None, None, None), + ) + .await + .unwrap(); + + let service = PublicationService::new(fixture.database.clone()); + service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("shared-a", 1), + Some(attribution_for("budget-member")), + ) + .await + .expect("first model-a request publishes"); + let model_a_again = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("shared-a2", 1), + Some(attribution_for("budget-member-2")), + ) + .await; + assert!( + matches!(model_a_again, Err(PublicationError::BudgetExceeded)), + "model-scoped global ceiling must deny, got {model_a_again:?}" + ); + // model-b shares the all-model global budget (one of two slots spent) + // but is outside the model-a ceiling. + service + .publish_attributed( + claim_for(&fixture, "model-b").await, + estimated_input("shared-b", 1), + Some(attribution_for("budget-member-2")), + ) + .await + .expect("model-b fits the remaining global slot"); + let global_spent = service + .publish_attributed( + claim_for(&fixture, "model-b").await, + estimated_input("shared-b2", 1), + Some(attribution_for("budget-member")), + ) + .await; + assert!(matches!( + global_spent, + Err(PublicationError::BudgetExceeded) + )); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn global_retry_and_settlement_count_one_logical_request() { + let fixture = fixture().await; + let global = InstanceBudgetService::new(fixture.database.clone()); + set_global( + &global, + "budget-admin", + None, + estimated_limits(Some(1), None, None, None), + ) + .await; + + let service = PublicationService::new(fixture.database.clone()); + let first = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-once", 1), + Some(attribution_for("budget-member")), + ) + .await + .expect("first attempt publishes"); + let PublicationOutcome::Published(first) = first else { + panic!("first attempt should publish"); + }; + DurableFinalizer::new(fixture.database.clone()) + .finalize_failed_attempt( + &first.identity, + FinalizationData { + outcome: FinalizationOutcome::UpstreamError, + status_code: Some(502), + input_tokens: 10, + output_tokens: 5, + input_tokens_reported: Some(10), + output_tokens_reported: Some(5), + cost_microdollars: 4, + provider_cost_microdollars: Some(4), + ..FinalizationData::default() + }, + Some(first.claim), + ) + .await + .expect("failed attempt keeps the global hold"); + // Retry reuses the logical hold: it must not consume a second global + // request even though the slot is the last one. + let retry = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-once", 2), + Some(attribution_for("budget-member")), + ) + .await + .expect("retry extends the same logical hold"); + let PublicationOutcome::Published(retry) = retry else { + panic!("retry should publish"); + }; + DurableFinalizer::new(fixture.database.clone()) + .finalize_request( + &retry.identity, + FinalizationData { + outcome: FinalizationOutcome::Completed, + input_tokens: 30, + output_tokens: 12, + input_tokens_reported: Some(30), + output_tokens_reported: Some(12), + cost_microdollars: 21, + provider_cost_microdollars: Some(21), + ..FinalizationData::default() + }, + Some(retry.claim), + ) + .await + .expect("terminal settlement folds one logical request"); + // One logical request settled into day + month buckets; the hold is gone. + assert_eq!(instance_counts(&fixture.database).await.1, 0); + assert_eq!(instance_usage_requests(&fixture.database).await, 2); + let usage = global + .usage("2026-01-01", 10) + .await + .expect("bounded instance usage reads succeed"); + assert_eq!(usage.len(), 2); + // Both attempts fold into one logical request: 10+5 failed, 30+12 + // terminal; provider-reported cost 4 + 21. + assert!(usage.iter().all(|bucket| bucket.request_count == 1 + && bucket.token_count == 57 + && bucket.cost_microdollars == 25)); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn global_concurrency_releases_only_after_terminal_settlement() { + let fixture = fixture().await; + let global = InstanceBudgetService::new(fixture.database.clone()); + set_global( + &global, + "budget-admin", + None, + estimated_limits(None, None, None, Some(1)), + ) + .await; + + let service = PublicationService::new(fixture.database.clone()); + let first = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-conc-a", 1), + Some(attribution_for("budget-member")), + ) + .await + .expect("first concurrent request publishes"); + let PublicationOutcome::Published(first) = first else { + panic!("first request should publish"); + }; + let second = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-conc-b", 1), + Some(attribution_for("budget-member-2")), + ) + .await; + assert!(matches!(second, Err(PublicationError::BudgetExceeded))); + DurableFinalizer::new(fixture.database.clone()) + .finalize_request( + &first.identity, + FinalizationData { + outcome: FinalizationOutcome::Completed, + input_tokens: 42, + input_tokens_reported: Some(42), + ..FinalizationData::default() + }, + Some(first.claim), + ) + .await + .expect("terminal settlement releases the global slot"); + service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-conc-c", 1), + Some(attribution_for("budget-member-2")), + ) + .await + .expect("settlement releases the concurrency slot"); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn global_window_projection_ignores_crossed_utc_boundaries() { + let fixture = fixture().await; + let global = InstanceBudgetService::new(fixture.database.clone()); + global + .set_rule( + "budget-admin", + None, + BudgetWindow::Day, + estimated_limits(Some(1), None, None, None), + ) + .await + .unwrap(); + global + .set_rule( + "budget-admin", + None, + BudgetWindow::Month, + estimated_limits(Some(1), None, None, None), + ) + .await + .unwrap(); + + let service = PublicationService::new(fixture.database.clone()); + let first = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-roll-a", 1), + Some(attribution_for("budget-member")), + ) + .await + .expect("first request publishes"); + let PublicationOutcome::Published(first) = first else { + panic!("first request should publish"); + }; + // Simulate the hold surviving across UTC resets: backdate both window + // starts. Today's windows must read as free; the concurrency count must + // still see the live hold. + fixture + .database + .call({ + let request_id = first.identity.db_request_id; + move |connection| { + connection.execute( + "UPDATE instance_budget_reservations + SET day_start = date('now', '-1 day'), + month_start = strftime('%Y-%m-01', 'now', '-1 month') + WHERE request_id = ?1", + [request_id], + )?; + Ok(()) + } + }) + .await + .unwrap(); + service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-roll-b", 1), + Some(attribution_for("budget-member-2")), + ) + .await + .expect("crossed windows free today's request slot"); + // Today's slot is now held again: a third request is denied. + let third = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-roll-c", 1), + Some(attribution_for("budget-member")), + ) + .await; + assert!(matches!(third, Err(PublicationError::BudgetExceeded))); + first.claim.release_all().expect("test claim is released"); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn legacy_unattributed_requests_count_against_global_ceiling() { + let fixture = fixture().await; + let global = InstanceBudgetService::new(fixture.database.clone()); + set_global( + &global, + "budget-admin", + None, + estimated_limits(Some(1), None, None, None), + ) + .await; + + let service = PublicationService::new(fixture.database.clone()); + service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-legacy", 1), + None, + ) + .await + .expect("unattributed request takes the global slot"); + // Switching to a named user cannot bypass the spent global ceiling, + // and the legacy hold settles into instance usage without a user row. + let bypass = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-legacy-b", 1), + Some(attribution_for("budget-member")), + ) + .await; + assert!( + matches!(bypass, Err(PublicationError::BudgetExceeded)), + "identity switch must not bypass the global ceiling, got {bypass:?}" + ); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn legacy_unattributed_settlement_folds_instance_usage() { + let fixture = fixture().await; + let global = InstanceBudgetService::new(fixture.database.clone()); + set_global( + &global, + "budget-admin", + None, + estimated_limits(Some(10), None, None, None), + ) + .await; + + let service = PublicationService::new(fixture.database.clone()); + let published = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-legacy-settle", 1), + None, + ) + .await + .expect("unattributed request publishes"); + let PublicationOutcome::Published(published) = published else { + panic!("request should publish"); + }; + DurableFinalizer::new(fixture.database.clone()) + .finalize_request( + &published.identity, + FinalizationData { + outcome: FinalizationOutcome::Completed, + input_tokens: 30, + output_tokens: 12, + input_tokens_reported: Some(30), + output_tokens_reported: Some(12), + cost_microdollars: 21, + provider_cost_microdollars: Some(21), + ..FinalizationData::default() + }, + Some(published.claim), + ) + .await + .expect("unattributed terminal settlement folds instance usage"); + assert_eq!(instance_usage_requests(&fixture.database).await, 2); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn disabled_user_does_not_erase_global_history() { + let fixture = fixture().await; + let global = InstanceBudgetService::new(fixture.database.clone()); + set_global( + &global, + "budget-admin", + None, + estimated_limits(Some(10), None, None, None), + ) + .await; + + let service = PublicationService::new(fixture.database.clone()); + let published = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-history", 1), + Some(attribution_for("budget-member")), + ) + .await + .expect("member request publishes"); + let PublicationOutcome::Published(published) = published else { + panic!("request should publish"); + }; + DurableFinalizer::new(fixture.database.clone()) + .finalize_request( + &published.identity, + FinalizationData { + outcome: FinalizationOutcome::Completed, + input_tokens: 42, + input_tokens_reported: Some(42), + ..FinalizationData::default() + }, + Some(published.claim), + ) + .await + .expect("settlement records global usage"); + fixture + .database + .call(|connection| { + connection.execute( + "UPDATE users SET state = 'disabled' WHERE id = 'budget-member'", + [], + )?; + Ok(()) + }) + .await + .unwrap(); + // Global counters never depended on the user row: history and remaining + // headroom are intact after disable. + assert_eq!(instance_usage_requests(&fixture.database).await, 2); + service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input("global-history-2", 1), + Some(attribution_for("budget-member-2")), + ) + .await + .expect("remaining global headroom is unaffected by the disable"); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn global_hard_token_and_usd_deny_unprovable_without_upstream_io() { + let fixture = fixture().await; + let global = InstanceBudgetService::new(fixture.database.clone()); + global + .set_rule( + "budget-admin", + None, + BudgetWindow::Day, + hard_limits(Some(1_000_000), None), + ) + .await + .unwrap(); + let service = PublicationService::new(fixture.database.clone()); + let unbounded = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + hard_input("global-hard-unbounded", 1, None, Some(3_000_000)), + Some(attribution_for("budget-member")), + ) + .await; + assert!(matches!( + unbounded, + Err(PublicationError::BudgetNotEnforceable) + )); + + global + .set_rule( + "budget-admin", + Some("model-b"), + BudgetWindow::Day, + hard_limits(None, Some(1_000_000)), + ) + .await + .unwrap(); + let unpriced = service + .publish_attributed( + claim_for(&fixture, "model-b").await, + hard_input("global-hard-unpriced", 1, Some(50), None), + Some(attribution_for("budget-member")), + ) + .await; + assert!(matches!( + unpriced, + Err(PublicationError::BudgetNotEnforceable) + )); + assert_eq!(instance_counts(&fixture.database).await, (0, 0, 0)); + + // The same hard scopes admit with proof: token worst case 150 × 2 and + // tariff-converted cost on the same hold. + let admitted = service + .publish_attributed( + claim_for(&fixture, "model-b").await, + hard_input("global-hard-ok", 1, Some(50), Some(1)), + Some(attribution_for("budget-member")), + ) + .await + .expect("proven hard request publishes"); + let PublicationOutcome::Published(admitted) = admitted else { + panic!("request should publish"); + }; + let held = fixture + .database + .call({ + let request_id = admitted.identity.db_request_id; + move |connection| { + connection.query_row( + "SELECT held_tokens, held_microdollars, enforcement, price_source + FROM instance_budget_reservations WHERE request_id = ?1", + [request_id], + |row| { + Ok(( + row.get::<_, i64>(0)?, + row.get::<_, i64>(1)?, + row.get::<_, String>(2)?, + row.get::<_, Option>(3)?, + )) + }, + ) + } + }) + .await + .unwrap(); + // Token-hard global rule is model-agnostic (matches model-b): 300-token + // hold. Cost-hard model-b rule rounds the tiny positive conversion up to + // one microdollar instead of admitting with a zero hold. + assert_eq!( + held, + ( + 300, + 1, + "hard".to_owned(), + Some("operator_tariff".to_owned()) + ) + ); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn global_rules_validate_admin_scope_and_reject_malformed() { + let fixture = fixture().await; + let global = InstanceBudgetService::new(fixture.database.clone()); + assert!(matches!( + global + .set_rule( + "budget-member", + None, + BudgetWindow::Day, + estimated_limits(Some(1), None, None, None), + ) + .await, + Err(InstanceBudgetError::ActorNotAdmin) + )); + assert!(matches!( + global + .remove_rule("budget-member", None, BudgetWindow::Day) + .await, + Err(InstanceBudgetError::ActorNotAdmin) + )); + for bad in [ + estimated_limits(Some(0), None, None, None), + estimated_limits(Some(-3), None, None, None), + estimated_limits(None, None, None, None), + ] { + assert!(matches!( + global + .set_rule("budget-admin", None, BudgetWindow::Day, bad) + .await, + Err(InstanceBudgetError::InvalidRule) + )); + } + global + .set_rule( + "budget-admin", + Some("model-a"), + BudgetWindow::Day, + estimated_limits(Some(5), None, None, None), + ) + .await + .unwrap(); + assert_eq!(global.rules().await.unwrap().len(), 1); + assert!( + global + .remove_rule("budget-admin", Some("model-a"), BudgetWindow::Day) + .await + .unwrap() + ); + assert!(global.rules().await.unwrap().is_empty()); + assert!(global.usage("2026-01-01", 501).await.is_err()); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn no_global_rules_preserves_single_admin_parity() { + let fixture = fixture().await; + let service = PublicationService::new(fixture.database.clone()); + // Attributed and legacy requests take no instance hold and write no + // instance rows when no global rule exists. + for (proxy, attribution) in [ + ("parity-a", Some(attribution_for("budget-member"))), + ("parity-b", None), + ] { + service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input(proxy, 1), + attribution, + ) + .await + .expect("rule-free publication is unchanged"); + } + assert_eq!(instance_counts(&fixture.database).await, (2, 0, 0)); + fixture.database.close().await.unwrap(); +} + +// --------------------------------------------------------------------------- +// M010 crash recovery: instance-global holds converge on restart without +// folding usage, and a released global hold never blocks the next user. +// --------------------------------------------------------------------------- + +#[tokio::test] +async fn crash_after_publication_releases_instance_hold_without_folding_usage() { + let fixture = fixture().await; + let global = InstanceBudgetService::new(fixture.database.clone()); + // One global concurrency slot shared by every user: a phantom hold + // would wedge the whole instance, not just one member. + set_global( + &global, + "budget-admin", + None, + BudgetLimits { + tokens: Some(1_000_000), + concurrent: Some(1), + enforcement: BudgetEnforcement::Hard, + ..Default::default() + }, + ) + .await; + let service = PublicationService::new(fixture.database.clone()); + let outcome = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + hard_input("global-crash-hold", 1, Some(50), None), + Some(attribution_for("budget-member")), + ) + .await + .expect("bounded global request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + let request_id = published.identity.db_request_id; + assert_eq!(instance_counts(&fixture.database).await, (1, 1, 0)); + drop(published.claim); + + let report = CrashReconciler::new(fixture.database.clone()) + .reconcile_once() + .await + .expect("restart reconciliation succeeds"); + assert_eq!(report.requests_interrupted, 1); + assert_eq!(report.instance_budget_holds_released, 1); + // The attributed member request also carries a rule-free user hold for + // usage accounting; it converges in the same pass. + assert_eq!(report.user_budget_holds_released, 1); + assert!( + report + .classification + .pending_requests_with_active_instance_hold + >= 1 + ); + // Compensation, not credit: released, never settled, no usage folded. + assert_eq!(instance_counts(&fixture.database).await, (1, 0, 0)); + let hold: (String, Option) = fixture + .database + .call(move |connection| { + connection.query_row( + "SELECT status, settled_at FROM instance_budget_reservations WHERE request_id = ?1", + [request_id], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + }) + .await + .unwrap(); + assert_eq!(hold, ("released".to_owned(), None)); + + // The freed global slot admits a different member's request: no + // instance-wide wedge from the phantom hold. + let next = service + .publish_attributed( + claim_for(&fixture, "model-a").await, + hard_input("global-crash-next", 1, Some(50), None), + Some(attribution_for("budget-member-2")), + ) + .await + .expect("next member admits after crash release"); + assert!( + matches!(next, PublicationOutcome::Published(_)), + "next member should publish" + ); + fixture.database.close().await.unwrap(); +} + +#[tokio::test] +async fn settled_global_request_survives_restart_without_double_fold() { + let fixture = fixture().await; + let global = InstanceBudgetService::new(fixture.database.clone()); + set_global( + &global, + "budget-admin", + None, + hard_limits(Some(1_000_000), None), + ) + .await; + let outcome = PublicationService::new(fixture.database.clone()) + .publish_attributed( + claim_for(&fixture, "model-a").await, + hard_input("global-crash-settled", 1, Some(50), None), + Some(attribution_for("budget-member")), + ) + .await + .expect("bounded global request publishes"); + let PublicationOutcome::Published(published) = outcome else { + panic!("request should publish"); + }; + DurableFinalizer::new(fixture.database.clone()) + .finalize_request( + &published.identity, + FinalizationData { + outcome: FinalizationOutcome::Completed, + ..FinalizationData::default() + }, + Some(published.claim), + ) + .await + .expect("terminal request settles"); + assert_eq!(instance_counts(&fixture.database).await, (1, 0, 2)); + assert_eq!(instance_usage_requests(&fixture.database).await, 2); + // A restart after settlement touches neither the settled global hold + // nor the folded buckets. + let report = CrashReconciler::new(fixture.database.clone()) + .reconcile_once() + .await + .expect("restart reconciliation succeeds"); + assert!(report.converged); + assert_eq!(report.instance_budget_holds_released, 0); + assert_eq!(report.user_budget_holds_released, 0); + assert_eq!(instance_counts(&fixture.database).await, (1, 0, 2)); + assert_eq!(instance_usage_requests(&fixture.database).await, 2); + fixture.database.close().await.unwrap(); +} + +// --------------------------------------------------------------------------- +// M010 storage-pressure observations: paired baseline vs budget-rule +// publish/settle latency, WAL footprint, and row counts for 1/10/49 +// distinct credential identities on ordinary local hardware. This is an +// observation record, not a performance gate: per-op assertions are +// generous anti-hang bounds only. Harness output lines prefixed with +// M010_GATE_OBSERVATION carry the JSON cells archived under +// artifacts/qualification/; M012 owns physical-SBC capacity proof. +// --------------------------------------------------------------------------- + +use std::fmt::Write as _; + +async fn insert_measurement_members(database: &Database, count: usize) -> Vec { + let names: Vec = (0..count).map(|index| format!("measure-{index}")).collect(); + let snapshot = names.clone(); + database + .call(move |connection| { + for name in &snapshot { + connection.execute( + "INSERT INTO users(id, username, normalized_username, role, password_hash) + VALUES (?1, ?1, ?1, 'member', 'test-hash')", + [name], + )?; + } + Ok(()) + }) + .await + .expect("measurement members insert"); + names +} + +fn percentile(sorted: &mut [u128], ratio: f64) -> u128 { + sorted.sort_unstable(); + if sorted.is_empty() { + return 0; + } + let rank = ((sorted.len() as f64 - 1.0) * ratio).round() as usize; + sorted[rank.min(sorted.len() - 1)] +} + +fn wal_footprint(dir: &std::path::Path, stem: &str) -> (u64, u64, u64) { + let size = |suffix: &str| { + std::fs::metadata(dir.join(format!("{stem}{suffix}"))) + .map(|metadata| metadata.len()) + .unwrap_or(0) + }; + (size(".sqlite3"), size(".sqlite3-wal"), size(".sqlite3-shm")) +} + +fn process_rss_kb() -> Option { + let output = std::process::Command::new("ps") + .args(["-o", "rss=", "-p", &std::process::id().to_string()]) + .output() + .ok()?; + if !output.status.success() { + return None; + } + String::from_utf8_lossy(&output.stdout).trim().parse().ok() +} + +#[tokio::test] +async fn paired_gate_observations_across_identities_and_rules() { + let work_root = tempfile::tempdir().expect("work root"); + for identities in [1_usize, 10, 49] { + for mode in ["baseline", "user_estimated", "user_hard_global_hard"] { + let stem = format!("m010-gate-{mode}-{identities}"); + let fixture = fixture_on(&DatabaseConfig { + path: work_root + .path() + .join(format!("{stem}.sqlite3")) + .display() + .to_string(), + ..Default::default() + }) + .await; + let members = insert_measurement_members(&fixture.database, identities).await; + if mode != "baseline" { + let users = UserBudgetService::new(fixture.database.clone()); + for member in &members { + users + .set_rule( + "budget-admin", + member, + None, + BudgetWindow::Day, + BudgetLimits { + tokens: Some(1_000_000_000), + microdollars: Some(1_000_000_000), + ..Default::default() + }, + ) + .await + .expect("measurement user rule is set"); + } + } + if mode == "user_hard_global_hard" { + let global = InstanceBudgetService::new(fixture.database.clone()); + global + .set_rule( + "budget-admin", + None, + BudgetWindow::Day, + BudgetLimits { + requests: Some(1_000_000), + tokens: Some(1_000_000_000), + enforcement: BudgetEnforcement::Hard, + ..Default::default() + }, + ) + .await + .expect("measurement global rule is set"); + } + // Hard mode needs worst-case proofs; the other modes ride the + // estimator projection exactly as production does. + let service = PublicationService::new(fixture.database.clone()); + let finalizer = DurableFinalizer::new(fixture.database.clone()); + let mut publish_us: Vec = Vec::with_capacity(identities); + let mut settle_us: Vec = Vec::with_capacity(identities); + for (index, member) in members.iter().enumerate() { + let proxy = format!("{stem}-{index}"); + let started = Instant::now(); + let outcome = if mode == "baseline" { + service + .publish( + claim_for(&fixture, "model-a").await, + estimated_input(&proxy, 1), + ) + .await + .expect("baseline request publishes") + } else if mode == "user_hard_global_hard" { + service + .publish_attributed( + claim_for(&fixture, "model-a").await, + hard_input(&proxy, 1, Some(50), Some(3_000_000)), + Some(attribution_for(member)), + ) + .await + .expect("hard request publishes") + } else { + service + .publish_attributed( + claim_for(&fixture, "model-a").await, + estimated_input(&proxy, 1), + Some(attribution_for(member)), + ) + .await + .expect("estimated request publishes") + }; + publish_us.push(started.elapsed().as_micros()); + let PublicationOutcome::Published(published) = outcome else { + panic!("measurement request should publish"); + }; + let started = Instant::now(); + finalizer + .finalize_request( + &published.identity, + FinalizationData { + outcome: FinalizationOutcome::Completed, + ..FinalizationData::default() + }, + Some(published.claim), + ) + .await + .expect("measurement request settles"); + settle_us.push(started.elapsed().as_micros()); + } + for sample in publish_us.iter().chain(settle_us.iter()) { + assert!( + *sample < 30_000_000, + "per-operation anti-hang bound exceeded: {sample}us" + ); + } + let (db_bytes, wal_bytes, shm_bytes) = wal_footprint(work_root.path(), &stem); + fixture + .database + .call(|connection| { + connection.query_row("PRAGMA wal_checkpoint(TRUNCATE)", [], |_| Ok(()))?; + Ok(()) + }) + .await + .expect("wal checkpoints"); + let (db_after, wal_after, shm_after) = wal_footprint(work_root.path(), &stem); + let rows: (i64, i64, i64, i64) = fixture + .database + .call(|connection| { + Ok(( + connection.query_row("SELECT COUNT(*) FROM requests", [], |row| { + row.get::<_, i64>(0) + })?, + connection.query_row( + "SELECT COUNT(*) FROM user_budget_reservations WHERE status = 'active'", + [], + |row| row.get::<_, i64>(0), + )?, + connection.query_row( + "SELECT COUNT(*) FROM instance_budget_reservations WHERE status = 'active'", + [], + |row| row.get::<_, i64>(0), + )?, + connection.query_row("SELECT COUNT(*) FROM user_budget_usage", [], |row| { + row.get::<_, i64>(0) + })?, + )) + }) + .await + .expect("row counts read"); + let mut cell = String::new(); + write!( + cell, + "{{\"mode\":\"{mode}\",\"identities\":{identities},\ + \"publish_us_min\":{},\"publish_us_p50\":{},\"publish_us_max\":{},\ + \"settle_us_min\":{},\"settle_us_p50\":{},\"settle_us_max\":{},\ + \"db_bytes\":{db_bytes},\"wal_bytes\":{wal_bytes},\"shm_bytes\":{shm_bytes},\ + \"wal_bytes_after_checkpoint\":{wal_after},\"shm_bytes_after_checkpoint\":{shm_after},\ + \"db_bytes_after_checkpoint\":{db_after},\ + \"requests\":{},\"active_user_holds\":{},\"active_instance_holds\":{},\ + \"user_usage_rows\":{},\"arch\":\"{}\",\"os\":\"{}\",\"profile\":\"debug\"}}", + percentile(&mut publish_us.clone(), 0.0), + percentile(&mut publish_us.clone(), 0.5), + percentile(&mut publish_us.clone(), 1.0), + percentile(&mut settle_us.clone(), 0.0), + percentile(&mut settle_us.clone(), 0.5), + percentile(&mut settle_us.clone(), 1.0), + rows.0, + rows.1, + rows.2, + rows.3, + std::env::consts::ARCH, + std::env::consts::OS, + ) + .expect("cell formats"); + println!("M010_GATE_OBSERVATION {cell}"); + fixture.database.close().await.expect("database closes"); + } + } + if let Some(rss) = process_rss_kb() { + println!("M010_GATE_OBSERVATION {{\"rss_kb_after_suite\":{rss}}}"); + } +} diff --git a/rust/tests/on_prem_identity.rs b/rust/tests/on_prem_identity.rs new file mode 100644 index 000000000..24eff5c89 --- /dev/null +++ b/rust/tests/on_prem_identity.rs @@ -0,0 +1,146 @@ +use eggpool::{ + db::{Database, DatabaseConfig, MigrationRunner}, + identity::{self, IdentityError}, +}; + +async fn database() -> Database { + let database = Database::open(DatabaseConfig::default()) + .await + .expect("in-memory DB opens"); + MigrationRunner::new(&database) + .run() + .await + .expect("canonical schema migrates"); + database +} + +#[tokio::test] +async fn bootstrap_is_atomic_and_personal_token_is_one_way() { + let database = database().await; + assert_eq!( + identity::access_state(&database).await.unwrap(), + "legacy_unmigrated" + ); + identity::mark_fresh_uninitialized(&database).await.unwrap(); + assert_eq!( + identity::access_state(&database).await.unwrap(), + "fresh_uninitialized" + ); + let (admin, credential) = + identity::bootstrap_admin(&database, "admin", "correct horse battery staple") + .await + .expect("first admin is created"); + + assert_eq!(admin.role, "admin"); + assert_eq!( + identity::access_state(&database).await.unwrap(), + "admin_initialized" + ); + assert!( + identity::verify_api_token(&database, &credential.token) + .await + .expect("valid token check") + ); + assert!( + !identity::verify_api_token(&database, "not-a-token") + .await + .expect("invalid token check") + ); + assert_eq!( + identity::verify_password(&database, "ADMIN", "correct horse battery staple") + .await + .expect("password check") + .expect("admin exists") + .id, + admin.id + ); + + let stored_hash = database + .call({ + let admin_id = admin.id.clone(); + move |connection| { + connection.query_row( + "SELECT password_hash FROM users WHERE id = ?1", + [&admin_id], + |row| row.get::<_, String>(0), + ) + } + }) + .await + .expect("hash is readable for the invariant check"); + assert!(!stored_hash.contains("correct horse battery staple")); + assert!(!stored_hash.is_empty()); + match identity::bootstrap_admin(&database, "other", "another valid password").await { + Err(IdentityError::AdminAlreadyInitialized) => {} + Err(error) => panic!("unexpected bootstrap error: {error}"), + Ok(_) => panic!("second bootstrap unexpectedly succeeded"), + } + assert_eq!(identity::list_users(&database).await.unwrap().len(), 1); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn simultaneous_bootstrap_commits_only_one_admin() { + let database = database().await; + let (first, second) = tokio::join!( + identity::bootstrap_admin(&database, "admin", "first safe password 123"), + identity::bootstrap_admin(&database, "admin", "second safe password 456"), + ); + assert_ne!(first.is_ok(), second.is_ok()); + assert_eq!(identity::list_users(&database).await.unwrap().len(), 1); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn password_reset_revokes_api_credentials_and_rejects_bad_passwords() { + let database = database().await; + let (_, credential) = + identity::bootstrap_admin(&database, "admin", "initial safe password 123") + .await + .unwrap(); + assert!( + identity::verify_api_token(&database, &credential.token) + .await + .unwrap() + ); + assert!(matches!( + identity::reset_password(&database, "admin", "short").await, + Err(IdentityError::InvalidPassword) + )); + + identity::reset_password(&database, "admin", "replacement password 456") + .await + .expect("password is changed"); + assert!( + !identity::verify_api_token(&database, &credential.token) + .await + .unwrap() + ); + assert!( + identity::verify_password(&database, "admin", "replacement password 456") + .await + .unwrap() + .is_some() + ); + assert!( + identity::verify_password(&database, "admin", "initial safe password 123") + .await + .unwrap() + .is_none() + ); + let replacement_credential = identity::issue_api_token(&database, "admin", "test client") + .await + .expect("personal credential can be issued"); + assert_eq!( + identity::revoke_api_tokens(&database, "admin") + .await + .expect("personal credentials can be revoked"), + 1 + ); + assert!( + !identity::verify_api_token(&database, &replacement_credential.token) + .await + .unwrap() + ); + database.close().await.unwrap(); +} diff --git a/rust/tests/on_prem_model_policy.rs b/rust/tests/on_prem_model_policy.rs new file mode 100644 index 000000000..1c79cd030 --- /dev/null +++ b/rust/tests/on_prem_model_policy.rs @@ -0,0 +1,225 @@ +use eggpool::{ + db::{Database, DatabaseConfig, MigrationRunner}, + identity, + model_policy::{ModelPolicyError, ModelPolicyService}, + web_session, +}; + +async fn database() -> Database { + let database = Database::open(DatabaseConfig::default()) + .await + .expect("in-memory database opens"); + MigrationRunner::new(&database) + .run() + .await + .expect("schema migrates"); + database +} + +async fn add_user(database: &Database, id: &'static str, role: &'static str) { + database + .call(move |connection| { + connection.execute( + "INSERT INTO users(id, username, normalized_username, role, password_hash)\ + VALUES (?1, ?1, ?1, ?2, 'test-hash')", + (id, role), + )?; + Ok(()) + }) + .await + .expect("user inserts"); +} + +#[tokio::test] +async fn profiles_are_default_deny_and_precedence_is_hard_deny_user_profile_default() { + let database = database().await; + add_user(&database, "admin-a", "admin").await; + add_user(&database, "member-a", "member").await; + let policy = ModelPolicyService::new(database.clone()); + + let admin = policy.snapshot("admin-a").await.unwrap(); + let unassigned_member = policy.snapshot("member-a").await.unwrap(); + assert!(admin.allows("unlisted-model", None)); + assert!(!unassigned_member.allows("unlisted-model", None)); + + policy + .create_profile("restricted", "Restricted", "deny", "admin-a") + .await + .unwrap(); + policy + .set_profile_rule("restricted", "model-a", "allow", "admin-a") + .await + .unwrap(); + policy + .set_profile_rule("restricted", "model-a/provider-a", "deny", "admin-a") + .await + .unwrap(); + policy + .set_profile_preference("restricted", Some("model-a"), "admin-a") + .await + .unwrap(); + let denied_preference = policy + .set_profile_preference("restricted", Some("model-b"), "admin-a") + .await; + assert!( + matches!( + denied_preference, + Err(ModelPolicyError::PreferredModelDenied) + ), + "unexpected preferred-model result: {denied_preference:?}" + ); + policy + .assign_profile("member-a", "restricted", "admin-a") + .await + .unwrap(); + + let profile = policy.snapshot("member-a").await.unwrap(); + assert_eq!(profile.preferred_model_id(), Some("model-a")); + assert!(profile.allows("model-a", None)); + assert!(!profile.allows("model-b", None)); + assert!(!profile.allows("model-a", Some("provider-a"))); + + policy + .set_user_override("member-a", "model-a", "allow", "admin-a") + .await + .unwrap(); + let user_override = policy.snapshot("member-a").await.unwrap(); + assert!(user_override.revision() > profile.revision()); + assert!(user_override.allows("model-a", Some("provider-a"))); + assert!( + profile.allows("model-a", None), + "old snapshots stay immutable" + ); + + policy + .set_instance_denial("model-a", "admin-a", true) + .await + .unwrap(); + let denied = policy.snapshot("member-a").await.unwrap(); + assert!(!denied.allows("model-a", None)); + assert!(!denied.allows("model-a", Some("provider-a"))); + assert!(admin.allows("model-a", None), "admitted policy is frozen"); + + database + .call(|connection| { + connection.execute( + "UPDATE users SET state = 'disabled' WHERE id = 'member-a'", + [], + )?; + Ok(()) + }) + .await + .unwrap(); + assert!(matches!( + policy.snapshot("member-a").await, + Err(ModelPolicyError::UserUnavailable) + )); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn wildcard_and_noncanonical_policy_keys_are_rejected() { + let database = database().await; + add_user(&database, "admin-a", "admin").await; + let policy = ModelPolicyService::new(database.clone()); + assert!( + policy + .create_profile("restricted", &"x".repeat(65), "deny", "admin-a") + .await + .is_err() + ); + assert!( + policy + .set_profile_rule("restricted", "model-*", "allow", "admin-a") + .await + .is_err() + ); + assert!( + policy + .set_profile_rule("restricted", " model-a", "allow", "admin-a") + .await + .is_err() + ); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn concurrent_policy_edit_keeps_each_snapshot_consistent_and_refreshes_new_work() { + let database = database().await; + add_user(&database, "admin-a", "admin").await; + add_user(&database, "member-a", "member").await; + let policy = ModelPolicyService::new(database.clone()); + let before = policy.snapshot("member-a").await.unwrap(); + assert!(!before.allows("model-a", None)); + + let (edit, observed) = tokio::join!( + policy.set_user_override("member-a", "model-a", "allow", "admin-a"), + policy.snapshot("member-a"), + ); + edit.unwrap(); + let observed = observed.unwrap(); + let after = policy.snapshot("member-a").await.unwrap(); + if observed.revision() == before.revision() { + assert!(!observed.allows("model-a", None)); + } else { + assert!(observed.allows("model-a", None)); + } + assert!(after.revision() > before.revision()); + assert!(after.allows("model-a", None)); + assert!( + !before.allows("model-a", None), + "in-flight policy is immutable" + ); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn revoked_admin_session_cannot_commit_a_policy_mutation() { + let database = database().await; + identity::mark_fresh_uninitialized(&database).await.unwrap(); + let (admin, _) = identity::bootstrap_admin( + &database, + "policy-admin", + "M015 admin password before reset", + ) + .await + .unwrap(); + let issued = web_session::issue(&database, &admin.id).await.unwrap(); + let stale_principal = web_session::authenticate(&database, &issued.session_token) + .await + .unwrap() + .expect("browser session is initially active"); + + identity::reset_password( + &database, + &admin.username, + "M015 admin password after reset", + ) + .await + .unwrap(); + let result = ModelPolicyService::new(database.clone()) + .with_browser_admin_session(&stale_principal) + .create_profile( + "stale-session-profile", + "Stale session profile", + "deny", + &admin.id, + ) + .await; + assert!(matches!(result, Err(ModelPolicyError::Database(_)))); + let profile_exists = database + .call(|connection| { + connection.query_row( + "SELECT EXISTS(SELECT 1 FROM policy_profiles WHERE id = 'stale-session-profile')", + [], + |row| row.get::<_, bool>(0), + ) + }) + .await + .unwrap(); + assert!( + !profile_exists, + "revoked session must not commit policy state" + ); + database.close().await.unwrap(); +} diff --git a/rust/tests/on_prem_sessions.rs b/rust/tests/on_prem_sessions.rs new file mode 100644 index 000000000..8844edc1c --- /dev/null +++ b/rust/tests/on_prem_sessions.rs @@ -0,0 +1,784 @@ +use std::{sync::Arc, time::Duration}; + +use eggpool::{ + Config, + db::{Database, DatabaseConfig, MigrationRunner}, + identity, + runtime_lifecycle::{ProcessRuntime, RuntimeGenerationFactory, RuntimeManager}, + server::{ServerRuntime, ShutdownReason}, + web_session, +}; +use tokio::{ + io::{AsyncReadExt, AsyncWriteExt}, + net::{TcpListener, TcpStream}, +}; + +async fn database() -> (tempfile::TempDir, Database) { + let directory = tempfile::tempdir().expect("temporary identity directory"); + let database = Database::open(DatabaseConfig { + path: directory + .path() + .join("eggpool.sqlite3") + .to_string_lossy() + .into_owned(), + ..DatabaseConfig::default() + }) + .await + .expect("database opens"); + MigrationRunner::new(&database) + .run() + .await + .expect("schema migrates"); + (directory, database) +} + +async fn create_admin(database: &Database) -> (String, String) { + identity::mark_fresh_uninitialized(database).await.unwrap(); + let (user, credential) = + identity::bootstrap_admin(database, "admin", "M002 strong test passphrase") + .await + .expect("administrator bootstraps"); + (user.id, credential.token) +} + +#[tokio::test] +async fn session_and_csrf_values_are_digest_only_and_revocable() { + let (_directory, database) = database().await; + let (user_id, _) = create_admin(&database).await; + let issued = web_session::issue(&database, &user_id).await.unwrap(); + let principal = web_session::authenticate(&database, &issued.session_token) + .await + .unwrap() + .expect("session authenticates"); + assert_eq!(principal.user_id, user_id); + assert!(web_session::verify_csrf(&principal, &issued.csrf_token)); + assert!(!web_session::verify_csrf(&principal, "forged-token")); + + let stored = database + .call(|connection| { + connection.query_row("SELECT digest, csrf_digest FROM web_sessions", [], |row| { + Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)) + }) + }) + .await + .unwrap(); + assert_ne!(stored.0, issued.session_token); + assert_ne!(stored.1, issued.csrf_token); + + web_session::revoke(&database, &issued.session_token) + .await + .unwrap(); + assert!( + web_session::authenticate(&database, &issued.session_token) + .await + .unwrap() + .is_none() + ); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn password_reset_revokes_browser_sessions() { + let (_directory, database) = database().await; + let (user_id, _) = create_admin(&database).await; + let issued = web_session::issue(&database, &user_id).await.unwrap(); + identity::reset_password(&database, "admin", "M002 replacement password 456") + .await + .expect("password reset succeeds"); + assert!( + web_session::authenticate(&database, &issued.session_token) + .await + .unwrap() + .is_none() + ); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn revoked_admin_session_cannot_commit_privileged_identity_or_budget_writes() { + let (_directory, database) = database().await; + let (admin_id, _) = create_admin(&database).await; + identity::enable_managed(&database).await.unwrap(); + let issued = web_session::issue(&database, &admin_id).await.unwrap(); + let principal = web_session::authenticate(&database, &issued.session_token) + .await + .unwrap() + .expect("admin session is initially active"); + let member = identity::create_member( + &database, + &admin_id, + "fenced-member", + "M015 fenced member temporary password", + ) + .await + .unwrap(); + let credential = + identity::issue_personal_api_token(&database, &admin_id, &member.id, "fenced member token") + .await + .unwrap(); + web_session::revoke(&database, &issued.session_token) + .await + .unwrap(); + + assert!( + identity::create_member_from_browser( + &database, + &principal, + "must-not-exist", + "M015 stale session temporary password", + ) + .await + .is_err() + ); + assert!( + identity::create_member_with_profile_from_browser( + &database, + &principal, + "must-not-exist-with-profile", + "M015 stale session temporary password", + "default", + ) + .await + .is_err() + ); + assert!( + identity::set_user_enabled_from_browser(&database, &principal, &member.id, false,) + .await + .is_err() + ); + assert!( + identity::admin_reset_password_from_browser( + &database, + &principal, + &member.id, + "M015 stale session replacement password", + ) + .await + .is_err() + ); + assert!( + identity::issue_personal_api_token_from_browser( + &database, + &principal, + &member.id, + "must-not-be-issued", + ) + .await + .is_err() + ); + assert!( + identity::revoke_user_api_tokens_from_browser(&database, &principal, &member.id,) + .await + .is_err() + ); + assert!( + identity::revoke_user_credential_from_browser( + &database, + &principal, + &member.id, + "missing-credential", + ) + .await + .is_err() + ); + assert!( + identity::reset_password_from_browser( + &database, + &principal, + "M015 stale session password replacement", + ) + .await + .is_err() + ); + + let limits = eggpool::user_budget::BudgetLimits { + requests: Some(1), + tokens: None, + microdollars: None, + concurrent: None, + enforcement: eggpool::user_budget::BudgetEnforcement::Estimated, + }; + let user_budgets = eggpool::user_budget::UserBudgetService::new(database.clone()) + .with_browser_admin_session(&principal); + assert!( + user_budgets + .set_rule( + &admin_id, + &member.id, + None, + eggpool::user_budget::BudgetWindow::Day, + limits, + ) + .await + .is_err() + ); + let instance_budgets = eggpool::instance_budget::InstanceBudgetService::new(database.clone()) + .with_browser_admin_session(&principal); + assert!( + instance_budgets + .set_rule( + &admin_id, + None, + eggpool::user_budget::BudgetWindow::Day, + limits, + ) + .await + .is_err() + ); + + let users = identity::list_users(&database).await.unwrap(); + assert!(!users.iter().any(|user| { + matches!( + user.username.as_str(), + "must-not-exist" | "must-not-exist-with-profile" + ) + })); + assert_eq!( + users + .iter() + .find(|user| user.id == member.id) + .expect("member remains present") + .state, + "active" + ); + assert!( + identity::verify_api_token_principal(&database, &credential.token) + .await + .unwrap() + .is_some(), + "stale admin requests do not revoke member credentials" + ); + assert!(user_budgets.rules(&member.id).await.unwrap().is_empty()); + assert!(instance_budgets.rules().await.unwrap().is_empty()); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn expired_sessions_and_disabled_users_are_rejected() { + let (_directory, database) = database().await; + let (user_id, _) = create_admin(&database).await; + let expired = web_session::issue(&database, &user_id).await.unwrap(); + database + .call(|connection| { + connection.execute( + "UPDATE web_sessions SET expires_at = datetime('now', '-1 second')", + [], + )?; + Ok(()) + }) + .await + .unwrap(); + assert!( + web_session::authenticate(&database, &expired.session_token) + .await + .unwrap() + .is_none() + ); + + let active = web_session::issue(&database, &user_id).await.unwrap(); + database + .call(move |connection| { + connection.execute( + "UPDATE users SET state = 'disabled' WHERE id = ?1", + [&user_id], + )?; + Ok(()) + }) + .await + .unwrap(); + assert!( + web_session::authenticate(&database, &active.session_token) + .await + .unwrap() + .is_none() + ); + database.close().await.unwrap(); +} + +#[test] +fn login_throttle_is_bounded_by_username_and_peer() { + let username = format!("login-test-{}", std::process::id()); + let peer = Some("192.0.2.19".parse().unwrap()); + let other_peer = Some("198.51.100.19".parse().unwrap()); + assert!(web_session::login_allowed(&username, peer)); + for _ in 0..10 { + web_session::record_login_failure(&username, peer); + } + assert!(!web_session::login_allowed(&username, peer)); + assert!(!web_session::login_allowed(&format!(" {username} "), peer)); + assert!(web_session::login_allowed(&username, other_peer)); +} + +#[test] +fn personal_token_failures_are_bounded_per_peer_before_database_lookup() { + let peer = Some("203.0.113.71".parse().unwrap()); + let other_peer = Some("203.0.113.72".parse().unwrap()); + for _ in 0..30 { + assert!(web_session::personal_token_attempt_allowed(peer)); + web_session::record_personal_token_failure(peer); + } + assert!(!web_session::personal_token_attempt_allowed(peer)); + assert!(web_session::personal_token_attempt_allowed(other_peer)); +} + +async fn http_request(address: std::net::SocketAddr, raw: &str) -> String { + let target = raw.split_whitespace().nth(1).unwrap_or("unknown"); + let mut stream = TcpStream::connect(address).await.expect("connect"); + stream + .write_all(raw.as_bytes()) + .await + .expect("write request"); + let mut response = Vec::new(); + tokio::time::timeout(Duration::from_secs(10), stream.read_to_end(&mut response)) + .await + .unwrap_or_else(|_| panic!("request to {target} did not finish")) + .expect("read response"); + String::from_utf8_lossy(&response).into_owned() +} + +fn cookie(response: &str, name: &str) -> String { + response + .lines() + .find_map(|line| { + let (header, value) = line.split_once(':')?; + if !header.eq_ignore_ascii_case("set-cookie") { + return None; + } + value + .trim() + .split(';') + .next()? + .strip_prefix(&format!("{name}=")) + .map(str::to_owned) + }) + .expect("cookie is set") +} + +#[tokio::test] +async fn managed_browser_login_account_csrf_and_logout_work_over_loopback() { + let (_directory, database) = database().await; + let (_user_id, agent_token) = create_admin(&database).await; + identity::enable_managed(&database).await.unwrap(); + + let process = ProcessRuntime::new(database.clone()).expect("process runtime"); + let mut config = Config::default(); + config.server.host = "127.0.0.1".to_owned(); + config.server.api_key = None; + config.dashboard.enabled = true; + config.dashboard.public = true; + let candidate = + RuntimeGenerationFactory::prepare(&process, config.clone(), "session-test".to_owned(), 1) + .await + .expect("initial generation"); + let manager = Arc::new(RuntimeManager::new( + candidate.transfer().expect("generation transfer"), + )); + process + .install_initial_tasks((*manager).clone(), &config) + .await + .expect("tasks install"); + let runtime = ServerRuntime::new(process, manager, config); + let handle = runtime.handle(); + let listener = TcpListener::bind(("127.0.0.1", 0)).await.expect("listener"); + let address = listener.local_addr().unwrap(); + let task = tokio::spawn(async move { runtime.serve_listener(listener).await }); + let origin = format!("http://{address}"); + + let public_dashboard = http_request( + address, + &format!("GET / HTTP/1.1\r\nHost: {address}\r\nConnection: close\r\n\r\n"), + ) + .await; + assert!( + public_dashboard.starts_with("HTTP/1.1 401"), + "{public_dashboard}" + ); + let agent_to_account = http_request(address, &format!( + "GET /account HTTP/1.1\r\nHost: {address}\r\nAuthorization: Bearer {agent_token}\r\nConnection: close\r\n\r\n" + )).await; + assert!( + agent_to_account.starts_with("HTTP/1.1 401"), + "{agent_to_account}" + ); + let agent_to_admin_api = http_request(address, &format!( + "GET /api/status HTTP/1.1\r\nHost: {address}\r\nAuthorization: Bearer {agent_token}\r\nConnection: close\r\n\r\n" + )).await; + assert!( + agent_to_admin_api.starts_with("HTTP/1.1 401"), + "{agent_to_admin_api}" + ); + + let login_page = http_request( + address, + &format!("GET /login HTTP/1.1\r\nHost: {address}\r\nConnection: close\r\n\r\n"), + ) + .await; + assert!(login_page.starts_with("HTTP/1.1 200"), "{login_page}"); + assert!(login_page.contains("name=\"password\""), "{login_page}"); + + let form = "username=admin&password=M002+strong+test+passphrase"; + let login = http_request(address, &format!( + "POST /login HTTP/1.1\r\nHost: {address}\r\nOrigin: {origin}\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{form}", form.len() + )).await; + assert!(login.starts_with("HTTP/1.1 303"), "{login}"); + assert!(login.to_ascii_lowercase().contains("httponly"), "{login}"); + assert!( + login.to_ascii_lowercase().contains("samesite=lax"), + "{login}" + ); + let session = cookie(&login, "eggpool_session"); + let csrf = cookie(&login, "eggpool_csrf"); + let cookie_header = format!("eggpool_session={session}; eggpool_csrf={csrf}"); + + let account = http_request(address, &format!("GET /account HTTP/1.1\r\nHost: {address}\r\nCookie: {cookie_header}\r\nConnection: close\r\n\r\n")).await; + assert!(account.starts_with("HTTP/1.1 200"), "{account}"); + assert!( + account + .to_ascii_lowercase() + .contains("cache-control: no-store"), + "authenticated account pages are not cacheable: {account}" + ); + assert!(account.contains("Change password"), "{account}"); + + // A forced first-login password change is available without JavaScript + // and invalidates the browser session when completed. + database + .call(|connection| { + connection.execute( + "UPDATE users SET must_change_password = 1 WHERE username = 'admin'", + [], + )?; + Ok(()) + }) + .await + .unwrap(); + let forced = http_request(address, &format!("GET / HTTP/1.1\r\nHost: {address}\r\nCookie: {cookie_header}\r\nConnection: close\r\n\r\n")).await; + assert!(forced.starts_with("HTTP/1.1 303"), "{forced}"); + assert!(forced.contains("/account?password=required"), "{forced}"); + let forced_account = http_request(address, &format!("GET /account HTTP/1.1\r\nHost: {address}\r\nCookie: {cookie_header}\r\nConnection: close\r\n\r\n")).await; + assert!( + forced_account.contains("Change your password to continue."), + "{forced_account}" + ); + + let new_password = "M002 new test password 789"; + let change_body = format!( + "current_password=M002+strong+test+passphrase&new_password=M002+new+test+password+789&csrf={csrf}" + ); + let changed = http_request(address, &format!( + "POST /account/password HTTP/1.1\r\nHost: {address}\r\nOrigin: {origin}\r\nCookie: {cookie_header}\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{change_body}", change_body.len() + )).await; + assert!(changed.starts_with("HTTP/1.1 303"), "{changed}"); + assert!( + web_session::authenticate(&database, &session) + .await + .unwrap() + .is_none(), + "password change revokes current session" + ); + let relogin_form = format!("username=admin&password={}", new_password.replace(' ', "+")); + let relogin = http_request(address, &format!( + "POST /login HTTP/1.1\r\nHost: {address}\r\nOrigin: {origin}\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{relogin_form}", relogin_form.len() + )).await; + assert!(relogin.starts_with("HTTP/1.1 303"), "{relogin}"); + + let session = cookie(&relogin, "eggpool_session"); + let csrf = cookie(&relogin, "eggpool_csrf"); + let cookie_header = format!("eggpool_session={session}; eggpool_csrf={csrf}"); + + let missing_body = "csrf="; + let missing_csrf = http_request(address, &format!( + "POST /logout HTTP/1.1\r\nHost: {address}\r\nOrigin: {origin}\r\nCookie: {cookie_header}\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{missing_body}", missing_body.len() + )).await; + assert!(missing_csrf.starts_with("HTTP/1.1 403"), "{missing_csrf}"); + + let logout_body = format!("csrf={csrf}"); + let logout = http_request(address, &format!( + "POST /logout HTTP/1.1\r\nHost: {address}\r\nOrigin: {origin}\r\nCookie: {cookie_header}\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{logout_body}", logout_body.len() + )).await; + assert!(logout.starts_with("HTTP/1.1 303"), "{logout}"); + + let stale = http_request(address, &format!("GET /account HTTP/1.1\r\nHost: {address}\r\nCookie: {cookie_header}\r\nConnection: close\r\n\r\n")).await; + assert!(stale.starts_with("HTTP/1.1 401"), "{stale}"); + handle.request_shutdown(ShutdownReason::Requested); + task.await.unwrap().unwrap(); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn trusted_reverse_proxy_can_complete_secure_browser_login() { + let (_directory, database) = database().await; + let (_user_id, _) = create_admin(&database).await; + identity::enable_managed(&database).await.unwrap(); + + let process = ProcessRuntime::new(database.clone()).expect("process runtime"); + let mut config = Config::default(); + config.server.host = "127.0.0.1".to_owned(); + config.server.api_key = None; + config.dashboard.enabled = true; + config.dashboard.public = true; + config.security.trusted_proxies = vec!["127.0.0.1".to_owned()]; + let candidate = RuntimeGenerationFactory::prepare( + &process, + config.clone(), + "proxy-session-test".to_owned(), + 1, + ) + .await + .expect("initial generation"); + let manager = Arc::new(RuntimeManager::new( + candidate.transfer().expect("generation transfer"), + )); + let runtime = ServerRuntime::new(process, manager, config); + let handle = runtime.handle(); + let listener = TcpListener::bind(("127.0.0.1", 0)).await.expect("listener"); + let address = listener.local_addr().unwrap(); + let task = tokio::spawn(async move { runtime.serve_listener(listener).await }); + let origin = format!("https://{address}"); + let form = "username=admin&password=M002+strong+test+passphrase"; + let login = http_request(address, &format!( + "POST /login HTTP/1.1\r\nHost: {address}\r\nOrigin: {origin}\r\nX-Forwarded-Proto: https\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{form}", form.len() + )).await; + + assert!(login.starts_with("HTTP/1.1 303"), "{login}"); + assert!( + login.to_ascii_lowercase().contains("eggpool_session=") + && login.to_ascii_lowercase().contains("; secure"), + "trusted proxy login must issue a Secure session cookie: {login}" + ); + handle.request_shutdown(ShutdownReason::Requested); + task.await.unwrap().unwrap(); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn browser_sessions_can_read_scoped_user_budget_api() { + let (_directory, database) = database().await; + let (admin_id, _) = create_admin(&database).await; + identity::enable_managed(&database).await.unwrap(); + identity::reset_password(&database, "admin", "M015 budget api admin password") + .await + .unwrap(); + let member = identity::create_member( + &database, + &admin_id, + "budget-api-member", + "M015 budget api member temporary password", + ) + .await + .unwrap(); + let other_member = identity::create_member( + &database, + &admin_id, + "budget-api-other", + "M015 budget api other temporary password", + ) + .await + .unwrap(); + identity::reset_password( + &database, + "budget-api-member", + "M015 budget api member password", + ) + .await + .unwrap(); + identity::reset_password( + &database, + "budget-api-other", + "M015 budget api other password", + ) + .await + .unwrap(); + + let limits = eggpool::user_budget::BudgetLimits { + requests: Some(5), + tokens: None, + microdollars: None, + concurrent: None, + enforcement: eggpool::user_budget::BudgetEnforcement::Estimated, + }; + let budgets = eggpool::user_budget::UserBudgetService::new(database.clone()); + budgets + .set_rule( + &admin_id, + &member.id, + None, + eggpool::user_budget::BudgetWindow::Day, + limits, + ) + .await + .unwrap(); + let other_limits = eggpool::user_budget::BudgetLimits { + requests: Some(10), + ..limits + }; + budgets + .set_rule( + &admin_id, + &other_member.id, + None, + eggpool::user_budget::BudgetWindow::Day, + other_limits, + ) + .await + .unwrap(); + eggpool::instance_budget::InstanceBudgetService::new(database.clone()) + .set_rule( + &admin_id, + None, + eggpool::user_budget::BudgetWindow::Day, + limits, + ) + .await + .unwrap(); + + let member_session = web_session::issue(&database, &member.id).await.unwrap(); + let admin_session = web_session::issue(&database, &admin_id).await.unwrap(); + let process = ProcessRuntime::new(database.clone()).expect("process runtime"); + let mut config = Config::default(); + config.server.host = "127.0.0.1".to_owned(); + config.server.api_key = Some("configured-shared-key-for-budget-api-test".to_owned()); + config.dashboard.enabled = true; + config.dashboard.public = true; + let candidate = RuntimeGenerationFactory::prepare( + &process, + config.clone(), + "browser-budget-api-test".to_owned(), + 1, + ) + .await + .expect("initial generation"); + let manager = Arc::new(RuntimeManager::new( + candidate.transfer().expect("generation transfer"), + )); + let runtime = ServerRuntime::new(process, manager, config); + let handle = runtime.handle(); + let listener = TcpListener::bind(("127.0.0.1", 0)).await.unwrap(); + let address = listener.local_addr().unwrap(); + let task = tokio::spawn(async move { runtime.serve_listener(listener).await }); + + let member_response = http_request( + address, + &format!( + "GET /api/user-budget HTTP/1.1\r\nHost: {address}\r\nCookie: eggpool_session={}\r\nConnection: close\r\n\r\n", + member_session.session_token + ), + ) + .await; + assert!( + member_response.starts_with("HTTP/1.1 200"), + "{member_response}" + ); + assert!( + member_response + .to_ascii_lowercase() + .contains("cache-control: no-store"), + "browser budget response must not be cached: {member_response}" + ); + assert!( + member_response.contains("\"max_requests\":5"), + "{member_response}" + ); + assert!( + !member_response.contains("instance_rules") + && !member_response.contains("\"max_requests\":10"), + "members see neither instance ceilings nor another user's rules: {member_response}" + ); + + let denied = http_request( + address, + &format!( + "GET /api/user-budget?user_id={} HTTP/1.1\r\nHost: {address}\r\nCookie: eggpool_session={}\r\nConnection: close\r\n\r\n", + other_member.id, member_session.session_token + ), + ) + .await; + assert!(denied.starts_with("HTTP/1.1 403"), "{denied}"); + + let admin_response = http_request( + address, + &format!( + "GET /api/user-budget?user_id={} HTTP/1.1\r\nHost: {address}\r\nCookie: eggpool_session={}\r\nConnection: close\r\n\r\n", + member.id, admin_session.session_token + ), + ) + .await; + assert!( + admin_response.starts_with("HTTP/1.1 200"), + "{admin_response}" + ); + assert!( + admin_response.contains("instance_rules"), + "{admin_response}" + ); + assert!( + admin_response.contains("\"max_requests\":5"), + "{admin_response}" + ); + + handle.request_shutdown(ShutdownReason::Requested); + task.await.unwrap().unwrap(); + database.close().await.unwrap(); +} + +#[tokio::test] +async fn token_guess_throttle_runs_before_access_state_and_database_lookup() { + let (_directory, database) = database().await; + let (_admin_id, _) = create_admin(&database).await; + identity::enable_managed(&database).await.unwrap(); + + let process = ProcessRuntime::new(database.clone()).expect("process runtime"); + let mut config = Config::default(); + config.server.host = "127.0.0.1".to_owned(); + config.server.api_key = Some("legacy-shared-key-for-test-only".to_owned()); + config.dashboard.enabled = true; + config.dashboard.public = true; + let candidate = + RuntimeGenerationFactory::prepare(&process, config.clone(), "token-test".to_owned(), 1) + .await + .expect("initial generation"); + let manager = Arc::new(RuntimeManager::new( + candidate.transfer().expect("generation transfer"), + )); + let runtime = ServerRuntime::new(process, manager, config); + let handle = runtime.handle(); + let listener = TcpListener::bind(("127.0.0.1", 0)).await.expect("listener"); + let address = listener.local_addr().unwrap(); + let task = tokio::spawn(async move { runtime.serve_listener(listener).await }); + + let before_missing = database.stats().calls; + let missing = http_request( + address, + &format!("GET /v1/models HTTP/1.1\r\nHost: {address}\r\nConnection: close\r\n\r\n"), + ) + .await; + assert!(missing.starts_with("HTTP/1.1 401"), "{missing}"); + assert_eq!( + database.stats().calls, + before_missing, + "missing credentials are rejected before the access-state read when the shared key is configured" + ); + + for index in 0..30_u64 { + let token = format!("epk1_{index:016x}_{:064x}", index + 1); + let response = http_request(address, &format!( + "GET /v1/models HTTP/1.1\r\nHost: {address}\r\nAuthorization: Bearer {token}\r\nConnection: close\r\n\r\n" + )).await; + assert!(response.starts_with("HTTP/1.1 401"), "{response}"); + } + let before_blocked = database.stats().calls; + let blocked_token = format!("epk1_{:016x}_{:064x}", 31_u64, 32_u64); + let blocked = http_request(address, &format!( + "GET /v1/models HTTP/1.1\r\nHost: {address}\r\nAuthorization: Bearer {blocked_token}\r\nConnection: close\r\n\r\n" + )).await; + assert!(blocked.starts_with("HTTP/1.1 401"), "{blocked}"); + assert_eq!( + database.stats().calls, + before_blocked, + "the exhausted peer bucket rejects before any SQLite operation" + ); + + handle.request_shutdown(ShutdownReason::Requested); + task.await.unwrap().unwrap(); + database.close().await.unwrap(); +} diff --git a/rust/tests/provider_transport.rs b/rust/tests/provider_transport.rs index e00c289ee..191231f90 100644 --- a/rust/tests/provider_transport.rs +++ b/rust/tests/provider_transport.rs @@ -1617,6 +1617,33 @@ async fn direct_https_uses_hostname_verified_explicit_test_root() { ); } +#[tokio::test(flavor = "current_thread")] +async fn direct_https_rejects_a_server_signed_by_an_untrusted_root() { + let server = FixtureServer::https(1); + let unrelated_server = FixtureServer::https(1); + let mut config = ProviderHttpConfig::new(&server.https_url()).expect("provider config"); + config.additional_root_certificates.push( + unrelated_server + .tls_certificate + .clone() + .expect("unrelated TLS root"), + ); + let client = ProviderHttpClient::new(config).expect("provider client"); + + assert_eq!( + client + .send( + Method::GET, + "/untrusted-root", + HeaderMap::new(), + Bytes::new(), + ) + .await + .expect_err("untrusted server certificate"), + TransportError::Tls + ); +} + #[test] fn transport_error_display_is_secret_free_and_uri_validation_is_fail_closed() { let marker = "provider-api-key-unique-marker"; @@ -1820,6 +1847,7 @@ async fn failed_pool_build_after_bind_closes_database_and_releases_listener() { // the canonical LAN bind default (which requires a server key). host: "127.0.0.1".to_owned(), port, + api_key: Some("fixture-server-key".to_owned()), ..Default::default() }, database: eggpool::config::DatabaseConfig { diff --git a/rust/tests/runtime_lifecycle_r004.rs b/rust/tests/runtime_lifecycle_r004.rs index c9e27190a..a4c80d0bc 100644 --- a/rust/tests/runtime_lifecycle_r004.rs +++ b/rust/tests/runtime_lifecycle_r004.rs @@ -195,6 +195,12 @@ async fn failed_finalization_keeps_old_transport_open_and_isolated() { client_protocol: "openai".into(), upstream_protocol: "openai".into(), attempt_number: 1, + user_id: None, + credential_id: None, + auth_epoch: None, + principal_kind: eggpool::identity::PrincipalKind::LegacySharedKey, + request_kind: eggpool::identity::AttributedRequestKind::Logical, + parent_proxy_request_id: None, }, data: FinalizationData { outcome: FinalizationOutcome::Completed, diff --git a/rust/tests/runtime_lifecycle_r005.rs b/rust/tests/runtime_lifecycle_r005.rs index e1f43062e..d6690bdfa 100644 --- a/rust/tests/runtime_lifecycle_r005.rs +++ b/rust/tests/runtime_lifecycle_r005.rs @@ -86,7 +86,7 @@ fn exact_r001_field_dispositions_are_ported() { .map(|(path, disposition)| ((*path).to_owned(), disposition.to_string())) .collect::>(); assert_eq!(actual, expected); - assert_eq!(actual.len(), 154); + assert_eq!(actual.len(), 155); assert_eq!( actual .iter() @@ -99,16 +99,17 @@ fn exact_r001_field_dispositions_are_ported() { .iter() .filter(|(_, disposition)| disposition == "restart_required") .count(), - 93 + 94 ); } #[test] fn serialized_schema_projection_is_the_coverage_guard() { - let expected = oracle_policy_rows() + let mut expected = oracle_policy_rows() .into_iter() .map(|(path, _)| path) .collect::>(); + expected.sort(); assert_eq!(schema_paths().expect("schema projection"), expected); } diff --git a/scripts/dashboard_parity/browser.py b/scripts/dashboard_parity/browser.py index 1066b43b4..813518471 100644 --- a/scripts/dashboard_parity/browser.py +++ b/scripts/dashboard_parity/browser.py @@ -10,7 +10,9 @@ # instead of being labelled with an unrelated correction. ORACLE_CANDIDATE_CORRECTIONS = { "static/dashboard.css": ( - "bound mobile panel intrinsic width within its table scroll wrapper" + "bound mobile panel intrinsic width within its table scroll wrapper, and " + "let the sixteen-entry top navigation reflow inside the topbar instead of " + "forcing one nowrap row that overflowed a 1280px viewport" ), "static/dashboard.js": ( "submit the timeseries filter form natively so the server renders the " diff --git a/scripts/dashboard_parity/cli.py b/scripts/dashboard_parity/cli.py index aa7d8f00d..528ee041c 100644 --- a/scripts/dashboard_parity/cli.py +++ b/scripts/dashboard_parity/cli.py @@ -10,6 +10,7 @@ uv run python scripts/qualification_dashboard_parity.py --skip-build uv run python scripts/qualification_dashboard_parity.py --skip-build --screenshots + uv run python scripts/qualification_dashboard_parity.py --skip-build --on-prem """ # ruff: noqa: F405 @@ -17,6 +18,7 @@ from ._shared import * # noqa: F403 from .fixtures import * # noqa: F403 +from .on_prem import * # noqa: F403 from .oracle import * # noqa: F403 from .process import * # noqa: F403 from .report import * # noqa: F403 @@ -47,6 +49,21 @@ def main() -> int: parser.add_argument("--output", type=Path, default=DEFAULT_JSON) parser.add_argument("--markdown", type=Path, default=DEFAULT_MARKDOWN) parser.add_argument("--screenshot-dir", type=Path) + parser.add_argument( + "--on-prem", + action="store_true", + help=( + "qualify the managed admin/member pages against the Rust-owned " + "approved reference instead of the historical Python oracle" + ), + ) + parser.add_argument( + "--write-on-prem-reference", + action="store_true", + help=( + "freeze the current on-prem browser observations as the approved reference" + ), + ) options = parser.parse_args() try: if options.write_manifest: @@ -57,6 +74,27 @@ def main() -> int: capture_oracle_snapshots(options.capture_oracle) print(f"Dashboard oracle captures written: {options.capture_oracle}") return 0 + if options.on_prem or options.write_on_prem_reference: + if not options.skip_build: + result = subprocess.run( + ["cargo", "build", "--manifest-path", str(RUST_MANIFEST)], + cwd=ROOT, + check=False, + ) + if result.returncode != 0: + raise QualificationError("Rust dashboard candidate failed to build") + with tempfile.TemporaryDirectory(prefix="eggpool-on-prem-browser-") as root: + print( + json.dumps( + qualify_on_prem_browser( + root=Path(root), + write_reference=options.write_on_prem_reference, + screenshot_dir=options.screenshot_dir, + ), + indent=2, + ) + ) + return 0 if options.shutdown_restart: if not options.skip_build: result = subprocess.run( diff --git a/scripts/dashboard_parity/on_prem.py b/scripts/dashboard_parity/on_prem.py new file mode 100644 index 000000000..d2ead67d3 --- /dev/null +++ b/scripts/dashboard_parity/on_prem.py @@ -0,0 +1,1006 @@ +# ruff: noqa: F405 +"""Browser qualification for the on-prem admin and member dashboard pages. + +The frozen Python oracle only ever implemented the historical M001-M012 +dashboard, so it cannot serve as the reference for the on-prem user-access pages +added afterwards. This module therefore keeps the gate split: + +* historical pages stay owned by the pinned Python oracle in ``runner.py``; +* on-prem pages are owned by EggPool itself and are qualified against a + Rust-owned approved reference that this module freezes and re-compares. + +Everything here reuses the existing CDP harness in ``browser.py`` and the +existing loopback fixtures in ``fixtures.py``; no second visual harness is +introduced. Only a managed on-prem instance is started here, because the +on-prem pages require a bootstrapped administrator and real member sessions +that the historical fixtures do not model. +""" + +from __future__ import annotations + +from ._shared import * # noqa: F403 +from .browser import _HeadlessScreenshotSession, _png_dimensions +from .fixtures import _start_server, _stop_server +from .process import _fetch, _port, _wait_for_tcp + +ON_PREM_DIR = ROOT / "tests" / "fixtures" / "dashboard-on-prem-browser" +ON_PREM_REFERENCE = ON_PREM_DIR / "reference.json" +ON_PREM_SCHEMA = 1 + +ADMIN_USERNAME = "admin" +MEMBER_USERNAME = "member" +PROFILE_ID = "team" +PROFILE_NAME = "Team" +TOKEN_LABEL = "qualification client" + +# Sentinel credentials planted in the managed fixture plus the real passwords the +# administrator and member use during the run. The gate proves that none of them +# ever reaches an admin or member HTML snapshot. +ACCOUNT_SECRET = "EGGPOOL_ONPREM_SECRET_ACCOUNT_KEY_7291" +STATIC_HEADER_SECRET = "EGGPOOL_ONPREM_SECRET_HEADER_VALUE_7291" +ADMIN_PASSWORD = "M011 qualification admin passphrase 7291" +MEMBER_TEMP_PASSWORD = "M011 temporary member passphrase 7291" +MEMBER_PASSWORD = "M011 rotated member passphrase 7291" +SECRET_MARKERS = ( + ACCOUNT_SECRET, + STATIC_HEADER_SECRET, + ADMIN_PASSWORD, + MEMBER_TEMP_PASSWORD, + MEMBER_PASSWORD, +) + +ADMIN_PAGES = ("/users", "/profiles", "/admin/audit") +MEMBER_PAGES = ("/my-account", "/my-account/models", "/usage") +# `/models` and `/latency` stay administrator-only, so a member must be refused +# there rather than served a member variant. +ADMIN_ONLY_PAGES = ("/users", "/profiles", "/admin/audit", "/models", "/latency") +# ``(label, width, height, device_scale_factor)``; HiDPI is qualified separately +# from the narrow viewport so a scaled desktop render is covered too. +ON_PREM_VIEWPORTS: tuple[tuple[str, int, int, float], ...] = ( + ("desktop", 1280, 800, 1.0), + ("mobile", 390, 844, 1.0), + ("desktop-hidpi", 1280, 800, 2.0), +) +# Every theme must render without clipping or overflow at every viewport. +ON_PREM_THEMES = ("default", "Cyber Red", "Catppuccin Latte", "Cyberpunk") + +_MEMBER_ROW_EXPRESSION = """(function () { + const rows = Array.from(document.querySelectorAll('table tbody tr')); + return JSON.stringify(rows.map(row => { + // The username is a row header (``), not a data cell. + const cells = Array.from(row.querySelectorAll('th, td')); + const stateField = row.querySelector( + 'form[action="/admin/users/state"] input[name="user_id"]'); + return { + username: cells.length ? cells[0].textContent.trim() : '', + role: cells.length > 1 ? cells[1].textContent.trim() : '', + state: cells.length > 2 ? cells[2].textContent.trim() : '', + must_change: cells.length > 3 + ? cells[3].textContent.includes('Must change password') : false, + id: stateField ? stateField.value : '', + }; + })); +})()""" + +_ONE_TIME_TOKEN_EXPRESSION = """(function () { + const pre = document.querySelector('pre'); + return pre ? pre.textContent.trim() : ''; +})()""" + +# Budget wording must stay truthful: enforced ceilings are hard limits, and +# cost/token ceilings are estimates until a request is priced. +_BUDGET_LABEL_EXPRESSION = """(function () { + // Budget controls live inside collapsed `
    `, whose text is not part of + // `innerText` until they are open. + document.querySelectorAll('details').forEach(node => { node.open = true; }); + const text = document.body.innerText; + const labels = []; + for (const needle of ['(hard)', 'estimated', 'hard', 'Hard', 'Estimated']) { + if (text.includes(needle)) labels.push(needle); + } + return JSON.stringify(labels); +})()""" + +_DOM_FACTS_TEMPLATE = r"""JSON.stringify((() => { + const markers = __SECRET_MARKERS__; + const ids = Array.from(document.querySelectorAll('[id]'), node => node.id); + const viewportWidth = document.documentElement.clientWidth; + const forms = Array.from(document.forms, form => ({ + action: new URL(form.action || location.href, location.href).pathname, + method: (form.method || 'get').toUpperCase(), + fields: Array.from(form.elements, element => element.name) + .filter(name => Boolean(name)).sort(), + unlabelled: Array.from( + form.querySelectorAll('input, select, textarea'), element => { + if (element.type === 'hidden') return null; + if (element.getAttribute('aria-label')) return null; + if (element.id && document.querySelector( + 'label[for="' + element.id + '"]')) return null; + if (element.closest('label')) return null; + return element.tagName.toLowerCase() + ':' + (element.name || '(unnamed)'); + }).filter(Boolean), + })); + const tables = Array.from(document.querySelectorAll('table'), table => { + const width = Math.round(table.getBoundingClientRect().width); + if (width <= viewportWidth) return null; + return {wrapped: Boolean(table.closest('.table-scroll')), width: width}; + }).filter(Boolean); + // `html { overflow-x: hidden }` makes the root a clipped scroller, so a + // programmatic `scrollTo` can still move it and would misreport. Measure the + // document against the viewport instead, and name the offending elements. + const root = document.documentElement; + const horizontalOverflow = root.scrollWidth > root.clientWidth + 1; + const overflowing = horizontalOverflow + ? Array.from(document.querySelectorAll('body *')).filter(element => { + const box = element.getBoundingClientRect(); + if (box.width <= viewportWidth + 1) return false; + return !element.closest('.table-scroll, .topnav-menu'); + }).slice(0, 5).map(element => { + const classes = typeof element.className === 'string' + ? element.className.trim().split(/\s+/).filter(Boolean) : []; + return element.tagName.toLowerCase() + + (classes.length ? '.' + classes.join('.') : ''); + }) + : []; + const html = document.documentElement.outerHTML; + return { + path: location.pathname + location.search, + title: document.title, + h1: Array.from(document.querySelectorAll('h1'), + node => node.textContent.trim()), + h2: Array.from(document.querySelectorAll('h2'), + node => node.textContent.trim()), + viewportMeta: Boolean(document.querySelector( + 'meta[name="viewport"]')), + duplicateIds: ids.filter((id, index) => ids.indexOf(id) !== index), + forms: forms, + tables: tables, + horizontalOverflow: horizontalOverflow, + overflowing: overflowing, + secretMarkers: markers.filter(marker => html.includes(marker)), + }; +})())""" + + +def _dom_facts_expression() -> str: + """Return the browser-side probe with the live credential markers embedded.""" + return _DOM_FACTS_TEMPLATE.replace( + "__SECRET_MARKERS__", json.dumps(list(SECRET_MARKERS)) + ) + + +def _on_prem_provider_config() -> str: + """Return fixture providers carrying sentinel credentials for the leak guard.""" + return ( + f"[providers.onprem-alpha]\n" + f'id = "onprem-alpha"\n' + f'base_url = "https://fixture.invalid/onprem-alpha"\n' + f'protocols = ["openai"]\n\n' + f"[providers.onprem-alpha.auth]\n" + f'mode = "bearer"\n\n' + f"[providers.onprem-alpha.models_endpoint]\n" + f'method = "GET"\n\n' + f"[[providers.onprem-alpha.headers]]\n" + f'name = "X-Fixture"\n' + f'value = "{STATIC_HEADER_SECRET}"\n\n' + f"[[providers.onprem-alpha.accounts]]\n" + f'name = "onprem-alpha-main"\n' + f'api_key = "{ACCOUNT_SECRET}"\n\n' + f"[[providers.onprem-alpha.static_models]]\n" + f'id = "onprem-chat-model"\n' + f'protocol = "openai"\n\n' + ) + + +def _write_on_prem_config(root: Path, port: int) -> Path: + """Write a managed-instance config with no server API key.""" + database = root / "on-prem.sqlite3" + path = root / "on-prem.toml" + path.write_text( + f"[server]\n" + f'host = "127.0.0.1"\nport = {port}\n\n[database]\n' + f'path = "{database}"\n\n[dashboard]\nenabled = true\n' + f'public = true\ntheme = "Cyber Red"\n' + "refresh_interval_s = 1\n\n[models]\n" + "startup_refresh = false\n\n[model_info]\n" + "enabled = false\nstartup_refresh = false\n\n" + f"{_on_prem_provider_config()}", + encoding="utf-8", + ) + return path + + +def _bootstrap_admin(config: Path, env: dict[str, str]) -> None: + """Create the managed administrator through the supported CLI surface.""" + completed = subprocess.run( + [ + str(RUST_BINARY), + "--config", + str(config), + "users", + "bootstrap", + ADMIN_USERNAME, + "--password-stdin", + ], + cwd=ROOT, + env=env, + input=f"{ADMIN_PASSWORD}\n", + capture_output=True, + text=True, + check=False, + timeout=60, + ) + if completed.returncode != 0: + detail = completed.stderr.strip() or completed.stdout.strip() + raise QualificationError(f"admin bootstrap failed: {detail}") + + +class _ManagedPage: + """Drive one browser context against the managed on-prem instance.""" + + def __init__(self, session: _HeadlessScreenshotSession, origin: str) -> None: + self._session = session + self._client = session.client + self.origin = origin + + def viewport(self, width: int, height: int, scale: float) -> None: + self._client.request( + "Emulation.setDeviceMetricsOverride", + { + "width": width, + "height": height, + "deviceScaleFactor": scale, + "mobile": width < 600, + }, + ) + + def _evaluate(self, expression: str) -> Any: + result = self._client.request( + "Runtime.evaluate", + {"expression": expression, "awaitPromise": True, "returnByValue": True}, + ) + remote = cast("dict[str, Any]", result.get("result", {})) + if remote.get("subtype") == "error": + raise QualificationError(f"page script failed: {remote.get('description')}") + return remote.get("value") + + def script(self, expression: str) -> Any: + """Evaluate one browser-side probe against the committed document.""" + return self._evaluate(expression) + + def rows(self) -> list[dict[str, Any]]: + """Return the rendered local-account rows with their stable shape.""" + return cast( + "list[dict[str, Any]]", + json.loads(str(self._evaluate(_MEMBER_ROW_EXPRESSION))), + ) + + def one_time_token(self) -> str: + return cast("str", self._evaluate(_ONE_TIME_TOKEN_EXPRESSION)) + + def budget_labels(self) -> list[str]: + return cast( + "list[str]", json.loads(str(self._evaluate(_BUDGET_LABEL_EXPRESSION))) + ) + + def has_selector(self, selector: str) -> bool: + return cast( + "bool", + self._evaluate(f"Boolean(document.querySelector({json.dumps(selector)}))"), + ) + + def text_length(self) -> int: + return cast("int", self._evaluate("document.body.innerText.length")) + + def _mark_document(self) -> None: + """Tag the outgoing document so a committed navigation is observable.""" + self._client.events.clear() + self._evaluate("window.__eggpoolNavigationMark = true") + + def wait_for_navigation(self) -> None: + """Block on a committed document rather than guessing with a sleep. + + A submitted form leaves the previous document ready until the redirect + commits, so polling ``document.readyState`` would happily observe the + outgoing page. The marker below only disappears when a new document has + actually replaced it, which keeps the gate on an observable fixture + boundary instead of a timing assumption. + """ + deadline = time.monotonic() + 20 + while time.monotonic() < deadline: + # Pumping a trivial evaluation drains buffered protocol traffic. + self._client.request("Runtime.evaluate", {"expression": "1"}) + committed = self._evaluate( + "typeof window.__eggpoolNavigationMark === 'undefined'" + " && document.readyState === 'complete'" + ) + if committed: + self._evaluate("document.fonts ? document.fonts.ready : true") + return + time.sleep(0.02) + raise QualificationError("navigation did not commit a new document") + + def goto(self, path: str) -> str: + self._mark_document() + self._client.request("Page.navigate", {"url": f"{self.origin}{path}"}) + self.wait_for_navigation() + return cast("str", self._evaluate("location.pathname + location.search")) + + def facts(self) -> dict[str, Any]: + raw = self._evaluate(_dom_facts_expression()) + return cast("dict[str, Any]", json.loads(str(raw))) + + def fill(self, selector: str, value: str) -> None: + """Set one control's value the way typing into it would.""" + self._evaluate( + "(function () {" + f" const field = document.querySelector({json.dumps(selector)});" + " if (!field) throw new Error('missing field ' + " + f"{json.dumps(selector)});" + f" field.value = {json.dumps(value)};" + " field.dispatchEvent(new Event('input', {bubbles: true}));" + " field.dispatchEvent(new Event('change', {bubbles: true}));" + " return true; })()" + ) + + def csrf(self, action: str) -> str: + """Return the CSRF token the page already minted for one form action.""" + escaped = json.dumps(action) + value = self._evaluate( + "(function () {" + f" const form = document.querySelector('form[action={escaped}]');" + " if (!form) return null;" + " const field = form.querySelector('input[name=\"csrf\"]');" + " return field ? field.value : null; })()" + ) + if not value: + raise QualificationError(f"no CSRF-fenced form for {action}") + return cast("str", value) + + def submit(self, action: str, fields: dict[str, str]) -> str: + """Fill and submit one server-rendered form exactly as a browser would.""" + escaped = json.dumps(action) + self._mark_document() + self._evaluate( + "(function () {" + f" const form = document.querySelector('form[action={escaped}]');" + " if (!form) throw new Error('missing form ' + " + escaped + ");" + f" const values = {json.dumps(fields)};" + " for (const [name, value] of Object.entries(values)) {" + " const field = form.elements.namedItem(name);" + " if (!field) throw new Error('missing field ' + name);" + " field.value = value;" + " field.dispatchEvent(new Event('input', {bubbles: true}));" + " field.dispatchEvent(new Event('change', {bubbles: true}));" + " }" + " form.requestSubmit();" + " return true; })()" + ) + self.wait_for_navigation() + return cast("str", self._evaluate("location.pathname + location.search")) + + def screenshot(self, artifact: Path) -> tuple[int, int, str]: + result = self._client.request( + "Page.captureScreenshot", {"format": "png", "fromSurface": True} + ) + payload = base64.b64decode(str(result.get("data", "")), validate=True) + artifact.parent.mkdir(parents=True, exist_ok=True) + artifact.write_bytes(payload) + return (*_png_dimensions(artifact), hashlib.sha256(payload).hexdigest()) + + def submit_button_point(self, action: str) -> tuple[float, float]: + """Return the viewport point of a form's submit button. + + Measured while scripting is still enabled so the button can then be + activated by a real mouse event after scripts are switched off. + """ + escaped = json.dumps(action) + point = self.script( + "(function () {" + f" const form = document.querySelector('form[action={escaped}]');" + " if (!form) throw new Error('missing form ' + " + escaped + ");" + " const button = form.querySelector(" + " 'button[type=submit], button:not([type])');" + " if (!button) throw new Error('missing submit button');" + " button.scrollIntoView({block: 'center', behavior: 'instant'});" + " const box = button.getBoundingClientRect();" + " return [box.left + box.width / 2, box.top + box.height / 2];" + " })()" + ) + coordinates = cast("list[float]", point) + return (coordinates[0], coordinates[1]) + + def click_at(self, x: float, y: float) -> None: + """Dispatch a real trusted-style mouse click, not a scripted call. + + `buttons` has to describe the held button state or Chrome drops the + press and no click is ever synthesised for the element under the point. + """ + for event_type, buttons in (("mousePressed", 1), ("mouseReleased", 0)): + self._client.request( + "Input.dispatchMouseEvent", + { + "type": event_type, + "x": x, + "y": y, + "button": "left", + "buttons": buttons, + "clickCount": 1, + }, + ) + + def sign_in_without_scripts( + self, username: str, password: str + ) -> tuple[str, dict[str, Any]]: + """Fill and submit the login form with page scripting switched off. + + The button is activated by a real mouse event, so nothing in this path + depends on the page's own JavaScript running. + """ + self.goto("/login") + self.fill("input[name=username]", username) + self.fill("input[name=password]", password) + point = self.submit_button_point("/login") + # Tag the outgoing document before scripting is switched off: the marker + # itself is page script, and a disabled page would never set it, leaving + # the wait below to return before the click had navigated. + self._mark_document() + self.disable_scripts(True) + try: + self.click_at(*point) + self.wait_for_navigation() + landed = cast("str", self._evaluate("location.pathname + location.search")) + notice = cast( + "str", + self.script( + "(function () { const alert =" + " document.querySelector('[role=alert]');" + " return alert ? alert.textContent : ''; })()" + ), + ).strip() + if landed.startswith("/login"): + raise QualificationError( + f"{username} could not sign in with scripting disabled: {notice!r}" + ) + finally: + self.disable_scripts(False) + return landed, {"sign_in": landed, "scripts_required": False} + + def sign_in(self, username: str, password: str) -> str: + """Sign in through the real form and prove the session took effect.""" + self.goto("/login") + landed = self.submit("/login", {"username": username, "password": password}) + if landed.startswith("/login"): + notice = cast( + "str", + self.script( + "(function () { const alert =" + " document.querySelector('[role=alert]');" + " return alert ? alert.textContent : ''; })()" + ), + ).strip() + raise QualificationError( + f"{username} was not signed in and stayed on the login form: {notice!r}" + ) + return landed + + def sign_out(self) -> None: + if self.has_selector('form[action="/logout"]'): + self.submit("/logout", {}) + + def disable_scripts(self, disabled: bool) -> None: + self._client.request( + "Emulation.setScriptExecutionDisabled", {"value": disabled} + ) + + +def _report_path(path: Path) -> str: + """Render a path for the run summary without failing outside the repo.""" + try: + return str(path.relative_to(ROOT)) + except ValueError: + return str(path) + + +def _stable_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: + """Drop per-run identifiers so the approved reference stays comparable.""" + return [{key: value for key, value in row.items() if key != "id"} for row in rows] + + +def _canonical(value: Any) -> str: + """Render one observation as stable JSON for the approved reference.""" + return json.dumps(value, sort_keys=True, ensure_ascii=False, separators=(",", ":")) + + +def _record(observations: dict[str, str], key: str, value: Any) -> dict[str, Any]: + observations[key] = _canonical(value) + return cast("dict[str, Any]", value) + + +def _assert_no_defects(key: str, facts: dict[str, Any], label: str) -> None: + """Fail on the bounded defect classes this plan is allowed to fix.""" + problems: list[str] = [] + if facts["duplicateIds"]: + problems.append(f"duplicate ids {facts['duplicateIds']}") + if facts["horizontalOverflow"]: + problems.append( + "page overflows the viewport" + + (f" at {facts['overflowing']}" if facts.get("overflowing") else "") + ) + if facts["tables"]: + unwrapped = [item["width"] for item in facts["tables"] if not item["wrapped"]] + if unwrapped: + problems.append(f"unbounded tables wider than viewport: {unwrapped}") + unlabelled = [ + f"{form['action']}:{name}" + for form in facts["forms"] + for name in form["unlabelled"] + ] + if unlabelled: + problems.append(f"unlabelled controls {unlabelled}") + if facts["secretMarkers"]: + problems.append(f"secret markers rendered {sorted(facts['secretMarkers'])}") + if not facts["viewportMeta"]: + problems.append("missing viewport meta") + if problems: + raise QualificationError(f"{label} {key}: " + "; ".join(problems)) + + +def _collect_page( + page: _ManagedPage, + observations: dict[str, str], + *, + role: str, + path: str, + viewport: tuple[str, int, int, float], + screenshot_dir: Path | None, + screenshots: list[dict[str, Any]], + theme: str = "", +) -> None: + name, width, height, scale = viewport + page.viewport(width, height, scale) + suffix = f"?theme={theme.replace(' ', '%20')}" if theme else "" + page.goto(f"{path}{suffix}") + key = f"page.{role}.{path}.{name}" + (f".{theme}" if theme else "") + facts = _record(observations, key, page.facts()) + _assert_no_defects(key, facts, role) + if screenshot_dir is not None: + width_px, height_px, digest = page.screenshot( + screenshot_dir / f"{role}{path.replace('/', '-')}-{name}" + f"{'-' + theme.replace(' ', '-') if theme else ''}.png" + ) + screenshots.append( + { + "key": key, + "width": width_px, + "height": height_px, + "sha256": digest, + } + ) + + +def qualify_on_prem_browser( + *, + root: Path, + write_reference: bool = False, + screenshot_dir: Path | None = None, +) -> dict[str, Any]: + """Qualify the managed admin/member pages against the approved reference.""" + instance = root / "on-prem" + instance.mkdir(parents=True) + home = instance / "home" + home.mkdir(exist_ok=True) + port = _port() + # The control socket has a hard path-length ceiling, so the runtime directory + # stays short like the historical fixtures instead of living under the + # already-deep temporary instance root. + runtime = Path(f"/tmp/eq11-onprem-{port}") + config = _write_on_prem_config(instance, port) + env = dict(os.environ) + for key in ( + "EGGPOOL_CONFIG", + "EGGPOOL_RUNTIME_DIR", + "EGGPOOL_PID_FILE", + "EGGPOOL_API_KEY", + ): + env.pop(key, None) + env.update( + { + "HOME": str(home), + "TZ": "UTC", + "EGGPOOL_RUNTIME_DIR": str(runtime), + "EGGPOOL_PID_FILE": str(instance / "on-prem.pid"), + } + ) + observations: dict[str, str] = {} + screenshots: list[dict[str, Any]] = [] + _bootstrap_admin(config, env) + server = _start_server([str(RUST_BINARY)], config, env) + session: _HeadlessScreenshotSession | None = None + try: + _wait_for_tcp(port, server, "managed on-prem dashboard") + origin = f"http://127.0.0.1:{port}" + session = _HeadlessScreenshotSession() + page = _ManagedPage(session, origin) + + # Anonymous browsers must never reach any managed page. + for path in (*ADMIN_ONLY_PAGES, "/my-account", "/usage"): + page.viewport(1280, 800, 1.0) + landed = page.goto(path) + _record(observations, f"anonymous.{path}", landed) + + # Sign in as the administrator with scripting disabled to prove the login + # and every later form work without JavaScript. + landed, no_js = page.sign_in_without_scripts(ADMIN_USERNAME, ADMIN_PASSWORD) + _record(observations, "admin.sign_in", landed) + _record(observations, "admin.no_js_login", no_js) + + page.goto("/profiles") + csrf = page.csrf("/admin/profiles/create") + _record( + observations, + "admin.profile_created", + page.submit( + "/admin/profiles/create", + { + "profile_id": PROFILE_ID, + "name": PROFILE_NAME, + "default_effect": "deny", + "csrf": csrf, + }, + ), + ) + page.goto("/users") + csrf = page.csrf("/admin/users/create") + _record( + observations, + "admin.member_created", + page.submit( + "/admin/users/create", + { + "username": MEMBER_USERNAME, + "password": MEMBER_TEMP_PASSWORD, + "profile_id": PROFILE_ID, + "csrf": csrf, + }, + ), + ) + issued = page.goto("/users") + _record(observations, "admin.after_member_created", issued) + member_state = page.rows() + _record(observations, "admin.member_row", _stable_rows(member_state)) + member = next( + (row for row in member_state if row["username"] == MEMBER_USERNAME), None + ) + if member is None or member["must_change"] is not True: + raise QualificationError( + f"member was not created as a forced password change: {member_state}" + ) + + page.goto("/users") + csrf = page.csrf("/admin/users/state") + _record( + observations, + "admin.member_suspended", + page.submit( + "/admin/users/state", + {"user_id": member["id"], "enabled": "0", "csrf": csrf}, + ), + ) + page.goto("/users") + csrf = page.csrf("/admin/users/state") + _record( + observations, + "admin.member_reactivated", + page.submit( + "/admin/users/state", + {"user_id": member["id"], "enabled": "1", "csrf": csrf}, + ), + ) + + for path in ADMIN_PAGES: + for viewport in ON_PREM_VIEWPORTS: + _collect_page( + page, + observations, + role="admin", + path=path, + viewport=viewport, + screenshot_dir=screenshot_dir, + screenshots=screenshots, + ) + for theme in ON_PREM_THEMES: + _collect_page( + page, + observations, + role="admin", + path="/users", + viewport=ON_PREM_VIEWPORTS[1], + screenshot_dir=screenshot_dir, + screenshots=screenshots, + theme=theme, + ) + + page.goto("/my-account") + csrf = page.csrf("/account/tokens") + reveal = page.submit("/account/tokens", {"label": TOKEN_LABEL, "csrf": csrf}) + _record(observations, "admin.token_reveal", reveal) + plaintext = page.one_time_token() + if len(plaintext) < 16: + raise QualificationError("one-time token reveal did not expose a token") + _record(observations, "admin.token_revealed_once", True) + # The reveal is the only place the plaintext may ever appear: returning to + # the account page must show the credential label, never the credential. + page.goto("/my-account") + _record( + observations, + "admin.token_absent_on_return", + plaintext + not in cast("str", page.script("document.documentElement.outerHTML")), + ) + csrf = page.csrf("/account/tokens") + second = page.submit( + "/account/tokens", {"label": f"{TOKEN_LABEL} again", "csrf": csrf} + ) + _record(observations, "admin.token_second_reveal", second) + second_plaintext = page.one_time_token() + _record( + observations, + "admin.token_reveal_is_fresh_each_time", + second_plaintext.startswith("epk1_") + and second_plaintext != plaintext + and plaintext not in second_plaintext, + ) + + page.goto("/users") + csrf = page.csrf("/admin/users/override") + _record( + observations, + "admin.override_saved", + page.submit( + "/admin/users/override", + { + "user_id": member["id"], + "model_id": "onprem-chat-model", + "effect": "allow", + "csrf": csrf, + }, + ), + ) + page.goto("/users") + csrf = page.csrf("/admin/users/budget") + _record( + observations, + "admin.budget_saved", + page.submit( + "/admin/users/budget", + { + "user_id": member["id"], + "model_id": "", + "window": "month", + "max_requests": "100", + "max_tokens": "100000", + "max_microdollars": "50000", + "max_concurrent": "4", + "csrf": csrf, + }, + ), + ) + _record( + observations, + "admin.budget_labels", + page.budget_labels(), + ) + page.goto("/users") + csrf = page.csrf("/admin/users/profile") + _record( + observations, + "admin.profile_conflict", + page.submit( + "/admin/users/profile", + { + "user_id": member["id"], + "profile_id": "missing-profile", + "csrf": csrf, + }, + ), + ) + + page.sign_out() + landed = page.sign_in(MEMBER_USERNAME, MEMBER_TEMP_PASSWORD) + _record(observations, "member.forced_password_landing", landed) + csrf = page.csrf("/account/password") + _record( + observations, + "member.password_changed", + page.submit( + "/account/password", + { + "current_password": MEMBER_TEMP_PASSWORD, + "new_password": MEMBER_PASSWORD, + "csrf": csrf, + }, + ), + ) + # Changing a password bumps the account auth epoch, so the browser + # session is invalidated on purpose and the member has to sign in again. + _record( + observations, + "member.reauthenticated_after_password_change", + page.sign_in(MEMBER_USERNAME, MEMBER_PASSWORD), + ) + for path in MEMBER_PAGES: + for viewport in ON_PREM_VIEWPORTS: + _collect_page( + page, + observations, + role="member", + path=path, + viewport=viewport, + screenshot_dir=screenshot_dir, + screenshots=screenshots, + ) + for path in ADMIN_ONLY_PAGES: + page.viewport(1280, 800, 1.0) + _record(observations, f"member_denied.{path}", page.goto(path)) + _record( + observations, + f"member_denied.{path}.message", + cast("str", page.script("document.body.innerText")).strip(), + ) + page.goto("/usage") + _record( + observations, + "member.budget_labelling", + page.budget_labels(), + ) + + page.sign_out() + for path in ADMIN_ONLY_PAGES: + page.viewport(1280, 800, 1.0) + _record(observations, f"after_logout.{path}", page.goto(path)) + + agent_token = _issue_agent_token(config, env) + for path in ("/users", "/my-account"): + status = _fetch( + f"{origin}{path}", headers={"Authorization": f"Bearer {agent_token}"} + ) + _record(observations, f"agent_token.{path}", status.status) + _stop_server(server) + server = _start_server([str(RUST_BINARY)], config, env) + _wait_for_tcp(port, server, "restarted managed on-prem dashboard") + # The signed-in browser session and every managed record live in the + # database, so a restart must not lose the account or its policy state. + _record( + observations, + "restart.session_survives_restart", + page.goto("/users"), + ) + page.goto("/profiles") + _record( + observations, + "restart.profiles_survive_restart", + PROFILE_ID in cast("str", page.script("document.body.innerText")), + ) + page.goto("/users") + _record( + observations, + "restart.member_row", + _stable_rows( + [ + row + for row in page.rows() + if row["username"] in {ADMIN_USERNAME, MEMBER_USERNAME} + ] + ), + ) + finally: + if session is not None: + session.close() + _stop_server(server) + shutil.rmtree(runtime, ignore_errors=True) + + return _reconcile_reference(observations, screenshots, write_reference) + + +def _issue_agent_token(config: Path, env: dict[str, str]) -> str: + """Issue a personal agent token to prove agent credentials stay off the UI.""" + completed = subprocess.run( + [ + str(RUST_BINARY), + "--config", + str(config), + "users", + "token", + ADMIN_USERNAME, + "--label", + "agent probe", + "--show-secret", + ], + cwd=ROOT, + env=env, + capture_output=True, + text=True, + check=False, + timeout=60, + ) + if completed.returncode != 0: + raise QualificationError( + f"agent token issue failed: {completed.stderr.strip()}" + ) + for line in completed.stdout.splitlines(): + candidate = line.strip() + if candidate.startswith("epk1_"): + return candidate + raise QualificationError(f"agent token plaintext not found: {completed.stdout}") + + +def _reconcile_reference( + observations: dict[str, str], + screenshots: list[dict[str, Any]], + write_reference: bool, +) -> dict[str, Any]: + """Freeze or compare this run against the Rust-owned approved reference.""" + candidate = { + "schema": ON_PREM_SCHEMA, + "observations": observations, + "screenshots": screenshots, + } + if write_reference: + ON_PREM_DIR.mkdir(parents=True, exist_ok=True) + ON_PREM_REFERENCE.write_text( + json.dumps(candidate, indent=2, sort_keys=True, ensure_ascii=False) + "\n", + encoding="utf-8", + ) + return { + "result": "reference-written", + "reference": _report_path(ON_PREM_REFERENCE), + "observations": len(observations), + "screenshots": len(screenshots), + } + if not ON_PREM_REFERENCE.is_file(): + raise QualificationError( + f"missing approved on-prem reference " + f"{_report_path(ON_PREM_REFERENCE)}; re-run with " + "--write-on-prem-reference after reviewing the diff" + ) + approved = json.loads(ON_PREM_REFERENCE.read_text(encoding="utf-8")) + if approved.get("schema") != ON_PREM_SCHEMA: + raise QualificationError( + f"approved on-prem reference schema {approved.get('schema')} " + f"does not match {ON_PREM_SCHEMA}" + ) + expected = cast("dict[str, str]", approved.get("observations", {})) + differences: list[str] = [] + for key in sorted(set(expected) | set(observations)): + if key not in observations: + differences.append(f"{key}: absent from this run") + elif key not in expected: + differences.append( + f"{key}: {observations[key]} is not in the approved reference" + ) + elif observations[key] != expected[key]: + differences.append(f"{key}: {expected[key]} -> {observations[key]}") + if differences: + raise QualificationError( + "on-prem browser drift from the approved reference: " + + "; ".join(differences[:20]) + + (f"; and {len(differences) - 20} more" if len(differences) > 20 else "") + ) + return { + "result": "passed", + "reference": _report_path(ON_PREM_REFERENCE), + "observations": len(observations), + "screenshots": len(screenshots), + } + + +__all__ = [ + "ON_PREM_DIR", + "ON_PREM_REFERENCE", + "ON_PREM_SCHEMA", + "ON_PREM_THEMES", + "ON_PREM_VIEWPORTS", + "QualificationError", + "qualify_on_prem_browser", +] diff --git a/scripts/qualification_live_provider.py b/scripts/qualification_live_provider.py index 0802b6a45..d542a09f5 100644 --- a/scripts/qualification_live_provider.py +++ b/scripts/qualification_live_provider.py @@ -55,7 +55,6 @@ DEFAULT_OUTPUT = ROOT / "artifacts/qualification/007-run.json" MANIFEST_VERSION = "runtime-q001.v1" SCHEMA_VERSION = "runtime-q007.v1" -SERVER_KEY = "q007-server-key" FAKE_PROVIDER_KEY = "q007-provider-key" OPENCODE_SESSION_HEADER = "x-opencode-session" MAX_REQUESTS = 8 @@ -909,6 +908,7 @@ def run_qualification( config = root / "config.toml" process: subprocess.Popen[bytes] | None = None fake_provider: FakeProvider | None = None + agent_token: str | None = None output_stream: Any | None = None error_stream: Any | None = None stdout_path = root / "candidate.stdout" @@ -949,7 +949,36 @@ def run_qualification( child_env["Q011_MINIMAX_API_KEY"] = secondary_key else: child_env["Q007_PROVIDER_API_KEY"] = FAKE_PROVIDER_KEY - child_env["SERVER_API_KEY"] = SERVER_KEY + bootstrap = subprocess.run( + [ + str(binary.resolve()), + "--config", + str(config), + "users", + "bootstrap", + "q007-admin", + "--password-stdin", + "--show-secret", + ], + input="Q007 synthetic bootstrap password\n", + capture_output=True, + text=True, + check=False, + cwd=ROOT, + env=child_env, + timeout=timeout, + ) + if bootstrap.returncode != 0: + raise QualificationError( + "synthetic administrator bootstrap failed: " + + bounded(bootstrap.stderr or bootstrap.stdout) + ) + token_match = re.search(r"epk1_[A-Za-z0-9_-]+", bootstrap.stdout) + if token_match is None: + raise QualificationError( + "synthetic administrator bootstrap returned no personal token" + ) + agent_token = token_match.group(0) child_env.setdefault("RUST_LOG", "debug") output_stream = stdout_path.open("wb") error_stream = stderr_path.open("wb") @@ -973,7 +1002,7 @@ def run_qualification( _wait_ready(process, f"{base_url}/v1/healthz", timeout) status, _, models_body = _http( f"{base_url}/v1/models", - headers={"Authorization": f"Bearer {SERVER_KEY}"}, + headers={"Authorization": f"Bearer {agent_token}"}, ) if status != 200: raise QualificationError(f"model catalog returned HTTP {status}") @@ -1007,7 +1036,7 @@ def run_qualification( for case in cases: request_body = json.dumps(_payload(case)).encode() headers = { - "Authorization": f"Bearer {SERVER_KEY}", + "Authorization": f"Bearer {agent_token}", "Content-Type": "application/json", OPENCODE_SESSION_HEADER: f"q007-{case.case_id}", } @@ -1038,7 +1067,7 @@ def run_qualification( raise QualificationError(f"{case.case_id} lacks terminal evidence") runtime_status, _, _ = _http( f"{base_url}/api/stats/runtime", - headers={"Authorization": f"Bearer {SERVER_KEY}"}, + headers={"Authorization": f"Bearer {agent_token}"}, ) report["runtime_status_http"] = runtime_status except (OSError, QualificationError) as error: @@ -1053,12 +1082,20 @@ def run_qualification( if stdout_path.exists(): report["candidate_stdout"] = bounded( stdout_path.read_text(encoding="utf-8", errors="replace"), - tuple(secret for secret in (provider_key, secondary_key) if secret), + tuple( + secret + for secret in (provider_key, secondary_key, agent_token) + if secret + ), ) if stderr_path.exists(): report["candidate_stderr"] = bounded( stderr_path.read_text(encoding="utf-8", errors="replace"), - tuple(secret for secret in (provider_key, secondary_key) if secret), + tuple( + secret + for secret in (provider_key, secondary_key, agent_token) + if secret + ), ) if fake_provider is not None: fake_provider.__exit__(None, None, None) diff --git a/scripts/qualification_sbc.py b/scripts/qualification_sbc.py index 5d79fc88f..aca84a968 100644 --- a/scripts/qualification_sbc.py +++ b/scripts/qualification_sbc.py @@ -10,6 +10,31 @@ --binary rust/target/release/eggpool \ --output artifacts/qualification/008-run.json +Extended attribution examples (each invocation is one independent matrix cell):: + + TMPDIR=/mnt/qualification-ssd uv run python scripts/qualification_sbc.py \ + --binary rust/target/release/eggpool \ + --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml \ + --benchmark-samples 30 --concurrency-workers 4 \ + --concurrency-requests 1000 --benchmark-principals 5 \ + --benchmark-budget-mode user-global \ + --output artifacts/qualification/m001-ssd-c4-u5-user-global-1.json + + uv run python scripts/qualification_sbc.py \ + --binary rust/target/release/eggpool \ + --config-fixture tests/tooling/fixtures/qualification/sbc-benchmark.toml \ + --benchmark-samples 30 --concurrency-workers 4 \ + --concurrency-requests 1000 --benchmark-principals 5 \ + --benchmark-budget-mode user-global \ + --competing-io-mib-per-second 1 --competing-io-max-mib 128 \ + --output artifacts/qualification/m001-mmc-competing-c4-u5-user-global-1.json + +The synthetic user count is a configured account count; requests are round-robin +authenticated across those personal tokens. The runner records only aggregate +counts and sanitized mount classes. p99 is emitted only with at least 1000 +concurrent request observations. Competing writes have explicit rate and total +volume caps and run only during the concurrent batch. + The command is intentionally refused unless Linux/aarch64 and a device-tree board model are visible. This prevents a hosted ARM VM from being reported as the mandatory physical SBC evidence. @@ -20,6 +45,7 @@ import argparse import contextlib import hashlib +import http.cookiejar import json import os import platform @@ -32,6 +58,7 @@ import threading import time import urllib.error +import urllib.parse import urllib.request import zipfile from collections.abc import Mapping, Sequence @@ -60,6 +87,12 @@ BENCHMARK_WARMUPS = 5 CONCURRENCY_BATCH_SIZE = 32 CONCURRENCY_WORKERS = 4 +CONCURRENCY_WORKERS_MAX = 8 +CONCURRENCY_REQUESTS_MAX = 4000 +P99_MIN_SAMPLES = 1000 +COMPETING_IO_MAX_MIB_PER_SECOND = 4 +COMPETING_IO_MAX_MIB = 128 +SYNTHETIC_ADMIN_PASSWORD = "SBC synthetic bootstrap password 123" STABILIZATION_SECONDS = 3.0 DIAGNOSTIC_MIN_SAMPLES = 10 DIAGNOSTIC_MAX_SAMPLES = 200 @@ -214,6 +247,7 @@ def bounded(value: str) -> str: r"(?i)(api[_-]?key|token|password|secret)=\S+", r"\1=", text ) text = re.sub(r"\bsk-[A-Za-z0-9_-]+", "", text) + text = re.sub(r"\bepk1_[A-Za-z0-9_-]+", "", text) text = re.sub(r"https?://[^\s/@:]+:[^\s/@]+@", "@", text) return text.encode("utf-8", "replace")[:MAX_REASON_BYTES].decode("utf-8", "replace") @@ -301,6 +335,50 @@ def _storage_device_class(device: str | None) -> str | None: return "block-device" +def _path_storage_metadata(path: Path) -> dict[str, str | None]: + """Describe the mount holding a qualification path without exposing it.""" + resolved = path.resolve() + matches: list[tuple[int, str, str]] = [] + mounts = _read_text(Path("/proc/mounts")) or "" + for line in mounts.splitlines(): + fields = line.split() + if len(fields) < 3: + continue + mountpoint = Path(fields[1].replace("\\040", " ")) + try: + resolved.relative_to(mountpoint) + except ValueError: + continue + matches.append((len(str(mountpoint)), fields[0], fields[2])) + if not matches: + return { + "filesystem": None, + "storage_medium_class": None, + "device_class": None, + } + _, source, filesystem = max(matches, key=lambda item: item[0]) + device = _root_block_device(source) + rotational = ( + _read_text(Path(f"/sys/class/block/{device}/queue/rotational")) + if device + else None + ) + medium = ( + "rotational" + if rotational == "1" + else "non-rotational" + if rotational == "0" + else "virtual" + if filesystem in {"tmpfs", "ramfs"} + else "unavailable" + ) + return { + "filesystem": filesystem[:64], + "storage_medium_class": medium, + "device_class": _storage_device_class(device), + } + + def board_metadata() -> tuple[dict[str, Any] | None, str | None]: """Return board facts, or why physical SBC evidence is unavailable.""" if platform.system().lower() != "linux": @@ -672,6 +750,39 @@ def _timed_http( ) +def _safe_http_error_class(body: bytes) -> str: + """Classify a client-visible error without retaining its text.""" + lowered = body[:MAX_HTTP_BODY_BYTES].decode("utf-8", errors="ignore").lower() + if "not enforceable" in lowered or "cannot be proven" in lowered: + return "budget-not-enforceable" + if "budget limit reached" in lowered: + return "budget-exceeded" + if "budget" in lowered or "limit reached" in lowered: + return "budget-other" + if "admission" in lowered or "backpressure" in lowered or "capacity" in lowered: + return "admission" + if "model" in lowered and ("forbidden" in lowered or "not allowed" in lowered): + return "model-policy" + if "authentication" in lowered or "unauthorized" in lowered: + return "authentication" + return "unclassified" + + +def _qualification_budget_rule_counts(database: Path) -> str: + """Return only rule-row counts from the disposable qualification database.""" + try: + with sqlite3.connect(f"file:{database}?mode=ro", uri=True, timeout=1) as db: + users = int( + db.execute("SELECT count(*) FROM user_budget_rules").fetchone()[0] + ) + instance = int( + db.execute("SELECT count(*) FROM instance_budget_rules").fetchone()[0] + ) + except (OSError, sqlite3.Error, TypeError, ValueError): + return "rules=unavailable" + return f"user_rules={users},instance_rules={instance}" + + def _run( command_id: str, argv: Sequence[str], @@ -1174,7 +1285,7 @@ def _percentile(values: Sequence[int], percentile: int) -> int | None: def _timing_summary( - elapsed_ms: Sequence[int], ttft_ms: Sequence[int] + elapsed_ms: Sequence[int], ttft_ms: Sequence[int], *, include_p99: bool = False ) -> dict[str, Any]: """Return aggregate timing scalars and discard per-request observations.""" summary: dict[str, Any] = { @@ -1191,6 +1302,10 @@ def _timing_summary( "p95_ttft_ms": _percentile(ttft_ms, 95), } ) + if include_p99 and len(elapsed_ms) >= P99_MIN_SAMPLES: + summary["p99_elapsed_ms"] = _percentile(elapsed_ms, 99) + if include_p99 and len(ttft_ms) >= P99_MIN_SAMPLES: + summary["p99_ttft_ms"] = _percentile(ttft_ms, 99) return summary @@ -2201,8 +2316,9 @@ def _benchmark_request( model: str, streaming: bool, client_surface: str = "responses", + api_token: str = "q008-server-key", ) -> tuple[int, bytes, int, int | None]: - return _request_timed(port, client_surface, model, streaming) + return _request_timed(port, client_surface, model, streaming, api_token) def _sequential_benchmark( @@ -2216,6 +2332,7 @@ def _sequential_benchmark( model: str | None = None, streaming: bool | None = None, terminal_marker: bytes | None = None, + api_token: str = "q008-server-key", ) -> tuple[dict[str, Any], bool]: """Run warm-ups and a sequential batch, retaining only aggregate scalars. @@ -2240,7 +2357,7 @@ def _sequential_benchmark( ) for _ in range(BENCHMARK_WARMUPS): status, body, _, _ = _benchmark_request( - port, active_model, active_streaming, case.client_surface + port, active_model, active_streaming, case.client_surface, api_token ) if ( status != 200 @@ -2261,13 +2378,13 @@ def _sequential_benchmark( }, False, ) - first_cpu = _proc_cpu(process.pid) started = time.monotonic() + first_cpu: tuple[int, int] | None = None elapsed_ms: list[int] = [] ttft_ms: list[int] = [] for _ in range(samples): status, body, elapsed, ttft = _benchmark_request( - port, active_model, active_streaming, case.client_surface + port, active_model, active_streaming, case.client_surface, api_token ) elapsed_ms.append(elapsed) if ttft is not None: @@ -2320,35 +2437,354 @@ def _sequential_benchmark( def _concurrent_finite_benchmark( - process: subprocess.Popen[str], port: int + process: subprocess.Popen[str], + port: int, + api_token: str = "q008-server-key", + *, + client_concurrency: int = CONCURRENCY_WORKERS, + request_count: int = CONCURRENCY_BATCH_SIZE, + competing_io_dir: Path | None = None, + competing_io_mib_per_second: int = 0, + competing_io_max_mib: int = 0, + api_tokens: Sequence[str] = (), ) -> dict[str, Any]: - """Run the fixed contention observation at client concurrency four.""" + """Run a bounded finite batch with optional same-filesystem I/O load.""" + if not 1 <= client_concurrency <= CONCURRENCY_WORKERS_MAX: + raise ValueError("concurrent benchmark workers must be between 1 and 8") + if not client_concurrency <= request_count <= CONCURRENCY_REQUESTS_MAX: + raise ValueError( + "concurrent benchmark requests must be between worker count and 4000" + ) + if (competing_io_mib_per_second == 0) != (competing_io_max_mib == 0): + raise ValueError("competing I/O rate and volume must both be zero or nonzero") + if not 0 <= competing_io_mib_per_second <= COMPETING_IO_MAX_MIB_PER_SECOND: + raise ValueError("competing I/O rate must be between 0 and 4 MiB/s") + if not 0 <= competing_io_max_mib <= COMPETING_IO_MAX_MIB: + raise ValueError("competing I/O volume must be between 0 and 128 MiB") + if competing_io_max_mib and competing_io_dir is None: + raise ValueError("competing I/O requires a qualification temporary root") + + io_stop = threading.Event() + io_ready = threading.Event() + io_result: dict[str, Any] = {"bytes_written": 0, "elapsed_ms": 0} + io_thread: threading.Thread | None = None + if competing_io_max_mib: + assert competing_io_dir is not None + io_path = competing_io_dir / "qualification-competing-io.tmp" + + def write_competing_io() -> None: + io_started = time.monotonic() + written = 0 + try: + with io_path.open("wb") as output: + io_ready.set() + limit = competing_io_max_mib * 1024 * 1024 + while not io_stop.is_set() and written < limit: + chunk_size = min(64 * 1024, limit - written) + output.write(b"\0" * chunk_size) + written += chunk_size + if written % (1024 * 1024) == 0: + output.flush() + os.fsync(output.fileno()) + target_elapsed = written / ( + competing_io_mib_per_second * 1024 * 1024 + ) + remaining = target_elapsed - (time.monotonic() - io_started) + if remaining > 0: + io_stop.wait(remaining) + output.flush() + os.fsync(output.fileno()) + except OSError as error: + io_result["error"] = type(error).__name__ + finally: + io_result["bytes_written"] = written + io_result["elapsed_ms"] = round((time.monotonic() - io_started) * 1000) + + io_thread = threading.Thread( + target=write_competing_io, name="qualification-competing-io", daemon=True + ) + io_thread.start() + if not io_ready.wait(timeout=5): + io_stop.set() + io_thread.join(timeout=5) + raise QualificationError("competing I/O worker did not become ready") + first_cpu = _proc_cpu(process.pid) started = time.monotonic() - def request_once(_item: int) -> tuple[int, bytes, int, int | None]: - return _benchmark_request(port, MODELS["responses"], False) + active_tokens = tuple(api_tokens) or (api_token,) - with ThreadPoolExecutor(max_workers=CONCURRENCY_WORKERS) as executor: - results = list(executor.map(request_once, range(CONCURRENCY_BATCH_SIZE))) + def request_once(item: int) -> tuple[int, bytes, int, int | None]: + return _benchmark_request( + port, + MODELS["responses"], + False, + api_token=active_tokens[item % len(active_tokens)], + ) + + try: + first_cpu = _proc_cpu(process.pid) + with ThreadPoolExecutor(max_workers=client_concurrency) as executor: + results = list(executor.map(request_once, range(request_count))) + finally: + io_stop.set() + if io_thread is not None: + io_thread.join(timeout=120) + if io_thread.is_alive(): + raise QualificationError("competing I/O worker did not stop") + if "error" in io_result: + raise QualificationError("competing I/O worker failed") elapsed_seconds = max(time.monotonic() - started, 0.001) second_cpu = _proc_cpu(process.pid) elapsed_ms = [result[2] for result in results] ttft_ms = [result[3] for result in results if result[3] is not None] completed = sum(1 for status, body, _, _ in results if status == 200 and body) + status_counts: dict[str, int] = {} + for status, _, _, _ in results: + key = str(status) + status_counts[key] = status_counts.get(key, 0) + 1 return { - "status": "measured" if completed == CONCURRENCY_BATCH_SIZE else "fail", - "request_count": CONCURRENCY_BATCH_SIZE, - "client_concurrency": CONCURRENCY_WORKERS, + "status": "measured" if completed == request_count else "fail", + "request_count": request_count, + "client_concurrency": client_concurrency, "completed_count": completed, - "failed_count": CONCURRENCY_BATCH_SIZE - completed, + "failed_count": request_count - completed, + "http_status_counts": status_counts, "batch_elapsed_ms": round(elapsed_seconds * 1000), - "requests_per_second": round(CONCURRENCY_BATCH_SIZE / elapsed_seconds, 3), + "requests_per_second": round(request_count / elapsed_seconds, 3), "cpu": _cpu_batch_summary(first_cpu, second_cpu, elapsed_seconds), - "timings": _timing_summary(elapsed_ms, ttft_ms), + "timings": _timing_summary(elapsed_ms, ttft_ms, include_p99=True), + "competing_io": { + "enabled": bool(competing_io_max_mib), + "configured_mib_per_second": competing_io_mib_per_second, + "maximum_mib": competing_io_max_mib, + "bytes_written": io_result["bytes_written"], + "elapsed_ms": io_result["elapsed_ms"], + "storage": ( + _path_storage_metadata(competing_io_dir) + if competing_io_dir is not None + else None + ), + }, } +def _command_token( + binary: Path, config: Path, env: Mapping[str, str], username: str, timeout: float +) -> str: + """Issue one synthetic personal token without retaining it in reports.""" + try: + result = subprocess.run( + [ + str(binary), + "--config", + str(config), + "users", + "token", + username, + "--label", + "SBC qualification", + "--show-secret", + ], + cwd=ROOT, + env=dict(env), + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as error: + raise QualificationError( + "synthetic personal token could not be issued" + ) from error + match = re.search(r"epk1_[A-Za-z0-9_-]+", result.stdout) + if result.returncode != 0 or match is None: + raise QualificationError("synthetic personal token issuance failed") + return match.group(0) + + +def _configure_benchmark_principals( + *, + binary: Path, + config: Path, + env: Mapping[str, str], + port: int, + admin_password: str, + admin_token: str, + principal_count: int, + budget_mode: str, + request_count: int, + timeout: float, +) -> list[str]: + """Create disposable managed principals and optional hard request budgets.""" + if not 1 <= principal_count <= 49: + raise ValueError("benchmark principal count must be between 1 and 49") + if budget_mode not in {"none", "user", "global", "user-global"}: + raise ValueError("invalid benchmark budget mode") + tokens = [admin_token] + if principal_count > 1: + try: + profile_result = subprocess.run( + [ + str(binary), + "--config", + str(config), + "users", + "policy", + "create", + "sbc-qualification", + "SBC Qualification", + "allow", + "--actor", + "sbc-admin", + ], + cwd=ROOT, + env=dict(env), + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as error: + raise QualificationError("synthetic model profile setup failed") from error + if profile_result.returncode != 0: + raise QualificationError("synthetic model profile setup failed") + jar = http.cookiejar.CookieJar() + opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(jar)) + origin = f"http://127.0.0.1:{port}" + try: + opener.open(f"{origin}/login", timeout=timeout).read() + login = urllib.request.Request( + f"{origin}/login", + data=urllib.parse.urlencode( + {"username": "sbc-admin", "password": admin_password} + ).encode(), + headers={ + "Content-Type": "application/x-www-form-urlencoded", + "Origin": origin, + "Referer": f"{origin}/login", + }, + method="POST", + ) + response = opener.open(login, timeout=timeout) + response.read() + if not any(cookie.name == "eggpool_session" for cookie in jar): + raise QualificationError("synthetic admin session was not issued") + for index in range(1, principal_count): + users_page = opener.open(f"{origin}/users", timeout=timeout).read() + html = users_page.decode("utf-8", errors="replace") + form = re.search( + r']+action="/admin/users/create"[^>]*>(.*?)', + html, + re.DOTALL, + ) + csrf = ( + re.search(r'name="csrf" value="([^"]+)"', form.group(1)) + if form is not None + else None + ) + if csrf is None: + raise QualificationError("synthetic user form was unavailable") + username = f"sbc-member-{index:02d}" + member_password = f"SBC qualification member passphrase {index:02d}" + create = urllib.request.Request( + f"{origin}/admin/users/create", + data=urllib.parse.urlencode( + { + "username": username, + "password": member_password, + "profile_id": "sbc-qualification", + "csrf": csrf.group(1), + } + ).encode(), + headers={ + "Content-Type": "application/x-www-form-urlencoded", + "Origin": origin, + "Referer": f"{origin}/users", + }, + method="POST", + ) + created = opener.open(create, timeout=timeout) + created.read() + tokens.append(_command_token(binary, config, env, username, timeout)) + except (OSError, urllib.error.URLError) as error: + raise QualificationError( + "synthetic managed principals could not be set up" + ) from error + + if budget_mode in {"user", "user-global"}: + # Keep the hard ceiling above the measured workload and Q008's bounded + # post-batch restart/recovery checks so those checks do not invalidate + # an otherwise complete budget-mode run. + max_requests = str(request_count + 100) + for username in ["sbc-admin"] + [ + f"sbc-member-{index:02d}" for index in range(1, principal_count) + ]: + try: + result = subprocess.run( + [ + str(binary), + "--config", + str(config), + "users", + "budget", + "set", + username, + "--window", + "day", + "--max-requests", + max_requests, + "--enforcement", + "hard", + "--actor", + "sbc-admin", + ], + cwd=ROOT, + env=dict(env), + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as error: + raise QualificationError( + "synthetic user budget setup failed" + ) from error + if result.returncode != 0: + raise QualificationError("synthetic user budget setup failed") + if budget_mode in {"global", "user-global"}: + try: + result = subprocess.run( + [ + str(binary), + "--config", + str(config), + "users", + "global-budget", + "set", + "--window", + "day", + "--max-requests", + str(request_count + 100), + "--enforcement", + "hard", + "--actor", + "sbc-admin", + ], + cwd=ROOT, + env=dict(env), + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as error: + raise QualificationError("synthetic global budget setup failed") from error + if result.returncode != 0: + raise QualificationError("synthetic global budget setup failed") + return tokens + + def _benchmark_sample_count(value: str) -> int: try: count = int(value) @@ -2492,7 +2928,11 @@ def _persist_journal_read_only_environment( def _request_timed( - port: int, surface: str, model: str, streaming: bool + port: int, + surface: str, + model: str, + streaming: bool, + api_token: str = "q008-server-key", ) -> tuple[int, bytes, int, int | None]: paths = { "chat_completions": "/v1/chat/completions", @@ -2516,7 +2956,7 @@ def _request_timed( method="POST", body=json.dumps(payload).encode(), headers={ - "Authorization": "Bearer q008-server-key", + "Authorization": f"Bearer {api_token}", "Content-Type": "application/json", }, ) @@ -2544,12 +2984,40 @@ def run_qualification( diagnose_persist_journal_steady_state: bool = False, diagnose_worker_io_attribution: bool = False, diagnostic_database_dir: Path | None = None, + concurrency_workers: int = CONCURRENCY_WORKERS, + concurrency_requests: int = CONCURRENCY_BATCH_SIZE, + competing_io_mib_per_second: int = 0, + competing_io_max_mib: int = 0, + benchmark_principals: int = 1, + benchmark_budget_mode: str = "none", ) -> dict[str, Any]: """Run SBC qualification and return a bounded machine-readable report.""" if benchmark_samples < 0 or benchmark_samples > BENCHMARK_MAX_SAMPLES: raise ValueError( f"benchmark sample count must be 0 or 1..{BENCHMARK_MAX_SAMPLES}" ) + if not 1 <= concurrency_workers <= CONCURRENCY_WORKERS_MAX: + raise ValueError("concurrency workers must be between 1 and 8") + if not concurrency_workers <= concurrency_requests <= CONCURRENCY_REQUESTS_MAX: + raise ValueError("concurrency requests must be between workers and 4000") + if (competing_io_mib_per_second == 0) != (competing_io_max_mib == 0): + raise ValueError("competing I/O rate and volume must both be zero or nonzero") + if not 0 <= competing_io_mib_per_second <= COMPETING_IO_MAX_MIB_PER_SECOND: + raise ValueError("competing I/O rate must be between 0 and 4 MiB/s") + if not 0 <= competing_io_max_mib <= COMPETING_IO_MAX_MIB: + raise ValueError("competing I/O volume must be between 0 and 128 MiB") + if not 1 <= benchmark_principals <= 49: + raise ValueError("benchmark principals must be between 1 and 49") + if benchmark_budget_mode not in {"none", "user", "global", "user-global"}: + raise ValueError("invalid benchmark budget mode") + if benchmark_samples == 0 and ( + concurrency_workers != CONCURRENCY_WORKERS + or concurrency_requests != CONCURRENCY_BATCH_SIZE + or competing_io_max_mib + or benchmark_principals != 1 + or benchmark_budget_mode != "none" + ): + raise ValueError("extended concurrency options require benchmark mode") if diagnose_finite_tail is not None and not ( DIAGNOSTIC_MIN_SAMPLES <= diagnose_finite_tail <= DIAGNOSTIC_MAX_SAMPLES ): @@ -2811,6 +3279,59 @@ def run_qualification( commands.append(result) if result.status != "pass": raise QualificationError(f"{command_id}: {result.reason}") + # On-prem identity (M001/M002) retired the shared server key for + # inference and catalog: fresh databases are `fresh_uninitialized`, + # so the runner bootstraps one synthetic administrator and uses + # the issued personal token for managed inference. The password + # and token never enter the report; only a redacted pass marker + # is recorded. Mirrors `qualification_live_provider.py` Q007. + bootstrap_started = time.monotonic() + try: + bootstrap_proc = subprocess.run( + [ + str(binary), + "--config", + str(config), + "users", + "bootstrap", + "sbc-admin", + "--password-stdin", + "--show-secret", + ], + cwd=ROOT, + env=dict(env), + input=f"{SYNTHETIC_ADMIN_PASSWORD}\n", + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as error: + raise QualificationError( + "synthetic administrator bootstrap could not start" + ) from error + bootstrap_ms = round((time.monotonic() - bootstrap_started) * 1000) + if bootstrap_proc.returncode != 0: + raise QualificationError( + "synthetic administrator bootstrap failed: " + + bounded(bootstrap_proc.stderr or bootstrap_proc.stdout) + ) + token_match = re.search(r"epk1_[A-Za-z0-9_-]+", bootstrap_proc.stdout) + if token_match is None: + raise QualificationError( + "synthetic administrator bootstrap returned no personal token" + ) + agent_token = token_match.group(0) + commands.append( + CommandResult( + "bootstrap-admin", + (str(binary), "--config", str(config), "users", "bootstrap"), + 0, + False, + bootstrap_ms, + "synthetic administrator bootstrap complete", + ) + ) server_env = _dedicated_checkpointer_server_environment( env, qualification_dedicated_checkpointer_mode ) @@ -2836,10 +3357,16 @@ def run_qualification( ("startup-readyz", "/v1/readyz"), ("model-listing", "/v1/models"), ): - status, body = _http( - f"http://127.0.0.1:{port}{path}", - headers={"Authorization": "Bearer q008-server-key"}, - ) + if step_id == "model-listing": + status, body = _http( + f"http://127.0.0.1:{port}{path}", + headers={"Authorization": f"Bearer {agent_token}"}, + ) + else: + status, body = _http( + f"http://127.0.0.1:{port}{path}", + headers={"Authorization": "Bearer q008-server-key"}, + ) passed = status == 200 and ( step_id != "model-listing" or all(model.encode() in body for model in MODELS.values()) @@ -2875,7 +3402,7 @@ def run_qualification( ) for surface, model in MODELS.items(): status, body, elapsed_ms, ttft_ms = _request_timed( - port, surface, model, False + port, surface, model, False, agent_token ) passed = status == 200 and len(body) > 0 report["functional"].append( @@ -2897,7 +3424,7 @@ def run_qualification( ) for surface, model in MODELS.items(): status, body, elapsed_ms, ttft_ms = _request_timed( - port, surface, model, True + port, surface, model, True, agent_token ) markers = { "chat_completions": b"[DONE]", @@ -2933,6 +3460,30 @@ def run_qualification( result = _command(command_id, binary, config, env, args, timeout) commands.append(result) if result.status != "pass": + # Managed mode (M001/M002) fences `/api/stats/runtime` + # behind a browser session: the server-key CLI is + # rejected with 401. That rejection is the expected + # managed fencing proof, not a qualification failure. + # Procfs/DB/WAL metrics are already captured via + # `resource_sample` (with `runtime_read_error` set); + # the full runtime projection remains available via + # browser session (M011 evidence). + if command_id == "runtime-status" and ( + "401" in result.reason + or "authentication failed" in result.reason.lower() + or "unauthorized" in result.reason.lower() + ): + report["functional"].append( + { + "id": command_id, + "status": "pass", + "note": ( + "managed-fencing: server-key rejected, " + "browser session required" + ), + } + ) + continue raise QualificationError(f"{command_id}: {result.reason}") report["functional"].append({"id": command_id, "status": "pass"}) for path in ( @@ -2947,6 +3498,24 @@ def run_qualification( f"http://127.0.0.1:{port}{path}", headers={"Authorization": "Bearer q008-server-key"}, ) + # Managed mode fences the dashboard shell (`/`) behind a + # browser session (M002/M011); a 401 without a session is + # the expected fencing proof. Static assets stay public + # and must still pass. + if path == "/" and status == 401: + report["functional"].append( + { + "id": "dashboard-page", + "status": "pass", + "http_status": status, + "body_bytes": len(body), + "note": ( + "managed-fencing: dashboard shell " + "requires browser session" + ), + } + ) + continue passed = status == 200 and len(body) > 0 dashboard_id = path.strip("/").replace("/", "-") or "page" report["functional"].append( @@ -2995,7 +3564,7 @@ def run_qualification( benchmark["paired_binary_sha256"] = candidate_hash benchmark["soft_threshold_frames"] = M007_SOFT_WAL_FRAMES concurrency = _concurrent_finite_benchmark( - benchmark_process, port + benchmark_process, port, agent_token ) if concurrency["status"] != "measured": raise QualificationError( @@ -3016,7 +3585,9 @@ def run_qualification( diagnose_worker_io_attribution ) if diagnose_persist_journal_steady_state: - concurrency = _concurrent_finite_benchmark(process, port) + concurrency = _concurrent_finite_benchmark( + process, port, agent_token + ) if concurrency["status"] != "measured": raise QualificationError( "M008 concurrency-4 workload did not complete" @@ -3695,8 +4266,9 @@ def run_qualification( benchmark_process = process benchmark: dict[str, Any] = { "sample_count": benchmark_samples, - "concurrency_batch_size": CONCURRENCY_BATCH_SIZE, - "client_concurrency": CONCURRENCY_WORKERS, + "concurrency_batch_size": concurrency_requests, + "client_concurrency": concurrency_workers, + "database_storage": _path_storage_metadata(database), "config_fixture": Path(config_fixture).name, "cadence": benchmark_cadence_facts(content), "runs": {}, @@ -3725,6 +4297,7 @@ def benchmark_sample( NATIVE_FINITE_CASE, benchmark_samples, provider, + api_token=agent_token, ) if not finite_measured: raise QualificationError( @@ -3738,6 +4311,7 @@ def benchmark_sample( NATIVE_STREAMING_CASE, benchmark_samples, provider, + api_token=agent_token, ) if not streaming_measured: raise QualificationError( @@ -3751,17 +4325,38 @@ def benchmark_sample( TRANSLATED_STREAMING_CASE, benchmark_samples, provider, + api_token=agent_token, ) benchmark["runs"]["translated_responses_to_messages_streaming"] = ( translated ) if translated_measured: benchmark_sample("after-translated-streaming") - concurrent = _concurrent_finite_benchmark(benchmark_process, port) - if concurrent["status"] != "measured": - raise QualificationError( - "concurrency-4 benchmark did not complete all requests" - ) + principal_tokens = _configure_benchmark_principals( + binary=binary, + config=config, + env=env, + port=port, + admin_password=SYNTHETIC_ADMIN_PASSWORD, + admin_token=agent_token, + principal_count=benchmark_principals, + budget_mode=benchmark_budget_mode, + request_count=concurrency_requests, + timeout=timeout, + ) + benchmark["principal_count"] = benchmark_principals + benchmark["budget_mode"] = benchmark_budget_mode + concurrent = _concurrent_finite_benchmark( + benchmark_process, + port, + agent_token, + client_concurrency=concurrency_workers, + request_count=concurrency_requests, + competing_io_dir=root, + competing_io_mib_per_second=competing_io_mib_per_second, + competing_io_max_mib=competing_io_max_mib, + api_tokens=principal_tokens, + ) benchmark["runs"]["concurrent_native_responses_finite"] = concurrent if diagnose_finite_tail is not None: benchmark_sample("before-finite-tail-diagnostic") @@ -3943,11 +4538,13 @@ def benchmark_sample( ) for surface, model in list(MODELS.items())[:2]: status, body, elapsed_ms, ttft_ms = _request_timed( - port, surface, model, False + port, surface, model, False, agent_token ) if status != 200 or not body: raise QualificationError( - f"second workload {surface} failed with HTTP {status}" + f"second workload {surface} failed with HTTP {status} " + f"({_safe_http_error_class(body)}; " + f"{_qualification_budget_rule_counts(database)})" ) workload_timings["second_workload_elapsed_ms"].append(elapsed_ms) if ttft_ms is not None: @@ -3985,6 +4582,17 @@ def benchmark_sample( ) report["durable"] = counts report["status"] = "pass" + if benchmark_samples: + benchmark_summary = cast( + "Mapping[str, Any]", report.get("benchmark", {}) + ) + runs = cast("Mapping[str, Any]", benchmark_summary.get("runs", {})) + concurrent = cast( + "Mapping[str, Any]", + runs.get("concurrent_native_responses_finite", {}), + ) + if concurrent.get("status") != "measured": + report["status"] = "fail" report["python_reference"] = { "status": "not-run", "reason": "optional same-board comparison was not supplied", @@ -4112,6 +4720,50 @@ def _parser() -> argparse.ArgumentParser: metavar="N", help="Run the optional physical-SBC benchmark with 1..100 samples.", ) + parser.add_argument( + "--concurrency-workers", + type=int, + default=CONCURRENCY_WORKERS, + metavar="N", + help="Concurrent finite-request workers, 1..8 (benchmark mode only).", + ) + parser.add_argument( + "--concurrency-requests", + type=int, + default=CONCURRENCY_BATCH_SIZE, + metavar="N", + help=( + "Concurrent finite requests, workers..4000 (benchmark mode only). " + "p99 is reported only at 1000 or more completed observations." + ), + ) + parser.add_argument( + "--competing-io-mib-per-second", + type=int, + default=0, + metavar="N", + help="Rate cap for same-filesystem competing writes, 1..4 MiB/s.", + ) + parser.add_argument( + "--competing-io-max-mib", + type=int, + default=0, + metavar="N", + help="Total competing-write cap, 1..128 MiB; requires a nonzero rate.", + ) + parser.add_argument( + "--benchmark-principals", + type=int, + default=1, + metavar="N", + help="Synthetic managed accounts used by the concurrent batch, 1..49.", + ) + parser.add_argument( + "--benchmark-budget-mode", + choices=("none", "user", "global", "user-global"), + default="none", + help="Apply no budget, hard per-user, hard global, or both request budgets.", + ) parser.add_argument( "--diagnose-finite-tail", type=_diagnose_sample_count, @@ -4262,6 +4914,12 @@ def main(argv: Sequence[str] | None = None) -> int: diagnose_persist_journal_steady_state=args.diagnose_persist_journal_steady_state, diagnose_worker_io_attribution=args.diagnose_worker_io_attribution, diagnostic_database_dir=args.diagnostic_database_dir, + concurrency_workers=args.concurrency_workers, + concurrency_requests=args.concurrency_requests, + competing_io_mib_per_second=args.competing_io_mib_per_second, + competing_io_max_mib=args.competing_io_max_mib, + benchmark_principals=args.benchmark_principals, + benchmark_budget_mode=args.benchmark_budget_mode, ) except (OSError, QualificationError, ValueError, sqlite3.Error) as error: requested_benchmark = ( diff --git a/tests/fixtures/cli/contract-matrix.json b/tests/fixtures/cli/contract-matrix.json index 42ecd3fc9..ef53455ad 100644 --- a/tests/fixtures/cli/contract-matrix.json +++ b/tests/fixtures/cli/contract-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "plan": "O001", - "command_count": 65, + "command_count": 72, "commands": [ {"path": "accounts", "owner": "O007", "options": []}, {"path": "accounts explain", "owner": "O007", "options": ["--model", "--provider", "--protocol", "--scores", "--gates"]}, @@ -51,7 +51,7 @@ {"path": "models", "owner": "O007", "options": []}, {"path": "models refresh", "owner": "O007", "options": []}, {"path": "newkey", "owner": "O004", "options": ["--show-old", "--show-secrets"]}, - {"path": "onboard", "owner": "O004", "options": ["--providers"]}, + {"path": "onboard", "owner": "O004", "options": ["--providers", "--admin-username", "--password-stdin", "--show-secret"]}, {"path": "recover", "owner": "O006", "options": []}, {"path": "rehash", "owner": "O003", "options": ["--json"]}, {"path": "restart", "owner": "O003", "options": ["--timeout"]}, @@ -67,6 +67,13 @@ {"path": "stop", "owner": "O003", "options": ["--timeout"]}, {"path": "uninstall", "owner": "O009", "options": ["--yes", "--keep-data", "--keep-config", "--keep-path", "--deploy-artifacts"]}, {"path": "update", "owner": "O008", "options": ["--check", "--from-source"]}, + {"path": "users", "owner": "O004", "options": []}, + {"path": "users enable-managed", "owner": "M002", "options": []}, + {"path": "users bootstrap", "owner": "O004", "options": ["--password-stdin", "--show-secret"]}, + {"path": "users list", "owner": "O004", "options": []}, + {"path": "users reset-password", "owner": "O004", "options": ["--password-stdin", "--show-secret"]}, + {"path": "users revoke", "owner": "O004", "options": []}, + {"path": "users token", "owner": "O004", "options": ["--label", "--show-secret"]}, {"path": "version", "owner": "O008", "options": []} ], "global_options": ["--config"], diff --git a/tests/fixtures/dashboard-on-prem-browser/reference.json b/tests/fixtures/dashboard-on-prem-browser/reference.json new file mode 100644 index 000000000..c1484ffd2 --- /dev/null +++ b/tests/fixtures/dashboard-on-prem-browser/reference.json @@ -0,0 +1,209 @@ +{ + "observations": { + "admin.after_member_created": "\"/users\"", + "admin.budget_labels": "[\"hard\",\"Hard\",\"Estimated\"]", + "admin.budget_saved": "\"/users\"", + "admin.member_created": "\"/users\"", + "admin.member_reactivated": "\"/admin/users/state\"", + "admin.member_row": "[{\"must_change\":false,\"role\":\"admin\",\"state\":\"active\",\"username\":\"admin\"},{\"must_change\":true,\"role\":\"member\",\"state\":\"active\",\"username\":\"member\"}]", + "admin.member_suspended": "\"/admin/users/state\"", + "admin.no_js_login": "{\"scripts_required\":false,\"sign_in\":\"/account\"}", + "admin.override_saved": "\"/users\"", + "admin.profile_conflict": "\"/admin/users/profile\"", + "admin.profile_created": "\"/profiles\"", + "admin.sign_in": "\"/account\"", + "admin.token_absent_on_return": "true", + "admin.token_reveal": "\"/account/tokens\"", + "admin.token_reveal_is_fresh_each_time": "true", + "admin.token_revealed_once": "true", + "admin.token_second_reveal": "\"/account/tokens\"", + "after_logout./admin/audit": "\"/admin/audit\"", + "after_logout./latency": "\"/latency\"", + "after_logout./models": "\"/models\"", + "after_logout./profiles": "\"/profiles\"", + "after_logout./users": "\"/users\"", + "agent_token./my-account": "401", + "agent_token./users": "401", + "anonymous./admin/audit": "\"/admin/audit\"", + "anonymous./latency": "\"/latency\"", + "anonymous./models": "\"/models\"", + "anonymous./my-account": "\"/my-account\"", + "anonymous./profiles": "\"/profiles\"", + "anonymous./usage": "\"/usage\"", + "anonymous./users": "\"/users\"", + "member.budget_labelling": "[\"estimated\",\"hard\",\"Estimated\"]", + "member.forced_password_landing": "\"/account?password=required\"", + "member.password_changed": "\"/login\"", + "member.reauthenticated_after_password_change": "\"/account\"", + "member_denied./admin/audit": "\"/admin/audit\"", + "member_denied./admin/audit.message": "\"Administrator access required.\"", + "member_denied./latency": "\"/latency\"", + "member_denied./latency.message": "\"Administrator access required.\"", + "member_denied./models": "\"/models\"", + "member_denied./models.message": "\"Administrator access required.\"", + "member_denied./profiles": "\"/profiles\"", + "member_denied./profiles.message": "\"Administrator access required.\"", + "member_denied./users": "\"/users\"", + "member_denied./users.message": "\"Administrator access required.\"", + "page.admin./admin/audit.desktop": "{\"duplicateIds\":[],\"forms\":[],\"h1\":[\"Administrative audit\"],\"h2\":[],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/admin/audit\",\"secretMarkers\":[],\"tables\":[],\"title\":\"Admin audit\",\"viewportMeta\":true}", + "page.admin./admin/audit.desktop-hidpi": "{\"duplicateIds\":[],\"forms\":[],\"h1\":[\"Administrative audit\"],\"h2\":[],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/admin/audit\",\"secretMarkers\":[],\"tables\":[],\"title\":\"Admin audit\",\"viewportMeta\":true}", + "page.admin./admin/audit.mobile": "{\"duplicateIds\":[],\"forms\":[],\"h1\":[\"Administrative audit\"],\"h2\":[],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/admin/audit\",\"secretMarkers\":[],\"tables\":[{\"width\":918,\"wrapped\":true}],\"title\":\"Admin audit\",\"viewportMeta\":true}", + "page.admin./profiles.desktop": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/profiles\",\"fields\":[\"period\",\"theme\"],\"method\":\"GET\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/create\",\"fields\":[\"csrf\",\"default_effect\",\"name\",\"profile_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/update\",\"fields\":[\"csrf\",\"default_effect\",\"name\",\"profile_id\",\"version\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/rule\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"profile_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/prefer\",\"fields\":[\"csrf\",\"model_id\",\"profile_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/delete\",\"fields\":[\"csrf\",\"profile_id\",\"version\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"EggPool\"],\"h2\":[\"Profiles\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/profiles\",\"secretMarkers\":[],\"tables\":[],\"title\":\"Profiles\",\"viewportMeta\":true}", + "page.admin./profiles.desktop-hidpi": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/profiles\",\"fields\":[\"period\",\"theme\"],\"method\":\"GET\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/create\",\"fields\":[\"csrf\",\"default_effect\",\"name\",\"profile_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/update\",\"fields\":[\"csrf\",\"default_effect\",\"name\",\"profile_id\",\"version\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/rule\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"profile_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/prefer\",\"fields\":[\"csrf\",\"model_id\",\"profile_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/delete\",\"fields\":[\"csrf\",\"profile_id\",\"version\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"EggPool\"],\"h2\":[\"Profiles\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/profiles\",\"secretMarkers\":[],\"tables\":[],\"title\":\"Profiles\",\"viewportMeta\":true}", + "page.admin./profiles.mobile": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/profiles\",\"fields\":[\"period\",\"theme\"],\"method\":\"GET\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/create\",\"fields\":[\"csrf\",\"default_effect\",\"name\",\"profile_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/update\",\"fields\":[\"csrf\",\"default_effect\",\"name\",\"profile_id\",\"version\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/rule\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"profile_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/prefer\",\"fields\":[\"csrf\",\"model_id\",\"profile_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/profiles/delete\",\"fields\":[\"csrf\",\"profile_id\",\"version\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"EggPool\"],\"h2\":[\"Profiles\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/profiles\",\"secretMarkers\":[],\"tables\":[],\"title\":\"Profiles\",\"viewportMeta\":true}", + "page.admin./users.desktop": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/users\",\"fields\":[\"period\",\"theme\"],\"method\":\"GET\",\"unlabelled\":[]},{\"action\":\"/admin/budget/instance\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/create\",\"fields\":[\"csrf\",\"password\",\"profile_id\",\"username\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"EggPool\"],\"h2\":[\"Instance budget ceilings\",\"Users\",\"Local accounts\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/users\",\"secretMarkers\":[],\"tables\":[],\"title\":\"Users\",\"viewportMeta\":true}", + "page.admin./users.desktop-hidpi": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/users\",\"fields\":[\"period\",\"theme\"],\"method\":\"GET\",\"unlabelled\":[]},{\"action\":\"/admin/budget/instance\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/create\",\"fields\":[\"csrf\",\"password\",\"profile_id\",\"username\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"EggPool\"],\"h2\":[\"Instance budget ceilings\",\"Users\",\"Local accounts\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/users\",\"secretMarkers\":[],\"tables\":[],\"title\":\"Users\",\"viewportMeta\":true}", + "page.admin./users.mobile": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/users\",\"fields\":[\"period\",\"theme\"],\"method\":\"GET\",\"unlabelled\":[]},{\"action\":\"/admin/budget/instance\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/create\",\"fields\":[\"csrf\",\"password\",\"profile_id\",\"username\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"EggPool\"],\"h2\":[\"Instance budget ceilings\",\"Users\",\"Local accounts\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/users\",\"secretMarkers\":[],\"tables\":[{\"width\":593,\"wrapped\":true},{\"width\":445,\"wrapped\":true}],\"title\":\"Users\",\"viewportMeta\":true}", + "page.admin./users.mobile.Catppuccin Latte": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/users\",\"fields\":[\"period\",\"theme\"],\"method\":\"GET\",\"unlabelled\":[]},{\"action\":\"/admin/budget/instance\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/create\",\"fields\":[\"csrf\",\"password\",\"profile_id\",\"username\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"EggPool\"],\"h2\":[\"Instance budget ceilings\",\"Users\",\"Local accounts\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/users?theme=Catppuccin%20Latte\",\"secretMarkers\":[],\"tables\":[{\"width\":593,\"wrapped\":true},{\"width\":445,\"wrapped\":true}],\"title\":\"Users\",\"viewportMeta\":true}", + "page.admin./users.mobile.Cyber Red": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/users\",\"fields\":[\"period\",\"theme\"],\"method\":\"GET\",\"unlabelled\":[]},{\"action\":\"/admin/budget/instance\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/create\",\"fields\":[\"csrf\",\"password\",\"profile_id\",\"username\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"EggPool\"],\"h2\":[\"Instance budget ceilings\",\"Users\",\"Local accounts\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/users?theme=Cyber%20Red\",\"secretMarkers\":[],\"tables\":[{\"width\":593,\"wrapped\":true},{\"width\":445,\"wrapped\":true}],\"title\":\"Users\",\"viewportMeta\":true}", + "page.admin./users.mobile.Cyberpunk": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/users\",\"fields\":[\"period\",\"theme\"],\"method\":\"GET\",\"unlabelled\":[]},{\"action\":\"/admin/budget/instance\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/create\",\"fields\":[\"csrf\",\"password\",\"profile_id\",\"username\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"EggPool\"],\"h2\":[\"Instance budget ceilings\",\"Users\",\"Local accounts\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/users?theme=Cyberpunk\",\"secretMarkers\":[],\"tables\":[{\"width\":593,\"wrapped\":true},{\"width\":445,\"wrapped\":true}],\"title\":\"Users\",\"viewportMeta\":true}", + "page.admin./users.mobile.default": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/users\",\"fields\":[\"period\",\"theme\"],\"method\":\"GET\",\"unlabelled\":[]},{\"action\":\"/admin/budget/instance\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/create\",\"fields\":[\"csrf\",\"password\",\"profile_id\",\"username\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/state\",\"fields\":[\"csrf\",\"enabled\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/password\",\"fields\":[\"csrf\",\"password\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/profile\",\"fields\":[\"csrf\",\"profile_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override\",\"fields\":[\"csrf\",\"effect\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/override/remove\",\"fields\":[\"csrf\",\"model_id\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/budget\",\"fields\":[\"csrf\",\"enforcement\",\"max_concurrent\",\"max_microdollars\",\"max_requests\",\"max_tokens\",\"model_id\",\"user_id\",\"window\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/token\",\"fields\":[\"csrf\",\"label\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/admin/users/revoke-tokens\",\"fields\":[\"csrf\",\"user_id\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"EggPool\"],\"h2\":[\"Instance budget ceilings\",\"Users\",\"Local accounts\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/users?theme=default\",\"secretMarkers\":[],\"tables\":[{\"width\":593,\"wrapped\":true},{\"width\":445,\"wrapped\":true}],\"title\":\"Users\",\"viewportMeta\":true}", + "page.member./my-account.desktop": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/account/tokens\",\"fields\":[\"csrf\",\"label\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/logout\",\"fields\":[\"csrf\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"My account\"],\"h2\":[\"Personal client credentials\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/my-account\",\"secretMarkers\":[],\"tables\":[],\"title\":\"My account\",\"viewportMeta\":true}", + "page.member./my-account.desktop-hidpi": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/account/tokens\",\"fields\":[\"csrf\",\"label\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/logout\",\"fields\":[\"csrf\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"My account\"],\"h2\":[\"Personal client credentials\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/my-account\",\"secretMarkers\":[],\"tables\":[],\"title\":\"My account\",\"viewportMeta\":true}", + "page.member./my-account.mobile": "{\"duplicateIds\":[],\"forms\":[{\"action\":\"/account/tokens\",\"fields\":[\"csrf\",\"label\"],\"method\":\"POST\",\"unlabelled\":[]},{\"action\":\"/logout\",\"fields\":[\"csrf\"],\"method\":\"POST\",\"unlabelled\":[]}],\"h1\":[\"My account\"],\"h2\":[\"Personal client credentials\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/my-account\",\"secretMarkers\":[],\"tables\":[],\"title\":\"My account\",\"viewportMeta\":true}", + "page.member./my-account/models.desktop": "{\"duplicateIds\":[],\"forms\":[],\"h1\":[\"My accessible models\"],\"h2\":[],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/my-account/models\",\"secretMarkers\":[],\"tables\":[],\"title\":\"My models\",\"viewportMeta\":true}", + "page.member./my-account/models.desktop-hidpi": "{\"duplicateIds\":[],\"forms\":[],\"h1\":[\"My accessible models\"],\"h2\":[],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/my-account/models\",\"secretMarkers\":[],\"tables\":[],\"title\":\"My models\",\"viewportMeta\":true}", + "page.member./my-account/models.mobile": "{\"duplicateIds\":[],\"forms\":[],\"h1\":[\"My accessible models\"],\"h2\":[],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/my-account/models\",\"secretMarkers\":[],\"tables\":[],\"title\":\"My models\",\"viewportMeta\":true}", + "page.member./usage.desktop": "{\"duplicateIds\":[],\"forms\":[],\"h1\":[\"My usage\"],\"h2\":[\"Configured limits\",\"Settled usage\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/usage\",\"secretMarkers\":[],\"tables\":[],\"title\":\"My usage\",\"viewportMeta\":true}", + "page.member./usage.desktop-hidpi": "{\"duplicateIds\":[],\"forms\":[],\"h1\":[\"My usage\"],\"h2\":[\"Configured limits\",\"Settled usage\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/usage\",\"secretMarkers\":[],\"tables\":[],\"title\":\"My usage\",\"viewportMeta\":true}", + "page.member./usage.mobile": "{\"duplicateIds\":[],\"forms\":[],\"h1\":[\"My usage\"],\"h2\":[\"Configured limits\",\"Settled usage\"],\"horizontalOverflow\":false,\"overflowing\":[],\"path\":\"/usage\",\"secretMarkers\":[],\"tables\":[{\"width\":579,\"wrapped\":true}],\"title\":\"My usage\",\"viewportMeta\":true}", + "restart.member_row": "[]", + "restart.profiles_survive_restart": "false", + "restart.session_survives_restart": "\"/users\"" + }, + "schema": 1, + "screenshots": [ + { + "height": 800, + "key": "page.admin./users.desktop", + "sha256": "a28af2d83bb825d34e082ce3aeefd3435f71504c7993ac4b9c865a77971b83fd", + "width": 1280 + }, + { + "height": 844, + "key": "page.admin./users.mobile", + "sha256": "c6cd7328dad32e19a741bdfa9c23cf3f12b334cd66d3ff1bce3e809a495767ae", + "width": 390 + }, + { + "height": 1600, + "key": "page.admin./users.desktop-hidpi", + "sha256": "4f9d3669b3b9f8d6ce95ed99a327c2ddd95225457e204fb91ad9f5cf98948556", + "width": 2560 + }, + { + "height": 800, + "key": "page.admin./profiles.desktop", + "sha256": "97b88f1489bfd8f02556c983cd651a82796d90c31d8eed19a852ae075408fdc9", + "width": 1280 + }, + { + "height": 844, + "key": "page.admin./profiles.mobile", + "sha256": "0a57216bce1dd4b8a97171d1a7218d3c67f2990aecfa1b09f7f9c3531bbb59b3", + "width": 390 + }, + { + "height": 1600, + "key": "page.admin./profiles.desktop-hidpi", + "sha256": "e3d6016e8afce40eaf697cdf04f82197b291287bb0815147325a32bf866e2ba1", + "width": 2560 + }, + { + "height": 800, + "key": "page.admin./admin/audit.desktop", + "sha256": "1a24770c09f63ddfe56b6122a77c3addb4d5fd64a4cb3c463eb3d49d4b8cc7bb", + "width": 1280 + }, + { + "height": 844, + "key": "page.admin./admin/audit.mobile", + "sha256": "cfe3c99e504f3aaf860c1b21fddc30d17d4c76f810c4efbb71d8e8c318871003", + "width": 390 + }, + { + "height": 1600, + "key": "page.admin./admin/audit.desktop-hidpi", + "sha256": "9483f21047977c1b9f8b784ac46cdd26753319d19059ef86e9661c812f648084", + "width": 2560 + }, + { + "height": 844, + "key": "page.admin./users.mobile.default", + "sha256": "c6cd7328dad32e19a741bdfa9c23cf3f12b334cd66d3ff1bce3e809a495767ae", + "width": 390 + }, + { + "height": 844, + "key": "page.admin./users.mobile.Cyber Red", + "sha256": "c6cd7328dad32e19a741bdfa9c23cf3f12b334cd66d3ff1bce3e809a495767ae", + "width": 390 + }, + { + "height": 844, + "key": "page.admin./users.mobile.Catppuccin Latte", + "sha256": "c6cd7328dad32e19a741bdfa9c23cf3f12b334cd66d3ff1bce3e809a495767ae", + "width": 390 + }, + { + "height": 844, + "key": "page.admin./users.mobile.Cyberpunk", + "sha256": "c6cd7328dad32e19a741bdfa9c23cf3f12b334cd66d3ff1bce3e809a495767ae", + "width": 390 + }, + { + "height": 800, + "key": "page.member./my-account.desktop", + "sha256": "7691fe0dcb6b564c1b0705f94197969e52f86b7719a07ca39b2bcdd161fb6cd8", + "width": 1280 + }, + { + "height": 844, + "key": "page.member./my-account.mobile", + "sha256": "1a000293d448c299807a8443ba37f37a6244d70b53a6a510130c0fa568a89cd1", + "width": 390 + }, + { + "height": 1600, + "key": "page.member./my-account.desktop-hidpi", + "sha256": "807df4fcf38f0fa2c289c0fbb9679f9270aec382f57d0f40e5cf157a346eec86", + "width": 2560 + }, + { + "height": 800, + "key": "page.member./my-account/models.desktop", + "sha256": "e40637f9fac94bbac0c95f221de13bafda87aed616e69e4c378a5d093b5e9c4f", + "width": 1280 + }, + { + "height": 844, + "key": "page.member./my-account/models.mobile", + "sha256": "40b06ed0413fc81fa940076105843f4a717a9ecdd88708c23a66ccaa3894ade7", + "width": 390 + }, + { + "height": 1600, + "key": "page.member./my-account/models.desktop-hidpi", + "sha256": "fe7fe1c487233deae2f135ef9ef18d9dc85e431764c3622076a8fe042941565c", + "width": 2560 + }, + { + "height": 800, + "key": "page.member./usage.desktop", + "sha256": "0994f9c01997b6fef82d4e53302a5dcdcad18ed51a17737e02461384b0b326ad", + "width": 1280 + }, + { + "height": 844, + "key": "page.member./usage.mobile", + "sha256": "37720a3c58fa44cd0a94c277f20d55a2a960e8382f51068ab5c42d49a17dcfb8", + "width": 390 + }, + { + "height": 1600, + "key": "page.member./usage.desktop-hidpi", + "sha256": "00d0ba324bcd512e127d741ad561802dd1029f4bff2acc4910e74d0a858c0495", + "width": 2560 + } + ] +} diff --git a/tests/fixtures/database/schema54-pre-upgrade.sql b/tests/fixtures/database/schema54-pre-upgrade.sql new file mode 100644 index 000000000..9dbccc6bd --- /dev/null +++ b/tests/fixtures/database/schema54-pre-upgrade.sql @@ -0,0 +1,588 @@ +PRAGMA foreign_keys=OFF; +BEGIN TRANSACTION; +CREATE TABLE _migrations ( +version INTEGER PRIMARY KEY, +name TEXT NOT NULL, +applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +INSERT INTO _migrations VALUES(1,'0001_initial.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(2,'0002_indexes.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(3,'0003_request_attempts.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(4,'0004_integration_hardening.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(5,'0005_price_microdollars.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(6,'0006_correct_price_microdollars.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(7,'0007_price_cache_rates.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(8,'0008_proxy_request_identity.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(9,'0009_model_protocol_source.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(10,'0010_health_probe.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(11,'0011_model_resolution_status.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(12,'0012_drop_reservations_estimated_microdollars.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(13,'0013_request_attempts_account_id_index.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(14,'0014_bandwidth_tracking.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(15,'0015_multi_provider.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(16,'0016_requests_provider_id.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(17,'0017_price_snapshots_provider_id.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(18,'0018_provider_pings.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(19,'0019_client_ip.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(20,'0020_performance_indexes.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(21,'0021_provider_model_metadata.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(22,'0022_dashboard_indexes.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(23,'0023_deprecated_model_placeholder.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(24,'0024_account_backoffs.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(25,'0025_stale_request_index.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(26,'0026_attempt_observability.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(27,'0027_routing_decisions.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(28,'0028_operational_events.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(29,'0029_latency_phases.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(30,'0030_model_pricing_aliases.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(31,'0031_price_snapshot_provenance.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(32,'0032_usage_rollups.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(33,'0033_request_provider_local_cost.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(34,'0034_transcoding_daily.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(35,'0035_routing_decision_score_components.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(36,'0036_model_info.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(37,'0037_model_info_failure_count.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(38,'0038_model_info_phase5.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(39,'0039_thinking_observability.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(40,'0040_cache_token_observability.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(41,'0041_canonical_request_segmentation.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(42,'0042_observe_mode_compression_accounting.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(43,'0043_safe_suffix_compression.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(44,'0044_compression_policy_resolver.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(45,'0045_synthetic_cache_controls.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(46,'0046_closed_loop_threshold_tuning.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(47,'0047_normalize_rollup_bucket_start.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(48,'0048_request_cost_repairs.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(49,'0049_model_info_match_evidence.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(50,'0050_routing_decisions_retention_index.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(51,'0051_model_quarantine.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(52,'0052_catalog_refresh_state.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(53,'0053_remove_attempt_status_analytics_index.sql','2026-10-14 16:14:21'); +INSERT INTO _migrations VALUES(54,'0054_model_quarantine_null_identity.sql','2026-10-14 16:14:21'); +CREATE TABLE accounts ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL UNIQUE, + api_key_env TEXT NOT NULL, + enabled INTEGER NOT NULL DEFAULT 1, + weight REAL NOT NULL DEFAULT 1.0, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +, provider_id TEXT NOT NULL DEFAULT 'opencode-go'); +INSERT INTO accounts VALUES(1,'account-a','D001_ACCOUNT_A_KEY',1,2.0,'2026-10-14 16:14:21','provider-a'); +INSERT INTO accounts VALUES(2,'account-b','D001_ACCOUNT_B_KEY',1,1.0,'2026-10-14 16:14:21','provider-b'); +INSERT INTO accounts VALUES(3,'disabled-a','D001_DISABLED_KEY',0,1.0,'2026-10-14 16:14:21','provider-a'); +CREATE TABLE models ( + model_id TEXT PRIMARY KEY, + display_name TEXT, + protocol TEXT NOT NULL DEFAULT 'openai', + capabilities TEXT NOT NULL DEFAULT '{}', + source_metadata TEXT NOT NULL DEFAULT '{}', + first_seen_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + last_seen_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +, protocol_source TEXT, endpoint_path TEXT, resolution_status TEXT NOT NULL DEFAULT 'resolved', provider_id TEXT NOT NULL DEFAULT 'opencode-go'); +INSERT INTO models VALUES('__deprecated__','Deprecated models','openai','{}','{}','2026-10-14 16:14:21','2026-10-14 16:14:21',NULL,NULL,'resolved','opencode-go'); +INSERT INTO models VALUES('shared-model','Shared Model','openai','{"supports_tools":true}','{"fixture":"d001"}','2026-10-14 16:14:21','2026-10-14 16:14:21','config',NULL,'resolved','provider-a'); +INSERT INTO models VALUES('withdrawn-model','Withdrawn Model','openai','{}','{"fixture":"d001"}','2026-10-14 16:14:21','2026-10-14 16:14:21','config',NULL,'resolved','provider-a'); +CREATE TABLE account_models ( + account_id INTEGER NOT NULL, + model_id TEXT NOT NULL, + enabled INTEGER NOT NULL DEFAULT 1, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (account_id, model_id), + FOREIGN KEY (account_id) REFERENCES accounts(id) ON DELETE CASCADE, + FOREIGN KEY (model_id) REFERENCES models(model_id) ON DELETE CASCADE +); +INSERT INTO account_models VALUES(1,'shared-model',1,'2026-10-14 16:14:21'); +INSERT INTO account_models VALUES(2,'shared-model',1,'2026-10-14 16:14:21'); +INSERT INTO account_models VALUES(1,'withdrawn-model',1,'2026-10-14 16:14:21'); +CREATE TABLE requests ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + account_id INTEGER NOT NULL, + model_id TEXT NOT NULL, + started_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + completed_at TIMESTAMP, + status TEXT NOT NULL DEFAULT 'pending', + input_tokens INTEGER DEFAULT 0, + output_tokens INTEGER DEFAULT 0, + cost_microdollars INTEGER DEFAULT 0, + upstream_latency_ms REAL DEFAULT 0, + error_message TEXT, protocol TEXT NOT NULL DEFAULT 'openai', streamed INTEGER NOT NULL DEFAULT 0, exactness TEXT NOT NULL DEFAULT 'unknown', cache_read_tokens INTEGER, cache_write_tokens INTEGER, reasoning_tokens INTEGER, thinking_characters INTEGER, reserved_microdollars INTEGER NOT NULL DEFAULT 0, first_byte_ms INTEGER, retry_count INTEGER NOT NULL DEFAULT 0, upstream_request_id TEXT, error_class TEXT, error_detail TEXT, status_code INTEGER, proxy_request_id TEXT, bytes_received INTEGER DEFAULT 0, bytes_emitted INTEGER DEFAULT 0, provider_id TEXT NOT NULL DEFAULT 'opencode-go', client_ip TEXT, original_model_id TEXT, first_attempt_at TIMESTAMP, last_attempt_id INTEGER, upstream_connect_ms INTEGER, upstream_read_ms INTEGER, coordinator_overhead_ms INTEGER, provider_cost_microdollars INTEGER, provider_cost_source TEXT, local_cost_microdollars INTEGER, local_cost_exactness TEXT, upstream_protocol TEXT, thinking_trace_json TEXT, cache_counter_status TEXT NOT NULL DEFAULT 'not_reported', cached_input_tokens INTEGER, cache_read_input_tokens INTEGER, cache_creation_input_tokens INTEGER, cache_write_input_tokens INTEGER, cache_write_input_reported INTEGER, input_tokens_reported INTEGER, output_tokens_reported INTEGER, total_tokens_reported INTEGER, request_shape_hash TEXT, stable_prefix_hash TEXT, transcoded INTEGER NOT NULL DEFAULT 0, raw_usage_json TEXT, segmentation_status TEXT NOT NULL DEFAULT 'empty_request', stable_prefix_estimated_tokens INTEGER, semi_stable_estimated_tokens INTEGER, volatile_estimated_tokens INTEGER, stable_prefix_bytes INTEGER, semi_stable_bytes INTEGER, volatile_bytes INTEGER, segmentation_summary_json TEXT, compression_status TEXT NOT NULL DEFAULT 'disabled', compression_mode TEXT, compression_candidate_count INTEGER NOT NULL DEFAULT 0, compression_eligible_candidate_count INTEGER NOT NULL DEFAULT 0, compression_suppressed_candidate_count INTEGER NOT NULL DEFAULT 0, compression_estimated_original_tokens INTEGER, compression_estimated_compressed_tokens INTEGER, compression_estimated_savings_tokens INTEGER, compression_analyzer_latency_ms REAL, compression_warning_count INTEGER NOT NULL DEFAULT 0, compression_reason_code_counts_json TEXT, compression_summary_json TEXT, compression_applied INTEGER NOT NULL DEFAULT 0, compression_transform_count INTEGER NOT NULL DEFAULT 0, compression_transforms_by_reason_json TEXT, compression_original_tokens INTEGER, compression_compressed_tokens INTEGER, compression_savings_tokens INTEGER, compression_pre_stable_prefix_hash TEXT, compression_post_stable_prefix_hash TEXT, compression_stable_prefix_preserved INTEGER NOT NULL DEFAULT 1, compression_warnings_json TEXT, compression_latency_ms REAL NOT NULL DEFAULT 0, compression_failed_fallback INTEGER NOT NULL DEFAULT 0, compression_applied_summary_json TEXT, compression_policy_name TEXT, compression_policy_source TEXT, compression_policy_warnings_json TEXT, synthetic_cache_status TEXT, synthetic_cache_dry_run INTEGER NOT NULL DEFAULT 1, synthetic_cache_candidate_count INTEGER NOT NULL DEFAULT 0, synthetic_cache_applied_count INTEGER NOT NULL DEFAULT 0, synthetic_cache_warning_count INTEGER NOT NULL DEFAULT 0, synthetic_cache_warnings_json TEXT, synthetic_cache_policy_name TEXT, synthetic_cache_policy_source TEXT, synthetic_cache_summary_json TEXT, + FOREIGN KEY (account_id) REFERENCES accounts(id), + FOREIGN KEY (model_id) REFERENCES models(model_id) +); +INSERT INTO requests VALUES(1,1,'shared-model','2026-10-14 16:14:21',NULL,'success',100,50,700,0.0,NULL,'openai',0,'unknown',NULL,NULL,NULL,NULL,0,NULL,0,NULL,NULL,NULL,NULL,NULL,0,0,'provider-a',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,'not_reported',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,0,NULL,'empty_request',NULL,NULL,NULL,NULL,NULL,NULL,NULL,'disabled',NULL,0,0,0,NULL,NULL,NULL,NULL,0,NULL,NULL,0,0,NULL,NULL,NULL,NULL,NULL,NULL,1,NULL,0.0,0,NULL,NULL,NULL,NULL,NULL,1,0,0,0,NULL,NULL,NULL,NULL); +INSERT INTO requests VALUES(2,2,'shared-model','2026-10-14 16:14:21',NULL,'pending',20,10,100,0.0,NULL,'anthropic',1,'unknown',NULL,NULL,NULL,NULL,0,NULL,0,NULL,NULL,NULL,NULL,NULL,0,0,'provider-b',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,'not_reported',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,0,NULL,'empty_request',NULL,NULL,NULL,NULL,NULL,NULL,NULL,'disabled',NULL,0,0,0,NULL,NULL,NULL,NULL,0,NULL,NULL,0,0,NULL,NULL,NULL,NULL,NULL,NULL,1,NULL,0.0,0,NULL,NULL,NULL,NULL,NULL,1,0,0,0,NULL,NULL,NULL,NULL); +CREATE TABLE reservations ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + request_id INTEGER NOT NULL, + account_id INTEGER NOT NULL, + model_id TEXT NOT NULL, + reserved_microdollars INTEGER NOT NULL DEFAULT 0, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + released_at TIMESTAMP, + status TEXT NOT NULL DEFAULT 'active', estimated_tokens INTEGER NOT NULL DEFAULT 0, expires_at TIMESTAMP, release_reason TEXT, original_model_id TEXT, + FOREIGN KEY (request_id) REFERENCES requests(id), + FOREIGN KEY (account_id) REFERENCES accounts(id), + FOREIGN KEY (model_id) REFERENCES models(model_id) +); +INSERT INTO reservations VALUES(1,1,1,'shared-model',700,'2026-10-14 16:14:21',NULL,'released',0,NULL,NULL,NULL); +INSERT INTO reservations VALUES(2,2,2,'shared-model',500,'2026-10-14 16:14:21',NULL,'active',0,NULL,NULL,NULL); +CREATE TABLE model_price_snapshots ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + model_id TEXT NOT NULL, + input_price_per_1k REAL, + output_price_per_1k REAL, + captured_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, input_per_million_microdollars INTEGER, output_per_million_microdollars INTEGER, source TEXT NOT NULL DEFAULT 'config', metadata_json TEXT, cache_read_per_million_microdollars INTEGER, cache_write_per_million_microdollars INTEGER, provider_id TEXT NOT NULL DEFAULT 'opencode-go', source_detail TEXT, source_confidence TEXT, catalog_source TEXT, + FOREIGN KEY (model_id) REFERENCES models(model_id) +); +CREATE TABLE account_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + account_id INTEGER NOT NULL, + event_type TEXT NOT NULL, + details TEXT NOT NULL DEFAULT '{}', + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + FOREIGN KEY (account_id) REFERENCES accounts(id) +); +CREATE TABLE request_attempts ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + request_id INTEGER NOT NULL, + attempt_number INTEGER NOT NULL, + account_id INTEGER NOT NULL, + started_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + completed_at TIMESTAMP, + status_code INTEGER, + error_class TEXT, + upstream_request_id TEXT, bytes_emitted INTEGER NOT NULL DEFAULT 0, error_detail TEXT, provider_id TEXT, model_id TEXT, protocol TEXT, retry_category TEXT, release_reason TEXT, bytes_received INTEGER NOT NULL DEFAULT 0, latency_ms INTEGER NOT NULL DEFAULT 0, streamed INTEGER NOT NULL DEFAULT 0, is_retry_outcome INTEGER NOT NULL DEFAULT 0, + FOREIGN KEY (request_id) REFERENCES requests(id) ON DELETE CASCADE, + FOREIGN KEY (account_id) REFERENCES accounts(id) +); +CREATE TABLE health_probe ( + id INTEGER PRIMARY KEY, + probe_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +CREATE TABLE providers ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + provider_id TEXT NOT NULL UNIQUE, + base_url TEXT NOT NULL, + protocols TEXT NOT NULL DEFAULT '["openai"]', + enabled INTEGER NOT NULL DEFAULT 1, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); +INSERT INTO providers VALUES(1,'opencode-go','https://opencode.ai/zen/go/v1','["openai", "anthropic"]',1,'2026-10-14 16:14:21'); +INSERT INTO providers VALUES(10,'provider-a','https://provider-a.invalid/v1','["openai","anthropic"]',1,'2026-10-14 16:14:21'); +INSERT INTO providers VALUES(11,'provider-b','https://provider-b.invalid/v1','["openai"]',1,'2026-10-14 16:14:21'); +CREATE TABLE provider_pings ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + provider_id TEXT NOT NULL, + account_name TEXT NOT NULL, + probed_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + latency_ms INTEGER, + status_code INTEGER, + error TEXT, + model_count INTEGER DEFAULT 0 +); +CREATE TABLE provider_model_metadata ( + model_id TEXT NOT NULL, + provider_id TEXT NOT NULL, + display_name TEXT, + protocol TEXT, + capabilities TEXT NOT NULL DEFAULT '{}', + source_metadata TEXT NOT NULL DEFAULT '{}', + protocol_source TEXT, + first_seen_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + last_seen_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + resolution_status TEXT NOT NULL DEFAULT 'resolved', + PRIMARY KEY (model_id, provider_id), + FOREIGN KEY (model_id) REFERENCES models(model_id) ON DELETE CASCADE +); +INSERT INTO provider_model_metadata VALUES('shared-model','provider-a','Shared Model','openai','{"supports_tools":true}','{"fixture":"d001"}','config','2026-10-14 16:14:21','2026-10-14 16:14:21','resolved'); +INSERT INTO provider_model_metadata VALUES('shared-model','provider-b','Shared Model','anthropic','{"supports_tools":true}','{"fixture":"d001"}','config','2026-10-14 16:14:21','2026-10-14 16:14:21','resolved'); +INSERT INTO provider_model_metadata VALUES('withdrawn-model','provider-a','Withdrawn Model','openai','{}','{"fixture":"d001"}','config','2026-10-14 16:14:21','2026-10-14 16:14:21','resolved'); +CREATE TABLE account_backoffs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + account_id INTEGER NOT NULL REFERENCES accounts(id) ON DELETE CASCADE, + model_id TEXT, + reason TEXT NOT NULL, + status_code INTEGER, + error_class TEXT, + consecutive_failures INTEGER NOT NULL DEFAULT 1, + backoff_until TEXT, + last_failure_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + UNIQUE(account_id, model_id, reason) +); +INSERT INTO account_backoffs VALUES(1,1,NULL,'rate_limited',429,'rate_limited',2,'2026-09-04 00:10:00','2026-09-04 00:00:00','2026-09-04 00:00:00'); +INSERT INTO account_backoffs VALUES(2,1,'withdrawn-model','model_unavailable',404,'model_not_found',1,'2026-09-04 00:05:00','2026-09-04 00:00:00','2026-09-04 00:00:00'); +CREATE TABLE routing_decisions ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + request_id INTEGER NOT NULL, + attempt_number INTEGER NOT NULL, + model_id TEXT NOT NULL, + provider_id TEXT, + protocol TEXT, + selected_account_id INTEGER, + selected_account_name TEXT, + selected_tier INTEGER, + selected_score REAL, + eligible_count INTEGER NOT NULL DEFAULT 0, + scored_count INTEGER NOT NULL DEFAULT 0, + attempted_excluded_count INTEGER NOT NULL DEFAULT 0, + top_score REAL, + top_score_account_name TEXT, + exclude_reasons_json TEXT NOT NULL DEFAULT '[]', + decision_made_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, score_components_json TEXT NOT NULL DEFAULT '{}', + FOREIGN KEY (request_id) REFERENCES requests(id) ON DELETE CASCADE, + FOREIGN KEY (selected_account_id) REFERENCES accounts(id) +); +CREATE TABLE operational_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + event_type TEXT NOT NULL, + details_json TEXT NOT NULL DEFAULT '{}', + occurred_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); +CREATE TABLE model_pricing_aliases ( + provider_id TEXT NOT NULL, + upstream_model_id TEXT NOT NULL, + catalog_source TEXT NOT NULL, + catalog_model_id TEXT NOT NULL, + confidence TEXT NOT NULL, + notes TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (provider_id, upstream_model_id, catalog_source) +); +CREATE TABLE usage_rollups ( + bucket_start TEXT NOT NULL, + bucket_size_s INTEGER NOT NULL, + provider_id TEXT NOT NULL, + model_id TEXT NOT NULL, + account_id INTEGER NOT NULL DEFAULT 0, + protocol TEXT NOT NULL, + streamed INTEGER NOT NULL DEFAULT 0, + status TEXT NOT NULL, + request_count INTEGER NOT NULL DEFAULT 0, + error_count INTEGER NOT NULL DEFAULT 0, + retry_count INTEGER NOT NULL DEFAULT 0, + input_tokens INTEGER NOT NULL DEFAULT 0, + output_tokens INTEGER NOT NULL DEFAULT 0, + cache_read_tokens INTEGER NOT NULL DEFAULT 0, + cache_write_tokens INTEGER NOT NULL DEFAULT 0, + reasoning_tokens INTEGER NOT NULL DEFAULT 0, + thinking_characters INTEGER NOT NULL DEFAULT 0, + cost_microdollars INTEGER NOT NULL DEFAULT 0, + bytes_received INTEGER NOT NULL DEFAULT 0, + bytes_emitted INTEGER NOT NULL DEFAULT 0, + latency_ms_sum INTEGER NOT NULL DEFAULT 0, + latency_ms_min INTEGER, + latency_ms_max INTEGER, + first_byte_ms_sum INTEGER NOT NULL DEFAULT 0, + first_byte_ms_count INTEGER NOT NULL DEFAULT 0, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY ( + bucket_start, + bucket_size_s, + provider_id, + model_id, + account_id, + protocol, + streamed, + status + ) +); +CREATE TABLE transcoding_daily ( + day TEXT NOT NULL, -- YYYY-MM-DD UTC + client_protocol TEXT NOT NULL, + upstream_protocol TEXT NOT NULL, + request_count INTEGER NOT NULL DEFAULT 0, + loss_warning_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (day, client_protocol, upstream_protocol) +) WITHOUT ROWID; +CREATE TABLE model_info_canonical ( + model_id TEXT PRIMARY KEY, + status TEXT NOT NULL, + summary TEXT, + detail_json TEXT NOT NULL DEFAULT '{}', + provenance_json TEXT NOT NULL DEFAULT '{}', + conflicts_json TEXT NOT NULL DEFAULT '{}', + sparse INTEGER NOT NULL DEFAULT 0, + first_seen_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + last_seen_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + last_refreshed_at TIMESTAMP, + next_refresh_at TIMESTAMP, + FOREIGN KEY (model_id) REFERENCES models(model_id) ON DELETE CASCADE +); +CREATE TABLE model_info_observations ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + model_id TEXT, + provider_id TEXT, + source TEXT NOT NULL, + source_model_id TEXT NOT NULL, + observed_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + expires_at TIMESTAMP, + confidence REAL NOT NULL DEFAULT 0.5, + raw_hash TEXT NOT NULL, + normalized_json TEXT NOT NULL DEFAULT '{}', + raw_json TEXT NOT NULL DEFAULT '{}', + FOREIGN KEY (model_id) REFERENCES models(model_id) ON DELETE CASCADE, + UNIQUE(source, source_model_id, raw_hash) +); +CREATE TABLE model_info_aliases ( + model_id TEXT NOT NULL, + provider_id TEXT, + alias TEXT NOT NULL, + source TEXT NOT NULL, + confidence REAL NOT NULL DEFAULT 0.5, + active INTEGER NOT NULL DEFAULT 1, + first_seen_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + last_seen_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, notes TEXT, match_method TEXT, discovered_by TEXT, diagnostics_json TEXT NOT NULL DEFAULT '{}', + PRIMARY KEY (provider_id, alias, source), + FOREIGN KEY (model_id) REFERENCES models(model_id) ON DELETE CASCADE +); +CREATE TABLE model_info_source_health ( + source TEXT PRIMARY KEY, + enabled INTEGER NOT NULL DEFAULT 1, + last_success_at TIMESTAMP, + last_error_at TIMESTAMP, + last_error_class TEXT, + last_error_message TEXT, + cooldown_until TIMESTAMP +, failure_count INTEGER NOT NULL DEFAULT 0, last_status_code INTEGER, rate_limited_until TIMESTAMP, last_success_duration_ms INTEGER, last_payload_count INTEGER); +CREATE TABLE model_info_overrides ( + model_id TEXT PRIMARY KEY, + summary TEXT, + family TEXT, + display_name TEXT, + notes TEXT, + hide_benchmark_sources INTEGER NOT NULL DEFAULT 0, + status_override TEXT, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + FOREIGN KEY (model_id) REFERENCES models(model_id) ON DELETE CASCADE +); +CREATE TABLE compression_tuning_recommendations ( + policy_name TEXT PRIMARY KEY, + status TEXT NOT NULL, + recommendation_json TEXT NOT NULL, + generated_at TEXT NOT NULL, + created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')) +); +CREATE TABLE compression_tuning_overrides ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + policy_name TEXT NOT NULL, + fields_json TEXT NOT NULL, + reason_codes_json TEXT NOT NULL, + generated_at TEXT NOT NULL, + expires_at TEXT +); +CREATE TABLE request_cost_repairs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + request_id INTEGER NOT NULL + REFERENCES requests(id) ON DELETE CASCADE, + old_cost_microdollars INTEGER NOT NULL, + new_cost_microdollars INTEGER NOT NULL, + old_exactness TEXT, + new_exactness TEXT NOT NULL, + reason TEXT NOT NULL, + provider_filter TEXT, + since_date TEXT, + repaired_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +CREATE TABLE model_info_match_evidence ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + model_id TEXT NOT NULL, + provider_id TEXT, + source TEXT NOT NULL, + alias TEXT NOT NULL, + match_method TEXT NOT NULL, + confidence REAL NOT NULL, + diagnostics_json TEXT NOT NULL DEFAULT '{}', + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + last_seen_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + FOREIGN KEY (model_id) REFERENCES models(model_id) ON DELETE CASCADE +); +CREATE TABLE model_quarantine ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + provider_id TEXT NOT NULL, + account_id TEXT NOT NULL, + canonical_model_id TEXT NOT NULL, + upstream_model_id TEXT, + upstream_protocol TEXT NOT NULL DEFAULT 'openai', + state TEXT NOT NULL DEFAULT 'suspected', + evidence_provenance TEXT NOT NULL DEFAULT 'runtime_http', + reason TEXT NOT NULL DEFAULT '', + first_observed TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + last_observed TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + observation_count INTEGER NOT NULL DEFAULT 1, + expiry TEXT, + cleared_at TEXT, + clear_reason TEXT, + last_status_code INTEGER, + last_error_class TEXT, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + UNIQUE(provider_id, account_id, canonical_model_id, upstream_model_id, upstream_protocol) +); +INSERT INTO model_quarantine VALUES(1,'provider-a','account-a','withdrawn-model','withdrawn-model','openai','quarantined','runtime_http','model_unavailable','2026-09-04 00:00:00','2026-09-04 00:01:00',2,'2026-09-04 00:06:00',NULL,NULL,404,'model_not_found','2026-10-14 16:14:21'); +INSERT INTO model_quarantine VALUES(2,'provider-b','account-b','shared-model',NULL,'anthropic','suspected','provider_catalog','partial_refresh','2026-09-04 00:00:00','2026-09-04 00:00:00',1,'2026-09-04 00:02:00',NULL,NULL,NULL,NULL,'2026-10-14 16:14:21'); +CREATE TABLE catalog_refresh_state ( + account_id INTEGER PRIMARY KEY, + provider_id TEXT NOT NULL, + last_successful_refresh_at TIMESTAMP NOT NULL, + last_outcome TEXT NOT NULL DEFAULT 'legacy_hydrated', + model_count INTEGER NOT NULL DEFAULT 0, + FOREIGN KEY (account_id) REFERENCES accounts(id) ON DELETE CASCADE +); +INSERT INTO catalog_refresh_state VALUES(1,'provider-a','2026-09-04 00:00:00','success_authoritative',2); +INSERT INTO catalog_refresh_state VALUES(2,'provider-b','2026-09-04 00:00:00','success_authoritative',1); +PRAGMA writable_schema=ON; +CREATE TABLE IF NOT EXISTS sqlite_sequence(name,seq); +DELETE FROM sqlite_sequence; +INSERT INTO sqlite_sequence VALUES('providers',11); +INSERT INTO sqlite_sequence VALUES('accounts',3); +INSERT INTO sqlite_sequence VALUES('requests',2); +INSERT INTO sqlite_sequence VALUES('reservations',2); +INSERT INTO sqlite_sequence VALUES('account_backoffs',2); +INSERT INTO sqlite_sequence VALUES('model_quarantine',2); +CREATE INDEX idx_accounts_enabled ON accounts(enabled); +CREATE INDEX idx_account_models_account ON account_models(account_id); +CREATE INDEX idx_account_models_model ON account_models(model_id); +CREATE INDEX idx_requests_account ON requests(account_id); +CREATE INDEX idx_requests_model ON requests(model_id); +CREATE INDEX idx_requests_started ON requests(started_at); +CREATE INDEX idx_reservations_account ON reservations(account_id); +CREATE INDEX idx_reservations_model ON reservations(model_id); +CREATE INDEX idx_reservations_status ON reservations(status); +CREATE INDEX idx_reservations_request ON reservations(request_id); +CREATE INDEX idx_model_price_snapshots_model ON model_price_snapshots(model_id); +CREATE INDEX idx_model_price_snapshots_captured ON model_price_snapshots(captured_at); +CREATE INDEX idx_account_events_account ON account_events(account_id); +CREATE INDEX idx_account_events_type ON account_events(event_type); +CREATE INDEX idx_account_events_created ON account_events(created_at); +CREATE INDEX idx_request_attempts_request + ON request_attempts(request_id); +CREATE INDEX idx_request_attempts_account + ON request_attempts(account_id, started_at); +CREATE INDEX idx_requests_exactness ON requests(exactness); +CREATE INDEX idx_requests_completed ON requests(completed_at); +CREATE INDEX idx_requests_protocol ON requests(protocol); +CREATE INDEX idx_reservations_expires ON reservations(expires_at); +CREATE INDEX idx_requests_status_started ON requests(status, started_at); +CREATE INDEX idx_price_snapshots_model_source + ON model_price_snapshots(model_id, source); +CREATE UNIQUE INDEX idx_requests_proxy_request_id + ON requests(proxy_request_id); +CREATE INDEX idx_provider_pings_provider ON provider_pings(provider_id); +CREATE INDEX idx_provider_pings_probed ON provider_pings(probed_at); +CREATE INDEX idx_provider_pings_provider_probed + ON provider_pings(provider_id, probed_at); +CREATE INDEX idx_requests_account_started + ON requests(account_id, started_at); +CREATE INDEX idx_price_snapshots_model_provider_captured + ON model_price_snapshots(model_id, provider_id, captured_at DESC, id DESC); +CREATE INDEX idx_requests_streamed_provider_model_started_ttft + ON requests(provider_id, model_id, started_at, first_byte_ms) + WHERE streamed = 1 AND first_byte_ms IS NOT NULL; +CREATE INDEX idx_reservations_status_expires + ON reservations(status, expires_at); +CREATE INDEX idx_provider_model_metadata_provider + ON provider_model_metadata(provider_id, model_id); +CREATE INDEX idx_requests_client_ip_started + ON requests(client_ip, started_at); +CREATE INDEX idx_requests_streamed_started_ttft + ON requests(started_at, first_byte_ms) + WHERE streamed = 1 AND first_byte_ms IS NOT NULL; +CREATE INDEX idx_requests_original_model_id + ON requests(original_model_id) WHERE original_model_id IS NOT NULL; +CREATE INDEX idx_reservations_original_model_id + ON reservations(original_model_id) WHERE original_model_id IS NOT NULL; +CREATE INDEX idx_account_backoffs_active + ON account_backoffs(backoff_until); +CREATE INDEX idx_account_backoffs_account_model + ON account_backoffs(account_id, model_id); +CREATE INDEX idx_request_attempts_provider_started + ON request_attempts(provider_id, started_at); +CREATE INDEX idx_request_attempts_model_started + ON request_attempts(model_id, started_at); +CREATE INDEX idx_request_attempts_retry_category_started + ON request_attempts(retry_category, started_at); +CREATE INDEX idx_routing_decisions_request + ON routing_decisions(request_id); +CREATE INDEX idx_routing_decisions_model_started + ON routing_decisions(model_id, decision_made_at); +CREATE INDEX idx_routing_decisions_provider_started + ON routing_decisions(provider_id, decision_made_at); +CREATE INDEX idx_routing_decisions_selected_account + ON routing_decisions(selected_account_name, decision_made_at); +CREATE INDEX idx_operational_events_type_occurred + ON operational_events(event_type, occurred_at); +CREATE INDEX idx_operational_events_occurred + ON operational_events(occurred_at); +CREATE INDEX idx_requests_ttfb_ms + ON requests(first_byte_ms) + WHERE first_byte_ms IS NOT NULL; +CREATE INDEX idx_pricing_aliases_catalog + ON model_pricing_aliases(catalog_source, catalog_model_id); +CREATE INDEX idx_pricing_aliases_provider + ON model_pricing_aliases(provider_id, upstream_model_id); +CREATE INDEX idx_usage_rollups_bucket ON usage_rollups(bucket_start, bucket_size_s); +CREATE INDEX idx_usage_rollups_provider_model ON usage_rollups(provider_id, model_id, bucket_start); +CREATE INDEX idx_usage_rollups_account ON usage_rollups(account_id, bucket_start); +CREATE INDEX idx_model_info_canonical_status + ON model_info_canonical(status); +CREATE INDEX idx_model_info_canonical_next_refresh + ON model_info_canonical(next_refresh_at); +CREATE INDEX idx_model_info_observations_model_source + ON model_info_observations(model_id, source); +CREATE INDEX idx_model_info_observations_source_model + ON model_info_observations(source, source_model_id); +CREATE INDEX idx_model_info_aliases_alias + ON model_info_aliases(alias); +CREATE INDEX idx_requests_cache_counter_status + ON requests(cache_counter_status, started_at); +CREATE INDEX idx_requests_transcoded + ON requests(transcoded, started_at); +CREATE INDEX idx_requests_segmentation_status + ON requests(segmentation_status, started_at); +CREATE INDEX idx_requests_compression_status + ON requests(compression_status, started_at); +CREATE INDEX idx_requests_compression_mode + ON requests(compression_mode, started_at); +CREATE INDEX idx_requests_compression_applied + ON requests(compression_applied, started_at); +CREATE INDEX idx_requests_compression_policy_name + ON requests(compression_policy_name, started_at); +CREATE INDEX idx_requests_synthetic_cache_status + ON requests(synthetic_cache_status, started_at); +CREATE INDEX idx_requests_synthetic_cache_policy_name + ON requests(synthetic_cache_policy_name, started_at); +CREATE INDEX idx_compression_tuning_recommendations_status + ON compression_tuning_recommendations(status, generated_at); +CREATE INDEX idx_compression_tuning_overrides_policy + ON compression_tuning_overrides(policy_name, generated_at); +CREATE INDEX idx_request_cost_repairs_request + ON request_cost_repairs(request_id, repaired_at DESC); +CREATE INDEX idx_request_cost_repairs_repaired_at + ON request_cost_repairs(repaired_at DESC); +CREATE INDEX idx_model_info_match_evidence_model + ON model_info_match_evidence(model_id, source); +CREATE INDEX idx_model_info_match_evidence_source + ON model_info_match_evidence(source, last_seen_at DESC); +CREATE INDEX idx_routing_decisions_retention + ON routing_decisions(decision_made_at, id); +CREATE INDEX idx_model_quarantine_state + ON model_quarantine(state); +CREATE INDEX idx_model_quarantine_expiry + ON model_quarantine(expiry); +CREATE INDEX idx_model_quarantine_model + ON model_quarantine(canonical_model_id); +CREATE INDEX idx_catalog_refresh_state_provider + ON catalog_refresh_state(provider_id); +CREATE UNIQUE INDEX uq_model_quarantine_null_upstream + ON model_quarantine( + provider_id, + account_id, + canonical_model_id, + upstream_protocol + ) + WHERE upstream_model_id IS NULL; +PRAGMA writable_schema=OFF; +COMMIT; diff --git a/tests/fixtures/runtime/compatibility-observations.json b/tests/fixtures/runtime/compatibility-observations.json index 9e4946184..96e3452c3 100644 --- a/tests/fixtures/runtime/compatibility-observations.json +++ b/tests/fixtures/runtime/compatibility-observations.json @@ -1 +1 @@ -{"active_authority": {"active_state_after_shutdown": "no active generation is acquirable", "app_state_rule": "mirrors are compatibility snapshots, never authority for leased request work", "generation_dataclass_fields": ["generation_id", "config", "config_digest", "registry", "catalog", "router", "coordinator", "client_pool", "outbound_manager", "health_manager", "cost_calculator", "transcoder_policy", "dispatch_overhead_recorder", "dispatch_span_recorder", "account_backoff_repo", "stats_service", "supervisor", "routing_trace_guard", "routing_trace_writer", "created_at_monotonic", "created_at_epoch", "effects_applier", "model_quarantine", "finalization_supervisor", "model_info", "model_router_registry", "local_pre_upstream_recorder", "stream_diagnostics", "immutable_request_state", "generation_cleanup_callbacks"], "generation_owned_app_state_mirrors": ["account_backoff_repo", "catalog", "client_pool", "cost_calculator", "dashboard_telemetry", "dispatch_overhead_recorder", "dispatch_span_recorder", "health_manager", "httpx_client", "local_pre_upstream_recorder", "metrics_coalescer", "model_info", "model_router_registry", "outbound_manager", "registry", "router", "routing_trace_guard", "runtime_metrics", "stats", "stream_diagnostics", "supervisor", "task_monitor", "transcoder_policy"], "live_request_authority": ["providers", "accounts", "model_routers", "model_overrides", "model_capabilities", "server.max_request_body_bytes", "models.expose_mode", "models.collapse_models", "models.stale_after_s", "models.allow_stale_catalog", "routing.* request/retry/health/fairness settings", "routing.wire_negotiation.*", "transcoder.*", "security.persist_redacted_error_detail", "task intervals and retention settings consumed per tick"], "process_runtime_fields": ["db", "stats_db", "config_path", "metrics_coalescer", "process_supervisor", "task_spec_version", "last_task_transition", "routing_trace_writer", "maintenance_state", "event_loop_lag_monitor", "readiness_probe", "wire_profile_resolver", "model_router_affinity", "model_router_metrics", "model_info"], "request_generation_rule": "one generation from admission through finite terminal registration or stream disconnect", "restart_required_constructor_authority": ["server.host/port/api_key/api_key_env/log_level/access_log/threads", "upstream transport settings", "database constructor and path settings", "dashboard route topology and public/auth mode", "security middleware settings", "readiness probe constructor settings", "listener and route-topology state"]}, "config_policy": {"change_order": "lexicographic path order", "default_unknown_disposition": "restart_required", "disposition_values": ["live", "restart_required", "ignored"], "dynamic_rules": [{"disposition": "live", "pattern": "providers."}, {"disposition": "live", "pattern": "accounts./"}, {"disposition": "live", "pattern": "model_overrides."}, {"disposition": "live", "pattern": "model_capabilities."}, {"disposition": "live", "pattern": "transcoder."}, {"disposition": "restart_required", "pattern": "cache."}, {"disposition": "live", "pattern": "models."}], "dynamic_values_are_collapsed": true, "field_dispositions": [{"disposition": "live", "path": "accounts"}, {"disposition": "restart_required", "path": "backup.directory"}, {"disposition": "live", "path": "backup.enabled"}, {"disposition": "restart_required", "path": "backup.include_env"}, {"disposition": "live", "path": "backup.interval_s"}, {"disposition": "live", "path": "backup.retain_count"}, {"disposition": "live", "path": "backup.startup_delay_s"}, {"disposition": "restart_required", "path": "dashboard.enabled"}, {"disposition": "restart_required", "path": "dashboard.public"}, {"disposition": "restart_required", "path": "dashboard.refresh_interval_s"}, {"disposition": "live", "path": "dashboard.retain_event_days"}, {"disposition": "live", "path": "dashboard.retain_request_stats_days"}, {"disposition": "restart_required", "path": "dashboard.store_request_content"}, {"disposition": "restart_required", "path": "dashboard.theme"}, {"disposition": "live", "path": "dashboard.themes_dir"}, {"disposition": "restart_required", "path": "database.busy_timeout_ms"}, {"disposition": "restart_required", "path": "database.journal_size_limit"}, {"disposition": "restart_required", "path": "database.path"}, {"disposition": "restart_required", "path": "database.synchronous"}, {"disposition": "restart_required", "path": "database.wal"}, {"disposition": "restart_required", "path": "database.worker_threads"}, {"disposition": "live", "path": "integrations.advertise_base_url"}, {"disposition": "restart_required", "path": "limits.five_hour_microdollars"}, {"disposition": "restart_required", "path": "limits.monthly_microdollars"}, {"disposition": "restart_required", "path": "limits.weekly_microdollars"}, {"disposition": "live", "path": "maintenance.contention_defer_above_lock_wait_p95_ms"}, {"disposition": "live", "path": "maintenance.max_batches_per_tick"}, {"disposition": "live", "path": "maintenance.max_deferral_age_s"}, {"disposition": "live", "path": "maintenance.max_rows_per_batch"}, {"disposition": "live", "path": "maintenance.max_tick_duration_ms"}, {"disposition": "live", "path": "maintenance.p0_max_batches_per_tick"}, {"disposition": "live", "path": "maintenance.p0_max_rows_per_batch"}, {"disposition": "live", "path": "maintenance.p0_max_tick_duration_ms"}, {"disposition": "restart_required", "path": "metrics.aggregate_only"}, {"disposition": "restart_required", "path": "metrics.cleanup_interval_s"}, {"disposition": "restart_required", "path": "metrics.cleanup_max_rows_per_pass"}, {"disposition": "live", "path": "metrics.detailed_span_sample_rate"}, {"disposition": "live", "path": "metrics.dispatch_spans.sample_rate"}, {"disposition": "restart_required", "path": "metrics.dispatch_spans.window_size"}, {"disposition": "restart_required", "path": "metrics.event_loop_lag_enabled"}, {"disposition": "live", "path": "metrics.flush_interval_s"}, {"disposition": "restart_required", "path": "metrics.max_buffered_events"}, {"disposition": "live", "path": "metrics.operational_event_retain_days"}, {"disposition": "restart_required", "path": "metrics.rollup_retain_days"}, {"disposition": "live", "path": "metrics.routing_decision_retain_days"}, {"disposition": "restart_required", "path": "metrics.timeseries_bucket_s"}, {"disposition": "restart_required", "path": "metrics.trace_sample_rate"}, {"disposition": "restart_required", "path": "metrics.write_mode"}, {"disposition": "live", "path": "model_capabilities"}, {"disposition": "restart_required", "path": "model_info.aliases"}, {"disposition": "restart_required", "path": "model_info.conflict_ttl_s"}, {"disposition": "live", "path": "model_info.enabled"}, {"disposition": "restart_required", "path": "model_info.include_in_models_endpoint"}, {"disposition": "restart_required", "path": "model_info.known_ttl_s"}, {"disposition": "restart_required", "path": "model_info.max_models_per_cycle"}, {"disposition": "restart_required", "path": "model_info.overrides"}, {"disposition": "restart_required", "path": "model_info.partial_ttl_s"}, {"disposition": "restart_required", "path": "model_info.refresh_interval_s"}, {"disposition": "restart_required", "path": "model_info.sources"}, {"disposition": "restart_required", "path": "model_info.sparse_new_accelerated_days"}, {"disposition": "restart_required", "path": "model_info.sparse_new_initial_ttl_s"}, {"disposition": "restart_required", "path": "model_info.sparse_new_later_ttl_s"}, {"disposition": "restart_required", "path": "model_info.startup_refresh"}, {"disposition": "restart_required", "path": "model_info.store_raw_observations"}, {"disposition": "live", "path": "model_overrides"}, {"disposition": "live", "path": "model_routers"}, {"disposition": "live", "path": "models.allow_stale_catalog"}, {"disposition": "restart_required", "path": "models.catalog_withdrawal_policy"}, {"disposition": "live", "path": "models.collapse_models"}, {"disposition": "live", "path": "models.expose_mode"}, {"disposition": "live", "path": "models.ping_retain_days"}, {"disposition": "live", "path": "models.refresh_interval_s"}, {"disposition": "live", "path": "models.stale_after_s"}, {"disposition": "restart_required", "path": "models.startup_refresh"}, {"disposition": "restart_required", "path": "network.connect_timeout_s"}, {"disposition": "restart_required", "path": "network.keepalive_expiry_s"}, {"disposition": "restart_required", "path": "network.max_connections"}, {"disposition": "restart_required", "path": "network.max_keepalive"}, {"disposition": "restart_required", "path": "network.read_timeout_s"}, {"disposition": "restart_required", "path": "pricing.catalogs.aliases"}, {"disposition": "restart_required", "path": "pricing.catalogs.opencode_zen.api_key"}, {"disposition": "restart_required", "path": "pricing.catalogs.opencode_zen.base_url"}, {"disposition": "restart_required", "path": "pricing.catalogs.opencode_zen.enabled"}, {"disposition": "restart_required", "path": "pricing.catalogs.opencode_zen.max_entries"}, {"disposition": "restart_required", "path": "pricing.catalogs.opencode_zen.options"}, {"disposition": "restart_required", "path": "pricing.catalogs.opencode_zen.priority"}, {"disposition": "restart_required", "path": "pricing.catalogs.opencode_zen.ttl_seconds"}, {"disposition": "restart_required", "path": "pricing.catalogs.openrouter.api_key"}, {"disposition": "restart_required", "path": "pricing.catalogs.openrouter.base_url"}, {"disposition": "restart_required", "path": "pricing.catalogs.openrouter.enabled"}, {"disposition": "restart_required", "path": "pricing.catalogs.openrouter.max_entries"}, {"disposition": "restart_required", "path": "pricing.catalogs.openrouter.options"}, {"disposition": "restart_required", "path": "pricing.catalogs.openrouter.priority"}, {"disposition": "restart_required", "path": "pricing.catalogs.openrouter.ttl_seconds"}, {"disposition": "restart_required", "path": "pricing.fallback"}, {"disposition": "live", "path": "providers"}, {"disposition": "restart_required", "path": "proxies"}, {"disposition": "restart_required", "path": "readiness_probe.enabled"}, {"disposition": "restart_required", "path": "readiness_probe.freshness_s"}, {"disposition": "restart_required", "path": "readiness_probe.initial_probe"}, {"disposition": "restart_required", "path": "readiness_probe.interval_s"}, {"disposition": "restart_required", "path": "readiness_probe.timeout_s"}, {"disposition": "live", "path": "routing.fairness_epsilon"}, {"disposition": "live", "path": "routing.fairness_mode"}, {"disposition": "live", "path": "routing.fairness_scope"}, {"disposition": "live", "path": "routing.health_penalty"}, {"disposition": "live", "path": "routing.inflight_penalty"}, {"disposition": "live", "path": "routing.local_quota_mode"}, {"disposition": "live", "path": "routing.max_retries_before_stream"}, {"disposition": "live", "path": "routing.near_tie_epsilon"}, {"disposition": "live", "path": "routing.quota_exhausted_cooldown_seconds"}, {"disposition": "live", "path": "routing.randomize_near_ties"}, {"disposition": "live", "path": "routing.strategy"}, {"disposition": "restart_required", "path": "routing.trace.flush_interval_s"}, {"disposition": "live", "path": "routing.trace.guard_cooldown_s"}, {"disposition": "live", "path": "routing.trace.guard_oldest_event_age_s"}, {"disposition": "live", "path": "routing.trace.guard_queue_occupancy_threshold"}, {"disposition": "live", "path": "routing.trace.include_score_components"}, {"disposition": "restart_required", "path": "routing.trace.max_batch_size"}, {"disposition": "live", "path": "routing.trace.mode"}, {"disposition": "restart_required", "path": "routing.trace.queue_capacity"}, {"disposition": "live", "path": "routing.trace.sample_rate"}, {"disposition": "restart_required", "path": "routing.trace.shutdown_flush_timeout_s"}, {"disposition": "live", "path": "routing.trace.skip_above_lock_wait_p95_ms"}, {"disposition": "live", "path": "routing.unknown_request_reservation_microdollars"}, {"disposition": "live", "path": "routing.wire_negotiation.cache_max_entries"}, {"disposition": "live", "path": "routing.wire_negotiation.enabled"}, {"disposition": "live", "path": "routing.wire_negotiation.learned_preference_ttl_s"}, {"disposition": "live", "path": "routing.wire_negotiation.max_concurrent_per_provider"}, {"disposition": "live", "path": "routing.wire_negotiation.min_negotiation_interval_s"}, {"disposition": "live", "path": "routing.wire_negotiation.rejection_cooldown_s"}, {"disposition": "restart_required", "path": "security.allowed_hosts"}, {"disposition": "restart_required", "path": "security.cors_origins"}, {"disposition": "live", "path": "security.persist_redacted_error_detail"}, {"disposition": "restart_required", "path": "security.redact_headers"}, {"disposition": "restart_required", "path": "security.trusted_proxies"}, {"disposition": "restart_required", "path": "server.access_log"}, {"disposition": "restart_required", "path": "server.api_key"}, {"disposition": "restart_required", "path": "server.api_key_env"}, {"disposition": "restart_required", "path": "server.host"}, {"disposition": "restart_required", "path": "server.log_level"}, {"disposition": "live", "path": "server.max_request_body_bytes"}, {"disposition": "restart_required", "path": "server.port"}, {"disposition": "restart_required", "path": "server.threads"}, {"disposition": "live", "path": "transcoder"}, {"disposition": "restart_required", "path": "update_checker.enabled"}, {"disposition": "restart_required", "path": "upstream.base_url"}, {"disposition": "restart_required", "path": "upstream.connect_timeout_s"}, {"disposition": "restart_required", "path": "upstream.keepalive_timeout_s"}, {"disposition": "restart_required", "path": "upstream.max_connections"}, {"disposition": "restart_required", "path": "upstream.max_keepalive"}, {"disposition": "restart_required", "path": "upstream.pool_timeout_s"}, {"disposition": "restart_required", "path": "upstream.read_timeout_s"}, {"disposition": "restart_required", "path": "upstream.write_timeout_s"}], "free_text_redaction_probe": "Authorization: ", "free_text_redaction_token": "", "ignored_paths": [], "runtime_tracked_default_paths": ["backup.directory", "backup.enabled", "backup.include_env", "backup.interval_s", "backup.retain_count", "backup.startup_delay_s", "dashboard.enabled", "dashboard.public", "dashboard.refresh_interval_s", "dashboard.retain_event_days", "dashboard.retain_request_stats_days", "dashboard.store_request_content", "dashboard.theme", "dashboard.themes_dir", "database.busy_timeout_ms", "database.journal_size_limit", "database.path", "database.synchronous", "database.wal", "database.worker_threads", "integrations.advertise_base_url", "limits.five_hour_microdollars", "limits.monthly_microdollars", "limits.weekly_microdollars", "maintenance.contention_defer_above_lock_wait_p95_ms", "maintenance.max_batches_per_tick", "maintenance.max_deferral_age_s", "maintenance.max_rows_per_batch", "maintenance.max_tick_duration_ms", "maintenance.p0_max_batches_per_tick", "maintenance.p0_max_rows_per_batch", "maintenance.p0_max_tick_duration_ms", "metrics.aggregate_only", "metrics.cleanup_interval_s", "metrics.cleanup_max_rows_per_pass", "metrics.detailed_span_sample_rate", "metrics.dispatch_spans.sample_rate", "metrics.dispatch_spans.window_size", "metrics.event_loop_lag_enabled", "metrics.flush_interval_s", "metrics.max_buffered_events", "metrics.operational_event_retain_days", "metrics.rollup_retain_days", "metrics.routing_decision_retain_days", "metrics.timeseries_bucket_s", "metrics.trace_sample_rate", "metrics.write_mode", "model_capabilities", "model_info.aliases", "model_info.conflict_ttl_s", "model_info.enabled", "model_info.include_in_models_endpoint", "model_info.known_ttl_s", "model_info.max_models_per_cycle", "model_info.overrides", "model_info.partial_ttl_s", "model_info.refresh_interval_s", "model_info.sources", "model_info.sparse_new_accelerated_days", "model_info.sparse_new_initial_ttl_s", "model_info.sparse_new_later_ttl_s", "model_info.startup_refresh", "model_info.store_raw_observations", "model_overrides", "model_routers", "models.allow_stale_catalog", "models.catalog_withdrawal_policy", "models.collapse_models", "models.expose_mode", "models.ping_retain_days", "models.refresh_interval_s", "models.stale_after_s", "models.startup_refresh", "network.connect_timeout_s", "network.keepalive_expiry_s", "network.max_connections", "network.max_keepalive", "network.read_timeout_s", "pricing.catalogs.aliases", "pricing.catalogs.opencode_zen.api_key", "pricing.catalogs.opencode_zen.base_url", "pricing.catalogs.opencode_zen.enabled", "pricing.catalogs.opencode_zen.max_entries", "pricing.catalogs.opencode_zen.options", "pricing.catalogs.opencode_zen.priority", "pricing.catalogs.opencode_zen.ttl_seconds", "pricing.catalogs.openrouter.api_key", "pricing.catalogs.openrouter.base_url", "pricing.catalogs.openrouter.enabled", "pricing.catalogs.openrouter.max_entries", "pricing.catalogs.openrouter.options", "pricing.catalogs.openrouter.priority", "pricing.catalogs.openrouter.ttl_seconds", "pricing.fallback", "proxies", "readiness_probe.enabled", "readiness_probe.freshness_s", "readiness_probe.initial_probe", "readiness_probe.interval_s", "readiness_probe.timeout_s", "routing.fairness_epsilon", "routing.fairness_mode", "routing.fairness_scope", "routing.health_penalty", "routing.inflight_penalty", "routing.local_quota_mode", "routing.max_retries_before_stream", "routing.near_tie_epsilon", "routing.quota_exhausted_cooldown_seconds", "routing.randomize_near_ties", "routing.strategy", "routing.trace.flush_interval_s", "routing.trace.guard_cooldown_s", "routing.trace.guard_oldest_event_age_s", "routing.trace.guard_queue_occupancy_threshold", "routing.trace.include_score_components", "routing.trace.max_batch_size", "routing.trace.mode", "routing.trace.queue_capacity", "routing.trace.sample_rate", "routing.trace.shutdown_flush_timeout_s", "routing.trace.skip_above_lock_wait_p95_ms", "routing.unknown_request_reservation_microdollars", "routing.wire_negotiation.cache_max_entries", "routing.wire_negotiation.enabled", "routing.wire_negotiation.learned_preference_ttl_s", "routing.wire_negotiation.max_concurrent_per_provider", "routing.wire_negotiation.min_negotiation_interval_s", "routing.wire_negotiation.rejection_cooldown_s", "security.allowed_hosts", "security.cors_origins", "security.persist_redacted_error_detail", "security.redact_headers", "security.trusted_proxies", "server.access_log", "server.api_key", "server.api_key_env", "server.host", "server.log_level", "server.max_request_body_bytes", "server.port", "server.threads", "transcoder", "update_checker.enabled", "upstream.base_url", "upstream.connect_timeout_s", "upstream.keepalive_timeout_s", "upstream.max_connections", "upstream.max_keepalive", "upstream.pool_timeout_s", "upstream.read_timeout_s", "upstream.write_timeout_s"], "secret_display_token": "", "secret_field_names": ["api_key", "api_key_env", "proxy_url", "proxy_url_env"], "top_level_config_sections": ["accounts", "backup", "dashboard", "database", "integrations", "limits", "maintenance", "metrics", "model_capabilities", "model_info", "model_overrides", "model_routers", "models", "network", "pricing", "providers", "proxies", "readiness_probe", "routing", "security", "server", "transcoder", "update_checker", "upstream"], "unknown_path_probe": "restart_required"}, "lifecycle": {"candidate": {"abort": {"abort_is_idempotent": true, "cleanup_errors_preserve_primary_failure": true, "closed_order": ["supervisor", "outbound_manager", "client_pool"], "primary_failure_stage": "build", "process_owned_resources_touched": false, "registered_order": ["client_pool", "outbound_manager", "supervisor"]}, "legal_transitions": [["building", "prepared"], ["prepared", "transferred"], ["building", "aborted"], ["prepared", "aborted"]], "states": ["building", "prepared", "transferred", "aborted"]}, "close_order": ["generation supervisor stop_all", "generation cleanup callbacks", "finalization supervisor shutdown", "provider client pool close", "outbound client manager aclose"], "generation_slot": {"legal_transitions": [["active", "retiring"], ["retiring", "closing"], ["closing", "closed"], ["closing", "failed_close"]], "retirement_ready_when": ["active_leases == 0", "terminal_references == 0"], "states": ["active", "retiring", "closing", "closed", "failed_close"]}, "lease": {"claim_and_increment_are_one_critical_section": true, "claim_predicate": "shutdown_in_progress OR (NOT lease_admission_gated AND active_slot_accepting)", "claim_timeout_s": 30.0, "release_is_exactly_once": true, "stream_release_boundary": "iterator finally after handler return"}, "pending_swap": {"commit_effects": ["candidate_becomes_active", "old_slot_stops_accepting_leases", "publication_epoch_increments", "lease_admission_reopens"], "legal_transitions": [["prepared", "staged"], ["staged", "committed"], ["staged", "rolled_back"], ["committed", "finalized"]], "rollback_effects": ["old_slot_restored", "candidate_slot_dropped", "lease_admission_reopens", "candidate_requires_abort"], "stage_effects": ["lease_admission_gated", "candidate_slot_non_accepting", "active_slot_unchanged"], "states": ["prepared", "staged", "committed", "rolled_back", "finalized"]}, "publication_gate": {"forbidden_inside_gate": ["provider/network calls", "catalog refresh", "backup", "unbounded work"], "inside_gate": ["stage candidate", "apply bounded mandatory acceptance changes", "commit runtime pointer", "commit task-spec state"], "pre_gate_work": ["config parsing and validation", "digest check and diff", "candidate construction", "persistence delta preparation", "task diff preflight"]}, "shutdown": {"adopts_unfinalized_committed_swap_old_slot": true, "forced_close_abandons_terminal_references_only_on_process_shutdown": true, "live_rehash_preserves_accepted_work": true, "manager_join_deadline_s": 10.0, "retirement_drain_deadline_s": 5.0, "stops_new_publication_and_acquisition": true}, "transaction": {"acceptance_states": ["not_accepted", "persistence_committed_runtime_pending", "accepted"], "accepted_reload_cannot_abort": true, "states": ["created", "validated", "diffed", "candidate_prepared", "persistence_prepared", "process_transitions_prepared", "process_transitions_preflighted", "commit_started", "runtime_staged", "runtime_swap_committed", "runtime_published", "process_transitions_applied", "persistence_committed", "observable_state_updated", "retirement_scheduled", "completed", "aborting", "aborted", "compensation_failed"], "transitions": {"aborted": [], "aborting": ["aborted", "compensation_failed"], "candidate_prepared": ["aborting", "persistence_prepared"], "commit_started": ["aborting", "runtime_published", "runtime_staged"], "compensation_failed": [], "completed": [], "created": ["aborting", "validated"], "diffed": ["aborting", "candidate_prepared"], "observable_state_updated": ["aborting", "retirement_scheduled"], "persistence_committed": ["aborting", "observable_state_updated"], "persistence_prepared": ["aborting", "process_transitions_prepared"], "process_transitions_applied": ["aborting", "persistence_committed"], "process_transitions_preflighted": ["aborting", "commit_started"], "process_transitions_prepared": ["aborting", "process_transitions_preflighted"], "retirement_scheduled": ["aborting", "completed"], "runtime_published": ["aborting", "process_transitions_applied"], "runtime_staged": ["aborting", "runtime_swap_committed"], "runtime_swap_committed": ["aborting", "process_transitions_applied"], "validated": ["aborting", "diffed"]}}}, "normalization": {"excluded": ["API keys", "proxy credentials", "raw config values", "request/provider bodies"], "normalized": ["timestamps", "runtime identifiers"], "preserved": ["state names and order", "booleans", "changed/restart path ordering", "task names and specs", "generation increments", "retirement pending state", "error category distinctions"]}, "oracle_modules": ["eggpool.runtime_manager", "eggpool.generation_factory", "eggpool.config_reload_policy", "eggpool.reload_transaction", "eggpool.runtime_task_inventory", "eggpool.runtime_tasks", "eggpool.reload_diagnostics", "eggpool.app", "eggpool.cli_rehash_helper"], "ownership_inventory": [{"authority": "ProcessRuntime.db/stats_db", "lifetime": "process", "name": "database_and_repositories", "ownership": "process-owned"}, {"authority": "RuntimeGeneration.client_pool", "lifetime": "generation retirement", "name": "provider_client_pool", "ownership": "generation-owned"}, {"authority": "RuntimeGeneration.outbound_manager", "lifetime": "generation retirement", "name": "outbound_client_manager", "ownership": "generation-owned"}, {"authority": "RuntimeGeneration.registry", "lifetime": "generation retirement", "name": "account_registry", "ownership": "generation-owned"}, {"authority": "RuntimeGeneration.catalog", "lifetime": "generation retirement", "name": "catalog_service", "ownership": "generation-owned"}, {"authority": "RuntimeGeneration.router/coordinator", "lifetime": "generation retirement", "name": "router_and_coordinator", "ownership": "generation-owned"}, {"authority": "RuntimeGeneration.finalization_supervisor", "lifetime": "after leases and terminal references drain", "name": "finalization_supervisor", "ownership": "generation-owned"}, {"authority": "ProcessRuntime.model_router_affinity", "lifetime": "process; revalidated per generation", "name": "model_router_affinity", "ownership": "process-owned"}, {"authority": "ProcessRuntime.wire_profile_resolver", "lifetime": "process; fingerprint-qualified", "name": "wire_profile_resolver", "ownership": "process-owned"}, {"authority": "RuntimeGeneration.supervisor", "lifetime": "generation retirement", "name": "generation_task_supervisor", "ownership": "generation-owned"}, {"authority": "ProcessRuntime.process_supervisor", "lifetime": "process singleton", "name": "process_task_supervisor", "ownership": "process-owned"}, {"authority": "startup lifecycle entry point", "lifetime": "one bounded startup pass", "name": "crash_reconciler", "ownership": "process-owned/startup-only"}, {"authority": "RuntimeMetricsService and reload metadata", "lifetime": "process; bounded snapshots", "name": "runtime_metrics_and_reload_diagnostics", "ownership": "process-owned"}, {"authority": "listener, middleware, route topology, auth, DB constructor", "lifetime": "process construction", "name": "server_constructor_state", "ownership": "startup-only/restart-required"}, {"authority": "RuntimeGenerationCandidate", "lifetime": "building/prepared until transfer or abort", "name": "candidate_resources_before_publication", "ownership": "candidate-owned"}, {"authority": "GenerationLease", "lifetime": "finite completion or stream disconnect", "name": "request_and_stream_lease", "ownership": "request/lease-owned"}, {"authority": "terminal reference on generation slot", "lifetime": "durable/runtime convergence", "name": "retained_finalization", "ownership": "retained-finalization-owned"}], "reload": {"cases": [{"accepted": true, "active_generation_unchanged": true, "category": "success_noop", "changed_sections": [], "generation_delta": 0, "legacy_wire_stage": "commit", "name": "identical_digest_noop", "publication_occurred": false, "restart_required_paths": [], "retirement_pending": false, "terminal_stage": "idle"}, {"accepted": true, "active_generation_unchanged": false, "category": "success_committed", "changed_sections": ["server"], "generation_delta": 1, "legacy_wire_stage": "retirement", "name": "valid_live_only_change", "publication_occurred": true, "restart_required_paths": [], "retirement_pending": "true_if_old_work_remains", "terminal_stage": "retirement"}, {"accepted": false, "active_generation_unchanged": true, "category": "rejected_restart_required", "changed_sections": ["server"], "generation_delta": 0, "legacy_wire_stage": "diff", "name": "restart_required_change", "publication_occurred": false, "restart_required_paths": ["server.port"], "retirement_pending": false, "terminal_stage": "diff"}, {"accepted": false, "active_generation_unchanged": true, "category": "rejected_restart_required", "changed_sections": ["server"], "generation_delta": 0, "legacy_wire_stage": "diff", "name": "mixed_live_and_restart_required_change", "publication_occurred": false, "restart_required_paths": ["server.port"], "retirement_pending": false, "terminal_stage": "diff"}, {"accepted": false, "active_generation_unchanged": true, "category": "rejected_validation", "changed_sections": [], "generation_delta": 0, "legacy_wire_stage": "validation", "name": "invalid_config", "publication_occurred": false, "restart_required_paths": [], "retirement_pending": false, "terminal_stage": "validation"}, {"accepted": false, "active_generation_unchanged": true, "category": "rejected_validation", "changed_sections": [], "generation_delta": 0, "legacy_wire_stage": "digest_check", "name": "expected_digest_mismatch", "publication_occurred": false, "restart_required_paths": [], "retirement_pending": false, "terminal_stage": "validation"}, {"accepted": false, "active_generation_unchanged": true, "category": "rejected_validation", "changed_sections": [], "generation_delta": 0, "legacy_wire_stage": "preparation", "name": "stale_caller_generation", "publication_occurred": false, "restart_required_paths": [], "retirement_pending": false, "terminal_stage": "preparation"}, {"accepted": false, "active_generation_unchanged": true, "category": "failed_candidate_prepare", "changed_sections": [], "generation_delta": 0, "legacy_wire_stage": "preparation", "name": "candidate_construction_failure", "publication_occurred": false, "restart_required_paths": [], "retirement_pending": false, "terminal_stage": "preparation"}, {"accepted": false, "active_generation_unchanged": true, "category": "failed_process_transition_prepare", "changed_sections": [], "generation_delta": 0, "legacy_wire_stage": "preparation", "name": "preflight_process_transition_failure", "publication_occurred": false, "restart_required_paths": [], "retirement_pending": false, "terminal_stage": "preparation"}, {"accepted": false, "active_generation_unchanged": true, "category": "aborted_cancelled", "changed_sections": [], "generation_delta": 0, "legacy_wire_stage": "preparation", "name": "cancellation_before_acceptance", "publication_occurred": false, "restart_required_paths": [], "retirement_pending": false, "terminal_stage": "preparation"}, {"accepted": true, "active_generation_unchanged": false, "category": "post_commit_finalization_pending", "changed_sections": [], "generation_delta": 1, "legacy_wire_stage": "retirement", "name": "cancellation_during_or_after_acceptance", "publication_occurred": true, "restart_required_paths": [], "retirement_pending": true, "terminal_stage": "retirement"}, {"accepted": true, "active_generation_unchanged": false, "category": "post_commit_finalization_pending", "changed_sections": [], "generation_delta": 1, "legacy_wire_stage": "retirement", "name": "successful_publication_retirement_pending", "publication_occurred": true, "restart_required_paths": [], "retirement_pending": true, "terminal_stage": "retirement"}, {"accepted": true, "active_generation_unchanged": false, "category": "success_committed", "changed_sections": [], "generation_delta": 1, "legacy_wire_stage": "retirement", "name": "successful_publication_already_drained", "publication_occurred": true, "restart_required_paths": [], "retirement_pending": false, "terminal_stage": "retirement"}], "classification_probes": {"retry_pending": "post_commit_finalization_pending", "success_committed": "success_committed", "success_noop": "success_noop", "validation": "rejected_validation"}, "diff_observations": {"identical": {"changes": [], "dispositions": [], "live": [], "paths": [], "restart_required": [], "sections": []}, "live_only": {"changes": [{"disposition": "live", "new_display": "10485761", "old_display": "10485760", "path": "server.max_request_body_bytes", "secret": false, "section": "server"}], "dispositions": ["live"], "live": ["server.max_request_body_bytes"], "paths": ["server.max_request_body_bytes"], "restart_required": [], "sections": ["server"]}, "mixed": {"changes": [{"disposition": "live", "new_display": "10485761", "old_display": "10485760", "path": "server.max_request_body_bytes", "secret": false, "section": "server"}, {"disposition": "restart_required", "new_display": "11301", "old_display": "11300", "path": "server.port", "secret": false, "section": "server"}], "dispositions": ["live", "restart_required"], "live": ["server.max_request_body_bytes"], "paths": ["server.max_request_body_bytes", "server.port"], "restart_required": ["server.port"], "sections": ["server"]}, "restart_only": {"changes": [{"disposition": "restart_required", "new_display": "11301", "old_display": "11300", "path": "server.port", "secret": false, "section": "server"}], "dispositions": ["restart_required"], "live": [], "paths": ["server.port"], "restart_required": ["server.port"], "sections": ["server"]}, "secret": {"changes": [{"disposition": "restart_required", "new_display": "", "old_display": "", "path": "pricing.catalogs.openrouter.api_key", "secret": true, "section": "pricing"}], "dispositions": ["restart_required"], "live": [], "paths": ["pricing.catalogs.openrouter.api_key"], "restart_required": ["pricing.catalogs.openrouter.api_key"], "sections": ["pricing"]}}, "generation_id_rule": "monotonic; no-op and rejected reloads do not publish", "legacy_stages": ["validation", "digest_check", "diff", "preparation", "reconciliation", "commit", "retirement"], "result_categories": ["success_committed", "success_noop", "success_ignored_only", "rejected_busy", "rejected_validation", "rejected_restart_required", "failed_candidate_prepare", "failed_persistence_prepare", "failed_process_transition_prepare", "failed_commit", "failed_publication", "failed_process_transition_apply", "failed_persistence_commit", "failed_runtime_stage", "failed_runtime_stage_rollback", "post_commit_finalization_pending", "retirement_schedule_failed", "shutdown_adopted", "aborted_cancelled", "aborted_shutdown", "compensation_failed", "internal_error"], "retirement_pending_rule": "accepted publication may be pending while old leases or terminal references drain", "terminal_stages": ["validation", "diff", "preparation", "reconciliation", "commit", "retirement", "idle"]}, "schema_version": "m8-runtime-lifecycle-r001-observations/v1", "tasks": {"conditions": {"automatic_backup": "backup.enabled and interval_s > 0 enables", "catalog_refresh": "models.refresh_interval_s <= 0 disables", "generation_leased_callbacks": "acquire active generation for every tick", "metrics_flush": "metrics.write_mode == immediate disables", "retention_cleanup": "always registered; reads active generation config per tick", "update_checker": "startup-only process task; update_checker.enabled and outbound handle required"}, "config_variants": {"automatic_backup_enabled": [{"enabled": true, "initial_delay_s": null, "interval_s": 300.0, "name": "catalog_refresh", "run_immediately": false}, {"enabled": true, "initial_delay_s": null, "interval_s": 86400.0, "name": "retention_cleanup", "run_immediately": false}, {"enabled": true, "initial_delay_s": null, "interval_s": 60.0, "name": "checkpoint", "run_immediately": true}, {"enabled": true, "initial_delay_s": 5.0, "interval_s": 120.0, "name": "metrics_flush", "run_immediately": false}, {"enabled": false, "initial_delay_s": null, "interval_s": 86400.0, "name": "update_checker", "run_immediately": true}, {"enabled": true, "initial_delay_s": 5.0, "interval_s": 600.0, "name": "automatic_backup", "run_immediately": false}], "catalog_refresh_disabled": [{"enabled": false, "initial_delay_s": null, "interval_s": 0.0, "name": "catalog_refresh", "run_immediately": false}, {"enabled": true, "initial_delay_s": null, "interval_s": 86400.0, "name": "retention_cleanup", "run_immediately": false}, {"enabled": true, "initial_delay_s": null, "interval_s": 60.0, "name": "checkpoint", "run_immediately": true}, {"enabled": true, "initial_delay_s": 5.0, "interval_s": 120.0, "name": "metrics_flush", "run_immediately": false}, {"enabled": false, "initial_delay_s": null, "interval_s": 86400.0, "name": "update_checker", "run_immediately": true}, {"enabled": false, "initial_delay_s": 300.0, "interval_s": 86400.0, "name": "automatic_backup", "run_immediately": false}], "default": [{"enabled": true, "initial_delay_s": null, "interval_s": 300.0, "name": "catalog_refresh", "run_immediately": false}, {"enabled": true, "initial_delay_s": null, "interval_s": 86400.0, "name": "retention_cleanup", "run_immediately": false}, {"enabled": true, "initial_delay_s": null, "interval_s": 60.0, "name": "checkpoint", "run_immediately": true}, {"enabled": true, "initial_delay_s": 5.0, "interval_s": 120.0, "name": "metrics_flush", "run_immediately": false}, {"enabled": false, "initial_delay_s": null, "interval_s": 86400.0, "name": "update_checker", "run_immediately": true}, {"enabled": false, "initial_delay_s": 300.0, "interval_s": 86400.0, "name": "automatic_backup", "run_immediately": false}], "metrics_immediate": [{"enabled": true, "initial_delay_s": null, "interval_s": 300.0, "name": "catalog_refresh", "run_immediately": false}, {"enabled": true, "initial_delay_s": null, "interval_s": 86400.0, "name": "retention_cleanup", "run_immediately": false}, {"enabled": true, "initial_delay_s": null, "interval_s": 60.0, "name": "checkpoint", "run_immediately": true}, {"enabled": false, "initial_delay_s": 5.0, "interval_s": 120.0, "name": "metrics_flush", "run_immediately": false}, {"enabled": false, "initial_delay_s": null, "interval_s": 86400.0, "name": "update_checker", "run_immediately": true}, {"enabled": false, "initial_delay_s": 300.0, "interval_s": 86400.0, "name": "automatic_backup", "run_immediately": false}], "update_checker_enabled": [{"enabled": true, "initial_delay_s": null, "interval_s": 300.0, "name": "catalog_refresh", "run_immediately": false}, {"enabled": true, "initial_delay_s": null, "interval_s": 86400.0, "name": "retention_cleanup", "run_immediately": false}, {"enabled": true, "initial_delay_s": null, "interval_s": 60.0, "name": "checkpoint", "run_immediately": true}, {"enabled": true, "initial_delay_s": 5.0, "interval_s": 120.0, "name": "metrics_flush", "run_immediately": false}, {"enabled": true, "initial_delay_s": null, "interval_s": 86400.0, "name": "update_checker", "run_immediately": true}, {"enabled": false, "initial_delay_s": 300.0, "interval_s": 86400.0, "name": "automatic_backup", "run_immediately": false}]}, "inventory": [{"callback_kind": "catalog_refresh", "default_enabled": true, "description": "Periodically refresh the model catalog from enabled accounts", "generation_dependencies": ["catalog", "model_info", "registry", "health_manager"], "initial_delay_s": null, "interval_s": 300.0, "name": "catalog_refresh", "ownership": "generation_leased", "process_dependencies": ["db"], "reloadable_fields": ["models.refresh_interval_s"], "run_immediately": false, "timeout_s": null}, {"callback_kind": "retention_cleanup", "default_enabled": true, "description": "Clean up old requests, events, pings, and expired reservations", "generation_dependencies": ["router"], "initial_delay_s": null, "interval_s": 86400.0, "name": "retention_cleanup", "ownership": "generation_leased", "process_dependencies": ["db"], "reloadable_fields": ["dashboard.retain_request_stats_days", "dashboard.retain_event_days", "metrics.operational_event_retain_days", "metrics.routing_decision_retain_days", "models.ping_retain_days"], "run_immediately": false, "timeout_s": null}, {"callback_kind": "checkpoint", "default_enabled": true, "description": "Periodic SQLite WAL checkpoint", "generation_dependencies": [], "initial_delay_s": null, "interval_s": 60.0, "name": "checkpoint", "ownership": "process", "process_dependencies": ["db"], "reloadable_fields": [], "run_immediately": true, "timeout_s": null}, {"callback_kind": "metrics_flush", "default_enabled": true, "description": "Flush buffered metrics analytics to SQLite", "generation_dependencies": [], "initial_delay_s": 5.0, "interval_s": 30.0, "name": "metrics_flush", "ownership": "process", "process_dependencies": ["metrics_coalescer"], "reloadable_fields": ["metrics.write_mode", "metrics.flush_interval_s"], "run_immediately": false, "timeout_s": null}, {"callback_kind": "update_checker", "default_enabled": true, "description": "Periodically check PyPI for new EggPool releases", "generation_dependencies": [], "initial_delay_s": null, "interval_s": 86400.0, "name": "update_checker", "ownership": "process", "process_dependencies": ["outbound_manager"], "reloadable_fields": [], "run_immediately": true, "timeout_s": null}, {"callback_kind": "automatic_backup", "default_enabled": true, "description": "Create periodic backup archives of config and database", "generation_dependencies": [], "initial_delay_s": 300.0, "interval_s": 86400.0, "name": "automatic_backup", "ownership": "process", "process_dependencies": ["db"], "reloadable_fields": ["backup.enabled", "backup.interval_s", "backup.startup_delay_s", "backup.retain_count"], "run_immediately": false, "timeout_s": null}], "names_unique": true, "singleton_process_tasks": ["checkpoint", "metrics_flush", "update_checker", "automatic_backup"]}} \ No newline at end of file +{"active_authority":{"active_state_after_shutdown":"no active generation is acquirable","app_state_rule":"mirrors are compatibility snapshots, never authority for leased request work","generation_dataclass_fields":["generation_id","config","config_digest","registry","catalog","router","coordinator","client_pool","outbound_manager","health_manager","cost_calculator","transcoder_policy","dispatch_overhead_recorder","dispatch_span_recorder","account_backoff_repo","stats_service","supervisor","routing_trace_guard","routing_trace_writer","created_at_monotonic","created_at_epoch","effects_applier","model_quarantine","finalization_supervisor","model_info","model_router_registry","local_pre_upstream_recorder","stream_diagnostics","immutable_request_state","generation_cleanup_callbacks"],"generation_owned_app_state_mirrors":["account_backoff_repo","catalog","client_pool","cost_calculator","dashboard_telemetry","dispatch_overhead_recorder","dispatch_span_recorder","health_manager","httpx_client","local_pre_upstream_recorder","metrics_coalescer","model_info","model_router_registry","outbound_manager","registry","router","routing_trace_guard","runtime_metrics","stats","stream_diagnostics","supervisor","task_monitor","transcoder_policy"],"live_request_authority":["providers","accounts","model_routers","model_overrides","model_capabilities","server.max_request_body_bytes","models.expose_mode","models.collapse_models","models.stale_after_s","models.allow_stale_catalog","routing.* request/retry/health/fairness settings","routing.wire_negotiation.*","transcoder.*","security.persist_redacted_error_detail","task intervals and retention settings consumed per tick"],"process_runtime_fields":["db","stats_db","config_path","metrics_coalescer","process_supervisor","task_spec_version","last_task_transition","routing_trace_writer","maintenance_state","event_loop_lag_monitor","readiness_probe","wire_profile_resolver","model_router_affinity","model_router_metrics","model_info"],"request_generation_rule":"one generation from admission through finite terminal registration or stream disconnect","restart_required_constructor_authority":["server.host/port/api_key/api_key_env/log_level/access_log/threads","upstream transport settings","database constructor and path settings","dashboard route topology and public/auth mode","security middleware settings","readiness probe constructor settings","listener and route-topology state"]},"config_policy":{"change_order":"lexicographic path order","default_unknown_disposition":"restart_required","disposition_values":["live","restart_required","ignored"],"dynamic_rules":[{"disposition":"live","pattern":"providers."},{"disposition":"live","pattern":"accounts./"},{"disposition":"live","pattern":"model_overrides."},{"disposition":"live","pattern":"model_capabilities."},{"disposition":"live","pattern":"transcoder."},{"disposition":"restart_required","pattern":"cache."},{"disposition":"live","pattern":"models."}],"dynamic_values_are_collapsed":true,"field_dispositions":[{"disposition":"live","path":"accounts"},{"disposition":"restart_required","path":"backup.directory"},{"disposition":"live","path":"backup.enabled"},{"disposition":"restart_required","path":"backup.include_env"},{"disposition":"live","path":"backup.interval_s"},{"disposition":"live","path":"backup.retain_count"},{"disposition":"live","path":"backup.startup_delay_s"},{"disposition":"restart_required","path":"dashboard.enabled"},{"disposition":"restart_required","path":"dashboard.public"},{"disposition":"restart_required","path":"dashboard.refresh_interval_s"},{"disposition":"live","path":"dashboard.retain_event_days"},{"disposition":"live","path":"dashboard.retain_request_stats_days"},{"disposition":"restart_required","path":"dashboard.store_request_content"},{"disposition":"restart_required","path":"dashboard.theme"},{"disposition":"live","path":"dashboard.themes_dir"},{"disposition":"restart_required","path":"database.busy_timeout_ms"},{"disposition":"restart_required","path":"database.journal_size_limit"},{"disposition":"restart_required","path":"database.path"},{"disposition":"restart_required","path":"database.synchronous"},{"disposition":"restart_required","path":"database.wal"},{"disposition":"restart_required","path":"database.worker_threads"},{"disposition":"live","path":"integrations.advertise_base_url"},{"disposition":"restart_required","path":"limits.five_hour_microdollars"},{"disposition":"restart_required","path":"limits.monthly_microdollars"},{"disposition":"restart_required","path":"limits.weekly_microdollars"},{"disposition":"live","path":"maintenance.contention_defer_above_lock_wait_p95_ms"},{"disposition":"live","path":"maintenance.max_batches_per_tick"},{"disposition":"live","path":"maintenance.max_deferral_age_s"},{"disposition":"live","path":"maintenance.max_rows_per_batch"},{"disposition":"live","path":"maintenance.max_tick_duration_ms"},{"disposition":"live","path":"maintenance.p0_max_batches_per_tick"},{"disposition":"live","path":"maintenance.p0_max_rows_per_batch"},{"disposition":"live","path":"maintenance.p0_max_tick_duration_ms"},{"disposition":"restart_required","path":"metrics.aggregate_only"},{"disposition":"restart_required","path":"metrics.cleanup_interval_s"},{"disposition":"restart_required","path":"metrics.cleanup_max_rows_per_pass"},{"disposition":"live","path":"metrics.detailed_span_sample_rate"},{"disposition":"live","path":"metrics.dispatch_spans.sample_rate"},{"disposition":"restart_required","path":"metrics.dispatch_spans.window_size"},{"disposition":"restart_required","path":"metrics.event_loop_lag_enabled"},{"disposition":"live","path":"metrics.flush_interval_s"},{"disposition":"restart_required","path":"metrics.max_buffered_events"},{"disposition":"live","path":"metrics.operational_event_retain_days"},{"disposition":"restart_required","path":"metrics.rollup_retain_days"},{"disposition":"live","path":"metrics.routing_decision_retain_days"},{"disposition":"restart_required","path":"metrics.timeseries_bucket_s"},{"disposition":"restart_required","path":"metrics.trace_sample_rate"},{"disposition":"restart_required","path":"metrics.write_mode"},{"disposition":"live","path":"model_capabilities"},{"disposition":"restart_required","path":"model_info.aliases"},{"disposition":"restart_required","path":"model_info.conflict_ttl_s"},{"disposition":"live","path":"model_info.enabled"},{"disposition":"restart_required","path":"model_info.include_in_models_endpoint"},{"disposition":"restart_required","path":"model_info.known_ttl_s"},{"disposition":"restart_required","path":"model_info.max_models_per_cycle"},{"disposition":"restart_required","path":"model_info.overrides"},{"disposition":"restart_required","path":"model_info.partial_ttl_s"},{"disposition":"restart_required","path":"model_info.refresh_interval_s"},{"disposition":"restart_required","path":"model_info.sources"},{"disposition":"restart_required","path":"model_info.sparse_new_accelerated_days"},{"disposition":"restart_required","path":"model_info.sparse_new_initial_ttl_s"},{"disposition":"restart_required","path":"model_info.sparse_new_later_ttl_s"},{"disposition":"restart_required","path":"model_info.startup_refresh"},{"disposition":"restart_required","path":"model_info.store_raw_observations"},{"disposition":"live","path":"model_overrides"},{"disposition":"live","path":"model_routers"},{"disposition":"live","path":"models.allow_stale_catalog"},{"disposition":"restart_required","path":"models.catalog_withdrawal_policy"},{"disposition":"live","path":"models.collapse_models"},{"disposition":"live","path":"models.expose_mode"},{"disposition":"live","path":"models.ping_retain_days"},{"disposition":"live","path":"models.refresh_interval_s"},{"disposition":"live","path":"models.stale_after_s"},{"disposition":"restart_required","path":"models.startup_refresh"},{"disposition":"restart_required","path":"network.connect_timeout_s"},{"disposition":"restart_required","path":"network.keepalive_expiry_s"},{"disposition":"restart_required","path":"network.max_connections"},{"disposition":"restart_required","path":"network.max_keepalive"},{"disposition":"restart_required","path":"network.read_timeout_s"},{"disposition":"restart_required","path":"pricing.catalogs.aliases"},{"disposition":"restart_required","path":"pricing.catalogs.opencode_zen.api_key"},{"disposition":"restart_required","path":"pricing.catalogs.opencode_zen.base_url"},{"disposition":"restart_required","path":"pricing.catalogs.opencode_zen.enabled"},{"disposition":"restart_required","path":"pricing.catalogs.opencode_zen.max_entries"},{"disposition":"restart_required","path":"pricing.catalogs.opencode_zen.options"},{"disposition":"restart_required","path":"pricing.catalogs.opencode_zen.priority"},{"disposition":"restart_required","path":"pricing.catalogs.opencode_zen.ttl_seconds"},{"disposition":"restart_required","path":"pricing.catalogs.openrouter.api_key"},{"disposition":"restart_required","path":"pricing.catalogs.openrouter.base_url"},{"disposition":"restart_required","path":"pricing.catalogs.openrouter.enabled"},{"disposition":"restart_required","path":"pricing.catalogs.openrouter.max_entries"},{"disposition":"restart_required","path":"pricing.catalogs.openrouter.options"},{"disposition":"restart_required","path":"pricing.catalogs.openrouter.priority"},{"disposition":"restart_required","path":"pricing.catalogs.openrouter.ttl_seconds"},{"disposition":"restart_required","path":"pricing.fallback"},{"disposition":"live","path":"providers"},{"disposition":"restart_required","path":"proxies"},{"disposition":"restart_required","path":"readiness_probe.enabled"},{"disposition":"restart_required","path":"readiness_probe.freshness_s"},{"disposition":"restart_required","path":"readiness_probe.initial_probe"},{"disposition":"restart_required","path":"readiness_probe.interval_s"},{"disposition":"restart_required","path":"readiness_probe.timeout_s"},{"disposition":"live","path":"routing.fairness_epsilon"},{"disposition":"live","path":"routing.fairness_mode"},{"disposition":"live","path":"routing.fairness_scope"},{"disposition":"live","path":"routing.health_penalty"},{"disposition":"live","path":"routing.inflight_penalty"},{"disposition":"live","path":"routing.local_quota_mode"},{"disposition":"live","path":"routing.max_retries_before_stream"},{"disposition":"live","path":"routing.near_tie_epsilon"},{"disposition":"live","path":"routing.quota_exhausted_cooldown_seconds"},{"disposition":"live","path":"routing.randomize_near_ties"},{"disposition":"live","path":"routing.strategy"},{"disposition":"restart_required","path":"routing.trace.flush_interval_s"},{"disposition":"live","path":"routing.trace.guard_cooldown_s"},{"disposition":"live","path":"routing.trace.guard_oldest_event_age_s"},{"disposition":"live","path":"routing.trace.guard_queue_occupancy_threshold"},{"disposition":"live","path":"routing.trace.include_score_components"},{"disposition":"restart_required","path":"routing.trace.max_batch_size"},{"disposition":"live","path":"routing.trace.mode"},{"disposition":"restart_required","path":"routing.trace.queue_capacity"},{"disposition":"live","path":"routing.trace.sample_rate"},{"disposition":"restart_required","path":"routing.trace.shutdown_flush_timeout_s"},{"disposition":"live","path":"routing.trace.skip_above_lock_wait_p95_ms"},{"disposition":"live","path":"routing.unknown_request_reservation_microdollars"},{"disposition":"live","path":"routing.wire_negotiation.cache_max_entries"},{"disposition":"live","path":"routing.wire_negotiation.enabled"},{"disposition":"live","path":"routing.wire_negotiation.learned_preference_ttl_s"},{"disposition":"live","path":"routing.wire_negotiation.max_concurrent_per_provider"},{"disposition":"live","path":"routing.wire_negotiation.min_negotiation_interval_s"},{"disposition":"live","path":"routing.wire_negotiation.rejection_cooldown_s"},{"disposition":"restart_required","path":"security.allowed_hosts"},{"path":"security.allow_insecure_web_auth","disposition":"restart_required"},{"disposition":"restart_required","path":"security.cors_origins"},{"disposition":"live","path":"security.persist_redacted_error_detail"},{"disposition":"restart_required","path":"security.redact_headers"},{"disposition":"restart_required","path":"security.trusted_proxies"},{"disposition":"restart_required","path":"server.access_log"},{"disposition":"restart_required","path":"server.api_key"},{"disposition":"restart_required","path":"server.api_key_env"},{"disposition":"restart_required","path":"server.host"},{"disposition":"restart_required","path":"server.log_level"},{"disposition":"live","path":"server.max_request_body_bytes"},{"disposition":"restart_required","path":"server.port"},{"disposition":"restart_required","path":"server.threads"},{"disposition":"live","path":"transcoder"},{"disposition":"restart_required","path":"update_checker.enabled"},{"disposition":"restart_required","path":"upstream.base_url"},{"disposition":"restart_required","path":"upstream.connect_timeout_s"},{"disposition":"restart_required","path":"upstream.keepalive_timeout_s"},{"disposition":"restart_required","path":"upstream.max_connections"},{"disposition":"restart_required","path":"upstream.max_keepalive"},{"disposition":"restart_required","path":"upstream.pool_timeout_s"},{"disposition":"restart_required","path":"upstream.read_timeout_s"},{"disposition":"restart_required","path":"upstream.write_timeout_s"}],"free_text_redaction_probe":"Authorization: ","free_text_redaction_token":"","ignored_paths":[],"runtime_tracked_default_paths":["backup.directory","backup.enabled","backup.include_env","backup.interval_s","backup.retain_count","backup.startup_delay_s","dashboard.enabled","dashboard.public","dashboard.refresh_interval_s","dashboard.retain_event_days","dashboard.retain_request_stats_days","dashboard.store_request_content","dashboard.theme","dashboard.themes_dir","database.busy_timeout_ms","database.journal_size_limit","database.path","database.synchronous","database.wal","database.worker_threads","integrations.advertise_base_url","limits.five_hour_microdollars","limits.monthly_microdollars","limits.weekly_microdollars","maintenance.contention_defer_above_lock_wait_p95_ms","maintenance.max_batches_per_tick","maintenance.max_deferral_age_s","maintenance.max_rows_per_batch","maintenance.max_tick_duration_ms","maintenance.p0_max_batches_per_tick","maintenance.p0_max_rows_per_batch","maintenance.p0_max_tick_duration_ms","metrics.aggregate_only","metrics.cleanup_interval_s","metrics.cleanup_max_rows_per_pass","metrics.detailed_span_sample_rate","metrics.dispatch_spans.sample_rate","metrics.dispatch_spans.window_size","metrics.event_loop_lag_enabled","metrics.flush_interval_s","metrics.max_buffered_events","metrics.operational_event_retain_days","metrics.rollup_retain_days","metrics.routing_decision_retain_days","metrics.timeseries_bucket_s","metrics.trace_sample_rate","metrics.write_mode","model_capabilities","model_info.aliases","model_info.conflict_ttl_s","model_info.enabled","model_info.include_in_models_endpoint","model_info.known_ttl_s","model_info.max_models_per_cycle","model_info.overrides","model_info.partial_ttl_s","model_info.refresh_interval_s","model_info.sources","model_info.sparse_new_accelerated_days","model_info.sparse_new_initial_ttl_s","model_info.sparse_new_later_ttl_s","model_info.startup_refresh","model_info.store_raw_observations","model_overrides","model_routers","models.allow_stale_catalog","models.catalog_withdrawal_policy","models.collapse_models","models.expose_mode","models.ping_retain_days","models.refresh_interval_s","models.stale_after_s","models.startup_refresh","network.connect_timeout_s","network.keepalive_expiry_s","network.max_connections","network.max_keepalive","network.read_timeout_s","pricing.catalogs.aliases","pricing.catalogs.opencode_zen.api_key","pricing.catalogs.opencode_zen.base_url","pricing.catalogs.opencode_zen.enabled","pricing.catalogs.opencode_zen.max_entries","pricing.catalogs.opencode_zen.options","pricing.catalogs.opencode_zen.priority","pricing.catalogs.opencode_zen.ttl_seconds","pricing.catalogs.openrouter.api_key","pricing.catalogs.openrouter.base_url","pricing.catalogs.openrouter.enabled","pricing.catalogs.openrouter.max_entries","pricing.catalogs.openrouter.options","pricing.catalogs.openrouter.priority","pricing.catalogs.openrouter.ttl_seconds","pricing.fallback","proxies","readiness_probe.enabled","readiness_probe.freshness_s","readiness_probe.initial_probe","readiness_probe.interval_s","readiness_probe.timeout_s","routing.fairness_epsilon","routing.fairness_mode","routing.fairness_scope","routing.health_penalty","routing.inflight_penalty","routing.local_quota_mode","routing.max_retries_before_stream","routing.near_tie_epsilon","routing.quota_exhausted_cooldown_seconds","routing.randomize_near_ties","routing.strategy","routing.trace.flush_interval_s","routing.trace.guard_cooldown_s","routing.trace.guard_oldest_event_age_s","routing.trace.guard_queue_occupancy_threshold","routing.trace.include_score_components","routing.trace.max_batch_size","routing.trace.mode","routing.trace.queue_capacity","routing.trace.sample_rate","routing.trace.shutdown_flush_timeout_s","routing.trace.skip_above_lock_wait_p95_ms","routing.unknown_request_reservation_microdollars","routing.wire_negotiation.cache_max_entries","routing.wire_negotiation.enabled","routing.wire_negotiation.learned_preference_ttl_s","routing.wire_negotiation.max_concurrent_per_provider","routing.wire_negotiation.min_negotiation_interval_s","routing.wire_negotiation.rejection_cooldown_s","security.allowed_hosts","security.cors_origins","security.persist_redacted_error_detail","security.redact_headers","security.trusted_proxies","server.access_log","server.api_key","server.api_key_env","server.host","server.log_level","server.max_request_body_bytes","server.port","server.threads","transcoder","update_checker.enabled","upstream.base_url","upstream.connect_timeout_s","upstream.keepalive_timeout_s","upstream.max_connections","upstream.max_keepalive","upstream.pool_timeout_s","upstream.read_timeout_s","upstream.write_timeout_s"],"secret_display_token":"","secret_field_names":["api_key","api_key_env","proxy_url","proxy_url_env"],"top_level_config_sections":["accounts","backup","dashboard","database","integrations","limits","maintenance","metrics","model_capabilities","model_info","model_overrides","model_routers","models","network","pricing","providers","proxies","readiness_probe","routing","security","server","transcoder","update_checker","upstream"],"unknown_path_probe":"restart_required"},"lifecycle":{"candidate":{"abort":{"abort_is_idempotent":true,"cleanup_errors_preserve_primary_failure":true,"closed_order":["supervisor","outbound_manager","client_pool"],"primary_failure_stage":"build","process_owned_resources_touched":false,"registered_order":["client_pool","outbound_manager","supervisor"]},"legal_transitions":[["building","prepared"],["prepared","transferred"],["building","aborted"],["prepared","aborted"]],"states":["building","prepared","transferred","aborted"]},"close_order":["generation supervisor stop_all","generation cleanup callbacks","finalization supervisor shutdown","provider client pool close","outbound client manager aclose"],"generation_slot":{"legal_transitions":[["active","retiring"],["retiring","closing"],["closing","closed"],["closing","failed_close"]],"retirement_ready_when":["active_leases == 0","terminal_references == 0"],"states":["active","retiring","closing","closed","failed_close"]},"lease":{"claim_and_increment_are_one_critical_section":true,"claim_predicate":"shutdown_in_progress OR (NOT lease_admission_gated AND active_slot_accepting)","claim_timeout_s":30.0,"release_is_exactly_once":true,"stream_release_boundary":"iterator finally after handler return"},"pending_swap":{"commit_effects":["candidate_becomes_active","old_slot_stops_accepting_leases","publication_epoch_increments","lease_admission_reopens"],"legal_transitions":[["prepared","staged"],["staged","committed"],["staged","rolled_back"],["committed","finalized"]],"rollback_effects":["old_slot_restored","candidate_slot_dropped","lease_admission_reopens","candidate_requires_abort"],"stage_effects":["lease_admission_gated","candidate_slot_non_accepting","active_slot_unchanged"],"states":["prepared","staged","committed","rolled_back","finalized"]},"publication_gate":{"forbidden_inside_gate":["provider/network calls","catalog refresh","backup","unbounded work"],"inside_gate":["stage candidate","apply bounded mandatory acceptance changes","commit runtime pointer","commit task-spec state"],"pre_gate_work":["config parsing and validation","digest check and diff","candidate construction","persistence delta preparation","task diff preflight"]},"shutdown":{"adopts_unfinalized_committed_swap_old_slot":true,"forced_close_abandons_terminal_references_only_on_process_shutdown":true,"live_rehash_preserves_accepted_work":true,"manager_join_deadline_s":10.0,"retirement_drain_deadline_s":5.0,"stops_new_publication_and_acquisition":true},"transaction":{"acceptance_states":["not_accepted","persistence_committed_runtime_pending","accepted"],"accepted_reload_cannot_abort":true,"states":["created","validated","diffed","candidate_prepared","persistence_prepared","process_transitions_prepared","process_transitions_preflighted","commit_started","runtime_staged","runtime_swap_committed","runtime_published","process_transitions_applied","persistence_committed","observable_state_updated","retirement_scheduled","completed","aborting","aborted","compensation_failed"],"transitions":{"aborted":[],"aborting":["aborted","compensation_failed"],"candidate_prepared":["aborting","persistence_prepared"],"commit_started":["aborting","runtime_published","runtime_staged"],"compensation_failed":[],"completed":[],"created":["aborting","validated"],"diffed":["aborting","candidate_prepared"],"observable_state_updated":["aborting","retirement_scheduled"],"persistence_committed":["aborting","observable_state_updated"],"persistence_prepared":["aborting","process_transitions_prepared"],"process_transitions_applied":["aborting","persistence_committed"],"process_transitions_preflighted":["aborting","commit_started"],"process_transitions_prepared":["aborting","process_transitions_preflighted"],"retirement_scheduled":["aborting","completed"],"runtime_published":["aborting","process_transitions_applied"],"runtime_staged":["aborting","runtime_swap_committed"],"runtime_swap_committed":["aborting","process_transitions_applied"],"validated":["aborting","diffed"]}}},"normalization":{"excluded":["API keys","proxy credentials","raw config values","request/provider bodies"],"normalized":["timestamps","runtime identifiers"],"preserved":["state names and order","booleans","changed/restart path ordering","task names and specs","generation increments","retirement pending state","error category distinctions"]},"oracle_modules":["eggpool.runtime_manager","eggpool.generation_factory","eggpool.config_reload_policy","eggpool.reload_transaction","eggpool.runtime_task_inventory","eggpool.runtime_tasks","eggpool.reload_diagnostics","eggpool.app","eggpool.cli_rehash_helper"],"ownership_inventory":[{"authority":"ProcessRuntime.db/stats_db","lifetime":"process","name":"database_and_repositories","ownership":"process-owned"},{"authority":"RuntimeGeneration.client_pool","lifetime":"generation retirement","name":"provider_client_pool","ownership":"generation-owned"},{"authority":"RuntimeGeneration.outbound_manager","lifetime":"generation retirement","name":"outbound_client_manager","ownership":"generation-owned"},{"authority":"RuntimeGeneration.registry","lifetime":"generation retirement","name":"account_registry","ownership":"generation-owned"},{"authority":"RuntimeGeneration.catalog","lifetime":"generation retirement","name":"catalog_service","ownership":"generation-owned"},{"authority":"RuntimeGeneration.router/coordinator","lifetime":"generation retirement","name":"router_and_coordinator","ownership":"generation-owned"},{"authority":"RuntimeGeneration.finalization_supervisor","lifetime":"after leases and terminal references drain","name":"finalization_supervisor","ownership":"generation-owned"},{"authority":"ProcessRuntime.model_router_affinity","lifetime":"process; revalidated per generation","name":"model_router_affinity","ownership":"process-owned"},{"authority":"ProcessRuntime.wire_profile_resolver","lifetime":"process; fingerprint-qualified","name":"wire_profile_resolver","ownership":"process-owned"},{"authority":"RuntimeGeneration.supervisor","lifetime":"generation retirement","name":"generation_task_supervisor","ownership":"generation-owned"},{"authority":"ProcessRuntime.process_supervisor","lifetime":"process singleton","name":"process_task_supervisor","ownership":"process-owned"},{"authority":"startup lifecycle entry point","lifetime":"one bounded startup pass","name":"crash_reconciler","ownership":"process-owned/startup-only"},{"authority":"RuntimeMetricsService and reload metadata","lifetime":"process; bounded snapshots","name":"runtime_metrics_and_reload_diagnostics","ownership":"process-owned"},{"authority":"listener, middleware, route topology, auth, DB constructor","lifetime":"process construction","name":"server_constructor_state","ownership":"startup-only/restart-required"},{"authority":"RuntimeGenerationCandidate","lifetime":"building/prepared until transfer or abort","name":"candidate_resources_before_publication","ownership":"candidate-owned"},{"authority":"GenerationLease","lifetime":"finite completion or stream disconnect","name":"request_and_stream_lease","ownership":"request/lease-owned"},{"authority":"terminal reference on generation slot","lifetime":"durable/runtime convergence","name":"retained_finalization","ownership":"retained-finalization-owned"}],"reload":{"cases":[{"accepted":true,"active_generation_unchanged":true,"category":"success_noop","changed_sections":[],"generation_delta":0,"legacy_wire_stage":"commit","name":"identical_digest_noop","publication_occurred":false,"restart_required_paths":[],"retirement_pending":false,"terminal_stage":"idle"},{"accepted":true,"active_generation_unchanged":false,"category":"success_committed","changed_sections":["server"],"generation_delta":1,"legacy_wire_stage":"retirement","name":"valid_live_only_change","publication_occurred":true,"restart_required_paths":[],"retirement_pending":"true_if_old_work_remains","terminal_stage":"retirement"},{"accepted":false,"active_generation_unchanged":true,"category":"rejected_restart_required","changed_sections":["server"],"generation_delta":0,"legacy_wire_stage":"diff","name":"restart_required_change","publication_occurred":false,"restart_required_paths":["server.port"],"retirement_pending":false,"terminal_stage":"diff"},{"accepted":false,"active_generation_unchanged":true,"category":"rejected_restart_required","changed_sections":["server"],"generation_delta":0,"legacy_wire_stage":"diff","name":"mixed_live_and_restart_required_change","publication_occurred":false,"restart_required_paths":["server.port"],"retirement_pending":false,"terminal_stage":"diff"},{"accepted":false,"active_generation_unchanged":true,"category":"rejected_validation","changed_sections":[],"generation_delta":0,"legacy_wire_stage":"validation","name":"invalid_config","publication_occurred":false,"restart_required_paths":[],"retirement_pending":false,"terminal_stage":"validation"},{"accepted":false,"active_generation_unchanged":true,"category":"rejected_validation","changed_sections":[],"generation_delta":0,"legacy_wire_stage":"digest_check","name":"expected_digest_mismatch","publication_occurred":false,"restart_required_paths":[],"retirement_pending":false,"terminal_stage":"validation"},{"accepted":false,"active_generation_unchanged":true,"category":"rejected_validation","changed_sections":[],"generation_delta":0,"legacy_wire_stage":"preparation","name":"stale_caller_generation","publication_occurred":false,"restart_required_paths":[],"retirement_pending":false,"terminal_stage":"preparation"},{"accepted":false,"active_generation_unchanged":true,"category":"failed_candidate_prepare","changed_sections":[],"generation_delta":0,"legacy_wire_stage":"preparation","name":"candidate_construction_failure","publication_occurred":false,"restart_required_paths":[],"retirement_pending":false,"terminal_stage":"preparation"},{"accepted":false,"active_generation_unchanged":true,"category":"failed_process_transition_prepare","changed_sections":[],"generation_delta":0,"legacy_wire_stage":"preparation","name":"preflight_process_transition_failure","publication_occurred":false,"restart_required_paths":[],"retirement_pending":false,"terminal_stage":"preparation"},{"accepted":false,"active_generation_unchanged":true,"category":"aborted_cancelled","changed_sections":[],"generation_delta":0,"legacy_wire_stage":"preparation","name":"cancellation_before_acceptance","publication_occurred":false,"restart_required_paths":[],"retirement_pending":false,"terminal_stage":"preparation"},{"accepted":true,"active_generation_unchanged":false,"category":"post_commit_finalization_pending","changed_sections":[],"generation_delta":1,"legacy_wire_stage":"retirement","name":"cancellation_during_or_after_acceptance","publication_occurred":true,"restart_required_paths":[],"retirement_pending":true,"terminal_stage":"retirement"},{"accepted":true,"active_generation_unchanged":false,"category":"post_commit_finalization_pending","changed_sections":[],"generation_delta":1,"legacy_wire_stage":"retirement","name":"successful_publication_retirement_pending","publication_occurred":true,"restart_required_paths":[],"retirement_pending":true,"terminal_stage":"retirement"},{"accepted":true,"active_generation_unchanged":false,"category":"success_committed","changed_sections":[],"generation_delta":1,"legacy_wire_stage":"retirement","name":"successful_publication_already_drained","publication_occurred":true,"restart_required_paths":[],"retirement_pending":false,"terminal_stage":"retirement"}],"classification_probes":{"retry_pending":"post_commit_finalization_pending","success_committed":"success_committed","success_noop":"success_noop","validation":"rejected_validation"},"diff_observations":{"identical":{"changes":[],"dispositions":[],"live":[],"paths":[],"restart_required":[],"sections":[]},"live_only":{"changes":[{"disposition":"live","new_display":"10485761","old_display":"10485760","path":"server.max_request_body_bytes","secret":false,"section":"server"}],"dispositions":["live"],"live":["server.max_request_body_bytes"],"paths":["server.max_request_body_bytes"],"restart_required":[],"sections":["server"]},"mixed":{"changes":[{"disposition":"live","new_display":"10485761","old_display":"10485760","path":"server.max_request_body_bytes","secret":false,"section":"server"},{"disposition":"restart_required","new_display":"11301","old_display":"11300","path":"server.port","secret":false,"section":"server"}],"dispositions":["live","restart_required"],"live":["server.max_request_body_bytes"],"paths":["server.max_request_body_bytes","server.port"],"restart_required":["server.port"],"sections":["server"]},"restart_only":{"changes":[{"disposition":"restart_required","new_display":"11301","old_display":"11300","path":"server.port","secret":false,"section":"server"}],"dispositions":["restart_required"],"live":[],"paths":["server.port"],"restart_required":["server.port"],"sections":["server"]},"secret":{"changes":[{"disposition":"restart_required","new_display":"","old_display":"","path":"pricing.catalogs.openrouter.api_key","secret":true,"section":"pricing"}],"dispositions":["restart_required"],"live":[],"paths":["pricing.catalogs.openrouter.api_key"],"restart_required":["pricing.catalogs.openrouter.api_key"],"sections":["pricing"]}},"generation_id_rule":"monotonic; no-op and rejected reloads do not publish","legacy_stages":["validation","digest_check","diff","preparation","reconciliation","commit","retirement"],"result_categories":["success_committed","success_noop","success_ignored_only","rejected_busy","rejected_validation","rejected_restart_required","failed_candidate_prepare","failed_persistence_prepare","failed_process_transition_prepare","failed_commit","failed_publication","failed_process_transition_apply","failed_persistence_commit","failed_runtime_stage","failed_runtime_stage_rollback","post_commit_finalization_pending","retirement_schedule_failed","shutdown_adopted","aborted_cancelled","aborted_shutdown","compensation_failed","internal_error"],"retirement_pending_rule":"accepted publication may be pending while old leases or terminal references drain","terminal_stages":["validation","diff","preparation","reconciliation","commit","retirement","idle"]},"schema_version":"m8-runtime-lifecycle-r001-observations/v1","tasks":{"conditions":{"automatic_backup":"backup.enabled and interval_s > 0 enables","catalog_refresh":"models.refresh_interval_s <= 0 disables","generation_leased_callbacks":"acquire active generation for every tick","metrics_flush":"metrics.write_mode == immediate disables","retention_cleanup":"always registered; reads active generation config per tick","update_checker":"startup-only process task; update_checker.enabled and outbound handle required"},"config_variants":{"automatic_backup_enabled":[{"enabled":true,"initial_delay_s":null,"interval_s":300.0,"name":"catalog_refresh","run_immediately":false},{"enabled":true,"initial_delay_s":null,"interval_s":86400.0,"name":"retention_cleanup","run_immediately":false},{"enabled":true,"initial_delay_s":null,"interval_s":60.0,"name":"checkpoint","run_immediately":true},{"enabled":true,"initial_delay_s":5.0,"interval_s":120.0,"name":"metrics_flush","run_immediately":false},{"enabled":false,"initial_delay_s":null,"interval_s":86400.0,"name":"update_checker","run_immediately":true},{"enabled":true,"initial_delay_s":5.0,"interval_s":600.0,"name":"automatic_backup","run_immediately":false}],"catalog_refresh_disabled":[{"enabled":false,"initial_delay_s":null,"interval_s":0.0,"name":"catalog_refresh","run_immediately":false},{"enabled":true,"initial_delay_s":null,"interval_s":86400.0,"name":"retention_cleanup","run_immediately":false},{"enabled":true,"initial_delay_s":null,"interval_s":60.0,"name":"checkpoint","run_immediately":true},{"enabled":true,"initial_delay_s":5.0,"interval_s":120.0,"name":"metrics_flush","run_immediately":false},{"enabled":false,"initial_delay_s":null,"interval_s":86400.0,"name":"update_checker","run_immediately":true},{"enabled":false,"initial_delay_s":300.0,"interval_s":86400.0,"name":"automatic_backup","run_immediately":false}],"default":[{"enabled":true,"initial_delay_s":null,"interval_s":300.0,"name":"catalog_refresh","run_immediately":false},{"enabled":true,"initial_delay_s":null,"interval_s":86400.0,"name":"retention_cleanup","run_immediately":false},{"enabled":true,"initial_delay_s":null,"interval_s":60.0,"name":"checkpoint","run_immediately":true},{"enabled":true,"initial_delay_s":5.0,"interval_s":120.0,"name":"metrics_flush","run_immediately":false},{"enabled":false,"initial_delay_s":null,"interval_s":86400.0,"name":"update_checker","run_immediately":true},{"enabled":false,"initial_delay_s":300.0,"interval_s":86400.0,"name":"automatic_backup","run_immediately":false}],"metrics_immediate":[{"enabled":true,"initial_delay_s":null,"interval_s":300.0,"name":"catalog_refresh","run_immediately":false},{"enabled":true,"initial_delay_s":null,"interval_s":86400.0,"name":"retention_cleanup","run_immediately":false},{"enabled":true,"initial_delay_s":null,"interval_s":60.0,"name":"checkpoint","run_immediately":true},{"enabled":false,"initial_delay_s":5.0,"interval_s":120.0,"name":"metrics_flush","run_immediately":false},{"enabled":false,"initial_delay_s":null,"interval_s":86400.0,"name":"update_checker","run_immediately":true},{"enabled":false,"initial_delay_s":300.0,"interval_s":86400.0,"name":"automatic_backup","run_immediately":false}],"update_checker_enabled":[{"enabled":true,"initial_delay_s":null,"interval_s":300.0,"name":"catalog_refresh","run_immediately":false},{"enabled":true,"initial_delay_s":null,"interval_s":86400.0,"name":"retention_cleanup","run_immediately":false},{"enabled":true,"initial_delay_s":null,"interval_s":60.0,"name":"checkpoint","run_immediately":true},{"enabled":true,"initial_delay_s":5.0,"interval_s":120.0,"name":"metrics_flush","run_immediately":false},{"enabled":true,"initial_delay_s":null,"interval_s":86400.0,"name":"update_checker","run_immediately":true},{"enabled":false,"initial_delay_s":300.0,"interval_s":86400.0,"name":"automatic_backup","run_immediately":false}]},"inventory":[{"callback_kind":"catalog_refresh","default_enabled":true,"description":"Periodically refresh the model catalog from enabled accounts","generation_dependencies":["catalog","model_info","registry","health_manager"],"initial_delay_s":null,"interval_s":300.0,"name":"catalog_refresh","ownership":"generation_leased","process_dependencies":["db"],"reloadable_fields":["models.refresh_interval_s"],"run_immediately":false,"timeout_s":null},{"callback_kind":"retention_cleanup","default_enabled":true,"description":"Clean up old requests, events, pings, and expired reservations","generation_dependencies":["router"],"initial_delay_s":null,"interval_s":86400.0,"name":"retention_cleanup","ownership":"generation_leased","process_dependencies":["db"],"reloadable_fields":["dashboard.retain_request_stats_days","dashboard.retain_event_days","metrics.operational_event_retain_days","metrics.routing_decision_retain_days","models.ping_retain_days"],"run_immediately":false,"timeout_s":null},{"callback_kind":"checkpoint","default_enabled":true,"description":"Periodic SQLite WAL checkpoint","generation_dependencies":[],"initial_delay_s":null,"interval_s":60.0,"name":"checkpoint","ownership":"process","process_dependencies":["db"],"reloadable_fields":[],"run_immediately":true,"timeout_s":null},{"callback_kind":"metrics_flush","default_enabled":true,"description":"Flush buffered metrics analytics to SQLite","generation_dependencies":[],"initial_delay_s":5.0,"interval_s":30.0,"name":"metrics_flush","ownership":"process","process_dependencies":["metrics_coalescer"],"reloadable_fields":["metrics.write_mode","metrics.flush_interval_s"],"run_immediately":false,"timeout_s":null},{"callback_kind":"update_checker","default_enabled":true,"description":"Periodically check PyPI for new EggPool releases","generation_dependencies":[],"initial_delay_s":null,"interval_s":86400.0,"name":"update_checker","ownership":"process","process_dependencies":["outbound_manager"],"reloadable_fields":[],"run_immediately":true,"timeout_s":null},{"callback_kind":"automatic_backup","default_enabled":true,"description":"Create periodic backup archives of config and database","generation_dependencies":[],"initial_delay_s":300.0,"interval_s":86400.0,"name":"automatic_backup","ownership":"process","process_dependencies":["db"],"reloadable_fields":["backup.enabled","backup.interval_s","backup.startup_delay_s","backup.retain_count"],"run_immediately":false,"timeout_s":null}],"names_unique":true,"singleton_process_tasks":["checkpoint","metrics_flush","update_checker","automatic_backup"]}} diff --git a/tests/tooling/test_dashboard_parity_on_prem.py b/tests/tooling/test_dashboard_parity_on_prem.py new file mode 100644 index 000000000..141d22a19 --- /dev/null +++ b/tests/tooling/test_dashboard_parity_on_prem.py @@ -0,0 +1,211 @@ +"""Contract tests for the on-prem admin/member browser qualification gate. + +These tests never start a server or a browser. They pin the ownership split +between the frozen Python oracle (historical pages) and the Rust-owned approved +reference (on-prem pages), the fixture's secret-leak sentinels, and the bounded +defect classes the gate is allowed to fix. +""" + +from __future__ import annotations + +import json +from typing import TYPE_CHECKING + +import pytest + +if TYPE_CHECKING: + from pathlib import Path + +from scripts.dashboard_parity import on_prem +from scripts.dashboard_parity.on_prem import ( + _DOM_FACTS_TEMPLATE, + ACCOUNT_SECRET, + ADMIN_ONLY_PAGES, + ADMIN_PAGES, + ADMIN_PASSWORD, + MEMBER_PAGES, + MEMBER_PASSWORD, + MEMBER_TEMP_PASSWORD, + ON_PREM_REFERENCE, + ON_PREM_SCHEMA, + ON_PREM_THEMES, + ON_PREM_VIEWPORTS, + SECRET_MARKERS, + STATIC_HEADER_SECRET, + QualificationError, + _assert_no_defects, + _canonical, + _dom_facts_expression, + _reconcile_reference, + _stable_rows, + _write_on_prem_config, + qualify_on_prem_browser, +) + +_CLEAN_FACTS = { + "duplicateIds": [], + "horizontalOverflow": False, + "tables": [], + "forms": [], + "secretMarkers": {}, + "viewportMeta": True, +} + + +def test_on_prem_module_owns_the_managed_browser_gate() -> None: + assert qualify_on_prem_browser.__module__ == "scripts.dashboard_parity.on_prem" + + +def test_managed_config_plants_sentinel_secrets_without_a_server_key( + tmp_path: Path, +) -> None: + config = _write_on_prem_config(tmp_path, 18571) + text = config.read_text(encoding="utf-8") + for marker in (ACCOUNT_SECRET, STATIC_HEADER_SECRET): + assert marker in text, f"fixture must plant {marker} for the leak guard" + # The passwords are typed by the administrator rather than configured, but + # they are still values that must never reach a rendered page. + for marker in (ADMIN_PASSWORD, MEMBER_TEMP_PASSWORD, MEMBER_PASSWORD): + assert marker not in text + # The managed instance has no server API key; provider credentials are the + # sentinels the leak guard is meant to catch. + server_section = text.split("[database]")[0] + assert "[server]" in server_section + assert "api_key" not in server_section + assert "api_key_env" not in server_section + + +def test_dom_facts_probe_reads_labels_tables_and_secret_markers() -> None: + probe = _dom_facts_expression() + for needle in ( + "duplicateIds", + "horizontalOverflow", + "table-scroll", + "unlabelled", + "viewportMeta", + "__SECRET_MARKERS__", + ): + assert needle in _DOM_FACTS_TEMPLATE, needle + # The placeholder is substituted at call time with the live markers. + assert "__SECRET_MARKERS__" not in probe + for marker in SECRET_MARKERS: + assert marker in probe, marker + + +def test_defect_guard_rejects_each_bounded_class() -> None: + defects = { + "duplicateIds": [["user-card"]], + "horizontalOverflow": True, + "tables": [{"wrapped": False, "width": 1600}], + "forms": [{"action": "/login", "unlabelled": ["input:username"]}], + "secretMarkers": {SECRET_MARKERS[0]: True}, + "viewportMeta": False, + } + for field, value in defects.items(): + facts = dict(_CLEAN_FACTS) + facts[field] = value + with pytest.raises(QualificationError): + _assert_no_defects("page.admin./users.desktop", facts, "admin") + + +def test_defect_guard_accepts_a_wrapped_scrolling_table() -> None: + facts = dict(_CLEAN_FACTS) + facts["tables"] = [{"wrapped": True, "width": 1600}] + _assert_no_defects("page.admin./users.mobile", facts, "admin") + + +def test_stable_rows_drops_per_run_identifiers() -> None: + rows = [{"username": "member", "id": "f0652a", "must_change": True}] + assert _stable_rows(rows) == [{"username": "member", "must_change": True}] + # The caller still needs the identifier to address the row's forms. + assert rows[0]["id"] == "f0652a" + + +def test_canonical_is_stable_and_key_ordered() -> None: + assert _canonical({"b": 1, "a": [2, {"d": 4, "c": 3}]}) == ( + '{"a":[2,{"c":3,"d":4}],"b":1}' + ) + + +def test_reconcile_rejects_drift_against_the_approved_reference( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + reference = tmp_path / "reference.json" + reference.write_text( + json.dumps({"schema": ON_PREM_SCHEMA, "observations": {"a": "1"}}), + encoding="utf-8", + ) + monkeypatch.setattr(on_prem, "ON_PREM_REFERENCE", reference) + with pytest.raises(QualificationError) as drift: + _reconcile_reference({"a": "2"}, [], False) + assert "a: 1 -> 2" in str(drift.value) + assert _reconcile_reference({"a": "1"}, [], False)["result"] == "passed" + with pytest.raises(QualificationError) as missing: + _reconcile_reference({}, [], False) + assert "absent from this run" in str(missing.value) + with pytest.raises(QualificationError) as unknown: + _reconcile_reference({"a": "1", "z": "9"}, [], False) + assert "not in the approved reference" in str(unknown.value) + + +def test_reconcile_rejects_a_schema_change_in_the_approved_reference( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + reference = tmp_path / "reference.json" + reference.write_text( + json.dumps({"schema": 0, "observations": {}}), encoding="utf-8" + ) + monkeypatch.setattr(on_prem, "ON_PREM_REFERENCE", reference) + with pytest.raises(QualificationError) as schema: + _reconcile_reference({}, [], False) + assert "schema" in str(schema.value) + + +def test_approved_reference_is_frozen_and_covers_the_required_surface() -> None: + assert ON_PREM_REFERENCE.is_file(), "run the gate with --write-on-prem-reference" + approved = json.loads(ON_PREM_REFERENCE.read_text(encoding="utf-8")) + assert approved["schema"] == ON_PREM_SCHEMA + observations: dict[str, str] = approved["observations"] + for role, pages in (("admin", ADMIN_PAGES), ("member", MEMBER_PAGES)): + for page in pages: + for name, _width, _height, _scale in ON_PREM_VIEWPORTS: + assert f"page.{role}.{page}.{name}" in observations + for page in ADMIN_ONLY_PAGES: + assert f"member_denied.{page}" in observations + assert f"member_denied.{page}.message" in observations + for page in ("/users", "/profiles", "/admin/audit"): + assert f"anonymous.{page}" in observations + assert f"after_logout.{page}" in observations + # A personal agent credential must never open the dashboard. + for page in ("/users", "/my-account"): + assert f"agent_token.{page}" in observations + for theme in ON_PREM_THEMES: + key = f"page.admin./users.mobile.{theme}" + assert key in observations + for key in ( + "admin.sign_in", + "admin.no_js_login", + "admin.profile_created", + "admin.member_created", + "admin.member_row", + "admin.member_suspended", + "admin.member_reactivated", + "admin.token_reveal", + "admin.token_revealed_once", + "admin.token_absent_on_return", + "admin.override_saved", + "admin.budget_saved", + "admin.budget_labels", + "admin.profile_conflict", + "member.forced_password_landing", + "member.password_changed", + "member.budget_labelling", + "restart.session_survives_restart", + "restart.profiles_survive_restart", + "restart.member_row", + ): + assert key in observations, key + for value in observations.values(): + assert "\n" not in value + for marker in SECRET_MARKERS: + assert marker not in value diff --git a/tests/tooling/test_on_prem_mmc_tail_attribution.py b/tests/tooling/test_on_prem_mmc_tail_attribution.py new file mode 100644 index 000000000..dcc0151ff --- /dev/null +++ b/tests/tooling/test_on_prem_mmc_tail_attribution.py @@ -0,0 +1,125 @@ +"""Evidence guard for the Plan M001 Pi 5 MMC/tmpfs paired benchmark.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).parents[2] +ARTIFACT_DIR = ROOT / "artifacts/qualification/on-prem-concurrency-performance" +MATRIX_PATH = ARTIFACT_DIR / "matrix.json" +TARGET_BOARD = "Raspberry Pi 5 Model B Rev 1.0" +TARGET_ATTESTATION = "Linux aarch64 plus device-tree board model" +RUNS_PER_STORAGE = 3 +EXPECTED_CONCURRENCY = 4 +EXPECTED_REQUESTS = 32 + + +def _load_run(name: str) -> dict[str, Any]: + path = ARTIFACT_DIR / name + assert path.is_file(), f"missing Plan M001 run artifact: {name}" + return json.loads(path.read_text(encoding="utf-8")) + + +def _matrix() -> dict[str, Any]: + assert MATRIX_PATH.is_file(), "missing Plan M001 matrix manifest" + return json.loads(MATRIX_PATH.read_text(encoding="utf-8")) + + +def test_m001_matrix_manifest_freezes_target_workload_and_thresholds() -> None: + matrix = _matrix() + assert matrix["schema_version"] == "on-prem-concurrency-matrix.v1" + assert matrix["plan"] == "on-prem-concurrency-performance M001" + assert matrix["thresholds"]["g3_p95_ms"] == 1_000 + assert matrix["thresholds"]["g3_max_ms"] == 3_000 + assert matrix["workload"]["concurrency"] == EXPECTED_CONCURRENCY + assert matrix["workload"]["request_count_per_run"] == EXPECTED_REQUESTS + assert matrix["workload"]["sample_count"] == 30 + assert matrix["target"]["board_model"] == TARGET_BOARD + assert matrix["target"]["architecture"] == "aarch64" + assert matrix["target"]["filesystem"] == "ext4" + assert matrix["target"]["attestation"] == TARGET_ATTESTATION + assert matrix["scope_limits"]["tested_concurrency_levels"] == [4] + assert matrix["scope_limits"]["tested_named_user_count"] == 1 + assert matrix["scope_limits"]["hard_user_or_global_budgets_enabled"] is False + + +def test_m001_has_three_same_candidate_runs_per_storage_condition() -> None: + matrix = _matrix() + all_runs: list[tuple[str, dict[str, Any]]] = [] + for storage in ("mmc", "tmpfs"): + runs = matrix["conditions"][storage]["runs"] + assert len(runs) == RUNS_PER_STORAGE + assert len(set(runs)) == RUNS_PER_STORAGE + for name in runs: + document = _load_run(name) + assert document["status"] == "pass", name + assert document["schema_version"] == "runtime-q008.v2", name + assert document["candidate"]["sha256"] == matrix["candidate"]["sha256"] + assert document["benchmark"]["config_fixture"] == "sbc-benchmark.toml" + env = document["environment"] + assert env["board_model"] == TARGET_BOARD + assert env["architecture"] == "aarch64" + assert env["filesystem"] == "ext4" + assert env["root_storage_device_class"] == "mmc" + assert env["attestation"] == TARGET_ATTESTATION + concurrent = document["benchmark"]["runs"][ + "concurrent_native_responses_finite" + ] + assert concurrent["status"] == "measured" + assert concurrent["client_concurrency"] == EXPECTED_CONCURRENCY + assert concurrent["request_count"] == EXPECTED_REQUESTS + assert concurrent["completed_count"] == EXPECTED_REQUESTS + assert concurrent["failed_count"] == 0 + assert document["durable"]["pending_requests"] == 0 + assert document["durable"]["active_reservations"] == 0 + all_runs.append((storage, document)) + + assert len({doc["candidate"]["sha256"] for _, doc in all_runs}) == 1 + assert ( + len( + { + ( + doc["candidate"]["repository_commit_sha"], + doc["candidate"]["rust_toolchain"], + ) + for _, doc in all_runs + } + ) + == 1 + ) + + +def test_m001_storage_control_reproduces_and_removes_the_g3_tail() -> None: + matrix = _matrix() + summaries: dict[str, list[tuple[int, int]]] = {"mmc": [], "tmpfs": []} + for storage in summaries: + for name in matrix["conditions"][storage]["runs"]: + benchmark = _load_run(name)["benchmark"]["runs"] + concurrent = benchmark["concurrent_native_responses_finite"]["timings"] + summaries[storage].append( + ( + int(concurrent["p95_elapsed_ms"]), + int(concurrent["maximum_elapsed_ms"]), + ) + ) + + assert all(p95 > 1_000 and maximum > 3_000 for p95, maximum in summaries["mmc"]) + assert all(p95 <= 1_000 and maximum <= 3_000 for p95, maximum in summaries["tmpfs"]) + + +def test_m001_reports_are_redacted_and_have_no_p99_claim() -> None: + matrix = _matrix() + for storage in ("mmc", "tmpfs"): + for name in matrix["conditions"][storage]["runs"]: + content = (ARTIFACT_DIR / name).read_text(encoding="utf-8") + assert "SBC synthetic bootstrap password" not in content + assert "epk1_" not in content + document = json.loads(content) + assert ( + "p99_elapsed_ms" + not in document["benchmark"]["runs"][ + "concurrent_native_responses_finite" + ]["timings"] + ) diff --git a/tests/tooling/test_on_prem_sbc_qualification.py b/tests/tooling/test_on_prem_sbc_qualification.py new file mode 100644 index 000000000..763c228d5 --- /dev/null +++ b/tests/tooling/test_on_prem_sbc_qualification.py @@ -0,0 +1,184 @@ +"""M012 contracts: the physical-SBC runner binds to an attested target, +records actual concurrency/completion only, and never fabricates the +1/5/10/25/49 named-user matrix or a small-team capacity claim. + +The runner covers a single-admin loopback corpus plus a fixed +contention observation; the named-user matrix in +`plans/implementation/on-prem-user-access/012-physical-sbc-small-team-qualification.md` +remains unimplemented until provisioned hardware exists. These tests +pin that boundary so a blocked off-target run can never be mistaken +for measured evidence. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from scripts.qualification_sbc import ( + BENCHMARK_FIXTURE, + CONCURRENCY_BATCH_SIZE, + CONCURRENCY_WORKERS, + DEFAULT_FIXTURE, + MANIFEST_VERSION, + SCHEMA_V1, + SCHEMA_VERSION, + board_metadata, + main, + run_qualification, +) + + +def test_m012_target_gate_requires_attested_physical_sbc() -> None: + board, reason = board_metadata() + if board is None: + assert reason, "gate must name the missing evidence" + lowered = reason.lower() + assert any( + token in lowered + for token in ("linux", "aarch64", "device-tree", "physical") + ) + else: + assert board["board_model"] + assert board["attestation"] == "Linux aarch64 plus device-tree board model" + assert board["architecture"] in {"aarch64", "arm64"} + + +def test_m012_off_target_run_is_blocked_before_measurement( + tmp_path: Path, +) -> None: + board, _ = board_metadata() + if board is not None: + pytest.skip("physical SBC target: gate passes, nothing to assert here") + report = run_qualification(binary=Path("/not/a/candidate")) + assert report["status"] == "blocked" + assert "candidate" not in report + assert "benchmark" not in report + output = tmp_path / "m012.json" + exit_code = main(["--binary", "/not/a/candidate", "--output", str(output)]) + assert exit_code == 1 + assert json.loads(output.read_text(encoding="utf-8"))["status"] == "blocked" + + +def test_m012_blocked_report_carries_manifest_and_bounded_environment() -> None: + board, _ = board_metadata() + if board is not None: + pytest.skip("physical SBC target: gate passes, nothing to assert here") + report = run_qualification(binary=Path("/not/a/candidate")) + assert report["schema_version"] == SCHEMA_V1 + assert report["schema_version"] == SCHEMA_VERSION + assert report["manifest"] == MANIFEST_VERSION + assert report["plan"] == "SBC qualification" + assert set(report["environment"]) == {"system", "architecture"} + + +def test_m012_blocked_report_is_secret_free_and_bounded() -> None: + board, _ = board_metadata() + if board is not None: + pytest.skip("physical SBC target: gate passes, nothing to assert here") + report = run_qualification(binary=Path("/not/a/candidate")) + payload = json.dumps(report, sort_keys=True) + assert "q008-provider-key" not in payload + assert "q008-server-key" not in payload + assert "http://" not in payload + assert "https://" not in payload + assert len(payload.encode()) < 2048 + + +def test_m012_runner_records_actual_concurrency_not_named_users() -> None: + # Fixed contention observation: four clients, 32 requests. This is an + # actual-concurrency measurement, never a named-user count. + assert CONCURRENCY_WORKERS == 4 + assert CONCURRENCY_BATCH_SIZE == 32 + report = run_qualification(binary=Path("/not/a/candidate")) + payload = json.dumps(report) + for claim in ( + "named_users", + "named-users", + "user_count", + "capacity", + "users_per_second", + ): + assert claim not in payload + + +def test_m012_no_named_user_or_concurrency_matrix_flags() -> None: + from scripts.qualification_sbc import _parser + + options = { + option for action in _parser()._actions for option in action.option_strings + } + for forbidden in ( + "--named-users", + "--users", + "--concurrency", + "--load-level", + "--user-matrix", + ): + assert forbidden not in options + + +def test_m012_benchmark_corpus_requires_benchmark_fixture() -> None: + with pytest.raises(ValueError, match="benchmark corpus"): + run_qualification(binary=Path("/not/a/candidate"), benchmark_samples=30) + + +def test_m012_control_candidate_modes_are_explicit_and_exclusive( + tmp_path: Path, +) -> None: + with pytest.raises(ValueError, match="publication phase"): + run_qualification( + binary=Path("/not/a/candidate"), + qualification_dedicated_checkpointer_mode="control", + ) + with pytest.raises(ValueError, match="mutually exclusive"): + run_qualification( + binary=Path("/not/a/candidate"), + config_fixture=BENCHMARK_FIXTURE, + diagnose_publication_phases=True, + qualification_dedicated_checkpointer_mode="control", + qualification_persist_journal_mode="control", + diagnostic_database_dir=tmp_path, + ) + + +def test_m012_sbc_fixtures_carry_no_user_identity_matrix() -> None: + for fixture in (DEFAULT_FIXTURE, BENCHMARK_FIXTURE): + text = fixture.read_text(encoding="utf-8") + assert "[users]" not in text + assert "personal_token" not in text + assert "password" not in text + assert "argon" not in text.lower() + assert "__SBC_UPSTREAM__" in text + assert "https://" not in text + + +def test_m012_managed_bootstrap_secret_is_redacted() -> None: + from scripts.qualification_sbc import bounded + + token = "epk1_0123456789abcdef_secret-part" + redacted = bounded(f"issued {token} for synthetic admin") + assert token not in redacted + assert "" in redacted + assert bounded("Bearer q008-server-key") == "Bearer " + + +def test_m012_inference_paths_accept_a_managed_personal_token() -> None: + import inspect + + from scripts.qualification_sbc import ( + _benchmark_request, + _concurrent_finite_benchmark, + _request_timed, + _sequential_benchmark, + ) + + for function in ( + _request_timed, + _benchmark_request, + _sequential_benchmark, + _concurrent_finite_benchmark, + ): + assert "api_token" in inspect.signature(function).parameters