Repository navigation
perf: cut slow pages and APIs found in Sentry traces - #92
Merged
Merged
Conversation
Production traces showed /experiments/{id}/ (p95 9.3 s), /runs/data/
(p95 3.1 s, the 502) and /experiments/new/ dominated by loading every
ScenarioResult.result transcript to read a severity.
Data layer
- ScenarioResult keeps trial counts (n_pass, n_judged, n_errors,
n_ungraded) and a small list_summary projection, kept in step by save();
migration 0011 backfills them in batches. pass_counts is one GROUP BY;
the run Results list, CSV export and compare read the projection.
- Indexes: AuditRun (project/archived/created_at, status/archived,
monitor/created_at, started_at), AuditEvent (run/kind/item; drop three
single-column ones), ScenarioResult (run/status), OtlpSpan (received_at).
- AuditEvent: turn events are bulk-inserted per batch; the sweeper prunes
per-turn events of runs finished > SIMPLEAUDIT_EVENT_RETENTION_DAYS (30);
manage.py prune_audit_events.
Web
- One membership query per request feeds access, admin flag, workspace
switcher and can_write; token-auth requests no longer write sessions.
- SSE: per-user and per-process stream caps (429 + retry), idle backoff,
keepalive, and an end for finished runs missing a terminal event.
- API: run list and results are paginated (limit/offset); run-list rows are
slim, ?expand=snapshots restores the full shape. Event poll capped at 1000.
- Exports stream (run JSON/CSV, runs CSV, API export encoding).
- Agent page Open WebUI sync throttled (SIMPLEAUDIT_OPENWEBUI_SYNC_TTL_S);
scenarios edit forms load on demand; compare/experiment/runs grid defer
unused JSON; worker counters are one aggregate; concurrent model and
health probes; DB-side prompt-version check; static Cache-Control;
Caddy compression and upstream timeouts.
docs/performance-audit.md records the audit, the Sentry evidence and status.
…summary-columns # Conflicts: # infra/ui.py # infra/worker.py
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
This was referenced Nov 11, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production traces (Sentry, 14 days) showed
/experiments/{id}/at p95 9.3 s,/runs/data/at p95 3.1 s (the dashboard 502) and/experiments/new/at p95 3.4 s. Span data put almost all of it on one query: loading everyScenarioResult.resulttranscript to read a severity. Full audit, evidence and status:docs/performance-audit.md.Data layer
ScenarioResultkeeps trial counts and a smalllist_summaryprojection, kept in step bysave(). Migrationaudits/0011backfills them in batches.pass_countsbecomes oneGROUP BY; the run Results list, CSV export and compare read the projection.AuditRun,AuditEvent,ScenarioResult,OtlpSpan(audits/0012,model_registry/0023).AuditEvent: batched turn-event inserts; hourly pruning of per-turn events of runs finished more thanSIMPLEAUDIT_EVENT_RETENTION_DAYS(30) ago;manage.py prune_audit_events.Web
limit/offset,{count, next, previous, results}); run-list rows are slim,?expand=snapshotsrestores the full shape. Breaking for external API clients of these two endpoints.Upgrade notes
audits/0011reads every stored result once; the new indexes lock their tables briefly.SIMPLEAUDIT_SSE_MAX_STREAMS_PER_USER,SIMPLEAUDIT_SSE_MAX_STREAMS,SIMPLEAUDIT_EVENT_RETENTION_DAYS,SIMPLEAUDIT_OPENWEBUI_SYNC_TTL_S(documented indocs/deployment.mdand the.envexamples).Checks
pytest -m "not embedded_hatchet": 1810 passed after mergingmain(the 126 collection errors are pre-existing duplicates under.claude/worktrees/).pytest -m embedded_hatchet(worker changed): passed.ruff check .clean,makemigrations --checkclean.caddynot installed locally).