A job's rate, the trials behind it, the controls, then a comparison.
- Open Runs, By launch or All runs.
- Open a job: its live board while it runs, its trial outcomes once finished.
Foreign results open the same way: Open existing results.
| Metric | Meaning |
|---|---|
| Raw solve rate | Resolved trials ÷ all trials; infrastructure failures count as misses |
| Adjusted solve rate | Resolved trials ÷ trials excluding infrastructure failures |
| Mean score | The average graded score, for scored benchmarks |
— |
No verdict, not a zero |
!!! note "The gap between raw and adjusted is infrastructure" It is what rate limits and setup failures cost; it does not predict the excluded tasks.
| Step | Open | Ask |
|---|---|---|
| 1 | Outcome and error | Did the attempt reach the task, or fail in setup or a provider call? |
| 2 | Verifier output | Which check rejected it? |
| 3 | Conversation and tool calls | Which actions led there? |
| 4 | Trace and artifacts | Which calls failed or ran slowly? What files were produced? |
- Controls: the
nopandoraclejobs beside the agent's (Core concepts). - Feasibility: a passing
nopor a failingoracleopens in the feasibility view. - Without controls: the run holds no evidence about task validity.
-
Open Compare: pick a workspace and a run in each column, Add a run for another, then choose Matched tasks. The first column is the baseline; every other number shows its difference from it.
-
Choose Task by task, start with the tasks whose outcomes differ and open the attempts side by side.
-
Choose Chart to draw the summary rows: each chart card picks its style (bars, dots or a scatter) and its metrics, and Add chart adds one.
-
Check dataset, model, agent and setup before attributing a change.
!!! warning "Different task sets move the aggregate" A score can change with the task set alone.
- Configure a judge in Settings.
- Request a judgement from a trial or a job.
- Read disagreements with the recorded outcome in Reconcile.
!!! warning "A judgement is a paid model call" Start with one trial; a batch button names how many calls it makes.
Questions, records and cost: Judgement.
Metric field guide, under Reference → Metrics on the documentation website.