Repository navigation
Conversation
python -m evals.online_mind2web runs a 16-task live-web slice (seed 116, 12 easy + 4 medium public-read ids). Task text is fetched on demand from the gated official file; success is WebJudge at threshold 3; failures stay in the denominator. Offline tests use a fixture file. No live numbers yet. Closes #116
Birfy
left a comment
There was a problem hiding this comment.
Review (Birfy) — comment only, own PR, not approving
Head c5491665, CI green (core 3.11/3.13, full, benchmarks). This matches #116: a committed 16-id slice (seed 116, 12 easy + 4 medium), task text downloaded at a pinned revision rather than vendored, every attempt kept in the denominator, and a finished episode without a judge recorded as unevaluated rather than a pass. Licence/NOTICE/CHANGELOG and the live-vs-#109/#104 labelling look right. Offline tests cover selection, fetch auth, failure reasons, and the result table. No live numbers in this PR, as stated.
One real bug in a documented path:
--judge-scores does not honour task_id__arm keys. evals/online_mind2web/__main__.py says scores may be keyed by task_id or task_id__arm, but load_score_judge never sets judge.arm. It therefore looks up f"{task.id}__" (empty arm) and falls back to task.id. A two-arm run (--arms jev llm) will apply the same score to both arms, and keys like 0632e496…__jev are ignored. Please thread the arm into the judge (or change the docstring and only accept task_id) and add a test. This is not a blocker for a first single-arm Jev run, but it will mis-score the chat-model comparison the docs already describe.
Two non-blocking notes for the first live run:
- WebJudge is applied after the fact via
--judge-scores; the runner does not call the official evaluator itself. That is documented and fine, but a results folder is not complete until that external step (or an in-process hook) exists. is_public_readis task-text only, as #116 asked. Sites in this slice (Ticketmaster, Coursera, Eventbrite, Apartments.com) can still present a login or consent wall. Worth marking in the results write-up if a live attempt dies on that, rather than treating it as a model failure.
I am not approving this PR (it is mine). The harness itself looks ready for review once the judge-key lookup is fixed or the docs are narrowed.
Required human contributor answers
Answer every question before implementation and PR submission, including drafts.
Reuse explicit, still-applicable human answers with a traceable source reference;
ask the human for anything missing and wait. An agent must not invent answers.
Blank fields, placeholders, or an agent-written rationale do not satisfy this gate.
Share only information and source links appropriate for this PR's audience.
What outcome are you trying to achieve?
Answer: A small, fixed, seeded live-web slice so Agents can report end-to-end browser completion on realistic sites, with failures and timeouts in the denominator. Source: issue [Eval] Add a small Online-Mind2Web live slice for browser agents (distinct from offline myjev steps) #116.
What is the highest-leverage variable for that outcome?
Answer: A committed task-id list and seed, plus a runner that records every planned attempt under a pinned model and budget. Source: issue [Eval] Add a small Online-Mind2Web live slice for browser agents (distinct from offline myjev steps) #116 proposed work and acceptance criteria.
How will this work move that variable rather than create busywork?
Answer: It adds
python -m evals.online_mind2webagainst a 16-task public-read subset of Official Online-Mind2Web (seed 116, 12 easy + 4 medium), downloads task text at a pinned revision instead of vendoring the 300-task set, and grades with WebJudge (threshold 3) rather than inventing a new site verifier. Source: issue [Eval] Add a small Online-Mind2Web live slice for browser agents (distinct from offline myjev steps) #116.Who is the customer or user, and what do they actually need?
Answer: Agents contributors comparing browser backends on live web tasks, distinct from the [Eval] Score browser decision backends step by step on a public Mind2Web/WebLINX set in the jev-ultrafast request format #109 offline step collector and from the [Eval] Add an Anthropic browser_toolset_20260801 baseline on WebVoyager Allrecipes task 0 #104 Anthropic browser_toolset baseline. Source: issue [Eval] Add a small Online-Mind2Web live slice for browser agents (distinct from offline myjev steps) #116.
After answering the four questions: is a PR needed at all to achieve this outcome?
Human decision and rationale (or explicit source reference): Yes. Issue #116 asks for a committed slice, a runner, docs, and licence/attribution; that has to land in the repo. Danyang asked scheduled coding runs to fill these intent items from the issue and not wait for a separate human answer.
For a bug fix: link actual observed failure evidence.
Not applicable: this is a new eval harness, not a bug fix.
Evidence: N/A
Why
Browser numbers in this repo come from a handful of live tasks (Allrecipes, Flights) plus the offline step collector in #109 / #111. That offline leg does not answer whether the full agent finishes a realistic live-web task. #116 asks for a small, seeded Online-Mind2Web slice as that second live denominator.
How
evals/online_mind2webfollows the WebHarbor slice pattern, adapted for the live web (no local site reset):slice.jsonpinsosunlp/Online-Mind2Webat6aa56e07c9d247fcc2b72fc3d94ad0f857f1e5d4(includes the 2026-05-15 task updates) and the 16 ids.random.Random(116).sampleof 12 easy + 4 medium, then sorted by id.fetchdownloads the official file (gated;HF_TOKEN);selectre-derives the ids.runstarts a cleanbrowseprofile per attempt, default arm hosted Jev, records Agents SHA, model, budget and date.unevaluated, never a pass. Timeouts, errors andBLOCKEDstay in the denominator.Review first:
evals/online_mind2web/slice.json,tasks.py,runner.py,docs/online-mind2web.md.What
python -m evals.online_mind2web {show,select,fetch,run,score}docs/online-mind2web.md, a results placeholder, a labelled live-suite section indocs/benchmarks.md, a row inevals/README.md, NOTICE / THIRD_PARTY_LICENSES (CC BY 4.0 dataset; WebJudge MIT, not vendored)--modelbackend. No live traces in this PR: a run needsHF_TOKEN, a Jev key, and the live web.Verification
uv run ruff format --check . && uv run ruff check . && uv run ty checkuv run pytest -q(1373 passed, 132 skipped) andscripts/smoke.shCHANGELOG.mdand the docs say what the code does nowDemo / evidence
HF_TOKEN, a Jev key, and live sites). Follow-up is a results folder underdocs/results/online-mind2web/once a run is recorded. Implementation is otherwise complete, so this is ready for review rather than left as a draft.c549166onissue-116-online-mind2web, against mainc4da7c3.Closes #116