Skip to content

[Eval] Seeded Online-Mind2Web live slice for browser agents (WebJudge-graded) - #125

Open
Birfy wants to merge 1 commit into
mainfrom
issue-116-online-mind2web
Open

Birfy wants to merge 1 commit into
mainfrom
issue-116-online-mind2web

Conversation

@Birfy

@Birfy Birfy commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Required human contributor answers

Answer every question before implementation and PR submission, including drafts.
Reuse explicit, still-applicable human answers with a traceable source reference;
ask the human for anything missing and wait. An agent must not invent answers.
Blank fields, placeholders, or an agent-written rationale do not satisfy this gate.
Share only information and source links appropriate for this PR's audience.

  1. What outcome are you trying to achieve?

    Answer: A small, fixed, seeded live-web slice so Agents can report end-to-end browser completion on realistic sites, with failures and timeouts in the denominator. Source: issue [Eval] Add a small Online-Mind2Web live slice for browser agents (distinct from offline myjev steps) #116.

  2. What is the highest-leverage variable for that outcome?

    Answer: A committed task-id list and seed, plus a runner that records every planned attempt under a pinned model and budget. Source: issue [Eval] Add a small Online-Mind2Web live slice for browser agents (distinct from offline myjev steps) #116 proposed work and acceptance criteria.

  3. How will this work move that variable rather than create busywork?

    Answer: It adds python -m evals.online_mind2web against a 16-task public-read subset of Official Online-Mind2Web (seed 116, 12 easy + 4 medium), downloads task text at a pinned revision instead of vendoring the 300-task set, and grades with WebJudge (threshold 3) rather than inventing a new site verifier. Source: issue [Eval] Add a small Online-Mind2Web live slice for browser agents (distinct from offline myjev steps) #116.

  4. Who is the customer or user, and what do they actually need?

    Answer: Agents contributors comparing browser backends on live web tasks, distinct from the [Eval] Score browser decision backends step by step on a public Mind2Web/WebLINX set in the jev-ultrafast request format #109 offline step collector and from the [Eval] Add an Anthropic browser_toolset_20260801 baseline on WebVoyager Allrecipes task 0 #104 Anthropic browser_toolset baseline. Source: issue [Eval] Add a small Online-Mind2Web live slice for browser agents (distinct from offline myjev steps) #116.

After answering the four questions: is a PR needed at all to achieve this outcome?

Human decision and rationale (or explicit source reference): Yes. Issue #116 asks for a committed slice, a runner, docs, and licence/attribution; that has to land in the repo. Danyang asked scheduled coding runs to fill these intent items from the issue and not wait for a separate human answer.

For a bug fix: link actual observed failure evidence.
Not applicable: this is a new eval harness, not a bug fix.

Evidence: N/A

Why

Browser numbers in this repo come from a handful of live tasks (Allrecipes, Flights) plus the offline step collector in #109 / #111. That offline leg does not answer whether the full agent finishes a realistic live-web task. #116 asks for a small, seeded Online-Mind2Web slice as that second live denominator.

How

evals/online_mind2web follows the WebHarbor slice pattern, adapted for the live web (no local site reset):

  • slice.json pins osunlp/Online-Mind2Web at 6aa56e07c9d247fcc2b72fc3d94ad0f857f1e5d4 (includes the 2026-05-15 task updates) and the 16 ids.
  • Selection is documented and checked: public-read tasks only (no login/payment/cart/favorites/account-write text), random.Random(116).sample of 12 easy + 4 medium, then sorted by id.
  • fetch downloads the official file (gated; HF_TOKEN); select re-derives the ids.
  • run starts a clean browse profile per attempt, default arm hosted Jev, records Agents SHA, model, budget and date.
  • Success is official WebJudge at threshold 3. Without a judge, a finished episode is unevaluated, never a pass. Timeouts, errors and BLOCKED stay in the denominator.

Review first: evals/online_mind2web/slice.json, tasks.py, runner.py, docs/online-mind2web.md.

What

  • python -m evals.online_mind2web {show,select,fetch,run,score}
  • Docs: docs/online-mind2web.md, a results placeholder, a labelled live-suite section in docs/benchmarks.md, a row in evals/README.md, NOTICE / THIRD_PARTY_LICENSES (CC BY 4.0 dataset; WebJudge MIT, not vendored)
  • No new --model backend. No live traces in this PR: a run needs HF_TOKEN, a Jev key, and the live web.

Verification

  • uv run ruff format --check . && uv run ruff check . && uv run ty check
  • uv run pytest -q (1373 passed, 132 skipped) and scripts/smoke.sh
  • CHANGELOG.md and the docs say what the code does now

Demo / evidence

  • Video link, or exemption/blocker with follow-up: no live run in this PR (needs HF_TOKEN, a Jev key, and live sites). Follow-up is a results folder under docs/results/online-mind2web/ once a run is recorded. Implementation is otherwise complete, so this is ready for review rather than left as a draft.
  • Application/task → agent actions → independently checked result: not yet; the harness records traces and applies WebJudge scores when provided.
  • System1-Agents / System1-Omni commits, checkpoint and worker/frontend provenance: this branch is c549166 on issue-116-online-mind2web, against main c4da7c3.

Closes #116

python -m evals.online_mind2web runs a 16-task live-web slice (seed 116,
12 easy + 4 medium public-read ids). Task text is fetched on demand from
the gated official file; success is WebJudge at threshold 3; failures
stay in the denominator. Offline tests use a fixture file. No live numbers yet.

Closes #116

@Birfy Birfy left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (Birfy) — comment only, own PR, not approving

Head c5491665, CI green (core 3.11/3.13, full, benchmarks). This matches #116: a committed 16-id slice (seed 116, 12 easy + 4 medium), task text downloaded at a pinned revision rather than vendored, every attempt kept in the denominator, and a finished episode without a judge recorded as unevaluated rather than a pass. Licence/NOTICE/CHANGELOG and the live-vs-#109/#104 labelling look right. Offline tests cover selection, fetch auth, failure reasons, and the result table. No live numbers in this PR, as stated.

One real bug in a documented path:

--judge-scores does not honour task_id__arm keys. evals/online_mind2web/__main__.py says scores may be keyed by task_id or task_id__arm, but load_score_judge never sets judge.arm. It therefore looks up f"{task.id}__" (empty arm) and falls back to task.id. A two-arm run (--arms jev llm) will apply the same score to both arms, and keys like 0632e496…__jev are ignored. Please thread the arm into the judge (or change the docstring and only accept task_id) and add a test. This is not a blocker for a first single-arm Jev run, but it will mis-score the chat-model comparison the docs already describe.

Two non-blocking notes for the first live run:

  • WebJudge is applied after the fact via --judge-scores; the runner does not call the official evaluator itself. That is documented and fine, but a results folder is not complete until that external step (or an in-process hook) exists.
  • is_public_read is task-text only, as #116 asked. Sites in this slice (Ticketmaster, Coursera, Eventbrite, Apartments.com) can still present a login or consent wall. Worth marking in the results write-up if a live attempt dies on that, rather than treating it as a model failure.

I am not approving this PR (it is mine). The harness itself looks ready for review once the judge-key lookup is fixed or the docs are narrowed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Eval] Add a small Online-Mind2Web live slice for browser agents (distinct from offline myjev steps)

1 participant