diff --git a/CHANGELOG.md b/CHANGELOG.md index 37b99a8..8728541 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver ## Unreleased +### Added + +- `python -m evals.online_mind2web`: a seeded 16-task live-web slice of Online-Mind2Web (12 easy + 4 medium, seed 116; + ids, websites and the official revision only in `evals/online_mind2web/slice.json`; task text downloaded on demand + from the gated `osunlp/Online-Mind2Web` file). Each attempt starts a clean `browse` profile, plays hosted Jev (or + another arm) under a pinned budget, and records every outcome; success is WebJudge at threshold 3 (or `unevaluated` + if no judge ran). The table names Agents SHA, model, budget and date. Offline tests use a fixture file and a + scripted player; no live numbers yet (#116). + ## 0.4.0 - 2026-11-13 ### Added diff --git a/NOTICE b/NOTICE index d66d358..5d6a588 100644 --- a/NOTICE +++ b/NOTICE @@ -23,3 +23,7 @@ This product includes software developed by third parties: Beurer-Kellner, Marc Fischer, and Florian Tramèr). Full license text: THIRD_PARTY_LICENSES.md - s1a/decision_models/onejev.py: HTTP client for OneJev / qev (OmniJev/OneJev), Apache-2.0; the weights and server are not vendored. Details: THIRD_PARTY_LICENSES.md +- evals/online_mind2web: task ids, websites and difficulty of a 16-task slice of Online-Mind2Web + (osunlp/Online-Mind2Web; Xue et al., 2025, arXiv:2504.01382), CC BY 4.0. The official task file is + not included. WebJudge evaluation code is MIT (OSU-NLP-Group/Online-Mind2Web) and is not vendored. + Details: THIRD_PARTY_LICENSES.md diff --git a/THIRD_PARTY_LICENSES.md b/THIRD_PARTY_LICENSES.md index ac25900..1b95300 100644 --- a/THIRD_PARTY_LICENSES.md +++ b/THIRD_PARTY_LICENSES.md @@ -126,3 +126,21 @@ OneJev and its serving code are Apache License 2.0 (Copyright OmniJev Team). Thi vendor OneJev weights, training code, or the `qev`/`flide` packages; install and run them separately per the model card. The unrelated PyPI package named `qev` is not the OneJev server — install from the OneJev GitHub repository as the model card instructs. + +## Online-Mind2Web + +`evals/online_mind2web` measures a seeded 16-task live-web slice of +[Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web) (OSU NLP Group; Xue, Qi, Shi, Song, Gou, +Song, Sun, and Su, *An Illusion of Progress? Assessing the Current State of Web Agents*, 2025, +[arXiv:2504.01382](https://arxiv.org/abs/2504.01382)). The official task file is gated and is not vendored; only +task ids, start URLs, difficulty labels and the pinned dataset revision are committed, in +`evals/online_mind2web/slice.json`. Downstream users must accept the dataset terms and download +`Online_Mind2Web.json` at that revision themselves. + +The dataset is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). The dataset card also +asks that [Mind2Web](https://arxiv.org/abs/2306.06070) (Deng et al., 2023) be cited. The WebJudge automatic +evaluator and the rest of the [OSU-NLP-Group/Online-Mind2Web](https://github.com/OSU-NLP-Group/Online-Mind2Web) +code are MIT License (Copyright (c) 2025 OSU Natural Language Processing). This repository does not vendor that +code; a run may call it separately. + +This work is used here for research evaluation of System 1 browser agents. diff --git a/docs/benchmarks.md b/docs/benchmarks.md index abdf50f..8699c76 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -64,6 +64,19 @@ the harness continues without it. Both sides pay it. ![Allrecipes, Jev left, the chat model right, at 8x](assets/demos/allrecipes-comparison-8x.gif) + +## Browser use: Online-Mind2Web live slice + +A seeded 16-task **live-web** slice of [Online-Mind2Web](https://github.com/OSU-NLP-Group/Online-Mind2Web) +(Xue et al., 2025, [arXiv:2504.01382](https://arxiv.org/abs/2504.01382)) lives in +`evals/online_mind2web/slice.json` (seed 116, 12 easy + 4 medium public-read tasks; official file downloaded at +revision `6aa56e07c9d247fcc2b72fc3d94ad0f857f1e5d4`). This is an end-to-end live suite: it is not the offline +step-level collector in issue #109, and it is not the planned Anthropic `browser_toolset` baseline in issue #104. + +Run it with `python -m evals.online_mind2web`; details, judging (WebJudge at threshold 3) and licence notes are in +[online-mind2web.md](online-mind2web.md). No live numbers are committed yet. Traces belong under +`docs/results/online-mind2web/`. + ## A longer game Over 150 moves of 2048 the cost gap widens with the transcript. Seed 0: Jev scored 1,104 in 244 s for $0.003; the diff --git a/docs/online-mind2web.md b/docs/online-mind2web.md new file mode 100644 index 0000000..45b3144 --- /dev/null +++ b/docs/online-mind2web.md @@ -0,0 +1,57 @@ +# Online-Mind2Web live slice + +A seeded 16-task **live-web** slice of [Online-Mind2Web](https://github.com/OSU-NLP-Group/Online-Mind2Web) +(Xue et al., 2025, [arXiv:2504.01382](https://arxiv.org/abs/2504.01382); dataset +[osunlp/Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web), CC BY 4.0). It answers "does the +full agent finish a realistic live-web task?", which is a different question from: + +- the offline step-level collector in issue #109 / PR #111 (frozen page, right operation/element) +- the Allrecipes / WebVoyager task 0 row already in [benchmarks.md](benchmarks.md) +- the planned Anthropic `browser_toolset` baseline in issue #104 + +## The slice + +Ids, levels, start URLs and the official revision live in `evals/online_mind2web/slice.json`. Task text is **not** +vendored. The official file is gated: accept the dataset terms, set `HF_TOKEN`, then download the pinned revision +(the 2026-05-15 task updates are in this SHA): + +```bash +# HF_TOKEN must already be able to read osunlp/Online-Mind2Web +uv run python -m evals.online_mind2web fetch +uv run python -m evals.online_mind2web select --official .external/online_mind2web/Online_Mind2Web.json +``` + +`select` re-derives the 16 ids from the official file and seed `116`. The rule is: take the official tasks whose +text does not ask for a login, payment, cart, favorites or similar account write; `random.Random(116).sample` 12 +sorted easy ids and 4 sorted medium ids; commit that list sorted by id. At this revision the public-read pool is 76 +easy and 135 medium tasks. + +## Running + +Each attempt starts a new `browse` session in its own `logs_dir` (no shared browser profile) and uses the official +start URL plus the official task text as the goal. The first backend is hosted Jev (`--model jev`); pass `--arms jev +llm` to add the chat-model arm under the same `--max-steps` and `--timeout`. + +```bash +uv run python -m evals.online_mind2web run --official .external/online_mind2web/Online_Mind2Web.json --arms jev +``` + +`results.md` and `config.json` name the Agents SHA, the dataset revision, the model, the budget and the date. Every +planned attempt is in the denominator, including timeouts, agent errors, `BLOCKED` and unevaluated episodes. + +## Judging + +Success is [WebJudge](https://github.com/OSU-NLP-Group/Online-Mind2Web), the official auto-eval, at the upstream +default threshold of 3 (upstream now recommends GPT-6 Luna as the backbone). This repository does not vendor +WebJudge. Without a judge, a finished episode is recorded as `unevaluated`, never as a pass. After an external +WebJudge run you can apply integer scores with `--judge-scores scores.json` (a JSON object keyed by task id). + +## Results + +No live numbers are committed yet: a run needs `HF_TOKEN`, a Jev (and optionally chat-model) key, and a live network. +Place traces under `docs/results/online-mind2web/` and redact secrets. + +## Licence + +The dataset is CC BY 4.0; the WebJudge / evaluation code in the official repository is MIT. Attribution is in +`NOTICE` and `THIRD_PARTY_LICENSES.md`. Only ids, websites, levels and the revision are committed here. diff --git a/docs/results/online-mind2web/README.md b/docs/results/online-mind2web/README.md new file mode 100644 index 0000000..9c56057 --- /dev/null +++ b/docs/results/online-mind2web/README.md @@ -0,0 +1,27 @@ +# Online-Mind2Web live slice results + +Live traces for the seeded 16-task slice in `evals/online_mind2web/slice.json` (seed 116, 12 easy + 4 medium, +dataset `osunlp/Online-Mind2Web` at `6aa56e07c9d247fcc2b72fc3d94ad0f857f1e5d4`). + +No run is committed yet. A later results folder should name the Agents SHA, the model or checkpoint, the budget +(`max_steps`, `timeout_s`) and the date, keep failures and timeouts in the denominator, and redact API keys from +raw traces. Success is WebJudge at threshold 3. + +Task ids (sorted): + +- `0632e496d37badee0350dad358f047c5` easy — https://cookpad.com/ +- `207e933d1bba815bcb58664b5d82c085` medium — https://www.apartments.com/ +- `2532fd402d3c741b79894e6ff2269f53` easy — https://www.thumbtack.com/ +- `34ccd15a8ea8fd3895af83f5ccf62369` easy — https://us.megabus.com/ +- `3621b099326c7aebd2e2dac6be3b52d1` easy — https://www.nvidia.com/ +- `5e1b8254c123c80178cc28e0afdb14f0_110325` easy — https://ticketmaster.com/ +- `6db4a0e346976f2729ba9afcd3208941` easy — https://www.fedex.com/en-us/home.html +- `783ce6a3499fa7cf25bc12f8f0ecbbbb` easy — https://www.mayoclinic.org/ +- `864244b6969e0f8733b0eb1ca06cd51f` medium — https://www.espn.com/ +- `a11ecdff735b51372d536c866011af6f` easy — https://www.coursera.org/ +- `a6f0434ce6aff5f9b03681241b03ad82` easy — https://finance.yahoo.com/ +- `bd1e3770b7181f6fce9c35e18caa9785` easy — https://craigslist.org/ +- `c698ff3fc0f6cbce39947c597ab5749b_121425` medium — https://www.eventbrite.com/ +- `d5c34bf39eb6096ae5d439325cde4d32` easy — https://www.dmv.virginia.gov/ +- `d71be72aa25c3eab8eea47a0e60382e2` easy — https://www.apple.com/ +- `fa9adb815b85d259f943d81874a052e5` medium — https://soundcloud.com/ diff --git a/evals/README.md b/evals/README.md index 857cc09..d691e35 100644 --- a/evals/README.md +++ b/evals/README.md @@ -18,6 +18,7 @@ folder per episode). `uv run python -m evals.table evals/results` aggregates the | Injection guard on page-length HTML (BrowseSafe-Bench sample) | precision, recall, failures and page coverage at the runtime clip | `s1a run injection_guard --labelled-set .external/browsesafe/test-sample-html.jsonl`, after `scripts/build_browsesafe_set.py build`; see `docs/benchmarks.md` | | Bounded recovery on/off (3 local form fixtures) | completion on/off in a real headless Chromium, judged by the fixture server | `python -m evals.recovery --repeat 1` | | Bounded recovery live (real decision/chat models, 3 local form fixtures) | completion on/off with the real models in a real headless Chromium, judged by the fixture server; cost null unless `CHAT_USD_PER_M_*` is set; optional `--validate-submission` variant | `python -m evals.recovery.live --model jev --repeat 1` | +| Online-Mind2Web live slice (16 seeded tasks) | live-web task completion, WebJudge-graded, failures in the denominator | `python -m evals.online_mind2web run`; see `docs/online-mind2web.md` | ## The loop diff --git a/evals/online_mind2web/__init__.py b/evals/online_mind2web/__init__.py new file mode 100644 index 0000000..876b95a --- /dev/null +++ b/evals/online_mind2web/__init__.py @@ -0,0 +1,8 @@ +# coding: utf-8 +"""A seeded Online-Mind2Web live slice: clean-profile browser tasks, WebJudge-graded (issue 116). + +``tasks`` loads the committed slice (ids, levels, websites and the official revision; never the task text) and the +official file downloaded on demand; ``runner`` plays every (task, arm) attempt on the live web and records each one, +failures included; ``score`` turns the attempts into the result table. Success is WebJudge (threshold 3), not an +Allrecipes-style site check and not the #109 offline step collector. +""" diff --git a/evals/online_mind2web/__main__.py b/evals/online_mind2web/__main__.py new file mode 100644 index 0000000..0e5cdf8 --- /dev/null +++ b/evals/online_mind2web/__main__.py @@ -0,0 +1,197 @@ +# coding: utf-8 +"""``python -m evals.online_mind2web {show,select,fetch,run,score}``: the seeded Online-Mind2Web live slice (#116). + +- ``show`` prints the committed slice (ids, levels, websites and the pinned revision). +- ``select --official Online_Mind2Web.json`` re-derives the ids from the official file and the slice's seed, to check them. +- ``fetch --dest FILE`` downloads the official file at the pinned revision (network, gated; needs ``HF_TOKEN``). + Nothing is vendored. +- ``run --official FILE --arms jev`` plays each task through ``browse`` with a decision model or the chat model under + the same ``--max-steps`` and ``--timeout``, writes ``attempts.jsonl`` and ``results.md``. Needs the models' keys and + a live network. Without a judge, finished episodes are recorded as ``unevaluated`` (not a pass). ``--judge-scores`` + applies WebJudge-style integer scores from a JSON object keyed by ``task_id`` (or ``task_id__arm``). +- ``score attempts.jsonl`` re-renders the table from recorded attempts. +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path + +from evals.online_mind2web import score, tasks + +RESULTS = Path("evals/results/online_mind2web") + + +def parser() -> argparse.ArgumentParser: + build = argparse.ArgumentParser( + prog="python -m evals.online_mind2web", + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + build.add_argument("--slice", type=Path, default=tasks.SLICE_PATH) + commands = build.add_subparsers(dest="command", required=True) + commands.add_parser("show") + sel = commands.add_parser("select") + sel.add_argument("--official", type=Path, required=True) + fetch = commands.add_parser("fetch") + fetch.add_argument("--dest", type=Path, default=Path(".external/online_mind2web/Online_Mind2Web.json")) + run = commands.add_parser("run") + run.add_argument("--official", type=Path, default=Path(".external/online_mind2web/Online_Mind2Web.json")) + run.add_argument("--arms", nargs="+", default=["jev"]) + run.add_argument("--repeat", type=int, default=1) + run.add_argument("--max-steps", type=int, default=40) + run.add_argument("--timeout", type=float, default=600.0) + run.add_argument("--headed", action="store_true") + run.add_argument("--judge-scores", type=Path, default=None, help="JSON object of WebJudge integer scores") + run.add_argument("--judge-model", default=None, help="name recorded next to --judge-scores") + run.add_argument("--threshold", type=int, default=score.JUDGE_THRESHOLD) + run.add_argument("--out", type=Path, default=None) + sc = commands.add_parser("score") + sc.add_argument("attempts", type=Path) + sc.add_argument("--agents-sha", default=None) + sc.add_argument("--model", default=None) + sc.add_argument("--budget", default=None) + sc.add_argument("--date", default=None) + return build + + +def agents_sha() -> str: + """The checkout's HEAD, or N/A when this is not a git work tree.""" + try: + result = subprocess.run(["git", "rev-parse", "HEAD"], check=True, capture_output=True, text=True, timeout=10) + except (OSError, subprocess.SubprocessError): + return score.NA + sha = result.stdout.strip() + return sha if sha else score.NA + + +def load_score_judge(path: Path, model: str | None): + """A judge that reads integer scores from a JSON object; used in tests and for applying an offline WebJudge run.""" + raw = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(raw, dict): + raise tasks.SliceError(f"{path} must be a JSON object of task ids to integer scores") + + def judge(task, outcome, logs_dir): + del outcome, logs_dir + value = raw.get(f"{task.id}__{getattr(judge, 'arm', '')}", raw.get(task.id)) + if not isinstance(value, int) or isinstance(value, bool): + raise tasks.SliceError(f"no integer WebJudge score for {task.id} in {path}") + return value + + judge.model = model + return judge + + +def main(argv: list[str] | None = None) -> int: + args = parser().parse_args(argv) + try: + slice_ = tasks.load_slice(args.slice) + except tasks.SliceError as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 + if args.command == "show": + print(f"{slice_.name}: {slice_.dataset} {slice_.source_file} @ {slice_.revision} (seed {slice_.seed})") + for task in slice_.tasks: + print(f"{task.id}\t{task.level}\t{task.reference_length}\t{task.website}") + return 0 + if args.command == "select": + try: + official = tasks.load_official(args.official) + tasks.check_selection(slice_, official) + except tasks.SliceError as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 + for task_id in slice_.ids: + print(task_id) + return 0 + if args.command == "fetch": + try: + path = tasks.fetch_official(slice_, args.dest) + except tasks.SliceError as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 + print(f"wrote {path} ({slice_.expected_tasks} tasks @ {slice_.revision})") + return 0 + if args.command == "score": + from evals.online_mind2web.runner import load_attempts + + rows = load_attempts(args.attempts) + sha = args.agents_sha or next((row.agents_sha for row in rows if row.agents_sha), score.NA) + print( + score.render( + rows, + revision=slice_.revision, + dataset=slice_.dataset, + agents_sha=sha, + model=args.model or score.NA, + budget=args.budget or score.NA, + date=args.date or score.NA, + ), + end="", + ) + return 0 + return asyncio.run(_run(args, slice_)) + + +async def _run(args: argparse.Namespace, slice_: tasks.Slice) -> int: + import s1a.entry # noqa: F401 # routes the harness logs to files before anything imports openjiuwen + from s1a.run import started_runner + + from evals.online_mind2web.live import LiveBudget, make_player + from evals.online_mind2web.runner import run_slice + + try: + official = tasks.bind(slice_, tasks.load_official(args.official)) + except (tasks.SliceError, FileNotFoundError) as exc: + print(f"error: {exc} (run `python -m evals.online_mind2web fetch` first)", file=sys.stderr) + return 2 + judge = load_score_judge(args.judge_scores, args.judge_model) if args.judge_scores else None + out = args.out or RESULTS / f"{datetime.now():%Y-%m-%d__%H-%M-%S}" + budget = LiveBudget(args.max_steps, args.timeout, not args.headed) + sha = agents_sha() + stamp = datetime.now(timezone.utc).astimezone().strftime("%Y-%m-%d") + budget_label = f"max_steps={budget.max_steps} timeout_s={budget.timeout_s:g}" + play = make_player(budget) + async with started_runner(): + attempts = await run_slice( + official, args.arms, play, out, repeat=args.repeat, judge=judge, agents_sha=sha, threshold=args.threshold + ) + table = score.render( + attempts, + revision=slice_.revision, + dataset=slice_.dataset, + agents_sha=sha, + model=",".join(args.arms), + budget=budget_label, + date=stamp, + threshold=args.threshold, + ) + (out / "results.md").write_text(table, encoding="utf-8") + (out / "config.json").write_text( + json.dumps( + { + "dataset": slice_.dataset, + "revision": slice_.revision, + "agents_sha": sha, + "arms": args.arms, + "budget": {"max_steps": budget.max_steps, "timeout_s": budget.timeout_s, "headless": budget.headless}, + "date": stamp, + "threshold": args.threshold, + "tasks": list(slice_.ids), + }, + indent=1, + ) + + "\n", + encoding="utf-8", + ) + print(table, end="") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/online_mind2web/live.py b/evals/online_mind2web/live.py new file mode 100644 index 0000000..8da694e --- /dev/null +++ b/evals/online_mind2web/live.py @@ -0,0 +1,78 @@ +# coding: utf-8 +"""The real player: one live-web task through the production ``browse`` front with a decision model or the chat model. + +``--model jev`` (or another decision model name) fills the model slot through ``build_model``; ``llm`` lets the chat +model decide every step. Both get the same budget (``--max-steps``, ``--timeout``). Each call uses its own ``logs_dir`` +as the browser workspace, so attempts do not share a profile. Decision time is the sum of the counted decision-model +calls (retries and failures included); the chat-model arm has none, so its decision time is N/A, not zero. Needs a +started Runner (``started_runner``). +""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path + +from evals.online_mind2web.runner import Outcome, Player +from evals.online_mind2web.tasks import OfficialTask, goal + + +@dataclass(frozen=True) +class LiveBudget: + max_steps: int = 40 + timeout_s: float = 600.0 + headless: bool = True + + +def make_player(budget: LiveBudget) -> Player: + from s1a.browser import browse, prompts + from s1a.browser.decision_model import BrowserPolicy + from s1a.config import chat_model_from_env + from s1a.decision_models import build_model + from s1a.spec import BrowserAgentSpec, Budget + + from evals.recovery.live import CountingDecisionModel + + chat = chat_model_from_env() + + async def play(task: OfficialTask, arm: str, logs_dir: Path) -> Outcome: + spec = BrowserAgentSpec( + name=f"online_mind2web_{task.id}", + description=f"Online-Mind2Web live task {task.id} on {task.website}.", + rules=prompts.OPERATION_RULES["en"], + language="en", + budget=Budget(max_steps=budget.max_steps, timeout_s=budget.timeout_s, stall_after=3), + goal=None, + ) + proxy = None if arm == "llm" else CountingDecisionModel(build_model(arm)) + try: + answer = await browse.browse( + spec, + BrowserPolicy(prefetch_values=False, batch_actions=False, goal_value_cache=False, rethink_on=False), + model_name=arm, + goal=goal(task), + timeout_s=budget.timeout_s, + max_steps=budget.max_steps, + logs_dir=logs_dir, + headless=budget.headless, + chat=chat, + decision_model=proxy, + ) + finally: + if proxy is not None: + await proxy.close() + error = answer.get("error") + if error and str(error).startswith("timeout"): + error = "timeout" + elif error: + error = "agent: " + str(error)[:120] + screenshot = answer.get("screenshot") + return Outcome( + answer=str(answer.get("final") or ""), + error=None if answer.get("ok") else (error or "agent: not ok"), + total_ms=answer.get("elapsed_ms"), + decision_ms=None if proxy is None else sum(int(call.get("ms") or 0) for call in proxy.calls), + screenshot=Path(screenshot) if screenshot else None, + ) + + return play diff --git a/evals/online_mind2web/runner.py b/evals/online_mind2web/runner.py new file mode 100644 index 0000000..c5e58b9 --- /dev/null +++ b/evals/online_mind2web/runner.py @@ -0,0 +1,167 @@ +# coding: utf-8 +"""Play every (task, arm) attempt on the live web and record each one, failures included. + +There is no site reset (this is the live web, not WebHarbor). Each attempt starts a new browser in its own +``logs_dir``, so profiles are not reused across tasks. A missing judge is ``unevaluated``, never a silent drop. +""" + +from __future__ import annotations + +import json +from collections.abc import Awaitable, Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Protocol + +from evals.online_mind2web.score import JUDGE_THRESHOLD, Attempt +from evals.online_mind2web.tasks import OfficialTask + +Player = Callable[["OfficialTask", str, Path], Awaitable["Outcome"]] + + +class Judge(Protocol): + """Score one finished episode; return a 1–5 WebJudge-style integer, or raise to record a judge error.""" + + def __call__(self, task: OfficialTask, outcome: "Outcome", logs_dir: Path) -> int: ... + + +@dataclass(frozen=True) +class Outcome: + answer: str + error: str | None = None # timeout / BLOCKED / agent: ... + total_ms: int | None = None + decision_ms: int | None = None + screenshot: Path | None = None + + +def classify(error: str | None) -> str | None: + """Map a player error to a table reason; None means the episode finished and still needs a judge.""" + if not error: + return None + text = error.strip() + lowered = text.lower() + if lowered.startswith("timeout"): + return "timeout" + if "blocked" in lowered: + return "BLOCKED" + return text if text.startswith("agent:") else "agent: " + text[:120] + + +def judged_attempt( + task: OfficialTask, + arm: str, + outcome: Outcome, + *, + judge: Judge | None, + logs_dir: Path, + agents_sha: str | None, + threshold: int = JUDGE_THRESHOLD, +) -> Attempt: + """An attempt row: player errors never reach the judge; a missing judge is unevaluated, not a pass.""" + failure = classify(outcome.error) + if failure is not None: + return Attempt( + task.id, + arm, + False, + failure, + outcome.total_ms, + outcome.decision_ms, + agents_sha=agents_sha, + ) + if judge is None: + return Attempt( + task.id, + arm, + False, + "unevaluated", + outcome.total_ms, + outcome.decision_ms, + agents_sha=agents_sha, + ) + try: + score = judge(task, outcome, logs_dir) + except Exception as exc: # a judge bug or missing key must not drop the attempt + return Attempt( + task.id, + arm, + False, + "judge: " + str(exc)[:120], + outcome.total_ms, + outcome.decision_ms, + agents_sha=agents_sha, + ) + passed = score >= threshold + return Attempt( + task.id, + arm, + passed, + None if passed else f"judge:{score}", + outcome.total_ms, + outcome.decision_ms, + judge_score=score, + judge_model=getattr(judge, "model", None), + agents_sha=agents_sha, + ) + + +async def attempt( + task: OfficialTask, + arm: str, + play: Player, + out: Path, + *, + judge: Judge | None = None, + agents_sha: str | None = None, + threshold: int = JUDGE_THRESHOLD, +) -> Attempt: + out.mkdir(parents=True, exist_ok=True) + try: + outcome = await play(task, arm, out) + except Exception as exc: + outcome = Outcome(answer="", error="agent: " + str(exc)[:120]) + row = judged_attempt(task, arm, outcome, judge=judge, logs_dir=out, agents_sha=agents_sha, threshold=threshold) + (out / "attempt.json").write_text(json.dumps(row.to_json(), indent=1), encoding="utf-8") + return row + + +async def run_slice( + tasks: list[OfficialTask], + arms: list[str], + play: Player, + out_root: Path, + *, + repeat: int = 1, + judge: Judge | None = None, + agents_sha: str | None = None, + threshold: int = JUDGE_THRESHOLD, +) -> list[Attempt]: + """Every attempt in a fixed order; arms alternate which goes first per task so neither always runs first.""" + if repeat < 1: + raise ValueError("repeat must be >= 1") + if not arms: + raise ValueError("at least one arm is required") + attempts: list[Attempt] = [] + results = out_root / "attempts.jsonl" + out_root.mkdir(parents=True, exist_ok=True) + for index, task in enumerate(tasks): + for r in range(repeat): + order = arms if (index + r) % 2 == 0 else list(reversed(arms)) + for arm in order: + row = await attempt( + task, + arm, + play, + out_root / f"{task.id}__{arm}__r{r}", + judge=judge, + agents_sha=agents_sha, + threshold=threshold, + ) + attempts.append(row) + with results.open("a", encoding="utf-8") as handle: + handle.write(json.dumps(row.to_json()) + "\n") + return attempts + + +def load_attempts(path: Path) -> list[Attempt]: + return [Attempt.from_json(json.loads(line)) for line in path.read_text(encoding="utf-8").splitlines() if line] diff --git a/evals/online_mind2web/score.py b/evals/online_mind2web/score.py new file mode 100644 index 0000000..dd17327 --- /dev/null +++ b/evals/online_mind2web/score.py @@ -0,0 +1,132 @@ +# coding: utf-8 +"""Attempts to the result table: completion with every attempt in the denominator, latency split, failure reasons. + +A number that was not measured is "N/A", never zero. Success is a WebJudge verdict at the official threshold (default +3); an episode that finished without a judge is ``unevaluated``, not a pass. Timeouts, agent errors, BLOCKED and +judge failures stay in the denominator. +""" + +from __future__ import annotations + +import math +import statistics +from collections import Counter +from dataclasses import asdict, dataclass +from typing import Any, Iterable + +NA = "N/A" +JUDGE_THRESHOLD = 3 + + +@dataclass(frozen=True) +class Attempt: + task_id: str + arm: str + passed: bool + failure: str | None # None only when passed + total_ms: int | None = None + decision_ms: int | None = None # None when the arm makes no decision-model calls + judge_score: int | None = None + judge_model: str | None = None + agents_sha: str | None = None + + @property + def environment_ms(self) -> int | None: + """Episode time outside decision calls: browser, page loads and the chat model.""" + if self.total_ms is None or self.decision_ms is None: + return None + return max(self.total_ms - self.decision_ms, 0) + + def to_json(self) -> dict[str, Any]: + return {**asdict(self), "environment_ms": self.environment_ms} + + @classmethod + def from_json(cls, data: dict[str, Any]) -> "Attempt": + keys = {key: data.get(key) for key in cls.__dataclass_fields__} + return cls(**keys) + + +def wilson(hits: int, total: int, z: float = 1.96) -> tuple[float, float] | None: + """The 95% Wilson score interval for ``hits`` of ``total``; None when ``total`` is 0.""" + if total <= 0: + return None + if not 0 <= hits <= total: + raise ValueError(f"hits {hits} outside 0..{total}") + p = hits / total + denom = 1 + z * z / total + centre = (p + z * z / (2 * total)) / denom + half = z * math.sqrt(p * (1 - p) / total + z * z / (4 * total * total)) / denom + return max(0.0, centre - half), min(1.0, centre + half) + + +def _median(values: Iterable[int | None]) -> float | None: + known = [v for v in values if v is not None] + return statistics.median(known) if known else None + + +def summarize(attempts: list[Attempt]) -> dict[str, dict[str, Any]]: + """Per arm, in first-seen order: attempts, passes, rate, Wilson interval, median latencies, failure counts.""" + arms: dict[str, list[Attempt]] = {} + for attempt in attempts: + arms.setdefault(attempt.arm, []).append(attempt) + out: dict[str, dict[str, Any]] = {} + for arm, rows in arms.items(): + passes = sum(1 for row in rows if row.passed) + out[arm] = { + "attempts": len(rows), + "passed": passes, + "rate": passes / len(rows), + "wilson95": wilson(passes, len(rows)), + "median_total_ms": _median(row.total_ms for row in rows), + "median_decision_ms": _median(row.decision_ms for row in rows), + "median_environment_ms": _median(row.environment_ms for row in rows), + "failures": dict(Counter(row.failure for row in rows if not row.passed and row.failure)), + } + return out + + +def _pct(value: float | None) -> str: + return NA if value is None else f"{value * 100:.0f}%" + + +def _ms(value: float | None) -> str: + return NA if value is None else f"{value / 1000:.1f} s" + + +def render( + attempts: list[Attempt], + *, + revision: str, + dataset: str, + agents_sha: str = NA, + model: str = NA, + budget: str = NA, + date: str = NA, + judge: str = "WebJudge", + threshold: int = JUDGE_THRESHOLD, +) -> str: + """The Markdown result table. Header names the dataset revision, Agents SHA, model, budget and date.""" + lines = [ + f"Online-Mind2Web live slice: `{dataset}` at revision `{revision}`.", + f"Agents SHA `{agents_sha or NA}`, model `{model or NA}`, budget `{budget or NA}`, date `{date or NA}`.", + f"Success is {judge} at threshold {threshold}; every attempt (failures, timeouts, errors, BLOCKED, " + "unevaluated) is in the denominator. This is a live-web suite, not the #109 offline step collector and not " + "the #104 Anthropic browser_toolset baseline.", + "", + "| arm | passed / attempts | rate | Wilson 95% | median total | median decision | median environment |" + " failures |", + "|---|---|---|---|---|---|---|---|", + ] + summary = summarize(attempts) + if not summary: + lines.append(f"| {NA} | 0 / 0 | {NA} | {NA} | {NA} | {NA} | {NA} | no attempts |") + for arm, row in summary.items(): + interval = row["wilson95"] + ci = NA if interval is None else f"{_pct(interval[0])}-{_pct(interval[1])}" + failures = ", ".join(f"{reason} x{count}" for reason, count in sorted(row["failures"].items())) or "none" + lines.append( + f"| {arm} | {row['passed']} / {row['attempts']} | {_pct(row['rate'])} | {ci} |" + f" {_ms(row['median_total_ms'])} | {_ms(row['median_decision_ms'])} | {_ms(row['median_environment_ms'])} |" + f" {failures} |" + ) + return "\n".join(lines) + "\n" diff --git a/evals/online_mind2web/slice.json b/evals/online_mind2web/slice.json new file mode 100644 index 0000000..572f74d --- /dev/null +++ b/evals/online_mind2web/slice.json @@ -0,0 +1,32 @@ +{ + "name": "online-mind2web-16", + "dataset": "osunlp/Online-Mind2Web", + "source_file": "Online_Mind2Web.json", + "revision": "6aa56e07c9d247fcc2b72fc3d94ad0f857f1e5d4", + "expected_tasks": 300, + "seed": 116, + "n_easy": 12, + "n_medium": 4, + "selection": "random.Random(116).sample of the sorted easy public-read ids (12) plus the sorted medium public-read ids (4); public-read drops login, payment, cart, favorites, account-write and similar task text (tasks.SKIP). Eligible pool at this revision: 76 easy, 135 medium. The committed list is then sorted by id. Task text is not vendored; fetch the official file at the pinned revision (gated; needs HF_TOKEN after accepting the dataset terms).", + "paper": "https://arxiv.org/abs/2504.01382", + "license": "CC BY 4.0", + "judge": "WebJudge (official auto-eval, default score threshold 3); not the Allrecipes site check and not the #109 offline step collector.", + "tasks": [ + {"id": "0632e496d37badee0350dad358f047c5", "level": "easy", "website": "https://cookpad.com/", "reference_length": 4}, + {"id": "207e933d1bba815bcb58664b5d82c085", "level": "medium", "website": "https://www.apartments.com/", "reference_length": 6}, + {"id": "2532fd402d3c741b79894e6ff2269f53", "level": "easy", "website": "https://www.thumbtack.com/", "reference_length": 3}, + {"id": "34ccd15a8ea8fd3895af83f5ccf62369", "level": "easy", "website": "https://us.megabus.com/", "reference_length": 3}, + {"id": "3621b099326c7aebd2e2dac6be3b52d1", "level": "easy", "website": "https://www.nvidia.com/", "reference_length": 5}, + {"id": "5e1b8254c123c80178cc28e0afdb14f0_110325", "level": "easy", "website": "https://ticketmaster.com/", "reference_length": 4}, + {"id": "6db4a0e346976f2729ba9afcd3208941", "level": "easy", "website": "https://www.fedex.com/en-us/home.html", "reference_length": 2}, + {"id": "783ce6a3499fa7cf25bc12f8f0ecbbbb", "level": "easy", "website": "https://www.mayoclinic.org/", "reference_length": 5}, + {"id": "864244b6969e0f8733b0eb1ca06cd51f", "level": "medium", "website": "https://www.espn.com/", "reference_length": 6}, + {"id": "a11ecdff735b51372d536c866011af6f", "level": "easy", "website": "https://www.coursera.org/", "reference_length": 4}, + {"id": "a6f0434ce6aff5f9b03681241b03ad82", "level": "easy", "website": "https://finance.yahoo.com/", "reference_length": 3}, + {"id": "bd1e3770b7181f6fce9c35e18caa9785", "level": "easy", "website": "https://craigslist.org/", "reference_length": 3}, + {"id": "c698ff3fc0f6cbce39947c597ab5749b_121425", "level": "medium", "website": "https://www.eventbrite.com/", "reference_length": 8}, + {"id": "d5c34bf39eb6096ae5d439325cde4d32", "level": "easy", "website": "https://www.dmv.virginia.gov/", "reference_length": 5}, + {"id": "d71be72aa25c3eab8eea47a0e60382e2", "level": "easy", "website": "https://www.apple.com/", "reference_length": 4}, + {"id": "fa9adb815b85d259f943d81874a052e5", "level": "medium", "website": "https://soundcloud.com/", "reference_length": 6} + ] +} diff --git a/evals/online_mind2web/tasks.py b/evals/online_mind2web/tasks.py new file mode 100644 index 0000000..5deebc2 --- /dev/null +++ b/evals/online_mind2web/tasks.py @@ -0,0 +1,287 @@ +# coding: utf-8 +"""The committed Online-Mind2Web slice and the official task file it points at. + +The slice names a Hugging Face dataset revision, a seed and task ids. Task text is downloaded on demand at that +revision and is not vendored: the official file is gated and released under CC BY 4.0. +""" + +from __future__ import annotations + +import json +import os +import random +import re +import urllib.error +import urllib.request +from dataclasses import dataclass +from pathlib import Path +from typing import Any +from urllib.parse import urlparse + +SLICE_PATH = Path(__file__).with_name("slice.json") +REVISION = re.compile(r"^[0-9a-f]{40}$") +TASK_ID = re.compile(r"^[0-9a-f]{32}(?:_[0-9a-z]+)?$", re.IGNORECASE) +LEVELS = ("easy", "medium", "hard") +HF_RESOLVE = "https://huggingface.co/datasets/{dataset}/resolve/{revision}/{source_file}" + +# Login, payment, cart, favorites and other account-writes: these stay out of the public-read pool. +SKIP = re.compile( + r"(" + r"\b(log[\s-]?in|sign[\s-]?in|sign[\s-]?up|create an account|my account|your account|" + r"check[\s-]?out|purchase|payment|credit card|home store|set (?:it|this) as|" + r"subscribe|password|register|book a |reserve |reservation|" + r"submit a (?:request|form|application)|liked by)\b" + r"|add .{0,80}to (?:the |my |your )?(?:cart|favorites?|wishlist)" + r"|to (?:the |my |your )?(?:cart|favorites?|wishlist)" + r"|\bbuy\b" + r")", + re.IGNORECASE, +) + + +class SliceError(ValueError): + """The slice file or the official task file is malformed or disagrees with the slice.""" + + +@dataclass(frozen=True) +class SliceTask: + id: str + level: str + website: str + reference_length: int + + +@dataclass(frozen=True) +class Slice: + name: str + dataset: str + source_file: str + revision: str + expected_tasks: int + seed: int + n_easy: int + n_medium: int + selection: str + tasks: tuple[SliceTask, ...] + + @property + def ids(self) -> tuple[str, ...]: + return tuple(task.id for task in self.tasks) + + +@dataclass(frozen=True) +class OfficialTask: + id: str + website: str + confirmed_task: str + reference_length: int + level: str + + +def _need(data: dict[str, Any], key: str, kind: type) -> Any: + value = data.get(key) + if not isinstance(value, kind) or (isinstance(value, bool) and kind is int): + raise SliceError(f"slice field {key!r} must be a {kind.__name__}, got {value!r}") + return value + + +def parse_slice(data: Any) -> Slice: + """A validated ``Slice``: the revision is a full SHA, ids are unique Online-Mind2Web hashes, levels are known.""" + if not isinstance(data, dict): + raise SliceError("the slice must be a JSON object") + if not REVISION.match(str(data.get("revision", ""))): + raise SliceError("slice field 'revision' must be a full 40-character commit SHA") + raw_tasks = _need(data, "tasks", list) + if not raw_tasks: + raise SliceError("the slice names no tasks") + tasks: list[SliceTask] = [] + for raw in raw_tasks: + if not isinstance(raw, dict) or not isinstance(raw.get("id"), str) or not TASK_ID.match(raw["id"]): + raise SliceError(f"bad task entry {raw!r}: want an id like a 32-character hex hash") + level = raw.get("level") + if level not in LEVELS: + raise SliceError(f"task {raw['id']}: level must be one of {LEVELS}, got {level!r}") + website = raw.get("website") + if not isinstance(website, str) or urlparse(website).scheme not in {"http", "https"}: + raise SliceError(f"task {raw['id']}: website must be an http(s) URL, got {website!r}") + length = raw.get("reference_length") + if not isinstance(length, int) or isinstance(length, bool) or length < 1: + raise SliceError(f"task {raw['id']}: reference_length must be a positive int, got {length!r}") + tasks.append(SliceTask(raw["id"], level, website, length)) + ids = [task.id for task in tasks] + if len(set(ids)) != len(ids): + raise SliceError("the slice repeats a task id") + n_easy, n_medium = _need(data, "n_easy", int), _need(data, "n_medium", int) + if n_easy < 0 or n_medium < 0 or n_easy + n_medium == 0: + raise SliceError("n_easy and n_medium must be non-negative and not both zero") + if sum(1 for task in tasks if task.level == "easy") != n_easy: + raise SliceError(f"the slice lists {sum(1 for t in tasks if t.level == 'easy')} easy tasks, expected {n_easy}") + if sum(1 for task in tasks if task.level == "medium") != n_medium: + raise SliceError( + f"the slice lists {sum(1 for t in tasks if t.level == 'medium')} medium tasks, expected {n_medium}" + ) + expected = _need(data, "expected_tasks", int) + if expected < n_easy + n_medium: + raise SliceError("expected_tasks is smaller than the slice") + return Slice( + name=_need(data, "name", str), + dataset=_need(data, "dataset", str), + source_file=_need(data, "source_file", str), + revision=data["revision"], + expected_tasks=expected, + seed=_need(data, "seed", int), + n_easy=n_easy, + n_medium=n_medium, + selection=_need(data, "selection", str), + tasks=tuple(tasks), + ) + + +def load_slice(path: Path = SLICE_PATH) -> Slice: + try: + data = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise SliceError(f"{path} is not JSON: {exc}") from None + return parse_slice(data) + + +def _field(row: dict[str, Any], *names: str) -> Any: + for name in names: + if name in row: + return row[name] + raise SliceError(f"task row is missing any of {names}: {row!r}") + + +def parse_official(data: Any) -> list[OfficialTask]: + """The official ``Online_Mind2Web.json`` list (or a dict keyed by task id).""" + if isinstance(data, dict) and all(isinstance(value, dict) for value in data.values()): + rows = list(data.values()) + elif isinstance(data, list): + rows = data + else: + raise SliceError("the official file must be a JSON list of tasks or an object keyed by task id") + tasks: list[OfficialTask] = [] + seen: set[str] = set() + for raw in rows: + if not isinstance(raw, dict): + raise SliceError(f"official task is not an object: {raw!r}") + task_id = str(_field(raw, "task_id", "id")).strip() + if not TASK_ID.match(task_id): + raise SliceError(f"bad official task id {task_id!r}") + if task_id in seen: + raise SliceError(f"official file repeats task id {task_id}") + seen.add(task_id) + level = str(_field(raw, "level")).lower() + if level not in LEVELS: + raise SliceError(f"task {task_id}: level must be one of {LEVELS}, got {level!r}") + website = str(_field(raw, "website")) + if urlparse(website).scheme not in {"http", "https"}: + raise SliceError(f"task {task_id}: website must be an http(s) URL, got {website!r}") + length = _field(raw, "reference_length") + if not isinstance(length, int) or isinstance(length, bool) or length < 1: + raise SliceError(f"task {task_id}: reference_length must be a positive int, got {length!r}") + text = str(_field(raw, "confirmed_task", "task_description", "task")).strip() + if not text: + raise SliceError(f"task {task_id} has an empty task description") + tasks.append(OfficialTask(task_id, website, text, length, level)) + if not tasks: + raise SliceError("the official file names no tasks") + return tasks + + +def load_official(path: Path) -> list[OfficialTask]: + try: + data = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise SliceError(f"{path} is not JSON: {exc}") from None + return parse_official(data) + + +def is_public_read(task: OfficialTask) -> bool: + """True when the task text does not ask for a login, payment, cart, favorites or similar account write.""" + return SKIP.search(task.confirmed_task) is None + + +def eligible_ids(tasks: list[OfficialTask], level: str) -> list[str]: + if level not in LEVELS: + raise SliceError(f"level must be one of {LEVELS}, got {level!r}") + return sorted(task.id for task in tasks if task.level == level and is_public_read(task)) + + +def select_ids(tasks: list[OfficialTask], *, seed: int, n_easy: int, n_medium: int) -> list[str]: + """The documented seed: ``sample`` from the sorted public-read easy ids, then the medium ids; return sorted ids.""" + easy, medium = eligible_ids(tasks, "easy"), eligible_ids(tasks, "medium") + if len(easy) < n_easy: + raise SliceError(f"need {n_easy} public-read easy tasks, the official file has {len(easy)}") + if len(medium) < n_medium: + raise SliceError(f"need {n_medium} public-read medium tasks, the official file has {len(medium)}") + rng = random.Random(seed) + return sorted(rng.sample(easy, n_easy) + rng.sample(medium, n_medium)) + + +def check_selection(slice_: Slice, tasks: list[OfficialTask]) -> None: + """The official file has the expected size and the seed reproduces the committed ids, levels and websites.""" + if len(tasks) != slice_.expected_tasks: + raise SliceError(f"official file has {len(tasks)} tasks, slice expected {slice_.expected_tasks}") + by_id = {task.id: task for task in tasks} + missing = [task_id for task_id in slice_.ids if task_id not in by_id] + if missing: + raise SliceError(f"official file is missing slice ids: {missing}") + chosen = select_ids(tasks, seed=slice_.seed, n_easy=slice_.n_easy, n_medium=slice_.n_medium) + if chosen != list(slice_.ids): + raise SliceError("the seed does not reproduce the committed slice ids from this official file") + for item in slice_.tasks: + official = by_id[item.id] + if (official.level, official.website, official.reference_length) != ( + item.level, + item.website, + item.reference_length, + ): + raise SliceError( + f"task {item.id} in the official file is {official.level} {official.website} " + f"len={official.reference_length}, slice says {item.level} {item.website} len={item.reference_length}" + ) + + +def official_url(slice_: Slice) -> str: + return HF_RESOLVE.format(dataset=slice_.dataset, revision=slice_.revision, source_file=slice_.source_file) + + +def fetch_official(slice_: Slice, dest: Path, *, token: str | None = None, opener=None) -> Path: + """Download the official JSON to ``dest`` (a file). Needs a Hugging Face token with access to the gated dataset.""" + dest.parent.mkdir(parents=True, exist_ok=True) + auth = token if token is not None else os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN") + if not auth: + raise SliceError( + "the official Online-Mind2Web file is gated; set HF_TOKEN after accepting the dataset terms at " + f"https://huggingface.co/datasets/{slice_.dataset}" + ) + request = urllib.request.Request( + official_url(slice_), + headers={"Authorization": f"Bearer {auth}", "User-Agent": "system1-agents-online-mind2web"}, + ) + try: + with (opener or urllib.request.urlopen)(request) as response: + body = response.read() + except urllib.error.HTTPError as exc: + raise SliceError(f"download failed ({exc.code}): {exc.reason}") from None + except urllib.error.URLError as exc: + raise SliceError(f"download failed: {exc.reason}") from None + dest.write_bytes(body) + check_selection(slice_, load_official(dest)) + return dest + + +def goal(task: OfficialTask) -> str: + """The browse goal: go to the official start URL and finish the official task text, then give a final answer.""" + return ( + f"Go to {task.website} and complete this task:\n\n{task.confirmed_task}\n\n" + "When finished, provide your final answer as text." + ) + + +def bind(slice_: Slice, official: list[OfficialTask]) -> list[OfficialTask]: + """The official records for the slice, in slice order, after ``check_selection``.""" + check_selection(slice_, official) + by_id = {task.id: task for task in official} + return [by_id[task_id] for task_id in slice_.ids] diff --git a/tests/test_online_mind2web_eval.py b/tests/test_online_mind2web_eval.py new file mode 100644 index 0000000..27bca00 --- /dev/null +++ b/tests/test_online_mind2web_eval.py @@ -0,0 +1,381 @@ +# coding: utf-8 +"""The seeded Online-Mind2Web live slice (issue 116), offline: no network, no model, no live sites. + +A fixture official file and a scripted player stand in for the gated Hugging Face download and ``browse``. +""" + +from __future__ import annotations + +import io +import json +from pathlib import Path + +import pytest + +from evals.online_mind2web import __main__ as cli +from evals.online_mind2web import runner, score, tasks + +REVISION = "6aa56e07c9d247fcc2b72fc3d94ad0f857f1e5d4" +EASY = [ + ("0632e496d37badee0350dad358f047c5", "https://cookpad.com/", "Browse gluten-free cookie recipes.", 4), + ("2532fd402d3c741b79894e6ff2269f53", "https://www.thumbtack.com/", "Find electricians near 10203.", 3), + ("34ccd15a8ea8fd3895af83f5ccf62369", "https://us.megabus.com/", "Find what to do when I lose an item.", 3), + ("3621b099326c7aebd2e2dac6be3b52d1", "https://www.nvidia.com/", "Open the lab leader's profile.", 5), + ( + "5e1b8254c123c80178cc28e0afdb14f0_110325", + "https://ticketmaster.com/", + "Find a help page about how presale works.", + 4, + ), + ( + "6db4a0e346976f2729ba9afcd3208941", + "https://www.fedex.com/en-us/home.html", + "Look up tracking for 3023858502.", + 2, + ), + ("783ce6a3499fa7cf25bc12f8f0ecbbbb", "https://www.mayoclinic.org/", "Find Florida internship programs.", 5), + ("a11ecdff735b51372d536c866011af6f", "https://www.coursera.org/", "Explore Psychology courses.", 4), + ("a6f0434ce6aff5f9b03681241b03ad82", "https://finance.yahoo.com/", "Find Tesla's close on March 17, 2023.", 3), + ("bd1e3770b7181f6fce9c35e18caa9785", "https://craigslist.org/", "Browse solar panel installer listings.", 3), + ("d5c34bf39eb6096ae5d439325cde4d32", "https://www.dmv.virginia.gov/", "Find a DMV center in Richmond.", 5), + ("d71be72aa25c3eab8eea47a0e60382e2", "https://www.apple.com/", "Find specs for the latest Macbook Air.", 4), + ("aa000000000000000000000000000001", "https://www.accuweather.com/", "Show daily weather for New York City.", 4), + ("aa000000000000000000000000000002", "https://www.drugs.com/", "Browse the natural products database.", 2), +] +MEDIUM = [ + ("207e933d1bba815bcb58664b5d82c085", "https://www.apartments.com/", "Find Ohio City apartments with parking.", 6), + ("864244b6969e0f8733b0eb1ca06cd51f", "https://www.espn.com/", "Find the last 2023 F1 race winner's time.", 6), + ( + "c698ff3fc0f6cbce39947c597ab5749b_121425", + "https://www.eventbrite.com/", + "Find school activity events in New York.", + 8, + ), + ("fa9adb815b85d259f943d81874a052e5", "https://soundcloud.com/", "Browse a user who reposted a Top 50 song.", 6), + ("bb000000000000000000000000000001", "https://www.nba.com/", "Find the league leader in blocked shots.", 5), +] +SKIPPED = [ + ( + "cc000000000000000000000000000001", + "easy", + "https://www.bestbuy.com/", + "Add a $50 Uber gift card to the cart.", + 4, + ), + ( + "cc000000000000000000000000000002", + "easy", + "https://www.apartments.com/", + "Add Elevate at Chicago, IL, to favorites.", + 4, + ), + ( + "cc000000000000000000000000000003", + "medium", + "https://www.amazon.com/", + "Find headphones and add them to the cart.", + 8, + ), + ( + "cc000000000000000000000000000004", + "easy", + "https://www.tumblr.com/", + "Find the posts liked by the user Taylor Swift.", + 4, + ), +] + + +def official_rows() -> list[dict]: + rows = [] + for task_id, website, text, length in EASY: + rows.append( + { + "task_id": task_id, + "website": website, + "confirmed_task": text, + "reference_length": length, + "level": "easy", + } + ) + for task_id, website, text, length in MEDIUM: + rows.append( + { + "task_id": task_id, + "website": website, + "confirmed_task": text, + "reference_length": length, + "level": "medium", + } + ) + for task_id, level, website, text, length in SKIPPED: + rows.append( + {"task_id": task_id, "website": website, "confirmed_task": text, "reference_length": length, "level": level} + ) + return rows + + +def fixture_slice() -> dict: + official = tasks.parse_official(official_rows()) + chosen = tasks.select_ids(official, seed=116, n_easy=12, n_medium=4) + by_id = {task.id: task for task in official} + return { + "name": "online-mind2web-16", + "dataset": "osunlp/Online-Mind2Web", + "source_file": "Online_Mind2Web.json", + "revision": REVISION, + "expected_tasks": len(official_rows()), + "seed": 116, + "n_easy": 12, + "n_medium": 4, + "selection": "test fixture", + "tasks": [ + { + "id": task_id, + "level": by_id[task_id].level, + "website": by_id[task_id].website, + "reference_length": by_id[task_id].reference_length, + } + for task_id in chosen + ], + } + + +def write_pair(tmp_path: Path) -> tuple[Path, Path]: + official = tmp_path / "Official.json" + official.write_text(json.dumps(official_rows()), encoding="utf-8") + slice_path = tmp_path / "slice.json" + slice_path.write_text(json.dumps(fixture_slice()), encoding="utf-8") + return slice_path, official + + +def test_committed_slice_loads_and_pins_the_revision(): + slice_ = tasks.load_slice() + assert slice_.dataset == "osunlp/Online-Mind2Web" + assert slice_.revision == REVISION + assert slice_.seed == 116 + assert (slice_.n_easy, slice_.n_medium, len(slice_.tasks)) == (12, 4, 16) + assert slice_.expected_tasks == 300 + assert list(slice_.ids) == sorted(slice_.ids) + assert len(set(slice_.ids)) == 16 + assert {task.level for task in slice_.tasks} == {"easy", "medium"} + + +def test_fixture_slice_is_reproducible_from_the_seed(): + slice_ = tasks.parse_slice(fixture_slice()) + official = tasks.parse_official(official_rows()) + tasks.check_selection(slice_, official) + again = tasks.select_ids(official, seed=116, n_easy=12, n_medium=4) + assert again == list(slice_.ids) + assert tasks.select_ids(official, seed=116, n_easy=12, n_medium=4) == again + + +@pytest.mark.parametrize( + "patch, message", + [ + ({"revision": "short"}, "40-character"), + ({"tasks": []}, "names no tasks"), + ({"n_easy": 99}, "easy tasks"), + ( + {"tasks": [{"id": "not-a-hash", "level": "easy", "website": "https://x.example/", "reference_length": 1}]}, + "hex", + ), + ], +) +def test_bad_slices_are_rejected(patch, message): + data = fixture_slice() + data.update(patch) + with pytest.raises(tasks.SliceError, match=message): + tasks.parse_slice(data) + + +def test_public_read_drops_cart_favorites_login_and_payment(): + official = tasks.parse_official(official_rows()) + skipped = {task.id for task in official if not tasks.is_public_read(task)} + assert skipped == {row[0] for row in SKIPPED} + assert "cc000000000000000000000000000001" not in tasks.eligible_ids(official, "easy") + + +def test_check_selection_rejects_a_missing_id_and_a_drifted_website(tmp_path): + data = fixture_slice() + rows = [row for row in official_rows() if row["task_id"] != data["tasks"][0]["id"]] + data["expected_tasks"] = len(rows) + slice_ = tasks.parse_slice(data) + with pytest.raises(tasks.SliceError, match="missing"): + tasks.check_selection(slice_, tasks.parse_official(rows)) + rows = official_rows() + for row in rows: + if row["task_id"] == slice_.ids[0]: + row["website"] = "https://example.invalid/" + with pytest.raises(tasks.SliceError, match="official file"): + tasks.check_selection(slice_, tasks.parse_official(rows)) + + +def test_fetch_needs_a_token_and_verifies_the_slice(tmp_path, monkeypatch): + slice_ = tasks.parse_slice(fixture_slice()) + dest = tmp_path / "Official.json" + monkeypatch.delenv("HF_TOKEN", raising=False) + monkeypatch.delenv("HUGGING_FACE_HUB_TOKEN", raising=False) + with pytest.raises(tasks.SliceError, match="HF_TOKEN"): + tasks.fetch_official(slice_, dest) + body = json.dumps(official_rows()).encode() + + class _Resp(io.BytesIO): + def __enter__(self): + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + return False + + def opener(request): + assert "Bearer secret-token" in request.headers["Authorization"] + return _Resp(body) + + path = tasks.fetch_official(slice_, dest, token="secret-token", opener=opener) + assert path == dest + tasks.check_selection(slice_, tasks.load_official(dest)) + + +def test_goal_names_the_site_and_the_task(): + task = tasks.parse_official(official_rows())[0] + text = tasks.goal(task) + assert task.website in text and task.confirmed_task in text + assert text.startswith("Go to ") and "final answer" in text + + +def _player(answers: dict[str, runner.Outcome | Exception]): + async def play(task, arm, logs_dir): + logs_dir.mkdir(parents=True, exist_ok=True) + (logs_dir / "profile").mkdir(exist_ok=True) # each attempt gets its own directory + key = f"{task.id}:{arm}" + result = answers[key] + if isinstance(result, Exception): + raise result + return result + + return play + + +@pytest.mark.asyncio +async def test_every_attempt_is_played_and_recorded(tmp_path): + official = tasks.bind(tasks.parse_slice(fixture_slice()), tasks.parse_official(official_rows())) + first, second = official[0], official[1] + scores = {first.id: 4, second.id: 2} + + def judge(task, outcome, logs_dir): + del outcome, logs_dir + return scores[task.id] + + attempts = await runner.run_slice( + [first, second], + ["jev", "llm"], + _player( + { + f"{first.id}:jev": runner.Outcome("ok", total_ms=1000, decision_ms=200), + f"{first.id}:llm": runner.Outcome("ok", total_ms=2000, decision_ms=None), + f"{second.id}:llm": runner.Outcome("ok", total_ms=1500, decision_ms=None), + f"{second.id}:jev": runner.Outcome("ok", total_ms=800, decision_ms=100), + } + ), + tmp_path / "out", + judge=judge, + agents_sha="abc", + ) + assert [row.task_id for row in attempts] == [first.id, first.id, second.id, second.id] + assert [row.arm for row in attempts] == ["jev", "llm", "llm", "jev"] # second task flips arm order + assert [row.passed for row in attempts] == [True, True, False, False] + assert attempts[2].failure == "judge:2" + recorded = runner.load_attempts(tmp_path / "out" / "attempts.jsonl") + assert len(recorded) == 4 + assert (tmp_path / "out" / f"{first.id}__jev__r0" / "attempt.json").is_file() + + +@pytest.mark.asyncio +async def test_failures_stay_in_the_denominator_with_their_reason(tmp_path): + official = tasks.parse_official(official_rows()) + a, b, c, d = official[:4] + attempts = await runner.run_slice( + [a, b, c, d], + ["jev"], + _player( + { + f"{a.id}:jev": runner.Outcome("", error="timeout after 600s", total_ms=600000), + f"{b.id}:jev": runner.Outcome("", error="BLOCKED by the site"), + f"{c.id}:jev": RuntimeError("browser crashed"), + f"{d.id}:jev": runner.Outcome("looks done", total_ms=3000, decision_ms=50), + } + ), + tmp_path / "out", + ) + assert [row.failure for row in attempts] == ["timeout", "BLOCKED", "agent: browser crashed", "unevaluated"] + assert all(not row.passed for row in attempts) + table = score.render(attempts, revision=REVISION, dataset="osunlp/Online-Mind2Web", agents_sha="deadbeef") + assert "0 / 4" in table and "timeout x1" in table and "unevaluated x1" in table and "BLOCKED x1" in table + assert "deadbeef" in table and "live-web" in table and "#109" in table and "#104" in table + assert "N/A" not in table.split("\n")[4] or "0%" in table # rate is 0%, not N/A + + +def test_missing_or_failing_judge_is_not_a_pass(): + task = tasks.parse_official(official_rows())[0] + finished = runner.Outcome("an answer", total_ms=10, decision_ms=1) + unevaluated = runner.judged_attempt(task, "jev", finished, judge=None, logs_dir=Path("."), agents_sha=None) + assert unevaluated == runner.Attempt(task.id, "jev", False, "unevaluated", 10, 1) + + def boom(task, outcome, logs_dir): + raise RuntimeError("no API key") + + failed = runner.judged_attempt(task, "jev", finished, judge=boom, logs_dir=Path("."), agents_sha=None) + assert failed.passed is False and failed.failure is not None and failed.failure.startswith("judge: no API key") + + +def test_wilson_interval(): + interval = score.wilson(0, 16) + assert interval is not None + lo, hi = interval + assert 0 <= lo <= hi <= 1 + assert score.wilson(0, 0) is None + + +def test_table_counts_every_attempt_and_writes_na(): + rows = [ + score.Attempt("a", "jev", True, None, 1000, 200, judge_score=4), + score.Attempt("b", "jev", False, "timeout", 600000, 10), + score.Attempt("c", "llm", False, "unevaluated", 2000, None), + ] + table = score.render( + rows, + revision=REVISION, + dataset="osunlp/Online-Mind2Web", + agents_sha="sha", + model="jev,llm", + budget="max_steps=40 timeout_s=600", + date="2026-11-18", + ) + assert "| jev | 1 / 2 | 50% |" in table + assert "| llm | 0 / 1 | 0% |" in table + assert "median decision" in table and "N/A" in table # llm has no decision time + assert "Agents SHA `sha`" in table and "2026-11-18" in table + + +def test_cli_show_select_and_score(tmp_path, capsys): + slice_path, official = write_pair(tmp_path) + assert cli.main(["--slice", str(slice_path), "show"]) == 0 + out = capsys.readouterr().out + assert REVISION in out and "0632e496d37badee0350dad358f047c5" in out + assert cli.main(["--slice", str(slice_path), "select", "--official", str(official)]) == 0 + selected = capsys.readouterr().out.splitlines() + assert selected == list(tasks.load_slice(slice_path).ids) + attempts = tmp_path / "attempts.jsonl" + attempts.write_text( + json.dumps(score.Attempt("x", "jev", False, "unevaluated", 10, 1, agents_sha="cafed00d").to_json()) + "\n", + encoding="utf-8", + ) + assert cli.main(["--slice", str(slice_path), "score", str(attempts)]) == 0 + scored = capsys.readouterr().out + assert "0 / 1" in scored and "unevaluated x1" in scored and "cafed00d" in scored + + +def test_cli_run_without_downloaded_tasks_explains(tmp_path, capsys): + slice_path, _ = write_pair(tmp_path) + assert cli.main(["--slice", str(slice_path), "run", "--official", str(tmp_path / "missing.json")]) == 2 + assert "fetch" in capsys.readouterr().err