Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,15 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver

## Unreleased

### Added

- `python -m evals.online_mind2web`: a seeded 16-task live-web slice of Online-Mind2Web (12 easy + 4 medium, seed 116;
ids, websites and the official revision only in `evals/online_mind2web/slice.json`; task text downloaded on demand
from the gated `osunlp/Online-Mind2Web` file). Each attempt starts a clean `browse` profile, plays hosted Jev (or
another arm) under a pinned budget, and records every outcome; success is WebJudge at threshold 3 (or `unevaluated`
if no judge ran). The table names Agents SHA, model, budget and date. Offline tests use a fixture file and a
scripted player; no live numbers yet (#116).

## 0.4.0 - 2026-11-13

### Added
Expand Down
4 changes: 4 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,7 @@ This product includes software developed by third parties:
Beurer-Kellner, Marc Fischer, and Florian Tramèr). Full license text: THIRD_PARTY_LICENSES.md
- s1a/decision_models/onejev.py: HTTP client for OneJev / qev (OmniJev/OneJev), Apache-2.0;
the weights and server are not vendored. Details: THIRD_PARTY_LICENSES.md
- evals/online_mind2web: task ids, websites and difficulty of a 16-task slice of Online-Mind2Web
(osunlp/Online-Mind2Web; Xue et al., 2025, arXiv:2504.01382), CC BY 4.0. The official task file is
not included. WebJudge evaluation code is MIT (OSU-NLP-Group/Online-Mind2Web) and is not vendored.
Details: THIRD_PARTY_LICENSES.md
18 changes: 18 additions & 0 deletions THIRD_PARTY_LICENSES.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,3 +126,21 @@ OneJev and its serving code are Apache License 2.0 (Copyright OmniJev Team). Thi
vendor OneJev weights, training code, or the `qev`/`flide` packages; install and run them separately
per the model card. The unrelated PyPI package named `qev` is not the OneJev server — install from the
OneJev GitHub repository as the model card instructs.

## Online-Mind2Web

`evals/online_mind2web` measures a seeded 16-task live-web slice of
[Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web) (OSU NLP Group; Xue, Qi, Shi, Song, Gou,
Song, Sun, and Su, *An Illusion of Progress? Assessing the Current State of Web Agents*, 2025,
[arXiv:2504.01382](https://arxiv.org/abs/2504.01382)). The official task file is gated and is not vendored; only
task ids, start URLs, difficulty labels and the pinned dataset revision are committed, in
`evals/online_mind2web/slice.json`. Downstream users must accept the dataset terms and download
`Online_Mind2Web.json` at that revision themselves.

The dataset is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). The dataset card also
asks that [Mind2Web](https://arxiv.org/abs/2306.06070) (Deng et al., 2023) be cited. The WebJudge automatic
evaluator and the rest of the [OSU-NLP-Group/Online-Mind2Web](https://github.com/OSU-NLP-Group/Online-Mind2Web)
code are MIT License (Copyright (c) 2025 OSU Natural Language Processing). This repository does not vendor that
code; a run may call it separately.

This work is used here for research evaluation of System 1 browser agents.
13 changes: 13 additions & 0 deletions docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,19 @@ the harness continues without it. Both sides pay it.

![Allrecipes, Jev left, the chat model right, at 8x](assets/demos/allrecipes-comparison-8x.gif)


## Browser use: Online-Mind2Web live slice

A seeded 16-task **live-web** slice of [Online-Mind2Web](https://github.com/OSU-NLP-Group/Online-Mind2Web)
(Xue et al., 2025, [arXiv:2504.01382](https://arxiv.org/abs/2504.01382)) lives in
`evals/online_mind2web/slice.json` (seed 116, 12 easy + 4 medium public-read tasks; official file downloaded at
revision `6aa56e07c9d247fcc2b72fc3d94ad0f857f1e5d4`). This is an end-to-end live suite: it is not the offline
step-level collector in issue #109, and it is not the planned Anthropic `browser_toolset` baseline in issue #104.

Run it with `python -m evals.online_mind2web`; details, judging (WebJudge at threshold 3) and licence notes are in
[online-mind2web.md](online-mind2web.md). No live numbers are committed yet. Traces belong under
`docs/results/online-mind2web/`.

## A longer game

Over 150 moves of 2048 the cost gap widens with the transcript. Seed 0: Jev scored 1,104 in 244 s for $0.003; the
Expand Down
57 changes: 57 additions & 0 deletions docs/online-mind2web.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Online-Mind2Web live slice

A seeded 16-task **live-web** slice of [Online-Mind2Web](https://github.com/OSU-NLP-Group/Online-Mind2Web)
(Xue et al., 2025, [arXiv:2504.01382](https://arxiv.org/abs/2504.01382); dataset
[osunlp/Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web), CC BY 4.0). It answers "does the
full agent finish a realistic live-web task?", which is a different question from:

- the offline step-level collector in issue #109 / PR #111 (frozen page, right operation/element)
- the Allrecipes / WebVoyager task 0 row already in [benchmarks.md](benchmarks.md)
- the planned Anthropic `browser_toolset` baseline in issue #104

## The slice

Ids, levels, start URLs and the official revision live in `evals/online_mind2web/slice.json`. Task text is **not**
vendored. The official file is gated: accept the dataset terms, set `HF_TOKEN`, then download the pinned revision
(the 2026-05-15 task updates are in this SHA):

```bash
# HF_TOKEN must already be able to read osunlp/Online-Mind2Web
uv run python -m evals.online_mind2web fetch
uv run python -m evals.online_mind2web select --official .external/online_mind2web/Online_Mind2Web.json
```

`select` re-derives the 16 ids from the official file and seed `116`. The rule is: take the official tasks whose
text does not ask for a login, payment, cart, favorites or similar account write; `random.Random(116).sample` 12
sorted easy ids and 4 sorted medium ids; commit that list sorted by id. At this revision the public-read pool is 76
easy and 135 medium tasks.

## Running

Each attempt starts a new `browse` session in its own `logs_dir` (no shared browser profile) and uses the official
start URL plus the official task text as the goal. The first backend is hosted Jev (`--model jev`); pass `--arms jev
llm` to add the chat-model arm under the same `--max-steps` and `--timeout`.

```bash
uv run python -m evals.online_mind2web run --official .external/online_mind2web/Online_Mind2Web.json --arms jev
```

`results.md` and `config.json` name the Agents SHA, the dataset revision, the model, the budget and the date. Every
planned attempt is in the denominator, including timeouts, agent errors, `BLOCKED` and unevaluated episodes.

## Judging

Success is [WebJudge](https://github.com/OSU-NLP-Group/Online-Mind2Web), the official auto-eval, at the upstream
default threshold of 3 (upstream now recommends GPT-6 Luna as the backbone). This repository does not vendor
WebJudge. Without a judge, a finished episode is recorded as `unevaluated`, never as a pass. After an external
WebJudge run you can apply integer scores with `--judge-scores scores.json` (a JSON object keyed by task id).

## Results

No live numbers are committed yet: a run needs `HF_TOKEN`, a Jev (and optionally chat-model) key, and a live network.
Place traces under `docs/results/online-mind2web/` and redact secrets.

## Licence

The dataset is CC BY 4.0; the WebJudge / evaluation code in the official repository is MIT. Attribution is in
`NOTICE` and `THIRD_PARTY_LICENSES.md`. Only ids, websites, levels and the revision are committed here.
27 changes: 27 additions & 0 deletions docs/results/online-mind2web/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Online-Mind2Web live slice results

Live traces for the seeded 16-task slice in `evals/online_mind2web/slice.json` (seed 116, 12 easy + 4 medium,
dataset `osunlp/Online-Mind2Web` at `6aa56e07c9d247fcc2b72fc3d94ad0f857f1e5d4`).

No run is committed yet. A later results folder should name the Agents SHA, the model or checkpoint, the budget
(`max_steps`, `timeout_s`) and the date, keep failures and timeouts in the denominator, and redact API keys from
raw traces. Success is WebJudge at threshold 3.

Task ids (sorted):

- `0632e496d37badee0350dad358f047c5` easy — https://cookpad.com/
- `207e933d1bba815bcb58664b5d82c085` medium — https://www.apartments.com/
- `2532fd402d3c741b79894e6ff2269f53` easy — https://www.thumbtack.com/
- `34ccd15a8ea8fd3895af83f5ccf62369` easy — https://us.megabus.com/
- `3621b099326c7aebd2e2dac6be3b52d1` easy — https://www.nvidia.com/
- `5e1b8254c123c80178cc28e0afdb14f0_110325` easy — https://ticketmaster.com/
- `6db4a0e346976f2729ba9afcd3208941` easy — https://www.fedex.com/en-us/home.html
- `783ce6a3499fa7cf25bc12f8f0ecbbbb` easy — https://www.mayoclinic.org/
- `864244b6969e0f8733b0eb1ca06cd51f` medium — https://www.espn.com/
- `a11ecdff735b51372d536c866011af6f` easy — https://www.coursera.org/
- `a6f0434ce6aff5f9b03681241b03ad82` easy — https://finance.yahoo.com/
- `bd1e3770b7181f6fce9c35e18caa9785` easy — https://craigslist.org/
- `c698ff3fc0f6cbce39947c597ab5749b_121425` medium — https://www.eventbrite.com/
- `d5c34bf39eb6096ae5d439325cde4d32` easy — https://www.dmv.virginia.gov/
- `d71be72aa25c3eab8eea47a0e60382e2` easy — https://www.apple.com/
- `fa9adb815b85d259f943d81874a052e5` medium — https://soundcloud.com/
1 change: 1 addition & 0 deletions evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ folder per episode). `uv run python -m evals.table evals/results` aggregates the
| Injection guard on page-length HTML (BrowseSafe-Bench sample) | precision, recall, failures and page coverage at the runtime clip | `s1a run injection_guard --labelled-set .external/browsesafe/test-sample-html.jsonl`, after `scripts/build_browsesafe_set.py build`; see `docs/benchmarks.md` |
| Bounded recovery on/off (3 local form fixtures) | completion on/off in a real headless Chromium, judged by the fixture server | `python -m evals.recovery --repeat 1` |
| Bounded recovery live (real decision/chat models, 3 local form fixtures) | completion on/off with the real models in a real headless Chromium, judged by the fixture server; cost null unless `CHAT_USD_PER_M_*` is set; optional `--validate-submission` variant | `python -m evals.recovery.live --model jev --repeat 1` |
| Online-Mind2Web live slice (16 seeded tasks) | live-web task completion, WebJudge-graded, failures in the denominator | `python -m evals.online_mind2web run`; see `docs/online-mind2web.md` |

## The loop

Expand Down
8 changes: 8 additions & 0 deletions evals/online_mind2web/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# coding: utf-8
"""A seeded Online-Mind2Web live slice: clean-profile browser tasks, WebJudge-graded (issue 116).

``tasks`` loads the committed slice (ids, levels, websites and the official revision; never the task text) and the
official file downloaded on demand; ``runner`` plays every (task, arm) attempt on the live web and records each one,
failures included; ``score`` turns the attempts into the result table. Success is WebJudge (threshold 3), not an
Allrecipes-style site check and not the #109 offline step collector.
"""
197 changes: 197 additions & 0 deletions evals/online_mind2web/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
# coding: utf-8
"""``python -m evals.online_mind2web {show,select,fetch,run,score}``: the seeded Online-Mind2Web live slice (#116).

- ``show`` prints the committed slice (ids, levels, websites and the pinned revision).
- ``select --official Online_Mind2Web.json`` re-derives the ids from the official file and the slice's seed, to check them.
- ``fetch --dest FILE`` downloads the official file at the pinned revision (network, gated; needs ``HF_TOKEN``).
Nothing is vendored.
- ``run --official FILE --arms jev`` plays each task through ``browse`` with a decision model or the chat model under
the same ``--max-steps`` and ``--timeout``, writes ``attempts.jsonl`` and ``results.md``. Needs the models' keys and
a live network. Without a judge, finished episodes are recorded as ``unevaluated`` (not a pass). ``--judge-scores``
applies WebJudge-style integer scores from a JSON object keyed by ``task_id`` (or ``task_id__arm``).
- ``score attempts.jsonl`` re-renders the table from recorded attempts.
"""

from __future__ import annotations

import argparse
import asyncio
import json
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path

from evals.online_mind2web import score, tasks

RESULTS = Path("evals/results/online_mind2web")


def parser() -> argparse.ArgumentParser:
build = argparse.ArgumentParser(
prog="python -m evals.online_mind2web",
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
build.add_argument("--slice", type=Path, default=tasks.SLICE_PATH)
commands = build.add_subparsers(dest="command", required=True)
commands.add_parser("show")
sel = commands.add_parser("select")
sel.add_argument("--official", type=Path, required=True)
fetch = commands.add_parser("fetch")
fetch.add_argument("--dest", type=Path, default=Path(".external/online_mind2web/Online_Mind2Web.json"))
run = commands.add_parser("run")
run.add_argument("--official", type=Path, default=Path(".external/online_mind2web/Online_Mind2Web.json"))
run.add_argument("--arms", nargs="+", default=["jev"])
run.add_argument("--repeat", type=int, default=1)
run.add_argument("--max-steps", type=int, default=40)
run.add_argument("--timeout", type=float, default=600.0)
run.add_argument("--headed", action="store_true")
run.add_argument("--judge-scores", type=Path, default=None, help="JSON object of WebJudge integer scores")
run.add_argument("--judge-model", default=None, help="name recorded next to --judge-scores")
run.add_argument("--threshold", type=int, default=score.JUDGE_THRESHOLD)
run.add_argument("--out", type=Path, default=None)
sc = commands.add_parser("score")
sc.add_argument("attempts", type=Path)
sc.add_argument("--agents-sha", default=None)
sc.add_argument("--model", default=None)
sc.add_argument("--budget", default=None)
sc.add_argument("--date", default=None)
return build


def agents_sha() -> str:
"""The checkout's HEAD, or N/A when this is not a git work tree."""
try:
result = subprocess.run(["git", "rev-parse", "HEAD"], check=True, capture_output=True, text=True, timeout=10)
except (OSError, subprocess.SubprocessError):
return score.NA
sha = result.stdout.strip()
return sha if sha else score.NA


def load_score_judge(path: Path, model: str | None):
"""A judge that reads integer scores from a JSON object; used in tests and for applying an offline WebJudge run."""
raw = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(raw, dict):
raise tasks.SliceError(f"{path} must be a JSON object of task ids to integer scores")

def judge(task, outcome, logs_dir):
del outcome, logs_dir
value = raw.get(f"{task.id}__{getattr(judge, 'arm', '')}", raw.get(task.id))
if not isinstance(value, int) or isinstance(value, bool):
raise tasks.SliceError(f"no integer WebJudge score for {task.id} in {path}")
return value

judge.model = model
return judge


def main(argv: list[str] | None = None) -> int:
args = parser().parse_args(argv)
try:
slice_ = tasks.load_slice(args.slice)
except tasks.SliceError as exc:
print(f"error: {exc}", file=sys.stderr)
return 2
if args.command == "show":
print(f"{slice_.name}: {slice_.dataset} {slice_.source_file} @ {slice_.revision} (seed {slice_.seed})")
for task in slice_.tasks:
print(f"{task.id}\t{task.level}\t{task.reference_length}\t{task.website}")
return 0
if args.command == "select":
try:
official = tasks.load_official(args.official)
tasks.check_selection(slice_, official)
except tasks.SliceError as exc:
print(f"error: {exc}", file=sys.stderr)
return 2
for task_id in slice_.ids:
print(task_id)
return 0
if args.command == "fetch":
try:
path = tasks.fetch_official(slice_, args.dest)
except tasks.SliceError as exc:
print(f"error: {exc}", file=sys.stderr)
return 2
print(f"wrote {path} ({slice_.expected_tasks} tasks @ {slice_.revision})")
return 0
if args.command == "score":
from evals.online_mind2web.runner import load_attempts

rows = load_attempts(args.attempts)
sha = args.agents_sha or next((row.agents_sha for row in rows if row.agents_sha), score.NA)
print(
score.render(
rows,
revision=slice_.revision,
dataset=slice_.dataset,
agents_sha=sha,
model=args.model or score.NA,
budget=args.budget or score.NA,
date=args.date or score.NA,
),
end="",
)
return 0
return asyncio.run(_run(args, slice_))


async def _run(args: argparse.Namespace, slice_: tasks.Slice) -> int:
import s1a.entry # noqa: F401 # routes the harness logs to files before anything imports openjiuwen
from s1a.run import started_runner

from evals.online_mind2web.live import LiveBudget, make_player
from evals.online_mind2web.runner import run_slice

try:
official = tasks.bind(slice_, tasks.load_official(args.official))
except (tasks.SliceError, FileNotFoundError) as exc:
print(f"error: {exc} (run `python -m evals.online_mind2web fetch` first)", file=sys.stderr)
return 2
judge = load_score_judge(args.judge_scores, args.judge_model) if args.judge_scores else None
out = args.out or RESULTS / f"{datetime.now():%Y-%m-%d__%H-%M-%S}"
budget = LiveBudget(args.max_steps, args.timeout, not args.headed)
sha = agents_sha()
stamp = datetime.now(timezone.utc).astimezone().strftime("%Y-%m-%d")
budget_label = f"max_steps={budget.max_steps} timeout_s={budget.timeout_s:g}"
play = make_player(budget)
async with started_runner():
attempts = await run_slice(
official, args.arms, play, out, repeat=args.repeat, judge=judge, agents_sha=sha, threshold=args.threshold
)
table = score.render(
attempts,
revision=slice_.revision,
dataset=slice_.dataset,
agents_sha=sha,
model=",".join(args.arms),
budget=budget_label,
date=stamp,
threshold=args.threshold,
)
(out / "results.md").write_text(table, encoding="utf-8")
(out / "config.json").write_text(
json.dumps(
{
"dataset": slice_.dataset,
"revision": slice_.revision,
"agents_sha": sha,
"arms": args.arms,
"budget": {"max_steps": budget.max_steps, "timeout_s": budget.timeout_s, "headless": budget.headless},
"date": stamp,
"threshold": args.threshold,
"tasks": list(slice_.ids),
},
indent=1,
)
+ "\n",
encoding="utf-8",
)
print(table, end="")
return 0


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading