[sriram687]

Summary

Adds "webarena_verified_hard" to DEFAULT_BENCHMARKS, filtering the 258 hard tasks from ServiceNow's official webarena-verified-hard.json subset.

Resolves the TODO at browsergym/experiments/src/browsergym/experiments/benchmark/configs.py:163.

Closes #415

Motivation

• Cheaper iteration: 258 of 812 tasks cuts eval compute and runtime by ~68%.
• Stress-testing: Easy navigation tasks saturate as agents improve; the hard subset targets planning, long-horizon tool use, and complex verification.
• Drop-in preset: No ad-hoc filtering scripts needed:

  bgym.DEFAULT_BENCHMARKS["webarena_verified_hard"]()

• No breaking changes: The existing 812-task "webarena_verified" benchmark is untouched.

Upstream Alignment

Item	Detail
Source	ServiceNow/webarena-verified/assets/dataset/subsets/webarena-verified-hard.json
Hard tasks	258
Non-hard tasks	554 (webarena-verified-non-hard.json)
Total	812

Changes

File	Change
configs.py	Added WEBARENA_VERIFIED_HARD_TASK_IDS; registered "webarena_verified_hard" using non-capturing regex filtering
test_benchmark.py	Added "webarena_verified_hard": 258 to expected_bench_size in test_build_benchmarks
webarena_verified/README.md	Documented how to instantiate the hard subset

Proof of work

Testing

• Task IDs checked against the live upstream JSON: 100% match
• Instantiation yields exactly 258 task runs (deterministic, seed=42)
• Benchmark.from_json() serialization roundtrip passes
• test_build_benchmarks passes

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[sriram687]

Hi @Surya-5555,

Hope you're doing well!

I've opened a PR addressing the TODO in configs.py:163 by adding the "webarena_verified_hard" benchmark preset (DEFAULT_BENCHMARKS).

It filters the 258 hard tasks from ServiceNow's official webarena-verified-hard.json subset, enabling faster (~68% less compute) evaluations on challenging tasks without modifying the full 812-task benchmark. All unit tests and JSON roundtrips have been verified locally.

Whenever you have a chance, could you please take a look? I'd really appreciate your feedback!

[Surya-5555]

Hi @sriram687, thanks for the PR!

The Unit tests / code-format check is currently failing. Could you please look into it and fix the formatting for the files you changed only (configs.py, test_benchmark.py and the README)? Please don't reformat unrelated files or code, so the diff stays focused on your change.

You can run the repo's formatter (see the code-format job / .pre-commit-config.yaml for the exact tool and settings) on just your modified files, commit, and push. The checks will re-run automatically.

Once that's green, I'll review it. Thanks!

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[sriram687]

Hi @sriram687, thanks for the PR!

The Unit tests / code-format check is currently failing. Could you please look into it and fix the formatting for the files you changed only (configs.py, test_benchmark.py and the README)? Please don't reformat unrelated files or code, so the diff stays focused on your change.

You can run the repo's formatter (see the code-format job / .pre-commit-config.yaml for the exact tool and settings) on just your modified files, commit, and push. The checks will re-run automatically.

Once that's green, I'll review it. Thanks!

Hi @Surya-5555 , thanks for the feedback! I’ve fixed the formatting issues in configs.py, test_benchmark.py, and the README using the repo’s formatter, limited to only the files I changed. The code-format check and all other checks are now passing. Please take another look whenever you get a chance. Thanks!

[Surya-5555]

LGTM.
Happy to merge!