Repository navigation
Conversation
WebArenaVerifiedEvaluator returned the mean of the per-evaluator scores. webarena_verified scores a task 1.0 only when every evaluator passes and 0.0 when any evaluator fails or errors, so a task that satisfied one of its two evaluators got a reward of 0.5 here and 0.0 in the benchmark's own harness. 488 of the 812 tasks have more than one evaluator. Return results.score, the harness's task-level score, as the docstring already describes.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #412
Problem
WebArenaVerifiedEvaluator.__call__returned the mean of the per-evaluator scores. webarena-verified's ownTaskEvalResultscores a task 1.0 only when every evaluator passes, and 0.0 when any evaluator fails or errors. So a task that satisfied one of its two evaluators got a reward of 0.5 in BrowserGym and 0.0 in the benchmark's harness. 488 of the 812 tasks have more than one evaluator, and AgentLab'savg_rewardaverages these partial rewards into the reported success rate.Change
browsergym/webarena_verified/.../evaluators.py: returnresults.score, the harness's task-level score, as the existing docstring ("1.0 or 0.0") describes. The per-evaluator results are still logged.tests/webarena/test_webarena_verified_evaluator.py: runs__call__on a real two-evaluator task fromwebarena-verified.json. The harness is mocked to returnTaskEvalResult.create(...)for pass/pass, pass/fail, fail/pass and pass/error, and the test checks that the reward equals the harness's task score. It skips whenbrowsergym.webarena_verifiedisn't installed, liketest_webarenalite_evaluators.py.Testing
main, 3 fail withassert 0.5 == 0.0(pass/fail, fail/pass, pass/error).browsergym.core.registration,browsergym.webarena(instance and task) andnltkwere replaced by small local modules, becauselibwebarenapulls in more than I could install on this machine. The test itself only touches the evaluator, the mocked harness and webarena-verified's result types.black --config=./pyproject.toml --check(24.2.0, as pinned in.pre-commit-config.yaml) passes on both files.tests/webarenaCI jobs against live WebArena instances.