[yl231]

Problem

/evaluate ran automation/process_pr_submission.py from the base branch, but that script then checked out the PR head and ran the PR branch's validator, evaluator and lockfile. Two consequences:

• Stale grading. Fixes merged to main (for example the LiveCodeBench fixes in Limit memory of LiveCodeBench test processes #239, Fix LiveCodeBench scoring of single-quoted __main__ guards #230 and Fix LiveCodeBench stdin tests for solutions that read sys.stdin.buffer #234) don't apply to a submission until its author rebases. The results posted on Divyam router submission #232, Add BizRouter submission (balanced Smart Routing) #229 and Add nuroutev2-artemis (NuRoute Artemis Task) #224 were graded this way; the re-grade in Re-grade the 25 leaderboard routers with the current evaluator #253 replaced them.
• Trust. Code from a submission branch ran in the same job as a token with pull-requests: write and checks: write, and the base checkout stored that token in .git/config. A submission could have changed its own scorer or posted a fake results comment. It could not push code, and no submission so far has modified llm_evaluation/.

Change

• Grade with the base branch's code. The script now creates its worktree from the base branch and copies in only the submission's data, read straight from git objects (symlinks are refused):

  • the router's prediction, robustness and config files;
  • prices for models the PR adds. Models already in model_cost.json keep main's price, and a PR that edits an existing price gets a warning in the log. Only the PR's own edits count, so a branch made from an older main doesn't undo later price changes.
  • names and mappings the PR adds to universal_model_names.py, which is parsed, not imported.

  Other files the PR changes are listed in the log and not used.

• Keep submission code away from the write token. The workflow is now three jobs:

  • start acknowledges the comment and creates the check;
  • evaluate runs the grading with contents: read and persist-credentials: false. This is the only job that runs submission code, because LiveCodeBench answers are executed;
  • report reads metrics.json from an artifact, accepts only numbers from it, posts the comment and completes the check.

• Safer inputs. The router name must match [A-Za-z0-9._-]+ and is passed to steps through env instead of being expanded into scripts. The entry-count check runs Python with -I, so a module planted in the PR checkout can't be imported.

• Evaluator in the comment. The results comment names the evaluator commit.

• README. The submission section says which files are taken from the branch.

Testing

• Real PRs. I staged six PRs (Add Corealis Router submission #235, Add DT-Router submission #241, feat(router): submit Krusch Cascade Router to RouterArena leaderboard #169, Add lemonade-liquid-router: local-first tri-ensemble router (~80% on-device) #198, Add iq-routing submission #209 and the merged Add BizRouter submission (balanced Smart Routing) #229) into worktrees of current main, and main's check_config_prediction_files.py --check-generated-result passed for each.
  • New models' prices and names were picked up: Add Corealis Router submission #235 adds 5, Add DT-Router submission #241 adds 3, Add lemonade-liquid-router: local-first tri-ensemble router (~80% on-device) #198 adds 3 and Add iq-routing submission #209 adds 1.
  • The router code in feat(router): submit Krusch Cascade Router to RouterArena leaderboard #169 and Add lemonade-liquid-router: local-first tri-ensemble router (~80% on-device) #198 was listed as unused.
  • The evaluator itself was not run locally, because it executes submitted code.
• Synthetic repo. A PR edit to an existing price is flagged and main's price kept; a price main changed after the PR branched is not flagged; names containing quotes round-trip; a symlinked prediction file and a names list built by code are both refused.
• Pre-commit. Passes, including actionlint.
• Live check after merge. issue_comment workflows always run from the default branch, so the new workflow can only run live after merge. I'll re-run /evaluate on Add Corealis Router submission #235, which was promised there anyway, and check the run log and comment.

The re-grade workflow (regrade-all.yml) only grades merged files on main, so it is unchanged.

🤖 Generated with Claude Code