From 5de466a91d67d0b385e9b235f25f6266109d9d7d Mon Sep 17 00:00:00 2001 From: song Date: Sun, 1 Nov 2026 09:55:12 +0800 Subject: [PATCH 1/2] fix(benchmark): pin the reviewed checkpoint provider patch pair Signed-off-by: song --- benchmark/README.md | 4 +- benchmark/edgebench/README.md | 19 ++++++---- ...v1.patch => nonfinite-checkpoint-v2.patch} | 14 +++---- benchmark/edgebench/source-default.json | 8 ++-- benchmark/tests/test_edgebench_source.py | 38 +++++++++++++++++++ 5 files changed, 63 insertions(+), 20 deletions(-) rename benchmark/edgebench/compatibility/{nonfinite-checkpoint-v1.patch => nonfinite-checkpoint-v2.patch} (97%) diff --git a/benchmark/README.md b/benchmark/README.md index bb61cc6b29..f3a816877c 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -407,8 +407,8 @@ python -m benchmark.edgebench.checkpoint_judge --checkpoint-capture \ Select the checkpoint provider through the same [shared EdgeBench source protocol](edgebench/README.md#select-and-freeze-the-native-provider) -as the workers and judge. A new qualification cohort resolves the canonical -integration branch and applies the reviewed native compatibility patch under +as the workers and judge. A new default qualification cohort fetches the reviewed +integration commit paired with the native compatibility patch under that protocol; its `.edgebench-source/receipt.json` records the effective provider commit, upstream revision and patch digest. Existing cohorts remain frozen. Patch representation changes receive a new digest and effective commit even diff --git a/benchmark/edgebench/README.md b/benchmark/edgebench/README.md index b094e70260..2ae9c5185d 100644 --- a/benchmark/edgebench/README.md +++ b/benchmark/edgebench/README.md @@ -12,7 +12,7 @@ iptables permissions; a macOS Docker socket alone does not provide that boundary ## Select and freeze the native provider -The first CLI entry for a log directory fetches the selected branch into +The first CLI entry for a log directory fetches the selected revision into `.edgebench-source/checkout` and atomically freezes the effective source in `.edgebench-source/receipt.json`. Concurrent worker/judge startup shares that lock. Later entries reuse the frozen checkout without fetching, even if the branch, @@ -22,8 +22,10 @@ before model/container entry. Use a new log directory to adopt a newer source; never repin an active or historical cohort. For a **new cohort with no explicit source options**, `source-default.json` -selects the shared branch and the versioned -[`nonfinite-checkpoint-v1` compatibility patch](compatibility/nonfinite-checkpoint-v1.patch). +selects the reviewed integration commit +[`e278cda3f4d2da3cf06aabf2df4daa99b9749501`](https://github.com/huangruiteng/EdgeBench/commit/e278cda3f4d2da3cf06aabf2df4daa99b9749501) +and the versioned +[`nonfinite-checkpoint-v2` compatibility patch](compatibility/nonfinite-checkpoint-v2.patch). This LoopX-owned patch fixes the native checkpoint representation of nonfinite released scores and metrics. Native scoring, report values and selection remain unchanged. Preparation verifies the asset digest and each touched file's exact @@ -31,10 +33,13 @@ upstream blob and mode, applies the patch before provider import/installation, and creates a clean, reproducible Git commit with the fetched commit as its parent. The snapshot uses an explicit LoopX compatibility identity and the upstream commit time; it does not attribute the patch to an upstream author. -Unrelated upstream changes can advance normally. A changed patch preimage, -including a later upstream implementation of the fix, stops preparation without -publishing a source directory. Review and update the bundled patch, or explicitly -select the original source, before starting that new cohort. +The default revision and patch are one reviewed pair: advancing the integration +branch does not change a new default cohort. Updating the pair requires reviewing +the new source and qualifying the native checkpoint transport again. A changed +patch preimage, including a later upstream implementation of the fix, still stops +preparation without publishing a source directory. Explicit branch/revision +selection remains available and uses unmodified provider code; it does not +silently inherit the default patch or its qualification. Patched sources use `edgebench_source_v2` receipts. `upstream_commit` is the fetched remote revision; `compatibility_patch` records the patch ID and SHA-256; diff --git a/benchmark/edgebench/compatibility/nonfinite-checkpoint-v1.patch b/benchmark/edgebench/compatibility/nonfinite-checkpoint-v2.patch similarity index 97% rename from benchmark/edgebench/compatibility/nonfinite-checkpoint-v1.patch rename to benchmark/edgebench/compatibility/nonfinite-checkpoint-v2.patch index d65f4f1dda..e0bec3ae75 100644 --- a/benchmark/edgebench/compatibility/nonfinite-checkpoint-v1.patch +++ b/benchmark/edgebench/compatibility/nonfinite-checkpoint-v2.patch @@ -1,5 +1,5 @@ diff --git a/docs/en/reference/judge-api.md b/docs/en/reference/judge-api.md -index 3b2b23fcf6b29770905be3cf24669cc83f0b144a..4209f8c6ac20c1de5c79495e56ad096649eb3499 100644 +index f0033a8a5dc31dc373272ae8affe157911b782bb..9f17a4966bef3a9f5560ee4e815c4d687de0a1d1 100644 --- a/docs/en/reference/judge-api.md +++ b/docs/en/reference/judge-api.md @@ -83,2 +83,19 @@ Content-Type: application/json @@ -32,7 +32,7 @@ index 3b2b23fcf6b29770905be3cf24669cc83f0b144a..4209f8c6ac20c1de5c79495e56ad0966 + If the restore response is lost, call `POST /api/v1/checkpoint/restore-result` with the **same** JSON body (`checkpoint_id`, `manifest_sha256`, `run_id`, and current `admin_secret`). It returns the original response shape and the already published submission token. It does not allocate another registration, replay grading, or restart cooldown. Readback also remains valid after subsequent submissions and preserves their consumed opportunities. The existing origin marker must match and the registration must still be live; a missing, mismatched, partial or lost registration returns 409. Wrong admin authority returns 403, and malformed bodies return 422. A persisted origin marker alone cannot recreate credentials after a judge process restart. This recovery does not recover a runner's phase deadline: its controller must retain the original activation-time deadline and must not initialize a new duration on retry. A new `run_id` represents a different branch, not recovery of this response. diff --git a/docs/zh/reference/judge-api.md b/docs/zh/reference/judge-api.md -index 19f9be961ddcf385b0639986c5434601318e2fad..6cb4d4bf36748ec212211f57b234ab4a8e2887e4 100644 +index 9eb2e7d2a6f1cb2be4d27c29c9941cbb0f46da58..b4a0dff68bd1d49a3fd585f418c7cc0dca17fa13 100644 --- a/docs/zh/reference/judge-api.md +++ b/docs/zh/reference/judge-api.md @@ -77,2 +77,20 @@ Token 是一个 32 字符的十六进制字符串，内部维护任务 ID、运 @@ -57,7 +57,7 @@ index 19f9be961ddcf385b0639986c5434601318e2fad..6cb4d4bf36748ec212211f57b234ab4a + 恢复不会评测或多消费一轮。后续提交仍由原生 `consume_round`、预算、冷却及选优规则决定。冷却在新注册发布时，从捕获时剩余的**时长**重新计时；离线等待和 capsule 物化既不会补发机会，也不会消耗这段时长；旧 capsule 不包含来源后来取得的提升。 diff --git a/sforge/harness/checkpoint.py b/sforge/harness/checkpoint.py -index 31d62ed85f68bea5bd2565f72b3b630d260e8a7a..42e46cd23ea329b099a99d2a32abb3dd9e7020b8 100644 +index 9a83144706ae2bddb067a4f89f989f288dafe974..d89ac61f4b26d837686b165e873da825652b5e63 100644 --- a/sforge/harness/checkpoint.py +++ b/sforge/harness/checkpoint.py @@ -45,3 +45,5 @@ if TYPE_CHECKING: @@ -199,12 +199,12 @@ index 31d62ed85f68bea5bd2565f72b3b630d260e8a7a..42e46cd23ea329b099a99d2a32abb3dd - ) + )) ): -@@ -413,3 +528,3 @@ class JudgeCheckpoints: +@@ -418,3 +533,3 @@ class JudgeCheckpoints: self._validate(packet) - encoded = _encode( + encoded = _encode_manifest( packet -@@ -530,3 +645,8 @@ class JudgeCheckpoints: +@@ -535,3 +650,8 @@ class JudgeCheckpoints: folder.mkdir(parents=True) - _write(folder / "report.json", _encode(report)) + # Native report.json uses json.dumps' Inf/NaN spelling @@ -215,7 +215,7 @@ index 31d62ed85f68bea5bd2565f72b3b630d260e8a7a..42e46cd23ea329b099a99d2a32abb3dd + ).encode()) incumbent = packet["incumbent"] diff --git a/tests/test_judge_checkpoint.py b/tests/test_judge_checkpoint.py -index be90734f9ec7939a8b7a89fafb6532e2a6e1e677..f991fd8c80501e50ab59738f156ced950373ce80 100644 +index 608effc16c9c032dd2a44b3375b5ea57c502e0fa..7dbdc68e47021c6ad42fac1462ac52647412eadc 100644 --- a/tests/test_judge_checkpoint.py +++ b/tests/test_judge_checkpoint.py @@ -20,2 +20,3 @@ import copy @@ -229,7 +229,7 @@ index be90734f9ec7939a8b7a89fafb6532e2a6e1e677..f991fd8c80501e50ab59738f156ced95 + CheckpointUnavailable, _decode_manifest, _encode, _encode_manifest, _sha, +) from sforge.harness.config import SForgeConfig -@@ -777 +780,244 @@ def test_restore_materialization_does_not_spend_remaining_cooldown( +@@ -810 +813,244 @@ def test_restore_materialization_does_not_spend_remaining_cooldown( assert submit(client, fresh, 12).status_code == 429 + + diff --git a/benchmark/edgebench/source-default.json b/benchmark/edgebench/source-default.json index ac35af2709..8aee0f341a 100644 --- a/benchmark/edgebench/source-default.json +++ b/benchmark/edgebench/source-default.json @@ -1,9 +1,9 @@ { "repository": "https://github.com/huangruiteng/EdgeBench.git", - "ref": "refs/heads/codex/edgebench-integration", + "ref": "e278cda3f4d2da3cf06aabf2df4daa99b9749501", "compatibility_patch": { - "id": "nonfinite-checkpoint-v1", - "relative_path": "compatibility/nonfinite-checkpoint-v1.patch", - "sha256": "44e5c9b3dacf65aaabfc036e26323aac27effbf933b9ff4477fd9a5e36831ffb" + "id": "nonfinite-checkpoint-v2", + "relative_path": "compatibility/nonfinite-checkpoint-v2.patch", + "sha256": "02d21096a2e55f625d3c8beee2335728caf08aa6c75a5a27b1ab412bc46323bf" } } diff --git a/benchmark/tests/test_edgebench_source.py b/benchmark/tests/test_edgebench_source.py index 02c8cd835f..d2400f7166 100644 --- a/benchmark/tests/test_edgebench_source.py +++ b/benchmark/tests/test_edgebench_source.py @@ -5,6 +5,7 @@ import hashlib import json import os +import re import subprocess import sys @@ -210,6 +211,9 @@ def test_bundled_patch_passes_required_public_checks(tmp_path): from loopx.contract import scan_public_boundary defaults = json.loads(source.DEFAULT_FILE.read_text()) + # The production bundle is qualified with an immutable upstream revision. + # A moving branch can invalidate its full-index preimages between cohorts. + assert re.fullmatch(r"[0-9a-f]{40}", defaults["ref"]) _, patch = source._bundled_patch(defaults["compatibility_patch"]) asset = tmp_path / "compatibility.patch" asset.write_bytes(patch) @@ -219,6 +223,40 @@ def test_bundled_patch_passes_required_public_checks(tmp_path): assert scan_public_boundary([asset], registry={})["ok"] +def test_pinned_default_and_patch_advance_only_as_a_reviewed_pair(repository, tmp_path): + install_test_patch(repository) + defaults = json.loads(source.DEFAULT_FILE.read_text()) + defaults["ref"] = git(repository, "rev-parse", "HEAD") + source.DEFAULT_FILE.write_text(json.dumps(defaults)) + old_checkout, original = source.prepare_source(tmp_path / "old") + old_receipt = (old_checkout.parent / "receipt.json").read_bytes() + + # Even a touched-file change on the remote branch cannot break a new cohort + # or silently opt it into an unqualified provider revision. + (repository / "sforge/__init__.py").write_text("MARKER = 'new-upstream'\n") + git(repository, "add", "sforge/__init__.py") + git(repository, "-c", "core.hooksPath=/dev/null", "commit", "-qm", "advance provider") + advanced = git(repository, "rev-parse", "HEAD") + new_checkout, same = source.prepare_source(tmp_path / "new") + assert same == original + assert (new_checkout / "sforge/__init__.py").read_text() == "MARKER = 'patched-source'\n" + + # Updating just the pin still fails closed; no partially prepared source + # can be consumed by a concurrent worker or judge. + defaults["ref"] = advanced + source.DEFAULT_FILE.write_text(json.dumps(defaults)) + with pytest.raises(ValueError, match="preimage differs"): + source.prepare_source(tmp_path / "unpaired") + assert not (tmp_path / "unpaired" / source.STATE_DIR).exists() + + install_test_patch(repository) + _, refreshed = source.prepare_source(tmp_path / "refreshed") + assert refreshed["upstream_commit"] == advanced != original["upstream_commit"] + assert refreshed["compatibility_patch"]["sha256"] != original["compatibility_patch"]["sha256"] + assert source.prepare_source(tmp_path / "old")[1] == original + assert (old_checkout.parent / "receipt.json").read_bytes() == old_receipt + + def test_default_patch_creates_reproducible_clean_source(repository, tmp_path, monkeypatch): asset = install_test_patch(repository) upstream = git(repository, "rev-parse", "HEAD") From 16b273f348e4c945599a8a0388c0a19586755230 Mon Sep 17 00:00:00 2001 From: song Date: Sun, 1 Nov 2026 09:59:21 +0800 Subject: [PATCH 2/2] docs(benchmark): describe the pinned default in the introduction Signed-off-by: song --- benchmark/edgebench/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/benchmark/edgebench/README.md b/benchmark/edgebench/README.md index 2ae9c5185d..ca463eeb19 100644 --- a/benchmark/edgebench/README.md +++ b/benchmark/edgebench/README.md @@ -3,10 +3,11 @@ This research adapter calls SForge's native `run_agent`. Task images, prompts, submission cooldowns, judge isolation and scoring remain owned by the pinned EdgeBench checkout. Best-only adds host capture/admission scheduling; native and -blind retain native automatic evaluation. The CLI defaults to the shared +blind retain native automatic evaluation. The CLI defaults to a reviewed commit +from the shared [EdgeBench integration branch](https://github.com/huangruiteng/EdgeBench/tree/codex/edgebench-integration), maintained jointly by `huangruiteng` and `loopx-agent`. The canonical repository -and branch live in [`source-default.json`](source-default.json). +and revision/compatibility-patch pair live in [`source-default.json`](source-default.json). Run the controller on Linux with Docker and native SForge iptables permissions; a macOS Docker socket alone does not provide that boundary.