Skip to content

fix(edgebench): default CLI cohorts to shared integration source - #7044

Merged
huangruiteng merged 1 commit into
mainfrom
codex/edgebench-shared-source-default-20261022
Oct 22, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/edgebench-shared-source-default-20261022

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

EdgeBench CLI workers previously imported whichever SForge happened to be installed, so selecting the shared integration branch in documentation did not ensure that workers and judges used it.

The worker, online judge and new native benchmark.edgebench.serve entry now resolve the shared source declared in benchmark/edgebench/source-default.json. The first entry atomically freezes a checkout and exact SHA per log directory; concurrent and later entries reuse it offline. Explicit repository/ref or SHA selection remains supported. Source drift, conflicting selections and legacy run directories without provenance are rejected. Worker runtime receipts include the selected source.

Validation: 337 tests passed across source selection, online judging, SForge runtime and source bundles; Ruff, bilingual diff and seven-file public-boundary checks passed. Real shared-source preparation and dependency installation passed; worker/online-judge CLI preflight and a native HTTP judge startup/shutdown passed with an isolated empty benchmark fixture. No model jobs or scoring submissions were launched for this change.

Existing frozen experiments and imported Python API callers retain their original environment. This local lock does not attest a judge running under a different log root. Dependencies are installed explicitly before starting the cohort. The new receipt remains local to this research adapter; the semantic inventory tool currently excludes benchmark/ sources. Native feedback defaults remain unchanged; the README example explicitly selects 30-minute active feedback and five-minute background evaluation.

CI currently hits two pre-existing fixture defects: both browser shard failures come from an unmigrated Goal attention fixture, and Windows fails two mocked cleanup deadline assertions on floating-point rounding. The attention failure also reproduces on current main. #7049 repairs these independently; its six canonical fixture variants, 62 related tests and compiled Goal masthead browser scenario pass locally. Windows and the complete PR checks remain pending; this PR is not yet merge-ready.

Signed-off-by: huangruiteng <huangrt01@163.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: actor_kind=model_agent; model=gpt-6.1-sol; provider=OpenAI; declaration_source=runtime_reported; reasoning_effort=max

详细中文评审

动机

运行EdgeBench比较的操作者即使按文档选了共享分支,worker和judge仍可能先导入环境中另一个SForge,得到来源不一致的运行。
例如环境的PYTHONPATH指向另一份SForge:基线worker先载入它;新CLI先按共同日志目录冻结精确源码,再导入所选provider,后续入口沿用同一份源码。
独立对照复现了基线误载环境包;当前worker、online judge及native serve均避开该包,实际导入冻结的共享源码并在预定参数预检处停止,日志没有试验任务目录。
本轮证明源码选择、来源拒绝和兼容入口,不启动模型、Docker任务或评分,不认证远端另一日志目录的judge,不宣称科学结果、安装版产品或完整研究收益。

改动思路

把同目录的源码选择集中在现有Python研究适配器,先锁定Git提交再导入;任务、评分和LoopX执行权限继续由原owner管理。仅修改文档或要求每个进程手动设置PYTHONPATH,不能消除已复现的误载。
本PR保留一个原子日志目录锁、一个共享来源配置和三个CLI入口的前置选择;直接Python API仍由调用者管理环境,历史试验不被重标或重新固定。
原来就需要在隔离环境安装SForge/Harbor;现在先固定来源、再显式安装其依赖,避免替每个进程重新抄写已知分支。源码选择不自动安装或启动工作,不把本地日志锁说成远端judge证明。相比新增安装器或第二套运行协议,这个provider内的小边界足以处理原问题。

具体改动

规范依据为 benchmark/edgebench/README.md,修订 9ddc9e6675350c45e3c1a1592f64d8aac2e218b3。Native EdgeBench worker trials 的固定原生源码及评分owner要求在这次CLI切片中实现;Feedback protocol (new-run default: native) 的冻结旧试验和原生反馈所有权保持。新README协议作为提案审查,不拿新增文字反向定义验收。完整七文件+425/-5已读;共享LoopX core、benchmark/runtime和native prompts均与基线同Git树/字节,评分或预算没有另一个决策owner。

关键代码讲解

prepare_source 先验证显式ref/完整SHA,锁住同一日志根,再校验已有receipt或在临时目录fetch、验证、原子rename。已有cohort不fetch,已有runs却没有来源证明则拒绝重标。bootstrap 在三个CLI的SForge导入前选择checkout,设置同一log环境并去除自身参数;直接导入Python模块不调用它。active_source_receipt 同时核Git HEAD、origin、干净状态及Python实际模块origin,并为既有runtime-receipt提供来源。serve.main 只把剩余参数交给原生SForge的serve,不实现第二个judge。

对主干的风险

最强反例是helper测试全过,但实际worker/judge仍先载入环境包。独立冻结脚本用实际SForge 193cecda05661380575b16c9fcc88a8c16ece9f3 与Harbor复现基线误载;当前三个CLI均避开错误PYTHONPATH,在明确的无效worker/slots/port预检处exit2,没有进入任务、模型或Docker。冲突SHA和旧无receipt日志也在进入前拒绝,旧文件和来源receipt逐字节保留。原生默认来源实际准备成功;并发首次选择、分支前进后旧cohort离线复用/新cohort取新源、六类漂移、tag与失败fetch后恢复均有实际Git覆盖。

初始Python3.14环境为40通过/274跳过,不能充当provider资格。另建兼容Python3.13私有环境显式安装所选SForge和Harbor后,当前337项、基线315项相关测试全部通过。直接Python API在两臂仍导入同一安装包;新runtime JSON会增加sforge_source,未激活CLI选择的API路径为null,因此没有宣称JSON形状完全相等。原生native默认3600/0不改;README例子显式选择1800/300,两者不可混淆。没有查询或等待CI。

标准预检整体仍exit1:5直接、9/10 catalog、7/8 risk及公共边界扫描通过,quota-plan和生命周期smoke的旧canonical来源fixture失败保留。同命令在精确基线/头均失败,内层quota的18条权威错误和汇总逐项相等,生命周期同为canonical_authority_required/todo_snapshot,相关owner和smoke源未变。benchmark敏感人工门仍在,不称预检绿或可以自行合并。错误详情包装器的一次调用语法错误和安装参数的一次shell展开错误均在检查执行前发生、已纠正并保留,不归因于PR。

我的整体评价

没有验证出的当前阻断发现。long_horizon为preserved:保持旧cohort与任务、反馈、预算和评分owner;user_experience为improved:同一logroot只需一次选择,后续直接复用,拒绝和恢复边界可读。把同目录的源码选择集中在现有Python研究适配器,先锁定Git提交再导入;任务、评分和LoopX执行权限继续由原owner管理。仅修改文档或要求每个进程手动设置PYTHONPATH,不能消除已复现的误载。
本PR保留一个原子日志目录锁、一个共享来源配置和三个CLI入口的前置选择;直接Python API仍由调用者管理环境,历史试验不被重标或重新固定。
局部简化已采用共享helper和原生CLI转发;扩大到通用源码框架、自动安装或产品配置页没有当前需求。实际Linux Docker执行、远端judge身份及科学countability仍应由原研究资格流程独立验证;本次APPROVE仅覆盖上述来源修复,保留两个预检红项和人工合并门。

英文简短结论

English verdict: APPROVE — 77de50a. An independent real-provider CLI control reproduces ambient-source loading on the base and verifies pinned-source imports for all three current entrypoints before deliberate parser stops. All337 current and315 original tests pass with the actual compatible provider environment. Two canonical-fixture premerge failures reproduce unchanged on the immutable base; benchmark/manual gates, Docker execution and scientific qualification remain separate.

@huangruiteng
huangruiteng merged commit 31ab292 into main Oct 22, 2026
24 of 34 checks passed
@huangruiteng
huangruiteng deleted the codex/edgebench-shared-source-default-20261022 branch October 22, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants