Skip to content

test(compat): exercise pinned AlphaEvolve examples in CI - #542

Merged
codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
Yushu-Li:test/alphaevolve-upstream-ci
Oct 24, 2026
Merged

codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
Yushu-Li:test/alphaevolve-upstream-ci

Conversation

@Yushu-Li

Copy link
Copy Markdown
Contributor

The local AlphaEvolve entrypoints currently have synthetic unit coverage, while validation against Google's actual examples is manual. Add a dedicated PR workflow that exercises the original circle-packing, TSP and signal-processing entrypoints against a single pinned upstream revision, without requiring model credentials or the Google SDK.

Each example runs seed-only and with one fixed comment-only proposal through the public CLI and real OE workers. The tests verify expected finite metrics, unchanged original/staged source hashes, cleaned import aliases and original report artifacts. Generation cases require an actual OE evolution trace containing an evaluated marked child, not merely a provider call or a successful exit. The TSP report must contain all five PNGs and its saved program; signal processing must save its evaluator archive in the run directory.

Run each case in a fresh subprocess with a small environment allowlist and deliberate dummy credential traps. Test-only startup tripwires block and immediately record socket/DNS and unscoped SDK/auth imports, including attempts caught by upstream code or made in workers that do not run atexit. Two negative tests verify that caught attempts still fail offline validation. These are safeguards for trusted test inputs, not a security sandbox. Explicit suite execution fails on a missing checkout path, an unpinned/dirty example checkout or a path outside the OE parent workspace. Temporary outputs are cleaned up, and failures include captured diagnostics.

The workflow uses a standard Ubuntu runner, Python 3.12, read-only permissions, a 15-minute timeout, cancellation of superseded runs and relevant PR path filters. It also supports manual dispatch after merging. Checkout credentials are not persisted, no account Secrets are used, and no artifacts or caches are uploaded. Scientific dependencies are pinned only for this opt-in job. The existing unit and real-model integration jobs, production code and base dependencies are unchanged.

Validation:

  • All eight opt-in checks passed locally, including a fresh Python 3.12 virtual environment installed from the workflow's dependency commands with neither google nor alpha_evolve installed. All six actual-example cases reported zero network/cloud attempts; each generation case recorded one evaluated proposal.
  • Existing full offline unit suite: 881 tests passed, including the local-loopback HTTP worker/resume test.
  • Official actionlint v1.7.7, YAML/embedded shell syntax checks, Black/isort, Python 3.10 grammar checks and Frame scans of all four added Python files passed.
  • Confirmed nonzero exits for a missing explicit upstream path and an unguarded helper; valid pinned checkout accepted and workspace escapes rejected.

Continues #468 without closing it. The upstream revision is 34cd919193945c6dc81053f9af594031b54f41a8; changing its pin requires reviewing the examples and expected metrics. Fixed proposals validate compatibility, not real-model optimization or the managed AlphaEvolve service. Local validation ran on macOS; the first GitHub-hosted Ubuntu result will come from this PR's new workflow. Complete round-trip run coverage and actual managed-service validation remain follow-ups.

@codelion
codelion merged commit 599f8b7 into algorithmicsuperintelligence:main Oct 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants