diff --git a/README.md b/README.md index 83e609a7d..4876c112f 100644 --- a/README.md +++ b/README.md @@ -492,6 +492,9 @@ Each call speaks newline-delimited JSON-RPC to a short-lived `codex app-server` ## Examples Gallery +To use unchanged local evaluators from Google's AlphaEvolve examples, see the +[AlphaEvolve evaluator interoperability walkthrough](examples/alphaevolve_interop/). +
### **Showcase Projects** diff --git a/examples/alphaevolve_interop/README.md b/examples/alphaevolve_interop/README.md new file mode 100644 index 000000000..bef4c8423 --- /dev/null +++ b/examples/alphaevolve_interop/README.md @@ -0,0 +1,96 @@ +# AlphaEvolve local evaluator interoperability + +This is the evaluator-contract slice of [#468](https://github.com/algorithmicsuperintelligence/openevolve/issues/468). +It runs the **unchanged** `program.py` and `evaluate.py` from Google's +[AlphaEvolve examples](https://github.com/Google-Cloud-AI/alphaevolve-on-googlecloud) +through an explicit adapter. It does not replace the cloud SDK, translate cloud +configuration, export bundles, or implement the rest of #468. + +## Evaluate an upstream seed locally + +From an installed OpenEvolve checkout: + +```bash +git clone https://github.com/Google-Cloud-AI/alphaevolve-on-googlecloud.git ../alphaevolve-reference + +# Evaluator-only dependencies; no Google client or authentication is used. +python -m pip install numpy scipy matplotlib pydantic +export PYTHONPATH="$(cd ../alphaevolve-reference/src && pwd)${PYTHONPATH:+:$PYTHONPATH}" +export MPLBACKEND=Agg + +python examples/alphaevolve_interop/run.py ../alphaevolve-reference/examples/circle_packing +python examples/alphaevolve_interop/run.py ../alphaevolve-reference/examples/tsp +python examples/alphaevolve_interop/run.py ../alphaevolve-reference/examples/signal_processing +``` + +Without `--config`, the runner only evaluates the seed and prints metrics and +artifacts. It makes **no LLM or Google Cloud requests**. The `alpha_evolve.models` +import used by circle packing resolves to the upstream source, not a fake SDK. +Upstream example dependencies can change; inspect their requirements if an import +fails. The signal-processing evaluator writes its normal `programs/` artifacts +under the current directory, so run it in a disposable working directory if needed. + +To generate and evaluate new candidates, provide **your own OpenEvolve config**: + +```bash +python examples/alphaevolve_interop/run.py ../alphaevolve-reference/examples/circle_packing \ + --config /path/to/your-openevolve-config.yaml --iterations 1 --output-dir ./interop-output +``` + +This second command invokes the model/provider you configured and may incur API +charges. It does not use or translate the Google example's `.env`, cloud client, +`run_evolution.py`, or Google credentials. Local model endpoints work as with any +other OpenEvolve run. Remote Cloud Run/GKE evaluators are not supported here. + +## Use the adapter directly + +```python +from openevolve.compat.alphaevolve import adapt_evaluator + +# imported_evaluator is the original synchronous AlphaEvolve evaluator callable. +evaluate = adapt_evaluator(imported_evaluator, objective_metric="sum_of_radii") +``` + +Use `evaluate` with `run_evolution(..., evaluator=evaluate)`, or expose it in an +evaluator module for the CLI. The helper: + +- Reads the candidate's source file as UTF-8 and constructs + `{"content": {"files": [{"path": filename, "content": source}]}}`. +- Accepts the nested `scores.scores` / `insights.insights` envelope as dictionaries + or SDK-style objects, without importing Google's package in OpenEvolve. +- Keeps all metrics for features and maps the explicitly selected objective to + `combined_score`, avoiding accidental averaging of unrelated metrics. +- Stores insights as JSON in the `alphaevolve_insights` artifact. +- Marks exact `-1e12` sentinel, null, and non-finite scores as failures, with + `error: 0.0`, an error artifact, and a `combined_score` of `-1e12`. This avoids + giving a failed candidate zero fitness when valid objectives are negative. + +The sentinel is an example convention, not a universal SDK rule. Set +`failure_sentinel=None` to disable it or choose a different sentinel. Choose a +finite `failure_score` **below every valid score** for your problem. A failed +secondary metric also marks the whole evaluation failed. Duplicate metrics, +missing objectives and malformed scores raise actionable errors. The names +`error` and `combined_score` are reserved by this mapping; select +`objective_metric="combined_score"` if that is already your original metric. + +Exceptions propagate to OpenEvolve's existing retry/error handling. That generic +handling may return zero fitness on exceptions or timeouts; this adapter does not +change that behavior. For negative objectives, inspect error/timeout results +before interpreting the best program. + +The adapter provides **no sandbox**. Like the original local evaluators, it runs +trusted Python code with the current process's privileges. Use a separate +execution environment for untrusted candidates. + +## Validation + +Default unit tests use synthetic envelopes and OpenEvolve's real evaluator and +callable-serialization path, without Google's SDK, keys, or network access: + +```bash +python -m unittest discover -s tests -p test_alphaevolve_compat.py +``` + +The walkthrough additionally permits direct validation against an upstream +checkout. An evaluator-only run demonstrates contract compatibility, not learned +improvement or a real-model evolution experiment. diff --git a/examples/alphaevolve_interop/run.py b/examples/alphaevolve_interop/run.py new file mode 100644 index 000000000..3e7bdec00 --- /dev/null +++ b/examples/alphaevolve_interop/run.py @@ -0,0 +1,68 @@ +"""Evaluate an unchanged AlphaEvolve example; optionally evolve it with an OE config.""" + +import argparse +import importlib.util +import json +from pathlib import Path + +from openevolve import run_evolution +from openevolve.compat.alphaevolve import adapt_evaluator + +EXAMPLES = { + "circle_packing": ("circle_packing_evaluation", "sum_of_radii"), + "tsp": ("tsp_evaluation_function", "neg_tour_length"), + "signal_processing": ("signal_processing_evaluation", "overall_score"), +} + + +def load_evaluator(example_dir): + """Load the example without importing its cloud client or experiment entrypoint.""" + function_name, objective = EXAMPLES[example_dir.name] + spec = importlib.util.spec_from_file_location( + "_alphaevolve_example", example_dir / "src" / "evaluate.py" + ) + if spec is None or spec.loader is None: + raise ImportError(f"Cannot load evaluator from {example_dir}") + module = importlib.util.module_from_spec(spec) + # Deliberately not registered in sys.modules: cloudpickle transports this + # file-loaded callable by value to OpenEvolve's worker processes. + spec.loader.exec_module(module) + return adapt_evaluator(getattr(module, function_name), objective_metric=objective) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("example_dir", type=Path, help="Path to the upstream example directory") + parser.add_argument( + "--config", type=Path, help="OE YAML config; omit to evaluate only (no LLM)" + ) + parser.add_argument( + "--iterations", type=int, default=1, help="Evolution iterations with --config" + ) + parser.add_argument("--output-dir", type=Path, default=Path("alphaevolve_output")) + args = parser.parse_args() + example_dir = args.example_dir.resolve() + if example_dir.name not in EXAMPLES: + parser.error(f"Supported example names: {', '.join(EXAMPLES)}") + if args.iterations < 1: + parser.error("--iterations must be positive") + program_path = example_dir / "src" / "program.py" + evaluator = load_evaluator(example_dir) + if args.config is None: + result = evaluator(program_path) + print(json.dumps({"metrics": result.metrics, "artifacts": result.artifacts}, indent=2)) + return + + result = run_evolution( + initial_program=program_path, + evaluator=evaluator, + config=args.config, + iterations=args.iterations, + output_dir=str(args.output_dir), + cleanup=True, + ) + print(json.dumps({"metrics": result.metrics, "best_score": result.best_score}, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/openevolve/compat/__init__.py b/openevolve/compat/__init__.py new file mode 100644 index 000000000..6a10159f1 --- /dev/null +++ b/openevolve/compat/__init__.py @@ -0,0 +1 @@ +"""Opt-in interoperability helpers for other evolution frameworks.""" diff --git a/openevolve/compat/alphaevolve.py b/openevolve/compat/alphaevolve.py new file mode 100644 index 000000000..d9beef688 --- /dev/null +++ b/openevolve/compat/alphaevolve.py @@ -0,0 +1,128 @@ +"""Adapt local Google Cloud AlphaEvolve evaluators without a Google SDK dependency.""" + +import json +import math +from collections.abc import Mapping +from dataclasses import dataclass +from numbers import Real +from pathlib import Path +from typing import Any, Callable, Optional + +from openevolve.evaluation_result import EvaluationResult + + +def _field(value: Any, name: str) -> Any: + """Read the same contract from model objects and their model_dump() dictionaries.""" + if isinstance(value, Mapping): + return value.get(name) + return getattr(value, name, None) + + +@dataclass +class _AlphaEvolveEvaluator: + evaluator: Callable + objective_metric: str + failure_sentinel: Optional[float] + failure_score: float + + def __call__(self, program_path: str) -> EvaluationResult: + path = Path(program_path) + candidate = { + "content": {"files": [{"path": path.name, "content": path.read_text(encoding="utf-8")}]} + } + result = self.evaluator(candidate) + scores = _field(_field(result, "scores"), "scores") + if not isinstance(scores, (list, tuple)) or not scores: + raise ValueError("AlphaEvolve evaluation must contain a non-empty scores.scores list") + + metrics = {} + failed_metrics = [] + for entry in scores: + metric, score = _field(entry, "metric"), _field(entry, "score") + if not isinstance(metric, str) or not metric.strip(): + raise ValueError("AlphaEvolve metric names must be non-empty strings") + if metric in metrics or metric == "error": + raise ValueError(f"Duplicate or reserved AlphaEvolve metric: {metric!r}") + if metric == "combined_score" and self.objective_metric != "combined_score": + raise ValueError( + "Select combined_score as objective_metric to preserve that metric" + ) + if score is not None and (isinstance(score, bool) or not isinstance(score, Real)): + raise TypeError(f"AlphaEvolve score for {metric!r} must be a number or None") + if ( + score is None + or not math.isfinite(score) + or (self.failure_sentinel is not None and score == self.failure_sentinel) + ): + failed_metrics.append(metric) + metrics[metric] = self.failure_score + else: + metrics[metric] = float(score) + + if self.objective_metric not in metrics: + raise ValueError( + f"AlphaEvolve evaluation is missing objective {self.objective_metric!r}" + ) + + artifacts = {} + insights = _field(_field(result, "insights"), "insights") + if insights: + artifacts["alphaevolve_insights"] = json.dumps( + [ + {"label": _field(item, "label"), "text": _field(item, "text")} + for item in insights + ], + ensure_ascii=False, + ) + + # Pick an explicit objective, rather than averaging unrelated feature metrics. + metrics["combined_score"] = metrics[self.objective_metric] + if failed_metrics: + # OpenEvolve error metrics alone have fitness zero, which could beat a + # valid negative objective (e.g. neg_tour_length). Retain a failure fitness. + metrics["combined_score"] = self.failure_score + metrics["error"] = 0.0 + artifacts["error"] = "AlphaEvolve evaluation failed for metrics: " + ", ".join( + failed_metrics + ) + return EvaluationResult(metrics=metrics, artifacts=artifacts) + + +def adapt_evaluator( + evaluator: Callable, + *, + objective_metric: str, + failure_sentinel: Optional[float] = -1e12, + failure_score: float = -1e12, +) -> Callable[[str], EvaluationResult]: + """Wrap a synchronous, single-file AlphaEvolve evaluator for OpenEvolve. + + The callable receives ``content.files[0].content`` instead of a file path. + Its ``scores.scores`` envelope may be a dictionary or SDK-style object. + All metrics are retained; ``objective_metric`` is also used as combined_score. + Insights become JSON in the ``alphaevolve_insights`` artifact. + + None/non-finite scores and the exact configurable failure sentinel mark the + evaluation failed. Choose a finite ``failure_score`` below all valid scores, + especially for negative objectives. Set ``failure_sentinel=None`` if -1e12 + is a legitimate score. Exceptions and malformed contracts propagate to the + existing OpenEvolve evaluator error handling. + + This only adapts evaluation: it does not provide cloud SDK emulation, config + translation, multi-file candidates, or execution isolation. The wrapped + evaluator executes trusted candidate code with its existing privileges. + """ + if not callable(evaluator): + raise TypeError("evaluator must be callable") + if not isinstance(objective_metric, str) or not objective_metric.strip(): + raise ValueError("objective_metric must be a non-empty string") + if objective_metric == "error": + raise ValueError("error is reserved for failure handling") + for name, value in (("failure_score", failure_score), ("failure_sentinel", failure_sentinel)): + if name == "failure_sentinel" and value is None: + continue + if isinstance(value, bool) or not isinstance(value, Real) or not math.isfinite(value): + raise ValueError(f"{name} must be finite") + return _AlphaEvolveEvaluator( + evaluator, objective_metric, failure_sentinel, float(failure_score) + ) diff --git a/tests/test_alphaevolve_compat.py b/tests/test_alphaevolve_compat.py new file mode 100644 index 000000000..0e4524455 --- /dev/null +++ b/tests/test_alphaevolve_compat.py @@ -0,0 +1,193 @@ +"""Offline tests for the opt-in AlphaEvolve evaluator contract (#468).""" + +import asyncio +import json +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace + +from openevolve.api import _prepare_evaluator, run_evolution +from openevolve.compat.alphaevolve import adapt_evaluator +from openevolve.config import Config, EvaluatorConfig, LLMConfig, LLMModelConfig +from openevolve.evaluator import Evaluator + + +def evaluation(scores, insights=None): + return { + "scores": {"scores": [{"metric": key, "score": value} for key, value in scores]}, + "insights": insights, + } + + +class ScriptedModel: + """An importable, deterministic provider, including inside spawned workers.""" + + def __init__(self, config): + self.model = config.name + + async def generate_with_context(self, system_message, messages, **kwargs): + return "<<<<<<< SEARCH\nx = 1\n=======\nx = 2\n>>>>>>> REPLACE" + + +def create_scripted_model(config): + return ScriptedModel(config) + + +class TestAlphaEvolveAdapter(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.program = Path(self.temp.name) / "program.py" + self.program.write_text("# 中文\nx = 1\n", encoding="utf-8") + + def adapt(self, result, **kwargs): + return adapt_evaluator(lambda candidate: result, objective_metric="fitness", **kwargs) + + def test_candidate_contains_original_source_and_file_name(self): + def score(candidate): + self.assertEqual( + candidate, + {"content": {"files": [{"path": "program.py", "content": "# 中文\nx = 1\n"}]}}, + ) + return evaluation([("fitness", 0.8), ("diversity", 2)]) + + result = adapt_evaluator(score, objective_metric="fitness")(self.program) + self.assertEqual(result.metrics, {"fitness": 0.8, "diversity": 2.0, "combined_score": 0.8}) + + def test_accepts_sdk_style_objects_without_importing_sdk(self): + result = SimpleNamespace( + scores=SimpleNamespace(scores=[SimpleNamespace(metric="fitness", score=-12.5)]), + insights=SimpleNamespace(insights=[SimpleNamespace(label="Hint", text="Try again")]), + ) + converted = self.adapt(result)(self.program) + self.assertEqual(converted.metrics["combined_score"], -12.5) + self.assertEqual( + json.loads(converted.artifacts["alphaevolve_insights"]), + [{"label": "Hint", "text": "Try again"}], + ) + + def test_dictionary_insights_are_preserved(self): + insights = {"insights": [{"label": "Runtime Error", "text": "bad input"}]} + result = self.adapt(evaluation([("fitness", 1.0)], insights))(self.program) + self.assertEqual(json.loads(result.artifacts["alphaevolve_insights"]), insights["insights"]) + + def test_objective_is_not_average_of_auxiliary_metrics(self): + result = self.adapt(evaluation([("fitness", -10), ("length", 1000)]))(self.program) + self.assertEqual(result.metrics["combined_score"], -10) + + def test_failure_sentinel_null_and_nonfinite_scores_fail_closed(self): + for value in (-1e12, None, float("nan"), float("inf"), -float("inf")): + with self.subTest(value=value): + result = self.adapt(evaluation([("fitness", value), ("length", 100)]))(self.program) + self.assertEqual(result.metrics["combined_score"], -1e12) + self.assertEqual(result.metrics["fitness"], -1e12) + self.assertEqual(result.metrics["length"], 100) + self.assertEqual(result.metrics["error"], 0.0) + self.assertIn("fitness", result.artifacts["error"]) + + def test_failed_auxiliary_metric_also_marks_evaluation_failed(self): + result = self.adapt(evaluation([("fitness", 1), ("length", None)]))(self.program) + self.assertEqual(result.metrics["combined_score"], -1e12) + + def test_failure_mapping_is_configurable(self): + result = self.adapt( + evaluation([("fitness", -999)]), failure_sentinel=-999, failure_score=-1e20 + )(self.program) + self.assertEqual(result.metrics["combined_score"], -1e20) + + def test_sentinel_can_be_disabled_for_a_valid_negative_score(self): + result = self.adapt(evaluation([("fitness", -1e12)]), failure_sentinel=None)(self.program) + self.assertNotIn("error", result.metrics) + + def test_missing_duplicate_or_reserved_metrics_raise(self): + for scores in ( + [], + [("other", 1)], + [("fitness", 1), ("fitness", 2)], + [("fitness", 1), ("combined_score", 2)], + [("error", 1), ("fitness", 2)], + ): + with self.subTest(scores=scores), self.assertRaises(ValueError): + self.adapt(evaluation(scores))(self.program) + + def test_malformed_names_scores_and_envelopes_raise(self): + for result in ( + {}, + {"scores": {}}, + {"scores": {"scores": "not a list"}}, + evaluation([(None, 1)]), + evaluation([("", 1)]), + evaluation([("fitness", True)]), + evaluation([("fitness", "1.0")]), + ): + with self.subTest(result=result), self.assertRaises((TypeError, ValueError)): + self.adapt(result)(self.program) + + def test_combined_score_can_be_the_original_objective(self): + adapter = adapt_evaluator( + lambda candidate: evaluation([("combined_score", 0.5)]), + objective_metric="combined_score", + ) + self.assertEqual(adapter(self.program).metrics, {"combined_score": 0.5}) + + def test_configuration_validation(self): + for options in ( + {"objective_metric": ""}, + {"objective_metric": "error"}, + {"objective_metric": "fitness", "failure_score": float("nan")}, + {"objective_metric": "fitness", "failure_sentinel": float("inf")}, + ): + with self.subTest(options=options), self.assertRaises(ValueError): + adapt_evaluator(lambda candidate: {}, **options) + with self.assertRaises(TypeError): + adapt_evaluator(None, objective_metric="fitness") + + def test_evaluator_errors_are_not_hidden(self): + def broken(candidate): + raise RuntimeError("broken evaluator") + + with self.assertRaisesRegex(RuntimeError, "broken evaluator"): + adapt_evaluator(broken, objective_metric="fitness")(self.program) + + def test_existing_callable_serialization_and_evaluator_pipeline(self): + adapter = self.adapt( + evaluation([("fitness", -3.0)], {"insights": [{"label": "Hint", "text": "Keep"}]}) + ) + files = [] + path = _prepare_evaluator(adapter, self.temp.name, files) + evaluator = Evaluator(EvaluatorConfig(cascade_evaluation=False, max_retries=0), path) + self.addCleanup(evaluator._executor.shutdown, wait=True) + result = asyncio.run(evaluator.evaluate_program("x = 1", "candidate")) + self.assertEqual(result, {"fitness": -3.0, "combined_score": -3.0}) + self.assertIn("alphaevolve_insights", evaluator.get_pending_artifacts("candidate")) + + def test_one_generation_through_real_controller_and_worker(self): + def score(candidate): + namespace = {} + exec(candidate["content"]["files"][0]["content"], namespace) + return evaluation([("fitness", namespace["x"])]) + + config = Config() + config.llm = LLMConfig( + models=[LLMModelConfig(name="offline-script", init_client=create_scripted_model)] + ) + config.database.num_islands = 1 + config.evaluator.cascade_evaluation = False + config.evaluator.max_retries = 0 + config.evaluator.parallel_evaluations = 1 + config.evaluator.use_llm_feedback = False + result = run_evolution( + "# EVOLVE-BLOCK-START\nx = 1\n# EVOLVE-BLOCK-END\n", + adapt_evaluator(score, objective_metric="fitness"), + config=config, + iterations=1, + output_dir=str(Path(self.temp.name) / "evolution"), + ) + self.assertEqual(result.best_score, 2.0) + self.assertEqual(result.best_program.generation, 1) + self.assertIn("x = 2", result.best_code) + + +if __name__ == "__main__": + unittest.main()