diff --git a/examples/alphaevolve_interop/README.md b/examples/alphaevolve_interop/README.md index b47ca1c1c..06092cd93 100644 --- a/examples/alphaevolve_interop/README.md +++ b/examples/alphaevolve_interop/README.md @@ -102,19 +102,29 @@ output. The completed `EvolutionResult` is available as `experiment.result`. `list_programs` returns **only the best result**, using the SDK's `alphaEvolvePrograms` content/evaluation envelope. It is not an archive listing; there is no pagination. `order_by=" asc|desc"` is accepted for -entrypoint migration but has no effect on a one-item result. Unknown metrics or -other query parameters raise errors. OE's internal `combined_score` is omitted +entrypoint migration but has no effect on a one-item result. The generic `score` +ordering names the client's explicit objective, not an average of metrics. +Unknown metrics or other query parameters raise errors. OE's internal `combined_score` is omitted unless it was explicitly selected as the objective; numeric objective/feature scores, including negatives, are retained. An error metric adds a generic -failure insight. `run_controller_loop` only checks successful completion and -accepts no cloud worker or timeout options. - -This is **not yet** enough to execute the original three upstream entrypoints -unchanged: complete archive access, remote client/controller methods and the -full SDK model surface remain unsupported. Local connection settings are still -required, and cloud-specific configuration/controller calls need adaptation. -The runner below remains available for unchanged upstream evaluators. -No managed Google Cloud run is implied by a local facade run. +failure insight. `run_controller_loop` only checks successful completion. +Explicit positive `num_samplers` and `num_evaluators` integers are accepted with +a warning that they are ignored: the synchronous OE run has already finished, +and no workers are created or concurrency changed. Other cloud controller +options, including idle timeouts, remain unsupported. + +`client.get_alpha_evolve_experiment(experiment_name=...)` is an in-memory lookup +of experiments created by that client. It returns a fresh `{"name": ..., "state": ...}` +dictionary, including for a created-but-not-started diagnostic experiment. +State reflects the local lifecycle; it is not a cloud resource/status schema. +Unknown names and names from another client raise `ValueError`. No configuration, +credentials, program data or requests are involved; the client retains its local +experiments for the client's lifetime. + +The scoped runner below now executes the three reviewed original entrypoints +with explicit configuration/factory swaps. Complete archive access, other +remote client/controller methods and the full SDK model surface remain +unsupported. No managed Google Cloud run is implied by a local facade run. ```bash python -m unittest discover -s tests -p 'test_alphaevolve_local_experiment.py' -v @@ -201,15 +211,15 @@ MPLBACKEND=Agg python examples/alphaevolve_interop/run.py \ python -m unittest discover -s tests -p 'test_alphaevolve_local_imports.py' -v ``` -This closes the local evaluator-type/import gap, not the complete SDK shim or -the original-entrypoint acceptance criterion for #468. It neither enables cloud -execution nor changes the facade's budget, concurrency or best-only query limits. +These types close the local evaluator-type/import gap, not the complete SDK shim +by themselves. They neither enable cloud execution nor change the facade's +budget, concurrency or best-only query limits. -## Run the original circle-packing or TSP entrypoint +## Run the original local-evaluation entrypoints `run_entrypoint.py` executes the upstream `src/run_evolution.py`'s original -`main()` for **circle_packing** and **tsp** with explicit local connection/model -settings. It stages byte-identical copies of `src/` and optional +`main()` for **circle_packing**, **tsp** and **signal_processing** with explicit +local connection/model settings. It stages byte-identical copies of `src/` and optional `instructions.md`, imports their relative helpers under temporary local aliases, and swaps imported client/experiment factories and configuration globals. It does not rewrite the source, replace the evaluator, or contact Google's SDK. @@ -231,6 +241,12 @@ MPLBACKEND=Agg python examples/alphaevolve_interop/run_entrypoint.py \ ../alphaevolve-reference/examples/circle_packing \ --api-base http://localhost:8000/v1 --model my-local-model \ --evolve --iterations 1 --output-dir ./circle-local-run + +# Run the original signal-processing main without model or Google Cloud requests. +python examples/alphaevolve_interop/run_entrypoint.py \ + ../alphaevolve-reference/examples/signal_processing \ + --api-base http://localhost:8000/v1 --model my-local-model \ + --output-dir ./signal-local-seed ``` The endpoint/model are required even for the default seed-only run; their live @@ -239,9 +255,8 @@ to one generation attempt. Explicit positive `--iterations` requires `--evolve`. These are OE generation attempts, not hard evaluation/request/time/cost caps; seed evaluation, retries and the original reporting code's own evaluations are additional. An upstream entrypoint can score its seed before OE scores it again. -This runner does not execute signal processing or remote/cloud-evaluation -examples; signal processing still needs its cloud-specific query/controller -calls adapted in a separate step. +Remote/cloud-evaluation examples such as adaptive sorting and LLM fine tuning +remain unsupported. Only a named `--api-key-env VARIABLE_NAME` selects a key, which must be non-empty. Without it, an explicit local placeholder is used, including for provider-client @@ -261,6 +276,22 @@ facade keeps its existing single-worker behavior and unsupported-field warnings. Missing expected configuration names cause a layout-drift error rather than a guessed adaptation. +Signal processing's reviewed entrypoint has no model configuration. For that +entrypoint only, a local experiment subclass supplies a single +`generation_settings.models` entry from the required `--model`, overriding any +upstream model list without mutating the original dictionary. The generic facade +still requires explicit models; it does not guess a default. The original main +first creates/queries a diagnostic experiment, then creates a separate experiment +to run. The runner requires precisely this `CREATED`, `COMPLETED` sequence; +an early return, caught diagnostic error or incomplete run is not success. + +While signal processing's main runs, `SAVE_PROGRAM_DIR` is set to the staged +`programs/` directory and restored afterwards, including on failure. This keeps +the evaluator's optional archive writes away from an inherited destination. +The original saver can reuse filenames; these files are not a complete archive +of evaluations. No other arbitrary environment or filesystem side effects are +isolated. + `.env*` files and bytecode caches are excluded from the staged `src/` copy; symlinks in the selected source tree or instructions are rejected. Original source files remain untouched, and reports operate on the staged copy. A new @@ -288,7 +319,7 @@ helpers, executes with normal process privileges; copying and alias cleanup provide no sandbox or general rollback of external side effects. Dynamic relative imports after initial loading are outside this reviewed subset. -This walkthrough covers two original-entrypoint paths toward #468. It is not a +This walkthrough covers three original-entrypoint paths toward #468. It is not a full SDK replacement, managed-service validation or completion of the issue. ```bash diff --git a/examples/alphaevolve_interop/run_entrypoint.py b/examples/alphaevolve_interop/run_entrypoint.py index e57f82518..eaf935362 100644 --- a/examples/alphaevolve_interop/run_entrypoint.py +++ b/examples/alphaevolve_interop/run_entrypoint.py @@ -9,6 +9,7 @@ import sys import tempfile import uuid +from collections.abc import Mapping from pathlib import Path from types import ModuleType from unittest.mock import patch @@ -19,7 +20,11 @@ local_imports, ) -EXAMPLES = {"circle_packing": "sum_of_radii", "tsp": "neg_tour_length"} +EXAMPLES = { + "circle_packing": "sum_of_radii", + "tsp": "neg_tour_length", + "signal_processing": "overall_score", +} def _load_entrypoint(root, settings): @@ -53,7 +58,7 @@ def _load_entrypoint(root, settings): def _validate_source(example_dir): example_dir = Path(example_dir).resolve() if example_dir.name not in EXAMPLES: - raise ValueError("Supported upstream entrypoints: circle_packing, tsp") + raise ValueError("Supported upstream entrypoints: " + ", ".join(EXAMPLES)) source = example_dir / "src" required = [source / name for name in ("run_evolution.py", "program.py", "evaluate.py")] if example_dir.name == "tsp": @@ -72,7 +77,7 @@ def _validate_source(example_dir): def run_entrypoint(example_dir, *, api_base, model, api_key=None, iterations=0, output_dir=None): """Run trusted, byte-identical staged sources; default to seed evaluation only. - Only the two reviewed entrypoints are supported. Imports are process-global; + Only the three reviewed entrypoints are supported. Imports are process-global; use this example runner in a fresh, single-threaded main process. Generation uses the chosen endpoint and may incur charges when iterations is positive. Reports operate on the facade's best-only result, not a full search history. @@ -96,8 +101,26 @@ def run_entrypoint(example_dir, *, api_base, model, api_key=None, iterations=0, experiments = [] + class SignalProcessingExperiment(AlphaEvolveExperiment): + def create_experiment(self, config): + # This upstream entrypoint has no generation_settings.models. Supply + # the caller's explicit model as a configuration swap, never a default + # model or an in-place change to the original experiment dictionary. + if not isinstance(config, Mapping): + raise ValueError("experiment must be a mapping") + generation = config.get("generation_settings", {}) + if not isinstance(generation, Mapping): + raise ValueError("generation_settings must be a mapping") + selected = dict(generation) | {"models": [{"name": model, "weight": 1.0}]} + super().create_experiment(dict(config) | {"generation_settings": selected}) + def experiment_factory(*args, **kwargs): - experiment = AlphaEvolveExperiment(*args, **kwargs) + experiment_type = ( + SignalProcessingExperiment + if example_dir.name == "signal_processing" + else AlphaEvolveExperiment + ) + experiment = experiment_type(*args, **kwargs) experiments.append(experiment) return experiment @@ -138,15 +161,23 @@ def experiment_factory(*args, **kwargs): "AlphaEvolveExperiment", "MAX_PROGRAMS_GENERATED", "MAX_PROGRAMS_EVALUATED", - "PARALLEL_EVALUATION", "CONCURRENCY", } if example_dir.name == "circle_packing": required.update( - {"MODEL_1", "MODEL_2", "MODEL_1_WEIGHT", "MODEL_2_WEIGHT", "WORKER_CONCURRENCY"} + { + "MODEL_1", + "MODEL_2", + "MODEL_1_WEIGHT", + "MODEL_2_WEIGHT", + "WORKER_CONCURRENCY", + "PARALLEL_EVALUATION", + } ) + elif example_dir.name == "tsp": + required.update({"MODEL", "PARALLEL_EVALUATION"}) else: - required.add("MODEL") + required.add("run_controller_loop") if any(not hasattr(entrypoint, key) for key in required) or not callable(entrypoint.main): raise ValueError("Upstream entrypoint layout changed; review its configuration first") @@ -168,11 +199,28 @@ def client_factory(*args, **kwargs): entrypoint.MODEL_1, entrypoint.MODEL_2 = model, "" entrypoint.MODEL_1_WEIGHT, entrypoint.MODEL_2_WEIGHT = 1.0, 0.0 entrypoint.WORKER_CONCURRENCY = 1 - else: + elif example_dir.name == "tsp": entrypoint.MODEL = model - entrypoint.main() - if not experiments or any(experiment.state != "COMPLETED" for experiment in experiments): + # The signal evaluator optionally writes archives to an environment path. + # Keep those artifacts in this run, rather than an inherited destination. + environment = ( + {"SAVE_PROGRAM_DIR": str(root / "programs")} + if example_dir.name == "signal_processing" + else {} + ) + with patch.dict(os.environ, environment): + entrypoint.main() + states = [experiment.state for experiment in experiments] + # The reviewed signal main first creates/queries a diagnostic experiment, + # then creates a separate one to run. Do not accept an early caught error + # or silently discard any other incomplete/failed experiment. + complete = ( + states == ["CREATED", "COMPLETED"] + if example_dir.name == "signal_processing" + else bool(states) and all(state == "COMPLETED" for state in states) + ) + if not complete: raise RuntimeError("Upstream entrypoint returned without completing a local experiment") result = experiments[-1].result return { diff --git a/openevolve/compat/alpha_evolve/client.py b/openevolve/compat/alpha_evolve/client.py index 1d70d3dbf..5f81b8593 100644 --- a/openevolve/compat/alpha_evolve/client.py +++ b/openevolve/compat/alpha_evolve/client.py @@ -57,3 +57,16 @@ def __init__( self.api_key = api_key self.iterations = iterations self.output_dir = str(output_dir) if output_dir is not None else None + self._experiments = {} + + def get_alpha_evolve_experiment(self, experiment_name: str) -> dict: + """Return name/state for an experiment created by this local client. + + This is an in-memory diagnostic lookup, not a cloud query or SDK + experiment resource. Configuration, credentials and program data are + intentionally not returned. The client retains its local experiments. + """ + if not isinstance(experiment_name, str) or experiment_name not in self._experiments: + raise ValueError("experiment_name must identify an experiment created by this client") + experiment = self._experiments[experiment_name] + return {"name": experiment.experiment_name, "state": experiment.state} diff --git a/openevolve/compat/alpha_evolve/controller.py b/openevolve/compat/alpha_evolve/controller.py index afc305a74..72eae0ed3 100644 --- a/openevolve/compat/alpha_evolve/controller.py +++ b/openevolve/compat/alpha_evolve/controller.py @@ -1,14 +1,36 @@ """Async-shaped completion check for synchronous local experiments.""" +import warnings +from typing import Optional + from .experiment import AlphaEvolveExperiment -async def run_controller_loop(experiment: AlphaEvolveExperiment) -> None: +async def run_controller_loop( + experiment: AlphaEvolveExperiment, + *, + num_samplers: Optional[int] = None, + num_evaluators: Optional[int] = None, +) -> None: """Check completion without running again, polling queues or creating workers. - The local ``start_experiment`` already ran OpenEvolve to completion. Cloud - sampler/evaluator/idle-timeout arguments are deliberately unsupported. + The local ``start_experiment`` already ran OpenEvolve to completion. + Explicit positive sampler/evaluator counts are accepted for entrypoint + migration and warn that they do not create workers or affect concurrency. + Other cloud controller options remain unsupported. """ if not isinstance(experiment, AlphaEvolveExperiment): raise TypeError("experiment must be the local AlphaEvolveExperiment") + for value in (num_samplers, num_evaluators): + if value is not None and ( + isinstance(value, bool) or not isinstance(value, int) or value < 1 + ): + raise ValueError("num_samplers and num_evaluators must be positive integers or None") experiment._require_state("COMPLETED") + if num_samplers is not None or num_evaluators is not None: + warnings.warn( + "Local run_controller_loop ignores num_samplers/num_evaluators; " + "start_experiment already completed the synchronous OE run", + UserWarning, + stacklevel=2, + ) diff --git a/openevolve/compat/alpha_evolve/experiment.py b/openevolve/compat/alpha_evolve/experiment.py index f15c7cfaa..df6d40826 100644 --- a/openevolve/compat/alpha_evolve/experiment.py +++ b/openevolve/compat/alpha_evolve/experiment.py @@ -93,6 +93,7 @@ def create_experiment(self, config: Mapping) -> None: self._config = local self.experiment_name = "local/experiments/" + uuid.uuid4().hex self.state = "CREATED" + self.client._experiments[self.experiment_name] = self def create_initial_program(self, initial_program: Mapping) -> None: """Accept one UTF-8 Python source; supplied seed scores are not trusted.""" @@ -179,12 +180,16 @@ def list_programs(self, params: Optional[Mapping] = None) -> dict[str, Any]: if "order_by" in params: order = params["order_by"] parts = order.rsplit(" ", 1) if isinstance(order, str) else [] + # Some upstream entrypoints use a generic score ordering. It names + # the explicitly selected objective, not an average of all metrics. + if len(parts) == 2 and parts[0] == "score": + parts[0] = self.client.objective_metric if ( len(parts) != 2 or parts[1] not in ("asc", "desc") or parts[0] not in {score["metric"] for score in scores} ): - raise ValueError("order_by must name a returned metric and asc or desc") + raise ValueError("order_by must name a returned metric (or score) and asc or desc") evaluation = {"scores": {"scores": scores}} if "error" in metrics: evaluation["insights"] = { diff --git a/tests/test_alphaevolve_entrypoint.py b/tests/test_alphaevolve_entrypoint.py index a6418eee0..f9a95cc5e 100644 --- a/tests/test_alphaevolve_entrypoint.py +++ b/tests/test_alphaevolve_entrypoint.py @@ -6,6 +6,7 @@ import json import os import secrets +import shutil import signal import sys import tempfile @@ -46,7 +47,8 @@ def setUp(self): name: module for name, module in sys.modules.items() if name.startswith("alpha_evolve") } self.options = {"api_base": "http://localhost:8000/v1", "model": "offline-model"} - for name, metric in runner.EXAMPLES.items(): + for name in ("circle_packing", "tsp"): + metric = runner.EXAMPLES[name] directory = self.root / name source = directory / "src" helpers = source / "utils" @@ -148,8 +150,164 @@ def run_offline(self, example, **options): warnings.simplefilter("ignore", UserWarning) return runner.run_entrypoint(example, **(self.options | options)) + def signal_example(self): + example = self.root / "signal_processing" + shutil.copytree(self.example, example) + source = example / "src" + evaluator = source / "evaluate.py" + evaluator.write_text( + "import os\n" + + evaluator.read_text() + .replace("'sum_of_radii'", "'overall_score'") + .replace("def evaluate(candidate):", "def signal_processing_evaluation(candidate):") + .replace( + " return AlphaEvolveProgramEvaluation", + " archive = Path(os.environ['SAVE_PROGRAM_DIR'])\n" + " archive.mkdir(exist_ok=True)\n" + " (archive / 'candidate.py').write_text(candidate['content']['files'][0]['content'])\n" + " return AlphaEvolveProgramEvaluation", + ) + ) + (source / "run_evolution.py").write_text( + "import asyncio\n" + "import json\n" + "import os\n" + "from pathlib import Path\n" + "from dotenv import load_dotenv\n" + "from alpha_evolve.client import AlphaEvolveClient\n" + "from alpha_evolve.experiment import AlphaEvolveExperiment\n" + "from alpha_evolve.controller import run_controller_loop\n" + "from .evaluate import signal_processing_evaluation, INITIAL_PROGRAM_CODE\n" + "load_dotenv()\n" + "MAX_PROGRAMS_GENERATED = MAX_PROGRAMS_EVALUATED = 10\n" + "CONCURRENCY = 2\n" + "def main():\n" + " client = AlphaEvolveClient(project_id='ignored-project')\n" + " config = {\n" + " 'program_language': 'python', 'problem_description': 'Maximize x',\n" + " 'run_settings': {'max_programs': MAX_PROGRAMS_GENERATED, 'concurrency': CONCURRENCY},\n" + " }\n" + " try:\n" + " probe = AlphaEvolveExperiment(client, signal_processing_evaluation, MAX_PROGRAMS_EVALUATED)\n" + " probe.create_experiment(config)\n" + " metadata = client.get_alpha_evolve_experiment(experiment_name=probe.experiment_name)\n" + " except Exception:\n" + " return\n" + " experiment = AlphaEvolveExperiment(client, signal_processing_evaluation, MAX_PROGRAMS_EVALUATED)\n" + " experiment.create_experiment(config)\n" + " assert 'generation_settings' not in config\n" + " experiment.create_initial_program({'content': {'files': [{'path': 'program.py', 'content': INITIAL_PROGRAM_CODE}]}})\n" + " experiment.start_experiment()\n" + " asyncio.run(run_controller_loop(experiment, num_samplers=CONCURRENCY, num_evaluators=CONCURRENCY))\n" + " result = experiment.list_programs({'order_by': 'score desc'})\n" + " output = Path(__file__).resolve().parents[1]\n" + " report = {'probe': metadata, 'result': result, 'budget': MAX_PROGRAMS_GENERATED,\n" + " 'concurrency': CONCURRENCY, 'save_dir': os.environ['SAVE_PROGRAM_DIR']}\n" + " (output / 'report.json').write_text(json.dumps(report))\n" + ) + return example + + def test_signal_original_lifecycle_runs_seed_and_real_worker_with_selected_model(self): + example = self.signal_example() + original = {p.relative_to(example): p.read_bytes() for p in example.rglob("*.py")} + for iterations in (0, 1): + output = self.root / f"signal-output-{iterations}" + with self.subTest(iterations=iterations): + result = self.run_offline(example, iterations=iterations, output_dir=output) + self.assertEqual(result["metrics"]["overall_score"], 1 + iterations) + report = json.loads((output / "report.json").read_text()) + self.assertEqual(report["probe"]["state"], "CREATED") + self.assertEqual(report["budget"], max(1, iterations)) + self.assertEqual(report["concurrency"], 1) + self.assertEqual(report["save_dir"], str(output / "programs")) + self.assertTrue((output / "programs/candidate.py").is_file()) + self.assertTrue((output / "oe-output/best/best_program.py").is_file()) + self.assertNotEqual( + report["probe"]["name"], + report["result"]["alphaEvolvePrograms"][0]["name"].rsplit("/programs/", 1)[0], + ) + for path, content in original.items(): + self.assertEqual((example / path).read_bytes(), content) + self.assertEqual((output / path).read_bytes(), content) + + def test_signal_explicit_model_overrides_supplied_models_without_mutating_config(self): + example = self.signal_example() + source = example / "src/run_evolution.py" + source.write_text( + source.read_text() + .replace( + "'program_language':", + "'generation_settings': {'models': [{'name': 'upstream-model'}]}, 'program_language':", + ) + .replace( + "assert 'generation_settings' not in config", + "assert config['generation_settings']['models'] == [{'name': 'upstream-model'}]", + ) + ) + configurations = [] + configure = self.offline_config + + def record(*args, **kwargs): + local = configure(*args, **kwargs) + configurations.append([(m.name, m.weight) for m in local.llm.models]) + return local + + with patch.object(self, "offline_config", side_effect=record): + self.run_offline(example, iterations=1) + self.assertEqual(configurations, [[("offline-model", 1.0)]] * 2) + + def test_signal_probe_query_failure_or_early_return_is_not_success(self): + example = self.signal_example() + with patch.object( + runner.AlphaEvolveClient, + "get_alpha_evolve_experiment", + side_effect=ValueError("lookup failed"), + ): + with self.assertRaisesRegex(RuntimeError, "without completing"): + self.run_offline(example) + source = example / "src/run_evolution.py" + source.write_text( + source.read_text().replace(" experiment.start_experiment()", " return") + ) + with self.assertRaisesRegex(RuntimeError, "without completing"): + self.run_offline(example) + + def test_signal_archive_environment_is_restored_on_success_and_failure(self): + example = self.signal_example() + inherited = self.root / "must-not-create-archives" + with patch.dict(os.environ, {"SAVE_PROGRAM_DIR": str(inherited)}): + self.run_offline(example) + self.assertEqual(os.environ["SAVE_PROGRAM_DIR"], str(inherited)) + source = example / "src/run_evolution.py" + source.write_text( + source.read_text().replace( + " experiment.start_experiment()", + " raise RuntimeError('entrypoint failed')", + ) + ) + with self.assertRaisesRegex(RuntimeError, "entrypoint failed"): + self.run_offline(example) + self.assertEqual(os.environ["SAVE_PROGRAM_DIR"], str(inherited)) + self.assertFalse(inherited.exists()) + + def test_signal_layout_drift_and_malformed_generation_settings_are_rejected(self): + example = self.signal_example() + source = example / "src/run_evolution.py" + original = source.read_text() + source.write_text(original.replace("CONCURRENCY = 2", "REMOVED_CONCURRENCY = 2")) + with self.assertRaisesRegex(ValueError, "layout changed"): + self.run_offline(example) + source.write_text( + original.replace( + "'program_language':", "'generation_settings': None, 'program_language':" + ) + ) + with self.assertRaisesRegex(RuntimeError, "without completing"): + self.run_offline(example) + def test_original_main_relative_helpers_and_real_workers_run_with_config_swaps(self): - for name, metric in runner.EXAMPLES.items(): + for name in ("circle_packing", "tsp"): + metric = runner.EXAMPLES[name] example = self.root / name originals = { path.relative_to(example): path.read_bytes() @@ -246,7 +404,7 @@ def test_bad_configuration_fails_before_creating_output_or_importing(self): self.assertFalse(output.exists()) def test_unsupported_layout_missing_sources_and_symlinks_are_rejected(self): - unsupported = self.root / "signal_processing" + unsupported = self.root / "adaptive_sort" unsupported.mkdir() with self.assertRaisesRegex(ValueError, "Supported upstream"): runner.run_entrypoint(unsupported, **self.options) diff --git a/tests/test_alphaevolve_local_experiment.py b/tests/test_alphaevolve_local_experiment.py index 8f077c340..5b3d8c149 100644 --- a/tests/test_alphaevolve_local_experiment.py +++ b/tests/test_alphaevolve_local_experiment.py @@ -368,11 +368,79 @@ def test_failed_evaluation_result_retains_failure_insight(self): self.assertEqual(evaluation["scores"]["scores"], [{"metric": "quality", "score": -1e12}]) self.assertIn("Error", evaluation["insights"]["insights"][0]["label"]) - def test_controller_rejects_other_sdk_and_cloud_worker_arguments(self): + def test_controller_rejects_other_sdk_and_unimplemented_cloud_options(self): with self.assertRaises(TypeError): asyncio.run(run_controller_loop(object())) with self.assertRaises(TypeError): - run_controller_loop(self.complete(), num_evaluators=4) + run_controller_loop(self.complete(), idle_timeout_s=10) + + def test_controller_accepts_counts_without_rerunning_or_adding_workers(self): + experiment = self.complete() + with ( + self.assertWarnsRegex(UserWarning, "ignores num_samplers/num_evaluators"), + patch(EXPERIMENT_MODULE + ".run_evolution") as evolve, + ): + asyncio.run(run_controller_loop(experiment, num_samplers=2, num_evaluators=4)) + evolve.assert_not_called() + self.assertEqual(experiment.state, "COMPLETED") + for key in ("num_samplers", "num_evaluators"): + for value in (0, -1, True, 1.5, "1"): + with self.subTest(key=key, value=value), self.assertRaises(ValueError): + asyncio.run(run_controller_loop(experiment, **{key: value})) + with self.assertRaises(RuntimeError): + asyncio.run(run_controller_loop(self.ready(), num_evaluators=1)) + + def test_generic_score_ordering_uses_selected_objective_and_preserves_metrics(self): + experiment = self.complete(result=self.fake_result(metrics={"quality": -3, "cost": -2})) + expected = experiment.list_programs({"order_by": "quality desc"}) + for direction in ("asc", "desc"): + self.assertEqual(experiment.list_programs({"order_by": "score " + direction}), expected) + missing = self.complete(result=self.fake_result(metrics={"cost": -2})) + with self.assertRaises(ValueError): + missing.list_programs({"order_by": "score desc"}) + with self.assertRaises(ValueError): + experiment.list_programs({"order_by": "score sideways"}) + + def test_local_query_tracks_lifecycle_without_io_and_returns_independent_metadata(self): + experiment = self.ready(api_key="private-selected-key") + client, name = experiment.client, experiment.experiment_name + with ( + patch("socket.socket.connect", side_effect=AssertionError("network")), + patch("builtins.open", side_effect=AssertionError("filesystem")), + ): + self.assertEqual( + client.get_alpha_evolve_experiment(experiment_name=name), + {"name": name, "state": "READY"}, + ) + response = client.get_alpha_evolve_experiment(name) + response["state"] = "WRONG" + self.assertEqual(client.get_alpha_evolve_experiment(name)["state"], "READY") + self.complete(experiment) + self.assertEqual(client.get_alpha_evolve_experiment(name)["state"], "COMPLETED") + for invalid in (None, [], "unknown", self.ready().experiment_name): + with self.subTest(invalid=invalid), self.assertRaises(ValueError): + client.get_alpha_evolve_experiment(invalid) + + def test_local_query_keeps_created_probes_separate_from_failed_runs(self): + probe = self.experiment() + with warnings.catch_warnings(): + warnings.simplefilter("ignore", UserWarning) + probe.create_experiment(self.payload) + client = probe.client + running = AlphaEvolveExperiment(client, evaluate_candidate, 2) + with warnings.catch_warnings(): + warnings.simplefilter("ignore", UserWarning) + running.create_experiment(self.payload) + running.create_initial_program(self.candidate) + with patch(EXPERIMENT_MODULE + ".run_evolution", side_effect=RuntimeError("run failed")): + with self.assertRaisesRegex(RuntimeError, "run failed"): + running.start_experiment() + self.assertEqual( + client.get_alpha_evolve_experiment(probe.experiment_name)["state"], "CREATED" + ) + self.assertEqual( + client.get_alpha_evolve_experiment(running.experiment_name)["state"], "FAILED" + ) def test_documented_seed_example_works_with_real_provider_client_and_no_ambient_key(self): readme = Path(__file__).resolve().parents[1] / "examples/alphaevolve_interop/README.md"