From aeb72e52903512278ac4d76c80c60443fbcc36dd Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:25:34 -0400 Subject: [PATCH 01/36] E1: receipt for leftover families, value-to-JSON copies and path identity Co-Authored-By: Claude Opus 5.5 --- .../E1-leftover-families.md | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 docs/refactor/consolidation-20261009/E1-leftover-families.md diff --git a/docs/refactor/consolidation-20261009/E1-leftover-families.md b/docs/refactor/consolidation-20261009/E1-leftover-families.md new file mode 100644 index 0000000000..b4e1d47b7d --- /dev/null +++ b/docs/refactor/consolidation-20261009/E1-leftover-families.md @@ -0,0 +1,61 @@ +# E1: Leftover families, value-to-JSON copies and path identity + +**Head audited:** `openhcs` `main` at `72a5c3287` (#1200, K4 merged). **Rules:** [00-RULES.md](00-RULES.md). **Step 4 sweep.** +**Shared abstractions** ([02-SHARED-ABSTRACTIONS.md](02-SHARED-ABSTRACTIONS.md)). *Builds:* none. *Uses:* `AutoRegisterMeta`, python-introspect `to_jsonable` / `RegisteredClassChoices` / `dataclass_from_mapping` (A2 as left by L2 and K6), zmqruntime `TypedWireMessage`, K4's `core/source_path_identity.py`. + +E1 owns what K4 and K6 reported outside their briefs and the whole-tree census of rule-1a violations that no surface owned. It does not touch `core/callable_contract.py` or the function-pattern code (K5), `core/artifacts.py` (K2), or the measurement code held in #1199 (K3); enums whose dispatch lives there are reassigned below. + +## What is wrong + +**Kinds that already have, or need, a family are still enums dispatched by their consumers; two modules hand-write the value-to-JSON switch; two modules compare dataset and store paths with a second identity rule.** + +1. **Source-binding enums (K4 handoff).** `core/source_bindings.py:62-118` declares six enums. Each is re-decided by its consumers: + - `SourceFilterMatchType` (12 members, `requires_value` smuggled through `__new__`) and `SourceFilterSubject` (3) are restated one-to-one as the strategy families `SourceFilterMatcher` and `SourceFilterTargetResolver` (`source_matching.py:113-270`), each leaf naming its member twice (`match_type = X`, `match_type_key = X.value`) and looked up by `.value`. CellProfiler's `LoadImages`/`NamesAndTypes` import keeps a third copy (`processing/backends/cellprofiler/infrastructure.py:86-160`, enums whose members carry our members). + - `MetadataSource` (2): `is MetadataSource.FOLDER_NAME` in `source_matching.metadata_source_texts` and twice in `infrastructure.py`. + - `SourceSetRole` (2): `is MATCHED`/`is BROADCAST` in `source_bindings.py` (3), `source_binding_selection.py` (2), `source_binding_workspace.py` (1). + - `SourceProjectionRole` (2): an `EnumKeyedStrategyMixin` family in `source_binding_workspace.py:507-580` with a totality check against `set(SourceProjectionRole)`, a `projection_role` ClassVar on the `source_projection.py` projection classes, and 14 `is PRIMARY_PLANE`/`is SOURCE_ARTIFACT` tests in `source_bindings.py`, `source_projection.py`, `source_binding_workspace.py`, `compiler.py`, `virtual_workspace_metadata.py`, `function_runtime.py`. + - `SourceBindingOrigin` (2): 8 `is STEP_INPUT`/`is PIPELINE_START` tests in `source_bindings.py` and `function_runtime.py`. +2. **GUI debug enums (K6 handoff).** `DebugActionPlacement` (4) and `DebugToolbarAuxiliaryAction` (1, a `str` enum used as an action id beside `type[DebugCommandType]`) in `pyqt_gui/widgets/shared/services/pipeline_debug_actions.py`; the toolbar groups buttons by placement member (`debug_toolbar.py:81-92`). `DebugSessionPhase` (`core/debug_session_projection.py:31`, 9 members) keys `DebugSessionPhaseDeclarationBase`, which already carries the behaviour; `PipelineDebugSessionPhase = DebugSessionPhase` is an alias. +3. **`CompiledArtifactInspectionControlPayload`** (`core/artifact_inspection.py:150-195`) copies `CompiledArtifactInspectionRequest`'s one field, converts both ways and hand-codes `to_dict`/`from_dict` with a `type` check against the one-member `ArtifactInspectionControlMessageType`, exactly the shape K6 deleted from the debug requests. +4. **numpy-to-JSON switches (K6 handoff).** `core/plate_image_inventory.py:1455-1500` (`_jsonable_array_values`, `_jsonable_metadata_value`, `_jsonable_scalar`) and `napari_roi_manager/_dataclasses.py:91-105` (`_json_value`) each re-decide None, scalars, numpy scalars, arrays, sequences and mappings beside `to_jsonable`, which lacks only the two numpy types. +5. **Dataset and store identity (K4 handoff).** `SourceDatasetIdentity.for_root`, `SourcePlaneStoreIdentity.container_paths`, `SourcePlaneDataset.root` and `.aggregate` (`source_projection.py:243, 259, 439, 479`) and `DatasetSource._register_virtual_workspace_backend` (`dataset_sources/source.py:363-367`) compare `Path.resolve()` results, a second identity rule beside K4's lexical one; `microscopes/bioformats_adapter.py` feeds them resolved paths. +6. **Whole-tree census.** `openhcs/` declares 373 `Enum` subclasses. 185 of them have members compared (`is`, `==`, `in`), matched or used as dict keys in our own code, at 751 sites. 100 of those are in `processing/backends/` (function parameter choices, mostly CellProfiler setting spellings); 85 are elsewhere. The table in *Census* classifies every one. + +## Target + +- **Source bindings.** Each enum becomes an `AutoRegisterMeta` family whose members own the behaviour now switched on; fields hold the member class (`type[F]`, `RegisteredClassChoices(F)` for forms; `to_jsonable` spells it by registry key on the wire and pycodify by class name in saved pipelines): + - `SourceFilterSubject` → `SourceFileName`, `SourceDirectory`, `SourceExtension`, each owning `text_of(path)`; `SourceFilterTargetResolver` is deleted. + - `SourceFilterMatchType` → text predicates (`TextContains`, `TextDoesNotContain`, `TextContainsRegex`, `TextDoesNotContainRegex`, `TextEquals`, `TextDoesNotEqual`, `TextStartsWith`, `TextDoesNotStartWith`, `TextEndsWith`, `TextDoesNotEndWith`) and path predicates (`PathIsImage`, `PathIsTif`); each owns `matches(...)`, and `requires_value` is which base it inherits. `SourceFilterMatcher` and its twelve leaves are deleted. CellProfiler's spelling table keeps CellProfiler's literals and points at our classes; a negated literal names its negated class through the family (`negated`). + - `MetadataSource` → `FromFileName`, `FromFolderName`, owning `texts(path)`. + - `SourceSetRole` → `MatchedSource`/`BroadcastSource`; `SourceProjectionRole` → `PrimaryPlaneProjection`/`SourceArtifactProjection`, keying the workspace strategies by class (`NominalTypeKeyedStrategyMixin`), with totality read from the registry; `SourceBindingOrigin` → `FromStepInput`/`FromPipelineStart`. The `is` tests become what the member answers. +- **GUI debug.** `DebugActionPlacement` → a placement family the toolbar iterates; `DebugToolbarAuxiliaryAction` → the auxiliary action class itself is the identity (its `action_id`). `DebugSessionPhase` is deleted: the `DebugSessionPhaseDeclarationBase` subclasses are the phases (registry key `name`), the alias goes. +- **Artifact inspection.** `CompiledArtifactInspectionRequest` subclasses `TypedWireMessage` (`wire_type = "openhcs_artifact_read_compiled"`); the payload class and `ArtifactInspectionControlMessageType` are deleted; the control strategy keys by `wire_type`. +- **numpy values.** `to_jsonable` registers `numpy.generic` (`.item()`) and `numpy.ndarray` (`.tolist()`) once, in `openhcs/core/numpy_jsonable.py`; both private switches are deleted and their callers call `to_jsonable` (the inventory keeps only its hashable view of the encoded value). +- **Path identity.** Dataset and store identities and roots use `source_path_identity` over `Path.absolute()`; no `resolve()` remains in an identity or equality in these modules or in the adapter that feeds them. +- **Census.** Every dispatched enum outside `processing/backends/` is converted, justified as an external-contract or boundary spelling, or reassigned to the surface holding its files. + +## Persisted state + +Saved `.py` pipelines and configs that name the six source-binding enums (`SourceFilterSubject.FILE`, …) are durable. **E1-Q1:** rewrite them? Default: no tool, as K4-Q1, G1-Q1 and G7-Q1; they fail loudly on load and are re-saved from the GUI. Two presets in `processing/presets/pipelines/` and `benchmark/validation/corpus.py` are rewritten in this PR; `benchmark/results/` is a frozen record and is not. Wire spellings (`to_jsonable` of a member is its registry key) keep the old `.value` strings. + +## Guards + +`tests/unit/test_e1_enum_dispatch_guards.py`: +- No module under `openhcs/` dispatches on an `Enum` member (an `is`/`is not`/`==`/`!=`/`in` comparison with `Enum.MEMBER`, a `match` case on one, a dict display keyed by one, or an `EnumKeyedStrategyMixin` key) unless the enum is listed as a boundary spelling or reassigned in the guard's table, each entry naming its module and reason; the table only shrinks. +- The deleted names (the six source-binding enums, `SourceFilterMatcher`, `SourceFilterTargetResolver`, `DebugSessionPhase`, `PipelineDebugSessionPhase`, `DebugActionPlacement`, `DebugToolbarAuxiliaryAction`, `CompiledArtifactInspectionControlPayload`, `ArtifactInspectionControlMessageType`, `_json_value`, `_jsonable_metadata_value`, `_jsonable_scalar`, `_jsonable_array_values`) stay deleted in the live trees. +- `core/source_projection.py`, `core/dataset_sources/` and `microscopes/bioformats_adapter.py` call no `.resolve(`. + +## Tests + +- One family test: every member of each new source-binding family round-trips through `to_jsonable`/`registered_class_from_key` and answers its behaviour (filter matching over a path table; metadata texts). +- One numpy round trip through `to_jsonable` (scalar, nested array, mapping of arrays). +- Existing source-binding, matching, workspace, CellProfiler import, editor, debug toolbar, artifact-inspection and inventory tests, updated to the families. + +## New-case experiments + +- A new filter predicate (say `TextMatchesGlob`). Today: an enum member, a matcher leaf naming it twice, a CellProfiler spelling row, an editor choice. After: one subclass; the editor and wire derive from the registry, CellProfiler adds a row only if CellProfiler spells it. +- A new numpy-bearing value in a preview: today, a third private switch. After: nothing. + +## Done when + +The brief's items are gone as named; every dispatched enum in the census is converted, justified or reassigned in the guard's table; the guards, the focused suites for touched code, the structural ratchet and nra-r1 pass with no regression. From 6be32aa344f042937b6ed494cb1bdddf001e8c6a Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:29:31 -0400 Subject: [PATCH 02/36] E1: numpy values through to_jsonable; delete two private JSON switches Co-Authored-By: Claude Opus 5.5 --- openhcs/core/numpy_jsonable.py | 21 ++++++ openhcs/core/plate_image_inventory.py | 75 ++++++---------------- openhcs/napari_roi_manager/_dataclasses.py | 21 ++---- 3 files changed, 46 insertions(+), 71 deletions(-) create mode 100644 openhcs/core/numpy_jsonable.py diff --git a/openhcs/core/numpy_jsonable.py b/openhcs/core/numpy_jsonable.py new file mode 100644 index 0000000000..52db230b16 --- /dev/null +++ b/openhcs/core/numpy_jsonable.py @@ -0,0 +1,21 @@ +"""numpy values in ``to_jsonable``, the one value-to-JSON encoder. + +Importing this module registers numpy scalars (as their Python value) and +arrays (as nested lists); modules that encode values which may be numpy import +it before calling ``to_jsonable``. +""" + +from __future__ import annotations + +import numpy as np +from python_introspect import JsonValue, to_jsonable + + +@to_jsonable.register(np.generic) +def _jsonable_numpy_scalar(value: np.generic) -> JsonValue: + return to_jsonable(value.item()) + + +@to_jsonable.register(np.ndarray) +def _jsonable_numpy_array(value: np.ndarray) -> JsonValue: + return to_jsonable(value.tolist()) diff --git a/openhcs/core/plate_image_inventory.py b/openhcs/core/plate_image_inventory.py index ded285a903..bf91a91c85 100644 --- a/openhcs/core/plate_image_inventory.py +++ b/openhcs/core/plate_image_inventory.py @@ -23,10 +23,9 @@ VirtualWorkspaceSourceProjection, WorkspaceSourceProjections, ) -from openhcs.core.virtual_workspace_metadata import ( - JsonScalar, - JsonValue, -) +from python_introspect import JsonValue, to_jsonable + +import openhcs.core.numpy_jsonable # noqa: F401 (numpy values in to_jsonable) if TYPE_CHECKING: from openhcs.core.orchestrator import PipelineOrchestrator @@ -621,12 +620,14 @@ def _semantic_roi_identity(roi: "ROI"): @staticmethod def _roi_shape_identity(shape: "ROIShape"): if not is_dataclass(shape): - return _hashable_jsonable_metadata_value(shape.shape_type.value) - return _hashable_jsonable_metadata_value( - { - "shape_type": shape.shape_type.value, - "fields": asdict(shape), - } + return _hashable(to_jsonable(shape.shape_type.value)) + return _hashable( + to_jsonable( + { + "shape_type": shape.shape_type.value, + "fields": asdict(shape), + } + ) ) @staticmethod @@ -637,7 +638,7 @@ def _roi_example(roi: "ROI") -> Mapping[str, JsonValue]: } for key in ("label", "area", "bbox", "centroid", "plane_indices"): if key in metadata: - example[key] = _jsonable_metadata_value(metadata[key]) + example[key] = to_jsonable(metadata[key]) return example @staticmethod @@ -1396,7 +1397,7 @@ def _summarize( omitted_reason = None sample_values: JsonValue = () if sample_included: - sample_values = _jsonable_array_values(sample) + sample_values = to_jsonable(sample) elif not include_array_values: omitted_reason = "include_array_values is false" else: @@ -1411,12 +1412,12 @@ def _summarize( resolution_shape=tuple(int(value) for value in resolution_shape), dtype=str(sample.dtype), minimum=( - _jsonable_scalar(statistics_array.min()) + to_jsonable(statistics_array.min()) if statistics_array.size else None ), maximum=( - _jsonable_scalar(statistics_array.max()) + to_jsonable(statistics_array.max()) if statistics_array.size else None ), @@ -1452,49 +1453,15 @@ def file_modified_label(file_path: Path) -> str: ) -def _jsonable_array_values(array: np.ndarray) -> JsonValue: - if array.ndim == 0: - return _jsonable_scalar(array.item()) - return [_jsonable_array_values(np.asarray(value)) for value in array] - - -def _jsonable_metadata_value(value) -> JsonValue: - if isinstance(value, np.generic): - return _jsonable_scalar(value.item()) - if isinstance(value, (str, int, float, bool)) or value is None: - return value - if isinstance(value, Mapping): - return {str(key): _jsonable_metadata_value(item) for key, item in value.items()} - if isinstance(value, list | tuple): - return tuple(_jsonable_metadata_value(item) for item in value) - return str(value) - - def semantic_roi_identity_from_metadata(metadata: Mapping): """Return a hashable semantic ROI identity from ROI metadata.""" - return _hashable_jsonable_metadata_value(metadata) + return _hashable(to_jsonable(metadata)) -def _hashable_jsonable_metadata_value(value): - jsonable = _jsonable_metadata_value(value) +def _hashable(jsonable: JsonValue): + """Freeze an encoded value: mappings become sorted pairs, lists tuples.""" if isinstance(jsonable, Mapping): - return tuple( - sorted( - ( - str(key), - _hashable_jsonable_metadata_value(item), - ) - for key, item in jsonable.items() - ) - ) - if isinstance(jsonable, list | tuple): - return tuple(_hashable_jsonable_metadata_value(item) for item in jsonable) + return tuple(sorted((key, _hashable(item)) for key, item in jsonable.items())) + if isinstance(jsonable, list): + return tuple(_hashable(item) for item in jsonable) return jsonable - - -def _jsonable_scalar(value: np.generic | JsonScalar) -> JsonLike: - if isinstance(value, np.generic): - value = value.item() - if isinstance(value, (str, int, float, bool)) or value is None: - return value - return str(value) diff --git a/openhcs/napari_roi_manager/_dataclasses.py b/openhcs/napari_roi_manager/_dataclasses.py index 46dde4039b..c5fefff3f1 100644 --- a/openhcs/napari_roi_manager/_dataclasses.py +++ b/openhcs/napari_roi_manager/_dataclasses.py @@ -6,6 +6,9 @@ import numpy as np from napari.layers.shapes._shapes_constants import ShapeType from numpy.typing import NDArray +from python_introspect import to_jsonable + +import openhcs.core.numpy_jsonable # noqa: F401 (numpy values in to_jsonable) @dataclass(frozen=True) @@ -44,7 +47,7 @@ def to_json_dict(self) -> dict[str, Any]: out["names"] = self.names if self.features is not None: out["features"] = { - column: [_json_value(value) for value in values] + column: to_jsonable(list(values)) for column, values in self.features.items() } return out @@ -87,19 +90,3 @@ def __post_init__(self): if not isinstance(self.shape_type, ShapeType): raise TypeError("ROI shape type must be a Napari ShapeType member") - -def _json_value(value: Any) -> Any: - """Project native feature values into JSON without stringifying semantics.""" - if value is None or isinstance(value, (str, bool, int, float)): - return value - if isinstance(value, np.generic): - return _json_value(value.item()) - if isinstance(value, np.ndarray): - return [_json_value(item) for item in value.tolist()] - if isinstance(value, (list, tuple)): - return [_json_value(item) for item in value] - if isinstance(value, dict): - return {str(key): _json_value(item) for key, item in value.items()} - raise TypeError( - f"ROI feature value of type {type(value).__name__!r} is not JSON serializable" - ) From 9366929571331fbb0ce8985b9d3154e49eb517ce Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:30:20 -0400 Subject: [PATCH 03/36] E1: hosted MCP authentication modes are a family Co-Authored-By: Claude Opus 5.5 --- openhcs/mcp/http.py | 4 +- openhcs/mcp/http_auth.py | 105 ++++++++++++++------- scripts/build_hosted_mcp_plugin.py | 27 ++---- tests/unit/agent/test_mcp_http_auth.py | 9 +- tests/unit/test_build_hosted_mcp_plugin.py | 11 +-- 5 files changed, 88 insertions(+), 68 deletions(-) diff --git a/openhcs/mcp/http.py b/openhcs/mcp/http.py index d00b7d7d6b..0b09fa3ccd 100644 --- a/openhcs/mcp/http.py +++ b/openhcs/mcp/http.py @@ -39,7 +39,7 @@ def observe( "schema_version": "openhcs.mcp.audit.v2", "event": "capability_invocation", "timestamp_unix": time.time(), - "authentication_mode": settings.authentication_mode.value, + "authentication_mode": settings.authentication_mode.name, "capability": capability.name, "transport": CapabilityTransport.HOSTED_STREAMABLE_HTTP.value, "outcome": outcome.value, @@ -163,7 +163,7 @@ async def healthz(_request): { "status": "ok", "transport": CapabilityTransport.HOSTED_STREAMABLE_HTTP.value, - "authentication_mode": settings.authentication_mode.value, + "authentication_mode": settings.authentication_mode.name, } ) diff --git a/openhcs/mcp/http_auth.py b/openhcs/mcp/http_auth.py index a88c05d0c6..6d79cb5da1 100644 --- a/openhcs/mcp/http_auth.py +++ b/openhcs/mcp/http_auth.py @@ -5,57 +5,96 @@ import hmac import os import time +from abc import ABC, abstractmethod from dataclasses import dataclass -from enum import Enum -from typing import Any, Callable +from typing import Any, Callable, ClassVar from urllib.parse import urlparse +from metaclass_registry import AutoRegisterMeta + class McpHttpConfigurationError(ValueError): """Raised when hosted MCP security configuration is incomplete or unsafe.""" -class McpHttpAuthenticationMode(str, Enum): - """Authentication contract for one hosted MCP deployment.""" +class McpHttpAuthenticationMode(ABC, metaclass=AutoRegisterMeta): + """Authentication contract for one hosted MCP deployment. - PUBLIC_READ_ONLY = "public_read_only" - OAUTH_INTROSPECTION = "oauth_introspection" + Each mode is a subclass; ``name`` is its spelling in + ``OPENHCS_MCP_HTTP_AUTH_MODE`` and in the hosted plugin packet. + """ - @property - def requires_oauth(self) -> bool: - """Return whether this mode requires the nested OAuth policy.""" - return self is McpHttpAuthenticationMode.OAUTH_INTROSPECTION + __registry_key__ = "name" + __skip_if_no_key__ = True - @property - def access_qualifier(self) -> str: - """Return auth-specific copy without restating capability access.""" - if self.requires_oauth: - return "authenticated, subject-isolated" - return "public" + name: ClassVar[str | None] = None + requires_oauth: ClassVar[bool] + """This mode requires the nested OAuth policy.""" + access_qualifier: ClassVar[str] + """Auth-specific copy without restating capability access.""" + @classmethod + @abstractmethod def tool_security_schemes( - self, + cls, required_scopes: tuple[str, ...], ) -> tuple[dict[str, object], ...]: """Project this mode into MCP tool security-scheme metadata.""" - if self.requires_oauth: - return ( - { - "type": "oauth2", - "scopes": list(required_scopes), - }, + + @staticmethod + def named(name: str) -> type["McpHttpAuthenticationMode"]: + """Return the mode spelled ``name``.""" + mode = McpHttpAuthenticationMode.__registry__.get(name) + if mode is None: + raise ValueError( + "Hosted MCP authentication mode must be one of: " + + ", ".join(McpHttpAuthenticationMode.__registry__) ) - return ({"type": "noauth"},) + return mode - @classmethod + @staticmethod def from_oauth_policy( - cls, oauth: "McpHttpOAuthSettings | None", - ) -> "McpHttpAuthenticationMode": + ) -> type["McpHttpAuthenticationMode"]: """Derive the mode from the materialized variant policy.""" if oauth is None: - return cls.PUBLIC_READ_ONLY - return cls.OAUTH_INTROSPECTION + return PublicReadOnlyAccess + return OAuthIntrospectionAccess + + +class PublicReadOnlyAccess(McpHttpAuthenticationMode): + """Anyone may call the read-only tools.""" + + name = "public_read_only" + requires_oauth = False + access_qualifier = "public" + + @classmethod + def tool_security_schemes( + cls, + required_scopes: tuple[str, ...], + ) -> tuple[dict[str, object], ...]: + return ({"type": "noauth"},) + + +class OAuthIntrospectionAccess(McpHttpAuthenticationMode): + """Callers present OAuth tokens checked by introspection.""" + + name = "oauth_introspection" + requires_oauth = True + access_qualifier = "authenticated, subject-isolated" + + @classmethod + def tool_security_schemes( + cls, + required_scopes: tuple[str, ...], + ) -> tuple[dict[str, object], ...]: + return ( + { + "type": "oauth2", + "scopes": list(required_scopes), + }, + ) def _required_environment(name: str) -> str: @@ -202,7 +241,7 @@ def __post_init__(self) -> None: raise McpHttpConfigurationError("bind_port must be between 1 and 65535.") @property - def authentication_mode(self) -> McpHttpAuthenticationMode: + def authentication_mode(self) -> type[McpHttpAuthenticationMode]: return McpHttpAuthenticationMode.from_oauth_policy(self.oauth) @property @@ -236,13 +275,13 @@ def from_environment(cls) -> "McpHttpResourceServerSettings": ) allowed_hosts = configured_hosts or ((public_host,) if public_host else ()) try: - authentication_mode = McpHttpAuthenticationMode( + authentication_mode = McpHttpAuthenticationMode.named( _required_environment("OPENHCS_MCP_HTTP_AUTH_MODE") ) except ValueError as exc: - valid_modes = ", ".join(mode.value for mode in McpHttpAuthenticationMode) raise McpHttpConfigurationError( - "OPENHCS_MCP_HTTP_AUTH_MODE must be one of: " + valid_modes + "OPENHCS_MCP_HTTP_AUTH_MODE must be one of: " + + ", ".join(McpHttpAuthenticationMode.__registry__) ) from exc oauth = None if authentication_mode.requires_oauth: diff --git a/scripts/build_hosted_mcp_plugin.py b/scripts/build_hosted_mcp_plugin.py index 21fe7fdb62..8a151406de 100644 --- a/scripts/build_hosted_mcp_plugin.py +++ b/scripts/build_hosted_mcp_plugin.py @@ -239,19 +239,6 @@ def normalize_registered_app_id(app_id: str) -> str: return value -def normalize_authentication_mode( - authentication_mode: str | McpHttpAuthenticationMode, -) -> McpHttpAuthenticationMode: - """Resolve the package mode through the hosted-server mode authority.""" - try: - return McpHttpAuthenticationMode(authentication_mode) - except ValueError as exc: - valid_modes = ", ".join(mode.value for mode in McpHttpAuthenticationMode) - raise ValueError( - f"Hosted MCP authentication mode must be one of: {valid_modes}" - ) from exc - - def validate_plugin_relative_path(path: str, *, label: str) -> str: """Validate one manifest path relative to the plugin root.""" value = _required_text(path, label=label) @@ -298,7 +285,7 @@ def _source_release_metadata(source_manifest: Path) -> dict[str, Any]: def _plugin_manifest( *, release_metadata: Mapping[str, Any], - authentication_mode: McpHttpAuthenticationMode, + authentication_mode: type[McpHttpAuthenticationMode], website_url: str, privacy_policy_url: str, terms_of_service_url: str, @@ -348,7 +335,7 @@ def _submission_packet( manifest: Mapping[str, Any], remote_mcp_url: str, registered_app_id: str, - authentication_mode: McpHttpAuthenticationMode, + authentication_mode: type[McpHttpAuthenticationMode], website_url: str, support_url: str, privacy_policy_url: str, @@ -416,11 +403,11 @@ def _submission_packet( def _submission_authentication( - authentication_mode: McpHttpAuthenticationMode, + authentication_mode: type[McpHttpAuthenticationMode], ) -> dict[str, object]: """Project the hosted auth owner into OpenAI reviewer metadata once.""" return { - "mode": authentication_mode.value, + "mode": authentication_mode.name, "type": "oauth" if authentication_mode.requires_oauth else "none", "user_login_required": authentication_mode.requires_oauth, } @@ -430,7 +417,7 @@ def build_hosted_plugin( *, remote_mcp_url: str, registered_app_id: str, - authentication_mode: str | McpHttpAuthenticationMode, + authentication_mode: str, website_url: str, support_url: str, privacy_policy_url: str, @@ -442,7 +429,7 @@ def build_hosted_plugin( """Project release metadata into a registered web-plugin review bundle.""" endpoint = validate_remote_mcp_url(remote_mcp_url) app_id = normalize_registered_app_id(registered_app_id) - auth_mode = normalize_authentication_mode(authentication_mode) + auth_mode = McpHttpAuthenticationMode.named(authentication_mode) website = validate_public_https_url(website_url, label="Public website URL") support = validate_public_https_url(support_url, label="Public support URL") privacy = validate_public_https_url( @@ -517,7 +504,7 @@ def main(argv: Sequence[str] | None = None) -> int: ) parser.add_argument( "--authentication-mode", - choices=[mode.value for mode in McpHttpAuthenticationMode], + choices=list(McpHttpAuthenticationMode.__registry__), default=McpHttpAuthenticationMode.PUBLIC_READ_ONLY.value, help=( "Hosted server authentication contract. The initial public plugin " diff --git a/tests/unit/agent/test_mcp_http_auth.py b/tests/unit/agent/test_mcp_http_auth.py index c9db6eab4a..d79c5211af 100644 --- a/tests/unit/agent/test_mcp_http_auth.py +++ b/tests/unit/agent/test_mcp_http_auth.py @@ -7,7 +7,8 @@ from openhcs.mcp.http_auth import ( IntrospectionTokenVerifier, - McpHttpAuthenticationMode, + OAuthIntrospectionAccess, + PublicReadOnlyAccess, McpHttpConfigurationError, McpHttpOAuthSettings, McpHttpResourceServerSettings, @@ -84,7 +85,7 @@ def test_http_settings_allow_explicit_loopback_development(): def test_public_read_only_settings_need_no_oauth_credentials(): settings = _settings(oauth=None) - assert settings.authentication_mode is McpHttpAuthenticationMode.PUBLIC_READ_ONLY + assert settings.authentication_mode is PublicReadOnlyAccess assert settings.authentication_mode.requires_oauth is False assert settings.authentication_mode.access_qualifier == "public" assert settings.required_scopes == () @@ -97,7 +98,7 @@ def test_public_read_only_settings_need_no_oauth_credentials(): def test_oauth_settings_project_auth_facets_from_the_mode_owner(): settings = _settings() - assert settings.authentication_mode is McpHttpAuthenticationMode.OAUTH_INTROSPECTION + assert settings.authentication_mode is OAuthIntrospectionAccess assert settings.authentication_mode.requires_oauth is True assert ( settings.authentication_mode.access_qualifier @@ -143,7 +144,7 @@ def test_environment_builds_public_read_only_settings(monkeypatch, tmp_path): settings = McpHttpResourceServerSettings.from_environment() assert settings.oauth is None - assert settings.authentication_mode is McpHttpAuthenticationMode.PUBLIC_READ_ONLY + assert settings.authentication_mode is PublicReadOnlyAccess assert settings.openai_domain_challenge_token == "openai-domain-token" diff --git a/tests/unit/test_build_hosted_mcp_plugin.py b/tests/unit/test_build_hosted_mcp_plugin.py index f8fafaa71c..f690bacdd7 100644 --- a/tests/unit/test_build_hosted_mcp_plugin.py +++ b/tests/unit/test_build_hosted_mcp_plugin.py @@ -7,7 +7,6 @@ from scripts import build_hosted_mcp_plugin as plugin_builder from scripts.build_hosted_mcp_plugin import ( build_hosted_plugin, - normalize_authentication_mode, normalize_registered_app_id, validate_plugin_relative_path, validate_public_https_url, @@ -203,15 +202,9 @@ def reject_hosted_surface(): _build(tmp_path) -def test_authentication_mode_uses_hosted_server_authority(): - assert normalize_authentication_mode("public_read_only").value == ( - "public_read_only" - ) - assert normalize_authentication_mode("oauth_introspection").value == ( - "oauth_introspection" - ) +def test_unknown_authentication_mode_is_rejected(tmp_path): with pytest.raises(ValueError, match="authentication mode"): - normalize_authentication_mode("oauth") + _build(tmp_path, authentication_mode="oauth") @pytest.mark.parametrize( From 4af05328d07ac6dc00fc50190e35b68bd74d1d66 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:31:15 -0400 Subject: [PATCH 04/36] E1: MCP process recovery reasons are a family that answers restart_required Co-Authored-By: Claude Opus 5.5 --- openhcs/mcp/lifecycle.py | 55 +++++++++++++++++++------- openhcs/mcp/server.py | 4 +- tests/unit/agent/test_mcp_lifecycle.py | 14 ++++--- 3 files changed, 50 insertions(+), 23 deletions(-) diff --git a/openhcs/mcp/lifecycle.py b/openhcs/mcp/lifecycle.py index 61466a5f0a..966213e2bc 100644 --- a/openhcs/mcp/lifecycle.py +++ b/openhcs/mcp/lifecycle.py @@ -9,6 +9,7 @@ from __future__ import annotations +from abc import ABC from collections.abc import Mapping, Sequence from dataclasses import dataclass import errno @@ -17,8 +18,11 @@ import json import os from pathlib import Path +from typing import ClassVar import stat +from metaclass_registry import AutoRegisterMeta + from openhcs.mcp.bootstrap import ( MCP_INSTALLATION_POINTER_ENVIRONMENT_VARIABLE, MCP_STABLE_LAUNCH_COMMAND_ENVIRONMENT_VARIABLE, @@ -51,28 +55,49 @@ class McpLifecycleConfigurationError(ValueError): """The launcher supplied an invalid local MCP lifecycle contract.""" -class McpProcessRecoveryReason(str, Enum): - """Why the running MCP process must be replaced.""" +class McpProcessRecoveryReason(ABC, metaclass=AutoRegisterMeta): + """Why the running MCP process must be replaced, or that it need not be. - CURRENT = "current" - SOURCE_CHANGED = "source_changed" - INSTALLATION_CHANGED = "installation_changed" - SOURCE_AND_INSTALLATION_CHANGED = "source_and_installation_changed" + Each reason is a subclass; ``name`` is its spelling in MCP results. + """ - @classmethod + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + restart_required: ClassVar[bool] = True + """The running process no longer matches its source or installation.""" + + @staticmethod def from_drift( - cls, *, source_changed: bool, installation_changed: bool, - ) -> "McpProcessRecoveryReason": + ) -> type["McpProcessRecoveryReason"]: if source_changed and installation_changed: - return cls.SOURCE_AND_INSTALLATION_CHANGED + return SourceAndInstallationChanged if source_changed: - return cls.SOURCE_CHANGED + return SourceChanged if installation_changed: - return cls.INSTALLATION_CHANGED - return cls.CURRENT + return InstallationChanged + return ProcessCurrent + + +class ProcessCurrent(McpProcessRecoveryReason): + name = "current" + restart_required = False + + +class SourceChanged(McpProcessRecoveryReason): + name = "source_changed" + + +class InstallationChanged(McpProcessRecoveryReason): + name = "installation_changed" + + +class SourceAndInstallationChanged(McpProcessRecoveryReason): + name = "source_and_installation_changed" class McpReconnectOwner(str, Enum): @@ -148,7 +173,7 @@ def from_path(cls, pointer_path: Path) -> "McpInstallationPointerSnapshot": class McpProcessRecoveryStatus: """One current recovery decision for the running local MCP process.""" - reason: McpProcessRecoveryReason + reason: type[McpProcessRecoveryReason] installation_pointer_path: str | None installation_pointer_changed_since_import: bool installation_pointer_available: bool | None @@ -276,7 +301,7 @@ def recovery_status( source_changed=source_changed, installation_changed=installation_changed, ) - restart_required = reason is not McpProcessRecoveryReason.CURRENT + restart_required = reason.restart_required command_is_stable = self.stable_launch_command is not None restart_command = ( self.stable_launch_command diff --git a/openhcs/mcp/server.py b/openhcs/mcp/server.py index 36c3850441..b133801f50 100644 --- a/openhcs/mcp/server.py +++ b/openhcs/mcp/server.py @@ -476,7 +476,7 @@ def _mcp_server_health() -> McpServerHealthResult: server_current_mtime_ns=current_source_mtime_ns, server_source_changed_since_import=bool(stale_source_paths), stale_source_paths=tuple(str(source_path) for source_path in stale_source_paths), - recovery_reason=recovery.reason.value, + recovery_reason=recovery.reason.name, installation_pointer_path=recovery.installation_pointer_path, installation_pointer_changed_since_import=( recovery.installation_pointer_changed_since_import @@ -907,7 +907,7 @@ def _mcp_server_stale_error(tool_name: str) -> JsonValue: stale_source_paths=tuple( str(source_path) for source_path in stale_source_paths ), - recovery_reason=recovery.reason.value, + recovery_reason=recovery.reason.name, installation_pointer_path=recovery.installation_pointer_path, installation_pointer_changed_since_import=( recovery.installation_pointer_changed_since_import diff --git a/tests/unit/agent/test_mcp_lifecycle.py b/tests/unit/agent/test_mcp_lifecycle.py index 437a6142ac..74d240a597 100644 --- a/tests/unit/agent/test_mcp_lifecycle.py +++ b/tests/unit/agent/test_mcp_lifecycle.py @@ -18,7 +18,9 @@ McpInstallationPointerSnapshot, McpLifecycleConfigurationError, McpProcessLifecycle, - McpProcessRecoveryReason, + InstallationChanged, + ProcessCurrent, + SourceChanged, ) import openhcs.mcp.server as server @@ -44,13 +46,13 @@ def test_source_change_requires_client_owned_reconnect(): current = lifecycle.recovery_status(source_changed=False) changed = lifecycle.recovery_status(source_changed=True) - assert current.reason is McpProcessRecoveryReason.CURRENT + assert current.reason is ProcessCurrent assert current.restart_required is False assert current.restart_command == () assert current.reconnect_owner is None assert current.retry_after_reconnect is False - assert changed.reason is McpProcessRecoveryReason.SOURCE_CHANGED + assert changed.reason is SourceChanged assert changed.restart_command == ( sys.executable, "-m", @@ -79,7 +81,7 @@ def test_installer_generation_change_uses_stable_launch_command(tmp_path): pointer.write_text("launch environment-b\n", encoding="utf-8") changed = lifecycle.recovery_status(source_changed=False) - assert changed.reason is McpProcessRecoveryReason.INSTALLATION_CHANGED + assert changed.reason is InstallationChanged assert changed.installation_pointer_path == str(pointer) assert changed.installation_pointer_changed_since_import is True assert changed.installation_pointer_available is True @@ -108,7 +110,7 @@ def test_deleted_installation_pointer_fails_closed_without_false_auto_recovery( pointer.unlink() changed = lifecycle.recovery_status(source_changed=False) - assert changed.reason is McpProcessRecoveryReason.INSTALLATION_CHANGED + assert changed.reason is InstallationChanged assert changed.installation_pointer_changed_since_import is True assert changed.installation_pointer_available is False assert changed.restart_command == (str(stable_launcher), "mcp") @@ -206,7 +208,7 @@ def test_deleted_environment_source_remains_health_reportable(monkeypatch, tmp_p assert stale_paths == (old_environment_source,) assert server._mcp_server_current_source_mtime_ns() is None - assert recovery.reason is McpProcessRecoveryReason.SOURCE_CHANGED + assert recovery.reason is SourceChanged assert recovery.restart_required is True assert recovery.retry_after_reconnect is True From 15208b062a46cba7811a12067062da8e7a0ea43d Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:31:35 -0400 Subject: [PATCH 05/36] E1: GUI shell kinds own their behaviour (startup readiness, startup events, ROI columns); platform and log-level spellings stay at their boundary Co-Authored-By: Claude Opus 5.5 --- openhcs/gui_startup.py | 95 +++++++------- .../widgets/_roi_manager.py | 118 +++++++++++------- openhcs/pyqt_gui/app.py | 104 +++++++++------ openhcs/pyqt_gui/config.py | 10 +- openhcs/pyqt_gui/launch.py | 35 ++---- openhcs/pyqt_gui/services/logging_config.py | 6 +- .../test_napari_roi_manager_integration.py | 13 +- 7 files changed, 208 insertions(+), 173 deletions(-) diff --git a/openhcs/gui_startup.py b/openhcs/gui_startup.py index 604668c452..e0b4d7cb3d 100644 --- a/openhcs/gui_startup.py +++ b/openhcs/gui_startup.py @@ -16,12 +16,11 @@ import traceback from abc import ABC, abstractmethod from collections.abc import Callable -from dataclasses import dataclass -from enum import Enum from importlib import import_module from threading import Event, Lock, Thread -from typing import IO +from typing import IO, ClassVar +from metaclass_registry import AutoRegisterMeta from zmqruntime.process_launch import BackgroundProcessLaunchPolicy _STARTUP_WINDOW_CHILD_ARGUMENT = "--startup-window-child" @@ -62,55 +61,53 @@ def present_parent_stream_closed(self) -> None: """Close a non-failure window after its parent stream ends.""" -@dataclass(frozen=True, slots=True) -class _StartupEventSemantics: - """Wire identity and leaf presentation owned by one event kind.""" +class _StartupEvent(ABC, metaclass=AutoRegisterMeta): + """A parent-to-window startup event; each kind presents itself. - wire_kind: str - present: Callable[[_StartupEventPresentationABC, dict[str, object]], None] + ``wire_kind`` is the kind's spelling in the JSON-lines stream. + """ + __registry_key__ = "wire_kind" + __skip_if_no_key__ = True -class _StartupEventKind(Enum): - """Closed parent-to-window events carrying their presentation behavior.""" - - OUTPUT = _StartupEventSemantics( - "output", - lambda presenter, event: presenter.present_output( - str(event.get("message", "")) - ), - ) - FAILURE = _StartupEventSemantics( - "failure", - lambda presenter, event: presenter.present_failure( - str(event.get("message", "")), - str(event.get("detail", "")), - ), - ) - READY = _StartupEventSemantics( - "ready", - lambda presenter, _event: presenter.present_ready(), - ) - - @property - def wire_kind(self) -> str: - return self.value.wire_kind + wire_kind: ClassVar[str | None] = None + @staticmethod + @abstractmethod def present( - self, - presenter: _StartupEventPresentationABC, - event: dict[str, object], + presenter: _StartupEventPresentationABC, event: dict[str, object] ) -> None: - self.value.present(presenter, event) + """Present one decoded event of this kind.""" - @classmethod - def from_wire_kind(cls, value: object) -> _StartupEventKind | None: - return next((member for member in cls if member.wire_kind == value), None) +class _OutputStartupEvent(_StartupEvent): + wire_kind = "output" + + @staticmethod + def present(presenter, event) -> None: + presenter.present_output(str(event.get("message", ""))) + + +class _FailureStartupEvent(_StartupEvent): + wire_kind = "failure" + + @staticmethod + def present(presenter, event) -> None: + presenter.present_failure( + str(event.get("message", "")), str(event.get("detail", "")) + ) + + +class _ReadyStartupEvent(_StartupEvent): + wire_kind = "ready" + + @staticmethod + def present(presenter, _event) -> None: + presenter.present_ready() -class _StartupWindowSignal(str, Enum): - """Closed child-to-parent startup-window signals.""" - PAINTED = "painted" +_PAINTED_EVENT_KIND = "painted" +"""Wire ``kind`` of the child's one startup-window signal: its first paint.""" def _present_startup_event( @@ -119,7 +116,7 @@ def _present_startup_event( ) -> bool: """Present one recognized wire event through its declaration owner.""" - kind = _StartupEventKind.from_wire_kind(event.get("kind")) + kind = _StartupEvent.__registry__.get(event.get("kind")) if kind is None: return False kind.present(presenter, event) @@ -180,7 +177,7 @@ def active(self) -> bool: def output(self, message: str, *, stream_name: str) -> None: """Mirror one real terminal line into the startup window.""" self._send( - _StartupEventKind.OUTPUT, + _OutputStartupEvent, message=message, stream_name=stream_name, ) @@ -197,14 +194,14 @@ def ready(self) -> None: # native Windows surface until the actual main-window readiness event. _signal_native_startup_handoff() self._run_finish_callbacks() - self._send(_StartupEventKind.READY) + self._send(_ReadyStartupEvent) self._finish_stream() def fail(self, message: str, detail: str = "") -> None: self._failed = True self._run_finish_callbacks() self._send( - _StartupEventKind.FAILURE, + _FailureStartupEvent, message=message, detail=detail, ) @@ -238,7 +235,7 @@ def _read_first_paint() -> None: event = json.loads(line) except (json.JSONDecodeError, TypeError): continue - if event.get("kind") == _StartupWindowSignal.PAINTED.value: + if event.get("kind") == _PAINTED_EVENT_KIND: painted.set() _signal_native_startup_handoff() return @@ -259,7 +256,7 @@ def _read_first_paint() -> None: observation_complete.wait(timeout=_STARTUP_FIRST_PAINT_TIMEOUT_SECONDS) return painted.is_set() - def _send(self, kind: _StartupEventKind, **payload: str) -> None: + def _send(self, kind: type[_StartupEvent], **payload: str) -> None: with self._lock: if self._closed or self._stream is None: return @@ -596,7 +593,7 @@ def paintEvent(self, event) -> None: return try: sys.stdout.write( - json.dumps({"kind": _StartupWindowSignal.PAINTED.value}) + "\n" + json.dumps({"kind": _PAINTED_EVENT_KIND}) + "\n" ) sys.stdout.flush() except (BrokenPipeError, OSError, ValueError): diff --git a/openhcs/napari_roi_manager/widgets/_roi_manager.py b/openhcs/napari_roi_manager/widgets/_roi_manager.py index 4f37ad8478..a33a2e2f33 100644 --- a/openhcs/napari_roi_manager/widgets/_roi_manager.py +++ b/openhcs/napari_roi_manager/widgets/_roi_manager.py @@ -2,9 +2,10 @@ import json from collections.abc import Iterable -from enum import IntEnum +from abc import ABC, abstractmethod from functools import partial from pathlib import Path +from typing import ClassVar import napari import numpy as np @@ -14,6 +15,7 @@ from napari.utils._proxies import PublicOnlyProxy from qtpy import QtCore from qtpy import QtWidgets as QtW +from metaclass_registry import AutoRegisterMeta from roifile import roiread, roiwrite from openhcs.napari_roi_manager._dataclasses import RoiData, RoiTuple @@ -87,17 +89,64 @@ def set_layer_available(self, available: bool) -> None: widget.setEnabled(available) -class RoiTableColumn(IntEnum): - """Columns projected directly from one native Napari Shapes layer.""" +class RoiTableColumn(ABC, metaclass=AutoRegisterMeta): + """One column of the ROI table, projected from a native Napari Shapes layer. - def __new__(cls, value: int, header: str): - member = int.__new__(cls, value) - member._value_ = value - member.header = header - return member + ``index`` is the column's Qt section; each column reads its own cells. + """ - NAME = (0, "name") - SHAPE_TYPE = (1, "type") + __registry_key__ = "index" + __skip_if_no_key__ = True + + index: ClassVar[int | None] = None + header: ClassVar[str] + editable: ClassVar[bool] = False + + @classmethod + def at(cls, section: int) -> type["RoiTableColumn"]: + return cls.__registry__[section] + + @staticmethod + @abstractmethod + def values(layer: Shapes) -> list[str]: + """Every cell of this column for ``layer``.""" + + @classmethod + def cell(cls, layer: Shapes, row: int) -> str: + return cls.values(layer)[row] + + +class RoiNameColumn(RoiTableColumn): + index = 0 + header = "name" + editable = True + + @staticmethod + def values(layer: Shapes) -> list[str]: + features = layer.features + if RoiNameColumn.header in features.columns: + return [str(value) for value in features[RoiNameColumn.header].tolist()] + return [f"ROI-{index:>04}" for index in range(len(layer.data))] + + @classmethod + def cell(cls, layer: Shapes, row: int) -> str: + features = layer.features + if cls.header in features.columns: + return str(features[cls.header].iat[row]) + return f"ROI-{row:>04}" + + +class RoiShapeTypeColumn(RoiTableColumn): + index = 1 + header = "type" + + @staticmethod + def values(layer: Shapes) -> list[str]: + return [str(value) for value in layer.shape_type] + + @classmethod + def cell(cls, layer: Shapes, row: int) -> str: + return str(layer.shape_type[row]) class QRoiTableModel(QtCore.QAbstractTableModel): @@ -129,7 +178,7 @@ def rowCount(self, parent=QtCore.QModelIndex()) -> int: return len(self._layer.data) def columnCount(self, parent=QtCore.QModelIndex()) -> int: - return 0 if parent.isValid() else len(RoiTableColumn) + return 0 if parent.isValid() else len(RoiTableColumn.__registry__) def data(self, index, role=QtCore.Qt.ItemDataRole.DisplayRole): if ( @@ -142,10 +191,7 @@ def data(self, index, role=QtCore.Qt.ItemDataRole.DisplayRole): ) ): return None - column = RoiTableColumn(index.column()) - if column is RoiTableColumn.NAME: - return self._name(index.row()) - return str(self._layer.shape_type[index.row()]) + return RoiTableColumn.at(index.column()).cell(self._layer, index.row()) def headerData( self, @@ -157,12 +203,12 @@ def headerData( orientation == QtCore.Qt.Orientation.Horizontal and role == QtCore.Qt.ItemDataRole.DisplayRole ): - return RoiTableColumn(section).header + return RoiTableColumn.at(section).header return super().headerData(section, orientation, role) def flags(self, index): flags = super().flags(index) - if index.isValid() and RoiTableColumn(index.column()) is RoiTableColumn.NAME: + if index.isValid() and RoiTableColumn.at(index.column()).editable: flags |= QtCore.Qt.ItemFlag.ItemIsEditable return flags @@ -171,38 +217,22 @@ def setData(self, index, value, role=QtCore.Qt.ItemDataRole.EditRole) -> bool: role != QtCore.Qt.ItemDataRole.EditRole or not index.isValid() or self._layer is None - or RoiTableColumn(index.column()) is not RoiTableColumn.NAME + or not RoiTableColumn.at(index.column()).editable ): return False - names = self.column_values(RoiTableColumn.NAME) + names = RoiNameColumn.values(self._layer) names[index.row()] = str(value) features = self._layer.features.copy() - features[RoiTableColumn.NAME.header] = names + features[RoiNameColumn.header] = names self._layer.features = features return True - def column_values(self, column: RoiTableColumn) -> list[str]: + def column_values(self, column: type[RoiTableColumn]) -> list[str]: """Project one complete column only for explicit bulk callers.""" if self._layer is None: return [] - if column is RoiTableColumn.NAME: - features = self._layer.features - if RoiTableColumn.NAME.header in features.columns: - return [ - str(value) - for value in features[RoiTableColumn.NAME.header].tolist() - ] - return [f"ROI-{index:>04}" for index in range(self.rowCount())] - return [str(value) for value in self._layer.shape_type] - - def _name(self, row: int) -> str: - if self._layer is None: - raise RuntimeError("ROI table model has no native Shapes layer.") - features = self._layer.features - if RoiTableColumn.NAME.header in features.columns: - return str(features[RoiTableColumn.NAME.header].iat[row]) - return f"ROI-{row:>04}" + return column.values(self._layer) class QRoiListWidget(QtW.QTableView): @@ -215,14 +245,14 @@ def __init__(self, parent=None): horizontal_header = self.horizontalHeader() horizontal_header.setFixedHeight(18) horizontal_header.setSectionResizeMode( - RoiTableColumn.NAME, + RoiNameColumn.index, QtW.QHeaderView.ResizeMode.Stretch, ) horizontal_header.setSectionResizeMode( - RoiTableColumn.SHAPE_TYPE, + RoiShapeTypeColumn.index, QtW.QHeaderView.ResizeMode.Interactive, ) - horizontal_header.resizeSection(RoiTableColumn.SHAPE_TYPE, 96) + horizontal_header.resizeSection(RoiShapeTypeColumn.index, 96) self.verticalHeader().setSectionResizeMode(QtW.QHeaderView.ResizeMode.Fixed) self.verticalHeader().setDefaultSectionSize(18) self.setSelectionBehavior(QtW.QAbstractItemView.SelectionBehavior.SelectRows) @@ -259,10 +289,10 @@ def select_rows(self, rows: Iterable[int]) -> None: span_stop = row continue selection.select( - self.model().index(span_start, RoiTableColumn.NAME), + self.model().index(span_start, RoiNameColumn.index), self.model().index( span_stop, - RoiTableColumn.SHAPE_TYPE, + RoiShapeTypeColumn.index, ), ) if row is not None: @@ -270,7 +300,7 @@ def select_rows(self, rows: Iterable[int]) -> None: selection_model.select(selection, flags) if valid_rows: self.scrollTo( - self.model().index(valid_rows[0], RoiTableColumn.NAME), + self.model().index(valid_rows[0], RoiNameColumn.index), QtW.QAbstractItemView.ScrollHint.EnsureVisible, ) finally: diff --git a/openhcs/pyqt_gui/app.py b/openhcs/pyqt_gui/app.py index da386d2271..af3fb15cbe 100644 --- a/openhcs/pyqt_gui/app.py +++ b/openhcs/pyqt_gui/app.py @@ -8,7 +8,6 @@ import logging import sys from collections.abc import Callable -from enum import Enum from typing import TYPE_CHECKING from objectstate import spawn_thread_with_context @@ -30,13 +29,64 @@ logger = logging.getLogger(__name__) -class MainWindowStartupReadinessState(Enum): - """Closed lifecycle for the initialized main-window paint boundary.""" +class StartupReadinessState: + """One state of the main-window readiness boundary; each owns its transitions. + + The base answers every event by staying put. + """ + + @staticmethod + def deferred_initialization_complete(readiness: "MainWindowStartupReadiness") -> None: + return None + + @staticmethod + def painted(readiness: "MainWindowStartupReadiness") -> None: + return None + + @staticmethod + def report_painted_ready(readiness: "MainWindowStartupReadiness") -> None: + return None + + @staticmethod + def fail(readiness: "MainWindowStartupReadiness", error: BaseException) -> None: + readiness._finish() + if readiness._on_failure is not None: + readiness._on_failure(error) + + +class WaitingForDeferredInitialization(StartupReadinessState): + @staticmethod + def deferred_initialization_complete(readiness: "MainWindowStartupReadiness") -> None: + readiness._state = WaitingForInitializedPaint + readiness._main_window.update() + + +class WaitingForInitializedPaint(StartupReadinessState): + @staticmethod + def painted(readiness: "MainWindowStartupReadiness") -> None: + readiness._state = PaintCompletionQueued + QtCore.QTimer.singleShot(0, readiness._report_painted_ready) + + +class PaintCompletionQueued(StartupReadinessState): + @staticmethod + def report_painted_ready(readiness: "MainWindowStartupReadiness") -> None: + readiness._finish() + logger.info("OpenHCS main window painted and ready") + if readiness._on_ready is None: + return + try: + readiness._on_ready() + except Exception as error: + if readiness._on_failure is None: + raise + readiness._on_failure(error) - WAITING_DEFERRED_INITIALIZATION = "waiting_deferred_initialization" - WAITING_INITIALIZED_PAINT = "waiting_initialized_paint" - PAINT_COMPLETION_QUEUED = "paint_completion_queued" - FINISHED = "finished" + +class StartupReadinessFinished(StartupReadinessState): + @staticmethod + def fail(readiness: "MainWindowStartupReadiness", error: BaseException) -> None: + return None class MainWindowStartupReadiness(QtCore.QObject): @@ -53,56 +103,30 @@ def __init__( self._main_window = main_window self._on_ready = on_ready self._on_failure = on_failure - self._state = MainWindowStartupReadinessState.WAITING_DEFERRED_INITIALIZATION + self._state: type[StartupReadinessState] = WaitingForDeferredInitialization main_window.installEventFilter(self) def deferred_initialization_complete(self) -> None: """Arm readiness for the first paint containing deferred UI state.""" - if ( - self._state - is not MainWindowStartupReadinessState.WAITING_DEFERRED_INITIALIZATION - ): - return - self._state = MainWindowStartupReadinessState.WAITING_INITIALIZED_PAINT - self._main_window.update() + self._state.deferred_initialization_complete(self) def fail(self, error: BaseException) -> None: """Terminate the readiness boundary with its originating failure.""" - if self._state is MainWindowStartupReadinessState.FINISHED: - return - self._finish() - if self._on_failure is not None: - self._on_failure(error) + self._state.fail(self, error) def eventFilter(self, watched, event) -> bool: # noqa: N802 if watched is self._main_window and event.type() == QtCore.QEvent.Type.Close: self._finish() return super().eventFilter(watched, event) - if ( - watched is self._main_window - and self._state is MainWindowStartupReadinessState.WAITING_INITIALIZED_PAINT - and event.type() == QtCore.QEvent.Type.Paint - ): - self._state = MainWindowStartupReadinessState.PAINT_COMPLETION_QUEUED - QtCore.QTimer.singleShot(0, self._report_painted_ready) + if watched is self._main_window and event.type() == QtCore.QEvent.Type.Paint: + self._state.painted(self) return super().eventFilter(watched, event) def _report_painted_ready(self) -> None: - if self._state is not MainWindowStartupReadinessState.PAINT_COMPLETION_QUEUED: - return - self._finish() - logger.info("OpenHCS main window painted and ready") - if self._on_ready is None: - return - try: - self._on_ready() - except Exception as error: - if self._on_failure is None: - raise - self._on_failure(error) + self._state.report_painted_ready(self) def _finish(self) -> None: - self._state = MainWindowStartupReadinessState.FINISHED + self._state = StartupReadinessFinished self._main_window.removeEventFilter(self) self.deleteLater() diff --git a/openhcs/pyqt_gui/config.py b/openhcs/pyqt_gui/config.py index eca20f9cc5..e1dc1ae1c9 100644 --- a/openhcs/pyqt_gui/config.py +++ b/openhcs/pyqt_gui/config.py @@ -64,9 +64,13 @@ class GuiLogLevel(str, Enum): @property def logging_level(self) -> int: - if self is GuiLogLevel.SILENT: - return logging.CRITICAL + 1 - return getattr(logging, self.value) + """The stdlib level; ``SILENT`` is one above ``CRITICAL``.""" + return getattr(logging, self.value, logging.CRITICAL + 1) + + @property + def emits_records(self) -> bool: + """Whether any record reaches a handler at this level.""" + return self.logging_level <= logging.CRITICAL @classmethod def choices(cls) -> tuple[str, ...]: diff --git a/openhcs/pyqt_gui/launch.py b/openhcs/pyqt_gui/launch.py index a27880dfc4..69446103d5 100644 --- a/openhcs/pyqt_gui/launch.py +++ b/openhcs/pyqt_gui/launch.py @@ -9,7 +9,6 @@ from __future__ import annotations import argparse -from enum import Enum import importlib.util import logging import os @@ -41,26 +40,6 @@ def is_wsl() -> bool: return False -class QtPlatformSystem(Enum): - """Closed host-platform axis for Qt platform setup.""" - - MACOS = "Darwin" - LINUX = "Linux" - DEFAULT = "default" - - @classmethod - def from_current(cls) -> "QtPlatformSystem": - current_system = platform.system() - for platform_system in (cls.MACOS, cls.LINUX): - if current_system == platform_system.value: - return platform_system - return cls.DEFAULT - - @property - def uses_default_qt_platform(self) -> bool: - return self is QtPlatformSystem.DEFAULT - - def _setup_macos_qt_platform() -> None: os.environ["QT_QPA_PLATFORM"] = "cocoa" logging.info("macOS detected - setting QT_QPA_PLATFORM=cocoa") @@ -93,10 +72,11 @@ def _setup_linux_qt_platform() -> None: os.environ["QT_X11_NO_MITSHM"] = "1" -QT_PLATFORM_SETUP: dict[QtPlatformSystem, Callable[[], None]] = { - QtPlatformSystem.MACOS: _setup_macos_qt_platform, - QtPlatformSystem.LINUX: _setup_linux_qt_platform, +QT_PLATFORM_SETUP_BY_SYSTEM: dict[str, Callable[[], None]] = { + "Darwin": _setup_macos_qt_platform, + "Linux": _setup_linux_qt_platform, } +"""Qt platform setup keyed by ``platform.system()``'s spelling; others use Qt's default.""" def setup_qt_platform(): @@ -108,13 +88,12 @@ def setup_qt_platform(): ) return - platform_system = QtPlatformSystem.from_current() - if platform_system.uses_default_qt_platform: + setup = QT_PLATFORM_SETUP_BY_SYSTEM.get(platform.system()) + if setup is None: # Windows and other platforms do not need QT_QPA_PLATFORM set. logging.debug(f"Platform {platform.system()} - using default Qt platform") return - - QT_PLATFORM_SETUP[platform_system]() + setup() def setup_logging(config, *, log_level=None, log_file=None): diff --git a/openhcs/pyqt_gui/services/logging_config.py b/openhcs/pyqt_gui/services/logging_config.py index 5291dceda7..f7d18fae55 100644 --- a/openhcs/pyqt_gui/services/logging_config.py +++ b/openhcs/pyqt_gui/services/logging_config.py @@ -46,7 +46,7 @@ def configure_gui_logging( handlers: list[logging.Handler] = [] log_file: Path | None = None - if level is not GuiLogLevel.SILENT: + if level.emits_records: if config.enable_console_logging: console_handler = GuiConsoleHandler(sys.stdout) console_handler.setFormatter(formatter) @@ -75,9 +75,9 @@ def configure_gui_logging( root_logger = logging.getLogger() previous_handlers = tuple(root_logger.handlers) root_logger.handlers = handlers - if level is GuiLogLevel.SILENT: + if not level.emits_records: logging.disable(logging.CRITICAL) - root_logger.setLevel(logging.CRITICAL + 1) + root_logger.setLevel(level.logging_level) else: logging.disable(logging.NOTSET) root_logger.setLevel(level.logging_level) diff --git a/tests/unit/test_napari_roi_manager_integration.py b/tests/unit/test_napari_roi_manager_integration.py index 98bea8e64b..ae5fd467f7 100644 --- a/tests/unit/test_napari_roi_manager_integration.py +++ b/tests/unit/test_napari_roi_manager_integration.py @@ -163,9 +163,9 @@ def bind(_layer) -> None: assert tuple(viewer.layers) == original_layers assert manager._layer is layer assert manager._roilist.rowCount() == 2 - from openhcs.napari_roi_manager.widgets._roi_manager import RoiTableColumn + from openhcs.napari_roi_manager.widgets._roi_manager import RoiNameColumn - assert manager._roilist._roi_model.column_values(RoiTableColumn.NAME) == [ + assert manager._roilist._roi_model.column_values(RoiNameColumn) == [ "first", "second", ] @@ -184,7 +184,7 @@ def bind(_layer) -> None: assert server.require_result_selection_surface().manager is original_manager assert manager._layer is second_layer - assert manager._roilist._roi_model.column_values(RoiTableColumn.NAME) == ["third"] + assert manager._roilist._roi_model.column_values(RoiNameColumn) == ["third"] viewer.close() @@ -280,7 +280,8 @@ def test_roi_manager_virtualizes_thousands_over_one_native_shapes_owner( from openhcs.napari_roi_manager.widgets._roi_manager import ( QRoiManager, - RoiTableColumn, + RoiNameColumn, + RoiShapeTypeColumn, ) member_count = 4_097 @@ -310,11 +311,11 @@ def test_roi_manager_virtualizes_thousands_over_one_native_shapes_owner( assert not isinstance(table, QTableWidget) assert manager.findChildren(QTableWidget) == [] assert ( - table.horizontalHeader().sectionResizeMode(RoiTableColumn.SHAPE_TYPE) + table.horizontalHeader().sectionResizeMode(RoiShapeTypeColumn.index) is QHeaderView.ResizeMode.Interactive ) assert model.rowCount() == member_count - last_name = model.index(member_count - 1, RoiTableColumn.NAME) + last_name = model.index(member_count - 1, RoiNameColumn.index) assert model.data(last_name) == f"ROI-{member_count - 1:04d}" feature_events = [] From d408cd2e4a3a1fe7774360a2d19b650e075153b3 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:32:22 -0400 Subject: [PATCH 06/36] E1: client registration outcomes are a family; the report encodes itself through to_jsonable Co-Authored-By: Claude Opus 5.5 --- openhcs/mcp/client_registration.py | 77 +++++++++++-------- .../agent/test_mcp_client_registration.py | 23 +++--- .../unit/agent/test_mcp_skill_registration.py | 10 +-- 3 files changed, 65 insertions(+), 45 deletions(-) diff --git a/openhcs/mcp/client_registration.py b/openhcs/mcp/client_registration.py index c17165cf68..39aa5cdc49 100644 --- a/openhcs/mcp/client_registration.py +++ b/openhcs/mcp/client_registration.py @@ -16,13 +16,13 @@ import tempfile from abc import ABC, abstractmethod from collections.abc import Callable, Mapping, Sequence -from dataclasses import asdict, dataclass -from enum import Enum +from dataclasses import dataclass from pathlib import Path from typing import Any, ClassVar import tomlkit from metaclass_registry import AutoRegisterMeta +from python_introspect import to_jsonable from tomlkit.items import InlineTable from openhcs.agent.runtime_platform import AgentRuntimePlatformKey @@ -41,13 +41,36 @@ _CONFIG_MODE = 0o600 -class ClientRegistrationStatus(str, Enum): - """Stable installer-facing result status.""" +class ClientRegistrationStatus(ABC, metaclass=AutoRegisterMeta): + """Installer-facing outcome of registering one client. - REGISTERED = "registered" - UPDATED = "updated" - UNCHANGED = "unchanged" - FAILED = "failed" + Each outcome is a subclass; ``name`` is its spelling in the installer + report. + """ + + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + failed: ClassVar[bool] = False + """Registration did not happen; the report counts this result as failed.""" + + +class ClientRegistered(ClientRegistrationStatus): + name = "registered" + + +class ClientUpdated(ClientRegistrationStatus): + name = "updated" + + +class ClientUnchanged(ClientRegistrationStatus): + name = "unchanged" + + +class ClientRegistrationFailed(ClientRegistrationStatus): + name = "failed" + failed = True @dataclass(frozen=True, slots=True) @@ -220,7 +243,7 @@ def installation_paths( class ClientConfigMutation: """Leaf-owned configuration mutation without duplicated client identity.""" - status: ClientRegistrationStatus + status: type[ClientRegistrationStatus] config_path: str | None backup_path: str | None message: str @@ -232,7 +255,7 @@ class ClientRegistrationResult: target_id: str display_name: str - status: str + status: type[ClientRegistrationStatus] required: bool detected: bool config_path: str | None @@ -252,7 +275,7 @@ def from_mutation( return cls( target_id=target.require_target_id(), display_name=target.display_name, - status=mutation.status.value, + status=mutation.status, required=required, detected=detected, config_path=mutation.config_path, @@ -275,7 +298,7 @@ def failure( return cls( target_id=target_id, display_name=display_name, - status=ClientRegistrationStatus.FAILED.value, + status=ClientRegistrationFailed, required=required, detected=detected, config_path=os.fspath(config_path) if config_path is not None else None, @@ -316,7 +339,7 @@ def from_results( failed = tuple( result for result in frozen_results - if result.status == ClientRegistrationStatus.FAILED.value + if result.status.failed ) return cls( schema_version=CLIENT_REGISTRATION_SCHEMA_VERSION, @@ -335,13 +358,7 @@ def from_results( def as_dict(self) -> dict[str, Any]: """Return a JSON-native installer payload.""" - return { - "schema_version": self.schema_version, - "ok": self.ok, - "required_ok": self.required_ok, - "results": [asdict(result) for result in self.results], - "skill_sync": [asdict(result) for result in self.skill_sync], - } + return dict(to_jsonable(self)) class ClientConfigFormatError(ValueError): @@ -717,7 +734,7 @@ def register( f"VS Code --add-mcp exited with {completed.returncode}{suffix}" ) return ClientConfigMutation( - status=ClientRegistrationStatus.REGISTERED, + status=ClientRegistered, config_path=None, backup_path=None, message=f"Registered OpenHCS through {executable} --add-mcp.", @@ -809,7 +826,7 @@ def _update_codex_toml( if _toml_server_matches(servers.get(OPENHCS_MCP_SERVER_NAME), launcher): return ClientConfigMutation( - status=ClientRegistrationStatus.UNCHANGED, + status=ClientUnchanged, config_path=os.fspath(path), backup_path=None, message="OpenHCS is already registered with ChatGPT desktop and Codex.", @@ -825,9 +842,9 @@ def _update_codex_toml( backup_path = _atomic_config_write(path, rendered, original=original) return ClientConfigMutation( status=( - ClientRegistrationStatus.UPDATED + ClientUpdated if original is not None - else ClientRegistrationStatus.REGISTERED + else ClientRegistered ), config_path=os.fspath(path), backup_path=backup_path, @@ -865,7 +882,7 @@ def _update_json_mcp_servers( desired = launcher.stdio_server_entry() if servers.get(OPENHCS_MCP_SERVER_NAME) == desired: return ClientConfigMutation( - status=ClientRegistrationStatus.UNCHANGED, + status=ClientUnchanged, config_path=os.fspath(path), backup_path=None, message="OpenHCS is already registered with this client.", @@ -876,9 +893,9 @@ def _update_json_mcp_servers( backup_path = _atomic_config_write(path, rendered, original=original) return ClientConfigMutation( status=( - ClientRegistrationStatus.UPDATED + ClientUpdated if original is not None - else ClientRegistrationStatus.REGISTERED + else ClientRegistered ), config_path=os.fspath(path), backup_path=backup_path, @@ -983,7 +1000,7 @@ def register_mcp_clients( skill_results: list[ClientSkillSyncResult] = [] if sync_client_skills: for result in results: - if result.status == ClientRegistrationStatus.FAILED.value: + if result.status.failed: continue target = McpClientRegistrationTarget.target_for_id(result.target_id) assert target is not None # Successful results came from registered owners. @@ -1099,10 +1116,10 @@ def main(argv: Sequence[str] | None = None) -> int: else: for result in report.results: print( - f"{result.display_name}: {result.status} - {result.message}", + f"{result.display_name}: {result.status.name} - {result.message}", file=( sys.stderr - if result.status == ClientRegistrationStatus.FAILED.value + if result.status.failed else sys.stdout ), ) diff --git a/tests/unit/agent/test_mcp_client_registration.py b/tests/unit/agent/test_mcp_client_registration.py index 3bc53e0dc3..712b7f4c43 100644 --- a/tests/unit/agent/test_mcp_client_registration.py +++ b/tests/unit/agent/test_mcp_client_registration.py @@ -12,7 +12,10 @@ from openhcs.mcp.client_registration import ( CLIENT_REGISTRATION_SCHEMA_VERSION, ClientRegistrationEnvironment, - ClientRegistrationStatus, + ClientRegistered, + ClientRegistrationFailed, + ClientUnchanged, + ClientUpdated, ClaudeDesktopClientRegistrationTarget, CodexClientRegistrationTarget, CursorClientRegistrationTarget, @@ -102,7 +105,7 @@ def test_codex_registration_preserves_unrelated_toml_and_creates_backup( assert report.ok assert report.required_ok (result,) = report.results - assert result.status == ClientRegistrationStatus.UPDATED.value + assert result.status == ClientUpdated assert result.config_path == str(config_path) assert result.backup_path == f"{config_path}.openhcs.bak" assert Path(result.backup_path).read_text(encoding="utf-8") == original @@ -138,7 +141,7 @@ def test_codex_unchanged_registration_does_not_rewrite_or_backup( environment=environment, ) - assert report.results[0].status == ClientRegistrationStatus.UNCHANGED.value + assert report.results[0].status == ClientUnchanged assert config_path.read_text(encoding="utf-8") == original assert not Path(f"{config_path}.openhcs.bak").exists() @@ -179,7 +182,7 @@ def test_new_cursor_config_uses_strict_mcp_servers_json(tmp_path: Path) -> None: ) result = report.results[0] - assert result.status == ClientRegistrationStatus.REGISTERED.value + assert result.status == ClientRegistered assert result.backup_path is None config_path = tmp_path / ".cursor" / "mcp.json" assert json.loads(config_path.read_text(encoding="utf-8")) == { @@ -227,7 +230,7 @@ def test_additional_home_json_clients_use_documented_user_config( assert report.ok (result,) = report.results - assert result.status == ClientRegistrationStatus.UPDATED.value + assert result.status == ClientUpdated assert result.config_path == str(config_path) updated = json.loads(config_path.read_text(encoding="utf-8")) assert updated["theme"] == "dark" @@ -265,7 +268,7 @@ def test_home_json_registration_is_idempotent_without_backup_churn( ) (result,) = report.results - assert result.status == ClientRegistrationStatus.UNCHANGED.value + assert result.status == ClientUnchanged assert result.backup_path is None assert config_path.read_text(encoding="utf-8") == original assert not Path(f"{config_path}.openhcs.bak").exists() @@ -295,7 +298,7 @@ def test_json_update_replaces_only_openhcs_and_preserves_unrelated_entries( ) result = report.results[0] - assert result.status == ClientRegistrationStatus.UPDATED.value + assert result.status == ClientUpdated assert Path(result.backup_path).read_text(encoding="utf-8") == original updated = json.loads(config_path.read_text(encoding="utf-8")) assert updated["theme"] == "dark" @@ -331,7 +334,7 @@ def test_malformed_config_fails_without_mutation_or_backup( assert not report.ok assert not report.required_ok - assert report.results[0].status == ClientRegistrationStatus.FAILED.value + assert report.results[0].status == ClientRegistrationFailed assert "left unchanged" in report.results[0].message assert config_path.read_text(encoding="utf-8") == malformed assert not Path(f"{config_path}.openhcs.bak").exists() @@ -420,8 +423,8 @@ def test_detected_failure_preserves_success_and_does_not_fail_required_contract( assert [ (result.target_id, result.status, result.required) for result in report.results ] == [ - ("codex", ClientRegistrationStatus.REGISTERED.value, True), - ("cursor", ClientRegistrationStatus.FAILED.value, False), + ("codex", ClientRegistered, True), + ("cursor", ClientRegistrationFailed, False), ] diff --git a/tests/unit/agent/test_mcp_skill_registration.py b/tests/unit/agent/test_mcp_skill_registration.py index 72cbcde7d9..7e54f88035 100644 --- a/tests/unit/agent/test_mcp_skill_registration.py +++ b/tests/unit/agent/test_mcp_skill_registration.py @@ -113,7 +113,7 @@ def test_legacy_skill_prevents_duplicate_discovery(bundle, environment, kind): report = register(environment, sync=True) assert not report.ok assert not report.required_ok - assert report.results[0].status == "registered" # MCP did succeed. + assert report.results[0].status is client_registration.ClientRegistered # MCP did succeed. assert "duplicate discovery" in report.skill_sync[0].error assert legacy.lstat() == before assert not (environment.home / ".agents").exists() @@ -124,7 +124,7 @@ def test_custom_default_skill_is_preserved_and_failure_is_separate(bundle, envir skill.mkdir(parents=True) (skill / "SKILL.md").write_text("user instructions") report = register(environment, sync=True) - assert report.results[0].status == "registered" + assert report.results[0].status is client_registration.ClientRegistered assert not report.required_ok assert "Unmanaged" in report.skill_sync[0].error assert (skill / "SKILL.md").read_text() == "user instructions" @@ -194,7 +194,7 @@ def failed_sync(*args, **kwargs): monkeypatch.setattr(client_registration, "sync_skills", failed_sync) report = register(environment, sync=True) - assert report.results[0].status == "registered" + assert report.results[0].status is client_registration.ClientRegistered assert not report.ok assert "not writable" in report.skill_sync[0].error @@ -268,7 +268,7 @@ def test_failed_mcp_registration_does_not_install_skills(bundle, environment): config.parent.mkdir(parents=True) config.write_text("[broken toml") report = register(environment, sync=True) - assert report.results[0].status == "failed" + assert report.results[0].status is client_registration.ClientRegistrationFailed assert not report.skill_sync assert not (environment.home / ".agents").exists() @@ -324,7 +324,7 @@ def skill_directory(cls, environment): @classmethod def register(cls, launcher, environment): return client_registration.ClientConfigMutation( - client_registration.ClientRegistrationStatus.REGISTERED, + client_registration.ClientRegistered, None, None, "registered test harness", From 79cdf436dceb5dc685fea1add5f6e7b457b9ca42 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:32:58 -0400 Subject: [PATCH 07/36] E1: viewer settlement phases are a family; settle progress is a declared wire view ViewerSettlePhase becomes SettlementRunning/SettlementComplete/SettlementFailed; each phase owns its validation and idle completion. ViewerSettleProgress decodes and encodes through its declared fields (WireView), so ViewerSettleField and the from_response/to_wire_mapping copies are deleted. Co-Authored-By: Claude Opus 5.5 --- openhcs/core/orchestrator/bundle_execution.py | 2 +- openhcs/runtime/fiji_viewer_server.py | 10 +- openhcs/runtime/napari_streaming_handlers.py | 17 +- openhcs/runtime/napari_viewer_server.py | 3 +- openhcs/runtime/viewer_control_actions.py | 5 +- openhcs/runtime/viewer_protocol.py | 147 ++++++++---------- tests/unit/test_fiji_viewer_server.py | 12 +- .../test_napari_accepted_work_settlement.py | 48 +++--- tests/unit/test_napari_reset_transitions.py | 8 +- tests/unit/test_napari_streaming_handlers.py | 36 ++--- tests/unit/test_napari_transport_ownership.py | 6 +- tests/unit/test_orchestrator_lane_planning.py | 6 +- tests/unit/test_viewer_control_actions.py | 2 +- tests/unit/test_viewer_protocol.py | 14 +- 14 files changed, 142 insertions(+), 174 deletions(-) diff --git a/openhcs/core/orchestrator/bundle_execution.py b/openhcs/core/orchestrator/bundle_execution.py index 07044777d6..5c291eb7fe 100644 --- a/openhcs/core/orchestrator/bundle_execution.py +++ b/openhcs/core/orchestrator/bundle_execution.py @@ -1166,7 +1166,7 @@ def __call__(self, progress: "ViewerSettleProgress") -> None: percent=percent, context={ "viewer_port": self.viewer_port, - **progress.to_wire_mapping(), + **progress.to_dict(), }, ) diff --git a/openhcs/runtime/fiji_viewer_server.py b/openhcs/runtime/fiji_viewer_server.py index 3658337c5e..7c45f24003 100644 --- a/openhcs/runtime/fiji_viewer_server.py +++ b/openhcs/runtime/fiji_viewer_server.py @@ -62,7 +62,9 @@ ViewerControlResponseField, ViewerProtocolStatus, ViewerServerLaunchRequest, - ViewerSettlePhase, + SettlementComplete, + SettlementFailed, + SettlementRunning, ViewerSettleProgress, ) from openhcs.runtime.zmq_application import OPENHCS_ENDPOINT_APPLICATION @@ -175,15 +177,15 @@ def progress(self) -> ViewerSettleProgress: with self._lock: if self._failure_message is not None: - phase = ViewerSettlePhase.FAILED + phase = SettlementFailed elif ( self._completed_update_count == self._total_update_count and not self._pending_item_count and not self._active_update ): - phase = ViewerSettlePhase.COMPLETE + phase = SettlementComplete else: - phase = ViewerSettlePhase.RUNNING + phase = SettlementRunning return ViewerSettleProgress( phase=phase, completed_update_count=self._completed_update_count, diff --git a/openhcs/runtime/napari_streaming_handlers.py b/openhcs/runtime/napari_streaming_handlers.py index 28c735e55b..102c095fb7 100644 --- a/openhcs/runtime/napari_streaming_handlers.py +++ b/openhcs/runtime/napari_streaming_handlers.py @@ -43,6 +43,9 @@ NapariLayerKind, ViewerComponentValueOrdering, ViewerSettlePhase, + SettlementComplete, + SettlementFailed, + SettlementRunning, ViewerSettleProgress, ViewerNativeWindowGeometry, ViewerNativeWindowState, @@ -382,17 +385,17 @@ def debounced_work_unit(self, route_key: str): self.active_route_work_unit_count = 0 @property - def phase(self) -> ViewerSettlePhase: + def phase(self) -> type[ViewerSettlePhase]: with self._lock: if self.failed: - return ViewerSettlePhase.FAILED + return SettlementFailed if ( self.updates is not None and self.completed_update_count == len(self.updates) and self.active_route is None ): - return ViewerSettlePhase.COMPLETE - return ViewerSettlePhase.RUNNING + return SettlementComplete + return SettlementRunning def begin_next( self, @@ -402,7 +405,7 @@ def begin_next( with self._lock: if self.updates is None: return None - if self.phase is not ViewerSettlePhase.RUNNING: + if not self.phase.running: return None if self.active_route is not None: raise RuntimeError( @@ -493,7 +496,7 @@ def require_retirement_boundary(self) -> None: """Requested settlement must finish; unrequested intake can be cancelled.""" with self._lock: if ( - (self.requested and self.phase is ViewerSettlePhase.RUNNING) + (self.requested and self.phase.running) or self.active_route is not None ): raise RuntimeError("Cannot retire layers during active settlement.") @@ -1827,7 +1830,7 @@ def set_pending_update( self.layer_settlement is not None and not self.layer_settlement.awaiting_updates ): - if self.layer_settlement.phase is ViewerSettlePhase.RUNNING: + if self.layer_settlement.phase.running: raise RuntimeError( "Cannot queue a Napari layer update while settlement is active." ) diff --git a/openhcs/runtime/napari_viewer_server.py b/openhcs/runtime/napari_viewer_server.py index 0ccce21643..2285d804e1 100644 --- a/openhcs/runtime/napari_viewer_server.py +++ b/openhcs/runtime/napari_viewer_server.py @@ -190,7 +190,6 @@ ViewerPayloadRecord, ViewerProtocolStatus, ViewerQtEnvironmentPolicy, - ViewerSettlePhase, ViewerSettleProgress, ViewerStateControlOptions, ) @@ -6866,7 +6865,7 @@ def process_messages(self) -> None: # The transport only admits the cycle. This Qt service binds route # updates and stops their timers after the accepted intake barrier. progress = self.layer_route_state.existing_settlement_progress(requested_only=True) - if progress is not None and progress.phase is ViewerSettlePhase.RUNNING: + if progress is not None and progress.phase.running: self.display_pipeline.settlement_progress() while True: try: diff --git a/openhcs/runtime/viewer_control_actions.py b/openhcs/runtime/viewer_control_actions.py index 056b735498..2f3cd509e9 100644 --- a/openhcs/runtime/viewer_control_actions.py +++ b/openhcs/runtime/viewer_control_actions.py @@ -32,7 +32,6 @@ from openhcs.runtime.viewer_protocol import ( OpenHCSViewerControlMessageType, ViewerControlField, - ViewerSettlePhase, ViewerSettleProgress, ) @@ -277,7 +276,7 @@ def reply(self, server: ViewerServerPort, progress_of) -> ControlReply: message=unavailable, ) progress = progress_of() - failed = progress.phase is ViewerSettlePhase.FAILED + failed = progress.phase.failed name = server.viewer_display_name return control_reply( ViewerProtocolStatus.ERROR if failed else ViewerProtocolStatus.SUCCESS, @@ -290,7 +289,7 @@ def reply(self, server: ViewerServerPort, progress_of) -> ControlReply: f"{progress.completed_update_count}/{progress.total_update_count}." ) ), - fields=progress.to_wire_mapping(), + fields=progress.to_dict(), ) diff --git a/openhcs/runtime/viewer_protocol.py b/openhcs/runtime/viewer_protocol.py index fddf72edee..25c21ceaa0 100644 --- a/openhcs/runtime/viewer_protocol.py +++ b/openhcs/runtime/viewer_protocol.py @@ -35,6 +35,7 @@ from zmqruntime.client import EndpointProcessGroup, endpoint_process from zmqruntime.config import NonBlankString, TransportMode, ZMQConfig from zmqruntime.messages import ( + WireView, ControlRequestHeader, ControlMessageType, EndpointApplicationCompatibility, @@ -120,25 +121,58 @@ register_cleanup_callback(_EXECUTION_OWNED_VIEWER_PROCESSES.stop_all) -class ViewerSettlePhase(str, Enum): - """Lifecycle phase for incremental viewer settlement.""" +class ViewerSettlePhase(ABC, metaclass=AutoRegisterMeta): + """Where one viewer's incremental settlement stands. - RUNNING = "running" - COMPLETE = "complete" - FAILED = "failed" + Each phase is a subclass; ``name`` is its wire spelling. + """ + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["ViewerSettlePhase"]]] = {} + + name: ClassVar[str | None] = None + running: ClassVar[bool] = False + """Display work is still queued or active.""" + settled: ClassVar[bool] = False + """Every queued update is displayed.""" + failed: ClassVar[bool] = False + """Display work failed; settlement will not complete.""" + idle_completion_percent: ClassVar[float] = 0.0 + """Completion reported when there were no updates to settle.""" + + @classmethod + def validate(cls, progress: "ViewerSettleProgress") -> None: + """Reject progress this phase cannot report.""" + + +class SettlementRunning(ViewerSettlePhase): + name = "running" + running = True + + +class SettlementComplete(ViewerSettlePhase): + name = "complete" + settled = True + idle_completion_percent = 100.0 + + @classmethod + def validate(cls, progress: "ViewerSettleProgress") -> None: + if progress.intake_item_active: + raise ValueError("Completed settlement cannot have active intake.") + if ( + progress.completed_update_count != progress.total_update_count + or progress.active_route is not None + ): + raise ValueError( + "Completed viewer settlement must have no active route and all " + "updates completed." + ) -class ViewerSettleField(str, Enum): - """Wire fields owned by viewer-settlement progress replies.""" - PHASE = "settle_phase" - COMPLETED_UPDATE_COUNT = "completed_update_count" - TOTAL_UPDATE_COUNT = "total_update_count" - ACTIVE_ROUTE = "active_route" - ACTIVE_ROUTE_WORK_UNIT_COUNT = "active_route_work_unit_count" - ACTIVE_ROUTE_WORK_UNIT_ACTIVE = "active_route_work_unit_active" - PROCESSED_INTAKE_ITEM_COUNT = "processed_intake_item_count" - INTAKE_ITEM_ACTIVE = "intake_item_active" +class SettlementFailed(ViewerSettlePhase): + name = "failed" + failed = True class ViewerPayloadSummaryField(str, Enum): @@ -661,10 +695,13 @@ def succeeded(self) -> bool: @dataclass(frozen=True, slots=True) -class ViewerSettleProgress: - """Typed progress for one viewer's incremental settlement cycle.""" +class ViewerSettleProgress(WireView): + """Typed progress for one viewer's incremental settlement cycle. + + It travels as declared fields beside the control reply's header. + """ - phase: ViewerSettlePhase + phase: type[ViewerSettlePhase] completed_update_count: int total_update_count: int active_route: str | None = None @@ -681,8 +718,6 @@ def __post_init__(self) -> None: raise ValueError("Viewer processed intake count must be non-negative.") if type(self.intake_item_active) is not bool: raise TypeError("Viewer intake activity must be boolean.") - if self.phase is ViewerSettlePhase.COMPLETE and self.intake_item_active: - raise ValueError("Completed settlement cannot have active intake.") if ( type(self.completed_update_count) is not int or type(self.total_update_count) is not int @@ -693,14 +728,6 @@ def __post_init__(self) -> None: raise ValueError( "Viewer settle progress counts must satisfy 0 <= completed <= total." ) - if self.phase is ViewerSettlePhase.COMPLETE and ( - self.completed_update_count != self.total_update_count - or self.active_route is not None - ): - raise ValueError( - "Completed viewer settlement must have no active route and all " - "updates completed." - ) if self.active_route is not None and not isinstance(self.active_route, str): raise TypeError("Viewer settle active_route must be a string or None.") if ( @@ -720,13 +747,14 @@ def __post_init__(self) -> None: raise ValueError( "Viewer settle active work-unit state requires an active route." ) + self.phase.validate(self) @classmethod def complete(cls, total_update_count: int = 0) -> "ViewerSettleProgress": """Return terminal successful settlement progress.""" return cls( - phase=ViewerSettlePhase.COMPLETE, + phase=SettlementComplete, completed_update_count=total_update_count, total_update_count=total_update_count, ) @@ -751,60 +779,7 @@ def heartbeat_due(self, elapsed_seconds: float, interval_seconds: float) -> bool def completion_percent(self) -> float: if self.total_update_count: return 100.0 * self.completed_update_count / self.total_update_count - return 100.0 if self.phase is ViewerSettlePhase.COMPLETE else 0.0 - - @classmethod - def from_response( - cls, - response: ViewerControlResponse, - ) -> "ViewerSettleProgress": - """Parse exact settlement progress from a control response.""" - - payload = response.payload - active_route_value = payload[ViewerSettleField.ACTIVE_ROUTE.value] - active_work_unit_value = payload[ - ViewerSettleField.ACTIVE_ROUTE_WORK_UNIT_ACTIVE.value - ] - if type(active_work_unit_value) is not bool: - raise TypeError("Viewer settle active work-unit state must be boolean.") - return cls( - phase=ViewerSettlePhase(str(payload[ViewerSettleField.PHASE.value])), - completed_update_count=int( - payload[ViewerSettleField.COMPLETED_UPDATE_COUNT.value] - ), - total_update_count=int(payload[ViewerSettleField.TOTAL_UPDATE_COUNT.value]), - active_route=( - None if active_route_value is None else str(active_route_value) - ), - active_route_work_unit_count=int( - payload[ViewerSettleField.ACTIVE_ROUTE_WORK_UNIT_COUNT.value] - ), - active_route_work_unit_active=active_work_unit_value, - processed_intake_item_count=cast(int, payload[ - ViewerSettleField.PROCESSED_INTAKE_ITEM_COUNT.value - ]), - intake_item_active=payload[ViewerSettleField.INTAKE_ITEM_ACTIVE.value], - ) - - def to_wire_mapping(self) -> dict[str, object]: - """Return exact fields for a viewer control reply.""" - - return { - ViewerSettleField.PHASE.value: self.phase.value, - ViewerSettleField.COMPLETED_UPDATE_COUNT.value: ( - self.completed_update_count - ), - ViewerSettleField.TOTAL_UPDATE_COUNT.value: self.total_update_count, - ViewerSettleField.ACTIVE_ROUTE.value: self.active_route, - ViewerSettleField.ACTIVE_ROUTE_WORK_UNIT_COUNT.value: ( - self.active_route_work_unit_count - ), - ViewerSettleField.ACTIVE_ROUTE_WORK_UNIT_ACTIVE.value: ( - self.active_route_work_unit_active - ), - ViewerSettleField.PROCESSED_INTAKE_ITEM_COUNT.value: self.processed_intake_item_count, - ViewerSettleField.INTAKE_ITEM_ACTIVE.value: self.intake_item_active, - } + return self.phase.idle_completion_percent class ViewerComponentValueOrdering: @@ -2185,7 +2160,7 @@ def settle_viewer_state( message_type=ViewerControlMessageType.SETTLE.value, timeout=timeout, ).send() - progress = ViewerSettleProgress.from_response(response) + progress = ViewerSettleProgress.from_dict(response.payload) if progress_callback is not None: progress_callback(progress) except Exception as error: @@ -2196,14 +2171,14 @@ def settle_viewer_state( ) return False - if not response.succeeded() or progress.phase is ViewerSettlePhase.FAILED: + if not response.succeeded() or progress.phase.failed: logger.warning( "%s viewer settlement failed: %s", self.viewer_process_label, response.payload, ) return False - if progress.phase is ViewerSettlePhase.COMPLETE: + if progress.phase.settled: logger.info( "%s viewer settled %d/%d layer updates", self.viewer_process_label, diff --git a/tests/unit/test_fiji_viewer_server.py b/tests/unit/test_fiji_viewer_server.py index 29f16608a4..82b2c798d9 100644 --- a/tests/unit/test_fiji_viewer_server.py +++ b/tests/unit/test_fiji_viewer_server.py @@ -56,7 +56,7 @@ from openhcs.runtime.viewer_protocol import ( ViewerControlMessageType, ViewerControlResponse, - ViewerSettlePhase, + SettlementRunning, ViewerSettleProgress, ) from tests.unit.viewer_axes_fixture import STREAM_AXES @@ -236,15 +236,13 @@ def test_fiji_settlement_tracks_queued_and_bounded_native_work() -> None: settlement.queue_items(2) settlement.queue_items(3) - queued = ViewerSettleProgress.from_response( - ViewerControlResponse( + queued = ViewerSettleProgress.from_dict( fiji_control_server(settlement=settlement).handle_control_message( {"type": ViewerControlMessageType.SETTLE.value} ) ) - ) - assert queued.phase is ViewerSettlePhase.RUNNING + assert queued.phase is SettlementRunning assert queued.completed_update_count == 0 assert queued.total_update_count == 5 assert queued.active_route is None @@ -258,14 +256,14 @@ def test_fiji_settlement_tracks_queued_and_bounded_native_work() -> None: settlement.complete_active_work_unit(route) active = settlement.progress() - assert active.phase is ViewerSettlePhase.RUNNING + assert active.phase is SettlementRunning assert active.active_route == route assert active.active_route_work_unit_count == 2 settlement.complete_active_update(route) between_batches = settlement.progress() - assert between_batches.phase is ViewerSettlePhase.RUNNING + assert between_batches.phase is SettlementRunning assert between_batches.completed_update_count == 5 assert between_batches.total_update_count == 9 diff --git a/tests/unit/test_napari_accepted_work_settlement.py b/tests/unit/test_napari_accepted_work_settlement.py index 958d9b25b5..d9ede645af 100644 --- a/tests/unit/test_napari_accepted_work_settlement.py +++ b/tests/unit/test_napari_accepted_work_settlement.py @@ -33,7 +33,9 @@ NapariViewerServerRequest, ViewerControlResponse, ViewerControlMessageRequest, - ViewerSettlePhase, + SettlementComplete, + SettlementFailed, + SettlementRunning, ViewerSettleProgress, ) from openhcs.core.streaming_config_declarations import NapariViewer @@ -140,7 +142,7 @@ def image_item(): def settle(receiver): response = NapariControlAction.for_message_type("settle").handle(receiver, {}) - return response, ViewerSettleProgress.from_response(ViewerControlResponse(response)) + return response, ViewerSettleProgress.from_dict(response) def saved_roi_item(tmp_path): @@ -176,28 +178,28 @@ def test_saved_roi_reopen_reports_pre_route_failure_and_recovers( assert not receiver.layer_route_state.layer_pending_updates response, progress = settle(receiver) assert response["status"] == "error" - assert progress.phase is ViewerSettlePhase.FAILED + assert progress.phase is SettlementFailed assert "channel" in response["message"] # A new accepted batch starts the next existing settlement cycle. The # previous failed cycle must not become a permanent unrelated failure. assert wire_receiver(image_item())["status"] == "success" - admitted = ViewerSettleProgress.from_response(ViewerControlResponse( + admitted = ViewerSettleProgress.from_dict( NapariControlAction.for_message_type("settle").transport_thread_response(receiver, {}) - )) - assert admitted.phase is ViewerSettlePhase.RUNNING + ) + assert admitted.phase is SettlementRunning assert admitted.total_update_count == 0 receiver.process_accepted_stream_messages() settle(receiver) qtbot.waitUntil( - lambda: settle(receiver)[1].phase is not ViewerSettlePhase.RUNNING, timeout=5000 + lambda: settle(receiver)[1].phase is not SettlementRunning, timeout=5000 ) - assert settle(receiver)[1].phase is ViewerSettlePhase.COMPLETE + assert settle(receiver)[1].phase is SettlementComplete assert len(receiver.viewer.layers) == 1 def test_previous_complete_cannot_settle_new_accepted_work(receiver): - assert settle(receiver)[1].phase is ViewerSettlePhase.COMPLETE + assert settle(receiver)[1].phase is SettlementComplete assert ( receiver.accept_stream_message(wire_batch(image_item())).to_wire_mapping()[ "status" @@ -205,10 +207,10 @@ def test_previous_complete_cannot_settle_new_accepted_work(receiver): == "success" ) assert not receiver.accepted_stream_batches.empty() - admitted = ViewerSettleProgress.from_response(ViewerControlResponse( + admitted = ViewerSettleProgress.from_dict( NapariControlAction.for_message_type("settle").transport_thread_response(receiver, {}) - )) - assert admitted.phase is ViewerSettlePhase.RUNNING + ) + assert admitted.phase is SettlementRunning assert admitted.total_update_count == 0 @@ -308,11 +310,11 @@ def deliver(): pass # The caller's original observation budget expired. expired_after = time.perf_counter() - started started = time.perf_counter() - progress = ViewerSettleProgress.from_response(ViewerControlMessageRequest( + progress = ViewerSettleProgress.from_dict((ViewerControlMessageRequest( client.runtime_endpoint, "settle", timeout=2, - ).send()) + ).send()).payload) control_observation.append((native_layer, expired_after, time.perf_counter() - started, progress)) - assert progress.phase is ViewerSettlePhase.RUNNING + assert progress.phase is SettlementRunning client_started.append(time.perf_counter()) return client.settle_viewer_state( progress_callback=lambda progress: observations.append((time.perf_counter(), progress)), @@ -322,16 +324,16 @@ def observe(): control.send(pickle.dumps(ViewerControlMessageRequest( client.runtime_endpoint, "settle", timeout=2, ).to_wire_mapping())) - return ViewerSettleProgress.from_response(ViewerControlResponse( + return ViewerSettleProgress.from_dict( pickle.loads(control.recv()) - )) + ) try: started = time.perf_counter() first = (receiver.layer_route_state.existing_settlement_progress() if late_observer else observe()) elapsed = time.perf_counter() - started - assert first.phase is ViewerSettlePhase.RUNNING + assert first.phase is SettlementRunning assert first.completed_update_count == first.total_update_count == 0 cycle = receiver.layer_route_state.layer_settlement assert cycle.awaiting_updates @@ -371,7 +373,7 @@ def observe(): "original navigation applied after observation expiry") assert final.processed_intake_item_count == (len(item) if isinstance(item, list) else 1) + 1 awaiting = [(t, p) for t, p in observations - if p.phase is ViewerSettlePhase.RUNNING and p.total_update_count == 0] + if p.phase is SettlementRunning and p.total_update_count == 0] awaiting_seconds = (awaiting[-1][0] - awaiting[0][0]) if awaiting else 0 receiver.viewer.screenshot(path=str(tmp_path / "native-settlement.png")) first_reply_seconds = observations[0][0] - client_started[0] if late_observer else elapsed @@ -431,10 +433,10 @@ def test_pre_route_failure_survives_later_batch_before_settlement( receiver.process_accepted_stream_messages() settle(receiver) qtbot.waitUntil( - lambda: settle(receiver)[1].phase is not ViewerSettlePhase.RUNNING, timeout=5000 + lambda: settle(receiver)[1].phase is not SettlementRunning, timeout=5000 ) response, progress = settle(receiver) - assert progress.phase is ViewerSettlePhase.FAILED + assert progress.phase is SettlementFailed assert "channel" in response["message"] assert len(receiver.viewer.layers) == 1 @@ -448,7 +450,7 @@ def test_clear_state_discards_pre_route_failure(receiver, tmp_path): ) receiver.process_accepted_stream_messages() receiver.clear_stream_state() - assert settle(receiver)[1].phase is ViewerSettlePhase.COMPLETE + assert settle(receiver)[1].phase is SettlementComplete assert receiver.layer_route_state.update_failure_message() is None @@ -464,7 +466,7 @@ def test_active_settlement_rejects_admission_without_retaining_copies( == "success" ) receiver.process_accepted_stream_messages() - assert settle(receiver)[1].phase is ViewerSettlePhase.RUNNING + assert settle(receiver)[1].phase is SettlementRunning settlement = receiver.layer_route_state.layer_settlement original_accept = receiver._accept_single_image copied_arrays = [] diff --git a/tests/unit/test_napari_reset_transitions.py b/tests/unit/test_napari_reset_transitions.py index 31bd0d8bd3..224affb742 100644 --- a/tests/unit/test_napari_reset_transitions.py +++ b/tests/unit/test_napari_reset_transitions.py @@ -61,7 +61,7 @@ ) from openhcs.core.roi_point_metadata import ROIFractionalZ from openhcs.runtime.viewer_protocol import ( - OpenHCSViewerControlMessageType, ViewerSettlePhase, + OpenHCSViewerControlMessageType, SettlementComplete, SettlementFailed, SettlementRunning, ) from openhcs.runtime.viewer_component_system import ( ViewerComponentAxisSemanticsFactory, @@ -549,9 +549,9 @@ def test_selected_retirement_uses_registered_queue_and_releases_payloads(receive # Genuine original terminal settlement, not a manually fabricated complete flag. receiver.display_pipeline.settlement_progress() app = QApplication.instance() - while receiver.layer_route_state.existing_settlement_progress().phase is ViewerSettlePhase.RUNNING: + while receiver.layer_route_state.existing_settlement_progress().phase is SettlementRunning: app.processEvents() - assert receiver.layer_route_state.existing_settlement_progress().phase is ViewerSettlePhase.COMPLETE + assert receiver.layer_route_state.existing_settlement_progress().phase is SettlementComplete native = receiver.layer_route_state.layer(route) native_ref = weakref.ref(native) receiver.batch_processors.get_or_create(layer_key=route, napari_server=receiver) @@ -693,7 +693,7 @@ def test_terminal_failed_candidate_is_retirable_without_erasing_other_failure(re receiver.layer_route_state.record_update_error(keep, ValueError("other original failure")) result = ViewerWindowService(QueuedRetirementGateway(receiver)).presentation(retirement_request(receiver, route)) assert result.applied and not result.errors - assert settlement.phase is ViewerSettlePhase.FAILED + assert settlement.phase is SettlementFailed assert not settlement.updates assert receiver.layer_route_state.layer_update_errors == {keep: "other original failure"} assert keep in receiver.layer_route_state.layers diff --git a/tests/unit/test_napari_streaming_handlers.py b/tests/unit/test_napari_streaming_handlers.py index 8b3d124c65..321c7d9b6f 100644 --- a/tests/unit/test_napari_streaming_handlers.py +++ b/tests/unit/test_napari_streaming_handlers.py @@ -34,7 +34,8 @@ ViewerPayloadProjectionOptions, ViewerShapePayloadProjection, ViewerProtocolStatus, - ViewerSettlePhase, + SettlementFailed, + SettlementRunning, ViewerSettleProgress, ViewerControlResponse, ViewerStateControlOptions, @@ -85,7 +86,6 @@ ViewerPayloadControlOptions, ViewerPayloadProjectionOptions, ViewerProtocolStatus, - ViewerSettlePhase, ViewerSettleProgress, ViewerShapePayloadProjection, ViewerStateControlOptions, @@ -5029,7 +5029,7 @@ def get_or_create(self, **_kwargs): update, ) failed_progress = pipeline.settlement_progress() - assert failed_progress.phase is ViewerSettlePhase.FAILED + assert failed_progress.phase is SettlementFailed assert ( server.layer_route_state.update_failure_message() == "failed-route: invalid payload axis" @@ -5085,25 +5085,23 @@ def get_or_create(self, **_kwargs): action = napari_viewer_server.NapariControlAction.for_message_type("settle") first_response = ViewerControlResponse(action.handle(server, {})) - first_progress = ViewerSettleProgress.from_response(first_response) - assert first_progress.phase is ViewerSettlePhase.RUNNING + first_progress = ViewerSettleProgress.from_dict(first_response.payload) + assert first_progress.phase is SettlementRunning assert first_progress.completed_update_count == 0 assert first_progress.total_update_count == 2 assert first_progress.active_route == "first-route" assert len(callbacks) == 1 callbacks.pop(0)() - middle_progress = ViewerSettleProgress.from_response( - ViewerControlResponse(action.handle(server, {})) - ) - assert middle_progress.phase is ViewerSettlePhase.RUNNING + middle_progress = ViewerSettleProgress.from_dict(action.handle(server, {})) + assert middle_progress.phase is SettlementRunning assert middle_progress.completed_update_count == 1 assert middle_progress.active_route == "second-route" assert len(callbacks) == 1 callbacks.pop(0)() final_response = ViewerControlResponse(action.handle(server, {})) - final_progress = ViewerSettleProgress.from_response(final_response) + final_progress = ViewerSettleProgress.from_dict(final_response.payload) assert final_response.succeeded() assert final_progress == ViewerSettleProgress.complete(2) assert callbacks == [] @@ -5158,30 +5156,22 @@ def get_or_create(self, **_kwargs): monkeypatch.setattr(napari_viewer_server, "QTimer", DeferredQtTimer) action = napari_viewer_server.NapariControlAction.for_message_type("settle") - initial = ViewerSettleProgress.from_response( - ViewerControlResponse(action.handle(server, {})) - ) + initial = ViewerSettleProgress.from_dict(action.handle(server, {})) assert initial.active_route == route_key assert initial.active_route_work_unit_count == 0 callbacks.pop(0)() - after_first = ViewerSettleProgress.from_response( - ViewerControlResponse(action.handle(server, {})) - ) + after_first = ViewerSettleProgress.from_dict(action.handle(server, {})) assert after_first.completed_update_count == 0 assert after_first.active_route_work_unit_count == 1 callbacks.pop(0)() - after_second = ViewerSettleProgress.from_response( - ViewerControlResponse(action.handle(server, {})) - ) + after_second = ViewerSettleProgress.from_dict(action.handle(server, {})) assert after_second.completed_update_count == 0 assert after_second.active_route_work_unit_count == 2 callbacks.pop(0)() - final = ViewerSettleProgress.from_response( - ViewerControlResponse(action.handle(server, {})) - ) + final = ViewerSettleProgress.from_dict(action.handle(server, {})) assert final == ViewerSettleProgress.complete(1) assert callbacks == [] @@ -5212,7 +5202,7 @@ def get_or_create(self, **_kwargs): pipeline.execute_scheduled_layer_update(route_key, update) failed_progress = pipeline.settlement_progress() - assert failed_progress.phase is ViewerSettlePhase.FAILED + assert failed_progress.phase is SettlementFailed assert ( server.layer_route_state.update_failure_message() == "failed-scheduled-route: invalid scheduled processor route" diff --git a/tests/unit/test_napari_transport_ownership.py b/tests/unit/test_napari_transport_ownership.py index aeac043055..a8d95d8d75 100644 --- a/tests/unit/test_napari_transport_ownership.py +++ b/tests/unit/test_napari_transport_ownership.py @@ -155,7 +155,7 @@ def test_napari_control_pump_reports_active_settlement_without_qt_dispatch() -> ) from openhcs.runtime.viewer_protocol import ( ViewerControlResponse, - ViewerSettlePhase, + SettlementRunning, ViewerSettleProgress, ) @@ -209,8 +209,8 @@ def stop(self) -> None: {"type": "settle"}, OperationDeadline.after_milliseconds(2000, operation="settlement observation"), ))) response = pickle.loads(socket.recv()) - progress = ViewerSettleProgress.from_response(ViewerControlResponse(response)) - assert progress.phase is ViewerSettlePhase.RUNNING + progress = ViewerSettleProgress.from_dict(response) + assert progress.phase is SettlementRunning assert progress.completed_update_count == 0 assert progress.active_route == "large-shapes" assert progress.active_route_work_unit_count == 0 diff --git a/tests/unit/test_orchestrator_lane_planning.py b/tests/unit/test_orchestrator_lane_planning.py index e4b2d24ec4..e7e4e5d838 100644 --- a/tests/unit/test_orchestrator_lane_planning.py +++ b/tests/unit/test_orchestrator_lane_planning.py @@ -69,7 +69,7 @@ from openhcs.runtime.viewer_protocol import ( ViewerControlResponse, ViewerPersistenceMode, - ViewerSettlePhase, + SettlementRunning, ViewerSettleProgress, ) from openhcs.core.progress.types import ( @@ -1363,7 +1363,7 @@ def settle_viewer_state(self, *, progress_callback=None): if progress_callback is not None: progress_callback( ViewerSettleProgress( - ViewerSettlePhase.RUNNING, + SettlementRunning, 1, 2, "large-rois", @@ -1414,7 +1414,7 @@ def test_viewer_settlement_progress_throttles_unchanged_active_observations( monkeypatch, ): active = ViewerSettleProgress( - ViewerSettlePhase.RUNNING, + SettlementRunning, 1, 2, "large-rois", diff --git a/tests/unit/test_viewer_control_actions.py b/tests/unit/test_viewer_control_actions.py index 7a8f9200aa..e4dd8a631c 100644 --- a/tests/unit/test_viewer_control_actions.py +++ b/tests/unit/test_viewer_control_actions.py @@ -88,7 +88,7 @@ def test_fiji_lifecycle_actions_answer_through_the_server_port() -> None: assert process["type"] == "process_launch_ack" assert ViewerProcessLaunchConfig.from_wire_mapping(process["process_launch"]) == launch - assert ViewerSettleProgress.from_response(ViewerControlResponse(settle)) == ( + assert ViewerSettleProgress.from_dict(settle) == ( ViewerSettleProgress.complete() ) assert shutdown["type"] == "shutdown_ack" diff --git a/tests/unit/test_viewer_protocol.py b/tests/unit/test_viewer_protocol.py index ef084790bd..a38880c30f 100644 --- a/tests/unit/test_viewer_protocol.py +++ b/tests/unit/test_viewer_protocol.py @@ -43,7 +43,7 @@ ViewerProcessPlatform, ViewerQtEnvironmentPolicy, ViewerRuntimeEndpoint, - ViewerSettlePhase, + SettlementRunning, ViewerSettleProgress, ) from openhcs.runtime.zmq_config import OPENHCS_ZMQ_CONFIG @@ -515,14 +515,14 @@ def detached_server_arguments( return DetachedViewerPythonArguments.from_literals(str(log_file)) progress_rows = ( - ViewerSettleProgress(ViewerSettlePhase.RUNNING, 0, 1, "large", 0, True), - ViewerSettleProgress(ViewerSettlePhase.RUNNING, 0, 1, "large", 0, True), - ViewerSettleProgress(ViewerSettlePhase.RUNNING, 0, 1, "large", 1), + ViewerSettleProgress(SettlementRunning, 0, 1, "large", 0, True), + ViewerSettleProgress(SettlementRunning, 0, 1, "large", 0, True), + ViewerSettleProgress(SettlementRunning, 0, 1, "large", 1), ViewerSettleProgress.complete(1), ) responses = [ ViewerControlResponse( - payload={"status": "success", **progress.to_wire_mapping()} + payload={"status": "success", **progress.to_dict()} ) for progress in progress_rows ] @@ -556,13 +556,13 @@ def send(request): def test_managed_viewer_settlement_rejects_no_progress(monkeypatch): progress = ViewerSettleProgress( - ViewerSettlePhase.RUNNING, + SettlementRunning, 0, 1, "stalled", ) response = ViewerControlResponse( - payload={"status": "success", **progress.to_wire_mapping()} + payload={"status": "success", **progress.to_dict()} ) viewer = type( "StalledViewer", From 08f0cf57de7af5c7945c22f0ff302ba1b1d07376 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:35:28 -0400 Subject: [PATCH 08/36] E1: the artifact inspection request is its wire message CompiledArtifactInspectionControlPayload copied the request's one field and hand-coded its codec against a one-member type enum; the request now subclasses TypedWireMessage and the control strategy keys by its wire_type. Co-Authored-By: Claude Opus 5.5 --- openhcs/core/artifact_inspection.py | 59 ++----------------- openhcs/runtime/zmq_control.py | 10 ++-- openhcs/runtime/zmq_execution_client.py | 5 +- .../test_compiled_artifact_inspection.py | 5 +- tests/unit/test_artifact_plan_view.py | 3 +- 5 files changed, 13 insertions(+), 69 deletions(-) diff --git a/openhcs/core/artifact_inspection.py b/openhcs/core/artifact_inspection.py index 3992e760a9..1031e9b6c5 100644 --- a/openhcs/core/artifact_inspection.py +++ b/openhcs/core/artifact_inspection.py @@ -4,8 +4,9 @@ from collections.abc import Mapping from dataclasses import dataclass -from enum import Enum -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, ClassVar + +from zmqruntime.messages import TypedWireMessage from openhcs.core.artifacts import ArtifactInputPlan, ArtifactOutputPlan, ArtifactSpec from openhcs.core.function_patterns import ( @@ -19,12 +20,6 @@ from openhcs.core.function_patterns import CompiledFunctionInvocation -class ArtifactInspectionControlMessageType(str, Enum): - """OpenHCS control message types owned by compiled artifact inspection.""" - - READ_COMPILED = "openhcs_artifact_read_compiled" - - @dataclass(frozen=True, slots=True) class CompiledArtifactInvocationInspection: """Exact artifact contracts and plans for one compiled invocation.""" @@ -139,9 +134,11 @@ def steps_for_index( @dataclass(frozen=True, slots=True) -class CompiledArtifactInspectionRequest: +class CompiledArtifactInspectionRequest(TypedWireMessage): """Control request for one retained compile artifact projection.""" + wire_type: ClassVar[str] = "openhcs_artifact_read_compiled" + compile_artifact_id: str def __post_init__(self) -> None: @@ -149,50 +146,6 @@ def __post_init__(self) -> None: raise ValueError("compile_artifact_id cannot be empty.") -@dataclass(frozen=True, slots=True) -class CompiledArtifactInspectionControlPayload: - """Wire payload for one compiled-artifact inspection request.""" - - compile_artifact_id: str - message_type: ArtifactInspectionControlMessageType = ( - ArtifactInspectionControlMessageType.READ_COMPILED - ) - - @classmethod - def from_request( - cls, - request: CompiledArtifactInspectionRequest, - ) -> "CompiledArtifactInspectionControlPayload": - return cls(compile_artifact_id=request.compile_artifact_id) - - @classmethod - def from_dict( - cls, - payload: Mapping[str, Any], - ) -> "CompiledArtifactInspectionControlPayload": - message_type = ArtifactInspectionControlMessageType(str(payload["type"])) - if message_type is not ArtifactInspectionControlMessageType.READ_COMPILED: - raise ValueError( - "Unsupported compiled artifact inspection control type: " - f"{message_type.value!r}." - ) - return cls( - compile_artifact_id=str(payload["compile_artifact_id"]), - message_type=message_type, - ) - - def to_request(self) -> CompiledArtifactInspectionRequest: - return CompiledArtifactInspectionRequest( - compile_artifact_id=self.compile_artifact_id - ) - - def to_dict(self) -> dict[str, Any]: - return { - "type": self.message_type.value, - "compile_artifact_id": self.compile_artifact_id, - } - - @dataclass(frozen=True, slots=True) class CompiledArtifactInspectionResponse: """Control response carrying the compiler-owned inspection projection.""" diff --git a/openhcs/runtime/zmq_control.py b/openhcs/runtime/zmq_control.py index df32723df8..e5f37581f6 100644 --- a/openhcs/runtime/zmq_control.py +++ b/openhcs/runtime/zmq_control.py @@ -142,9 +142,9 @@ def handle( class CompiledArtifactInspectionMessageStrategy(ZMQControlMessageStrategy): """Read the exact artifact plans retained by one compile artifact.""" - from openhcs.core.artifact_inspection import ArtifactInspectionControlMessageType + from openhcs.core.artifact_inspection import CompiledArtifactInspectionRequest - registry_key = ArtifactInspectionControlMessageType.READ_COMPILED.value + registry_key = CompiledArtifactInspectionRequest.wire_type def handle( self, @@ -153,14 +153,12 @@ def handle( ) -> dict: from openhcs.core.artifact_inspection import ( CompiledArtifactInspection, - CompiledArtifactInspectionControlPayload, + CompiledArtifactInspectionRequest, CompiledArtifactInspectionResponse, ) try: - request = CompiledArtifactInspectionControlPayload.from_dict( - message - ).to_request() + request = CompiledArtifactInspectionRequest.from_dict(message) record = context.compiled_artifacts.get(request.compile_artifact_id) if record is None: raise KeyError( diff --git a/openhcs/runtime/zmq_execution_client.py b/openhcs/runtime/zmq_execution_client.py index c601442f0b..14efb43965 100644 --- a/openhcs/runtime/zmq_execution_client.py +++ b/openhcs/runtime/zmq_execution_client.py @@ -1196,7 +1196,6 @@ def get_compiled_artifact_inspection( """Read the compiler-owned artifact inspection projection.""" from openhcs.core.artifact_inspection import ( - CompiledArtifactInspectionControlPayload, CompiledArtifactInspectionRequest, CompiledArtifactInspectionResponse, ) @@ -1206,9 +1205,7 @@ def get_compiled_artifact_inspection( request = CompiledArtifactInspectionRequest( compile_artifact_id=compile_artifact_id ) - response = self._send_control_request( - CompiledArtifactInspectionControlPayload.from_request(request).to_dict() - ) + response = self._send_control_request(request.to_dict()) return CompiledArtifactInspectionResponse.from_control_response( response ).inspection diff --git a/tests/integration/test_compiled_artifact_inspection.py b/tests/integration/test_compiled_artifact_inspection.py index 53e439f644..3c0421d8fe 100644 --- a/tests/integration/test_compiled_artifact_inspection.py +++ b/tests/integration/test_compiled_artifact_inspection.py @@ -15,7 +15,6 @@ from openhcs.core.config import GlobalPipelineConfig, PipelineConfig from openhcs.runtime.zmq_execution_signature import OpenHCSExecutionConfigBundle from openhcs.core.artifact_inspection import ( - CompiledArtifactInspectionControlPayload, CompiledArtifactInspectionRequest, CompiledArtifactInspectionResponse, ) @@ -217,9 +216,7 @@ def wait_for_completion(self, execution_id): def get_compiled_artifact_inspection(self, execution_id): request = CompiledArtifactInspectionRequest(execution_id) response = ZMQControlMessageRouter.handle( - CompiledArtifactInspectionControlPayload.from_request( - request - ).to_dict(), + request.to_dict(), context, ) return CompiledArtifactInspectionResponse.from_control_response( diff --git a/tests/unit/test_artifact_plan_view.py b/tests/unit/test_artifact_plan_view.py index 56bc7119d3..26c073ead1 100644 --- a/tests/unit/test_artifact_plan_view.py +++ b/tests/unit/test_artifact_plan_view.py @@ -14,7 +14,6 @@ from openhcs.runtime.zmq_execution_signature import OpenHCSExecutionConfigBundle from openhcs.core.artifact_inspection import ( CompiledArtifactInspection, - CompiledArtifactInspectionControlPayload, CompiledArtifactInspectionRequest, CompiledArtifactInspectionResponse, ) @@ -223,7 +222,7 @@ def test_registered_control_router_reads_retained_compile_artifact() -> None: compilation=ZMQCompilationResult(execution_bundle=bundle), ) request = CompiledArtifactInspectionRequest("compile-1") - message = CompiledArtifactInspectionControlPayload.from_request(request).to_dict() + message = request.to_dict() response = ZMQControlMessageRouter.handle( message, From feee7e56be39b94248234e67a2e1ab1c48cc3db4 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:35:28 -0400 Subject: [PATCH 09/36] E1: debug session phases, toolbar placements and the inspect action are families DebugSessionPhase was an enum keying the phase declarations that already carried the behaviour; the declarations are now the phases. Toolbar placement is a family the toolbar iterates; the runtime-values action is identified by its own declaration. The Pipeline* aliases of the core debug types are gone. Co-Authored-By: Claude Opus 5.5 --- openhcs/authoring/session/dataset_status.py | 12 -- openhcs/authoring/session/views.py | 8 +- openhcs/core/debug_session_projection.py | 78 ++++--------- .../services/ui_bridge_pipeline_editor.py | 4 +- openhcs/pyqt_gui/widgets/debug_toolbar.py | 56 +++------ .../services/debug_session_projection.py | 29 ++--- .../shared/services/pipeline_debug_actions.py | 108 +++++++++++------- tests/unit/pyqt_gui/test_debug_toolbar.py | 19 +-- tests/unit/pyqt_gui/test_ui_agent_bridge.py | 12 +- 9 files changed, 131 insertions(+), 195 deletions(-) diff --git a/openhcs/authoring/session/dataset_status.py b/openhcs/authoring/session/dataset_status.py index 79734ebecb..0a9075f3c7 100644 --- a/openhcs/authoring/session/dataset_status.py +++ b/openhcs/authoring/session/dataset_status.py @@ -5,10 +5,6 @@ from typing import Optional from openhcs.constants.constants import OrchestratorState -from openhcs.core.debug_session_projection import ( - DebugSessionPhase, - DebugSessionPhaseDeclarationBase, -) from openhcs.core.progress.projection import SubjectRuntimeProjection from openhcs.core.execution_state import ( TerminalExecutionStatus, @@ -18,14 +14,6 @@ class PlateStatusPresenter: """Render deterministic plate status text from execution + orchestrator signals.""" - @classmethod - def build_debug_status_prefix( - cls, - *, - debug_phase: DebugSessionPhase, - ) -> str: - return DebugSessionPhaseDeclarationBase.for_phase(debug_phase).status_prefix - @classmethod def build_status_prefix( cls, diff --git a/openhcs/authoring/session/views.py b/openhcs/authoring/session/views.py index 4f52d57c13..d9aa6129ee 100644 --- a/openhcs/authoring/session/views.py +++ b/openhcs/authoring/session/views.py @@ -217,10 +217,8 @@ def debug_session_id(self) -> str | None: @property def status_prefix(self) -> str: phase = self.debug_phase - if phase is not None: - prefix = PlateStatusPresenter.build_debug_status_prefix(debug_phase=phase) - if prefix: - return prefix + if phase is not None and phase.status_prefix: + return phase.status_prefix return PlateStatusPresenter.build_status_prefix( orchestrator_state=self.orchestrator_state, is_init_pending=self.scope_id in self.session.init_pending, @@ -305,7 +303,7 @@ def row_state( output_root=relation.output_root, source_scope_id=relation.source_scope_id, source_root=relation.source_root, - debug_phase=None if debug_phase is None else debug_phase.value, + debug_phase=None if debug_phase is None else debug_phase.name, debug_session_id=activity.debug_session_id, finished_execution_id=next( ( diff --git a/openhcs/core/debug_session_projection.py b/openhcs/core/debug_session_projection.py index 5e20dad67e..466968799e 100644 --- a/openhcs/core/debug_session_projection.py +++ b/openhcs/core/debug_session_projection.py @@ -4,7 +4,6 @@ from abc import ABC from dataclasses import dataclass -from enum import Enum from typing import ClassVar from metaclass_registry import AutoRegisterMeta @@ -28,20 +27,6 @@ class DebugSessionTargetState: terminal_status: str | None = None -class DebugSessionPhase(str, Enum): - """Closed debugger phase serialized by UI/MCP adapters.""" - - NO_PLATE = "no_plate" - NEEDS_INITIALIZATION = "needs_initialization" - NEEDS_COMPILE = "needs_compile" - PENDING_EXECUTION = "pending_execution" - ACTIVE_SESSION = "active_session" - TERMINAL_COMPLETE = "terminal_complete" - TERMINAL_FAILED = "terminal_failed" - TERMINAL_CANCELLED = "terminal_cancelled" - READY = "ready" - - @dataclass(frozen=True, slots=True) class DebugPauseBoundaryState: """Debug-pause markers visible to action projection and command dispatch.""" @@ -85,48 +70,33 @@ def snapshot_for_id(self, snapshot_id: str | None) -> DebugSnapshot | None: return None @property - def phase(self) -> DebugSessionPhase: - return DebugSessionPhaseDeclarationBase.for_context(self).require_phase() + def phase(self) -> type[DebugSessionPhase]: + return DebugSessionPhase.for_context(self) -class DebugSessionPhaseDeclarationBase(ABC, metaclass=AutoRegisterMeta): - """Nominal core declaration for one debugger session phase.""" +class DebugSessionPhase(ABC, metaclass=AutoRegisterMeta): + """One debugger session phase; ``name`` is its spelling for UI and MCP.""" - __registry_key__ = "phase" + __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[ - dict[DebugSessionPhase, type["DebugSessionPhaseDeclarationBase"]] - ] = {} + __registry__: ClassVar[dict[str, type["DebugSessionPhase"]]] = {} - phase: ClassVar[DebugSessionPhase | None] = None + name: ClassVar[str | None] = None priority: ClassVar[int] title: ClassVar[str] detail: ClassVar[str] status_prefix: ClassVar[str] = "" is_terminal: ClassVar[bool] = False - @classmethod - def require_phase(cls) -> DebugSessionPhase: - if cls.phase is None: - raise TypeError(f"{cls.__name__} does not declare a debug session phase.") - return cls.phase - @classmethod def matches(cls, context: DebugSessionProjectionContext) -> bool: raise NotImplementedError - @classmethod - def for_phase( - cls, - phase: DebugSessionPhase, - ) -> type["DebugSessionPhaseDeclarationBase"]: - return cls.__registry__[phase] - @classmethod def for_context( cls, context: DebugSessionProjectionContext, - ) -> type["DebugSessionPhaseDeclarationBase"]: + ) -> type["DebugSessionPhase"]: for declaration in sorted( cls.__registry__.values(), key=lambda candidate: candidate.priority, @@ -136,8 +106,8 @@ def for_context( raise ValueError("No debugger session phase matched the projection context.") -class NoPlateDebugSessionPhase(DebugSessionPhaseDeclarationBase): - phase = DebugSessionPhase.NO_PLATE +class NoPlateDebugSessionPhase(DebugSessionPhase): + name = "no_plate" priority = 0 title = "No Plate" detail = "Select a plate to debug." @@ -147,8 +117,8 @@ def matches(cls, context: DebugSessionProjectionContext) -> bool: return context.target is None -class NeedsInitializationDebugSessionPhase(DebugSessionPhaseDeclarationBase): - phase = DebugSessionPhase.NEEDS_INITIALIZATION +class NeedsInitializationDebugSessionPhase(DebugSessionPhase): + name = "needs_initialization" priority = 10 title = "Needs Init" detail = "Initialize the selected plate before debugging." @@ -158,8 +128,8 @@ def matches(cls, context: DebugSessionProjectionContext) -> bool: return context.target is not None and not context.target.initialized -class NeedsCompileDebugSessionPhase(DebugSessionPhaseDeclarationBase): - phase = DebugSessionPhase.NEEDS_COMPILE +class NeedsCompileDebugSessionPhase(DebugSessionPhase): + name = "needs_compile" priority = 20 title = "Needs Compile" detail = "Compile the selected plate before debugging." @@ -173,8 +143,8 @@ def matches(cls, context: DebugSessionProjectionContext) -> bool: ) -class PendingExecutionDebugSessionPhase(DebugSessionPhaseDeclarationBase): - phase = DebugSessionPhase.PENDING_EXECUTION +class PendingExecutionDebugSessionPhase(DebugSessionPhase): + name = "pending_execution" priority = 30 title = "Debug Starting" detail = "Debug execution is queued or starting." @@ -191,8 +161,8 @@ def matches(cls, context: DebugSessionProjectionContext) -> bool: ) -class ActiveDebugSessionPhase(DebugSessionPhaseDeclarationBase): - phase = DebugSessionPhase.ACTIVE_SESSION +class ActiveDebugSessionPhase(DebugSessionPhase): + name = "active_session" priority = 40 title = "Debug Active" detail = "Debug session has an active cursor or paused worker." @@ -208,7 +178,7 @@ def matches(cls, context: DebugSessionProjectionContext) -> bool: ) -class TerminalDebugSessionPhase(DebugSessionPhaseDeclarationBase): +class TerminalDebugSessionPhase(DebugSessionPhase): """Shared terminal debugger phase matching a terminal execution status.""" terminal_status: ClassVar[TerminalExecutionStatus | None] = None @@ -237,7 +207,7 @@ def matches(cls, context: DebugSessionProjectionContext) -> bool: class TerminalCompleteDebugSessionPhase(TerminalDebugSessionPhase): - phase = DebugSessionPhase.TERMINAL_COMPLETE + name = "terminal_complete" priority = 50 title = "Debug Complete" detail = "Last debug command completed." @@ -246,7 +216,7 @@ class TerminalCompleteDebugSessionPhase(TerminalDebugSessionPhase): class TerminalFailedDebugSessionPhase(TerminalDebugSessionPhase): - phase = DebugSessionPhase.TERMINAL_FAILED + name = "terminal_failed" priority = 51 title = "Debug Failed" detail = "Last debug command failed." @@ -255,7 +225,7 @@ class TerminalFailedDebugSessionPhase(TerminalDebugSessionPhase): class TerminalCancelledDebugSessionPhase(TerminalDebugSessionPhase): - phase = DebugSessionPhase.TERMINAL_CANCELLED + name = "terminal_cancelled" priority = 52 title = "Debug Cancelled" detail = "Last debug command was cancelled." @@ -263,8 +233,8 @@ class TerminalCancelledDebugSessionPhase(TerminalDebugSessionPhase): terminal_status = TerminalExecutionStatus.CANCELLED -class ReadyDebugSessionPhase(DebugSessionPhaseDeclarationBase): - phase = DebugSessionPhase.READY +class ReadyDebugSessionPhase(DebugSessionPhase): + name = "ready" priority = 100 title = "Ready" detail = "Debug controls are ready." diff --git a/openhcs/pyqt_gui/services/ui_bridge_pipeline_editor.py b/openhcs/pyqt_gui/services/ui_bridge_pipeline_editor.py index bacfc06baa..5ea566d9a8 100644 --- a/openhcs/pyqt_gui/services/ui_bridge_pipeline_editor.py +++ b/openhcs/pyqt_gui/services/ui_bridge_pipeline_editor.py @@ -310,7 +310,7 @@ def _state(self) -> UiPipelineDebugSessionState: manager_execution_state=context.manager_execution_state.value, initialized=False if target is None else target.initialized, compiled=False if target is None else target.compiled, - phase=DebugToolbarActionProjector.phase(context).value, + phase=context.phase.name, active_session_id=None if session is None else session.debug_session_id, execution_id=None if session is None else session.execution_id, axis_id=None if session is None else session.axis_id, @@ -375,7 +375,7 @@ def _action_state(cls, model: DebugActionRenderModel) -> UiDebugActionState: return UiDebugActionState( action_id=model.action_id, label=model.label, - placement=model.placement.value, + placement=model.placement.name, enabled=model.enabled, side_effects=model.side_effects, confirmation_required=model.confirmation_required, diff --git a/openhcs/pyqt_gui/widgets/debug_toolbar.py b/openhcs/pyqt_gui/widgets/debug_toolbar.py index 19a2376caa..593422391b 100644 --- a/openhcs/pyqt_gui/widgets/debug_toolbar.py +++ b/openhcs/pyqt_gui/widgets/debug_toolbar.py @@ -18,12 +18,13 @@ DebugActionRenderModel, DebugSessionPanelText, DebugToolbarActionProjector, - PipelineDebugSessionContext, ) +from openhcs.core.debug_session_projection import DebugSessionProjectionContext from openhcs.pyqt_gui.widgets.shared.services.pipeline_debug_actions import ( - DebugActionPlacement, - DebugToolbarAuxiliaryAction, + DebugToolbarPlacement, + InspectRuntimeValuesAction, PipelineDebugActionDeclarationBase, + PrimaryPanel, ) from pyqt_reactive.widgets.shared.button_panel import ButtonPanel from pyqt_reactive.theming import ColorScheme @@ -55,10 +56,6 @@ def __init__( color_scheme: ColorScheme | None = None, ) -> None: super().__init__(parent) - self.button_panel: ButtonPanel | None = None - self.primary_panel: ButtonPanel | None = None - self.session_panel: ButtonPanel | None = None - self.inspector_panel: ButtonPanel | None = None self.phase_label = QLabel("No Plate") self.cursor_label = QLabel("Select a plate to debug.") self._buttons_by_id: dict[str, QPushButton] = {} @@ -68,7 +65,7 @@ def __init__( layout.setContentsMargins(0, 0, 0, 0) layout.setSpacing(2) - initial_context = PipelineDebugSessionContext( + initial_context = DebugSessionProjectionContext( target=None, session=None, manager_execution_state=ManagerExecutionState.IDLE, @@ -76,31 +73,19 @@ def __init__( initial_models = DebugToolbarActionProjector.render_models(initial_context) layout.addWidget(self.phase_label) layout.addWidget(self.cursor_label) - self.primary_panel = self._build_panel( - initial_models, - DebugActionPlacement.PRIMARY, - color_scheme, - ) - self.button_panel = self.primary_panel - self.session_panel = self._build_panel( - initial_models, - DebugActionPlacement.SESSION, - color_scheme, - ) - self.inspector_panel = self._build_panel( - initial_models, - DebugActionPlacement.INSPECTOR, - color_scheme, - ) - layout.addWidget(self.primary_panel) - layout.addWidget(self.session_panel) - layout.addWidget(self.inspector_panel) + self.panels: dict[type[DebugToolbarPlacement], ButtonPanel] = { + placement: self._build_panel(initial_models, placement, color_scheme) + for placement in DebugToolbarPlacement.panels() + } + self.button_panel = self.panels[PrimaryPanel] + for panel in self.panels.values(): + layout.addWidget(panel) self.set_action_models(initial_models) def _build_panel( self, initial_models: tuple[DebugActionRenderModel, ...], - placement: DebugActionPlacement, + placement: type[DebugToolbarPlacement], color_scheme: ColorScheme | None, ) -> ButtonPanel: models = tuple( @@ -133,12 +118,12 @@ def buttons(self) -> dict[type[DebugCommandType], QPushButton]: def runtime_inspection_button(self) -> QPushButton | None: """Return the declared runtime-inspection button when rendered.""" - return self._buttons_by_id.get(DebugToolbarAuxiliaryAction.RUNTIME_VALUES.value) + return self._buttons_by_id.get(InspectRuntimeValuesAction.action_id()) def emit_debug_command(self, action_id: str) -> None: PipelineDebugActionDeclarationBase.for_action_id(action_id).emit_from(self) - def set_debug_session_context(self, context: PipelineDebugSessionContext) -> None: + def set_debug_session_context(self, context: DebugSessionProjectionContext) -> None: """Project toolbar controls from the current PipelineEditor debug context.""" panel_text = DebugSessionPanelText.from_context(context) @@ -168,17 +153,6 @@ def command_enabled(self, command_type: type[DebugCommandType]) -> bool: ) return model.enabled - def auxiliary_action_enabled( - self, - action_type: DebugToolbarAuxiliaryAction, - ) -> bool: - model = self._action_models.get(action_type.value) - if model is None: - raise ValueError( - f"Debug auxiliary action is not exposed by the toolbar: {action_type}" - ) - return model.enabled - @staticmethod def _tooltip_for_model(model: DebugActionRenderModel) -> str: if model.disabled_reason is None: diff --git a/openhcs/pyqt_gui/widgets/shared/services/debug_session_projection.py b/openhcs/pyqt_gui/widgets/shared/services/debug_session_projection.py index 80b2e1839e..0ca294149f 100644 --- a/openhcs/pyqt_gui/widgets/shared/services/debug_session_projection.py +++ b/openhcs/pyqt_gui/widgets/shared/services/debug_session_projection.py @@ -5,23 +5,16 @@ from dataclasses import dataclass from openhcs.core.debug_session_projection import ( - DebugPauseBoundaryState, DebugSessionPhase, - DebugSessionPhaseDeclarationBase, DebugSessionProjectionContext, - DebugSessionTargetState, + PendingExecutionDebugSessionPhase, ) from openhcs.pyqt_gui.widgets.shared.services.pipeline_debug_actions import ( DebugActionDisabledReason, - DebugActionPlacement, + DebugToolbarPlacement, PipelineDebugActionDeclarationBase, ) -PipelineDebugTargetState = DebugSessionTargetState -PipelineDebugSessionPhase = DebugSessionPhase -PipelineDebugPauseBoundaryState = DebugPauseBoundaryState -PipelineDebugSessionContext = DebugSessionProjectionContext - @dataclass(frozen=True, slots=True) class DebugActionRenderModel: @@ -31,10 +24,10 @@ class DebugActionRenderModel: action_id: str label: str tooltip: str - placement: DebugActionPlacement + placement: type[DebugToolbarPlacement] enabled: bool disabled_reason: DebugActionDisabledReason | None - phase: DebugSessionPhase + phase: type[DebugSessionPhase] side_effects: tuple[str, ...] confirmation_required: bool requires_active_debug_session: bool @@ -63,7 +56,7 @@ def render_model( declaration: type[PipelineDebugActionDeclarationBase], context: DebugSessionProjectionContext, ) -> DebugActionRenderModel: - phase = cls.phase(context) + phase = context.phase disabled_reason = cls.disabled_reason(declaration, context) return DebugActionRenderModel( declaration=declaration, @@ -164,11 +157,7 @@ def execution_pending_without_session( cls, context: DebugSessionProjectionContext, ) -> bool: - return context.phase is DebugSessionPhase.PENDING_EXECUTION - - @classmethod - def phase(cls, context: DebugSessionProjectionContext) -> DebugSessionPhase: - return context.phase + return context.phase is PendingExecutionDebugSessionPhase @dataclass(frozen=True, slots=True) @@ -182,7 +171,7 @@ class DebugSessionPanelText: def from_context( cls, context: DebugSessionProjectionContext ) -> "DebugSessionPanelText": - phase_declaration = DebugSessionPhaseDeclarationBase.for_context(context) + phase_declaration = context.phase cursor = None command_type = None active_session = context.active_session @@ -221,8 +210,4 @@ def from_context( "DebugActionRenderModel", "DebugSessionPanelText", "DebugToolbarActionProjector", - "PipelineDebugPauseBoundaryState", - "PipelineDebugSessionContext", - "PipelineDebugSessionPhase", - "PipelineDebugTargetState", ) diff --git a/openhcs/pyqt_gui/widgets/shared/services/pipeline_debug_actions.py b/openhcs/pyqt_gui/widgets/shared/services/pipeline_debug_actions.py index c9df2f2543..be6e81479b 100644 --- a/openhcs/pyqt_gui/widgets/shared/services/pipeline_debug_actions.py +++ b/openhcs/pyqt_gui/widgets/shared/services/pipeline_debug_actions.py @@ -4,7 +4,6 @@ from abc import ABC, abstractmethod from dataclasses import dataclass -from enum import Enum from typing import TYPE_CHECKING, ClassVar from metaclass_registry import AutoRegisterMeta @@ -26,9 +25,7 @@ if TYPE_CHECKING: from openhcs.pyqt_gui.widgets.debug_toolbar import DebugToolbarWidget - from openhcs.pyqt_gui.widgets.shared.services.debug_session_projection import ( - PipelineDebugSessionContext, - ) + from openhcs.core.debug_session_projection import DebugSessionProjectionContext from openhcs.pyqt_gui.widgets.shared.services.pipeline_editor_workflows import ( PipelineEditorDebugWorkflow, ) @@ -46,19 +43,51 @@ def as_agent_error(self) -> AgentError: return AgentError(code=self.code, message=self.message, hint=self.hint) -class DebugActionPlacement(Enum): - """Toolbar placement declared by one debug action.""" +class DebugToolbarPlacement(ABC, metaclass=AutoRegisterMeta): + """Where the debug toolbar shows an action; ``name`` is its UI-bridge spelling.""" + + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["DebugToolbarPlacement"]]] = {} + + name: ClassVar[str | None] = None + shown: ClassVar[bool] = True + """Actions placed here get a toolbar button.""" + panel_order: ClassVar[int] = 0 + """Position of this placement's button panel in the toolbar.""" + + @classmethod + def panels(cls) -> tuple[type["DebugToolbarPlacement"], ...]: + """The placements the toolbar renders as button panels, in order.""" + + return tuple( + sorted( + (placement for placement in cls.__registry__.values() if placement.shown), + key=lambda placement: placement.panel_order, + ) + ) + + +class PrimaryPanel(DebugToolbarPlacement): + name = "primary" + panel_order = 0 + + +class SessionPanel(DebugToolbarPlacement): + name = "session" + panel_order = 1 - PRIMARY = "primary" - INSPECTOR = "inspector" - SESSION = "session" - HIDDEN = "hidden" +class InspectorPanel(DebugToolbarPlacement): + name = "inspector" + panel_order = 2 -class DebugToolbarAuxiliaryAction(str, Enum): - """Non-command debug toolbar actions.""" - RUNTIME_VALUES = "runtime_values" +class OffToolbar(DebugToolbarPlacement): + """Actions invoked through the editor or the UI bridge, without a button.""" + + name = "hidden" + shown = False class PipelineDebugActionDeclarationBase(ABC, metaclass=AutoRegisterMeta): @@ -66,15 +95,16 @@ class PipelineDebugActionDeclarationBase(ABC, metaclass=AutoRegisterMeta): __registry__: ClassVar[ dict[ - type[DebugCommandType] | DebugToolbarAuxiliaryAction, + type[DebugCommandType] | str, type["PipelineDebugActionDeclarationBase"], ] ] = {} __registry_key__ = "identity" __skip_if_no_key__ = True - identity: ClassVar[type[DebugCommandType] | DebugToolbarAuxiliaryAction | None] = None - toolbar_placement: ClassVar[DebugActionPlacement] = DebugActionPlacement.HIDDEN + identity: ClassVar[type[DebugCommandType] | str | None] = None + """The debug command this action sends, or an auxiliary action's id.""" + toolbar_placement: ClassVar[type[DebugToolbarPlacement]] = OffToolbar toolbar_order: ClassVar[int] = 0 label: ClassVar[str] tooltip: ClassVar[str] @@ -89,7 +119,7 @@ def action_id(cls) -> str: return to_jsonable(cls.require_identity()) @classmethod - def require_identity(cls) -> type[DebugCommandType] | DebugToolbarAuxiliaryAction: + def require_identity(cls) -> type[DebugCommandType] | str: if cls.identity is None: raise ValueError( f"{cls.__name__} does not declare a debug action identity." @@ -98,7 +128,7 @@ def require_identity(cls) -> type[DebugCommandType] | DebugToolbarAuxiliaryActio @classmethod def is_toolbar_action(cls) -> bool: - return cls.toolbar_placement is not DebugActionPlacement.HIDDEN + return cls.toolbar_placement.shown @classmethod def toolbar_actions(cls) -> tuple[type["PipelineDebugActionDeclarationBase"], ...]: @@ -110,7 +140,7 @@ def toolbar_actions(cls) -> tuple[type["PipelineDebugActionDeclarationBase"], .. if declaration.is_toolbar_action() ), key=lambda declaration: ( - declaration.toolbar_placement.value, + declaration.toolbar_placement.panel_order, declaration.toolbar_order, declaration.action_id(), ), @@ -141,19 +171,19 @@ def for_command_type( return declaration @classmethod - def label_for(cls, context: "PipelineDebugSessionContext") -> str: + def label_for(cls, context: "DebugSessionProjectionContext") -> str: del context return cls.label @classmethod - def tooltip_for(cls, context: "PipelineDebugSessionContext") -> str: + def tooltip_for(cls, context: "DebugSessionProjectionContext") -> str: del context return cls.tooltip @classmethod def availability_override( cls, - context: "PipelineDebugSessionContext", + context: "DebugSessionProjectionContext", ) -> DebugActionDisabledReason | None: del context return None @@ -190,19 +220,10 @@ def emit_from(cls, toolbar: "DebugToolbarWidget") -> None: class PipelineDebugAuxiliaryActionDeclaration(PipelineDebugActionDeclarationBase): """Debug action owned by the toolbar but not represented by DebugCommandType.""" - identity: ClassVar[DebugToolbarAuxiliaryAction | None] = None + identity: ClassVar[str | None] = None side_effects: ClassVar[tuple[str, ...]] = ("opens_debug_runtime_inspector",) confirmation_required: ClassVar[bool] = False - @classmethod - def auxiliary_action_type(cls) -> DebugToolbarAuxiliaryAction: - identity = cls.require_identity() - if not isinstance(identity, DebugToolbarAuxiliaryAction): - raise TypeError( - f"{cls.__name__} is not backed by DebugToolbarAuxiliaryAction." - ) - return identity - @classmethod def invoke(cls, workflow: "PipelineEditorDebugWorkflow") -> None: workflow.show_runtime_inspection() @@ -224,13 +245,13 @@ def invoke(cls, workflow: "PipelineEditorDebugWorkflow") -> None: class StartOrContinueDebugAction(PipelineDebugCommandActionDeclaration): identity = RunDebugCommand - toolbar_placement = DebugActionPlacement.PRIMARY + toolbar_placement = PrimaryPanel toolbar_order = 10 label = "Debug" tooltip = "Start or continue debug execution for the selected plate" @classmethod - def label_for(cls, context: "PipelineDebugSessionContext") -> str: + def label_for(cls, context: "DebugSessionProjectionContext") -> str: if context.active_session is not None: return "Continue" target = context.target @@ -241,7 +262,7 @@ def label_for(cls, context: "PipelineDebugSessionContext") -> str: class StepDebugAction(PipelineDebugCommandActionDeclaration): identity = StepDebugCommand - toolbar_placement = DebugActionPlacement.PRIMARY + toolbar_placement = PrimaryPanel toolbar_order = 20 label = "Step" tooltip = "Run one debug step" @@ -249,7 +270,7 @@ class StepDebugAction(PipelineDebugCommandActionDeclaration): class RunToPauseDebugAction(PipelineDebugCommandActionDeclaration): identity = RunToPauseDebugCommand - toolbar_placement = DebugActionPlacement.PRIMARY + toolbar_placement = PrimaryPanel toolbar_order = 30 label = "Run to Pause" tooltip = "Run until the next pause marker" @@ -257,7 +278,7 @@ class RunToPauseDebugAction(PipelineDebugCommandActionDeclaration): @classmethod def availability_override( cls, - context: "PipelineDebugSessionContext", + context: "DebugSessionProjectionContext", ) -> DebugActionDisabledReason | None: if not context.pause_boundaries.has_pause_boundaries: return DebugActionDisabledReason( @@ -273,7 +294,7 @@ def availability_override( class RestartDebugAction(PipelineDebugCommandActionDeclaration): identity = RestartDebugCommand - toolbar_placement = DebugActionPlacement.SESSION + toolbar_placement = SessionPanel toolbar_order = 40 label = "Restart" tooltip = "Restart the current debug session" @@ -281,8 +302,8 @@ class RestartDebugAction(PipelineDebugCommandActionDeclaration): class InspectRuntimeValuesAction(PipelineDebugAuxiliaryActionDeclaration): - identity = DebugToolbarAuxiliaryAction.RUNTIME_VALUES - toolbar_placement = DebugActionPlacement.INSPECTOR + identity = "runtime_values" + toolbar_placement = InspectorPanel toolbar_order = 10 label = "Inspect Runtime" tooltip = "Inspect live runtime values for the paused debug worker" @@ -291,7 +312,7 @@ class InspectRuntimeValuesAction(PipelineDebugAuxiliaryActionDeclaration): class ChooseSourceGroupDebugAction(PipelineDebugCommandActionDeclaration): identity = ChooseSourceGroupDebugCommand - toolbar_placement = DebugActionPlacement.SESSION + toolbar_placement = SessionPanel toolbar_order = 10 label = "Choose source group" tooltip = "Choose a well/image set for debug execution" @@ -299,7 +320,7 @@ class ChooseSourceGroupDebugAction(PipelineDebugCommandActionDeclaration): class StopDebugSessionAction(PipelineDebugCommandActionDeclaration): identity = StopDebugCommand - toolbar_placement = DebugActionPlacement.SESSION + toolbar_placement = SessionPanel toolbar_order = 20 label = "Stop debug session" tooltip = "Stop the active debug execution" @@ -320,8 +341,7 @@ class RandomSourceGroupDebugAction(PipelineDebugCommandActionDeclaration): __all__ = ( "ChooseSourceGroupDebugAction", "DebugActionDisabledReason", - "DebugActionPlacement", - "DebugToolbarAuxiliaryAction", + "DebugToolbarPlacement", "InspectRuntimeValuesAction", "PipelineDebugActionDeclarationBase", "PipelineDebugAuxiliaryActionDeclaration", diff --git a/tests/unit/pyqt_gui/test_debug_toolbar.py b/tests/unit/pyqt_gui/test_debug_toolbar.py index 5bf15c63f1..d4a042ea1e 100644 --- a/tests/unit/pyqt_gui/test_debug_toolbar.py +++ b/tests/unit/pyqt_gui/test_debug_toolbar.py @@ -29,10 +29,11 @@ from openhcs.core.steps.function_step import FunctionStep from openhcs.pyqt_gui.widgets.debug_toolbar import DebugToolbarWidget from openhcs.pyqt_gui.widgets.pipeline_editor import PipelineEditorWidget -from openhcs.pyqt_gui.widgets.shared.services.debug_session_projection import ( - PipelineDebugPauseBoundaryState, - PipelineDebugSessionContext, - PipelineDebugTargetState, +from openhcs.core.debug_session_projection import ( + DebugPauseBoundaryState, + DebugSessionProjectionContext, + DebugSessionTargetState, + TerminalCompleteDebugSessionPhase, ) from openhcs.pyqt_gui.widgets.shared.services.pipeline_debug_actions import ( PipelineDebugActionDeclarationBase, @@ -87,9 +88,9 @@ def debug_toolbar_context( terminal_summary: DebugTerminalSummary | None = None, manager_execution_state: ManagerExecutionState = ManagerExecutionState.IDLE, pause_step_indices: tuple[int, ...] = (1,), -) -> PipelineDebugSessionContext: - return PipelineDebugSessionContext( - target=PipelineDebugTargetState( +) -> DebugSessionProjectionContext: + return DebugSessionProjectionContext( + target=DebugSessionTargetState( current_plate_scope_id="plate", pipeline_scope_id="plate::pipeline", initialized=initialized, @@ -97,7 +98,7 @@ def debug_toolbar_context( ), session=session, terminal_summary=terminal_summary, - pause_boundaries=PipelineDebugPauseBoundaryState(pause_step_indices), + pause_boundaries=DebugPauseBoundaryState(pause_step_indices), manager_execution_state=manager_execution_state, ) @@ -536,7 +537,7 @@ def test_debug_context_prefers_terminal_state_over_a_stale_inspected_session( context = harness.editor.debug_session_context() assert context.active_session is None - assert context.phase.value == "terminal_complete" + assert context.phase is TerminalCompleteDebugSessionPhase assert context.terminal_summary is summary diff --git a/tests/unit/pyqt_gui/test_ui_agent_bridge.py b/tests/unit/pyqt_gui/test_ui_agent_bridge.py index 620d575856..c78a96ce22 100644 --- a/tests/unit/pyqt_gui/test_ui_agent_bridge.py +++ b/tests/unit/pyqt_gui/test_ui_agent_bridge.py @@ -208,7 +208,7 @@ from openhcs.pyqt_gui.widgets.pipeline_editor import PipelineEditorWidget from openhcs.pyqt_gui.widgets.plate_manager import PlateManagerWidget from openhcs.pyqt_gui.widgets.shared.services.pipeline_debug_actions import ( - DebugToolbarAuxiliaryAction, + InspectRuntimeValuesAction, PipelineDebugActionDeclarationBase, StepDebugAction, ) @@ -4489,7 +4489,7 @@ def test_pipeline_debug_toolbar_actions_are_exposed_from_toolbar_declarations() for action_id in ( RestartDebugCommand.name, StopDebugCommand.name, - DebugToolbarAuxiliaryAction.RUNTIME_VALUES.value, + InspectRuntimeValuesAction.action_id(), ): assert actions[action_id].enabled is False assert actions[action_id].disabled_error.code == "debug_session_required" @@ -4562,12 +4562,12 @@ def test_pipeline_debug_toolbar_runtime_values_action_requires_debug_session( ) bridge = _pipeline_editor_bridge(gui.pipeline_editor) widget_id = PipelineDebugToolbarWidgetIdentity.require_value() - action = _debug_actions(bridge)[DebugToolbarAuxiliaryAction.RUNTIME_VALUES.value] + action = _debug_actions(bridge)[InspectRuntimeValuesAction.action_id()] result = bridge.invoke_action( UiActionInvokeRequest( widget_id=widget_id, - action_id=DebugToolbarAuxiliaryAction.RUNTIME_VALUES.value, + action_id=InspectRuntimeValuesAction.action_id(), selected_scope_ids=action.target_scope_ids, observed_selection_revision_token=action.selection_revision_token, confirmation_requirement=UiBridgeConfirmationRequirement.from_flag( @@ -4622,7 +4622,7 @@ def test_pipeline_debug_session_state_surface_projects_context_and_actions() -> assert state.payload["cursor"]["step_scope_id"] == "plate-1::functionstep_1" assert actions[RestartDebugCommand.name]["enabled"] is True assert ( - actions[DebugToolbarAuxiliaryAction.RUNTIME_VALUES.value]["enabled"] is True + actions[InspectRuntimeValuesAction.action_id()]["enabled"] is True ) assert actions[StepDebugCommand.name]["label"] == StepDebugAction.label @@ -4735,5 +4735,5 @@ def test_pipeline_debug_session_terminal_summary_retires_the_inspected_session() assert actions[RestartDebugCommand.name]["enabled"] is False assert actions[StopDebugCommand.name]["enabled"] is False assert ( - actions[DebugToolbarAuxiliaryAction.RUNTIME_VALUES.value]["enabled"] is False + actions[InspectRuntimeValuesAction.action_id()]["enabled"] is False ) From b3ba85eec5a853090276390c5659f026bb7c34c1 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:36:12 -0400 Subject: [PATCH 10/36] E1: host platforms are a family; per-platform client and deployment leaves key by platform class AgentRuntimePlatformKey is deleted; the *AgentRuntimePlatformAuthority classes are the platforms (AgentRuntimePlatform, LinuxRuntimePlatform, ...), and Claude Desktop paths and desktop deployment select their leaf by nominal type. Co-Authored-By: Claude Opus 5.5 --- openhcs/agent/path_policy.py | 6 +- openhcs/agent/runtime_platform.py | 80 ++++++++----------- openhcs/agent/services/ui_bridge_service.py | 18 ++--- openhcs/desktop/deployment.py | 46 ++++++----- openhcs/mcp/client_registration.py | 38 +++++---- openhcs/mcp/dev_client_core.py | 4 +- openhcs/pyqt_gui/config.py | 6 +- openhcs/pyqt_gui/services/ui_bridge_server.py | 6 +- openhcs/utils/environment.py | 8 +- .../check_viewer_feature_measurement_live.py | 6 +- .../agent/test_mcp_client_registration.py | 27 ++++--- .../agent/test_mcp_distribution_runtime.py | 34 ++++---- tests/unit/agent/test_mcp_server.py | 14 ++-- .../unit/agent/test_mcp_skill_registration.py | 4 +- tests/unit/agent/test_ui_bridge_service.py | 12 +-- tests/unit/desktop/test_desktop_deployment.py | 25 +++--- tests/unit/test_process_environment.py | 6 +- 17 files changed, 165 insertions(+), 175 deletions(-) diff --git a/openhcs/agent/path_policy.py b/openhcs/agent/path_policy.py index 55759f5bfc..4be39cebf3 100644 --- a/openhcs/agent/path_policy.py +++ b/openhcs/agent/path_policy.py @@ -8,7 +8,7 @@ from pathlib import Path from openhcs.agent.exceptions import AgentFacingErrorMixin -from openhcs.agent.runtime_platform import AgentRuntimePlatformAuthority +from openhcs.agent.runtime_platform import AgentRuntimePlatform class AgentPathLocationAuthority: @@ -27,11 +27,11 @@ def source_checkout_root(cls) -> Path | None: @staticmethod def temporary_root() -> Path: - return AgentRuntimePlatformAuthority.current().temporary_root() + return AgentRuntimePlatform.current().temporary_root() @staticmethod def user_data_root() -> Path: - return AgentRuntimePlatformAuthority.current().application_data_root("OpenHCS") + return AgentRuntimePlatform.current().application_data_root("OpenHCS") @classmethod def default_output_root(cls) -> Path: diff --git a/openhcs/agent/runtime_platform.py b/openhcs/agent/runtime_platform.py index 859668f1e1..a14141ebfc 100644 --- a/openhcs/agent/runtime_platform.py +++ b/openhcs/agent/runtime_platform.py @@ -1,4 +1,4 @@ -"""Nominal host-platform authorities for headless agent runtime paths.""" +"""Host-platform families for headless agent runtime paths.""" from __future__ import annotations @@ -7,52 +7,40 @@ import tempfile from abc import ABC from collections.abc import Mapping -from enum import Enum from pathlib import Path from typing import ClassVar from metaclass_registry import AutoRegisterMeta -from metaclass_registry.strategies import EnumKeyedStrategyMixin +class AgentRuntimePlatform(ABC, metaclass=AutoRegisterMeta): + """Filesystem, process and session semantics of one host platform. -class AgentRuntimePlatformKey(str, Enum): - """Closed host-platform axis used by agent runtime authorities.""" + Each platform is a subclass; ``name`` is its spelling in reports. + """ - WINDOWS = "windows" - MACOS = "macos" - LINUX = "linux" - POSIX = "posix" - OTHER = "other" + __registry_key__ = "name" + __skip_if_no_key__ = True - @classmethod - def current(cls) -> "AgentRuntimePlatformKey": - """Return the nominal platform key for the running interpreter.""" + name: ClassVar[str | None] = None + + @staticmethod + def current_type() -> type["AgentRuntimePlatform"]: + """Return the platform class of the running interpreter.""" if sys.platform.startswith("win"): - return cls.WINDOWS + return WindowsRuntimePlatform if sys.platform == "darwin": - return cls.MACOS + return MacOSRuntimePlatform if sys.platform.startswith("linux"): - return cls.LINUX + return LinuxRuntimePlatform if os.name == "posix": - return cls.POSIX - return cls.OTHER - - -class AgentRuntimePlatformAuthority( - EnumKeyedStrategyMixin[AgentRuntimePlatformKey], - ABC, - metaclass=AutoRegisterMeta, -): - """Registered authority for platform-owned agent filesystem semantics.""" - - __enum_member_attr__ = "platform_key" - platform_key: ClassVar[AgentRuntimePlatformKey] + return PosixRuntimePlatform + return OtherRuntimePlatform @classmethod - def current(cls) -> "AgentRuntimePlatformAuthority": - """Instantiate the authority declared for the current platform.""" - return cls.for_enum_member(AgentRuntimePlatformKey.current()) + def current(cls) -> "AgentRuntimePlatform": + """Instantiate the platform of the running interpreter.""" + return cls.current_type()() @staticmethod def resolved_path(path: str | Path) -> Path: @@ -199,10 +187,10 @@ def application_runtime_dirs( return tuple(dict.fromkeys(candidates)) -class OtherAgentRuntimePlatformAuthority(AgentRuntimePlatformAuthority): +class OtherRuntimePlatform(AgentRuntimePlatform): """Portable conservative defaults for unknown host platforms.""" - platform_key = AgentRuntimePlatformKey.OTHER + name = "other" class PosixAgentRuntimePlatformMixin: @@ -271,22 +259,22 @@ def graphical_session_available( return True -class PosixAgentRuntimePlatformAuthority( +class PosixRuntimePlatform( PosixAgentRuntimePlatformMixin, - AgentRuntimePlatformAuthority, + AgentRuntimePlatform, ): """Registered authority for otherwise-unclassified POSIX hosts.""" - platform_key = AgentRuntimePlatformKey.POSIX + name = "posix" -class LinuxAgentRuntimePlatformAuthority( +class LinuxRuntimePlatform( PosixAgentRuntimePlatformMixin, - AgentRuntimePlatformAuthority, + AgentRuntimePlatform, ): """Linux runtime paths, including the standard per-user runtime root.""" - platform_key = AgentRuntimePlatformKey.LINUX + name = "linux" proc_root: ClassVar[Path] = Path("/proc") def _process_environment(self, pid: int) -> Mapping[str, str] | None: @@ -323,14 +311,14 @@ def _application_runtime_candidates( return tuple(candidates) -class MacOSAgentRuntimePlatformAuthority( +class MacOSRuntimePlatform( NativeGraphicalSessionMixin, PosixAgentRuntimePlatformMixin, - AgentRuntimePlatformAuthority, + AgentRuntimePlatform, ): """macOS application-data convention plus POSIX runtime ownership.""" - platform_key = AgentRuntimePlatformKey.MACOS + name = "macos" def application_data_root(self, application_name: str) -> Path: return ( @@ -338,13 +326,13 @@ def application_data_root(self, application_name: str) -> Path: ).resolve(strict=False) -class WindowsAgentRuntimePlatformAuthority( +class WindowsRuntimePlatform( NativeGraphicalSessionMixin, - AgentRuntimePlatformAuthority, + AgentRuntimePlatform, ): """Windows user-data and local runtime path conventions.""" - platform_key = AgentRuntimePlatformKey.WINDOWS + name = "windows" def child_process_environment_keys(self) -> tuple[str, ...]: """Preserve native Windows process, home, app-data, and temp roots.""" diff --git a/openhcs/agent/services/ui_bridge_service.py b/openhcs/agent/services/ui_bridge_service.py index 4e659db252..916c4ef155 100644 --- a/openhcs/agent/services/ui_bridge_service.py +++ b/openhcs/agent/services/ui_bridge_service.py @@ -99,7 +99,7 @@ UiWindowSnapshotResult, ) from openhcs.agent.path_policy import AgentPathPolicy, AgentPathPolicyError -from openhcs.agent.runtime_platform import AgentRuntimePlatformAuthority +from openhcs.agent.runtime_platform import AgentRuntimePlatform from openhcs.agent.services.object_state_field_projection import ( ObjectStateFieldListProjector, ) @@ -1046,9 +1046,9 @@ def descriptor_dirs(cls) -> tuple[Path, ...]: UiBridgeDescriptorEnvironment.descriptor_directory_path_key ) if configured: - return (AgentRuntimePlatformAuthority.resolved_path(configured),) + return (AgentRuntimePlatform.resolved_path(configured),) - return AgentRuntimePlatformAuthority.current().application_runtime_dirs( + return AgentRuntimePlatform.current().application_runtime_dirs( "OpenHCS", "ui-bridge", ) @@ -1509,7 +1509,7 @@ def _validate_descriptor_file_path(path: Path) -> None: stat_result = path.stat() if not stat.S_ISREG(stat_result.st_mode): raise PermissionError("UI bridge descriptor must be a regular file.") - platform_authority = AgentRuntimePlatformAuthority.current() + platform_authority = AgentRuntimePlatform.current() if not platform_authority.supports_posix_permissions(): return uid = platform_authority.current_user_id() @@ -1533,7 +1533,7 @@ def _validate_descriptor_file_path(path: Path) -> None: @staticmethod def _validate_descriptor_process(descriptor: UiBridgeDescriptorFile) -> None: - process_started_at_unix = AgentRuntimePlatformAuthority.process_started_at_unix( + process_started_at_unix = AgentRuntimePlatform.process_started_at_unix( descriptor.pid ) if process_started_at_unix is None: @@ -1556,7 +1556,7 @@ def selected_descriptor_path() -> Path | None: configured = environ.get(UiBridgeDescriptorEnvironment.descriptor_file_path_key) if not configured: return None - return AgentRuntimePlatformAuthority.resolved_path(configured) + return AgentRuntimePlatform.resolved_path(configured) @classmethod def live_descriptors(cls) -> tuple[UiBridgeDescriptorFile, ...]: @@ -1599,7 +1599,7 @@ def _read_descriptor_results(cls) -> tuple[UiBridgeDescriptorReadResult, ...]: ) for path in dict.fromkeys( - AgentRuntimePlatformAuthority.resolved_path(path) + AgentRuntimePlatform.resolved_path(path) for path in descriptor_paths ): result = UiBridgeDescriptorReader.read(path) @@ -1671,7 +1671,7 @@ def resolve( ) -> UiBridgeConnectionResolution: if connection.descriptor_file_path is not None: return self._resolve_explicit_file( - AgentRuntimePlatformAuthority.resolved_path( + AgentRuntimePlatform.resolved_path( connection.descriptor_file_path ), connection, @@ -1834,7 +1834,7 @@ def viewer_launch_context( connection: UiBridgeConnectionSpec = DEFAULT_UI_BRIDGE_CONNECTION_SPEC, ) -> ViewerLaunchContext: """Resolve one typed detached-viewer context from UI or current process.""" - platform_authority = AgentRuntimePlatformAuthority.current() + platform_authority = AgentRuntimePlatform.current() resolution = self._descriptor_resolver.resolve(connection) if resolution.ok and resolution.process_id is not None: environment = platform_authority.graphical_process_environment( diff --git a/openhcs/desktop/deployment.py b/openhcs/desktop/deployment.py index 9f5d609ad8..93e42c4f94 100644 --- a/openhcs/desktop/deployment.py +++ b/openhcs/desktop/deployment.py @@ -20,15 +20,16 @@ from typing import ClassVar from uuid import uuid4 -from metaclass_registry import AutoRegisterMeta +from metaclass_registry import AutoRegisterMeta, RegistryFamily, RegistryKeyAttribute from packaging.utils import canonicalize_name from openhcs import __version__ as OPENHCS_VERSION from openhcs.agent.runtime_platform import ( - AgentRuntimePlatformAuthority, - AgentRuntimePlatformKey, + AgentRuntimePlatform, + MacOSRuntimePlatform, + WindowsRuntimePlatform, ) -from metaclass_registry.strategies import EnumKeyedStrategyMixin +from metaclass_registry.strategies import NominalTypeKeyedStrategyMixin from openhcs.mcp.bootstrap import ( MCP_INSTALLATION_POINTER_ENVIRONMENT_VARIABLE, MCP_STABLE_LAUNCH_COMMAND_ENVIRONMENT_VARIABLE, @@ -155,7 +156,7 @@ def from_runtime( class DesktopDeploymentReport: """Paths refreshed by one platform's desktop deployment.""" - platform: AgentRuntimePlatformKey + platform: type[AgentRuntimePlatform] launcher_path: str desktop_shortcut_path: str application_path: str | None @@ -295,34 +296,33 @@ def _powershell_literal(value: str) -> str: class DesktopDeployment( - EnumKeyedStrategyMixin[AgentRuntimePlatformKey], + NominalTypeKeyedStrategyMixin, ABC, metaclass=AutoRegisterMeta, ): """Launcher, shortcut and icon files installed for one platform.""" - __enum_member_attr__ = "platform_key" - platform_key: ClassVar[AgentRuntimePlatformKey] + value_type: ClassVar[type[AgentRuntimePlatform] | None] = None + value_type_label: ClassVar[str | None] = None + __registry_family__ = RegistryFamily(RegistryKeyAttribute.VALUE_TYPE_LABEL) @classmethod def current(cls) -> "DesktopDeployment": - platform_key = AgentRuntimePlatformKey.current() - try: - return cls.for_enum_member(platform_key) - except KeyError as exc: + deployments = cls.strategy_types_for_nominal_type( + AgentRuntimePlatform.current_type() + ) + if not deployments: raise DesktopDeploymentError( "Native desktop deployment refresh is supported only for the " "Windows and macOS installers." - ) from exc + ) + return deployments[0]() @classmethod def numba_cache_path(cls) -> Path: """Return this platform's user-local compiled-code cache.""" - runtime_platform = AgentRuntimePlatformAuthority.for_enum_member( - cls.platform_key - ) - return OpenHCSProcessEnvironment.numba_cache_path(runtime_platform) + return OpenHCSProcessEnvironment.numba_cache_path(cls.value_type()) @abstractmethod def refresh( @@ -344,7 +344,7 @@ def update_candidate( class WindowsDesktopDeployment(DesktopDeployment): """Windows native GUI, stable MCP launcher, and Shell Link.""" - platform_key = AgentRuntimePlatformKey.WINDOWS + value_type = WindowsRuntimePlatform _application_launcher_name = "OpenHCS.exe" _current_environment_pointer_name = "current-environment" _launcher_fingerprint_name = "OpenHCS-launcher-fingerprint.json" @@ -917,7 +917,7 @@ def refresh( for candidate in candidates: _remove_path(candidate) return DesktopDeploymentReport( - platform=self.platform_key, + platform=self.value_type, launcher_path=str(context.installation_pointer), desktop_shortcut_path=str(shortcut_path), application_path=str(application_launcher_path), @@ -929,7 +929,7 @@ def refresh( class MacOSDesktopDeployment(DesktopDeployment): """macOS environment launcher, app bundle, and Desktop link.""" - platform_key = AgentRuntimePlatformKey.MACOS + value_type = MacOSRuntimePlatform def update_candidate( self, @@ -1101,7 +1101,7 @@ def refresh( for candidate in candidates: _discard_transaction_path(candidate) return DesktopDeploymentReport( - platform=self.platform_key, + platform=self.value_type, launcher_path=str(environment_launcher), desktop_shortcut_path=str(desktop_link), application_path=str(application_path), @@ -1174,7 +1174,9 @@ def main(argv: list[str] | None = None) -> int: file=sys.stderr, ) if arguments.json: - print(json.dumps(asdict(report), sort_keys=True)) + from python_introspect import to_jsonable + + print(json.dumps(to_jsonable(report), sort_keys=True)) else: print(f"Refreshed OpenHCS desktop integration: {report.launcher_path}") return 0 diff --git a/openhcs/mcp/client_registration.py b/openhcs/mcp/client_registration.py index 39aa5cdc49..f84a9ca6e6 100644 --- a/openhcs/mcp/client_registration.py +++ b/openhcs/mcp/client_registration.py @@ -21,11 +21,14 @@ from typing import Any, ClassVar import tomlkit -from metaclass_registry import AutoRegisterMeta -from python_introspect import to_jsonable +from metaclass_registry import AutoRegisterMeta, RegistryFamily, RegistryKeyAttribute from tomlkit.items import InlineTable -from openhcs.agent.runtime_platform import AgentRuntimePlatformKey +from openhcs.agent.runtime_platform import ( + AgentRuntimePlatform, + MacOSRuntimePlatform, + WindowsRuntimePlatform, +) from openhcs.agent.skill_bundle import path_is_redirected from openhcs.agent.skill_sync import ( SkillSyncPartialFailure, @@ -34,7 +37,7 @@ installed_skill_bundle, sync_skills, ) -from metaclass_registry.strategies import EnumKeyedStrategyMixin +from metaclass_registry.strategies import NominalTypeKeyedStrategyMixin CLIENT_REGISTRATION_SCHEMA_VERSION = "openhcs.mcp.client-registration.v1" OPENHCS_MCP_SERVER_NAME = "openhcs" @@ -106,7 +109,7 @@ class ClientRegistrationEnvironment: home: Path environ: Mapping[str, str] - platform_key: AgentRuntimePlatformKey + platform: type[AgentRuntimePlatform] executable_resolver: ExecutableResolver process_runner: ProcessRunner @@ -116,7 +119,7 @@ def current(cls) -> "ClientRegistrationEnvironment": return cls( home=Path.home(), environ=dict(os.environ), - platform_key=AgentRuntimePlatformKey.current(), + platform=AgentRuntimePlatform.current_type(), executable_resolver=shutil.which, process_runner=subprocess.run, ) @@ -131,14 +134,15 @@ def executable(self, candidates: Sequence[str]) -> str | None: class ClaudeDesktopPlatformSemanticsABC( - EnumKeyedStrategyMixin[AgentRuntimePlatformKey], + NominalTypeKeyedStrategyMixin, ABC, metaclass=AutoRegisterMeta, ): """Registered owner of Claude Desktop paths for one supported platform.""" - __enum_member_attr__ = "platform_key" - platform_key: ClassVar[AgentRuntimePlatformKey] + value_type: ClassVar[type[AgentRuntimePlatform] | None] = None + value_type_label: ClassVar[str | None] = None + __registry_family__ = RegistryFamily(RegistryKeyAttribute.VALUE_TYPE_LABEL) @abstractmethod def config_path(self, environment: ClientRegistrationEnvironment) -> Path: @@ -164,7 +168,7 @@ def desktop_app_installed( class WindowsClaudeDesktopPlatformSemantics(ClaudeDesktopPlatformSemanticsABC): """Claude Desktop configuration and installation paths on Windows.""" - platform_key = AgentRuntimePlatformKey.WINDOWS + value_type = WindowsRuntimePlatform def config_path(self, environment: ClientRegistrationEnvironment) -> Path: app_data = environment.environ.get("APPDATA") @@ -218,7 +222,7 @@ def desktop_app_installed( class MacOSClaudeDesktopPlatformSemantics(ClaudeDesktopPlatformSemanticsABC): """Claude Desktop configuration and installation paths on macOS.""" - platform_key = AgentRuntimePlatformKey.MACOS + value_type = MacOSRuntimePlatform def config_path(self, environment: ClientRegistrationEnvironment) -> Path: return ( @@ -358,6 +362,8 @@ def from_results( def as_dict(self) -> dict[str, Any]: """Return a JSON-native installer payload.""" + from python_introspect import to_jsonable + return dict(to_jsonable(self)) @@ -553,12 +559,10 @@ class ClaudeDesktopClientRegistrationTarget(McpClientRegistrationTarget): def _platform_semantics( environment: ClientRegistrationEnvironment, ) -> ClaudeDesktopPlatformSemanticsABC | None: - try: - return ClaudeDesktopPlatformSemanticsABC.for_enum_member( - environment.platform_key - ) - except KeyError: - return None + semantics = ClaudeDesktopPlatformSemanticsABC.strategy_types_for_nominal_type( + environment.platform + ) + return semantics[0]() if semantics else None @classmethod def config_path( diff --git a/openhcs/mcp/dev_client_core.py b/openhcs/mcp/dev_client_core.py index e975db7572..6afabc45ee 100644 --- a/openhcs/mcp/dev_client_core.py +++ b/openhcs/mcp/dev_client_core.py @@ -54,7 +54,7 @@ UiStateSurfaceRequest, ) from openhcs.agent.path_policy import AgentPathPolicy -from openhcs.agent.runtime_platform import AgentRuntimePlatformAuthority +from openhcs.agent.runtime_platform import AgentRuntimePlatform from openhcs.agent.ui_bridge_environment import UiBridgeDescriptorEnvironment from openhcs.constants.constants import OrchestratorState from openhcs.core.execution_state import ( @@ -309,7 +309,7 @@ class McpDevServerSpec: def environment(self) -> dict[str, str]: """Environment entries inherited by the fresh MCP subprocess.""" return ( - AgentRuntimePlatformAuthority.current().project_child_process_environment( + AgentRuntimePlatform.current().project_child_process_environment( os.environ, include_graphical_session=True, additional_keys=( diff --git a/openhcs/pyqt_gui/config.py b/openhcs/pyqt_gui/config.py index eca20f9cc5..39cc4f0dfa 100644 --- a/openhcs/pyqt_gui/config.py +++ b/openhcs/pyqt_gui/config.py @@ -34,7 +34,7 @@ from zmqruntime.transport import get_default_transport_mode from openhcs.agent.dto.execution import ExecutionConnectionSpec -from openhcs.agent.runtime_platform import AgentRuntimePlatformAuthority +from openhcs.agent.runtime_platform import AgentRuntimePlatform from openhcs.agent.services.ui_bridge_service import ( UiBridgeDescriptorDirectoryAuthority, ) @@ -306,7 +306,7 @@ def resolve_auth_token(self) -> str: def descriptor_path_for(self, bridge_instance_id: str) -> Path: if self.descriptor_file_path is not None: - return AgentRuntimePlatformAuthority.resolved_path( + return AgentRuntimePlatform.resolved_path( self.descriptor_file_path ) return ( @@ -316,7 +316,7 @@ def descriptor_path_for(self, bridge_instance_id: str) -> Path: def descriptor_directory_or_default(self) -> Path: if self.descriptor_directory_path is not None: - return AgentRuntimePlatformAuthority.resolved_path( + return AgentRuntimePlatform.resolved_path( self.descriptor_directory_path ) return UiBridgeDescriptorDirectoryAuthority.default_descriptor_dir() diff --git a/openhcs/pyqt_gui/services/ui_bridge_server.py b/openhcs/pyqt_gui/services/ui_bridge_server.py index 38c39630ac..b57e350fdd 100644 --- a/openhcs/pyqt_gui/services/ui_bridge_server.py +++ b/openhcs/pyqt_gui/services/ui_bridge_server.py @@ -42,7 +42,7 @@ UiBridgeRequestEnvelope, UiBridgeResponseEnvelope, ) -from openhcs.agent.runtime_platform import AgentRuntimePlatformAuthority +from openhcs.agent.runtime_platform import AgentRuntimePlatform from openhcs.agent.services.ui_bridge_service import ( UI_BRIDGE_PROTOCOL_VERSION, UiBridgeOperation, @@ -525,7 +525,7 @@ def _require_available_endpoint(self, endpoint: TransportEndpoint) -> None: raise UiBridgeEndpointInUseError(endpoint, occupied_ports) def _write_descriptor_file(self, descriptor: UiBridgeDescriptorFile) -> Path: - path = AgentRuntimePlatformAuthority.resolved_path( + path = AgentRuntimePlatform.resolved_path( descriptor.descriptor_file_path ) path.parent.mkdir(parents=True, mode=0o700, exist_ok=True) @@ -539,7 +539,7 @@ def _write_descriptor_file(self, descriptor: UiBridgeDescriptorFile) -> Path: + "\n", encoding="utf-8", ) - if AgentRuntimePlatformAuthority.current().supports_posix_permissions(): + if AgentRuntimePlatform.current().supports_posix_permissions(): path.chmod(0o600) return path diff --git a/openhcs/utils/environment.py b/openhcs/utils/environment.py index ace4eb8623..de0b92e4ca 100644 --- a/openhcs/utils/environment.py +++ b/openhcs/utils/environment.py @@ -13,7 +13,7 @@ from openhcs.resources.brand import BRAND_PRODUCT_NAME if TYPE_CHECKING: - from openhcs.agent.runtime_platform import AgentRuntimePlatformAuthority + from openhcs.agent.runtime_platform import AgentRuntimePlatform class OpenHCSProcessEnvironment: @@ -32,7 +32,7 @@ class OpenHCSProcessEnvironment: @staticmethod def numba_cache_path( - platform_authority: "AgentRuntimePlatformAuthority", + platform_authority: "AgentRuntimePlatform", ) -> Path: """Return the user-local cache path for compiled Numba artifacts.""" @@ -44,9 +44,9 @@ def numba_cache_path( def current_numba_cache_path(cls) -> Path: """Return the compiled-code cache for the current host platform.""" - from openhcs.agent.runtime_platform import AgentRuntimePlatformAuthority + from openhcs.agent.runtime_platform import AgentRuntimePlatform - return cls.numba_cache_path(AgentRuntimePlatformAuthority.current()) + return cls.numba_cache_path(AgentRuntimePlatform.current()) @classmethod def child_process_environment_keys(cls) -> tuple[str, ...]: diff --git a/tests/diagnostics/check_viewer_feature_measurement_live.py b/tests/diagnostics/check_viewer_feature_measurement_live.py index dbf7f36812..15dac4e33e 100644 --- a/tests/diagnostics/check_viewer_feature_measurement_live.py +++ b/tests/diagnostics/check_viewer_feature_measurement_live.py @@ -201,7 +201,7 @@ def save() -> None: from openhcs.core.streaming_config_declarations import NapariViewer from zmqruntime.config import TransportMode from python_introspect import to_jsonable - from openhcs.agent.runtime_platform import AgentRuntimePlatformAuthority + from openhcs.agent.runtime_platform import AgentRuntimePlatform from polystore.imagej_distribution import ( FijiArchiveDistribution, ImageJArchiveDownloadPolicy, @@ -309,7 +309,7 @@ def call(name: str, arguments: dict, *, expect_error: bool = False) -> dict: owned_identity = heartbeat.process_identity receipt["viewer_identity"] = to_jsonable(owned_identity) receipt["heartbeat"] = to_jsonable(heartbeat) - child_environment = AgentRuntimePlatformAuthority.current().graphical_process_environment( + child_environment = AgentRuntimePlatform.current().graphical_process_environment( owned_identity.pid, additional_keys=( FijiArchiveDistribution.cache_root_environment_key, @@ -328,7 +328,7 @@ def call(name: str, arguments: dict, *, expect_error: bool = False) -> dict: receipt["child_policy"] = { "imagej_cache_root": str(child_cache), "downloads_allowed": child_downloads, - "owner": "AgentRuntimePlatformAuthority process environment + FijiArchiveDistribution/ImageJArchiveDownloadPolicy", + "owner": "AgentRuntimePlatform process environment + FijiArchiveDistribution/ImageJArchiveDownloadPolicy", } save() payloads = call( diff --git a/tests/unit/agent/test_mcp_client_registration.py b/tests/unit/agent/test_mcp_client_registration.py index 712b7f4c43..1f4b824f1f 100644 --- a/tests/unit/agent/test_mcp_client_registration.py +++ b/tests/unit/agent/test_mcp_client_registration.py @@ -7,7 +7,12 @@ import pytest import tomlkit -from openhcs.agent.runtime_platform import AgentRuntimePlatformKey +from openhcs.agent.runtime_platform import ( + AgentRuntimePlatform, + LinuxRuntimePlatform, + MacOSRuntimePlatform, + WindowsRuntimePlatform, +) from openhcs.mcp import client_registration from openhcs.mcp.client_registration import ( CLIENT_REGISTRATION_SCHEMA_VERSION, @@ -31,7 +36,7 @@ def _environment( home: Path, *, - platform_key: AgentRuntimePlatformKey = AgentRuntimePlatformKey.LINUX, + platform: type[AgentRuntimePlatform] = LinuxRuntimePlatform, environ: dict[str, str] | None = None, executables: dict[str, str] | None = None, process_runner=None, @@ -51,7 +56,7 @@ def default_runner( return ClientRegistrationEnvironment( home=home, environ=environ or {}, - platform_key=platform_key, + platform=platform, executable_resolver=resolve, process_runner=process_runner or default_runner, ) @@ -429,15 +434,15 @@ def test_detected_failure_preserves_success_and_does_not_fail_required_contract( @pytest.mark.parametrize( - ("platform_key", "environ", "expected_relative"), + ("platform", "environ", "expected_relative"), ( ( - AgentRuntimePlatformKey.WINDOWS, + WindowsRuntimePlatform, {"APPDATA": "APP_DATA"}, Path("APP_DATA/Claude/claude_desktop_config.json"), ), ( - AgentRuntimePlatformKey.MACOS, + MacOSRuntimePlatform, {}, Path("Library/Application Support/Claude/claude_desktop_config.json"), ), @@ -445,7 +450,7 @@ def test_detected_failure_preserves_success_and_does_not_fail_required_contract( ) def test_claude_config_path_is_owned_by_platform_specific_leaf_hook( tmp_path: Path, - platform_key: AgentRuntimePlatformKey, + platform: type[AgentRuntimePlatform], environ: dict[str, str], expected_relative: Path, ) -> None: @@ -453,7 +458,7 @@ def test_claude_config_path_is_owned_by_platform_specific_leaf_hook( environ = {"APPDATA": str(tmp_path / environ["APPDATA"])} environment = _environment( tmp_path, - platform_key=platform_key, + platform=platform, environ=environ, ) @@ -470,7 +475,7 @@ def test_claude_windows_leaf_owns_desktop_installation_forms( program_files = tmp_path / "ProgramFiles" environment = _environment( tmp_path, - platform_key=AgentRuntimePlatformKey.WINDOWS, + platform=WindowsRuntimePlatform, environ={ "LOCALAPPDATA": str(local_app_data), "PROGRAMFILES": str(program_files), @@ -491,7 +496,7 @@ def test_claude_windows_leaf_owns_desktop_installation_forms( def test_claude_desktop_is_explicitly_unsupported_on_linux(tmp_path: Path) -> None: environment = _environment( tmp_path, - platform_key=AgentRuntimePlatformKey.LINUX, + platform=LinuxRuntimePlatform, ) assert ClaudeDesktopClientRegistrationTarget.config_path(environment) is None @@ -504,7 +509,7 @@ def test_claude_code_cli_alone_does_not_detect_claude_desktop( ) -> None: environment = _environment( tmp_path, - platform_key=AgentRuntimePlatformKey.MACOS, + platform=MacOSRuntimePlatform, executables={"claude": str(tmp_path / ".local" / "bin" / "claude")}, ) diff --git a/tests/unit/agent/test_mcp_distribution_runtime.py b/tests/unit/agent/test_mcp_distribution_runtime.py index bf6a3f38c9..40da11e2fc 100644 --- a/tests/unit/agent/test_mcp_distribution_runtime.py +++ b/tests/unit/agent/test_mcp_distribution_runtime.py @@ -9,9 +9,9 @@ from openhcs.agent.dto.common import SCHEMA_VERSION from openhcs.agent.runtime_platform import ( - AgentRuntimePlatformAuthority, - AgentRuntimePlatformKey, - LinuxAgentRuntimePlatformAuthority, + AgentRuntimePlatform, + LinuxRuntimePlatform, + WindowsRuntimePlatform, ) from openhcs.agent.services.ui_bridge_service import ( UI_BRIDGE_PROTOCOL_VERSION, @@ -53,9 +53,9 @@ def test_windows_descriptor_default_uses_local_application_data( monkeypatch.delenv("OPENHCS_UI_BRIDGE_DESCRIPTOR_DIR", raising=False) monkeypatch.setenv("LOCALAPPDATA", str(local_application_data)) monkeypatch.setattr( - AgentRuntimePlatformKey, - "current", - classmethod(lambda cls: cls.WINDOWS), + AgentRuntimePlatform, + "current_type", + staticmethod(lambda: WindowsRuntimePlatform), ) descriptor_dirs = UiBridgeDescriptorDirectoryAuthority.descriptor_dirs() @@ -65,12 +65,8 @@ def test_windows_descriptor_default_uses_local_application_data( ).resolve(strict=False) -def test_linux_platform_authority_is_registered_under_linux_key(): - authority = AgentRuntimePlatformAuthority.for_enum_member( - AgentRuntimePlatformKey.LINUX - ) - - assert isinstance(authority, LinuxAgentRuntimePlatformAuthority) +def test_linux_platform_is_registered_under_its_name(): + assert AgentRuntimePlatform.__registry__["linux"] is LinuxRuntimePlatform def test_linux_platform_authority_projects_only_declared_graphical_environment( @@ -84,7 +80,7 @@ def test_linux_platform_authority_projects_only_declared_graphical_environment( b"XDG_RUNTIME_DIR=/run/user/1000\0PATH=/usr/bin\0" b"OPENHCS_CPU_ONLY=true\0SECRET_TOKEN=do-not-forward\0" ) - authority = LinuxAgentRuntimePlatformAuthority() + authority = LinuxRuntimePlatform() monkeypatch.setattr(authority, "proc_root", tmp_path) environment = authority.graphical_process_environment( @@ -103,7 +99,7 @@ def test_linux_platform_authority_projects_only_declared_graphical_environment( def test_posix_platform_authority_preserves_declared_temporary_root(): - authority = LinuxAgentRuntimePlatformAuthority() + authority = LinuxRuntimePlatform() environment = authority.project_child_process_environment( { @@ -122,7 +118,7 @@ def test_posix_platform_authority_preserves_declared_temporary_root(): def test_linux_platform_authority_owns_graphical_session_availability(): - authority = LinuxAgentRuntimePlatformAuthority() + authority = LinuxRuntimePlatform() assert authority.graphical_session_available({"DISPLAY": ":7"}) is True assert ( @@ -141,16 +137,14 @@ def test_descriptor_reader_uses_cross_platform_process_and_permission_authority( descriptor_path = tmp_path / "ui_bridge_windows.json" _write_descriptor(descriptor_path) descriptor_path.chmod(0o644) - windows_authority = AgentRuntimePlatformAuthority.for_enum_member( - AgentRuntimePlatformKey.WINDOWS - ) + windows_authority = WindowsRuntimePlatform() monkeypatch.setattr( - AgentRuntimePlatformAuthority, + AgentRuntimePlatform, "current", classmethod(lambda cls: windows_authority), ) monkeypatch.setattr( - AgentRuntimePlatformAuthority, + AgentRuntimePlatform, "process_started_at_unix", staticmethod(lambda pid: 1.0 if pid == os.getpid() else None), ) diff --git a/tests/unit/agent/test_mcp_server.py b/tests/unit/agent/test_mcp_server.py index 6cf70f5288..31c58bee11 100644 --- a/tests/unit/agent/test_mcp_server.py +++ b/tests/unit/agent/test_mcp_server.py @@ -8484,12 +8484,12 @@ async def call_health_through_dev_client(): def test_mcp_dev_client_server_spec_preserves_gui_session_environment(monkeypatch): import openhcs.mcp.dev_client as dev_client from openhcs.agent.path_policy import AgentPathPolicy - from openhcs.agent.runtime_platform import AgentRuntimePlatformAuthority + from openhcs.agent.runtime_platform import AgentRuntimePlatform from openhcs.agent.ui_bridge_environment import UiBridgeDescriptorEnvironment from openhcs.core.native_threading import native_thread_count_environment_keys from openhcs.utils.environment import OpenHCSProcessEnvironment - platform_authority = AgentRuntimePlatformAuthority.current() + platform_authority = AgentRuntimePlatform.current() for key in platform_authority.graphical_session_environment_keys(): monkeypatch.delenv(key, raising=False) for key in AgentPathPolicy.environment_keys(): @@ -8560,13 +8560,11 @@ def test_mcp_dev_client_server_spec_preserves_gui_session_environment(monkeypatc def test_mcp_dev_client_server_spec_preserves_windows_host_environment(monkeypatch): import openhcs.mcp.dev_client as dev_client from openhcs.agent.runtime_platform import ( - AgentRuntimePlatformAuthority, - AgentRuntimePlatformKey, + AgentRuntimePlatform, + WindowsRuntimePlatform, ) - windows_authority = AgentRuntimePlatformAuthority.for_enum_member( - AgentRuntimePlatformKey.WINDOWS - ) + windows_authority = WindowsRuntimePlatform() expected_keys = { "HOME", "LOGNAME", @@ -8587,7 +8585,7 @@ def test_mcp_dev_client_server_spec_preserves_windows_host_environment(monkeypat } assert set(windows_authority.child_process_environment_keys()) == expected_keys monkeypatch.setattr( - AgentRuntimePlatformAuthority, + AgentRuntimePlatform, "current", classmethod(lambda cls: windows_authority), ) diff --git a/tests/unit/agent/test_mcp_skill_registration.py b/tests/unit/agent/test_mcp_skill_registration.py index 7e54f88035..eee330b598 100644 --- a/tests/unit/agent/test_mcp_skill_registration.py +++ b/tests/unit/agent/test_mcp_skill_registration.py @@ -7,7 +7,7 @@ import pytest from openhcs.agent import skill_sync -from openhcs.agent.runtime_platform import AgentRuntimePlatformKey +from openhcs.agent.runtime_platform import LinuxRuntimePlatform from openhcs.agent.skill_bundle import AgentSkillBundle from openhcs.agent.skill_sync import SkillSyncReceipt, sync_skills from openhcs.mcp import client_registration @@ -43,7 +43,7 @@ def no_process(*args, **kwargs): return ClientRegistrationEnvironment( home=tmp_path / "home", environ={"CODEX_HOME": str(tmp_path / "custom-codex")}, - platform_key=AgentRuntimePlatformKey.LINUX, + platform=LinuxRuntimePlatform, executable_resolver=lambda _: None, process_runner=no_process, ) diff --git a/tests/unit/agent/test_ui_bridge_service.py b/tests/unit/agent/test_ui_bridge_service.py index 986576bdf2..1b1d2b5cb4 100644 --- a/tests/unit/agent/test_ui_bridge_service.py +++ b/tests/unit/agent/test_ui_bridge_service.py @@ -91,7 +91,7 @@ UiWindowSnapshotResult, UiWindowSummary, ) -from openhcs.agent.runtime_platform import AgentRuntimePlatformAuthority +from openhcs.agent.runtime_platform import AgentRuntimePlatform from openhcs.agent.services.ui_bridge_service import ( UI_BRIDGE_PROTOCOL_VERSION, UiBridgeApplyDocumentOperation, @@ -880,7 +880,7 @@ def test_ui_bridge_service_resolves_projected_graphical_viewer_launch_context( monkeypatch.setattr( "openhcs.agent.runtime_platform." - "LinuxAgentRuntimePlatformAuthority._process_environment", + "LinuxRuntimePlatform._process_environment", lambda self, pid: ( { "DISPLAY": ":19", @@ -1015,7 +1015,7 @@ def test_descriptor_resolver_rejects_dead_process(monkeypatch, tmp_path): ).write() monkeypatch.setenv("OPENHCS_UI_BRIDGE_DESCRIPTOR", str(descriptor_path)) monkeypatch.setattr( - AgentRuntimePlatformAuthority, + AgentRuntimePlatform, "process_started_at_unix", staticmethod(lambda _pid: None), ) @@ -1054,7 +1054,7 @@ def test_descriptor_catalog_removes_dead_process_before_application_compatibilit classmethod(lambda cls: ()), ) monkeypatch.setattr( - AgentRuntimePlatformAuthority, + AgentRuntimePlatform, "process_started_at_unix", staticmethod(lambda _pid: None), ) @@ -1077,7 +1077,7 @@ def test_descriptor_resolver_rejects_reused_process_identity(monkeypatch, tmp_pa descriptor_path.write_text(json.dumps(payload), encoding="utf-8") monkeypatch.setenv("OPENHCS_UI_BRIDGE_DESCRIPTOR", str(descriptor_path)) monkeypatch.setattr( - AgentRuntimePlatformAuthority, + AgentRuntimePlatform, "process_started_at_unix", staticmethod(lambda pid: 20.0 if pid == os.getpid() else None), ) @@ -1104,7 +1104,7 @@ def test_status_rechecks_descriptor_process_liveness_after_gateway_call( process_start_times = iter((descriptor_started_at - 1.0, None)) monkeypatch.setenv("OPENHCS_UI_BRIDGE_DESCRIPTOR", str(descriptor_path)) monkeypatch.setattr( - AgentRuntimePlatformAuthority, + AgentRuntimePlatform, "process_started_at_unix", staticmethod(lambda _pid: next(process_start_times)), ) diff --git a/tests/unit/desktop/test_desktop_deployment.py b/tests/unit/desktop/test_desktop_deployment.py index a42f52c6dd..ab041789b8 100644 --- a/tests/unit/desktop/test_desktop_deployment.py +++ b/tests/unit/desktop/test_desktop_deployment.py @@ -11,7 +11,10 @@ import pytest from openhcs.desktop import deployment as desktop_deployment -from openhcs.agent.runtime_platform import AgentRuntimePlatformKey +from openhcs.agent.runtime_platform import ( + MacOSRuntimePlatform, + WindowsRuntimePlatform, +) from openhcs.desktop.deployment import ( DesktopDeployment, DesktopDeploymentContext, @@ -33,7 +36,7 @@ def test_optional_skill_failure_retains_published_restart_target( tmp_path, monkeypatch, capsys ): report = DesktopDeploymentReport( - platform=AgentRuntimePlatformKey.MACOS, + platform=MacOSRuntimePlatform, launcher_path=str(tmp_path / "launcher"), desktop_shortcut_path=str(tmp_path / "shortcut"), application_path=None, @@ -73,7 +76,7 @@ def test_desktop_refresh_runs_skill_updates_only_after_successful_publication( context = _context(tmp_path, pointer_name="current") events = [] report = DesktopDeploymentReport( - platform=AgentRuntimePlatformKey.MACOS, + platform=MacOSRuntimePlatform, launcher_path=str(tmp_path / "launcher"), desktop_shortcut_path=str(tmp_path / "shortcut"), application_path=None, @@ -135,7 +138,7 @@ def test_native_setup_can_explicitly_skip_receipt_enrolled_refresh( ): kwargs = {} report = DesktopDeploymentReport( - platform=AgentRuntimePlatformKey.MACOS, + platform=MacOSRuntimePlatform, launcher_path=str(tmp_path / "launcher"), desktop_shortcut_path=str(tmp_path / "shortcut"), application_path=None, @@ -167,7 +170,7 @@ def test_unexpected_optional_refresh_error_cannot_invalidate_publication( ): context = _context(tmp_path, pointer_name="current") report = DesktopDeploymentReport( - platform=AgentRuntimePlatformKey.MACOS, + platform=MacOSRuntimePlatform, launcher_path=str(tmp_path / "launcher"), desktop_shortcut_path=str(tmp_path / "shortcut"), application_path=None, @@ -237,16 +240,12 @@ def _context(tmp_path: Path, *, pointer_name: str) -> DesktopDeploymentContext: def test_desktop_deployment_platforms_use_the_registered_host_axis() -> None: assert ( - DesktopDeployment.strategy_type_for_enum_member( - AgentRuntimePlatformKey.WINDOWS - ) - is WindowsDesktopDeployment + DesktopDeployment.strategy_types_for_nominal_type(WindowsRuntimePlatform) + == (WindowsDesktopDeployment,) ) assert ( - DesktopDeployment.strategy_type_for_enum_member( - AgentRuntimePlatformKey.MACOS - ) - is MacOSDesktopDeployment + DesktopDeployment.strategy_types_for_nominal_type(MacOSRuntimePlatform) + == (MacOSDesktopDeployment,) ) diff --git a/tests/unit/test_process_environment.py b/tests/unit/test_process_environment.py index 7aa01bdba6..73a0f47071 100644 --- a/tests/unit/test_process_environment.py +++ b/tests/unit/test_process_environment.py @@ -5,7 +5,7 @@ ImageJArchiveDownloadPolicy, ) -from openhcs.agent.runtime_platform import WindowsAgentRuntimePlatformAuthority +from openhcs.agent.runtime_platform import WindowsRuntimePlatform from openhcs.resources.brand import BRAND_PRODUCT_NAME from openhcs.utils.environment import OpenHCSProcessEnvironment @@ -29,7 +29,7 @@ def test_process_environment_owns_numba_cache_location(monkeypatch, tmp_path) -> monkeypatch.setenv("LOCALAPPDATA", str(local_data)) cache_path = OpenHCSProcessEnvironment.numba_cache_path( - WindowsAgentRuntimePlatformAuthority() + WindowsRuntimePlatform() ) assert cache_path == (local_data / BRAND_PRODUCT_NAME / "numba").resolve() @@ -41,7 +41,7 @@ def test_windows_numba_cache_preserves_legacy_path_budget( local_data = tmp_path / "local-data" monkeypatch.setenv("LOCALAPPDATA", str(local_data)) cache_path = OpenHCSProcessEnvironment.numba_cache_path( - WindowsAgentRuntimePlatformAuthority() + WindowsRuntimePlatform() ) relative_cache = cache_path.relative_to(local_data.resolve()) windows_cache = PureWindowsPath( From d32c8331cad46995f438715d213b07d6626055ad Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:37:56 -0400 Subject: [PATCH 11/36] E1: dataset and store identity use the one source path identity source_projection, dataset_sources and the Bio-Formats adapter compared resolve() results, a second identity rule beside K4's lexical one. They now use absolute_source_path (the lexical identity made absolute, no symlink following). A plate reached through a symlink alias is a different dataset root; the registration test now checks reuse for a lexically equal spelling. Co-Authored-By: Claude Opus 5.5 --- openhcs/core/dataset_sources/plane_stores.py | 3 +- openhcs/core/dataset_sources/source.py | 7 ++-- openhcs/core/source_path_identity.py | 10 ++++++ openhcs/core/source_projection.py | 10 +++--- openhcs/microscopes/bioformats_adapter.py | 32 +++++++++---------- ...t_microscope_virtual_workspace_metadata.py | 6 ++-- 6 files changed, 40 insertions(+), 28 deletions(-) diff --git a/openhcs/core/dataset_sources/plane_stores.py b/openhcs/core/dataset_sources/plane_stores.py index 3bee3bf608..27c7c17ec8 100644 --- a/openhcs/core/dataset_sources/plane_stores.py +++ b/openhcs/core/dataset_sources/plane_stores.py @@ -20,6 +20,7 @@ from openhcs.core.dataset_sources.discovery import domain_registry_config from openhcs.core.config import SourceBindingsConfig from openhcs.core.source_matching import merge_source_metadata +from openhcs.core.source_path_identity import absolute_source_path from openhcs.core.source_metadata import SourceMetadataMapping from openhcs.core.source_projection import ( SourceCandidate, @@ -100,7 +101,7 @@ def discover_dataset( cls, root: str | Path, *, source_bindings: SourceBindingsConfig | None = None ) -> SourcePlaneDataset: """Aggregate every registered store's planes under ``root``.""" - root_path = Path(root).resolve(strict=False) + root_path = absolute_source_path(root) if not root_path.exists(): raise PlaneStoreUnavailableError( f"Plane-store collection does not exist: {root_path}" diff --git a/openhcs/core/dataset_sources/source.py b/openhcs/core/dataset_sources/source.py index 5f84095275..9d48b6def9 100644 --- a/openhcs/core/dataset_sources/source.py +++ b/openhcs/core/dataset_sources/source.py @@ -24,6 +24,7 @@ from openhcs.core.axes import Axis from openhcs.core.dataset_sources.choice import DatasetSourceChoice from openhcs.core.dataset_sources.discovery import domain_registry_config +from openhcs.core.source_path_identity import absolute_source_path from openhcs.core.dataset_sources.interfaces import ( FilenameParser, MetadataHandler, @@ -360,17 +361,17 @@ def _register_virtual_workspace_backend( from openhcs.core.virtual_workspace_metadata import METADATA_CONFIG - resolved_root = Path(dataset_root).resolve() + root = absolute_source_path(dataset_root) registered = filemanager.registry.get(Backend.VIRTUAL_WORKSPACE.value) if ( isinstance(registered, VirtualWorkspaceBackend) - and registered.plate_root.resolve() == resolved_root + and absolute_source_path(registered.plate_root) == root and registered.metadata_config == METADATA_CONFIG ): return backend = VirtualWorkspaceBackend( - plate_root=resolved_root, metadata_config=METADATA_CONFIG + plate_root=root, metadata_config=METADATA_CONFIG ) filemanager.register_backend(Backend.VIRTUAL_WORKSPACE.value, backend) logger.info(f"Registered virtual workspace backend for {dataset_root}") diff --git a/openhcs/core/source_path_identity.py b/openhcs/core/source_path_identity.py index 28779ae9eb..823134321b 100644 --- a/openhcs/core/source_path_identity.py +++ b/openhcs/core/source_path_identity.py @@ -37,6 +37,16 @@ def source_path_identity(path: str | PathLike[str]) -> Path: return _normal_form(str(path)) +def absolute_source_path(path: str | PathLike[str]) -> Path: + """Return the identity of ``path`` made absolute against the working directory. + + Datasets, stores and workspaces are identified by absolute paths; this is + that identity, still without following symlinks. + """ + + return _normal_form(str(Path(path).absolute())) + + def source_path_identity_key(path: str | PathLike[str]) -> str: """Return the identity of ``path`` as a string key.""" diff --git a/openhcs/core/source_projection.py b/openhcs/core/source_projection.py index cf7ed596e1..e6868d9482 100644 --- a/openhcs/core/source_projection.py +++ b/openhcs/core/source_projection.py @@ -41,7 +41,7 @@ source_metadata_scalar, ) from python_introspect import to_jsonable -from openhcs.core.source_path_identity import source_path_posix +from openhcs.core.source_path_identity import absolute_source_path, source_path_posix from openhcs.core.axes import Axis, AxisFamily, PartitionAxis, StackAxis, TileAxis @@ -240,7 +240,7 @@ def __post_init__(self) -> None: def for_root(cls, root: str | Path) -> "SourceDatasetIdentity": """Return the exact collection identity for one submitted source root.""" - return cls(Path(root).resolve(strict=False).as_uri()) + return cls(absolute_source_path(root).as_uri()) @dataclass(frozen=True, slots=True) @@ -256,7 +256,7 @@ class SourcePlaneStoreIdentity: def __post_init__(self) -> None: container_paths = tuple( sorted( - {Path(path).resolve(strict=False) for path in self.container_paths}, + {absolute_source_path(path) for path in self.container_paths}, key=str, ) ) @@ -436,7 +436,7 @@ class SourcePlaneDataset: diagnostics: tuple[SourceDatasetDiagnostic, ...] = () def __post_init__(self) -> None: - root = Path(self.root).resolve(strict=False) + root = absolute_source_path(self.root) if not isinstance(self.identity, SourceDatasetIdentity): raise TypeError( "SourcePlaneDataset.identity must be SourceDatasetIdentity." @@ -476,7 +476,7 @@ def aggregate( if not datasets: raise ValueError("SourcePlaneDataset.aggregate requires datasets.") - roots = {dataset.root.resolve(strict=False) for dataset in datasets} + roots = {dataset.root for dataset in datasets} if len(roots) != 1: raise SourceDatasetConflictError( "Source stores cannot aggregate across different collection roots." diff --git a/openhcs/microscopes/bioformats_adapter.py b/openhcs/microscopes/bioformats_adapter.py index 5202643a68..b001ae64fd 100644 --- a/openhcs/microscopes/bioformats_adapter.py +++ b/openhcs/microscopes/bioformats_adapter.py @@ -28,6 +28,7 @@ with_source_component_metadata, ) from openhcs.core.source_metadata import SourceVoxelSpacing +from openhcs.core.source_path_identity import absolute_source_path from openhcs.core.source_projection import ( OpenHCSPlaneAddress, SourceCandidate, @@ -70,7 +71,7 @@ class BioFormatsPackedRgbSeriesExclusion(SourceDatasetDiagnostic): def __post_init__(self) -> None: source_files = tuple( sorted( - {Path(path).resolve(strict=False) for path in self.source_files}, + {absolute_source_path(path) for path in self.source_files}, key=str, ) ) @@ -400,14 +401,14 @@ def _image_candidates( container_paths = tuple( sorted( { - image.source_path.resolve(strict=False), - *(path.resolve(strict=False) for path in image.source_files), + absolute_source_path(image.source_path), + *(absolute_source_path(path) for path in image.source_files), }, key=str, ) ) - normalized_source = image.source_path.resolve(strict=False) - backend_source = _relative_path(self.root, normalized_source) + source_path = absolute_source_path(image.source_path) + backend_source = _relative_path(self.root, source_path) filter_paths = tuple( value for path in container_paths @@ -436,7 +437,7 @@ def _image_candidates( if image.reader == "npy": source_ref = SourcePixelRef( backend=Backend.DISK.value, - backend_address=_relative_path(self.root, normalized_source), + backend_address=_relative_path(self.root, source_path), source_axis_indices=(plane.t - 1, plane.z - 1, plane.c - 1), ) source_axis_shape = ( @@ -457,7 +458,7 @@ def _image_candidates( candidates.append( SourceCandidate( source_ref=source_ref, - relative_path=_relative_path(self.root, normalized_source), + relative_path=_relative_path(self.root, source_path), metadata=metadata, source_axis_shape=source_axis_shape, source_filter_paths=filter_paths, @@ -659,8 +660,7 @@ def discover_stores(self, root: Path) -> tuple[SourcePlaneDataset, ...]: claimed_paths: set[Path] = set() container_claims: list[BioFormatsContainerClaim] = [] for source_path in candidate_source_paths(root): - resolved_path = source_path.resolve(strict=False) - if resolved_path in claimed_paths: + if absolute_source_path(source_path) in claimed_paths: continue if not self._declares_path(context, source_path): continue @@ -1176,7 +1176,7 @@ def _ngff_image_candidates( channel_count = shape[axes.index("c")] if "c" in axes else 1 channel_labels = _ngff_channel_labels(attrs, channel_count) pixel_size = _ngff_pixel_size(dataset, len(axes), image_location.fmt) - container_paths = (store_root.resolve(strict=False),) + container_paths = (absolute_source_path(store_root),) filter_paths = _physical_path_identities(collection_root, store_root) candidates: list[SourceCandidate] = [] for source_axis_indices in product(*(range(size) for size in source_axis_shape)): @@ -1380,7 +1380,7 @@ def from_dataset( """Project one successful dataset onto its physical container claim.""" return cls( - entrypoint=entrypoint.resolve(strict=False), + entrypoint=absolute_source_path(entrypoint), container_paths=frozenset( path for candidate in dataset.candidates @@ -1391,7 +1391,7 @@ def from_dataset( def proves_redundant_failed_entrypoint(self, candidate: Path) -> bool: """Return whether this claim proves a failed nested descriptor is redundant.""" - candidate = candidate.resolve(strict=False) + candidate = absolute_source_path(candidate) if ( self.entrypoint.name != candidate.name or self.entrypoint.parent not in candidate.parents @@ -1444,16 +1444,16 @@ def _physical_path_identities(root: Path, source_path: Path) -> tuple[str, ...]: dict.fromkeys( ( _relative_path(root, source_path), - str(source_path.resolve(strict=False)), + str(absolute_source_path(source_path)), ) ) ) def _relative_path(root: Path, path: Path) -> str: - resolved_root = root.resolve(strict=False) - collection_root = resolved_root.parent if root.is_file() else resolved_root - return path.resolve(strict=False).relative_to(collection_root).as_posix() + root_path = absolute_source_path(root) + collection_root = root_path.parent if root.is_file() else root_path + return absolute_source_path(path).relative_to(collection_root).as_posix() def _absolute_path(root: Path, value: str) -> Path: diff --git a/tests/unit/test_microscope_virtual_workspace_metadata.py b/tests/unit/test_microscope_virtual_workspace_metadata.py index 6b66186586..ecd9aaea33 100644 --- a/tests/unit/test_microscope_virtual_workspace_metadata.py +++ b/tests/unit/test_microscope_virtual_workspace_metadata.py @@ -278,9 +278,9 @@ def test_workspace_registration_reuses_owner_and_refreshes_changed_mapping(tmp_p manager = FileManager({}) DatasetSource._register_virtual_workspace_backend(tmp_path, manager) owner = manager.registry[Backend.VIRTUAL_WORKSPACE.value] - alias = tmp_path / "plate-alias" - alias.symlink_to(tmp_path, target_is_directory=True) - DatasetSource._register_virtual_workspace_backend(alias, manager) + (tmp_path / "sub").mkdir() + same_root = tmp_path / "sub" / ".." + DatasetSource._register_virtual_workspace_backend(same_root, manager) assert manager.registry[Backend.VIRTUAL_WORKSPACE.value] is owner assert owner._resolve_ref("image.tif").backend_address == "first.tif" From 484004f393471793aab12e02c83b11edb9a769ef Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:40:03 -0400 Subject: [PATCH 12/36] E1: viewer lifecycle, launch session, napari layer kinds, label routes and variable-size handling are families ViewerLifecycleMode and ViewerLifecycleState become the ViewerLifecycle family (ViewerStopped, ViewerConnectedExternal, ViewerOwnedProcess); each owns is_running and bootstrap rollback. ViewerLaunchContextMode becomes ViewerLaunchSession. NapariLayerKind members own layer creation, replacing the creator and logger tables. NapariDimensionLabelRouteSource members carry their logged reason. NapariVariableSizeHandling members (PadToMaxSize, SeparateLayers) own routing and materialization; the EnumKeyed strategy family is deleted and the config field holds the class (RegisteredClassChoices). Co-Authored-By: Claude Opus 5.5 --- openhcs/core/viewer_streaming_service.py | 3 +- openhcs/demo/synthetic_plate_pipeline.py | 4 +- openhcs/runtime/napari_streaming_handlers.py | 139 ++++------- openhcs/runtime/napari_viewer_server.py | 210 +++++------------ openhcs/runtime/viewer_display.py | 135 ++++++++++- openhcs/runtime/viewer_protocol.py | 221 +++++++++--------- scripts/mcp_thesis_demo_live.py | 4 +- tests/integration/test_main.py | 4 +- tests/unit/agent/test_ui_bridge_service.py | 6 +- .../test_napari_accepted_work_settlement.py | 3 +- tests/unit/test_napari_reset_transitions.py | 4 +- tests/unit/test_napari_streaming_handlers.py | 33 ++- tests/unit/test_streaming_service.py | 8 +- tests/unit/test_viewer_protocol.py | 11 +- 14 files changed, 370 insertions(+), 415 deletions(-) diff --git a/openhcs/core/viewer_streaming_service.py b/openhcs/core/viewer_streaming_service.py index 8ea8da76f6..0b82362121 100644 --- a/openhcs/core/viewer_streaming_service.py +++ b/openhcs/core/viewer_streaming_service.py @@ -206,6 +206,7 @@ def get_or_create_visualizer( from zmqruntime.queue_tracker import GlobalQueueTrackerRegistry from openhcs.runtime.viewer_protocol import ( + ViewerConnectedExternal, ViewerGraphicalSessionUnavailableError, ViewerLaunchContext, ViewerProcessLaunchReuseCheck, @@ -237,7 +238,7 @@ def create_viewer() -> ManagedViewerLifecycleMixin: launch_context=resolved_launch_context, ) if not fresh and created_viewer.existing_viewer_is_ready(): - created_viewer.lifecycle_state.mark_connected_external() + created_viewer.lifecycle = ViewerConnectedExternal return created_viewer try: diff --git a/openhcs/demo/synthetic_plate_pipeline.py b/openhcs/demo/synthetic_plate_pipeline.py index 7c7689728f..fc490eee95 100644 --- a/openhcs/demo/synthetic_plate_pipeline.py +++ b/openhcs/demo/synthetic_plate_pipeline.py @@ -13,7 +13,7 @@ LazyProcessingConfig, PipelineConfig, ) -from openhcs.runtime.viewer_display import NapariVariableSizeHandling +from openhcs.runtime.viewer_display import PadToMaxSize from openhcs.core.steps.function_step import FunctionStep from openhcs.processing.backends.analysis.cell_counting_cpu import ( DetectionMethod, @@ -102,7 +102,7 @@ ), name="Cell Counting", napari_streaming_config=LazyNapariStreamingConfig( - variable_size_handling=NapariVariableSizeHandling.PAD_TO_MAX + variable_size_handling=PadToMaxSize ), ) pipeline_steps.append(step_8) diff --git a/openhcs/runtime/napari_streaming_handlers.py b/openhcs/runtime/napari_streaming_handlers.py index 102c095fb7..8cd957fc70 100644 --- a/openhcs/runtime/napari_streaming_handlers.py +++ b/openhcs/runtime/napari_streaming_handlers.py @@ -40,7 +40,6 @@ ViewerLayerAxisProjection, ) from openhcs.runtime.viewer_protocol import ( - NapariLayerKind, ViewerComponentValueOrdering, ViewerSettlePhase, SettlementComplete, @@ -868,101 +867,55 @@ def _shape_aggregate_extents(data: LayerData) -> tuple[int, ...]: )) -NapariLayerCreator: TypeAlias = Callable[ - [NapariViewerLayerCreator, LayerData, str, Mapping[str, LayerKwargValue]], - NapariLayerHandle, -] -NapariLayerCreationLogger: TypeAlias = Callable[ - [NapariLayerKind, str, LayerData], - None, -] - - -def _log_created_layer( - layer_kind: NapariLayerKind, - layer_name: str, - data: LayerData, -) -> None: - del data - logger.info( - "🔬 NAPARI PROCESS: Created %s layer %s", - layer_kind.value, - layer_name, - ) +class NapariLayerKind(ABC): + """One napari layer type a stream route is displayed as; each is a subclass.""" + name: ClassVar[str] + """Napari's name for the layer type.""" + adder: ClassVar[str] + """The ``napari.Viewer`` method that creates the layer.""" + holds_items: ClassVar[bool] = False + """The layer's data is a sequence of items (shapes, points), not an array.""" -def _log_created_item_layer( - layer_kind: NapariLayerKind, - layer_name: str, - data: LayerData, -) -> None: - logger.info( - "🔬 NAPARI PROCESS: Created %s layer %s with %d %s", - layer_kind.value, - layer_name, - len(data), - layer_kind.value, - ) + @classmethod + def create( + cls, + viewer: NapariViewerLayerCreator, + data: LayerData, + layer_name: str, + layer_kwargs: Mapping[str, LayerKwargValue], + ) -> NapariLayerHandle: + layer = getattr(viewer, cls.adder)(data, name=layer_name, **dict(layer_kwargs)) + if cls.holds_items: + logger.info( + "🔬 NAPARI PROCESS: Created %s layer %s with %d %s", + cls.name, layer_name, len(data), cls.name, + ) + else: + logger.info("🔬 NAPARI PROCESS: Created %s layer %s", cls.name, layer_name) + return layer -def _complete_layer_kind_mapping( - mapping: Mapping[NapariLayerKind, NapariLayerCreator], -) -> Mapping[NapariLayerKind, NapariLayerCreator]: - missing = set(NapariLayerKind) - set(mapping) - if missing: - raise ValueError(f"Missing Napari layer creators for {missing!r}.") - return mapping +class NapariImageLayer(NapariLayerKind): + name = "image" + adder = "add_image" -def _complete_layer_log_mapping( - mapping: Mapping[NapariLayerKind, NapariLayerCreationLogger], -) -> Mapping[NapariLayerKind, NapariLayerCreationLogger]: - missing = set(NapariLayerKind) - set(mapping) - if missing: - raise ValueError(f"Missing Napari layer loggers for {missing!r}.") - return mapping +class NapariShapesLayer(NapariLayerKind): + name = "shapes" + adder = "add_shapes" + holds_items = True -NAPARI_LAYER_CREATORS = _complete_layer_kind_mapping( - { - NapariLayerKind.IMAGE: ( - lambda viewer, data, layer_name, layer_kwargs: viewer.add_image( - data, - name=layer_name, - **dict(layer_kwargs), - ) - ), - NapariLayerKind.SHAPES: ( - lambda viewer, data, layer_name, layer_kwargs: viewer.add_shapes( - data, - name=layer_name, - **dict(layer_kwargs), - ) - ), - NapariLayerKind.POINTS: ( - lambda viewer, data, layer_name, layer_kwargs: viewer.add_points( - data, - name=layer_name, - **dict(layer_kwargs), - ) - ), - NapariLayerKind.LABELS: ( - lambda viewer, data, layer_name, layer_kwargs: viewer.add_labels( - data, - name=layer_name, - **dict(layer_kwargs), - ) - ), - } -) -NAPARI_LAYER_CREATED_LOGGERS = _complete_layer_log_mapping( - { - NapariLayerKind.IMAGE: _log_created_layer, - NapariLayerKind.SHAPES: _log_created_item_layer, - NapariLayerKind.POINTS: _log_created_item_layer, - NapariLayerKind.LABELS: _log_created_layer, - } -) +class NapariPointsLayer(NapariLayerKind): + name = "points" + adder = "add_points" + holds_items = True + + +class NapariLabelsLayer(NapariLayerKind): + name = "labels" + adder = "add_labels" class NapariImageLayerPresentationPolicy(ABC): @@ -1104,7 +1057,7 @@ class NapariLayerUpdates: def create_or_update( self, *, - layer_kind: NapariLayerKind, + layer_kind: type[NapariLayerKind], viewer: NapariViewerLayerCreator, layers: dict[str, NapariLayerHandle], route_key: str, @@ -1116,12 +1069,7 @@ def create_or_update( viewer, self._existing_layer(viewer=viewer, layers=layers, route_key=route_key), ) - new_layer = NAPARI_LAYER_CREATORS[layer_kind]( - viewer, - data, - layer_name, - layer_kwargs, - ) + new_layer = layer_kind.create(viewer, data, layer_name, layer_kwargs) self.mount( viewer=viewer, layers=layers, @@ -1129,7 +1077,6 @@ def create_or_update( layer=new_layer, selection=selection, ) - NAPARI_LAYER_CREATED_LOGGERS[layer_kind](layer_kind, layer_name, data) return new_layer def mount( diff --git a/openhcs/runtime/napari_viewer_server.py b/openhcs/runtime/napari_viewer_server.py index 2285d804e1..2e96bb3035 100644 --- a/openhcs/runtime/napari_viewer_server.py +++ b/openhcs/runtime/napari_viewer_server.py @@ -83,7 +83,6 @@ from openhcs.runtime.viewer_display import ( NapariDisplaySettings, NapariSlots, - NapariVariableSizeHandling, ) from metaclass_registry.strategies import EnumKeyedStrategyMixin from openhcs.core.runtime_image_values import ( @@ -97,6 +96,9 @@ ViewerServerPort, ) from openhcs.runtime.napari_streaming_handlers import ( + NapariImageLayer, + NapariLayerKind, + NapariPointsLayer, DimensionLabelMap, LayerData, LayerKwargValue, @@ -160,7 +162,6 @@ ) from python_introspect import to_jsonable from openhcs.runtime.viewer_protocol import ( - NapariLayerKind, NapariViewerServerRequest, OpenHCSViewerServerABC, OpenHCSViewerControlMessageType, @@ -929,136 +930,6 @@ class NapariLayerRoute(ViewerComponentAxisSemantics): ) -class NapariVariableSizeDisplayStrategy( - EnumKeyedStrategyMixin[NapariVariableSizeHandling], - ABC, - metaclass=AutoRegisterMeta, -): - """Apply one declared variable-size policy at routing and display time.""" - - __enum_member_attr__ = "variable_size_handling" - - variable_size_handling: ClassVar[NapariVariableSizeHandling] - - def routing_context( - self, - context: NapariStreamLayerContext, - ) -> NapariStreamLayerContext: - """Return the effective route context for one streamed payload.""" - - return context - - @abstractmethod - def materialize_layer_items( - self, - layer_items: list[NapariStreamLayerItem], - *, - route_key: str, - ) -> list[NapariStreamLayerItem]: - """Return image items that can share one native Napari layer.""" - - -class NapariPadToMaxDisplayStrategy(NapariVariableSizeDisplayStrategy): - """Keep one component route and pad its images to a common shape.""" - - variable_size_handling = NapariVariableSizeHandling.PAD_TO_MAX - - def materialize_layer_items( - self, - layer_items: list[NapariStreamLayerItem], - *, - route_key: str, - ) -> list[NapariStreamLayerItem]: - shapes = {item.data.shape for item in layer_items} - if len(shapes) > 1: - logger.info( - "🔬 NAPARI PROCESS: Images in layer %s have different shapes - " - "padding to max size", - route_key, - ) - max_shape = tuple( - max(shape[axis] for shape in shapes) - for axis in range(len(layer_items[0].data.shape)) - ) - for item_index, item in enumerate(layer_items): - if item.data.shape == max_shape: - continue - pad_width = tuple( - (0, maximum - current) - for current, maximum in zip( - item.data.shape, - max_shape, - strict=True, - ) - ) - layer_items[item_index] = replace( - item, - data=np.pad( - item.data, - pad_width, - mode="constant", - constant_values=0, - ), - ) - logger.debug( - "🔬 NAPARI PROCESS: Padded image from %s to %s", - item.data.shape, - max_shape, - ) - return layer_items - - -class NapariSeparateLayersDisplayStrategy(NapariVariableSizeDisplayStrategy): - """Give each partition-axis value its own route and keep native shapes.""" - - variable_size_handling = NapariVariableSizeHandling.SEPARATE_LAYERS - - def routing_context( - self, - context: NapariStreamLayerContext, - ) -> NapariStreamLayerContext: - if context.address.stream_layer_data_type is not StreamingDataType.IMAGE: - return context - - partition_components = context.layout.names_with_role(PartitionAxis) - for component in partition_components: - ViewerComponentCoordinates.required_value( - context.address.components, - component, - context="Napari separate-layer routing", - ) - layered = context.layout.components_in(NapariSlots.Layer) - if all(component in layered for component in partition_components): - return context - return replace( - context, - layout=context.layout.with_modes( - { - **context.layout.component_modes, - **{ - component: NapariSlots.Layer.wire_value - for component in partition_components - }, - } - ), - ) - - def materialize_layer_items( - self, - layer_items: list[NapariStreamLayerItem], - *, - route_key: str, - ) -> list[NapariStreamLayerItem]: - shapes = {item.data.shape for item in layer_items} - if len(shapes) > 1: - raise ValueError( - "Napari separate-layer route " - f"{route_key!r} still contains mixed image shapes: " - f"{sorted(shapes)!r}." - ) - return layer_items - - class NapariComponentAwareDisplayCoordinator: """Route loaded payload data into debounced Napari layer updates.""" @@ -1102,9 +973,11 @@ def _route( stream_layer_context: NapariStreamLayerContext, server: "NapariViewerServer", ) -> NapariLayerRoute: - effective_context = NapariVariableSizeDisplayStrategy.for_enum_member( - stream_layer_context.display_config.variable_size_handling - ).routing_context(stream_layer_context) + effective_context = ( + stream_layer_context.display_config.variable_size_handling.routing_context( + stream_layer_context + ) + ) payload_layout_role = NapariImagePayloadLayoutRole.for_stream_layer_context( effective_context, ) @@ -1482,15 +1355,38 @@ def disambiguate( return f"{title} [{StreamProducerDisplayNameAuthority.disambiguation_label(producer)}]" -class NapariDimensionLabelRouteSource(str, Enum): - """Provenance for the dimension-label route currently applied to Napari.""" +class NapariDimensionLabelRouteSource(ABC): + """Why a route owns Napari's dimension labels; each reason is a subclass.""" + + description: ClassVar[str] + """The reason as logged.""" + unresolved: ClassVar[bool] = False + """No route could be chosen from the viewer's context.""" + + +class SelectedOpenHCSLayerRoute(NapariDimensionLabelRouteSource): + description = "selected OpenHCS layer" + + +class UpdatedOpenHCSLayerRoute(NapariDimensionLabelRouteSource): + description = "updated OpenHCS layer" + + +class ActiveStreamRoute(NapariDimensionLabelRouteSource): + description = "active stream layer context" + + +class CompatibleStreamRoute(NapariDimensionLabelRouteSource): + description = "compatible stream layer context" + + +class ActiveNonOpenHCSLayerRoute(NapariDimensionLabelRouteSource): + description = "active non-OpenHCS layer" + - SELECTED_OPENHCS_LAYER = "selected_openhcs_layer" - UPDATED_OPENHCS_LAYER = "updated_openhcs_layer" - ACTIVE_STREAM_ROUTE = "active_stream_layer_context" - COMPATIBLE_STREAM_ROUTE = "compatible_stream_layer_context" - ACTIVE_NON_OPENHCS_LAYER = "active_non_openhcs_layer" - MISSING = "missing" +class MissingRoute(NapariDimensionLabelRouteSource): + description = "missing" + unresolved = True @dataclass(frozen=True) @@ -1498,7 +1394,7 @@ class NapariDimensionLabelRouteResolution: """Resolved dimension-label route plus the reason it was selected.""" route_key: str | None - source: NapariDimensionLabelRouteSource + source: type[NapariDimensionLabelRouteSource] class NapariDimensionLabelStore: @@ -1613,7 +1509,7 @@ def resolve( if selected_route_key is None: return NapariDimensionLabelRouteResolution( route_key=None, - source=NapariDimensionLabelRouteSource.ACTIVE_NON_OPENHCS_LAYER, + source=ActiveNonOpenHCSLayerRoute, ) if self._route_matches_viewer_context( @@ -1623,7 +1519,7 @@ def resolve( ): return NapariDimensionLabelRouteResolution( route_key=updated_route_key, - source=NapariDimensionLabelRouteSource.UPDATED_OPENHCS_LAYER, + source=UpdatedOpenHCSLayerRoute, ) if selected_route_key is not None: @@ -1634,26 +1530,26 @@ def resolve( ): return NapariDimensionLabelRouteResolution( route_key=selected_route_key, - source=NapariDimensionLabelRouteSource.SELECTED_OPENHCS_LAYER, + source=SelectedOpenHCSLayerRoute, ) route_key = self.server.layer_route_state.active_dimension_label_route if self._route_matches_viewer_context(route_key, viewer_ndim, current_step): return NapariDimensionLabelRouteResolution( route_key=route_key, - source=NapariDimensionLabelRouteSource.ACTIVE_STREAM_ROUTE, + source=ActiveStreamRoute, ) route_key = self._latest_compatible_route(viewer_ndim, current_step) if route_key is not None: return NapariDimensionLabelRouteResolution( route_key=route_key, - source=NapariDimensionLabelRouteSource.COMPATIBLE_STREAM_ROUTE, + source=CompatibleStreamRoute, ) return NapariDimensionLabelRouteResolution( route_key=None, - source=NapariDimensionLabelRouteSource.MISSING, + source=MissingRoute, ) def _route_key_for_layer(self, active_layer: NapariLayerHandle) -> str | None: @@ -1735,7 +1631,7 @@ def setup_for_layer(self, layer_key: str) -> None: self._connect_handlers() resolution = self.route_resolver.resolve(layer_key) - if resolution.source is NapariDimensionLabelRouteSource.MISSING: + if resolution.source.unresolved: state = self.server.layer_route_state.dimension_state_for(layer_key) if self._apply_axis_labels(layer_key, state): self.server.layer_route_state.set_active_dimension_label_route( @@ -1752,7 +1648,7 @@ def setup_for_layer(self, layer_key: str) -> None: logger.info( "🔬 NAPARI PROCESS: Active dimension label route resolved to %s (%s)", resolution.route_key, - resolution.source.value, + resolution.source.description, ) def _connect_handlers(self) -> None: @@ -1863,7 +1759,7 @@ class NapariLayerDisplayRequest: def create_or_update_layer( self, *, - layer_kind: NapariLayerKind, + layer_kind: type[NapariLayerKind], data: LayerData, layer_kwargs: Mapping[str, LayerKwargValue], ) -> NapariLayerHandle: @@ -2103,9 +1999,7 @@ def handle(self, request: NapariLayerDisplayRequest) -> None: f"Layer {presentation.route_key} contains mixed-rank image payloads: " f"{sorted(set(shapes))}" ) - layer_items = NapariVariableSizeDisplayStrategy.for_enum_member( - request.display_config.variable_size_handling - ).materialize_layer_items( + layer_items = request.display_config.variable_size_handling.materialize_layer_items( layer_items, route_key=presentation.route_key, ) @@ -2149,7 +2043,7 @@ def handle(self, request: NapariLayerDisplayRequest) -> None: ) request.create_or_update_layer( - layer_kind=NapariLayerKind.IMAGE, + layer_kind=NapariImageLayer, data=stacked_data, layer_kwargs=layer_kwargs, ) @@ -2436,7 +2330,7 @@ def handle(self, request: NapariLayerDisplayRequest) -> None: if axis_labels is not None: layer_kwargs["axis_labels"] = axis_labels request.create_or_update_layer( - layer_kind=NapariLayerKind.POINTS, + layer_kind=NapariPointsLayer, data=points_data, layer_kwargs=layer_kwargs, ) diff --git a/openhcs/runtime/viewer_display.py b/openhcs/runtime/viewer_display.py index 46792d72bc..876b25857e 100644 --- a/openhcs/runtime/viewer_display.py +++ b/openhcs/runtime/viewer_display.py @@ -7,16 +7,20 @@ an axis into, and which roles default to which slot. """ -from abc import ABC +import logging +from abc import ABC, abstractmethod from collections.abc import Iterator, Mapping, Sequence -from dataclasses import dataclass, field, fields +from dataclasses import dataclass, field, fields, replace from enum import Enum from functools import cache -from typing import ClassVar +from typing import Annotated, ClassVar +from metaclass_registry import AutoRegisterMeta from python_introspect import ( AnnotatedDataclassValidationMixin, + RegisteredClassChoices, dataclass_from_mapping, + to_jsonable, ) from zmqruntime.config import NonBlankString from zmqruntime.viewer_protocol import ViewerWireMapping, ViewerWirePayload, ViewerWireValue @@ -31,6 +35,8 @@ TimeAxis, ) +logger = logging.getLogger(__name__) + DECLARED_AXES_FIELD = "declared_axes" """Display-config wire field carrying the stream's axis declarations.""" @@ -358,18 +364,124 @@ def from_display_payload(cls, payload: Mapping[str, ViewerWireValue]): def settings_wire_mapping(self) -> ViewerWireMapping: return ViewerWirePayload.mapping( { - declared.name: getattr(self, declared.name) + declared.name: to_jsonable(getattr(self, declared.name)) for declared in fields(self.settings_type()) }, context=f"{type(self).__name__} display settings", ) -class NapariVariableSizeHandling(Enum): - """How to handle images with different sizes in the same layer.""" +class NapariVariableSizeHandling(ABC, metaclass=AutoRegisterMeta): + """How Napari shows images of different sizes in one stream; each way is a subclass. + + ``name`` is its spelling in forms and on the wire. + """ + + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["NapariVariableSizeHandling"]]] = {} + + name: ClassVar[str | None] = None + + @classmethod + def routing_context(cls, context): + """Return the effective route context for one streamed payload.""" + + return context + + @classmethod + @abstractmethod + def materialize_layer_items(cls, layer_items: list, *, route_key: str) -> list: + """Return image items that can share one native Napari layer.""" + + +class PadToMaxSize(NapariVariableSizeHandling): + """Keep one component route and pad its images to a common shape.""" - SEPARATE_LAYERS = "separate_layers" - PAD_TO_MAX = "pad_to_max" + name = "pad_to_max" + + @classmethod + def materialize_layer_items(cls, layer_items: list, *, route_key: str) -> list: + import numpy as np + + shapes = {item.data.shape for item in layer_items} + if len(shapes) <= 1: + return layer_items + logger.info( + "🔬 NAPARI PROCESS: Images in layer %s have different shapes - " + "padding to max size", + route_key, + ) + max_shape = tuple( + max(shape[axis] for shape in shapes) + for axis in range(len(layer_items[0].data.shape)) + ) + for item_index, item in enumerate(layer_items): + if item.data.shape == max_shape: + continue + pad_width = tuple( + (0, maximum - current) + for current, maximum in zip(item.data.shape, max_shape, strict=True) + ) + layer_items[item_index] = replace( + item, + data=np.pad(item.data, pad_width, mode="constant", constant_values=0), + ) + logger.debug( + "🔬 NAPARI PROCESS: Padded image from %s to %s", + item.data.shape, + max_shape, + ) + return layer_items + + +class SeparateLayers(NapariVariableSizeHandling): + """Give each partition-axis value its own route and keep native shapes.""" + + name = "separate_layers" + + @classmethod + def routing_context(cls, context): + from polystore.streaming_constants import StreamingDataType + + from openhcs.core.axes import PartitionAxis + from openhcs.runtime.viewer_component_system import ViewerComponentCoordinates + + if context.address.stream_layer_data_type is not StreamingDataType.IMAGE: + return context + partition_components = context.layout.names_with_role(PartitionAxis) + for component in partition_components: + ViewerComponentCoordinates.required_value( + context.address.components, + component, + context="Napari separate-layer routing", + ) + layered = context.layout.components_in(NapariSlots.Layer) + if all(component in layered for component in partition_components): + return context + return replace( + context, + layout=context.layout.with_modes( + { + **context.layout.component_modes, + **{ + component: NapariSlots.Layer.wire_value + for component in partition_components + }, + } + ), + ) + + @classmethod + def materialize_layer_items(cls, layer_items: list, *, route_key: str) -> list: + shapes = {item.data.shape for item in layer_items} + if len(shapes) > 1: + raise ValueError( + "Napari separate-layer route " + f"{route_key!r} still contains mixed image shapes: " + f"{sorted(shapes)!r}." + ) + return layer_items @dataclass(frozen=True) @@ -386,8 +498,11 @@ class NapariDisplaySettings(ViewerDisplaySettings): ) }, ) - variable_size_handling: NapariVariableSizeHandling = field( - default=NapariVariableSizeHandling.PAD_TO_MAX, + variable_size_handling: Annotated[ + type[NapariVariableSizeHandling], + RegisteredClassChoices(NapariVariableSizeHandling), + ] = field( + default=PadToMaxSize, metadata={ "description": ( "How Napari handles streamed images with different spatial " diff --git a/openhcs/runtime/viewer_protocol.py b/openhcs/runtime/viewer_protocol.py index 25c21ceaa0..f04d0196b1 100644 --- a/openhcs/runtime/viewer_protocol.py +++ b/openhcs/runtime/viewer_protocol.py @@ -922,19 +922,32 @@ def qt_environment_policy(self) -> "ViewerQtPlatformEnvironmentPolicy": ) -class ViewerLaunchContextMode(Enum): - """Provenance and availability of one detached viewer launch session.""" +class ViewerLaunchSession(ABC): + """Where a detached viewer's graphical session comes from; each source is a subclass.""" - INHERITED_GRAPHICAL_SESSION = "inherited_graphical_session" - PROJECTED_GRAPHICAL_SESSION = "projected_graphical_session" - HEADLESS = "headless" + graphical: ClassVar[bool] = True + """A graphical session is available to the child process.""" + + +class InheritedGraphicalSession(ViewerLaunchSession): + """The launching process already owns a GUI session.""" + + +class ProjectedGraphicalSession(ViewerLaunchSession): + """A graphical environment validated by its owner is carried to the child.""" + + +class HeadlessSession(ViewerLaunchSession): + """No graphical session is available.""" + + graphical = False @dataclass(frozen=True, slots=True) class ViewerLaunchContext: """Typed graphical-session context consumed by detached viewer launch.""" - mode: ViewerLaunchContextMode + session: type[ViewerLaunchSession] environment_overlay: Mapping[str, str] = field(default_factory=dict) environment_unset_keys: tuple[str, ...] = () @@ -953,7 +966,7 @@ def __post_init__(self) -> None: @classmethod def inherited_graphical_session(cls) -> "ViewerLaunchContext": """Declare that the launching process already owns a GUI session.""" - return cls(ViewerLaunchContextMode.INHERITED_GRAPHICAL_SESSION) + return cls(InheritedGraphicalSession) @classmethod def projected_graphical_session( @@ -969,7 +982,7 @@ def projected_graphical_session( environment_overlay.pop("QT_QPA_PLATFORM") environment_unset_keys = ("QT_QPA_PLATFORM",) return cls( - ViewerLaunchContextMode.PROJECTED_GRAPHICAL_SESSION, + ProjectedGraphicalSession, environment_overlay, environment_unset_keys, ) @@ -977,11 +990,11 @@ def projected_graphical_session( @classmethod def headless(cls) -> "ViewerLaunchContext": """Declare that no authoritative graphical session is available.""" - return cls(ViewerLaunchContextMode.HEADLESS) + return cls(HeadlessSession) @property def graphical_session_available(self) -> bool: - return self.mode is not ViewerLaunchContextMode.HEADLESS + return self.session.graphical def child_environment( self, @@ -1009,15 +1022,6 @@ def __str__(self) -> str: ) -class NapariLayerKind(Enum): - """Napari layer creation families used by streaming display.""" - - IMAGE = "image" - SHAPES = "shapes" - POINTS = "points" - LABELS = "labels" - - class FijiPayloadKind(Enum): """Payload strings sent to the Fiji viewer process.""" @@ -1559,44 +1563,96 @@ def policy(self) -> "ViewerControlPingPolicy": ) -class ViewerLifecycleMode(Enum): - """Runtime ownership state for a managed viewer.""" +class ViewerLifecycle(ABC): + """Who holds a managed viewer right now; each holder is a subclass.""" - STOPPED = "stopped" - CONNECTED_EXTERNAL = "connected_external" - OWNED_PROCESS = "owned_process" + active: ClassVar[bool] = True + """A viewer is expected to answer on the port.""" + owns_process: ClassVar[bool] = False + """This manager launched the viewer process and must stop it.""" + @classmethod + @abstractmethod + def is_running(cls, viewer: "ManagedViewerLifecycleMixin") -> bool: + """Whether the viewer still runs; stops the viewer when it does not.""" -@dataclass(slots=True) -class ViewerLifecycleState: - """Nominal lifecycle state for viewer process managers.""" + @classmethod + def rollback_failed_bootstrap(cls, viewer: "ManagedViewerLifecycleMixin") -> None: + """Discard only resources owned by an incomplete bootstrap attempt.""" + + from zmqruntime import ViewerStateManager + + manager = ViewerStateManager.get_instance() + port = viewer.required_port + if manager.get_viewer(viewer.viewer_family.wire_value, port) is viewer: + manager.release_viewer( + viewer.viewer_family.wire_value, port, stop=True, force=True, + ) + return + if cls.owns_process or viewer.process is not None: + viewer.force_stop() + return + viewer.lifecycle = ViewerStopped - mode: ViewerLifecycleMode = ViewerLifecycleMode.STOPPED + +class ViewerStopped(ViewerLifecycle): + active = False @classmethod - def stopped(cls) -> "ViewerLifecycleState": - return cls() + def is_running(cls, viewer: "ManagedViewerLifecycleMixin") -> bool: + return False - @property - def is_active(self) -> bool: - return self.mode is not ViewerLifecycleMode.STOPPED - @property - def is_connected_external(self) -> bool: - return self.mode is ViewerLifecycleMode.CONNECTED_EXTERNAL +class ViewerConnectedExternal(ViewerLifecycle): + """Connected to a viewer some other process launched.""" - @property - def is_owned_process(self) -> bool: - return self.mode is ViewerLifecycleMode.OWNED_PROCESS + @classmethod + def is_running(cls, viewer: "ManagedViewerLifecycleMixin") -> bool: + if viewer.check_connected_viewer(): + return True + logging.getLogger(type(viewer).__module__).debug( + "%s viewer on port %s is no longer responsive", + viewer.viewer_process_label, + viewer.port, + ) + viewer.lifecycle = ViewerStopped + return False - def mark_connected_external(self) -> None: - self.mode = ViewerLifecycleMode.CONNECTED_EXTERNAL + @classmethod + def rollback_failed_bootstrap(cls, viewer: "ManagedViewerLifecycleMixin") -> None: + viewer.cleanup_viewer_client() + viewer.lifecycle = ViewerStopped - def mark_owned_process(self) -> None: - self.mode = ViewerLifecycleMode.OWNED_PROCESS - def mark_stopped(self) -> None: - self.mode = ViewerLifecycleMode.STOPPED +class ViewerOwnedProcess(ViewerLifecycle): + """Running a viewer process this manager launched.""" + + owns_process = True + + @classmethod + def is_running(cls, viewer: "ManagedViewerLifecycleMixin") -> bool: + log = logging.getLogger(type(viewer).__module__) + if viewer.process is None: + viewer.lifecycle = ViewerStopped + return False + try: + alive = endpoint_process(viewer.process).is_alive() + except Exception as error: + log.warning( + "Error checking %s process status: %s", + viewer.viewer_process_label, + error, + ) + viewer.lifecycle = ViewerStopped + return False + if not alive: + log.debug( + "%s process on port %s is no longer alive", + viewer.viewer_process_label, + viewer.port, + ) + viewer.lifecycle = ViewerStopped + return alive @dataclass(frozen=True, slots=True) @@ -1756,7 +1812,7 @@ def __init__( transport=runtime_config.transport_endpoint, config=runtime_config.transport_config, ) - self.lifecycle_state: ViewerLifecycleState = ViewerLifecycleState.stopped() + self.lifecycle: type[ViewerLifecycle] = ViewerStopped @property def required_port(self) -> int: @@ -1802,7 +1858,7 @@ def start_viewer_now(self) -> None: with self._lock: port = self.required_port self.prepare_fresh_viewer_start() - if self.lifecycle_state.is_active: + if self.lifecycle.active: logger.warning("%s viewer is already running.", self.viewer_process_label) return logger.info( @@ -1813,7 +1869,7 @@ def start_viewer_now(self) -> None: ) self.process = self.launch_detached_viewer() if self.owned_viewer_process_is_alive(): - self.lifecycle_state.mark_owned_process() + self.lifecycle = ViewerOwnedProcess logger.info( "🔬 VISUALIZER: %s viewer process started (PID: %s)", self.viewer_process_label, @@ -1844,7 +1900,7 @@ def stop_viewer(self) -> None: "🔬 VISUALIZER: Force killing %s viewer process", self.viewer_process_label, ) - self.lifecycle_state.mark_stopped() + self.lifecycle = ViewerStopped def stop(self, timeout: float = 5.0) -> None: self.stop_viewer() @@ -2049,37 +2105,12 @@ def force_stop(self, timeout: float = 5.0) -> None: self.viewer_process_label, ) self.runtime_endpoint.force_release_addresses() - self.lifecycle_state.mark_stopped() + self.lifecycle = ViewerStopped def rollback_failed_bootstrap(self) -> None: """Discard only resources owned by an incomplete bootstrap attempt.""" - lifecycle_state = self.lifecycle_state - if lifecycle_state.is_connected_external: - self.cleanup_viewer_client() - lifecycle_state.mark_stopped() - return - - from zmqruntime import ViewerStateManager - - manager = ViewerStateManager.get_instance() - managed_viewer = manager.get_viewer( - self.viewer_family.wire_value, - self.required_port, - ) - if managed_viewer is self: - manager.release_viewer( - self.viewer_family.wire_value, - self.required_port, - stop=True, - force=True, - ) - return - - if lifecycle_state.is_owned_process or self.process is not None: - self.force_stop() - return - lifecycle_state.mark_stopped() + self.lifecycle.rollback_failed_bootstrap(self) def start(self, detached: bool = True) -> subprocess.Popen[bytes]: self.start_viewer_now() @@ -2240,43 +2271,7 @@ def read_viewer_state(self, timeout: float = 30.0) -> ViewerControlResponse: @property def is_running(self) -> bool: - lifecycle_state = self.lifecycle_state - if not lifecycle_state.is_active: - return False - - if lifecycle_state.is_connected_external: - if not self.check_connected_viewer(): - logging.getLogger(self.__class__.__module__).debug( - "%s viewer on port %s is no longer responsive", - self.viewer_process_label, - self.port, - ) - lifecycle_state.mark_stopped() - return False - return True - - if self.process is None: - lifecycle_state.mark_stopped() - return False - - try: - alive = endpoint_process(self.process).is_alive() - if not alive: - logging.getLogger(self.__class__.__module__).debug( - "%s process on port %s is no longer alive", - self.viewer_process_label, - self.port, - ) - lifecycle_state.mark_stopped() - return alive - except Exception as error: - logging.getLogger(self.__class__.__module__).warning( - "Error checking %s process status: %s", - self.viewer_process_label, - error, - ) - lifecycle_state.mark_stopped() - return False + return self.lifecycle.is_running(self) @dataclass(frozen=True, slots=True) diff --git a/scripts/mcp_thesis_demo_live.py b/scripts/mcp_thesis_demo_live.py index 80714f7ef3..15204b301e 100755 --- a/scripts/mcp_thesis_demo_live.py +++ b/scripts/mcp_thesis_demo_live.py @@ -1228,7 +1228,7 @@ def demo_source( LazyStepPartitionFilterConfig, PipelineConfig, ) -from openhcs.runtime.viewer_display import NapariVariableSizeHandling +from openhcs.runtime.viewer_display import PadToMaxSize from openhcs.core.source_bindings import ( ComponentSelector, NamedSourceBinding, @@ -1390,7 +1390,7 @@ def demo_source( default_dtype_conversion=DtypeConversion.UINT8 ), napari_streaming_config=LazyNapariStreamingConfig( - variable_size_handling=NapariVariableSizeHandling.PAD_TO_MAX + variable_size_handling=PadToMaxSize ) ) ] diff --git a/tests/integration/test_main.py b/tests/integration/test_main.py index 3df43c6f52..8567e38232 100644 --- a/tests/integration/test_main.py +++ b/tests/integration/test_main.py @@ -27,7 +27,7 @@ Backend, ZarrConfig, ) -from openhcs.runtime.viewer_display import NapariVariableSizeHandling +from openhcs.runtime.viewer_display import PadToMaxSize from openhcs.runtime.zmq_execution_client import OpenHCSExecutionSubmission from openhcs.core.config import ( LazyStepMaterializationConfig, @@ -360,7 +360,7 @@ def create_test_pipeline( ), napari_streaming_config=LazyNapariStreamingConfig( port=ports.napari[2], - variable_size_handling=NapariVariableSizeHandling.PAD_TO_MAX, + variable_size_handling=PadToMaxSize, enabled=enable_napari, ), fiji_streaming_config=LazyFijiStreamingConfig( diff --git a/tests/unit/agent/test_ui_bridge_service.py b/tests/unit/agent/test_ui_bridge_service.py index 986576bdf2..52696da0dc 100644 --- a/tests/unit/agent/test_ui_bridge_service.py +++ b/tests/unit/agent/test_ui_bridge_service.py @@ -120,7 +120,7 @@ UiBridgeWidgetTreeOperation, ) from openhcs.agent.services.ui_bridge_transport import UiBridgeControlClient -from openhcs.runtime.viewer_protocol import ViewerLaunchContextMode +from openhcs.runtime.viewer_protocol import HeadlessSession, ProjectedGraphicalSession from openhcs.runtime.zmq_application import OPENHCS_ENDPOINT_APPLICATION from python_introspect import to_jsonable @@ -896,7 +896,7 @@ def test_ui_bridge_service_resolves_projected_graphical_viewer_launch_context( launch_context = UiBridgeService(path_policy=object()).viewer_launch_context() - assert launch_context.mode is ViewerLaunchContextMode.PROJECTED_GRAPHICAL_SESSION + assert launch_context.session is ProjectedGraphicalSession assert launch_context.environment_overlay == { "DISPLAY": ":19", "XDG_RUNTIME_DIR": "/run/user/1000", @@ -1172,7 +1172,7 @@ def test_descriptor_resolver_reports_ambiguous_live_descriptors(monkeypatch, tmp first_bridge_id, second_bridge_id, } - assert launch_context.mode is ViewerLaunchContextMode.HEADLESS + assert launch_context.session is HeadlessSession def test_descriptor_resolver_reports_live_descriptors_for_missing_instance( diff --git a/tests/unit/test_napari_accepted_work_settlement.py b/tests/unit/test_napari_accepted_work_settlement.py index d9ede645af..10791fc561 100644 --- a/tests/unit/test_napari_accepted_work_settlement.py +++ b/tests/unit/test_napari_accepted_work_settlement.py @@ -30,6 +30,7 @@ NapariViewerServer, ) from openhcs.runtime.viewer_protocol import ( + ViewerConnectedExternal, NapariViewerServerRequest, ViewerControlResponse, ViewerControlMessageRequest, @@ -273,7 +274,7 @@ def test_initial_settlement_is_observable_before_qt_intake_and_completes( ), persistent=False, viewer_family=NapariViewer, ), ) - client.lifecycle_state.mark_connected_external() + client.lifecycle = ViewerConnectedExternal observations = [] client_worker = ThreadPoolExecutor(max_workers=1) late_observer = os.environ.get("OPENHCS_SETTLEMENT_QUALIFICATION_LATE_OBSERVER") == "true" diff --git a/tests/unit/test_napari_reset_transitions.py b/tests/unit/test_napari_reset_transitions.py index 224affb742..142808f2c0 100644 --- a/tests/unit/test_napari_reset_transitions.py +++ b/tests/unit/test_napari_reset_transitions.py @@ -21,7 +21,7 @@ NapariDimensionMode, NapariDisplayConfig, ) -from openhcs.runtime.viewer_display import NapariVariableSizeHandling +from openhcs.runtime.viewer_display import PadToMaxSize from openhcs.core.runtime_image_values import ImagePayloadMetadata from openhcs.core.source_metadata import SourceVoxelSpacing from openhcs.runtime.napari_streaming_handlers import ( @@ -129,7 +129,7 @@ def enqueue( colour_mode=NapariDimensionMode.LAYER, stack_mode=(NapariDimensionMode.STACK if z_domain else NapariDimensionMode.LAYER), time_mode=NapariDimensionMode.LAYER, - variable_size_handling=NapariVariableSizeHandling.PAD_TO_MAX, + variable_size_handling=PadToMaxSize, ) semantics = ViewerComponentAxisSemanticsFactory.from_display_config( ViewerMappingDisplayConfigInput( diff --git a/tests/unit/test_napari_streaming_handlers.py b/tests/unit/test_napari_streaming_handlers.py index 321c7d9b6f..01764a27a6 100644 --- a/tests/unit/test_napari_streaming_handlers.py +++ b/tests/unit/test_napari_streaming_handlers.py @@ -20,14 +20,13 @@ NapariDisplayConfig, NapariDimensionMode, ) -from openhcs.runtime.viewer_display import NapariVariableSizeHandling +from openhcs.runtime.viewer_display import PadToMaxSize, SeparateLayers from openhcs.core.runtime_image_values import ( ImagePayloadMetadata, ) from openhcs.core.runtime_plane_projection import RuntimePlaneAxis from openhcs.core.source_spatial_domain import SourceSpatialDomain from openhcs.runtime.viewer_protocol import ( - NapariLayerKind, ViewerControlResponseField, ViewerNavigationControlOptions, ViewerPayloadControlOptions, @@ -44,6 +43,9 @@ from openhcs.runtime.napari_viewer_server import NapariViewerServer from openhcs.runtime.viewer_protocol import NapariViewerServerRequest from openhcs.runtime.napari_streaming_handlers import ( + NapariImageLayer, + NapariPointsLayer, + NapariShapesLayer, NapariAggregateAxisBinding, NapariAggregateAxisBindingBuilder, NapariAggregateAxisBindingSet, @@ -77,7 +79,6 @@ ViewerRouteComponentValueTracker, ) from openhcs.runtime.viewer_protocol import ( - NapariLayerKind, ViewerComponentValueOrdering, ViewerControlResponse, ViewerControlResponseField, @@ -1313,7 +1314,7 @@ def test_napari_layer_update_authority_replaces_existing_image_without_global_ax authority = NapariLayerUpdates() first = authority.create_or_update( - layer_kind=NapariLayerKind.IMAGE, + layer_kind=NapariImageLayer, viewer=viewer, layers=layers, route_key="route-nuclei", @@ -1327,7 +1328,7 @@ def test_napari_layer_update_authority_replaces_existing_image_without_global_ax }, ) second = authority.create_or_update( - layer_kind=NapariLayerKind.IMAGE, + layer_kind=NapariImageLayer, viewer=viewer, layers=layers, route_key="route-nuclei", @@ -1374,7 +1375,7 @@ def test_napari_layer_update_authority_preserves_user_selected_layer(): authority = NapariLayerUpdates() new_layer = authority.create_or_update( - layer_kind=NapariLayerKind.IMAGE, + layer_kind=NapariImageLayer, viewer=viewer, layers=layers, route_key="route-overlay", @@ -1396,7 +1397,7 @@ def test_napari_layer_update_authority_transfers_active_replaced_layer(): authority = NapariLayerUpdates() first = authority.create_or_update( - layer_kind=NapariLayerKind.IMAGE, + layer_kind=NapariImageLayer, viewer=viewer, layers=layers, route_key="route-nuclei", @@ -1409,7 +1410,7 @@ def test_napari_layer_update_authority_transfers_active_replaced_layer(): ) viewer.layers.selection.active = first second = authority.create_or_update( - layer_kind=NapariLayerKind.IMAGE, + layer_kind=NapariImageLayer, viewer=viewer, layers=layers, route_key="route-nuclei", @@ -1432,7 +1433,7 @@ def test_napari_layer_update_authority_marks_color_images_as_rgb(): image = np.zeros((2, 16, 16, 3), dtype=np.uint8) layer = authority.create_or_update( - layer_kind=NapariLayerKind.IMAGE, + layer_kind=NapariImageLayer, viewer=viewer, layers=layers, route_key="route-overlay", @@ -4459,7 +4460,7 @@ def test_napari_layer_update_authority_declares_shapes_and_points_kwargs(): authority = NapariLayerUpdates() shapes = authority.create_or_update( - layer_kind=NapariLayerKind.SHAPES, + layer_kind=NapariShapesLayer, viewer=viewer, layers=layers, route_key="route-rois", @@ -4474,7 +4475,7 @@ def test_napari_layer_update_authority_declares_shapes_and_points_kwargs(): }, ) points = authority.create_or_update( - layer_kind=NapariLayerKind.POINTS, + layer_kind=NapariPointsLayer, viewer=viewer, layers=layers, route_key="route-spots", @@ -4837,15 +4838,13 @@ def _stream(config: NapariDisplayConfig): padded_server = _stream( replace( NapariDisplayConfig(), - variable_size_handling=NapariVariableSizeHandling.PAD_TO_MAX, + variable_size_handling=PadToMaxSize, ) ) assert len(padded_server.layer_route_state.layer_pending_updates) == 1 padded_route = padded_server.display_pipeline.scheduled[-1][2] padded_items = ( - napari_viewer_server.NapariVariableSizeDisplayStrategy.for_enum_member( - padded_route.display_config.variable_size_handling - ).materialize_layer_items( + padded_route.display_config.variable_size_handling.materialize_layer_items( padded_server.layer_route_state.pending_update_for( padded_route.route_key ).items, @@ -4859,7 +4858,7 @@ def _stream(config: NapariDisplayConfig): separate_server = _stream( replace( NapariDisplayConfig(), - variable_size_handling=NapariVariableSizeHandling.SEPARATE_LAYERS, + variable_size_handling=SeparateLayers, ) ) assert len(separate_server.layer_route_state.layer_pending_updates) == 2 @@ -4907,7 +4906,7 @@ def test_napari_configured_colormap_reaches_native_layer_and_invalid_name_fails( try: with pytest.raises((KeyError, ValueError)): NapariLayerUpdates().create_or_update( - layer_kind=NapariLayerKind.IMAGE, + layer_kind=NapariImageLayer, viewer=viewer, layers={}, route_key="invalid-colormap", diff --git a/tests/unit/test_streaming_service.py b/tests/unit/test_streaming_service.py index 4acc8a7537..9bdc72e709 100644 --- a/tests/unit/test_streaming_service.py +++ b/tests/unit/test_streaming_service.py @@ -69,6 +69,8 @@ NapariStreamLayerItem, ) from openhcs.runtime.viewer_protocol import ( + ViewerConnectedExternal, + ViewerStopped, DetachedViewerLaunchFailure, DetachedViewerServerEntrypointSpec, ManagedViewerLifecycleMixin, @@ -1271,7 +1273,7 @@ def lifecycle_manager(monkeypatch): ) monkeypatch.setattr( ManagedViewerLifecycleMixin, "force_stop", - lambda self: self.lifecycle_state.mark_stopped(), + lambda self: setattr(self, "lifecycle", ViewerStopped), ) monkeypatch.setattr( ManagedViewerLifecycleMixin, "start", @@ -1299,7 +1301,7 @@ def test_streaming_viewer_lifecycle_attaches_existing_viewer_without_restart( ) assert isinstance(viewer, NapariStreamVisualizer) - assert viewer.lifecycle_state.is_connected_external + assert viewer.lifecycle is ViewerConnectedExternal @pytest.mark.parametrize( @@ -1398,7 +1400,7 @@ def test_streaming_viewer_lifecycle_admits_new_launch_inside_managed_acquisition ) -> None: active_config = config_type(enabled=True, port=5563, persistent=True, listen_host="*") existing_viewer = active_config.create_visualizer(FakeFileManager()) - existing_viewer.lifecycle_state.mark_connected_external() + existing_viewer.lifecycle = ViewerConnectedExternal monkeypatch.setattr( ManagedViewerLifecycleMixin, "owned_viewer_process_is_alive", lambda self: owns_process, diff --git a/tests/unit/test_viewer_protocol.py b/tests/unit/test_viewer_protocol.py index a38880c30f..fd1412e6ae 100644 --- a/tests/unit/test_viewer_protocol.py +++ b/tests/unit/test_viewer_protocol.py @@ -27,6 +27,7 @@ ViewerStateControlOptions, ) from openhcs.runtime.viewer_protocol import ( + ViewerConnectedExternal, DetachedViewerLaunchLog, DetachedViewerLaunchRequest, DetachedViewerPythonArguments, @@ -451,7 +452,7 @@ def send(request): monkeypatch.setattr(ViewerControlMessageRequest, "send", send) viewer = StateViewer() - viewer.lifecycle_state.mark_connected_external() + viewer.lifecycle = ViewerConnectedExternal observed = viewer.read_viewer_state(timeout=7.5) @@ -539,7 +540,7 @@ def send(request): ) monkeypatch.setattr(viewer_protocol.time, "sleep", lambda _seconds: None) viewer = ProgressViewer() - viewer.lifecycle_state.mark_connected_external() + viewer.lifecycle = ViewerConnectedExternal observed_progress = [] assert viewer.settle_viewer_state( @@ -590,7 +591,7 @@ def test_managed_viewer_settlement_rejects_no_progress(monkeypatch): viewer_family=NapariViewer, ) ) - viewer.lifecycle_state.mark_connected_external() + viewer.lifecycle = ViewerConnectedExternal monotonic_values = iter((0.0, 0.0, 1.1)) monkeypatch.setattr(ViewerControlMessageRequest, "send", lambda _request: response) monkeypatch.setattr( @@ -1097,12 +1098,12 @@ def detached_server_arguments( viewer = ExternalViewer() assert not viewer.is_running - viewer.lifecycle_state.mark_connected_external() + viewer.lifecycle = ViewerConnectedExternal assert viewer.is_running viewer.connected = False assert not viewer.is_running - assert not viewer.lifecycle_state.is_active + assert not viewer.lifecycle.active def test_prepare_fresh_viewer_start_releases_endpoint_after_shutdown_ack(): From 974394b27ca47ce58a983d2d19f796838519693a Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:42:25 -0400 Subject: [PATCH 13/36] E1: viewer process-launch declaration is a declared wire message ViewerProcessLaunchConfig encodes and decodes through its fields (WireMessage); ViewerProcessLaunchField and the hand-written mapping are deleted. Decoding still requires every declared field. Co-Authored-By: Claude Opus 5.5 --- openhcs/core/streaming_config_factory.py | 45 ++++++++--------------- openhcs/runtime/viewer_control_actions.py | 2 +- tests/unit/test_streaming_service.py | 4 +- tests/unit/test_viewer_protocol.py | 8 ++-- 4 files changed, 24 insertions(+), 35 deletions(-) diff --git a/openhcs/core/streaming_config_factory.py b/openhcs/core/streaming_config_factory.py index 7a53b5b21a..bb6e36b0f4 100644 --- a/openhcs/core/streaming_config_factory.py +++ b/openhcs/core/streaming_config_factory.py @@ -3,8 +3,7 @@ from __future__ import annotations from collections.abc import Mapping -from dataclasses import dataclass, field -from enum import Enum +from dataclasses import dataclass, field, fields from typing import TYPE_CHECKING, ClassVar from objectstate import DataclassFieldAccess, get_base_config_type @@ -22,6 +21,7 @@ ViewerDisplayConfigABC, ) from zmqruntime.config import TransportMode, ZMQConfig +from zmqruntime.messages import WireMessage from zmqruntime.viewer_protocol import ViewerTransportEndpoint from openhcs.core.streaming_config_declarations import ViewerFamily @@ -77,16 +77,12 @@ def viewer_backend_kwargs( ) -class ViewerProcessLaunchField(str, Enum): - """Wire fields projected from the viewer process-launch declaration.""" - - QT_FONT_DPI = "qt_font_dpi" - LISTEN_HOST = "listen_host" - - @dataclass(frozen=True, slots=True) -class ViewerProcessLaunchConfig: - """Process-global settings that must exist before viewer construction.""" +class ViewerProcessLaunchConfig(WireMessage): + """Process-global settings that must exist before viewer construction. + + Its declared fields are its wire spelling on the viewer control boundary. + """ qt_font_dpi: int | None = None listen_host: str = "127.0.0.1" @@ -96,17 +92,13 @@ def __post_init__(self) -> None: raise TypeError("Viewer listen host must be a string.") if not self.listen_host.strip(): raise ValueError("Viewer listen host must not be blank.") + if isinstance(self.qt_font_dpi, bool) or not isinstance( + self.qt_font_dpi, int | None + ): + raise TypeError("Viewer Qt font DPI must be an integer or None.") if self.qt_font_dpi is not None and self.qt_font_dpi <= 0: raise ValueError("Viewer Qt font DPI must be positive when provided.") - def to_wire_mapping(self) -> dict[str, str | int | None]: - """Project the launch declaration onto the viewer control boundary.""" - - return { - ViewerProcessLaunchField.QT_FONT_DPI.value: self.qt_font_dpi, - ViewerProcessLaunchField.LISTEN_HOST.value: self.listen_host, - } - def matches_existing_viewer( self, active: "ViewerProcessLaunchConfig", @@ -128,17 +120,12 @@ def from_wire_mapping( cls, payload: Mapping[str, object], ) -> "ViewerProcessLaunchConfig": - """Rehydrate an exact launch declaration from viewer state.""" + """Rehydrate an exact launch declaration: every declared field is required.""" - value = payload[ViewerProcessLaunchField.QT_FONT_DPI.value] - if value is not None and ( - not isinstance(value, int) or isinstance(value, bool) - ): - raise TypeError("Viewer Qt font DPI must be an integer or None.") - return cls( - qt_font_dpi=value, - listen_host=payload[ViewerProcessLaunchField.LISTEN_HOST.value], - ) + missing = tuple(name.name for name in fields(cls) if name.name not in payload) + if missing: + raise KeyError(f"Viewer process-launch declaration is missing {missing!r}.") + return cls.from_dict(payload) @dataclass(frozen=True, slots=True) diff --git a/openhcs/runtime/viewer_control_actions.py b/openhcs/runtime/viewer_control_actions.py index 2f3cd509e9..7426dcb243 100644 --- a/openhcs/runtime/viewer_control_actions.py +++ b/openhcs/runtime/viewer_control_actions.py @@ -241,7 +241,7 @@ def handle(self, server: ViewerServerPort, message: Mapping[str, object]) -> Con response_type=self.response_type, fields={ ViewerControlField.PROCESS_LAUNCH.value: ( - server.process_launch.to_wire_mapping() + server.process_launch.to_dict() ) }, ) diff --git a/tests/unit/test_streaming_service.py b/tests/unit/test_streaming_service.py index 9bdc72e709..5098c48a61 100644 --- a/tests/unit/test_streaming_service.py +++ b/tests/unit/test_streaming_service.py @@ -234,7 +234,7 @@ def test_napari_viewer_reuse_requires_matching_process_launch(monkeypatch) -> No lambda _request: ViewerControlResponse( { "status": "success", - "process_launch": active_launch.to_wire_mapping(), + "process_launch": active_launch.to_dict(), } ), ) @@ -1409,7 +1409,7 @@ def test_streaming_viewer_lifecycle_admits_new_launch_inside_managed_acquisition ViewerControlMessageRequest, "send", lambda self: ViewerControlResponse({ "status": "success", - "process_launch": active_config.viewer_process_launch_config().to_wire_mapping(), + "process_launch": active_config.viewer_process_launch_config().to_dict(), }), ) lifecycle_manager.get_or_create_viewer( diff --git a/tests/unit/test_viewer_protocol.py b/tests/unit/test_viewer_protocol.py index fd1412e6ae..6b347f6a81 100644 --- a/tests/unit/test_viewer_protocol.py +++ b/tests/unit/test_viewer_protocol.py @@ -678,17 +678,19 @@ def test_viewer_process_launch_config_round_trips_exact_wire_declaration(): config = ViewerProcessLaunchConfig(qt_font_dpi=96, listen_host="192.0.2.1") assert ( - ViewerProcessLaunchConfig.from_wire_mapping(config.to_wire_mapping()) == config + ViewerProcessLaunchConfig.from_wire_mapping(config.to_dict()) == config ) with pytest.raises(KeyError): ViewerProcessLaunchConfig.from_wire_mapping({}) with pytest.raises(TypeError): - ViewerProcessLaunchConfig.from_wire_mapping({"qt_font_dpi": True}) + ViewerProcessLaunchConfig.from_wire_mapping( + {"qt_font_dpi": True, "listen_host": "127.0.0.1"} + ) with pytest.raises(ValueError): ViewerProcessLaunchConfig(qt_font_dpi=0) - with pytest.raises(TypeError, match="listen host must be a string"): + with pytest.raises(TypeError, match="listen_host must be str"): ViewerProcessLaunchConfig.from_wire_mapping( {"qt_font_dpi": None, "listen_host": True} ) From 39765316461ec035fb7c044bc8a939117284e9f7 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:43:01 -0400 Subject: [PATCH 14/36] E1: selection scope, update origin and update failure response are families; PlateManager document assignments are a record SelectedAllSelectionMode and its UiCodeDocumentSelectionMode alias are deleted: SelectedItems/AllItems own keeps() and pick(), and the wire keeps their registry names. DesktopUpdateCheckOrigin members declare quiet; DesktopUpdateProgressAction becomes ReopenAfterFailedUpdate/ ExitAfterFailedUpdate, each following itself. PlateManagerCodeNamespaceField is replaced by PlateManagerDocumentAssignments, whose fields are the names. Co-Authored-By: Claude Opus 5.5 --- openhcs/agent/dto/__init__.py | 1 - openhcs/agent/dto/ui_bridge.py | 10 +- openhcs/agent/services/llm_context_service.py | 6 +- .../agent/services/selected_plate_service.py | 4 +- openhcs/agent/services/ui_bridge_service.py | 6 +- openhcs/authoring/session/dataset_document.py | 18 +- openhcs/core/selection.py | 77 +++++++-- openhcs/desktop/update.py | 37 ++-- openhcs/desktop/update_worker.py | 64 ++++--- openhcs/mcp/dev_client_commands/ui.py | 10 +- openhcs/pyqt_gui/main.py | 17 +- openhcs/pyqt_gui/services/ui_agent_bridge.py | 22 ++- .../services/ui_bridge_live_overview.py | 8 +- .../services/ui_bridge_pipeline_editor.py | 15 +- .../services/ui_bridge_plate_manager.py | 44 +++-- openhcs/pyqt_gui/widgets/plate_manager.py | 14 +- .../ui/shared/plate_manager_code_document.py | 161 ++++++++---------- tests/unit/agent/test_ui_bridge_service.py | 5 +- tests/unit/desktop/test_desktop_update.py | 17 +- .../desktop/test_desktop_update_worker.py | 8 +- .../pyqt_gui/test_main_window_file_actions.py | 6 +- .../pyqt_gui/test_plate_manager_widget.py | 19 +-- tests/unit/pyqt_gui/test_ui_agent_bridge.py | 13 +- .../test_ui_bridge_live_measurements.py | 10 +- 24 files changed, 316 insertions(+), 276 deletions(-) diff --git a/openhcs/agent/dto/__init__.py b/openhcs/agent/dto/__init__.py index 144a967fce..e2ffa57a89 100644 --- a/openhcs/agent/dto/__init__.py +++ b/openhcs/agent/dto/__init__.py @@ -211,7 +211,6 @@ "UiCodeDocumentId", "UiCodeDocumentOptionalBaseRevision", "UiCodeDocumentRequest", - "UiCodeDocumentSelectionMode", "UiCodeDocumentSummary", "UiCodeDocumentValidationRequest", "UiCodeDocumentValidationResult", diff --git a/openhcs/agent/dto/ui_bridge.py b/openhcs/agent/dto/ui_bridge.py index 638bba5c3a..d587841eed 100644 --- a/openhcs/agent/dto/ui_bridge.py +++ b/openhcs/agent/dto/ui_bridge.py @@ -40,6 +40,7 @@ TransportMode, ) +from openhcs.core.selection import AllItems, SelectedItems from openhcs.agent.dto.session import DatasetRowState, PipelineStepState from openhcs.agent.dto.common import ( AgentDataclassCliRequest, @@ -101,9 +102,6 @@ from openhcs.agent.ui_bridge_identities import ( UiWidgetIdentityDeclaration as UiWidgetIdentityDeclaration, ) -from openhcs.core.selection import ( - SelectedAllSelectionMode as UiCodeDocumentSelectionMode, -) from openhcs.core.selection import ( SelectedScopeIdsArgument, SelectedScopeIdsCarrier, @@ -680,7 +678,7 @@ def from_fields( cls, *, document_id: str, - selection_mode: str = UiCodeDocumentSelectionMode.SELECTED.value, + selection_mode: str = SelectedItems.name, clean: bool = True, ) -> "UiCodeDocumentRequest": return cls( @@ -709,7 +707,7 @@ def from_fields( cls, *, surface_id: str = PlateManagerStateSurfaceIdentityDeclaration.require_value(), - selection_mode: str = UiCodeDocumentSelectionMode.ALL.value, + selection_mode: str = AllItems.name, base_revision_token: str | None = None, ) -> "UiStateSurfaceRequest": return cls( @@ -2405,7 +2403,7 @@ def from_fields( document_id: str, source: str, base_revision_token: str, - selection_mode: str = UiCodeDocumentSelectionMode.SELECTED.value, + selection_mode: str = SelectedItems.name, selected_scope_ids: tuple[str, ...] = (), require_confirmation: bool = True, snapshot_label: str | None = None, diff --git a/openhcs/agent/services/llm_context_service.py b/openhcs/agent/services/llm_context_service.py index 3a01816cdb..5e179742bb 100644 --- a/openhcs/agent/services/llm_context_service.py +++ b/openhcs/agent/services/llm_context_service.py @@ -62,7 +62,7 @@ UiLiveOverviewStateSurfaceIdentityDeclaration, ) from openhcs.ui.shared.plate_manager_code_document import ( - PlateManagerCodeNamespaceField, + PlateManagerDocumentAssignments, ) @@ -199,9 +199,7 @@ def render(cls, service: "AgentAuthoringContextService") -> str: del service code_document_id = PlateManagerOrchestratorCodeDocumentIdentity.require_value() state_surface_id = PlateManagerStateSurfaceIdentityDeclaration.require_value() - document_fields = ", ".join( - field.value for field in PlateManagerCodeNamespaceField - ) + document_fields = ", ".join(PlateManagerDocumentAssignments.names()) workflow_names = ", ".join( operation.operation_id for operation in SessionOperation.all() diff --git a/openhcs/agent/services/selected_plate_service.py b/openhcs/agent/services/selected_plate_service.py index ffdce1016e..753e7f57d8 100644 --- a/openhcs/agent/services/selected_plate_service.py +++ b/openhcs/agent/services/selected_plate_service.py @@ -5,6 +5,7 @@ from collections.abc import Mapping from dataclasses import dataclass +from openhcs.core.selection import SelectedItems from openhcs.agent.dto.common import AgentError, AgentWarning, SCHEMA_VERSION from python_introspect import JsonValue from openhcs.agent.dto.plate import ( @@ -35,7 +36,6 @@ UiBridgeService, ) from openhcs.core.plate_image_inventory import PlateFileKind -from openhcs.core.selection import SelectedAllSelectionMode @dataclass(frozen=True, slots=True) @@ -290,7 +290,7 @@ def selected_plate_state( UiBridgeGetStateSurfaceOperation, UiStateSurfaceRequest( surface_id=PlateManagerStateSurfaceIdentityDeclaration.require_value(), - selection_mode=SelectedAllSelectionMode.SELECTED.value, + selection_mode=SelectedItems.name, ), connection, ) diff --git a/openhcs/agent/services/ui_bridge_service.py b/openhcs/agent/services/ui_bridge_service.py index 4e659db252..2b4f5bd278 100644 --- a/openhcs/agent/services/ui_bridge_service.py +++ b/openhcs/agent/services/ui_bridge_service.py @@ -29,6 +29,7 @@ TransportMode, ) +from openhcs.core.selection import AllItems, SelectedItems from openhcs.agent.dto.common import SCHEMA_VERSION, AgentError from openhcs.agent.dto.session import SessionEventBatch, SessionEventsRequest from openhcs.agent.dto.ui_bridge import ( @@ -60,7 +61,6 @@ UiCodeDocumentCatalog, UiCodeDocumentIdentity, UiCodeDocumentRequest, - UiCodeDocumentSelectionMode, UiCodeDocumentSummary, UiCodeDocumentValidationRequest, UiCodeDocumentValidationResult, @@ -580,7 +580,7 @@ def failed(cls, request, errors): current_revision_token=None, current_snapshot=None, selection_mode=request.resolved_selection_mode( - UiCodeDocumentSelectionMode.SELECTED + SelectedItems ), selected_scope_ids=(), errors=errors, @@ -606,7 +606,7 @@ def failed(cls, request, errors): payload_schema="openhcs.ui.unavailable_state_surface.v1", payload={}, selection_mode=request.resolved_selection_mode( - UiCodeDocumentSelectionMode.ALL + AllItems ), selected_scope_ids=(), current_revision_token=None, diff --git a/openhcs/authoring/session/dataset_document.py b/openhcs/authoring/session/dataset_document.py index 600f4aa03a..5d8483234f 100644 --- a/openhcs/authoring/session/dataset_document.py +++ b/openhcs/authoring/session/dataset_document.py @@ -15,8 +15,9 @@ from objectstate.object_state import ObjectState, ObjectStateRegistry from objectstate.value_semantics import semantic_values_equal +from openhcs.core.selection import AllItems, SelectedItems, SelectionScope from openhcs.core.config import GlobalPipelineConfig, PipelineConfig -from openhcs.core.selection import SelectedAllSelectionMode, SelectedScopeIdsCarrier +from openhcs.core.selection import SelectedScopeIdsCarrier from objectstate import retain_only_authored_paths from openhcs.ui.shared.plate_manager_code_document import ( PlateManagerCodeDocumentAuthority, @@ -35,7 +36,7 @@ class DatasetDocumentScope(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "mode" __skip_if_no_key__ = True - mode: ClassVar[SelectedAllSelectionMode | None] = None + mode: ClassVar[type[SelectionScope] | None] = None selected_scope_ids: tuple[str, ...] = () @classmethod @@ -43,10 +44,9 @@ def from_carrier( cls, carrier: SelectedScopeIdsCarrier, *, - default: SelectedAllSelectionMode = SelectedAllSelectionMode.SELECTED, + default: type[SelectionScope] = SelectedItems, ) -> "DatasetDocumentScope": - mode = SelectedAllSelectionMode(carrier.resolved_selection_mode(default)) - return cls.__registry__[mode](selected_scope_ids=tuple(carrier.selected_scope_ids)) + return cls.__registry__[carrier.selection_scope(default)](selected_scope_ids=tuple(carrier.selected_scope_ids)) def require_payload_scope(self, scope_ids: tuple[str, ...]) -> None: if self.selected_scope_ids and frozenset(scope_ids) != frozenset( @@ -71,7 +71,7 @@ def synchronize( class SelectedDatasetDocumentScope(DatasetDocumentScope): """Upsert the selected datasets; keep every other dataset.""" - mode = SelectedAllSelectionMode.SELECTED + mode = SelectedItems def commits_global_draft(self, state: ObjectState | None) -> bool: return False @@ -84,7 +84,7 @@ def synchronize(self, session, payload) -> None: class AllDatasetDocumentScope(DatasetDocumentScope): """Make the dataset list exactly the document's.""" - mode = SelectedAllSelectionMode.ALL + mode = AllItems def commits_global_draft(self, state: ObjectState | None) -> bool: return bool(state and state.dirty_fields) @@ -125,7 +125,7 @@ def render_dataset_document( session: "Session", scope_ids: tuple[str, ...], *, - selection_mode: SelectedAllSelectionMode, + selection_mode: type[SelectionScope], ) -> DatasetDocument: payload = PlateManagerCodeDocumentAuthority.from_values( plate_paths=list(scope_ids), @@ -140,7 +140,7 @@ def render_dataset_document( return DatasetDocument( source=PlateManagerCodeDocumentAuthority.render(payload), payload=payload, - selection_mode=selection_mode.value, + selection_mode=selection_mode.name, selected_scope_ids=tuple(scope_ids), ) diff --git a/openhcs/core/selection.py b/openhcs/core/selection.py index 14808cc832..f90c316e14 100644 --- a/openhcs/core/selection.py +++ b/openhcs/core/selection.py @@ -2,29 +2,82 @@ from __future__ import annotations +from abc import ABC, abstractmethod from collections.abc import Iterable from dataclasses import dataclass -from enum import Enum +from typing import ClassVar +from metaclass_registry import AutoRegisterMeta -class SelectedAllSelectionMode(str, Enum): - SELECTED = "selected" - ALL = "all" + +class SelectionScope(ABC, metaclass=AutoRegisterMeta): + """Whether a view or document covers the selected items or all of them. + + ``name`` is the scope's spelling on the wire and in CLI choices. + """ + + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + + @staticmethod + def named(name: str) -> type["SelectionScope"]: + scope = SelectionScope.__registry__.get(name) + if scope is None: + raise ValueError( + f"Unknown selection mode {name!r}; expected one of " + f"{tuple(SelectionScope.__registry__)!r}." + ) + return scope + + @staticmethod + @abstractmethod + def keeps(selected: bool) -> bool: + """Whether an item with this selection state is in scope.""" + + @staticmethod + @abstractmethod + def pick(selected: tuple[str, ...], every: tuple[str, ...]) -> tuple[str, ...]: + """The ids in scope, given the selected ids (in selection order) and all ids.""" + + +class SelectedItems(SelectionScope): + name = "selected" + + @staticmethod + def keeps(selected: bool) -> bool: + return selected + + @staticmethod + def pick(selected: tuple[str, ...], every: tuple[str, ...]) -> tuple[str, ...]: + return selected + + +class AllItems(SelectionScope): + name = "all" + + @staticmethod + def keeps(selected: bool) -> bool: + return True + + @staticmethod + def pick(selected: tuple[str, ...], every: tuple[str, ...]) -> tuple[str, ...]: + return every @dataclass(frozen=True, kw_only=True, slots=True) class SelectionModeCarrier: selection_mode: str | None = None - def resolved_selection_mode( - self, - default: str | SelectedAllSelectionMode, - ) -> str: + def resolved_selection_mode(self, default: type[SelectionScope]) -> str: + """The requested mode's wire spelling, or ``default``'s.""" if self.selection_mode is None: - if isinstance(default, SelectedAllSelectionMode): - return default.value - return default - return self.selection_mode + return default.name + return SelectionScope.named(self.selection_mode).name + + def selection_scope(self, default: type[SelectionScope]) -> type[SelectionScope]: + return SelectionScope.named(self.resolved_selection_mode(default)) @dataclass(frozen=True, kw_only=True, slots=True) diff --git a/openhcs/desktop/update.py b/openhcs/desktop/update.py index cbcc0afee1..b9b6bc02bd 100644 --- a/openhcs/desktop/update.py +++ b/openhcs/desktop/update.py @@ -14,7 +14,7 @@ from enum import Enum from importlib.metadata import PackageNotFoundError, distribution from pathlib import Path -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, ClassVar from urllib.parse import urlparse from uuid import uuid4 @@ -27,6 +27,7 @@ from python_introspect import dataclass_from_mapping, to_jsonable from zmqruntime.process_launch import BackgroundProcessLaunchPolicy +from openhcs.core.selection import AllItems from openhcs import __version__ as OPENHCS_VERSION from openhcs.desktop.deployment import ( DESKTOP_RESTART_EXECUTABLE_ENVIRONMENT_VARIABLE, @@ -42,7 +43,6 @@ apply_dataset_document, render_dataset_document, ) -from openhcs.core.selection import SelectedAllSelectionMode from openhcs.ui.shared.plate_manager_code_document import ( PlateManagerCodeDocumentAuthority, ) @@ -71,11 +71,23 @@ class DesktopUpdateError(RuntimeError): """Raised when official release metadata cannot produce a safe update.""" -class DesktopUpdateCheckOrigin(Enum): +class DesktopUpdateCheckOrigin(ABC): """User-visible context that requested one release check.""" - EXPLICIT = "explicit" - STARTUP = "startup" + quiet: ClassVar[bool] + """Only an available update is shown; "up to date" and failures are logged.""" + + +class ExplicitUpdateCheck(DesktopUpdateCheckOrigin): + """The user asked for the check.""" + + quiet = False + + +class StartupUpdateCheck(DesktopUpdateCheckOrigin): + """The application checked on its own after startup.""" + + quiet = True class DesktopRestartPurpose(Enum): @@ -471,7 +483,7 @@ def capture( context = render_dataset_document( main_window.session, tuple(main_window.session.dataset_scope_ids()), - selection_mode=SelectedAllSelectionMode.ALL, + selection_mode=AllItems, ) session = cls.pending() session.directory.mkdir(parents=True) @@ -658,7 +670,7 @@ class DesktopUpdateCheckResult: """Successful update check plus the context that requested it.""" update: DesktopUpdate - origin: DesktopUpdateCheckOrigin + origin: type[DesktopUpdateCheckOrigin] @dataclass(frozen=True, slots=True) @@ -666,7 +678,7 @@ class DesktopUpdateCheckFailure: """Failed update check plus the context that requested it.""" message: str - origin: DesktopUpdateCheckOrigin + origin: type[DesktopUpdateCheckOrigin] class DesktopUpdateDialogPresenter: @@ -825,19 +837,14 @@ def __init__( self._network_manager = network_manager or QNetworkAccessManager(self) self._url_opener = url_opener or QDesktopServices.openUrl self._active_reply: QNetworkReply | None = None - self._active_origin: DesktopUpdateCheckOrigin | None = None + self._active_origin: type[DesktopUpdateCheckOrigin] | None = None def check_for_updates( self, - origin: DesktopUpdateCheckOrigin = DesktopUpdateCheckOrigin.EXPLICIT, + origin: type[DesktopUpdateCheckOrigin] = ExplicitUpdateCheck, ) -> bool: """Start a nonblocking check, returning false if one is already active.""" - if not isinstance(origin, DesktopUpdateCheckOrigin): - raise TypeError( - "DesktopUpdateService.check_for_updates requires " - f"DesktopUpdateCheckOrigin; got {type(origin).__name__}." - ) if self._active_reply is not None: return False diff --git a/openhcs/desktop/update_worker.py b/openhcs/desktop/update_worker.py index 7c99824254..70ab19e429 100644 --- a/openhcs/desktop/update_worker.py +++ b/openhcs/desktop/update_worker.py @@ -39,11 +39,25 @@ class DesktopUpdateProgressMode(Enum): COMPLETE = "determinate" -class DesktopUpdateProgressAction(Enum): - """Closed failure response axis sent from the progress UI to orchestration.""" +class DesktopUpdateFailureResponse(ABC): + """The user's answer to a failed update, sent from the progress UI.""" - REOPEN = "reopen" - EXIT = "exit" + @staticmethod + @abstractmethod + def follow(reopen: Callable[[], None]) -> None: + """Carry out this answer; ``reopen`` restarts the saved session.""" + + +class ReopenAfterFailedUpdate(DesktopUpdateFailureResponse): + @staticmethod + def follow(reopen: Callable[[], None]) -> None: + reopen() + + +class ExitAfterFailedUpdate(DesktopUpdateFailureResponse): + @staticmethod + def follow(reopen: Callable[[], None]) -> None: + return None class DesktopUpdateProgressUnavailable(RuntimeError): @@ -136,7 +150,7 @@ def output(self, message: str) -> None: """Present one line emitted by the active update command.""" @abstractmethod - def failure(self, message: str) -> DesktopUpdateProgressAction: + def failure(self, message: str) -> type[DesktopUpdateFailureResponse]: """Present a failure and return the user's declared recovery action.""" @abstractmethod @@ -610,7 +624,7 @@ def __init__( self._tk = tk self._events: queue.Queue[DesktopUpdateProgressEventABC] = queue.Queue() - self._actions: queue.Queue[DesktopUpdateProgressAction] = queue.Queue() + self._actions: queue.Queue[type[DesktopUpdateFailureResponse]] = queue.Queue() self._theme = theme self._completed = False self._failed = False @@ -746,13 +760,13 @@ def __init__( button_frame, text="Reopen OpenHCS", style="OpenHCS.TButton", - command=lambda: self._finish_with(DesktopUpdateProgressAction.REOPEN), + command=lambda: self._finish_with(ReopenAfterFailedUpdate), ) self._exit_button = ttk.Button( button_frame, text="Exit", style="OpenHCS.TButton", - command=lambda: self._finish_with(DesktopUpdateProgressAction.EXIT), + command=lambda: self._finish_with(ExitAfterFailedUpdate), ) self._root.protocol("WM_DELETE_WINDOW", self._on_close) @@ -762,7 +776,7 @@ def phase(self, phase: DesktopUpdatePhase) -> None: def output(self, message: str) -> None: self._events.put(DesktopUpdateOutputEvent(message)) - def failure(self, message: str) -> DesktopUpdateProgressAction: + def failure(self, message: str) -> type[DesktopUpdateFailureResponse]: self._events.put(DesktopUpdateFailureEvent(message)) return self._actions.get() @@ -786,7 +800,7 @@ def execute() -> None: self._root.after(40, self._poll_events) self._root.mainloop() if worker.is_alive(): - self._actions.put(DesktopUpdateProgressAction.EXIT) + self._actions.put(ExitAfterFailedUpdate) worker.join(timeout=5) if worker.is_alive() or results.empty(): return 1 @@ -850,13 +864,13 @@ def _poll_events(self) -> None: event.apply(self) self._root.after(40, self._poll_events) - def _finish_with(self, action: DesktopUpdateProgressAction) -> None: + def _finish_with(self, action: type[DesktopUpdateFailureResponse]) -> None: self._actions.put(action) self._root.destroy() def _on_close(self) -> None: if self._failed: - self._finish_with(DesktopUpdateProgressAction.EXIT) + self._finish_with(ExitAfterFailedUpdate) elif self._completed: self._root.destroy() else: @@ -914,19 +928,23 @@ def _recover_from_failure( restart_executable: str | None = None, ) -> None: _write_update_error(arguments.error_file, error_message) - action = progress.failure(error_message) - if not reopen_available or action is not DesktopUpdateProgressAction.REOPEN: + response = progress.failure(error_message) + if not reopen_available: return - restart_error = _restart_saved_session( - arguments, - launch_spec=launch_spec, - restart_executable=restart_executable, - ) - if restart_error is not None: - _write_update_error( - arguments.error_file, - f"{error_message}\n\n{restart_error}", + + def reopen() -> None: + restart_error = _restart_saved_session( + arguments, + launch_spec=launch_spec, + restart_executable=restart_executable, ) + if restart_error is not None: + _write_update_error( + arguments.error_file, + f"{error_message}\n\n{restart_error}", + ) + + response.follow(reopen) def _perform_update_transaction( diff --git a/openhcs/mcp/dev_client_commands/ui.py b/openhcs/mcp/dev_client_commands/ui.py index 368505fb91..f2bb25fb97 100644 --- a/openhcs/mcp/dev_client_commands/ui.py +++ b/openhcs/mcp/dev_client_commands/ui.py @@ -12,6 +12,7 @@ WindowSnapshotFrameCondition, ) +from openhcs.core.selection import AllItems, SelectedItems, SelectionScope from openhcs.agent.dto.session import ( DatasetRowState, SessionEventBatch, @@ -40,7 +41,6 @@ from openhcs.agent.ui_bridge_identities import ( PlateManagerStateSurfaceIdentityDeclaration, ) -from openhcs.core.selection import SelectedAllSelectionMode from openhcs.mcp.dev_client_commanding import ( CapabilityBackedCommandSpec, McpDevCommandSpec, @@ -102,8 +102,8 @@ def configure_parser(self, parser: argparse.ArgumentParser) -> None: ) parser.add_argument( "--selection-mode", - choices=tuple(mode.value for mode in SelectedAllSelectionMode), - default=SelectedAllSelectionMode.ALL.value, + choices=tuple(SelectionScope.__registry__), + default=AllItems.name, ) parser.add_argument("--base-revision-token") add_ui_connection_options(parser) @@ -185,8 +185,8 @@ def configure_parser(self, parser: argparse.ArgumentParser) -> None: "--wait-selection-mode", "--poll-selection-mode", dest="poll_selection_mode", - choices=tuple(mode.value for mode in SelectedAllSelectionMode), - default=SelectedAllSelectionMode.SELECTED.value, + choices=tuple(SelectionScope.__registry__), + default=SelectedItems.name, help=(f"Selection mode used while waiting on {state_surface_id}."), ) parser.add_argument( diff --git a/openhcs/pyqt_gui/main.py b/openhcs/pyqt_gui/main.py index 0e88fc96f2..8561d33b03 100644 --- a/openhcs/pyqt_gui/main.py +++ b/openhcs/pyqt_gui/main.py @@ -44,6 +44,7 @@ from pyqt_reactive.widgets.system_monitor import SystemMonitorWidget from zmqruntime import EndpointConnectionCancelledError +from openhcs.core.selection import AllItems, SelectedItems from openhcs.agent.services.endpoint_function_catalog_service import ( ZMQFunctionCatalogService, ) @@ -67,7 +68,6 @@ ) from openhcs.authoring.session.session import DispatcherThread, Session from openhcs.core.config import GlobalPipelineConfig -from openhcs.core.selection import SelectedAllSelectionMode from openhcs.agent.dto.common import AgentError from openhcs.authoring.session.operations.application import ( CheckForUpdates, @@ -85,7 +85,8 @@ DesktopRestartSession, DesktopRuntimeEnvironment, DesktopUpdateCheckFailure, - DesktopUpdateCheckOrigin, + ExplicitUpdateCheck, + StartupUpdateCheck, DesktopUpdateCheckResult, DesktopUpdateDialogPresenter, DesktopUpdateError, @@ -1073,11 +1074,11 @@ def save_pipeline(self): def load_orchestrator_configuration(self) -> None: """Open the dataset code document for the selected datasets.""" - self.plate_manager_widget.show_dataset_code(SelectedAllSelectionMode.SELECTED) + self.plate_manager_widget.show_dataset_code(SelectedItems) def save_orchestrator_configuration(self) -> None: """Open the dataset code document with every dataset, for export.""" - self.plate_manager_widget.show_dataset_code(SelectedAllSelectionMode.ALL) + self.plate_manager_widget.show_dataset_code(AllItems) def show_configuration(self): """Open the registered application configuration window.""" @@ -1415,7 +1416,7 @@ def restart_session(self) -> None: def check_for_updates(self) -> None: """Start an explicit, asynchronous stable-release check.""" if not self.desktop_update_service.check_for_updates( - DesktopUpdateCheckOrigin.EXPLICIT + ExplicitUpdateCheck ): return self.check_for_updates_action.setEnabled(False) @@ -1427,7 +1428,7 @@ def _check_for_updates_on_startup(self) -> None: if not self.runtime_context.ui_config.check_for_updates_on_startup: return if self.desktop_update_service.check_for_updates( - DesktopUpdateCheckOrigin.STARTUP + StartupUpdateCheck ): self.check_for_updates_action.setEnabled(False) @@ -1438,7 +1439,7 @@ def _on_update_check_completed( self.check_for_updates_action.setEnabled(True) update = result.update if not update.update_available: - if result.origin is DesktopUpdateCheckOrigin.EXPLICIT: + if not result.origin.quiet: self.status_message.emit("OpenHCS is up to date") self.desktop_update_presenter.show_up_to_date(update) return @@ -1484,7 +1485,7 @@ def _on_update_check_completed( def _on_update_check_failed(self, failure: DesktopUpdateCheckFailure) -> None: self.check_for_updates_action.setEnabled(True) - if failure.origin is DesktopUpdateCheckOrigin.STARTUP: + if failure.origin.quiet: logger.warning( "Startup OpenHCS update check failed: %s", failure.message, diff --git a/openhcs/pyqt_gui/services/ui_agent_bridge.py b/openhcs/pyqt_gui/services/ui_agent_bridge.py index 913d91547c..a4ab5cfb10 100644 --- a/openhcs/pyqt_gui/services/ui_agent_bridge.py +++ b/openhcs/pyqt_gui/services/ui_agent_bridge.py @@ -161,7 +161,7 @@ from openhcs.ui.shared.plate_manager_code_document import ( PlateManagerCodeDocumentAuthority, PlateManagerCodeNamespace, - PlateManagerCodeNamespaceField, + PlateManagerDocumentAssignments, PlateManagerOrchestratorCodePayload, ) @@ -171,15 +171,19 @@ UiBridgeObservedResultT = TypeVar("UiBridgeObservedResultT", bound=AgentResultEnvelope) +_ORCHESTRATOR_CODE_DOCUMENT_EXAMPLE = PlateManagerDocumentAssignments( + plate_paths="['/path/to/plate']", + global_config="GlobalPipelineConfig(...)", + per_plate_configs="{'/path/to/plate': PipelineConfig(...)}", + pipeline_data="{'/path/to/plate': [FunctionStep(...), ...]}", +) ORCHESTRATOR_CODE_DOCUMENT_PAYLOAD_HINT = ( "Use the plate-manager orchestrator code document shape: " - f"{PlateManagerCodeNamespaceField.PLATE_PATHS.value} = ['/path/to/plate']; " - f"{PlateManagerCodeNamespaceField.GLOBAL_CONFIG.value} = " - "GlobalPipelineConfig(...); " - f"{PlateManagerCodeNamespaceField.PER_PLATE_CONFIGS.value} = " - "{'/path/to/plate': PipelineConfig(...)}; " - f"{PlateManagerCodeNamespaceField.PIPELINE_DATA.value} = " - "{'/path/to/plate': [FunctionStep(...), ...]}. " + + "; ".join( + f"{name} = {getattr(_ORCHESTRATOR_CODE_DOCUMENT_EXAMPLE, name)}" + for name in PlateManagerDocumentAssignments.names() + ) + + ". " f"Read {PlateManagerOrchestratorCodeDocumentIdentity.require_value()} " "for the current template." ) @@ -331,7 +335,7 @@ def visit_Assign(self, node: ast.Assign) -> None: self._error("unsafe_assignment", "Only one named assignment is allowed.") return target = node.targets[0] - if target.id in PlateManagerCodeNamespaceField.allowed_assignment_names(): + if target.id in PlateManagerDocumentAssignments.names(): self.visit(node.value) return if target.id in self._helper_bindings: diff --git a/openhcs/pyqt_gui/services/ui_bridge_live_overview.py b/openhcs/pyqt_gui/services/ui_bridge_live_overview.py index 0e7b583550..99813cdcd1 100644 --- a/openhcs/pyqt_gui/services/ui_bridge_live_overview.py +++ b/openhcs/pyqt_gui/services/ui_bridge_live_overview.py @@ -7,9 +7,9 @@ from zmqruntime.viewer_state import ViewerInstance, ViewerStateManager +from openhcs.core.selection import AllItems from openhcs.agent.dto.common import AgentError, SCHEMA_VERSION from openhcs.agent.dto.ui_bridge import ( - UiCodeDocumentSelectionMode, UiLiveOverviewItem, UiLiveOverviewMetric, UiLiveOverviewSection, @@ -184,14 +184,14 @@ def summary(self) -> UiStateSurfaceSummary: title=self.identity.title, widget_id=self.identity.widget_id, readable=True, - supported_selection_modes=(UiCodeDocumentSelectionMode.ALL.value,), + supported_selection_modes=(AllItems.name,), current_selection_count=0, total_scope_count=1, ) def read(self, request: UiStateSurfaceRequest) -> UiStateSurfaceDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.ALL + AllItems ) try: state = UiLiveOverviewState( @@ -272,7 +272,7 @@ def _state_error( errors: tuple[AgentError, ...], ) -> UiStateSurfaceDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.ALL + AllItems ) state = UiLiveOverviewState( schema_version=SCHEMA_VERSION, diff --git a/openhcs/pyqt_gui/services/ui_bridge_pipeline_editor.py b/openhcs/pyqt_gui/services/ui_bridge_pipeline_editor.py index bacfc06baa..02edd49e3b 100644 --- a/openhcs/pyqt_gui/services/ui_bridge_pipeline_editor.py +++ b/openhcs/pyqt_gui/services/ui_bridge_pipeline_editor.py @@ -5,12 +5,12 @@ import hashlib from dataclasses import replace +from openhcs.core.selection import AllItems, SelectionScope from openhcs.agent.dto.common import AgentError, SCHEMA_VERSION from openhcs.agent.dto.ui_bridge import ( UiActionIdentity, UiActionInvokeRequest, UiActionSummary, - UiCodeDocumentSelectionMode, UiDebugActionState, UiDebugCursorState, UiDebugRuntimeFrameState, @@ -225,7 +225,7 @@ def summary(self) -> UiStateSurfaceSummary: def read(self, request: UiStateSurfaceRequest) -> UiStateSurfaceDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.ALL + AllItems ) try: state = self._state() @@ -559,7 +559,7 @@ def _state_error( errors: tuple[AgentError, ...], ) -> UiStateSurfaceDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.ALL + AllItems ) state = UiPipelineDebugSessionState( schema_version=SCHEMA_VERSION, @@ -644,7 +644,7 @@ def summary(self) -> UiStateSurfaceSummary: def read(self, request: UiStateSurfaceRequest) -> UiStateSurfaceDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.ALL + AllItems ) try: state = self._state(selection_mode=selection_mode) @@ -676,9 +676,8 @@ def _state(self, *, selection_mode: str) -> UiPipelineEditorState: view = PipelineStepsView.state_of( self._manager.session, self._manager.current_plate, selected ) - steps = view.steps - if selection_mode == UiCodeDocumentSelectionMode.SELECTED.value: - steps = tuple(step for step in steps if step.selected) + scope = SelectionScope.named(selection_mode) + steps = tuple(step for step in view.steps if scope.keeps(step.selected)) return UiPipelineEditorState( schema_version=SCHEMA_VERSION, summary=self.summary(), @@ -733,7 +732,7 @@ def _state_error( errors: tuple[AgentError, ...], ) -> UiStateSurfaceDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.ALL + AllItems ) state = UiPipelineEditorState( schema_version=SCHEMA_VERSION, diff --git a/openhcs/pyqt_gui/services/ui_bridge_plate_manager.py b/openhcs/pyqt_gui/services/ui_bridge_plate_manager.py index ba59cd15a6..a46f292d43 100644 --- a/openhcs/pyqt_gui/services/ui_bridge_plate_manager.py +++ b/openhcs/pyqt_gui/services/ui_bridge_plate_manager.py @@ -5,13 +5,13 @@ import hashlib from dataclasses import replace +from openhcs.core.selection import AllItems, SelectedItems, SelectionScope from openhcs.agent.dto.common import AgentError, SCHEMA_VERSION from openhcs.agent.dto.ui_bridge import ( UiCodeDocument, UiCodeDocumentApplyRequest, UiCodeDocumentApplyResult, UiCodeDocumentRequest, - UiCodeDocumentSelectionMode, UiCodeDocumentSummary, UiCodeDocumentValidationRequest, UiCodeDocumentValidationResult, @@ -35,7 +35,6 @@ ) from python_introspect import to_jsonable from objectstate.object_state import ObjectStateRegistry -from openhcs.core.selection import SelectedAllSelectionMode from openhcs.agent.dto.session import DatasetRowState from openhcs.authoring.session.dataset_document import ( AllDatasetDocumentScope, @@ -217,10 +216,10 @@ def summary(self) -> UiCodeDocumentSummary: def read(self, request: UiCodeDocumentRequest) -> UiCodeDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.SELECTED + SelectedItems ) if ( - selection_mode == UiCodeDocumentSelectionMode.SELECTED.value + selection_mode == SelectedItems.name and not self._manager.get_selected_items() ): return self._document_error( @@ -238,7 +237,7 @@ def read(self, request: UiCodeDocumentRequest) -> UiCodeDocument: ) try: context = self._manager.dataset_document( - SelectedAllSelectionMode(selection_mode) + SelectionScope.named(selection_mode) ) except Exception as exc: return self._document_error( @@ -315,7 +314,7 @@ def apply(self, request: UiCodeDocumentApplyRequest) -> UiCodeDocumentApplyResul operations = self._manager.code_document_operations( DatasetDocumentScope.from_carrier( request, - default=SelectedAllSelectionMode.ALL, + default=AllItems, ) ) result = self._execution_service.validate_source(request.source, operations) @@ -372,7 +371,7 @@ def _document_error( errors: tuple[AgentError, ...], ) -> UiCodeDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.SELECTED + SelectedItems ) return UiCodeDocument( schema_version=SCHEMA_VERSION, @@ -441,8 +440,7 @@ def _project(self, selection_mode: str) -> UiPlateManagerState: .lower(), ) for row in view.rows - if SelectedAllSelectionMode(selection_mode) is SelectedAllSelectionMode.ALL - or row.selected + if SelectionScope.named(selection_mode).keeps(row.selected) ) return UiPlateManagerState( schema_version=SCHEMA_VERSION, @@ -468,7 +466,7 @@ def summary(self) -> UiStateSurfaceSummary: def read(self, request: UiStateSurfaceRequest) -> UiStateSurfaceDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.ALL + AllItems ) try: state = self._project(selection_mode) @@ -488,7 +486,7 @@ def read(self, request: UiStateSurfaceRequest) -> UiStateSurfaceDocument: return self._document_from_state(state) def overview_sections(self) -> tuple[UiLiveOverviewSection, ...]: - state = self._project(UiCodeDocumentSelectionMode.ALL.value) + state = self._project(AllItems.name) rows = state.rows return ( UiLiveOverviewSection( @@ -563,7 +561,7 @@ def _state_error( errors: tuple[AgentError, ...], ) -> UiStateSurfaceDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.ALL + AllItems ) state = UiPlateManagerState( schema_version=SCHEMA_VERSION, @@ -660,8 +658,8 @@ def summary(self) -> UiStateSurfaceSummary: widget_id=self.identity.widget_id, readable=True, supported_selection_modes=( - UiCodeDocumentSelectionMode.SELECTED.value, - UiCodeDocumentSelectionMode.ALL.value, + SelectedItems.name, + AllItems.name, ), current_selection_count=sum( entry.subject_id in selected_plate_ids for entry in all_entries @@ -671,7 +669,7 @@ def summary(self) -> UiStateSurfaceSummary: def read(self, request: UiStateSurfaceRequest) -> UiStateSurfaceDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.ALL + AllItems ) try: state = self._project_state(selection_mode) @@ -732,19 +730,17 @@ def overview_sections(self) -> tuple[UiLiveOverviewSection, ...]: def _project_state(self, selection_mode: str) -> UiLiveMeasurementsState: all_entries = self._manager.session.live_measurements.semantic_entries() selected_plate_ids = self._selected_plate_ids() - mode = UiCodeDocumentSelectionMode(selection_mode) - entries = ( - all_entries - if mode is UiCodeDocumentSelectionMode.ALL - else tuple( - entry for entry in all_entries if entry.subject_id in selected_plate_ids - ) + scope = SelectionScope.named(selection_mode) + entries = tuple( + entry + for entry in all_entries + if scope.keeps(entry.subject_id in selected_plate_ids) ) entry_states = tuple(self._entry_state(entry) for entry in entries) return UiLiveMeasurementsState( schema_version=SCHEMA_VERSION, summary=self.summary(), - selection_mode=mode.value, + selection_mode=scope.name, selected_scope_ids=self._manager.selection_scope_ids(), object_state_token=ObjectStateRegistry.get_token(), retained_entry_count=len(all_entries), @@ -791,7 +787,7 @@ def _state_error( errors: tuple[AgentError, ...], ) -> UiStateSurfaceDocument: selection_mode = request.resolved_selection_mode( - UiCodeDocumentSelectionMode.ALL + AllItems ) state = UiLiveMeasurementsState( schema_version=SCHEMA_VERSION, diff --git a/openhcs/pyqt_gui/widgets/plate_manager.py b/openhcs/pyqt_gui/widgets/plate_manager.py index 426fa41c09..dd8f737f23 100644 --- a/openhcs/pyqt_gui/widgets/plate_manager.py +++ b/openhcs/pyqt_gui/widgets/plate_manager.py @@ -35,6 +35,7 @@ from pyqt_reactive.widgets.shared.scope_visual_config import ListItemType from typing_extensions import override +from openhcs.core.selection import SelectedItems, SelectionScope from openhcs.agent.dto.knowledge import KnowledgeBaseDocumentTarget from openhcs.agent.dto.session import DatasetRootsRequest from openhcs.agent.ui_bridge_identities import ( @@ -84,7 +85,6 @@ ) from openhcs.core.config import PipelineConfig from openhcs.constants.constants import OrchestratorState -from openhcs.core.selection import SelectedAllSelectionMode from openhcs.pyqt_gui.services.ui_bridge_contracts import ( UiOwnedStateSurfaceDeclaration, ) @@ -396,16 +396,12 @@ def show_item_editor(self, item: DatasetRow) -> None: def dataset_document( self, - selection_mode: SelectedAllSelectionMode = SelectedAllSelectionMode.SELECTED, + selection_mode: type[SelectionScope] = SelectedItems, ) -> DatasetDocument: """The dataset code document: the selection, else every dataset.""" every = tuple(row.scope_id for row in self.plates) - scope_ids = ( - every - if selection_mode is SelectedAllSelectionMode.ALL - else self.selection_scope_ids() or every - ) + scope_ids = selection_mode.pick(self.selection_scope_ids(), every) or every return render_dataset_document( self.session, scope_ids, selection_mode=selection_mode ) @@ -420,7 +416,7 @@ def code_document_operations(self, scope: DatasetDocumentScope): ).apply_namespace, ) - def show_dataset_code(self, selection_mode: SelectedAllSelectionMode) -> None: + def show_dataset_code(self, selection_mode: type[SelectionScope]) -> None: """Open the dataset code document in the code editor.""" document = self.dataset_document(selection_mode) @@ -550,7 +546,7 @@ class ShowDatasetCodePresenter(OperationPresenter): def present(self, renderer: GuiRenderer, request) -> None: del request - renderer.plate_manager.show_dataset_code(SelectedAllSelectionMode.SELECTED) + renderer.plate_manager.show_dataset_code(SelectedItems) class ShowLiveResultsPresenter(OperationPresenter): diff --git a/openhcs/ui/shared/plate_manager_code_document.py b/openhcs/ui/shared/plate_manager_code_document.py index 50e1d2f683..c0f6343aee 100644 --- a/openhcs/ui/shared/plate_manager_code_document.py +++ b/openhcs/ui/shared/plate_manager_code_document.py @@ -4,7 +4,6 @@ from collections.abc import Mapping, Sequence from dataclasses import dataclass, fields, is_dataclass -from enum import Enum from functools import cache from pathlib import Path from typing import Any, Self, get_args, get_origin, get_type_hints @@ -15,32 +14,37 @@ from openhcs.core.dataset_sources.dataset_scopes import DatasetScope -class PlateManagerCodeNamespaceField(str, Enum): - """Exact public assignments in a PlateManager code document.""" +class PlateManagerCodeNamespace(dict): + """Nominal execution namespace for PlateManager documents.""" - PLATE_PATHS = "plate_paths" - GLOBAL_CONFIG = "global_config" - PER_PLATE_CONFIGS = "per_plate_configs" - PIPELINE_DATA = "pipeline_data" - @classmethod - def allowed_assignment_names(cls) -> frozenset[str]: - return frozenset(field.value for field in cls) +@dataclass(frozen=True, slots=True) +class PlateManagerDocumentAssignments: + """The public assignments of a PlateManager code document, one field each. + The field names are the assigned names; nothing else may be assigned. + """ -class PlateManagerCodeNamespace(dict): - """Nominal execution namespace for PlateManager documents.""" + plate_paths: object + global_config: object + per_plate_configs: object + pipeline_data: object @classmethod - def from_mapping(cls, namespace: Mapping[str, object]) -> Self: - code_namespace = cls() - code_namespace.update(namespace) - return code_namespace + def names(cls) -> tuple[str, ...]: + return tuple(declared.name for declared in fields(cls)) - def require(self, field: PlateManagerCodeNamespaceField) -> object: - if field.value not in self: - raise ValueError(f"PlateManager code document must define {field.value!r}.") - return self[field.value] + @classmethod + def from_namespace(cls, namespace: Mapping[str, object]) -> Self: + for name in cls.names(): + if name not in namespace: + raise ValueError(f"PlateManager code document must define {name!r}.") + return cls(**{name: namespace[name] for name in cls.names()}) + + def to_namespace(self) -> PlateManagerCodeNamespace: + return PlateManagerCodeNamespace( + {name: getattr(self, name) for name in self.names()} + ) @dataclass(frozen=True, slots=True) @@ -79,38 +83,30 @@ def from_values( per_plate_configs: Mapping[str | Path, PipelineConfig], pipeline_data: Mapping[str | Path, Sequence[FunctionStep]], ) -> PlateManagerOrchestratorCodePayload: - namespace = PlateManagerCodeNamespace( - { - PlateManagerCodeNamespaceField.PLATE_PATHS.value: list(plate_paths), - PlateManagerCodeNamespaceField.GLOBAL_CONFIG.value: ( - global_pipeline_config - ), - PlateManagerCodeNamespaceField.PER_PLATE_CONFIGS.value: dict( - per_plate_configs - ), - PlateManagerCodeNamespaceField.PIPELINE_DATA.value: { - path: list(steps) for path, steps in pipeline_data.items() - }, - } + return cls._payload( + PlateManagerDocumentAssignments( + plate_paths=list(plate_paths), + global_config=global_pipeline_config, + per_plate_configs=dict(per_plate_configs), + pipeline_data={path: list(steps) for path, steps in pipeline_data.items()}, + ) ) - return cls.from_namespace(namespace) @classmethod def from_namespace( cls, namespace: Mapping[str, object], ) -> PlateManagerOrchestratorCodePayload: - values = PlateManagerCodeNamespace.from_mapping(namespace) - plate_paths_value = values.require(PlateManagerCodeNamespaceField.PLATE_PATHS) - global_config_value = values.require( - PlateManagerCodeNamespaceField.GLOBAL_CONFIG - ) - per_plate_configs_value = values.require( - PlateManagerCodeNamespaceField.PER_PLATE_CONFIGS - ) - pipeline_data_value = values.require( - PlateManagerCodeNamespaceField.PIPELINE_DATA - ) + return cls._payload(PlateManagerDocumentAssignments.from_namespace(namespace)) + + @classmethod + def _payload( + cls, assignments: PlateManagerDocumentAssignments + ) -> PlateManagerOrchestratorCodePayload: + plate_paths_value = assignments.plate_paths + global_config_value = assignments.global_config + per_plate_configs_value = assignments.per_plate_configs + pipeline_data_value = assignments.pipeline_data if not isinstance(plate_paths_value, list): raise TypeError("plate_paths must be a list of strings or Paths.") @@ -179,23 +175,14 @@ def to_namespace( per_plate_configs=payload.per_plate_configs, pipeline_data=payload.pipeline_data, ) - return PlateManagerCodeNamespace( - { - PlateManagerCodeNamespaceField.PLATE_PATHS.value: list( - normalized.plate_paths - ), - PlateManagerCodeNamespaceField.GLOBAL_CONFIG.value: ( - normalized.global_pipeline_config - ), - PlateManagerCodeNamespaceField.PER_PLATE_CONFIGS.value: dict( - normalized.per_plate_configs - ), - PlateManagerCodeNamespaceField.PIPELINE_DATA.value: { - path: list(steps) - for path, steps in normalized.pipeline_data.items() - }, - } - ) + return PlateManagerDocumentAssignments( + plate_paths=list(normalized.plate_paths), + global_config=normalized.global_pipeline_config, + per_plate_configs=dict(normalized.per_plate_configs), + pipeline_data={ + path: list(steps) for path, steps in normalized.pipeline_data.items() + }, + ).to_namespace() @classmethod def render( @@ -217,38 +204,26 @@ def render( scope_id: DatasetScope.parse(scope_id).code_value() for scope_id in normalized.plate_paths } - body = CodeBlock.from_items( - ( - Assignment( - PlateManagerCodeNamespaceField.PLATE_PATHS.value, - [ - code_value_by_scope[scope_id] - for scope_id in normalized.plate_paths - ], - ), - BlankLine(), - Assignment( - PlateManagerCodeNamespaceField.GLOBAL_CONFIG.value, - normalized.global_pipeline_config, - ), - BlankLine(), - Assignment( - PlateManagerCodeNamespaceField.PER_PLATE_CONFIGS.value, - { - code_value_by_scope[scope_id]: config - for scope_id, config in normalized.per_plate_configs.items() - }, - ), - BlankLine(), - Assignment( - PlateManagerCodeNamespaceField.PIPELINE_DATA.value, - { - code_value_by_scope[scope_id]: steps - for scope_id, steps in normalized.pipeline_data.items() - }, - ), - ) + document = PlateManagerDocumentAssignments( + plate_paths=[ + code_value_by_scope[scope_id] for scope_id in normalized.plate_paths + ], + global_config=normalized.global_pipeline_config, + per_plate_configs={ + code_value_by_scope[scope_id]: config + for scope_id, config in normalized.per_plate_configs.items() + }, + pipeline_data={ + code_value_by_scope[scope_id]: steps + for scope_id, steps in normalized.pipeline_data.items() + }, ) + items = [] + for name in document.names(): + if items: + items.append(BlankLine()) + items.append(Assignment(name, getattr(document, name))) + body = CodeBlock.from_items(tuple(items)) return generate_python_source( body, header=cls.HEADER, diff --git a/tests/unit/agent/test_ui_bridge_service.py b/tests/unit/agent/test_ui_bridge_service.py index 986576bdf2..2de2a13856 100644 --- a/tests/unit/agent/test_ui_bridge_service.py +++ b/tests/unit/agent/test_ui_bridge_service.py @@ -1,4 +1,5 @@ from __future__ import annotations +from openhcs.core.selection import AllItems, SelectedItems from openhcs.agent.dto.session import DatasetRowState import inspect @@ -629,7 +630,7 @@ def get_document( sha256="abc", current_revision_token="rev-1", current_snapshot=None, - selection_mode=request.resolved_selection_mode("selected"), + selection_mode=request.resolved_selection_mode(SelectedItems), selected_scope_ids=(), ) @@ -650,7 +651,7 @@ def get_state_surface( UiPlateManagerState( schema_version=SCHEMA_VERSION, summary=summary, - selection_mode=request.resolved_selection_mode("all"), + selection_mode=request.resolved_selection_mode(AllItems), selected_scope_ids=(PLATE_SCOPE_ID,), object_state_token=1, manager_execution_state="idle", diff --git a/tests/unit/desktop/test_desktop_update.py b/tests/unit/desktop/test_desktop_update.py index c9549fe798..1a6f80d1a2 100644 --- a/tests/unit/desktop/test_desktop_update.py +++ b/tests/unit/desktop/test_desktop_update.py @@ -41,7 +41,8 @@ DesktopRestartUiState, DesktopRuntimeEnvironment, DesktopUpdateCheckFailure, - DesktopUpdateCheckOrigin, + ExplicitUpdateCheck, + StartupUpdateCheck, DesktopUpdateCheckResult, DesktopUpdateDialogPresenter, DesktopUpdateError, @@ -222,7 +223,7 @@ def test_unavailable_release_service_emits_failure_without_blocking() -> None: failures = [] service.check_failed.connect(failures.append) - assert service.check_for_updates(DesktopUpdateCheckOrigin.STARTUP) + assert service.check_for_updates(StartupUpdateCheck) assert manager.request.url().toString() == LATEST_RELEASE_API_URL assert ( manager.request.attribute(QNetworkRequest.Attribute.Http2AllowedAttribute) @@ -235,7 +236,7 @@ def test_unavailable_release_service_emits_failure_without_blocking() -> None: assert failures == [ DesktopUpdateCheckFailure( message="Host not found", - origin=DesktopUpdateCheckOrigin.STARTUP, + origin=StartupUpdateCheck, ) ] assert reply.deleted @@ -312,7 +313,7 @@ def test_main_window_starts_check_only_from_explicit_action() -> None: assert not action.enabled assert main_like.desktop_update_service.origins == [ - DesktopUpdateCheckOrigin.EXPLICIT + ExplicitUpdateCheck ] assert status.messages == ["Checking for OpenHCS updates…"] @@ -320,7 +321,7 @@ def test_main_window_starts_check_only_from_explicit_action() -> None: @pytest.mark.parametrize( ("enabled", "expected_origins", "expected_action_enabled"), [ - (True, [DesktopUpdateCheckOrigin.STARTUP], False), + (True, [StartupUpdateCheck], False), (False, [], True), ], ) @@ -385,7 +386,7 @@ def test_main_window_available_update_saves_and_starts_restart(monkeypatch) -> N main_like, DesktopUpdateCheckResult( update=update, - origin=DesktopUpdateCheckOrigin.STARTUP, + origin=StartupUpdateCheck, ), ) @@ -421,14 +422,14 @@ def test_startup_check_is_quiet_when_current_or_unavailable() -> None: main_like, DesktopUpdateCheckResult( update=current, - origin=DesktopUpdateCheckOrigin.STARTUP, + origin=StartupUpdateCheck, ), ) main_module.OpenHCSMainWindow._on_update_check_failed( main_like, DesktopUpdateCheckFailure( message="offline", - origin=DesktopUpdateCheckOrigin.STARTUP, + origin=StartupUpdateCheck, ), ) diff --git a/tests/unit/desktop/test_desktop_update_worker.py b/tests/unit/desktop/test_desktop_update_worker.py index 0165a67239..3d252f5ea9 100644 --- a/tests/unit/desktop/test_desktop_update_worker.py +++ b/tests/unit/desktop/test_desktop_update_worker.py @@ -45,8 +45,8 @@ class _ProgressProbe: def __init__( self, *, - action: desktop_update_worker.DesktopUpdateProgressAction = ( - desktop_update_worker.DesktopUpdateProgressAction.EXIT + action: type[desktop_update_worker.DesktopUpdateFailureResponse] = ( + desktop_update_worker.ExitAfterFailedUpdate ), ) -> None: self.action = action @@ -638,7 +638,7 @@ def test_worker_relaunches_and_preserves_session_after_update_failure( ) -> None: calls = [] progress = _ProgressProbe( - action=desktop_update_worker.DesktopUpdateProgressAction.REOPEN + action=desktop_update_worker.ReopenAfterFailedUpdate ) monkeypatch.setattr( desktop_update_worker.DesktopUpdateProgressWindow, @@ -811,7 +811,7 @@ def test_unexpected_orchestration_exception_uses_visible_recovery_boundary( tmp_path: Path, ) -> None: progress = _ProgressProbe( - action=desktop_update_worker.DesktopUpdateProgressAction.REOPEN + action=desktop_update_worker.ReopenAfterFailedUpdate ) error_file = tmp_path / "update-error.txt" arguments = desktop_update_worker.parse_arguments(_worker_arguments(tmp_path)) diff --git a/tests/unit/pyqt_gui/test_main_window_file_actions.py b/tests/unit/pyqt_gui/test_main_window_file_actions.py index aaa1a11b50..5cecbf11dd 100644 --- a/tests/unit/pyqt_gui/test_main_window_file_actions.py +++ b/tests/unit/pyqt_gui/test_main_window_file_actions.py @@ -6,7 +6,7 @@ import pytest -from openhcs.core.selection import SelectedAllSelectionMode +from openhcs.core.selection import AllItems, SelectedItems from openhcs.pyqt_gui.main import OpenHCSMainWindow @@ -21,8 +21,8 @@ def show_dataset_code(self, selection_mode) -> None: @pytest.mark.parametrize( "action, mode", ( - (OpenHCSMainWindow.load_orchestrator_configuration, SelectedAllSelectionMode.SELECTED), - (OpenHCSMainWindow.save_orchestrator_configuration, SelectedAllSelectionMode.ALL), + (OpenHCSMainWindow.load_orchestrator_configuration, SelectedItems), + (OpenHCSMainWindow.save_orchestrator_configuration, AllItems), ), ) def test_file_actions_open_the_dataset_code_document(action, mode) -> None: diff --git a/tests/unit/pyqt_gui/test_plate_manager_widget.py b/tests/unit/pyqt_gui/test_plate_manager_widget.py index 3ba0b534cd..d9d2cd00f3 100644 --- a/tests/unit/pyqt_gui/test_plate_manager_widget.py +++ b/tests/unit/pyqt_gui/test_plate_manager_widget.py @@ -16,12 +16,12 @@ from zmqruntime.startup import EndpointStartupPhase, EndpointStartupStatus import openhcs.processing.backends.cellprofiler as cellprofiler_backend +from openhcs.core.selection import AllItems, SelectedItems from openhcs.agent.dto.ui_bridge import ( UiBridgeConfirmationRequirement, UiCodeDocumentApplyRequest, UiCodeDocumentId, UiCodeDocumentRequest, - UiCodeDocumentSelectionMode, UiStateSurfaceId, UiStateSurfaceRequest, UiWindowNavigateRequest, @@ -76,7 +76,6 @@ from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.core.progress import ProgressEvent -from openhcs.core.selection import SelectedAllSelectionMode from openhcs.core.source_bindings import ( MetadataSelector, NamedSourceBinding, @@ -172,11 +171,11 @@ def _bridge(manager) -> UiAgentBridgeService: ) -def _apply(bridge, source: str, *, selection_mode=UiCodeDocumentSelectionMode.ALL): +def _apply(bridge, source: str, *, selection_mode=AllItems): document = bridge.get_document( UiCodeDocumentRequest( document_id=UiCodeDocumentId.PLATE_MANAGER_ORCHESTRATOR.value, - selection_mode=selection_mode.value, + selection_mode=selection_mode.name, ) ) return bridge.apply_document( @@ -729,7 +728,7 @@ def test_restart_restores_a_missing_dataset_without_initializing_it(tmp_path) -> assert isinstance(outcome, DesktopRestartSucceeded) assert orchestrator.state is OrchestratorState.CREATED assert not orchestrator.is_initialized() - restored = gui.plate_manager.dataset_document(SelectedAllSelectionMode.ALL) + restored = gui.plate_manager.dataset_document(AllItems) assert restored.payload.plate_paths == (scope_id,) assert restored.payload.per_plate_configs == payload.per_plate_configs assert restored.payload.pipeline_data == payload.pipeline_data @@ -758,7 +757,7 @@ def test_a_missing_dataset_declaration_stays_editable_until_initialization( assert orchestrator.state is OrchestratorState.CREATED assert not orchestrator.is_initialized() rendered = render_dataset_document( - session, (scope_id,), selection_mode=SelectedAllSelectionMode.ALL + session, (scope_id,), selection_mode=AllItems ) assert rendered.payload.plate_paths == (scope_id,) with pytest.raises(FileNotFoundError, match="unavailable-plate"): @@ -951,7 +950,7 @@ def test_selected_code_document_replaces_only_the_selected_pipeline( [selected], {selected: [FunctionStep(name="New selected")]}, ), - selection_mode=UiCodeDocumentSelectionMode.SELECTED, + selection_mode=SelectedItems, ) assert result.applied @@ -1044,7 +1043,7 @@ def test_adopting_a_global_config_preserves_a_dirty_dataset_override(tmp_path) - def _render(session, *scope_ids): return render_dataset_document( - session, tuple(scope_ids), selection_mode=SelectedAllSelectionMode.ALL + session, tuple(scope_ids), selection_mode=AllItems ) @@ -1268,7 +1267,7 @@ def test_dataset_document_edits_during_a_run_and_recovers_after_its_failure( state = bridge.get_state_surface( UiStateSurfaceRequest( surface_id=UiStateSurfaceId.PLATE_MANAGER.value, - selection_mode=UiCodeDocumentSelectionMode.ALL.value, + selection_mode=AllItems.name, ) ) @@ -1312,7 +1311,7 @@ def apply(source): document = bridge.get_document( UiCodeDocumentRequest( document_id="window_code_document:pipeline_editor", - selection_mode=UiCodeDocumentSelectionMode.SELECTED.value, + selection_mode=SelectedItems.name, ) ) return bridge.apply_document( diff --git a/tests/unit/pyqt_gui/test_ui_agent_bridge.py b/tests/unit/pyqt_gui/test_ui_agent_bridge.py index 620d575856..61d93f9a39 100644 --- a/tests/unit/pyqt_gui/test_ui_agent_bridge.py +++ b/tests/unit/pyqt_gui/test_ui_agent_bridge.py @@ -56,6 +56,7 @@ from zmqruntime import DataControlPortPairAuthority, EndpointApplication from zmqruntime.config import TransportMode +from openhcs.core.selection import AllItems, SelectedItems from openhcs.agent.dto.common import SCHEMA_VERSION from openhcs.agent.dto.ui_bridge import ( UiBridgeOperationStatusRequest, @@ -67,7 +68,6 @@ UiCodeDocumentApplyRequest, UiCodeDocumentId, UiCodeDocumentRequest, - UiCodeDocumentSelectionMode, UiCodeDocumentValidationRequest, UiObjectStateFieldFilter, UiObjectStateScopeListRequest, @@ -219,7 +219,6 @@ PlateManagerCodeDocumentAuthority, ) from openhcs.ui.shared.plate_scope_identity import PipelineScopeIdentity -from openhcs.core.selection import SelectedAllSelectionMode from openhcs.runtime.zmq_config import OpenHCSZMQConfig from tests.unit.pyqt_gui.session_harness import session_gui, session_port from openhcs.core.progress.types import ( @@ -238,8 +237,8 @@ PLATE_SCOPE_ID = "plate-1" DATASET_SCOPE_ID = "/tmp/openhcs-ui-bridge-test/plate-1" PLATE_NAME = "plate 1" -ALL_SELECTION_MODE = UiCodeDocumentSelectionMode.ALL.value -SELECTED_SELECTION_MODE = UiCodeDocumentSelectionMode.SELECTED.value +ALL_SELECTION_MODE = AllItems.name +SELECTED_SELECTION_MODE = SelectedItems.name BRIDGE_INSTANCE_ID = "bridge-test" BRIDGE_AUTH_TOKEN = "secret-token" VALID_SOURCE = ( @@ -3631,11 +3630,7 @@ def selection_scope_ids(self) -> tuple[str, ...]: return tuple(self.selected) def dataset_document(self, selection_mode) -> FakeCodeDocumentContext: - scope_ids = ( - self.plate_scope_ids - if selection_mode is SelectedAllSelectionMode.ALL - else self.selected - ) + scope_ids = selection_mode.pick(tuple(self.selected), tuple(self.plate_scope_ids)) return FakeCodeDocumentContext( source=VALID_SOURCE, selected_scope_ids=tuple(scope_ids) ) diff --git a/tests/unit/pyqt_gui/test_ui_bridge_live_measurements.py b/tests/unit/pyqt_gui/test_ui_bridge_live_measurements.py index b0f41cb7bb..7dc5da7495 100644 --- a/tests/unit/pyqt_gui/test_ui_bridge_live_measurements.py +++ b/tests/unit/pyqt_gui/test_ui_bridge_live_measurements.py @@ -4,8 +4,8 @@ from dataclasses import dataclass from pathlib import Path +from openhcs.core.selection import AllItems, SelectedItems from openhcs.agent.dto.ui_bridge import ( - UiCodeDocumentSelectionMode, UiStateSurfaceRequest, ) from openhcs.agent.ui_bridge_identities import ( @@ -220,13 +220,13 @@ def test_live_measurement_surface_projects_exact_bounded_preview_and_revision() state = bridge.get_state_surface( UiStateSurfaceRequest( surface_id=declaration.surface_id, - selection_mode=UiCodeDocumentSelectionMode.ALL.value, + selection_mode=AllItems.name, ) ) poll = bridge.get_state_surface( UiStateSurfaceRequest( surface_id=declaration.surface_id, - selection_mode=UiCodeDocumentSelectionMode.ALL.value, + selection_mode=AllItems.name, base_revision_token=state.current_revision_token, ) ) @@ -273,7 +273,7 @@ def test_live_measurement_surface_honors_plate_selection_and_populates_overview( selected = bridge.get_state_surface( UiStateSurfaceRequest( surface_id=declaration.surface_id, - selection_mode=UiCodeDocumentSelectionMode.SELECTED.value, + selection_mode=SelectedItems.name, ) ) overview = bridge.get_state_surface( @@ -299,7 +299,7 @@ def test_live_measurement_surface_honors_plate_selection_and_populates_overview( filtered = bridge.get_state_surface( UiStateSurfaceRequest( surface_id=declaration.surface_id, - selection_mode=UiCodeDocumentSelectionMode.SELECTED.value, + selection_mode=SelectedItems.name, ) ) From 9c3ee00d698a167182d47be700a6375e87cb7d05 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:45:11 -0400 Subject: [PATCH 15/36] E1: CellProfiler interop kinds as families; grid ordering carries its numbering CellProfilerPipelineStage, ModuleSettingCoverageStatus, CellProfilerMeasurementFeatureKind (merged with its parser family) and MeasurementImageReferenceDomain become AutoRegisterMeta families whose members own what consumers switched on; CellProfilerPipelineFileKind and the CellProfilerMeasurementImageDomain alias are deleted. SpatialGridOrdering members carry whether numbers fill a row first, so the grid stops comparing members. Co-Authored-By: Claude Opus 5.5 --- benchmark/cellprofiler_comparison.py | 4 +- benchmark/converter/compatibility_matrix.py | 2 +- openhcs/core/measurement_image_alignment.py | 90 ++++--- openhcs/core/runtime_spatial_grid.py | 19 +- openhcs/interop/cellprofiler/__init__.py | 7 +- .../cellprofiler/measurement_lookup.py | 227 +++++++----------- .../module_artifact_declarations.py | 13 +- .../interop/cellprofiler/module_settings.py | 57 +++-- .../interop/cellprofiler/plate_workspace.py | 110 +++++---- .../cellprofiler/runtime/invocation.py | 25 +- .../cellprofiler/runtime/module_execution.py | 18 +- .../cellprofiler/runtime/profile_fields.py | 4 +- .../runtime/relationship_measurement_rows.py | 4 +- .../backends/cellprofiler/object_filtering.py | 12 +- .../test_cellprofiler_compatibility_matrix.py | 6 +- .../test_cellprofiler_module_execution.py | 43 ++-- .../test_cellprofiler_spreadsheet_export.py | 7 +- ...filer_uncovered_object_output_contracts.py | 6 +- .../test_exported_relationship_equivalence.py | 7 +- .../unit/test_measurement_image_alignment.py | 26 +- ...test_measurement_image_plane_projection.py | 4 +- tests/unit/test_settings_binder.py | 11 +- 22 files changed, 351 insertions(+), 351 deletions(-) diff --git a/benchmark/cellprofiler_comparison.py b/benchmark/cellprofiler_comparison.py index 81d97ddc29..0e12fa430e 100644 --- a/benchmark/cellprofiler_comparison.py +++ b/benchmark/cellprofiler_comparison.py @@ -271,7 +271,7 @@ def from_report( unmapped_cppipe_setting_row_count=sum( 1 for setting in report.cppipe_settings - if setting.coverage.value == "unmapped" + if not setting.coverage.is_covered ), supported_absorbed_processing_modules=( coverage.supported_absorbed_processing_modules @@ -304,7 +304,7 @@ def from_report( module_num=setting.module_num, setting_name=setting.setting_name, normalized_setting_name=setting.normalized_setting_name, - coverage=setting.coverage.value, + coverage=setting.coverage.name, value=setting.value, ) for setting in report.cppipe_settings diff --git a/benchmark/converter/compatibility_matrix.py b/benchmark/converter/compatibility_matrix.py index c3a3485896..1cb3122eb1 100644 --- a/benchmark/converter/compatibility_matrix.py +++ b/benchmark/converter/compatibility_matrix.py @@ -150,7 +150,7 @@ class CPPipeSettingCompatibilityCoverage(ModuleSettingRowRecord): case_name: str canonical_module_name: str - coverage: ModuleSettingCoverageStatus + coverage: type[ModuleSettingCoverageStatus] @dataclass(frozen=True, slots=True) diff --git a/openhcs/core/measurement_image_alignment.py b/openhcs/core/measurement_image_alignment.py index 1528fc95ea..ac6bfcfd81 100644 --- a/openhcs/core/measurement_image_alignment.py +++ b/openhcs/core/measurement_image_alignment.py @@ -5,8 +5,10 @@ import logging from abc import ABC, abstractmethod from dataclasses import dataclass, replace +from typing import ClassVar import numpy as np +from metaclass_registry import AutoRegisterMeta from openhcs.core.aligned_image_payload import AlignedImageStack, ImageOutputBundle from openhcs.core.runtime_array_values import RuntimeArrayData @@ -29,7 +31,6 @@ from openhcs.core.runtime_slice_projection import RuntimeSliceProjection from openhcs.core.source_spatial_domain import SourceSpatialDomainAdapter -from enum import Enum from openhcs.core.runtime_image_values import ImagePayload, owned_runtime_value logger = logging.getLogger(__name__) @@ -44,9 +45,9 @@ def alignment_image(self) -> RuntimeArrayData | AlignedImageStack: """Return the image-like payload used for measurement alignment.""" @property - def alignment_reference_domain(self) -> MeasurementImageReferenceDomain: + def alignment_reference_domain(self) -> type[MeasurementImageReferenceDomain]: """Return the domain that owns alignment shape decisions.""" - return MeasurementImageReferenceDomain.SOURCE_IMAGE + return SourceImageReference @property def alignment_source_aliases(self) -> tuple[str, ...]: @@ -119,11 +120,9 @@ def image_data(self) -> RuntimeArrayData | AlignedImageStack: return self.image.data @property - def reference_domain(self) -> MeasurementImageReferenceDomain: + def reference_domain(self) -> type[MeasurementImageReferenceDomain]: """Return the source-owned alignment reference domain.""" - return MeasurementImageReferenceDomain( - self.source.alignment_reference_domain, - ) + return self.source.alignment_reference_domain @property def source_aliases(self) -> tuple[str, ...]: @@ -322,17 +321,7 @@ def aligned(self) -> "MeasurementImageLabelAlignmentRequest": def _with_aligned_source(self) -> "MeasurementImageLabelAlignmentRequest": """Align an already projected request without repeating label projection.""" - request = self - if request.reference_domain is MeasurementImageReferenceDomain.SOURCE_IMAGE: - request = request.with_source_projected_image() - image = request.image - elif request.reference_domain is MeasurementImageReferenceDomain.OBJECT_LABELS: - image = request.object_label_reference_image() - else: - raise ValueError( - "Measurement image alignment has no declared reference-domain behavior " - f"for {request.reference_domain!r}." - ) + request, image = self.reference_domain.reference_image(self) label_payload = request.label_payload image_runtime_projection = RuntimeSliceProjection.preserved_context_for_value( image @@ -354,7 +343,7 @@ def _with_aligned_source(self) -> "MeasurementImageLabelAlignmentRequest": request.labels, label_payload=label_payload, ) - if request.reference_domain is MeasurementImageReferenceDomain.SOURCE_IMAGE + if request.reference_domain.labels_follow_image else request.labels ) self.validate_alignment( @@ -575,7 +564,7 @@ def from_request( logger, "measurement_object_labels_source_projection", source_projection_timer.elapsed(), - reference_domain=request.reference_domain.value, + reference_domain=request.reference_domain.name, source_type=type(request.image).__name__, labels_type=type(request.labels).__name__, source_aliases=request.source_aliases, @@ -591,7 +580,7 @@ def from_request( logger, "measurement_object_labels_source_context", source_context_timer.elapsed(), - reference_domain=request.reference_domain.value, + reference_domain=request.reference_domain.name, payload_reused=source_projected_payload is source_payload, ) image_request = replace( @@ -612,7 +601,7 @@ def from_request( logger, "measurement_object_labels_image_alignment", image_align_timer.elapsed(), - reference_domain=request.reference_domain.value, + reference_domain=request.reference_domain.name, align_image_to_labels=request.align_image_to_labels, aligned_type=type(aligned_image).__name__, ) @@ -620,7 +609,7 @@ def from_request( aligned_measurement_labels = ( aligned_request.labels if request.align_image_to_labels - and request.reference_domain is MeasurementImageReferenceDomain.SOURCE_IMAGE + and request.reference_domain.labels_follow_image else image_request.labels_for_image( aligned_image, source_projected_labels, @@ -637,7 +626,7 @@ def from_request( logger, "measurement_object_labels_label_alignment", label_align_timer.elapsed(), - reference_domain=request.reference_domain.value, + reference_domain=request.reference_domain.name, labels_reused=aligned_measurement_labels is source_projected_labels, ) completion_timer = RuntimeProfileTimer.start() @@ -649,7 +638,7 @@ def from_request( logger, "measurement_object_labels_completion_payload", completion_timer.elapsed(), - reference_domain=request.reference_domain.value, + reference_domain=request.reference_domain.name, payload_reused=completion_payload is source_projected_payload, ) return cls( @@ -743,8 +732,51 @@ def measurement_image_source_spatial_adapter( return SourceSpatialDomainAdapter.for_value(image) -class MeasurementImageReferenceDomain(str, Enum): - """Semantic image domain used as the reference for object measurement.""" +class MeasurementImageReferenceDomain(ABC, metaclass=AutoRegisterMeta): + """The image whose shape object measurement aligns to. + + Each domain is a subclass; ``name`` is its spelling in profiles. + """ + + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["MeasurementImageReferenceDomain"]]] = {} + + name: ClassVar[str | None] = None + labels_follow_image: ClassVar[bool] + """Object labels are projected onto the measurement image's shape.""" + bundles_source_metadata: ClassVar[bool] + """Several measurement images keep their source metadata side by side.""" + + @classmethod + @abstractmethod + def reference_image( + cls, + request: MeasurementImageLabelAlignmentRequest, + ) -> tuple[MeasurementImageLabelAlignmentRequest, RuntimeArrayData | AlignedImageStack]: + """Return the request projected for this domain and its reference image.""" + + +class SourceImageReference(MeasurementImageReferenceDomain): + """The source image owns the shape; labels follow it.""" + + name = "source_image" + labels_follow_image = True + bundles_source_metadata = False - SOURCE_IMAGE = "source_image" - OBJECT_LABELS = "object_labels" + @classmethod + def reference_image(cls, request): + projected = request.with_source_projected_image() + return projected, projected.image + + +class ObjectLabelsReference(MeasurementImageReferenceDomain): + """The object labels own the shape; the image is read in label space.""" + + name = "object_labels" + labels_follow_image = False + bundles_source_metadata = True + + @classmethod + def reference_image(cls, request): + return request, request.object_label_reference_image() diff --git a/openhcs/core/runtime_spatial_grid.py b/openhcs/core/runtime_spatial_grid.py index 5ad6f6e4c3..bde9e8a39f 100644 --- a/openhcs/core/runtime_spatial_grid.py +++ b/openhcs/core/runtime_spatial_grid.py @@ -54,10 +54,21 @@ def normalized(self, count: int, field_name: str) -> "SpatialGridAxis": class SpatialGridOrdering(str, Enum): - """Primary axis used when numbering positions in a spatial grid.""" + """Primary axis used when numbering positions in a spatial grid. - BY_ROWS = "rows" - BY_COLUMNS = "columns" + The members are DefineGrid's spot-ordering choices; each carries whether + object numbers fill one grid row before the next. + """ + + def __new__(cls, value: str, fills_each_row_first: bool): + obj = str.__new__(cls, value) + obj._value_ = value + obj._fills_each_row_first = fills_each_row_first + return obj + + BY_ROWS = ("rows", False) + BY_COLUMNS = ("columns", True) + fills_each_row_first = AliasProperty[bool]("_fills_each_row_first") class SpatialGridOrigin(str, Enum): @@ -389,7 +400,7 @@ def measurement_values(self) -> dict[str, int | float]: def derived_spot_table(self) -> tuple[tuple[int, ...], ...]: """Return the object-number topology declared by this grid.""" object_ids = np.arange(1, self.rows * self.columns + 1, dtype=np.int32) - if self.ordering is SpatialGridOrdering.BY_COLUMNS: + if self.ordering.fills_each_row_first: table = object_ids.reshape(self.rows, self.columns) else: table = object_ids.reshape(self.columns, self.rows).T diff --git a/openhcs/interop/cellprofiler/__init__.py b/openhcs/interop/cellprofiler/__init__.py index 6163d09cc5..51a5f20269 100644 --- a/openhcs/interop/cellprofiler/__init__.py +++ b/openhcs/interop/cellprofiler/__init__.py @@ -11,11 +11,11 @@ cellprofiler_runtime_equivalence_policy, ) from openhcs.interop.cellprofiler.measurement_lookup import ( - CellProfilerChildCountFeatureParser, CellProfilerMeasurementFeature, CellProfilerMeasurementFeatureKind, - CellProfilerMeasurementFeatureParser, - CellProfilerObjectCountFeatureParser, + ChildCountFeatureName, + ObjectCountFeatureName, + OtherFeatureName, child_count_feature_child_name, count_feature_object_name, ) @@ -31,7 +31,6 @@ from openhcs.interop.cellprofiler.runtime.invocation import ( CellProfilerImageRequest, CellProfilerMeasurementImage, - CellProfilerMeasurementImageDomain, CellProfilerSliceAlignedValues, ) from openhcs.interop.cellprofiler.setting_names import ( diff --git a/openhcs/interop/cellprofiler/measurement_lookup.py b/openhcs/interop/cellprofiler/measurement_lookup.py index d7eb4f6a63..24baec96b2 100644 --- a/openhcs/interop/cellprofiler/measurement_lookup.py +++ b/openhcs/interop/cellprofiler/measurement_lookup.py @@ -2,9 +2,8 @@ from __future__ import annotations -from abc import ABC, abstractmethod +from abc import ABC from dataclasses import dataclass -from enum import Enum from typing import ClassVar from metaclass_registry import AutoRegisterMeta @@ -82,186 +81,126 @@ def feature_name(cls, identity: str) -> str: -class CellProfilerMeasurementFeatureKind(Enum): - """CellProfiler measurement feature families with structured semantics.""" +class CellProfilerMeasurementFeatureKind(ABC, metaclass=AutoRegisterMeta): + """A CellProfiler measurement feature-name convention. - OBJECT_COUNT = "object_count" - CHILD_COUNT = "child_count" - OTHER = "other" + Each convention is a subclass that parses and renders its names. A kind + whose names target an object set reads the object name out of them. + """ + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["CellProfilerMeasurementFeatureKind"]]] = {} -@dataclass(frozen=True, slots=True) -class CellProfilerMeasurementFeature: - """Structured view of one CellProfiler measurement feature name.""" - - name: str - kind: CellProfilerMeasurementFeatureKind - object_name: str | None = None - - @classmethod - def parse(cls, feature_name: str | None) -> "CellProfilerMeasurementFeature | None": - """Parse a CellProfiler feature name into nominal feature semantics.""" - if feature_name is None: - return None - normalized = feature_name.strip() - if not normalized: - return None - for parser_type in CellProfilerMeasurementFeatureParser.__registry__.values(): - parsed = parser_type().parse_feature(normalized) - if parsed is not None: - return parsed - return cls( - normalized, - CellProfilerMeasurementFeatureKind.OTHER, - ) + name: ClassVar[str | None] = None + needs_prior_measurement_rows: ClassVar[bool] = True + """Its values come from earlier measurement rows, not from object labels.""" @classmethod - def object_count(cls, object_name: str) -> "CellProfilerMeasurementFeature": - return CellProfilerMeasurementFeatureParser.for_kind( - CellProfilerMeasurementFeatureKind.OBJECT_COUNT - ).feature_from_object_name(object_name) + def object_name_in(cls, feature_name: str) -> str | None: + """Return the object name ``feature_name`` targets in this convention.""" + return None @classmethod - def child_count(cls, child_object_name: str) -> "CellProfilerMeasurementFeature": - return CellProfilerMeasurementFeatureParser.for_kind( - CellProfilerMeasurementFeatureKind.CHILD_COUNT - ).feature_from_object_name(child_object_name) + def feature(cls, object_name: str) -> "CellProfilerMeasurementFeature": + """Return the feature this convention names for ``object_name``.""" + raise TypeError(f"{cls.__name__} does not name features after object sets.") - @classmethod - def child_count_object_names( - cls, - feature_names: tuple[object, ...], - ) -> tuple[str, ...]: - """Return ordered unique child object names referenced by count features.""" - child_names = tuple( - parsed.object_name - for feature_name in feature_names - for parsed in (cls.parse(str(feature_name)),) - if ( - parsed is not None - and parsed.kind is CellProfilerMeasurementFeatureKind.CHILD_COUNT - and parsed.object_name is not None - ) - ) - return tuple(dict.fromkeys(child_names)) +class ObjectCountFeatureName(CellProfilerMeasurementFeatureKind): + """CellProfiler ``Count_`` image-level object counts.""" -class CellProfilerMeasurementFeatureParser(ABC, metaclass=AutoRegisterMeta): - """Registered parser/renderer for one CellProfiler measurement feature family.""" + name = "object_count" + needs_prior_measurement_rows = False + prefix: ClassVar[str] = "Count_" - __registry_key__ = "kind_key" - __skip_if_no_key__ = True - kind: ClassVar[CellProfilerMeasurementFeatureKind | None] = None - kind_key: ClassVar[str | None] = None + @classmethod + def object_name_in(cls, feature_name: str) -> str | None: + if not feature_name.startswith(cls.prefix): + return None + return feature_name[len(cls.prefix) :].strip() or None @classmethod - def for_kind( - cls, - kind: CellProfilerMeasurementFeatureKind, - ) -> "CellProfilerMeasurementFeatureParser": - parser_type = cls.__registry__.get(kind.value) - if parser_type is None: - raise KeyError(f"No CellProfiler measurement parser registered for {kind}.") - return parser_type() - - @abstractmethod - def parse_feature( - self, - feature_name: str, - ) -> CellProfilerMeasurementFeature | None: - """Return parsed feature semantics when this parser owns the name.""" + def feature(cls, object_name: str) -> "CellProfilerMeasurementFeature": + normalized = object_name.strip() + if not normalized: + raise ValueError("Object-count feature requires a non-empty object name.") + return CellProfilerMeasurementFeature( + name=f"{cls.prefix}{normalized}", kind=cls, object_name=normalized + ) - @abstractmethod - def feature_from_object_name( - self, - object_name: str, - ) -> CellProfilerMeasurementFeature: - """Render a feature name for an object-targeted feature family.""" +class ChildCountFeatureName(CellProfilerMeasurementFeatureKind): + """CellProfiler ``Children__Count`` relationship counts.""" -class CellProfilerObjectCountFeatureParser(CellProfilerMeasurementFeatureParser): - """Parser for CellProfiler ``Count_`` image-level object counts.""" + name = "child_count" - kind = CellProfilerMeasurementFeatureKind.OBJECT_COUNT - kind_key = kind.value - prefix = "Count_" + @classmethod + def object_name_in(cls, feature_name: str) -> str | None: + return ChildCountFeatureDeclaration.from_feature_name(feature_name) - def parse_feature( - self, - feature_name: str, - ) -> CellProfilerMeasurementFeature | None: - if not feature_name.startswith(self.prefix): - return None - object_name = feature_name[len(self.prefix) :].strip() - if not object_name: - return None + @classmethod + def feature(cls, object_name: str) -> "CellProfilerMeasurementFeature": return CellProfilerMeasurementFeature( - name=feature_name, - kind=CellProfilerMeasurementFeatureKind.OBJECT_COUNT, - object_name=object_name, + name=ChildCountFeatureDeclaration.feature_name(object_name), + kind=cls, + object_name=object_name.strip(), ) - def feature_from_object_name( - self, - object_name: str, - ) -> CellProfilerMeasurementFeature: - normalized = object_name.strip() - if not normalized: - raise ValueError("Object-count feature requires a non-empty object name.") - return CellProfilerMeasurementFeature( - name=f"{self.prefix}{normalized}", - kind=CellProfilerMeasurementFeatureKind.OBJECT_COUNT, - object_name=normalized, - ) +class OtherFeatureName(CellProfilerMeasurementFeatureKind): + """Any other feature name; it names no object set.""" -class CellProfilerChildCountFeatureParser(CellProfilerMeasurementFeatureParser): - """Parser for CellProfiler ``Children__Count`` relationships.""" + name = "other" - kind = CellProfilerMeasurementFeatureKind.CHILD_COUNT - kind_key = kind.value - def parse_feature(self, feature_name: str) -> CellProfilerMeasurementFeature | None: - object_name = ChildCountFeatureDeclaration.from_feature_name(feature_name) - if object_name is None: +@dataclass(frozen=True, slots=True) +class CellProfilerMeasurementFeature: + """Structured view of one CellProfiler measurement feature name.""" + + name: str + kind: type[CellProfilerMeasurementFeatureKind] + object_name: str | None = None + + @classmethod + def parse(cls, feature_name: str | None) -> "CellProfilerMeasurementFeature | None": + """Parse a CellProfiler feature name into its convention and object.""" + if feature_name is None: return None - return CellProfilerMeasurementFeature( - name=feature_name, - kind=CellProfilerMeasurementFeatureKind.CHILD_COUNT, - object_name=object_name, - ) + normalized = feature_name.strip() + if not normalized: + return None + for kind in CellProfilerMeasurementFeatureKind.__registry__.values(): + object_name = kind.object_name_in(normalized) + if object_name is not None: + return cls(normalized, kind, object_name) + return cls(normalized, OtherFeatureName) - def feature_from_object_name( - self, object_name: str - ) -> CellProfilerMeasurementFeature: - feature_name = ChildCountFeatureDeclaration.feature_name(object_name) - return CellProfilerMeasurementFeature( - name=feature_name, - kind=CellProfilerMeasurementFeatureKind.CHILD_COUNT, - object_name=object_name.strip(), + @classmethod + def child_count_object_names( + cls, + feature_names: tuple[object, ...], + ) -> tuple[str, ...]: + """Return ordered unique child object names referenced by count features.""" + child_names = ( + ChildCountFeatureName.object_name_in(str(feature_name).strip()) + for feature_name in feature_names ) + return tuple(dict.fromkeys(name for name in child_names if name is not None)) def count_feature_object_name(feature_name: str | None) -> str | None: """Return the object-set name encoded by a CellProfiler Count_* feature.""" - parsed = CellProfilerMeasurementFeature.parse(feature_name) - if ( - parsed is None - or parsed.kind is not CellProfilerMeasurementFeatureKind.OBJECT_COUNT - ): + if feature_name is None: return None - return parsed.object_name + return ObjectCountFeatureName.object_name_in(feature_name.strip()) def child_count_feature_child_name(feature_name: str | None) -> str | None: """Return the child object name encoded by Children__Count.""" - parsed = CellProfilerMeasurementFeature.parse(feature_name) - if ( - parsed is None - or parsed.kind is not CellProfilerMeasurementFeatureKind.CHILD_COUNT - ): + if feature_name is None: return None - return parsed.object_name + return ChildCountFeatureName.object_name_in(feature_name.strip()) __all__ = declared_public_names( diff --git a/openhcs/interop/cellprofiler/module_artifact_declarations.py b/openhcs/interop/cellprofiler/module_artifact_declarations.py index 98420c562e..29d976e04c 100644 --- a/openhcs/interop/cellprofiler/module_artifact_declarations.py +++ b/openhcs/interop/cellprofiler/module_artifact_declarations.py @@ -389,7 +389,7 @@ def measurement_record_rows( from openhcs.core.runtime_measurements import MeasurementRowAxisField from openhcs.core.runtime_tabular_values import FieldSpec from openhcs.interop.cellprofiler.measurement_lookup import ( - CellProfilerMeasurementFeature, + ObjectCountFeatureName, ) row_batches = [] @@ -399,9 +399,7 @@ def measurement_record_rows( object_name=object_spec.name, domain_scope=request.object_label_output_domain_scope(), ) - count_feature = CellProfilerMeasurementFeature.object_count( - object_spec.name - ).name + count_feature = ObjectCountFeatureName.feature(object_spec.name).name label_plane_domains = location_rows.label_plane_domains() count_fields = ( FieldSpec(MeasurementRowAxisField.SLICE_INDEX.value, int), @@ -520,7 +518,6 @@ def prior_measurement_feature_names( from openhcs.interop.cellprofiler.measurement_lookup import ( CellProfilerMeasurementFeature, - CellProfilerMeasurementFeatureKind, ) return tuple( @@ -529,11 +526,7 @@ def prior_measurement_feature_names( for binding in cls.measurement_feature_setting_bindings() for feature_name in binding.feature_names(module) for feature in (CellProfilerMeasurementFeature.parse(feature_name),) - if ( - feature is not None - and feature.kind - is not CellProfilerMeasurementFeatureKind.OBJECT_COUNT - ) + if feature is not None and feature.kind.needs_prior_measurement_rows ) ) diff --git a/openhcs/interop/cellprofiler/module_settings.py b/openhcs/interop/cellprofiler/module_settings.py index 9b439eb120..9aa19f5bd2 100644 --- a/openhcs/interop/cellprofiler/module_settings.py +++ b/openhcs/interop/cellprofiler/module_settings.py @@ -3,6 +3,7 @@ from __future__ import annotations import inspect +from abc import ABC from collections.abc import Mapping from dataclasses import dataclass, field from enum import Enum @@ -17,6 +18,8 @@ get_type_hints, ) +from metaclass_registry import AutoRegisterMeta + from openhcs.interop.cellprofiler.setting_names import ( SettingNameFamily, setting_name_matches, @@ -53,35 +56,55 @@ class ModuleSettingRowRecord: class ModuleSettingCoverageRecord(ModuleSettingRowRecord): """Coverage status for one concrete CellProfiler setting row.""" - status: "ModuleSettingCoverageStatus" + status: type["ModuleSettingCoverageStatus"] -class ModuleSettingCoverageStatus(str, Enum): - """How one CellProfiler setting row was accounted for by import binding.""" +class ModuleSettingCoverageStatus(ABC, metaclass=AutoRegisterMeta): + """How one CellProfiler setting row was accounted for by import binding. - BOUND = "bound" - IGNORED = "ignored" - UNMAPPED = "unmapped" + Each status is a subclass; ``name`` is its spelling in coverage reports. + """ - @property - def is_covered(self) -> bool: - """Return whether the declaration accounted for this setting row.""" - return self is not ModuleSettingCoverageStatus.UNMAPPED + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["ModuleSettingCoverageStatus"]]] = {} - @classmethod + name: ClassVar[str | None] = None + is_covered: ClassVar[bool] = True + """The declaration accounted for this setting row.""" + + @staticmethod def for_setting( - cls, normalized_name: str, *, unmapped_kwargs: Mapping[str, Any], ignored_setting_names: frozenset[str], - ) -> "ModuleSettingCoverageStatus": - """Return the coverage status owned by this status enum.""" + ) -> type["ModuleSettingCoverageStatus"]: + """Return the status of one setting row after binding.""" if normalized_name not in unmapped_kwargs: - return cls.BOUND + return SettingBound if normalized_name in ignored_setting_names: - return cls.IGNORED - return cls.UNMAPPED + return SettingIgnored + return SettingUnmapped + + +class SettingBound(ModuleSettingCoverageStatus): + """The row was bound to a function keyword.""" + + name = "bound" + + +class SettingIgnored(ModuleSettingCoverageStatus): + """The module declaration deliberately ignores the row.""" + + name = "ignored" + + +class SettingUnmapped(ModuleSettingCoverageStatus): + """No binding accounted for the row.""" + + name = "unmapped" + is_covered = False @dataclass(frozen=True, slots=True) diff --git a/openhcs/interop/cellprofiler/plate_workspace.py b/openhcs/interop/cellprofiler/plate_workspace.py index 86f7e14541..d48b146564 100644 --- a/openhcs/interop/cellprofiler/plate_workspace.py +++ b/openhcs/interop/cellprofiler/plate_workspace.py @@ -2,11 +2,12 @@ from __future__ import annotations +from abc import ABC from dataclasses import dataclass -from enum import Enum from pathlib import Path from typing import ClassVar +from metaclass_registry import AutoRegisterMeta from polystore.base import _create_storage_registry from polystore.filemanager import FileManager @@ -26,43 +27,55 @@ from openhcs.core.dataset_sources.source_schema import SourceSchemaFilenameParser -class CellProfilerPipelineStage(Enum): - """Tutorial-stage semantics encoded in CellProfiler pipeline filenames.""" +class CellProfilerPipelineStage(ABC, metaclass=AutoRegisterMeta): + """Tutorial stage a CellProfiler pipeline filename declares. - START = 0 - FINAL = 1 - UNLABELED = 2 + Each stage is a subclass; registration order is the order pipelines of a + folder are listed in. + """ - @classmethod - def from_path(cls, cppipe_path: Path) -> "CellProfilerPipelineStage": - tokens = tuple( + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["CellProfilerPipelineStage"]]] = {} + + name: ClassVar[str | None] = None + filename_token: ClassVar[str | None] = None + """Filename token naming this stage; ``None`` for the unlabelled stage.""" + chosen_when_unique: ClassVar[bool] = False + """A folder's only pipeline of this stage is its default pipeline.""" + + @staticmethod + def of(cppipe_path: Path) -> type["CellProfilerPipelineStage"]: + tokens = { token.casefold() - for token in ( - cppipe_path.stem.replace("-", "_").replace(" ", "_").split("_") - ) + for token in cppipe_path.stem.replace("-", "_").replace(" ", "_").split("_") if token + } + stages = tuple(CellProfilerPipelineStage.__registry__.values()) + return next( + (stage for stage in stages if stage.filename_token in tokens), + UnlabeledCellProfilerPipeline, ) - if "start" in tokens: - return cls.START - if "final" in tokens: - return cls.FINAL - return cls.UNLABELED + @classmethod + def rank(cls) -> int: + return tuple(CellProfilerPipelineStage.__registry__.values()).index(cls) -class CellProfilerPipelineFileKind(Enum): - """Nominal file categories considered during `.cppipe` discovery.""" - PIPELINE = "pipeline" - APPLEDOUBLE_SIDECAR = "appledouble_sidecar" - OTHER = "other" +class StartCellProfilerPipeline(CellProfilerPipelineStage): + name = "start" + filename_token = "start" - @classmethod - def from_path(cls, path: Path) -> "CellProfilerPipelineFileKind": - if path.suffix != ".cppipe": - return cls.OTHER - if AppleDoubleResourceForkFilename.matches(path.name): - return cls.APPLEDOUBLE_SIDECAR - return cls.PIPELINE + +class FinalCellProfilerPipeline(CellProfilerPipelineStage): + name = "final" + filename_token = "final" + chosen_when_unique = True + + +class UnlabeledCellProfilerPipeline(CellProfilerPipelineStage): + name = "unlabeled" + chosen_when_unique = True class AppleDoubleResourceForkFilename: @@ -80,24 +93,24 @@ class CellProfilerPipelineFile: """Visible `.cppipe` file with ordering semantics for one source folder.""" path: Path - stage: CellProfilerPipelineStage + stage: type[CellProfilerPipelineStage] @classmethod def from_path(cls, cppipe_path: Path) -> "CellProfilerPipelineFile": return cls( path=cppipe_path, - stage=CellProfilerPipelineStage.from_path(cppipe_path), + stage=CellProfilerPipelineStage.of(cppipe_path), ) @property def sort_key(self) -> tuple[int, str]: - return (self.stage.value, self.path.name.casefold()) + return (self.stage.rank(), self.path.name.casefold()) @staticmethod def is_visible_cppipe(cppipe_path: Path) -> bool: - return ( - CellProfilerPipelineFileKind.from_path(cppipe_path) - is CellProfilerPipelineFileKind.PIPELINE + """A ``.cppipe`` file that is not a macOS AppleDouble sidecar.""" + return cppipe_path.suffix == ".cppipe" and not ( + AppleDoubleResourceForkFilename.matches(cppipe_path.name) ) @@ -188,23 +201,16 @@ def default_cppipe_path(self) -> Path | None: if same_named_cppipe in candidates: return same_named_cppipe - final_pipelines = tuple( - path - for path in candidates - if CellProfilerPipelineStage.from_path(path) - is CellProfilerPipelineStage.FINAL - ) - if len(final_pipelines) == 1: - return final_pipelines[0] - - unlabeled_pipelines = tuple( - path - for path in candidates - if CellProfilerPipelineStage.from_path(path) - is CellProfilerPipelineStage.UNLABELED - ) - if len(unlabeled_pipelines) == 1: - return unlabeled_pipelines[0] + for stage in CellProfilerPipelineStage.__registry__.values(): + if not stage.chosen_when_unique: + continue + staged = tuple( + path + for path in candidates + if CellProfilerPipelineStage.of(path) is stage + ) + if len(staged) == 1: + return staged[0] return candidates[0] diff --git a/openhcs/interop/cellprofiler/runtime/invocation.py b/openhcs/interop/cellprofiler/runtime/invocation.py index 25d0030abf..8f63960d0f 100644 --- a/openhcs/interop/cellprofiler/runtime/invocation.py +++ b/openhcs/interop/cellprofiler/runtime/invocation.py @@ -15,6 +15,7 @@ MeasurementImageAlignmentSource, MeasurementImageReferenceDomain, PreparedMeasurementObjectLabels, + SourceImageReference, ) from openhcs.core.function_patterns import InvocationArtifactInputEdgePlan from openhcs.core.runtime_adapters import ( @@ -433,9 +434,6 @@ def invocation_kwargs( } -CellProfilerMeasurementImageDomain = MeasurementImageReferenceDomain - - @dataclass(frozen=True, slots=True, kw_only=True) class CellProfilerMeasurementImage( CellProfilerSourceIdentityMixin, @@ -452,18 +450,18 @@ class CellProfilerMeasurementImage( payload: RuntimeCallableArgument source_aliases: tuple[str, ...] = () align_to_labels: bool = True - reference_domain: CellProfilerMeasurementImageDomain = ( - CellProfilerMeasurementImageDomain.SOURCE_IMAGE - ) + reference_domain: type[MeasurementImageReferenceDomain] = SourceImageReference def __post_init__(self) -> None: object.__setattr__(self, "payload", owned_runtime_value(self.payload)) self.validate_source_identity() - if not isinstance(self.reference_domain, CellProfilerMeasurementImageDomain): + if not ( + isinstance(self.reference_domain, type) + and issubclass(self.reference_domain, MeasurementImageReferenceDomain) + ): raise TypeError( - "CellProfilerMeasurementImage.reference_domain must be " - "CellProfilerMeasurementImageDomain, got " - f"{type(self.reference_domain).__name__}." + "CellProfilerMeasurementImage.reference_domain must be a " + f"MeasurementImageReferenceDomain class, got {self.reference_domain!r}." ) object.__setattr__( self, "payload", self.execution_mode.executable_payload(self.payload) @@ -475,10 +473,7 @@ def source_metadata_composition_mode( sources: tuple["CellProfilerMeasurementImage", ...], ) -> ImagePayloadMetadataCompositionMode: """Return source metadata topology declared by measurement reference domains.""" - if all( - source.reference_domain is CellProfilerMeasurementImageDomain.OBJECT_LABELS - for source in sources - ): + if all(source.reference_domain.bundles_source_metadata for source in sources): return ImagePayloadMetadataCompositionMode.BUNDLE return ImagePayloadMetadataCompositionMode.STACK @@ -488,7 +483,7 @@ def alignment_image(self) -> RuntimeCallableArgument: return self.payload @property - def alignment_reference_domain(self) -> CellProfilerMeasurementImageDomain: + def alignment_reference_domain(self) -> type[MeasurementImageReferenceDomain]: """Return the reference domain that owns alignment shape decisions.""" return self.reference_domain diff --git a/openhcs/interop/cellprofiler/runtime/module_execution.py b/openhcs/interop/cellprofiler/runtime/module_execution.py index d7fa4dae61..a0dd7ac063 100644 --- a/openhcs/interop/cellprofiler/runtime/module_execution.py +++ b/openhcs/interop/cellprofiler/runtime/module_execution.py @@ -74,7 +74,11 @@ from openhcs.interop.cellprofiler.runtime.invocation import ( CellProfilerImageRequest, CellProfilerMeasurementImage, - CellProfilerMeasurementImageDomain, +) +from openhcs.core.measurement_image_alignment import ( + MeasurementImageReferenceDomain, + ObjectLabelsReference, + SourceImageReference, ) from openhcs.interop.cellprofiler.runtime.measurement_execution_support import ( CellProfilerRuntimeProfiler, @@ -630,7 +634,7 @@ def _measurement_image_inputs( image_inputs, adapter, current_image, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) def _independent_measurement_image_inputs( @@ -656,7 +660,7 @@ def _independent_measurement_image_inputs( source_image_name=None, source_aliases=(), payload=current_image, - reference_domain=CellProfilerMeasurementImageDomain.SOURCE_IMAGE, + reference_domain=SourceImageReference, ), ) @@ -666,8 +670,8 @@ def _resolved_measurement_images( adapter: CellProfilerRuntimeAdapter, current_image: RuntimeCallableArgument, *, - reference_domain: CellProfilerMeasurementImageDomain = ( - CellProfilerMeasurementImageDomain.SOURCE_IMAGE + reference_domain: type[MeasurementImageReferenceDomain] = ( + SourceImageReference ), ) -> tuple[CellProfilerMeasurementImage, ...]: source_aliases = ArtifactSpecCollection(image_inputs).names() @@ -689,7 +693,7 @@ def _resolved_measurement_image( current_image: RuntimeCallableArgument, source_aliases: tuple[str, ...], *, - reference_domain: CellProfilerMeasurementImageDomain, + reference_domain: type[MeasurementImageReferenceDomain], ) -> CellProfilerMeasurementImage: value = RuntimeInputBindingRequest( adapter=adapter, @@ -742,7 +746,7 @@ def _object_label_measurement_image( source_image_name=None, source_aliases=(), payload=reference_image, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, plane_projection=preserved_image_plane_projection( reference_image, adapter, diff --git a/openhcs/interop/cellprofiler/runtime/profile_fields.py b/openhcs/interop/cellprofiler/runtime/profile_fields.py index 22e01c17c0..9ef4f5d20a 100644 --- a/openhcs/interop/cellprofiler/runtime/profile_fields.py +++ b/openhcs/interop/cellprofiler/runtime/profile_fields.py @@ -31,7 +31,7 @@ def object_label_stage_profile_fields( ("stage", stage), ("object", object_spec.name), ("source_image_name", measurement_image.source_image_name), - ("reference_domain", measurement_image.reference_domain.value), + ("reference_domain", measurement_image.reference_domain.name), ("value_type", type(value).__name__), ("data_type", type(label_data).__name__), ("data_shape", _shape_for(label_data)), @@ -62,7 +62,7 @@ def dense_label_argument_stage_profile_fields( ("stage", stage), ("object", object_spec.name), ("source_image_name", measurement_image.source_image_name), - ("reference_domain", measurement_image.reference_domain.value), + ("reference_domain", measurement_image.reference_domain.name), ("value_type", type(value).__name__), ("data_type", type(data).__name__), ("data_shape", _shape_for(data)), diff --git a/openhcs/interop/cellprofiler/runtime/relationship_measurement_rows.py b/openhcs/interop/cellprofiler/runtime/relationship_measurement_rows.py index 257bb2c1d1..8d29007b36 100644 --- a/openhcs/interop/cellprofiler/runtime/relationship_measurement_rows.py +++ b/openhcs/interop/cellprofiler/runtime/relationship_measurement_rows.py @@ -39,7 +39,7 @@ DirectParentReferenceMeasurementFeature, ) from openhcs.interop.cellprofiler.measurement_lookup import ( - CellProfilerMeasurementFeature, + ChildCountFeatureName, ) from openhcs.core.steps.function_runtime import RuntimeCallableArgument from openhcs.interop.cellprofiler.runtime.runtime_profile import ( @@ -295,7 +295,7 @@ def child_count_rows_for_ids( ) parent_number = parent_numbers[parent_id] counts[parent_number] = counts[parent_number] + 1 - feature_name = CellProfilerMeasurementFeature.child_count(child_spec.name).name + feature_name = ChildCountFeatureName.feature(child_spec.name).name rows = tuple( { MeasurementRowAxisField.OBJECT_NAME.value: parent_spec.name, diff --git a/openhcs/processing/backends/cellprofiler/object_filtering.py b/openhcs/processing/backends/cellprofiler/object_filtering.py index 5afd104623..69d0017a05 100644 --- a/openhcs/processing/backends/cellprofiler/object_filtering.py +++ b/openhcs/processing/backends/cellprofiler/object_filtering.py @@ -3,7 +3,7 @@ from __future__ import annotations from openhcs.interop.cellprofiler.measurement_lookup import ( CellProfilerMeasurementFeature, - CellProfilerMeasurementFeatureKind, + child_count_feature_child_name, ) from openhcs.interop.cellprofiler.setting_names import ( RepeatedSettingSequence, @@ -1419,12 +1419,10 @@ def measurement_values_for_feature( ): return self.measurement_values - feature = CellProfilerMeasurementFeature.parse(feature_name) - if ( - feature is not None - and feature.kind is CellProfilerMeasurementFeatureKind.CHILD_COUNT - ): - return self.child_count_measurement_values(feature) + if child_count_feature_child_name(feature_name) is not None: + return self.child_count_measurement_values( + CellProfilerMeasurementFeature.parse(feature_name) + ) if not self.measurement_tables: raise ValueError( "FilterObjects measurement feature " diff --git a/tests/unit/test_cellprofiler_compatibility_matrix.py b/tests/unit/test_cellprofiler_compatibility_matrix.py index 512fccfbd0..1d6e78ff58 100644 --- a/tests/unit/test_cellprofiler_compatibility_matrix.py +++ b/tests/unit/test_cellprofiler_compatibility_matrix.py @@ -14,7 +14,7 @@ ) from openhcs.core.callable_contract import FunctionStepExecutionScope from openhcs.interop.cellprofiler.module_settings import ( - ModuleSettingCoverageStatus, + SettingBound, ) @@ -164,12 +164,12 @@ def test_compatibility_matrix_summarizes_benchmark_coverage( ( "IdentifyPrimaryObjects", "Select the input image", - ModuleSettingCoverageStatus.BOUND, + SettingBound, ), ( "IdentifyPrimaryObjects", "Name the primary objects to be identified", - ModuleSettingCoverageStatus.BOUND, + SettingBound, ), } diff --git a/tests/unit/test_cellprofiler_module_execution.py b/tests/unit/test_cellprofiler_module_execution.py index 67031ebb85..af023aa593 100644 --- a/tests/unit/test_cellprofiler_module_execution.py +++ b/tests/unit/test_cellprofiler_module_execution.py @@ -192,9 +192,12 @@ from openhcs.interop.cellprofiler.runtime.invocation import ( CellProfilerImageRequest, CellProfilerMeasurementImage, - CellProfilerMeasurementImageDomain, CellProfilerSliceAlignedValues, ) +from openhcs.core.measurement_image_alignment import ( + ObjectLabelsReference, + SourceImageReference, +) from openhcs.interop.cellprofiler.runtime.main_flow import ( cellprofiler_main_flow_output, ) @@ -962,7 +965,7 @@ def _measurement_image_for_labels( label_payload=None, plane_projector=None, source_aliases=(), - reference_domain=CellProfilerMeasurementImageDomain.SOURCE_IMAGE, + reference_domain=SourceImageReference, ): source = CellProfilerMeasurementImage( source_image_name=None, @@ -6764,7 +6767,7 @@ def test_resolved_measurement_image_preserves_runtime_slice_projection() -> None runtime, image, ("DNA",), - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) assert measurement_image.plane_projection == ( @@ -6806,7 +6809,7 @@ def test_multi_image_photometry_consumes_each_named_current_source(aliases): ) measurement_images = executor._resolved_measurement_images( image_specs, runtime, payload, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) labels = ObjectLabelSet( name="Cells", variant_data=ObjectLabelVariantData(labels=np.ones((4, 5), dtype=np.int32)), @@ -9304,12 +9307,12 @@ def test_object_label_measurement_images_bundle_source_metadata() -> None: CellProfilerMeasurementImage( source_image_name=None, payload=dna_payload, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ), CellProfilerMeasurementImage( source_image_name=None, payload=rna_payload, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ), ) ) @@ -11285,11 +11288,11 @@ def test_object_only_measurement_carrier_preserves_color_stack() -> None: source_image_name=None, source_aliases=(), payload=color_stack, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) assert carrier.payload.data is color_stack - assert carrier.reference_domain is CellProfilerMeasurementImageDomain.OBJECT_LABELS + assert carrier.reference_domain is ObjectLabelsReference def test_compose_image_payload_rejects_unowned_singleton_broadcast(): @@ -12695,7 +12698,7 @@ def test_object_only_measurement_carrier_preserves_payload_stack_context() -> No source_image_name=None, source_aliases=(), payload=payload, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) assert carrier.payload is payload @@ -12720,7 +12723,7 @@ def test_object_only_measurement_carrier_preserves_aligned_stack() -> None: source_image_name=None, source_aliases=(), payload=payload, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, execution_mode=AlignedStackExecution, ) @@ -12752,7 +12755,7 @@ def test_literal_measurement_stack_preserves_pixels_in_object_reference_domain( ) carrier = CellProfilerMeasurementImage( source_image_name="Source", payload=stack, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, execution_mode=execution_mode, ) prepared = carrier.prepare_object_labels(labels) @@ -15650,7 +15653,7 @@ def test_measurement_image_for_labels_rejects_undeclared_object_domain_stack() - _measurement_image_for_labels( image, labels, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) @@ -15662,7 +15665,7 @@ def test_measurement_image_for_labels_rejects_undeclared_object_domain_shape() - _measurement_image_for_labels( image, labels, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) @@ -16695,7 +16698,7 @@ def test_measurement_labels_project_runtime_slice_stack_for_object_domain() -> N source_image_name=None, source_aliases=(), payload=np.ones((4, 5), dtype=np.float32), - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) measurement_labels = _prepared_measurement_labels( @@ -16731,7 +16734,7 @@ def test_measurement_labels_reject_runtime_slice_stack_without_projection() -> N source_image_name=None, source_aliases=(), payload=np.ones((4, 5), dtype=np.float32), - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) with pytest.raises(ValueError, match="Measurement image alignment"): @@ -16766,7 +16769,7 @@ def test_measurement_labels_project_runtime_slice_stack_for_unrelated_source_ima source_image_name="DF_image", source_aliases=("DF_image",), payload=np.ones((4, 5), dtype=np.float32), - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) measurement_labels = _prepared_measurement_labels( @@ -16803,7 +16806,7 @@ def test_measurement_labels_project_source_binding_stack_for_object_domain() -> source_image_name=None, source_aliases=(), payload=np.ones((4, 5), dtype=np.float32), - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) measurement_labels = _prepared_measurement_labels( @@ -16838,7 +16841,7 @@ def test_measurement_labels_project_repeated_source_binding_stack_for_object_dom source_image_name=None, source_aliases=(), payload=np.ones((4, 5), dtype=np.float32), - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) measurement_labels = _prepared_measurement_labels( @@ -16876,7 +16879,7 @@ def test_measurement_labels_reject_undeclared_nested_runtime_stack_axis() -> Non source_image_name=None, source_aliases=(), payload=np.ones((4, 5), dtype=np.float32), - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) with pytest.raises(ValueError, match="Measurement image alignment"): @@ -17206,7 +17209,7 @@ def test_measurement_image_for_labels_uses_object_label_source_spatial_crop() -> image, label_payload.labels, label_payload=label_payload, - reference_domain=CellProfilerMeasurementImageDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) np.testing.assert_array_equal( diff --git a/tests/unit/test_cellprofiler_spreadsheet_export.py b/tests/unit/test_cellprofiler_spreadsheet_export.py index 4f00818f5a..7bedb8b689 100644 --- a/tests/unit/test_cellprofiler_spreadsheet_export.py +++ b/tests/unit/test_cellprofiler_spreadsheet_export.py @@ -73,7 +73,8 @@ CellProfilerModule, ) from openhcs.interop.cellprofiler.module_settings import ( - ModuleSettingCoverageStatus, + SettingBound, + SettingIgnored, ) from openhcs.interop.cellprofiler.worm_measurements import ( WormControlPointAxis, @@ -251,8 +252,8 @@ def test_export_to_spreadsheet_binds_scalars_and_repeated_file_rows() -> None: ) assert not bound.unmapped_kwargs assert {record.status for record in bound.setting_coverage} <= { - ModuleSettingCoverageStatus.BOUND, - ModuleSettingCoverageStatus.IGNORED, + SettingBound, + SettingIgnored, } diff --git a/tests/unit/test_cellprofiler_uncovered_object_output_contracts.py b/tests/unit/test_cellprofiler_uncovered_object_output_contracts.py index d5fa258cbf..fbfc72d2a8 100644 --- a/tests/unit/test_cellprofiler_uncovered_object_output_contracts.py +++ b/tests/unit/test_cellprofiler_uncovered_object_output_contracts.py @@ -45,9 +45,7 @@ from openhcs.interop.cellprofiler.database_column_dialect import ( CellProfilerObjectCoreMeasurementFeature, ) -from openhcs.interop.cellprofiler.measurement_lookup import ( - CellProfilerMeasurementFeature, -) +from openhcs.interop.cellprofiler.measurement_lookup import ObjectCountFeatureName from openhcs.interop.cellprofiler.parser import ModuleBlock, ModuleSetting from openhcs.interop.cellprofiler.pipeline_import import import_cellprofiler_pipeline from openhcs.interop.cellprofiler.runtime.measurement_recording import ( @@ -332,7 +330,7 @@ def test_identify_dead_worms_projects_exact_native_measurement_features() -> Non assert table.source_image_name == "CellMask" assert IdentifyDeadWormsModule.owns_measurement_feature_name("Worm_Angle") assert not IdentifyDeadWormsModule.owns_measurement_feature_name("Worm_Unknown") - count_name = CellProfilerMeasurementFeature.object_count("DeadWorms").name + count_name = ObjectCountFeatureName.feature("DeadWorms").name assert tuple(row[count_name] for row in rows if count_name in row) == (2,) object_rows = tuple( row for row in rows if MeasurementRowAxisField.OBJECT_LABEL.value in row diff --git a/tests/unit/test_exported_relationship_equivalence.py b/tests/unit/test_exported_relationship_equivalence.py index 10b79f331c..f2ec6df752 100644 --- a/tests/unit/test_exported_relationship_equivalence.py +++ b/tests/unit/test_exported_relationship_equivalence.py @@ -349,13 +349,10 @@ def test_child_count_parser_and_core_declaration_share_original_grammar( feature: str, identity: str | None ) -> None: from openhcs.interop.cellprofiler.measurement_lookup import ChildCountFeatureDeclaration - from openhcs.interop.cellprofiler.measurement_lookup import ( - CellProfilerChildCountFeatureParser, - ) + from openhcs.interop.cellprofiler.measurement_lookup import ChildCountFeatureName assert ChildCountFeatureDeclaration.from_feature_name(feature) == identity - parsed = CellProfilerChildCountFeatureParser().parse_feature(feature) - assert (None if parsed is None else parsed.object_name) == identity + assert ChildCountFeatureName.object_name_in(feature) == identity @pytest.mark.parametrize("swapped", (False, True)) diff --git a/tests/unit/test_measurement_image_alignment.py b/tests/unit/test_measurement_image_alignment.py index d244cc5b44..11c8431024 100644 --- a/tests/unit/test_measurement_image_alignment.py +++ b/tests/unit/test_measurement_image_alignment.py @@ -15,6 +15,8 @@ ) from openhcs.core.measurement_image_alignment import ( MeasurementImageReferenceDomain, + ObjectLabelsReference, + SourceImageReference, ) from openhcs.core.runtime_object_label_domains import ( ObjectLabelDomain, @@ -45,9 +47,7 @@ @dataclass(frozen=True, slots=True) class _MeasurementImageSource(MeasurementImageAlignmentSource): payload: object - reference_domain: MeasurementImageReferenceDomain = ( - MeasurementImageReferenceDomain.SOURCE_IMAGE - ) + reference_domain: type[MeasurementImageReferenceDomain] = SourceImageReference source_aliases: tuple[str, ...] = () @property @@ -55,7 +55,7 @@ def alignment_image(self) -> object: return self.payload @property - def alignment_reference_domain(self) -> MeasurementImageReferenceDomain: + def alignment_reference_domain(self) -> type[MeasurementImageReferenceDomain]: return self.reference_domain @property @@ -202,7 +202,7 @@ def test_payload_scoped_labels_reject_non_runtime_image_plane_axis() -> None: def test_object_reference_requires_nominal_label_payload() -> None: source = _MeasurementImageSource( np.ones((4, 5), dtype=np.float32), - reference_domain=MeasurementImageReferenceDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) with pytest.raises(ValueError, match="requires an ObjectLabelValue"): @@ -220,7 +220,7 @@ def test_object_reference_uses_declared_source_spatial_adapter() -> None: ) source = _MeasurementImageSource( image, - reference_domain=MeasurementImageReferenceDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) aligned = source.object_label_alignment_request(labels).aligned().image @@ -254,7 +254,7 @@ def test_object_reference_preserves_nominal_image_payload_context() -> None: plane_axis=RuntimePlaneAxis.RUNTIME_SLICE, ), ), - reference_domain=MeasurementImageReferenceDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) aligned = source.object_label_alignment_request(labels).aligned().image @@ -318,7 +318,7 @@ def test_object_reference_projects_declared_runtime_slice_only_with_context() -> ) source = _MeasurementImageSource( np.ones((4, 5), dtype=np.float32), - reference_domain=MeasurementImageReferenceDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) with pytest.raises(ValueError, match="incompatible declared domains"): @@ -342,7 +342,7 @@ def test_object_reference_projects_declared_single_runtime_slice() -> None: ) source = _MeasurementImageSource( np.ones((4, 5), dtype=np.float32), - reference_domain=MeasurementImageReferenceDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) prepared = PreparedMeasurementObjectLabels.from_source( @@ -376,7 +376,7 @@ def test_object_reference_projects_image_with_declared_single_runtime_slice() -> plane_axis=RuntimePlaneAxis.RUNTIME_SLICE, ), ), - reference_domain=MeasurementImageReferenceDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) prepared = PreparedMeasurementObjectLabels.from_source( @@ -399,7 +399,7 @@ def test_object_reference_projects_local_singleton_in_wider_runtime_scope() -> N ) source = _MeasurementImageSource( np.ones((4, 5), dtype=np.float32), - reference_domain=MeasurementImageReferenceDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) prepared = PreparedMeasurementObjectLabels.from_source( @@ -433,7 +433,7 @@ def test_projected_payload_domain_is_not_projected_twice() -> None: ) source = _MeasurementImageSource( np.ones((4, 5), dtype=np.float32), - reference_domain=MeasurementImageReferenceDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) with pytest.raises(ValueError, match="incompatible declared domains"): @@ -620,7 +620,7 @@ def test_object_reference_replaces_unrelated_aligned_image_stack() -> None: ) source = _MeasurementImageSource( image, - reference_domain=MeasurementImageReferenceDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) prepared = PreparedMeasurementObjectLabels.from_source( diff --git a/tests/unit/test_measurement_image_plane_projection.py b/tests/unit/test_measurement_image_plane_projection.py index 93f719b639..fabce8aac7 100644 --- a/tests/unit/test_measurement_image_plane_projection.py +++ b/tests/unit/test_measurement_image_plane_projection.py @@ -9,7 +9,7 @@ ObjectLabelPayload, ObjectLabelVariantData, ) -from openhcs.core.measurement_image_alignment import MeasurementImageReferenceDomain +from openhcs.core.measurement_image_alignment import ObjectLabelsReference from openhcs.core.runtime_object_label_domains import ( ObjectLabelDomain, ObjectLabelDomainScope, @@ -59,7 +59,7 @@ def test_singleton_object_projection_consumes_measurement_image_plane_proof() -> axis=RuntimePlaneAxis.RUNTIME_SLICE, axis_size=1, ), - reference_domain=MeasurementImageReferenceDomain.OBJECT_LABELS, + reference_domain=ObjectLabelsReference, ) ( diff --git a/tests/unit/test_settings_binder.py b/tests/unit/test_settings_binder.py index e7a4233232..25c2ec79b1 100644 --- a/tests/unit/test_settings_binder.py +++ b/tests/unit/test_settings_binder.py @@ -1945,11 +1945,12 @@ def test_module_settings_binding_rejects_unmapped_settings(): def test_module_setting_coverage_has_only_explicit_final_states(): - assert [(member.name, member.value) for member in ModuleSettingCoverageStatus] == [ - ("BOUND", "bound"), - ("IGNORED", "ignored"), - ("UNMAPPED", "unmapped"), - ] + statuses = ModuleSettingCoverageStatus.__registry__ + assert {name: status.is_covered for name, status in statuses.items()} == { + "bound": True, + "ignored": True, + "unmapped": False, + } def test_setting_to_keyword_binding_owns_keyword_derivation_and_row_reconstruction(): From 87f6c08f275527d592775904e3f519417c3829f7 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:46:52 -0400 Subject: [PATCH 16/36] E1: payload summaries are spelled by their declared record fields The napari state projection builds summary mappings that ViewerPayloadSummary and ViewerArrayValueSummary decode strictly; ViewerPayloadSummaryField restated those field names and is deleted. Co-Authored-By: Claude Opus 5.5 --- openhcs/runtime/napari_viewer_server.py | 21 ++++++++++----------- openhcs/runtime/viewer_protocol.py | 11 ----------- 2 files changed, 10 insertions(+), 22 deletions(-) diff --git a/openhcs/runtime/napari_viewer_server.py b/openhcs/runtime/napari_viewer_server.py index 2e96bb3035..2204fbd8c7 100644 --- a/openhcs/runtime/napari_viewer_server.py +++ b/openhcs/runtime/napari_viewer_server.py @@ -185,7 +185,6 @@ ViewerNavigationControlOptions, ViewerPayloadField, ViewerPayloadProjectionOptions, - ViewerPayloadSummaryField, ViewerPayloadSummary, ViewerArrayValueSummary, ViewerPayloadRecord, @@ -4324,7 +4323,7 @@ def payload_summary( summary.update(cls.array_summary(data)) elif isinstance(data, (list, tuple)): summary["item_count"] = len(data) - summary[ViewerPayloadSummaryField.NONZERO_COUNT.value] = len(data) + summary["nonzero_count"] = len(data) summary.update(cls.shape_payload_summary(data)) return cls.payload_summary_type.from_wire_mapping(summary) @@ -4380,12 +4379,12 @@ def shape_source_spatial_shape( def array_summary(cls, array: np.ndarray) -> dict[str, NapariWireValue]: nonzero_count = int(np.count_nonzero(array)) summary: dict[str, NapariWireValue] = { - ViewerPayloadSummaryField.SHAPE.value: tuple( + "shape": tuple( int(axis) for axis in array.shape ), "dtype": str(array.dtype), "size": int(array.size), - ViewerPayloadSummaryField.NONZERO_COUNT.value: nonzero_count, + "nonzero_count": nonzero_count, } summary.update( cls.nonzero_coordinate_summary( @@ -4409,7 +4408,7 @@ def nonzero_coordinate_summary( if nonzero_count > cls.MAX_NONZERO_COORDINATE_PROJECTION_COUNT: return { - ViewerPayloadSummaryField.NONZERO_COORDINATE_OMISSION_REASON.value: ( + "nonzero_coordinate_omission_reason": ( "nonzero count exceeds bounded coordinate projection limit " f"{cls.MAX_NONZERO_COORDINATE_PROJECTION_COUNT}" ) @@ -4418,7 +4417,7 @@ def nonzero_coordinate_summary( coordinates = np.argwhere(array != 0) if nonzero_count == 0: return { - ViewerPayloadSummaryField.NONZERO_EXAMPLE_COORDINATES.value: (), + "nonzero_example_coordinates": (), } example_count = min( @@ -4432,13 +4431,13 @@ def nonzero_coordinate_summary( dtype=np.int64, ) return { - ViewerPayloadSummaryField.NONZERO_MIN_COORDINATE.value: tuple( + "nonzero_min_coordinate": tuple( int(value) for value in coordinates.min(axis=0) ), - ViewerPayloadSummaryField.NONZERO_MAX_COORDINATE.value: tuple( + "nonzero_max_coordinate": tuple( int(value) for value in coordinates.max(axis=0) ), - ViewerPayloadSummaryField.NONZERO_EXAMPLE_COORDINATES.value: tuple( + "nonzero_example_coordinates": tuple( tuple(int(value) for value in coordinates[index]) for index in example_indices ), @@ -4771,11 +4770,11 @@ def array_value_projection( summary.update( { "dtype": str(sample.dtype), - ViewerPayloadSummaryField.SHAPE.value: tuple( + "shape": tuple( int(axis_size) for axis_size in sample.shape ), "size": int(sample.size), - ViewerPayloadSummaryField.NONZERO_COUNT.value: int( + "nonzero_count": int( np.count_nonzero(sample) ), } diff --git a/openhcs/runtime/viewer_protocol.py b/openhcs/runtime/viewer_protocol.py index f04d0196b1..ffcfce2118 100644 --- a/openhcs/runtime/viewer_protocol.py +++ b/openhcs/runtime/viewer_protocol.py @@ -175,17 +175,6 @@ class SettlementFailed(ViewerSettlePhase): failed = True -class ViewerPayloadSummaryField(str, Enum): - """Viewer payload-summary response fields.""" - - SHAPE = "shape" - NONZERO_COUNT = "nonzero_count" - NONZERO_MIN_COORDINATE = "nonzero_min_coordinate" - NONZERO_MAX_COORDINATE = "nonzero_max_coordinate" - NONZERO_EXAMPLE_COORDINATES = "nonzero_example_coordinates" - NONZERO_COORDINATE_OMISSION_REASON = "nonzero_coordinate_omission_reason" - - class ViewerFieldAbsent: """An omitted protocol field, distinct from JSON null, false and zero.""" From b1c4099dd5fd4695865c414256e39bb39d04df1d Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:52:13 -0400 Subject: [PATCH 17/36] E1: catalog views and ranks, parameter sources, selected-plate targets, field filters and bridge operation states are families Members own what consumers switched on: parameter sources answer agent_supplied and their runtime description; selected-plate targets resolve their plate root; field filters match a field summary (the separate ObjectStateFieldFilterDeclaration family is deleted); operation states answer is_terminal, severity and completion selection. Catalog signature and summary views render themselves; the unused search_rank is deleted. Co-Authored-By: Claude Opus 5.5 --- openhcs/agent/dto/functions.py | 85 +++--- openhcs/agent/dto/plate.py | 112 ++++++-- openhcs/agent/dto/ui_bridge.py | 234 +++++++++++------ .../services/function_catalog_service.py | 247 ++++++++++-------- .../object_state_field_help_service.py | 11 +- .../services/object_state_field_projection.py | 107 +------- .../agent/services/selected_plate_service.py | 33 +-- openhcs/agent/services/ui_bridge_service.py | 8 +- openhcs/mcp/dev_client_commands/plate.py | 8 +- openhcs/mcp/dev_client_commands/ui.py | 2 +- openhcs/mcp/dev_client_core.py | 22 +- openhcs/mcp/dev_client_renderers/knowledge.py | 5 +- .../mcp/dev_client_renderers/object_state.py | 5 +- openhcs/mcp/dev_client_renderers/plate.py | 9 +- openhcs/mcp/dev_client_rendering.py | 6 +- openhcs/pyqt_gui/services/ui_agent_bridge.py | 55 ++-- .../services/ui_bridge_object_state.py | 13 +- tests/unit/agent/test_agent_services.py | 12 +- .../test_declared_callable_documentation.py | 9 +- .../agent/test_function_declaration_help.py | 9 +- ...cellprofiler_function_contract_friction.py | 6 +- ...test_mcp_dev_client_knowledge_renderers.py | 7 +- .../test_mcp_dev_client_plate_renderers.py | 13 +- tests/unit/agent/test_mcp_server.py | 4 +- .../test_object_state_field_help_service.py | 9 +- tests/unit/agent/test_ui_bridge_service.py | 31 ++- tests/unit/pyqt_gui/test_ui_agent_bridge.py | 4 +- tests/unit/test_catalog_current_readiness.py | 4 +- tests/unit/test_function_catalog_zmq.py | 6 +- 29 files changed, 589 insertions(+), 487 deletions(-) diff --git a/openhcs/agent/dto/functions.py b/openhcs/agent/dto/functions.py index 6528f22aee..7f5fc6bb76 100644 --- a/openhcs/agent/dto/functions.py +++ b/openhcs/agent/dto/functions.py @@ -11,6 +11,7 @@ from pathlib import Path from typing import Any, ClassVar, Generic, Self, TypeVar, cast +from metaclass_registry import AutoRegisterMeta from zmqruntime.config import TransportMode from zmqruntime.messages import MessageFields, ProcessIdentity, ResponseType from zmqruntime.startup import EndpointStartupStatus @@ -390,48 +391,57 @@ def require_result( raise ValueError("Registration returned a different persistence destination.") -class FunctionParameterSource(str, Enum): - """Nominal source of one callable parameter in agent-facing docs.""" +class FunctionParameterSource(ABC, metaclass=AutoRegisterMeta): + """Who supplies one callable parameter, as agent-facing docs describe it. - _runtime_description: str | None + Each source is a subclass; ``name`` is its spelling on the wire. + """ - def __new__( - cls, - value: str, - runtime_description: str | None, - ) -> FunctionParameterSource: - member = str.__new__(cls, value) - member._value_ = value - member._runtime_description = runtime_description - return member - - AGENT = ("agent", None) - PRIMARY_INPUT = ( - "runtime_primary_input", + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + agent_supplied: ClassVar[bool] = False + """The agent passes this parameter as a FunctionStep kwarg.""" + runtime_description: ClassVar[str | None] = None + """Why the agent must not pass a runtime-supplied parameter.""" + + +class SuppliedByAgent(FunctionParameterSource): + name = "agent" + agent_supplied = True + + +class SuppliedFromPrimaryInput(FunctionParameterSource): + name = "runtime_primary_input" + runtime_description = ( "Supplied by OpenHCS from the FunctionStep input image payload; do not " - "pass this as a function kwarg.", + "pass this as a function kwarg." ) - ARTIFACT_INPUT = ( - "runtime_artifact_input", + + +class SuppliedFromArtifactInput(FunctionParameterSource): + name = "runtime_artifact_input" + runtime_description = ( "Supplied by OpenHCS from a declared artifact input during pipeline " - "execution; do not pass this as a function kwarg.", + "execution; do not pass this as a function kwarg." ) - RUNTIME_PARAMETER = ( - "runtime_parameter", + + +class SuppliedByRuntime(FunctionParameterSource): + name = "runtime_parameter" + runtime_description = ( "Supplied by OpenHCS runtime execution infrastructure; do not pass this " - "as a function kwarg.", - ) - RUNTIME_ADAPTER = ( - "runtime_adapter", - "Supplied by OpenHCS as a runtime adapter object; do not pass this as a " - "function kwarg.", + "as a function kwarg." ) - @property - def runtime_description(self) -> str | None: - """Explain the runtime-owned source directly from its declaration.""" - return self._runtime_description +class SuppliedAsRuntimeAdapter(FunctionParameterSource): + name = "runtime_adapter" + runtime_description = ( + "Supplied by OpenHCS as a runtime adapter object; do not pass this as a " + "function kwarg." + ) @dataclass(frozen=True, slots=True) @@ -465,17 +475,20 @@ class FunctionParameterSpec: annotation: str | None default_repr: str | None required: bool - supplied_by: FunctionParameterSource = FunctionParameterSource.AGENT + supplied_by: type[FunctionParameterSource] = SuppliedByAgent description: str | None = None enum_import_path: str | None = None enum_members: tuple[str, ...] = () enum_values: tuple[str, ...] = () def __post_init__(self) -> None: - if not isinstance(self.supplied_by, FunctionParameterSource): + if not ( + isinstance(self.supplied_by, type) + and issubclass(self.supplied_by, FunctionParameterSource) + ): raise TypeError( - "FunctionParameterSpec.supplied_by requires " - f"FunctionParameterSource, got {type(self.supplied_by).__name__}." + "FunctionParameterSpec.supplied_by requires a " + f"FunctionParameterSource class, got {self.supplied_by!r}." ) diff --git a/openhcs/agent/dto/plate.py b/openhcs/agent/dto/plate.py index 3341c0459e..39d1857acc 100644 --- a/openhcs/agent/dto/plate.py +++ b/openhcs/agent/dto/plate.py @@ -2,9 +2,13 @@ from __future__ import annotations +from abc import ABC, abstractmethod +from collections.abc import Mapping from dataclasses import dataclass, field from enum import Enum +from typing import ClassVar +from metaclass_registry import AutoRegisterMeta from zmqruntime.config import TransportMode from openhcs.agent.dto.common import ( @@ -86,12 +90,80 @@ class PlateInspectionIssueCode(str, Enum): PROBABLE_NATIVE_HANDLER = "plate_probable_native_handler" -class SelectedPlateFileQueryTarget(str, Enum): - """Plate root used when querying files from a selected PlateManager row.""" +class SelectedPlateFileQueryTarget(ABC, metaclass=AutoRegisterMeta): + """Plate root used when querying files from a selected PlateManager row. - SELECTED = "selected" - OUTPUT = "output" - SOURCE = "source" + Each target is a subclass; ``name`` is its spelling in tool arguments. + """ + + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + is_selected_row_root: ClassVar[bool] = False + """The query reads the selected row's own root, not a related plate.""" + streams_from_source_context: ClassVar[bool] = False + """Streamed results resolve against the row's source plate.""" + + @staticmethod + def named(name: str) -> type["SelectedPlateFileQueryTarget"]: + target = SelectedPlateFileQueryTarget.__registry__.get(name) + if target is None: + raise ValueError( + f"Unknown selected-plate target {name!r}; expected one of " + f"{tuple(SelectedPlateFileQueryTarget.__registry__)!r}." + ) + return target + + @classmethod + @abstractmethod + def plate_root( + cls, + selected_row: Mapping[str, JsonValue], + plate_root: str, + state_surface_id: str, + ) -> tuple[str | None, AgentError | None]: + """Return the plate root this target reads from one selected row.""" + + +class SelectedRowRoot(SelectedPlateFileQueryTarget): + name = "selected" + is_selected_row_root = True + + @classmethod + def plate_root(cls, selected_row, plate_root, state_surface_id): + return plate_root, None + + +class OutputPlateRoot(SelectedPlateFileQueryTarget): + name = "output" + streams_from_source_context = True + + @classmethod + def plate_root(cls, selected_row, plate_root, state_surface_id): + output_root = selected_row.get("output_root") + if isinstance(output_root, str) and output_root: + return output_root, None + return None, AgentError( + code="ui_selected_plate_output_root_unavailable", + message="The selected PlateManager row does not expose an output_root.", + hint=( + "Call openhcs_ui_get_state_surface(surface_id=" + f"{state_surface_id!r}) to inspect source/output plate " + "relationships." + ), + ) + + +class SourcePlateRoot(SelectedPlateFileQueryTarget): + name = "source" + + @classmethod + def plate_root(cls, selected_row, plate_root, state_surface_id): + source_root = selected_row.get("source_root") + if isinstance(source_root, str) and source_root: + return source_root, None + return plate_root, None class PlateWorkspacePreparationOperation(str, Enum): @@ -163,11 +235,11 @@ class SelectedPlateTargetOptions(DatasetTarget): """Shared target controls for PlateManager-selected plate operations.""" pattern_format: str | None = None - target: SelectedPlateFileQueryTarget = SelectedPlateFileQueryTarget.SELECTED + target: type[SelectedPlateFileQueryTarget] = SelectedRowRoot @staticmethod - def target_from_value(value: str) -> SelectedPlateFileQueryTarget: - return SelectedPlateFileQueryTarget(value) + def target_from_value(value: str) -> type[SelectedPlateFileQueryTarget]: + return SelectedPlateFileQueryTarget.named(value) @dataclass(frozen=True, slots=True) @@ -512,7 +584,7 @@ def from_fields( *, source_format: str = AutoDetectedSource.source_name, pattern_format: str | None = None, - target: str = SelectedPlateFileQueryTarget.SELECTED.value, + target: str = SelectedRowRoot.name, max_sample_files: int = PlateInspectionDefaults.DEFAULT_MAX_SAMPLE_FILES, max_component_values: int = PlateInspectionDefaults.DEFAULT_MAX_COMPONENT_VALUES, max_parse_failure_samples: int = ( @@ -534,7 +606,7 @@ def as_tool_arguments(self) -> dict[str, JsonValue]: return { "source_format": self.source_format, "pattern_format": self.pattern_format, - "target": self.target.value, + "target": self.target.name, "max_sample_files": self.max_sample_files, "max_component_values": self.max_component_values, "max_parse_failure_samples": self.max_parse_failure_samples, @@ -575,7 +647,7 @@ def from_fields( source_format: str = AutoDetectedSource.source_name, pattern_format: str | None = None, kind: PlateFileKindSelection = PlateFileKind.IMAGE, - target: str = SelectedPlateFileQueryTarget.SELECTED.value, + target: str = SelectedRowRoot.name, path_contains: str | None = None, component_filters: dict[str, list[str]] | None = None, offset: int = 0, @@ -603,7 +675,7 @@ def as_tool_arguments(self) -> dict[str, JsonValue]: "source_format": self.source_format, "pattern_format": self.pattern_format, "kind": PlateFileInventoryQuery.kind_value(self.kind), - "target": self.target.value, + "target": self.target.name, "path_contains": self.path_contains, "component_filters": self.component_filters.as_mapping(), "offset": self.offset, @@ -657,7 +729,7 @@ def from_fields( image_path: str | None = None, source_format: str = AutoDetectedSource.source_name, pattern_format: str | None = None, - target: str = SelectedPlateFileQueryTarget.SELECTED.value, + target: str = SelectedRowRoot.name, y: int = 0, x: int = 0, height: int = 32, @@ -689,7 +761,7 @@ def as_tool_arguments(self) -> dict[str, JsonValue]: "image_path": self.image_path, "source_format": self.source_format, "pattern_format": self.pattern_format, - "target": self.target.value, + "target": self.target.name, "y": self.y, "x": self.x, "height": self.height, @@ -740,7 +812,7 @@ def from_fields( source_format: str = AutoDetectedSource.source_name, pattern_format: str | None = None, kind: PlateFileKindSelection = PlateFileKind.IMAGE, - target: str = SelectedPlateFileQueryTarget.SELECTED.value, + target: str = SelectedRowRoot.name, path_contains: str | None = None, component_filters: dict[str, list[str]] | None = None, limit: int = 1, @@ -776,7 +848,7 @@ def as_tool_arguments(self) -> dict[str, JsonValue]: "source_format": self.source_format, "pattern_format": self.pattern_format, "kind": PlateFileInventoryQuery.kind_value(self.kind), - "target": self.target.value, + "target": self.target.name, "path_contains": self.path_contains, "component_filters": self.component_filters.as_mapping(), "limit": self.limit, @@ -1239,7 +1311,7 @@ class SelectedPlateImageInspectionResult(AgentResultEnvelope): """Image inspection for the single plate currently selected in the UI.""" selected_plate: JsonObject = field(default_factory=dict) - target: SelectedPlateFileQueryTarget = SelectedPlateFileQueryTarget.SELECTED + target: type[SelectedPlateFileQueryTarget] = SelectedRowRoot inspection: PlatePathInspectionResult | None = None @@ -1248,7 +1320,7 @@ class SelectedPlateFileQueryResult(AgentResultEnvelope): """File query for the single plate currently selected in the UI.""" selected_plate: JsonObject = field(default_factory=dict) - target: SelectedPlateFileQueryTarget = SelectedPlateFileQueryTarget.SELECTED + target: type[SelectedPlateFileQueryTarget] = SelectedRowRoot query: PlateFileQueryResult | None = None @@ -1257,7 +1329,7 @@ class SelectedPlateImageSampleResult(AgentResultEnvelope): """Bounded image sample for the single plate currently selected in the UI.""" selected_plate: JsonObject = field(default_factory=dict) - target: SelectedPlateFileQueryTarget = SelectedPlateFileQueryTarget.SELECTED + target: type[SelectedPlateFileQueryTarget] = SelectedRowRoot image_path: str | None = None auto_selected_image_path: bool = False sample: PlateImageSampleResult | None = None @@ -1268,5 +1340,5 @@ class SelectedPlateFileStreamResult(AgentResultEnvelope): """Streaming result for the single plate currently selected in the UI.""" selected_plate: JsonObject = field(default_factory=dict) - target: SelectedPlateFileQueryTarget = SelectedPlateFileQueryTarget.SELECTED + target: type[SelectedPlateFileQueryTarget] = SelectedRowRoot stream: PlateFileStreamResult | None = None diff --git a/openhcs/agent/dto/ui_bridge.py b/openhcs/agent/dto/ui_bridge.py index 638bba5c3a..8005301dc1 100644 --- a/openhcs/agent/dto/ui_bridge.py +++ b/openhcs/agent/dto/ui_bridge.py @@ -2,12 +2,14 @@ from __future__ import annotations -from collections.abc import Callable, Mapping +from abc import ABC, abstractmethod +from collections.abc import Mapping from dataclasses import dataclass, field, replace -from enum import Enum, nonmember +from enum import Enum from math import isfinite -from typing import TYPE_CHECKING, ClassVar, TypeVar, cast +from typing import TYPE_CHECKING, ClassVar, TypeVar +from metaclass_registry import AutoRegisterMeta from objectstate import DottedFieldPath from pyqt_reactive.services.widget_tree_projection_config import ( COMPACT_FIELD_PROJECTION_METADATA_KEY, @@ -155,75 +157,72 @@ def _dataset_workflow_enum() -> type[Enum]: UiSelectedPlateWorkflowKind = _dataset_workflow_enum() -class UiBridgeOperationStatus(str, Enum): - """Closed operation state with declaration-owned completion semantics.""" +class UiBridgeOperationStatus(ABC, metaclass=AutoRegisterMeta): + """State of one bridge operation; each state is a subclass. - is_terminal: bool - live_overview_severity: str - _completion_selector: Callable[[object, object, object], object] + ``name`` is the state's spelling in ``UiBridgeOperationRef.status``. A + state that is not running or completed failed: it is terminal, reported as + an error, and selects the caller's failed output. + """ - _select_active = nonmember(lambda active, _succeeded, _failed: active) - _select_succeeded = nonmember(lambda _active, succeeded, _failed: succeeded) - _select_failed = nonmember(lambda _active, _succeeded, failed: failed) + __registry_key__ = "name" + __skip_if_no_key__ = True - RUNNING = ( - "running", - False, - UiLiveOverviewSeverity.INFO.value, - _select_active, - ) - COMPLETED = ( - "completed", - True, - UiLiveOverviewSeverity.INFO.value, - _select_succeeded, - ) - FAILED = ( - "failed", - True, - UiLiveOverviewSeverity.ERROR.value, - _select_failed, - ) - NOT_FOUND = ( - "not_found", - True, - UiLiveOverviewSeverity.ERROR.value, - _select_failed, - ) - UNAVAILABLE = ( - "unavailable", - True, - UiLiveOverviewSeverity.ERROR.value, - _select_failed, - ) + name: ClassVar[str | None] = None + is_terminal: ClassVar[bool] = True + live_overview_severity: ClassVar[str] = UiLiveOverviewSeverity.ERROR.value - def __new__( - cls, - value: str, - is_terminal: bool, - live_overview_severity: str, - completion_selector: Callable[[object, object, object], object], - ) -> "UiBridgeOperationStatus": - member = str.__new__(cls, value) - member._value_ = value - member.is_terminal = is_terminal - member.live_overview_severity = live_overview_severity - member._completion_selector = completion_selector - return member + @staticmethod + def named(name: str) -> type["UiBridgeOperationStatus"]: + """Return the state spelled ``name``; ``ValueError`` when unknown.""" + + status = UiBridgeOperationStatus.__registry__.get(name) + if status is None: + raise ValueError(f"Unknown UI bridge operation status {name!r}.") + return status + @classmethod def select_completion( - self, + cls, *, active: _UiBridgeOperationSelectionT, succeeded: _UiBridgeOperationSelectionT, failed: _UiBridgeOperationSelectionT, ) -> _UiBridgeOperationSelectionT: - """Select caller-owned output through this status member's leaf.""" + """Select caller-owned output for this state.""" - return cast( - _UiBridgeOperationSelectionT, - self._completion_selector(active, succeeded, failed), - ) + return failed + + +class OperationRunning(UiBridgeOperationStatus): + name = "running" + is_terminal = False + live_overview_severity = UiLiveOverviewSeverity.INFO.value + + @classmethod + def select_completion(cls, *, active, succeeded, failed): + return active + + +class OperationCompleted(UiBridgeOperationStatus): + name = "completed" + live_overview_severity = UiLiveOverviewSeverity.INFO.value + + @classmethod + def select_completion(cls, *, active, succeeded, failed): + return succeeded + + +class OperationFailed(UiBridgeOperationStatus): + name = "failed" + + +class OperationNotFound(UiBridgeOperationStatus): + name = "not_found" + + +class OperationUnavailable(UiBridgeOperationStatus): + name = "unavailable" if TYPE_CHECKING: @@ -320,15 +319,86 @@ def from_mapping( return cls() -class UiObjectStateFieldFilter(str, Enum): - """Agent-facing filters over existing ObjectState field semantics.""" +class UiObjectStateFieldFilter(ABC, metaclass=AutoRegisterMeta): + """Agent-facing filter over existing ObjectState field semantics. + + Each filter is a subclass; ``name`` is its spelling in tool arguments. + """ - ALL = "all" - DIRTY = "dirty" - DEFAULT_DIFF = "default_diff" - INHERITED = "inherited" - RAW_RESOLVED = "raw_resolved" - SEMANTIC = "semantic" + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + keeps_every_field: ClassVar[bool] = False + + @staticmethod + def named(name: str) -> type["UiObjectStateFieldFilter"]: + field_filter = UiObjectStateFieldFilter.__registry__.get(name) + if field_filter is None: + raise ValueError( + f"Unsupported ObjectState field filter {name!r}; expected one of " + f"{tuple(UiObjectStateFieldFilter.__registry__)!r}." + ) + return field_filter + + @classmethod + @abstractmethod + def matches(cls, field: "UiObjectStateFieldSummary") -> bool: + """Return whether ``field`` passes this filter.""" + + +class AllFields(UiObjectStateFieldFilter): + name = "all" + keeps_every_field = True + + @classmethod + def matches(cls, field: "UiObjectStateFieldSummary") -> bool: + return True + + +class DirtyFields(UiObjectStateFieldFilter): + name = "dirty" + + @classmethod + def matches(cls, field: "UiObjectStateFieldSummary") -> bool: + return field.dirty + + +class DefaultDiffFields(UiObjectStateFieldFilter): + name = "default_diff" + + @classmethod + def matches(cls, field: "UiObjectStateFieldSummary") -> bool: + return field.signature_diff + + +class InheritedFields(UiObjectStateFieldFilter): + name = "inherited" + + @classmethod + def matches(cls, field: "UiObjectStateFieldSummary") -> bool: + return field.inherited_value + + +class RawResolvedFields(UiObjectStateFieldFilter): + name = "raw_resolved" + + @classmethod + def matches(cls, field: "UiObjectStateFieldSummary") -> bool: + return field.raw_value_is_none != field.resolved_value_is_none + + +class SemanticFields(UiObjectStateFieldFilter): + name = "semantic" + + @classmethod + def matches(cls, field: "UiObjectStateFieldSummary") -> bool: + return ( + field.dirty + or field.signature_diff + or field.inherited_value + or RawResolvedFields.matches(field) + ) @dataclass(frozen=True, slots=True) @@ -339,7 +409,7 @@ class UiObjectStateFieldListOptions: include_field_values: bool = False include_field_descriptions: bool = False field_paths: tuple[str, ...] = () - field_filter: UiObjectStateFieldFilter = UiObjectStateFieldFilter.ALL + field_filter: type[UiObjectStateFieldFilter] = AllFields @dataclass(frozen=True, kw_only=True) @@ -1792,9 +1862,9 @@ def resolve_operation( """Resolve an accepted placeholder from its authoritative operation.""" try: - status = UiBridgeOperationStatus(operation.status) + status = UiBridgeOperationStatus.named(operation.status) except ValueError: - status = UiBridgeOperationStatus.UNAVAILABLE + status = OperationUnavailable try: outcome = ( UiWidgetActionInvocationOutcome(operation.outcome) @@ -1804,7 +1874,7 @@ def resolve_operation( except ValueError: outcome = None errors = operation.errors - if status is UiBridgeOperationStatus.COMPLETED and outcome is None: + if status is OperationCompleted and outcome is None: errors = ( *errors, AgentError( @@ -1817,7 +1887,7 @@ def resolve_operation( ) invoked = ( outcome.invoked - if status is UiBridgeOperationStatus.COMPLETED and outcome is not None + if status is OperationCompleted and outcome is not None else False ) return replace( @@ -2085,7 +2155,7 @@ class UiObjectStateScopeListRequest( include_field_values: bool = False include_field_descriptions: bool = False field_paths: tuple[str, ...] = () - field_filter: UiObjectStateFieldFilter = UiObjectStateFieldFilter.ALL + field_filter: type[UiObjectStateFieldFilter] = AllFields @classmethod def from_visibility_options( @@ -2115,7 +2185,7 @@ def from_fields( include_system_scopes: bool = False, include_fields: bool = False, include_field_values: bool = False, - field_filter: str = UiObjectStateFieldFilter.ALL.value, + field_filter: str = AllFields.name, field_limit: int = 200, field_offset: int = 0, ) -> "UiObjectStateScopeListRequest": @@ -2128,7 +2198,7 @@ def from_fields( include_field_values=include_field_values, field_limit=field_limit, field_offset=field_offset, - field_filter=UiObjectStateFieldFilter(field_filter), + field_filter=UiObjectStateFieldFilter.named(field_filter), ), scope_ids=tuple(scope_ids), ) @@ -2155,7 +2225,7 @@ def as_tool_arguments(self) -> JsonObject: "include_system_scopes": self.include_system_scopes, "include_fields": self.include_fields, "include_field_values": self.include_field_values, - "field_filter": self.field_filter.value, + "field_filter": self.field_filter.name, "field_limit": self.field_limit, "field_offset": self.field_offset, } @@ -2196,7 +2266,7 @@ class UiObjectStateFieldListResult(AgentResultEnvelope): requested_scope_ids: tuple[str, ...] field_paths: tuple[str, ...] field_path_contains: tuple[str, ...] - field_filter: str + field_filter: type[UiObjectStateFieldFilter] include_container_fields: bool matched_scope_count: int matched_field_count: int @@ -2220,7 +2290,7 @@ class UiObjectStateFieldListQuery(AgentCliRequest): include_system_scopes: bool = False include_clean_fields: bool = True include_container_fields: bool = False - field_filter: UiObjectStateFieldFilter = UiObjectStateFieldFilter.ALL + field_filter: type[UiObjectStateFieldFilter] = AllFields include_field_values: bool = False field_limit: int = 200 field_offset: int = 0 @@ -2238,7 +2308,7 @@ def from_fields( include_system_scopes: bool = False, include_clean_fields: bool = True, include_container_fields: bool = False, - field_filter: str = UiObjectStateFieldFilter.ALL.value, + field_filter: str = AllFields.name, include_field_values: bool = False, field_limit: int = 200, field_offset: int = 0, @@ -2253,7 +2323,7 @@ def from_fields( include_system_scopes=include_system_scopes, include_clean_fields=include_clean_fields, include_container_fields=include_container_fields, - field_filter=UiObjectStateFieldFilter(field_filter), + field_filter=UiObjectStateFieldFilter.named(field_filter), include_field_values=include_field_values, field_limit=field_limit, field_offset=field_offset, @@ -2324,7 +2394,7 @@ def as_tool_arguments(self) -> JsonObject: "include_system_scopes": self.include_system_scopes, "include_clean_fields": self.include_clean_fields, "include_container_fields": self.include_container_fields, - "field_filter": self.field_filter.value, + "field_filter": self.field_filter.name, "include_field_values": self.include_field_values, "field_limit": self.field_limit, "field_offset": self.field_offset, diff --git a/openhcs/agent/services/function_catalog_service.py b/openhcs/agent/services/function_catalog_service.py index f406b1429e..7e40b29e32 100644 --- a/openhcs/agent/services/function_catalog_service.py +++ b/openhcs/agent/services/function_catalog_service.py @@ -11,7 +11,7 @@ from dataclasses import dataclass, replace from enum import Enum from pathlib import Path -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, ClassVar from pyqt_reactive.services.help_document import HelpDocument from pyqt_reactive.services.parameter_help_service import ( @@ -30,15 +30,15 @@ import openhcs.processing.custom_functions.manager as custom_function_manager from openhcs.agent.dto.common import SCHEMA_VERSION from openhcs.agent.dto.functions import ( - DEFAULT_FUNCTION_DETAIL_DOC_CHARS, CellProfilerArtifactBindingSummary, CellProfilerModuleDeclarationSummary, CustomFunctionRegistrationDestination, CustomFunctionRegistrationDestinationRequest, - CustomFunctionRegistrationRequest, - CustomFunctionRegistrationResult, CustomFunctionRegistrationHandle, CustomFunctionRegistrationObservation, + CustomFunctionRegistrationRequest, + CustomFunctionRegistrationResult, + DEFAULT_FUNCTION_DETAIL_DOC_CHARS, FunctionArtifactSpec, FunctionCatalogEntry, FunctionCatalogPage, @@ -46,6 +46,11 @@ FunctionParameterSource, FunctionParameterSpec, FunctionRuntimeContractSummary, + SuppliedAsRuntimeAdapter, + SuppliedByAgent, + SuppliedByRuntime, + SuppliedFromArtifactInput, + SuppliedFromPrimaryInput, catalog_page, ) from openhcs.agent.exceptions import AgentFacingErrorMixin @@ -82,35 +87,85 @@ def __init__(self, function_id: str) -> None: super().__init__(f"Unknown OpenHCS function_id: {function_id}") -class CatalogSearchRank(Enum): - ALL_PASS = 0 - EXACT_NAME = 10 - PREFIX_NAME = 20 - NAME_CONTAINS = 30 - ID_CONTAINS = 40 - IMPORT_CONTAINS = 50 - TAG_CONTAINS = 60 - PARAMETER_CONTAINS = 70 - SUMMARY_CONTAINS = 80 - DOC_CONTAINS = 90 - NO_MATCH = 1000 +class CatalogSearchRank(ABC): + """Which catalog text a search matched; lower ``order`` ranks first.""" - @property - def matched(self) -> bool: - return self is not CatalogSearchRank.NO_MATCH + order: ClassVar[int] + matched: ClassVar[bool] = True -class SignatureView(Enum): - FULL = 10000 - COMPACT = 4 +class AllPass(CatalogSearchRank): + order = 0 - @property - def parameter_limit(self) -> int: - return self.value - @property - def compact(self) -> bool: - return self is SignatureView.COMPACT +class ExactName(CatalogSearchRank): + order = 10 + + +class PrefixName(CatalogSearchRank): + order = 20 + + +class NameContains(CatalogSearchRank): + order = 30 + + +class IdContains(CatalogSearchRank): + order = 40 + + +class ImportContains(CatalogSearchRank): + order = 50 + + +class TagContains(CatalogSearchRank): + order = 60 + + +class ParameterContains(CatalogSearchRank): + order = 70 + + +class SummaryContains(CatalogSearchRank): + order = 80 + + +class DocContains(CatalogSearchRank): + order = 90 + + +class NoMatch(CatalogSearchRank): + order = 1000 + matched = False + + +class SignatureView(ABC): + """How much of a callable signature a catalog entry shows.""" + + @classmethod + @abstractmethod + def render(cls, signature: inspect.Signature, display_name: str) -> str: + """Render ``signature`` under ``display_name``.""" + + +class FullSignature(SignatureView): + @classmethod + def render(cls, signature: inspect.Signature, display_name: str) -> str: + return f"{display_name}{signature}" + + +class CompactSignature(SignatureView): + parameter_limit: ClassVar[int] = 4 + + @classmethod + def render(cls, signature: inspect.Signature, display_name: str) -> str: + visible_parameters = tuple(signature.parameters.values()) + if len(visible_parameters) <= cls.parameter_limit: + return f"{display_name}{signature}" + parameter_names = ", ".join( + parameter.name for parameter in visible_parameters[: cls.parameter_limit] + ) + return f"{display_name}({parameter_names}, ...)" class AgentFunctionSearchPolicy: @@ -201,11 +256,6 @@ def accepts_library_or_tag( _normalized_search_text(value) for value in (library, *backend_tags) } - def search_rank( - self, entry: FunctionCatalogEntry, metadata: FunctionMetadata - ) -> CatalogSearchRank: - return self.search_match(entry, metadata).rank - def search_match( self, entry: FunctionCatalogEntry, @@ -213,28 +263,28 @@ def search_match( parameters: tuple[FunctionParameterSpec, ...] = (), ) -> "CatalogTextMatch": if not self.text: - return CatalogTextMatch(CatalogSearchRank.ALL_PASS, 0) + return CatalogTextMatch(AllPass, 0) name_text = _normalized_search_text(entry.name) function_id_text = _normalized_search_text(entry.function_id) import_path_text = _normalized_search_text(entry.import_path) tag_text = _normalized_search_text(" ".join(entry.backend_tags)) if name_text == self.text: - return CatalogTextMatch(CatalogSearchRank.EXACT_NAME, 10000) + return CatalogTextMatch(ExactName, 10000) if name_text.startswith(self.text): - return CatalogTextMatch(CatalogSearchRank.PREFIX_NAME, 9000) + return CatalogTextMatch(PrefixName, 9000) matches = [] for rank, value in ( - (CatalogSearchRank.NAME_CONTAINS, name_text), - (CatalogSearchRank.ID_CONTAINS, function_id_text), - (CatalogSearchRank.IMPORT_CONTAINS, import_path_text), - (CatalogSearchRank.TAG_CONTAINS, tag_text), + (NameContains, name_text), + (IdContains, function_id_text), + (ImportContains, import_path_text), + (TagContains, tag_text), ( - CatalogSearchRank.PARAMETER_CONTAINS, + ParameterContains, _parameter_search_text(parameters), ), - (CatalogSearchRank.SUMMARY_CONTAINS, entry.summary or ""), + (SummaryContains, entry.summary or ""), ( - CatalogSearchRank.DOC_CONTAINS, + DocContains, PARAMETER_DOCUMENTATION_POLICY.detail_doc(metadata.func, metadata.doc) or "", ), ): @@ -242,8 +292,8 @@ def search_match( if score: matches.append(CatalogTextMatch(rank, score)) if not matches: - return CatalogTextMatch(CatalogSearchRank.NO_MATCH, 0) - return min(matches, key=lambda match: (-match.score, match.rank.value)) + return CatalogTextMatch(NoMatch, 0) + return min(matches, key=lambda match: (-match.score, match.rank.order)) def _matches_text(self, value: str) -> bool: return self._text_score(value) > 0 @@ -286,7 +336,7 @@ def _token_variants(token: str) -> tuple[str, ...]: @dataclass(frozen=True, slots=True) class CatalogTextMatch: - rank: CatalogSearchRank + rank: type[CatalogSearchRank] score: int @property @@ -294,22 +344,34 @@ def matched(self) -> bool: return self.rank.matched -class SummaryView(Enum): - FULL = 10000 - COMPACT = 180 +class SummaryView(ABC): + """How much of a callable summary a catalog entry shows.""" - @property - def character_limit(self) -> int: - return self.value + @classmethod + @abstractmethod + def bounded(cls, text: str) -> str: + """Return ``text`` as this view shows it.""" + + +class FullSummary(SummaryView): + @classmethod + def bounded(cls, text: str) -> str: + return text - @property - def compact(self) -> bool: - return self is SummaryView.COMPACT + +class CompactSummary(SummaryView): + character_limit: ClassVar[int] = 180 + + @classmethod + def bounded(cls, text: str) -> str: + if len(text) <= cls.character_limit: + return text + return f"{text[: cls.character_limit - 3].rstrip()}..." @dataclass(frozen=True, slots=True) class CatalogSearchCandidate: - rank: CatalogSearchRank + rank: type[CatalogSearchRank] score: int entry: FunctionCatalogEntry @@ -317,7 +379,7 @@ class CatalogSearchCandidate: def sort_key(self) -> tuple[int, int, str, str]: return ( -self.score, - self.rank.value, + self.rank.order, self.entry.name.casefold(), self.entry.function_id, ) @@ -365,7 +427,7 @@ def summary( self, func: Callable, metadata_doc: str | None, - view: SummaryView, + view: type[SummaryView], contract: CallableContract | None = None, ) -> str | None: doc = self.detail_doc(func, metadata_doc, contract) @@ -374,7 +436,7 @@ def summary( for line in doc.splitlines(): stripped = line.strip() if stripped: - return _bounded_summary(stripped, view) + return view.bounded(stripped) return None def should_document( @@ -404,26 +466,11 @@ def visible_signature( if self.should_document( name, parameter, skip_keyword_bag=True, hidden_names=hidden_names ) - and supplied_by.get(name) is FunctionParameterSource.AGENT + and supplied_by[name].agent_supplied ) ) return sig.replace(parameters=visible_parameters) - @staticmethod - def render_signature( - signature: inspect.Signature, display_name: str, view: SignatureView - ) -> str: - visible_parameters = tuple(signature.parameters.values()) - if view.compact and len(visible_parameters) > view.parameter_limit: - parameter_names = ", ".join( - ( - parameter.name - for parameter in visible_parameters[: view.parameter_limit] - ) - ) - return f"{display_name}({parameter_names}, ...)" - return f"{display_name}{signature}" - def parameter_specs( self, func: Callable, contract: CallableContract | None = None ) -> tuple[FunctionParameterSpec, ...]: @@ -441,7 +488,7 @@ def parameter_specs( name=name, annotation=_format_annotation(parameter.annotation), default_repr=_format_default(parameter.default), - required=supplier is FunctionParameterSource.AGENT + required=supplier.agent_supplied and parameter.default is inspect.Parameter.empty, supplied_by=supplier, description=self.parameter_description( @@ -476,7 +523,7 @@ def agent_parameter_names( name for name, parameter in sig.parameters.items() if self.should_document(name, parameter, hidden_names=hidden_names) - and supplied_by[name] is FunctionParameterSource.AGENT + and supplied_by[name].agent_supplied ) ) runtime_names = ( @@ -498,26 +545,26 @@ def supplied_by( callable_contract = contract or CallableContract.from_callable(func) hidden_names = parameter_exclusions(func) supplied_by = { - name: FunctionParameterSource.AGENT + name: SuppliedByAgent for name, parameter in sig.parameters.items() if self.should_document(name, parameter) } for name in hidden_names: if name in supplied_by: - supplied_by[name] = FunctionParameterSource.RUNTIME_PARAMETER + supplied_by[name] = SuppliedByRuntime primary_input_name = callable_contract.primary_input_parameter_name if primary_input_name in supplied_by: - supplied_by[primary_input_name] = FunctionParameterSource.PRIMARY_INPUT + supplied_by[primary_input_name] = SuppliedFromPrimaryInput for name in callable_contract.artifact_input_parameter_names: if name in supplied_by: - supplied_by[name] = FunctionParameterSource.ARTIFACT_INPUT + supplied_by[name] = SuppliedFromArtifactInput for name in callable_contract.runtime_bound_parameters: if name in supplied_by: - supplied_by[name] = FunctionParameterSource.RUNTIME_PARAMETER + supplied_by[name] = SuppliedByRuntime runtime_context_parameter = callable_contract.runtime_context_parameter if runtime_context_parameter in supplied_by: supplied_by[runtime_context_parameter] = ( - FunctionParameterSource.RUNTIME_PARAMETER + SuppliedByRuntime ) runtime_adapter = callable_contract.runtime_adapter if ( @@ -525,7 +572,7 @@ def supplied_by( and runtime_adapter.parameter_name in supplied_by ): supplied_by[runtime_adapter.parameter_name] = ( - FunctionParameterSource.RUNTIME_ADAPTER + SuppliedAsRuntimeAdapter ) return supplied_by @@ -584,11 +631,11 @@ def prepare_projections( ) -> None: """Prepare both public views against the implementation's authority.""" while True: - for signature_view in SignatureView: + for compact_signatures in (False, True): if cancellation is not None and cancellation.requested(): raise CancelledError self.catalog( - compact_signatures=signature_view.compact, + compact_signatures=compact_signatures, status_callback=status_callback, cancellation=cancellation, ) @@ -934,7 +981,7 @@ def _search_projections( metadata.func, contract ) entry = self._entry( - function_id, metadata, SignatureView.FULL, SummaryView.FULL, + function_id, metadata, FullSignature, FullSummary, contract=contract, visible_signature=signature, ) projections.append( @@ -947,12 +994,10 @@ def _search_projections( ), compact_entry=replace( entry, - signature=PARAMETER_DOCUMENTATION_POLICY.render_signature( - signature, entry.name, SignatureView.COMPACT - ), + signature=CompactSignature.render(signature, entry.name), summary=( None if entry.summary is None - else _bounded_summary(entry.summary, SummaryView.COMPACT) + else CompactSummary.bounded(entry.summary) ), ), ) @@ -988,9 +1033,9 @@ def get( function_id, metadata, signature_view=( - SignatureView.COMPACT if compact_signature else SignatureView.FULL + CompactSignature if compact_signature else FullSignature ), - summary_view=SummaryView.COMPACT, + summary_view=CompactSummary, contract=contract, ), parameters=PARAMETER_DOCUMENTATION_POLICY.parameter_specs(func, contract), @@ -1109,8 +1154,8 @@ def _entry( self, function_id: str, metadata: FunctionMetadata, - signature_view: SignatureView = SignatureView.FULL, - summary_view: SummaryView = SummaryView.FULL, + signature_view: type[SignatureView] = FullSignature, + summary_view: type[SummaryView] = FullSummary, contract: CallableContract | None = None, visible_signature: inspect.Signature | None = None, ) -> FunctionCatalogEntry: @@ -1122,9 +1167,7 @@ def _entry( visible_signature = PARAMETER_DOCUMENTATION_POLICY.visible_signature( metadata.func, callable_contract ) - signature = PARAMETER_DOCUMENTATION_POLICY.render_signature( - visible_signature, name, signature_view - ) + signature = signature_view.render(visible_signature, name) summary = PARAMETER_DOCUMENTATION_POLICY.summary( metadata.func, metadata.doc, summary_view, callable_contract ) @@ -1162,14 +1205,6 @@ def _format_default(default) -> str | None: return repr(default) -def _bounded_summary(text: str, view: SummaryView) -> str: - if not view.compact: - return text - if len(text) <= view.character_limit: - return text - return f"{text[: view.character_limit - 3].rstrip()}..." - - def _import_path(module: str, name: str) -> str: if not module: return name diff --git a/openhcs/agent/services/object_state_field_help_service.py b/openhcs/agent/services/object_state_field_help_service.py index 374e03f68c..ea399ecddb 100644 --- a/openhcs/agent/services/object_state_field_help_service.py +++ b/openhcs/agent/services/object_state_field_help_service.py @@ -19,7 +19,6 @@ from openhcs.agent.dto.functions import ( FunctionArtifactSpec, FunctionDetail, - FunctionParameterSource, ) from openhcs.agent.dto.ui_bridge import ( UiBridgeConnectionSpec, @@ -382,14 +381,14 @@ def _function_parameter_context( ), None, ) - if parameter is None or parameter.supplied_by is FunctionParameterSource.AGENT: + if parameter is None or parameter.supplied_by.agent_supplied: return None pieces = [ ( "Function parameter contract: " f"{parameter.name} is supplied by OpenHCS " - f"({parameter.supplied_by.value}); do not pass it as a " + f"({parameter.supplied_by.name}); do not pass it as a " "FunctionStep kwarg." ) ] @@ -618,7 +617,7 @@ def _active_kwarg_context_lines( parameter_by_name = { parameter.name: parameter for parameter in detail.parameters - if parameter.supplied_by is FunctionParameterSource.AGENT + if parameter.supplied_by.agent_supplied } lines = [" active kwargs:"] for name in active_kwarg_names: @@ -641,9 +640,9 @@ def _active_kwarg_context_lines( def _runtime_parameter_context_lines(detail: FunctionDetail) -> list[str]: return [ - f"{parameter.name} ({parameter.supplied_by.value})" + f"{parameter.name} ({parameter.supplied_by.name})" for parameter in detail.parameters - if parameter.supplied_by is not FunctionParameterSource.AGENT + if not parameter.supplied_by.agent_supplied ] diff --git a/openhcs/agent/services/object_state_field_projection.py b/openhcs/agent/services/object_state_field_projection.py index aa253b0744..145ad900ce 100644 --- a/openhcs/agent/services/object_state_field_projection.py +++ b/openhcs/agent/services/object_state_field_projection.py @@ -2,16 +2,13 @@ from __future__ import annotations -from abc import ABC, abstractmethod from collections.abc import Mapping -from typing import ClassVar -from metaclass_registry import AutoRegisterMeta from python_introspect import JsonValue, project_dataclass from openhcs.agent.dto.common import SCHEMA_VERSION from openhcs.agent.dto.ui_bridge import ( - UiObjectStateFieldFilter, + SemanticFields, UiObjectStateFieldListQuery, UiObjectStateFieldListResult, UiObjectStateFieldPathIndex, @@ -22,94 +19,6 @@ ) -class ObjectStateFieldFilterDeclaration(ABC, metaclass=AutoRegisterMeta): - """Nominal semantic predicate for one agent-facing ObjectState filter.""" - - __registry_key__ = "field_filter" - __skip_if_no_key__ = True - - field_filter: ClassVar[UiObjectStateFieldFilter | None] = None - - @classmethod - @abstractmethod - def matches(cls, field: UiObjectStateFieldSummary) -> bool: - raise NotImplementedError - - @classmethod - def declaration_for( - cls, - field_filter: UiObjectStateFieldFilter, - ) -> type["ObjectStateFieldFilterDeclaration"]: - try: - return cls.__registry__[field_filter] - except KeyError as exc: - raise ValueError( - f"Unsupported ObjectState field filter: {field_filter.value}" - ) from exc - - @classmethod - def matches_filter( - cls, - field_filter: UiObjectStateFieldFilter, - field: UiObjectStateFieldSummary, - ) -> bool: - return cls.declaration_for(field_filter).matches(field) - - -class AllObjectStateFieldFilter(ObjectStateFieldFilterDeclaration): - field_filter = UiObjectStateFieldFilter.ALL - - @classmethod - def matches(cls, field: UiObjectStateFieldSummary) -> bool: - del field - return True - - -class DirtyObjectStateFieldFilter(ObjectStateFieldFilterDeclaration): - field_filter = UiObjectStateFieldFilter.DIRTY - - @classmethod - def matches(cls, field: UiObjectStateFieldSummary) -> bool: - return field.dirty - - -class DefaultDiffObjectStateFieldFilter(ObjectStateFieldFilterDeclaration): - field_filter = UiObjectStateFieldFilter.DEFAULT_DIFF - - @classmethod - def matches(cls, field: UiObjectStateFieldSummary) -> bool: - return field.signature_diff - - -class InheritedObjectStateFieldFilter(ObjectStateFieldFilterDeclaration): - field_filter = UiObjectStateFieldFilter.INHERITED - - @classmethod - def matches(cls, field: UiObjectStateFieldSummary) -> bool: - return field.inherited_value - - -class RawResolvedObjectStateFieldFilter(ObjectStateFieldFilterDeclaration): - field_filter = UiObjectStateFieldFilter.RAW_RESOLVED - - @classmethod - def matches(cls, field: UiObjectStateFieldSummary) -> bool: - return field.raw_value_is_none != field.resolved_value_is_none - - -class SemanticObjectStateFieldFilter(ObjectStateFieldFilterDeclaration): - field_filter = UiObjectStateFieldFilter.SEMANTIC - - @classmethod - def matches(cls, field: UiObjectStateFieldSummary) -> bool: - return ( - field.dirty - or field.signature_diff - or field.inherited_value - or RawResolvedObjectStateFieldFilter.matches(field) - ) - - class ObjectStateFieldListProjector: """Project scope catalogs through an ObjectState field query.""" @@ -152,17 +61,11 @@ def project_catalog( and field_path in container_field_paths ): continue - if not query.include_clean_fields and not ( - ObjectStateFieldFilterDeclaration.matches_filter( - UiObjectStateFieldFilter.SEMANTIC, - field, - ) + if not query.include_clean_fields and not SemanticFields.matches( + field ): continue - if not ObjectStateFieldFilterDeclaration.matches_filter( - query.field_filter, - field, - ): + if not query.field_filter.matches(field): continue matched_field_count += 1 if remaining_offset: @@ -197,7 +100,7 @@ def project_catalog( requested_scope_ids=query.scope_ids, field_paths=query.field_paths, field_path_contains=query.field_path_contains, - field_filter=query.field_filter.value, + field_filter=query.field_filter, include_container_fields=query.include_container_fields, matched_scope_count=len(matched_scopes), matched_field_count=matched_field_count, diff --git a/openhcs/agent/services/selected_plate_service.py b/openhcs/agent/services/selected_plate_service.py index ffdce1016e..bff865bd71 100644 --- a/openhcs/agent/services/selected_plate_service.py +++ b/openhcs/agent/services/selected_plate_service.py @@ -321,34 +321,19 @@ def selected_plate_target_root( self, selected_plate: SelectedPlateStateResolution, *, - target: SelectedPlateFileQueryTarget, + target: type[SelectedPlateFileQueryTarget], source_format: str, ) -> tuple[SelectedPlateTargetRoot | None, AgentError | None]: state_surface_id = PlateManagerStateSurfaceIdentityDeclaration.require_value() plate_root = selected_plate.plate_root if plate_root is None: raise RuntimeError("Selected row plate_root was validated as a string.") - selected_row = selected_plate.selected_plate or {} - if target is SelectedPlateFileQueryTarget.OUTPUT: - output_root = selected_row.get("output_root") - if not isinstance(output_root, str) or not output_root: - return None, AgentError( - code="ui_selected_plate_output_root_unavailable", - message=( - "The selected PlateManager row does not expose an " - "output_root." - ), - hint=( - "Call openhcs_ui_get_state_surface(surface_id=" - f"{state_surface_id!r}) to inspect source/output plate " - "relationships." - ), - ) - plate_root = output_root - elif target is SelectedPlateFileQueryTarget.SOURCE: - source_root = selected_row.get("source_root") - if isinstance(source_root, str) and source_root: - plate_root = source_root + resolved_root, error = target.plate_root( + selected_plate.selected_plate or {}, plate_root, state_surface_id + ) + if error is not None: + return None, error + plate_root = resolved_root return ( SelectedPlateTargetRoot( plate_root=plate_root, @@ -487,11 +472,11 @@ def first_selected_plate_image_path( def stream_context_plate_root( selected_plate: SelectedPlateStateResolution, *, - target: SelectedPlateFileQueryTarget, + target: type[SelectedPlateFileQueryTarget], kind: PlateFileKind | None, ) -> str | None: if ( - target is not SelectedPlateFileQueryTarget.OUTPUT + not target.streams_from_source_context or kind is not PlateFileKind.RESULT ): return None diff --git a/openhcs/agent/services/ui_bridge_service.py b/openhcs/agent/services/ui_bridge_service.py index 916c4ef155..2b5f3b2de4 100644 --- a/openhcs/agent/services/ui_bridge_service.py +++ b/openhcs/agent/services/ui_bridge_service.py @@ -32,6 +32,8 @@ from openhcs.agent.dto.common import SCHEMA_VERSION, AgentError from openhcs.agent.dto.session import SessionEventBatch, SessionEventsRequest from openhcs.agent.dto.ui_bridge import ( + OperationRunning, + OperationUnavailable, UI_BRIDGE_UNKNOWN_WIDGET, UNKNOWN_UI_BRIDGE_OPERATION_ROUTE, UiActionCatalog, @@ -370,7 +372,7 @@ def failed(cls, request, errors): operation_id=request.operation_id, route=UNKNOWN_UI_BRIDGE_OPERATION_ROUTE, ), - status=UiBridgeOperationStatus.UNAVAILABLE.value, + status=OperationUnavailable.name, started_at_unix=0.0, errors=errors, ) @@ -811,7 +813,7 @@ def call(cls, service, connection, request): service, connection, status_request ) try: - status = UiBridgeOperationStatus(operation.status) + status = UiBridgeOperationStatus.named(operation.status) except ValueError: return replace( operation, @@ -947,7 +949,7 @@ def accepted(cls, request, bridge_operation): document_id=request.document_id, applied=False, base_revision_token=request.base_revision_token, - outcome=UiBridgeOperationStatus.RUNNING.value, + outcome=OperationRunning.name, operation_id=bridge_operation.identity.operation_id, receipt=UiMutationReceipt.accepted_for( request.request_token, diff --git a/openhcs/mcp/dev_client_commands/plate.py b/openhcs/mcp/dev_client_commands/plate.py index f69059874c..22732eec8c 100644 --- a/openhcs/mcp/dev_client_commands/plate.py +++ b/openhcs/mcp/dev_client_commands/plate.py @@ -479,7 +479,7 @@ def configure_parser( request_type, "target", "--target", - choices=tuple(target.value for target in SelectedPlateFileQueryTarget), + choices=tuple(SelectedPlateFileQueryTarget.__registry__), help="Plate root to stream from the selected PlateManager row.", ) parser.add_argument("--path-contains") @@ -604,7 +604,7 @@ def configure_parser(self, parser: argparse.ArgumentParser) -> None: SelectedPlateImageInspectionRequest, "target", "--target", - choices=tuple(target.value for target in SelectedPlateFileQueryTarget), + choices=tuple(SelectedPlateFileQueryTarget.__registry__), help="Plate root to inspect from the selected PlateManager row.", ) add_request_field_option( @@ -685,7 +685,7 @@ def configure_parser(self, parser: argparse.ArgumentParser) -> None: SelectedPlateFileQueryRequest, "target", "--target", - choices=tuple(target.value for target in SelectedPlateFileQueryTarget), + choices=tuple(SelectedPlateFileQueryTarget.__registry__), help="Plate root to query from the selected PlateManager row.", ) add_request_field_option( @@ -792,7 +792,7 @@ def configure_parser(self, parser: argparse.ArgumentParser) -> None: SelectedPlateImageSampleRequest, "target", "--target", - choices=tuple(target.value for target in SelectedPlateFileQueryTarget), + choices=tuple(SelectedPlateFileQueryTarget.__registry__), help="Plate root to sample from the selected PlateManager row.", ) add_request_field_option(parser, SelectedPlateImageSampleRequest, "y", "--y") diff --git a/openhcs/mcp/dev_client_commands/ui.py b/openhcs/mcp/dev_client_commands/ui.py index 368505fb91..1350208daa 100644 --- a/openhcs/mcp/dev_client_commands/ui.py +++ b/openhcs/mcp/dev_client_commands/ui.py @@ -968,7 +968,7 @@ def calls_from_args( field_path_contains=tuple(args.contains), include_system_scopes=args.include_system_scopes, include_clean_fields=( - args.field_filter == UiObjectStateFieldFilter.ALL.value + UiObjectStateFieldFilter.named(args.field_filter).keeps_every_field ), include_container_fields=args.include_container_fields, field_filter=args.field_filter, diff --git a/openhcs/mcp/dev_client_core.py b/openhcs/mcp/dev_client_core.py index 6afabc45ee..5aa04f4a86 100644 --- a/openhcs/mcp/dev_client_core.py +++ b/openhcs/mcp/dev_client_core.py @@ -41,6 +41,12 @@ from openhcs.agent.dto.common import AgentCliRequest, AgentError, AgentResultEnvelope from openhcs.agent.dto.mcp import McpBoundaryFailure, McpToolErrorResult from openhcs.agent.dto.ui_bridge import ( + AllFields, + DefaultDiffFields, + DirtyFields, + InheritedFields, + RawResolvedFields, + SemanticFields, UiActionInvocationStatus, UiBridgeOperationRef, UiBridgeOperationStatus, @@ -2172,17 +2178,17 @@ def code_document_source_from_args(args: argparse.Namespace) -> str: def add_object_state_field_filter_options(parser: argparse.ArgumentParser) -> None: filter_group = parser.add_mutually_exclusive_group() - parser.set_defaults(field_filter=UiObjectStateFieldFilter.ALL.value) + parser.set_defaults(field_filter=AllFields.name) filter_group.add_argument( "--field-filter", - choices=tuple(field_filter.value for field_filter in UiObjectStateFieldFilter), + choices=tuple(UiObjectStateFieldFilter.__registry__), help="Return fields matching one ObjectState semantic filter.", ) filter_group.add_argument( "--dirty-only", dest="field_filter", action="store_const", - const=UiObjectStateFieldFilter.DIRTY.value, + const=DirtyFields.name, help="Only return unsaved ObjectState fields marked with * semantics.", ) filter_group.add_argument( @@ -2190,7 +2196,7 @@ def add_object_state_field_filter_options(parser: argparse.ArgumentParser) -> No "--semantic-only", dest="field_filter", action="store_const", - const=UiObjectStateFieldFilter.SEMANTIC.value, + const=SemanticFields.name, help=( "Only return dirty, default-diff, inherited, or raw/resolved-none " "semantic fields." @@ -2200,21 +2206,21 @@ def add_object_state_field_filter_options(parser: argparse.ArgumentParser) -> No "--default-diff-only", dest="field_filter", action="store_const", - const=UiObjectStateFieldFilter.DEFAULT_DIFF.value, + const=DefaultDiffFields.name, help="Only return fields that differ from constructor/default code mode.", ) filter_group.add_argument( "--inherited-only", dest="field_filter", action="store_const", - const=UiObjectStateFieldFilter.INHERITED.value, + const=InheritedFields.name, help="Only return inherited/resolved ObjectState fields.", ) filter_group.add_argument( "--raw-resolved-only", dest="field_filter", action="store_const", - const=UiObjectStateFieldFilter.RAW_RESOLVED.value, + const=RawResolvedFields.name, help="Only return fields whose raw None/resolved None state differs.", ) @@ -2362,7 +2368,7 @@ def workflow_operation_receipt_skip_reason( if operation is None or operation.errors: return WorkflowPollSkipReason.OPERATION_RECEIPT_FAILED try: - status = UiBridgeOperationStatus(operation.status) + status = UiBridgeOperationStatus.named(operation.status) except ValueError: return WorkflowPollSkipReason.OPERATION_RECEIPT_FAILED return status.select_completion( diff --git a/openhcs/mcp/dev_client_renderers/knowledge.py b/openhcs/mcp/dev_client_renderers/knowledge.py index cd49d3523c..a1e8211c03 100644 --- a/openhcs/mcp/dev_client_renderers/knowledge.py +++ b/openhcs/mcp/dev_client_renderers/knowledge.py @@ -17,7 +17,6 @@ FunctionCatalogEntry, FunctionCatalogPage, FunctionDetail, - FunctionParameterSource, FunctionParameterSpec, ) from openhcs.agent.dto.knowledge import ( @@ -372,12 +371,12 @@ def render_payload( agent_parameters = tuple( parameter for parameter in payload.parameters - if parameter.supplied_by is FunctionParameterSource.AGENT + if parameter.supplied_by.agent_supplied ) runtime_parameters = tuple( parameter for parameter in payload.parameters - if parameter.supplied_by is not FunctionParameterSource.AGENT + if not parameter.supplied_by.agent_supplied ) artifact_outputs = ( () diff --git a/openhcs/mcp/dev_client_renderers/object_state.py b/openhcs/mcp/dev_client_renderers/object_state.py index 5e7aa9e5b6..3f77731329 100644 --- a/openhcs/mcp/dev_client_renderers/object_state.py +++ b/openhcs/mcp/dev_client_renderers/object_state.py @@ -4,7 +4,6 @@ from openhcs.agent.dto.ui_bridge import ( UiCatalogPageMetadata, - UiObjectStateFieldFilter, UiObjectStateFieldHelpResult, UiObjectStateFieldListResult, UiObjectStateFieldMutationResult, @@ -255,8 +254,8 @@ def _filter_lines(payload: UiObjectStateFieldListResult) -> tuple[str, ...]: filters.append("field_paths=" + ",".join(payload.field_paths)) if payload.field_path_contains: filters.append("contains=" + ",".join(payload.field_path_contains)) - if payload.field_filter != UiObjectStateFieldFilter.ALL.value: - filters.append(f"field_filter={payload.field_filter}") + if not payload.field_filter.keeps_every_field: + filters.append(f"field_filter={payload.field_filter.name}") if payload.include_container_fields: filters.append("include_container_fields=True") return ("Filters: " + " ".join(filters),) if filters else () diff --git a/openhcs/mcp/dev_client_renderers/plate.py b/openhcs/mcp/dev_client_renderers/plate.py index e4a05b8010..badb1e0ed9 100644 --- a/openhcs/mcp/dev_client_renderers/plate.py +++ b/openhcs/mcp/dev_client_renderers/plate.py @@ -25,7 +25,6 @@ PlateInspectionResultFileRecordSummary, PlatePathInspectionResult, SelectedPlateFileQueryResult, - SelectedPlateFileQueryTarget, SelectedPlateFileStreamResult, SelectedPlateImageInspectionResult, SelectedPlateImageSampleResult, @@ -843,7 +842,7 @@ def selected_plate_line(cls, payload) -> str: "Selected plate: " f"{cls.text(None if row is None else row.name)} " f"root={cls.text(None if row is None else row.root)} " - f"target={payload.target.value}" + f"target={payload.target.name}" ) @classmethod @@ -876,8 +875,8 @@ def render_payload( return "\n".join(lines) target_prefix = ( "" - if payload.target is SelectedPlateFileQueryTarget.SELECTED - else f"--target {payload.target.value} " + if payload.target.is_selected_row_root + else f"--target {payload.target.name} " ) lines.append( PlateInspectionRenderer.inspection_text( @@ -910,7 +909,7 @@ def render_payload( lines.append(PlateFileQueryRenderer.render_payload(query, options)) row = cls.selected_row(payload) if ( - payload.target is SelectedPlateFileQueryTarget.SELECTED + payload.target.is_selected_row_root and query.total_count == 0 and row is not None and row.output_root diff --git a/openhcs/mcp/dev_client_rendering.py b/openhcs/mcp/dev_client_rendering.py index ec49f58683..7fbea544df 100644 --- a/openhcs/mcp/dev_client_rendering.py +++ b/openhcs/mcp/dev_client_rendering.py @@ -16,7 +16,7 @@ from typing import ClassVar, TypeAlias, TypeVar from metaclass_registry import AutoRegisterMeta -from python_introspect import JsonValue +from python_introspect import JsonValue, to_jsonable from openhcs.agent.capabilities import ( AgentCapabilityDeclaration, @@ -350,8 +350,8 @@ def diagnostic_lines(errors: Iterable[AgentError]) -> tuple[str, ...]: def text(value: object, *, absent_text: str = "") -> str: if value is None: return absent_text - if isinstance(value, Enum): - return str(value.value) + if isinstance(value, Enum | AutoRegisterMeta): + return str(to_jsonable(value)) return str(value) @staticmethod diff --git a/openhcs/pyqt_gui/services/ui_agent_bridge.py b/openhcs/pyqt_gui/services/ui_agent_bridge.py index 913d91547c..b6e88e0652 100644 --- a/openhcs/pyqt_gui/services/ui_agent_bridge.py +++ b/openhcs/pyqt_gui/services/ui_agent_bridge.py @@ -35,6 +35,11 @@ RegistryFunctionReference, ) from openhcs.agent.dto.ui_bridge import ( + OperationCompleted, + OperationFailed, + OperationNotFound, + OperationRunning, + OperationUnavailable, UNKNOWN_UI_BRIDGE_OPERATION_ROUTE, UiActionCatalog, UiActionIdentity, @@ -701,7 +706,7 @@ def complete( try: self._tracker.complete( operation.identity.operation_id, - status=UiBridgeOperationStatus.COMPLETED, + status=OperationCompleted, outcome=operation_type.outcome(result), errors=result.errors, warnings=result.warnings, @@ -718,7 +723,7 @@ def fail( try: self._tracker.complete( operation.identity.operation_id, - status=UiBridgeOperationStatus.FAILED, + status=OperationFailed, outcome="error", errors=( AgentError.from_exception("ui_bridge_operation_failed", exception), @@ -751,7 +756,7 @@ def start(self, operation_name: str, target_id: str | None) -> UiBridgeOperation target_id=target_id, ), ), - status=UiBridgeOperationStatus.RUNNING.value, + status=OperationRunning.name, started_at_unix=time.time(), ) with self._lock: @@ -762,7 +767,7 @@ def complete( self, operation_id: str, *, - status: UiBridgeOperationStatus, + status: type[UiBridgeOperationStatus], outcome: str | None, errors: tuple[AgentError, ...] = (), warnings: tuple[AgentWarning, ...] = (), @@ -772,7 +777,7 @@ def complete( operation = self._operations[operation_id] completed = replace( operation, - status=status.value, + status=status.name, completed_at_unix=time.time(), outcome=outcome, errors=errors, @@ -797,9 +802,9 @@ def complete_result( return self.complete( operation_id, status=( - UiBridgeOperationStatus.FAILED + OperationFailed if result.errors - else UiBridgeOperationStatus.COMPLETED + else OperationCompleted ), outcome="error" if result.errors else contract.success_outcome, errors=result.errors, @@ -816,7 +821,7 @@ def fail_exception( ) return self.complete( operation_id, - status=UiBridgeOperationStatus.FAILED, + status=OperationFailed, outcome="error", errors=(AgentError.from_exception(contract.failure_error_code, error),), ) @@ -855,7 +860,7 @@ def get(self, operation_id: str) -> UiBridgeOperationRef: operation_id=operation_id, route=UNKNOWN_UI_BRIDGE_OPERATION_ROUTE, ), - status=UiBridgeOperationStatus.NOT_FOUND.value, + status=OperationNotFound.name, started_at_unix=0.0, completed_at_unix=0.0, outcome="not_found", @@ -874,10 +879,8 @@ def operations(self) -> tuple[UiBridgeOperationRef, ...]: def overview_sections(self) -> tuple[UiLiveOverviewSection, ...]: operations = self.operations() status_counts = { - status: sum( - 1 for operation in operations if operation.status == status.value - ) - for status in UiBridgeOperationStatus + status: sum(1 for operation in operations if operation.status == status.name) + for status in (OperationRunning, OperationFailed, OperationCompleted) } recent_operations = tuple(reversed(operations[-5:])) return ( @@ -885,25 +888,25 @@ def overview_sections(self) -> tuple[UiLiveOverviewSection, ...]: section_id=self.overview_identity.section_id, title=self.overview_identity.title, summary=( - f"{status_counts[UiBridgeOperationStatus.RUNNING]} running, " - f"{status_counts[UiBridgeOperationStatus.FAILED]} failed, " - f"{status_counts[UiBridgeOperationStatus.COMPLETED]} completed" + f"{status_counts[OperationRunning]} running, " + f"{status_counts[OperationFailed]} failed, " + f"{status_counts[OperationCompleted]} completed" ), metrics=( UiLiveOverviewMetric( - key=UiBridgeOperationStatus.RUNNING.value, + key=OperationRunning.name, label="Running", - value=str(status_counts[UiBridgeOperationStatus.RUNNING]), + value=str(status_counts[OperationRunning]), ), UiLiveOverviewMetric( - key=UiBridgeOperationStatus.FAILED.value, + key=OperationFailed.name, label="Failed", - value=str(status_counts[UiBridgeOperationStatus.FAILED]), + value=str(status_counts[OperationFailed]), ), UiLiveOverviewMetric( - key=UiBridgeOperationStatus.COMPLETED.value, + key=OperationCompleted.name, label="Completed", - value=str(status_counts[UiBridgeOperationStatus.COMPLETED]), + value=str(status_counts[OperationCompleted]), ), ), items=tuple( @@ -919,11 +922,13 @@ def overview_sections(self) -> tuple[UiLiveOverviewSection, ...]: ) @staticmethod - def _operation_status(operation: UiBridgeOperationRef) -> UiBridgeOperationStatus: + def _operation_status( + operation: UiBridgeOperationRef, + ) -> type[UiBridgeOperationStatus]: try: - return UiBridgeOperationStatus(operation.status) + return UiBridgeOperationStatus.named(operation.status) except ValueError: - return UiBridgeOperationStatus.UNAVAILABLE + return OperationUnavailable @classmethod def _operation_severity(cls, operation: UiBridgeOperationRef) -> str: diff --git a/openhcs/pyqt_gui/services/ui_bridge_object_state.py b/openhcs/pyqt_gui/services/ui_bridge_object_state.py index b5f318eafd..8860cf92c7 100644 --- a/openhcs/pyqt_gui/services/ui_bridge_object_state.py +++ b/openhcs/pyqt_gui/services/ui_bridge_object_state.py @@ -29,6 +29,7 @@ from openhcs.agent.dto.common import AgentError, SCHEMA_VERSION from openhcs.agent.dto.ui_bridge import ( + UiCatalogPageMetadata, UiCodeDocument, UiCodeDocumentApplyRequest, UiCodeDocumentApplyResult, @@ -36,12 +37,10 @@ UiCodeDocumentSummary, UiCodeDocumentValidationRequest, UiCodeDocumentValidationResult, - UiCatalogPageMetadata, UiMutationReceipt, - UiObjectStateFieldFilter, - UiObjectStateFieldProvenance, UiObjectStateFieldMutationRequest, UiObjectStateFieldMutationResult, + UiObjectStateFieldProvenance, UiObjectStateFieldSemanticCarrier, UiObjectStateFieldSummary, UiObjectStateScopeCatalog, @@ -51,9 +50,6 @@ UiObjectStateValuePreview, UiSemanticAddress, ) -from openhcs.agent.services.object_state_field_projection import ( - ObjectStateFieldFilterDeclaration, -) from objectstate import ObjectState, ObjectStateRegistry from objectstate.inspection import FieldInspection, type_name, type_qualname from openhcs.pyqt_gui.services.ui_bridge_contracts import ( @@ -425,12 +421,11 @@ def _field_page( ) else: all_field_paths = tuple(sorted(state.parameters.keys())) - if request.field_filter is not UiObjectStateFieldFilter.ALL: + if not request.field_filter.keeps_every_field: all_field_paths = tuple( field_path for field_path in all_field_paths - if ObjectStateFieldFilterDeclaration.matches_filter( - request.field_filter, + if request.field_filter.matches( self._field_summary( state, field_path, diff --git a/tests/unit/agent/test_agent_services.py b/tests/unit/agent/test_agent_services.py index 3d1f960602..cfacfe9863 100644 --- a/tests/unit/agent/test_agent_services.py +++ b/tests/unit/agent/test_agent_services.py @@ -33,7 +33,11 @@ PipelineSourceArtifactPlanInspectionRequest, RuntimeServerExecutionStatusRequest, ) -from openhcs.agent.dto.functions import FunctionParameterSource +from openhcs.agent.dto.functions import ( + SuppliedByAgent, + SuppliedByRuntime, + SuppliedFromPrimaryInput, +) from openhcs.agent.dto.pipeline import CreatePipelineRequest from openhcs.agent.dto.viewer import ( ViewerWindowCloseRequest, @@ -1173,8 +1177,8 @@ def test_function_catalog_search_and_describe_use_registry_ids(monkeypatch): assert detail.entry.signature == "sample_processing_function(sigma=1.0)" assert [parameter.name for parameter in detail.parameters] == ["image", "sigma"] assert [parameter.supplied_by for parameter in detail.parameters] == [ - FunctionParameterSource.PRIMARY_INPUT, - FunctionParameterSource.AGENT, + SuppliedFromPrimaryInput, + SuppliedByAgent, ] assert detail.parameters[0].required is False assert detail.parameters[1].required is False @@ -1236,7 +1240,7 @@ def test_function_catalog_projects_runtime_context_from_callable_contract( == "context" ) assert ( - parameters["context"].supplied_by is FunctionParameterSource.RUNTIME_PARAMETER + parameters["context"].supplied_by is SuppliedByRuntime ) assert parameters["context"].required is False assert parameters["context"].description == ( diff --git a/tests/unit/agent/test_declared_callable_documentation.py b/tests/unit/agent/test_declared_callable_documentation.py index e21dd0afc5..368c6e43c4 100644 --- a/tests/unit/agent/test_declared_callable_documentation.py +++ b/tests/unit/agent/test_declared_callable_documentation.py @@ -9,7 +9,10 @@ import numpy as np import pytest -from openhcs.agent.dto.functions import FunctionParameterSource +from openhcs.agent.dto.functions import ( + SuppliedByAgent, + SuppliedByRuntime, +) from openhcs.agent.services.function_catalog_service import ( FunctionCatalogService, PARAMETER_DOCUMENTATION_POLICY, @@ -81,7 +84,7 @@ def independent_doc_probe(image: np.ndarray, *, offset: float = 0.75) -> np.ndar assert "slice_by_slice" not in {parameter.name for parameter in detail.parameters} offset = next(parameter for parameter in detail.parameters if parameter.name == "offset") assert offset.default_repr == "0.75" - assert offset.supplied_by is FunctionParameterSource.AGENT + assert offset.supplied_by is SuppliedByAgent assert offset.description == "Declared intensity offset in original image units." assert catalog.search(query="original image units").total == 1 @@ -101,7 +104,7 @@ def independent_flexible_doc_probe(image: np.ndarray, *, offset: float = 0.5) -> parameters = {parameter.name: parameter for parameter in detail.parameters} assert parameters["slice_by_slice"].default_repr == "False" assert "Process 3D arrays slice-by-slice" in parameters["slice_by_slice"].description - assert parameters["dtype_config"].supplied_by is FunctionParameterSource.RUNTIME_PARAMETER + assert parameters["dtype_config"].supplied_by is SuppliedByRuntime pixels = np.zeros((2, 4, 6)) result = metadata.func(pixels, slice_by_slice=True) assert seen == [(4, 6), (4, 6)] diff --git a/tests/unit/agent/test_function_declaration_help.py b/tests/unit/agent/test_function_declaration_help.py index 9d749e2188..66b73b7698 100644 --- a/tests/unit/agent/test_function_declaration_help.py +++ b/tests/unit/agent/test_function_declaration_help.py @@ -178,7 +178,10 @@ def test_policy_consumer_executes_cooperative_hooks_once_in_declared_mro(): from pyqt_reactive.services.parameter_help_service import ( dataclass_type_from_annotation, ) - from openhcs.agent.dto.functions import FunctionParameterSource + from openhcs.agent.dto.functions import ( + SuppliedByAgent, + SuppliedFromPrimaryInput, + ) policy = AuditedPolicy() specs = policy.parameter_specs(independent_probe) @@ -190,8 +193,8 @@ def test_policy_consumer_executes_cooperative_hooks_once_in_declared_mro(): dataclass_type_from_annotation(policy.annotations[controls_index]) is IndependentControls ) - assert policy.suppliers[controls_index] is FunctionParameterSource.AGENT - assert policy.suppliers[0] is FunctionParameterSource.PRIMARY_INPUT + assert policy.suppliers[controls_index] is SuppliedByAgent + assert policy.suppliers[0] is SuppliedFromPrimaryInput assert "Acquisition duration in seconds" in specs[controls_index].description assert "Search radius in millimeters" in specs[controls_index].description diff --git a/tests/unit/agent/test_mcp_cellprofiler_function_contract_friction.py b/tests/unit/agent/test_mcp_cellprofiler_function_contract_friction.py index b8994c015a..d24a2019cf 100644 --- a/tests/unit/agent/test_mcp_cellprofiler_function_contract_friction.py +++ b/tests/unit/agent/test_mcp_cellprofiler_function_contract_friction.py @@ -3,7 +3,7 @@ from collections.abc import Callable from dataclasses import dataclass -from openhcs.agent.dto.functions import FunctionParameterSource +from openhcs.agent.dto.functions import SuppliedFromArtifactInput from openhcs.agent.services.function_catalog_service import FunctionCatalogService from openhcs.processing.backends.cellprofiler.feature_enhancement import ( enhance_or_suppress_features, @@ -157,7 +157,7 @@ def test_function_detail_classifies_normalized_artifact_fed_parameter(monkeypatc assert "seed_labels" not in detail.entry.signature assert ( - parameters["seed_labels"].supplied_by is FunctionParameterSource.ARTIFACT_INPUT + parameters["seed_labels"].supplied_by is SuppliedFromArtifactInput ) assert parameters["seed_labels"].required is False assert "do not pass this as a function kwarg" in ( @@ -191,7 +191,7 @@ def test_catalog_exposes_declared_exact_object_selector_and_authoring_guidance( assert "one-element tuple selects one producer" in contract.source_binding_rule assert "do not pass labels" in contract.source_binding_rule parameters = {parameter.name: parameter for parameter in detail.parameters} - assert parameters["labels"].supplied_by is FunctionParameterSource.ARTIFACT_INPUT + assert parameters["labels"].supplied_by is SuppliedFromArtifactInput def test_catalog_projects_registered_and_new_binding_declarations_without_a_roster(): diff --git a/tests/unit/agent/test_mcp_dev_client_knowledge_renderers.py b/tests/unit/agent/test_mcp_dev_client_knowledge_renderers.py index 50a8cad40a..183712084a 100644 --- a/tests/unit/agent/test_mcp_dev_client_knowledge_renderers.py +++ b/tests/unit/agent/test_mcp_dev_client_knowledge_renderers.py @@ -20,9 +20,9 @@ FunctionCatalogEntry, FunctionCatalogPage, FunctionDetail, - FunctionParameterSource, FunctionParameterSpec, FunctionRuntimeContractSummary, + SuppliedFromPrimaryInput, ) from openhcs.agent.dto.knowledge import ( KnowledgeBaseCatalog, @@ -34,6 +34,7 @@ KnowledgeBaseSourceSpan, ) from openhcs.agent.dto.ui_bridge import ( + DirtyFields, UiCatalogPageMetadata, UiMutationReceipt, UiMutationRequestToken, @@ -155,7 +156,7 @@ def _field_summary(path: str, **semantics) -> UiObjectStateFieldSummary: requested_scope_ids=("scope-1",), field_paths=(), field_path_contains=("worker",), - field_filter="dirty", + field_filter=DirtyFields, include_container_fields=False, matched_scope_count=1, matched_field_count=2, @@ -273,7 +274,7 @@ def _field_summary(path: str, **semantics) -> UiObjectStateFieldSummary: FunctionParameterSpec("threshold", "float", "0.5", False), FunctionParameterSpec( "image", "ndarray", None, True, - supplied_by=FunctionParameterSource.PRIMARY_INPUT, + supplied_by=SuppliedFromPrimaryInput, description="From the step input.", ), ), diff --git a/tests/unit/agent/test_mcp_dev_client_plate_renderers.py b/tests/unit/agent/test_mcp_dev_client_plate_renderers.py index 39db8df27b..fe3992bbd1 100644 --- a/tests/unit/agent/test_mcp_dev_client_plate_renderers.py +++ b/tests/unit/agent/test_mcp_dev_client_plate_renderers.py @@ -30,7 +30,8 @@ PlateInspectionValueSource, PlatePathInspectionResult, SelectedPlateFileQueryResult, - SelectedPlateFileQueryTarget, + OutputPlateRoot, + SelectedRowRoot, SelectedPlateFileStreamResult, SelectedPlateImageInspectionResult, SelectedPlateImageSampleResult, @@ -302,8 +303,8 @@ def test_stream_bounds_paths_and_suggests_viewer_commands(): @pytest.mark.parametrize( "target,command", [ - (SelectedPlateFileQueryTarget.SELECTED, "Next: selected-plate-sample A01_w2.tif"), - (SelectedPlateFileQueryTarget.OUTPUT, "Next: selected-plate-sample --target output A01_w2.tif"), + (SelectedRowRoot, "Next: selected-plate-sample A01_w2.tif"), + (OutputPlateRoot, "Next: selected-plate-sample --target output A01_w2.tif"), ], ) def test_selected_plate_images_render_the_nested_inspection(target, command): @@ -313,7 +314,7 @@ def test_selected_plate_images_render_the_nested_inspection(target, command): target=target, inspection=inspection(), ) ) - assert text.startswith(f"Selected plate: plate-a root=/plates/a target={target.value}\nPlate: /plates/a") + assert text.startswith(f"Selected plate: plate-a root=/plates/a target={target.name}\nPlate: /plates/a") assert command in text assert "sample-plate-image" not in text assert text.count("plate_parse_failures") == 1 @@ -344,7 +345,7 @@ def test_selected_plate_sample_and_stream_keep_target_context_on_failure(): sample_text = render( SelectedPlateImageSampleResult( schema_version=SCHEMA_VERSION, selected_plate=to_jsonable(row()), - target=SelectedPlateFileQueryTarget.OUTPUT, image_path="a.tif", errors=(error,), + target=OutputPlateRoot, image_path="a.tif", errors=(error,), ) ) assert sample_text.startswith( @@ -362,7 +363,7 @@ def test_selected_plate_sample_and_stream_keep_target_context_on_failure(): nested = render( SelectedPlateFileStreamResult( schema_version=SCHEMA_VERSION, selected_plate=to_jsonable(row()), - target=SelectedPlateFileQueryTarget.OUTPUT, + target=OutputPlateRoot, stream=stream(warnings=(AgentWarning("viewer_slow", "slow start"),)), ) ) diff --git a/tests/unit/agent/test_mcp_server.py b/tests/unit/agent/test_mcp_server.py index 31c58bee11..224c7ed009 100644 --- a/tests/unit/agent/test_mcp_server.py +++ b/tests/unit/agent/test_mcp_server.py @@ -62,7 +62,7 @@ UiCodeDocumentSummary, UiCatalogPageMetadata, UiMutationReceipt, - UiObjectStateFieldFilter, + SemanticFields, UiObjectStateFieldHelpRequest, UiObjectStateFieldMutationResult, UiObjectStateFieldProvenance, @@ -2284,7 +2284,7 @@ def list_object_state_scopes(self, request, connection): assert connection == "ui-connection" assert request.scope_ids == ("/tmp/plate",) assert request.include_fields is True - assert request.field_filter is UiObjectStateFieldFilter.SEMANTIC + assert request.field_filter is SemanticFields return request.filtered_catalog( UiObjectStateScopeCatalog( schema_version=SCHEMA_VERSION, diff --git a/tests/unit/agent/test_object_state_field_help_service.py b/tests/unit/agent/test_object_state_field_help_service.py index 12198fd0a3..d13faedb8e 100644 --- a/tests/unit/agent/test_object_state_field_help_service.py +++ b/tests/unit/agent/test_object_state_field_help_service.py @@ -6,9 +6,10 @@ FunctionArtifactSpec, FunctionCatalogEntry, FunctionDetail, - FunctionParameterSource, FunctionParameterSpec, FunctionRuntimeContractSummary, + SuppliedFromArtifactInput, + SuppliedFromPrimaryInput, ) from openhcs.agent.dto.ui_bridge import ( UiObjectStateFieldHelpRequest, @@ -141,7 +142,7 @@ def get_by_import_path( annotation="ndarray", default_repr=None, required=False, - supplied_by=FunctionParameterSource.PRIMARY_INPUT, + supplied_by=SuppliedFromPrimaryInput, ), FunctionParameterSpec( name="min_cell_area", @@ -329,14 +330,14 @@ def get_by_import_path( annotation="ndarray", default_repr=None, required=False, - supplied_by=FunctionParameterSource.PRIMARY_INPUT, + supplied_by=SuppliedFromPrimaryInput, ), FunctionParameterSpec( name="grid_dimensions", annotation="tuple[int, int]", default_repr=None, required=False, - supplied_by=FunctionParameterSource.ARTIFACT_INPUT, + supplied_by=SuppliedFromArtifactInput, ), FunctionParameterSpec( name="overlap_ratio", diff --git a/tests/unit/agent/test_ui_bridge_service.py b/tests/unit/agent/test_ui_bridge_service.py index 1b1d2b5cb4..271629e893 100644 --- a/tests/unit/agent/test_ui_bridge_service.py +++ b/tests/unit/agent/test_ui_bridge_service.py @@ -15,6 +15,12 @@ from openhcs.agent.path_policy import AgentPathPolicy from openhcs.agent.dto.common import SCHEMA_VERSION, AgentError, AgentResourceRef from openhcs.agent.dto.ui_bridge import ( + SemanticFields, + OperationCompleted, + OperationFailed, + OperationNotFound, + OperationRunning, + OperationUnavailable, UiActionCatalog, UiActionIdentity, UiActionInvocationStatus, @@ -22,7 +28,6 @@ UiActionInvokeResult, UiBranchCatalog, UiBranchSwitchRequest, - UiBridgeOperationStatus, UiBridgeConfirmationRequirement, UiBridgeConnectionSpec, UiBridgeOperationIdentity, @@ -1581,7 +1586,7 @@ def get_operation_status(connection, request): operation_id=request.operation_id, route=UiBridgeOperationRoute(operation_name="invoke_widget_action"), ), - status=UiBridgeOperationStatus.COMPLETED.value, + status=OperationCompleted.name, started_at_unix=1.0, completed_at_unix=2.0, outcome="invoked", @@ -1645,7 +1650,7 @@ def get_operation_status(connection, request): operation_id=request.operation_id, route=UiBridgeOperationRoute(operation_name="invoke_widget_action"), ), - status=UiBridgeOperationStatus.FAILED.value, + status=OperationFailed.name, started_at_unix=1.0, completed_at_unix=2.0, outcome="error", @@ -1823,7 +1828,7 @@ def list_object_state_scopes( ) ) - assert result.field_filter == "semantic" + assert result.field_filter is SemanticFields assert result.matched_field_count == 1 assert result.returned_field_count == 1 assert result.scopes[0].fields[0].field_path == "napari_streaming_config.enabled" @@ -2073,24 +2078,24 @@ def test_ui_bridge_operation_status_owns_completion_selection(): } assert ( - UiBridgeOperationStatus.RUNNING.select_completion(**choices) + OperationRunning.select_completion(**choices) is choices["active"] ) assert ( - UiBridgeOperationStatus.COMPLETED.select_completion(**choices) + OperationCompleted.select_completion(**choices) is choices["succeeded"] ) for status in ( - UiBridgeOperationStatus.FAILED, - UiBridgeOperationStatus.NOT_FOUND, - UiBridgeOperationStatus.UNAVAILABLE, + OperationFailed, + OperationNotFound, + OperationUnavailable, ): assert status.select_completion(**choices) is choices["failed"] - assert UiBridgeOperationStatus.RUNNING.is_terminal is False - assert UiBridgeOperationStatus.COMPLETED.is_terminal is True - assert UiBridgeOperationStatus.RUNNING.live_overview_severity == "info" - assert UiBridgeOperationStatus.FAILED.live_overview_severity == "error" + assert OperationRunning.is_terminal is False + assert OperationCompleted.is_terminal is True + assert OperationRunning.live_overview_severity == "info" + assert OperationFailed.live_overview_severity == "error" def test_ui_requests_own_mcp_tool_argument_projection(): diff --git a/tests/unit/pyqt_gui/test_ui_agent_bridge.py b/tests/unit/pyqt_gui/test_ui_agent_bridge.py index 620d575856..5ef06d9ed3 100644 --- a/tests/unit/pyqt_gui/test_ui_agent_bridge.py +++ b/tests/unit/pyqt_gui/test_ui_agent_bridge.py @@ -69,7 +69,7 @@ UiCodeDocumentRequest, UiCodeDocumentSelectionMode, UiCodeDocumentValidationRequest, - UiObjectStateFieldFilter, + DirtyFields, UiObjectStateScopeListRequest, UiSelectedPlateWorkflowKind, UiSelectedPlateWorkflowRequest, @@ -2608,7 +2608,7 @@ class MultiFieldDummy: UiObjectStateScopeListRequest( include_system_scopes=True, include_fields=True, - field_filter=UiObjectStateFieldFilter.DIRTY, + field_filter=DirtyFields, field_limit=1, ) ) diff --git a/tests/unit/test_catalog_current_readiness.py b/tests/unit/test_catalog_current_readiness.py index 6dd712f033..37c7d3d655 100644 --- a/tests/unit/test_catalog_current_readiness.py +++ b/tests/unit/test_catalog_current_readiness.py @@ -225,10 +225,10 @@ def entry(*args, **kwargs): preparation.ensure_started().result(timeout=2) assert preparation.observe(handle()).outcome is Outcome.READY assert service.projections_current() - from openhcs.agent.services.function_catalog_service import SignatureView + from openhcs.agent.services.function_catalog_service import FullSignature current = RegistryService._metadata_cache["probe:declaration"] - assert (current, SignatureView.FULL) in calls + assert (current, FullSignature) in calls assert all(projection.metadata is current for projection in service._projections) for compact in (False, True): assert service.catalog(compact_signatures=compact).items diff --git a/tests/unit/test_function_catalog_zmq.py b/tests/unit/test_function_catalog_zmq.py index 24ab4baef2..3d1724af55 100644 --- a/tests/unit/test_function_catalog_zmq.py +++ b/tests/unit/test_function_catalog_zmq.py @@ -48,6 +48,8 @@ FunctionReferenceControlRequest, FunctionReferenceControlResponse, FunctionSearchRequest, + SuppliedByAgent, + SuppliedFromPrimaryInput, catalog_page, ) from openhcs.agent.services.function_catalog_service import FunctionCatalogService @@ -64,9 +66,9 @@ def test_function_parameter_source_owns_runtime_presentation() -> None: - assert FunctionParameterSource.AGENT.runtime_description is None + assert SuppliedByAgent.runtime_description is None assert "FunctionStep input image payload" in ( - FunctionParameterSource.PRIMARY_INPUT.runtime_description or "" + SuppliedFromPrimaryInput.runtime_description or "" ) From 8c565253da7da71bb28769d0bde72cf20f0ba918 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:54:26 -0400 Subject: [PATCH 18/36] E1: source-binding kinds are families; filter, metadata and role behaviour on the members SourceFilterMatchType, SourceFilterSubject, MetadataSource, SourceSetRole, SourceProjectionRole and SourceBindingOrigin were enums whose meaning each consumer re-decided; the matcher and target-resolver strategy families restated two of them one-to-one, keyed by .value. Each is now an AutoRegisterMeta family: match types own matches() (text comparisons and their negations, path predicates), subjects own text_of(), metadata sources own texts(), the primary-plane role checks its artifact kind. Fields hold the member class with RegisteredClassChoices; the wire spelling is the registry name (unchanged strings). CellProfiler's subject literals decode through the family registry; its operator literals are one table naming our classes. Workspace projection strategies and source-universe requests key by the member class. Co-Authored-By: Claude Opus 5.5 --- benchmark/equivalence/outputs.py | 6 +- benchmark/validation/corpus.py | 16 +- openhcs/core/pipeline/compiler.py | 4 +- openhcs/core/source_binding_selection.py | 20 +- openhcs/core/source_binding_workspace.py | 48 +- openhcs/core/source_bindings.py | 455 +++++++++++++----- openhcs/core/source_matching.py | 294 +---------- openhcs/core/source_projection.py | 18 +- openhcs/core/source_workspace_projection.py | 9 +- openhcs/core/steps/function_runtime.py | 8 +- openhcs/core/virtual_workspace_metadata.py | 11 +- .../backends/cellprofiler/infrastructure.py | 222 ++++----- .../loose_operaphenix_neurite_outgrowth.py | 8 +- ...uroncyto_ii_crossover_neurite_outgrowth.py | 8 +- .../widgets/source_bindings_editor.py | 13 +- tests/diagnostics/owned_bootstrap_readback.py | 9 +- .../test_cellprofiler_generated_pipeline.py | 22 +- .../test_chained_callable_lineage_journey.py | 8 +- .../test_image_math_publication_620.py | 7 +- .../test_measurement_declaration_journey.py | 10 +- ...rimary_segmentation_diagnostics_journey.py | 8 +- ...est_source_schema_aggregate_publication.py | 4 +- .../test_source_store_code_mode_zmq.py | 8 +- .../pyqt_gui/test_source_bindings_editor.py | 306 ++++++------ .../test_time_travel_viewport_context.py | 8 +- .../test_bioformats_preparation_selection.py | 14 +- .../unit/test_cellprofiler_analyst_export.py | 6 +- ...test_cellprofiler_artifact_declarations.py | 13 +- ...ofiler_artifact_identity_reconstruction.py | 4 +- ...ellprofiler_combineobjects_source_scope.py | 7 +- .../test_cellprofiler_export_to_database.py | 4 +- ...llprofiler_generated_pipeline_execution.py | 8 +- ...lprofiler_generic_special_input_binding.py | 7 +- ...ellprofiler_morphology_public_contracts.py | 7 +- .../test_cellprofiler_neighbors_contract.py | 7 +- .../unit/test_cellprofiler_pipeline_import.py | 4 +- .../unit/test_cellprofiler_runtime_adapter.py | 16 +- tests/unit/test_cellprofiler_save_images.py | 9 +- .../unit/test_cellprofiler_source_bindings.py | 65 +-- ..._cellprofiler_source_bindings_ingestion.py | 22 +- tests/unit/test_compilation_session.py | 4 +- tests/unit/test_function_io.py | 32 +- ...st_function_output_manifest_input_scope.py | 13 +- ...test_function_runtime_source_projection.py | 48 +- tests/unit/test_function_step_transport.py | 7 +- .../unit/test_invocation_contract_provider.py | 4 +- ...nvocation_input_source_context_identity.py | 7 +- .../test_loaded_pattern_group_ownership.py | 9 +- ...t_microscope_virtual_workspace_metadata.py | 8 +- .../unit/test_path_planner_materialization.py | 10 +- tests/unit/test_plane_store_sources.py | 8 +- .../test_runtime_adapter_source_artifacts.py | 17 +- ...test_runtime_source_resolution_snapshot.py | 6 +- tests/unit/test_runtime_values.py | 7 +- .../test_source_binding_path_admission.py | 26 +- tests/unit/test_source_binding_workspace.py | 188 ++++---- tests/unit/test_source_bindings.py | 123 +++-- tests/unit/test_source_bindings_preview.py | 56 +-- ..._source_bound_matched_image_set_anchors.py | 12 +- tests/unit/test_source_matching.py | 30 +- tests/unit/test_source_projection.py | 13 +- tests/unit/test_source_tile_geometry.py | 7 +- tests/unit/test_step_input_artifact_origin.py | 4 +- 63 files changed, 1207 insertions(+), 1155 deletions(-) diff --git a/benchmark/equivalence/outputs.py b/benchmark/equivalence/outputs.py index ee5accd3d8..a2fa0a3577 100644 --- a/benchmark/equivalence/outputs.py +++ b/benchmark/equivalence/outputs.py @@ -24,7 +24,9 @@ RuntimeArtifactExecutionObservation, ) from openhcs.core.runtime_exports import RuntimeExportObservation -from openhcs.core.source_bindings import SourceProjectionRole +from openhcs.core.source_bindings import ( + SourceArtifactRole, +) from openhcs.core.source_matching import SourceImageSetIdentityPolicy from openhcs.core.source_projection import SourceProjectionSet from openhcs.core.source_workspace_projection import VirtualWorkspaceSourceProjection @@ -159,7 +161,7 @@ def image_snapshots( for projection in ( workspace.source_projections_by_virtual_path[virtual_path], ) - if projection.projection_role is SourceProjectionRole.SOURCE_ARTIFACT + if projection.projection_role is SourceArtifactRole and (root / projection.ref.backend_address).absolute() in expected_paths ) if not projections: diff --git a/benchmark/validation/corpus.py b/benchmark/validation/corpus.py index 646b715247..0e498a2990 100644 --- a/benchmark/validation/corpus.py +++ b/benchmark/validation/corpus.py @@ -43,16 +43,16 @@ ImportedMetadataTable, MetadataExtractionRule, MetadataSelector, - MetadataSource, + FromFileName, + FromPipelineStart, NamedSourceBinding, SourceBindingMatchDimension, SourceBindingMatchField, PairByMetadata, PairByOrder, - SourceBindingOrigin, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, + PathIsImage, + SourceFileName, SourceSelector, ) from openhcs.domains.microscopy.axes import Microscopy @@ -533,7 +533,7 @@ def source_bindings_for_validation( metadata=(MetadataSelector(field="channel", value=channel.value),), inherit_current_scope=True, ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=( ComponentSelector( component=Microscopy.Channel, @@ -562,14 +562,14 @@ def source_bindings_for_validation( return SourceBindingsConfig( source_filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.IS_IMAGE, + subject=SourceFileName, + match_type=PathIsImage, ), ), bindings=bindings, metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=( r"^(?:plate-(?P[^_]+)_)?" r"well-(?P[A-P]\d{2})_site-(?P[^_]+)_" diff --git a/openhcs/core/pipeline/compiler.py b/openhcs/core/pipeline/compiler.py index 9dfa0fe33d..b443f29425 100644 --- a/openhcs/core/pipeline/compiler.py +++ b/openhcs/core/pipeline/compiler.py @@ -93,7 +93,7 @@ ) from openhcs.core.source_bindings import ( CompiledSourceBindingPlan, - SourceProjectionRole, + PrimaryPlaneRole, ) from openhcs.core.steps.abstract import AbstractStep from openhcs.core.partition_filter import PartitionFilterProcessor @@ -1395,7 +1395,7 @@ def _validate_pipeline_start_payload_axis_requirement( for projection in ( workspace_projection.source_projections_by_virtual_path.values() ) - if projection.projection_role is SourceProjectionRole.PRIMARY_PLANE + if projection.projection_role is PrimaryPlaneRole ) if not selected: failures.append( diff --git a/openhcs/core/source_binding_selection.py b/openhcs/core/source_binding_selection.py index 77264b9db3..9b77c9281a 100644 --- a/openhcs/core/source_binding_selection.py +++ b/openhcs/core/source_binding_selection.py @@ -22,7 +22,10 @@ PairByMetadata, PairByOrder, SourceSetPairing, - SourceSetRole, + FromPipelineStart, + FromStepInput, + MatchedSource, + SourceBindingOrigin, ) from openhcs.core.source_image_provenance import ( SourceImageIdentity, @@ -536,7 +539,7 @@ def execution_anchor_bindings( return tuple( binding for binding in resolvable_bindings - if binding.source_set_role is SourceSetRole.MATCHED + if binding.source_set_role is MatchedSource ) @staticmethod @@ -1157,7 +1160,7 @@ def matched_bindings(self) -> tuple[NamedSourceBinding, ...]: return tuple( binding for binding in self.bindings - if binding.source_set_role is SourceSetRole.MATCHED + if binding.source_set_role is MatchedSource ) @classmethod @@ -1604,10 +1607,11 @@ def require_load_universe(self) -> SourceFileUniverse: class SourceUniverseRequest(metaclass=AutoRegisterMeta): """Plan-artifact request for source-binding runtime universe resolution.""" - __registry_key__ = "universe_request_kind" + __registry_key__ = "origin" __skip_if_no_key__ = True - universe_request_kind: ClassVar[str | None] = None + origin: ClassVar[type[SourceBindingOrigin] | None] = None + """The binding origin whose source universe this request resolves.""" context: "ProcessingContext" plan: "CompiledStepPlan" matching_files: tuple[str, ...] @@ -1760,7 +1764,7 @@ def admit_source_artifact_cohort( @classmethod def for_binding(cls, binding: NamedSourceBinding) -> type[SourceUniverseRequest]: """Select the already registered owner of a binding's origin.""" - return cls.__registry__[binding.origin.value] + return cls.__registry__[binding.origin] def runtime_universe_state(self) -> SourceUniverseRuntimeState: """Return cached source-universe state for this request.""" @@ -1906,7 +1910,7 @@ def axis_files(self) -> tuple[str, ...]: class StepInputSourceUniverseRequest(SourceUniverseRequest): """Request for the source universe represented by the current step input.""" - universe_request_kind = "step_input" + origin = FromStepInput @classmethod def source_artifact_payload( @@ -1941,7 +1945,7 @@ def contribute_runtime_state( class PipelineStartSourceUniverseRequest(SourceUniverseRequest): """Request for the source universe represented by the pipeline start.""" - universe_request_kind = "pipeline_start" + origin = FromPipelineStart def contribute_runtime_state( self, diff --git a/openhcs/core/source_binding_workspace.py b/openhcs/core/source_binding_workspace.py index 2a439a25c9..0a85852221 100644 --- a/openhcs/core/source_binding_workspace.py +++ b/openhcs/core/source_binding_workspace.py @@ -15,19 +15,18 @@ from openhcs.constants.constants import Backend from openhcs.core.runtime_array_values import array_geometry -from metaclass_registry.strategies import ( - EnumKeyedStrategyMixin, - NominalTypeKeyedStrategyMixin, -) +from metaclass_registry.strategies import NominalTypeKeyedStrategyMixin from openhcs.core.source_bindings import ( ComponentSelector, ImportedMetadataTable, NamedSourceBinding, PairByMetadata, PairByOrder, + BroadcastSource, + PrimaryPlaneRole, + SourceArtifactRole, SourceProjectionRole, SourceSetPairing, - SourceSetRole, ) from openhcs.core.config import SourceBindingsConfig from openhcs.core.source_path_identity import source_path_posix @@ -503,14 +502,16 @@ def _matched_source_sets( ) -class SourceBindingProjectionStrategy( - EnumKeyedStrategyMixin[SourceProjectionRole], - ABC, - metaclass=AutoRegisterMeta, -): +class SourceBindingProjectionStrategy(ABC, metaclass=AutoRegisterMeta): """Project one binding member through its declared projection role.""" - strategy_key: ClassVar[SourceProjectionRole | None] = None + __registry_key__ = "role" + __skip_if_no_key__ = True + __registry__: ClassVar[ + dict[type[SourceProjectionRole], type["SourceBindingProjectionStrategy"]] + ] = {} + + role: ClassVar[type[SourceProjectionRole] | None] = None @abstractmethod def projection( @@ -523,7 +524,7 @@ def projection( class PrimaryPlaneBindingProjection(SourceBindingProjectionStrategy): - strategy_key = SourceProjectionRole.PRIMARY_PLANE + role = PrimaryPlaneRole def projection( self, @@ -549,7 +550,7 @@ def projection( class SourceArtifactBindingProjection(SourceBindingProjectionStrategy): - strategy_key = SourceProjectionRole.SOURCE_ARTIFACT + role = SourceArtifactRole def projection( self, @@ -568,11 +569,8 @@ def projection( def _validate_source_binding_projection_registry() -> None: - strategy_keys = { - strategy_type.strategy_key - for strategy_type in SourceBindingProjectionStrategy.registered_strategy_types() - } - expected = set(SourceProjectionRole) + strategy_keys = set(SourceBindingProjectionStrategy.__registry__) + expected = set(SourceProjectionRole.__registry__.values()) if strategy_keys != expected: raise RuntimeError( "Source binding projection registry does not exactly cover " @@ -890,12 +888,12 @@ def materialize( plane_mappings = { path: projection.ref.to_workspace_mapping() for projection, path in projection_paths - if projection.projection_role is SourceProjectionRole.PRIMARY_PLANE + if projection.projection_role is PrimaryPlaneRole } artifact_mappings = { path: projection.ref.to_workspace_mapping() for projection, path in projection_paths - if projection.projection_role is SourceProjectionRole.SOURCE_ARTIFACT + if projection.projection_role is SourceArtifactRole } return SourceBindingWorkspaceMaterialization( source_root=source_root, @@ -1110,7 +1108,7 @@ def _expanded_binding_candidates( ) -> tuple[SourceCandidate, ...]: if candidate.declared_address is not None: return (candidate,) - if binding.source_set_role is SourceSetRole.BROADCAST: + if binding.source_set_role is BroadcastSource: return (candidate,) components = self.source_bindings.source_stack_components if not components: @@ -1195,7 +1193,9 @@ def _source_set_projections( projections: list[SourceProjection] = [] for source_set in source_sets: well = self._source_set_well(source_set) - projection_indexes = {role: 0 for role in SourceProjectionRole} + projection_indexes = { + role: 0 for role in SourceProjectionRole.__registry__.values() + } for binding in bindings: candidate = source_set.candidates_by_alias.get(binding.alias) if candidate is None: @@ -1224,9 +1224,9 @@ def _source_set_projections( for axis in AxisFamily.active().axes ) projections.append( - SourceBindingProjectionStrategy.for_enum_member( + SourceBindingProjectionStrategy.__registry__[ binding.projection_role - ).projection( + ]().projection( binding, replace( candidate, diff --git a/openhcs/core/source_bindings.py b/openhcs/core/source_bindings.py index a2a16273cf..2a3c310950 100644 --- a/openhcs/core/source_bindings.py +++ b/openhcs/core/source_bindings.py @@ -10,7 +10,6 @@ from abc import ABC, abstractmethod from collections.abc import Iterable, Mapping, Sequence from dataclasses import dataclass, field, replace -from enum import Enum from pathlib import Path, PureWindowsPath from types import MappingProxyType from typing import TYPE_CHECKING, Annotated, Any, ClassVar, Self, TypeVar @@ -27,7 +26,8 @@ ImageArtifactType, ) from openhcs.core.component_set import ComponentSet -from python_introspect import FieldLabel +from polystore.formats import get_format_from_extension +from python_introspect import FieldLabel, RegisteredClassChoices from openhcs.core.runtime_tabular_values import FieldSpec from openhcs.core.source_metadata import ( SourceMetadataFields, @@ -38,7 +38,7 @@ source_metadata_scalar, ) from openhcs.core.xdg_paths import get_openhcs_cache_dir -from openhcs.core.source_path_identity import source_paths_equal +from openhcs.core.source_path_identity import source_path_posix, source_paths_equal from openhcs.core.source_spatial_domain import SourceSpatialDomain from openhcs.core.axis_boundaries import AxisChoices from openhcs.core.axes import Axis, AxisFamily, GroupingDeclaration @@ -59,77 +59,341 @@ SourceBindingValue = TypeVar("SourceBindingValue") -class SourceBindingOrigin(Enum): - """Where a named binding should be resolved from.""" +class SourceBindingOrigin(ABC, metaclass=AutoRegisterMeta): + """Where a named binding is resolved from.""" - STEP_INPUT = "step_input" - PIPELINE_START = "pipeline_start" + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["SourceBindingOrigin"]]] = {} + + name: ClassVar[str | None] = None + """Spelling in forms and on the wire.""" + + +class FromStepInput(SourceBindingOrigin): + """The planes the prior step hands this step.""" + name = "step_input" -class SourceSetRole(Enum): - """How one source binding participates in source-set assembly.""" - MATCHED = "matched" - BROADCAST = "broadcast" +class FromPipelineStart(SourceBindingOrigin): + """The source planes as they were before the pipeline's first step.""" + name = "pipeline_start" + + +class SourceSetRole(ABC, metaclass=AutoRegisterMeta): + """How one source binding takes part in source-set assembly.""" + + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["SourceSetRole"]]] = {} -class SourceProjectionRole(Enum): + name: ClassVar[str | None] = None + """Spelling in forms and on the wire.""" + + +class MatchedSource(SourceSetRole): + """The alias determines which source sets exist and how many.""" + + name = "matched" + + +class BroadcastSource(SourceSetRole): + """The alias is appended to every assembled source set.""" + + name = "broadcast" + + +class SourceProjectionRole(ABC, metaclass=AutoRegisterMeta): """How one source binding is represented in a materialized workspace.""" - PRIMARY_PLANE = "primary_plane" - SOURCE_ARTIFACT = "source_artifact" + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["SourceProjectionRole"]]] = {} + + name: ClassVar[str | None] = None + """Spelling in forms and on the wire.""" + + @classmethod + def check_artifact_kind(cls, artifact_kind: type[ArtifactType]) -> None: + """Reject an artifact kind this projection cannot carry.""" + + +class PrimaryPlaneRole(SourceProjectionRole): + """The source becomes a primary image plane of the step's stack.""" + + name = "primary_plane" + + @classmethod + def check_artifact_kind(cls, artifact_kind: type[ArtifactType]) -> None: + if not issubclass(artifact_kind, ImageArtifactType): + raise ValueError( + "NamedSourceBinding primary-plane projections require an image " + f"artifact kind, got {artifact_kind.__name__}." + ) + + +class SourceArtifactRole(SourceProjectionRole): + """The source is handed to functions as a named artifact.""" + + name = "source_artifact" + + +class MetadataSource(ABC, metaclass=AutoRegisterMeta): + """Which text of a source path metadata extraction rules search.""" + + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["MetadataSource"]]] = {} + + name: ClassVar[str | None] = None + """Spelling in forms and on the wire.""" + + @staticmethod + @abstractmethod + def texts(file_path: str) -> tuple[str, ...]: + """The texts a rule's pattern is searched in, in order.""" + + +class FromFileName(MetadataSource): + name = "file_name" + + @staticmethod + def texts(file_path: str) -> tuple[str, ...]: + return (Path(file_path).name,) + + +class FromFolderName(MetadataSource): + name = "folder_name" + + @staticmethod + def texts(file_path: str) -> tuple[str, ...]: + parent = Path(file_path).parent + parent_path = source_path_posix(parent) + if parent_path == parent.name: + return (parent.name,) + return (parent.name, parent_path) + + +class SourceFilterSubject(ABC, metaclass=AutoRegisterMeta): + """Which part of a source path one filter clause inspects.""" + + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["SourceFilterSubject"]]] = {} + + name: ClassVar[str | None] = None + """Spelling in forms and on the wire.""" + + @staticmethod + @abstractmethod + def text_of(file_path: str) -> str: + """The text of ``file_path`` this subject names.""" + + +class SourceFileName(SourceFilterSubject): + name = "file" + + @staticmethod + def text_of(file_path: str) -> str: + return os.path.basename(file_path) + + +class SourceDirectory(SourceFilterSubject): + name = "directory" + + @staticmethod + def text_of(file_path: str) -> str: + return os.path.dirname(file_path) + + +class SourceExtension(SourceFilterSubject): + name = "extension" + + @staticmethod + def text_of(file_path: str) -> str: + return os.path.splitext(file_path)[1].lower() + + +class SourceFilterMatchType(ABC, metaclass=AutoRegisterMeta): + """How one source filter clause matches a source path.""" + + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["SourceFilterMatchType"]]] = {} + + name: ClassVar[str | None] = None + """Spelling in forms and on the wire.""" + + requires_value: ClassVar[bool] + """The clause carries a comparison operand.""" + + @classmethod + @abstractmethod + def matches(cls, file_path: str, text: str, value: str | None) -> bool: + """Whether ``file_path``, whose subject text is ``text``, matches.""" + + +class TextMatchType(SourceFilterMatchType): + """Compares the subject text with the clause's value.""" + + requires_value = True + + @staticmethod + @abstractmethod + def compare(text: str, value: str) -> bool: + """Whether ``text`` satisfies this comparison with ``value``.""" + + @classmethod + def matches(cls, file_path: str, text: str, value: str | None) -> bool: + if value is None: + raise ValueError(f"{cls.name} source filters require a value.") + return cls.compare(text, value) + + @classmethod + def negation(cls) -> type["TextMatchType"]: + """The comparison that holds exactly when this one does not.""" + + (negation,) = ( + member + for member in SourceFilterMatchType.__registry__.values() + if issubclass(member, NegatedTextMatch) and member.negates is cls + ) + return negation + + +class NegatedTextMatch(TextMatchType): + """Holds exactly when the comparison it negates does not.""" + + negates: ClassVar[type[TextMatchType]] + + @classmethod + def compare(cls, text: str, value: str) -> bool: + return not cls.negates.compare(text, value) + + +class TextContains(TextMatchType): + name = "contains" + + @staticmethod + def compare(text: str, value: str) -> bool: + return value in text + + +class TextContainsRegex(TextMatchType): + name = "contains_regex" + + @staticmethod + def compare(text: str, value: str) -> bool: + return re.search(value, text) is not None + + +class TextDoesNotContain(NegatedTextMatch): + name = "does_not_contain" + negates = TextContains + + +class TextDoesNotContainRegex(NegatedTextMatch): + name = "does_not_contain_regex" + negates = TextContainsRegex + + +class TextEquals(TextMatchType): + name = "equals" + + @staticmethod + def compare(text: str, value: str) -> bool: + return text == value + + +class TextDoesNotEqual(NegatedTextMatch): + name = "does_not_equal" + negates = TextEquals + +class TextStartsWith(TextMatchType): + name = "starts_with" -class MetadataSource(Enum): - """Where metadata extraction rules read source text from.""" + @staticmethod + def compare(text: str, value: str) -> bool: + return text.startswith(value) + + +class TextDoesNotStartWith(NegatedTextMatch): + name = "does_not_start_with" + negates = TextStartsWith - FILE_NAME = "file_name" - FOLDER_NAME = "folder_name" + +class TextEndsWith(TextMatchType): + name = "ends_with" + + @staticmethod + def compare(text: str, value: str) -> bool: + return text.endswith(value) -class SourceFilterSubject(Enum): - """Which part of a source path one filter clause targets.""" +class TextDoesNotEndWith(NegatedTextMatch): + name = "does_not_end_with" + negates = TextEndsWith - FILE = "file" - DIRECTORY = "directory" - EXTENSION = "extension" +class PathMatchType(SourceFilterMatchType): + """A predicate of the whole path; the clause carries no value.""" -class SourceFilterMatchType(Enum): - """How one source filter clause matches its target text.""" + requires_value = False + + @staticmethod + @abstractmethod + def holds_for(file_path: str) -> bool: + """Whether ``file_path`` satisfies this predicate.""" + + @classmethod + def matches(cls, file_path: str, text: str, value: str | None) -> bool: + if value is not None: + raise ValueError(f"{cls.name} source filters do not accept a value.") + return cls.holds_for(file_path) - def __new__(cls, value: str, requires_value: bool = True): - obj = object.__new__(cls) - obj._value_ = value - obj.requires_value = requires_value - return obj - CONTAINS = ("contains", True) - CONTAINS_REGEX = ("contains_regex", True) - DOES_NOT_CONTAIN = ("does_not_contain", True) - DOES_NOT_CONTAIN_REGEX = ("does_not_contain_regex", True) - EQUALS = ("equals", True) - DOES_NOT_EQUAL = ("does_not_equal", True) - STARTS_WITH = ("starts_with", True) - DOES_NOT_START_WITH = ("does_not_start_with", True) - ENDS_WITH = ("ends_with", True) - DOES_NOT_END_WITH = ("does_not_end_with", True) - IS_IMAGE = ("is_image", False) - IS_TIF = ("is_tif", False) +class PathIsImage(PathMatchType): + """The path's extension names a loadable pixel format.""" + + name = "is_image" + + @staticmethod + def holds_for(file_path: str) -> bool: + suffix = os.path.splitext(file_path)[1].lower() + try: + file_format = get_format_from_extension(suffix) + except ValueError: + return False + return file_format.is_pixel_payload + + +class PathIsTif(PathMatchType): + name = "is_tif" + + @staticmethod + def holds_for(file_path: str) -> bool: + return os.path.splitext(file_path)[1].lower() in {".tif", ".tiff"} @dataclass(frozen=True, slots=True) class SourceFilterClause: """Typed filter clause applied before metadata extraction.""" - subject: SourceFilterSubject + subject: Annotated[ + type[SourceFilterSubject], RegisteredClassChoices(SourceFilterSubject) + ] """Part of the source path inspected by this clause.""" - match_type: SourceFilterMatchType + match_type: Annotated[ + type[SourceFilterMatchType], RegisteredClassChoices(SourceFilterMatchType) + ] """Comparison applied to the selected path text.""" value: str | None = None - """Comparison operand; omitted for predicate matches such as ``IS_IMAGE``.""" + """Comparison operand; omitted for path predicates such as ``PathIsImage``.""" any_group: int | None = None """Optional OR-group identifier. @@ -139,31 +403,26 @@ class SourceFilterClause: """ def __post_init__(self) -> None: - object.__setattr__( - self, - "subject", - SourceFilterSubject( - self.subject, - ), - ) - match_type = SourceFilterMatchType( - self.match_type, - ) - object.__setattr__(self, "match_type", match_type) if self.any_group is not None: if not isinstance(self.any_group, int) or self.any_group < 0: raise ValueError( "SourceFilterClause.any_group must be a nonnegative integer." ) - normalized_value = None if self.value is None else str(self.value) - if not match_type.requires_value: + if not self.match_type.requires_value: object.__setattr__(self, "value", None) return - if normalized_value is None: + if self.value is None: raise ValueError( - "SourceFilterClause.value is required unless match_type is IS_IMAGE." + f"SourceFilterClause.value is required for {self.match_type.name}." ) - object.__setattr__(self, "value", normalized_value) + object.__setattr__(self, "value", str(self.value)) + + def matches(self, file_path: str) -> bool: + """Whether one source path satisfies this clause.""" + + return self.match_type.matches( + file_path, self.subject.text_of(file_path), self.value + ) def _normalized_optional_source_text(value: str | None) -> str | None: @@ -439,7 +698,7 @@ def normalize_source_binding_values( class MetadataExtractionRule: """Regex-backed metadata extraction rule for source binding resolution.""" - source: MetadataSource + source: Annotated[type[MetadataSource], RegisteredClassChoices(MetadataSource)] """Source-path text searched by the regular expression.""" pattern: str @@ -449,13 +708,6 @@ class MetadataExtractionRule: """Optional path clauses limiting which sources this extraction rule processes.""" def __post_init__(self) -> None: - object.__setattr__( - self, - "source", - MetadataSource( - self.source, - ), - ) if not self.pattern: raise ValueError("MetadataExtractionRule.pattern cannot be empty.") compiled_pattern = re.compile(str(self.pattern)) @@ -771,7 +1023,9 @@ class SourceAssignmentBase(metaclass=AutoRegisterMeta): selector: SourceSelector = SourceSelector() """Source-plane selection criteria for this named input.""" - origin: SourceBindingOrigin = SourceBindingOrigin.STEP_INPUT + origin: Annotated[ + type[SourceBindingOrigin], RegisteredClassChoices(SourceBindingOrigin) + ] = FromStepInput """Whether selection starts from the prior step input or pipeline-start sources.""" component_identity: Annotated[ @@ -812,13 +1066,6 @@ def __post_init__(self) -> None: "component_identity", tuple(dict.fromkeys(component_identity)), ) - object.__setattr__( - self, - "origin", - SourceBindingOrigin( - self.origin, - ), - ) @property def artifact_kind(self) -> ArtifactType: @@ -922,10 +1169,14 @@ class NamedSourceBinding(SourceAssignmentBase): required: bool = True """Whether compilation fails when this alias cannot supply an artifact.""" - source_set_role: SourceSetRole = SourceSetRole.MATCHED + source_set_role: Annotated[ + type[SourceSetRole], RegisteredClassChoices(SourceSetRole) + ] = MatchedSource """Whether this alias determines source-set membership or broadcasts to every set.""" - projection_role: SourceProjectionRole = SourceProjectionRole.PRIMARY_PLANE + projection_role: Annotated[ + type[SourceProjectionRole], RegisteredClassChoices(SourceProjectionRole) + ] = PrimaryPlaneRole """Whether the resolved source becomes a primary image plane or source artifact.""" explicit_source: ImagePlaneSource | None = None @@ -950,19 +1201,7 @@ def __post_init__(self) -> None: "artifact_kind", ArtifactType.coerce(self.artifact_kind), ) - source_set_role = SourceSetRole( - self.source_set_role, - ) - projection_role = SourceProjectionRole( - self.projection_role, - ) - if projection_role is SourceProjectionRole.PRIMARY_PLANE and not issubclass( - self.artifact_kind, ImageArtifactType - ): - raise ValueError( - "NamedSourceBinding PRIMARY_PLANE projections require an image " - f"artifact kind, got {self.artifact_kind.__name__}." - ) + self.projection_role.check_artifact_kind(self.artifact_kind) if self.explicit_source is not None and not isinstance( self.explicit_source, ImagePlaneSource, @@ -970,8 +1209,6 @@ def __post_init__(self) -> None: raise TypeError( "NamedSourceBinding.explicit_source must be ImagePlaneSource or None." ) - object.__setattr__(self, "source_set_role", source_set_role) - object.__setattr__(self, "projection_role", projection_role) object.__setattr__(self, "required", bool(self.required)) object.__setattr__(self, "load_as_monochrome", bool(self.load_as_monochrome)) object.__setattr__(self, "load_as_mask", bool(self.load_as_mask)) @@ -1165,11 +1402,11 @@ def project_step_input_payload(self, payload: RuntimeArrayData) -> RuntimeArrayD metadata = payload.metadata provenance = metadata.source_provenance if not provenance.has_values: - raise ValueError(f"STEP_INPUT binding {self.alias!r} requires source provenance.") + raise ValueError(f"Step-input binding {self.alias!r} requires source provenance.") if not provenance.source_plane_count: if not self.selector.matches_provenance(provenance): - raise ValueError(f"STEP_INPUT binding {self.alias!r} selects no current planes.") + raise ValueError(f"Step-input binding {self.alias!r} selects no current planes.") selected = payload else: named_planes = provenance.source_plane_selection(self.alias) @@ -1182,7 +1419,7 @@ def project_step_input_payload(self, payload: RuntimeArrayData) -> RuntimeArrayD if self.selector.matches_provenance(provenance.for_source_plane(index)) ) if not selection: - raise ValueError(f"STEP_INPUT binding {self.alias!r} selects no current planes.") + raise ValueError(f"Step-input binding {self.alias!r} selects no current planes.") selected = metadata.project_source_planes(payload, selection) return self.apply_loaded_payload(selected, source_context=None) @@ -1190,8 +1427,8 @@ def project_step_input_payload(self, payload: RuntimeArrayData) -> RuntimeArrayD def requires_current_pixels(self) -> bool: """Whether this source projects the actual current main-flow pixels.""" return ( - self.origin is SourceBindingOrigin.STEP_INPUT - and self.projection_role is SourceProjectionRole.PRIMARY_PLANE + self.origin is FromStepInput + and self.projection_role is PrimaryPlaneRole ) @staticmethod @@ -1331,7 +1568,7 @@ def requires_step_input_component_stack( components: tuple[type[Axis], ...], ) -> bool: """Whether resolving this binding needs component-varying step input.""" - if self.origin is not SourceBindingOrigin.STEP_INPUT: + if self.origin is not FromStepInput: return False if not components: return False @@ -1362,7 +1599,7 @@ def matched_source_bindings(self) -> tuple[NamedSourceBinding, ...]: return tuple( binding for binding in self.binding_declarations - if binding.source_set_role is SourceSetRole.MATCHED + if binding.source_set_role is MatchedSource ) @property @@ -1372,7 +1609,7 @@ def broadcast_source_bindings(self) -> tuple[NamedSourceBinding, ...]: return tuple( binding for binding in self.binding_declarations - if binding.source_set_role is SourceSetRole.BROADCAST + if binding.source_set_role is BroadcastSource ) @property @@ -1382,7 +1619,7 @@ def primary_plane_bindings(self) -> tuple[NamedSourceBinding, ...]: return tuple( binding for binding in self.binding_declarations - if binding.projection_role is SourceProjectionRole.PRIMARY_PLANE + if binding.projection_role is PrimaryPlaneRole ) def binding_for_alias( @@ -1447,7 +1684,7 @@ def requires_step_input_component_stack( def requires_pipeline_start_resolution(self) -> bool: """Whether any declaration resolves from the pipeline-start universe.""" return any( - binding.origin is SourceBindingOrigin.PIPELINE_START + binding.origin is FromPipelineStart for binding in self.binding_declarations ) @@ -1458,8 +1695,8 @@ def requires_pipeline_start_source_set(self) -> bool: sum( 1 for binding in self.binding_declarations - if binding.origin is SourceBindingOrigin.PIPELINE_START - and binding.source_set_role is SourceSetRole.MATCHED + if binding.origin is FromPipelineStart + and binding.source_set_role is MatchedSource ) > 1 ) @@ -1468,7 +1705,7 @@ def requires_pipeline_start_source_set(self) -> bool: def requires_step_input_selector_resolution(self) -> bool: """Whether step-input declarations need selector-aware matching.""" return any( - binding.origin is SourceBindingOrigin.STEP_INPUT + binding.origin is FromStepInput and binding.requires_selector_resolution for binding in self.binding_declarations ) @@ -1975,7 +2212,7 @@ def requires_step_input_selector_resolution(self) -> bool: """Whether any step-input binding needs selector-aware source matching.""" return any( - binding.origin is SourceBindingOrigin.STEP_INPUT + binding.origin is FromStepInput and binding.requires_selector_resolution for binding in self.bindings ) @@ -2015,7 +2252,7 @@ def from_source_binding_plan( uses_pipeline_start_binding_origin = ( source_binding_plan.has_primary_content and any( - binding.origin is SourceBindingOrigin.PIPELINE_START + binding.origin is FromPipelineStart for binding in source_binding_plan.bindings ) ) diff --git a/openhcs/core/source_matching.py b/openhcs/core/source_matching.py index bbc4393981..09a98947c1 100644 --- a/openhcs/core/source_matching.py +++ b/openhcs/core/source_matching.py @@ -2,26 +2,20 @@ from __future__ import annotations -import os import re from abc import ABC, abstractmethod from dataclasses import dataclass from enum import Enum from functools import lru_cache -from pathlib import Path -from typing import Callable, ClassVar, Mapping, Sequence, TYPE_CHECKING, TypeAlias +from typing import ClassVar, Mapping, Sequence, TYPE_CHECKING, TypeAlias from metaclass_registry import AutoRegisterMeta -from polystore.formats import get_format_from_extension from openhcs.core.component_set import ComponentSet from openhcs.core.source_bindings import ( MetadataExtractionRule, - MetadataSource, SourceBindingDeclarationsMixin, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, ) from openhcs.core.source_metadata import ( DECLARED_SOURCE_METADATA_FIELD, @@ -38,240 +32,13 @@ source_metadata_dict, source_metadata_scalar, ) -from openhcs.core.source_path_identity import source_path_identity_key, source_path_posix +from openhcs.core.source_path_identity import source_path_identity_key from openhcs.core.axes import Axis, AxisFamily if TYPE_CHECKING: from openhcs.core.config import PipelineConfig -@dataclass(frozen=True, slots=True) -class SourceFilterMatchRequest: - """Typed request for one source-filter match evaluation.""" - - file_path: str - clause: SourceFilterClause - target: str - - -def _string_contains(target: str, value: str) -> bool: - return value in target - - -def _string_does_not_contain(target: str, value: str) -> bool: - return value not in target - - -def _string_contains_regex(target: str, value: str) -> bool: - return re.search(value, target) is not None - - -def _string_does_not_contain_regex(target: str, value: str) -> bool: - return re.search(value, target) is None - - -def _string_equals(target: str, value: str) -> bool: - return target == value - - -def _string_does_not_equal(target: str, value: str) -> bool: - return target != value - - -def _string_starts_with(target: str, value: str) -> bool: - return target.startswith(value) - - -def _string_does_not_start_with(target: str, value: str) -> bool: - return not target.startswith(value) - - -def _string_ends_with(target: str, value: str) -> bool: - return target.endswith(value) - - -def _string_does_not_end_with(target: str, value: str) -> bool: - return not target.endswith(value) - - -def is_image_path(file_path: str) -> bool: - """Return whether the path extension is a loadable image source.""" - suffix = os.path.splitext(file_path)[1].lower() - try: - file_format = get_format_from_extension(suffix) - except ValueError: - return False - return file_format.is_pixel_payload - - -def is_tif_path(file_path: str) -> bool: - """Return whether the path extension is a TIFF source.""" - - return os.path.splitext(file_path)[1].lower() in {".tif", ".tiff"} - - -class SourceFilterMatcher(ABC, metaclass=AutoRegisterMeta): - """Nominal family for typed source-filter match behavior.""" - - __registry_key__ = "match_type_key" - __skip_if_no_key__ = True - match_type: ClassVar[SourceFilterMatchType | None] = None - match_type_key: ClassVar[str | None] = None - - @classmethod - def for_match_type( - cls, - match_type: SourceFilterMatchType, - ) -> "SourceFilterMatcher": - return cls.__registry__[match_type.value]() - - @abstractmethod - def matches(self, request: SourceFilterMatchRequest) -> bool: - """Return whether one file path satisfies the filter clause.""" - - -class ValuePredicateSourceFilterMatcher(SourceFilterMatcher): - """Declarative matcher for source-filter clauses with scalar values.""" - - value_predicate: ClassVar[Callable[[str, str], bool]] - - def matches(self, request: SourceFilterMatchRequest) -> bool: - return type(self).value_predicate( - request.target, - _require_filter_value(request.clause), - ) - - -class PathPredicateSourceFilterMatcher(SourceFilterMatcher): - """Declarative matcher for source-filter clauses that inspect the path.""" - - path_predicate: ClassVar[Callable[[str], bool]] - - def matches(self, request: SourceFilterMatchRequest) -> bool: - if request.clause.value is not None: - raise ValueError( - f"{request.clause.match_type.value} source filters do not accept " - "a scalar clause value." - ) - return type(self).path_predicate(request.file_path) - - -class ContainsSourceFilterMatcher(ValuePredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.CONTAINS - match_type_key = SourceFilterMatchType.CONTAINS.value - value_predicate = staticmethod(_string_contains) - - -class DoesNotContainSourceFilterMatcher(ValuePredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.DOES_NOT_CONTAIN - match_type_key = SourceFilterMatchType.DOES_NOT_CONTAIN.value - value_predicate = staticmethod(_string_does_not_contain) - - -class ContainsRegexSourceFilterMatcher(ValuePredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.CONTAINS_REGEX - match_type_key = SourceFilterMatchType.CONTAINS_REGEX.value - value_predicate = staticmethod(_string_contains_regex) - - -class DoesNotContainRegexSourceFilterMatcher(ValuePredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.DOES_NOT_CONTAIN_REGEX - match_type_key = SourceFilterMatchType.DOES_NOT_CONTAIN_REGEX.value - value_predicate = staticmethod(_string_does_not_contain_regex) - - -class EqualsSourceFilterMatcher(ValuePredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.EQUALS - match_type_key = SourceFilterMatchType.EQUALS.value - value_predicate = staticmethod(_string_equals) - - -class DoesNotEqualSourceFilterMatcher(ValuePredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.DOES_NOT_EQUAL - match_type_key = SourceFilterMatchType.DOES_NOT_EQUAL.value - value_predicate = staticmethod(_string_does_not_equal) - - -class StartsWithSourceFilterMatcher(ValuePredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.STARTS_WITH - match_type_key = SourceFilterMatchType.STARTS_WITH.value - value_predicate = staticmethod(_string_starts_with) - - -class DoesNotStartWithSourceFilterMatcher(ValuePredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.DOES_NOT_START_WITH - match_type_key = SourceFilterMatchType.DOES_NOT_START_WITH.value - value_predicate = staticmethod(_string_does_not_start_with) - - -class EndsWithSourceFilterMatcher(ValuePredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.ENDS_WITH - match_type_key = SourceFilterMatchType.ENDS_WITH.value - value_predicate = staticmethod(_string_ends_with) - - -class DoesNotEndWithSourceFilterMatcher(ValuePredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.DOES_NOT_END_WITH - match_type_key = SourceFilterMatchType.DOES_NOT_END_WITH.value - value_predicate = staticmethod(_string_does_not_end_with) - - -class IsImageSourceFilterMatcher(PathPredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.IS_IMAGE - match_type_key = SourceFilterMatchType.IS_IMAGE.value - path_predicate = staticmethod(is_image_path) - - -class IsTifSourceFilterMatcher(PathPredicateSourceFilterMatcher): - match_type = SourceFilterMatchType.IS_TIF - match_type_key = SourceFilterMatchType.IS_TIF.value - path_predicate = staticmethod(is_tif_path) - - -class SourceFilterTargetResolver(ABC, metaclass=AutoRegisterMeta): - """Nominal family for source-filter target text resolution.""" - - __registry_key__ = "subject_key" - __skip_if_no_key__ = True - subject: ClassVar[SourceFilterSubject | None] = None - subject_key: ClassVar[str | None] = None - - @classmethod - def for_subject( - cls, - subject: SourceFilterSubject, - ) -> "SourceFilterTargetResolver": - return cls.__registry__[subject.value]() - - @abstractmethod - def resolve_text(self, file_path: str) -> str: - """Return the subject-specific text inspected by one filter clause.""" - - -class FileSourceFilterTargetResolver(SourceFilterTargetResolver): - subject = SourceFilterSubject.FILE - subject_key = SourceFilterSubject.FILE.value - - def resolve_text(self, file_path: str) -> str: - return os.path.basename(file_path) - - -class DirectorySourceFilterTargetResolver(SourceFilterTargetResolver): - subject = SourceFilterSubject.DIRECTORY - subject_key = SourceFilterSubject.DIRECTORY.value - - def resolve_text(self, file_path: str) -> str: - return os.path.dirname(file_path) - - -class ExtensionSourceFilterTargetResolver(SourceFilterTargetResolver): - subject = SourceFilterSubject.EXTENSION - subject_key = SourceFilterSubject.EXTENSION.value - - def resolve_text(self, file_path: str) -> str: - return os.path.splitext(file_path)[1].lower() - - def metadata_from_rules( file_path: str, metadata_rules: tuple[MetadataExtractionRule, ...], @@ -283,10 +50,10 @@ def metadata_from_rules( extracted: dict[str, str] = {} filter_candidate = file_path if filter_path is None else filter_path for rule in metadata_rules: - if not rule_filters_match(filter_candidate, rule.filters): + if not source_filters_match(filter_candidate, rule.filters): continue match = None - for target in metadata_source_texts(file_path, rule.source): + for target in rule.source.texts(file_path): match = re.search(rule.pattern, target) if match is not None: break @@ -310,31 +77,6 @@ def metadata_from_rules( ) -def metadata_source_texts( - file_path: str, - source: MetadataSource, -) -> tuple[str, ...]: - """Return ordered path texts inspected by one metadata extraction rule.""" - - path = Path(file_path) - if source is MetadataSource.FOLDER_NAME: - parent_path = source_path_posix(path.parent) - parent_name = path.parent.name - if parent_path == parent_name: - return (parent_name,) - return (parent_name, parent_path) - return (path.name,) - - -def rule_filters_match( - file_path: str, - filters: tuple[SourceFilterClause, ...], -) -> bool: - """Return whether one source path satisfies metadata-rule filters.""" - - return source_filters_match(file_path, filters) - - def source_filters_match( file_path: str, filters: tuple[SourceFilterClause, ...], @@ -351,7 +93,7 @@ def _source_filters_match_cached( ) -> bool: any_group_matches: dict[int, bool] = {} for clause in filters: - matches = filter_clause_matches(file_path, clause) + matches = clause.matches(file_path) if clause.any_group is None: if not matches: return False @@ -362,24 +104,6 @@ def _source_filters_match_cached( return all(any_group_matches.values()) -def filter_clause_matches( - file_path: str, - clause: SourceFilterClause, -) -> bool: - """Return whether one source path satisfies one source-filter clause.""" - - target = SourceFilterTargetResolver.for_subject(clause.subject).resolve_text( - file_path - ) - return SourceFilterMatcher.for_match_type(clause.match_type).matches( - SourceFilterMatchRequest( - file_path=file_path, - clause=clause, - target=target, - ) - ) - - class SourceImageSetComponentRole(Enum): """Semantic role of a source metadata component within an image set.""" @@ -943,11 +667,3 @@ def constraint_matches_metadata( and source_metadata_values_equal(str(metadata_value), value) for metadata_value in SourceMetadataFields.scalar_values(metadata) ) - - -def _require_filter_value(clause: SourceFilterClause) -> str: - if clause.value is None: - raise ValueError( - "SourceFilterClause.value must be set unless match_type is IS_IMAGE." - ) - return clause.value diff --git a/openhcs/core/source_projection.py b/openhcs/core/source_projection.py index e6868d9482..4da55413dd 100644 --- a/openhcs/core/source_projection.py +++ b/openhcs/core/source_projection.py @@ -21,6 +21,8 @@ from openhcs.core.source_bindings import ( SOURCE_BINDING_ALIAS_METADATA_FIELD, NamedSourceBinding, + PrimaryPlaneRole, + SourceArtifactRole, SourceProjectionRole, ) from openhcs.core.source_matching import ( @@ -597,7 +599,7 @@ def _record_identity( class SourceProjection: """Nominal root for typed source projections.""" - projection_role: ClassVar[SourceProjectionRole] + projection_role: ClassVar[type[SourceProjectionRole]] address: OpenHCSPlaneAddress | None ref: SourcePixelRef source_alias: str | None @@ -725,7 +727,7 @@ def declared_optional_payload_field(cls, payload_type: type) -> str: class SourcePlaneProjection(SourceProjection): """One canonical primary plane mapped to one source pixel reference.""" - projection_role: ClassVar[SourceProjectionRole] = SourceProjectionRole.PRIMARY_PLANE + projection_role: ClassVar[type[SourceProjectionRole]] = PrimaryPlaneRole artifact_kind: ClassVar[type[ArtifactType]] = ImageArtifactType address: OpenHCSPlaneAddress @@ -766,9 +768,7 @@ def payload_composition_alias(self) -> str | None: class SourceArtifactProjection(SourceProjection): """One typed source artifact attached to a source-set execution address.""" - projection_role: ClassVar[SourceProjectionRole] = ( - SourceProjectionRole.SOURCE_ARTIFACT - ) + projection_role: ClassVar[type[SourceProjectionRole]] = SourceArtifactRole address: OpenHCSPlaneAddress | None ref: SourcePixelRef @@ -968,7 +968,7 @@ def plane_projections(self) -> tuple[SourcePlaneProjection, ...]: tuple( projection for projection in self.projections - if projection.projection_role is SourceProjectionRole.PRIMARY_PLANE + if projection.projection_role is PrimaryPlaneRole ), ) @@ -981,7 +981,7 @@ def artifact_projections(self) -> tuple[SourceArtifactProjection, ...]: tuple( projection for projection in self.projections - if projection.projection_role is SourceProjectionRole.SOURCE_ARTIFACT + if projection.projection_role is SourceArtifactRole ), ) @@ -1358,7 +1358,7 @@ def _source_projection_payload( "virtual_path": path, "address": projection.serialized_address(), "ref": projection.ref.to_workspace_mapping(), - "projection_role": projection.projection_role.value, + "projection_role": projection.projection_role.name, } if projection.source_alias is not None: payload["source_alias"] = projection.source_alias @@ -1409,7 +1409,7 @@ def _normalize_projection(projection: SourceProjection) -> None: ): raise TypeError("Source projection address must be OpenHCSPlaneAddress.") if projection.address is None and ( - projection.projection_role is not SourceProjectionRole.SOURCE_ARTIFACT + projection.projection_role is not SourceArtifactRole or projection.execution_scope is None ): raise TypeError( diff --git a/openhcs/core/source_workspace_projection.py b/openhcs/core/source_workspace_projection.py index 5f67a165c1..c22d455f3a 100644 --- a/openhcs/core/source_workspace_projection.py +++ b/openhcs/core/source_workspace_projection.py @@ -369,17 +369,12 @@ def pipeline_start_files(self, *, axis_id: str | None = None) -> tuple[str, ...] def files_for_projection_role( self, - projection_role: SourceProjectionRole, + projection_role: type[SourceProjectionRole], *, axis_id: str | None = None, ) -> tuple[str, ...]: """Return source files belonging to one exact declared projection role.""" - role = ( - projection_role - if isinstance(projection_role, SourceProjectionRole) - else SourceProjectionRole(projection_role) - ) return tuple( self._loadable_virtual_path(virtual_path) for virtual_path in self.relative_virtual_paths() @@ -390,7 +385,7 @@ def files_for_projection_role( self._loadable_virtual_path(virtual_path), ) ).projection_role - is role + is projection_role ) def source_occurrences_for_binding( diff --git a/openhcs/core/steps/function_runtime.py b/openhcs/core/steps/function_runtime.py index 39008a42ad..f5b26f25f1 100644 --- a/openhcs/core/steps/function_runtime.py +++ b/openhcs/core/steps/function_runtime.py @@ -105,9 +105,9 @@ ) from openhcs.core.source_bindings import ( CompiledSourceBindingPlan, - SourceBindingOrigin, + FromPipelineStart, SOURCE_BINDING_ALIAS_METADATA_FIELD, - SourceProjectionRole, + PrimaryPlaneRole, ) from openhcs.core.runtime_image_values import ( ImagePayload, @@ -792,7 +792,7 @@ def _filter_matching_files_for_source_bindings( bindings = tuple( binding for binding in source_binding_plan.bindings - if binding.projection_role is SourceProjectionRole.PRIMARY_PLANE + if binding.projection_role is PrimaryPlaneRole ) if not bindings: return matching_files @@ -1681,7 +1681,7 @@ def loaded_source_artifact_payload( or self.invocation.key.position != 0 or self.group_data.execution_plan.main_input_dependency.kind is not StepInputDependencyKind.PIPELINE_START - or binding.origin is not SourceBindingOrigin.PIPELINE_START + or binding.origin is not FromPipelineStart or request.context is not self.group_data.context ): return None diff --git a/openhcs/core/virtual_workspace_metadata.py b/openhcs/core/virtual_workspace_metadata.py index 21cd8b65cd..35cf72df25 100644 --- a/openhcs/core/virtual_workspace_metadata.py +++ b/openhcs/core/virtual_workspace_metadata.py @@ -14,11 +14,12 @@ from polystore.atomic import FileLockError, atomic_update_json from polystore.metadata_writer import MetadataConfig from polystore.virtual_workspace import SourcePixelRef +from python_introspect import registered_class_from_key from openhcs.core.artifacts import ArtifactType from openhcs.core.component_group_scope import RuntimeExecutionAxisScope from openhcs.core.runtime_image_values import ImagePayloadMetadata -from openhcs.core.source_bindings import SourceProjectionRole +from openhcs.core.source_bindings import PrimaryPlaneRole, SourceProjectionRole from openhcs.core.source_metadata import ( DurableSourceMetadata, SourceMetadataMapping, @@ -659,8 +660,8 @@ def _projection_record( ) virtual_path = cls._required_text(record, "virtual_path") try: - projection_role = SourceProjectionRole( - cls._required_text(record, "projection_role") + projection_role = registered_class_from_key( + SourceProjectionRole, cls._required_text(record, "projection_role") ) except ValueError as exc: raise RuntimeError( @@ -680,7 +681,7 @@ def _projection_record( "virtual_workspace source_projection address must be a mapping or " "null." ) - if projection_role is SourceProjectionRole.PRIMARY_PLANE and address is None: + if projection_role is PrimaryPlaneRole and address is None: raise RuntimeError( "Primary source projections require a complete plane address." ) @@ -696,7 +697,7 @@ def _projection_record( None if image_metadata_value is None else ImagePayloadMetadata.from_mapping(image_metadata_value) ) - if projection_role is SourceProjectionRole.PRIMARY_PLANE: + if projection_role is PrimaryPlaneRole: projection: SourceProjection = SourcePlaneProjection( address=address, ref=ref, diff --git a/openhcs/processing/backends/cellprofiler/infrastructure.py b/openhcs/processing/backends/cellprofiler/infrastructure.py index 9e25b88a37..91d1dd5018 100644 --- a/openhcs/processing/backends/cellprofiler/infrastructure.py +++ b/openhcs/processing/backends/cellprofiler/infrastructure.py @@ -11,6 +11,8 @@ from pathlib import Path from typing import Callable, ClassVar +from python_introspect import registered_class_from_key + from openhcs.core.artifacts import ( ArtifactType, ImageArtifactType, @@ -24,19 +26,37 @@ MetadataExtractionRule, MetadataSelector, MetadataSource, + FromFileName, + FromFolderName, NamedSourceBinding, SourceBindingMatchDimension, SourceBindingMatchField, PairByMetadata, PairByOrder, SourceSetPairing, + BroadcastSource, + FromPipelineStart, + MatchedSource, + PathIsImage, + PathIsTif, + PrimaryPlaneRole, + SourceArtifactRole, SourceBindingOrigin, + SourceExtension, + SourceFileName, SourceFilterClause, SourceFilterMatchType, SourceFilterSubject, SourceProjectionRole, SourceSelector, SourceSetRole, + TextContains, + TextContainsRegex, + TextDoesNotContain, + TextEquals, + TextEndsWith, + TextMatchType, + TextStartsWith, ) from openhcs.core.config import SourceBindingsConfig from openhcs.core.source_matching import source_metadata_component @@ -83,88 +103,32 @@ _SOURCE_FILTER_TOKEN_PATTERN = re.compile(r'\s*(\(|\)|"(?:\\.|[^"\\])*"|[^\s()]+)') -class CellProfilerSourceFilterSubject(Enum): - """CellProfiler filter subject translated to one generic subject.""" - - def __new__(cls, literal: str, subject: SourceFilterSubject): - member = object.__new__(cls) - member._value_ = literal - member.subject = subject - return member - - FILE = ("file", SourceFilterSubject.FILE) - DIRECTORY = ("directory", SourceFilterSubject.DIRECTORY) - EXTENSION = ("extension", SourceFilterSubject.EXTENSION) - - @classmethod - def from_literal(cls, value: str) -> "CellProfilerSourceFilterSubject": - normalized = value.strip().casefold() - for member in cls: - if member.value == normalized: - return member - raise ValueError(f"Unsupported source filter subject: {value!r}.") - - -class CellProfilerSourceFilterOperator(Enum): - """CellProfiler filter operator translated to generic positive/negative forms.""" - - def __new__( - cls, - literal: str, - match_type: SourceFilterMatchType, - negated_match_type: SourceFilterMatchType | None, - ): - member = object.__new__(cls) - member._value_ = literal - member.match_type = match_type - member.negated_match_type = negated_match_type - return member - - CONTAINS = ( - "contain", - SourceFilterMatchType.CONTAINS, - SourceFilterMatchType.DOES_NOT_CONTAIN, - ) - CONTAINS_REGEX = ( - "containregexp", - SourceFilterMatchType.CONTAINS_REGEX, - SourceFilterMatchType.DOES_NOT_CONTAIN_REGEX, - ) - EQUALS = ( - "eq", - SourceFilterMatchType.EQUALS, - SourceFilterMatchType.DOES_NOT_EQUAL, - ) - STARTS_WITH = ( - "startwith", - SourceFilterMatchType.STARTS_WITH, - SourceFilterMatchType.DOES_NOT_START_WITH, - ) - ENDS_WITH = ( - "endwith", - SourceFilterMatchType.ENDS_WITH, - SourceFilterMatchType.DOES_NOT_END_WITH, - ) - IS_IMAGE = ("isimage", SourceFilterMatchType.IS_IMAGE, None) - IS_TIF = ("istif", SourceFilterMatchType.IS_TIF, None) - - @classmethod - def from_literal(cls, value: str) -> "CellProfilerSourceFilterOperator": - normalized = value.strip().casefold() - for member in cls: - if member.value == normalized: - return member - raise ValueError(f"Unsupported source filter operator: {value!r}.") - - def generic_match_type(self, *, negated: bool) -> SourceFilterMatchType: - if not negated: - return self.match_type - if self.negated_match_type is None: - raise ValueError( - "Unsupported source filter operator/negation pair: " - f"{self.value!r}, negated=True." - ) - return self.negated_match_type +_CELLPROFILER_FILTER_OPERATORS: Mapping[str, type[SourceFilterMatchType]] = { + "contain": TextContains, + "containregexp": TextContainsRegex, + "eq": TextEquals, + "startwith": TextStartsWith, + "endwith": TextEndsWith, + "isimage": PathIsImage, + "istif": PathIsTif, +} +"""CellProfiler's filter-operator literals and the match each spells.""" + + +def _cellprofiler_filter_match( + operator_literal: str, *, negated: bool +) -> type[SourceFilterMatchType]: + match_type = _CELLPROFILER_FILTER_OPERATORS.get(operator_literal) + if match_type is None: + raise ValueError(f"Unsupported source filter operator: {operator_literal!r}.") + if not negated: + return match_type + if not issubclass(match_type, TextMatchType): + raise ValueError( + "Unsupported source filter operator/negation pair: " + f"{operator_literal!r}, negated=True." + ) + return match_type.negation() class CellProfilerSourceImageType(Enum): @@ -174,7 +138,7 @@ def __new__( cls, literal: str, artifact_kind: type[ArtifactType], - projection_role: SourceProjectionRole, + projection_role: type[SourceProjectionRole], load_as_monochrome: bool, load_as_mask: bool, source_channel_axis: int | None, @@ -193,7 +157,7 @@ def __new__( GRAYSCALE = ( "grayscale image", ImageArtifactType, - SourceProjectionRole.PRIMARY_PLANE, + PrimaryPlaneRole, True, False, None, @@ -202,7 +166,7 @@ def __new__( COLOR = ( "color image", ImageArtifactType, - SourceProjectionRole.PRIMARY_PLANE, + PrimaryPlaneRole, False, False, -1, @@ -211,7 +175,7 @@ def __new__( BINARY_IMAGE = ( "binary image", ImageArtifactType, - SourceProjectionRole.PRIMARY_PLANE, + PrimaryPlaneRole, True, True, None, @@ -220,7 +184,7 @@ def __new__( BINARY_MASK = ( "binary mask", ImageArtifactType, - SourceProjectionRole.PRIMARY_PLANE, + PrimaryPlaneRole, True, True, None, @@ -229,7 +193,7 @@ def __new__( MASK = ( "mask", ImageArtifactType, - SourceProjectionRole.PRIMARY_PLANE, + PrimaryPlaneRole, True, True, None, @@ -238,7 +202,7 @@ def __new__( ILLUMINATION_FUNCTION = ( "illumination function", ImageArtifactType, - SourceProjectionRole.SOURCE_ARTIFACT, + SourceArtifactRole, False, False, None, @@ -247,7 +211,7 @@ def __new__( OBJECTS = ( "objects", ObjectLabelsArtifactType, - SourceProjectionRole.SOURCE_ARTIFACT, + SourceArtifactRole, False, False, None, @@ -266,9 +230,9 @@ def binding( self, alias: str, selector: SourceSelector, - origin: SourceBindingOrigin, + origin: type[SourceBindingOrigin], *, - source_set_role: SourceSetRole, + source_set_role: type[SourceSetRole], explicit_source: ImagePlaneSource | None = None, ) -> NamedSourceBinding: return NamedSourceBinding( @@ -430,23 +394,26 @@ def _source_filter_clause_from_parts( negated: bool, value: str | None, ) -> SourceFilterClause: - subject = CellProfilerSourceFilterSubject.from_literal(subject_literal).subject + subject = registered_class_from_key( + SourceFilterSubject, subject_literal.strip().casefold() + ) normalized_operator = operator_literal.casefold() if ( - subject is SourceFilterSubject.EXTENSION + subject is SourceExtension and not negated and normalized_operator.startswith("is") and normalized_operator not in {"isimage", "istif"} ): return SourceFilterClause( subject=subject, - match_type=SourceFilterMatchType.EQUALS, + match_type=TextEquals, value=f".{normalized_operator.removeprefix('is')}", ) - operator = CellProfilerSourceFilterOperator.from_literal(normalized_operator) return SourceFilterClause( subject=subject, - match_type=operator.generic_match_type(negated=negated), + match_type=_cellprofiler_filter_match( + normalized_operator.strip(), negated=negated + ), value=value, ) @@ -510,14 +477,11 @@ def _images_filters(module: ModuleBlock) -> tuple[SourceFilterClause, ...]: ) filtering_mode = module.get_setting("Filter images?", "").strip().casefold() if "images" in filtering_mode and not any( - clause.match_type is SourceFilterMatchType.IS_IMAGE for clause in filters + clause.match_type is PathIsImage for clause in filters ): filters.insert( 0, - SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.IS_IMAGE, - ), + SourceFilterClause(subject=SourceFileName, match_type=PathIsImage), ) return tuple(dict.fromkeys(filters)) @@ -559,13 +523,13 @@ def contribute_source_bindings( ) -def _load_images_match_type(module: ModuleBlock) -> SourceFilterMatchType: +def _load_images_match_type(module: ModuleBlock) -> type[SourceFilterMatchType]: mode = module.get_setting("How do you want to load these files?", "") normalized = mode.strip().casefold() if not normalized or "exact" in normalized: - return SourceFilterMatchType.CONTAINS + return TextContains if "regular" in normalized or "regex" in normalized: - return SourceFilterMatchType.CONTAINS_REGEX + return TextContainsRegex raise ValueError(f"Unsupported LoadImages matching mode: {mode!r}.") @@ -578,7 +542,7 @@ def _load_images_filters( if match_text: filters.append( SourceFilterClause( - subject=SourceFilterSubject.FILE, + subject=SourceFileName, match_type=_load_images_match_type(module), value=decode_cellprofiler_setting_literal(match_text), ) @@ -591,23 +555,23 @@ def _load_images_filters( if exclusion: filters.append( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.DOES_NOT_CONTAIN, + subject=SourceFileName, + match_type=TextDoesNotContain, value=decode_cellprofiler_setting_literal(exclusion), ) ) return tuple(filters) -def _load_images_metadata_sources(value: str) -> tuple[MetadataSource, ...]: +def _load_images_metadata_sources(value: str) -> tuple[type[MetadataSource], ...]: normalized = value.strip().casefold() if not normalized or normalized == "none": return () - sources: list[MetadataSource] = [] + sources: list[type[MetadataSource]] = [] if "file" in normalized or "both" in normalized: - sources.append(MetadataSource.FILE_NAME) + sources.append(FromFileName) if any(token in normalized for token in ("folder", "subfolder", "path", "both")): - sources.append(MetadataSource.FOLDER_NAME) + sources.append(FromFolderName) if not sources: raise ValueError(f"Unsupported LoadImages metadata extraction mode: {value!r}.") return tuple(dict.fromkeys(sources)) @@ -615,11 +579,11 @@ def _load_images_metadata_sources(value: str) -> tuple[MetadataSource, ...]: def _load_images_metadata_pattern( block: Sequence[ModuleSetting], - source: MetadataSource, + source: type[MetadataSource], ) -> str: family = ( _LOAD_IMAGES_FOLDER_PATTERN - if source is MetadataSource.FOLDER_NAME + if source is FromFolderName else _LOAD_IMAGES_FILE_PATTERN ) pattern = decode_cellprofiler_setting_literal( @@ -628,7 +592,7 @@ def _load_images_metadata_pattern( if not pattern or pattern.strip().casefold() == "none": raise ValueError( "LoadImages metadata extraction requires a non-empty " - f"{source.value} regular expression." + f"{source.name} regular expression." ) return pattern @@ -668,8 +632,8 @@ def contribute_source_bindings( source_type.binding( alias, selector, - SourceBindingOrigin.PIPELINE_START, - source_set_role=SourceSetRole.MATCHED, + FromPipelineStart, + source_set_role=MatchedSource, ) ) for source in _load_images_metadata_sources( @@ -711,11 +675,11 @@ class CellProfilerMetadataPatternBlock: """Typed access to one CellProfiler path-metadata regex block.""" block: Sequence[ModuleSetting] - source: MetadataSource + source: type[MetadataSource] @property def pattern(self) -> str: - if self.source is MetadataSource.FOLDER_NAME: + if self.source is FromFolderName: value = block_setting_value( self.block, "Regular expression to extract from folder name", @@ -732,7 +696,7 @@ def required_pattern(self) -> str: if not self.pattern: raise ValueError( "CellProfiler path metadata extraction requires a non-empty " - f"{self.source.value} regular expression." + f"{self.source.name} regular expression." ) return self.pattern @@ -840,14 +804,14 @@ def is_active(self, *, extraction_enabled: bool) -> bool: return True class ExtractionSource(Enum): - def __new__(cls, literal: str, source: MetadataSource): + def __new__(cls, literal: str, source: type[MetadataSource]): member = object.__new__(cls) member._value_ = literal member.source = source return member - FILE_NAME = ("File name", MetadataSource.FILE_NAME) - FOLDER_NAME = ("Folder name", MetadataSource.FOLDER_NAME) + FILE_NAME = ("File name", FromFileName) + FOLDER_NAME = ("Folder name", FromFolderName) class ExtractionScope(str, Enum): ALL_IMAGES = "All images" @@ -1316,10 +1280,10 @@ def contribute_source_bindings( binding = source_type.binding( alias, selector, - SourceBindingOrigin.PIPELINE_START, - source_set_role=SourceSetRole.MATCHED, + FromPipelineStart, + source_set_role=MatchedSource, ) - if source_type.projection_role is SourceProjectionRole.PRIMARY_PLANE: + if source_type.projection_role is PrimaryPlaneRole: channel_identity = _selector_component_identity( selector, colour_axis, @@ -1364,11 +1328,11 @@ def contribute_source_bindings( binding = source_type.binding( alias, SourceSelector(), - SourceBindingOrigin.PIPELINE_START, - source_set_role=SourceSetRole.BROADCAST, + FromPipelineStart, + source_set_role=BroadcastSource, explicit_source=explicit_source, ) - if source_type.projection_role is SourceProjectionRole.PRIMARY_PLANE: + if source_type.projection_role is PrimaryPlaneRole: while str(next_channel_index) in used_channel_values: next_channel_index += 1 channel_identity = ComponentSelector( diff --git a/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth.py b/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth.py index 4227bde834..b8fa040381 100644 --- a/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth.py +++ b/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth.py @@ -43,8 +43,8 @@ ComponentSelector, NamedSourceBinding, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, + SourceFileName, + TextEquals, SourceSelector, ) from openhcs.core.compiled_step_plan import CompiledStepPlan @@ -142,8 +142,8 @@ def _exact_image_binding( selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.EQUALS, + subject=SourceFileName, + match_type=TextEquals, value=source.filename, ), ), diff --git a/openhcs/processing/presets/pipelines/neuroncyto_ii_crossover_neurite_outgrowth.py b/openhcs/processing/presets/pipelines/neuroncyto_ii_crossover_neurite_outgrowth.py index 2676f4f227..a1f5bc9b19 100644 --- a/openhcs/processing/presets/pipelines/neuroncyto_ii_crossover_neurite_outgrowth.py +++ b/openhcs/processing/presets/pipelines/neuroncyto_ii_crossover_neurite_outgrowth.py @@ -41,8 +41,8 @@ ComponentSelector, NamedSourceBinding, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, + SourceFileName, + TextEquals, SourceSelector, ) from openhcs.core.compiled_step_plan import CompiledStepPlan @@ -89,8 +89,8 @@ def _exact_source_binding( selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.EQUALS, + subject=SourceFileName, + match_type=TextEquals, value=filename, ), ), diff --git a/openhcs/pyqt_gui/widgets/source_bindings_editor.py b/openhcs/pyqt_gui/widgets/source_bindings_editor.py index b72c2ed760..4f75b5b3bc 100644 --- a/openhcs/pyqt_gui/widgets/source_bindings_editor.py +++ b/openhcs/pyqt_gui/widgets/source_bindings_editor.py @@ -99,13 +99,13 @@ ComponentSelector, MetadataExtractionRule, MetadataSelector, - MetadataSource, + FromFileName, NamedSourceBinding, SourceBindingMatchDimension, SourceBindingMatchField, + PathIsImage, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSetPairing, PairByMetadata, PairByOrder, @@ -1287,10 +1287,7 @@ def add_source_filter_row( try: self.source_filters_controller.append( clause - or SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.IS_IMAGE, - ) + or SourceFilterClause(SourceFileName, PathIsImage) ) finally: self._updating_ui = False @@ -1332,7 +1329,7 @@ def add_metadata_rule_row( self.metadata_rules_controller.append( rule or MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?P.+)", ) ) diff --git a/tests/diagnostics/owned_bootstrap_readback.py b/tests/diagnostics/owned_bootstrap_readback.py index dc2b08a8bb..fa6f3a69e8 100644 --- a/tests/diagnostics/owned_bootstrap_readback.py +++ b/tests/diagnostics/owned_bootstrap_readback.py @@ -13,7 +13,10 @@ from openhcs.core.roi_source_metadata import ROIArchiveSourceMetadata from openhcs.core.runtime_object_labels import object_label_dense_array from openhcs.core.source_matching import source_component_metadata_value -from openhcs.core.source_bindings import SourceProjectionRole +from openhcs.core.source_bindings import ( + PrimaryPlaneRole, + SourceArtifactRole, +) from openhcs.core.source_projection import OpenHCSPlaneAddress from openhcs.core.virtual_workspace_metadata import ( METADATA_CONFIG, OpenHCSMetadataSubdirectories, @@ -46,13 +49,13 @@ def addresses(metadata): def require_projection_inventory(saved, source_addresses, artifact_plans): """Require primary AND named image planes from the compiled declarations.""" - expected = Counter((SourceProjectionRole.PRIMARY_PLANE, None, ImageArtifactType, address) + expected = Counter((PrimaryPlaneRole, None, ImageArtifactType, address) for address in source_addresses) for plan in artifact_plans: artifact_type = ArtifactType.coerce(plan.kind) if (issubclass(artifact_type, ImageArtifactType) and plan.materialization is not None and plan.materialization.persistent_enabled): - expected.update((SourceProjectionRole.SOURCE_ARTIFACT, plan.name, artifact_type, address) + expected.update((SourceArtifactRole, plan.name, artifact_type, address) for address in source_addresses) actual = Counter((item.projection_role, item.source_alias, item.artifact_kind, item.address) for item in saved) diff --git a/tests/integration/test_cellprofiler_generated_pipeline.py b/tests/integration/test_cellprofiler_generated_pipeline.py index 8cc702c685..a436ba87f3 100644 --- a/tests/integration/test_cellprofiler_generated_pipeline.py +++ b/tests/integration/test_cellprofiler_generated_pipeline.py @@ -73,12 +73,12 @@ ) from openhcs.core.source_bindings import ( ComponentSelector, + FromPipelineStart, NamedSourceBinding, - SourceBindingOrigin, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, + TextContains, ) from openhcs.core.steps.function_step import FunctionStep from openhcs.interop.cellprofiler.pipeline_import import import_cellprofiler_pipeline @@ -369,13 +369,13 @@ def test_invalid_public_cellprofiler_step_fails_during_zmq_compilation( selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="w1", ), ), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=( ComponentSelector(Microscopy.Channel, "1"), ), @@ -551,8 +551,8 @@ def test_loadimages_cppipe_preserves_source_artifact_bindings_through_zmq( illum_assignment = source_bindings.binding_for_alias("Illum") assert raw_assignment is not None assert illum_assignment is not None - assert raw_assignment.origin is SourceBindingOrigin.PIPELINE_START - assert illum_assignment.origin is SourceBindingOrigin.PIPELINE_START + assert raw_assignment.origin is FromPipelineStart + assert illum_assignment.origin is FromPipelineStart _, export = _execute_imported_cppipe_via_zmq( tmp_path, @@ -1004,14 +1004,14 @@ def test_official_illumination_preserves_rule_row_binding_over_zmq( orig_green = source_bindings.binding_for_alias("OrigGreen") assert orig_green is not None assert source_bindings.binding_for_alias("DNA") is None - assert orig_green.origin is SourceBindingOrigin.PIPELINE_START + assert orig_green.origin is FromPipelineStart assert orig_green.component_identity == ( ComponentSelector(Microscopy.Channel, "1"), ) assert len(orig_green.selector.filters) == 1 source_filter = orig_green.selector.filters[0] - assert source_filter.subject is SourceFilterSubject.FILE - assert source_filter.match_type is SourceFilterMatchType.CONTAINS + assert source_filter.subject is SourceFileName + assert source_filter.match_type is TextContains assert source_filter.value == "AS_09047_" _, export = _execute_imported_cppipe_via_zmq( diff --git a/tests/integration/test_chained_callable_lineage_journey.py b/tests/integration/test_chained_callable_lineage_journey.py index b57f976444..56ede7c673 100644 --- a/tests/integration/test_chained_callable_lineage_journey.py +++ b/tests/integration/test_chained_callable_lineage_journey.py @@ -30,10 +30,10 @@ from openhcs.core.source_bindings import ( ComponentSelector, NamedSourceBinding, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, + TextContains, ) from openhcs.core.steps.function_step import FunctionStep from openhcs.processing.custom_functions.runtime_registry import ( @@ -73,8 +73,8 @@ def test_chained_public_callable_uses_declared_main_flow_not_storage_argument( alias="label-fixture", selector=SourceSelector(filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="label-fixture", ), )), diff --git a/tests/integration/test_image_math_publication_620.py b/tests/integration/test_image_math_publication_620.py index b849e8e233..9cb890c651 100644 --- a/tests/integration/test_image_math_publication_620.py +++ b/tests/integration/test_image_math_publication_620.py @@ -20,7 +20,10 @@ from openhcs.core.orchestrator.execution_result import RuntimeObservationMode from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.pipeline_document import PipelineDocumentCodec -from openhcs.core.source_bindings import MetadataExtractionRule, MetadataSource +from openhcs.core.source_bindings import ( + FromFileName, + MetadataExtractionRule, +) from openhcs.core.source_metadata import SourceVoxelSpacing from openhcs.core.source_workspace_projection import VirtualWorkspaceSourceProjection from openhcs.core.virtual_workspace_metadata import ( @@ -51,7 +54,7 @@ def test_registered_mixed_image_math_publishes_once_in_both_operand_orders(tmp_p variable_components=[Microscopy.ZIndex], group_by=Microscopy.Channel), path_planning_config=LazyPathPlanningConfig(global_output_folder=tmp_path / 'results'), source_bindings_config=replace(imported.source_bindings_config, - metadata_rules=(MetadataExtractionRule(source=MetadataSource.FILE_NAME, + metadata_rules=(MetadataExtractionRule(source=FromFileName, pattern=r'(?P[A-H]\d{2})_s(?P\d+)_w(?P\d+)_z(?P\d+)_t(?P\d+)_(?:Gray|Color)\.tif'),), source_stack_components=(), source_voxel_spacing=SourceVoxelSpacing((0.5, 0.5))), napari_streaming_config=LazyNapariStreamingConfig(enabled=False), diff --git a/tests/integration/test_measurement_declaration_journey.py b/tests/integration/test_measurement_declaration_journey.py index 01a9a187ee..cacac25c62 100644 --- a/tests/integration/test_measurement_declaration_journey.py +++ b/tests/integration/test_measurement_declaration_journey.py @@ -55,11 +55,11 @@ from openhcs.core.source_bindings import ( ComponentSelector, NamedSourceBinding, + PairByOrder, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, - PairByOrder, + TextContains, ) from openhcs.core.steps.function_step import FunctionStep from openhcs.interop.cellprofiler.measurement_dialect import ( @@ -186,8 +186,8 @@ def _source(alias, channel): selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value=f"_w{channel}_", ), ), diff --git a/tests/integration/test_primary_segmentation_diagnostics_journey.py b/tests/integration/test_primary_segmentation_diagnostics_journey.py index 88c75b18f9..6eb8399d94 100644 --- a/tests/integration/test_primary_segmentation_diagnostics_journey.py +++ b/tests/integration/test_primary_segmentation_diagnostics_journey.py @@ -132,10 +132,10 @@ def test_normal_compiled_runtime_persists_diagnostics_and_preserves_secondary_bi from openhcs.core.source_bindings import ( ComponentSelector, NamedSourceBinding, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, + TextContains, ) from openhcs.core.config import StepSourceBindingsConfig from openhcs.core.source_image_provenance import SourceImageProvenance @@ -236,8 +236,8 @@ def step(func, name, kwargs): selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="DNA", ), ) diff --git a/tests/integration/test_source_schema_aggregate_publication.py b/tests/integration/test_source_schema_aggregate_publication.py index 37c4614558..114d50e7f3 100644 --- a/tests/integration/test_source_schema_aggregate_publication.py +++ b/tests/integration/test_source_schema_aggregate_publication.py @@ -34,8 +34,8 @@ from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.core.source_bindings import ( ComponentSelector, + FromFileName, MetadataExtractionRule, - MetadataSource, NamedSourceBinding, SourceSelector, ) @@ -84,7 +84,7 @@ def test_real_z_aggregation_persists_scalar_and_unchanged_source_planes(tmp_path dataset_source=SourceBindingsSource, source_bindings_config=LazySourceBindingsConfig( metadata_rules=(MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?PA[0-9]+)_s(?P[0-9]+)_w(?P[0-9]+)_z(?P[0-9]+)_t(?P[0-9]+)", ),), bindings=(NamedSourceBinding( diff --git a/tests/integration/test_source_store_code_mode_zmq.py b/tests/integration/test_source_store_code_mode_zmq.py index d1673258ab..cf5ddd401b 100644 --- a/tests/integration/test_source_store_code_mode_zmq.py +++ b/tests/integration/test_source_store_code_mode_zmq.py @@ -35,10 +35,10 @@ from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.core.source_bindings import ( NamedSourceBinding, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, + TextEquals, ) from openhcs.core.source_metadata import SourceMetadataFields from openhcs.core.steps.function_step import FunctionStep @@ -90,8 +90,8 @@ def _binding(alias: str, file_name: str | None = None) -> NamedSourceBinding: selector = SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.EQUALS, + subject=SourceFileName, + match_type=TextEquals, value=file_name, ), ), diff --git a/tests/unit/pyqt_gui/test_source_bindings_editor.py b/tests/unit/pyqt_gui/test_source_bindings_editor.py index 424f8a24ee..abc260a3e4 100644 --- a/tests/unit/pyqt_gui/test_source_bindings_editor.py +++ b/tests/unit/pyqt_gui/test_source_bindings_editor.py @@ -14,24 +14,30 @@ QStyle, QWidget, ) -from python_introspect import is_enableable +from metaclass_registry import AutoRegisterMeta +from python_introspect import is_enableable, to_jsonable from openhcs.core.source_bindings import ( + BroadcastSource, ComponentSelector, + FromFileName, + FromPipelineStart, MetadataExtractionRule, - MetadataSource, MetadataSelector, + NamedSourceBinding, + PairByMetadata, + PathIsImage, + SourceArtifactRole, SourceBindingMatchDimension, SourceBindingMatchField, - PairByMetadata, + SourceDirectory, + SourceExtension, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, - SourceBindingOrigin, - SourceProjectionRole, - SourceSetRole, - NamedSourceBinding, + TextContains, + TextDoesNotContainRegex, + TextEquals, ) from openhcs.core.config import ( GlobalPipelineConfig, @@ -165,8 +171,8 @@ def table_cell_text(table, row: int, column: int) -> str: return widget.line_edit.text() if isinstance(widget, QComboBox): value = widget.currentData() - if isinstance(value, Enum): - return str(value.value) + if isinstance(value, (Enum, AutoRegisterMeta)): + return str(to_jsonable(value)) return widget.currentText().strip() item = table.item(row, column) if item is None: @@ -266,8 +272,8 @@ def test_source_bindings_editor_renders_pipeline_source_filter_table() -> None: SourceBindingsConfig( source_filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ), @@ -308,8 +314,8 @@ def test_source_bindings_editor_edits_pipeline_source_filters() -> None: assert widget.get_value().source_filters == ( SourceFilterClause( - SourceFilterSubject.EXTENSION, - SourceFilterMatchType.EQUALS, + SourceExtension, + TextEquals, ".tif", ), ) @@ -321,8 +327,8 @@ def test_source_bindings_source_filter_dropdown_ignores_wheel() -> None: widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) ) @@ -332,7 +338,7 @@ def test_source_bindings_source_filter_dropdown_ignores_wheel() -> None: column_index(SourceFilterClause, "match_type"), ) assert isinstance(combo, NoScrollComboBox) - assert combo.currentData() is SourceFilterMatchType.CONTAINS + assert combo.currentData() is TextContains wheel_event = QWheelEvent( QPointF(5, 5), @@ -346,11 +352,11 @@ def test_source_bindings_source_filter_dropdown_ignores_wheel() -> None: ) QApplication.sendEvent(combo, wheel_event) - assert combo.currentData() is SourceFilterMatchType.CONTAINS + assert combo.currentData() is TextContains assert widget.get_value().source_filters == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ) @@ -366,8 +372,8 @@ def test_source_bindings_editor_accepts_lazy_pipeline_source_bindings() -> None: assert get_base_type_for_lazy(type(edited)) is SourceBindingsConfig assert edited.source_filters == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.IS_IMAGE, + SourceFileName, + PathIsImage, ), ) @@ -426,8 +432,8 @@ def test_inline_source_bindings_root_reset_all_resets_child_fields() -> None: assert isinstance(widget, SourceBindingsEditorWidget) widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) ) @@ -502,12 +508,12 @@ def test_pipeline_source_bindings_table_edits_recreate_container_and_children() ), ) expected_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) expected_rule = MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?PA\d{2})_(?PDNA)\.tif", ) expected_match_plan = PairByMetadata(dimensions=( @@ -775,8 +781,8 @@ def test_pipeline_step_source_filter_time_travel_refreshes_open_step_editor() -> assert isinstance(step_widget, SourceBindingsEditorWidget) first_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) pipeline_widget.add_source_filter_row(first_filter) @@ -812,8 +818,8 @@ def test_pipeline_step_source_filter_time_travel_refreshes_open_step_editor() -> QApplication.processEvents() second_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, "DNA", ) assert plate_state.parameters["step_source_bindings_config.source_filters"] == ( @@ -1191,8 +1197,8 @@ def test_pipeline_source_bindings_preview_preserves_inherited_table_rows() -> No "step_source_bindings_config.source_filters" ) == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ) @@ -1273,8 +1279,8 @@ def test_pipeline_source_bindings_preview_preserves_inherited_table_rows() -> No def test_source_bindings_enableable_only_updates_skip_table_rebuild() -> None: QtApplicationHarness.app() inherited_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) widget = SourceBindingsEditorWidget.from_bindings( @@ -1390,8 +1396,8 @@ def test_source_bindings_table_child_reset_restores_inherited_placeholder_rows() assert isinstance(step_widget, SourceBindingsEditorWidget) inherited_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) source_widget.add_source_filter_row(inherited_filter) @@ -1423,8 +1429,8 @@ def test_source_bindings_table_child_reset_restores_inherited_placeholder_rows() QApplication.processEvents() local_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "RNA", ) assert state.parameters["step_source_bindings_config.source_filters"] == ( @@ -1514,8 +1520,8 @@ def test_source_bindings_inherited_table_combo_activation_materializes_child() - assert isinstance(step_widget, SourceBindingsEditorWidget) inherited_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) source_widget.add_source_filter_row(inherited_filter) @@ -1606,8 +1612,8 @@ def test_source_bindings_inherited_table_value_edit_undo_restores_lazy_child() - assert isinstance(step_widget, SourceBindingsEditorWidget) inherited_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) source_widget.add_source_filter_row(inherited_filter) @@ -1627,8 +1633,8 @@ def test_source_bindings_inherited_table_value_edit_undo_restores_lazy_child() - QApplication.processEvents() edited_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "RNA", ) assert state.parameters["step_source_bindings_config.source_filters"] == ( @@ -1694,8 +1700,8 @@ def test_source_bindings_inherited_table_edits_flash_only_changed_cell() -> None assert isinstance(step_widget, SourceBindingsEditorWidget) inherited_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) source_widget.add_source_filter_row(inherited_filter) @@ -1715,8 +1721,8 @@ def test_source_bindings_inherited_table_edits_flash_only_changed_cell() -> None QApplication.processEvents() materialized_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "RNA", ) assert state.parameters["step_source_bindings_config.source_filters"] == ( @@ -1756,8 +1762,8 @@ def test_source_bindings_inherited_table_edits_flash_only_changed_cell() -> None QApplication.processEvents() edited_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "RNA2", ) assert state.parameters["step_source_bindings_config.source_filters"] == ( @@ -2117,12 +2123,12 @@ def test_source_bindings_editor_uses_resolved_placeholder_tables() -> None: QtApplicationHarness.app() inherited_binding = NamedSourceBinding(alias="InheritedDNA") inherited_rule = MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?PA\d{2})_(?PDNA)\.tif", ) inherited_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) local_raw = LazyStepSourceBindingsConfig() @@ -2151,7 +2157,7 @@ def test_source_bindings_editor_uses_resolved_placeholder_tables() -> None: column_index(SourceFilterClause, "match_type"), ) assert isinstance(match_type_widget, QComboBox) - assert match_type_widget.currentData() is SourceFilterMatchType.CONTAINS + assert match_type_widget.currentData() is TextContains assert match_type_widget.currentText() == "contains" assert PlaceholderConfig.text_color_name() in match_type_widget.styleSheet() @@ -2167,8 +2173,8 @@ def test_source_bindings_editor_uses_resolved_placeholder_tables() -> None: def test_source_bindings_table_row_value_read_does_not_emit_item_changed() -> None: QtApplicationHarness.app() inherited_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) widget = SourceBindingsEditorWidget.from_bindings( @@ -2190,7 +2196,7 @@ def test_source_bindings_table_row_value_read_does_not_emit_item_changed() -> No widget.source_filters_table.itemChanged.connect(emitted.append) assert widget.source_filters_controller.row_values() == ( - (SourceFilterSubject.FILE, SourceFilterMatchType.CONTAINS, "DNA", ""), + (SourceFileName, TextContains, "DNA", ""), ) assert emitted == [] @@ -2200,8 +2206,8 @@ def test_source_bindings_editor_editing_inherited_table_makes_lazy_child_concret ): QtApplicationHarness.app() inherited_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) local_raw = LazyStepSourceBindingsConfig() @@ -2229,8 +2235,8 @@ def test_source_bindings_editor_editing_inherited_table_makes_lazy_child_concret assert DataclassFieldAccess.raw_value(edited, "bindings") is None assert DataclassFieldAccess.raw_value(edited, "source_filters") == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "RNA", ), ) @@ -2263,16 +2269,16 @@ def test_source_bindings_child_reset_restores_lazy_inheritance_slot() -> None: widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) ) QApplication.processEvents() assert state.parameters["source_bindings.source_filters"] == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ) @@ -2304,8 +2310,8 @@ def test_source_bindings_child_reset_noops_for_already_inherited_table_preview() ObjectStateRegistry.clear() ensure_global_config_context(GlobalPipelineConfig, GlobalPipelineConfig()) inherited_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) plate_state = ObjectState(PipelineConfig(), scope_id="plate") @@ -2397,7 +2403,7 @@ def test_source_bindings_editor_preserves_unedited_lazy_inheritance_slots() -> N QtApplicationHarness.app() inherited_binding = NamedSourceBinding(alias="InheritedDNA") inherited_rule = MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?PA\d{2})_(?PDNA)\.tif", ) local_raw = LazyStepSourceBindingsConfig() @@ -2415,7 +2421,7 @@ def test_source_bindings_editor_preserves_unedited_lazy_inheritance_slots() -> N widget.add_metadata_rule_row( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?Ps\d+)", ) ) @@ -2426,7 +2432,7 @@ def test_source_bindings_editor_preserves_unedited_lazy_inheritance_slots() -> N assert edited.metadata_rules == ( inherited_rule, MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?Ps\d+)", ), ) @@ -2537,8 +2543,8 @@ def test_source_filters_fit_complete_rows_and_bar_inside_scoped_section() -> Non app = QtApplicationHarness.app() filters = tuple( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.IS_IMAGE, + SourceFileName, + PathIsImage, f"source-{index}", ) for index in range(3) @@ -2608,7 +2614,7 @@ def test_step_metadata_rule_cells_show_inherited_and_local_edit_markers() -> Non ObjectStateRegistry.clear() ensure_global_config_context(GlobalPipelineConfig, GlobalPipelineConfig()) inherited_rule = MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?PA\d{2})_(?PDNA)\.tif", ) state = ObjectState( @@ -2675,7 +2681,7 @@ def test_step_metadata_rule_cells_show_inherited_and_local_edit_markers() -> Non ) assert state.parameters["step_source_bindings_config.metadata_rules"] == ( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=local_pattern, ), ) @@ -2948,8 +2954,8 @@ def test_inline_step_source_bindings_time_travel_preserves_dirty_marker() -> Non widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, "DNA", ) ) @@ -2983,8 +2989,8 @@ def test_inline_step_source_bindings_time_travel_preserves_dirty_marker() -> Non QApplication.processEvents() assert state.parameters["source_bindings.source_filters"] == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, "DNA", ), ) @@ -2993,7 +2999,7 @@ def test_inline_step_source_bindings_time_travel_preserves_dirty_marker() -> Non column_index(SourceFilterClause, "match_type"), ) assert isinstance(match_type_widget, QComboBox) - assert match_type_widget.currentData() is SourceFilterMatchType.EQUALS + assert match_type_widget.currentData() is TextEquals assert match_type_widget.currentText() == "*_equals" assert match_type_widget.property("objectstate_dirty") is True assert "source_bindings.source_filters" in state.dirty_fields @@ -3008,8 +3014,8 @@ def test_inline_step_source_bindings_undo_one_of_two_cell_edits_keeps_owner_dirt QtApplicationHarness.app() ObjectStateRegistry.clear() saved_filter = SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, "DNA", ) step = FunctionStep( @@ -3071,8 +3077,8 @@ def test_inline_step_source_bindings_undo_one_of_two_cell_edits_keeps_owner_dirt QApplication.processEvents() assert state.parameters["source_bindings.source_filters"] == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ) @@ -3082,7 +3088,7 @@ def test_inline_step_source_bindings_undo_one_of_two_cell_edits_keeps_owner_dirt column_index(SourceFilterClause, "match_type"), ) assert isinstance(match_type_widget, QComboBox) - assert match_type_widget.currentData() is SourceFilterMatchType.CONTAINS + assert match_type_widget.currentData() is TextContains assert match_type_widget.currentText() == "*_contains" assert match_type_widget.property("objectstate_dirty") is True @@ -3151,8 +3157,8 @@ def test_inline_source_bindings_dropdown_edit_flashes_only_changed_cell() -> Non widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, "DNA", ) ) @@ -3178,8 +3184,8 @@ def test_inline_source_bindings_dropdown_edit_flashes_only_changed_cell() -> Non assert queued == [expected_path] assert state.parameters["source_bindings_config.source_filters"] == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ) @@ -3189,7 +3195,7 @@ def test_inline_source_bindings_dropdown_edit_flashes_only_changed_cell() -> Non 0, column_index(SourceFilterClause, "match_type") ) assert isinstance(combo, QComboBox) - assert combo.currentData() is SourceFilterMatchType.CONTAINS + assert combo.currentData() is TextContains combo.setCurrentIndex(combo.currentIndex()) QApplication.processEvents() assert queued == [] @@ -3312,12 +3318,12 @@ def test_inline_source_bindings_initial_source_filter_cells_show_signature_diff( source_bindings=LazyStepSourceBindingsConfig( source_filters=( SourceFilterClause( - SourceFilterSubject.EXTENSION, - SourceFilterMatchType.IS_IMAGE, + SourceExtension, + PathIsImage, ), SourceFilterClause( - SourceFilterSubject.DIRECTORY, - SourceFilterMatchType.DOES_NOT_CONTAIN_REGEX, + SourceDirectory, + TextDoesNotContainRegex, value=r"[\\/]\.", ), ), @@ -3400,8 +3406,8 @@ def test_inline_source_bindings_structural_path_flash_targets_table_cell() -> No assert isinstance(widget, SourceBindingsEditorWidget) widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) ) @@ -3479,8 +3485,8 @@ def __init__(self, form_manager: ParameterFormManager) -> None: for value in ("DNA", "RNA", "Protein"): widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, value, ) ) @@ -3589,8 +3595,8 @@ def test_source_bindings_cell_flash_element_masks_cell_and_child_label() -> None assert isinstance(widget, SourceBindingsEditorWidget) widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) ) @@ -3654,8 +3660,8 @@ def test_source_bindings_child_section_flash_masks_changed_child_section() -> No assert isinstance(widget, SourceBindingsEditorWidget) widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) ) @@ -3727,8 +3733,8 @@ def test_source_bindings_owner_flash_masks_descendant_fields() -> None: assert isinstance(widget, SourceBindingsEditorWidget) widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) ) @@ -3800,8 +3806,8 @@ def test_source_bindings_dropdown_time_travel_restores_widget_value() -> None: widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, "DNA", ) ) @@ -3836,7 +3842,7 @@ def test_source_bindings_dropdown_time_travel_restores_widget_value() -> None: column_index(SourceFilterClause, "match_type"), ) assert isinstance(match_type_widget, QComboBox) - assert match_type_widget.currentData() is SourceFilterMatchType.CONTAINS + assert match_type_widget.currentData() is TextContains assert match_type_widget.currentText() == "*_contains" assert match_type_widget.property("objectstate_dirty") is True assert ( @@ -3848,8 +3854,8 @@ def test_source_bindings_dropdown_time_travel_restores_widget_value() -> None: QApplication.processEvents() assert state.parameters["source_bindings_config.source_filters"] == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, "DNA", ), ) @@ -3866,7 +3872,7 @@ def test_source_bindings_dropdown_time_travel_restores_widget_value() -> None: column_index(SourceFilterClause, "match_type"), ) assert isinstance(match_type_widget, QComboBox) - assert match_type_widget.currentData() is SourceFilterMatchType.EQUALS + assert match_type_widget.currentData() is TextEquals assert match_type_widget.currentText() == "*_equals" assert match_type_widget.property("objectstate_dirty") is True assert ( @@ -3878,8 +3884,8 @@ def test_source_bindings_dropdown_time_travel_restores_widget_value() -> None: QApplication.processEvents() assert state.parameters["source_bindings_config.source_filters"] == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ) @@ -3896,7 +3902,7 @@ def test_source_bindings_dropdown_time_travel_restores_widget_value() -> None: column_index(SourceFilterClause, "match_type"), ) assert isinstance(match_type_widget, QComboBox) - assert match_type_widget.currentData() is SourceFilterMatchType.CONTAINS + assert match_type_widget.currentData() is TextContains assert match_type_widget.currentText() == "*_contains" assert match_type_widget.property("objectstate_dirty") is True assert ( @@ -3935,8 +3941,8 @@ def test_source_bindings_text_time_travel_restores_widget_value() -> None: widget.add_source_filter_row( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ) ) @@ -3975,8 +3981,8 @@ def test_source_bindings_text_time_travel_restores_widget_value() -> None: QApplication.processEvents() assert state.parameters["source_bindings_config.source_filters"] == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ) @@ -3997,8 +4003,8 @@ def test_source_bindings_text_time_travel_restores_widget_value() -> None: QApplication.processEvents() assert state.parameters["source_bindings_config.source_filters"] == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "RNA", ), ) @@ -4046,8 +4052,8 @@ def test_source_bindings_child_state_update_refreshes_widget_value() -> None: first_filters = ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, "DNA", ), ) @@ -4090,8 +4096,8 @@ def test_source_bindings_child_state_update_refreshes_widget_value() -> None: second_filters = ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "RNA", ), ) @@ -4305,8 +4311,8 @@ def test_source_bindings_editor_set_value_resets_stale_preview_table() -> None: raw_config = StepSourceBindingsConfig( source_filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ), @@ -4314,8 +4320,8 @@ def test_source_bindings_editor_set_value_resets_stale_preview_table() -> None: preview_config = StepSourceBindingsConfig( source_filters=( SourceFilterClause( - SourceFilterSubject.EXTENSION, - SourceFilterMatchType.CONTAINS, + SourceExtension, + TextContains, ".tif", ), ), @@ -4353,12 +4359,12 @@ def test_source_bindings_editor_renders_preview_context(tmp_path) -> None: bindings=( NamedSourceBinding( alias="DNA", - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ), @@ -4408,8 +4414,8 @@ def test_source_bindings_editor_preserves_selector_on_basic_edits() -> None: metadata=(MetadataSelector("Well", "A01"),), filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ), @@ -4435,7 +4441,7 @@ def test_source_bindings_editor_preserves_binding_identity_on_basic_edits() -> N binding = NamedSourceBinding( alias="DNA", component_identity=(ComponentSelector(Microscopy.Channel, "1"),), - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ) widget = SourceBindingsEditorWidget.from_bindings( StepSourceBindingsConfig(bindings=(binding,)) @@ -4448,7 +4454,7 @@ def test_source_bindings_editor_preserves_binding_identity_on_basic_edits() -> N edited = widget.get_value().bindings[0] assert edited.alias == "OrigDNA" assert edited.component_identity == (ComponentSelector(Microscopy.Channel, "1"),) - assert edited.projection_role is SourceProjectionRole.SOURCE_ARTIFACT + assert edited.projection_role is SourceArtifactRole def test_source_bindings_editor_edits_binding_identity_columns() -> None: @@ -4467,13 +4473,13 @@ def test_source_bindings_editor_edits_binding_identity_columns() -> None: table, 0, "source_set_role", - SourceSetRole.BROADCAST.value, + BroadcastSource.name, ) set_binding_cell_text( table, 0, "projection_role", - SourceProjectionRole.SOURCE_ARTIFACT.value, + SourceArtifactRole.name, ) widget._apply_step_bindings(dialog.bindings()) @@ -4482,8 +4488,8 @@ def test_source_bindings_editor_edits_binding_identity_columns() -> None: ComponentSelector(Microscopy.Channel, "2"), ComponentSelector(Microscopy.Site, "1"), ) - assert binding.source_set_role is SourceSetRole.BROADCAST - assert binding.projection_role is SourceProjectionRole.SOURCE_ARTIFACT + assert binding.source_set_role is BroadcastSource + assert binding.projection_role is SourceArtifactRole def test_source_bindings_editor_edits_selector_columns() -> None: @@ -4506,7 +4512,7 @@ def test_source_bindings_editor_edits_selector_columns() -> None: set_editable_cell_text( table, *binding_cell_position(0, "filters"), - (SourceFilterClause(SourceFilterSubject.FILE, SourceFilterMatchType.CONTAINS, "DNA"),), + (SourceFilterClause(SourceFileName, TextContains, "DNA"),), ) set_binding_cell_text(table, 0, "inherit_current_scope", False) widget._apply_step_bindings(dialog.bindings()) @@ -4519,8 +4525,8 @@ def test_source_bindings_editor_edits_selector_columns() -> None: assert selector.metadata == (MetadataSelector("Well", "A01"),) assert selector.filters == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ) @@ -4535,13 +4541,13 @@ def test_source_bindings_editor_uses_free_form_selector_pickers(tmp_path) -> Non bindings=( NamedSourceBinding( alias="DNA", - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, selector=SourceSelector(), ), ), metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?PA\d{2})_(?PDNA)\.tif", ), ), @@ -4611,18 +4617,18 @@ def test_source_bindings_editor_edits_metadata_rules() -> None: widget.metadata_rules_table, 0, column_index(MetadataExtractionRule, "filters"), - (SourceFilterClause(SourceFilterSubject.FILE, SourceFilterMatchType.CONTAINS, "DNA"),), + (SourceFilterClause(SourceFileName, TextContains, "DNA"),), ) rules = widget.get_value().metadata_rules assert rules == ( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?PA\d{2})_(?PDNA)\.tif", filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ), @@ -4685,5 +4691,5 @@ def test_source_bindings_editor_enum_columns_use_typed_combos() -> None: binding = widget.get_value().bindings[0] assert binding.artifact_kind.value == "object_labels" - assert binding.origin is SourceBindingOrigin.PIPELINE_START - assert binding.projection_role is SourceProjectionRole.SOURCE_ARTIFACT + assert binding.origin is FromPipelineStart + assert binding.projection_role is SourceArtifactRole diff --git a/tests/unit/pyqt_gui/test_time_travel_viewport_context.py b/tests/unit/pyqt_gui/test_time_travel_viewport_context.py index 38171c0936..d156da37c8 100644 --- a/tests/unit/pyqt_gui/test_time_travel_viewport_context.py +++ b/tests/unit/pyqt_gui/test_time_travel_viewport_context.py @@ -5,9 +5,9 @@ from openhcs.core.config import PipelineConfig from openhcs.core.source_bindings import ( + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, + TextEquals, ) from openhcs.pyqt_gui.widgets.source_bindings_editor import ( SourceBindingsEditorWidget, @@ -72,8 +72,8 @@ def test_removed_structural_row_time_travel_preserves_visible_viewport() -> None def source_filter(value: str) -> SourceFilterClause: return SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, value, ) diff --git a/tests/unit/test_bioformats_preparation_selection.py b/tests/unit/test_bioformats_preparation_selection.py index e56de30f80..6750950efc 100644 --- a/tests/unit/test_bioformats_preparation_selection.py +++ b/tests/unit/test_bioformats_preparation_selection.py @@ -21,10 +21,12 @@ ImagePlaneSource, MetadataSelector, NamedSourceBinding, + SourceDirectory, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, + TextContains, + TextEquals, ) from openhcs.microscopes.bioformats import BioFormatsHandler from openhcs.microscopes.bioformats_adapter import BioFormatsJavaAdapter @@ -39,7 +41,7 @@ def _file_filter(name): return SourceFilterClause( - SourceFilterSubject.FILE, SourceFilterMatchType.EQUALS, name + SourceFileName, TextEquals, name ) @@ -259,13 +261,13 @@ def test_explicit_source_and_metadata_only_discovery_stay_declared(tmp_path): def test_discovery_reuses_directory_or_group_filter_semantics(tmp_path): filters = ( SourceFilterClause( - SourceFilterSubject.DIRECTORY, SourceFilterMatchType.CONTAINS, "selected" + SourceDirectory, TextContains, "selected" ), SourceFilterClause( - SourceFilterSubject.FILE, SourceFilterMatchType.EQUALS, "a.czi", any_group=0 + SourceFileName, TextEquals, "a.czi", any_group=0 ), SourceFilterClause( - SourceFilterSubject.FILE, SourceFilterMatchType.EQUALS, "b.czi", any_group=0 + SourceFileName, TextEquals, "b.czi", any_group=0 ), ) config = SourceBindingsConfig(source_filters=filters) diff --git a/tests/unit/test_cellprofiler_analyst_export.py b/tests/unit/test_cellprofiler_analyst_export.py index 98145d376e..c0186bac08 100644 --- a/tests/unit/test_cellprofiler_analyst_export.py +++ b/tests/unit/test_cellprofiler_analyst_export.py @@ -55,7 +55,7 @@ from openhcs.core.source_bindings import ( CompiledSourceBindingPlan, NamedSourceBinding, - SourceProjectionRole, + SourceArtifactRole, ) from openhcs.core.config import StepSourceBindingsConfig from openhcs.core.source_matching import SourceImageSetIdentityPolicy @@ -904,7 +904,7 @@ def _borrowed_source_export_fixture( bindings=( NamedSourceBinding(alias="DNA"), NamedSourceBinding( - alias="IllumDNA", projection_role=SourceProjectionRole.SOURCE_ARTIFACT + alias="IllumDNA", projection_role=SourceArtifactRole ), ) ) @@ -1638,7 +1638,7 @@ def test_module_contract_selects_ordered_tables_and_cpa_images_and_declares_bund bindings=( NamedSourceBinding( alias="DNA", - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ), ), diff --git a/tests/unit/test_cellprofiler_artifact_declarations.py b/tests/unit/test_cellprofiler_artifact_declarations.py index b14a5abef2..addc612bec 100644 --- a/tests/unit/test_cellprofiler_artifact_declarations.py +++ b/tests/unit/test_cellprofiler_artifact_declarations.py @@ -57,6 +57,7 @@ from openhcs.core.source_bindings import ( ComponentSelector, NamedSourceBinding, + SourceArtifactRole, SourceProjectionRole, ) from openhcs.core.steps.function_step import FunctionStep @@ -747,10 +748,12 @@ def test_declarations_carry_cross_step_object_and_measurement_flow() -> None: @pytest.mark.parametrize("in_main_flow", (True, False)) -@pytest.mark.parametrize("projection_role", tuple(SourceProjectionRole)) +@pytest.mark.parametrize( + "projection_role", tuple(SourceProjectionRole.__registry__.values()) +) def test_source_bound_input_is_declared_once_regardless_of_main_flow_membership( in_main_flow: bool, - projection_role: SourceProjectionRole, + projection_role: type[SourceProjectionRole], ) -> None: source_image = ArtifactSpec.input("OrigBlue", ImageArtifactType) module = _module( @@ -887,7 +890,7 @@ def test_group_artifact_context_reseeds_main_flow_from_exact_primary_bindings() illumination = NamedSourceBinding( alias="Illumination", artifact_kind=ImageArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, component_identity=(ComponentSelector(Microscopy.Channel, "2"),), ) bindings = (dna, rna, illumination) @@ -966,7 +969,7 @@ def test_measure_image_quality_all_loaded_uses_only_primary_plane_bindings() -> NamedSourceBinding( alias="Illumination", artifact_kind=ImageArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ) source_specs = tuple(binding.input_spec() for binding in bindings) @@ -1963,7 +1966,7 @@ def test_declaration_resolves_exact_source_binding_without_cursor_artifact() -> NamedSourceBinding( alias="Nuclei", artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ), ), diff --git a/tests/unit/test_cellprofiler_artifact_identity_reconstruction.py b/tests/unit/test_cellprofiler_artifact_identity_reconstruction.py index 8174da1634..08554a3596 100644 --- a/tests/unit/test_cellprofiler_artifact_identity_reconstruction.py +++ b/tests/unit/test_cellprofiler_artifact_identity_reconstruction.py @@ -31,7 +31,7 @@ from openhcs.core.source_bindings import ( ComponentSelector, NamedSourceBinding, - SourceProjectionRole, + SourceArtifactRole, ) from openhcs.core.config import StepSourceBindingsConfig from openhcs.core.steps.function_step import FunctionStep @@ -178,7 +178,7 @@ def test_authored_identity_remains_visible_through_compiled_artifact_graph() -> NamedSourceBinding( "Cells", artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, component_identity=(object_scope,), ), ), diff --git a/tests/unit/test_cellprofiler_combineobjects_source_scope.py b/tests/unit/test_cellprofiler_combineobjects_source_scope.py index ecd2fe016a..6ab1727561 100644 --- a/tests/unit/test_cellprofiler_combineobjects_source_scope.py +++ b/tests/unit/test_cellprofiler_combineobjects_source_scope.py @@ -30,7 +30,10 @@ PathPlannerGroupScope, ) from openhcs.core.progress import set_progress_queue -from openhcs.core.source_bindings import NamedSourceBinding, SourceProjectionRole +from openhcs.core.source_bindings import ( + NamedSourceBinding, + SourceArtifactRole, +) from openhcs.core.steps.function_step import FunctionStep from openhcs.interop.cellprofiler.plate_workspace import ( CellProfilerPlateWorkspacePreparer, @@ -84,7 +87,7 @@ def combine(object_labels): NamedSourceBinding( alias=source.name, artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ), ) diff --git a/tests/unit/test_cellprofiler_export_to_database.py b/tests/unit/test_cellprofiler_export_to_database.py index ee22800df2..f654d25e42 100644 --- a/tests/unit/test_cellprofiler_export_to_database.py +++ b/tests/unit/test_cellprofiler_export_to_database.py @@ -68,7 +68,7 @@ from openhcs.core.source_bindings import ( CompiledSourceBindingPlan, NamedSourceBinding, - SourceProjectionRole, + SourceArtifactRole, ) from openhcs.core.source_matching import SourceImageSetIdentityPolicy from openhcs.interop.cellprofiler.analyst_export import ( @@ -665,7 +665,7 @@ def test_official30_database_modules_bind_and_build_exact_contracts( bindings=tuple( NamedSourceBinding( alias=name, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ) for name in source_names ), diff --git a/tests/unit/test_cellprofiler_generated_pipeline_execution.py b/tests/unit/test_cellprofiler_generated_pipeline_execution.py index 23043add2a..860197ecc1 100644 --- a/tests/unit/test_cellprofiler_generated_pipeline_execution.py +++ b/tests/unit/test_cellprofiler_generated_pipeline_execution.py @@ -57,10 +57,10 @@ from openhcs.core.source_bindings import ( ComponentSelector, NamedSourceBinding, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, + TextContains, ) from openhcs.core.steps.function_step import FunctionStep from openhcs.interop.cellprofiler.module_declarations import CellProfilerModule @@ -163,8 +163,8 @@ def test_compiler_derives_runtime_executor_after_generic_transport( selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="OrigStain1", ), ) diff --git a/tests/unit/test_cellprofiler_generic_special_input_binding.py b/tests/unit/test_cellprofiler_generic_special_input_binding.py index b7a96dc119..18cc75e33c 100644 --- a/tests/unit/test_cellprofiler_generic_special_input_binding.py +++ b/tests/unit/test_cellprofiler_generic_special_input_binding.py @@ -34,7 +34,10 @@ ) from openhcs.core.pipeline.path_planner import PathPlanner, PathPlannerArtifactStage from openhcs.core.runtime_object_labels import ObjectLabelValue -from openhcs.core.source_bindings import NamedSourceBinding, SourceProjectionRole +from openhcs.core.source_bindings import ( + NamedSourceBinding, + SourceArtifactRole, +) from openhcs.core.config import StepSourceBindingsConfig from openhcs.core.steps.function_step import FunctionStep from openhcs.interop.cellprofiler.module_declarations import CellProfilerModule @@ -57,7 +60,7 @@ def _source_binding(alias: str, artifact_type) -> NamedSourceBinding: return NamedSourceBinding( alias, artifact_kind=artifact_type, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ) diff --git a/tests/unit/test_cellprofiler_morphology_public_contracts.py b/tests/unit/test_cellprofiler_morphology_public_contracts.py index 1292539ac6..4aae4f172b 100644 --- a/tests/unit/test_cellprofiler_morphology_public_contracts.py +++ b/tests/unit/test_cellprofiler_morphology_public_contracts.py @@ -37,7 +37,10 @@ from openhcs.core.pipeline.compilation_session import CompilationSession from openhcs.core.pipeline.artifact_planning import artifact_producers_for_outputs from openhcs.core.pipeline.function_contracts import artifact_outputs -from openhcs.core.source_bindings import NamedSourceBinding, SourceProjectionRole +from openhcs.core.source_bindings import ( + NamedSourceBinding, + SourceArtifactRole, +) from openhcs.core.steps.function_step import FunctionStep from openhcs.core.memory.decorators import numpy from openhcs.core.runtime_object_labels import ObjectLabelValue @@ -77,7 +80,7 @@ def _objects(name: str) -> NamedSourceBinding: return NamedSourceBinding( alias=name, artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ) diff --git a/tests/unit/test_cellprofiler_neighbors_contract.py b/tests/unit/test_cellprofiler_neighbors_contract.py index 9c0cb36c99..387fe6d706 100644 --- a/tests/unit/test_cellprofiler_neighbors_contract.py +++ b/tests/unit/test_cellprofiler_neighbors_contract.py @@ -42,7 +42,10 @@ ObjectLabelVariantData, ) from openhcs.core.runtime_relationships import ObjectRelationshipDeclaration -from openhcs.core.source_bindings import NamedSourceBinding, SourceProjectionRole +from openhcs.core.source_bindings import ( + NamedSourceBinding, + SourceArtifactRole, +) from openhcs.core.steps.function_step import FunctionStep from openhcs.interop.cellprofiler.compile_time_contracts import ( CellProfilerInvocationContractProviderFactory, @@ -67,7 +70,7 @@ def _neighbor_step() -> FunctionStep: NamedSourceBinding( alias="Cells", artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ), ) diff --git a/tests/unit/test_cellprofiler_pipeline_import.py b/tests/unit/test_cellprofiler_pipeline_import.py index 112fd56c90..c9aa0b2e1e 100644 --- a/tests/unit/test_cellprofiler_pipeline_import.py +++ b/tests/unit/test_cellprofiler_pipeline_import.py @@ -42,7 +42,7 @@ ComponentSelector, NamedSourceBinding, PairByOrder, - SourceProjectionRole, + PrimaryPlaneRole, ) from openhcs.core.steps.function_step import FunctionStep from openhcs.core.vfs_protocol import FileManagerLike @@ -1783,7 +1783,7 @@ def test_save_images_target_preserves_cross_group_source_and_main_flow() -> None "ColorLung" ) assert color_lung_binding is not None - assert color_lung_binding.projection_role is SourceProjectionRole.PRIMARY_PLANE + assert color_lung_binding.projection_role is PrimaryPlaneRole save_invocation = next( normalize_function_pattern(pipeline_steps[2].func).iter_items() ) diff --git a/tests/unit/test_cellprofiler_runtime_adapter.py b/tests/unit/test_cellprofiler_runtime_adapter.py index c05639c2c2..7776dd6294 100644 --- a/tests/unit/test_cellprofiler_runtime_adapter.py +++ b/tests/unit/test_cellprofiler_runtime_adapter.py @@ -109,13 +109,13 @@ ) from openhcs.core.source_bindings import ( CompiledSourceBindingPlan, + FromPipelineStart, NamedSourceBinding, - SourceBindingOrigin, + SourceArtifactRole, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, - SourceProjectionRole, SourceSelector, + TextContains, ) from openhcs.core.source_image_provenance import ( SourceImageProvenance, @@ -679,13 +679,13 @@ def _pipeline_start_contains_binding(alias): selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, alias, ), ) ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ) @@ -4059,7 +4059,7 @@ def test_cellprofiler_adapter_allows_measurements_for_source_bound_objects(): NamedSourceBinding( alias=NUCLEI, artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ) ) diff --git a/tests/unit/test_cellprofiler_save_images.py b/tests/unit/test_cellprofiler_save_images.py index b82f775b25..756d287762 100644 --- a/tests/unit/test_cellprofiler_save_images.py +++ b/tests/unit/test_cellprofiler_save_images.py @@ -44,7 +44,10 @@ RuntimePlaneAxis, RuntimePlaneAxisValueProjection, ) -from openhcs.core.source_bindings import NamedSourceBinding, SourceProjectionRole +from openhcs.core.source_bindings import ( + NamedSourceBinding, + SourceArtifactRole, +) from openhcs.core.config import StepSourceBindingsConfig from openhcs.core.source_image_provenance import SourceImageProvenancePlanes from openhcs.interop.cellprofiler.module_declarations import ( @@ -108,7 +111,7 @@ def _contract( bindings=tuple( NamedSourceBinding( alias=name, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ) for name in source_names ), @@ -286,7 +289,7 @@ def test_save_images_public_contract_selects_upstream_runtime_image() -> None: bindings=( NamedSourceBinding( alias="OrigBlue", - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ), ), diff --git a/tests/unit/test_cellprofiler_source_bindings.py b/tests/unit/test_cellprofiler_source_bindings.py index 773fea0425..0def3b4d51 100644 --- a/tests/unit/test_cellprofiler_source_bindings.py +++ b/tests/unit/test_cellprofiler_source_bindings.py @@ -31,18 +31,23 @@ from openhcs.core.source_image_provenance import SourceImageIdentity from openhcs.core.source_matching import source_filters_match from openhcs.core.source_bindings import ( + BroadcastSource, ComponentSelector, + FromFileName, + FromPipelineStart, ImagePlaneSource, - MetadataSource, + MatchedSource, NamedSourceBinding, - SourceBindingDeclarationsMixin, PairByMetadata, - SourceBindingOrigin, + PathIsImage, + PrimaryPlaneRole, + SourceArtifactRole, + SourceBindingDeclarationsMixin, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, - SourceProjectionRole, - SourceSetRole, + TextContains, + TextContainsRegex, + TextDoesNotContain, ) from openhcs.interop.cellprofiler.module_declarations import CellProfilerModule from openhcs.interop.cellprofiler.parser import ( @@ -218,7 +223,7 @@ def test_complete_lazy_setup_facts_pycodify_and_reconstruct_in_fresh_process() - alias="Mask", artifact_kind=ObjectLabelsArtifactType, required=False, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, load_as_mask=True, source_channel_axis=-1, source_channel_counts=frozenset({3}), @@ -375,9 +380,9 @@ def test_images_setup_preserves_pipeline_owned_embedded_planes() -> None: ), ) - assert config.source_filters[0].match_type is SourceFilterMatchType.IS_IMAGE - assert config.source_filters[1].subject is SourceFilterSubject.FILE - assert config.source_filters[1].match_type is SourceFilterMatchType.CONTAINS_REGEX + assert config.source_filters[0].match_type is PathIsImage + assert config.source_filters[1].subject is SourceFileName + assert config.source_filters[1].match_type is TextContainsRegex assert config.source_filters[1].value == "A01" assert config.image_plane_sources[0].uri == "/data/A01.npy" assert config.image_plane_sources[0].index == "2" @@ -446,8 +451,8 @@ def test_names_and_types_keeps_metadata_presence_out_of_path_filters() -> None: assert config.bindings[0].selector.filters == ( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "dapi", ), ) @@ -480,7 +485,7 @@ def test_metadata_setup_contributes_path_rules_and_imported_tables() -> None: config = _fold_setup_modules(metadata) assert len(config.metadata_rules) == 1 - assert config.metadata_rules[0].source is MetadataSource.FILE_NAME + assert config.metadata_rules[0].source is FromFileName assert "Well" in config.metadata_rules[0].pattern assert config.metadata_rules[0].filters[0].value == "DNA" assert config.imported_metadata_tables[0].location == "metadata/plate.csv" @@ -720,13 +725,13 @@ def test_names_and_types_contributes_typed_bindings_and_metadata_match_plan() -> illumination = config.binding_for_alias("DAPIillum") assert dapi.artifact_kind is ImageArtifactType - assert dapi.origin is SourceBindingOrigin.PIPELINE_START + assert dapi.origin is FromPipelineStart assert dapi.selector.metadata[0].field == "channel" assert dapi.selector.metadata[0].value == "1" assert illumination.artifact_kind is ImageArtifactType - assert illumination.origin is SourceBindingOrigin.PIPELINE_START - assert illumination.source_set_role is SourceSetRole.MATCHED - assert illumination.projection_role is SourceProjectionRole.SOURCE_ARTIFACT + assert illumination.origin is FromPipelineStart + assert illumination.source_set_role is MatchedSource + assert illumination.projection_role is SourceArtifactRole assert isinstance(config.match_plan, PairByMetadata) assert config.match_plan.dimensions[0].field_for_alias("DAPI") == "folder" assert ( @@ -755,10 +760,10 @@ def test_names_and_types_parses_declared_single_image_as_broadcast_member() -> N dapi = config.binding_for_alias("DAPI") flatfield = config.binding_for_alias("Flatfield") - assert dapi.source_set_role is SourceSetRole.MATCHED - assert dapi.projection_role is SourceProjectionRole.PRIMARY_PLANE - assert flatfield.source_set_role is SourceSetRole.BROADCAST - assert flatfield.projection_role is SourceProjectionRole.SOURCE_ARTIFACT + assert dapi.source_set_role is MatchedSource + assert dapi.projection_role is PrimaryPlaneRole + assert flatfield.source_set_role is BroadcastSource + assert flatfield.projection_role is SourceArtifactRole assert flatfield.explicit_source == ImagePlaneSource( uri="file:/data/flatfield.tif", series="2", @@ -869,8 +874,8 @@ def test_names_and_types_contributes_payload_loading_semantics( assert mask.artifact_kind is ImageArtifactType assert mask.load_as_mask is True assert objects.artifact_kind is ObjectLabelsArtifactType - assert objects.source_set_role is SourceSetRole.MATCHED - assert objects.projection_role is SourceProjectionRole.SOURCE_ARTIFACT + assert objects.source_set_role is MatchedSource + assert objects.projection_role is SourceArtifactRole def test_monochrome_source_normalizes_before_collapsing_rgb_channels() -> None: @@ -997,15 +1002,15 @@ def test_load_images_contributes_binding_filters_metadata_and_grouping() -> None config = _fold_setup_modules(module) binding = config.binding_for_alias("DNA") - assert binding.origin is SourceBindingOrigin.PIPELINE_START - assert binding.selector.filters[0].subject is SourceFilterSubject.FILE - assert binding.selector.filters[0].match_type is SourceFilterMatchType.CONTAINS + assert binding.origin is FromPipelineStart + assert binding.selector.filters[0].subject is SourceFileName + assert binding.selector.filters[0].match_type is TextContains assert binding.selector.filters[0].value == "Channel2" assert ( - binding.selector.filters[1].match_type is SourceFilterMatchType.DOES_NOT_CONTAIN + binding.selector.filters[1].match_type is TextDoesNotContain ) assert binding.selector.filters[1].value == "ILLUM" - assert config.metadata_rules[0].source is MetadataSource.FILE_NAME + assert config.metadata_rules[0].source is FromFileName assert config.metadata_rules[0].pattern == ( r"^.*-(?P.+)-(?P[0-9]{2})" ) @@ -1056,7 +1061,7 @@ def test_setup_fold_is_source_ordered_and_uses_module_registry() -> None: config = _fold_setup_modules(images, names, groups) assert tuple(binding.alias for binding in config.bindings) == ("DNA",) - assert config.source_filters[0].match_type is SourceFilterMatchType.IS_IMAGE + assert config.source_filters[0].match_type is PathIsImage assert config.grouping_metadata_fields == ("Well",) assert all( CellProfilerModule.require_module(module.name).emits_function_step() is False diff --git a/tests/unit/test_cellprofiler_source_bindings_ingestion.py b/tests/unit/test_cellprofiler_source_bindings_ingestion.py index 2d70c7f91e..31eff15a10 100644 --- a/tests/unit/test_cellprofiler_source_bindings_ingestion.py +++ b/tests/unit/test_cellprofiler_source_bindings_ingestion.py @@ -17,14 +17,16 @@ from openhcs.core.source_projection import SourceCandidate from openhcs.core.source_bindings import ( ComponentSelector, + FromFileName, MetadataExtractionRule, - MetadataSource, NamedSourceBinding, PairByOrder, + PathIsImage, + SourceExtension, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, + TextContains, ) from openhcs.core.config import SourceBindingsConfig from openhcs.core.axes import Axis @@ -42,7 +44,7 @@ def _two_channel_config( return SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=( r"^(?P[A-H][0-9]{2})_s(?P[0-9]+)_" r"(?P[^.]+)" ), @@ -51,8 +53,8 @@ def _two_channel_config( match_plan=PairByOrder(), source_filters=( SourceFilterClause( - subject=SourceFilterSubject.EXTENSION, - match_type=SourceFilterMatchType.IS_IMAGE, + subject=SourceExtension, + match_type=PathIsImage, ), ), bindings=( @@ -61,8 +63,8 @@ def _two_channel_config( selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="DNA", ), ), @@ -74,8 +76,8 @@ def _two_channel_config( selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="RNA", ), ), diff --git a/tests/unit/test_compilation_session.py b/tests/unit/test_compilation_session.py index 4ddb4c0122..076da849da 100644 --- a/tests/unit/test_compilation_session.py +++ b/tests/unit/test_compilation_session.py @@ -45,8 +45,8 @@ CompiledSourceBindingPlan, CompiledSourceUniversePlan, ComponentSelector, + FromFileName, MetadataExtractionRule, - MetadataSource, NamedSourceBinding, PairByOrder, SourceSelector, @@ -506,7 +506,7 @@ def test_compiler_keeps_variable_components_as_stack_source(): def test_path_planner_source_binding_plan_comes_from_objectstate_snapshot(): ObjectStateRegistry.clear() metadata_rule = MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?P[A-H]\d{2})\.tif", ) match_plan = PairByOrder() diff --git a/tests/unit/test_function_io.py b/tests/unit/test_function_io.py index 332f035c90..43a81e039d 100644 --- a/tests/unit/test_function_io.py +++ b/tests/unit/test_function_io.py @@ -21,11 +21,11 @@ CompiledSourceUniversePlan, NamedSourceBinding, PairByOrder, + SourceArtifactRole, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, - SourceProjectionRole, SourceSelector, + TextEquals, ) from openhcs.core.config import SourceBindingsConfig from openhcs.core.source_workspace_projection import VirtualWorkspaceSourceProjection @@ -254,8 +254,8 @@ def test_bulk_preload_preserves_nested_virtual_workspace_paths(tmp_path: Path) - selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, source_paths[0].name, ), ), @@ -264,12 +264,12 @@ def test_bulk_preload_preserves_nested_virtual_workspace_paths(tmp_path: Path) - NamedSourceBinding( alias="ObjectLabels", artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, source_paths[1].name, ), ), @@ -331,12 +331,12 @@ def test_object_only_source_anchors_match_compiled_pattern_after_loading( NamedSourceBinding( alias=alias, artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, source_path.name, ), ), @@ -400,12 +400,12 @@ def test_virtual_pipeline_source_universe_does_not_mix_physical_paths( NamedSourceBinding( alias=alias, artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, source_path.name, ), ) @@ -486,8 +486,8 @@ def test_bulk_preload_loads_matlab_pixels_through_declared_vfs_backend( selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, path.name, ), ), diff --git a/tests/unit/test_function_output_manifest_input_scope.py b/tests/unit/test_function_output_manifest_input_scope.py index 51a2b10476..0e629ba66b 100644 --- a/tests/unit/test_function_output_manifest_input_scope.py +++ b/tests/unit/test_function_output_manifest_input_scope.py @@ -617,8 +617,13 @@ def test_same_scope_parameter_bound_input_does_not_select_lifecycle_output( def test_named_producer_members_keep_acquisition_filters_and_site_correlations(): from openhcs.core.source_bindings import ( - CompiledSourceBindingPlan, ComponentSelector, NamedSourceBinding, - SourceFilterClause, SourceFilterMatchType, SourceFilterSubject, SourceSelector, + CompiledSourceBindingPlan, + ComponentSelector, + NamedSourceBinding, + SourceFileName, + SourceFilterClause, + SourceSelector, + TextContains, ) from openhcs.core.source_matching import SourceImageSetIdentityPolicy from openhcs.core.source_metadata import SourceFilterPathMetadata @@ -652,7 +657,7 @@ def test_named_producer_members_keep_acquisition_filters_and_site_correlations() bindings = CompiledSourceBindingPlan(bindings=(NamedSourceBinding( alias="OrigStain1", selector=SourceSelector(filters=(SourceFilterClause( - SourceFilterSubject.FILE, SourceFilterMatchType.CONTAINS, "N_R", + SourceFileName, TextContains, "N_R", ),)), component_identity=(ComponentSelector(Microscopy.Channel, "1"),), ),)) @@ -666,7 +671,7 @@ def test_named_producer_members_keep_acquisition_filters_and_site_correlations() assert tuple(record.component_values["site"] for record in selected) == (1, 2) wrong_binding = replace(bindings, bindings=(replace( bindings.bindings[0], selector=SourceSelector(filters=(SourceFilterClause( - SourceFilterSubject.FILE, SourceFilterMatchType.CONTAINS, "absent", + SourceFileName, TextContains, "absent", ),)), ),)) assert index.source_binding_members( diff --git a/tests/unit/test_function_runtime_source_projection.py b/tests/unit/test_function_runtime_source_projection.py index 3334140ba4..610ad45624 100644 --- a/tests/unit/test_function_runtime_source_projection.py +++ b/tests/unit/test_function_runtime_source_projection.py @@ -68,17 +68,19 @@ from openhcs.core.runtime_pattern_cache import RuntimePatternDiscoveryCache from openhcs.core.steps.function_output_manifest import _STEP_OUTPUT_MANIFESTS from openhcs.core.source_bindings import ( - SOURCE_BINDING_ALIAS_METADATA_FIELD, CompiledSourceBindingPlan, ComponentSelector, + FromPipelineStart, NamedSourceBinding, PairByOrder, - SourceBindingOrigin, + PrimaryPlaneRole, + SOURCE_BINDING_ALIAS_METADATA_FIELD, + SourceArtifactRole, + SourceExtension, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, - SourceProjectionRole, SourceSelector, + TextContains, ) from openhcs.core.source_image_provenance import ( SourceImageProvenance, @@ -319,11 +321,11 @@ def test_workspace_source_files_select_exact_projection_roles( workspace_root=str(plate_path), ) - assert projection.files_for_projection_role(SourceProjectionRole.PRIMARY_PLANE) == ( + assert projection.files_for_projection_role(PrimaryPlaneRole) == ( str(plate_path / canonical_path), ) assert projection.files_for_projection_role( - SourceProjectionRole.SOURCE_ARTIFACT + SourceArtifactRole ) == (str(plate_path / artifact_path),) @@ -1693,7 +1695,7 @@ def test_source_bound_anchor_filter_combines_ordered_non_grouped_source_sets() - selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, "1"),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, "1"),), ), NamedSourceBinding( @@ -1701,7 +1703,7 @@ def test_source_bound_anchor_filter_combines_ordered_non_grouped_source_sets() - selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, "2"),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, "2"),), ), ) @@ -1887,7 +1889,7 @@ def save_image(image, *, image_to_save: np.ndarray): selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, "1"),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, "1"),), ), NamedSourceBinding( @@ -1895,7 +1897,7 @@ def save_image(image, *, image_to_save: np.ndarray): selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, "3"),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, "3"),), ), ), @@ -1953,7 +1955,7 @@ def exact_source_input(image): selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, "1"),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, "1"),), ), ), @@ -2007,7 +2009,7 @@ def exact_source_input(image): selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, "1"),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, "1"),), ), ), @@ -2052,7 +2054,7 @@ def test_pipeline_start_anchors_project_raw_selectors_to_semantic_groups() -> No selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, "1"),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, "MCP_DNA"),), ), NamedSourceBinding( @@ -2060,7 +2062,7 @@ def test_pipeline_start_anchors_project_raw_selectors_to_semantic_groups() -> No selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, "2"),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, "MCP_AGP"),), ), ) @@ -2227,7 +2229,7 @@ def source_bound_module(image, *, runtime): selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, "1"),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, "1"),), ), ), @@ -2819,7 +2821,7 @@ def test_main_flow_input_owns_runtime_binding_scope_with_auxiliary_source() -> N NamedSourceBinding( alias="OrigStain1", component_identity=(ComponentSelector(Microscopy.Channel, "1"),), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), NamedSourceBinding( alias="OrigStain2", @@ -3634,8 +3636,8 @@ def test_grouped_runtime_source_expansion_uses_scoped_bindings( selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "N_R", ), ) @@ -3647,8 +3649,8 @@ def test_grouped_runtime_source_expansion_uses_scoped_bindings( selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "N_G", ), ) @@ -3811,7 +3813,7 @@ def test_unbound_workspace_source_keeps_filename_component_provenance( Microscopy.ZIndex.name: 3, Microscopy.Timepoint.name: 4, Microscopy.Well.name: "A01", - SourceFilterSubject.EXTENSION.value: ".tif", + SourceExtension.name: ".tif", } projection = VirtualWorkspaceSourceProjection( source_refs_by_virtual_path={virtual_path: source_ref}, @@ -3879,7 +3881,7 @@ def resolve_address(backend_address, backend, *, base_path): Microscopy.ZIndex.name: 3, Microscopy.Timepoint.name: 4, Microscopy.Well.name: "A01", - SourceFilterSubject.EXTENSION.value: ".tif", + SourceExtension.name: ".tif", } assert projected.metadata.source_provenance.represented_source_image_names == ("OrigDNA",) diff --git a/tests/unit/test_function_step_transport.py b/tests/unit/test_function_step_transport.py index 6f867571a5..b8771481c7 100644 --- a/tests/unit/test_function_step_transport.py +++ b/tests/unit/test_function_step_transport.py @@ -29,7 +29,10 @@ FunctionReferenceTransportAuthority, ) from openhcs.core.function_step_transport import FunctionStepTransportAuthority -from openhcs.core.source_bindings import NamedSourceBinding, SourceBindingOrigin +from openhcs.core.source_bindings import ( + FromPipelineStart, + NamedSourceBinding, +) from openhcs.core.steps.function_step import FunctionStep from openhcs.processing.backends import cellprofiler as cellprofiler_backend from openhcs.processing.backends.lib_registry.registry_service import RegistryService @@ -285,7 +288,7 @@ def test_function_step_source_emits_source_bindings_once() -> None: bindings=( NamedSourceBinding( alias="OrigBlue", - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), ), ) diff --git a/tests/unit/test_invocation_contract_provider.py b/tests/unit/test_invocation_contract_provider.py index 47f83eaa31..bf2a06b571 100644 --- a/tests/unit/test_invocation_contract_provider.py +++ b/tests/unit/test_invocation_contract_provider.py @@ -1226,7 +1226,7 @@ def test_calculate_math_provider_keeps_object_identity_and_output_name_public() from openhcs.core.config import GlobalPipelineConfig, PipelineConfig from openhcs.core.context.processing_context import ProcessingContext from openhcs.core.pipeline.compilation_session import CompilationSession - from openhcs.core.source_bindings import NamedSourceBinding, SourceProjectionRole + from openhcs.core.source_bindings import NamedSourceBinding, SourceArtifactRole from openhcs.core.config import StepSourceBindingsConfig from openhcs.core.steps.function_step import FunctionStep from openhcs.interop.cellprofiler.compile_time_contracts import ( @@ -1250,7 +1250,7 @@ def test_calculate_math_provider_keeps_object_identity_and_output_name_public() NamedSourceBinding( alias="Nuclei", artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ), ), diff --git a/tests/unit/test_invocation_input_source_context_identity.py b/tests/unit/test_invocation_input_source_context_identity.py index afee352f86..55c81141fa 100644 --- a/tests/unit/test_invocation_input_source_context_identity.py +++ b/tests/unit/test_invocation_input_source_context_identity.py @@ -35,7 +35,8 @@ from openhcs.core.source_bindings import ( CompiledSourceBindingPlan, NamedSourceBinding, - SourceProjectionRole, + PrimaryPlaneRole, + SourceArtifactRole, ) from openhcs.interop.cellprofiler.runtime.artifact_binding import ( RuntimeInputBindingRequest, @@ -193,7 +194,7 @@ def test_named_source_binding_owns_unstored_input_over_main_flow_projection( bindings=( NamedSourceBinding( alias=plate_template.name, - projection_role=SourceProjectionRole.PRIMARY_PLANE, + projection_role=PrimaryPlaneRole, ), ) ) @@ -327,7 +328,7 @@ def test_compiled_producer_edge_overrides_matching_source_binding() -> None: bindings=( NamedSourceBinding( alias=combined_image.name, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ), ), diff --git a/tests/unit/test_loaded_pattern_group_ownership.py b/tests/unit/test_loaded_pattern_group_ownership.py index cebb2169d7..459455f3b4 100644 --- a/tests/unit/test_loaded_pattern_group_ownership.py +++ b/tests/unit/test_loaded_pattern_group_ownership.py @@ -13,7 +13,10 @@ from openhcs.core.context.processing_context import ProcessingContext from openhcs.core.function_patterns import compile_function_pattern from openhcs.core.runtime_image_values import ImagePayloadMetadata -from openhcs.core.source_bindings import CompiledSourceBindingPlan +from openhcs.core.source_bindings import ( + CompiledSourceBindingPlan, + FromPipelineStart, +) from openhcs.core.source_image_provenance import SourceImageProvenancePlanes from openhcs.core.runtime_profile import RuntimeProfileLogger from openhcs.core.steps import function_runtime @@ -774,11 +777,11 @@ def test_initial_source_input_uses_loaded_declared_cohort_without_workspace_read from openhcs.core.pipeline.function_contracts import artifact_inputs from openhcs.core.memory.decorators import numpy as numpy_memory from openhcs.core.runtime_plane_projection import RuntimePlaneAxis, RuntimePlaneProjection - from openhcs.core.source_bindings import NamedSourceBinding, SourceBindingOrigin + from openhcs.core.source_bindings import NamedSourceBinding from openhcs.core.step_dependencies import StepInputDependency request, paths, payload = _fixture() - binding = NamedSourceBinding(alias="Raw", origin=SourceBindingOrigin.PIPELINE_START) + binding = NamedSourceBinding(alias="Raw", origin=FromPipelineStart) source = binding.input_spec() @artifact_inputs(source) diff --git a/tests/unit/test_microscope_virtual_workspace_metadata.py b/tests/unit/test_microscope_virtual_workspace_metadata.py index ecd9aaea33..e4648e7ecd 100644 --- a/tests/unit/test_microscope_virtual_workspace_metadata.py +++ b/tests/unit/test_microscope_virtual_workspace_metadata.py @@ -71,9 +71,9 @@ def test_compiler_source_queries_borrow_config_while_public_queries_stay_live( from openhcs.core.config import GlobalPipelineConfig, PipelineConfig from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.source_bindings import ( + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, + TextContains, ) from openhcs.core.config import SourceBindingsConfig, LazySourceBindingsConfig @@ -101,12 +101,12 @@ def test_compiler_source_queries_borrow_config_while_public_queries_stay_live( metadata_path.write_text(json.dumps(metadata_document)) source_a = SourceBindingsConfig(source_filters=( SourceFilterClause( - SourceFilterSubject.FILE, SourceFilterMatchType.CONTAINS, "source-a" + SourceFileName, TextContains, "source-a" ), )) source_b = SourceBindingsConfig(source_filters=( SourceFilterClause( - SourceFilterSubject.FILE, SourceFilterMatchType.CONTAINS, "source-b" + SourceFileName, TextContains, "source-b" ), )) held_config = GlobalPipelineConfig(source_bindings_config=source_a) diff --git a/tests/unit/test_path_planner_materialization.py b/tests/unit/test_path_planner_materialization.py index a8989d066d..6c073d0415 100644 --- a/tests/unit/test_path_planner_materialization.py +++ b/tests/unit/test_path_planner_materialization.py @@ -104,9 +104,9 @@ CompiledSourceBindingPlan, CompiledSourceUniversePlan, ComponentSelector, + FromPipelineStart, NamedSourceBinding, - SourceBindingOrigin, - SourceProjectionRole, + SourceArtifactRole, ) from openhcs.core.step_dependencies import StepInputDependency from openhcs.core.step_dependencies import StepInputDependencyKind @@ -2664,7 +2664,7 @@ def measure_green(image, *, runtime): bindings=tuple( NamedSourceBinding( alias=name, - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, channel),), ) for name, channel in ((blue.name, "1"), (green.name, "2")) @@ -4154,7 +4154,7 @@ def test_main_flow_source_anchor_restricts_execution_to_its_exact_channel(): NamedSourceBinding( alias="DF_image", component_identity=(ComponentSelector(Microscopy.Channel, "2"),), - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ), ) @@ -4315,7 +4315,7 @@ def test_execution_anchor_ignores_source_artifact_lineage(): NamedSourceBinding( alias="OrigRed", component_identity=(ComponentSelector(Microscopy.Channel, "3"),), - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ), ) diff --git a/tests/unit/test_plane_store_sources.py b/tests/unit/test_plane_store_sources.py index 3335f872c3..4579ad3c6b 100644 --- a/tests/unit/test_plane_store_sources.py +++ b/tests/unit/test_plane_store_sources.py @@ -32,10 +32,10 @@ from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.source_bindings import ( NamedSourceBinding, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, + TextEquals, ) from openhcs.core.steps.function_io import ( save_materialized_data, @@ -199,8 +199,8 @@ def _binding(alias: str, file_name: str) -> NamedSourceBinding: selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, file_name, ), ), diff --git a/tests/unit/test_runtime_adapter_source_artifacts.py b/tests/unit/test_runtime_adapter_source_artifacts.py index 578f14b7a5..8bb0c2817c 100644 --- a/tests/unit/test_runtime_adapter_source_artifacts.py +++ b/tests/unit/test_runtime_adapter_source_artifacts.py @@ -20,11 +20,12 @@ from openhcs.core.source_bindings import ( NamedSourceBinding, PairByOrder, + PrimaryPlaneRole, + SourceArtifactRole, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, - SourceProjectionRole, SourceSelector, + TextEquals, ) from openhcs.core.config import SourceBindingsConfig, StepSourceBindingsConfig from openhcs.core.source_image_provenance import SourceImageIdentity @@ -75,7 +76,7 @@ def binding( source_path: Path, *, artifact_kind=ImageArtifactType, - projection_role=SourceProjectionRole.PRIMARY_PLANE, + projection_role=PrimaryPlaneRole, ) -> NamedSourceBinding: return NamedSourceBinding( alias=alias, @@ -84,8 +85,8 @@ def binding( selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, source_path.name, ), ) @@ -96,13 +97,13 @@ def binding( illumination_binding = binding( "Illumination", illumination_path, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ) labels_binding = binding( "Labels", labels_path, artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ) source_bindings = SourceBindingsConfig( bindings=( diff --git a/tests/unit/test_runtime_source_resolution_snapshot.py b/tests/unit/test_runtime_source_resolution_snapshot.py index 87198f3581..82ce985496 100644 --- a/tests/unit/test_runtime_source_resolution_snapshot.py +++ b/tests/unit/test_runtime_source_resolution_snapshot.py @@ -14,8 +14,8 @@ SourcePatternResolutionContext, ) from openhcs.core.source_bindings import ( + FromFileName, MetadataExtractionRule, - MetadataSource, PairByOrder, ) from openhcs.core.source_matching import SourceImageSetIdentityPolicy @@ -181,7 +181,7 @@ def test_snapshot_replacement_tracks_projection_parser_semantics_and_rules(): assert replaced_parser.metadata_for_path(PATH)["site"] == 1 rules = ( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, pattern=r"(?PA01)" + source=FromFileName, pattern=r"(?PA01)" ), ) changed_rules = snapshot(cache, source_projection, parser, rules) @@ -347,7 +347,7 @@ def test_durable_workspace_mapping_still_runs_selector_parser_and_rule_fallbacks source_metadata_by_path={PATH: metadata}, metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, pattern=r"(?PA01)" + source=FromFileName, pattern=r"(?PA01)" ), ), ) diff --git a/tests/unit/test_runtime_values.py b/tests/unit/test_runtime_values.py index ef7e293a2f..eff0d0d02c 100644 --- a/tests/unit/test_runtime_values.py +++ b/tests/unit/test_runtime_values.py @@ -109,7 +109,10 @@ from openhcs.core.runtime_tabular_values import ( ColumnarRows, ) -from openhcs.core.source_bindings import NamedSourceBinding, SourceProjectionRole +from openhcs.core.source_bindings import ( + NamedSourceBinding, + SourceArtifactRole, +) from openhcs.core.source_image_provenance import ( SourceImageIdentity, SourceImageProvenancePlanes, @@ -2362,7 +2365,7 @@ def test_object_label_source_image_semantics_treats_rgb_image_as_label_plane() - payload = NamedSourceBinding( alias="Objects", artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, source_channel_axis=-1, source_channel_counts=frozenset((3, 4)), ).apply_loaded_payload( diff --git a/tests/unit/test_source_binding_path_admission.py b/tests/unit/test_source_binding_path_admission.py index 0f5dd5f13c..5c229c0b0c 100644 --- a/tests/unit/test_source_binding_path_admission.py +++ b/tests/unit/test_source_binding_path_admission.py @@ -12,10 +12,12 @@ ImagePlaneSource, MetadataSelector, NamedSourceBinding, + SourceDirectory, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, + TextContainsRegex, + TextEquals, ) from openhcs.core.config import SourceBindingsConfig from openhcs.core.source_projection import SourceCandidate @@ -74,19 +76,19 @@ def test_binding_directory_regex_and_or_group_use_the_same_owner( selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.DIRECTORY, - SourceFilterMatchType.CONTAINS_REGEX, + SourceDirectory, + TextContainsRegex, r"(?:^|/)selected$", ), SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, "a.czi", any_group=0, ), SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, "b.czi", any_group=0, ), @@ -141,8 +143,8 @@ def test_decoded_companion_filters_do_not_replace_exact_entrypoint_identity(tmp_ selector = SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, companion.name, ), ) @@ -179,8 +181,8 @@ def test_unrestricted_alias_keeps_union_without_erasing_other_alias_filters(tmp_ selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, "selected.czi", ), ) diff --git a/tests/unit/test_source_binding_workspace.py b/tests/unit/test_source_binding_workspace.py index 03519aa7e3..621768ad4b 100644 --- a/tests/unit/test_source_binding_workspace.py +++ b/tests/unit/test_source_binding_workspace.py @@ -23,23 +23,29 @@ ) from openhcs.core.dataset_sources.interfaces import PixelSizeMetadataArtifactProvider from openhcs.core.source_bindings import ( + BroadcastSource, ComponentSelector, + FromFileName, + FromFolderName, ImagePlaneSource, ImportedMetadataJoin, ImportedMetadataTable, MetadataExtractionRule, - MetadataSource, NamedSourceBinding, - SourceBindingMatchDimension, - SourceBindingMatchField, PairByMetadata, PairByOrder, + PathIsImage, + SourceArtifactRole, + SourceBindingMatchDimension, + SourceBindingMatchField, + SourceExtension, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, - SourceProjectionRole, - SourceSetRole, SourceSelector, + TextContains, + TextEndsWith, + TextEquals, + TextStartsWith, ) from openhcs.core.config import SourceBindingsConfig from openhcs.core.dataset_sources.choice import DatasetSourceChoice @@ -124,7 +130,7 @@ def test_prepared_workspace_admits_declared_source_universe_without_rewriting_pr workspace = tmp_path / "prepared" materialization = SourceBindingWorkspaceProjector( SourceBindingsConfig(metadata_rules=(MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?PA\d+)_s(?P\d+)_w(?P\d+)", ),)), parser=SourceSchemaFilenameParser(), @@ -134,8 +140,8 @@ def test_prepared_workspace_admits_declared_source_universe_without_rewriting_pr (workspace / FIELDS.DEFAULT_SUBDIRECTORY).mkdir(exist_ok=True) original_bytes = materialization.metadata_path.read_bytes() config = SourceBindingsConfig(source_filters=() if selection is None else ( - SourceFilterClause(subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + SourceFilterClause(subject=SourceFileName, + match_type=TextContains, value=f"_s{selection:03}_"), )) handler = DatasetSourceChoice.named("auto").open(workspace, filemanager=filemanager, source_bindings_config=config) @@ -198,8 +204,8 @@ def reject_redecode(_cls, _subdirectory): assert context.runtime_source_workspace_projections.projection_or_empty(axis_id="A01") is compiled_views["A01"] prior_authority = context.runtime_source_workspace_projections handler._source_bindings_config = replace(config, source_filters=( - SourceFilterClause(subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, value="_s008_"), + SourceFilterClause(subject=SourceFileName, + match_type=TextContains, value="_s008_"), )) changed = context.runtime_source_workspace_projections assert changed is not prior_authority @@ -207,8 +213,8 @@ def reject_redecode(_cls, _subdirectory): handler._source_bindings_config = replace(config, source_filters=()) assert context.runtime_source_workspace_projections.projection_or_empty() is full handler._source_bindings_config = replace(config, source_filters=( - SourceFilterClause(subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, value="_s099_"), + SourceFilterClause(subject=SourceFileName, + match_type=TextContains, value="_s099_"), )) with pytest.raises(ValueError, match="matched no prepared workspace sources"): context.runtime_source_workspace_projections.projection_or_empty() @@ -240,8 +246,8 @@ def reject_redecode(_cls, _subdirectory): WorkspaceSourceProjections.from_dataset_metadata( dataset_root=workspace, metadata_handler=handler.metadata_handler, filemanager=filemanager, source_bindings=SourceBindingsConfig(source_filters=( - SourceFilterClause(subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + SourceFilterClause(subject=SourceFileName, + match_type=TextContains, value="_s099_"), )), ).projection_if_available() @@ -394,8 +400,8 @@ def test_source_workspace_excludes_non_pixel_sidecars_before_projection(tmp_path SourceBindingsConfig( source_filters=( SourceFilterClause( - subject=SourceFilterSubject.EXTENSION, - match_type=SourceFilterMatchType.IS_IMAGE, + subject=SourceExtension, + match_type=PathIsImage, ), ), bindings=( @@ -441,7 +447,7 @@ def test_source_binding_workspace_projector_assigns_selector_channels(tmp_path): SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r".*_(?P[A-Z]\d+)_s(?P\d+)\.png", ), ), @@ -451,8 +457,8 @@ def test_source_binding_workspace_projector_assigns_selector_channels(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.STARTS_WITH, + subject=SourceFileName, + match_type=TextStartsWith, value="nuclei_", ), ), @@ -464,8 +470,8 @@ def test_source_binding_workspace_projector_assigns_selector_channels(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.STARTS_WITH, + subject=SourceFileName, + match_type=TextStartsWith, value="membrane_", ), ), @@ -533,7 +539,7 @@ def test_source_binding_workspace_projects_semantic_identity_after_raw_selection SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=( r"^(?P[A-Z][0-9]+)_s(?P[0-9]+)_w" r"(?P[0-9]+)[.]tif$" @@ -581,8 +587,8 @@ def binding(alias: str, channel: str) -> NamedSourceBinding: selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.ENDS_WITH, + SourceFileName, + TextEndsWith, f"_{alias}.tif", ), ), @@ -594,7 +600,7 @@ def binding(alias: str, channel: str) -> NamedSourceBinding: SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^(?P[A-H][0-9]{2})_s" r"(?P[0-9]+)_(?:DNA|RNA)[.]tif$", ), @@ -644,11 +650,11 @@ def test_source_binding_workspace_projects_declared_groups_to_wells(tmp_path): SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"^.*_(?P[0-9]+)\.tif$", ), MetadataExtractionRule( - source=MetadataSource.FOLDER_NAME, + source=FromFolderName, pattern=r".*[\\/](?P[^\\/]+)$", ), ), @@ -658,8 +664,8 @@ def test_source_binding_workspace_projects_declared_groups_to_wells(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="GFP", ), ), @@ -711,7 +717,7 @@ def test_source_binding_workspace_projects_registered_well_parts(tmp_path): SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=(r"^image-(?P[A-Z])-(?P[0-9]{2})\.tif$"), ), ), @@ -745,11 +751,11 @@ def test_group_address_preserves_distinct_source_well_as_literal_metadata(tmp_pa SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"^(?P[A-Z][0-9]+)_.*_(?P[0-9]+)\.tif$", ), MetadataExtractionRule( - source=MetadataSource.FOLDER_NAME, + source=FromFolderName, pattern=r".*[\\/](?P[^\\/]+)$", ), ), @@ -821,8 +827,8 @@ def test_source_binding_workspace_projector_order_matches_aliases(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.STARTS_WITH, + subject=SourceFileName, + match_type=TextStartsWith, value="DAPI_", ), ), @@ -833,8 +839,8 @@ def test_source_binding_workspace_projector_order_matches_aliases(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.STARTS_WITH, + subject=SourceFileName, + match_type=TextStartsWith, value="Actin_", ), ), @@ -872,23 +878,23 @@ def test_order_source_sets_join_imported_metadata_across_aliases(tmp_path): config = SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^DNA_(?P[A-Z][0-9]+)\.tif$", filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.STARTS_WITH, + SourceFileName, + TextStartsWith, "DNA_", ), ), ), MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^Actin_(?P[0-9]+)\.tif$", filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.STARTS_WITH, + SourceFileName, + TextStartsWith, "Actin_", ), ), @@ -900,8 +906,8 @@ def test_order_source_sets_join_imported_metadata_across_aliases(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.STARTS_WITH, + SourceFileName, + TextStartsWith, "DNA_", ), ), @@ -913,8 +919,8 @@ def test_order_source_sets_join_imported_metadata_across_aliases(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.STARTS_WITH, + SourceFileName, + TextStartsWith, "Actin_", ), ), @@ -967,8 +973,8 @@ def test_order_source_sets_preserve_shared_virtual_stack_coordinates(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.STARTS_WITH, + SourceFileName, + TextStartsWith, "DNA", ), ), @@ -980,8 +986,8 @@ def test_order_source_sets_preserve_shared_virtual_stack_coordinates(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.STARTS_WITH, + SourceFileName, + TextStartsWith, "Membrane", ), ), @@ -1021,7 +1027,7 @@ def test_metadata_source_sets_propagate_imported_metadata(tmp_path): config = SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^(?P[A-Z][0-9]+)_(?:DNA|Actin)\.tif$", ), ), @@ -1031,8 +1037,8 @@ def test_metadata_source_sets_propagate_imported_metadata(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.ENDS_WITH, + SourceFileName, + TextEndsWith, f"_{alias}.tif", ), ), @@ -1085,7 +1091,7 @@ def test_imported_metadata_duplicate_join_uses_first_matching_row(tmp_path): config = SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^(?P[0-9]+)_(?P[A-Z][0-9]+)_DNA\.tif$", ), ), @@ -1135,7 +1141,7 @@ def test_imported_metadata_later_stage_overrides_extracted_field(tmp_path): config = SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^(?P[^_]+)_(?P[A-Z][0-9]+)_s(?P[0-9]+)_w(?P[0-9]+)\.tif$", ), ), @@ -1178,7 +1184,7 @@ def test_imported_metadata_coerces_join_and_payload_values_through_declared_type config = SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^(?P[0-9]+)_DNA\.tif$", ), ), @@ -1251,7 +1257,7 @@ def test_imported_metadata_rejects_conflicting_or_unmatched_complete_joins( config = SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^(?P[A-Z][0-9]+)_(?PDNA|Actin)\.tif$", ), ), @@ -1261,8 +1267,8 @@ def test_imported_metadata_rejects_conflicting_or_unmatched_complete_joins( selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.ENDS_WITH, + SourceFileName, + TextEndsWith, f"_{stain}.tif", ), ), @@ -1330,7 +1336,7 @@ def test_imported_metadata_location_is_resolved_exactly(tmp_path): config = SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^(?P[A-Z][0-9]+)_DNA\.tif$", ), ), @@ -1384,7 +1390,7 @@ def test_imported_metadata_rejects_invalid_table_contracts( config = SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^(?P[A-Z][0-9]+)_DNA\.tif$", ), ), @@ -1410,7 +1416,7 @@ def test_source_bindings_reinitialization_resolves_current_imported_metadata(tmp config = SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^(?P[A-Z][0-9]+)_DNA\.tif$", ), ), @@ -1529,7 +1535,7 @@ def test_source_reinitialization_sees_new_physical_files_not_virtual_aliases(tmp bindings=(NamedSourceBinding(alias="Images"),), metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^(?P[A-Z]\d{2})_s(?P\d+)_w(?P\d+)\.tif$", ), ), @@ -1624,8 +1630,8 @@ def test_order_source_sets_pair_expanded_stack_planes_across_aliases(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.STARTS_WITH, + subject=SourceFileName, + match_type=TextStartsWith, value="DNA_", ), ), @@ -1636,8 +1642,8 @@ def test_order_source_sets_pair_expanded_stack_planes_across_aliases(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.STARTS_WITH, + subject=SourceFileName, + match_type=TextStartsWith, value="Membrane_", ), ), @@ -1688,8 +1694,8 @@ def test_source_binding_workspace_projector_order_uses_shortest_member_count( selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.STARTS_WITH, + subject=SourceFileName, + match_type=TextStartsWith, value="DAPI_", ), ), @@ -1700,8 +1706,8 @@ def test_source_binding_workspace_projector_order_uses_shortest_member_count( selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.STARTS_WITH, + subject=SourceFileName, + match_type=TextStartsWith, value="Actin_", ), ), @@ -1745,8 +1751,8 @@ def test_source_binding_workspace_broadcasts_explicit_single_members(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.STARTS_WITH, + SourceFileName, + TextStartsWith, "DAPI_", ), ), @@ -1755,8 +1761,8 @@ def test_source_binding_workspace_broadcasts_explicit_single_members(tmp_path): NamedSourceBinding( alias="Flatfield", artifact_kind=ImageArtifactType, - source_set_role=SourceSetRole.BROADCAST, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + source_set_role=BroadcastSource, + projection_role=SourceArtifactRole, explicit_source=explicit_source, ), ), @@ -1799,7 +1805,7 @@ def test_metadata_source_sets_reuse_declared_partial_match_members(tmp_path): SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"^(?P[A-Z][0-9]+)_s(?P[0-9]+)_DAPI\.tif$", ), ), @@ -1809,8 +1815,8 @@ def test_metadata_source_sets_reuse_declared_partial_match_members(tmp_path): selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.ENDS_WITH, + SourceFileName, + TextEndsWith, "DAPI.tif", ), ), @@ -1819,12 +1825,12 @@ def test_metadata_source_sets_reuse_declared_partial_match_members(tmp_path): NamedSourceBinding( alias="Illumination", artifact_kind=ImageArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, illumination.name, ), ), @@ -1882,12 +1888,12 @@ def test_source_binding_workspace_materializes_artifact_only_matched_groups( NamedSourceBinding( alias=alias, artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.EQUALS, + SourceFileName, + TextEquals, path.name, ), ), @@ -2030,8 +2036,8 @@ def test_source_bindings_handler_materializes_non_stack_source_artifacts(tmp_pat selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="Channel 1", ), ), @@ -2043,13 +2049,13 @@ def test_source_bindings_handler_materializes_non_stack_source_artifacts(tmp_pat selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.EQUALS, + subject=SourceFileName, + match_type=TextEquals, value="illumination.npy", ), ), ), - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ), ), ), diff --git a/tests/unit/test_source_bindings.py b/tests/unit/test_source_bindings.py index c88b097807..96b7a40512 100644 --- a/tests/unit/test_source_bindings.py +++ b/tests/unit/test_source_bindings.py @@ -39,27 +39,29 @@ SourcePatternResolutionContext, ) from openhcs.core.source_bindings import ( + BroadcastSource, CompiledSourceUniversePlan, ComponentSelector, + FromFileName, + FromFolderName, + FromPipelineStart, + FromStepInput, ImagePlaneSource, ImportedMetadataJoin, ImportedMetadataTable, MetadataExtractionRule, MetadataSelector, - MetadataSource, NamedSourceBinding, - SourceBindingMatchDimension, - SourceBindingMatchField, PairByMetadata, PairByOrder, - SourceBindingOrigin, + source_binding_group_keys_for_group_by, + SourceArtifactRole, + SourceBindingMatchDimension, + SourceBindingMatchField, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, - SourceProjectionRole, SourceSelector, - SourceSetRole, - source_binding_group_keys_for_group_by, + TextContains, ) from openhcs.core.source_image_provenance import ( RuntimeSourceImageProvenancePlane, @@ -156,19 +158,19 @@ def test_source_image_set_policy_uses_binding_and_source_stack_declarations(): ) -def test_named_source_binding_normalizes_origin_and_requires_alias(): +def test_named_source_binding_normalizes_artifact_kind_and_requires_alias(): binding = NamedSourceBinding( alias="OrigBlue", - origin="pipeline_start", + origin=FromPipelineStart, ) - assert binding.origin is SourceBindingOrigin.PIPELINE_START + assert binding.origin is FromPipelineStart assert binding.artifact_kind is ImageArtifactType objects_binding = NamedSourceBinding( alias="Nuclei", artifact_kind="object_labels", - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ) assert objects_binding.artifact_kind is ObjectLabelsArtifactType @@ -742,7 +744,7 @@ def test_source_bindings_expose_generic_resolution_requirements(): ), NamedSourceBinding( alias="IllumDNA", - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), ) ) @@ -833,12 +835,12 @@ def test_compiled_source_binding_plan_preserves_named_selectors(): ), metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r".*(?PPlateA)\.tif", filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="PlateA", ), ), @@ -865,7 +867,7 @@ def test_compiled_source_binding_plan_preserves_named_selectors(): assert plan.has_primary_content assert len(plan.bindings) == 1 - assert plan.metadata_rules[0].source is MetadataSource.FILE_NAME + assert plan.metadata_rules[0].source is FromFileName assert isinstance(plan.match_plan, PairByMetadata) binding = plan.bindings[0] assert binding.alias == "OrigBlue" @@ -883,7 +885,7 @@ def test_pipeline_start_binding_does_not_force_full_source_universe(): selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, "1"),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), ), enabled=True, @@ -906,26 +908,26 @@ def test_matched_image_set_without_match_plan_uses_selector_compatible_sources() selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="w1", ), ) ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), NamedSourceBinding( alias="PH3", selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="w2", ), ) ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), ), match_plan=PairByOrder(), @@ -1002,7 +1004,7 @@ def test_source_set_without_matched_bindings_preserves_anchor_candidates() -> No bindings=( NamedSourceBinding( alias="PlateTemplate", - source_set_role=SourceSetRole.BROADCAST, + source_set_role=BroadcastSource, ), ), match_plan=PairByOrder(), @@ -1164,7 +1166,7 @@ def test_source_binding_members_use_exact_source_identity_without_metadata_axes( alias="SavedIPO", component_identity=coordinates, artifact_kind=ObjectLabelsArtifactType, - projection_role=SourceProjectionRole.SOURCE_ARTIFACT, + projection_role=SourceArtifactRole, ) refs = { primary_path: SourcePixelRef("disk", source_path), @@ -1554,8 +1556,8 @@ def test_virtual_workspace_source_matching_uses_declared_filter_identity(): selector=SourceSelector( filters=( SourceFilterClause( - subject=SourceFilterSubject.FILE, - match_type=SourceFilterMatchType.CONTAINS, + subject=SourceFileName, + match_type=TextContains, value="N_R", ), ), @@ -1619,7 +1621,7 @@ def test_rule_metadata_preserves_original_source_metadata_mapping(): source_paths_by_virtual_path={}, metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?P[^_]+)_(?P\d+)", ), ), @@ -1642,13 +1644,13 @@ def test_compiled_source_binding_plan_round_trips_through_pickle(): selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, "1"),), ), - origin=SourceBindingOrigin.STEP_INPUT, + origin=FromStepInput, ), ), enabled=True, metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FOLDER_NAME, + source=FromFolderName, pattern=r"(?PPlateA)", ), ), @@ -1670,3 +1672,60 @@ def test_compiled_source_binding_plan_round_trips_through_pickle(): assert restored.metadata_rules == plan.metadata_rules assert restored.match_plan == plan.match_plan assert restored.binding_for_alias("OrigBlue") == plan.binding_for_alias("OrigBlue") + + +SOURCE_BINDING_FAMILIES = ( + source_bindings_module.SourceBindingOrigin, + source_bindings_module.SourceSetRole, + source_bindings_module.SourceProjectionRole, + source_bindings_module.MetadataSource, + source_bindings_module.SourceFilterSubject, + source_bindings_module.SourceFilterMatchType, +) + + +@pytest.mark.parametrize("family", SOURCE_BINDING_FAMILIES, ids=lambda f: f.__name__) +def test_source_binding_kinds_travel_as_their_registry_names(family): + from python_introspect import registered_class_from_key, to_jsonable + + members = tuple(family.__registry__.values()) + assert members + for member in members: + spelling = to_jsonable(member) + assert spelling == member.name + assert registered_class_from_key(family, spelling) is member + + +def test_source_filter_match_types_answer_the_path_table(): + sb = source_bindings_module + path = "/plate/A01/VitraChannel1ILLUM.tif" + expected = { + sb.TextContains: ("ILLUM", True), + sb.TextContainsRegex: (r"Channel\d", True), + sb.TextEquals: ("VitraChannel1ILLUM.tif", True), + sb.TextStartsWith: ("Vitra", True), + sb.TextEndsWith: (".npy", False), + sb.PathIsImage: (None, True), + sb.PathIsTif: (None, True), + } + for match_type, (value, holds) in expected.items(): + clause = sb.SourceFilterClause(sb.SourceFileName, match_type, value) + assert clause.matches(path) is holds, match_type + if issubclass(match_type, sb.TextMatchType): + negated = sb.SourceFilterClause( + sb.SourceFileName, match_type.negation(), value + ) + assert negated.matches(path) is not holds, match_type.negation() + text_matches = { + member + for member in sb.SourceFilterMatchType.__registry__.values() + if issubclass(member, sb.TextMatchType) + } + negations = { + member for member in text_matches if issubclass(member, sb.NegatedTextMatch) + } + assert {positive.negation() for positive in text_matches - negations} == negations + assert sb.SourceDirectory.text_of(path) == "/plate/A01" + assert sb.SourceExtension.text_of(path) == ".tif" + assert FromFileName.texts(path) == ("VitraChannel1ILLUM.tif",) + assert FromFolderName.texts(path) == ("A01", "/plate/A01") diff --git a/tests/unit/test_source_bindings_preview.py b/tests/unit/test_source_bindings_preview.py index 15dbdeba6e..8c46a0245a 100644 --- a/tests/unit/test_source_bindings_preview.py +++ b/tests/unit/test_source_bindings_preview.py @@ -2,17 +2,19 @@ from openhcs.constants.constants import Backend from openhcs.core.source_bindings import ( + FromFileName, + FromPipelineStart, ImagePlaneSource, MetadataExtractionRule, - MetadataSource, MetadataSelector, NamedSourceBinding, PairByOrder, - SourceBindingOrigin, + PathIsTif, + SourceExtension, + SourceFileName, SourceFilterClause, - SourceFilterMatchType, - SourceFilterSubject, SourceSelector, + TextContains, ) from openhcs.core.config import SourceBindingsConfig, StepSourceBindingsConfig from openhcs.core.source_bindings_preview import ( @@ -54,8 +56,8 @@ def test_source_bindings_preview_reuses_typed_filter_and_order_matching(tmp_path source_bindings = SourceBindingsConfig( source_filters=( SourceFilterClause( - SourceFilterSubject.EXTENSION, - SourceFilterMatchType.IS_TIF, + SourceExtension, + PathIsTif, ), ), bindings=( @@ -64,26 +66,26 @@ def test_source_bindings_preview_reuses_typed_filter_and_order_matching(tmp_path selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), NamedSourceBinding( alias="GFP", selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "GFP", ), ), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), ), match_plan=PairByOrder(), @@ -124,7 +126,7 @@ def test_source_inventory_uses_declared_image_plane_sources(tmp_path): image_plane_sources=(ImagePlaneSource(uri=str(source_path)),), metadata_rules=( MetadataExtractionRule( - source=MetadataSource.FILE_NAME, + source=FromFileName, pattern=r"(?PA\d{2})_(?Ps\d+)_(?PDNA)\.tif", ), ), @@ -134,7 +136,7 @@ def test_source_inventory_uses_declared_image_plane_sources(tmp_path): selector=SourceSelector( metadata=(MetadataSelector("channel", "DNA"),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), ), ) @@ -168,12 +170,12 @@ def test_source_inventory_from_paths_applies_source_filters(tmp_path): source_bindings = SourceBindingsConfig( source_filters=( SourceFilterClause( - SourceFilterSubject.EXTENSION, - SourceFilterMatchType.IS_TIF, + SourceExtension, + PathIsTif, ), SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ), @@ -200,7 +202,7 @@ def test_source_inventory_and_preview_use_resolved_step_override(tmp_path): bindings=( NamedSourceBinding( alias="DNA", - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), ), ) @@ -208,15 +210,15 @@ def test_source_inventory_and_preview_use_resolved_step_override(tmp_path): enabled=True, source_filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "GFP", ), ), bindings=( NamedSourceBinding( alias="GFP", - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), ), ) @@ -251,13 +253,13 @@ def test_source_binding_preview_reports_required_alias_without_matches(): selector=SourceSelector( filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, ), ), ) @@ -284,8 +286,8 @@ def test_source_inventory_from_filemanager_uses_vfs_file_listing(): source_bindings = SourceBindingsConfig( source_filters=( SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "DNA", ), ), diff --git a/tests/unit/test_source_bound_matched_image_set_anchors.py b/tests/unit/test_source_bound_matched_image_set_anchors.py index df900d067d..33f4358746 100644 --- a/tests/unit/test_source_bound_matched_image_set_anchors.py +++ b/tests/unit/test_source_bound_matched_image_set_anchors.py @@ -29,12 +29,12 @@ from openhcs.core.source_bindings import ( CompiledSourceBindingPlan, ComponentSelector, + FromPipelineStart, NamedSourceBinding, - SourceBindingMatchDimension, - SourceBindingMatchField, PairByMetadata, PairByOrder, - SourceBindingOrigin, + SourceBindingMatchDimension, + SourceBindingMatchField, SourceSelector, ) from openhcs.core.source_projection import OpenHCSPlaneAddress, SourcePlaneProjection @@ -131,7 +131,7 @@ def _source_bindings() -> tuple[NamedSourceBinding, ...]: selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, channel),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, channel),), ) for alias, channel in SOURCE_ALIASES @@ -553,7 +553,7 @@ def test_complete_source_set_templates_preserve_each_execution_group_anchor( selector=SourceSelector( components=(ComponentSelector(Microscopy.Channel, channel),), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, channel),), ) for alias, channel in (("OrigStain1", "1"), ("OrigStain2", "2")) @@ -658,7 +658,7 @@ def test_static_site_groups_do_not_cross_project_natural_source_set_templates( selector=SourceSelector( filters=(), ), - origin=SourceBindingOrigin.PIPELINE_START, + origin=FromPipelineStart, component_identity=(ComponentSelector(Microscopy.Channel, channel),), ) for alias, channel in (("OrigStain1", "1"), ("OrigStain2", "2")) diff --git a/tests/unit/test_source_matching.py b/tests/unit/test_source_matching.py index 9edab3e64d..aef8afeff7 100644 --- a/tests/unit/test_source_matching.py +++ b/tests/unit/test_source_matching.py @@ -1,9 +1,13 @@ import pytest from openhcs.core.source_bindings import ( + SourceDirectory, + SourceFileName, SourceFilterClause, SourceFilterMatchType, - SourceFilterSubject, + TextContains, + TextDoesNotEqual, + TextEquals, ) from openhcs.core.source_matching import ( DECLARED_SOURCE_METADATA_FIELD, @@ -27,29 +31,29 @@ ("match_type", "path", "expected"), ( ( - SourceFilterMatchType.EQUALS, + TextEquals, "/plate/VitraChannel1ILLUM.npy", True, ), ( - SourceFilterMatchType.EQUALS, + TextEquals, "/plate/VitraChannel2ILLUM.npy", False, ), ( - SourceFilterMatchType.DOES_NOT_EQUAL, + TextDoesNotEqual, "/plate/VitraChannel1ILLUM.npy", False, ), ( - SourceFilterMatchType.DOES_NOT_EQUAL, + TextDoesNotEqual, "/plate/VitraChannel2ILLUM.npy", True, ), ), ) def test_file_source_filters_match_exact_file_names( - match_type: SourceFilterMatchType, + match_type: type[SourceFilterMatchType], path: str, expected: bool, ): @@ -58,7 +62,7 @@ def test_file_source_filters_match_exact_file_names( path, ( SourceFilterClause( - subject=SourceFilterSubject.FILE, + subject=SourceFileName, match_type=match_type, value="VitraChannel1ILLUM.npy", ), @@ -71,19 +75,19 @@ def test_file_source_filters_match_exact_file_names( def test_source_filters_match_flat_conjunction_of_disjunction_groups() -> None: filters = ( SourceFilterClause( - SourceFilterSubject.DIRECTORY, - SourceFilterMatchType.CONTAINS, + SourceDirectory, + TextContains, "plate", ), SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "_s1_", any_group=0, ), SourceFilterClause( - SourceFilterSubject.FILE, - SourceFilterMatchType.CONTAINS, + SourceFileName, + TextContains, "_s2_", any_group=0, ), diff --git a/tests/unit/test_source_projection.py b/tests/unit/test_source_projection.py index 9ab891dbd9..eafc61394d 100644 --- a/tests/unit/test_source_projection.py +++ b/tests/unit/test_source_projection.py @@ -7,7 +7,10 @@ from polystore.bioformats_storage import BioFormatsPlaneRef from openhcs.core.artifacts import ImageArtifactType, ObjectLabelsArtifactType -from openhcs.core.source_bindings import SourceProjectionRole +from openhcs.core.source_bindings import ( + PrimaryPlaneRole, + SourceArtifactRole, +) from openhcs.core.source_metadata import ( DECLARED_SOURCE_METADATA_FIELD, SourceMetadataFields, @@ -44,20 +47,20 @@ def test_projection_nominal_owners_declare_projection_semantics() -> None: artifact_kind=ObjectLabelsArtifactType, ) - assert plane.projection_role is SourceProjectionRole.PRIMARY_PLANE + assert plane.projection_role is PrimaryPlaneRole assert plane.source_alias == "DNA" assert plane.artifact_kind is ImageArtifactType - assert plane.identity_key == (SourceProjectionRole.PRIMARY_PLANE, address) + assert plane.identity_key == (PrimaryPlaneRole, address) assert plane.payload_composition_alias == "DNA" assert plane.virtual_workspace_path("plane.tif", execution_anchor=False) == ( "plane.tif" ) - assert artifact.projection_role is SourceProjectionRole.SOURCE_ARTIFACT + assert artifact.projection_role is SourceArtifactRole assert artifact.source_alias == "Nuclei" assert artifact.artifact_kind is ObjectLabelsArtifactType assert artifact.identity_key == ( - SourceProjectionRole.SOURCE_ARTIFACT, + SourceArtifactRole, address, "Nuclei", ) diff --git a/tests/unit/test_source_tile_geometry.py b/tests/unit/test_source_tile_geometry.py index c97b3f3091..ff4dd162ba 100644 --- a/tests/unit/test_source_tile_geometry.py +++ b/tests/unit/test_source_tile_geometry.py @@ -21,7 +21,10 @@ from openhcs.core.runtime_plane_projection import RuntimePlaneAxis from openhcs.utils.environment import OpenHCSProcessEnvironment from openhcs.core.source_binding_workspace import SourceBindingWorkspaceProjector -from openhcs.core.source_bindings import MetadataExtractionRule, MetadataSource +from openhcs.core.source_bindings import ( + FromFileName, + MetadataExtractionRule, +) from openhcs.core.config import SourceBindingsConfig from openhcs.core.source_image_provenance import SourceImageProvenancePlanes from openhcs.core.source_matching import source_component_metadata_value @@ -88,7 +91,7 @@ def _config(): return SourceBindingsConfig( metadata_rules=( MetadataExtractionRule( - MetadataSource.FILE_NAME, + FromFileName, r"(?PA\d+)_s(?P\d+)_w(?P\d+)_z(?P\d+)_t(?P\d+)\.tif", ), ) diff --git a/tests/unit/test_step_input_artifact_origin.py b/tests/unit/test_step_input_artifact_origin.py index 27eebe0e0d..c6960cc20b 100644 --- a/tests/unit/test_step_input_artifact_origin.py +++ b/tests/unit/test_step_input_artifact_origin.py @@ -17,8 +17,8 @@ from openhcs.core.source_bindings import ( CompiledSourceBindingPlan, ComponentSelector, + FromPipelineStart, NamedSourceBinding, - SourceBindingOrigin, SourceSelector, ) from openhcs.core.source_image_provenance import SourceImageProvenancePlanes @@ -79,7 +79,7 @@ def test_step_input_selects_current_component_planes_not_original_aliases(): def test_binding_origin_uses_existing_registered_universe_family(): assert SourceUniverseRequest.for_binding(NamedSourceBinding(alias="Response")) is StepInputSourceUniverseRequest assert SourceUniverseRequest.for_binding( - NamedSourceBinding(alias="Raw", origin=SourceBindingOrigin.PIPELINE_START) + NamedSourceBinding(alias="Raw", origin=FromPipelineStart) ) is PipelineStartSourceUniverseRequest From 81b2c44f8d84d9038f938f1e7d0b67ab8af9d83a Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:58:00 -0400 Subject: [PATCH 19/36] E1: threshold diagnostics, watershed runtime, experimental formats, DefineGrid and SplitOrMerge variants keyed by class ThresholdDiagnosticDomain, WatershedRuntimeFamily, ExperimentalAnalysisScope, DefineGridVariant and SplitOrMergeInputTopology are deleted as enums: their strategy leaves (or new variant classes) are the family members, keyed by name or function name, and own what was switched on. DefineGrid's resolve_function asks the variant instead of re-deciding it. Co-Authored-By: Claude Opus 5.5 --- .../processing/backends/cellprofiler/grid.py | 64 ++++++++++------- .../backends/cellprofiler/morphology.py | 57 ++++++++++----- .../backends/cellprofiler/thresholding.py | 72 +++++++++---------- .../backends/cellprofiler/watershed.py | 29 +++----- .../experimental_analysis/analysis.py | 5 +- .../experimental_analysis/layout_rows.py | 13 ---- .../experimental_analysis/result_formats.py | 32 +++++---- .../unified_analysis_engine.py | 13 ++-- ...test_cellprofiler_artifact_declarations.py | 10 +-- tests/unit/test_experimental_layout_rows.py | 17 ++--- 10 files changed, 157 insertions(+), 155 deletions(-) diff --git a/openhcs/processing/backends/cellprofiler/grid.py b/openhcs/processing/backends/cellprofiler/grid.py index a422ce87d3..5349a541a0 100644 --- a/openhcs/processing/backends/cellprofiler/grid.py +++ b/openhcs/processing/backends/cellprofiler/grid.py @@ -123,19 +123,22 @@ def from_setting( return cls(normalized) -class DefineGridVariant(str, Enum): - """Absorbed DefineGrid function variants.""" +class DefineGridVariant(ABC, metaclass=AutoRegisterMeta): + """Absorbed DefineGrid function variant. - MANUAL = "define_grid_manual" - AUTOMATIC = "define_grid_automatic" + Each variant is a subclass keyed by the function it runs. + """ - @property - def requires_object_input(self) -> bool: - """Return whether this variant derives the grid from object labels.""" - return self is DefineGridVariant.AUTOMATIC + __registry_key__ = "function_name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["DefineGridVariant"]]] = {} - @classmethod - def from_module(cls, module: ModuleBlock) -> "DefineGridVariant": + function_name: ClassVar[str | None] = None + requires_object_input: ClassVar[bool] + """The variant derives the grid from object labels.""" + + @staticmethod + def from_module(module: ModuleBlock) -> type["DefineGridVariant"]: method = DefineGridManualModule.DefinitionMethod.from_literal( _setting_value( module, @@ -144,11 +147,29 @@ def from_module(cls, module: ModuleBlock) -> "DefineGridVariant": ) ) return ( - cls.AUTOMATIC + AutomaticGrid if method is DefineGridManualModule.DefinitionMethod.automatic - else cls.MANUAL + else ManualGrid ) + @staticmethod + def for_function(function_name: str) -> type["DefineGridVariant"]: + return DefineGridVariant.__registry__[function_name] + + +class ManualGrid(DefineGridVariant): + """A grid placed from two coordinate pairs on an image.""" + + function_name = "define_grid_manual" + requires_object_input = False + + +class AutomaticGrid(DefineGridVariant): + """A grid derived from previously identified objects.""" + + function_name = "define_grid_automatic" + requires_object_input = True + def define_grid_bound_kwargs( module: ModuleBlock, binder: SettingsBinder @@ -183,7 +204,7 @@ def define_grid_bound_kwargs( ) ), } - if DefineGridVariant.from_module(module) is DefineGridVariant.MANUAL: + if not DefineGridVariant.from_module(module).requires_object_input: first_x, first_y = _coordinate_pair( _setting_value( module, @@ -509,16 +530,7 @@ def resolve_function( source_bindings: "StepSourceBindingsConfig", ) -> Callable[..., object]: del contract, source_bindings - method = cls.DefinitionMethod.from_literal( - cls.setting_value(module, cls.definition_method_setting) - or cls.definition_method_default.value - ) - function_name = ( - cls.function_variants[0] - if method is cls.DefinitionMethod.automatic - else str(cls.function_name) - ) - return cls.require_callable(function_name) + return cls.require_callable(DefineGridVariant.from_module(module).function_name) @classmethod def active_artifact_bindings( @@ -537,7 +549,7 @@ def active_artifact_bindings( return bindings if invocation_key is None: raise TypeError("DefineGrid requires its exact invocation key.") - requires_objects = DefineGridVariant( + requires_objects = DefineGridVariant.for_function( invocation_key.function_name ).requires_object_input retain_image = optional_setting_value(module, cls.retain_grid_image_setting) @@ -576,7 +588,7 @@ def artifact_output_relations( output_position=output_position, ) del module, name, step_context, output_position - variant = DefineGridVariant(invocation_key.function_name) + variant = DefineGridVariant.for_function(invocation_key.function_name) source_type = ( ObjectLabelsArtifactType if variant.requires_object_input @@ -585,7 +597,7 @@ def artifact_output_relations( sources = artifact_inputs.of_artifact_type(source_type) if len(sources) != 1: raise ValueError( - f"{variant.value} requires exactly one {source_type.value} " + f"{variant.function_name} requires exactly one {source_type.value} " f"grid-domain source, got {tuple(spec.ref() for spec in sources)!r}." ) return (SourceStackLineageSourceRelation(source=sources[0].ref()),) diff --git a/openhcs/processing/backends/cellprofiler/morphology.py b/openhcs/processing/backends/cellprofiler/morphology.py index 6937848eaf..ae97922906 100644 --- a/openhcs/processing/backends/cellprofiler/morphology.py +++ b/openhcs/processing/backends/cellprofiler/morphology.py @@ -5878,26 +5878,46 @@ class SplitOrMergeIntensityMethod(Enum): CLOSEST_POINT = "closest_point" -class SplitOrMergeInputTopology(Enum): - """Exact public callable ABI selected by SplitOrMergeObjects settings.""" +class SplitOrMergeInputTopology(ABC, metaclass=AutoRegisterMeta): + """Exact public callable ABI selected by SplitOrMergeObjects settings. - LABELS_ONLY = "split_or_merge_objects" - GUIDE_IMAGE = "split_or_merge_objects_with_guide_image" - PARENT_OBJECTS = "split_or_merge_objects_per_parent" + Each topology is a subclass keyed by the function it runs. + """ - @classmethod + __registry_key__ = "function_name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["SplitOrMergeInputTopology"]]] = {} + + function_name: ClassVar[str | None] = None + uses_guide_image: ClassVar[bool] = False + uses_parent_objects: ClassVar[bool] = False + + @staticmethod def from_values( - cls, *, operation: SplitOrMergeOperation, merge_method: SplitOrMergeMergeMethod, use_guide_image: bool, - ) -> "SplitOrMergeInputTopology": + ) -> type["SplitOrMergeInputTopology"]: if operation is SplitOrMergeOperation.SPLIT: - return cls.LABELS_ONLY + return SplitOrMergeLabelsOnly if merge_method is SplitOrMergeMergeMethod.PER_PARENT: - return cls.PARENT_OBJECTS - return cls.GUIDE_IMAGE if use_guide_image else cls.LABELS_ONLY + return SplitOrMergePerParent + return SplitOrMergeWithGuideImage if use_guide_image else SplitOrMergeLabelsOnly + + +class SplitOrMergeLabelsOnly(SplitOrMergeInputTopology): + function_name = "split_or_merge_objects" + + +class SplitOrMergeWithGuideImage(SplitOrMergeInputTopology): + function_name = "split_or_merge_objects_with_guide_image" + uses_guide_image = True + + +class SplitOrMergePerParent(SplitOrMergeInputTopology): + function_name = "split_or_merge_objects_per_parent" + uses_parent_objects = True @dataclass(frozen=True, slots=True) @@ -7243,7 +7263,9 @@ class SplitOrMergeObjectsModule( ) @classmethod - def input_topology(cls, module: "ModuleBlock") -> SplitOrMergeInputTopology: + def input_topology( + cls, module: "ModuleBlock" + ) -> type[SplitOrMergeInputTopology]: return SplitOrMergeInputTopology.from_values( operation=coerce_cellprofiler_enum( SplitOrMergeOperation, @@ -7276,9 +7298,8 @@ def active_artifact_bindings(cls, module=None, *, invocation_key=None): return tuple( binding for binding in bindings - if topology is SplitOrMergeInputTopology.GUIDE_IMAGE - or binding is not cls.guide_image_binding - if topology is SplitOrMergeInputTopology.PARENT_OBJECTS + if topology.uses_guide_image or binding is not cls.guide_image_binding + if topology.uses_parent_objects or binding is not cls.parent_objects_binding ) @@ -7329,7 +7350,7 @@ def resolve_function( source_bindings: "StepSourceBindingsConfig", ) -> Callable[..., object]: del contract, source_bindings - return cls.require_callable(cls.input_topology(module).value) + return cls.require_callable(cls.input_topology(module).function_name) @classmethod def finalize_module_blocks_for_invocation( @@ -7342,10 +7363,10 @@ def finalize_module_blocks_for_invocation( ) for block in blocks: topology = cls.input_topology(block) - if invocation.contract.function_name != topology.value: + if invocation.contract.function_name != topology.function_name: raise ValueError( f"{cls.module_name} callable {invocation.contract.function_name!r} " - f"does not match declared input topology {topology.name}." + f"does not match declared input topology {topology.__name__}." ) return blocks diff --git a/openhcs/processing/backends/cellprofiler/thresholding.py b/openhcs/processing/backends/cellprofiler/thresholding.py index 2fc2d83fff..c45f93cb1f 100644 --- a/openhcs/processing/backends/cellprofiler/thresholding.py +++ b/openhcs/processing/backends/cellprofiler/thresholding.py @@ -910,19 +910,12 @@ def sum_of_entropies( return _numpy_threshold_sum_of_entropies(image, mask, binary_image) -class ThresholdDiagnosticDomain(Enum): - """Nominal image domains for CellProfiler threshold diagnostics.""" - - PLANAR_IMAGE = "planar_image" - ND_IMAGE = "nd_image" - - @dataclass(frozen=True, slots=True) class ThresholdDiagnosticRequest: """Typed inputs for one CellProfiler threshold diagnostic measurement.""" backend: "NumbaNumpyThresholdDiagnosticsBackendStrategy" - domain: ThresholdDiagnosticDomain + domain: type["ThresholdDiagnosticDomain"] image: np.ndarray mask: np.ndarray | None binary_image: np.ndarray @@ -952,11 +945,7 @@ def from_inputs( ) return cls( backend=backend, - domain=( - ThresholdDiagnosticDomain.PLANAR_IMAGE - if image_array.ndim == 2 - else ThresholdDiagnosticDomain.ND_IMAGE - ), + domain=ThresholdDiagnosticDomain.of_rank(image_array.ndim), image=image_array, mask=mask_array, binary_image=binary_array, @@ -964,40 +953,46 @@ def from_inputs( ) -class ThresholdDiagnosticDomainStrategy( - EnumKeyedStrategyMixin[ThresholdDiagnosticDomain], - ABC, - metaclass=AutoRegisterMeta, -): - """Apply CellProfiler threshold diagnostics in the correct image domain.""" +class ThresholdDiagnosticDomain(ABC, metaclass=AutoRegisterMeta): + """Image domain CellProfiler threshold diagnostics are measured in. - __enum_member_attr__ = "domain" - domain: ClassVar[ThresholdDiagnosticDomain | None] = None + Each domain is a subclass that applies the diagnostics in that domain. + """ - @classmethod - def evaluate(cls, request: ThresholdDiagnosticRequest) -> tuple[float, float]: - return cls.for_enum_member(request.domain).diagnostics(request) + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["ThresholdDiagnosticDomain"]]] = {} + name: ClassVar[str | None] = None + + @staticmethod + def of_rank(ndim: int) -> type["ThresholdDiagnosticDomain"]: + """Planar images use the 2-D path; every other rank is one image.""" + return PlanarThresholdDiagnostics if ndim == 2 else WholeImageThresholdDiagnostics + + @classmethod @abstractmethod - def diagnostics(self, request: ThresholdDiagnosticRequest) -> tuple[float, float]: + def diagnostics(cls, request: ThresholdDiagnosticRequest) -> tuple[float, float]: """Return weighted variance and sum of entropies for ``request``.""" -class PlanarThresholdDiagnosticDomainStrategy(ThresholdDiagnosticDomainStrategy): +class PlanarThresholdDiagnostics(ThresholdDiagnosticDomain): """Use the optimized 2-D backend path for planar CellProfiler images.""" - domain = ThresholdDiagnosticDomain.PLANAR_IMAGE + name = "planar_image" - def diagnostics(self, request: ThresholdDiagnosticRequest) -> tuple[float, float]: + @classmethod + def diagnostics(cls, request: ThresholdDiagnosticRequest) -> tuple[float, float]: return request.backend.diagnostics_planar(request) -class WholeImageThresholdDiagnosticDomainStrategy(ThresholdDiagnosticDomainStrategy): +class WholeImageThresholdDiagnostics(ThresholdDiagnosticDomain): """Measure ND images as one CellProfiler image domain.""" - domain = ThresholdDiagnosticDomain.ND_IMAGE + name = "nd_image" - def diagnostics(self, request: ThresholdDiagnosticRequest) -> tuple[float, float]: + @classmethod + def diagnostics(cls, request: ThresholdDiagnosticRequest) -> tuple[float, float]: return request.backend.diagnostics_whole_image(request) @@ -1063,15 +1058,14 @@ def diagnostics( *, proven_unit_interval_scale: int | None = None, ) -> tuple[float, float]: - return ThresholdDiagnosticDomainStrategy.evaluate( - ThresholdDiagnosticRequest.from_inputs( - backend=self, - image=image, - mask=mask, - binary_image=binary_image, - proven_unit_interval_scale=proven_unit_interval_scale, - ) + request = ThresholdDiagnosticRequest.from_inputs( + backend=self, + image=image, + mask=mask, + binary_image=binary_image, + proven_unit_interval_scale=proven_unit_interval_scale, ) + return request.domain.diagnostics(request) def diagnostics_planar( self, diff --git a/openhcs/processing/backends/cellprofiler/watershed.py b/openhcs/processing/backends/cellprofiler/watershed.py index 961782bc7d..7c102eae8d 100644 --- a/openhcs/processing/backends/cellprofiler/watershed.py +++ b/openhcs/processing/backends/cellprofiler/watershed.py @@ -608,13 +608,6 @@ class WatershedSeedMethod(str, Enum): CONNECTED_COMPONENTS = "connected_components" -class WatershedRuntimeFamily(str, Enum): - """CellProfiler Watershed implementation family selected by module revision.""" - - CELLPROFILER4 = "cellprofiler4" - LIBRARY = "library" - - @dataclass(frozen=True, slots=True) class WatershedSpecialInputs: """Method-resolved auxiliary images for one Watershed invocation.""" @@ -666,7 +659,7 @@ def topology(self) -> WatershedInputTopology: def execute( self, - runtime_family: WatershedRuntimeFamily, + runtime: type["WatershedRuntimeStrategy"], special_inputs: tuple[object, ...] = (), ) -> tuple[np.ndarray, DataclassMeasurementColumnarRows, ObjectLabelValue]: """Apply this request through its declaration-owned runtime family.""" @@ -724,7 +717,7 @@ def execute( structuring_element=self.structuring_element, structuring_element_size=self.structuring_element_size, ) - labels = WatershedRuntimeStrategy.for_enum_member(runtime_family).labels( + labels = runtime().labels( image_array, intensity_image_array, markers_array, @@ -1193,14 +1186,14 @@ def markers( ) -class WatershedRuntimeStrategy( - EnumKeyedStrategyMixin[WatershedRuntimeFamily], ABC, metaclass=AutoRegisterMeta -): - """Execute one nominal CellProfiler Watershed implementation family.""" +class WatershedRuntimeStrategy(ABC, metaclass=AutoRegisterMeta): + """One CellProfiler Watershed implementation family, selected by module revision. + + Each family is a subclass; ``strategy_label`` is its registry key. + """ __registry_key__ = WATERSHED_STRATEGY_REGISTRY_KEY __skip_if_no_key__ = True - strategy_key: ClassVar[WatershedRuntimeFamily | None] = None strategy_label: ClassVar[str | None] = None @abstractmethod @@ -1401,7 +1394,7 @@ def labels( class CellProfiler4WatershedRuntimeStrategy(WatershedRuntimeStrategy): """CellProfiler 4.2 module-level Watershed semantics.""" - strategy_key = WatershedRuntimeFamily.CELLPROFILER4 + strategy_label = "cellprofiler4" def labels( self, @@ -1516,7 +1509,7 @@ def labels( class LibraryWatershedRuntimeStrategy(WatershedRuntimeStrategy): """CellProfiler library-style Watershed semantics.""" - strategy_key = WatershedRuntimeFamily.LIBRARY + strategy_label = "library" def labels( self, @@ -1867,7 +1860,7 @@ def watershed_library( watershed method and supplied by OpenHCS. """ - return request.execute(WatershedRuntimeFamily.LIBRARY, topology_inputs) + return request.execute(LibraryWatershedRuntimeStrategy, topology_inputs) @runtime_image_execution_mode(FullStackExecution) @@ -1881,7 +1874,7 @@ def watershed_cellprofiler4( ) -> tuple[np.ndarray, DataclassMeasurementColumnarRows, ObjectLabelValue]: """Apply CellProfiler 4 Watershed with topology-resolved auxiliary inputs.""" - return request.execute(WatershedRuntimeFamily.CELLPROFILER4, topology_inputs) + return request.execute(CellProfiler4WatershedRuntimeStrategy, topology_inputs) set_signature_analysis_target(watershed_cellprofiler4, watershed_library) diff --git a/openhcs/processing/backends/experimental_analysis/analysis.py b/openhcs/processing/backends/experimental_analysis/analysis.py index ee871bb570..c27be9d8fb 100755 --- a/openhcs/processing/backends/experimental_analysis/analysis.py +++ b/openhcs/processing/backends/experimental_analysis/analysis.py @@ -23,7 +23,6 @@ from openhcs.domains.microscopy.config import NormalizationMethod from openhcs.processing.backends.experimental_analysis.layout_rows import ( - ExperimentalAnalysisScope, ExperimentalLayoutRowRole, ) from openhcs.processing.backends.experimental_analysis.result_formats import ExperimentalResultFormatStrategy @@ -99,9 +98,7 @@ def is_well_tuple(item) -> bool: def result_format_strategy(scope: object) -> ExperimentalResultFormatStrategy: """Resolve one declared workbook scope through the nominal strategy root.""" - return ExperimentalResultFormatStrategy.for_enum_member( - ExperimentalAnalysisScope.coerce(scope) - ) + return ExperimentalResultFormatStrategy.for_scope(scope) def read_results(results_path: str, scope: Optional[str] = None) -> pd.DataFrame: diff --git a/openhcs/processing/backends/experimental_analysis/layout_rows.py b/openhcs/processing/backends/experimental_analysis/layout_rows.py index 14f6cd1e80..f3ccb94fe5 100644 --- a/openhcs/processing/backends/experimental_analysis/layout_rows.py +++ b/openhcs/processing/backends/experimental_analysis/layout_rows.py @@ -2,7 +2,6 @@ from __future__ import annotations -from enum import Enum class ExperimentalLayoutRowRole: @@ -34,15 +33,3 @@ def specific_replicate(self) -> int | None: return None -class ExperimentalAnalysisScope(Enum): - """Stable workbook declarations for supported experimental result formats.""" - - CX5 = "EDDU_CX5" - METAXPRESS = "EDDU_metaxpress" - - @classmethod - def coerce(cls, scope: object) -> "ExperimentalAnalysisScope": - for member in cls: - if scope == member or scope == member.value: - return member - raise ValueError(f"microscope {scope} not known") diff --git a/openhcs/processing/backends/experimental_analysis/result_formats.py b/openhcs/processing/backends/experimental_analysis/result_formats.py index d3039f5574..8e0c62fee1 100644 --- a/openhcs/processing/backends/experimental_analysis/result_formats.py +++ b/openhcs/processing/backends/experimental_analysis/result_formats.py @@ -5,13 +5,11 @@ from abc import abstractmethod from pathlib import Path import string -from typing import Any +from typing import Any, ClassVar from metaclass_registry import AutoRegisterMeta import pandas as pd -from metaclass_registry.strategies import EnumKeyedStrategyMixin -from openhcs.processing.backends.experimental_analysis.layout_rows import ExperimentalAnalysisScope PlateDictionary = dict[str, dict[str, dict[str, Any]]] @@ -25,13 +23,23 @@ def plate_well_ids() -> tuple[str, ...]: ) -class ExperimentalResultFormatStrategy( - EnumKeyedStrategyMixin[ExperimentalAnalysisScope], - metaclass=AutoRegisterMeta, -): - """Own all result-file behavior for one declared workbook scope.""" +class ExperimentalResultFormatStrategy(metaclass=AutoRegisterMeta): + """Own all result-file behavior for one declared workbook scope. - __enum_member_attr__ = "scope" + Each format is a subclass; ``scope_name`` is the scope a workbook declares. + """ + + __registry_key__ = "scope_name" + __skip_if_no_key__ = True + scope_name: ClassVar[str | None] = None + + @classmethod + def for_scope(cls, scope: object) -> "ExperimentalResultFormatStrategy": + """Return the format for the scope a workbook declares.""" + format_type = cls.__registry__.get(scope) + if format_type is None: + raise ValueError(f"microscope {scope} not known") + return format_type() @abstractmethod def read_results(self, results_path: str | Path) -> pd.DataFrame: @@ -68,7 +76,7 @@ def process(self, results_path: str | Path) -> dict[str, Any]: "features": features, "plate_names": list(plates_dict), "plates_dict": plates_dict, - "format_name": self.scope.value, + "format_name": self.scope_name, } def empty_wells(self, raw_df: pd.DataFrame) -> dict[str, dict[str, Any]]: @@ -92,7 +100,7 @@ def measurement_value(value: Any) -> Any: class CX5ExperimentalResultFormat(ExperimentalResultFormatStrategy): """ThermoFisher CX5 result-file behavior.""" - scope = ExperimentalAnalysisScope.CX5 + scope_name = "EDDU_CX5" def read_results(self, results_path: str | Path) -> pd.DataFrame: return pd.read_excel(results_path, sheet_name="Rawdata") @@ -128,7 +136,7 @@ def fill_plates_dict( class MetaXpressExperimentalResultFormat(ExperimentalResultFormatStrategy): """MetaXpress consolidated CSV and workbook result behavior.""" - scope = ExperimentalAnalysisScope.METAXPRESS + scope_name = "EDDU_metaxpress" def read_results(self, results_path: str | Path) -> pd.DataFrame: path = Path(results_path) diff --git a/openhcs/processing/backends/experimental_analysis/unified_analysis_engine.py b/openhcs/processing/backends/experimental_analysis/unified_analysis_engine.py index b1b2487cf7..be3349c2f6 100644 --- a/openhcs/processing/backends/experimental_analysis/unified_analysis_engine.py +++ b/openhcs/processing/backends/experimental_analysis/unified_analysis_engine.py @@ -4,7 +4,6 @@ from typing import Dict, List, Any, Optional, Tuple from openhcs.domains.microscopy.config import ExperimentalAnalysisConfig -from openhcs.processing.backends.experimental_analysis.layout_rows import ExperimentalAnalysisScope from openhcs.processing.backends.experimental_analysis.result_formats import ( ExperimentalResultFormatStrategy, ) @@ -47,13 +46,11 @@ def run_analysis( try: # Parse the workbook declaration that owns result format. experiment_config = self._parse_experiment_config(config_file) - scope = ExperimentalAnalysisScope.coerce(experiment_config["scope"]) - - # Process microscope data through the scope-keyed leaf. - processed_data = ExperimentalResultFormatStrategy.for_enum_member( - scope - ).process(results_path) - format_name = scope.value + result_format = ExperimentalResultFormatStrategy.for_scope( + experiment_config["scope"] + ) + processed_data = result_format.process(results_path) + format_name = result_format.scope_name # Create the experiment data structure. experiment_dict_locations = self._make_experiment_dict_locations( diff --git a/tests/unit/test_cellprofiler_artifact_declarations.py b/tests/unit/test_cellprofiler_artifact_declarations.py index b14a5abef2..bac3ebe281 100644 --- a/tests/unit/test_cellprofiler_artifact_declarations.py +++ b/tests/unit/test_cellprofiler_artifact_declarations.py @@ -69,7 +69,9 @@ ) from openhcs.processing.backends.cellprofiler.grid import ( DefineGridManualModule, + AutomaticGrid, DefineGridVariant, + ManualGrid, ) from openhcs.processing.backends.cellprofiler.classification import ( ClassifyObjectsSingleMeasurementModule, @@ -1594,19 +1596,19 @@ def test_relate_objects_measurements_depend_on_declared_relationship_output() -> ( ( DefineGridManualModule.DefinitionMethod.manual, - DefineGridVariant.MANUAL, + ManualGrid, ("Brightfield", ImageArtifactType), ), ( DefineGridManualModule.DefinitionMethod.automatic, - DefineGridVariant.AUTOMATIC, + AutomaticGrid, ("Guides", ObjectLabelsArtifactType), ), ), ) def test_define_grid_declares_its_exact_geometry_source( definition_method: DefineGridManualModule.DefinitionMethod, - variant: DefineGridVariant, + variant: type[DefineGridVariant], expected_source: tuple[str, type], ) -> None: current_image = ArtifactSpec.input("Brightfield", ImageArtifactType) @@ -1624,7 +1626,7 @@ def test_define_grid_declares_its_exact_geometry_source( contract = DefineGridManualModule.callable_contract( module=module, invocation_key=FunctionInvocationKey( - function_name=variant.value, + function_name=variant.function_name, group_key=DEFAULT_GROUP_KEY, position=0, ), diff --git a/tests/unit/test_experimental_layout_rows.py b/tests/unit/test_experimental_layout_rows.py index b3303b676e..f2f04a8ba1 100644 --- a/tests/unit/test_experimental_layout_rows.py +++ b/tests/unit/test_experimental_layout_rows.py @@ -13,7 +13,6 @@ write_values_heat_map, ) from openhcs.processing.backends.experimental_analysis.layout_rows import ( - ExperimentalAnalysisScope, ExperimentalLayoutRowRole, ) from openhcs.processing.backends.experimental_analysis.result_formats import ( @@ -39,25 +38,17 @@ def test_experimental_layout_row_role_classifies_well_rows() -> None: assert ExperimentalLayoutRowRole("wells_12").specific_replicate == 12 -def test_experimental_analysis_scope_coerces_public_scope_values() -> None: - assert ExperimentalAnalysisScope.coerce("EDDU_CX5") is ExperimentalAnalysisScope.CX5 - assert ( - ExperimentalAnalysisScope.coerce("EDDU_metaxpress") - is ExperimentalAnalysisScope.METAXPRESS - ) - - def test_experimental_analysis_scope_selects_nominal_result_format() -> None: assert isinstance( - ExperimentalResultFormatStrategy.for_enum_member(ExperimentalAnalysisScope.CX5), + ExperimentalResultFormatStrategy.for_scope("EDDU_CX5"), CX5ExperimentalResultFormat, ) assert isinstance( - ExperimentalResultFormatStrategy.for_enum_member( - ExperimentalAnalysisScope.METAXPRESS - ), + ExperimentalResultFormatStrategy.for_scope("EDDU_metaxpress"), MetaXpressExperimentalResultFormat, ) + with pytest.raises(ValueError, match="not known"): + ExperimentalResultFormatStrategy.for_scope("EDDU_unknown") def test_cx5_result_format_owns_rawdata_sheet(monkeypatch) -> None: From 9f59e934b3053bd18582e5ca1d63ecd7c3fb0cb2 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 10:58:25 -0400 Subject: [PATCH 20/36] E1: step input dependencies are a family; delete three status/role enums StepInputDependency packed four variants into a kind enum plus optional fields that every consumer re-checked; each case is now a frozen subclass (UnresolvedStepInput, NoMainFlowInput, PipelineStartInput, StepOutputInput), and only StepOutputInput carries the producer step. SourceImageSetComponentRole restated a membership test; SourceAnchorSelectionStatus was written and never read; SourceBindingMatchResolutionStatus restated whether a binding matched. Co-Authored-By: Claude Opus 5.5 --- openhcs/core/compiled_step_plan.py | 4 +- openhcs/core/pipeline/artifact_planning.py | 18 ++- .../pipeline/funcstep_contract_validator.py | 7 +- openhcs/core/pipeline/path_planner.py | 38 ++--- openhcs/core/source_binding_selection.py | 38 +---- openhcs/core/source_matching.py | 20 +-- openhcs/core/step_dependencies.py | 135 ++++++------------ openhcs/core/steps/function_execution.py | 5 +- .../core/steps/function_output_manifest.py | 11 +- openhcs/core/steps/function_runtime.py | 16 +-- tests/unit/test_compilation_session.py | 15 +- ...st_function_execution_lifecycle_anchors.py | 8 +- ...st_function_output_manifest_input_scope.py | 12 +- tests/unit/test_function_outputs.py | 6 +- ...test_function_runtime_source_projection.py | 80 +++++------ .../unit/test_function_step_execution_plan.py | 9 +- .../test_function_step_execution_scope.py | 18 +-- ..._gray_to_color_payload_axis_compile_432.py | 6 +- .../test_implicit_main_flow_compilation.py | 4 +- .../test_loaded_pattern_group_ownership.py | 28 ++-- .../test_opaque_main_flow_image_domain.py | 6 +- .../unit/test_path_planner_materialization.py | 27 ++-- tests/unit/test_payload_axis_compile_gate.py | 34 ++--- tests/unit/test_source_bindings.py | 11 +- ..._source_bound_matched_image_set_anchors.py | 12 +- 25 files changed, 227 insertions(+), 341 deletions(-) diff --git a/openhcs/core/compiled_step_plan.py b/openhcs/core/compiled_step_plan.py index 4c51c4a0cf..7aa2bcada4 100644 --- a/openhcs/core/compiled_step_plan.py +++ b/openhcs/core/compiled_step_plan.py @@ -27,7 +27,7 @@ CompiledSourceBindingPlan, CompiledSourceUniversePlan, ) -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import StepInputDependency, UnresolvedStepInput from openhcs.core.axes import Axis if TYPE_CHECKING: @@ -276,7 +276,7 @@ class CompiledStepPlan: default_factory=SequentialRuntimeFilterPlan.disabled ) main_input_dependency: StepInputDependency = field( - default_factory=StepInputDependency.unresolved + default_factory=UnresolvedStepInput ) source_binding_plan: CompiledSourceBindingPlan = field( default_factory=CompiledSourceBindingPlan diff --git a/openhcs/core/pipeline/artifact_planning.py b/openhcs/core/pipeline/artifact_planning.py index b6db353918..d9af3216f5 100644 --- a/openhcs/core/pipeline/artifact_planning.py +++ b/openhcs/core/pipeline/artifact_planning.py @@ -50,7 +50,13 @@ from openhcs.core.callable_contract import FunctionStepExecutionScope from openhcs.core.source_bindings import CompiledSourceBindingPlan from openhcs.core.config import StepSourceBindingsConfig -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import ( + NoMainFlowInput, + PipelineStartInput, + StepInputDependency, + StepOutputInput, + UnresolvedStepInput, +) from openhcs.core.steps.function_step import FunctionStep if TYPE_CHECKING: @@ -240,7 +246,7 @@ class ArtifactGraph: field(default_factory=dict, repr=False) ) main_input_dependency: StepInputDependency = field( - default_factory=StepInputDependency.unresolved + default_factory=UnresolvedStepInput ) source_binding_plan: CompiledSourceBindingPlan = field( default_factory=CompiledSourceBindingPlan @@ -271,13 +277,13 @@ def resolve_main_input_dependency( isinstance(step, FunctionStep) and execution_scope is FunctionStepExecutionScope.PLATE ): - return StepInputDependency.no_main_flow() + return NoMainFlowInput() if ( step_index == 0 or step.processing_config.input_source == InputSource.PIPELINE_START ): - return StepInputDependency.pipeline_start() + return PipelineStartInput() local_output_refs = frozenset( producer.spec.ref() for producer in self.producers @@ -352,7 +358,7 @@ def resolve_main_input_dependency( f"Main-flow artifact producer step {producer_index} has no " "compiled scope identity." ) - return StepInputDependency.step_output( + return StepOutputInput( source_step_index=producer_index, source_step_scope_id=producer_scope_id, ) @@ -367,7 +373,7 @@ def resolve_main_input_dependency( return previous_dependency producer_scope_id = step_scope_ids[producer_index] - return StepInputDependency.step_output( + return StepOutputInput( source_step_index=producer_index, source_step_scope_id=producer_scope_id, ) diff --git a/openhcs/core/pipeline/funcstep_contract_validator.py b/openhcs/core/pipeline/funcstep_contract_validator.py index e9b97c74a6..50940b4e8a 100644 --- a/openhcs/core/pipeline/funcstep_contract_validator.py +++ b/openhcs/core/pipeline/funcstep_contract_validator.py @@ -38,7 +38,7 @@ ) from openhcs.core.steps.function_step import FunctionStep from openhcs.core.runtime_stores import RuntimeArtifactBatch -from openhcs.core.step_dependencies import StepInputDependencyKind +from openhcs.core.step_dependencies import NoMainFlowInput from openhcs.core.variable_component_stack_requirement import ( VariableComponentStackRequirementRequest, ) @@ -661,12 +661,11 @@ def validate_compiled_step_plan(step_plan) -> None: f"Plate-scoped FunctionStep {step_name!r} cannot use a dict pattern." ) if ( - step_plan.main_input_dependency.kind - is not StepInputDependencyKind.NO_MAIN_FLOW + not isinstance(step_plan.main_input_dependency, NoMainFlowInput) ): raise ValueError( f"Plate-scoped FunctionStep {step_name!r} must compile with " - "StepInputDependency.no_main_flow()." + "NoMainFlowInput()." ) FuncStepContractValidator.validate_dataset_callable_contracts( contracts, diff --git a/openhcs/core/pipeline/path_planner.py b/openhcs/core/pipeline/path_planner.py index 720fd216f5..fc3bb9d352 100644 --- a/openhcs/core/pipeline/path_planner.py +++ b/openhcs/core/pipeline/path_planner.py @@ -72,8 +72,10 @@ source_binding_group_keys_for_group_by, ) from openhcs.core.step_dependencies import ( + NoMainFlowInput, + PipelineStartInput, StepInputDependency, - StepInputDependencyKind, + StepOutputInput, ) from openhcs.core.steps.abstract import AbstractStep from openhcs.core.steps.function_step import FunctionStep @@ -1851,18 +1853,11 @@ def validate(self) -> None: for i in range(1, self.planner.session.step_count): curr = self.planner.session.pipeline.steps[i] dependency = self.planner.plans[i].main_input_dependency - if dependency.kind in ( - StepInputDependencyKind.NO_MAIN_FLOW, - StepInputDependencyKind.PIPELINE_START, - ): - continue - if dependency.kind is not StepInputDependencyKind.STEP_OUTPUT: + if not dependency.is_resolved: raise ValueError(f"Step {curr.name} has unresolved main input dependency.") + if not isinstance(dependency, StepOutputInput): + continue source_step_index = dependency.source_step_index - if source_step_index is None: - raise ValueError( - f"Step {curr.name} main input dependency is missing source_step_index." - ) curr_in = self.planner.plans[i].input_dir source_out = self.planner.plans[source_step_index].output_dir if curr_in != source_out: @@ -2170,9 +2165,7 @@ def input_component_scopes( main_input_dependency: StepInputDependency, ) -> PathPlannerComponentScopes: """Return component scopes visible on a step's main-flow input branch.""" - if main_input_dependency.kind is StepInputDependencyKind.PIPELINE_START: - return PathPlannerComponentScopes.empty() - source_step_index = main_input_dependency.source_step_index + source_step_index = main_input_dependency.predecessor_step_index() if source_step_index is None: return PathPlannerComponentScopes.empty() return self.planner.main_flow_component_scopes.get( @@ -2256,30 +2249,23 @@ def step_io_dirs( ) -> tuple[Path, Path]: """Resolve read/write directories for one step.""" plan = self.planner.plans.get(step_index) - reads_from_pipeline_start = ( - main_input_dependency.kind is StepInputDependencyKind.PIPELINE_START - ) - has_no_main_flow = ( - main_input_dependency.kind is StepInputDependencyKind.NO_MAIN_FLOW - ) - if plan is not None and plan.input_dir is not None: input_dir = Path(plan.input_dir) - elif reads_from_pipeline_start: + elif isinstance(main_input_dependency, PipelineStartInput): input_dir = self.planner.initial_input - elif has_no_main_flow: + elif isinstance(main_input_dependency, NoMainFlowInput): input_dir = self.planner.paths.build_output_path() else: - source_step_index = main_input_dependency.source_step_index + source_step_index = main_input_dependency.predecessor_step_index() if source_step_index is None: raise ValueError( - f"Step {step_index} main input dependency is missing source_step_index." + f"Step {step_index} main input dependency is unresolved." ) input_dir = Path(self.planner.plans[source_step_index].output_dir) if plan is not None and plan.output_dir is not None: output_dir = Path(plan.output_dir) - elif reads_from_pipeline_start or has_no_main_flow: + elif isinstance(main_input_dependency, PipelineStartInput | NoMainFlowInput): output_dir = self.planner.paths.build_output_path() else: output_dir = input_dir diff --git a/openhcs/core/source_binding_selection.py b/openhcs/core/source_binding_selection.py index 9b77c9281a..ba14297035 100644 --- a/openhcs/core/source_binding_selection.py +++ b/openhcs/core/source_binding_selection.py @@ -4,7 +4,6 @@ from abc import ABC, abstractmethod from dataclasses import dataclass, field, replace -from enum import Enum from functools import lru_cache from pathlib import Path from types import MappingProxyType @@ -603,19 +602,11 @@ def compatible_candidates( ) -class SourceAnchorSelectionStatus(str, Enum): - """Outcome of source-bound execution-anchor resolution.""" - - SELECTED = "selected" - DEFERRED_TO_RUNTIME = "deferred_to_runtime" - - @dataclass(frozen=True, slots=True) class SourceAnchorPatternSelection: """Resolved source-compatible anchors plus the authority that owns them.""" patterns: tuple[SourceCandidatePath, ...] - status: SourceAnchorSelectionStatus reason: str @classmethod @@ -625,11 +616,7 @@ def selected( *, reason: str = "source selectors resolved at anchor boundary", ) -> "SourceAnchorPatternSelection": - return cls( - patterns=tuple(patterns), - status=SourceAnchorSelectionStatus.SELECTED, - reason=reason, - ) + return cls(patterns=tuple(patterns), reason=reason) @classmethod def deferred_to_runtime( @@ -638,15 +625,7 @@ def deferred_to_runtime( *, reason: str, ) -> "SourceAnchorPatternSelection": - return cls( - patterns=tuple(patterns), - status=SourceAnchorSelectionStatus.DEFERRED_TO_RUNTIME, - reason=reason, - ) - - @property - def owns_runtime_resolution(self) -> bool: - return self.status is SourceAnchorSelectionStatus.DEFERRED_TO_RUNTIME + return cls(patterns=tuple(patterns), reason=reason) @dataclass(frozen=True, slots=True) @@ -664,19 +643,12 @@ def allows_runtime_completion(self) -> bool: ) -class SourceBindingMatchResolutionStatus(str, Enum): - """Outcome of matching one anchor pattern to source-binding aliases.""" - - MATCHED = "matched" - DEFERRED_TO_RUNTIME = "deferred_to_runtime" - - @dataclass(frozen=True, slots=True) class SourceBindingMatchResolution: """Alias match result for one source-bound anchor pattern.""" - status: SourceBindingMatchResolutionStatus binding: NamedSourceBinding | None + """The one matching binding; none when resolution is deferred to runtime.""" reason: str @classmethod @@ -685,7 +657,6 @@ def matched( binding: NamedSourceBinding, ) -> "SourceBindingMatchResolution": return cls( - status=SourceBindingMatchResolutionStatus.MATCHED, binding=binding, reason="exactly one selector binding matched the anchor", ) @@ -697,7 +668,6 @@ def deferred_to_runtime( reason: str, ) -> "SourceBindingMatchResolution": return cls( - status=SourceBindingMatchResolutionStatus.DEFERRED_TO_RUNTIME, binding=None, reason=reason, ) @@ -711,7 +681,7 @@ def require_binding(self) -> NamedSourceBinding: @property def owns_runtime_resolution(self) -> bool: - return self.status is SourceBindingMatchResolutionStatus.DEFERRED_TO_RUNTIME + return self.binding is None class SourceBoundAnchorPatternPolicy( diff --git a/openhcs/core/source_matching.py b/openhcs/core/source_matching.py index 09a98947c1..dfb5851b49 100644 --- a/openhcs/core/source_matching.py +++ b/openhcs/core/source_matching.py @@ -5,7 +5,6 @@ import re from abc import ABC, abstractmethod from dataclasses import dataclass -from enum import Enum from functools import lru_cache from typing import ClassVar, Mapping, Sequence, TYPE_CHECKING, TypeAlias @@ -104,13 +103,6 @@ def _source_filters_match_cached( return all(any_group_matches.values()) -class SourceImageSetComponentRole(Enum): - """Semantic role of a source metadata component within an image set.""" - - IMAGE_SET_AXIS = "image_set_axis" - IMAGE_PLANE_MEMBER = "image_plane_member" - - @dataclass(frozen=True, slots=True) class SourceImageSetIdentityPolicy: """Nominal policy for reducing source plane metadata to image-set identity.""" @@ -124,18 +116,12 @@ def __post_init__(self) -> None: frozenset(self.plane_member_components), ) - def role(self, component: type[Axis]) -> SourceImageSetComponentRole: - """Return whether a component identifies the image set or a plane in it.""" - if component in self.plane_member_components: - return SourceImageSetComponentRole.IMAGE_PLANE_MEMBER - return SourceImageSetComponentRole.IMAGE_SET_AXIS - def identity_components(self) -> tuple[type[Axis], ...]: """Return generated source components that identify one image set.""" return tuple( component for component in AxisFamily.active().axes - if self.role(component) is SourceImageSetComponentRole.IMAGE_SET_AXIS + if self.is_identity_component(component) ) @classmethod @@ -157,8 +143,8 @@ def from_plane_member_fields( return cls(frozenset(plane_member_components)) def is_identity_component(self, component: type[Axis]) -> bool: - """Return whether a metadata component participates in image-set identity.""" - return self.role(component) is SourceImageSetComponentRole.IMAGE_SET_AXIS + """Whether a component identifies the image set, not a plane within it.""" + return component not in self.plane_member_components def including_source_bindings( self, diff --git a/openhcs/core/step_dependencies.py b/openhcs/core/step_dependencies.py index fe5d188831..de09a44ad9 100644 --- a/openhcs/core/step_dependencies.py +++ b/openhcs/core/step_dependencies.py @@ -2,111 +2,70 @@ from __future__ import annotations +from abc import ABC from dataclasses import dataclass -from enum import Enum +from typing import ClassVar -class StepInputDependencyKind(str, Enum): - """Closed family of main-input dependency kinds.""" +@dataclass(frozen=True, slots=True) +class StepInputDependency(ABC): + """Where one compiled step's main input comes from; each case is a subclass.""" - UNRESOLVED = "unresolved" - NO_MAIN_FLOW = "no_main_flow" - PIPELINE_START = "pipeline_start" - STEP_OUTPUT = "step_output" + is_resolved: ClassVar[bool] = True + """Whether compiled planning can use this dependency.""" - @property - def has_predecessor(self) -> bool: - """Whether this dependency kind names an upstream compiled step.""" + def predecessor_step_index(self) -> int | None: + """The exact upstream compiled step, if one exists.""" - return self is type(self).STEP_OUTPUT + return None - @property - def is_pipeline_start(self) -> bool: - """Whether this dependency kind terminates at the pipeline source.""" + def uses_pipeline_start_anchors(self) -> bool: + """Only a pipeline-start main input is filtered against raw source bindings.""" - return self is type(self).PIPELINE_START + return False - @property - def is_resolved(self) -> bool: - """Whether this dependency kind is usable for compiled planning.""" + def require_pipeline_start(self) -> None: + """Prove that this terminal dependency reaches the pipeline source.""" - return self is not type(self).UNRESOLVED + raise ValueError( + "Terminal step-input dependency is not pipeline-start: " + f"{type(self).__name__}." + ) @dataclass(frozen=True, slots=True) -class StepInputDependency: - """Authoritative main-input edge for one compiled step.""" - - kind: StepInputDependencyKind - source_step_index: int | None = None - source_step_scope_id: str | None = None - - @classmethod - def unresolved(cls) -> "StepInputDependency": - return cls(StepInputDependencyKind.UNRESOLVED) - - @classmethod - def pipeline_start(cls) -> "StepInputDependency": - return cls(StepInputDependencyKind.PIPELINE_START) - - @classmethod - def no_main_flow(cls) -> "StepInputDependency": - return cls(StepInputDependencyKind.NO_MAIN_FLOW) - - @classmethod - def step_output( - cls, - *, - source_step_index: int, - source_step_scope_id: str, - ) -> "StepInputDependency": - return cls( - kind=StepInputDependencyKind.STEP_OUTPUT, - source_step_index=source_step_index, - source_step_scope_id=source_step_scope_id, - ) +class UnresolvedStepInput(StepInputDependency): + """Not yet resolved by compilation.""" - def __post_init__(self) -> None: - if self.kind.has_predecessor: - if self.source_step_index is None: - raise ValueError( - "StepInputDependency.step_output requires source_step_index." - ) - if not self.source_step_scope_id: - raise ValueError( - "StepInputDependency.step_output requires source_step_scope_id." - ) - return - - if self.source_step_index is not None or self.source_step_scope_id is not None: - raise ValueError( - f"StepInputDependency kind {self.kind.value!r} cannot carry a source step." - ) - - @property - def is_resolved(self) -> bool: - return self.kind.is_resolved + is_resolved = False - def predecessor_step_index(self) -> int | None: - """Project the exact upstream compiled step, if one exists.""" - - if not self.kind.has_predecessor: - return None - if self.source_step_index is None: - raise ValueError( - "StepInputDependency.step_output requires source_step_index." - ) - return self.source_step_index + +@dataclass(frozen=True, slots=True) +class NoMainFlowInput(StepInputDependency): + """The step reads no main-flow image stack.""" + + +@dataclass(frozen=True, slots=True) +class PipelineStartInput(StepInputDependency): + """The step reads the pipeline's source planes.""" def uses_pipeline_start_anchors(self) -> bool: - """Only a pipeline-start main input is filtered against raw source bindings.""" - return self.kind.is_pipeline_start + return True def require_pipeline_start(self) -> None: - """Prove that this terminal dependency reaches the pipeline source.""" + return None + - if not self.kind.is_pipeline_start: - raise ValueError( - "Terminal step-input dependency is not pipeline-start: " - f"{self.kind.value!r}." - ) +@dataclass(frozen=True, slots=True) +class StepOutputInput(StepInputDependency): + """The step reads one upstream step's main-flow output.""" + + source_step_index: int + source_step_scope_id: str + + def __post_init__(self) -> None: + if not self.source_step_scope_id: + raise ValueError("StepOutputInput requires source_step_scope_id.") + + def predecessor_step_index(self) -> int: + return self.source_step_index diff --git a/openhcs/core/steps/function_execution.py b/openhcs/core/steps/function_execution.py index feb3070afc..224534ef8b 100644 --- a/openhcs/core/steps/function_execution.py +++ b/openhcs/core/steps/function_execution.py @@ -37,7 +37,7 @@ SourcePatternResolutionContext, ) from openhcs.core.source_bindings import CompiledSourceBindingPlan -from openhcs.core.step_dependencies import StepInputDependencyKind +from openhcs.core.step_dependencies import PipelineStartInput from openhcs.core.steps.function_io import ( generate_materialized_paths, get_all_image_paths, @@ -794,8 +794,7 @@ def _detect_patterns(self) -> dict[str, DiscoveredPatternCollection]: if source_files is None: source_projection = self.context.runtime_source_workspace_projections.projection_if_available() if ( - plan.main_input_dependency.kind - is StepInputDependencyKind.PIPELINE_START + isinstance(plan.main_input_dependency, PipelineStartInput) and source_projection is not None ): source_files = source_projection.pipeline_start_files( diff --git a/openhcs/core/steps/function_output_manifest.py b/openhcs/core/steps/function_output_manifest.py index 58513540ce..09587f438d 100644 --- a/openhcs/core/steps/function_output_manifest.py +++ b/openhcs/core/steps/function_output_manifest.py @@ -22,7 +22,7 @@ from openhcs.core.runtime_plane_projection import RuntimePlaneAxis from openhcs.core.source_metadata import DurableSourceMetadata, SourceMetadataValue -from openhcs.core.step_dependencies import StepInputDependencyKind +from openhcs.core.step_dependencies import StepOutputInput from openhcs.core.steps.function_output_identity import ( FunctionOutputIdentity, ) @@ -359,9 +359,7 @@ def _main_input_producer_key( plan: CompiledStepPlan, ) -> StepOutputManifestKey | None: dependency = plan.main_input_dependency - if dependency.kind is not StepInputDependencyKind.STEP_OUTPUT: - return None - if dependency.source_step_scope_id is None: + if not isinstance(dependency, StepOutputInput): return None return StepOutputManifestKey(dependency.source_step_scope_id, plan.axis_id) @@ -566,10 +564,7 @@ def _requested_producer_outputs( plan: CompiledStepPlan, ) -> frozenset[tuple[str, str, str | None]]: dependency = plan.main_input_dependency - if ( - dependency.kind is not StepInputDependencyKind.STEP_OUTPUT - or dependency.source_step_scope_id is None - ): + if not isinstance(dependency, StepOutputInput): return frozenset() producer_scope_id = dependency.source_step_scope_id return frozenset( diff --git a/openhcs/core/steps/function_runtime.py b/openhcs/core/steps/function_runtime.py index f5b26f25f1..aa2c67291b 100644 --- a/openhcs/core/steps/function_runtime.py +++ b/openhcs/core/steps/function_runtime.py @@ -127,7 +127,7 @@ RuntimePlaneAxisValueProjection, RuntimePlaneProjection, ) -from openhcs.core.step_dependencies import StepInputDependencyKind +from openhcs.core.step_dependencies import PipelineStartInput, StepOutputInput from openhcs.core.steps.function_output_manifest import ( NoStepOutputManifestMatch, ProducedOutputSemantics, @@ -679,8 +679,7 @@ def load_input_stack( tuple(full_file_paths), ( Backend.MEMORY - if plan.main_input_dependency.kind - is StepInputDependencyKind.STEP_OUTPUT + if isinstance(plan.main_input_dependency, StepOutputInput) else Backend(plan.read_backend) ), ).load_images(context.filemanager, zarr_config=plan.zarr_config) @@ -744,8 +743,7 @@ def _filter_matching_files_for_group( ) -> list[str]: """Constrain grouped executions to files from the current component.""" if ( - self.execution_plan.main_input_dependency.kind - is StepInputDependencyKind.STEP_OUTPUT + isinstance(self.execution_plan.main_input_dependency, StepOutputInput) or self.compiled_group.runtime_domain is RuntimeInvocationDomain.ARTIFACT_MANAGED ): @@ -781,8 +779,7 @@ def _filter_matching_files_for_source_bindings( """Constrain the loaded main stack to declared image source bindings.""" if ( - self.execution_plan.main_input_dependency.kind - is StepInputDependencyKind.STEP_OUTPUT + isinstance(self.execution_plan.main_input_dependency, StepOutputInput) ): return matching_files @@ -1679,8 +1676,9 @@ def loaded_source_artifact_payload( if ( binding is None or self.invocation.key.position != 0 - or self.group_data.execution_plan.main_input_dependency.kind - is not StepInputDependencyKind.PIPELINE_START + or not isinstance( + self.group_data.execution_plan.main_input_dependency, PipelineStartInput + ) or binding.origin is not FromPipelineStart or request.context is not self.group_data.context ): diff --git a/tests/unit/test_compilation_session.py b/tests/unit/test_compilation_session.py index 076da849da..526070ad67 100644 --- a/tests/unit/test_compilation_session.py +++ b/tests/unit/test_compilation_session.py @@ -51,7 +51,12 @@ PairByOrder, SourceSelector, ) -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import ( + PipelineStartInput, + StepInputDependency, + StepOutputInput, + UnresolvedStepInput, +) from openhcs.core.steps.function_step import FunctionStep from openhcs.processing.backends.analysis.neurite_outgrowth import ( neurite_outgrowth_metaxpress, @@ -298,7 +303,7 @@ def _compile_source_plans_for_contract( session: CompilationSession, snapshot: AbstractStep, func, - main_input_dependency: StepInputDependency = StepInputDependency.pipeline_start(), + main_input_dependency: StepInputDependency = PipelineStartInput(), ): execution_bindings = snapshot.source_bindings.compiled_plan_for_contracts((CallableContract.from_callable(func),), main_input_dependency, ArtifactDeclarationStepContext.empty().available_artifacts) fields = snapshot.source_bindings.metadata_fields_for_literal_types( @@ -382,7 +387,7 @@ def test_compilation_session_shares_resolved_pipeline_and_owns_axis_plans(monkey ) first_graph = session.pipeline.artifact_graphs[0] - assert first_graph.main_input_dependency == StepInputDependency.pipeline_start() + assert first_graph.main_input_dependency == PipelineStartInput() assert first_graph.source_binding_plan.bindings == ( session.pipeline.steps[0].source_bindings.bindings[0], ) @@ -752,7 +757,7 @@ def counted_provider(cls, pipeline): ) assert plan is not None - assert dependency_before_provider == StepInputDependency.unresolved() + assert dependency_before_provider == UnresolvedStepInput() assert session.plan(0).main_input_dependency == dependency_before_provider assert plan.contract.artifact_inputs.names() == ("DNA",) @@ -810,7 +815,7 @@ def test_path_planner_step_output_projects_only_exact_source_artifacts() -> None session, snapshot, _previous_step_and_external_source, - StepInputDependency.step_output( + StepOutputInput( source_step_index=0, source_step_scope_id="plate::functionstep_0", ), diff --git a/tests/unit/test_function_execution_lifecycle_anchors.py b/tests/unit/test_function_execution_lifecycle_anchors.py index c13e1c0da6..8b9fe2ce4f 100644 --- a/tests/unit/test_function_execution_lifecycle_anchors.py +++ b/tests/unit/test_function_execution_lifecycle_anchors.py @@ -26,7 +26,7 @@ from openhcs.core.source_workspace_projection import WorkspaceSourceProjections from openhcs.core.steps.function_output_manifest import _STEP_OUTPUT_MANIFESTS from openhcs.core.source_bindings import CompiledSourceBindingPlan, NamedSourceBinding -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import NoMainFlowInput, StepOutputInput from openhcs.core.steps.function_execution import FunctionStepExecutor from openhcs.domains.microscopy.axes import Microscopy @@ -70,11 +70,11 @@ def _anchor_filter(plan: object, output_manifest: object) -> FunctionStepExecuto @pytest.mark.parametrize( "dependency", ( - StepInputDependency.step_output( + StepOutputInput( source_step_index=4, source_step_scope_id="non-adjacent-main-flow-producer", ), - StepInputDependency.no_main_flow(), + NoMainFlowInput(), ), ) def test_storage_backed_cross_group_uses_producer_lifecycle_anchor(dependency) -> None: @@ -170,7 +170,7 @@ def test_source_anchored_group_uses_exact_main_flow_producer_manifest() -> None: plan = SimpleNamespace( step_index=6, step_name="SourceConsumer", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=2, source_step_scope_id="exact-main-flow-producer", ), diff --git a/tests/unit/test_function_output_manifest_input_scope.py b/tests/unit/test_function_output_manifest_input_scope.py index 0e629ba66b..736a537471 100644 --- a/tests/unit/test_function_output_manifest_input_scope.py +++ b/tests/unit/test_function_output_manifest_input_scope.py @@ -27,7 +27,7 @@ compile_function_pattern, ) from openhcs.core.pipeline.function_contracts import artifact_inputs, artifact_outputs -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import StepOutputInput from openhcs.core.steps.function_output_identity import FunctionOutputIdentity from openhcs.core.steps.function_output_manifest import ( ProducedOutputSemantics, @@ -253,7 +253,7 @@ def test_foreign_artifact_inputs_do_not_filter_lifecycle_producer( ) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id=dependency_scope, ), @@ -339,7 +339,7 @@ def consume(image): ) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="align", ), @@ -436,7 +436,7 @@ def calculate_illumination(image): ) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="color_to_gray", ), @@ -494,7 +494,7 @@ def test_storage_backed_input_does_not_reclassify_lifecycle_output( stored_input = ArtifactSpec.input("StoredLabels", ObjectLabelsArtifactType) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="artifact_producer", ), @@ -574,7 +574,7 @@ def test_same_scope_parameter_bound_input_does_not_select_lifecycle_output( ) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="artifact_producer", ), diff --git a/tests/unit/test_function_outputs.py b/tests/unit/test_function_outputs.py index 1f39d69f59..164cfabe2d 100644 --- a/tests/unit/test_function_outputs.py +++ b/tests/unit/test_function_outputs.py @@ -62,7 +62,7 @@ ) from openhcs.core.source_projection import SourceProjectionMetadataSerializer from openhcs.core.source_spatial_domain import SourceSpatialDomain -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import NoMainFlowInput, StepOutputInput from openhcs.core.steps.function_output_identity import ( FunctionOutputIdentity, FunctionOutputPathRequest, @@ -427,7 +427,7 @@ def function_step_plan( output_dir=Path("/tmp/output"), pipeline_position=pipeline_position, step_scope_id=f"step-scope-{pipeline_position}", - main_input_dependency=StepInputDependency.no_main_flow(), + main_input_dependency=NoMainFlowInput(), variable_components=variable_components, compiled_function_pattern=compile_function_pattern(lambda image: image, {}, {}), ) @@ -574,7 +574,7 @@ def test_step_output_manifest_prefers_main_dependency_over_auxiliary_artifact_in ) consumer = function_step_plan("Watershed") - consumer.main_input_dependency = StepInputDependency.step_output( + consumer.main_input_dependency = StepOutputInput( source_step_index=24, source_step_scope_id="main-producer", ) diff --git a/tests/unit/test_function_runtime_source_projection.py b/tests/unit/test_function_runtime_source_projection.py index 610ad45624..a395417c40 100644 --- a/tests/unit/test_function_runtime_source_projection.py +++ b/tests/unit/test_function_runtime_source_projection.py @@ -100,7 +100,7 @@ VirtualWorkspaceSourceProjection, VirtualWorkspaceSourceProjectionCache, ) -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import PipelineStartInput, StepOutputInput from openhcs.core.steps.function_execution import ( FunctionStepExecutor, ) @@ -931,7 +931,7 @@ def test_step_output_manifest_scopes_previous_step_inputs(tmp_path: Path) -> Non ) consumer = SimpleNamespace( axis_id="A14", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=3, source_step_scope_id="enhance", ), @@ -1019,7 +1019,7 @@ def test_step_output_manifest_pattern_lookup_returns_producer_memory_paths( ) consumer = SimpleNamespace( axis_id="A14", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=4, source_step_scope_id="mask_image", ), @@ -1097,7 +1097,7 @@ def test_step_output_manifest_does_not_treat_artifact_inputs_as_main_flow( ) consumer = SimpleNamespace( axis_id="A14", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=CompiledSourceBindingPlan(), artifact_inputs={ plan.ref(): plan @@ -1197,7 +1197,7 @@ def test_step_output_manifest_uses_declared_artifact_producer_scope( ) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=3, source_step_scope_id="crop_red", ), @@ -1293,7 +1293,7 @@ def test_special_artifact_input_does_not_narrow_step_output_main_flow( ) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=2, source_step_scope_id="image_set", ), @@ -1354,7 +1354,7 @@ def test_special_artifact_input_does_not_narrow_step_output_main_flow( def test_step_output_manifest_ignores_sidecar_artifact_for_anchor_filtering() -> None: consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=CompiledSourceBindingPlan(), artifact_inputs={ plan.ref(): plan @@ -1382,7 +1382,7 @@ def test_step_output_manifest_ignores_sidecar_artifact_for_anchor_filtering() -> def test_step_output_manifest_preserves_pipeline_start_source_bound_anchor() -> None: consumer = SimpleNamespace( axis_id="A14", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=CompiledSourceBindingPlan( bindings=(NamedSourceBinding(alias="OrigActin_Golgi_Membrane"),) ), @@ -1410,7 +1410,7 @@ def test_step_output_manifest_preserves_pipeline_start_source_bound_anchor() -> def test_step_output_anchor_filter_skips_source_binding_filter() -> None: plan = SimpleNamespace( - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=4, source_step_scope_id="mask_image", ), @@ -1435,7 +1435,7 @@ def test_step_output_anchor_filter_skips_source_binding_filter() -> None: def test_step_output_anchor_uses_compiler_owned_component_scope() -> None: plan = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=4, source_step_scope_id="object_to_image", ), @@ -1475,7 +1475,7 @@ def identify_secondary(image): axis_id="A01", step_index=4, step_name="IdentifySecondaryObjects", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=3, source_step_scope_id="identify_primary", ), @@ -1549,7 +1549,7 @@ def identify_primary_objects(image, *, runtime): axis_id="A01", step_index=2, step_name="IdentifyPrimaryObjects", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="crop_green_red", ), @@ -1624,7 +1624,7 @@ def producer_output_contexts_for_paths(self, *_args): def test_step_output_anchor_resolves_dynamic_component_scope_from_patterns() -> None: plan = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=4, source_step_scope_id="crop", ), @@ -1660,7 +1660,7 @@ def test_step_output_anchor_resolves_dynamic_component_scope_from_patterns() -> def test_source_anchor_uses_compiler_owned_static_component_scope() -> None: plan = SimpleNamespace( - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), execution_group_scope=ComponentGroupScope.from_raw( ("1",), component=Microscopy.Channel, @@ -1741,7 +1741,7 @@ def measure_source_set(image): axis_id="A01", step_index=0, step_name="SourceBoundAnchorFilter", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=source_binding_plan, execution_group_scope=ComponentGroupScope.from_raw( ("1", "2"), @@ -1907,7 +1907,7 @@ def save_image(image, *, image_to_save: np.ndarray): axis_id="A01", step_index=16, step_name="SaveImages", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=source_binding_plan, execution_group_scope=ComponentGroupScope.from_raw( ("3",), @@ -1965,7 +1965,7 @@ def exact_source_input(image): axis_id="A01", step_index=0, step_name="ExactSourceInput", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=source_binding_plan, execution_group_scope=ComponentGroupScope.from_raw( ("1",), @@ -2019,7 +2019,7 @@ def exact_source_input(image): axis_id="A01", step_index=0, step_name="ExactSourceInput", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=source_binding_plan, execution_group_scope=ComponentGroupScope.from_raw( ("1",), @@ -2068,7 +2068,7 @@ def test_pipeline_start_anchors_project_raw_selectors_to_semantic_groups() -> No ) plan = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=CompiledSourceBindingPlan(bindings=bindings), execution_group_scope=ComponentGroupScope.from_raw( ("MCP_DNA", "MCP_AGP"), @@ -2185,7 +2185,7 @@ def test_first_step_prepares_raw_source_anchors_under_semantic_binding_groups( executor = FunctionStepExecutor(context, 0) grouped_patterns = executor._prepare_groups(executor._detect_patterns()) - assert executor.plan.main_input_dependency == StepInputDependency.pipeline_start() + assert executor.plan.main_input_dependency == PipelineStartInput() assert executor.plan.source_binding_plan.has_primary_content assert grouped_patterns == { "MCP_DNA": ("A01_s{iii}_w1_z001_t001.tif",), @@ -2251,7 +2251,7 @@ def source_bound_module(image, *, runtime): ) plan = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=source_binding_plan, execution_group_value=None, execution_group_scope=ComponentGroupScope.ungrouped(), @@ -3282,7 +3282,7 @@ def test_pipeline_start_main_flow_survives_prior_producer_image_input( component_value="1", ) - assert consumer_plan.main_input_dependency == StepInputDependency.pipeline_start() + assert consumer_plan.main_input_dependency == PipelineStartInput() assert tuple( binding.alias for binding in consumer_plan.source_binding_plan.binding_declarations @@ -3673,7 +3673,7 @@ def test_grouped_runtime_source_expansion_uses_scoped_bindings( execution_plan=SimpleNamespace( axis_id="A01", execution_group_scope=ComponentGroupScope.dynamic(Microscopy.Channel), - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=source_binding_plan, compiled_function_pattern=compiled_pattern, variable_component_values=(Microscopy.Site.name,), @@ -3769,7 +3769,7 @@ def test_alias_only_workspace_filter_excludes_unselected_source_and_orders_stack step_index=0, step_name="MetaXpress", axis_id="A01", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=selected_plan, ), compiled_group=compile_function_pattern( @@ -3890,7 +3890,7 @@ def test_step_output_load_filter_skips_source_binding_filter() -> None: from openhcs.core.steps.function_runtime import PatternGroupExecutionRequest plan = SimpleNamespace( - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=4, source_step_scope_id="mask_image", ), @@ -3925,7 +3925,7 @@ def test_empty_source_binding_plan_does_not_filter_runtime_inputs() -> None: plan = SimpleNamespace( axis_id="A01", execution_group_scope=ComponentGroupScope.ungrouped(), - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=CompiledSourceBindingPlan(), ) runtime = PatternGroupExecutionRequest( @@ -4058,7 +4058,7 @@ def resolve_address(backend_address, backend, *, base_path): read_backend="memory", input_memory_type="numpy", variable_components=(), - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=source_binding_plan, ) context = SimpleNamespace( @@ -4165,7 +4165,7 @@ def load_batch(self, paths, backend): input_memory_type="numpy", variable_components=(Microscopy.ZIndex,), compiled_function_pattern=compile_function_pattern(lambda image: image, {}, {}), - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="producer", ), @@ -4238,7 +4238,7 @@ def test_artifact_managed_group_uses_compiler_group_without_filtering_anchor_fil step_index=0, step_name="source fixture", axis_id="A01", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), ), component_value="1", context=SimpleNamespace(), @@ -4262,7 +4262,7 @@ def test_step_output_group_does_not_reinterpret_producer_path_component() -> Non step_index=0, step_name="source fixture", axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=4, source_step_scope_id="object_to_image", ), @@ -4312,7 +4312,7 @@ def test_step_output_manifest_does_not_filter_main_flow_by_artifact_input( ) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="correct_illumination", ), @@ -4431,7 +4431,7 @@ def identify_stain_1(image): ) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=2, source_step_scope_id="align", ), @@ -4500,7 +4500,7 @@ def test_step_output_manifest_updates_selected_slot_and_preserves_other_componen pipeline_position=2, axis_id="A01", output_dir=output_dir, - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="producer", ), @@ -4595,7 +4595,7 @@ def test_step_output_manifest_collapsed_domain_replaces_inherited_components( pipeline_position=2, axis_id="A01", output_dir=output_dir, - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="producer", ), @@ -4666,7 +4666,7 @@ def test_step_output_manifest_new_output_address_replaces_inherited_components( pipeline_position=2, axis_id="A01", output_dir=output_dir, - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="producer", ), @@ -4750,7 +4750,7 @@ def test_step_output_manifest_grouped_subset_replaces_inherited_components( pipeline_position=2, axis_id="A01", output_dir=output_dir, - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="producer", ), @@ -4818,7 +4818,7 @@ def test_step_output_manifest_preserves_anonymous_side_effect_main_flow( ) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=2, source_step_scope_id="identify_primary", ), @@ -4879,7 +4879,7 @@ def test_step_output_manifest_accepts_source_anchor_for_qualified_output( ) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=4, source_step_scope_id="correct_illumination_apply", ), @@ -5628,7 +5628,7 @@ def qualified_producer_manifest(tmp_path): ) consumer = SimpleNamespace( axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="producer" ), compiled_function_pattern=compile_function_pattern(lambda image: image, {}, {}), @@ -5830,7 +5830,7 @@ def test_producer_loader_validates_ambiguity_before_cache( input_dir=Path("/memory"), input_memory_type="numpy", compiled_function_pattern=compile_function_pattern(lambda image: image, {}, {}), - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="producer", ), diff --git a/tests/unit/test_function_step_execution_plan.py b/tests/unit/test_function_step_execution_plan.py index 53d320cb0f..769cc30ee0 100644 --- a/tests/unit/test_function_step_execution_plan.py +++ b/tests/unit/test_function_step_execution_plan.py @@ -36,10 +36,7 @@ from openhcs.core.artifact_key_selection import AdapterRecordedArtifactOutputPolicy from openhcs.core.runtime_adapters import RuntimeAdapterRequest, runtime_adapter from openhcs.core.source_bindings import CompiledSourceBindingPlan -from openhcs.core.step_dependencies import ( - StepInputDependency, - StepInputDependencyKind, -) +from openhcs.core.step_dependencies import StepOutputInput from openhcs.core.steps.function_artifact_materialization import ( RuntimeArtifactMaterialization, ) @@ -71,7 +68,7 @@ def _compiled_plan(**overrides): output_dir=Path("/tmp/output"), variable_components=(Microscopy.Site,), group_by=None, - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="plate::functionstep_1", ), @@ -126,7 +123,7 @@ def test_compiled_step_plan_is_the_runtime_plan_owner(): assert plan.step_scope_id == "plate::functionstep_2" assert plan.execution_group_scope.keys == ("2",) assert plan.variable_components == (Microscopy.Site,) - assert plan.main_input_dependency.kind is StepInputDependencyKind.STEP_OUTPUT + assert isinstance(plan.main_input_dependency, StepOutputInput) assert plan.main_input_dependency.source_step_scope_id == "plate::functionstep_1" assert plan.source_binding_plan == CompiledSourceBindingPlan() assert plan.device_id_for(plan.input_memory_type) is None diff --git a/tests/unit/test_function_step_execution_scope.py b/tests/unit/test_function_step_execution_scope.py index 9799262e90..31ff638fa0 100644 --- a/tests/unit/test_function_step_execution_scope.py +++ b/tests/unit/test_function_step_execution_scope.py @@ -77,10 +77,7 @@ ) from openhcs.core.source_bindings import CompiledSourceBindingPlan, NamedSourceBinding from openhcs.core.source_matching import SourceImageSetIdentityPolicy -from openhcs.core.step_dependencies import ( - StepInputDependency, - StepInputDependencyKind, -) +from openhcs.core.step_dependencies import NoMainFlowInput from openhcs.core.steps.function_step import FunctionStep from openhcs.core.dataset_sources.source_schema import SourceSchemaFilenameParser from openhcs.processing.materialization import FileBundleOptions, MaterializationSpec @@ -210,12 +207,9 @@ def test_compiled_step_plan_does_not_store_execution_scope() -> None: def test_plate_dependency_has_no_main_flow() -> None: - assert StepInputDependencyKind.NO_MAIN_FLOW.value == "no_main_flow" - dependency = StepInputDependency.no_main_flow() + dependency = NoMainFlowInput() - assert dependency.kind is StepInputDependencyKind.NO_MAIN_FLOW - assert dependency.source_step_index is None - assert dependency.source_step_scope_id is None + assert dependency.predecessor_step_index() is None assert dependency.is_resolved @@ -242,7 +236,7 @@ def test_plate_scope_drives_no_main_flow_paths() -> None: ) input_dir, resolved_output_dir = planner.steps.step_io_dirs(dependency, 1) - assert dependency == StepInputDependency.no_main_flow() + assert dependency == NoMainFlowInput() assert input_dir == output_dir assert resolved_output_dir == output_dir @@ -262,7 +256,7 @@ def test_path_validation_accepts_explicit_no_main_flow_dependency( step_index=1, step_name="export", axis_id="A01", - main_input_dependency=StepInputDependency.no_main_flow(), + main_input_dependency=NoMainFlowInput(), ) } validation = PathPlannerValidationStage(planner) @@ -417,7 +411,7 @@ def _plate_step_plan( analysis_results_dir="/plate/analysis", pipeline_position=step_index + 1, variable_components=(), - main_input_dependency=StepInputDependency.no_main_flow(), + main_input_dependency=NoMainFlowInput(), artifact_inputs=input_plans, artifact_outputs=output_plans, compiled_function_pattern=compiled_pattern, diff --git a/tests/unit/test_gray_to_color_payload_axis_compile_432.py b/tests/unit/test_gray_to_color_payload_axis_compile_432.py index b2a214eddc..33f29f2424 100644 --- a/tests/unit/test_gray_to_color_payload_axis_compile_432.py +++ b/tests/unit/test_gray_to_color_payload_axis_compile_432.py @@ -19,7 +19,7 @@ from openhcs.core.pipeline.compiler import PipelineCompiler from openhcs.core.aligned_image_payload import ImagePayloadBundleContext from openhcs.core.image_payload_execution_mode import NaturalExecution -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import StepOutputInput from openhcs.interop.cellprofiler.runtime.function_contract_execution import CellProfilerFunctionContractExecutor from openhcs.processing.backends.cellprofiler.color import ( ColorToGrayMode, ImageChannelType, color_to_gray, gray_to_color, @@ -41,7 +41,7 @@ def _gray_creator_session(tmp_path): ) consumer = CompiledStepPlan( step_index=1, step_name="SplitComposedRoles", axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="step-0", ), compiled_function_pattern=_compiled_pattern(color_to_gray), @@ -263,7 +263,7 @@ def test_color_to_gray_does_not_preserve_consumed_payload_axis_for_later_consume consumer = session.plans[1] session.plans[2] = replace( consumer, step_index=2, step_name="RejectSecondScalarSplit", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="step-1", ), ) diff --git a/tests/unit/test_implicit_main_flow_compilation.py b/tests/unit/test_implicit_main_flow_compilation.py index 91fb138d72..11d80a7557 100644 --- a/tests/unit/test_implicit_main_flow_compilation.py +++ b/tests/unit/test_implicit_main_flow_compilation.py @@ -17,7 +17,7 @@ from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.pipeline.path_planner import PathPlannerGroupScope from openhcs.core.progress import set_progress_queue -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import StepOutputInput from openhcs.core.steps.function_step import FunctionStep from openhcs.processing.backends.cellprofiler.thresholding import threshold from openhcs.processing.backends.processors.numpy_processor import percentile_normalize @@ -84,7 +84,7 @@ def test_native_grouped_main_flow_compiles_into_cellprofiler_consumer(tmp_path): context = compilation.runtime_contexts["A01"] native_plan = context.step_plans[0] threshold_plan = context.step_plans[1] - assert threshold_plan.main_input_dependency == StepInputDependency.step_output( + assert threshold_plan.main_input_dependency == StepOutputInput( source_step_index=0, source_step_scope_id=native_plan.step_scope_id, ) diff --git a/tests/unit/test_loaded_pattern_group_ownership.py b/tests/unit/test_loaded_pattern_group_ownership.py index 459455f3b4..3a165c5ee1 100644 --- a/tests/unit/test_loaded_pattern_group_ownership.py +++ b/tests/unit/test_loaded_pattern_group_ownership.py @@ -9,6 +9,7 @@ import pytest from openhcs.core.compiled_step_plan import CompiledStepPlan +from openhcs.core.step_dependencies import PipelineStartInput, StepOutputInput from openhcs.core.component_group_scope import ComponentGroupScope from openhcs.core.context.processing_context import ProcessingContext from openhcs.core.function_patterns import compile_function_pattern @@ -535,7 +536,6 @@ def test_stored_primary_admission_owns_independent_buffer_and_live_primary_bindi def test_checkpoint_demand_follows_preserved_input_to_native_reader(): from openhcs.constants.constants import Backend - from openhcs.core.step_dependencies import StepInputDependency request, _ = _stored_primary_fixture() source = replace( @@ -551,7 +551,7 @@ def test_checkpoint_demand_follows_preserved_input_to_native_reader(): step_index=1, step_scope_id="preserving", compiled_function_pattern=preserving_pattern, - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="source" ), ) @@ -560,7 +560,7 @@ def test_checkpoint_demand_follows_preserved_input_to_native_reader(): step_index=2, step_scope_id="native", compiled_function_pattern=compile_function_pattern(_identity, {}, {}), - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="preserving" ), ) @@ -576,7 +576,6 @@ def test_checkpoint_demand_follows_preserved_input_to_native_reader(): @pytest.mark.parametrize("pattern_kind", ("unknown", "missing", "malformed")) def test_checkpoint_demand_preserves_dependent_group_admission(pattern_kind): from openhcs.constants.constants import Backend - from openhcs.core.step_dependencies import StepInputDependency request, _ = _stored_primary_fixture() source = replace( @@ -598,7 +597,7 @@ def test_checkpoint_demand_preserves_dependent_group_admission(pattern_kind): is_grouped=pattern_kind == "missing", ) ), - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="source" ), ) @@ -608,7 +607,7 @@ def test_checkpoint_demand_preserves_dependent_group_admission(pattern_kind): source.requires_main_flow_checkpoint(plans) else: assert source.requires_main_flow_checkpoint(plans) - consumer.main_input_dependency = StepInputDependency.step_output( + consumer.main_input_dependency = StepOutputInput( source_step_index=99, source_step_scope_id="absent" ) assert not source.requires_main_flow_checkpoint(plans) @@ -618,7 +617,6 @@ def test_checkpoint_demand_preserves_dependent_group_admission(pattern_kind): def test_checkpoint_demand_propagates_only_through_preserved_outputs(): from openhcs.constants.constants import Backend - from openhcs.core.step_dependencies import StepInputDependency request, _ = _stored_primary_fixture() source = replace( @@ -632,7 +630,7 @@ def test_checkpoint_demand_propagates_only_through_preserved_outputs(): step_index=1, step_scope_id="consumer", visualize=True, - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="source" ), ) @@ -648,7 +646,6 @@ def test_checkpoint_demand_propagates_only_through_preserved_outputs(): def test_checkpoint_demand_terminates_on_preserving_cycles_and_self_producers(): from openhcs.constants.constants import Backend - from openhcs.core.step_dependencies import StepInputDependency request, _ = _stored_primary_fixture(preserves_main_flow=True) first = replace( @@ -656,7 +653,7 @@ def test_checkpoint_demand_terminates_on_preserving_cycles_and_self_producers(): step_index=1, step_scope_id="first", write_backend=Backend.MEMORY.value, - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=2, source_step_scope_id="second" ), ) @@ -664,7 +661,7 @@ def test_checkpoint_demand_terminates_on_preserving_cycles_and_self_producers(): first, step_index=2, step_scope_id="second", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="first" ), ) @@ -674,7 +671,7 @@ def test_checkpoint_demand_terminates_on_preserving_cycles_and_self_producers(): second.streaming_configs = {"viewer": object()} assert first.requires_main_flow_checkpoint(plans) assert second.requires_main_flow_checkpoint(plans) - first.main_input_dependency = StepInputDependency.step_output( + first.main_input_dependency = StepOutputInput( source_step_index=1, source_step_scope_id="first" ) assert not first.requires_main_flow_checkpoint({1: first}) @@ -682,7 +679,7 @@ def test_checkpoint_demand_terminates_on_preserving_cycles_and_self_producers(): first, step_index=0, step_scope_id="self", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="self" ), ) @@ -778,7 +775,6 @@ def test_initial_source_input_uses_loaded_declared_cohort_without_workspace_read from openhcs.core.memory.decorators import numpy as numpy_memory from openhcs.core.runtime_plane_projection import RuntimePlaneAxis, RuntimePlaneProjection from openhcs.core.source_bindings import NamedSourceBinding - from openhcs.core.step_dependencies import StepInputDependency request, paths, payload = _fixture() binding = NamedSourceBinding(alias="Raw", origin=FromPipelineStart) @@ -805,7 +801,7 @@ def consume(image): ), source_bindings=bindings, ) - request.execution_plan.main_input_dependency = StepInputDependency.pipeline_start() + request.execution_plan.main_input_dependency = PipelineStartInput() request.execution_plan.source_binding_plan = CompiledSourceBindingPlan(bindings=(binding,)) request.execution_plan.compiled_function_pattern = pattern request = replace(request, compiled_group=pattern.default_group) @@ -862,7 +858,7 @@ def forbidden_original_read(*args, **kwargs): # A source role by itself cannot turn a predecessor's transformed pixels # into the original pipeline-start source. - request.execution_plan.main_input_dependency = StepInputDependency.step_output( + request.execution_plan.main_input_dependency = StepOutputInput( source_step_index=0, source_step_scope_id="predecessor", ) assert edge.resolve_unstored_payload(executor, payload, request=adapter) is original diff --git a/tests/unit/test_opaque_main_flow_image_domain.py b/tests/unit/test_opaque_main_flow_image_domain.py index 6ec39c90d1..8ea44f32bc 100644 --- a/tests/unit/test_opaque_main_flow_image_domain.py +++ b/tests/unit/test_opaque_main_flow_image_domain.py @@ -19,7 +19,7 @@ from openhcs.core.context.processing_context import ProcessingContext from openhcs.core.steps.function_output_manifest import step_output_manifest, StepOutputManifestStore from openhcs.core.source_bindings import CompiledSourceBindingPlan -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import StepOutputInput from openhcs.core.dataset_sources.source_schema import SourceSchemaFilenameParser from openhcs.core.source_image_provenance import SourceImageProvenancePlanes from openhcs.core.payload_axes import PayloadAxes @@ -207,7 +207,7 @@ def test_save_and_next_load_preserve_domain_and_independent_cache(tmp_path, monk context.runtime_image_stack_cache.clear() consumer_plan = replace( plan, step_index=1, step_scope_id="whole-consumer", input_dir=tmp_path, - main_input_dependency=StepInputDependency.step_output(source_step_index=0, source_step_scope_id=plan.step_scope_id), + main_input_dependency=StepOutputInput(source_step_index=0, source_step_scope_id=plan.step_scope_id), ) consumer = PatternGroupExecutionRequest( context=context, execution_plan=consumer_plan, compiled_group=pattern.default_group, @@ -366,7 +366,7 @@ def test_saved_mixed_named_cohort_rejects_joint_load_and_preserves_selected_doma def consumer_for(compiled): consumer_plan = replace(plan, step_index=1, step_scope_id="mixed-consumer", input_dir=tmp_path, compiled_function_pattern=compiled, - main_input_dependency=StepInputDependency.step_output(source_step_index=0, source_step_scope_id=plan.step_scope_id)) + main_input_dependency=StepOutputInput(source_step_index=0, source_step_scope_id=plan.step_scope_id)) consumer = PatternGroupExecutionRequest(context=context, execution_plan=consumer_plan, compiled_group=compiled.default_group, diff --git a/tests/unit/test_path_planner_materialization.py b/tests/unit/test_path_planner_materialization.py index 6c073d0415..9743ee7b25 100644 --- a/tests/unit/test_path_planner_materialization.py +++ b/tests/unit/test_path_planner_materialization.py @@ -108,8 +108,11 @@ NamedSourceBinding, SourceArtifactRole, ) -from openhcs.core.step_dependencies import StepInputDependency -from openhcs.core.step_dependencies import StepInputDependencyKind +from openhcs.core.step_dependencies import ( + NoMainFlowInput, + PipelineStartInput, + StepOutputInput, +) from openhcs.core.steps.abstract import AbstractStep from openhcs.core.steps.function_step import FunctionStep from openhcs.core.steps.function_runtime import ( @@ -378,7 +381,7 @@ def metadata_consumer(image, grid_dimensions): declarations, pattern, _, contracts = _prepare_step_declarations(planner, snapshot, 3 ) - execution_bindings = snapshot.source_bindings.compiled_plan_for_contracts(contracts, StepInputDependency.step_output( + execution_bindings = snapshot.source_bindings.compiled_plan_for_contracts(contracts, StepOutputInput( source_step_index=2, source_step_scope_id="plate::functionstep_2", ), planner.artifact_context.available_artifacts) @@ -629,7 +632,7 @@ def export_measurements(*, artifact_batch: RuntimeArtifactBatch): callable_scope, contracts, ) = _prepare_step_declarations(planner, snapshot, 3) - execution_bindings = snapshot.source_bindings.compiled_plan_for_contracts(contracts, StepInputDependency.no_main_flow(), planner.artifact_context.available_artifacts) + execution_bindings = snapshot.source_bindings.compiled_plan_for_contracts(contracts, NoMainFlowInput(), planner.artifact_context.available_artifacts) maps = planner.artifacts.compile_plan_maps( snapshot, 3, @@ -651,7 +654,7 @@ def export_measurements(*, artifact_batch: RuntimeArtifactBatch): ) plan = planner.plans[3] plan.step_name = snapshot.name - plan.main_input_dependency = StepInputDependency.no_main_flow() + plan.main_input_dependency = NoMainFlowInput() plan.artifact_inputs = maps.inputs plan.artifact_outputs = maps.outputs plan.compiled_function_pattern = compiled @@ -4179,7 +4182,7 @@ def test_main_flow_source_anchor_restricts_execution_to_its_exact_channel(): ), ) - contract_bindings = snapshot.source_bindings.compiled_plan_for_contracts((source_contract,), StepInputDependency.pipeline_start(), planner.artifact_context.available_artifacts) + contract_bindings = snapshot.source_bindings.compiled_plan_for_contracts((source_contract,), PipelineStartInput(), planner.artifact_context.available_artifacts) source_anchor_specs = tuple( binding.input_spec() for binding in contract_bindings.primary_plane_bindings ) @@ -4331,7 +4334,7 @@ def test_execution_anchor_ignores_source_artifact_lineage(): ), ) - contract_bindings = snapshot.source_bindings.compiled_plan_for_contracts((source_contract,), StepInputDependency.pipeline_start(), planner.artifact_context.available_artifacts) + contract_bindings = snapshot.source_bindings.compiled_plan_for_contracts((source_contract,), PipelineStartInput(), planner.artifact_context.available_artifacts) source_anchor_specs = tuple( binding.input_spec() for binding in contract_bindings.primary_plane_bindings ) @@ -4625,7 +4628,7 @@ def test_compiled_group_by_preserves_dynamic_execution_scope(): planner.steps.update_core_step_plan( snapshot, 3, - StepInputDependency.step_output( + StepOutputInput( source_step_index=2, source_step_scope_id="plate::functionstep_2", ), @@ -5742,7 +5745,7 @@ def test_main_input_dependency_uses_scope_identity_for_step_output_edges(): 1, ) - assert dependency.kind is StepInputDependencyKind.STEP_OUTPUT + assert isinstance(dependency, StepOutputInput) assert dependency.source_step_index == 0 assert dependency.source_step_scope_id == "plate::functionstep_0" @@ -5800,7 +5803,7 @@ def test_main_input_dependency_uses_declared_artifact_producer_not_previous_step declarations=declarations, ) - assert dependency == StepInputDependency.step_output( + assert dependency == StepOutputInput( source_step_index=0, source_step_scope_id="plate::functionstep_0", ) @@ -5841,7 +5844,7 @@ def measure(image, *, runtime): ), is_grouped=False, ) - source_dependency = StepInputDependency.step_output( + source_dependency = StepOutputInput( source_step_index=0, source_step_scope_id="plate::functionstep_0", ) @@ -5911,7 +5914,7 @@ def test_main_input_dependency_preserves_pipeline_start_edges(): 1, ) - assert dependency.kind is StepInputDependencyKind.PIPELINE_START + assert isinstance(dependency, PipelineStartInput) input_dir, output_dir = planner.steps.step_io_dirs(dependency, 1) assert input_dir == Path("/data/plate1/images") assert output_dir == Path("/data/plate1_processed/images") diff --git a/tests/unit/test_payload_axis_compile_gate.py b/tests/unit/test_payload_axis_compile_gate.py index ea2ba16dc5..96bdc7508e 100644 --- a/tests/unit/test_payload_axis_compile_gate.py +++ b/tests/unit/test_payload_axis_compile_gate.py @@ -36,7 +36,7 @@ ) from openhcs.core.source_projection import OpenHCSPlaneAddress, SourcePlaneProjection from openhcs.core.source_workspace_projection import VirtualWorkspaceSourceProjection -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import PipelineStartInput, StepOutputInput from openhcs.processing.backends.cellprofiler.color import color_to_gray from openhcs.processing.backends.cellprofiler.crop import crop from openhcs.domains.microscopy.axes import Microscopy @@ -93,8 +93,8 @@ def test_crop_declares_payload_axis_preservation() -> None: def test_step_input_dependency_owns_ancestry_and_pipeline_start_proofs() -> None: - pipeline_start = StepInputDependency.pipeline_start() - step_output = StepInputDependency.step_output( + pipeline_start = PipelineStartInput() + step_output = StepOutputInput( source_step_index=3, source_step_scope_id="step-3", ) @@ -138,7 +138,7 @@ def _session( step_index=0, step_name="Convert colour", axis_id="A01", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=CompiledSourceBindingPlan(bindings=(binding,)), compiled_function_pattern=_compiled_pattern(), ) @@ -273,7 +273,7 @@ def identity(image): step_index=0, step_name="Grouped conversion", axis_id="A01", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=CompiledSourceBindingPlan( bindings=(grayscale_binding, rgb_binding), ), @@ -368,7 +368,7 @@ def crop_like(image): step_name="Crop", axis_id="A01", step_scope_id="step-0", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=binding_plan, compiled_function_pattern=_compiled_pattern(crop_like), ) @@ -376,7 +376,7 @@ def crop_like(image): step_index=1, step_name="Convert colour", axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="step-0", ), @@ -409,7 +409,7 @@ def crop_like(image): step_name="First crop", axis_id="A01", step_scope_id="step-0", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=binding_plan, compiled_function_pattern=_compiled_pattern(crop_like), ) @@ -418,7 +418,7 @@ def crop_like(image): step_name="Second crop", axis_id="A01", step_scope_id="step-1", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="step-0", ), @@ -428,7 +428,7 @@ def crop_like(image): step_index=2, step_name="Convert colour", axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=1, source_step_scope_id="step-1", ), @@ -458,7 +458,7 @@ def unknown_transform(image): step_name="Unknown transform", axis_id="A01", step_scope_id="step-0", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=binding_plan, compiled_function_pattern=_compiled_pattern(unknown_transform), ) @@ -466,7 +466,7 @@ def unknown_transform(image): step_index=1, step_name="Convert colour", axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="step-0", ), @@ -509,9 +509,9 @@ def crop_like(image): axis_id="A01", step_scope_id=f"step-{index}", main_input_dependency=( - StepInputDependency.pipeline_start() + PipelineStartInput() if index == 0 - else StepInputDependency.step_output( + else StepOutputInput( source_step_index=index - 1, source_step_scope_id=f"step-{index - 1}", ) @@ -561,7 +561,7 @@ def generic_flow_preserver(image): step_name="Generic flow producer", axis_id="A01", step_scope_id="step-0", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=binding_plan, compiled_function_pattern=_compiled_pattern(generic_flow_preserver), ) @@ -569,7 +569,7 @@ def generic_flow_preserver(image): step_index=1, step_name="Convert colour", axis_id="A01", - main_input_dependency=StepInputDependency.step_output( + main_input_dependency=StepOutputInput( source_step_index=0, source_step_scope_id="step-0", ), @@ -636,7 +636,7 @@ def test_unknown_routed_group_cannot_fall_back_to_all_primary_sources( step_index=0, step_name="Unknown routed conversion", axis_id="A01", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=CompiledSourceBindingPlan( bindings=(grayscale_binding, rgb_binding), ), diff --git a/tests/unit/test_source_bindings.py b/tests/unit/test_source_bindings.py index 96b7a40512..751225edce 100644 --- a/tests/unit/test_source_bindings.py +++ b/tests/unit/test_source_bindings.py @@ -71,7 +71,6 @@ SourceImageProvenancePlanes, ) from openhcs.core.source_matching import ( - SourceImageSetComponentRole, SourceImageSetIdentityPolicy, ) from openhcs.core.source_metadata import ( @@ -130,14 +129,8 @@ def test_source_image_set_plane_members_are_exactly_the_step_stack_axes(): frozenset((Microscopy.Site.name,)) ) - assert ( - site_stack.role(Microscopy.Site) - is SourceImageSetComponentRole.IMAGE_PLANE_MEMBER - ) - assert ( - site_stack.role(Microscopy.Channel) - is SourceImageSetComponentRole.IMAGE_SET_AXIS - ) + assert not site_stack.is_identity_component(Microscopy.Site) + assert site_stack.is_identity_component(Microscopy.Channel) def test_source_image_set_policy_uses_binding_and_source_stack_declarations(): diff --git a/tests/unit/test_source_bound_matched_image_set_anchors.py b/tests/unit/test_source_bound_matched_image_set_anchors.py index 33f4358746..cff0262884 100644 --- a/tests/unit/test_source_bound_matched_image_set_anchors.py +++ b/tests/unit/test_source_bound_matched_image_set_anchors.py @@ -41,7 +41,7 @@ from openhcs.core.source_workspace_projection import ( VirtualWorkspaceSourceProjection, WorkspaceSourceProjections, ) -from openhcs.core.step_dependencies import StepInputDependency +from openhcs.core.step_dependencies import PipelineStartInput from openhcs.core.steps.function_execution import ( FunctionStepExecutor, ) @@ -251,7 +251,7 @@ def measure_images(image): axis_id="A01", step_index=0, step_name="MeasureImages", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=source_binding_plan, execution_group_scope=ComponentGroupScope.from_raw( tuple(binding.component_identity[0].value for binding in compiled_bindings), @@ -420,7 +420,7 @@ def measure_hoechst(image): axis_id="A01", step_index=0, step_name="GroupedMeasurements", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=source_binding_plan, execution_group_scope=ComponentGroupScope.from_raw( ("2", "1"), @@ -480,7 +480,7 @@ def identify_bodies(image): axis_id="A01", step_index=0, step_name="NeuronBodies", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=CompiledSourceBindingPlan(bindings=(binding,)), execution_group_scope=ComponentGroupScope.from_raw( ("2",), @@ -586,7 +586,7 @@ def align_source_set(image): axis_id="A01", step_index=0, step_name="Align", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=CompiledSourceBindingPlan( bindings=bindings, match_plan=PairByOrder(), @@ -708,7 +708,7 @@ def align_source_set(image): axis_id="A01", step_index=0, step_name="Align", - main_input_dependency=StepInputDependency.pipeline_start(), + main_input_dependency=PipelineStartInput(), source_binding_plan=CompiledSourceBindingPlan( bindings=bindings, match_plan=PairByOrder(), From 8e666c1da90b82d649e661693b0da21eb1462b42 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:00:08 -0400 Subject: [PATCH 21/36] E1: action invocation, workflow poll and widget-tree output formats are families; descriptor-set size picks its resolver directly; error dialogs are a window field DescriptorSetCardinality and UiWindowSemanticMarker are deleted: the descriptor set returns the resolver for its size, and a window summary carries error_dialog instead of a one-member marker string. Co-Authored-By: Claude Opus 5.5 --- openhcs/agent/dto/__init__.py | 1 - openhcs/agent/dto/ui_bridge.py | 33 ++++++--- openhcs/agent/services/ui_bridge_service.py | 58 +++++---------- openhcs/mcp/dev_client_commands/ui.py | 33 +++++---- openhcs/mcp/dev_client_core.py | 70 ++++++++++++------- openhcs/mcp/dev_client_renderers/ui_bridge.py | 4 +- openhcs/mcp/dev_client_rendering.py | 41 ++++++++--- openhcs/pyqt_gui/services/ui_agent_bridge.py | 4 +- .../pyqt_gui/services/ui_bridge_contracts.py | 27 ++++--- .../pyqt_gui/services/ui_bridge_windows.py | 17 ++--- tests/unit/agent/test_mcp_server.py | 13 ++-- tests/unit/agent/test_ui_bridge_service.py | 10 +-- tests/unit/pyqt_gui/test_ui_agent_bridge.py | 36 +++++----- 13 files changed, 188 insertions(+), 159 deletions(-) diff --git a/openhcs/agent/dto/__init__.py b/openhcs/agent/dto/__init__.py index 144a967fce..aa5d4bcdcb 100644 --- a/openhcs/agent/dto/__init__.py +++ b/openhcs/agent/dto/__init__.py @@ -277,7 +277,6 @@ "UiWindowFocusResult", "UiWindowIdentity", "UiWindowManagerScope", - "UiWindowSemanticMarker", "UiWindowNavigateRequest", "UiWindowNavigateResult", "UiWindowOpenPolicy", diff --git a/openhcs/agent/dto/ui_bridge.py b/openhcs/agent/dto/ui_bridge.py index 8005301dc1..82c3cf0478 100644 --- a/openhcs/agent/dto/ui_bridge.py +++ b/openhcs/agent/dto/ui_bridge.py @@ -1029,10 +1029,27 @@ class UiStateSurfaceDocument( payload: JsonObject -class UiActionInvocationStatus(str, Enum): - ACCEPTED = "accepted" - REJECTED = "rejected" - UNAVAILABLE = "unavailable" +class UiActionInvocationStatus(ABC, metaclass=AutoRegisterMeta): + """Whether the UI took a requested action; ``name`` is its wire spelling.""" + + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + accepted: ClassVar[bool] = False + + +class ActionAccepted(UiActionInvocationStatus): + name = "accepted" + accepted = True + + +class ActionRejected(UiActionInvocationStatus): + name = "rejected" + + +class ActionUnavailable(UiActionInvocationStatus): + name = "unavailable" class UiWidgetActionInvocationOutcome(str, Enum): @@ -1289,7 +1306,7 @@ def as_tool_arguments(self) -> JsonObject: class UiActionInvokeResult: schema_version: str identity: UiActionIdentity - status: str + status: type[UiActionInvocationStatus] receipt: UiMutationReceipt target_scope_ids: tuple[str, ...] = () selection_revision_token: str | None = None @@ -1319,10 +1336,6 @@ def from_identity(cls, identity: UiWindowIdentity) -> "UiWindowManagerScope": return cls(value=identity.window_id) -class UiWindowSemanticMarker(str, Enum): - ERROR_DIALOG = "error_dialog" - - @dataclass(frozen=True, slots=True, kw_only=True) class UiWindowSemanticCarrier: """Shared semantic state carried by agent-visible window summaries.""" @@ -1353,6 +1366,8 @@ class UiWindowSummary(UiWindowSemanticCarrier): visible: bool focusable: bool manager_scope: UiWindowManagerScope | None = None + error_dialog: bool = False + """The window is a critical message box reporting an error.""" @property def window_id(self) -> str: diff --git a/openhcs/agent/services/ui_bridge_service.py b/openhcs/agent/services/ui_bridge_service.py index 2b5f3b2de4..fdb07e7376 100644 --- a/openhcs/agent/services/ui_bridge_service.py +++ b/openhcs/agent/services/ui_bridge_service.py @@ -9,7 +9,6 @@ from abc import ABC, abstractmethod from collections.abc import Callable from dataclasses import dataclass, replace -from enum import Enum from os import environ from pathlib import Path from typing import Annotated, ClassVar @@ -32,13 +31,14 @@ from openhcs.agent.dto.common import SCHEMA_VERSION, AgentError from openhcs.agent.dto.session import SessionEventBatch, SessionEventsRequest from openhcs.agent.dto.ui_bridge import ( + ActionAccepted, + ActionUnavailable, OperationRunning, OperationUnavailable, UI_BRIDGE_UNKNOWN_WIDGET, UNKNOWN_UI_BRIDGE_OPERATION_ROUTE, UiActionCatalog, UiActionIdentity, - UiActionInvocationStatus, UiActionInvokeRequest, UiActionInvokeResult, UiBranchCatalog, @@ -630,7 +630,7 @@ def failed(cls, request, errors): widget_id=request.widget_id, action_id=request.action_id, ), - status=UiActionInvocationStatus.UNAVAILABLE.value, + status=ActionUnavailable, receipt=UiMutationReceipt.rejected_for(request.request_token), errors=errors, ) @@ -643,7 +643,7 @@ def accepted(cls, request, bridge_operation): widget_id=request.widget_id, action_id=request.action_id, ), - status=UiActionInvocationStatus.ACCEPTED.value, + status=ActionAccepted, receipt=UiMutationReceipt.accepted_for( request.request_token, bridge_operation_id=bridge_operation.identity.operation_id, @@ -654,7 +654,7 @@ def accepted(cls, request, bridge_operation): @classmethod def outcome(cls, result) -> str: - return result.status + return result.status.name class UiBridgeSelectedPlateWorkflowOperation(UiBridgeOperation): @@ -674,7 +674,7 @@ def failed(cls, request, errors): widget_id=UI_BRIDGE_UNKNOWN_WIDGET, action_id=request.workflow.value, ), - status=UiActionInvocationStatus.UNAVAILABLE.value, + status=ActionUnavailable, receipt=UiMutationReceipt.rejected_for(request.request_token), errors=errors, ), @@ -1302,27 +1302,21 @@ def __str__(self) -> str: ) -class DescriptorSetCardinality(Enum): - NONE = "none" - ONE = "one" - MANY = "many" - - @dataclass(frozen=True, slots=True) class LiveUiBridgeDescriptorSet: descriptors: tuple[UiBridgeDescriptorFile, ...] - @property - def cardinality(self) -> DescriptorSetCardinality: + def resolution_runner(self) -> "DescriptorSetResolutionRunner": + """Return the resolver for this set's size: none, one or many.""" count = len(self.descriptors) if count == 0: - return DescriptorSetCardinality.NONE + return NoDescriptorSetResolutionRunner() if count == 1: - return DescriptorSetCardinality.ONE - return DescriptorSetCardinality.MANY + return SingleDescriptorSetResolutionRunner() + return AmbiguousDescriptorSetResolutionRunner() def only_descriptor(self) -> UiBridgeDescriptorFile: - if self.cardinality is not DescriptorSetCardinality.ONE: + if len(self.descriptors) != 1: raise ValueError( "Live UI bridge descriptor set does not contain exactly one descriptor." ) @@ -1363,20 +1357,8 @@ def apply(self, connection: UiBridgeConnectionSpec) -> UiBridgeConnectionSpec: ) -class DescriptorSetResolutionRunner(ABC, metaclass=AutoRegisterMeta): - """Registered resolver behavior for live UI bridge descriptor cardinality.""" - - __registry_key__ = "cardinality" - __skip_if_no_key__ = True - - cardinality: ClassVar[DescriptorSetCardinality | None] = None - - @classmethod - def for_cardinality( - cls, - cardinality: DescriptorSetCardinality, - ) -> "DescriptorSetResolutionRunner": - return cls.__registry__[cardinality]() +class DescriptorSetResolutionRunner(ABC): + """Resolver behavior for a live UI bridge descriptor set of one size.""" @abstractmethod def resolve( @@ -1389,8 +1371,6 @@ def resolve( class NoDescriptorSetResolutionRunner(DescriptorSetResolutionRunner): - cardinality = DescriptorSetCardinality.NONE - def resolve( self, resolver: "UiBridgeDescriptorResolver", @@ -1405,8 +1385,6 @@ def resolve( class SingleDescriptorSetResolutionRunner(DescriptorSetResolutionRunner): - cardinality = DescriptorSetCardinality.ONE - def resolve( self, resolver: "UiBridgeDescriptorResolver", @@ -1421,8 +1399,6 @@ def resolve( class AmbiguousDescriptorSetResolutionRunner(DescriptorSetResolutionRunner): - cardinality = DescriptorSetCardinality.MANY - def resolve( self, resolver: "UiBridgeDescriptorResolver", @@ -1692,9 +1668,9 @@ def resolve( descriptor_set = LiveUiBridgeDescriptorSet( UiBridgeDescriptorCatalog.live_descriptors() ) - return DescriptorSetResolutionRunner.for_cardinality( - descriptor_set.cardinality - ).resolve(self, descriptor_set, connection) + return descriptor_set.resolution_runner().resolve( + self, descriptor_set, connection + ) def _resolve_explicit_file( self, diff --git a/openhcs/mcp/dev_client_commands/ui.py b/openhcs/mcp/dev_client_commands/ui.py index 1350208daa..c837c937a9 100644 --- a/openhcs/mcp/dev_client_commands/ui.py +++ b/openhcs/mcp/dev_client_commands/ui.py @@ -56,6 +56,8 @@ McpDevToolBatchResponse, McpDevToolCall, McpDevToolResult, + PollFailed, + PollTimedOut, WorkflowPollBaseline, WorkflowPollSkipReason, WorkflowPollSummary, @@ -70,8 +72,8 @@ parse_cli_json_value, parse_json_object, selected_workflow_tool_arguments, - state_surface_tool_arguments, state_surface_document, + state_surface_tool_arguments, ui_request_tool_arguments, ui_tool_arguments, workflow_operation_receipt_skip_reason, @@ -86,7 +88,9 @@ workflow_result_was_accepted, ) from openhcs.mcp.dev_client_rendering import ( + JsonWidgetTree, McpDevOutputRenderer, + OutlineWidgetTree, WidgetTreeOutputFormat, ) @@ -357,7 +361,7 @@ async def _poll_workflow_state( baseline: WorkflowPollBaseline | None, results: list[McpDevToolResult], transient_poll_error_count: int, - ) -> tuple[WorkflowPollSummaryStatus, bool, int, int]: + ) -> tuple[type[WorkflowPollSummaryStatus], bool, int, int]: """Re-read the workflow's state each time the session publishes events. Starts after the bridge receipt completes; between reads it waits on @@ -392,7 +396,7 @@ async def await_events(after_sequence: int) -> int: return after_sequence if batch is None else batch.last_sequence poll_count = 0 - terminal_status: WorkflowPollSummaryStatus | None = None + terminal_status: type[WorkflowPollSummaryStatus] | None = None while True: poll_result = await call_mcp_tool( session, @@ -411,7 +415,7 @@ async def await_events(after_sequence: int) -> int: events_after = await await_events(events_after) continue results.append(poll_result) - terminal_status = WorkflowPollSummaryStatus.FAILED + terminal_status = PollFailed break results.append(poll_result) if policy.can_evaluate(poll_result, baseline): @@ -426,10 +430,10 @@ async def await_events(after_sequence: int) -> int: break events_after = await await_events(events_after) - status = terminal_status or WorkflowPollSummaryStatus.TIMEOUT + status = terminal_status or PollTimedOut return ( status, - status is WorkflowPollSummaryStatus.COMPLETED, + status.completed, poll_count, transient_poll_error_count, ) @@ -455,7 +459,7 @@ def render_compact(self, response, args: argparse.Namespace) -> str: ( "Action: " f"{McpDevOutputRenderer.text(summary.action_status)} " - f"poll={summary.status.value} count={summary.poll_count}" + f"poll={summary.status.name} count={summary.poll_count}" ), ] if summary.target_scope_ids: @@ -699,12 +703,12 @@ def configure_json_option(self, parser: argparse.ArgumentParser) -> None: "--json", dest="output", action="store_const", - const=WidgetTreeOutputFormat.JSON.value, + const=JsonWidgetTree.name, help="Alias for --output json.", ) def requests_json_output(self, args: argparse.Namespace) -> bool: - return WidgetTreeOutputFormat(args.output).is_json + return WidgetTreeOutputFormat.named(args.output).is_json @staticmethod def _effective_max_depth(args: argparse.Namespace) -> int | None: @@ -716,7 +720,7 @@ def _effective_max_depth(args: argparse.Namespace) -> int | None: def _effective_max_nodes(args: argparse.Namespace) -> int | None: if args.max_nodes is not None: return args.max_nodes - return 800 if args.output == WidgetTreeOutputFormat.OUTLINE.value else 40 + return WidgetTreeOutputFormat.named(args.output).default_max_nodes def configure_parser(self, parser: argparse.ArgumentParser) -> None: parser.add_argument("window_id") @@ -727,8 +731,8 @@ def configure_parser(self, parser: argparse.ArgumentParser) -> None: parser.add_argument("--create-if-missing", action="store_true") parser.add_argument( "--output", - choices=WidgetTreeOutputFormat.choices(), - default=WidgetTreeOutputFormat.OUTLINE.value, + choices=tuple(WidgetTreeOutputFormat.__registry__), + default=OutlineWidgetTree.name, help="Render JSON or a clean indented widget outline.", ) parser.add_argument( @@ -767,12 +771,13 @@ def calls_from_args( if args.actionable_only else ( False - if args.output == WidgetTreeOutputFormat.OUTLINE.value + if WidgetTreeOutputFormat.named(args.output).shows_whole_tree else not args.include_non_actionable ) ), include_tree=( - args.include_tree or args.output == WidgetTreeOutputFormat.OUTLINE.value + args.include_tree + or WidgetTreeOutputFormat.named(args.output).shows_whole_tree ), max_depth=self._effective_max_depth(args), max_nodes=self._effective_max_nodes(args), diff --git a/openhcs/mcp/dev_client_core.py b/openhcs/mcp/dev_client_core.py index 5aa04f4a86..ab0134862f 100644 --- a/openhcs/mcp/dev_client_core.py +++ b/openhcs/mcp/dev_client_core.py @@ -47,7 +47,6 @@ InheritedFields, RawResolvedFields, SemanticFields, - UiActionInvocationStatus, UiBridgeOperationRef, UiBridgeOperationStatus, UiBridgeOperationWaitRequest, @@ -181,45 +180,63 @@ class McpDevProtocolError(RuntimeError): """Malformed or unsupported MCP wire response.""" -class WorkflowPollSummaryStatus(str, Enum): - """Machine-readable selected-workflow polling outcome.""" +class WorkflowPollSummaryStatus(ABC, metaclass=AutoRegisterMeta): + """Machine-readable selected-workflow polling outcome; ``name`` is its spelling.""" - COMPLETED = "completed" - FAILED = "failed" - TIMEOUT = "timeout" - SKIPPED = "skipped" + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + completed: ClassVar[bool] = False + + +class PollCompleted(WorkflowPollSummaryStatus): + name = "completed" + completed = True + + +class PollFailed(WorkflowPollSummaryStatus): + name = "failed" + + +class PollTimedOut(WorkflowPollSummaryStatus): + name = "timeout" + + +class PollSkipped(WorkflowPollSummaryStatus): + name = "skipped" class WorkflowPollSkipReason(str, Enum): """Machine-readable reason selected-workflow polling did not start.""" - poll_status: WorkflowPollSummaryStatus + poll_status: type[WorkflowPollSummaryStatus] WORKFLOW_NOT_ACCEPTED = ( "workflow_not_accepted", - WorkflowPollSummaryStatus.SKIPPED, + PollSkipped, ) WORKFLOW_TOOL_ERROR = ( "workflow_tool_error", - WorkflowPollSummaryStatus.SKIPPED, + PollSkipped, ) OPERATION_RECEIPT_MISSING = ( "operation_receipt_missing", - WorkflowPollSummaryStatus.FAILED, + PollFailed, ) OPERATION_RECEIPT_FAILED = ( "operation_receipt_failed", - WorkflowPollSummaryStatus.FAILED, + PollFailed, ) OPERATION_RECEIPT_TIMEOUT = ( "operation_receipt_timeout", - WorkflowPollSummaryStatus.TIMEOUT, + PollTimedOut, ) def __new__( cls, value: str, - poll_status: WorkflowPollSummaryStatus, + poll_status: type[WorkflowPollSummaryStatus], ) -> "WorkflowPollSkipReason": member = str.__new__(cls, value) member._value_ = value @@ -638,7 +655,7 @@ class WorkflowPollSummary: tool_name: ClassVar[str] = "mcp_dev_selected_workflow_poll" workflow: str | None - status: WorkflowPollSummaryStatus + status: type[WorkflowPollSummaryStatus] poll_requested: bool poll_completed: bool poll_count: int @@ -649,11 +666,11 @@ class WorkflowPollSummary: @property def mcp_error(self) -> bool: - return self.status is not WorkflowPollSummaryStatus.COMPLETED + return not self.status.completed def as_payload(self) -> dict[str, JsonValue]: payload: dict[str, JsonValue] = { - "poll_status": self.status.value, + "poll_status": self.status.name, "poll_requested": self.poll_requested, "poll_completed": self.poll_completed, "poll_count": self.poll_count, @@ -2312,14 +2329,13 @@ def state_surface_tool_arguments( def workflow_result_was_accepted(result: McpDevToolResult) -> bool: - return ( - workflow_result_action_status(result) == UiActionInvocationStatus.ACCEPTED.value - ) + payload = workflow_result_payload(result) + return payload is not None and payload.action_result.status.accepted def workflow_result_action_status(result: McpDevToolResult) -> str | None: payload = workflow_result_payload(result) - return None if payload is None else payload.action_result.status + return None if payload is None else payload.action_result.status.name def workflow_result_event_sequence(result: McpDevToolResult) -> int: @@ -2414,7 +2430,7 @@ def workflow_poll_terminal_status( *, target_scope_ids: tuple[str, ...], policy: WorkflowStatePollPolicy, -) -> WorkflowPollSummaryStatus | None: +) -> type[WorkflowPollSummaryStatus] | None: if not workflow_poll_manager_is_idle(result): return None rows = workflow_poll_target_rows( @@ -2424,9 +2440,9 @@ def workflow_poll_terminal_status( if not rows: return None if any(policy.failed_for_row(row) for row in rows): - return WorkflowPollSummaryStatus.FAILED + return PollFailed if all(policy.terminal_for_row(row) for row in rows): - return WorkflowPollSummaryStatus.COMPLETED + return PollCompleted return None @@ -2480,7 +2496,7 @@ def parse_orchestrator_state( def workflow_poll_summary_result( *, workflow: str | None = None, - status: WorkflowPollSummaryStatus | None = None, + status: type[WorkflowPollSummaryStatus] | None = None, poll_requested: bool, poll_completed: bool, poll_count: int, @@ -2491,9 +2507,9 @@ def workflow_poll_summary_result( ) -> McpDevToolResult: if status is None: status = ( - WorkflowPollSummaryStatus.COMPLETED + PollCompleted if poll_completed - else WorkflowPollSummaryStatus.TIMEOUT + else PollTimedOut ) summary = WorkflowPollSummary( workflow=workflow, diff --git a/openhcs/mcp/dev_client_renderers/ui_bridge.py b/openhcs/mcp/dev_client_renderers/ui_bridge.py index 782e3b6994..df90ed377a 100644 --- a/openhcs/mcp/dev_client_renderers/ui_bridge.py +++ b/openhcs/mcp/dev_client_renderers/ui_bridge.py @@ -540,6 +540,8 @@ def _summary_lines(cls, summary: UiWindowSummary) -> list[str]: ] if summary.semantic_markers: status_parts.append("markers=" + ",".join(summary.semantic_markers)) + if summary.error_dialog: + status_parts.append("error_dialog") return [f"Window: {summary.title}", "Status: " + " ".join(status_parts)] @classmethod @@ -870,7 +872,7 @@ def render_payload(cls, payload, options: McpDevOutputRenderOptions) -> str: *cls.introduction_lines(payload), f"UI action invoke: " f"action={action.identity.widget_id}/{action.identity.action_id} " - f"status={action.status}", + f"status={action.status.name}", cls.acknowledgement_line(action.receipt), f"Selection: targets={cls.sequence_text(action.target_scope_ids)} " f"selection_rev={cls.text(action.selection_revision_token)}", diff --git a/openhcs/mcp/dev_client_rendering.py b/openhcs/mcp/dev_client_rendering.py index 7fbea544df..c2934f9492 100644 --- a/openhcs/mcp/dev_client_rendering.py +++ b/openhcs/mcp/dev_client_rendering.py @@ -9,6 +9,7 @@ from __future__ import annotations import argparse +from abc import ABC import json from collections.abc import Callable, Iterable, Mapping, Sequence from dataclasses import dataclass, fields, is_dataclass @@ -30,19 +31,39 @@ PresentationValue = TypeVar("PresentationValue") -class WidgetTreeOutputFormat(str, Enum): - """CLI presentation formats for widget-tree command output.""" +class WidgetTreeOutputFormat(ABC, metaclass=AutoRegisterMeta): + """CLI presentation format for widget-tree command output. - JSON = "json" - OUTLINE = "outline" + Each format is a subclass; ``name`` is its ``--output`` spelling. + """ - @property - def is_json(self) -> bool: - return self is WidgetTreeOutputFormat.JSON + __registry_key__ = "name" + __skip_if_no_key__ = True - @classmethod - def choices(cls) -> tuple[str, ...]: - return tuple(output_format.value for output_format in cls) + name: ClassVar[str | None] = None + is_json: ClassVar[bool] + """The command prints the JSON payload instead of rendering it.""" + default_max_nodes: ClassVar[int] + shows_whole_tree: ClassVar[bool] + """The request includes the nested tree and non-actionable branches.""" + + @staticmethod + def named(name: str) -> type["WidgetTreeOutputFormat"]: + return WidgetTreeOutputFormat.__registry__[name] + + +class JsonWidgetTree(WidgetTreeOutputFormat): + name = "json" + is_json = True + default_max_nodes = 40 + shows_whole_tree = False + + +class OutlineWidgetTree(WidgetTreeOutputFormat): + name = "outline" + is_json = False + default_max_nodes = 800 + shows_whole_tree = True @dataclass(frozen=True, slots=True) diff --git a/openhcs/pyqt_gui/services/ui_agent_bridge.py b/openhcs/pyqt_gui/services/ui_agent_bridge.py index b6e88e0652..ea75b3565c 100644 --- a/openhcs/pyqt_gui/services/ui_agent_bridge.py +++ b/openhcs/pyqt_gui/services/ui_agent_bridge.py @@ -35,6 +35,7 @@ RegistryFunctionReference, ) from openhcs.agent.dto.ui_bridge import ( + ActionRejected, OperationCompleted, OperationFailed, OperationNotFound, @@ -43,7 +44,6 @@ UNKNOWN_UI_BRIDGE_OPERATION_ROUTE, UiActionCatalog, UiActionIdentity, - UiActionInvocationStatus, UiActionInvokeRequest, UiActionInvokeResult, UiBranchCatalog, @@ -1878,7 +1878,7 @@ def _action_error( widget_id=request.widget_id, action_id=request.action_id, ), - status=UiActionInvocationStatus.REJECTED.value, + status=ActionRejected, receipt=UiMutationReceipt.rejected_for(request.request_token), target_scope_ids=request.selected_scope_ids, selection_revision_token=request.observed_selection_revision_token, diff --git a/openhcs/pyqt_gui/services/ui_bridge_contracts.py b/openhcs/pyqt_gui/services/ui_bridge_contracts.py index b79f9eba63..24c36b3447 100644 --- a/openhcs/pyqt_gui/services/ui_bridge_contracts.py +++ b/openhcs/pyqt_gui/services/ui_bridge_contracts.py @@ -9,6 +9,13 @@ from openhcs.agent.dto.common import SCHEMA_VERSION, AgentError, AgentWarning from openhcs.agent.dto.ui_bridge import ( + ActionAccepted, + ActionRejected, + UiActionCatalog, + UiActionIdentity, + UiActionInvokeRequest, + UiActionInvokeResult, + UiActionSummary, UiCodeDocument, UiCodeDocumentApplyRequest, UiCodeDocumentApplyResult, @@ -21,20 +28,16 @@ UiMutationReceipt, UiObjectStateScopeCatalog, UiObjectStateScopeListRequest, - UiActionCatalog, - UiActionInvocationStatus, - UiActionInvokeRequest, - UiActionInvokeResult, - UiActionSummary, - UiActionIdentity, + UiSnapshotRef, UiStateSurfaceDocument, UiStateSurfaceIdentity, UiStateSurfaceRequest, UiStateSurfaceSummary, - UiSnapshotRef, - UiWidgetIdentity, UiWidgetActionInvokeRequest, UiWidgetActionInvokeResult, + UiWidgetIdentity, + UiWidgetTreeRequest, + UiWidgetTreeResult, UiWindowCatalog, UiWindowCloseRequest, UiWindowCloseResult, @@ -44,8 +47,6 @@ UiWindowNavigateResult, UiWindowSnapshotRequest, UiWindowSnapshotResult, - UiWidgetTreeRequest, - UiWidgetTreeResult, ) from openhcs.agent.ui_bridge_identities import ( UiCodeDocumentIdentityDeclaration, @@ -281,11 +282,7 @@ def _result( widget_id=request.widget_id, action_id=request.action_id, ), - status=( - UiActionInvocationStatus.ACCEPTED - if accepted - else UiActionInvocationStatus.REJECTED - ).value, + status=ActionAccepted if accepted else ActionRejected, receipt=( UiMutationReceipt.accepted_for(request.request_token) if accepted diff --git a/openhcs/pyqt_gui/services/ui_bridge_windows.py b/openhcs/pyqt_gui/services/ui_bridge_windows.py index 350a34cc5d..0d6dedc5e5 100644 --- a/openhcs/pyqt_gui/services/ui_bridge_windows.py +++ b/openhcs/pyqt_gui/services/ui_bridge_windows.py @@ -99,7 +99,6 @@ UiWindowNavigateResult, UiWindowOperationRequest, UiWindowSemanticCarrier, - UiWindowSemanticMarker, UiWindowSnapshotRequest, UiWindowSnapshotResult, UiWindowSummary, @@ -371,7 +370,7 @@ def overview_sections(self) -> tuple[UiLiveOverviewSection, ...]: error_dialogs = tuple( window for window in windows - if UiWindowSemanticMarker.ERROR_DIALOG.value in window.semantic_markers + if window.error_dialog ) return ( UiLiveOverviewSection( @@ -401,9 +400,7 @@ def overview_sections(self) -> tuple[UiLiveOverviewSection, ...]: @staticmethod def _overview_window_item(window: UiWindowSummary) -> UiLiveOverviewItem: - is_error_dialog = ( - UiWindowSemanticMarker.ERROR_DIALOG.value in window.semantic_markers - ) + is_error_dialog = window.error_dialog return UiLiveOverviewItem( label=window.title, status=window.window_kind, @@ -1188,7 +1185,7 @@ def summary(self, widget: QWidget) -> UiWindowSummary: visible=widget.isVisible(), focusable=True, manager_scope=None, - semantic_markers=self._semantic_markers(widget), + error_dialog=self._is_error_dialog(widget), ) def _projected_top_level_widgets(self) -> tuple[QWidget, ...]: @@ -1230,13 +1227,11 @@ def _title(widget: QWidget) -> str: return type(widget).__name__ @staticmethod - def _semantic_markers(widget: QWidget) -> tuple[str, ...]: - if ( + def _is_error_dialog(widget: QWidget) -> bool: + return ( isinstance(widget, QMessageBox) and widget.icon() is QMessageBox.Icon.Critical - ): - return (UiWindowSemanticMarker.ERROR_DIALOG.value,) - return () + ) class UiWindowSnapshotResultFactory: diff --git a/tests/unit/agent/test_mcp_server.py b/tests/unit/agent/test_mcp_server.py index 224c7ed009..0481e1e9b1 100644 --- a/tests/unit/agent/test_mcp_server.py +++ b/tests/unit/agent/test_mcp_server.py @@ -1,4 +1,5 @@ from openhcs.agent.dto.session import DatasetRowState +from openhcs.mcp.dev_client_core import PollCompleted, PollFailed, PollTimedOut import asyncio import importlib.util import inspect @@ -6076,7 +6077,7 @@ def test_mcp_dev_client_workflow_poll_reports_failed_terminal_rows(): target_scope_ids=("scope-a",), policy=run_policy, ) - is dev_client.WorkflowPollSummaryStatus.FAILED + is PollFailed ) assert ( dev_client.workflow_poll_terminal_status( @@ -6084,7 +6085,7 @@ def test_mcp_dev_client_workflow_poll_reports_failed_terminal_rows(): target_scope_ids=("scope-a",), policy=compile_policy, ) - is dev_client.WorkflowPollSummaryStatus.FAILED + is PollFailed ) @@ -6355,7 +6356,7 @@ async def fake_call_tool(session, call, timeout_seconds): "openhcs_ui_selected_plate_workflow", "openhcs_ui_wait_for_operation_receipt", ] - assert response.results[-1].payloads[0].status is dev_client.WorkflowPollSummaryStatus.FAILED + assert response.results[-1].payloads[0].status is PollFailed from openhcs.mcp.dev_client_core import WorkflowPollSkipReason assert response.results[-1].payloads[0].skip_reason is WorkflowPollSkipReason.OPERATION_RECEIPT_FAILED @@ -6433,7 +6434,7 @@ async def fake_call_tool(session, call, timeout_seconds): ] summary = response.results[-1] assert summary.mcp_error is True - assert summary.payloads[0].status is dev_client.WorkflowPollSummaryStatus.TIMEOUT + assert summary.payloads[0].status is PollTimedOut assert summary.payloads[0].poll_count == 1 @@ -6514,7 +6515,7 @@ async def fake_call_tool(session, call, timeout_seconds): ] summary = response.results[-1] assert summary.mcp_error is False - assert summary.payloads[0].status is dev_client.WorkflowPollSummaryStatus.COMPLETED + assert summary.payloads[0].status is PollCompleted assert summary.payloads[0].poll_count == 1 @@ -7115,7 +7116,7 @@ def test_mcp_dev_client_workflow_poll_timeout_summary_is_error(): summary = dev_client.workflow_poll_summary_result( workflow="compile_datasets", - status=dev_client.WorkflowPollSummaryStatus.TIMEOUT, + status=PollTimedOut, poll_requested=True, poll_completed=False, poll_count=2, diff --git a/tests/unit/agent/test_ui_bridge_service.py b/tests/unit/agent/test_ui_bridge_service.py index 271629e893..3bb2b10280 100644 --- a/tests/unit/agent/test_ui_bridge_service.py +++ b/tests/unit/agent/test_ui_bridge_service.py @@ -15,15 +15,15 @@ from openhcs.agent.path_policy import AgentPathPolicy from openhcs.agent.dto.common import SCHEMA_VERSION, AgentError, AgentResourceRef from openhcs.agent.dto.ui_bridge import ( - SemanticFields, + ActionAccepted, OperationCompleted, OperationFailed, OperationNotFound, OperationRunning, OperationUnavailable, + SemanticFields, UiActionCatalog, UiActionIdentity, - UiActionInvocationStatus, UiActionInvokeRequest, UiActionInvokeResult, UiBranchCatalog, @@ -336,7 +336,7 @@ def invoke_action( widget_id=request.widget_id, action_id=request.action_id, ), - status=UiActionInvocationStatus.ACCEPTED.value, + status=ActionAccepted, receipt=UiMutationReceipt( request_token=request.request_token, accepted=True, @@ -360,7 +360,7 @@ def selected_plate_workflow( widget_id=WIDGET_ID, action_id=request.workflow.value, ), - status=UiActionInvocationStatus.ACCEPTED.value, + status=ActionAccepted, receipt=UiMutationReceipt( request_token=request.request_token, accepted=True, @@ -1505,7 +1505,7 @@ def test_service_forwards_fake_gateway_requests(monkeypatch, tmp_path): assert close_result.closed is True assert gateway.close_requests == [UiWindowCloseRequest(window_id=WINDOW_ID)] assert ( - workflow_result.action_result.status == UiActionInvocationStatus.ACCEPTED.value + workflow_result.action_result.status is ActionAccepted ) assert gateway.selected_plate_workflow_requests == [ UiSelectedPlateWorkflowRequest( diff --git a/tests/unit/pyqt_gui/test_ui_agent_bridge.py b/tests/unit/pyqt_gui/test_ui_agent_bridge.py index 5ef06d9ed3..cf2b26bc7b 100644 --- a/tests/unit/pyqt_gui/test_ui_agent_bridge.py +++ b/tests/unit/pyqt_gui/test_ui_agent_bridge.py @@ -58,18 +58,20 @@ from openhcs.agent.dto.common import SCHEMA_VERSION from openhcs.agent.dto.ui_bridge import ( - UiBridgeOperationStatusRequest, + ActionAccepted, + ActionRejected, + DirtyFields, UiActionInvokeRequest, UiBranchSwitchRequest, UiBridgeConfirmationRequirement, UiBridgeConnectionSpec, + UiBridgeOperationStatusRequest, UiBridgeRequestEnvelope, UiCodeDocumentApplyRequest, UiCodeDocumentId, UiCodeDocumentRequest, UiCodeDocumentSelectionMode, UiCodeDocumentValidationRequest, - DirtyFields, UiObjectStateScopeListRequest, UiSelectedPlateWorkflowKind, UiSelectedPlateWorkflowRequest, @@ -2696,7 +2698,7 @@ def test_managed_window_save_action_returns_before_deferred_save_runs() -> None: ) assert action.invocation_mode == "async" - assert result.status == "accepted" + assert result.status is ActionAccepted assert result.receipt.bridge_operation_id is not None assert result.target_scope_ids == (window.scope_id,) assert window.save_count == 0 @@ -3801,7 +3803,7 @@ def test_main_window_actions_are_application_operations_presented_by_the_window( ), ) ) - assert result.status == "accepted", result.errors + assert result.status is ActionAccepted, result.errors assert result.warnings == operation.warnings counts.append( ( @@ -3836,7 +3838,7 @@ def test_main_window_action_is_unavailable_while_its_window_action_is_disabled() assert action.enabled is False assert action.disabled_error.code == "update_check_in_progress" - assert result.status == "rejected" + assert result.status is ActionRejected assert main_window.update_check_count == 0 @@ -4141,12 +4143,12 @@ def invoke(token): assert action.selection_revision_token assert action.target_scope_ids == (DATASET_SCOPE_ID,) - assert accepted.status == "accepted" + assert accepted.status is ActionAccepted assert accepted.selection_revision_token == action.selection_revision_token assert [operation for operation, _request in renderer.presented] == [ ShowDatasetCode ] - assert stale.status == "rejected" + assert stale.status is ActionRejected assert stale.errors[0].code == "stale_ui_action_revision" @@ -4182,7 +4184,7 @@ def test_selected_run_workflow_dispatches_stop_while_execution_is_active( assert action.enabled is True assert action.title == ManagerExecutionState.RUNNING.run_button_text assert action.disabled_error is None - assert result.action_result.status == "accepted" + assert result.action_result.status is ActionAccepted assert forces == [False] assert session.execution_state is ManagerExecutionState.FORCE_KILL_READY @@ -4208,7 +4210,7 @@ def test_selected_workflow_returns_before_the_queued_operation_runs( ) operation_id = result.action_result.receipt.bridge_operation_id - assert result.action_result.status == "accepted" + assert result.action_result.status is ActionAccepted assert operation_id is not None assert result.action_result.workflow_status_surface_ids == ( "plate_manager.state", @@ -4288,7 +4290,7 @@ def test_selected_compile_reports_the_init_precondition(monkeypatch) -> None: assert error.code == "dataset_not_initialized" assert InitializeDatasets.operation_id in error.hint assert not action.enabled - assert result.action_result.status == "rejected" + assert result.action_result.status is ActionRejected def test_selected_workflow_confirmation_rejection_avoids_ui_preflight( @@ -4311,7 +4313,7 @@ def test_selected_workflow_confirmation_rejection_avoids_ui_preflight( ) ) - assert result.action_result.status == "rejected" + assert result.action_result.status is ActionRejected assert result.errors[0].code == "confirmation_required" assert result.action_result.workflow_status_surface_ids == ( "plate_manager.state", @@ -4397,10 +4399,10 @@ def invoke(action, *, confirm, targets=None): edit_result = invoke(edit_action, confirm=True) stale_result = invoke(edit_action, confirm=False, targets=("wrong-target",)) - assert code_result.status == "accepted" - assert edit_result.status == "rejected" + assert code_result.status is ActionAccepted + assert edit_result.status is ActionRejected assert edit_result.errors[0].code == "confirmation_required" - assert stale_result.status == "rejected" + assert stale_result.status is ActionRejected assert stale_result.errors[0].code == "stale_ui_action_selection" assert [operation for operation, _request in renderer.presented] == [ ShowPipelineCode @@ -4537,9 +4539,9 @@ def invoke(confirm): rejected = invoke(True) accepted = invoke(False) - assert rejected.status == "rejected" + assert rejected.status is ActionRejected assert rejected.errors[0].code == "confirmation_required" - assert accepted.status == "accepted" + assert accepted.status is ActionAccepted ((work, args, kwargs),) = started assert work == gui.session.run_debug assert args == (DATASET_SCOPE_ID,) @@ -4578,7 +4580,7 @@ def test_pipeline_debug_toolbar_runtime_values_action_requires_debug_session( assert action.enabled is True assert action.confirmation_required is False - assert result.status == "accepted" + assert result.status is ActionAccepted assert len(inspections) == 1 From 917c97de1cf3051b1f586926fccbed318c44777a Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:00:17 -0400 Subject: [PATCH 22/36] E1: orchestrator state, terminal status, axis outcome, observation mode and export scope are families OrchestratorState, TerminalExecutionStatus, ExecutionStatus, RuntimeObservationMode and ZMQRuntimeObservationExportScope become class families whose members carry what consumers switched on (retain_records, returns_every_record, completion_payload, export, succeeded). Wire spellings are the registry names; the agent run request keeps a boundary Enum view derived from the export-scope registry. ExecutionResult.is_error is deleted (derived from status). Co-Authored-By: Claude Opus 5.5 --- benchmark/cellprofiler_benchmark_cli.py | 7 +- benchmark/contracts/measured_run_receipt.py | 16 +- benchmark/control.py | 2 +- benchmark/matched_cellprofiler_batch.py | 4 +- benchmark/openhcs_measured_run.py | 4 +- benchmark/well_throughput_scaling.py | 4 +- openhcs/agent/dto/session.py | 17 +- openhcs/authoring/session/compile_batch.py | 10 +- openhcs/authoring/session/dataset_status.py | 4 +- openhcs/authoring/session/events.py | 6 +- openhcs/authoring/session/execution_batch.py | 14 +- .../authoring/session/execution_control.py | 10 +- openhcs/authoring/session/result_summaries.py | 4 +- openhcs/authoring/session/session.py | 38 ++-- openhcs/authoring/session/submission.py | 10 +- openhcs/authoring/session/views.py | 4 +- openhcs/constants/constants.py | 155 +++++++++------- openhcs/core/debug_session_projection.py | 12 +- openhcs/core/execution_state.py | 166 +++++++++--------- openhcs/core/orchestrator/bundle_execution.py | 27 +-- openhcs/core/orchestrator/execution_result.py | 119 ++++++++----- openhcs/core/orchestrator/orchestrator.py | 28 +-- openhcs/core/orchestrator/worker_execution.py | 11 +- openhcs/core/orchestrator/worker_lanes.py | 2 +- openhcs/core/pipeline/compiler.py | 7 +- openhcs/mcp/dev_client_core.py | 26 +-- openhcs/mcp/dev_client_session.py | 5 +- openhcs/mcp/installed_demo.py | 4 +- .../pyqt_gui/services/reactor_providers.py | 4 +- .../services/ui_bridge_plate_manager.py | 4 +- openhcs/pyqt_gui/widgets/plate_manager.py | 6 +- openhcs/runtime/zmq_execution_observation.py | 8 +- openhcs/runtime/zmq_execution_server.py | 43 +---- openhcs/runtime/zmq_execution_signature.py | 117 +++++++++--- openhcs/runtime/zmq_worker_execution.py | 2 +- .../test_agent_observation_export_zmq.py | 12 +- .../test_chained_callable_lineage_journey.py | 4 +- .../test_image_math_publication_620.py | 4 +- .../test_measurement_declaration_journey.py | 4 +- ...rimary_segmentation_diagnostics_journey.py | 4 +- ...est_source_schema_aggregate_publication.py | 4 +- tests/unit/pyqt_gui/test_debug_toolbar.py | 11 +- ...pipeline_code_document_transactionality.py | 4 +- .../pyqt_gui/test_pipeline_editor_widget.py | 6 +- .../test_plate_manager_cppipe_import.py | 3 +- .../pyqt_gui/test_plate_manager_widget.py | 54 +++--- .../pyqt_gui/test_plate_status_presenter.py | 14 +- tests/unit/pyqt_gui/test_reactor_providers.py | 4 +- .../pyqt_gui/test_session_batch_workflows.py | 26 +-- tests/unit/pyqt_gui/test_ui_agent_bridge.py | 6 +- tests/unit/test_compiled_execution.py | 30 ++-- ..._completed_output_publication_lifecycle.py | 10 +- tests/unit/test_deleted_core_names_absent.py | 2 +- .../test_function_artifact_materialization.py | 8 +- .../test_function_step_execution_scope.py | 10 +- .../test_materialization_batch_outcomes.py | 2 +- ...est_nonpersistent_viewer_state_evidence.py | 6 +- tests/unit/test_openhcs_measured_run.py | 6 +- tests/unit/test_orchestrator_lane_planning.py | 31 ++-- tests/unit/test_plane_store_sources.py | 4 +- tests/unit/test_well_throughput_scaling.py | 4 +- tests/unit/test_zmq_compilation_identity.py | 2 +- tests/unit/test_zmq_execution_server.py | 32 ++-- tests/unit/test_zmq_runtime_outcome_export.py | 4 +- 64 files changed, 680 insertions(+), 531 deletions(-) diff --git a/benchmark/cellprofiler_benchmark_cli.py b/benchmark/cellprofiler_benchmark_cli.py index 7a149cf0d0..30fc3e6eb3 100644 --- a/benchmark/cellprofiler_benchmark_cli.py +++ b/benchmark/cellprofiler_benchmark_cli.py @@ -173,6 +173,7 @@ def configure( ) -> argparse.ArgumentParser: parser = self._parser(subparsers) from openhcs.runtime.zmq_execution_signature import ( + ExportRuntimeValues, ZMQRuntimeObservationExportScope, ) @@ -197,8 +198,8 @@ def configure( parser.add_argument("--pipeline-name") parser.add_argument( "--observation-scope", - choices=tuple(scope.value for scope in ZMQRuntimeObservationExportScope), - default=ZMQRuntimeObservationExportScope.VALUES.value, + choices=tuple(ZMQRuntimeObservationExportScope.__registry__), + default=ExportRuntimeValues.name, help="Retain full runtime values or outcome-only execution evidence.", ) parser.add_argument("--host", default="localhost") @@ -280,7 +281,7 @@ def run(self, args: argparse.Namespace) -> int: runtime_observation_export_path=str( MeasuredPipelineRunArtifact.RUNTIME_OBSERVATION.path_in(output_dir) ), - runtime_observation_export_scope=ZMQRuntimeObservationExportScope( + runtime_observation_export_scope=ZMQRuntimeObservationExportScope.named( args.observation_scope ), submit_timeout_ms=args.submit_timeout_ms, diff --git a/benchmark/contracts/measured_run_receipt.py b/benchmark/contracts/measured_run_receipt.py index 9d5624e62a..3ca43ce763 100644 --- a/benchmark/contracts/measured_run_receipt.py +++ b/benchmark/contracts/measured_run_receipt.py @@ -14,7 +14,10 @@ from benchmark.contracts.run_artifacts import write_new_measured_artifact from benchmark.timing import PhaseTimingRecord from openhcs.runtime.environment_provenance import RuntimeEnvironmentSnapshot -from openhcs.runtime.zmq_execution_signature import ZMQRuntimeObservationExportScope +from openhcs.runtime.zmq_execution_signature import ( + ExportRuntimeValues, + ZMQRuntimeObservationExportScope, +) MEASURED_PIPELINE_RUN_RECEIPT_SCHEMA_VERSION = "openhcs.benchmark.measured-pipeline.v2" _ARCHIVED_MEASURED_PIPELINE_RUN_RECEIPT_SCHEMA_VERSION = ( @@ -65,8 +68,8 @@ class MeasuredPipelineRunReceipt: completed_at_epoch_seconds: float compile_artifact_id: str | None = None server_environment: RuntimeEnvironmentSnapshot | None = None - observation_export_scope: ZMQRuntimeObservationExportScope = ( - ZMQRuntimeObservationExportScope.VALUES + observation_export_scope: type[ZMQRuntimeObservationExportScope] = ( + ExportRuntimeValues ) expected_axis_count: int | None = None observed_axis_count: int | None = None @@ -138,11 +141,8 @@ def read(cls, path: Path) -> Self: cls, { **payload, - "observation_export_scope": ZMQRuntimeObservationExportScope( - payload.get( - "observation_export_scope", - ZMQRuntimeObservationExportScope.VALUES.value, - ) + "observation_export_scope": ZMQRuntimeObservationExportScope.named( + payload.get("observation_export_scope", ExportRuntimeValues.name) ), "phase_timings": tuple( PhaseTimingRecord.from_payload(record) for record in timing_payloads diff --git a/benchmark/control.py b/benchmark/control.py index a5cd05c2cd..3e8fb1d6ae 100644 --- a/benchmark/control.py +++ b/benchmark/control.py @@ -273,7 +273,7 @@ def report_measured_pipeline_run( f"- Compile artifact: `{receipt.compile_artifact_id or 'none; server job may include compilation'}`", f"- Output roots: {len(receipt.output_roots)}", "- Output files: not inspected; use output/equivalence evidence for output claims", - f"- Runtime observation scope: `{receipt.observation_export_scope.value}`", + f"- Runtime observation scope: `{receipt.observation_export_scope.name}`", f"- Execution axes: {receipt.observed_axis_count if receipt.observed_axis_count is not None else 'not recorded'}" + ( f" / {receipt.expected_axis_count} expected" diff --git a/benchmark/matched_cellprofiler_batch.py b/benchmark/matched_cellprofiler_batch.py index afa64c2235..a5189ae72e 100644 --- a/benchmark/matched_cellprofiler_batch.py +++ b/benchmark/matched_cellprofiler_batch.py @@ -121,7 +121,7 @@ ) from openhcs.runtime.zmq_execution_signature import ( ZMQAuxiliaryExecutionParams, - ZMQRuntimeObservationExportScope, + ExportRuntimeOutcomes, ) from python_introspect import to_jsonable from openhcs.domains.microscopy.axes import Microscopy @@ -1659,7 +1659,7 @@ def capture_progress_event(event: Mapping[str, Any]) -> None: axis_events.clear() progress_events.clear() print(f"OpenHCS batch {repetition} starting.", flush=True) - export_scope = ZMQRuntimeObservationExportScope.OUTCOMES + export_scope = ExportRuntimeOutcomes evidence_dir = root / "candidate_evidence" / str(repetition) output_dir = root / "candidate" / str(repetition) global_config = _global_config( diff --git a/benchmark/openhcs_measured_run.py b/benchmark/openhcs_measured_run.py index 2a40080799..9523226820 100644 --- a/benchmark/openhcs_measured_run.py +++ b/benchmark/openhcs_measured_run.py @@ -48,7 +48,7 @@ ) from openhcs.runtime.zmq_execution_signature import ( ZMQAuxiliaryExecutionParams, - ZMQRuntimeObservationExportScope, + ExportRuntimeOutcomes, ) from .timing import ( @@ -381,7 +381,7 @@ def retain_measured_openhcs_completion( ) if ( auxiliary_params.runtime_observation_export_scope - is ZMQRuntimeObservationExportScope.OUTCOMES + is ExportRuntimeOutcomes ): observation_export = ZMQRuntimeExecutionOutcomeExport.read( observation_export_path diff --git a/benchmark/well_throughput_scaling.py b/benchmark/well_throughput_scaling.py index 7a370ef0b6..4f133d1146 100644 --- a/benchmark/well_throughput_scaling.py +++ b/benchmark/well_throughput_scaling.py @@ -76,7 +76,7 @@ from openhcs.runtime.zmq_execution_observation import ZMQRuntimeExecutionOutcomeExport from openhcs.runtime.zmq_execution_signature import ( ZMQAuxiliaryExecutionParams, - ZMQRuntimeObservationExportScope, + ExportRuntimeOutcomes, ) from python_introspect import to_jsonable from openhcs.domains.microscopy.axes import Microscopy @@ -2146,7 +2146,7 @@ def run_case_well_throughput( ).with_auxiliary_params( ZMQAuxiliaryExecutionParams( runtime_observation_export_path=observation_path, - runtime_observation_export_scope=ZMQRuntimeObservationExportScope.OUTCOMES, + runtime_observation_export_scope=ExportRuntimeOutcomes, ) ) progress_events: list[dict[str, Any]] = [] diff --git a/openhcs/agent/dto/session.py b/openhcs/agent/dto/session.py index 9de3fa92a5..6a42bbb456 100644 --- a/openhcs/agent/dto/session.py +++ b/openhcs/agent/dto/session.py @@ -10,11 +10,24 @@ AgentError, AgentWarning, ) +from enum import Enum + from python_introspect import validate_annotated_dataclass from zmqruntime.config import PositiveInteger from openhcs.runtime.zmq_execution_signature import ZMQRuntimeObservationExportScope +RuntimeObservationExportScopeName = Enum( + "RuntimeObservationExportScopeName", + [(name.upper(), name) for name in ZMQRuntimeObservationExportScope.__registry__], + type=str, + module=__name__, +) +"""Boundary view of the export-scope registry, for agent request fields.""" +RuntimeObservationExportScopeName.scope = property( + lambda member: ZMQRuntimeObservationExportScope.named(member.value) +) + @dataclass(frozen=True, slots=True) class NoArgumentsRequest(AgentDataclassCliRequest): @@ -49,8 +62,8 @@ class DatasetRunRequest(AgentDataclassCliRequest): scope_ids: tuple[str, ...] runtime_observation_export_path: str | None = None - runtime_observation_export_scope: ZMQRuntimeObservationExportScope = ( - ZMQRuntimeObservationExportScope.VALUES + runtime_observation_export_scope: RuntimeObservationExportScopeName = ( + RuntimeObservationExportScopeName.VALUES ) submit_timeout_ms: PositiveInteger | None = None """Bound on each execution submission; the transport default when unset.""" diff --git a/openhcs/authoring/session/compile_batch.py b/openhcs/authoring/session/compile_batch.py index b0594a3d0d..003deaf30a 100644 --- a/openhcs/authoring/session/compile_batch.py +++ b/openhcs/authoring/session/compile_batch.py @@ -21,8 +21,8 @@ StatusReported, ) from openhcs.authoring.session.run_requests import dataset_pipeline_request -from openhcs.core.execution_state import TerminalExecutionStatus -from openhcs.constants.constants import OrchestratorState +from openhcs.core.execution_state import ExecutionFailed +from openhcs.constants.constants import OrchestratorCompileFailed, OrchestratorCompiled from openhcs.runtime.zmq_execution_signature import TransportValue if TYPE_CHECKING: @@ -74,7 +74,7 @@ def on_wait_start(_request, _index: int, total: int) -> None: ) def on_wait_success(request, _execution_id, _index, _total) -> None: - session.set_dataset_state(request.scope_id, OrchestratorState.COMPILED) + session.set_dataset_state(request.scope_id, OrchestratorCompiled) logger.info("Successfully compiled %s", request.scope_id) def on_failure(request, error, _index, _total) -> None: @@ -139,7 +139,7 @@ def on_failure(request, error, _index, _total) -> None: error, exc_info=True, ) - session.batch.mark_terminal(request.scope_id, TerminalExecutionStatus.FAILED) + session.batch.mark_terminal(request.scope_id, ExecutionFailed) session.publish(ErrorReported(f"Compile failed for {request.scope_id}: {error}")) session.publish(DatasetsChanged()) @@ -196,7 +196,7 @@ def _compile_failed(self, scope_id: str, error: Exception) -> None: logger.error("COMPILATION ERROR: %s: %s", scope_id, error, exc_info=True) session = self._session session.clear_compile_pending((scope_id,)) - session.set_dataset_state(scope_id, OrchestratorState.COMPILE_FAILED) + session.set_dataset_state(scope_id, OrchestratorCompileFailed) session.publish( CompilationFailed(scope_id, session.dataset_name(scope_id), str(error)) ) diff --git a/openhcs/authoring/session/dataset_status.py b/openhcs/authoring/session/dataset_status.py index 79734ebecb..aa7bf87088 100644 --- a/openhcs/authoring/session/dataset_status.py +++ b/openhcs/authoring/session/dataset_status.py @@ -30,11 +30,11 @@ def build_debug_status_prefix( def build_status_prefix( cls, *, - orchestrator_state: Optional[OrchestratorState], + orchestrator_state: Optional[type[OrchestratorState]], is_init_pending: bool, is_compile_pending: bool, is_execution_active: bool, - terminal_status: Optional[TerminalExecutionStatus], + terminal_status: Optional[type[TerminalExecutionStatus]], runtime_projection: Optional[SubjectRuntimeProjection], ) -> str: # Runtime projection is canonical whenever present. diff --git a/openhcs/authoring/session/events.py b/openhcs/authoring/session/events.py index 1687356af7..b9e1b576c2 100644 --- a/openhcs/authoring/session/events.py +++ b/openhcs/authoring/session/events.py @@ -96,11 +96,11 @@ def scope_id(self) -> str: @dataclass(frozen=True, slots=True) class DatasetStateChanged(DatasetScopedEvent): - state: OrchestratorState + state: type[OrchestratorState] @property def message(self) -> str: - return f"{self.dataset_scope_id}: {self.state.value}" + return f"{self.dataset_scope_id}: {self.state.name}" @dataclass(frozen=True, slots=True) @@ -155,7 +155,7 @@ class DatasetExecutionFinished(DatasetScopedEvent): @property def message(self) -> str: - return f"{self.dataset_scope_id}: {self.completion.status.value}" + return f"{self.dataset_scope_id}: {self.completion.status.name}" @dataclass(frozen=True, slots=True) diff --git a/openhcs/authoring/session/execution_batch.py b/openhcs/authoring/session/execution_batch.py index 193f7970db..f338e31246 100644 --- a/openhcs/authoring/session/execution_batch.py +++ b/openhcs/authoring/session/execution_batch.py @@ -6,7 +6,7 @@ from dataclasses import dataclass, replace from openhcs.core.execution_state import ( - TerminalExecutionStatus, + TerminalExecutionStatus, ExecutionCompleted, parse_terminal_status, ) @@ -16,11 +16,11 @@ class ExecutionBatchMember: """All runtime facts owned for one plate in the current batch.""" execution_id: str | None = None - terminal_status: TerminalExecutionStatus | None = None + terminal_status: type[TerminalExecutionStatus] | None = None superseded: bool = False @property - def current_terminal_status(self) -> TerminalExecutionStatus | None: + def current_terminal_status(self) -> type[TerminalExecutionStatus] | None: """Current row activity, distinct from this member's batch outcome.""" return None if self.superseded else self.terminal_status @@ -33,7 +33,7 @@ def with_execution(self, execution_id: str) -> ExecutionBatchMember: def with_terminal_status( self, - status: str | TerminalExecutionStatus, + status: str | type[TerminalExecutionStatus], ) -> ExecutionBatchMember: return replace(self, terminal_status=parse_terminal_status(status)) @@ -66,7 +66,7 @@ def execution_id(self, plate_path: str) -> str | None: def mark_terminal( self, plate_path: str, - status: str | TerminalExecutionStatus, + status: str | type[TerminalExecutionStatus], ) -> None: member = self._require_batch_member(plate_path) self._members_by_plate[plate_path] = member.with_terminal_status(status) @@ -85,7 +85,7 @@ def active_plates(self) -> tuple[str, ...]: if member.active ) - def terminal_status(self, plate_path: str) -> TerminalExecutionStatus | None: + def terminal_status(self, plate_path: str) -> type[TerminalExecutionStatus] | None: member = self._members_by_plate.get(plate_path) return None if member is None else member.current_terminal_status @@ -134,7 +134,7 @@ def all_batch_terminal(self) -> bool: def terminal_counts(self) -> tuple[int, int]: statuses = tuple(status for _plate_path, status in self.terminal_items()) completed = sum( - 1 for status in statuses if status is TerminalExecutionStatus.COMPLETE + 1 for status in statuses if status is ExecutionCompleted ) failed = sum(1 for status in statuses if status.counts_as_failed) return completed, failed diff --git a/openhcs/authoring/session/execution_control.py b/openhcs/authoring/session/execution_control.py index ad0d71e994..59b8c2ab0a 100644 --- a/openhcs/authoring/session/execution_control.py +++ b/openhcs/authoring/session/execution_control.py @@ -10,10 +10,10 @@ from zmqruntime.shutdown import EndpointShutdownService from openhcs.authoring.session.events import ErrorReported -from openhcs.constants.constants import OrchestratorState +from openhcs.constants.constants import OrchestratorExecFailed from openhcs.core.execution_state import ( ManagerExecutionState, - TerminalExecutionStatus, + ExecutionCancelled, ExecutionFailed, ) if TYPE_CHECKING: @@ -47,8 +47,8 @@ def check_all_completed(self) -> None: async def handle_execution_failure(self) -> None: session = self._session for scope_id in tuple(session.batch.active_plates): - session.batch.mark_terminal(scope_id, TerminalExecutionStatus.FAILED) - session.set_dataset_state(scope_id, OrchestratorState.EXEC_FAILED) + session.batch.mark_terminal(scope_id, ExecutionFailed) + session.set_dataset_state(scope_id, OrchestratorExecFailed) session.execution_state = ManagerExecutionState.IDLE try: await session.client.disconnect() @@ -92,7 +92,7 @@ def cancel_active_datasets(self) -> None: session = self._session for scope_id in session.batch.cancellable_plates(): session.finish_dataset_execution( - TerminalExecutionStatus.CANCELLED.completion_payload( + ExecutionCancelled.completion_payload( execution_id=session.batch.execution_id(scope_id), execution_payload={}, ), diff --git a/openhcs/authoring/session/result_summaries.py b/openhcs/authoring/session/result_summaries.py index eebaf28ca3..8cd881aa5a 100644 --- a/openhcs/authoring/session/result_summaries.py +++ b/openhcs/authoring/session/result_summaries.py @@ -6,7 +6,7 @@ from pathlib import Path from typing import TYPE_CHECKING -from openhcs.core.execution_state import TerminalExecutionStatus +from openhcs.core.execution_state import ExecutionCompleted from openhcs.processing.backends.analysis.consolidate_analysis_results import ( consolidate_multi_plate_summaries, ) @@ -26,7 +26,7 @@ def consolidate_batch_results(session: "Session") -> None: summary_paths: list[str] = [] names: list[str] = [] for scope_id, status in session.batch.terminal_items(): - if status is not TerminalExecutionStatus.COMPLETE: + if status is not ExecutionCompleted: continue root = Path(scope_id) base = ( diff --git a/openhcs/authoring/session/session.py b/openhcs/authoring/session/session.py index 46feaa7e76..f695ede04e 100644 --- a/openhcs/authoring/session/session.py +++ b/openhcs/authoring/session/session.py @@ -124,10 +124,14 @@ from openhcs.core.xdg_paths import get_openhcs_data_dir from openhcs.core.pipeline_document import PipelineDocument, PipelineDocumentCodec from openhcs.core.pipeline_import import PipelineImporter -from openhcs.core.orchestrator.orchestrator import ( +from openhcs.constants.constants import ( + OrchestratorCreated, + OrchestratorExecuting, + OrchestratorInitFailed, + OrchestratorReady, OrchestratorState, - PipelineOrchestrator, ) +from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.progress import registry from openhcs.core.progress.debug_projection import ( DebugRuntimeProjection, @@ -608,7 +612,7 @@ def _create_orchestrator( parent_state=ObjectStateRegistry.get_by_scope(""), ) ObjectStateRegistry.register(state) - self.publish(DatasetStateChanged(scope.scope_id, OrchestratorState.CREATED)) + self.publish(DatasetStateChanged(scope.scope_id, OrchestratorCreated)) return state def _split_dataset_scopes(self) -> None: @@ -827,9 +831,9 @@ def invalidate_compilation(self, scope_id: str) -> None: if ( orchestrator is not None and orchestrator.state.has_completed_initialization - and orchestrator.state is not OrchestratorState.READY + and orchestrator.state is not OrchestratorReady ): - self.set_dataset_state(scope_id, OrchestratorState.READY) + self.set_dataset_state(scope_id, OrchestratorReady) def import_workspace_pipeline( self, @@ -923,7 +927,7 @@ def execution_state(self, state: ManagerExecutionState) -> None: self.publish(ExecutionStateChanged(state)) self.publish(AvailabilityChanged()) - def set_dataset_state(self, scope_id: str, state: OrchestratorState) -> None: + def set_dataset_state(self, scope_id: str, state: type[OrchestratorState]) -> None: orchestrator = ObjectStateRegistry.get_object(scope_id) if orchestrator is not None: orchestrator._state = state @@ -1048,7 +1052,7 @@ def run() -> InputWorkspacePreparationResult | None: "Failed to initialize %s: %s", scope_id, error, exc_info=True ) self.init_pending.discard(scope_id) - self.set_dataset_state(scope_id, OrchestratorState.INIT_FAILED) + self.set_dataset_state(scope_id, OrchestratorInitFailed) self.publish( InitializationFailed(scope_id, scope.display_name, str(error)) ) @@ -1057,7 +1061,7 @@ def run() -> InputWorkspacePreparationResult | None: self.main_thread.call( lambda: self.import_workspace_pipeline(scope_id, workspace) ) - self.set_dataset_state(scope_id, OrchestratorState.READY) + self.set_dataset_state(scope_id, OrchestratorReady) if scope_id == self.current_scope_id or not self.current_scope_id: self.main_thread.call( lambda: self._reselect(scope_id) @@ -1077,8 +1081,8 @@ def run() -> InputWorkspacePreparationResult | None: for scope_id in scope_ids if ObjectStateRegistry.get_object(scope_id) is not None ] - succeeded = sum(state is OrchestratorState.READY for state in states) - failed = sum(state is OrchestratorState.INIT_FAILED for state in states) + succeeded = sum(state is OrchestratorReady for state in states) + failed = sum(state is OrchestratorInitFailed for state in states) self.publish( StatusReported( f"Successfully initialized {succeeded} dataset(s)" @@ -1166,7 +1170,7 @@ async def run_datasets( await self.connect_client() for scope_id in scope_ids: self.debug_terminal_summaries.pop(scope_id, None) - self.set_dataset_state(scope_id, OrchestratorState.EXECUTING) + self.set_dataset_state(scope_id, OrchestratorExecuting) self.publish( StatusReported( f"Compiling {len(scope_ids)} dataset(s) before execution..." @@ -1200,13 +1204,13 @@ async def run_datasets( def observation_export_params( self, export_path: str | None, - export_scope: ZMQRuntimeObservationExportScope | str, + export_scope: type[ZMQRuntimeObservationExportScope], ) -> ZMQAuxiliaryExecutionParams | None: """Runtime observation export options for a run, under the access policy.""" - scope = ZMQRuntimeObservationExportScope(export_scope) + scope = export_scope if export_path is None: - if scope is not ZMQRuntimeObservationExportScope.VALUES: + if not scope.retains_runtime_values: raise ValueError("Outcome observation requires an export path.") return None path = self.dataset_access.require_writable(Path(export_path)) @@ -1253,7 +1257,7 @@ def _finish_dataset_execution( scope_id: str, ) -> None: status = completion.status - logger.info("Dataset %s finished with status %s", scope_id, status.value) + logger.info("Dataset %s finished with status %s", scope_id, status.name) self.batch.mark_terminal(scope_id, status) try: self.set_dataset_state(scope_id, status.orchestrator_state) @@ -1463,7 +1467,7 @@ def _close_debug_session(self, scope_id: str) -> None: else: summary = DebugTerminalSummary.from_session( active, - terminal_status=terminal_status.value, + terminal_status=terminal_status.name, completed_at_unix=time.time(), ) snapshots = self.debug_snapshots.get(scope_id, ()) @@ -1490,7 +1494,7 @@ def debug_context(self, scope_id: str) -> DebugSessionProjectionContext: terminal_status=( None if (status := self.batch.terminal_status(scope_id)) is None - else status.value + else status.name ), ), session=self.displayed_debug_session(scope_id), diff --git a/openhcs/authoring/session/submission.py b/openhcs/authoring/session/submission.py index da4e2f9b39..ea17c88939 100644 --- a/openhcs/authoring/session/submission.py +++ b/openhcs/authoring/session/submission.py @@ -22,11 +22,11 @@ ) from openhcs.authoring.session.events import ErrorReported, StatusReported from openhcs.core.execution_state import ( - TerminalExecutionStatus, + ExecutionFailed, parse_terminal_status, ) from openhcs.core.debug import DebugExecutionConfig -from openhcs.constants.constants import OrchestratorState +from openhcs.constants.constants import OrchestratorExecFailed from openhcs.runtime.zmq_execution_client import ZMQExecutionRequestBuilder from openhcs.runtime.zmq_execution_signature import ZMQAuxiliaryExecutionParams @@ -133,8 +133,8 @@ async def _submit( error_text = response.require_failure_text(f"{label} submission") logger.error("%s submission failed for %s: %s", label, scope_id, error_text) session.publish(ErrorReported(f"Submission failed for {scope_id}: {error_text}")) - session.batch.mark_terminal(scope_id, TerminalExecutionStatus.FAILED) - session.set_dataset_state(scope_id, OrchestratorState.EXEC_FAILED) + session.batch.mark_terminal(scope_id, ExecutionFailed) + session.set_dataset_state(scope_id, OrchestratorExecFailed) return None def follow(self, execution_id: str, scope_id: str) -> None: @@ -177,7 +177,7 @@ def on_terminal(terminal_id: str, status: str, payload: dict) -> None: def on_status_error(error_id: str, message: str) -> None: if current(error_id): session.finish_dataset_execution( - TerminalExecutionStatus.FAILED.completion_payload( + ExecutionFailed.completion_payload( execution_id=error_id, execution_payload={MessageFields.ERROR: message}, ), diff --git a/openhcs/authoring/session/views.py b/openhcs/authoring/session/views.py index 4f52d57c13..787d2be46f 100644 --- a/openhcs/authoring/session/views.py +++ b/openhcs/authoring/session/views.py @@ -294,10 +294,10 @@ def row_state( execution_active=activity.execution_active, status_prefix=activity.status_prefix, orchestrator_state=( - None if orchestrator_state is None else orchestrator_state.value + None if orchestrator_state is None else orchestrator_state.name ), execution_id=activity.execution_id, - terminal_status=None if terminal is None else terminal.value, + terminal_status=None if terminal is None else terminal.name, runtime_state=None if runtime is None else runtime.state.state_name, runtime_percent=None if runtime is None else runtime.percent, queue_position=None if runtime is None else runtime.queue_position, diff --git a/openhcs/constants/constants.py b/openhcs/constants/constants.py index c2c9b0731f..46de57722e 100644 --- a/openhcs/constants/constants.py +++ b/openhcs/constants/constants.py @@ -6,7 +6,10 @@ """ from enum import Enum -from typing import Set +from abc import ABC +from typing import ClassVar, Set + +from metaclass_registry import AutoRegisterMeta from arraybridge.types import ( CPU_MEMORY_TYPES as CPU_MEMORY_TYPES, @@ -47,68 +50,94 @@ from polystore.constants import Backend -class OrchestratorState(Enum): - """Simple orchestrator state tracking - no complex state machine.""" - - has_completed_initialization: bool - skips_initialization: bool - allows_execution: bool - status_prefix: str - - CREATED = ("created", False, False, False, "") # Object exists, not initialized - READY = ("ready", True, True, False, "✓ Init") # Initialized, ready for compilation - COMPILED = ("compiled", True, True, True, "✓ Compiled") # Compilation complete - EXECUTING = ( - "executing", - True, - False, - False, - "🔄 Executing", - ) # Execution in progress - COMPLETED = ( - "completed", - True, - True, - True, - "✅ Complete", - ) # Execution completed successfully - INIT_FAILED = ( - "init_failed", - False, - False, - False, - "❌ Init Failed", - ) # Initialization failed - COMPILE_FAILED = ( - "compile_failed", - True, - False, - False, - "❌ Compile Failed", - ) # Compilation failed - EXEC_FAILED = ( - "exec_failed", - True, - False, - False, - "❌ Exec Failed", - ) # Execution failed - - def __new__( - cls, - value: str, - has_completed_initialization: bool, - skips_initialization: bool, - allows_execution: bool, - status_prefix: str, - ): - obj = object.__new__(cls) - obj._value_ = value - obj.has_completed_initialization = has_completed_initialization - obj.skips_initialization = skips_initialization - obj.allows_execution = allows_execution - obj.status_prefix = status_prefix - return obj +class OrchestratorState(ABC, metaclass=AutoRegisterMeta): + """One lifecycle state of a dataset's orchestrator. + + Each state is a subclass declaring what it allows; ``name`` is its spelling + on the wire and in status text. + """ + + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + has_completed_initialization: ClassVar[bool] + skips_initialization: ClassVar[bool] + allows_execution: ClassVar[bool] + status_prefix: ClassVar[str] + + @staticmethod + def named(name: str) -> type["OrchestratorState"] | None: + """The state spelled ``name``, or ``None`` for an unknown spelling.""" + return OrchestratorState.__registry__.get(name) + + +class OrchestratorCreated(OrchestratorState): + """The orchestrator exists and is not initialized.""" + + name = "created" + has_completed_initialization = False + skips_initialization = False + allows_execution = False + status_prefix = "" + + +class OrchestratorReady(OrchestratorState): + """Initialized and ready for compilation.""" + + name = "ready" + has_completed_initialization = True + skips_initialization = True + allows_execution = False + status_prefix = "✓ Init" + + +class OrchestratorCompiled(OrchestratorState): + name = "compiled" + has_completed_initialization = True + skips_initialization = True + allows_execution = True + status_prefix = "✓ Compiled" + + +class OrchestratorExecuting(OrchestratorState): + name = "executing" + has_completed_initialization = True + skips_initialization = False + allows_execution = False + status_prefix = "🔄 Executing" + + +class OrchestratorCompleted(OrchestratorState): + name = "completed" + has_completed_initialization = True + skips_initialization = True + allows_execution = True + status_prefix = "✅ Complete" + + +class OrchestratorInitFailed(OrchestratorState): + name = "init_failed" + has_completed_initialization = False + skips_initialization = False + allows_execution = False + status_prefix = "❌ Init Failed" + + +class OrchestratorCompileFailed(OrchestratorState): + name = "compile_failed" + has_completed_initialization = True + skips_initialization = False + allows_execution = False + status_prefix = "❌ Compile Failed" + + +class OrchestratorExecFailed(OrchestratorState): + name = "exec_failed" + has_completed_initialization = True + skips_initialization = False + allows_execution = False + status_prefix = "❌ Exec Failed" # I/O-related constants diff --git a/openhcs/core/debug_session_projection.py b/openhcs/core/debug_session_projection.py index 5e20dad67e..48a9b0ad11 100644 --- a/openhcs/core/debug_session_projection.py +++ b/openhcs/core/debug_session_projection.py @@ -12,7 +12,7 @@ from openhcs.core.debug import DebugSession, DebugSnapshot, DebugTerminalSummary from openhcs.core.execution_state import ( ManagerExecutionState, - TerminalExecutionStatus, + TerminalExecutionStatus, ExecutionCancelled, ExecutionCompleted, ExecutionFailed, parse_terminal_status, ) @@ -211,11 +211,11 @@ def matches(cls, context: DebugSessionProjectionContext) -> bool: class TerminalDebugSessionPhase(DebugSessionPhaseDeclarationBase): """Shared terminal debugger phase matching a terminal execution status.""" - terminal_status: ClassVar[TerminalExecutionStatus | None] = None + terminal_status: ClassVar[type[TerminalExecutionStatus] | None] = None is_terminal = True @classmethod - def require_terminal_status(cls) -> TerminalExecutionStatus: + def require_terminal_status(cls) -> type[TerminalExecutionStatus]: if cls.terminal_status is None: raise TypeError(f"{cls.__name__} does not declare a terminal status.") return cls.terminal_status @@ -242,7 +242,7 @@ class TerminalCompleteDebugSessionPhase(TerminalDebugSessionPhase): title = "Debug Complete" detail = "Last debug command completed." status_prefix = "Debug complete" - terminal_status = TerminalExecutionStatus.COMPLETE + terminal_status = ExecutionCompleted class TerminalFailedDebugSessionPhase(TerminalDebugSessionPhase): @@ -251,7 +251,7 @@ class TerminalFailedDebugSessionPhase(TerminalDebugSessionPhase): title = "Debug Failed" detail = "Last debug command failed." status_prefix = "Debug failed" - terminal_status = TerminalExecutionStatus.FAILED + terminal_status = ExecutionFailed class TerminalCancelledDebugSessionPhase(TerminalDebugSessionPhase): @@ -260,7 +260,7 @@ class TerminalCancelledDebugSessionPhase(TerminalDebugSessionPhase): title = "Debug Cancelled" detail = "Last debug command was cancelled." status_prefix = "Debug cancelled" - terminal_status = TerminalExecutionStatus.CANCELLED + terminal_status = ExecutionCancelled class ReadyDebugSessionPhase(DebugSessionPhaseDeclarationBase): diff --git a/openhcs/core/execution_state.py b/openhcs/core/execution_state.py index e5824c9b63..282bbce038 100644 --- a/openhcs/core/execution_state.py +++ b/openhcs/core/execution_state.py @@ -2,14 +2,21 @@ from __future__ import annotations +from abc import ABC, abstractmethod from collections.abc import Callable, Mapping from dataclasses import asdict, dataclass, fields from enum import Enum from typing import Any, ClassVar +from metaclass_registry import AutoRegisterMeta from zmqruntime.messages import MessageFields -from openhcs.constants.constants import OrchestratorState +from openhcs.constants.constants import ( + OrchestratorCompleted, + OrchestratorExecFailed, + OrchestratorReady, + OrchestratorState, +) @dataclass(frozen=True, slots=True) @@ -62,7 +69,7 @@ def is_present(self) -> bool: class ExecutionCompletionPayload: """One typed terminal result passed from polling through UI presentation.""" - status: TerminalExecutionStatus + status: type[TerminalExecutionStatus] execution_id: str | None results: Mapping[str, Any] output_dataset: ExecutionOutputSummary @@ -80,7 +87,7 @@ def auto_add_output_dataset(self) -> bool | None: @classmethod def completed( cls, - status: TerminalExecutionStatus, + status: type[TerminalExecutionStatus], execution_id: str | None, execution_payload: Mapping[str, Any], ) -> ExecutionCompletionPayload: @@ -100,7 +107,7 @@ def completed( @classmethod def failed( cls, - status: TerminalExecutionStatus, + status: type[TerminalExecutionStatus], execution_id: str | None, execution_payload: Mapping[str, Any], ) -> ExecutionCompletionPayload: @@ -118,7 +125,7 @@ def failed( @classmethod def cancelled( cls, - status: TerminalExecutionStatus, + status: type[TerminalExecutionStatus], execution_id: str | None, execution_payload: Mapping[str, Any], ) -> ExecutionCompletionPayload: @@ -230,98 +237,87 @@ def stop_request( return next_state, requested_force -class TerminalExecutionStatus(str, Enum): - """Terminal workflow status shared by UI, MCP, and dev-client polling.""" +class TerminalExecutionStatus(ABC, metaclass=AutoRegisterMeta): + """Terminal workflow status shared by UI, MCP, and dev-client polling. - counts_as_failed: bool - orchestrator_state: OrchestratorState - status_prefix: str - emit_failure: bool - auto_add_output_dataset: bool - serialized_aliases: frozenset[str] + Each status is a subclass; ``name`` is its spelling on the wire, and + ``serialized_aliases`` are other runtime spellings it is read from. + """ - COMPLETE = ( - "complete", - False, - OrchestratorState.COMPLETED, - "✅ Complete", - False, - True, - frozenset(), - ExecutionCompletionPayload.completed, - ) - FAILED = ( - "failed", - True, - OrchestratorState.EXEC_FAILED, - "❌ Exec Failed", - True, - False, - frozenset({"error"}), - ExecutionCompletionPayload.failed, - ) - CANCELLED = ( - "cancelled", - True, - OrchestratorState.READY, - "✗ Cancelled", - False, - False, - frozenset(), - ExecutionCompletionPayload.cancelled, - ) + __registry_key__ = "name" + __skip_if_no_key__ = True - def __new__( - cls, - value: str, - counts_as_failed: bool, - orchestrator_state: OrchestratorState, - status_prefix: str, - emit_failure: bool, - auto_add_output_dataset: bool, - serialized_aliases: frozenset[str], - completion_factory: Callable[ - [TerminalExecutionStatus, str | None, Mapping[str, Any]], - ExecutionCompletionPayload, - ], - ): - obj = str.__new__(cls, value) - obj._value_ = value - obj.counts_as_failed = counts_as_failed - obj.orchestrator_state = orchestrator_state - obj.status_prefix = status_prefix - obj.emit_failure = emit_failure - obj.auto_add_output_dataset = auto_add_output_dataset - obj.serialized_aliases = serialized_aliases - obj._completion_factory = completion_factory - return obj + name: ClassVar[str | None] = None + counts_as_failed: ClassVar[bool] + orchestrator_state: ClassVar[type[OrchestratorState]] + status_prefix: ClassVar[str] + emit_failure: ClassVar[bool] + auto_add_output_dataset: ClassVar[bool] + serialized_aliases: ClassVar[frozenset[str]] = frozenset() - @classmethod - def _missing_(cls, value: object) -> TerminalExecutionStatus | None: - return next( - ( - status - for status in cls - if isinstance(value, str) and value in status.serialized_aliases - ), - None, - ) + @staticmethod + def named(value: str) -> type["TerminalExecutionStatus"]: + """The status spelled ``value`` or one of its aliases.""" + for status in TerminalExecutionStatus.__registry__.values(): + if value == status.name or value in status.serialized_aliases: + return status + raise ValueError(f"{value!r} is not a terminal execution status.") + @classmethod + @abstractmethod def completion_payload( - self, + cls, *, execution_id: str | None, execution_payload: Mapping[str, Any], ) -> ExecutionCompletionPayload: - """Decode this member's runtime payload into the canonical terminal result.""" + """Decode this status's runtime payload into the terminal result.""" + + +class ExecutionCompleted(TerminalExecutionStatus): + name = "complete" + counts_as_failed = False + orchestrator_state = OrchestratorCompleted + status_prefix = "✅ Complete" + emit_failure = False + auto_add_output_dataset = True + + @classmethod + def completion_payload(cls, *, execution_id, execution_payload): + return ExecutionCompletionPayload.completed(cls, execution_id, execution_payload) - return self._completion_factory(self, execution_id, execution_payload) + +class ExecutionFailed(TerminalExecutionStatus): + name = "failed" + counts_as_failed = True + orchestrator_state = OrchestratorExecFailed + status_prefix = "❌ Exec Failed" + emit_failure = True + auto_add_output_dataset = False + serialized_aliases = frozenset({"error"}) + + @classmethod + def completion_payload(cls, *, execution_id, execution_payload): + return ExecutionCompletionPayload.failed(cls, execution_id, execution_payload) + + +class ExecutionCancelled(TerminalExecutionStatus): + name = "cancelled" + counts_as_failed = True + orchestrator_state = OrchestratorReady + status_prefix = "✗ Cancelled" + emit_failure = False + auto_add_output_dataset = False + + @classmethod + def completion_payload(cls, *, execution_id, execution_payload): + return ExecutionCompletionPayload.cancelled(cls, execution_id, execution_payload) def parse_terminal_status( - status: str | TerminalExecutionStatus, -) -> TerminalExecutionStatus: - """Return the canonical terminal status for a UI/runtime status value.""" - if isinstance(status, TerminalExecutionStatus): + status: str | type[TerminalExecutionStatus], +) -> type[TerminalExecutionStatus]: + """Return the terminal status for a UI/runtime status value.""" + if isinstance(status, type): return status - return TerminalExecutionStatus(status) + return TerminalExecutionStatus.named(status) diff --git a/openhcs/core/orchestrator/bundle_execution.py b/openhcs/core/orchestrator/bundle_execution.py index 07044777d6..434104bda7 100644 --- a/openhcs/core/orchestrator/bundle_execution.py +++ b/openhcs/core/orchestrator/bundle_execution.py @@ -19,7 +19,7 @@ Optional, ) -from openhcs.constants.constants import Backend, OrchestratorState +from openhcs.constants.constants import Backend, OrchestratorCompleted, OrchestratorExecFailed, OrchestratorExecuting, OrchestratorReady from openhcs.core.artifacts import ( SpecialArtifactType, ) @@ -45,6 +45,7 @@ ExecutionResult, RuntimeContextObservation, RuntimeExecutionObservation, + MergeAllRuntimeRecords, RuntimeObservationMode, ) from openhcs.core.orchestrator.worker_execution import ( @@ -146,7 +147,7 @@ def is_success(self) -> bool: def require_success(self) -> None: """Reject failed axes at terminal boundaries without dropping their outputs.""" failures = [ - f"{axis_id}: {result.status.value}: {result.error_message or 'no error detail'}" + f"{axis_id}: {result.status.name}: {result.error_message or 'no error detail'}" for axis_id, result in self.items() if not result.is_success() ] @@ -170,7 +171,7 @@ class BundleExecutionRequest(ProgressExecutionContext): visualizer: ExecutionVisualizerABC | None log_file_base: Optional[str] progress_queue: ProgressQueue - runtime_observation_mode: RuntimeObservationMode + runtime_observation_mode: type[RuntimeObservationMode] debug_execution_policy: DebugExecutionPolicy prepared_worker_runner: PreparedForkWorkerLaneRunner | None = None @@ -236,7 +237,7 @@ def execute_bundle_request( for context in validated.compiled_contexts.values(): context.reset_completed_step_outputs() try: - orchestrator._state = OrchestratorState.EXECUTING + orchestrator._state = OrchestratorExecuting logger.info( f"Starting execution for {len(validated.compiled_contexts)} axis values " f"with max_workers={validated.actual_max_workers}." @@ -360,11 +361,11 @@ def execute_bundle_request( ), ) except ExecutionCancelledError: - orchestrator._state = OrchestratorState.READY + orchestrator._state = OrchestratorReady logger.info("Compiled bundle execution cancelled") raise except Exception as exc: - orchestrator._state = OrchestratorState.EXEC_FAILED + orchestrator._state = OrchestratorExecFailed logger.error(f"Failed to execute compiled bundle: {exc}") raise finally: @@ -395,12 +396,12 @@ def validate_bundle_execution( dataset_step_indexes = validate_dataset_scoped_contexts(compiled_contexts) if ( dataset_step_indexes - and request.runtime_observation_mode is RuntimeObservationMode.OMIT + and not request.runtime_observation_mode.collects_records ): raise ValueError( - "Dataset-scoped FunctionSteps require RuntimeObservationMode." - "MERGE_INTO_PARENT so compiled artifact inputs are available after " - "worker execution." + "Dataset-scoped FunctionSteps require a runtime observation mode " + "that returns records, so compiled artifact inputs are available " + "after worker execution." ) runtime_environment = request.execution_bundle.runtime_environment return ValidatedBundleExecution( @@ -656,7 +657,7 @@ def execute_dataset_scoped_steps( context_key=context_key, context=context, records=records, - runtime_observation_mode=RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode=MergeAllRuntimeRecords, outputs=StepExecutionObservation.combine( observations_by_context[context_key] ), @@ -1252,6 +1253,6 @@ def project_execution_state( """Project worker-lane results back into orchestrator lifecycle state.""" if execution_results.is_success(): - orchestrator._state = OrchestratorState.COMPLETED + orchestrator._state = OrchestratorCompleted else: - orchestrator._state = OrchestratorState.EXEC_FAILED + orchestrator._state = OrchestratorExecFailed diff --git a/openhcs/core/orchestrator/execution_result.py b/openhcs/core/orchestrator/execution_result.py index 157ca6db78..d08fbd2523 100644 --- a/openhcs/core/orchestrator/execution_result.py +++ b/openhcs/core/orchestrator/execution_result.py @@ -7,9 +7,11 @@ import copyreg from dataclasses import dataclass, field -from enum import Enum +from abc import ABC, abstractmethod from types import MappingProxyType -from typing import TYPE_CHECKING, Mapping, Optional +from typing import TYPE_CHECKING, ClassVar, Mapping, Optional + +from metaclass_registry import AutoRegisterMeta from openhcs.constants.constants import Backend from openhcs.core.context.processing_context import ProcessingContext @@ -46,12 +48,27 @@ def register(cls) -> None: RuntimeExecutionTransportSerialization.register() -class ExecutionStatus(Enum): - """Status of pipeline execution for an axis or combination.""" +class ExecutionStatus(ABC, metaclass=AutoRegisterMeta): + """How pipeline execution ended for one axis; ``name`` is its spelling.""" + + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + succeeded: ClassVar[bool] = False + + +class AxisSucceeded(ExecutionStatus): + name = "success" + succeeded = True + + +class AxisFailed(ExecutionStatus): + name = "error" + - SUCCESS = "success" - ERROR = "error" - CANCELLED = "cancelled" +class AxisCancelled(ExecutionStatus): + name = "cancelled" @dataclass(frozen=True) @@ -69,7 +86,7 @@ def from_context( context_key: str, context: ProcessingContext, records: tuple[StoredRuntimeValue, ...], - runtime_observation_mode: "RuntimeObservationMode", + runtime_observation_mode: "type[RuntimeObservationMode]", outputs: StepExecutionObservation, ) -> "RuntimeContextObservation": """Retain requested records beside the completed output projections.""" @@ -110,52 +127,62 @@ def merge_into(self, execution_contexts: Mapping[str, ProcessingContext]) -> Non context.record_completed_step_outputs(context_observation.outputs) -class RuntimeObservationMode(Enum): - """Controls whether worker runtime records are returned to the parent.""" +class RuntimeObservationMode(ABC): + """Which worker runtime records are returned to the parent; each mode keeps its own.""" - MERGE_INTO_PARENT = "merge_into_parent" - MERGE_DATASET_INPUTS = "merge_dataset_inputs" - OMIT = "omit" + collects_records: ClassVar[bool] = True + returns_every_record: ClassVar[bool] = False + """Workers keep each record to the end, so consumed inputs are never released.""" - @property - def collects_records(self) -> bool: - return self is not RuntimeObservationMode.OMIT - - @classmethod - def from_parent_requirement(cls, required: bool) -> "RuntimeObservationMode": + @staticmethod + def from_parent_requirement(required: bool) -> type["RuntimeObservationMode"]: """Select the mode implied by compiled parent-side execution needs.""" - return cls.MERGE_INTO_PARENT if required else cls.OMIT + return MergeAllRuntimeRecords if required else OmitRuntimeRecords - @classmethod + @staticmethod def for_compiled_bundle( - cls, bundle: "CompiledExecutionBundle" - ) -> "RuntimeObservationMode": + bundle: "CompiledExecutionBundle", + ) -> type["RuntimeObservationMode"]: """Consolidation carries rendered tables; only dataset-scoped inputs need records.""" if bundle.requires_parent_runtime_observation: - return cls.MERGE_DATASET_INPUTS - return cls.OMIT + return MergeDatasetInputRecords + return OmitRuntimeRecords + @classmethod def including_parent_requirement( - self, + cls, required: bool, - ) -> "RuntimeObservationMode": + ) -> type["RuntimeObservationMode"]: """Return this mode strengthened by an additional retention requirement.""" - return type(self).MERGE_INTO_PARENT if required else self + return MergeAllRuntimeRecords if required else cls + @staticmethod + @abstractmethod def retain_records( - self, records: tuple[StoredRuntimeValue, ...], context: ProcessingContext, ) -> tuple[StoredRuntimeValue, ...]: """Keep records consumed by the selected parent-side execution route.""" - if self is RuntimeObservationMode.OMIT: - return () - if self is RuntimeObservationMode.MERGE_INTO_PARENT: - return records + +class MergeAllRuntimeRecords(RuntimeObservationMode): + """Return every worker record to the parent.""" + + returns_every_record = True + + @staticmethod + def retain_records(records, context): + return records + + +class MergeDatasetInputRecords(RuntimeObservationMode): + """Return only the records dataset-scoped steps read as inputs.""" + + @staticmethod + def retain_records(records, context): selected_addresses = set() for plan in context.step_plans.values(): if not plan.execution_scope.requires_parent_runtime_observation: @@ -186,6 +213,16 @@ def retain_records( ) +class OmitRuntimeRecords(RuntimeObservationMode): + """Return no worker records.""" + + collects_records = False + + @staticmethod + def retain_records(records, context): + return () + + @dataclass(frozen=True) class ExecutionResult: """ @@ -201,7 +238,7 @@ class ExecutionResult: error_message: Optional error message if status is ERROR """ - status: ExecutionStatus + status: type[ExecutionStatus] axis_id: str failed_combination: Optional[str] = None error_message: Optional[str] = None @@ -211,14 +248,10 @@ class ExecutionResult: def is_success(self) -> bool: """Check if execution was successful.""" - return self.status == ExecutionStatus.SUCCESS - - def is_error(self) -> bool: - """Check if execution failed.""" - return self.status == ExecutionStatus.ERROR + return self.status.succeeded def is_cancelled(self) -> bool: - return self.status == ExecutionStatus.CANCELLED + return self.status is AxisCancelled @classmethod def success( @@ -228,7 +261,7 @@ def success( ) -> "ExecutionResult": """Create a successful execution result.""" return cls( - status=ExecutionStatus.SUCCESS, + status=AxisSucceeded, axis_id=axis_id, runtime_observation=runtime_observation or RuntimeExecutionObservation(), ) @@ -243,7 +276,7 @@ def error( ) -> "ExecutionResult": """Create an error execution result.""" return cls( - status=ExecutionStatus.ERROR, + status=AxisFailed, axis_id=axis_id, failed_combination=failed_combination, error_message=error_message, @@ -263,7 +296,7 @@ def cancelled( ) -> "ExecutionResult": """Return completed outputs from a cooperative cancellation boundary.""" return cls( - status=ExecutionStatus.CANCELLED, + status=AxisCancelled, axis_id=axis_id, error_message=error_message, runtime_observation=( diff --git a/openhcs/core/orchestrator/orchestrator.py b/openhcs/core/orchestrator/orchestrator.py index 755cc8b5bb..dd3286e639 100644 --- a/openhcs/core/orchestrator/orchestrator.py +++ b/openhcs/core/orchestrator/orchestrator.py @@ -9,7 +9,13 @@ from pathlib import Path from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Union -from openhcs.constants.constants import Backend, OrchestratorState +from openhcs.constants.constants import ( + Backend, + OrchestratorCreated, + OrchestratorInitFailed, + OrchestratorReady, + OrchestratorState, +) from openhcs.core.compiled_execution import CompiledExecutionBundle from openhcs.core.config import GlobalPipelineConfig, source_bindings_defaults_to_base from openhcs.core.execution_visualizer import ExecutionVisualizerABC @@ -81,7 +87,7 @@ class PipelineOrchestrator: __objectstate_delegate__ = "pipeline_config" _dataset_root: Optional[Path] = None _dataset_root_frozen: bool = False - state: AliasProperty[OrchestratorState] = AliasProperty("_state") + state: AliasProperty[type[OrchestratorState]] = AliasProperty("_state") def __init__( self, @@ -219,7 +225,7 @@ def _initialize_runtime_services( self._microscope_handler_rebuild_type: type[DatasetSource] | None = None self.default_pipeline_definition: Optional[List[AbstractStep]] = None self._initialized: bool = False - self._state: OrchestratorState = OrchestratorState.CREATED + self._state: type[OrchestratorState] = OrchestratorCreated # Progress callback for real-time execution updates self.progress_callback = progress_callback @@ -359,7 +365,7 @@ def bind_input_workspace( ) -> None: """Bind a caller-prepared generic input workspace before initialization.""" - if self._initialized or self.state is not OrchestratorState.CREATED: + if self._initialized or self.state is not OrchestratorCreated: raise RuntimeError( "An input workspace can only be bound before initialization." ) @@ -395,7 +401,7 @@ def _rebind_dataset_root_for_prepared_workspace( ) -> None: """Bind CREATED orchestrator execution to its prepared input workspace.""" - if self._initialized or self.state is not OrchestratorState.CREATED: + if self._initialized or self.state is not OrchestratorCreated: raise RuntimeError( "Prepared input workspace can only rebind a CREATED orchestrator." ) @@ -476,7 +482,7 @@ def initialize( # Mark as initialized BEFORE caching to avoid chicken-and-egg problem self._initialized = True - self._state = OrchestratorState.READY + self._state = OrchestratorReady # Auto-cache component keys and metadata for instant access logger.info("Caching component keys and metadata...") @@ -493,7 +499,7 @@ def initialize( ) return self except Exception as e: - self._state = OrchestratorState.INIT_FAILED + self._state = OrchestratorInitFailed logger.error(f"Failed to initialize orchestrator: {e}") raise @@ -502,7 +508,7 @@ def adopt_compiled_execution( execution_bundle: CompiledExecutionBundle, ) -> "PipelineOrchestrator": """Bind an admitted compiled source domain to this fresh runtime owner.""" - if self._initialized or self.state is not OrchestratorState.CREATED: + if self._initialized or self.state is not OrchestratorCreated: raise RuntimeError( "Compiled execution can only be adopted before initialization." ) @@ -553,7 +559,7 @@ def adopt_compiled_execution( execution_bundle.axis_ids ) self._initialized = True - self._state = OrchestratorState.READY + self._state = OrchestratorReady return self def is_initialized(self) -> bool: @@ -762,7 +768,7 @@ def execute_compiled_bundle( max_workers: Optional[int] = None, visualizer: ExecutionVisualizerABC | None = None, log_file_base: Optional[str] = None, - runtime_observation_mode: RuntimeObservationMode | None = None, + runtime_observation_mode: type[RuntimeObservationMode] | None = None, debug_execution_policy: DebugExecutionPolicy = NoOpDebugExecutionPolicy(), prepared_worker_runner: "PreparedForkWorkerLaneRunner | None" = None, ) -> Dict[str, ExecutionResult]: @@ -1030,7 +1036,7 @@ def _invalidate_source_projection(self) -> None: self.microscope_handler = None self.input_dir = None self._initialized = False - self._state = OrchestratorState.CREATED + self._state = OrchestratorCreated self._component_keys_cache.clear() self.metadata_cache.clear_cache() diff --git a/openhcs/core/orchestrator/worker_execution.py b/openhcs/core/orchestrator/worker_execution.py index e8973bb225..b55c419b8b 100644 --- a/openhcs/core/orchestrator/worker_execution.py +++ b/openhcs/core/orchestrator/worker_execution.py @@ -33,6 +33,7 @@ ExecutionResult, RuntimeContextObservation, RuntimeExecutionObservation, + MergeAllRuntimeRecords, RuntimeObservationMode, ) from openhcs.core.orchestrator.cancellation import ( @@ -690,7 +691,7 @@ class ExecuteInheritedLane(AnsweredWorkerCommand): lane_axis_context_keys: list[tuple[str, list[str]]] lane_context: WorkerLaneExecutionContext - runtime_observation_mode: RuntimeObservationMode + runtime_observation_mode: type[RuntimeObservationMode] def run(self, connection: Any) -> Dict[str, ExecutionResult]: execution_bundle = ForkInheritedWorkerExecutionState.require_current() @@ -1286,7 +1287,7 @@ def _execute_axis_with_sequential_combinations( pipeline_definition: List[AbstractStep], axis_contexts: TransportAxisContexts, lane_context: WorkerLaneExecutionContext, - runtime_observation_mode: RuntimeObservationMode, + runtime_observation_mode: type[RuntimeObservationMode], cancellation: ExecutionCancellationSignal | None = None, release_axis_resources: bool = True, ) -> ExecutionResult: @@ -1463,7 +1464,7 @@ def _execute_single_axis_static( *, context_key: str, cancellation: ExecutionCancellationSignal | None = None, - runtime_observation_mode: RuntimeObservationMode = RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode: type[RuntimeObservationMode] = MergeAllRuntimeRecords, ) -> ExecutionResult: """Execute one frozen axis context against the compiled pipeline.""" @@ -1487,7 +1488,7 @@ def _execute_single_axis_static( # original retention contract. Ordinary execution can retire dead inputs # only after the step's output/progress consumers have completed. release_consumed = ( - runtime_observation_mode is not RuntimeObservationMode.MERGE_INTO_PARENT + not runtime_observation_mode.returns_every_record and isinstance(lane_context.debug_execution_policy, NoOpDebugExecutionPolicy) and not any( plan.visualize or plan.streaming_configs @@ -1635,7 +1636,7 @@ def execute_worker_lane( pipeline_definition: List[AbstractStep], lane_axis_contexts: WorkerLaneAxisContexts, lane_context: WorkerLaneExecutionContext, - runtime_observation_mode: RuntimeObservationMode, + runtime_observation_mode: type[RuntimeObservationMode], cancellation: ExecutionCancellationSignal | None = None, release_axis_resources: bool = True, ) -> Dict[str, ExecutionResult]: diff --git a/openhcs/core/orchestrator/worker_lanes.py b/openhcs/core/orchestrator/worker_lanes.py index be38c7686b..14196e32e7 100644 --- a/openhcs/core/orchestrator/worker_lanes.py +++ b/openhcs/core/orchestrator/worker_lanes.py @@ -118,7 +118,7 @@ class WorkerLaneExecutionPlan(ProgressExecutionContext): debug_execution_policy: DebugExecutionPolicy assignments: WorkerAssignmentPlan - runtime_observation_mode: RuntimeObservationMode + runtime_observation_mode: type[RuntimeObservationMode] def active_lane_items(self) -> tuple[tuple[str, WorkerLaneAxisContexts], ...]: """Return worker lanes that own at least one axis context.""" diff --git a/openhcs/core/pipeline/compiler.py b/openhcs/core/pipeline/compiler.py index 9dfa0fe33d..2aaa8ab6ac 100644 --- a/openhcs/core/pipeline/compiler.py +++ b/openhcs/core/pipeline/compiler.py @@ -22,7 +22,8 @@ import dill as pickle from openhcs.constants.constants import ( - OrchestratorState, + OrchestratorCompileFailed, + OrchestratorCompiled, READ_BACKEND, WRITE_BACKEND, Backend, @@ -1477,7 +1478,7 @@ def _finalize_compilation( PipelineCompiler._log_path_planning_summary(compiled_contexts) logger.info("Stripping attributes from pipeline definition steps.") StepAttributeStripper.strip_step_attributes(pipeline_definition, {}) - orchestrator._state = OrchestratorState.COMPILED + orchestrator._state = OrchestratorCompiled logger.info( f"Execution config: {effective_config.num_workers} workers configured for pipeline execution" ) @@ -1695,7 +1696,7 @@ def compile_pipelines( ) return execution_bundle except Exception as e: - orchestrator._state = OrchestratorState.COMPILE_FAILED + orchestrator._state = OrchestratorCompileFailed logger.error(f"Failed to compile pipelines: {e}") raise diff --git a/openhcs/mcp/dev_client_core.py b/openhcs/mcp/dev_client_core.py index e975db7572..4e33085948 100644 --- a/openhcs/mcp/dev_client_core.py +++ b/openhcs/mcp/dev_client_core.py @@ -56,7 +56,12 @@ from openhcs.agent.path_policy import AgentPathPolicy from openhcs.agent.runtime_platform import AgentRuntimePlatformAuthority from openhcs.agent.ui_bridge_environment import UiBridgeDescriptorEnvironment -from openhcs.constants.constants import OrchestratorState +from openhcs.constants.constants import ( + OrchestratorCompileFailed, + OrchestratorExecFailed, + OrchestratorInitFailed, + OrchestratorState, +) from openhcs.core.execution_state import ( ManagerExecutionState, TerminalExecutionStatus, @@ -536,7 +541,7 @@ class WorkflowTerminalStateCriterion(ABC, metaclass=AutoRegisterMeta): __skip_if_no_key__ = True workflow: ClassVar[UiSelectedPlateWorkflowKind] - failed_orchestrator_states: ClassVar[tuple[OrchestratorState, ...]] = () + failed_orchestrator_states: ClassVar[tuple[type[OrchestratorState], ...]] = () terminal_state_is_idempotent: ClassVar[bool] = False @classmethod @@ -565,7 +570,7 @@ def failed_for_row(self, row: DatasetRowState) -> bool: class InitWorkflowTerminalStateCriterion(WorkflowTerminalStateCriterion): workflow = UiSelectedPlateWorkflowKind.INIT - failed_orchestrator_states = (OrchestratorState.INIT_FAILED,) + failed_orchestrator_states = (OrchestratorInitFailed,) terminal_state_is_idempotent = True def terminal_for_row(self, row: DatasetRowState) -> bool: @@ -574,7 +579,7 @@ def terminal_for_row(self, row: DatasetRowState) -> bool: class CompileWorkflowTerminalStateCriterion(WorkflowTerminalStateCriterion): workflow = UiSelectedPlateWorkflowKind.COMPILE - failed_orchestrator_states = (OrchestratorState.COMPILE_FAILED,) + failed_orchestrator_states = (OrchestratorCompileFailed,) def terminal_for_row(self, row: DatasetRowState) -> bool: return row.compile_pending is False and row.compiled is True @@ -582,7 +587,7 @@ def terminal_for_row(self, row: DatasetRowState) -> bool: class RunWorkflowTerminalStateCriterion(WorkflowTerminalStateCriterion): workflow = UiSelectedPlateWorkflowKind.RUN - failed_orchestrator_states = (OrchestratorState.EXEC_FAILED,) + failed_orchestrator_states = (OrchestratorExecFailed,) def terminal_for_row(self, row: DatasetRowState) -> bool: return ( @@ -2451,24 +2456,21 @@ def workflow_poll_target_rows( def terminal_execution_status( value: str | None, -) -> TerminalExecutionStatus | None: +) -> type[TerminalExecutionStatus] | None: if value is None: return None try: - return TerminalExecutionStatus(value) + return TerminalExecutionStatus.named(value) except ValueError: return None def parse_orchestrator_state( value: str | None, -) -> OrchestratorState | None: +) -> type[OrchestratorState] | None: if value is None: return None - try: - return OrchestratorState(value) - except ValueError: - return None + return OrchestratorState.named(value) def workflow_poll_summary_result( diff --git a/openhcs/mcp/dev_client_session.py b/openhcs/mcp/dev_client_session.py index 2fae6f25bf..079ef6f067 100644 --- a/openhcs/mcp/dev_client_session.py +++ b/openhcs/mcp/dev_client_session.py @@ -15,6 +15,7 @@ from python_introspect import JsonValue from openhcs.agent.capabilities import agent_capabilities +from openhcs.constants.constants import OrchestratorCompileFailed, OrchestratorInitFailed from openhcs.agent.dto.execution_connection import ExecutionConnectionSpec from openhcs.agent.dto.session import DatasetListState, DatasetRowState from openhcs.mcp.dev_client_core import ( @@ -48,7 +49,7 @@ class InitializeStage(DatasetStage): @classmethod def finished(cls, row: DatasetRowState) -> bool: return not row.init_pending and ( - row.initialized or row.orchestrator_state == "init_failed" + row.initialized or row.orchestrator_state == OrchestratorInitFailed.name ) @classmethod @@ -62,7 +63,7 @@ class CompileStage(DatasetStage): @classmethod def finished(cls, row: DatasetRowState) -> bool: return not row.compile_pending and ( - row.compiled or row.orchestrator_state == "compile_failed" + row.compiled or row.orchestrator_state == OrchestratorCompileFailed.name ) @classmethod diff --git a/openhcs/mcp/installed_demo.py b/openhcs/mcp/installed_demo.py index 04ae0a01c6..c3726b313f 100644 --- a/openhcs/mcp/installed_demo.py +++ b/openhcs/mcp/installed_demo.py @@ -34,7 +34,7 @@ ) from openhcs.agent.dto.viewer import ViewerWindowValidationSummaryResult from openhcs.core.config import LazyNapariStreamingConfig -from openhcs.core.execution_state import TerminalExecutionStatus +from openhcs.core.execution_state import ExecutionCompleted from openhcs.core.native_threading import configure_native_thread_environment from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.core.plate_file_inventory import PlateFileKind @@ -396,7 +396,7 @@ def _execute_pipeline( timeout_seconds=None, ) (row,) = (row for row in datasets.rows if row.root == str(plate_path)) - if row.terminal_status != TerminalExecutionStatus.COMPLETE.value: + if row.terminal_status != ExecutionCompleted.name: raise InstalledDemoFailure( f"Portable neurite execution ended with {row.terminal_status}: {row}" ) diff --git a/openhcs/pyqt_gui/services/reactor_providers.py b/openhcs/pyqt_gui/services/reactor_providers.py index 17922c59d0..76887b99b9 100644 --- a/openhcs/pyqt_gui/services/reactor_providers.py +++ b/openhcs/pyqt_gui/services/reactor_providers.py @@ -225,13 +225,13 @@ def _get_current_orchestrator(self) -> PipelineOrchestrator | None: from objectstate import ObjectStateRegistry - from openhcs.constants.constants import OrchestratorState + from openhcs.constants.constants import OrchestratorCreated from openhcs.core.orchestrator import PipelineOrchestrator orchestrator = ObjectStateRegistry.get_object(plate_manager.session.current_scope_id) if not isinstance(orchestrator, PipelineOrchestrator): return None - if orchestrator.state is OrchestratorState.CREATED: + if orchestrator.state is OrchestratorCreated: return None return orchestrator diff --git a/openhcs/pyqt_gui/services/ui_bridge_plate_manager.py b/openhcs/pyqt_gui/services/ui_bridge_plate_manager.py index a46f292d43..54a7b937ca 100644 --- a/openhcs/pyqt_gui/services/ui_bridge_plate_manager.py +++ b/openhcs/pyqt_gui/services/ui_bridge_plate_manager.py @@ -72,7 +72,7 @@ ) from openhcs.pyqt_gui.widgets.plate_manager import PlateManagerWidget from openhcs.core.execution_state import ( - TerminalExecutionStatus, + ExecutionFailed, ) ORCHESTRATOR_DOCUMENT_TITLE = "Plate manager orchestrator config" @@ -549,7 +549,7 @@ def _overview_row_detail(row: DatasetRowState) -> str: @staticmethod def _overview_row_severity(row: DatasetRowState) -> UiLiveOverviewSeverity: - if row.terminal_status == TerminalExecutionStatus.FAILED.value: + if row.terminal_status == ExecutionFailed.name: return UiLiveOverviewSeverity.ERROR if row.execution_active or row.queue_position is not None: return UiLiveOverviewSeverity.WARNING diff --git a/openhcs/pyqt_gui/widgets/plate_manager.py b/openhcs/pyqt_gui/widgets/plate_manager.py index dd8f737f23..665a69a182 100644 --- a/openhcs/pyqt_gui/widgets/plate_manager.py +++ b/openhcs/pyqt_gui/widgets/plate_manager.py @@ -84,7 +84,7 @@ output_relations, ) from openhcs.core.config import PipelineConfig -from openhcs.constants.constants import OrchestratorState +from openhcs.constants.constants import OrchestratorCreated from openhcs.pyqt_gui.services.ui_bridge_contracts import ( UiOwnedStateSurfaceDeclaration, ) @@ -355,7 +355,7 @@ def _format_item_content(self, item: DatasetRow, index: int, context: None): @override def _get_list_item_tooltip(self, item: DatasetRow) -> str: orchestrator = ObjectStateRegistry.get_object(item.scope_id) - return f"Status: {orchestrator.state.value}" if orchestrator else "" + return f"Status: {orchestrator.state.name}" if orchestrator else "" @override def _get_item_scope_id(self, item: DatasetRow, index: int) -> Optional[str]: @@ -450,7 +450,7 @@ def results_orchestrator(self, scope_id: str | None): orchestrator = ObjectStateRegistry.get_object(scope_id) if ( orchestrator is not None - and orchestrator.state is not OrchestratorState.CREATED + and orchestrator.state is not OrchestratorCreated ): return orchestrator return None diff --git a/openhcs/runtime/zmq_execution_observation.py b/openhcs/runtime/zmq_execution_observation.py index 52a1d22888..0fedf0abb6 100644 --- a/openhcs/runtime/zmq_execution_observation.py +++ b/openhcs/runtime/zmq_execution_observation.py @@ -52,7 +52,7 @@ def _axis_membership_failures( class ZMQExecutionAxisOutcome: """Small per-axis result retained without worker runtime values.""" - status: ExecutionStatus + status: type[ExecutionStatus] failed_combination: str | None = None error_message: str | None = None @@ -130,7 +130,7 @@ def axis_count(self) -> int: @property def successful_axis_count(self) -> int: return sum( - outcome.status is ExecutionStatus.SUCCESS + outcome.status.succeeded for outcome in self.outcomes_by_axis.values() ) @@ -140,9 +140,9 @@ def require_successful_axes(self) -> None: self.outcomes_by_axis, ) unsuccessful = { - axis_id: outcome.status.value + axis_id: outcome.status.name for axis_id, outcome in self.outcomes_by_axis.items() - if outcome.status is not ExecutionStatus.SUCCESS + if not outcome.status.succeeded } failures = list(membership_failures) if unsuccessful: diff --git a/openhcs/runtime/zmq_execution_server.py b/openhcs/runtime/zmq_execution_server.py index 5f71b1e779..49cd9c8607 100644 --- a/openhcs/runtime/zmq_execution_server.py +++ b/openhcs/runtime/zmq_execution_server.py @@ -814,45 +814,20 @@ def _export_runtime_observation( return from openhcs.core.runtime_execution_validation import runtime_output_roots - from openhcs.core.runtime_exports import RuntimeExportObservation - from openhcs.runtime.zmq_execution_observation import ( - ZMQRuntimeExecutionObservationExport, - ZMQRuntimeExecutionOutcomeExport, - ) - from openhcs.runtime.zmq_execution_signature import ( - ZMQAuxiliaryParamField, - ZMQRuntimeObservationExportScope, - ) + from openhcs.runtime.zmq_execution_signature import ZMQAuxiliaryParamField execution_bundle = compilation.execution_bundle output_roots = runtime_output_roots( execution_bundle.runtime_contexts, compilation.output_dataset.output_dataset_root, ) - if ( - request_context.auxiliary_params.runtime_observation_export_scope - is ZMQRuntimeObservationExportScope.OUTCOMES - ): - observed_exports = RuntimeExportObservation.from_runtime_observations( - execution_results.runtime_observations - ) - export = ZMQRuntimeExecutionOutcomeExport.from_execution( - compiled_axis_ids=execution_bundle.runtime_contexts, - execution_results=execution_results, - output_roots=output_roots, - server_environment=self._server_environment, - execution_id=request_context.execution_id, - exports=observed_exports, - ) - else: - export = ZMQRuntimeExecutionObservationExport.from_execution( - compiled_contexts=execution_bundle.runtime_contexts, - execution_results=execution_results, - runtime_observations=execution_results.runtime_observations, - output_roots=output_roots, - server_environment=self._server_environment, - execution_id=request_context.execution_id, - ) + export = request_context.auxiliary_params.runtime_observation_export_scope.export( + execution_bundle=execution_bundle, + execution_results=execution_results, + output_roots=output_roots, + server_environment=self._server_environment, + execution_id=request_context.execution_id, + ) export.write(export_path) self.active_executions[request_context.execution_id].set_extra( ZMQAuxiliaryParamField.RUNTIME_OBSERVATION_EXPORT_PATH.value, @@ -860,7 +835,7 @@ def _export_runtime_observation( ) self.active_executions[request_context.execution_id].set_extra( ZMQAuxiliaryParamField.RUNTIME_OBSERVATION_EXPORT_SCOPE.value, - request_context.auxiliary_params.runtime_observation_export_scope.value, + request_context.auxiliary_params.runtime_observation_export_scope.name, ) logger.info( "[%s] Exported runtime observation to %s", diff --git a/openhcs/runtime/zmq_execution_signature.py b/openhcs/runtime/zmq_execution_signature.py index 52ea245922..d95fde7239 100644 --- a/openhcs/runtime/zmq_execution_signature.py +++ b/openhcs/runtime/zmq_execution_signature.py @@ -5,10 +5,13 @@ import hashlib import json from collections.abc import Mapping +from abc import ABC, abstractmethod from dataclasses import dataclass from enum import Enum from pathlib import Path -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, ClassVar + +from metaclass_registry import AutoRegisterMeta from zmqruntime.messages import ExecuteRequest, MessageFields @@ -39,11 +42,88 @@ class ZMQAuxiliaryParamField(Enum): RUNTIME_OBSERVATION_EXPORT_SCOPE = "runtime_observation_export_scope" -class ZMQRuntimeObservationExportScope(Enum): - """Amount of runtime evidence requested from an ordinary execution.""" +class ZMQRuntimeObservationExportScope(ABC, metaclass=AutoRegisterMeta): + """Amount of runtime evidence requested from an ordinary execution. + + ``name`` is the scope's spelling on the wire and in agent requests. + """ + + __registry_key__ = "name" + __skip_if_no_key__ = True + + name: ClassVar[str | None] = None + retains_runtime_values: ClassVar[bool] + """Workers return their full runtime values for the export.""" - VALUES = "values" - OUTCOMES = "outcomes" + @staticmethod + def named(value: str) -> type["ZMQRuntimeObservationExportScope"]: + scope = ZMQRuntimeObservationExportScope.__registry__.get(value) + if scope is None: + raise ValueError(f"Unknown runtime observation export scope: {value!r}.") + return scope + + @staticmethod + @abstractmethod + def export( + *, + execution_bundle: CompiledExecutionBundle, + execution_results, + output_roots, + server_environment, + execution_id: str, + ): + """Build this scope's runtime export from one finished execution.""" + + +class ExportRuntimeValues(ZMQRuntimeObservationExportScope): + """Export the full runtime values.""" + + name = "values" + retains_runtime_values = True + + @staticmethod + def export( + *, execution_bundle, execution_results, output_roots, server_environment, execution_id + ): + from openhcs.runtime.zmq_execution_observation import ( + ZMQRuntimeExecutionObservationExport, + ) + + return ZMQRuntimeExecutionObservationExport.from_execution( + compiled_contexts=execution_bundle.runtime_contexts, + execution_results=execution_results, + runtime_observations=execution_results.runtime_observations, + output_roots=output_roots, + server_environment=server_environment, + execution_id=execution_id, + ) + + +class ExportRuntimeOutcomes(ZMQRuntimeObservationExportScope): + """Export per-axis outcomes and exports, without array values.""" + + name = "outcomes" + retains_runtime_values = False + + @staticmethod + def export( + *, execution_bundle, execution_results, output_roots, server_environment, execution_id + ): + from openhcs.core.runtime_exports import RuntimeExportObservation + from openhcs.runtime.zmq_execution_observation import ( + ZMQRuntimeExecutionOutcomeExport, + ) + + return ZMQRuntimeExecutionOutcomeExport.from_execution( + compiled_axis_ids=execution_bundle.runtime_contexts, + execution_results=execution_results, + output_roots=output_roots, + server_environment=server_environment, + execution_id=execution_id, + exports=RuntimeExportObservation.from_runtime_observations( + execution_results.runtime_observations + ), + ) @dataclass(frozen=True, slots=True) @@ -53,14 +133,13 @@ class ZMQAuxiliaryExecutionParams: partition_values: tuple[str, ...] | None = None debug_execution_config: DebugExecutionConfig | None = None runtime_observation_export_path: Path | None = None - runtime_observation_export_scope: ZMQRuntimeObservationExportScope = ( - ZMQRuntimeObservationExportScope.VALUES + runtime_observation_export_scope: type[ZMQRuntimeObservationExportScope] = ( + ExportRuntimeValues ) def __post_init__(self) -> None: if ( - self.runtime_observation_export_scope - is ZMQRuntimeObservationExportScope.OUTCOMES + not self.runtime_observation_export_scope.retains_runtime_values and self.runtime_observation_export_path is None ): raise ValueError("Outcome observation requires an export path.") @@ -68,15 +147,14 @@ def __post_init__(self) -> None: def runtime_observation_mode_for( self, execution_bundle: CompiledExecutionBundle, - ) -> RuntimeObservationMode: + ) -> type[RuntimeObservationMode]: """Resolve retention from compiled needs plus an explicit export request.""" return RuntimeObservationMode.for_compiled_bundle( execution_bundle ).including_parent_requirement( self.runtime_observation_export_path is not None - and self.runtime_observation_export_scope - is ZMQRuntimeObservationExportScope.VALUES + and self.runtime_observation_export_scope.retains_runtime_values ) def to_transport(self) -> dict[str, Any]: @@ -105,7 +183,7 @@ def runtime_observation_items(self) -> TransportRequestItems: ), ( ZMQAuxiliaryParamField.RUNTIME_OBSERVATION_EXPORT_SCOPE.value, - self.runtime_observation_export_scope.value, + self.runtime_observation_export_scope.name, ), ) @@ -150,15 +228,12 @@ def from_transport( ) @staticmethod - def _scope_from_transport(value: str | None) -> ZMQRuntimeObservationExportScope: + def _scope_from_transport( + value: str | None, + ) -> type[ZMQRuntimeObservationExportScope]: if value is None: - return ZMQRuntimeObservationExportScope.VALUES - try: - return ZMQRuntimeObservationExportScope(value) - except ValueError as exc: - raise ValueError( - f"Unknown runtime observation export scope: {value!r}." - ) from exc + return ExportRuntimeValues + return ZMQRuntimeObservationExportScope.named(value) @staticmethod def _partition_values_from_transport( diff --git a/openhcs/runtime/zmq_worker_execution.py b/openhcs/runtime/zmq_worker_execution.py index 3858572861..34d648cd24 100644 --- a/openhcs/runtime/zmq_worker_execution.py +++ b/openhcs/runtime/zmq_worker_execution.py @@ -31,7 +31,7 @@ class ZMQWorkerExecutionRequest: execution_id: str orchestrator: Any execution_bundle: CompiledExecutionBundle - runtime_observation_mode: RuntimeObservationMode + runtime_observation_mode: type[RuntimeObservationMode] progress_context: ProgressExecutionContext debug_execution_policy: Any active_execution_record: Any diff --git a/tests/integration/test_agent_observation_export_zmq.py b/tests/integration/test_agent_observation_export_zmq.py index 9fc672778f..57d1bd6252 100644 --- a/tests/integration/test_agent_observation_export_zmq.py +++ b/tests/integration/test_agent_observation_export_zmq.py @@ -51,7 +51,7 @@ ) from openhcs.runtime.zmq_execution_signature import ( ZMQAuxiliaryExecutionParams, - ZMQRuntimeObservationExportScope, + ExportRuntimeOutcomes, ) @@ -319,7 +319,7 @@ def test_ordinary_execution_can_export_outcomes_without_value_observation( ).with_auxiliary_params( ZMQAuxiliaryExecutionParams( runtime_observation_export_path=export_path, - runtime_observation_export_scope=ZMQRuntimeObservationExportScope.OUTCOMES, + runtime_observation_export_scope=ExportRuntimeOutcomes, ) ) @@ -344,7 +344,7 @@ def test_ordinary_execution_can_export_outcomes_without_value_observation( assert completed.observation is None assert ( completed.receipt.observation_export_scope - is ZMQRuntimeObservationExportScope.OUTCOMES + is ExportRuntimeOutcomes ) assert { BenchmarkPhase.COMPILE_OPENHCS, @@ -394,7 +394,7 @@ def test_measured_repetitions_share_one_owned_ordinary_server(tmp_path: Path) -> evidence_dir / "observation.pkl.gz" ), runtime_observation_export_scope=( - ZMQRuntimeObservationExportScope.OUTCOMES + ExportRuntimeOutcomes ), ) ) @@ -473,7 +473,7 @@ def test_well_throughput_wrapper_runs_a_synthetic_ordinary_plate( assert receipt.subject_id == str(source_identity) assert receipt.execution_subject_id == str(plate) assert receipt.expected_axis_count == receipt.observed_axis_count == 1 - assert receipt.observation_export_scope is ZMQRuntimeObservationExportScope.OUTCOMES + assert receipt.observation_export_scope is ExportRuntimeOutcomes def test_paper_two_worker_geometry_uses_one_ordinary_measured_run( @@ -566,7 +566,7 @@ def test_measured_cli_uses_ordinary_source_session_and_shared_finalizer( assert receipt.execution_subject_id == str(plate) assert receipt.run_id == "ordinary-cli" assert receipt.pipeline_name == "pipeline" - assert receipt.observation_export_scope is ZMQRuntimeObservationExportScope.OUTCOMES + assert receipt.observation_export_scope is ExportRuntimeOutcomes # The CLI measures on its own client, so client phases precede the server's. assert BenchmarkPhase.SERVER_PIPELINE_JOB in { timing.phase for timing in receipt.phase_timings diff --git a/tests/integration/test_chained_callable_lineage_journey.py b/tests/integration/test_chained_callable_lineage_journey.py index b57f976444..510276ec69 100644 --- a/tests/integration/test_chained_callable_lineage_journey.py +++ b/tests/integration/test_chained_callable_lineage_journey.py @@ -22,7 +22,7 @@ PipelineConfig, ) from openhcs.core.function_patterns import MainFlowInputProjection -from openhcs.core.orchestrator.execution_result import RuntimeObservationMode +from openhcs.core.orchestrator.execution_result import MergeAllRuntimeRecords from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.core.runtime_object_labels import object_label_dense_array @@ -137,7 +137,7 @@ def test_chained_public_callable_uses_declared_main_flow_not_storage_argument( results = orchestrator.execute_compiled_bundle( execution_bundle=bundle, max_workers=1, - runtime_observation_mode=RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode=MergeAllRuntimeRecords, progress_queue=AgentProgressQueue(), progress_context=ProgressExecutionContext( execution_id="chained-public-callable-synthetic", diff --git a/tests/integration/test_image_math_publication_620.py b/tests/integration/test_image_math_publication_620.py index b849e8e233..f327c074cf 100644 --- a/tests/integration/test_image_math_publication_620.py +++ b/tests/integration/test_image_math_publication_620.py @@ -17,7 +17,7 @@ GlobalPipelineConfig, LazyNapariStreamingConfig, LazyPathPlanningConfig, LazyProcessingConfig, ) -from openhcs.core.orchestrator.execution_result import RuntimeObservationMode +from openhcs.core.orchestrator.execution_result import MergeAllRuntimeRecords from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.core.source_bindings import MetadataExtractionRule, MetadataSource @@ -64,7 +64,7 @@ def test_registered_mixed_image_math_publishes_once_in_both_operand_orders(tmp_p )).execution_bundle outcomes = PipelineOrchestrator(source, pipeline_config=config).initialize().execute_compiled_bundle( execution_bundle=bundle, max_workers=1, - runtime_observation_mode=RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode=MergeAllRuntimeRecords, progress_queue=AgentProgressQueue(), progress_context=ProgressExecutionContext( execution_id='engineering620-registered', diff --git a/tests/integration/test_measurement_declaration_journey.py b/tests/integration/test_measurement_declaration_journey.py index 01a9a187ee..c980db02e7 100644 --- a/tests/integration/test_measurement_declaration_journey.py +++ b/tests/integration/test_measurement_declaration_journey.py @@ -43,7 +43,7 @@ from openhcs.core.measurement_feature_queries import measurement_values_for_feature from openhcs.core.measurement_row_materialization import DataclassMeasurementColumnarRows from openhcs.core.memory import numpy -from openhcs.core.orchestrator.execution_result import RuntimeObservationMode +from openhcs.core.orchestrator.execution_result import MergeAllRuntimeRecords from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.core.pipeline.function_contracts import artifact_outputs @@ -285,7 +285,7 @@ def _execute(path, document, bundle): return orchestrator.execute_compiled_bundle( execution_bundle=bundle, max_workers=1, - runtime_observation_mode=RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode=MergeAllRuntimeRecords, progress_queue=AgentProgressQueue(), progress_context=ProgressExecutionContext( execution_id="measurement-contract-synthetic", diff --git a/tests/integration/test_primary_segmentation_diagnostics_journey.py b/tests/integration/test_primary_segmentation_diagnostics_journey.py index 88c75b18f9..1a4889e236 100644 --- a/tests/integration/test_primary_segmentation_diagnostics_journey.py +++ b/tests/integration/test_primary_segmentation_diagnostics_journey.py @@ -125,7 +125,7 @@ def test_normal_compiled_runtime_persists_diagnostics_and_preserves_secondary_bi LazySourceBindingsConfig, PipelineConfig, ) - from openhcs.core.orchestrator.execution_result import RuntimeObservationMode + from openhcs.core.orchestrator.execution_result import MergeAllRuntimeRecords from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.core.runtime_object_labels import object_label_dense_array @@ -298,7 +298,7 @@ def step(func, name, kwargs): results = orchestrator.execute_compiled_bundle( execution_bundle=bundle, max_workers=1, - runtime_observation_mode=RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode=MergeAllRuntimeRecords, progress_queue=AgentProgressQueue(), progress_context=ProgressExecutionContext( execution_id="primary-diagnostics-synthetic", diff --git a/tests/integration/test_source_schema_aggregate_publication.py b/tests/integration/test_source_schema_aggregate_publication.py index 37c4614558..31f3e24714 100644 --- a/tests/integration/test_source_schema_aggregate_publication.py +++ b/tests/integration/test_source_schema_aggregate_publication.py @@ -28,7 +28,7 @@ PipelineConfig, ) from openhcs.core.memory import numpy as numpy_function -from openhcs.core.orchestrator.execution_result import RuntimeObservationMode +from openhcs.core.orchestrator.execution_result import MergeAllRuntimeRecords from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator from openhcs.core.pipeline.function_contracts import artifact_outputs from openhcs.core.pipeline_document import PipelineDocumentCodec @@ -112,7 +112,7 @@ def test_real_z_aggregation_persists_scalar_and_unchanged_source_planes(tmp_path orchestrator = PipelineOrchestrator(source, pipeline_config=document.pipeline_config).initialize() outcomes = orchestrator.execute_compiled_bundle( execution_bundle=bundle, max_workers=1, - runtime_observation_mode=RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode=MergeAllRuntimeRecords, progress_queue=AgentProgressQueue(), progress_context=ProgressExecutionContext( execution_id="source-schema-aggregate-synthetic", diff --git a/tests/unit/pyqt_gui/test_debug_toolbar.py b/tests/unit/pyqt_gui/test_debug_toolbar.py index 5bf15c63f1..da75ccbeaa 100644 --- a/tests/unit/pyqt_gui/test_debug_toolbar.py +++ b/tests/unit/pyqt_gui/test_debug_toolbar.py @@ -8,6 +8,7 @@ from PyQt6.QtWidgets import QApplication from pyqt_reactive.theming import ColorScheme +from openhcs.constants.constants import OrchestratorReady from openhcs.core.artifacts import MeasurementsArtifactType from openhcs.core.config import NapariStreamingConfig from openhcs.core.debug import ( @@ -24,7 +25,7 @@ from openhcs.authoring.session.progress_notifications import ( DebugSnapshotAvailableNotification, ) -from openhcs.core.execution_state import ManagerExecutionState, TerminalExecutionStatus +from openhcs.core.execution_state import ManagerExecutionState, ExecutionFailed from openhcs.core.progress import ProgressEvent from openhcs.core.steps.function_step import FunctionStep from openhcs.pyqt_gui.widgets.debug_toolbar import DebugToolbarWidget @@ -359,7 +360,7 @@ def debug_editor(tmp_path, monkeypatch, *, run_started_work: bool = False): with caller_session() as session: (scope_id,) = add_datasets(session, tmp_path, "plate") orchestrator = session.orchestrator(scope_id) - orchestrator._state = type(orchestrator.state).READY + orchestrator._state = OrchestratorReady session.set_pipeline( scope_id, [ @@ -781,7 +782,7 @@ def test_session_reuses_the_persistent_paused_worker_across_commands( with caller_session() as session: (scope_id,) = add_datasets(session, tmp_path, "plate") orchestrator = session.orchestrator(scope_id) - orchestrator._state = type(orchestrator.state).READY + orchestrator._state = OrchestratorReady compiled, submitted, worker_commands = [], [], [] async def connect(): @@ -852,7 +853,7 @@ def test_dataset_completion_retires_its_debug_session(tmp_path) -> None: session.execution_state = ManagerExecutionState.RUNNING session.finish_dataset_execution( - TerminalExecutionStatus.FAILED.completion_payload( + ExecutionFailed.completion_payload( execution_id="exec-1", execution_payload={} ), scope_id, @@ -861,4 +862,4 @@ def test_dataset_completion_retires_its_debug_session(tmp_path) -> None: assert scope_id not in session.debug_sessions summary = session.debug_terminal_summaries[scope_id] assert summary.debug_session_id == debug_session.debug_session_id - assert summary.terminal_status == TerminalExecutionStatus.FAILED.value + assert summary.terminal_status == ExecutionFailed.name diff --git a/tests/unit/pyqt_gui/test_pipeline_code_document_transactionality.py b/tests/unit/pyqt_gui/test_pipeline_code_document_transactionality.py index 88e6421267..243eeee4ca 100644 --- a/tests/unit/pyqt_gui/test_pipeline_code_document_transactionality.py +++ b/tests/unit/pyqt_gui/test_pipeline_code_document_transactionality.py @@ -7,7 +7,7 @@ import pytest from openhcs.authoring.session.events import PipelineChanged -from openhcs.constants.constants import OrchestratorState +from openhcs.constants.constants import OrchestratorReady from openhcs.domains.microscopy.axes import Microscopy from tests.unit.pyqt_gui.session_harness import add_datasets, session_gui @@ -18,7 +18,7 @@ def _editor(tmp_path): with session_gui() as gui: (scope_id,) = add_datasets(gui.session, tmp_path, "plate") - gui.session.set_dataset_state(scope_id, OrchestratorState.READY) + gui.session.set_dataset_state(scope_id, OrchestratorReady) gui.settle() changes = [] gui.session.subscribe( diff --git a/tests/unit/pyqt_gui/test_pipeline_editor_widget.py b/tests/unit/pyqt_gui/test_pipeline_editor_widget.py index 1c56192601..eee3ef12f2 100644 --- a/tests/unit/pyqt_gui/test_pipeline_editor_widget.py +++ b/tests/unit/pyqt_gui/test_pipeline_editor_widget.py @@ -37,7 +37,7 @@ PipelineObjectStateBinding, ) from openhcs.authoring.session.step_scopes import StepEditorScope -from openhcs.constants.constants import OrchestratorState +from openhcs.constants.constants import OrchestratorCompiled, OrchestratorReady from openhcs.core.artifact_inspection import CompiledArtifactInspection from openhcs.core.axes import Ungrouped from openhcs.core.config import ( @@ -105,7 +105,7 @@ def editor_gui(tmp_path, *, initialized: bool = True): with session_gui() as gui: (scope_id,) = add_datasets(gui.session, tmp_path, "plate") if initialized: - gui.session.set_dataset_state(scope_id, OrchestratorState.READY) + gui.session.set_dataset_state(scope_id, OrchestratorReady) gui.settle() yield EditorGui(gui, scope_id) @@ -1308,7 +1308,7 @@ def test_debug_toolbar_follows_the_current_dataset_compilation(tmp_path) -> None assert toolbar.command_enabled(StepDebugCommand) is False harness.session.compiled[harness.scope_id] = _compiled(harness.scope_id) - harness.session.set_dataset_state(harness.scope_id, OrchestratorState.COMPILED) + harness.session.set_dataset_state(harness.scope_id, OrchestratorCompiled) harness.gui.settle() assert toolbar.command_enabled(StepDebugCommand) is True diff --git a/tests/unit/pyqt_gui/test_plate_manager_cppipe_import.py b/tests/unit/pyqt_gui/test_plate_manager_cppipe_import.py index ff709f1e91..e450aca55a 100644 --- a/tests/unit/pyqt_gui/test_plate_manager_cppipe_import.py +++ b/tests/unit/pyqt_gui/test_plate_manager_cppipe_import.py @@ -7,6 +7,7 @@ from objectstate.object_state import ObjectStateRegistry from PyQt6.QtCore import Qt +from openhcs.constants.constants import OrchestratorReady from openhcs.agent.dto.session import DatasetRootsRequest from openhcs.authoring.session.datasets import dataset_scope_ids from openhcs.authoring.session.operations.datasets import ( @@ -178,7 +179,7 @@ def activateWindow(self): with session_gui() as gui: gui.session.invoke(AddDatasets, DatasetRootsRequest(roots=(str(root),))) orchestrator = gui.session.orchestrator(scope_id) - orchestrator._state = type(orchestrator.state).READY + orchestrator._state = OrchestratorReady gui.session.refresh() gui.settle() emissions = [] diff --git a/tests/unit/pyqt_gui/test_plate_manager_widget.py b/tests/unit/pyqt_gui/test_plate_manager_widget.py index d9d2cd00f3..e890e9b31c 100644 --- a/tests/unit/pyqt_gui/test_plate_manager_widget.py +++ b/tests/unit/pyqt_gui/test_plate_manager_widget.py @@ -53,7 +53,7 @@ ) from openhcs.authoring.session.pipelines import PipelineObjectStateBinding from openhcs.authoring.session.views import DatasetListView -from openhcs.constants.constants import OrchestratorState +from openhcs.constants.constants import OrchestratorCompleted, OrchestratorCreated, OrchestratorExecFailed, OrchestratorExecuting, OrchestratorInitFailed, OrchestratorReady from openhcs.core.config import ( GlobalPipelineConfig, LazyFijiStreamingConfig, @@ -70,7 +70,7 @@ ExecutionCompletionPayload, ExecutionOutputSummary, ManagerExecutionState, - TerminalExecutionStatus, + ExecutionCancelled, ExecutionCompleted, ExecutionFailed, ) from openhcs.core.input_workspace import InputWorkspacePreparationResult from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator @@ -133,7 +133,7 @@ def post(self, callback) -> None: def _ready(session, *scope_ids: str) -> None: for scope_id in scope_ids: - session.set_dataset_state(scope_id, OrchestratorState.READY) + session.set_dataset_state(scope_id, OrchestratorReady) def _running(session, scope_id: str, execution_id: str) -> None: @@ -347,18 +347,18 @@ def test_initialization_publishes_the_workspace_config_while_another_dataset_run def complete_initialization(): assert scope_id in session.init_pending - orchestrator._state = OrchestratorState.READY + orchestrator._state = OrchestratorReady monkeypatch.setattr(orchestrator, "initialize", complete_initialization) if already_initialized: orchestrator.bind_input_workspace(workspace) - orchestrator._state = OrchestratorState.READY + orchestrator._state = OrchestratorReady _running(session, running, "owned") asyncio.run(session.initialize_datasets((scope_id,))) assert not session.init_pending - assert orchestrator.state is OrchestratorState.READY + assert orchestrator.state is OrchestratorReady assert ( ObjectStateRegistry.get_by_scope(scope_id).get_saved_resolved_value( "num_workers" @@ -389,7 +389,7 @@ def test_initialization_imports_steps_before_announcing_the_selection( monkeypatch.setattr( orchestrator, "initialize", - lambda: setattr(orchestrator, "_state", OrchestratorState.READY), + lambda: setattr(orchestrator, "_state", OrchestratorReady), ) observed = [] gui.plate_manager.plate_selected.connect( @@ -532,7 +532,7 @@ def test_workspace_import_never_writes_into_the_displayed_dataset(tmp_path) -> N assert _names(gui.pipeline_editor.displayed_steps) == ["StartOnly"] -@pytest.mark.parametrize("state", (OrchestratorState.READY, OrchestratorState.EXECUTING)) +@pytest.mark.parametrize("state", (OrchestratorReady, OrchestratorExecuting)) def test_source_binding_context_projects_the_current_orchestrator_config( tmp_path, state ) -> None: @@ -580,7 +580,7 @@ def test_source_binding_context_projects_the_current_orchestrator_config( after = orchestrator.source_binding_context(logical_scope) assert tuple(b.alias for b in after.source_bindings.bindings) == ("After",) assert tuple(b.alias for b in before.source_bindings.bindings) == ("Before",) - assert orchestrator.state is OrchestratorState.CREATED + assert orchestrator.state is OrchestratorCreated def test_compile_needs_an_initialized_dataset(tmp_path) -> None: @@ -603,7 +603,7 @@ def test_live_results_show_the_source_orchestrator_for_source_and_output_rows( source, output = add_datasets( gui.session, tmp_path, "source_plate", "source_plate_openhcs" ) - gui.session.set_dataset_state(source, OrchestratorState.COMPLETED) + gui.session.set_dataset_state(source, OrchestratorCompleted) source_orchestrator = gui.session.orchestrator(source) assert gui.plate_manager.results_orchestrator(source) is source_orchestrator @@ -726,7 +726,7 @@ def test_restart_restores_a_missing_dataset_without_initializing_it(tmp_path) -> orchestrator = gui.session.orchestrator(scope_id) assert isinstance(outcome, DesktopRestartSucceeded) - assert orchestrator.state is OrchestratorState.CREATED + assert orchestrator.state is OrchestratorCreated assert not orchestrator.is_initialized() restored = gui.plate_manager.dataset_document(AllItems) assert restored.payload.plate_paths == (scope_id,) @@ -754,7 +754,7 @@ def test_a_missing_dataset_declaration_stays_editable_until_initialization( apply_dataset_document(session, payload, AllDatasetDocumentScope()) orchestrator = session.orchestrator(scope_id) - assert orchestrator.state is OrchestratorState.CREATED + assert orchestrator.state is OrchestratorCreated assert not orchestrator.is_initialized() rendered = render_dataset_document( session, (scope_id,), selection_mode=AllItems @@ -762,7 +762,7 @@ def test_a_missing_dataset_declaration_stays_editable_until_initialization( assert rendered.payload.plate_paths == (scope_id,) with pytest.raises(FileNotFoundError, match="unavailable-plate"): orchestrator.initialize() - assert orchestrator.state is OrchestratorState.INIT_FAILED + assert orchestrator.state is OrchestratorInitFailed def test_drag_reorder_persists_the_dataset_order(tmp_path) -> None: @@ -824,7 +824,7 @@ def test_progress_from_a_worker_thread_reaches_the_live_row(tmp_path) -> None: (scope_id,) = add_datasets(gui.session, tmp_path, "plate") orchestrator = gui.session.orchestrator(scope_id) orchestrator._initialized = True - orchestrator._state = OrchestratorState.EXECUTING + orchestrator._state = OrchestratorExecuting gui.session.batch.begin_batch((scope_id,)) gui.session.batch.record_execution(scope_id, "execution-1") gui.settle() @@ -1181,7 +1181,7 @@ def test_a_pipeline_change_clears_stale_compilation_and_terminal_tracking( with caller_session() as session: (scope_id,) = add_datasets(session, tmp_path, "plate") - session.orchestrator(scope_id)._state = OrchestratorState.COMPLETED + session.orchestrator(scope_id)._state = OrchestratorCompleted session.set_compiled( scope_id, CompiledDataset( @@ -1194,7 +1194,7 @@ def test_a_pipeline_change_clears_stale_compilation_and_terminal_tracking( ) session.batch.begin_batch((scope_id,)) session.batch.record_execution(scope_id, "execution-1") - session.batch.mark_terminal(scope_id, TerminalExecutionStatus.COMPLETE) + session.batch.mark_terminal(scope_id, ExecutionCompleted) apply_dataset_document( session, @@ -1212,15 +1212,15 @@ def test_a_pipeline_change_clears_stale_compilation_and_terminal_tracking( assert scope_id not in session.compiled assert session.batch.execution_id(scope_id) is None assert session.batch.terminal_status(scope_id) is None - assert session.orchestrator(scope_id).state is OrchestratorState.READY + assert session.orchestrator(scope_id).state is OrchestratorReady assert _names(session.pipeline_steps(scope_id)) == ["Replacement"] def _failed(execution_id: str, detail: str) -> ExecutionCompletionPayload: - return TerminalExecutionStatus.FAILED.completion_payload( + return ExecutionFailed.completion_payload( execution_id=execution_id, execution_payload={ - "status": TerminalExecutionStatus.FAILED.value, + "status": ExecutionFailed.name, "error": detail, }, ) @@ -1232,7 +1232,7 @@ def _executing_dataset(gui, tmp_path) -> str: session.set_pipeline(scope_id, [FunctionStep(func=percentile_normalize, name="Failing")]) orchestrator = session.orchestrator(scope_id) orchestrator._initialized = True - orchestrator._state = OrchestratorState.EXECUTING + orchestrator._state = OrchestratorExecuting session.compiled[scope_id] = object() _running(session, scope_id, "execution-1") gui.settle() @@ -1260,7 +1260,7 @@ def test_dataset_document_edits_during_a_run_and_recovers_after_its_failure( session.finish_dataset_execution(_failed("execution-1", "expected"), scope_id) gui.settle() assert session.execution_state is ManagerExecutionState.IDLE - assert session.orchestrator(scope_id).state is OrchestratorState.EXEC_FAILED + assert session.orchestrator(scope_id).state is OrchestratorExecFailed assert session.batch.execution_id(scope_id) is None result = _apply(bridge, replacement) @@ -1273,7 +1273,7 @@ def test_dataset_document_edits_during_a_run_and_recovers_after_its_failure( (row,) = state.payload["rows"] assert result.applied - assert row["orchestrator_state"] == OrchestratorState.READY.value + assert row["orchestrator_state"] == OrchestratorReady.name assert row["compiled"] is False assert row["execution_active"] is False assert row["execution_id"] is None @@ -1344,10 +1344,10 @@ def apply(source): time.sleep(0.01) assert presented == [(ManagerExecutionState.IDLE, None)] - assert session.orchestrator(scope_id).state is OrchestratorState.EXEC_FAILED + assert session.orchestrator(scope_id).state is OrchestratorExecFailed assert apply(replacement).applied assert _names(session.pipeline_steps(scope_id)) == ["Replacement"] - assert session.orchestrator(scope_id).state is OrchestratorState.READY + assert session.orchestrator(scope_id).state is OrchestratorReady assert scope_id not in session.compiled finally: WindowManager.unregister(window_id, editor) @@ -1367,7 +1367,7 @@ def test_stop_completion_resets_the_force_kill_state(tmp_path, monkeypatch) -> N assert session.execution_state is ManagerExecutionState.FORCE_KILL_READY session.finish_dataset_execution( - TerminalExecutionStatus.CANCELLED.completion_payload( + ExecutionCancelled.completion_payload( execution_id="execution-1", execution_payload={} ), scope_id, @@ -1423,7 +1423,7 @@ def test_a_produced_output_dataset_selects_prepared_replay_once( (source,) = add_datasets(session, tmp_path, "source") output_root = str(tmp_path / "produced") completion = ExecutionCompletionPayload( - status=TerminalExecutionStatus.COMPLETE, + status=ExecutionCompleted, execution_id="produced", results={}, output_dataset=ExecutionOutputSummary( @@ -1447,7 +1447,7 @@ def test_a_produced_output_dataset_selects_prepared_replay_once( _running(session, source, "produced-again") session.finish_dataset_execution( ExecutionCompletionPayload( - status=TerminalExecutionStatus.COMPLETE, + status=ExecutionCompleted, execution_id="produced-again", results={}, output_dataset=completion.output_dataset, diff --git a/tests/unit/pyqt_gui/test_plate_status_presenter.py b/tests/unit/pyqt_gui/test_plate_status_presenter.py index efd3ea9e1d..603449ca19 100644 --- a/tests/unit/pyqt_gui/test_plate_status_presenter.py +++ b/tests/unit/pyqt_gui/test_plate_status_presenter.py @@ -1,4 +1,4 @@ -from openhcs.core.orchestrator.orchestrator import OrchestratorState +from openhcs.constants.constants import OrchestratorCompiled, OrchestratorExecFailed from openhcs.core.progress.projection import ( SubjectRuntimeIdentity, SubjectRuntimeProjection, @@ -7,7 +7,7 @@ CompilingSubjectRuntimeState, ) from openhcs.core.execution_state import ( - TerminalExecutionStatus, + ExecutionFailed, ) from openhcs.authoring.session.dataset_status import ( PlateStatusPresenter, @@ -31,11 +31,11 @@ def _runtime_plate( def test_runtime_projection_is_canonical_over_local_flags(): prefix = PlateStatusPresenter.build_status_prefix( - orchestrator_state=OrchestratorState.EXEC_FAILED, + orchestrator_state=OrchestratorExecFailed, is_init_pending=True, is_compile_pending=True, is_execution_active=False, - terminal_status=TerminalExecutionStatus.FAILED, + terminal_status=ExecutionFailed, runtime_projection=_runtime_plate(CompilingSubjectRuntimeState, 42.0), ) @@ -73,7 +73,7 @@ def test_pending_status_when_active_without_runtime_or_queue(): def test_orchestrator_fallback_for_idle_case(): prefix = PlateStatusPresenter.build_status_prefix( - orchestrator_state=OrchestratorState.COMPILED, + orchestrator_state=OrchestratorCompiled, is_init_pending=False, is_compile_pending=False, is_execution_active=False, @@ -88,5 +88,5 @@ def test_status_presenter_has_no_parallel_state_label_tables(): assert not hasattr(PlateStatusPresenter, "TERMINAL_LABELS") assert not hasattr(PlateStatusPresenter, "ORCHESTRATOR_LABELS") - assert TerminalExecutionStatus.FAILED.status_prefix == "❌ Exec Failed" - assert OrchestratorState.COMPILED.status_prefix == "✓ Compiled" + assert ExecutionFailed.status_prefix == "❌ Exec Failed" + assert OrchestratorCompiled.status_prefix == "✓ Compiled" diff --git a/tests/unit/pyqt_gui/test_reactor_providers.py b/tests/unit/pyqt_gui/test_reactor_providers.py index e1d3602af0..637578dc61 100644 --- a/tests/unit/pyqt_gui/test_reactor_providers.py +++ b/tests/unit/pyqt_gui/test_reactor_providers.py @@ -2,7 +2,7 @@ from objectstate import ObjectStateRegistry -from openhcs.constants.constants import OrchestratorState +from openhcs.constants.constants import OrchestratorReady from openhcs.core.orchestrator import PipelineOrchestrator from openhcs.pyqt_gui.services.reactor_providers import ( OpenHCSComponentSelectionProvider, @@ -38,7 +38,7 @@ def test_component_provider_resolves_the_public_orchestrator_declaration( assert provider.has_components_available(Microscopy.Channel) is False - orchestrator.state = OrchestratorState.READY + orchestrator.state = OrchestratorReady assert provider.has_components_available(Microscopy.Channel) is True assert provider.get_component_keys(Microscopy.Channel) == ["1", "2"] diff --git a/tests/unit/pyqt_gui/test_session_batch_workflows.py b/tests/unit/pyqt_gui/test_session_batch_workflows.py index 7962026952..ca6888a53c 100644 --- a/tests/unit/pyqt_gui/test_session_batch_workflows.py +++ b/tests/unit/pyqt_gui/test_session_batch_workflows.py @@ -30,8 +30,12 @@ from openhcs.core.component_group_scope import RuntimeExecutionAxisScope from openhcs.core.config import GlobalPipelineConfig, PipelineConfig from openhcs.core.debug import DebugCursor, DebugEvent, DebugProgressEventRequest -from openhcs.core.execution_state import ManagerExecutionState, TerminalExecutionStatus -from openhcs.core.orchestrator.orchestrator import OrchestratorState +from openhcs.core.execution_state import ManagerExecutionState, ExecutionCompleted, ExecutionFailed +from openhcs.constants.constants import ( + OrchestratorCompileFailed, + OrchestratorCompiled, + OrchestratorReady, +) from openhcs.core.progress import ProgressExecutionContext, ProgressEvent from openhcs.core.progress.live_measurements import ( LiveMeasurementProgressPayload, @@ -95,7 +99,7 @@ async def wait(self, client, *, execution_id: str, scope_id: str): def _ready(session, *scope_ids: str) -> None: for scope_id in scope_ids: - session.orchestrator(scope_id)._state = OrchestratorState.READY + session.orchestrator(scope_id)._state = OrchestratorReady def _running(session, scope_id: str, execution_id: str) -> None: @@ -181,7 +185,7 @@ async def connect(): asyncio.run(_run(session, target, debug)) assert session.execution_state is ManagerExecutionState.IDLE assert not session.batch.active_plates - assert session.batch.terminal_status(target) is TerminalExecutionStatus.FAILED + assert session.batch.terminal_status(target) is ExecutionFailed assert session.compile_pending == {unrelated} assert len(errors) == 1 @@ -219,9 +223,9 @@ def during_compile(scope_id): assert session.execution_state is ManagerExecutionState.RUNNING assert (other in session.compiled) is not fails assert session.orchestrator(other).state is ( - OrchestratorState.COMPILE_FAILED if fails else OrchestratorState.COMPILED + OrchestratorCompileFailed if fails else OrchestratorCompiled ) - session.batch.mark_terminal(running, TerminalExecutionStatus.COMPLETE) + session.batch.mark_terminal(running, ExecutionCompleted) assert session.batch.all_batch_terminal() assert session.batch.terminal_counts() == (1, 0) @@ -272,7 +276,7 @@ def during_compile(_scope_id): @pytest.mark.parametrize( - "terminal", (TerminalExecutionStatus.COMPLETE, TerminalExecutionStatus.FAILED) + "terminal", (ExecutionCompleted, ExecutionFailed) ) @pytest.mark.parametrize("fails", (False, True)) def test_recompile_supersedes_row_activity_without_erasing_the_batch_outcome( @@ -308,7 +312,7 @@ def during_compile(_scope_id): asyncio.run(session.compile_datasets((finished,))) expected = ( - OrchestratorState.COMPILE_FAILED if fails else OrchestratorState.COMPILED + OrchestratorCompileFailed if fails else OrchestratorCompiled ) assert session.orchestrator(finished).state is expected assert activity.orchestrator_state is expected @@ -316,9 +320,9 @@ def during_compile(_scope_id): assert session.batch.terminal_items() == outcome_before assert session.batch.execution_id(active) == "active-A" assert session.batch.active_plates == (active,) - session.batch.mark_terminal(active, TerminalExecutionStatus.COMPLETE) + session.batch.mark_terminal(active, ExecutionCompleted) assert session.batch.terminal_counts() == ( - (2, 0) if terminal is TerminalExecutionStatus.COMPLETE else (1, 1) + (2, 0) if terminal is ExecutionCompleted else (1, 1) ) @@ -672,5 +676,5 @@ def test_a_standard_run_retires_the_previous_debug_summary(tmp_path, monkeypatch asyncio.run(session.run_datasets((scope_id,))) assert scope_id not in session.debug_terminal_summaries - assert session.batch.terminal_status(scope_id) is TerminalExecutionStatus.FAILED + assert session.batch.terminal_status(scope_id) is ExecutionFailed assert session.execution_state is ManagerExecutionState.IDLE diff --git a/tests/unit/pyqt_gui/test_ui_agent_bridge.py b/tests/unit/pyqt_gui/test_ui_agent_bridge.py index 61d93f9a39..7b11a67b15 100644 --- a/tests/unit/pyqt_gui/test_ui_agent_bridge.py +++ b/tests/unit/pyqt_gui/test_ui_agent_bridge.py @@ -120,7 +120,7 @@ ) from openhcs.authoring.session.session import CallerThread, Renderer, Session from openhcs.authoring.session.views import DatasetListView, PipelineStepsView -from openhcs.constants.constants import OrchestratorState +from openhcs.constants.constants import OrchestratorExecuting, OrchestratorReady from openhcs.core.config import ( GlobalPipelineConfig, LazyNapariStreamingConfig, @@ -3666,7 +3666,7 @@ def pipeline_gui(selected_steps: tuple[int, ...] = (0,), *, compiled: bool = Tru with session_gui() as gui: session = gui.session session.ensure_datasets((DATASET_SCOPE_ID,)) - session.set_dataset_state(DATASET_SCOPE_ID, OrchestratorState.READY) + session.set_dataset_state(DATASET_SCOPE_ID, OrchestratorReady) session.set_pipeline( DATASET_SCOPE_ID, [ @@ -4085,7 +4085,7 @@ def test_plate_manager_state_ignores_stale_runtime_without_current_execution_id( def test_plate_manager_state_terminal_status_overrides_stale_executing_state() -> None: with dataset_gui(DATASET_SCOPE_ID) as (gui, _scope_ids): session = gui.session - session.orchestrator(DATASET_SCOPE_ID)._state = OrchestratorState.EXECUTING + session.orchestrator(DATASET_SCOPE_ID)._state = OrchestratorExecuting session.batch.begin_batch((DATASET_SCOPE_ID,)) session.batch.record_execution(DATASET_SCOPE_ID, "failed-execution") session.batch.mark_terminal(DATASET_SCOPE_ID, "failed") diff --git a/tests/unit/test_compiled_execution.py b/tests/unit/test_compiled_execution.py index ed2eda228a..ec2a0a5b73 100644 --- a/tests/unit/test_compiled_execution.py +++ b/tests/unit/test_compiled_execution.py @@ -38,7 +38,7 @@ ExecutionResult, RuntimeContextObservation, RuntimeExecutionObservation, - RuntimeObservationMode, + RuntimeObservationMode, MergeAllRuntimeRecords, MergeDatasetInputRecords, OmitRuntimeRecords, ) from openhcs.core.orchestrator.worker_lanes import WorkerLaneExecutionContext from openhcs.core.runtime_artifact_values import RuntimeValue @@ -118,13 +118,13 @@ def test_compiled_execution_bundle_derives_axis_ids_from_contexts() -> None: @pytest.mark.parametrize( ("execution_scopes", "expected_mode"), ( - ((FunctionStepExecutionScope.AXIS,), RuntimeObservationMode.OMIT), + ((FunctionStepExecutionScope.AXIS,), OmitRuntimeRecords), ( ( FunctionStepExecutionScope.AXIS, FunctionStepExecutionScope.PLATE, ), - RuntimeObservationMode.MERGE_DATASET_INPUTS, + MergeDatasetInputRecords, ), ), ) @@ -203,18 +203,18 @@ def test_consolidation_does_not_require_parent_payload_records( assert bundle.requires_parent_runtime_observation is False assert ( RuntimeObservationMode.for_compiled_bundle(bundle) - is RuntimeObservationMode.OMIT + is OmitRuntimeRecords ) def test_runtime_observation_mode_can_only_be_strengthened() -> None: assert ( - RuntimeObservationMode.OMIT.including_parent_requirement(True) - is RuntimeObservationMode.MERGE_INTO_PARENT + OmitRuntimeRecords.including_parent_requirement(True) + is MergeAllRuntimeRecords ) assert ( - RuntimeObservationMode.MERGE_INTO_PARENT.including_parent_requirement(False) - is RuntimeObservationMode.MERGE_INTO_PARENT + MergeAllRuntimeRecords.including_parent_requirement(False) + is MergeAllRuntimeRecords ) @@ -262,7 +262,7 @@ def execute_axis(_pipeline, context, _lane, **_kwargs): worker_slot="worker", worker_assignments={"worker": [context.axis_id for context in contexts]}, ), - runtime_observation_mode=RuntimeObservationMode.OMIT, + runtime_observation_mode=OmitRuntimeRecords, release_axis_resources=False, ) @@ -354,10 +354,10 @@ def execute_current_run( *, context_key, cancellation=None, - runtime_observation_mode=RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode=MergeAllRuntimeRecords, ): assert context_key == "A01" - assert runtime_observation_mode is RuntimeObservationMode.MERGE_INTO_PARENT + assert runtime_observation_mode is MergeAllRuntimeRecords current_records.append( context.runtime_value_store.replace( measurement_value(2), @@ -395,7 +395,7 @@ def execute_current_run( worker_slot="worker-0", worker_assignments={"worker-0": ["A01"]}, ), - runtime_observation_mode=RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode=MergeAllRuntimeRecords, ) assert result.runtime_observation.contexts == ( @@ -460,14 +460,14 @@ def fail_reset() -> None: worker_slot="worker-0", worker_assignments={"worker-0": ["A01"]}, ), - runtime_observation_mode=RuntimeObservationMode.OMIT, + runtime_observation_mode=OmitRuntimeRecords, ) assert cleaned == [(MemoryType.CUPY, 2)] assert pixel_reference() is None -@pytest.mark.parametrize("observation_mode", tuple(RuntimeObservationMode)) +@pytest.mark.parametrize("observation_mode", tuple(RuntimeObservationMode.__subclasses__())) @pytest.mark.parametrize("release_process_resources", (True, False)) def test_worker_lane_releases_previous_axis_stack_before_next_axis( monkeypatch, @@ -512,7 +512,7 @@ def execute_axis(_pipeline, context, _lane, **_kwargs): @pytest.mark.parametrize( "observation_mode", - (RuntimeObservationMode.OMIT, RuntimeObservationMode.MERGE_DATASET_INPUTS), + (OmitRuntimeRecords, MergeDatasetInputRecords), ) @pytest.mark.parametrize("release_process_resources", (True, False)) def test_worker_lane_releases_unconsumed_image_records_before_next_axis( diff --git a/tests/unit/test_completed_output_publication_lifecycle.py b/tests/unit/test_completed_output_publication_lifecycle.py index dd9746b8d8..7d21d898ff 100644 --- a/tests/unit/test_completed_output_publication_lifecycle.py +++ b/tests/unit/test_completed_output_publication_lifecycle.py @@ -16,7 +16,11 @@ from openhcs.core.debug import NoOpDebugExecutionPolicy from openhcs.core.orchestrator.cancellation import ExecutionCancellationSignal from openhcs.core.orchestrator import worker_execution -from openhcs.core.orchestrator.execution_result import RuntimeObservationMode +from openhcs.core.orchestrator.execution_result import ( + AxisCancelled, + AxisFailed, + OmitRuntimeRecords, +) from openhcs.core.orchestrator.worker_lanes import WorkerLaneExecutionContext from openhcs.core.runtime_image_values import ImagePayloadMetadata from openhcs.core.source_projection import OpenHCSPlaneAddress, SourcePlaneProjection @@ -106,10 +110,10 @@ def process(self, context, index): context = _context(tmp_path, 2) result = worker_execution._execute_axis_with_sequential_combinations( - [Save(), Fail()], [("A01", context)], _lane(), RuntimeObservationMode.OMIT, + [Save(), Fail()], [("A01", context)], _lane(), OmitRuntimeRecords, cancellation=signal, release_axis_resources=False, ) - assert result.is_cancelled() if cancel else result.is_error() + assert result.status is (AxisCancelled if cancel else AxisFailed) assert not context.runtime_value_store.observed_values assert pixel_refs[0]() is None assert len(result.runtime_observation.contexts) == 1 diff --git a/tests/unit/test_deleted_core_names_absent.py b/tests/unit/test_deleted_core_names_absent.py index 5313b87a5c..d724040fc7 100644 --- a/tests/unit/test_deleted_core_names_absent.py +++ b/tests/unit/test_deleted_core_names_absent.py @@ -94,5 +94,5 @@ def test_deleted_core_names_stay_absent_from_openhcs() -> None: def test_deleted_core_members_stay_absent() -> None: assert "step_type" not in {field.name for field in fields(CompiledStepPlan)} - assert "PENDING" not in ExecutionStatus.__members__ + assert "pending" not in ExecutionStatus.__registry__ assert not hasattr(function_contracts, "validate_artifact_input_parameter_bindings") diff --git a/tests/unit/test_function_artifact_materialization.py b/tests/unit/test_function_artifact_materialization.py index f212b6b5bc..d89e94bdf5 100644 --- a/tests/unit/test_function_artifact_materialization.py +++ b/tests/unit/test_function_artifact_materialization.py @@ -65,7 +65,7 @@ from openhcs.core.orchestrator.execution_result import ( RuntimeContextObservation, RuntimeExecutionObservation, - RuntimeObservationMode, + MergeAllRuntimeRecords, OmitRuntimeRecords, ) from openhcs.core.pipeline.function_contracts import artifact_outputs from openhcs.core.pipeline.artifact_planning import ( @@ -1974,7 +1974,7 @@ def test_consolidation_preserves_export_bundle_and_text_tables(options, payload) @pytest.mark.parametrize( "observation_mode", - (RuntimeObservationMode.OMIT, RuntimeObservationMode.MERGE_INTO_PARENT), + (OmitRuntimeRecords, MergeAllRuntimeRecords), ) def test_completed_observation_projects_tables_before_worker_payload_release( observation_mode, monkeypatch, @@ -2036,10 +2036,10 @@ def test_completed_observation_projects_tables_before_worker_payload_release( context.runtime_value_store.clear() del pixels assert (pixel_reference() is not None) is ( - observation_mode is RuntimeObservationMode.MERGE_INTO_PARENT + observation_mode is MergeAllRuntimeRecords ) assert len(observation.records) == ( - 2 if observation_mode is RuntimeObservationMode.MERGE_INTO_PARENT else 0 + 2 if observation_mode is MergeAllRuntimeRecords else 0 ) assert len(observation.outputs.runtime_export_paths) == 1 assert observation.outputs.runtime_export_paths[0].suffix == ".csv" diff --git a/tests/unit/test_function_step_execution_scope.py b/tests/unit/test_function_step_execution_scope.py index 9799262e90..1e8e1cc7b5 100644 --- a/tests/unit/test_function_step_execution_scope.py +++ b/tests/unit/test_function_step_execution_scope.py @@ -46,7 +46,7 @@ execute_dataset_scoped_steps, validate_dataset_scoped_contexts, ) -from openhcs.core.orchestrator.execution_result import RuntimeObservationMode +from openhcs.core.orchestrator.execution_result import MergeAllRuntimeRecords, MergeDatasetInputRecords, OmitRuntimeRecords from openhcs.core.progress import ProgressEvent, ProgressExecutionContext from openhcs.core.pipeline.function_contracts import ( artifact_inputs, @@ -684,7 +684,7 @@ def export(*, artifact_batch: RuntimeArtifactBatch): count=3, ) observations = context.runtime_value_store.observed_values - retained = RuntimeObservationMode.MERGE_DATASET_INPUTS.retain_records( + retained = MergeDatasetInputRecords.retain_records( observations, context, ) @@ -692,13 +692,13 @@ def export(*, artifact_batch: RuntimeArtifactBatch): (["raw"] if stored_image else []) + ["Measurements", "Measurements"] ) assert ( - RuntimeObservationMode.MERGE_INTO_PARENT.retain_records( + MergeAllRuntimeRecords.retain_records( observations, context, ) is observations ) - assert RuntimeObservationMode.OMIT.retain_records(observations, context) == () + assert OmitRuntimeRecords.retain_records(observations, context) == () full_store = type(context.runtime_value_store)() full_store.merge_observed_values(observations) @@ -752,7 +752,7 @@ def export(*, artifact_batch: RuntimeArtifactBatch): metadata_writer=True, ) context = _plate_context("A01", (plan,)) - assert RuntimeObservationMode.MERGE_DATASET_INPUTS.retain_records((), context) == () + assert MergeDatasetInputRecords.retain_records((), context) == () with pytest.raises(ValueError, match="missing required input"): _dataset_artifact_batch( compiled_contexts={"A01": context}, diff --git a/tests/unit/test_materialization_batch_outcomes.py b/tests/unit/test_materialization_batch_outcomes.py index 16961b7e02..219cef46be 100644 --- a/tests/unit/test_materialization_batch_outcomes.py +++ b/tests/unit/test_materialization_batch_outcomes.py @@ -159,7 +159,7 @@ def release_execution_image_cache(self): self.released = True -@pytest.mark.parametrize("mode", tuple(RuntimeObservationMode)) +@pytest.mark.parametrize("mode", tuple(RuntimeObservationMode.__subclasses__())) def test_worker_exports_actual_step_outcomes_after_resource_release( tmp_path, monkeypatch, mode ): diff --git a/tests/unit/test_nonpersistent_viewer_state_evidence.py b/tests/unit/test_nonpersistent_viewer_state_evidence.py index 55f042a152..0580422605 100644 --- a/tests/unit/test_nonpersistent_viewer_state_evidence.py +++ b/tests/unit/test_nonpersistent_viewer_state_evidence.py @@ -15,7 +15,7 @@ ) from openhcs.core.orchestrator.execution_result import ( ExecutionResult, - RuntimeObservationMode, + OmitRuntimeRecords, ) from openhcs.runtime.viewer_protocol import ViewerControlResponse from openhcs.runtime.zmq_server_hooks import ZMQResultsSummaryEnricher @@ -90,7 +90,7 @@ def execute_compiled_bundle(self, **kwargs): execution_id="exec", orchestrator=Orchestrator(), execution_bundle=execution_bundle, - runtime_observation_mode=RuntimeObservationMode.OMIT, + runtime_observation_mode=OmitRuntimeRecords, progress_context=ProgressExecutionContext( execution_id="exec", subject_id="plate", @@ -101,7 +101,7 @@ def execute_compiled_bundle(self, **kwargs): ).execute() assert returned is compiled_results - assert execution_calls[0]["runtime_observation_mode"] is RuntimeObservationMode.OMIT + assert execution_calls[0]["runtime_observation_mode"] is OmitRuntimeRecords assert progress_queue.values == [None] assert record.get_extra(BundleExecutionExtras.EXECUTION_RECORD_KEY) is ( compiled_results.extras diff --git a/tests/unit/test_openhcs_measured_run.py b/tests/unit/test_openhcs_measured_run.py index b5f47b386c..c23e6275ad 100644 --- a/tests/unit/test_openhcs_measured_run.py +++ b/tests/unit/test_openhcs_measured_run.py @@ -27,7 +27,7 @@ from openhcs.runtime.zmq_execution_observation import ZMQRuntimeExecutionOutcomeExport from openhcs.runtime.zmq_execution_signature import ( ZMQAuxiliaryExecutionParams, - ZMQRuntimeObservationExportScope, + ExportRuntimeOutcomes, ) @@ -480,7 +480,7 @@ def test_outcome_only_run_uses_the_shared_receipt_finalizer(tmp_path: Path) -> N ).with_auxiliary_params( ZMQAuxiliaryExecutionParams( runtime_observation_export_path=observation_path, - runtime_observation_export_scope=ZMQRuntimeObservationExportScope.OUTCOMES, + runtime_observation_export_scope=ExportRuntimeOutcomes, ) ) endpoint = measured_run.measured_endpoint_provenance( @@ -564,7 +564,7 @@ def test_outcome_only_run_uses_the_shared_receipt_finalizer(tmp_path: Path) -> N assert completion.receipt.observed_axis_count == 1 assert ( completion.receipt.observation_export_scope - is ZMQRuntimeObservationExportScope.OUTCOMES + is ExportRuntimeOutcomes ) assert ( MeasuredPipelineRunReceipt.read( diff --git a/tests/unit/test_orchestrator_lane_planning.py b/tests/unit/test_orchestrator_lane_planning.py index e4b2d24ec4..be0490891d 100644 --- a/tests/unit/test_orchestrator_lane_planning.py +++ b/tests/unit/test_orchestrator_lane_planning.py @@ -19,6 +19,7 @@ bundle_execution as compiled_plate_execution_module, ) from openhcs.core.orchestrator import orchestrator as orchestrator_module +from openhcs.constants.constants import OrchestratorCompleted, OrchestratorExecFailed from openhcs.core.orchestrator import worker_execution as worker_execution_module from openhcs.domains.microscopy import ( analysis_consolidation as analysis_consolidation_module, @@ -47,7 +48,7 @@ from openhcs.core.orchestrator.execution_result import ( ExecutionResult, RuntimeExecutionObservation, - RuntimeObservationMode, + MergeAllRuntimeRecords, OmitRuntimeRecords, ) from openhcs.core.orchestrator.worker_execution import ( ForkInheritedWorkerExecutorResources, @@ -120,7 +121,7 @@ def test_fork_resources_reuse_lane_processes_for_ordered_plate_partitions(monkey worker_assignments={"worker_0": ["A01"], "worker_1": ["A02"]}, lane_axis_contexts={"worker_0": [("A01", [])], "worker_1": [("A02", [])]}, ), - runtime_observation_mode=RuntimeObservationMode.OMIT, + runtime_observation_mode=OmitRuntimeRecords, ) processes = [] try: @@ -177,7 +178,7 @@ def test_fork_resources_join_all_lanes_after_plate_failure(monkeypatch, failure) worker_assignments={"worker_0": ["A01"], "worker_1": ["A02"]}, lane_axis_contexts={"worker_0": [("A01", [])], "worker_1": [("A02", [])]}, ), - runtime_observation_mode=RuntimeObservationMode.OMIT, + runtime_observation_mode=OmitRuntimeRecords, ) processes = [] try: @@ -250,7 +251,7 @@ def _execute_with_visualizer(monkeypatch, visualizer, *, progress_queue=None): visualizer=None, log_file_base=None, progress_queue=progress_queue or SimpleNamespace(put=lambda _event: None), - runtime_observation_mode=RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode=MergeAllRuntimeRecords, debug_execution_policy=NoOpDebugExecutionPolicy(), ) @@ -387,7 +388,7 @@ def test_compiled_plate_execution_request_uses_bundle_as_runtime_authority(): visualizer=None, log_file_base=None, progress_queue="queue", - runtime_observation_mode=RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode=MergeAllRuntimeRecords, debug_execution_policy=NoOpDebugExecutionPolicy(), ) orchestrator = SimpleNamespace( @@ -424,7 +425,7 @@ def test_plate_scope_rejects_omitted_runtime_observations(monkeypatch): visualizer=None, log_file_base=None, progress_queue="queue", - runtime_observation_mode=RuntimeObservationMode.OMIT, + runtime_observation_mode=OmitRuntimeRecords, debug_execution_policy=NoOpDebugExecutionPolicy(), ) monkeypatch.setattr( @@ -433,7 +434,7 @@ def test_plate_scope_rejects_omitted_runtime_observations(monkeypatch): lambda _contexts: (1,), ) - with pytest.raises(ValueError, match="MERGE_INTO_PARENT"): + with pytest.raises(ValueError, match="runtime observation mode that returns records"): validate_bundle_execution( SimpleNamespace(is_initialized=lambda: True), request, @@ -767,7 +768,7 @@ def held_lane(**_kwargs): writer, (reader,), worker_execution_module.ExecuteInheritedLane( - [], lane, RuntimeObservationMode.OMIT + [], lane, OmitRuntimeRecords ), ), ) @@ -837,7 +838,7 @@ def fake_execute_worker_lane( worker_assignments={"worker_0": ["A01"]}, lane_axis_contexts={"worker_0": [("A01", [("A01", object())])]}, ), - runtime_observation_mode=RuntimeObservationMode.OMIT, + runtime_observation_mode=OmitRuntimeRecords, ) results = PooledWorkerLaneRunner(FakeExecutor(), cancellation=None).run( @@ -888,7 +889,7 @@ def execute_axis( pipeline_definition=[], lane_axis_contexts=lane_axis_contexts, lane_context=lane_context, - runtime_observation_mode=RuntimeObservationMode.OMIT, + runtime_observation_mode=OmitRuntimeRecords, cancellation=cancellation, ) @@ -962,7 +963,7 @@ def fake_execute_worker_lane( worker_assignments={"worker_0": ["A01"]}, lane_axis_contexts={"worker_0": [("A01", [("A01", object())])]}, ), - runtime_observation_mode=RuntimeObservationMode.OMIT, + runtime_observation_mode=OmitRuntimeRecords, ) PooledWorkerLaneRunner(FakeExecutor(), cancellation=None).run( @@ -1030,7 +1031,7 @@ def fake_execute_worker_lane( worker_assignments={"worker_0": ["A01"]}, lane_axis_contexts={"worker_0": [("A01", ["A01"])]}, ), - runtime_observation_mode=RuntimeObservationMode.MERGE_INTO_PARENT, + runtime_observation_mode=MergeAllRuntimeRecords, ) results = ForkInheritedWorkerLaneRunner(ForbiddenMultiprocessingContext()).run( @@ -1071,7 +1072,7 @@ def submit(self, fn, *args): worker_assignments={"worker_0": ["A01"]}, lane_axis_contexts={"worker_0": [("A01", [("A01", object())])]}, ), - runtime_observation_mode=RuntimeObservationMode.OMIT, + runtime_observation_mode=OmitRuntimeRecords, ) with pytest.raises(RuntimeError, match="lane exploded"): @@ -1139,13 +1140,13 @@ def test_execution_state_projector_maps_success_and_failure(): orchestrator, BundleExecutionResults({"A01": ExecutionResult.success("A01")}), ) - assert orchestrator._state is orchestrator_module.OrchestratorState.COMPLETED + assert orchestrator._state is OrchestratorCompleted project_execution_state( orchestrator, BundleExecutionResults({"A01": ExecutionResult.error("A01")}), ) - assert orchestrator._state is orchestrator_module.OrchestratorState.EXEC_FAILED + assert orchestrator._state is OrchestratorExecFailed @pytest.mark.parametrize( diff --git a/tests/unit/test_plane_store_sources.py b/tests/unit/test_plane_store_sources.py index 3335f872c3..085eb6baa7 100644 --- a/tests/unit/test_plane_store_sources.py +++ b/tests/unit/test_plane_store_sources.py @@ -21,7 +21,7 @@ ZarrBatchLayout, ) -from openhcs.constants.constants import Backend, OrchestratorState +from openhcs.constants.constants import Backend, OrchestratorCreated from openhcs.core.config import ( GlobalPipelineConfig, LazySourceBindingsConfig, @@ -406,7 +406,7 @@ def reject_editor(*args, **kwargs): else: orchestrator.apply_pipeline_config(replacement) - assert orchestrator.state is OrchestratorState.CREATED + assert orchestrator.state is OrchestratorCreated assert not orchestrator.is_initialized() assert orchestrator.microscope_handler is None assert orchestrator.get_effective_config().source_bindings_config == edited_bindings diff --git a/tests/unit/test_well_throughput_scaling.py b/tests/unit/test_well_throughput_scaling.py index 644cf7b1c8..8d784fece1 100644 --- a/tests/unit/test_well_throughput_scaling.py +++ b/tests/unit/test_well_throughput_scaling.py @@ -60,7 +60,7 @@ ) from openhcs.core.dataset_sources.source_schema import SourceSchemaFilenameParser from openhcs.runtime.zmq_execution_observation import ZMQRuntimeExecutionOutcomeExport -from openhcs.runtime.zmq_execution_signature import ZMQRuntimeObservationExportScope +from openhcs.runtime.zmq_execution_signature import ExportRuntimeOutcomes from openhcs.domains.microscopy.axes import Microscopy @@ -1456,7 +1456,7 @@ def fake_execute( assert pipeline_config.materialize_runtime_artifacts is False assert submissions[0].global_pipeline_config.materialize_runtime_artifacts is False assert submissions[0].config_params["runtime_observation_export_scope"] == ( - ZMQRuntimeObservationExportScope.OUTCOMES.value + ExportRuntimeOutcomes.name ) diff --git a/tests/unit/test_zmq_compilation_identity.py b/tests/unit/test_zmq_compilation_identity.py index 4790af0cca..f8a668f924 100644 --- a/tests/unit/test_zmq_compilation_identity.py +++ b/tests/unit/test_zmq_compilation_identity.py @@ -29,7 +29,7 @@ def _payload(config_params=None): ) -@pytest.mark.parametrize("scope", tuple(ZMQRuntimeObservationExportScope)) +@pytest.mark.parametrize("scope", tuple(ZMQRuntimeObservationExportScope.__registry__.values())) @pytest.mark.parametrize("path", ("/evidence/first.pkl.gz", "/evidence/second.pkl.gz")) def test_observation_options_change_request_but_not_compilation_identity(scope, path): options = ZMQAuxiliaryExecutionParams( diff --git a/tests/unit/test_zmq_execution_server.py b/tests/unit/test_zmq_execution_server.py index 2f58dbb20d..762dc552f9 100644 --- a/tests/unit/test_zmq_execution_server.py +++ b/tests/unit/test_zmq_execution_server.py @@ -30,7 +30,7 @@ ExecutionResult, RuntimeContextObservation, RuntimeExecutionObservation, - RuntimeObservationMode, + MergeAllRuntimeRecords, MergeDatasetInputRecords, OmitRuntimeRecords, ) from openhcs.core.progress import ProgressEvent, ProgressExecutionContext from openhcs.runtime.zmq_execution_observation import ( @@ -47,7 +47,7 @@ ZMQExecutionConfigTransport, ZMQExecutionIdentity, ZMQExecutionRequestPayload, - ZMQRuntimeObservationExportScope, + ExportRuntimeOutcomes, ) from openhcs.domains.microscopy.axes import Microscopy from openhcs.core.progress.types import ( @@ -131,7 +131,7 @@ def test_axis_outcomes_qualify_server_terminal_after_export(monkeypatch, failed) compile_only=False, execution_id=record.execution_id, auxiliary_params=SimpleNamespace( - runtime_observation_mode_for=lambda bundle: RuntimeObservationMode.OMIT + runtime_observation_mode_for=lambda bundle: OmitRuntimeRecords ), ) monkeypatch.setattr( @@ -222,26 +222,26 @@ def test_terminal_notification_contains_openhcs_summary_before_cleanup( @pytest.mark.parametrize( ("compiled_mode", "export_path", "expected_mode"), ( - (RuntimeObservationMode.OMIT, None, RuntimeObservationMode.OMIT), + (OmitRuntimeRecords, None, OmitRuntimeRecords), ( - RuntimeObservationMode.OMIT, + OmitRuntimeRecords, "/tmp/runtime-observation.pkl", - RuntimeObservationMode.MERGE_INTO_PARENT, + MergeAllRuntimeRecords, ), ( - RuntimeObservationMode.MERGE_DATASET_INPUTS, + MergeDatasetInputRecords, None, - RuntimeObservationMode.MERGE_DATASET_INPUTS, + MergeDatasetInputRecords, ), ( - RuntimeObservationMode.MERGE_DATASET_INPUTS, + MergeDatasetInputRecords, "/tmp/runtime-observation.pkl", - RuntimeObservationMode.MERGE_INTO_PARENT, + MergeAllRuntimeRecords, ), ( - RuntimeObservationMode.MERGE_INTO_PARENT, + MergeAllRuntimeRecords, None, - RuntimeObservationMode.MERGE_DATASET_INPUTS, + MergeDatasetInputRecords, ), ), ) @@ -265,7 +265,7 @@ def test_zmq_auxiliary_params_strengthen_compiled_observation_requirement( def test_outcome_export_does_not_strengthen_worker_runtime_value_retention() -> None: params = ZMQAuxiliaryExecutionParams( runtime_observation_export_path=Path("/tmp/outcomes.pkl.gz"), - runtime_observation_export_scope=ZMQRuntimeObservationExportScope.OUTCOMES, + runtime_observation_export_scope=ExportRuntimeOutcomes, ) execution_bundle = SimpleNamespace( requires_parent_runtime_observation=False, @@ -273,7 +273,7 @@ def test_outcome_export_does_not_strengthen_worker_runtime_value_retention() -> assert ( params.runtime_observation_mode_for(execution_bundle) - is RuntimeObservationMode.OMIT + is OmitRuntimeRecords ) assert ZMQAuxiliaryExecutionParams.from_transport(params.to_transport()) == params @@ -281,7 +281,7 @@ def test_outcome_export_does_not_strengthen_worker_runtime_value_retention() -> def test_outcome_export_requires_a_path() -> None: with pytest.raises(ValueError, match="requires an export path"): ZMQAuxiliaryExecutionParams( - runtime_observation_export_scope=ZMQRuntimeObservationExportScope.OUTCOMES + runtime_observation_export_scope=ExportRuntimeOutcomes ) @@ -302,7 +302,7 @@ def test_server_exports_outcomes_without_projecting_compiled_values( execution_id=record.execution_id, auxiliary_params=ZMQAuxiliaryExecutionParams( runtime_observation_export_path=export_path, - runtime_observation_export_scope=ZMQRuntimeObservationExportScope.OUTCOMES, + runtime_observation_export_scope=ExportRuntimeOutcomes, ), ) compilation = SimpleNamespace( diff --git a/tests/unit/test_zmq_runtime_outcome_export.py b/tests/unit/test_zmq_runtime_outcome_export.py index 3f624d105e..07a6bd37f5 100644 --- a/tests/unit/test_zmq_runtime_outcome_export.py +++ b/tests/unit/test_zmq_runtime_outcome_export.py @@ -6,7 +6,7 @@ from openhcs.core.orchestrator.execution_result import ( ExecutionResult, - ExecutionStatus, + AxisFailed, RuntimeContextObservation, RuntimeExecutionObservation, ) @@ -61,7 +61,7 @@ def test_outcome_export_round_trip_excludes_runtime_values(tmp_path: Path) -> No assert restored.execution_id == "execution-1" assert restored.axis_count == 2 assert restored.successful_axis_count == 1 - assert restored.outcomes_by_axis["B01"].status is ExecutionStatus.ERROR + assert restored.outcomes_by_axis["B01"].status is AxisFailed assert restored.outcomes_by_axis["B01"].failed_combination == "field-2" assert restored.exports is not None assert restored.exports.output_files == (declared_output,) From 66593f69b4dd0d098cb24ebf34963f71915b4e25 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:01:00 -0400 Subject: [PATCH 23/36] E1: function-catalog control requests carry their wire_type; responses use one result key FunctionCatalogControlMessageType, FunctionCatalogControlField and FunctionCatalogPreparationStatus are deleted: each request declares its wire type (the execution server keys its strategies by it), every response carries its value under one result key, and the pending reply declares its status. Co-Authored-By: Claude Opus 5.5 --- openhcs/agent/dto/functions.py | 106 ++++++------------ openhcs/runtime/zmq_control.py | 20 ++-- .../check_custom_registration_live.py | 15 ++- .../unit/test_catalog_endpoint_incarnation.py | 7 +- tests/unit/test_function_catalog_zmq.py | 4 +- 5 files changed, 55 insertions(+), 97 deletions(-) diff --git a/openhcs/agent/dto/functions.py b/openhcs/agent/dto/functions.py index 7f5fc6bb76..01f3b11acd 100644 --- a/openhcs/agent/dto/functions.py +++ b/openhcs/agent/dto/functions.py @@ -32,32 +32,6 @@ DEFAULT_FUNCTION_DETAIL_DOC_CHARS = 6_000 -class FunctionCatalogControlMessageType(str, Enum): - """Execution-server control messages for the live function catalog.""" - - READ_CATALOG = "openhcs_function_catalog_read" - SEARCH_CATALOG = "openhcs_function_catalog_search" - READ_DETAIL = "openhcs_function_detail_read" - READ_REFERENCE = "openhcs_function_reference_read" - REGISTER_CUSTOM = "openhcs_custom_function_register" - OBSERVE_CUSTOM_REGISTRATION = "openhcs_custom_function_registration_observe" - CUSTOM_REGISTRATION_DESTINATION = "openhcs_custom_function_registration_destination" - START_PREPARATION = "openhcs_function_catalog_prepare" - READ_PREPARATION = "openhcs_function_catalog_preparation_status" - CANCEL_PREPARATION = "openhcs_function_catalog_preparation_cancel" - - -class FunctionCatalogControlField(str, Enum): - """OpenHCS-specific fields carried by function-catalog control messages.""" - - REQUEST = "request" - CATALOG = "catalog" - PREPARATION = "preparation" - DETAIL = "detail" - REFERENCE = "reference" - RESULT = "result" - - @dataclass(frozen=True) class FunctionIdentity: function_id: str @@ -72,27 +46,26 @@ class ImportPathRef: class FunctionCatalogControlRequestABC(ABC): """Nominal owner of one function-catalog control request identity.""" - message_type: ClassVar[FunctionCatalogControlMessageType] + wire_type: ClassVar[str] + """This request's control message type on the execution-server wire.""" def __init_subclass__(cls) -> None: - if "message_type" not in cls.__dict__: + if "wire_type" not in cls.__dict__: raise TypeError( - f"{cls.__name__} must declare its function-catalog message_type." + f"{cls.__name__} must declare its function-catalog wire_type." ) @classmethod def from_control_payload(cls, payload: Mapping[str, Any]) -> Self: """Recover this exact request type from its declared wire message.""" - actual_message_type = FunctionCatalogControlMessageType( - str(payload[MessageFields.TYPE]) - ) - if actual_message_type is not cls.message_type: + actual_wire_type = str(payload[MessageFields.TYPE]) + if actual_wire_type != cls.wire_type: raise ValueError( - f"Expected function catalog control type {cls.message_type.value!r}, " - f"got {actual_message_type.value!r}." + f"Expected function catalog control type {cls.wire_type!r}, " + f"got {actual_wire_type!r}." ) - request = payload[FunctionCatalogControlField.REQUEST.value] + request = payload[FunctionCatalogControlPayload.request_key] if not isinstance(request, cls): raise TypeError( f"{cls.__name__} control payload requires {cls.__name__}, got " @@ -108,7 +81,7 @@ class FunctionSearchRequest(FunctionCatalogControlRequestABC): limit: int = 50 compact_signatures: bool = True - message_type = FunctionCatalogControlMessageType.SEARCH_CATALOG + wire_type = "openhcs_function_catalog_search" @dataclass(frozen=True, slots=True) @@ -237,19 +210,19 @@ def __init__(self, state: FunctionCatalogPreparationState) -> None: @dataclass(frozen=True, slots=True) class FunctionCatalogPreparationStartRequest(FunctionCatalogControlRequestABC): connection: ExecutionConnectionSpec - message_type = FunctionCatalogControlMessageType.START_PREPARATION + wire_type = "openhcs_function_catalog_prepare" @dataclass(frozen=True, slots=True) class FunctionCatalogPreparationStatusRequest(FunctionCatalogControlRequestABC): handle: FunctionCatalogPreparationHandle - message_type = FunctionCatalogControlMessageType.READ_PREPARATION + wire_type = "openhcs_function_catalog_preparation_status" @dataclass(frozen=True, slots=True) class FunctionCatalogPreparationCancelRequest(FunctionCatalogControlRequestABC): handle: FunctionCatalogPreparationHandle - message_type = FunctionCatalogControlMessageType.CANCEL_PREPARATION + wire_type = "openhcs_function_catalog_preparation_cancel" @dataclass(frozen=True, slots=True) @@ -278,7 +251,7 @@ class CustomFunctionRegistrationRequest(FunctionCatalogControlRequestABC): admission_policy: AgentPathPolicy | None = field(default=None, repr=False, compare=False) server_identity: ProcessIdentity | None = None - message_type = FunctionCatalogControlMessageType.REGISTER_CUSTOM + wire_type = "openhcs_custom_function_register" @classmethod def from_fields( @@ -339,13 +312,13 @@ class CustomFunctionRegistrationDestinationRequest(FunctionCatalogControlRequest """Read the persistence owner's exact path without evaluating source.""" function_name: str | None = None - message_type = FunctionCatalogControlMessageType.CUSTOM_REGISTRATION_DESTINATION + wire_type = "openhcs_custom_function_registration_destination" @dataclass(frozen=True, slots=True) class CustomFunctionRegistrationObservationRequest(FunctionCatalogControlRequestABC): handle: CustomFunctionRegistrationHandle - message_type = FunctionCatalogControlMessageType.OBSERVE_CUSTOM_REGISTRATION + wire_type = "openhcs_custom_function_registration_observe" @dataclass(frozen=True, slots=True) @@ -576,7 +549,7 @@ class FunctionCatalogControlRequest(FunctionCatalogControlRequestABC): compact_signatures: bool = True - message_type = FunctionCatalogControlMessageType.READ_CATALOG + wire_type = "openhcs_function_catalog_read" @dataclass(frozen=True, slots=True) @@ -590,7 +563,7 @@ class FunctionDetailControlRequest( max_doc_chars: int | None = DEFAULT_FUNCTION_DETAIL_DOC_CHARS compact_signature: bool = True - message_type = FunctionCatalogControlMessageType.READ_DETAIL + wire_type = "openhcs_function_detail_read" @dataclass(frozen=True, slots=True) @@ -602,7 +575,7 @@ class FunctionReferenceControlRequest( catalog_revision: str - message_type = FunctionCatalogControlMessageType.READ_REFERENCE + wire_type = "openhcs_function_reference_read" @dataclass(frozen=True, slots=True) @@ -610,6 +583,7 @@ class FunctionCatalogControlPayload(Generic[ControlRequestT]): """Wire projection of one request declaration and its enum identity.""" request: ControlRequestT + request_key: ClassVar[str] = "request" @classmethod def from_request(cls, request: ControlRequestT) -> Self: @@ -622,8 +596,8 @@ def from_request(cls, request: ControlRequestT) -> Self: def to_dict(self) -> dict[str, Any]: return { - MessageFields.TYPE: self.request.message_type.value, - FunctionCatalogControlField.REQUEST.value: self.request, + MessageFields.TYPE: self.request.wire_type, + self.request_key: self.request, } @@ -632,11 +606,11 @@ class FunctionCatalogControlResponseBase(Generic[ControlResponseT]): """Shared typed transport for one declared successful control response.""" value: ControlResponseT - field: ClassVar[FunctionCatalogControlField] + result_key: ClassVar[str] = "result" value_type: ClassVar[type] def __init_subclass__(cls) -> None: - for attribute in ("field", "value_type"): + for attribute in ("value_type",): if attribute not in cls.__dict__: raise TypeError( f"{cls.__name__} must declare function-catalog {attribute}." @@ -645,17 +619,17 @@ def __init_subclass__(cls) -> None: def to_control_response(self) -> dict[str, Any]: return { MessageFields.STATUS: ResponseType.OK.value, - self.field.value: self.value, + self.result_key: self.value, } @classmethod def from_control_response(cls, payload: Mapping[str, Any]) -> Self: if payload.get(MessageFields.STATUS) != ResponseType.OK.value: raise RuntimeError(str(payload.get(MessageFields.ERROR) or payload)) - value = payload[cls.field.value] + value = payload[cls.result_key] if not isinstance(value, cls.value_type): raise TypeError( - f"Function catalog response field {cls.field.value!r} requires " + f"Function catalog response {cls.__name__} requires " f"{cls.value_type.__name__}, got {type(value).__name__}." ) return cls(value=cast(ControlResponseT, value)) @@ -666,7 +640,6 @@ class FunctionCatalogControlResponse( ): """Typed response carrying one endpoint-owned catalog projection.""" - field = FunctionCatalogControlField.CATALOG value_type = FunctionCatalogPage @property @@ -677,27 +650,22 @@ def catalog(self) -> FunctionCatalogPage: class FunctionCatalogPreparationStateControlResponse( FunctionCatalogControlResponseBase[FunctionCatalogPreparationState] ): - field = FunctionCatalogControlField.PREPARATION value_type = FunctionCatalogPreparationState -class FunctionCatalogPreparationStatus(str, Enum): - """Control response state while endpoint catalog preparation is active.""" - - PENDING = "pending" - - @dataclass(frozen=True, slots=True) class FunctionCatalogPreparationControlResponse: """Typed signal that the endpoint is actively preparing its catalog.""" status: EndpointStartupStatus retry_after_seconds: float = 0.1 + pending_status: ClassVar[str] = "pending" + """The control reply status while the endpoint prepares its catalog.""" def to_control_response(self) -> dict[str, Any]: return { - MessageFields.STATUS: FunctionCatalogPreparationStatus.PENDING.value, - FunctionCatalogControlField.PREPARATION.value: self, + MessageFields.STATUS: self.pending_status, + FunctionCatalogControlResponseBase.result_key: self, } @classmethod @@ -705,12 +673,9 @@ def from_control_response( cls, payload: Mapping[str, Any], ) -> FunctionCatalogPreparationControlResponse | None: - if ( - payload.get(MessageFields.STATUS) - != FunctionCatalogPreparationStatus.PENDING.value - ): + if payload.get(MessageFields.STATUS) != cls.pending_status: return None - preparation = payload.get(FunctionCatalogControlField.PREPARATION.value) + preparation = payload.get(FunctionCatalogControlResponseBase.result_key) if not isinstance(preparation, cls): raise TypeError( "Pending function catalog response requires " @@ -723,7 +688,6 @@ def from_control_response( class FunctionDetailControlResponse(FunctionCatalogControlResponseBase[FunctionDetail]): """Typed response carrying one revision-consistent function detail.""" - field = FunctionCatalogControlField.DETAIL value_type = FunctionDetail @property @@ -736,7 +700,6 @@ class FunctionReferenceControlResponse( ): """Typed response carrying one server-owned compiler reference.""" - field = FunctionCatalogControlField.REFERENCE value_type = FunctionReference @property @@ -782,7 +745,6 @@ class CustomFunctionRegistrationControlResponse( ): """Typed response carrying endpoint-owned custom registration results.""" - field = FunctionCatalogControlField.RESULT value_type = CustomFunctionRegistrationResult @property @@ -793,7 +755,6 @@ def result(self) -> CustomFunctionRegistrationResult: class CustomFunctionRegistrationDestinationControlResponse( FunctionCatalogControlResponseBase[CustomFunctionRegistrationDestination] ): - field = FunctionCatalogControlField.RESULT value_type = CustomFunctionRegistrationDestination @property @@ -804,7 +765,6 @@ def destination(self) -> CustomFunctionRegistrationDestination: class CustomFunctionRegistrationObservationControlResponse( FunctionCatalogControlResponseBase[CustomFunctionRegistrationObservation] ): - field = FunctionCatalogControlField.RESULT value_type = CustomFunctionRegistrationObservation diff --git a/openhcs/runtime/zmq_control.py b/openhcs/runtime/zmq_control.py index df32723df8..3ad93be762 100644 --- a/openhcs/runtime/zmq_control.py +++ b/openhcs/runtime/zmq_control.py @@ -221,7 +221,7 @@ class FunctionCatalogReadMessageStrategy(FunctionCatalogMessageStrategy): from openhcs.agent.dto.functions import FunctionCatalogControlRequest request_type = FunctionCatalogControlRequest - registry_key = request_type.message_type.value + registry_key = request_type.wire_type def handle_ready( self, @@ -245,7 +245,7 @@ class FunctionCatalogSearchMessageStrategy(FunctionCatalogMessageStrategy): from openhcs.agent.dto.functions import FunctionSearchRequest request_type = FunctionSearchRequest - registry_key = request_type.message_type.value + registry_key = request_type.wire_type def handle_ready( self, @@ -272,7 +272,7 @@ class FunctionDetailReadMessageStrategy(FunctionCatalogMessageStrategy): from openhcs.agent.dto.functions import FunctionDetailControlRequest request_type = FunctionDetailControlRequest - registry_key = request_type.message_type.value + registry_key = request_type.wire_type def handle_ready( self, @@ -299,7 +299,7 @@ class FunctionReferenceReadMessageStrategy(FunctionCatalogMessageStrategy): from openhcs.agent.dto.functions import FunctionReferenceControlRequest request_type = FunctionReferenceControlRequest - registry_key = request_type.message_type.value + registry_key = request_type.wire_type def handle_ready( self, @@ -321,7 +321,7 @@ class CustomFunctionRegistrationDestinationMessageStrategy(ZMQControlMessageStra from openhcs.agent.dto.functions import CustomFunctionRegistrationDestinationRequest request_type = CustomFunctionRegistrationDestinationRequest - registry_key = request_type.message_type.value + registry_key = request_type.wire_type def handle(self, message: dict, context: ZMQControlRequestContext) -> dict: from openhcs.agent.dto.functions import ( @@ -382,7 +382,7 @@ class StartFunctionCatalogPreparationStrategy( from openhcs.agent.dto.functions import FunctionCatalogPreparationStartRequest request_type = FunctionCatalogPreparationStartRequest - registry_key = request_type.message_type.value + registry_key = request_type.wire_type def preparation_state( self, @@ -400,7 +400,7 @@ class ReadFunctionCatalogPreparationStrategy( from openhcs.agent.dto.functions import FunctionCatalogPreparationStatusRequest request_type = FunctionCatalogPreparationStatusRequest - registry_key = request_type.message_type.value + registry_key = request_type.wire_type def preparation_state( self, @@ -418,7 +418,7 @@ class CancelFunctionCatalogPreparationStrategy( from openhcs.agent.dto.functions import FunctionCatalogPreparationCancelRequest request_type = FunctionCatalogPreparationCancelRequest - registry_key = request_type.message_type.value + registry_key = request_type.wire_type def preparation_state( self, @@ -434,7 +434,7 @@ class CustomFunctionRegistrationMessageStrategy(ZMQControlMessageStrategy): from openhcs.agent.dto.functions import CustomFunctionRegistrationRequest request_type = CustomFunctionRegistrationRequest - registry_key = request_type.message_type.value + registry_key = request_type.wire_type def handle(self, message: dict, context: ZMQControlRequestContext) -> dict: from openhcs.agent.dto.functions import ( @@ -465,7 +465,7 @@ class CustomFunctionRegistrationObservationMessageStrategy(ZMQControlMessageStra from openhcs.agent.dto.functions import CustomFunctionRegistrationObservationRequest request_type = CustomFunctionRegistrationObservationRequest - registry_key = request_type.message_type.value + registry_key = request_type.wire_type def handle(self, message: dict, context: ZMQControlRequestContext) -> dict: from openhcs.agent.dto.functions import CustomFunctionRegistrationObservationControlResponse diff --git a/tests/diagnostics/check_custom_registration_live.py b/tests/diagnostics/check_custom_registration_live.py index 90564e47c7..d342c10933 100644 --- a/tests/diagnostics/check_custom_registration_live.py +++ b/tests/diagnostics/check_custom_registration_live.py @@ -34,8 +34,7 @@ def child_server(role: str, audit: Path, native_argv: list[str]) -> None: from openhcs.agent.dto.functions import ( CustomFunctionRegistrationControlResponse, - FunctionCatalogControlField, - FunctionCatalogControlMessageType, + CustomFunctionRegistrationDestinationRequest, ) from openhcs.runtime.zmq_execution_server import ZMQExecutionServer from openhcs.runtime.zmq_execution_server_launcher import main @@ -65,14 +64,14 @@ def handle_control_message(self, message): + "\n" ) if role == "unsupported" and message[MessageFields.TYPE] == ( - FunctionCatalogControlMessageType.CUSTOM_REGISTRATION_DESTINATION.value + CustomFunctionRegistrationDestinationRequest.wire_type ): return ControlErrorResponse( message="Unsupported destination proof fixture" ).to_dict() response = super().handle_control_message(message) - if role == "owned" and FunctionCatalogControlField.RESULT.value in response: - result = response[FunctionCatalogControlField.RESULT.value] + if role == "owned" and CustomFunctionRegistrationControlResponse.result_key in response: + result = response[CustomFunctionRegistrationControlResponse.result_key] if ( isinstance( result, CustomFunctionRegistrationControlResponse.value_type @@ -556,7 +555,7 @@ def finish_job(ref): .splitlines() ] assert all( - row.get("type") != CustomFunctionRegistrationRequest.message_type.value + row.get("type") != CustomFunctionRegistrationRequest.wire_type for row in rows ) receipt["protected_sentinel"] = { @@ -605,7 +604,7 @@ def finish_job(ref): for row in (receipt_dir / "owned-control.jsonl").read_text().splitlines() ] assert all( - row.get("type") != CustomFunctionRegistrationRequest.message_type.value + row.get("type") != CustomFunctionRegistrationRequest.wire_type for row in audit_before_mutation ) receipt["zero_source_dispatch_before_ready"] = True @@ -757,7 +756,7 @@ def finish_job(ref): ] assert ( sum( - row.get("type") == CustomFunctionRegistrationRequest.message_type.value + row.get("type") == CustomFunctionRegistrationRequest.wire_type for row in audit_rows ) == 2 diff --git a/tests/unit/test_catalog_endpoint_incarnation.py b/tests/unit/test_catalog_endpoint_incarnation.py index 37c8fda1a8..ffce8bd834 100644 --- a/tests/unit/test_catalog_endpoint_incarnation.py +++ b/tests/unit/test_catalog_endpoint_incarnation.py @@ -24,7 +24,6 @@ from openhcs.agent.dto.functions import ( CustomFunctionRegistrationDestinationRequest, CustomFunctionRegistrationRequest, - FunctionCatalogControlMessageType, FunctionCatalogControlRequest, FunctionCatalogControlRequestABC, FunctionCatalogControlResponse, @@ -225,7 +224,7 @@ def test_new_request_declaration_and_cooperative_capability_need_no_consumer_edi @dataclass(frozen=True, slots=True) class IndependentCatalogProbe(FunctionCatalogControlRequestABC): marker: str = "independent" - message_type = FunctionCatalogControlMessageType.READ_CATALOG + wire_type = FunctionCatalogControlRequest.wire_type events = [] @@ -348,8 +347,8 @@ def send(payload, *, timeout_ms): if payload[MessageFields.TYPE] == ControlMessageType.PING.value: return expected.to_dict() if ( - payload["request"].message_type - is FunctionCatalogControlMessageType.START_PREPARATION + payload["request"].wire_type + == FunctionCatalogPreparationStartRequest.wire_type ): return FunctionCatalogPreparationStateControlResponse( state diff --git a/tests/unit/test_function_catalog_zmq.py b/tests/unit/test_function_catalog_zmq.py index 3d1724af55..f4830d5c6b 100644 --- a/tests/unit/test_function_catalog_zmq.py +++ b/tests/unit/test_function_catalog_zmq.py @@ -83,8 +83,8 @@ def test_function_parameter_spec_requires_nominal_source() -> None: ) -def test_function_catalog_control_request_requires_declared_message_type() -> None: - with pytest.raises(TypeError, match="message_type"): +def test_function_catalog_control_request_requires_declared_wire_type() -> None: + with pytest.raises(TypeError, match="wire_type"): class MissingMessageType(FunctionCatalogControlRequestABC): pass From baf84e2869780078d10c234bfd7fbd4cef1498ab Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:05:59 -0400 Subject: [PATCH 24/36] E1: shrink the kernel domain-reference pin (DebugSessionPhase.NO_PLATE is gone) Co-Authored-By: Claude Opus 5.5 --- tests/unit/test_axis_family_guards.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/unit/test_axis_family_guards.py b/tests/unit/test_axis_family_guards.py index 8d7335781b..e7de2a479b 100644 --- a/tests/unit/test_axis_family_guards.py +++ b/tests/unit/test_axis_family_guards.py @@ -330,7 +330,6 @@ def test_runtime_foreign_names_name_existing_modules() -> None: ("openhcs/core/dataset_sources/openhcs_format.py", "get_plate_acquisition_datetime"), ("openhcs/core/dataset_sources/openhcs_format.py", "get_plate_name"), ("openhcs/core/dataset_sources/source.py", "plate_root"), - ("openhcs/core/debug_session_projection.py", "NO_PLATE"), ("openhcs/core/debug_session_projection.py", "NoPlateDebugSessionPhase"), ("openhcs/core/debug_session_projection.py", "current_plate_scope_id"), ("openhcs/core/debug_session_projection.py", "no_plate"), From 4da43bf196315b14f7a03855fdefa020fea9f8cd Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:07:06 -0400 Subject: [PATCH 25/36] E1: materialization backend is a family; dead get_required_backend deleted Each MaterializationBackend member declares its PolyStore backend and whether it writes Zarr; the compiler asks writes_zarr instead of comparing members. DatasetSource.get_required_backend had no caller. Co-Authored-By: Claude Opus 5.5 --- benchmark/adapters/openhcs.py | 4 +- benchmark/matched_cellprofiler_batch.py | 4 +- benchmark/well_throughput_scaling.py | 4 +- openhcs/core/config.py | 49 ++++++++++++++++--- openhcs/core/dataset_sources/source.py | 20 +------- openhcs/core/pipeline/compiler.py | 11 ++--- .../pipeline/materialization_flag_planner.py | 8 +-- openhcs/core/pipeline/path_planner.py | 4 +- .../check_custom_registration_live.py | 4 +- .../diagnostics/check_owned_bootstrap_live.py | 4 +- tests/integration/helpers/fixture_utils.py | 2 +- .../test_cellprofiler_generated_pipeline.py | 6 +-- .../test_count_cells_simple_dual_channel.py | 4 +- tests/integration/test_main.py | 12 ++--- .../test_neurite_outgrowth_synthetic_plate.py | 4 +- .../test_source_store_code_mode_zmq.py | 4 +- tests/unit/test_compilation_session.py | 6 +-- ...test_materialization_filter_inheritance.py | 4 +- .../unit/test_materialization_flag_planner.py | 10 ++-- tests/unit/test_openhcs_adapter.py | 4 +- tests/unit/test_source_tile_geometry.py | 2 +- ...test_zarr_input_conversion_subdirectory.py | 6 +-- 22 files changed, 95 insertions(+), 81 deletions(-) diff --git a/benchmark/adapters/openhcs.py b/benchmark/adapters/openhcs.py index 6448fa115f..2461152009 100644 --- a/benchmark/adapters/openhcs.py +++ b/benchmark/adapters/openhcs.py @@ -44,7 +44,7 @@ CompilationDebugConfig, GlobalPipelineConfig, LazyCompilationDebugConfig, - MaterializationBackend, + MaterializeToDisk, PipelineConfig, ) from openhcs.core.equivalence.policy import RuntimeEquivalencePolicy @@ -325,7 +325,7 @@ def _execution_global_config( ), vfs_config=replace( self.global_config.vfs_config, - materialization_backend=MaterializationBackend.DISK, + materialization_backend=MaterializeToDisk, ), compilation_debug_config=compilation_debug_config, materialize_runtime_artifacts=request.materialize_runtime_artifacts, diff --git a/benchmark/matched_cellprofiler_batch.py b/benchmark/matched_cellprofiler_batch.py index a5189ae72e..d4cb5be31a 100644 --- a/benchmark/matched_cellprofiler_batch.py +++ b/benchmark/matched_cellprofiler_batch.py @@ -78,7 +78,7 @@ GlobalPipelineConfig, LazyPathPlanningConfig, LazyPartitionFilterConfig, - MaterializationBackend, + MaterializeToDisk, MultiprocessingStartMethod, PathPlanningConfig, PipelineConfig, @@ -1112,7 +1112,7 @@ def _global_config( global_output_folder=output_dir, output_dir_suffix="_matched_pilot", ), - vfs_config=VFSConfig(materialization_backend=MaterializationBackend.DISK), + vfs_config=VFSConfig(materialization_backend=MaterializeToDisk), analysis_consolidation_config=AnalysisConsolidationConfig(enabled=False), materialize_runtime_artifacts=False, ) diff --git a/benchmark/well_throughput_scaling.py b/benchmark/well_throughput_scaling.py index 4f133d1146..a088abe9b6 100644 --- a/benchmark/well_throughput_scaling.py +++ b/benchmark/well_throughput_scaling.py @@ -42,7 +42,7 @@ LazyPathPlanningConfig, LazyVFSConfig, LazyPartitionFilterConfig, - MaterializationBackend, + MaterializeToDisk, MultiprocessingStartMethod, PartitionFilterConfig, ) @@ -2122,7 +2122,7 @@ def run_case_well_throughput( output_dir_suffix="_well_throughput", ), vfs_config=LazyVFSConfig( - materialization_backend=MaterializationBackend.DISK + materialization_backend=MaterializeToDisk ), ), global_config, diff --git a/openhcs/core/config.py b/openhcs/core/config.py index d5161a41cf..cd622b3434 100644 --- a/openhcs/core/config.py +++ b/openhcs/core/config.py @@ -125,13 +125,44 @@ class StreamingConfigMeta(EnableableMeta, AutoRegisterMeta): TiffCompression = _polystore_config.TiffCompression -class MaterializationBackend(Enum): - """Available backends for materialization (persistent storage only).""" +class MaterializationBackend(ABC, metaclass=AutoRegisterMeta): + """A persistent storage backend materialization may write to. - AUTO = "auto" - ZARR = "zarr" - DISK = "disk" - OMERO_LOCAL = "omero_local" + Each choice declares its PolyStore ``backend``; ``name`` is its spelling in + forms and on the wire. + """ + + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["MaterializationBackend"]]] = {} + + name: ClassVar[str | None] = None + backend: ClassVar[Backend] + writes_zarr: ClassVar[bool] = False + """Outputs are a Zarr store, which needs a Zarr config and converted inputs.""" + + +class MaterializeAuto(MaterializationBackend): + """Use the dataset source's primary backend.""" + + name = "auto" + backend = Backend.AUTO + + +class MaterializeToZarr(MaterializationBackend): + name = "zarr" + backend = Backend.ZARR + writes_zarr = True + + +class MaterializeToDisk(MaterializationBackend): + name = "disk" + backend = Backend.DISK + + +class MaterializeToOmeroLocal(MaterializationBackend): + name = "omero_local" + backend = Backend.OMERO_LOCAL class PartitionFilterMode(ABC, metaclass=AutoRegisterMeta): @@ -481,8 +512,10 @@ class VFSConfig(AnnotatedDataclassValidationMixin): """ materialization_backend: Annotated[ - MaterializationBackend, abbreviation("materialize") - ] = MaterializationBackend.DISK + type[MaterializationBackend], + abbreviation("materialize"), + RegisteredClassChoices(MaterializationBackend), + ] = MaterializeToDisk """Persistent backend used by compiled main-flow and artifact materialization plans. This setting selects how requested outputs are stored; it does not itself diff --git a/openhcs/core/dataset_sources/source.py b/openhcs/core/dataset_sources/source.py index 5f84095275..7c5a2aed48 100644 --- a/openhcs/core/dataset_sources/source.py +++ b/openhcs/core/dataset_sources/source.py @@ -33,7 +33,7 @@ logger = logging.getLogger(__name__) if TYPE_CHECKING: - from openhcs.core.config import MaterializationBackend, PipelineConfig + from openhcs.core.config import PipelineConfig from openhcs.core.runtime_pattern_cache import RuntimePatternDiscoveryCache from openhcs.core.config import SourceBindingsConfig @@ -315,24 +315,6 @@ def compatible_backends(self) -> List[Backend]: def initialize_workspace(self, dataset_root: Path, filemanager: FileManager) -> Path: """Prepare the dataset for reading and return its image directory.""" - def get_required_backend(self) -> Optional["MaterializationBackend"]: - """The materialization backend this source requires, when it has one.""" - from openhcs.core.config import MaterializationBackend - - if len(self.compatible_backends) == 1: - backend_value = self.compatible_backends[0].value - materialization_values = { - materialization_backend.value - for materialization_backend in MaterializationBackend - } - if backend_value not in materialization_values: - raise RuntimeError( - f"{self.source_name} declares single backend " - f"{backend_value!r}, but it is not a materialization backend." - ) - return MaterializationBackend(backend_value) - return None - def available_backends(self, dataset_root: Union[str, Path]) -> List[Backend]: """Backends available for this dataset (default: every compatible one).""" del dataset_root diff --git a/openhcs/core/pipeline/compiler.py b/openhcs/core/pipeline/compiler.py index 2aaa8ab6ac..00ddd03029 100644 --- a/openhcs/core/pipeline/compiler.py +++ b/openhcs/core/pipeline/compiler.py @@ -43,7 +43,7 @@ ) from openhcs.core.config import ( GlobalPipelineConfig, - MaterializationBackend, + MaterializeToZarr, StreamingConfig, VFSConfig, PartitionFilterConfig, @@ -287,7 +287,7 @@ def _configure_input_conversion_if_needed( return vfs_config = session.global_config.vfs_config - if vfs_config.materialization_backend != MaterializationBackend.ZARR: + if not vfs_config.materialization_backend.writes_zarr: return available_backends = context.microscope_handler.available_backends( @@ -312,7 +312,7 @@ def _configure_input_conversion_if_needed( zarr_subdir = "zarr" if uses_virtual_workspace else original_subdir context.step_plans[0].input_conversion = InputConversionPlan( output_dir=dataset_root / zarr_subdir, - backend=MaterializationBackend.ZARR.value, + backend=MaterializeToZarr.backend.value, uses_virtual_workspace=uses_virtual_workspace, original_subdir=original_subdir, ) @@ -507,8 +507,7 @@ def declare_zarr_stores(session: CompilationSession) -> None: session.plan(step_index).zarr_config = None if ( not session.step_count - or session.global_config.vfs_config.materialization_backend - != MaterializationBackend.ZARR + or not session.global_config.vfs_config.materialization_backend.writes_zarr ): return last_step_index = session.step_count - 1 @@ -591,7 +590,7 @@ def _compile_runtime_artifact_materialization_plans( if persistent_step_indexes: persistent_backend = ( materialization_planner.resolve_backend( - session.global_config.vfs_config.materialization_backend, + session.global_config.vfs_config.materialization_backend.backend, ) ) for step_index, step_plan in session.plans.items(): diff --git a/openhcs/core/pipeline/materialization_flag_planner.py b/openhcs/core/pipeline/materialization_flag_planner.py index 7181dcc313..cb4abba576 100644 --- a/openhcs/core/pipeline/materialization_flag_planner.py +++ b/openhcs/core/pipeline/materialization_flag_planner.py @@ -15,7 +15,7 @@ from openhcs.constants.constants import Backend from openhcs.core.context.processing_context import ProcessingContext from openhcs.core.steps.abstract import AbstractStep -from openhcs.core.config import GlobalPipelineConfig, MaterializationBackend +from openhcs.core.config import GlobalPipelineConfig from openhcs.core.partition_filter import PartitionFilterProcessor from openhcs.core.vfs_protocol import FileManagerLike @@ -57,7 +57,7 @@ def __post_init__(self) -> None: ), ) - def resolve_backend(self, declaration: Backend | MaterializationBackend) -> str: + def resolve_backend(self, declaration: Backend) -> str: """Resolve explicit declarations directly and admit AUTO once per submission.""" if declaration.value != Backend.AUTO.value: return declaration.value @@ -110,7 +110,7 @@ def prepare_pipeline_flags( or i == last_image_materialization_step ): step_plan.write_backend = self.resolve_backend( - vfs_config.materialization_backend, + vfs_config.materialization_backend.backend, ) else: step_plan.write_backend = Backend.MEMORY.value @@ -118,7 +118,7 @@ def prepare_pipeline_flags( if step_plan.materialized_output is not None: step_plan.materialized_output = replace( step_plan.materialized_output, - backend=self.resolve_backend(vfs_config.materialization_backend), + backend=self.resolve_backend(vfs_config.materialization_backend.backend), ) if not materializes_main_flow_axis: diff --git a/openhcs/core/pipeline/path_planner.py b/openhcs/core/pipeline/path_planner.py index 720fd216f5..f169ccbbd1 100644 --- a/openhcs/core/pipeline/path_planner.py +++ b/openhcs/core/pipeline/path_planner.py @@ -1796,7 +1796,7 @@ def input_conversion_plan_for_step( return InputConversionPlan( output_dir=output_dir, - backend=self.planner.vfs.materialization_backend.value, + backend=self.planner.vfs.materialization_backend.backend.value, uses_virtual_workspace=False, original_subdir=input_dir.name, ) @@ -1819,7 +1819,7 @@ def apply_materialization_plan( ) self.planner.plans[step_index].materialized_output = MaterializedOutputPlan( output_dir=materialized_output_dir, - backend=self.planner.vfs.materialization_backend.value, + backend=self.planner.vfs.materialization_backend.backend.value, dataset_root=str(materialized_dataset_root), sub_dir=materialization_config.sub_dir, analysis_results_dir=str( diff --git a/tests/diagnostics/check_custom_registration_live.py b/tests/diagnostics/check_custom_registration_live.py index 90564e47c7..00e6362b7f 100644 --- a/tests/diagnostics/check_custom_registration_live.py +++ b/tests/diagnostics/check_custom_registration_live.py @@ -640,7 +640,7 @@ def finish_job(ref): LazyPathPlanningConfig, LazyStepMaterializationConfig, LazyVFSConfig, - MaterializationBackend, + MaterializeToDisk, PipelineConfig, ) from openhcs.core.pipeline_document import PipelineDocumentCodec @@ -655,7 +655,7 @@ def finish_job(ref): global_output_folder=owned / "outputs" ), vfs_config=LazyVFSConfig( - materialization_backend=MaterializationBackend.DISK + materialization_backend=MaterializeToDisk ), ), pipeline_steps=[ diff --git a/tests/diagnostics/check_owned_bootstrap_live.py b/tests/diagnostics/check_owned_bootstrap_live.py index 6e8005b256..d41e468d6f 100644 --- a/tests/diagnostics/check_owned_bootstrap_live.py +++ b/tests/diagnostics/check_owned_bootstrap_live.py @@ -315,7 +315,7 @@ def finish(ref): assert registered.functions[0].function_id in {entry.function_id for entry in page.items} save() from openhcs.core.config import (PipelineConfig, LazyPathPlanningConfig, - LazyVFSConfig, MaterializationBackend, + LazyVFSConfig, MaterializeToDisk, LazyStepMaterializationConfig, LazyProcessingConfig) from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.core.steps.function_step import FunctionStep @@ -335,7 +335,7 @@ def finish(ref): pipeline_config=PipelineConfig(num_workers=1, use_threading=True, dataset_source=BioFormatsHandler, path_planning_config=LazyPathPlanningConfig(global_output_folder=owned/'outputs'), - vfs_config=LazyVFSConfig(materialization_backend=MaterializationBackend.DISK)), + vfs_config=LazyVFSConfig(materialization_backend=MaterializeToDisk)), pipeline_steps=steps, ) source = PipelineDocumentCodec.render(document) diff --git a/tests/integration/helpers/fixture_utils.py b/tests/integration/helpers/fixture_utils.py index 19a3bb4fb3..ced2c39d3d 100644 --- a/tests/integration/helpers/fixture_utils.py +++ b/tests/integration/helpers/fixture_utils.py @@ -505,7 +505,7 @@ def debug_global_config(execution_mode, backend_config): sub_dir="images", # Default subdirectory for processed data output_dir_suffix="_outputs", # Suffix for output directories ), - vfs=VFSConfig(materialization_backend=MaterializationBackend(backend_config)), + vfs=VFSConfig(materialization_backend=MaterializationBackend.__registry__[backend_config]), zarr=ZarrConfig(), use_threading=use_threading, ) diff --git a/tests/integration/test_cellprofiler_generated_pipeline.py b/tests/integration/test_cellprofiler_generated_pipeline.py index 8cc702c685..5844c17874 100644 --- a/tests/integration/test_cellprofiler_generated_pipeline.py +++ b/tests/integration/test_cellprofiler_generated_pipeline.py @@ -49,7 +49,7 @@ LazyProcessingConfig, LazyVFSConfig, LazyPartitionFilterConfig, - MaterializationBackend, + MaterializeToDisk, PipelineConfig, LazySourceBindingsConfig, source_bindings_defaults_to_base, @@ -177,7 +177,7 @@ def _execute_imported_cppipe_via_zmq( output_dir_suffix="_imported_cppipe_zmq", ), vfs_config=LazyVFSConfig( - materialization_backend=MaterializationBackend.DISK, + materialization_backend=MaterializeToDisk, ), partition_filter_config=( LazyPartitionFilterConfig() @@ -387,7 +387,7 @@ def test_invalid_public_cellprofiler_step_fails_during_zmq_compilation( global_output_folder=tmp_path / "invalid_compile_outputs", ), vfs_config=LazyVFSConfig( - materialization_backend=MaterializationBackend.DISK, + materialization_backend=MaterializeToDisk, ), ) observation_path = tmp_path / "invalid_compile_observation.pkl" diff --git a/tests/integration/test_count_cells_simple_dual_channel.py b/tests/integration/test_count_cells_simple_dual_channel.py index bc9bc2ada2..f3ce1f5256 100644 --- a/tests/integration/test_count_cells_simple_dual_channel.py +++ b/tests/integration/test_count_cells_simple_dual_channel.py @@ -26,7 +26,7 @@ LazyPathPlanningConfig, LazyProcessingConfig, LazyVFSConfig, - MaterializationBackend, + MaterializeToDisk, PathPlanningConfig, PipelineConfig, VFSConfig, @@ -115,7 +115,7 @@ def test_dual_channel_count_runs_on_synthetic_plate_with_channel_stack( _write_known_dual_channel_images(plate_dir) suffix = "_dual_channel_test" - vfs_config = VFSConfig(materialization_backend=MaterializationBackend.DISK) + vfs_config = VFSConfig(materialization_backend=MaterializeToDisk) global_config = GlobalPipelineConfig( num_workers=2, dataset_source=ImageXpressHandler, diff --git a/tests/integration/test_main.py b/tests/integration/test_main.py index 3df43c6f52..9b77bf17bd 100644 --- a/tests/integration/test_main.py +++ b/tests/integration/test_main.py @@ -464,9 +464,9 @@ def _create_pipeline_config(test_config: TestConfig) -> GlobalPipelineConfig: # For OMERO tests, use omero_local backend for materialization # For other tests, use the configured backend if test_config.is_omero: - materialization_backend = MaterializationBackend("omero_local") + materialization_backend = MaterializationBackend.__registry__["omero_local"] else: - materialization_backend = MaterializationBackend(test_config.backend_config) + materialization_backend = MaterializationBackend.__registry__[test_config.backend_config] return GlobalPipelineConfig( num_workers=CONSTANTS.DEFAULT_WORKERS, @@ -525,9 +525,9 @@ def _initialize_orchestrator( # For OMERO tests, use omero_local backend for materialization # For other tests, use the configured backend if test_config.is_omero: - materialization_backend = MaterializationBackend("omero_local") + materialization_backend = MaterializationBackend.__registry__["omero_local"] else: - materialization_backend = MaterializationBackend(test_config.backend_config) + materialization_backend = MaterializationBackend.__registry__[test_config.backend_config] # Create PipelineConfig with lazy configs for proper hierarchical inheritance pipeline_config = PipelineConfig( @@ -784,9 +784,9 @@ def _execute_pipeline_with_mode( # For OMERO tests, use omero_local backend for materialization # For other tests, use the configured backend if test_config.is_omero: - materialization_backend = MaterializationBackend("omero_local") + materialization_backend = MaterializationBackend.__registry__["omero_local"] else: - materialization_backend = MaterializationBackend(test_config.backend_config) + materialization_backend = MaterializationBackend.__registry__[test_config.backend_config] # Create pipeline config with lazy configs from openhcs.core.config import ( diff --git a/tests/integration/test_neurite_outgrowth_synthetic_plate.py b/tests/integration/test_neurite_outgrowth_synthetic_plate.py index f96301ea0a..8b92ca8bba 100644 --- a/tests/integration/test_neurite_outgrowth_synthetic_plate.py +++ b/tests/integration/test_neurite_outgrowth_synthetic_plate.py @@ -43,7 +43,7 @@ LazyPathPlanningConfig, LazyProcessingConfig, LazyVFSConfig, - MaterializationBackend, + MaterializeToDisk, NapariDisplayConfig, PathPlanningConfig, PipelineConfig, @@ -121,7 +121,7 @@ def test_neurite_outgrowth_runs_on_synthetic_plate_as_2d_channel_stack( source_stack = _write_known_neurite_images(plate_dir) suffix = "_neurite_test" - vfs_config = VFSConfig(materialization_backend=MaterializationBackend.DISK) + vfs_config = VFSConfig(materialization_backend=MaterializeToDisk) global_config = GlobalPipelineConfig( num_workers=1, dataset_source=ImageXpressHandler, diff --git a/tests/integration/test_source_store_code_mode_zmq.py b/tests/integration/test_source_store_code_mode_zmq.py index d1673258ab..a462a02f1f 100644 --- a/tests/integration/test_source_store_code_mode_zmq.py +++ b/tests/integration/test_source_store_code_mode_zmq.py @@ -23,7 +23,7 @@ LazyPathPlanningConfig, LazyProcessingConfig, LazyVFSConfig, - MaterializationBackend, + MaterializeToDisk, PipelineConfig, LazySourceBindingsConfig, LazyStepSourceBindingsConfig, @@ -149,7 +149,7 @@ def _code_mode_round_trip( global_output_folder=output_root, ), vfs_config=LazyVFSConfig( - materialization_backend=MaterializationBackend.DISK, + materialization_backend=MaterializeToDisk, ), ) payload = PlateManagerCodeDocumentAuthority.from_values( diff --git a/tests/unit/test_compilation_session.py b/tests/unit/test_compilation_session.py index 4ddb4c0122..7e19a9824a 100644 --- a/tests/unit/test_compilation_session.py +++ b/tests/unit/test_compilation_session.py @@ -15,7 +15,7 @@ from openhcs.domains.microscopy.config import AnalysisConsolidationConfig from openhcs.core.config import ( GlobalPipelineConfig, - MaterializationBackend, + MaterializeToDisk, MaterializeToZarr, LazyNapariStreamingConfig, LazyProcessingConfig, LazySourceBindingsConfig, @@ -96,7 +96,7 @@ def full_plate_keys(component, *, resolved_config): @pytest.mark.parametrize( - "backend", [MaterializationBackend.DISK, MaterializationBackend.ZARR] + "backend", [MaterializeToDisk, MaterializeToZarr] ) def test_zarr_store_planning_queries_whole_plate_only_for_zarr(backend): config = GlobalPipelineConfig() @@ -123,7 +123,7 @@ def full_plate_keys(component, *, resolved_config): ) PipelineCompiler.declare_zarr_stores(session) assert plans[0].zarr_config is None - if backend is MaterializationBackend.ZARR: + if backend is MaterializeToZarr: assert plans[1].zarr_config == { "all_wells": ("A01", "A02"), "needs_initialization": True } diff --git a/tests/unit/test_materialization_filter_inheritance.py b/tests/unit/test_materialization_filter_inheritance.py index 0f9a2ff1cd..0ac8b1f6db 100644 --- a/tests/unit/test_materialization_filter_inheritance.py +++ b/tests/unit/test_materialization_filter_inheritance.py @@ -15,7 +15,7 @@ LazyPathPlanningConfig, LazyStepMaterializationConfig, LazyPartitionFilterConfig, - MaterializationBackend, + MaterializeToDisk, PipelineConfig, VFSConfig, ) @@ -111,7 +111,7 @@ def test_path_zero_keeps_inherited_step_checkpoint_independent(tmp_path) -> None pipeline_config=SimpleNamespace( vfs_config=VFSConfig( read_backend=Backend.DISK, - materialization_backend=MaterializationBackend.DISK, + materialization_backend=MaterializeToDisk, ), path_planning_config=effective_config.path_planning_config, ), diff --git a/tests/unit/test_materialization_flag_planner.py b/tests/unit/test_materialization_flag_planner.py index e2302a2409..ab667f8fe3 100644 --- a/tests/unit/test_materialization_flag_planner.py +++ b/tests/unit/test_materialization_flag_planner.py @@ -10,7 +10,7 @@ ) from openhcs.core.compiled_step_plan import CompiledStepPlan from openhcs.core.config import ( - MaterializationBackend, + MaterializeAuto, MaterializeToDisk, PathPlanningConfig, VFSConfig, ) @@ -27,7 +27,7 @@ def _pipeline_config( return SimpleNamespace( vfs_config=VFSConfig( read_backend=Backend.DISK, - materialization_backend=MaterializationBackend.DISK, + materialization_backend=MaterializeToDisk, ), path_planning_config=path_planning_config, ) @@ -220,7 +220,7 @@ def primary_backend(path, manager): config = _pipeline_config() config.vfs_config = VFSConfig( read_backend=Backend.AUTO, - materialization_backend=MaterializationBackend.AUTO, + materialization_backend=MaterializeAuto, ) def new_submission(): @@ -243,7 +243,7 @@ def new_submission(): assert context.step_plans[0].write_backend == "virtual_workspace" assert requests == [(input_dir, filemanager)] source_backend = "zarr" - assert planner.resolve_backend(MaterializationBackend.AUTO) == "virtual_workspace" + assert planner.resolve_backend(MaterializeAuto.backend) == "virtual_workspace" assert new_submission().resolve_backend(Backend.AUTO) == "zarr" assert requests == [(input_dir, filemanager), (input_dir, filemanager)] @@ -252,7 +252,7 @@ def test_unused_materialization_auto_does_not_query_source(): config = _pipeline_config(path_planning_config=PathPlanningConfig(partition_filter=0)) config.vfs_config = VFSConfig( read_backend=Backend.DISK, - materialization_backend=MaterializationBackend.AUTO, + materialization_backend=MaterializeAuto, ) planner = MaterializationFlagPlanner( pipeline_config=config, diff --git a/tests/unit/test_openhcs_adapter.py b/tests/unit/test_openhcs_adapter.py index 81d555b7e6..766cc3ac75 100644 --- a/tests/unit/test_openhcs_adapter.py +++ b/tests/unit/test_openhcs_adapter.py @@ -35,7 +35,7 @@ from openhcs.core.config import ( CompilationDebugConfig, GlobalPipelineConfig, - MaterializationBackend, + MaterializeToDisk, PathPlanningConfig, PipelineConfig, VFSConfig, @@ -162,7 +162,7 @@ def test_openhcs_adapter_preserves_declared_main_flow_output_policy( assert configured.path_planning_config.global_output_folder == request.output_dir assert configured.path_planning_config.output_dir_suffix == "_candidate" assert configured.vfs_config.intermediate_backend is Backend.DISK - assert configured.vfs_config.materialization_backend is MaterializationBackend.DISK + assert configured.vfs_config.materialization_backend is MaterializeToDisk assert configured.analysis_consolidation_config.enabled is False assert configured.analysis_consolidation_config.metaxpress_style is False assert ( diff --git a/tests/unit/test_source_tile_geometry.py b/tests/unit/test_source_tile_geometry.py index c97b3f3091..d3c7b38e86 100644 --- a/tests/unit/test_source_tile_geometry.py +++ b/tests/unit/test_source_tile_geometry.py @@ -541,7 +541,7 @@ def test_synthetic_pipeline_compiles_and_executes_positions_artifact_then_paired config = PipelineConfig( source_bindings_config=LazySourceBindingsConfig.from_config(_config()), vfs_config=LazyVFSConfig( - materialization_backend=MaterializationBackend(storage_backend) + materialization_backend=MaterializationBackend.__registry__[storage_backend] ), ) orchestrator = PipelineOrchestrator(plate, pipeline_config=config).initialize() diff --git a/tests/unit/test_zarr_input_conversion_subdirectory.py b/tests/unit/test_zarr_input_conversion_subdirectory.py index c391a6e00f..8900bdcee5 100644 --- a/tests/unit/test_zarr_input_conversion_subdirectory.py +++ b/tests/unit/test_zarr_input_conversion_subdirectory.py @@ -13,7 +13,7 @@ from openhcs.core.config import ( GlobalPipelineConfig, LazyVFSConfig, - MaterializationBackend, + MaterializeToZarr, PipelineConfig, VFSConfig, ) @@ -56,13 +56,13 @@ def test_zarr_conversion_keys_store_outputs_by_plate_relative_virtual_path(tmp_p num_workers=1, use_threading=True, dataset_source=DatasetSourceChoice.named("openhcsdata"), - vfs_config=VFSConfig(materialization_backend=MaterializationBackend.ZARR), + vfs_config=VFSConfig(materialization_backend=MaterializeToZarr), ), ) orchestrator = PipelineOrchestrator( plate, pipeline_config=PipelineConfig( - vfs_config=LazyVFSConfig(materialization_backend=MaterializationBackend.ZARR) + vfs_config=LazyVFSConfig(materialization_backend=MaterializeToZarr) ), ).initialize() wells = orchestrator.get_component_keys(AxisFamily.active().partition_axis()) From cddd60c519577a3a8248b3818e799ab512c43301 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:15:20 -0400 Subject: [PATCH 26/36] E1: custom-function lifetime and MFD presets without enums CustomFunctionLifetime members own whether a source file must be checked. MfdPresetKey keyed a table restating each preset and MfdPresetFamily keyed the materializer registry; each preset is now one definition naming its materializer. Co-Authored-By: Claude Opus 5.5 --- .../processing/custom_functions/manager.py | 3 +- .../custom_functions/runtime_registry.py | 54 +++++----- openhcs/processing/presets/mfd_specs.py | 99 ++++++------------- .../presets/pipelines/10x_mfd_crop_analyze.py | 4 +- .../10x_mfd_crop_analyze_dapi-fitc-cy5.py | 4 +- .../pipelines/10x_mfd_stitch_ashlar_cpu.py | 4 +- .../presets/pipelines/10x_mfd_stitch_gpu.py | 4 +- .../agent/test_compile_selector_authoring.py | 4 +- .../unit/agent/test_knowledge_base_service.py | 4 +- ...test_knowledge_lazy_conversion_boundary.py | 4 +- .../test_public_callable_enum_ingress.py | 4 +- tests/unit/test_mfd_preset_specs.py | 23 +++-- 12 files changed, 94 insertions(+), 117 deletions(-) diff --git a/openhcs/processing/custom_functions/manager.py b/openhcs/processing/custom_functions/manager.py index e4934e2a11..eea25e3ec7 100644 --- a/openhcs/processing/custom_functions/manager.py +++ b/openhcs/processing/custom_functions/manager.py @@ -29,6 +29,7 @@ from openhcs.processing.custom_functions.events import custom_function_changed from openhcs.processing.custom_functions.runtime_registry import ( CustomFunctionLifetime, + PersistedLifetime, CustomFunctionRuntimeRegistry, project_custom_function, ) @@ -307,7 +308,7 @@ def load_custom_function( return 1 CustomFunctionRuntimeRegistry.publish( metadata, - CustomFunctionLifetime.PERSISTED, + PersistedLifetime, ) functions = [metadata.func] diff --git a/openhcs/processing/custom_functions/runtime_registry.py b/openhcs/processing/custom_functions/runtime_registry.py index 106b3eabdc..825957e5ff 100644 --- a/openhcs/processing/custom_functions/runtime_registry.py +++ b/openhcs/processing/custom_functions/runtime_registry.py @@ -9,7 +9,7 @@ from concurrent.futures import Future from contextlib import contextmanager from dataclasses import dataclass -from enum import Enum, auto +from abc import ABC from traceback import TracebackException from typing import TYPE_CHECKING @@ -79,24 +79,32 @@ def _canonical_metadata_claims( ) -class CustomFunctionLifetime(Enum): +class CustomFunctionLifetime(ABC): """Lifetime owned by one process-local custom-function declaration.""" - PERSISTED = auto() - EPHEMERAL = auto() + @classmethod + def require_current_source(cls, source: CustomFunctionSource) -> None: + """Validate the durable source; only a persisted function owns a file.""" + + @staticmethod + def from_persist(persist: bool) -> type[CustomFunctionLifetime]: + return PersistedLifetime if persist else EphemeralLifetime - def require_current_source(self, source: CustomFunctionSource) -> None: - """Validate the durable source only for the lifetime that owns a file.""" - if self is CustomFunctionLifetime.PERSISTED: - from openhcs.processing.custom_functions.manager import ( - CustomFunctionManager, - ) - CustomFunctionManager(create_storage=False).require_source(source) +class PersistedLifetime(CustomFunctionLifetime): + """Declared by a source file in the custom-function store.""" @classmethod - def from_persist(cls, persist: bool) -> CustomFunctionLifetime: - return cls.PERSISTED if persist else cls.EPHEMERAL + def require_current_source(cls, source: CustomFunctionSource) -> None: + from openhcs.processing.custom_functions.manager import ( + CustomFunctionManager, + ) + + CustomFunctionManager(create_storage=False).require_source(source) + + +class EphemeralLifetime(CustomFunctionLifetime): + """Declared only in this process.""" @dataclass(frozen=True, slots=True) @@ -104,7 +112,7 @@ class CustomFunctionRuntimeDeclaration: """One projected custom function together with its source lifetime.""" metadata: FunctionMetadata - lifetime: CustomFunctionLifetime + lifetime: type[CustomFunctionLifetime] def require_source_namespace( self, @@ -304,14 +312,14 @@ def prepare_source_once( def publish( cls, metadata: FunctionMetadata, - lifetime: CustomFunctionLifetime, + lifetime: type[CustomFunctionLifetime], ) -> None: """Publish one prepared declaration with its exact source lifetime.""" with cls._lock: metadata = cls._canonical_metadata_for_publication(metadata) cls._publish_locked(metadata, lifetime) - if lifetime is CustomFunctionLifetime.PERSISTED: + if lifetime is PersistedLifetime: cls._source_revision = None @classmethod @@ -371,7 +379,7 @@ def replace(cls, old_name: str, metadata: FunctionMetadata) -> None: if old_name != new_name: cls._remove_preparation_outcomes_locked(old_name) cls._remove_module_exports_locked((old_name,)) - cls._publish_locked(metadata, CustomFunctionLifetime.PERSISTED) + cls._publish_locked(metadata, PersistedLifetime) cls._source_revision = None @classmethod @@ -392,7 +400,7 @@ def replace_all( ephemeral_declarations = { name: declaration for name, declaration in cls._declarations_by_name.items() - if declaration.lifetime is CustomFunctionLifetime.EPHEMERAL + if declaration.lifetime is EphemeralLifetime } for function_name, metadata in metadata_by_name.items(): ephemeral = ephemeral_declarations.get(function_name) @@ -411,7 +419,7 @@ def replace_all( persisted_names = { name for name, declaration in cls._declarations_by_name.items() - if declaration.lifetime is CustomFunctionLifetime.PERSISTED + if declaration.lifetime is PersistedLifetime } for function_name in persisted_names - metadata_by_name.keys(): cls._remove_preparation_outcomes_locked(function_name) @@ -421,7 +429,7 @@ def replace_all( **{ name: CustomFunctionRuntimeDeclaration( metadata=metadata, - lifetime=CustomFunctionLifetime.PERSISTED, + lifetime=PersistedLifetime, ) for name, metadata in metadata_by_name.items() }, @@ -440,7 +448,7 @@ def remove(cls, function_name: str) -> None: declaration = cls._declarations_by_name.pop(function_name, None) if ( declaration is not None - and declaration.lifetime is CustomFunctionLifetime.PERSISTED + and declaration.lifetime is PersistedLifetime ): cls._source_revision = None @@ -467,7 +475,7 @@ def source_revision(cls) -> CustomFunctionSourceRevision | None: def _publish_locked( cls, metadata: FunctionMetadata, - lifetime: CustomFunctionLifetime, + lifetime: type[CustomFunctionLifetime], ) -> None: existing = cls._declarations_by_name.get(metadata.original_name) replacing_name = ( @@ -595,6 +603,6 @@ def register_custom_function(func: Callable) -> Callable: metadata = project_custom_function(func) CustomFunctionRuntimeRegistry.publish( metadata, - CustomFunctionLifetime.EPHEMERAL, + EphemeralLifetime, ) return metadata.func diff --git a/openhcs/processing/presets/mfd_specs.py b/openhcs/processing/presets/mfd_specs.py index a1c08273ef..717bb44cc1 100644 --- a/openhcs/processing/presets/mfd_specs.py +++ b/openhcs/processing/presets/mfd_specs.py @@ -5,11 +5,9 @@ from abc import ABC, abstractmethod from collections.abc import Callable from dataclasses import dataclass -from enum import Enum from pathlib import Path from typing import Any -from metaclass_registry import AutoRegisterMeta from arraybridge.decorators import DtypeConversion from openhcs.constants.input_source import InputSource @@ -96,35 +94,21 @@ def materialize(self) -> list[FunctionStep]: return [step.materialize() for step in self.steps] -class MfdPresetKey(str, Enum): - """Named MFD preset variants materialized by this spec authority.""" - - CROP_ANALYZE = "10x_mfd_crop_analyze" - CROP_ANALYZE_DAPI_FITC_CY5 = "10x_mfd_crop_analyze_dapi_fitc_cy5" - STITCH_ASHLAR_CPU = "10x_mfd_stitch_ashlar_cpu" - STITCH_GPU = "10x_mfd_stitch_gpu" - - -class MfdPresetFamily(str, Enum): - """High-level MFD preset template families.""" - - CROP_ANALYZE = "crop_analyze" - STITCH = "stitch" - - @dataclass(frozen=True, slots=True) class MfdPresetDefinition: """Variant row for MFD preset materialization.""" - key: MfdPresetKey - family: MfdPresetFamily + name: str + """The preset's pipeline name.""" + materializer: type["MfdPresetMaterializer"] + """The template family that builds this preset's steps.""" include_cy5_cell_count: bool = False stitch_func: Callable[..., Any] | None = None def required_stitch_func(self) -> Callable[..., Any]: if self.stitch_func is None: raise ValueError( - f"MFD stitch preset {self.key.value} is missing stitch_func" + f"MFD stitch preset {self.name} is missing stitch_func" ) return self.stitch_func @@ -285,41 +269,8 @@ def materialize_spec(self, definition: MfdPresetDefinition) -> PresetStepSpec: ) -MFD_PRESET_DEFINITIONS: dict[MfdPresetKey, MfdPresetDefinition] = { - MfdPresetKey.CROP_ANALYZE: MfdPresetDefinition( - key=MfdPresetKey.CROP_ANALYZE, - family=MfdPresetFamily.CROP_ANALYZE, - ), - MfdPresetKey.CROP_ANALYZE_DAPI_FITC_CY5: MfdPresetDefinition( - key=MfdPresetKey.CROP_ANALYZE_DAPI_FITC_CY5, - family=MfdPresetFamily.CROP_ANALYZE, - include_cy5_cell_count=True, - ), - MfdPresetKey.STITCH_ASHLAR_CPU: MfdPresetDefinition( - key=MfdPresetKey.STITCH_ASHLAR_CPU, - family=MfdPresetFamily.STITCH, - stitch_func=ashlar_compute_tile_positions_cpu, - ), - MfdPresetKey.STITCH_GPU: MfdPresetDefinition( - key=MfdPresetKey.STITCH_GPU, - family=MfdPresetFamily.STITCH, - stitch_func=ashlar_compute_tile_positions_gpu, - ), -} - - -class MfdPresetMaterializer(ABC, metaclass=AutoRegisterMeta): - """Nominal strategy family for materializing MFD preset families.""" - - __registry_key__ = "family" - __skip_if_no_key__ = True - __registry__: dict[MfdPresetFamily, type["MfdPresetMaterializer"]] = {} - - family: MfdPresetFamily | None = None - - @classmethod - def for_family(cls, family: MfdPresetFamily) -> "MfdPresetMaterializer": - return cls.__registry__[family]() +class MfdPresetMaterializer(ABC): + """One MFD preset template family; it builds a preset's steps.""" @abstractmethod def materialize(self, definition: MfdPresetDefinition) -> list[FunctionStep]: @@ -329,11 +280,9 @@ def materialize(self, definition: MfdPresetDefinition) -> list[FunctionStep]: class CropAnalyzeMfdPresetMaterializer(MfdPresetMaterializer): """Materialize MFD crop/analyze preset variants.""" - family = MfdPresetFamily.CROP_ANALYZE - def materialize(self, definition: MfdPresetDefinition) -> list[FunctionStep]: return PresetPipelineSpec( - name=definition.key.value, + name=definition.name, steps=( PresetStepSpec( binding=PresetStepBinding( @@ -359,11 +308,9 @@ def materialize(self, definition: MfdPresetDefinition) -> list[FunctionStep]: class StitchMfdPresetMaterializer(MfdPresetMaterializer): """Materialize MFD stitch preset variants.""" - family = MfdPresetFamily.STITCH - def materialize(self, definition: MfdPresetDefinition) -> list[FunctionStep]: return PresetPipelineSpec( - name=definition.key.value, + name=definition.name, steps=tuple( template.materialize_spec(definition) for template in STITCH_STEP_TEMPLATES @@ -371,8 +318,26 @@ def materialize(self, definition: MfdPresetDefinition) -> list[FunctionStep]: ).materialize() -def build_mfd_preset(key: MfdPresetKey | str) -> list[FunctionStep]: - """Materialize an MFD preset by typed key.""" - preset_key = MfdPresetKey(key) - definition = MFD_PRESET_DEFINITIONS[preset_key] - return MfdPresetMaterializer.for_family(definition.family).materialize(definition) +MFD_CROP_ANALYZE = MfdPresetDefinition( + "10x_mfd_crop_analyze", CropAnalyzeMfdPresetMaterializer +) +MFD_CROP_ANALYZE_DAPI_FITC_CY5 = MfdPresetDefinition( + "10x_mfd_crop_analyze_dapi_fitc_cy5", + CropAnalyzeMfdPresetMaterializer, + include_cy5_cell_count=True, +) +MFD_STITCH_ASHLAR_CPU = MfdPresetDefinition( + "10x_mfd_stitch_ashlar_cpu", + StitchMfdPresetMaterializer, + stitch_func=ashlar_compute_tile_positions_cpu, +) +MFD_STITCH_GPU = MfdPresetDefinition( + "10x_mfd_stitch_gpu", + StitchMfdPresetMaterializer, + stitch_func=ashlar_compute_tile_positions_gpu, +) + + +def build_mfd_preset(preset: MfdPresetDefinition) -> list[FunctionStep]: + """Materialize one MFD preset into its steps.""" + return preset.materializer().materialize(preset) diff --git a/openhcs/processing/presets/pipelines/10x_mfd_crop_analyze.py b/openhcs/processing/presets/pipelines/10x_mfd_crop_analyze.py index f8bea827c8..0a06ef6d5f 100644 --- a/openhcs/processing/presets/pipelines/10x_mfd_crop_analyze.py +++ b/openhcs/processing/presets/pipelines/10x_mfd_crop_analyze.py @@ -1,5 +1,5 @@ # Edit this pipeline and save to apply changes -from openhcs.processing.presets.mfd_specs import MfdPresetKey, build_mfd_preset +from openhcs.processing.presets.mfd_specs import MFD_CROP_ANALYZE, build_mfd_preset -pipeline_steps = build_mfd_preset(MfdPresetKey.CROP_ANALYZE) +pipeline_steps = build_mfd_preset(MFD_CROP_ANALYZE) diff --git a/openhcs/processing/presets/pipelines/10x_mfd_crop_analyze_dapi-fitc-cy5.py b/openhcs/processing/presets/pipelines/10x_mfd_crop_analyze_dapi-fitc-cy5.py index 78ad1b0aa0..34ebd44304 100644 --- a/openhcs/processing/presets/pipelines/10x_mfd_crop_analyze_dapi-fitc-cy5.py +++ b/openhcs/processing/presets/pipelines/10x_mfd_crop_analyze_dapi-fitc-cy5.py @@ -1,5 +1,5 @@ # Edit this pipeline and save to apply changes -from openhcs.processing.presets.mfd_specs import MfdPresetKey, build_mfd_preset +from openhcs.processing.presets.mfd_specs import MFD_CROP_ANALYZE_DAPI_FITC_CY5, build_mfd_preset -pipeline_steps = build_mfd_preset(MfdPresetKey.CROP_ANALYZE_DAPI_FITC_CY5) +pipeline_steps = build_mfd_preset(MFD_CROP_ANALYZE_DAPI_FITC_CY5) diff --git a/openhcs/processing/presets/pipelines/10x_mfd_stitch_ashlar_cpu.py b/openhcs/processing/presets/pipelines/10x_mfd_stitch_ashlar_cpu.py index b605b9b0e7..ef4332d367 100644 --- a/openhcs/processing/presets/pipelines/10x_mfd_stitch_ashlar_cpu.py +++ b/openhcs/processing/presets/pipelines/10x_mfd_stitch_ashlar_cpu.py @@ -1,5 +1,5 @@ # Edit this pipeline and save to apply changes -from openhcs.processing.presets.mfd_specs import MfdPresetKey, build_mfd_preset +from openhcs.processing.presets.mfd_specs import MFD_STITCH_ASHLAR_CPU, build_mfd_preset -pipeline_steps = build_mfd_preset(MfdPresetKey.STITCH_ASHLAR_CPU) +pipeline_steps = build_mfd_preset(MFD_STITCH_ASHLAR_CPU) diff --git a/openhcs/processing/presets/pipelines/10x_mfd_stitch_gpu.py b/openhcs/processing/presets/pipelines/10x_mfd_stitch_gpu.py index ab86eb0cf5..1f300754ee 100644 --- a/openhcs/processing/presets/pipelines/10x_mfd_stitch_gpu.py +++ b/openhcs/processing/presets/pipelines/10x_mfd_stitch_gpu.py @@ -1,5 +1,5 @@ # Edit this pipeline and save to apply changes -from openhcs.processing.presets.mfd_specs import MfdPresetKey, build_mfd_preset +from openhcs.processing.presets.mfd_specs import MFD_STITCH_GPU, build_mfd_preset -pipeline_steps = build_mfd_preset(MfdPresetKey.STITCH_GPU) +pipeline_steps = build_mfd_preset(MFD_STITCH_GPU) diff --git a/tests/unit/agent/test_compile_selector_authoring.py b/tests/unit/agent/test_compile_selector_authoring.py index a937944aa3..492add53bd 100644 --- a/tests/unit/agent/test_compile_selector_authoring.py +++ b/tests/unit/agent/test_compile_selector_authoring.py @@ -38,7 +38,7 @@ from openhcs.processing.backends.cellprofiler.intensity import measure_object_intensity from openhcs.processing.backends.lib_registry.registry_service import RegistryService from openhcs.processing.custom_functions.runtime_registry import ( - CustomFunctionLifetime, + EphemeralLifetime, CustomFunctionRuntimeRegistry, project_custom_function, ) @@ -288,7 +288,7 @@ class NewAuthoringModule(CellProfilerModule, ExtraOutputSettings): ) < NewAuthoringModule.__mro__.index(CellProfilerModuleSettings) metadata = project_custom_function(new_authoring_callable) CustomFunctionRuntimeRegistry.publish( - metadata, CustomFunctionLifetime.EPHEMERAL + metadata, EphemeralLifetime ) monkeypatch.setitem( RegistryService._metadata_cache, metadata.composite_key, metadata diff --git a/tests/unit/agent/test_knowledge_base_service.py b/tests/unit/agent/test_knowledge_base_service.py index 0274bb487c..f636461b22 100644 --- a/tests/unit/agent/test_knowledge_base_service.py +++ b/tests/unit/agent/test_knowledge_base_service.py @@ -446,7 +446,7 @@ def test_example_corpus_document_exposes_native_python_source(): assert "from openhcs.core.steps.function_step import FunctionStep" in ( source.content ) - assert "class MfdPresetKey" in source.content + assert "class MfdPresetDefinition" in source.content assert "def build_mfd_preset" in source.content @@ -530,7 +530,7 @@ def test_knowledge_base_search_covers_native_python_source(): result = service.search( KnowledgeBaseSearchRequest( - query="MfdPresetKey build_mfd_preset FunctionStep", + query="MfdPresetDefinition build_mfd_preset FunctionStep", limit=5, ) ) diff --git a/tests/unit/agent/test_knowledge_lazy_conversion_boundary.py b/tests/unit/agent/test_knowledge_lazy_conversion_boundary.py index 7a22b4c4ca..5367f7e8aa 100644 --- a/tests/unit/agent/test_knowledge_lazy_conversion_boundary.py +++ b/tests/unit/agent/test_knowledge_lazy_conversion_boundary.py @@ -8,7 +8,7 @@ from openhcs.interop.cellprofiler.module_declarations import CellProfilerModule from openhcs.processing.backends.lib_registry.registry_service import RegistryService from openhcs.processing.custom_functions.runtime_registry import ( - CustomFunctionLifetime, + EphemeralLifetime, CustomFunctionRuntimeRegistry, project_custom_function, ) @@ -44,7 +44,7 @@ def independent_knowledge_source_probe(image): monkeypatch.setattr(RegistryService, "_metadata_cache", {}) metadata = project_custom_function(independent_knowledge_source_probe) - CustomFunctionRuntimeRegistry.publish(metadata, CustomFunctionLifetime.EPHEMERAL) + CustomFunctionRuntimeRegistry.publish(metadata, EphemeralLifetime) try: assert get_function(metadata.composite_key) is metadata.func with pytest.raises(KeyError, match="Unknown canonical function ID"): diff --git a/tests/unit/agent/test_public_callable_enum_ingress.py b/tests/unit/agent/test_public_callable_enum_ingress.py index d9b1515160..7d0bde4371 100644 --- a/tests/unit/agent/test_public_callable_enum_ingress.py +++ b/tests/unit/agent/test_public_callable_enum_ingress.py @@ -16,7 +16,7 @@ from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.processing.backends.lib_registry.registry_service import RegistryService from openhcs.processing.custom_functions.runtime_registry import ( - CustomFunctionLifetime, + EphemeralLifetime, CustomFunctionRuntimeRegistry, project_custom_function, ) @@ -62,7 +62,7 @@ def independent_request_projection(request: IndependentProjectionRequest): def author(monkeypatch, request): # Declarations outside the registry's modules are custom functions. metadata = project_custom_function(request.param) - CustomFunctionRuntimeRegistry.publish(metadata, CustomFunctionLifetime.EPHEMERAL) + CustomFunctionRuntimeRegistry.publish(metadata, EphemeralLifetime) function_id = metadata.composite_key monkeypatch.setattr(RegistryService, "_metadata_cache", {function_id: metadata}) monkeypatch.setattr(RegistryService, "_resolved_reference_callables", {}) diff --git a/tests/unit/test_mfd_preset_specs.py b/tests/unit/test_mfd_preset_specs.py index de3ac624ff..3c83db2701 100644 --- a/tests/unit/test_mfd_preset_specs.py +++ b/tests/unit/test_mfd_preset_specs.py @@ -36,7 +36,10 @@ ) from openhcs.processing.presets.mfd_specs import ( MFD_WHOLE_DEVICE_TEMPLATE_PATH, - MfdPresetKey, + MFD_CROP_ANALYZE, + MFD_CROP_ANALYZE_DAPI_FITC_CY5, + MFD_STITCH_ASHLAR_CPU, + MFD_STITCH_GPU, build_mfd_preset, ) from openhcs.domains.microscopy.axes import Microscopy @@ -80,7 +83,7 @@ def _normalize_call(): def test_crop_analyze_spec_matches_expected_step_contract(): - steps = build_mfd_preset(MfdPresetKey.CROP_ANALYZE) + steps = build_mfd_preset(MFD_CROP_ANALYZE) assert [step.name for step in steps] == [ "crop_device", @@ -142,8 +145,8 @@ def test_crop_analyze_spec_matches_expected_step_contract(): def test_crop_analyze_cy5_overlay_adds_channel_4_cell_count(): - base_steps = build_mfd_preset(MfdPresetKey.CROP_ANALYZE) - overlay_steps = build_mfd_preset(MfdPresetKey.CROP_ANALYZE_DAPI_FITC_CY5) + base_steps = build_mfd_preset(MFD_CROP_ANALYZE) + overlay_steps = build_mfd_preset(MFD_CROP_ANALYZE_DAPI_FITC_CY5) assert base_steps[0].func == overlay_steps[0].func assert base_steps[1].func == overlay_steps[1].func @@ -162,8 +165,8 @@ def test_crop_analyze_cy5_overlay_adds_channel_4_cell_count(): def test_stitch_specs_share_structure_and_vary_backend(): - cpu_steps = build_mfd_preset(MfdPresetKey.STITCH_ASHLAR_CPU) - gpu_steps = build_mfd_preset(MfdPresetKey.STITCH_GPU) + cpu_steps = build_mfd_preset(MFD_STITCH_ASHLAR_CPU) + gpu_steps = build_mfd_preset(MFD_STITCH_GPU) assert [step.name for step in cpu_steps] == [ "process", @@ -199,10 +202,10 @@ def test_stitch_specs_share_structure_and_vary_backend(): def test_preset_wrapper_files_materialize_pipeline_steps(): expected_builders = { - "10x_mfd_crop_analyze.py": MfdPresetKey.CROP_ANALYZE, - "10x_mfd_crop_analyze_dapi-fitc-cy5.py": MfdPresetKey.CROP_ANALYZE_DAPI_FITC_CY5, - "10x_mfd_stitch_ashlar_cpu.py": MfdPresetKey.STITCH_ASHLAR_CPU, - "10x_mfd_stitch_gpu.py": MfdPresetKey.STITCH_GPU, + "10x_mfd_crop_analyze.py": MFD_CROP_ANALYZE, + "10x_mfd_crop_analyze_dapi-fitc-cy5.py": MFD_CROP_ANALYZE_DAPI_FITC_CY5, + "10x_mfd_stitch_ashlar_cpu.py": MFD_STITCH_ASHLAR_CPU, + "10x_mfd_stitch_gpu.py": MFD_STITCH_GPU, } for filename, preset_key in expected_builders.items(): From 0fdc095fbe07d1a3fdf372a02ff2b9ea85720513 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:18:05 -0400 Subject: [PATCH 27/36] E1: input source is a family InputSource's two members were compared by every planner; ReadPreviousStep and ReadPipelineStart now declare whether the step reads the pipeline-start main flow, and the config field holds the class (RegisteredClassChoices). Wire spellings unchanged ('previous', 'start'). The steps.abstract re-export and a re-check of the declared field type are deleted. Co-Authored-By: Claude Opus 5.5 --- openhcs/constants/input_source.py | 127 +++++------------- openhcs/core/config.py | 17 +-- openhcs/core/invocation_artifacts.py | 13 +- openhcs/core/pipeline/artifact_planning.py | 3 +- .../pipeline/materialization_flag_planner.py | 4 +- openhcs/core/pipeline/path_planner.py | 5 +- openhcs/core/steps/abstract.py | 9 +- openhcs/demo/basic_pipeline.py | 4 +- openhcs/demo/synthetic_plate_pipeline.py | 4 +- .../module_artifact_declarations.py | 6 +- .../cellprofiler/module_declarations.py | 12 +- .../interop/cellprofiler/pipeline_import.py | 16 +-- openhcs/processing/presets/mfd_specs.py | 6 +- .../cy5_axon_cell_body_crop_analysis.py | 4 +- .../pipelines/czi_brain_axon_cellbody.py | 10 +- ..._96_well_neurite_outgrowth_pipeline_cpu.py | 4 +- ..._96_well_neurite_outgrowth_pipeline_gpu.py | 4 +- .../loose_operaphenix_neurite_outgrowth.py | 8 +- ...peraphenix_neurite_outgrowth_metaxpress.py | 4 +- ...uroncyto_ii_crossover_neurite_outgrowth.py | 8 +- openhcs/pyqt_gui/widgets/pipeline_editor.py | 2 +- scripts/mcp_assay_showcase.py | 14 +- scripts/mcp_thesis_demo_live.py | 4 +- .../test_cellprofiler_generated_pipeline.py | 4 +- tests/integration/test_main.py | 4 +- .../test_measurement_declaration_journey.py | 6 +- ...rimary_segmentation_diagnostics_journey.py | 4 +- ...est_source_schema_aggregate_publication.py | 4 +- .../test_source_store_code_mode_zmq.py | 4 +- .../test_artifact_declaration_step_context.py | 12 +- ...test_cellprofiler_artifact_declarations.py | 6 +- ...ellprofiler_combineobjects_source_scope.py | 4 +- .../test_cellprofiler_export_to_database.py | 8 +- ...llprofiler_generated_pipeline_execution.py | 4 +- tests/unit/test_cellprofiler_image_quality.py | 4 +- ...t_cellprofiler_module_processing_config.py | 4 +- .../unit/test_cellprofiler_pipeline_import.py | 19 ++- ...profiler_runtime_callable_introspection.py | 30 ++--- tests/unit/test_compilation_session.py | 12 +- .../unit/test_czi_brain_axon_cellbody_demo.py | 9 +- ...test_function_runtime_source_projection.py | 4 +- .../test_implicit_main_flow_compilation.py | 6 +- .../unit/test_invocation_contract_provider.py | 28 ++-- ..._operaphenix_neurite_metaxpress_example.py | 3 +- ...test_materialization_filter_inheritance.py | 4 +- .../unit/test_materialization_flag_planner.py | 4 +- tests/unit/test_mfd_preset_specs.py | 5 +- tests/unit/test_neuroncyto_ii_neurite_demo.py | 7 +- .../unit/test_path_planner_materialization.py | 52 +++---- .../unit/test_resolved_pipeline_definition.py | 6 +- tests/unit/test_routed_channel_selection.py | 6 +- tests/unit/test_routed_composite_execution.py | 4 +- tests/unit/test_source_bindings.py | 6 +- tests/unit/test_source_tile_geometry.py | 7 +- 54 files changed, 245 insertions(+), 323 deletions(-) diff --git a/openhcs/constants/input_source.py b/openhcs/constants/input_source.py index d62c12c688..fead3ed8a5 100644 --- a/openhcs/constants/input_source.py +++ b/openhcs/constants/input_source.py @@ -1,106 +1,49 @@ +"""Where a pipeline step's main flow comes from. + +A step reads either the main-flow result of the previous step or the +pipeline-start main flow (for example to compute tile positions from the +original images). Separately named images or other typed values are artifact +inputs, satisfied through step source bindings or prior artifact producers; they +are not another input source. + +```python +step = FunctionStep( + func=ashlar_compute_tile_positions_gpu, + name="compute_positions", + processing_config=LazyProcessingConfig(input_source=ReadPipelineStart), +) +``` """ -Input Source Strategy Enum for OpenHCS. -This module defines the InputSource enum for explicit input source declaration -in pipeline steps, replacing the @chain_breaker decorator system with a cleaner, -more declarative approach. +from __future__ import annotations -Doctrinal Clauses: -- Clause 3 — Declarative Primacy -- Clause 88 — No Inferred Capabilities -- Clause 245 — Declarative Enforcement -""" - -from enum import Enum - - -class InputSource(Enum): - """ - Main-flow source strategies for pipeline steps. +from abc import ABC +from typing import ClassVar - This enum replaces the @chain_breaker decorator system with explicit - input source declaration, providing cleaner and more predictable - pipeline behavior. +from metaclass_registry import AutoRegisterMeta - The InputSource enum supports two strategies: - 1. **PREVIOUS_STEP** (Default): Standard pipeline chaining where each step - receives the ordinary main-flow result of the previous step. +class InputSource(ABC, metaclass=AutoRegisterMeta): + """Main-flow source of one step; ``name`` is its spelling in forms and on the wire.""" - 2. **PIPELINE_START**: The step receives the pipeline-start main-flow input, - bypassing previous step results. This replaces the @chain_breaker - decorator functionality. + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["InputSource"]]] = {} - The enum does not select separately named images or other typed values. A - callable declares those as artifact inputs, and compilation satisfies them - through step source bindings or prior artifact producers alongside the main - flow. + name: ClassVar[str | None] = None + reads_pipeline_start: ClassVar[bool] + """The step reads the pipeline-start main flow, not the previous step's.""" - Usage Examples: - Standard chaining (default behavior): - ```python - step = FunctionStep( - func=my_processing_function, - name="process_images", - processing_config=LazyProcessingConfig( - input_source=InputSource.PREVIOUS_STEP, - ), - ) - ``` +class ReadPreviousStep(InputSource): + """Standard chaining: the previous step's main-flow result.""" - Chain breaking for position generation: - ```python - step = FunctionStep( - func=ashlar_compute_tile_positions_gpu, - name="compute_positions", - processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, - ), - ) - ``` + name = "previous" + reads_pipeline_start = False - Quality control accessing original data: - ```python - step = FunctionStep( - func=quality_control_function, - name="qc_check", - processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, - ), - ) - ``` - """ - PREVIOUS_STEP = "previous" - """ - Standard pipeline chaining strategy. - - The step consumes the ordinary main-flow result of the previous step. This - is the default behavior and maintains normal pipeline chaining. - - This strategy: - - Maintains sequential data flow - - Enables progressive image processing - - Is the default for all steps - """ +class ReadPipelineStart(InputSource): + """The pipeline-start main flow, bypassing previous step results.""" - PIPELINE_START = "start" - """ - Pipeline start input strategy (replaces @chain_breaker). - - The step consumes the pipeline-start main-flow input, bypassing all previous - step results. This is equivalent to the @chain_breaker behavior but declared - explicitly. - - This strategy: - - Accesses original input data - - Bypasses all previous processing steps - - Is required for position generation and quality control - - Common use cases: - - Position generation functions (MIST, Ashlar) - - Quality control and validation steps - - Analysis requiring original image data - - Debugging and comparison operations - """ + name = "start" + reads_pipeline_start = True diff --git a/openhcs/core/config.py b/openhcs/core/config.py index cd622b3434..9aabbd42e7 100644 --- a/openhcs/core/config.py +++ b/openhcs/core/config.py @@ -37,7 +37,7 @@ TimeAxis, ) from metaclass_registry import AutoRegisterMeta -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import InputSource, ReadPreviousStep from python_introspect import ( AnnotatedDataclassValidationMixin, Enableable, @@ -580,9 +580,9 @@ class ProcessingConfig(AnnotatedDataclassValidationMixin): selection. This field does not define the assembled stack axis. """ - input_source: Annotated[InputSource, abbreviation("source")] = ( - InputSource.PREVIOUS_STEP - ) + input_source: Annotated[ + type[InputSource], abbreviation("source"), RegisteredClassChoices(InputSource) + ] = ReadPreviousStep """Main-flow source: previous step or pipeline start. Additional named sources are callable artifact inputs satisfied through @@ -997,7 +997,7 @@ class StepSourceBindingsConfig( enabled: bool = False """Whether this step uses source-binding resolution instead of the prior step image stack.""" - def for_input_source(self, input_source: InputSource) -> Self: + def for_input_source(self, input_source: type[InputSource]) -> Self: """Return the source declarations active for one resolved input source.""" if self.enabled is None: @@ -1005,12 +1005,7 @@ def for_input_source(self, input_source: InputSource) -> Self: "StepSourceBindingsConfig.for_input_source requires " "ObjectState-resolved enabled state." ) - resolved_input_source = ( - input_source - if isinstance(input_source, InputSource) - else InputSource(input_source) - ) - if self.enabled or resolved_input_source is InputSource.PIPELINE_START: + if self.enabled or input_source.reads_pipeline_start: return self return type(self)() diff --git a/openhcs/core/invocation_artifacts.py b/openhcs/core/invocation_artifacts.py index be2f0e13da..69169f5262 100644 --- a/openhcs/core/invocation_artifacts.py +++ b/openhcs/core/invocation_artifacts.py @@ -9,7 +9,7 @@ from metaclass_registry import AutoRegisterMeta -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import InputSource, ReadPreviousStep from openhcs.core.artifact_key_selection import ArtifactPlanKeySelector from openhcs.core.artifacts import ( ArtifactInputPlan, @@ -68,7 +68,7 @@ class ArtifactDeclarationStepContext: step_index: int | None = None source_bindings: StepSourceBindingsConfig = EMPTY_SOURCE_BINDINGS group_by: type[GroupingDeclaration] = Ungrouped - input_source: InputSource = InputSource.PREVIOUS_STEP + input_source: type[InputSource] = ReadPreviousStep available_artifacts: ArtifactSpecCollection = field( default_factory=lambda: ArtifactSpecCollection(()) ) @@ -99,11 +99,6 @@ def __post_init__(self) -> None: f"Ungrouped, got {self.group_by!r}." ) - if not isinstance(self.input_source, InputSource): - raise TypeError( - "ArtifactDeclarationStepContext.input_source must be InputSource, got " - f"{type(self.input_source).__name__}." - ) for field_name, collection in ( ("available_artifacts", self.available_artifacts), ("main_flow_artifacts", self.main_flow_artifacts), @@ -150,7 +145,7 @@ def with_source_declarations( binding.input_spec().ref() for binding in self.source_bindings.primary_plane_bindings ) - if self.input_source is InputSource.PIPELINE_START or ( + if self.input_source.reads_pipeline_start or ( self.step_index == 0 and not main_flow_artifacts ): main_flow_artifacts = ArtifactSpecCollection( @@ -212,7 +207,7 @@ def with_source_binding_scope( *, source_bindings: StepSourceBindingsConfig, group_by: type[GroupingDeclaration], - input_source: InputSource, + input_source: type[InputSource], source_groups: tuple[str | None, ...] = (), ) -> "ArtifactDeclarationStepContext": """Apply one resolved step's source scope and declared source artifacts.""" diff --git a/openhcs/core/pipeline/artifact_planning.py b/openhcs/core/pipeline/artifact_planning.py index d9af3216f5..c609b33847 100644 --- a/openhcs/core/pipeline/artifact_planning.py +++ b/openhcs/core/pipeline/artifact_planning.py @@ -46,7 +46,6 @@ TerminalMaterializationSpec, ) -from openhcs.constants.input_source import InputSource from openhcs.core.callable_contract import FunctionStepExecutionScope from openhcs.core.source_bindings import CompiledSourceBindingPlan from openhcs.core.config import StepSourceBindingsConfig @@ -281,7 +280,7 @@ def resolve_main_input_dependency( if ( step_index == 0 - or step.processing_config.input_source == InputSource.PIPELINE_START + or step.processing_config.input_source.reads_pipeline_start ): return PipelineStartInput() diff --git a/openhcs/core/pipeline/materialization_flag_planner.py b/openhcs/core/pipeline/materialization_flag_planner.py index cb4abba576..af4e4b0b8e 100644 --- a/openhcs/core/pipeline/materialization_flag_planner.py +++ b/openhcs/core/pipeline/materialization_flag_planner.py @@ -94,9 +94,7 @@ def prepare_pipeline_flags( if i == 0: step_plan.read_backend = self.resolve_backend(vfs_config.read_backend) elif step_plan.read_backend is None: - from openhcs.core.steps.abstract import InputSource - - if step.processing_config.input_source == InputSource.PIPELINE_START: + if step.processing_config.input_source.reads_pipeline_start: if step_plans[0].input_conversion is not None: step_plan.read_backend = Backend.ZARR.value step_plan.input_dir = step_plans[0].input_conversion.output_dir diff --git a/openhcs/core/pipeline/path_planner.py b/openhcs/core/pipeline/path_planner.py index 4cfa06b0a6..10c0a6b04c 100644 --- a/openhcs/core/pipeline/path_planner.py +++ b/openhcs/core/pipeline/path_planner.py @@ -14,7 +14,6 @@ from pathlib import Path from typing import Dict, Mapping, Optional -from openhcs.constants.input_source import InputSource from openhcs.core.axis_filter import StepAxisFilterSet from openhcs.core.artifacts import ( ArtifactInputProjectionPlan, @@ -519,7 +518,7 @@ def dynamic_execution_scope_for_group_by( resolved_config=self.planner.session.global_config, ) ) - if step.processing_config.input_source is InputSource.PIPELINE_START + if step.processing_config.input_source.reads_pipeline_start else () ) if source_keys: @@ -2275,7 +2274,7 @@ def step_io_dirs( @staticmethod def input_source(step: AbstractStep) -> str: """Get input source string.""" - if step.processing_config.input_source == InputSource.PIPELINE_START: + if step.processing_config.input_source.reads_pipeline_start: return "PIPELINE_START" return "PREVIOUS_STEP" diff --git a/openhcs/core/steps/abstract.py b/openhcs/core/steps/abstract.py index 2e891d1379..d5deb44bbf 100644 --- a/openhcs/core/steps/abstract.py +++ b/openhcs/core/steps/abstract.py @@ -16,7 +16,6 @@ from objectstate import get_base_type_for_lazy, semantic_values_equal -from openhcs.constants.input_source import InputSource as InputSource # Import LazyStepMaterializationConfig for type hints from openhcs.core.config import ( @@ -204,11 +203,11 @@ class AbstractStep(abc.ABC): ``processing_config.input_source`` controls the ordinary main-flow source: - - InputSource.PREVIOUS_STEP (default): Standard pipeline chaining where the step + - ReadPreviousStep (default): Standard pipeline chaining where the step consumes the previous step's main-flow result. This maintains normal sequential data flow. - - InputSource.PIPELINE_START: The step consumes pipeline-start main flow, + - ReadPipelineStart: The step consumes pipeline-start main flow, bypassing previous step results. This replaces the @chain_breaker decorator functionality and is used for position generation and quality control. @@ -224,7 +223,7 @@ class AbstractStep(abc.ABC): func=my_processing_function, name="process_images", processing_config=LazyProcessingConfig( - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ), ) ``` @@ -235,7 +234,7 @@ class AbstractStep(abc.ABC): func=ashlar_compute_tile_positions_gpu, name="compute_positions", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), ) ``` diff --git a/openhcs/demo/basic_pipeline.py b/openhcs/demo/basic_pipeline.py index e7857a4c93..60f473a472 100644 --- a/openhcs/demo/basic_pipeline.py +++ b/openhcs/demo/basic_pipeline.py @@ -7,7 +7,7 @@ # Edit this pipeline and save to apply changes # Automatically collected imports -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import ( LazyFijiStreamingConfig, LazyNapariStreamingConfig, @@ -72,7 +72,7 @@ step_5 = FunctionStep( func=(stack_percentile_normalize, {"low_percentile": 0.5, "high_percentile": 99.5}), name="Secondary Enhancement", - processing_config=LazyProcessingConfig(input_source=InputSource.PIPELINE_START), + processing_config=LazyProcessingConfig(input_source=ReadPipelineStart), ) pipeline_steps.append(step_5) diff --git a/openhcs/demo/synthetic_plate_pipeline.py b/openhcs/demo/synthetic_plate_pipeline.py index fc490eee95..7d7a8951d4 100644 --- a/openhcs/demo/synthetic_plate_pipeline.py +++ b/openhcs/demo/synthetic_plate_pipeline.py @@ -7,7 +7,7 @@ # Edit this pipeline and save to apply changes # Automatically collected imports -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import ( LazyNapariStreamingConfig, LazyProcessingConfig, @@ -71,7 +71,7 @@ step_5 = FunctionStep( func=(stack_percentile_normalize, {"low_percentile": 0.5, "high_percentile": 99.5}), name="Secondary Enhancement", - processing_config=LazyProcessingConfig(input_source=InputSource.PIPELINE_START), + processing_config=LazyProcessingConfig(input_source=ReadPipelineStart), ) pipeline_steps.append(step_5) diff --git a/openhcs/interop/cellprofiler/module_artifact_declarations.py b/openhcs/interop/cellprofiler/module_artifact_declarations.py index 29d976e04c..161be72020 100644 --- a/openhcs/interop/cellprofiler/module_artifact_declarations.py +++ b/openhcs/interop/cellprofiler/module_artifact_declarations.py @@ -12,7 +12,7 @@ ClassVar, ) -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import InputSource, ReadPipelineStart from openhcs.core.artifacts import ( ArtifactInputPlan, ArtifactOutputPlan, @@ -1234,7 +1234,7 @@ def input_source_for_contract( callable_contract: CallableContract, *, step_context: "ArtifactDeclarationStepContext", - ) -> InputSource: + ) -> type[InputSource]: """Keep original measurement images alongside produced image subjects.""" produced_refs = frozenset( producer.spec.ref().for_plan_type(ArtifactInputPlan) @@ -1245,7 +1245,7 @@ def input_source_for_contract( and step_context.source_bindings.binding_for_artifact_ref(spec.ref()) is not None for spec in callable_contract.artifact_inputs.of_artifact_type(ImageArtifactType) ): - return InputSource.PIPELINE_START + return ReadPipelineStart return super().input_source_for_contract( callable_contract, step_context=step_context, diff --git a/openhcs/interop/cellprofiler/module_declarations.py b/openhcs/interop/cellprofiler/module_declarations.py index 0f9f0b41d4..a35d8e1c93 100644 --- a/openhcs/interop/cellprofiler/module_declarations.py +++ b/openhcs/interop/cellprofiler/module_declarations.py @@ -19,7 +19,7 @@ from metaclass_registry import AutoRegisterMeta, LazyDiscoveryDict, RegistryConfig from openhcs.core.measurement_dialect import MeasurementDialect -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import InputSource, ReadPipelineStart, ReadPreviousStep from openhcs.core.artifacts import ArtifactSpec, ArtifactSpecRef from openhcs.core.callable_contract import ( CallableContract, @@ -937,7 +937,7 @@ def input_source_for_contract( callable_contract: CallableContract, *, step_context: ArtifactDeclarationStepContext, - ) -> InputSource: + ) -> type[InputSource]: """Resolve the main-flow anchor independently of auxiliary source inputs.""" produced_refs = frozenset( producer.spec.ref().for_plan_type(ArtifactInputPlan) @@ -948,13 +948,13 @@ def input_source_for_contract( and spec.ref() in produced_refs for spec in callable_contract.artifact_inputs ): - return InputSource.PREVIOUS_STEP + return ReadPreviousStep if any( step_context.source_bindings.binding_for_artifact_ref(spec.ref()) is not None for spec in callable_contract.artifact_inputs ): - return InputSource.PIPELINE_START - return InputSource.PREVIOUS_STEP + return ReadPipelineStart + return ReadPreviousStep @classmethod def processing_config( @@ -978,7 +978,7 @@ def processing_config( if callable_contract.execution_scope is FunctionStepExecutionScope.PLATE: variable_components: tuple[type[Axis], ...] = () group_by: type[GroupingDeclaration] | None = Ungrouped - input_source = InputSource.PREVIOUS_STEP + input_source = ReadPreviousStep else: family = AxisFamily.active() required_axes = tuple( diff --git a/openhcs/interop/cellprofiler/pipeline_import.py b/openhcs/interop/cellprofiler/pipeline_import.py index f9350de333..40a35d3c18 100644 --- a/openhcs/interop/cellprofiler/pipeline_import.py +++ b/openhcs/interop/cellprofiler/pipeline_import.py @@ -14,7 +14,7 @@ ) from openhcs.constants import Backend -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import InputSource, ReadPipelineStart, ReadPreviousStep from openhcs.core.artifacts import ( ArtifactInputPlan, ArtifactOutputPlan, @@ -335,7 +335,7 @@ def _parsed_pipeline_declaration( forward_context = ArtifactDeclarationStepContext().with_source_binding_scope( source_bindings=step_source_bindings, group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) target_units = _parsed_target_units( executable_modules, @@ -400,11 +400,11 @@ def _pipeline_processing_config( def _public_step_source_bindings( source_bindings: StepSourceBindingsConfig, artifact_inputs: Iterable[ArtifactSpec], - input_source: InputSource, + input_source: type[InputSource], ) -> StepSourceBindingsConfig: """Project only direct source inputs loaded beside the resolved main flow.""" - if input_source is not InputSource.PREVIOUS_STEP: + if input_source.reads_pipeline_start: return source_bindings direct_source_refs = tuple( spec.ref() @@ -894,12 +894,12 @@ def _lower_module_batch( source_bindings = step_context.source_bindings raw_callable = units[0].raw_callable input_source = ( - InputSource.PIPELINE_START + ReadPipelineStart if any( - unit.processing_config.input_source is InputSource.PIPELINE_START + unit.processing_config.input_source.reads_pipeline_start for unit in units ) - else InputSource.PREVIOUS_STEP + else ReadPreviousStep ) processing_config = replace(units[0].processing_config, input_source=input_source) if any( @@ -964,7 +964,7 @@ def _lower_module_batch( ) == Counter(source_group_keys) has_unscoped_single_invocation = len(units) == 1 and not lineage_keys[0] follows_previous_main_flow = ( - len(units) == 1 and processing_config.input_source is InputSource.PREVIOUS_STEP + len(units) == 1 and not processing_config.input_source.reads_pipeline_start ) public_pattern: FunctionPatternSyntax | None = None lowered_units = units diff --git a/openhcs/processing/presets/mfd_specs.py b/openhcs/processing/presets/mfd_specs.py index 717bb44cc1..6d8806ec1a 100644 --- a/openhcs/processing/presets/mfd_specs.py +++ b/openhcs/processing/presets/mfd_specs.py @@ -10,7 +10,7 @@ from arraybridge.decorators import DtypeConversion -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import InputSource, ReadPipelineStart from openhcs.core.config import LazyDtypeConfig, LazyProcessingConfig from openhcs.core.steps.function_step import FunctionStep from openhcs.processing.backends.analysis.cell_counting_cpu import ( @@ -56,7 +56,7 @@ class PresetStepBinding: name: str variable_components: tuple[Any, ...] = () - input_source: InputSource | None = None + input_source: type[InputSource] | None = None @dataclass(frozen=True, slots=True, kw_only=True) @@ -256,7 +256,7 @@ def materialize_spec(self, definition: MfdPresetDefinition) -> PresetStepSpec: PresetStepTemplate( binding=PresetStepBinding( name="process_2", - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), func_factory_factory=lambda definition: ( lambda: _channel_preprocess_func(include_channel_4=True) diff --git a/openhcs/processing/presets/pipelines/cy5_axon_cell_body_crop_analysis.py b/openhcs/processing/presets/pipelines/cy5_axon_cell_body_crop_analysis.py index d3f6bd5cf7..104d378370 100755 --- a/openhcs/processing/presets/pipelines/cy5_axon_cell_body_crop_analysis.py +++ b/openhcs/processing/presets/pipelines/cy5_axon_cell_body_crop_analysis.py @@ -3,7 +3,7 @@ # Automatically collected imports from pathlib import Path -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import LazyDtypeConfig, LazyStepMaterializationConfig from arraybridge.decorators import DtypeConversion from openhcs.core.steps.function_step import FunctionStep @@ -106,7 +106,7 @@ ), name="crop_device", variable_components=[Microscopy.Channel], - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) pipeline_steps.append(step_5) diff --git a/openhcs/processing/presets/pipelines/czi_brain_axon_cellbody.py b/openhcs/processing/presets/pipelines/czi_brain_axon_cellbody.py index c8710ae585..406beaa8e8 100644 --- a/openhcs/processing/presets/pipelines/czi_brain_axon_cellbody.py +++ b/openhcs/processing/presets/pipelines/czi_brain_axon_cellbody.py @@ -31,7 +31,7 @@ from polystore.filemanager import FileManager from polystore.streaming.identity import StreamProducerIdentity -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart, ReadPreviousStep from openhcs.core.artifacts import ArtifactSpec from openhcs.core.config import ( LazyNapariStreamingConfig, @@ -171,7 +171,7 @@ def build_czi_brain_axon_cellbody_demo( processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), ), FunctionStep( @@ -197,7 +197,7 @@ def build_czi_brain_axon_cellbody_demo( processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ), napari_streaming_config=_analysis_stream( inputs, @@ -218,7 +218,7 @@ def build_czi_brain_axon_cellbody_demo( processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=LazyStepSourceBindingsConfig( enabled=True, @@ -246,7 +246,7 @@ def build_czi_brain_axon_cellbody_demo( processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ), napari_streaming_config=_analysis_stream( inputs, diff --git a/openhcs/processing/presets/pipelines/imx_96_well_neurite_outgrowth_pipeline_cpu.py b/openhcs/processing/presets/pipelines/imx_96_well_neurite_outgrowth_pipeline_cpu.py index d0e0dc4f9a..0f51ca8e43 100644 --- a/openhcs/processing/presets/pipelines/imx_96_well_neurite_outgrowth_pipeline_cpu.py +++ b/openhcs/processing/presets/pipelines/imx_96_well_neurite_outgrowth_pipeline_cpu.py @@ -1,7 +1,7 @@ # Edit this pipeline and save to apply changes # Automatically collected imports -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import LazyNapariStreamingConfig from openhcs.core.steps.function_step import FunctionStep from openhcs.processing.backends.assemblers.assemble_stack_cupy import ( @@ -61,7 +61,7 @@ step_5 = FunctionStep( func=[stack_percentile_normalize_torch_processor, tophat], name="preprocess2", - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) pipeline_steps.append(step_5) diff --git a/openhcs/processing/presets/pipelines/imx_96_well_neurite_outgrowth_pipeline_gpu.py b/openhcs/processing/presets/pipelines/imx_96_well_neurite_outgrowth_pipeline_gpu.py index 20345c42c2..8973da25e4 100755 --- a/openhcs/processing/presets/pipelines/imx_96_well_neurite_outgrowth_pipeline_gpu.py +++ b/openhcs/processing/presets/pipelines/imx_96_well_neurite_outgrowth_pipeline_gpu.py @@ -1,7 +1,7 @@ # Edit this pipeline and save to apply changes # Automatically collected imports -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import LazyNapariStreamingConfig from openhcs.core.steps.function_step import FunctionStep from openhcs.processing.backends.assemblers.assemble_stack_cupy import ( @@ -61,7 +61,7 @@ step_5 = FunctionStep( func=[stack_percentile_normalize_torch_processor, tophat], name="preprocess2", - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) pipeline_steps.append(step_5) diff --git a/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth.py b/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth.py index b8fa040381..5ac4754662 100644 --- a/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth.py +++ b/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth.py @@ -24,7 +24,7 @@ from polystore.streaming.identity import StreamProducerIdentity from openhcs.core.dataset_sources.source_bindings_source import SourceBindingsSource -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.aligned_image_payload import AlignedImageSliceContext from openhcs.core.artifacts import ImageArtifactType, ObjectLabelsArtifactType from openhcs.core.config import ( @@ -230,7 +230,7 @@ def build_loose_operaphenix_neurite_pipeline( engine.body_detection_kwargs(), ), processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=_named_source(inputs.cell_body_source.alias), napari_streaming_config=_qc_stream(inputs, "viridis"), @@ -245,7 +245,7 @@ def build_loose_operaphenix_neurite_pipeline( }, ), processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=_named_source(inputs.smi312.alias), step_materialization_config=_qc_checkpoint( @@ -305,7 +305,7 @@ def build_loose_operaphenix_neurite_pipeline( engine.secondary_kwargs(), ), processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=_named_source(inputs.smi312.alias), napari_streaming_config=_qc_stream(inputs, "magma"), diff --git a/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth_metaxpress.py b/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth_metaxpress.py index f686248a98..9605f876b5 100644 --- a/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth_metaxpress.py +++ b/openhcs/processing/presets/pipelines/loose_operaphenix_neurite_outgrowth_metaxpress.py @@ -13,7 +13,7 @@ from pathlib import Path -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import ( LazyNapariStreamingConfig, LazyProcessingConfig, @@ -76,7 +76,7 @@ def build_loose_operaphenix_neurite_metaxpress_pipeline( processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), napari_streaming_config=LazyNapariStreamingConfig( enabled=True, diff --git a/openhcs/processing/presets/pipelines/neuroncyto_ii_crossover_neurite_outgrowth.py b/openhcs/processing/presets/pipelines/neuroncyto_ii_crossover_neurite_outgrowth.py index a1f5bc9b19..41cb1890bb 100644 --- a/openhcs/processing/presets/pipelines/neuroncyto_ii_crossover_neurite_outgrowth.py +++ b/openhcs/processing/presets/pipelines/neuroncyto_ii_crossover_neurite_outgrowth.py @@ -27,7 +27,7 @@ from polystore.streaming.identity import StreamProducerIdentity from openhcs.core.dataset_sources.source_bindings_source import SourceBindingsSource -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart, ReadPreviousStep from openhcs.core.aligned_image_payload import AlignedImageSliceContext from openhcs.core.config import ( LazyNapariStreamingConfig, @@ -168,7 +168,7 @@ def build_neuroncyto_ii_crossover_demo( processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), napari_streaming_config=LazyNapariStreamingConfig( enabled=True, @@ -267,7 +267,7 @@ def prepare() -> None: processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), napari_streaming_config=LazyNapariStreamingConfig( enabled=True, @@ -283,7 +283,7 @@ def prepare() -> None: processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ), napari_streaming_config=compact_step.napari_streaming_config, ) diff --git a/openhcs/pyqt_gui/widgets/pipeline_editor.py b/openhcs/pyqt_gui/widgets/pipeline_editor.py index 7f030ed519..cb99806d97 100644 --- a/openhcs/pyqt_gui/widgets/pipeline_editor.py +++ b/openhcs/pyqt_gui/widgets/pipeline_editor.py @@ -185,7 +185,7 @@ def group_by_line(self, group_by: type[GroupingDeclaration] | None) -> str: return "Group By: None" return f"Group By: {group_by.name}" - def input_source_line(self, input_source: InputSource) -> str: + def input_source_line(self, input_source: type[InputSource]) -> str: if not input_source: return "Input Source: None" return f"Input Source: {input_source.name}" diff --git a/scripts/mcp_assay_showcase.py b/scripts/mcp_assay_showcase.py index f534cc61b2..898394d982 100644 --- a/scripts/mcp_assay_showcase.py +++ b/scripts/mcp_assay_showcase.py @@ -390,7 +390,7 @@ def _common_source_header( from pathlib import Path -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import ( GlobalPipelineConfig, LazyPathPlanningConfig, @@ -490,7 +490,7 @@ def _pipeline_start_processing_config_source() -> str: """Render the ordinary pipeline-start processing declaration.""" return """ processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), """ @@ -499,7 +499,7 @@ def _z_stack_processing_config_source() -> str: """Render the pipeline-start declaration whose variable axis is Z.""" return """ processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, variable_components=[Microscopy.ZIndex], ), """ @@ -600,7 +600,7 @@ def _dual_channel_source(plate_path: Path, output_path: Path) -> str: processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), """ + _napari_streaming_config_source() @@ -653,7 +653,7 @@ def _colocalization_source(plate_path: Path, output_path: Path) -> str: processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), ), FunctionStep( @@ -667,7 +667,7 @@ def _colocalization_source(plate_path: Path, output_path: Path) -> str: processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), """ + _napari_streaming_config_source() @@ -862,7 +862,7 @@ def _foreground_skeleton_topology_source(plate_path: Path, output_path: Path) -> }, ), processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), """ + _napari_streaming_config_source() diff --git a/scripts/mcp_thesis_demo_live.py b/scripts/mcp_thesis_demo_live.py index 15204b301e..8676e262b5 100755 --- a/scripts/mcp_thesis_demo_live.py +++ b/scripts/mcp_thesis_demo_live.py @@ -1215,7 +1215,7 @@ def demo_source( from pathlib import Path from arraybridge.decorators import DtypeConversion -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.axes import Ungrouped from openhcs.core.config import ( GlobalPipelineConfig, @@ -1357,7 +1357,7 @@ def demo_source( }}), name='Secondary Enhancement', processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START + input_source=ReadPipelineStart ) ), FunctionStep( diff --git a/tests/integration/test_cellprofiler_generated_pipeline.py b/tests/integration/test_cellprofiler_generated_pipeline.py index 39ece7f465..3649b19750 100644 --- a/tests/integration/test_cellprofiler_generated_pipeline.py +++ b/tests/integration/test_cellprofiler_generated_pipeline.py @@ -32,7 +32,7 @@ resolve_benchmark_path_root, ) from openhcs.constants import Backend -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ( ArtifactType, ImageArtifactType, @@ -348,7 +348,7 @@ def test_invalid_public_cellprofiler_step_fails_during_zmq_compilation( func=align, name="Align", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), ) ] diff --git a/tests/integration/test_main.py b/tests/integration/test_main.py index 2868d3bd2f..73dd9c41de 100644 --- a/tests/integration/test_main.py +++ b/tests/integration/test_main.py @@ -17,7 +17,7 @@ from pathlib import Path from typing import Dict, List, Union -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import ( GlobalPipelineConfig, MaterializationBackend, @@ -317,7 +317,7 @@ def create_test_pipeline( ) ], processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START + input_source=ReadPipelineStart ), ), Step(name="CPU Assembly", func=assemble_stack_cpu), diff --git a/tests/integration/test_measurement_declaration_journey.py b/tests/integration/test_measurement_declaration_journey.py index e3be2ec474..8986aee880 100644 --- a/tests/integration/test_measurement_declaration_journey.py +++ b/tests/integration/test_measurement_declaration_journey.py @@ -16,7 +16,7 @@ CompileInspectionInput, InProcessCompileInspectionGateway, ) -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart, ReadPreviousStep from openhcs.core.artifacts import ( ArtifactInputPlan, ArtifactOutputPlan, @@ -203,7 +203,7 @@ def _step(func, name, kwargs): processing_config=LazyProcessingConfig( variable_components=[Microscopy.Site], group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=StepSourceBindingsConfig(enabled=True), ) @@ -324,7 +324,7 @@ def test_photometry_carrier_preserves_raw_aliases_and_produced_pixels( )) measurement = document.pipeline_steps[-1] measurement.processing_config = replace( - measurement.processing_config, input_source=InputSource.PREVIOUS_STEP, + measurement.processing_config, input_source=ReadPreviousStep, ) document = PipelineDocumentCodec.from_source(PipelineDocumentCodec.render(document)) bundle = _compile(tmp_path, document, GlobalPipelineConfig(num_workers=1, use_threading=True)) diff --git a/tests/integration/test_primary_segmentation_diagnostics_journey.py b/tests/integration/test_primary_segmentation_diagnostics_journey.py index 9e327bde58..e5cada6cd6 100644 --- a/tests/integration/test_primary_segmentation_diagnostics_journey.py +++ b/tests/integration/test_primary_segmentation_diagnostics_journey.py @@ -112,7 +112,7 @@ def test_normal_compiled_runtime_persists_diagnostics_and_preserves_secondary_bi CompileInspectionInput, InProcessCompileInspectionGateway, ) - from openhcs.constants.input_source import InputSource + from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ( ArtifactInputPlan, ArtifactOutputPlan, @@ -179,7 +179,7 @@ def step(func, name, kwargs): processing_config=LazyProcessingConfig( variable_components=[Microscopy.Site], group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=StepSourceBindingsConfig(enabled=True), ) diff --git a/tests/integration/test_source_schema_aggregate_publication.py b/tests/integration/test_source_schema_aggregate_publication.py index 43ea632652..d520223dc9 100644 --- a/tests/integration/test_source_schema_aggregate_publication.py +++ b/tests/integration/test_source_schema_aggregate_publication.py @@ -19,7 +19,7 @@ CompileInspectionInput, InProcessCompileInspectionGateway, ) -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ImageArtifactType, MainFlowStackOutputSpec, SpecialArtifactType from openhcs.core.config import ( GlobalPipelineConfig, @@ -98,7 +98,7 @@ def test_real_z_aggregation_persists_scalar_and_unchanged_source_planes(tmp_path processing_config=LazyProcessingConfig( variable_components=[Microscopy.ZIndex], group_by=group_by, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), )], ) diff --git a/tests/integration/test_source_store_code_mode_zmq.py b/tests/integration/test_source_store_code_mode_zmq.py index 22dc376b05..4956ed9579 100644 --- a/tests/integration/test_source_store_code_mode_zmq.py +++ b/tests/integration/test_source_store_code_mode_zmq.py @@ -17,7 +17,7 @@ ) from openhcs.constants.constants import Backend -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import ( GlobalPipelineConfig, LazyPathPlanningConfig, @@ -130,7 +130,7 @@ def _source_step() -> FunctionStep: name="Normalize named source store", func=stack_percentile_normalize, processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=LazyStepSourceBindingsConfig(enabled=True), ) diff --git a/tests/unit/test_artifact_declaration_step_context.py b/tests/unit/test_artifact_declaration_step_context.py index 0f7827927d..ca7e5a63b7 100644 --- a/tests/unit/test_artifact_declaration_step_context.py +++ b/tests/unit/test_artifact_declaration_step_context.py @@ -2,7 +2,7 @@ from dataclasses import fields -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart, ReadPreviousStep from openhcs.core.artifacts import ( ArtifactInputPlan, ArtifactSpec, @@ -30,7 +30,7 @@ def test_context_owns_exact_step_artifact_projection_facts() -> None: assert "processing_config" not in field_names assert ArtifactDeclarationStepContext.empty().group_by is Ungrouped assert ( - ArtifactDeclarationStepContext.empty().input_source is InputSource.PREVIOUS_STEP + not ArtifactDeclarationStepContext.empty().input_source.reads_pipeline_start ) @@ -43,7 +43,7 @@ def test_context_pipeline_start_sources_replace_main_flow() -> None: source_bindings=StepSourceBindingsConfig( bindings=(NamedSourceBinding(alias="DNA"),), ), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, main_flow_artifacts=ArtifactSpecCollection((upstream_image,)), ).with_source_declarations((source_image, measurement)) @@ -62,7 +62,7 @@ def test_context_pipeline_start_main_flow_uses_declared_source_subset() -> None: NamedSourceBinding(alias="Actin"), ), ), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ).with_source_declarations((dna,)) assert advanced.available_artifacts.specs == (dna,) @@ -78,7 +78,7 @@ def test_context_supplemental_step_sources_do_not_replace_previous_main_flow() - enabled=True, bindings=(NamedSourceBinding(alias="DNA"),), ), - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, main_flow_artifacts=ArtifactSpecCollection((upstream_image,)), ).with_source_declarations((source_image,)) @@ -91,7 +91,7 @@ def test_context_preserves_main_flow_for_previous_step_source_declarations() -> source_image = ArtifactSpec.input("DNA", ImageArtifactType) advanced = ArtifactDeclarationStepContext( - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, main_flow_artifacts=ArtifactSpecCollection((prior_image,)), ).with_source_declarations((source_image,)) diff --git a/tests/unit/test_cellprofiler_artifact_declarations.py b/tests/unit/test_cellprofiler_artifact_declarations.py index dd2e3a4b42..ba488b6779 100644 --- a/tests/unit/test_cellprofiler_artifact_declarations.py +++ b/tests/unit/test_cellprofiler_artifact_declarations.py @@ -13,7 +13,7 @@ from objectstate.lazy_factory import ensure_global_config_context from objectstate.object_state import ObjectState from objectstate.object_state_registry import ObjectStateRegistry -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ( ArtifactInputPlan, ArtifactOutputPlan, @@ -899,7 +899,7 @@ def test_group_artifact_context_reseeds_main_flow_from_exact_primary_bindings() context = ArtifactDeclarationStepContext( source_bindings=StepSourceBindingsConfig(enabled=True, bindings=bindings), group_by=Microscopy.Channel, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ).with_source_declarations(binding.input_spec() for binding in bindings) scoped = CellProfilerModule._artifact_context_for_group(context, group_key="2") @@ -994,7 +994,7 @@ def test_measure_image_quality_all_loaded_uses_only_primary_plane_bindings() -> enabled=True, bindings=bindings, ), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, available_artifacts=ArtifactSpecCollection(source_specs), main_flow_artifacts=ArtifactSpecCollection(source_specs), ), diff --git a/tests/unit/test_cellprofiler_combineobjects_source_scope.py b/tests/unit/test_cellprofiler_combineobjects_source_scope.py index 6ab1727561..78eac6590e 100644 --- a/tests/unit/test_cellprofiler_combineobjects_source_scope.py +++ b/tests/unit/test_cellprofiler_combineobjects_source_scope.py @@ -10,7 +10,7 @@ import tifffile from objectstate.lazy_factory import ensure_global_config_context -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ( ArtifactSpec, ArtifactSpecCollection, @@ -97,7 +97,7 @@ def combine(object_labels): processing_config=LazyProcessingConfig( variable_components=[Microscopy.Site], group_by=Microscopy.Channel, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=source_bindings, ) diff --git a/tests/unit/test_cellprofiler_export_to_database.py b/tests/unit/test_cellprofiler_export_to_database.py index f654d25e42..911051608f 100644 --- a/tests/unit/test_cellprofiler_export_to_database.py +++ b/tests/unit/test_cellprofiler_export_to_database.py @@ -13,7 +13,7 @@ import openhcs.serialization.pycodify_formatters # noqa: F401 from objectstate.lazy_factory import ensure_global_config_context -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ( ArtifactInputPlan, ObjectLabelsArtifactType, @@ -208,10 +208,10 @@ def test_registered_plate_scoped_modules_resolve_previous_step_input() -> None: for module_type, callable_contract in plate_declarations: resolved = module_type.processing_config( callable_contract=callable_contract, - inherited=ProcessingConfig(input_source=InputSource.PIPELINE_START), + inherited=ProcessingConfig(input_source=ReadPipelineStart), ) - assert resolved.input_source is InputSource.PREVIOUS_STEP + assert not resolved.input_source.reads_pipeline_start @pytest.mark.parametrize( @@ -450,7 +450,7 @@ def test_official30_database_pipelines_import_and_round_trip_public_steps( assert [step.name for step in steps].count("ExportToDatabase") == 1 with config_context(pipeline_config): export_step = next(step for step in steps if step.name == "ExportToDatabase") - assert export_step.processing_config.input_source is InputSource.PREVIOUS_STEP + assert not export_step.processing_config.input_source.reads_pipeline_start assert export_step.source_bindings.enabled is True assert export_step.source_bindings.binding_declarations diff --git a/tests/unit/test_cellprofiler_generated_pipeline_execution.py b/tests/unit/test_cellprofiler_generated_pipeline_execution.py index 860197ecc1..49dcecd85d 100644 --- a/tests/unit/test_cellprofiler_generated_pipeline_execution.py +++ b/tests/unit/test_cellprofiler_generated_pipeline_execution.py @@ -11,7 +11,7 @@ import tifffile from objectstate.lazy_factory import ensure_global_config_context -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ( ArtifactInputPlan, ArtifactOutputPlan, @@ -183,7 +183,7 @@ def test_compiler_derives_runtime_executor_after_generic_transport( ), name="CorrectIlluminationCalculate", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=StepSourceBindingsConfig(enabled=True), ) diff --git a/tests/unit/test_cellprofiler_image_quality.py b/tests/unit/test_cellprofiler_image_quality.py index 310a5799c0..124fced1cb 100644 --- a/tests/unit/test_cellprofiler_image_quality.py +++ b/tests/unit/test_cellprofiler_image_quality.py @@ -11,7 +11,7 @@ ) -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ( ArtifactSpec, ArtifactSpecCollection, @@ -217,7 +217,7 @@ def test_measure_image_quality_image_selection_resolves_from_its_typed_scope() - step_name=MeasureImageQualityModule.module_name, step_index=0, source_bindings=source_bindings, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, available_artifacts=source_specs, main_flow_artifacts=source_specs, ) diff --git a/tests/unit/test_cellprofiler_module_processing_config.py b/tests/unit/test_cellprofiler_module_processing_config.py index 7492ee072e..0c5b316d55 100644 --- a/tests/unit/test_cellprofiler_module_processing_config.py +++ b/tests/unit/test_cellprofiler_module_processing_config.py @@ -4,7 +4,7 @@ import pytest -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.callable_contract import CallableContract from openhcs.core.config import ProcessingConfig from openhcs.interop.cellprofiler.module_declarations import CellProfilerModule @@ -44,7 +44,7 @@ def test_processing_config_preserves_resolved_input_source() -> None: inherited = ProcessingConfig( variable_components=[Microscopy.Channel], group_by=Microscopy.Site, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) config = _processing_config( correct_illumination_apply, diff --git a/tests/unit/test_cellprofiler_pipeline_import.py b/tests/unit/test_cellprofiler_pipeline_import.py index c9aa0b2e1e..0431d6811d 100644 --- a/tests/unit/test_cellprofiler_pipeline_import.py +++ b/tests/unit/test_cellprofiler_pipeline_import.py @@ -11,7 +11,7 @@ from objectstate import config_context from openhcs.constants import Backend -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart, ReadPreviousStep from openhcs.core.artifacts import ( ArtifactInputPlan, ArtifactSpec, @@ -745,8 +745,7 @@ def test_direct_import_preserves_3d_image_variant_for_derived_input() -> None: Microscopy.ZIndex ] assert ( - pipeline_steps[1].processing_config.input_source - is InputSource.PREVIOUS_STEP + not pipeline_steps[1].processing_config.input_source.reads_pipeline_start ) remove_holes_invocation = next( normalize_function_pattern(pipeline_steps[1].func).iter_items() @@ -786,8 +785,7 @@ def test_direct_import_preserves_3d_object_variant_for_derived_input() -> None: ] with config_context(pipeline_config): assert ( - pipeline_steps[1].processing_config.input_source - is InputSource.PREVIOUS_STEP + not pipeline_steps[1].processing_config.input_source.reads_pipeline_start ) dilate_invocation = next( normalize_function_pattern(pipeline_steps[1].func).iter_items() @@ -1369,7 +1367,7 @@ def test_repeated_measurement_images_preserve_one_complete_module_roster() -> No assert invocations[0].kwargs_dict["select_images_to_measure"] == ("DNA", "PH3") with config_context(pipeline_config): assert measurement_step.processing_config.group_by is Ungrouped - assert measurement_step.processing_config.input_source is InputSource.PIPELINE_START + assert measurement_step.processing_config.input_source.reads_pipeline_start def test_measurement_rosters_preserve_exact_subjects_across_public_transport() -> None: @@ -1530,7 +1528,7 @@ def test_object_measurement_subjects_across_channels_share_one_invocation() -> N ) with config_context(config): assert measurement_step.processing_config.group_by is Ungrouped - assert measurement_step.processing_config.input_source is InputSource.PREVIOUS_STEP + assert not measurement_step.processing_config.input_source.reads_pipeline_start def test_mixed_source_and_produced_measurements_share_one_step() -> None: @@ -1597,8 +1595,7 @@ def test_mixed_source_and_produced_measurements_share_one_step() -> None: assert FunctionStepTransportAuthority.source_from_pipeline(reconstructed) == source with config_context(pipeline_config): assert ( - measurement_steps[0].processing_config.input_source - is InputSource.PIPELINE_START + measurement_steps[0].processing_config.input_source.reads_pipeline_start ) @@ -1620,7 +1617,7 @@ def test_multi_image_source_contract_keeps_every_source_input_exact() -> None: step_index=13, source_bindings=source_bindings, group_by=Microscopy.Channel, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ).with_source_declarations(binding.input_spec() for binding in bindings) module = ModuleBlock( name="ImageMath", @@ -1674,7 +1671,7 @@ def mixed_inputs(image: object) -> object: projected = _public_step_source_bindings( source_bindings, contract.artifact_inputs, - InputSource.PREVIOUS_STEP, + ReadPreviousStep, ) assert projected.enabled is True diff --git a/tests/unit/test_cellprofiler_runtime_callable_introspection.py b/tests/unit/test_cellprofiler_runtime_callable_introspection.py index e5611227b5..72d6986e25 100644 --- a/tests/unit/test_cellprofiler_runtime_callable_introspection.py +++ b/tests/unit/test_cellprofiler_runtime_callable_introspection.py @@ -8,7 +8,7 @@ import pytest -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.compiled_step_plan import CompiledStepPlan from openhcs.core.config import ( GlobalPipelineConfig, @@ -122,7 +122,7 @@ def test_imported_function_step_values_remain_signature_diffs_in_object_state(): debug_pause=True, processing_config=LazyProcessingConfig( variable_components=[Microscopy.Site], - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), ), scope_id="plate::functionstep_0", @@ -139,7 +139,7 @@ def test_imported_function_step_values_remain_signature_diffs_in_object_state(): ] assert state.signature_default("processing_config.variable_components") is None assert ( - state.parameters["processing_config.input_source"] is InputSource.PIPELINE_START + state.parameters["processing_config.input_source"].reads_pipeline_start ) assert state.signature_default("processing_config.input_source") is None assert { @@ -166,7 +166,7 @@ def test_step_invocation_contract_provider_validates_public_cellprofiler_config( ), name="MaskObjects", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=StepSourceBindingsConfig( enabled=True, @@ -205,7 +205,7 @@ def test_public_compile_time_provider_binds_cellprofiler_runtime_from_step_decla ), name="CorrectIlluminationCalculate", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=source_bindings, ) @@ -217,7 +217,7 @@ def test_public_compile_time_provider_binds_cellprofiler_runtime_from_step_decla step_context = ArtifactDeclarationStepContext( step_index=0, source_bindings=source_bindings, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) compiled = compile_function_pattern( @@ -275,7 +275,7 @@ def test_adapter_free_cellprofiler_module_keeps_raw_runtime_callable() -> None: ), name="SaveImages", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=source_bindings, ) @@ -289,7 +289,7 @@ def test_adapter_free_cellprofiler_module_keeps_raw_runtime_callable() -> None: ArtifactDeclarationStepContext( step_index=0, source_bindings=source_bindings, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), ) @@ -328,7 +328,7 @@ def test_public_compile_time_provider_uses_step_order_for_repeated_modules(): ), name="CorrectIlluminationCalculate", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=source_bindings, ), @@ -342,7 +342,7 @@ def test_public_compile_time_provider_uses_step_order_for_repeated_modules(): ), name="CorrectIlluminationCalculate", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=source_bindings, ), @@ -356,7 +356,7 @@ def test_public_compile_time_provider_uses_step_order_for_repeated_modules(): step_context = ArtifactDeclarationStepContext( step_index=1, source_bindings=source_bindings, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) compiled = compile_function_pattern( @@ -391,7 +391,7 @@ def test_cellprofiler_compile_time_contract_provider_derives_single_source_input ), name="CorrectIlluminationCalculate", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=source_bindings, ) @@ -440,7 +440,7 @@ def test_cellprofiler_compile_time_contract_provider_scopes_grouped_source_bindi }, name="CorrectIlluminationCalculate", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=source_bindings, ) @@ -496,7 +496,7 @@ def test_compile_time_provider_derives_group_contracts_from_public_pattern_after }, name="CorrectIlluminationCalculate", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=source_bindings, ) @@ -514,7 +514,7 @@ def test_compile_time_provider_derives_group_contracts_from_public_pattern_after step_context = ArtifactDeclarationStepContext( step_index=0, source_bindings=source_bindings, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) compiled = compile_function_pattern( module.pipeline_steps[0].func, diff --git a/tests/unit/test_compilation_session.py b/tests/unit/test_compilation_session.py index 1fa77c1583..dd014ba89f 100644 --- a/tests/unit/test_compilation_session.py +++ b/tests/unit/test_compilation_session.py @@ -8,7 +8,7 @@ from objectstate.object_state import ObjectState from objectstate.object_state_registry import ObjectStateRegistry -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import InputSource, ReadPipelineStart, ReadPreviousStep from openhcs.core.artifacts import ArtifactSpec, ImageArtifactType from openhcs.core.callable_contract import CallableContract from openhcs.core.compiled_step_plan import CompiledStepPlan @@ -268,7 +268,7 @@ def _resolved_step( step: FunctionStep, variable_components=(Microscopy.Site,), source_bindings=EMPTY_SOURCE_BINDINGS, - input_source: InputSource = InputSource.PREVIOUS_STEP, + input_source: InputSource = ReadPreviousStep, ) -> AbstractStep: step.source_bindings = source_bindings step.processing_config = ProcessingConfig( @@ -663,7 +663,7 @@ def test_path_planner_activates_declared_source_binding_for_pipeline_start(): snapshot = _resolved_step( step, source_bindings=StepSourceBindingsConfig(bindings=(binding,)), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) session = CompilationSession.from_context( context=_context(), @@ -702,7 +702,7 @@ def test_plate_export_contract_construction_projects_inputs_to_runtime_batch(mon enabled=True, bindings=(binding,), ), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) session = CompilationSession.from_context( context=_context(), @@ -768,7 +768,7 @@ def test_path_planner_preserves_pipeline_start_bindings_for_implicit_main_flow() snapshot = _resolved_step( step, source_bindings=StepSourceBindingsConfig(bindings=(binding,)), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) session = CompilationSession.from_context( context=_context(), @@ -853,7 +853,7 @@ def test_path_planner_preserves_metaxpress_primary_source_order( enabled=True, bindings=selected_bindings, ), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) pipeline_config = PipelineConfig( source_bindings_config=LazySourceBindingsConfig(bindings=pipeline_bindings) diff --git a/tests/unit/test_czi_brain_axon_cellbody_demo.py b/tests/unit/test_czi_brain_axon_cellbody_demo.py index 53f02de4c0..be815af224 100644 --- a/tests/unit/test_czi_brain_axon_cellbody_demo.py +++ b/tests/unit/test_czi_brain_axon_cellbody_demo.py @@ -6,7 +6,6 @@ from polystore.bioformats_storage import BioFormatsPlaneRef from polystore.virtual_workspace import SourcePixelRef -from openhcs.constants.input_source import InputSource from openhcs.core.artifacts import ImageArtifactType from openhcs.core.callable_contract import CallableContract from openhcs.core.function_patterns import get_core_callable @@ -168,23 +167,23 @@ def test_czi_demo_declares_bounded_three_channel_analysis_and_artifacts( assert steps[0].processing_config.variable_components == [ Microscopy.Channel ] - assert steps[0].processing_config.input_source is InputSource.PIPELINE_START + assert steps[0].processing_config.input_source.reads_pipeline_start assert steps[1].processing_config.variable_components == [ Microscopy.Channel ] - assert steps[1].processing_config.input_source is InputSource.PREVIOUS_STEP + assert not steps[1].processing_config.input_source.reads_pipeline_start assert steps[2].processing_config.group_by is Ungrouped assert steps[2].processing_config.variable_components == [ Microscopy.Channel ] - assert steps[2].processing_config.input_source is InputSource.PIPELINE_START + assert steps[2].processing_config.input_source.reads_pipeline_start assert steps[2].source_bindings.enabled is True assert [binding.alias for binding in steps[2].source_bindings.bindings] == ["Axon"] assert steps[3].processing_config.group_by is Ungrouped assert steps[3].processing_config.variable_components == [ Microscopy.Channel ] - assert steps[3].processing_config.input_source is InputSource.PREVIOUS_STEP + assert not steps[3].processing_config.input_source.reads_pipeline_start count_kwargs = steps[1].func[1] assert count_kwargs["cell_body"].channel_index == 1 diff --git a/tests/unit/test_function_runtime_source_projection.py b/tests/unit/test_function_runtime_source_projection.py index a395417c40..8433b1df56 100644 --- a/tests/unit/test_function_runtime_source_projection.py +++ b/tests/unit/test_function_runtime_source_projection.py @@ -3194,7 +3194,7 @@ def test_pipeline_start_main_flow_survives_prior_producer_image_input( from objectstate import ObjectStateRegistry from objectstate.lazy_factory import ensure_global_config_context - from openhcs.constants.input_source import InputSource + from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import ( LazyProcessingConfig, PipelineConfig, @@ -3225,7 +3225,7 @@ def test_pipeline_start_main_flow_survives_prior_producer_image_input( pipeline_start = LazyProcessingConfig( variable_components=[Microscopy.Site], group_by=Microscopy.Channel, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) steps = [ FunctionStep( diff --git a/tests/unit/test_implicit_main_flow_compilation.py b/tests/unit/test_implicit_main_flow_compilation.py index 11d80a7557..81a1bb09ec 100644 --- a/tests/unit/test_implicit_main_flow_compilation.py +++ b/tests/unit/test_implicit_main_flow_compilation.py @@ -7,7 +7,7 @@ import tifffile from objectstate.lazy_factory import ensure_global_config_context -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart, ReadPreviousStep from openhcs.core.callable_contract import FunctionStepExecutionScope from openhcs.core.config import ( GlobalPipelineConfig, @@ -45,12 +45,12 @@ def test_native_grouped_main_flow_compiles_into_cellprofiler_consumer(tmp_path): pipeline_start_processing = LazyProcessingConfig( variable_components=[Microscopy.Site], group_by=Microscopy.Channel, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) previous_step_processing = LazyProcessingConfig( variable_components=[Microscopy.Site], group_by=Microscopy.Channel, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) steps = [ FunctionStep( diff --git a/tests/unit/test_invocation_contract_provider.py b/tests/unit/test_invocation_contract_provider.py index bf2a06b571..8b11148198 100644 --- a/tests/unit/test_invocation_contract_provider.py +++ b/tests/unit/test_invocation_contract_provider.py @@ -637,7 +637,7 @@ def test_invocation_contract_plan_rejects_unwritten_consumed_kwarg_with_context( def test_cellprofiler_provider_reconstructs_exact_contract_from_public_step() -> None: - from openhcs.constants.input_source import InputSource + from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ImageArtifactType from openhcs.core.compiled_step_plan import CompiledStepPlan from openhcs.core.config import ( @@ -670,7 +670,7 @@ def test_cellprofiler_provider_reconstructs_exact_contract_from_public_step() -> ), name="ColorToGray", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=StepSourceBindingsConfig( enabled=True, @@ -843,7 +843,7 @@ def test_cellprofiler_provider_leaves_native_same_name_callable_unclaimed() -> N def test_cellprofiler_invocation_contract_uses_one_complete_step_context() -> None: - from openhcs.constants.input_source import InputSource + from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ( ArtifactSpec, ArtifactSpecCollection, @@ -881,7 +881,7 @@ def test_cellprofiler_invocation_contract_uses_one_complete_step_context() -> No step_index=3, source_bindings=source_bindings, group_by=Microscopy.Channel, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, available_artifacts=source_artifacts, main_flow_artifacts=source_artifacts, ) @@ -1007,7 +1007,7 @@ def test_grouped_contracts_project_prior_main_flow_by_producer_ownership() -> No def test_cellprofiler_invocation_contract_accepts_main_flow_input_without_binding() -> ( None ): - from openhcs.constants.input_source import InputSource + from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ( ArtifactSpec, ArtifactSpecCollection, @@ -1039,7 +1039,7 @@ def test_cellprofiler_invocation_contract_accepts_main_flow_input_without_bindin step_name="ColorToGray", step_index=7, source_bindings=source_bindings, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, available_artifacts=available_artifacts, main_flow_artifacts=available_artifacts, ) @@ -1057,7 +1057,7 @@ def test_cellprofiler_invocation_contract_accepts_main_flow_input_without_bindin def test_align_omission_probe_rejects_insufficient_static_source_without_throwing() -> ( None ): - from openhcs.constants.input_source import InputSource + from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ( ArtifactSpec, ArtifactSpecCollection, @@ -1083,7 +1083,7 @@ def test_align_omission_probe_rejects_insufficient_static_source_without_throwin step_name="Align", step_index=4, source_bindings=source_bindings, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, available_artifacts=source_artifacts, main_flow_artifacts=source_artifacts, ), @@ -1094,7 +1094,7 @@ def test_align_omission_probe_rejects_insufficient_static_source_without_throwin def test_cellprofiler_provider_rejects_under_specified_one_image_align() -> None: - from openhcs.constants.input_source import InputSource + from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ImageArtifactType from openhcs.core.compiled_step_plan import CompiledStepPlan from openhcs.core.config import ( @@ -1116,7 +1116,7 @@ def test_cellprofiler_provider_rejects_under_specified_one_image_align() -> None func=align, name="Align", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), source_bindings=StepSourceBindingsConfig( enabled=True, @@ -1157,7 +1157,7 @@ def test_cellprofiler_provider_rejects_under_specified_one_image_align() -> None def test_cellprofiler_invocation_contract_allows_source_binding_supersets() -> None: - from openhcs.constants.input_source import InputSource + from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.artifacts import ( ArtifactSpec, ArtifactSpecCollection, @@ -1197,7 +1197,7 @@ def test_cellprofiler_invocation_contract_allows_source_binding_supersets() -> N step_name="ColorToGray", step_index=8, source_bindings=source_bindings, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, available_artifacts=available_artifacts, main_flow_artifacts=available_artifacts, ) @@ -1339,7 +1339,7 @@ def test_calculate_math_provider_keeps_object_identity_and_output_name_public() def test_native_unnamed_main_flow_remains_a_canonical_contract_input() -> None: - from openhcs.constants.input_source import InputSource + from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.compiled_step_plan import CompiledStepPlan from openhcs.core.config import ( GlobalPipelineConfig, @@ -1412,7 +1412,7 @@ def session_for( func=(color_to_gray, {"mode": ColorToGrayMode.COMBINE}), name="ColorToGray", processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START + input_source=ReadPipelineStart ), ) ) diff --git a/tests/unit/test_loose_operaphenix_neurite_metaxpress_example.py b/tests/unit/test_loose_operaphenix_neurite_metaxpress_example.py index c5e264015f..e177d66ec8 100644 --- a/tests/unit/test_loose_operaphenix_neurite_metaxpress_example.py +++ b/tests/unit/test_loose_operaphenix_neurite_metaxpress_example.py @@ -9,7 +9,6 @@ from openhcs.agent.dto.knowledge import KnowledgeBaseDocumentRequest from openhcs.agent.services.knowledge_base_service import KnowledgeBaseService -from openhcs.constants.input_source import InputSource from openhcs.core.config import GlobalPipelineConfig from openhcs.core.function_patterns import get_core_callable from openhcs.core.orchestrator.orchestrator import PipelineOrchestrator @@ -92,7 +91,7 @@ def test_compact_example_uses_owned_channel_order_and_one_function_step( assert get_core_callable(step.func) is neurite_outgrowth_metaxpress assert step.processing_config.variable_components == [Microscopy.Channel] assert step.processing_config.group_by is Ungrouped - assert step.processing_config.input_source is InputSource.PIPELINE_START + assert step.processing_config.input_source.reads_pipeline_start assert step.napari_streaming_config.enabled is True assert step.napari_streaming_config.partition_filter is None assert step.napari_streaming_config.port == 5999 diff --git a/tests/unit/test_materialization_filter_inheritance.py b/tests/unit/test_materialization_filter_inheritance.py index 0ac8b1f6db..db172f3a6c 100644 --- a/tests/unit/test_materialization_filter_inheritance.py +++ b/tests/unit/test_materialization_filter_inheritance.py @@ -5,7 +5,7 @@ from objectstate.lazy_factory import ensure_global_config_context from objectstate.object_state_registry import ObjectStateRegistry from openhcs.constants.constants import Backend -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPreviousStep from openhcs.core.compiled_step_plan import ( CompiledStepPlan, MaterializedOutputPlan, @@ -124,7 +124,7 @@ def test_path_zero_keeps_inherited_step_checkpoint_independent(tmp_path) -> None [ SimpleNamespace( processing_config=SimpleNamespace( - input_source=InputSource.PREVIOUS_STEP + input_source=ReadPreviousStep ) ) ], diff --git a/tests/unit/test_materialization_flag_planner.py b/tests/unit/test_materialization_flag_planner.py index ab667f8fe3..8745543d2b 100644 --- a/tests/unit/test_materialization_flag_planner.py +++ b/tests/unit/test_materialization_flag_planner.py @@ -2,7 +2,7 @@ from types import SimpleNamespace from openhcs.constants.constants import Backend -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import InputSource, ReadPreviousStep from openhcs.core.artifacts import ( ArtifactOutputPlan, ObjectLabelsArtifactType, @@ -33,7 +33,7 @@ def _pipeline_config( ) -def _step(input_source: InputSource = InputSource.PREVIOUS_STEP) -> SimpleNamespace: +def _step(input_source: InputSource = ReadPreviousStep) -> SimpleNamespace: return SimpleNamespace(processing_config=SimpleNamespace(input_source=input_source)) diff --git a/tests/unit/test_mfd_preset_specs.py b/tests/unit/test_mfd_preset_specs.py index 3c83db2701..e657d0e1d4 100644 --- a/tests/unit/test_mfd_preset_specs.py +++ b/tests/unit/test_mfd_preset_specs.py @@ -3,7 +3,6 @@ from arraybridge.decorators import DtypeConversion -from openhcs.constants.input_source import InputSource from openhcs.core.config import LazyDtypeConfig from openhcs.processing.backends.analysis.cell_counting_cpu import ( DetectionMethod, @@ -188,8 +187,8 @@ def test_stitch_specs_share_structure_and_vary_backend(): ashlar_compute_tile_positions_gpu, {"stitch_alpha": 0.2}, ) - assert cpu_steps[3].processing_config.input_source == InputSource.PIPELINE_START - assert gpu_steps[3].processing_config.input_source == InputSource.PIPELINE_START + assert cpu_steps[3].processing_config.input_source.reads_pipeline_start + assert gpu_steps[3].processing_config.input_source.reads_pipeline_start assert cpu_steps[4].func is assemble_stack_cupy assert gpu_steps[4].func is assemble_stack_cupy assert cpu_steps[0].func["1"] == [ diff --git a/tests/unit/test_neuroncyto_ii_neurite_demo.py b/tests/unit/test_neuroncyto_ii_neurite_demo.py index 54e06edc97..3f654071ab 100644 --- a/tests/unit/test_neuroncyto_ii_neurite_demo.py +++ b/tests/unit/test_neuroncyto_ii_neurite_demo.py @@ -10,7 +10,6 @@ from objectstate import ObjectStateRegistry from objectstate.lazy_factory import ensure_global_config_context -from openhcs.constants.input_source import InputSource from openhcs.core.callable_contract import CallableContract from openhcs.core.artifacts import ArtifactViewerStreaming from openhcs.core.config import ( @@ -95,7 +94,7 @@ def test_neuroncyto_demo_declares_exact_crossover_channel_semantics( assert get_core_callable(step.func) is neurite_outgrowth_metaxpress_pixels assert step.processing_config.variable_components == [Microscopy.Channel] assert step.processing_config.group_by is Ungrouped - assert step.processing_config.input_source is InputSource.PIPELINE_START + assert step.processing_config.input_source.reads_pipeline_start assert step.napari_streaming_config.enabled is True assert step.napari_streaming_config.persistent is True assert step.napari_streaming_config.port == 5999 @@ -351,9 +350,9 @@ def test_neuroncyto_demo_contributor_prepares_only_declared_pair( assert len(contribution.pipeline_steps) == 2 contrast_step, analysis_step = contribution.pipeline_steps assert get_core_callable(contrast_step.func) is percentile_normalize_plane - assert contrast_step.processing_config.input_source is InputSource.PIPELINE_START + assert contrast_step.processing_config.input_source.reads_pipeline_start assert get_core_callable(analysis_step.func) is neurite_outgrowth_metaxpress_pixels - assert analysis_step.processing_config.input_source is InputSource.PREVIOUS_STEP + assert not analysis_step.processing_config.input_source.reads_pipeline_start assert contribution.presentation_identity.output_key == "neurite_morphology" assert contribution.presentation_identity.artifact_kind == "spatial_graph" assert contribution.presentation_identity.step_name == analysis_step.name diff --git a/tests/unit/test_path_planner_materialization.py b/tests/unit/test_path_planner_materialization.py index 9743ee7b25..2ae66707c9 100644 --- a/tests/unit/test_path_planner_materialization.py +++ b/tests/unit/test_path_planner_materialization.py @@ -7,7 +7,7 @@ import numpy as np from openhcs.processing.backends.cellprofiler.crop import CropMask -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import InputSource, ReadPipelineStart, ReadPreviousStep from openhcs.core.artifacts import ( ArtifactMeasurementSubjectRelation, ArtifactInputPlan, @@ -322,7 +322,7 @@ def _resolved_step( source_bindings: StepSourceBindingsConfig = EMPTY_SOURCE_BINDINGS, group_by: type[GroupingDeclaration] = Microscopy.Channel, variable_components: tuple[type[Axis], ...] = (Microscopy.Site,), - input_source: InputSource = InputSource.PREVIOUS_STEP, + input_source: InputSource = ReadPreviousStep, processing_config: ProcessingConfig | None = None, step_materialization_config=None, ) -> AbstractStep: @@ -624,7 +624,7 @@ def export_measurements(*, artifact_batch: RuntimeArtifactBatch): ), group_by=Ungrouped, variable_components=(), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) ( declarations, @@ -878,7 +878,7 @@ def produce_shared(): func=produce_shared, group_by=Ungrouped, variable_components=(), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) ( producer_declarations, @@ -1187,7 +1187,7 @@ def test_implicit_native_main_flow_provenance_drives_artifact_owned_scope(): processing_config = LazyProcessingConfig( variable_components=[Microscopy.Site], group_by=Microscopy.Channel, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) steps = ( FunctionStep( @@ -1257,7 +1257,7 @@ def test_implicit_native_main_flow_provenance_drives_artifact_owned_scope(): step_name=steps[1].name, step_index=1, group_by=Microscopy.Channel, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) planner.artifact_context = consumer_context consumer_invocation = next(normalize_function_pattern(steps[1].func).iter_items()) @@ -1507,7 +1507,7 @@ def test_enabled_source_bindings_preserve_previous_step_main_flow(): enabled=True, bindings=(NamedSourceBinding(alias="Input"),), ), - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, main_flow_artifacts=ArtifactSpecCollection( (ArtifactSpec.input("Previous", ImageArtifactType),) ), @@ -2680,7 +2680,7 @@ def measure_green(image, *, runtime): source_bindings=source_bindings, group_by=Microscopy.Channel, variable_components=(Microscopy.Site,), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) execution_scope = PathPlannerGroupScope.from_raw( ("1", "2"), @@ -3395,7 +3395,7 @@ def declared_artifact_owner(image): variable_components=(Microscopy.Site,), name="identify_cells", source_bindings=EMPTY_SOURCE_BINDINGS, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) pipeline = ResolvedPipelineDefinition( @@ -3455,7 +3455,7 @@ def filter_objects(image, *, runtime): variable_components=(Microscopy.Site,), name="FilterObjects", source_bindings=EMPTY_SOURCE_BINDINGS, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) input_component_scopes = PathPlannerComponentScopes( { @@ -3596,7 +3596,7 @@ def identify_primary_objects(*, runtime): variable_components=(Microscopy.Site,), name="IdentifyPrimaryObjects", source_bindings=EMPTY_SOURCE_BINDINGS, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) main_flow_scopes = PathPlannerComponentScopes( { @@ -3652,7 +3652,7 @@ def relate_objects(image, *, runtime): variable_components=(Microscopy.Site,), name="RelateObjects", source_bindings=EMPTY_SOURCE_BINDINGS, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) main_flow_scopes = PathPlannerComponentScopes( { @@ -3706,7 +3706,7 @@ def measure_object_size_shape(image, *, runtime): variable_components=(Microscopy.Site,), name="MeasureObjectSizeShape", source_bindings=EMPTY_SOURCE_BINDINGS, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) main_flow_scopes = PathPlannerComponentScopes( { @@ -3760,7 +3760,7 @@ def test_non_dict_group_by_declares_dynamic_scope_without_plate_key_lookup(): variable_components=(Microscopy.Site,), name="enhance", source_bindings=EMPTY_SOURCE_BINDINGS, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) source_scope = planner.execution_groups.get_execution_groups( @@ -3782,7 +3782,7 @@ def test_non_dict_group_by_uses_dynamic_source_scope_for_pipeline_start(): variable_components=(Microscopy.Site,), name="source_loaded_channel_callable", source_bindings=EMPTY_SOURCE_BINDINGS, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) source_scope = planner.execution_groups.get_execution_groups( @@ -3883,7 +3883,7 @@ def test_non_dict_group_by_preserves_explicitly_collapsed_input_axis(): variable_components=(Microscopy.Site,), name="measure_channel_named_artifacts_over_site_stack", source_bindings=EMPTY_SOURCE_BINDINGS, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) scope = planner.execution_groups.get_execution_groups(snapshot, input_scopes) @@ -4024,7 +4024,7 @@ def test_non_dict_group_by_namespaces_artifact_outputs_with_dynamic_component(): variable_components=(Microscopy.Site,), name="single_callable_channel_artifacts", source_bindings=EMPTY_SOURCE_BINDINGS, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) maps = planner.artifacts.compile_plan_maps( @@ -4065,7 +4065,7 @@ def test_non_dict_group_by_uses_source_binding_identity_for_pipeline_start_scope ), ), ), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) scope = planner.execution_groups.get_execution_groups( @@ -4127,7 +4127,7 @@ def save_declared_payload(image): source_bindings=StepSourceBindingsConfig( enabled=True, bindings=tuple(bindings) ), - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) scope = planner.execution_groups.get_execution_groups( @@ -4168,7 +4168,7 @@ def test_main_flow_source_anchor_restricts_execution_to_its_exact_channel(): variable_components=(Microscopy.Site,), name="bf_source_consumer", source_bindings=source_bindings, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) source_contract = CallableContract.from_callable(lambda image: image) source_contract = replace( @@ -4323,7 +4323,7 @@ def test_execution_anchor_ignores_source_artifact_lineage(): ), ) snapshot = _resolved_step( - source_bindings=source_bindings, input_source=InputSource.PIPELINE_START + source_bindings=source_bindings, input_source=ReadPipelineStart ) source_contract = CallableContract.from_callable(lambda image: image) source_contract = replace( @@ -4588,7 +4588,7 @@ def test_non_dict_group_by_ignores_source_binding_identity_for_other_components( ), ), ), - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) scope = planner.execution_groups.get_execution_groups( @@ -4613,7 +4613,7 @@ def test_compiled_group_by_preserves_dynamic_execution_scope(): group_by=Microscopy.Channel, variable_components=(Microscopy.Site,), name="measure_after_channel_collapse", - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ) artifact_maps = ArtifactPlanMaps( declarations=ArtifactGraph.empty(), @@ -5095,7 +5095,7 @@ def produce(image, source_value): name=f"produce_{output_name}", func=produce, source_bindings=source_bindings, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ) execution_scope = planner.execution_groups.get_execution_groups( snapshot, @@ -5795,7 +5795,7 @@ def test_main_input_dependency_uses_declared_artifact_producer_not_previous_step dependency = planner.steps.main_input_dependency( _resolved_step( - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, is_function_step=True, name="Identify", ), @@ -5910,7 +5910,7 @@ def test_main_input_dependency_preserves_pipeline_start_edges(): planner.steps = PathPlannerStepAssemblyStage(planner) dependency = planner.steps.main_input_dependency( - _resolved_step(input_source=InputSource.PIPELINE_START, is_function_step=False), + _resolved_step(input_source=ReadPipelineStart, is_function_step=False), 1, ) diff --git a/tests/unit/test_resolved_pipeline_definition.py b/tests/unit/test_resolved_pipeline_definition.py index f6493fa57f..e1b5bd2911 100644 --- a/tests/unit/test_resolved_pipeline_definition.py +++ b/tests/unit/test_resolved_pipeline_definition.py @@ -1,7 +1,7 @@ from openhcs.core.pipeline.compilation_session import ResolvedPipelineDefinition import pytest -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import ( ProcessingConfig, StepMaterializationConfig, @@ -39,7 +39,7 @@ def test_resolved_pipeline_reads_steps_without_object_conversion(): processing_config=ProcessingConfig( variable_components=[Microscopy.Site], group_by=None, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), step_materialization_config=StepMaterializationConfig(enabled=False), ) @@ -52,7 +52,7 @@ def test_resolved_pipeline_reads_steps_without_object_conversion(): assert pipeline.step_scope_ids[0] == scope assert pipeline.steps[0].source_bindings is source_bindings assert ( - pipeline.steps[0].processing_config.input_source is InputSource.PIPELINE_START + pipeline.steps[0].processing_config.input_source.reads_pipeline_start ) assert pipeline.steps[0].processing_config.variable_components == [ Microscopy.Site diff --git a/tests/unit/test_routed_channel_selection.py b/tests/unit/test_routed_channel_selection.py index ba516fd42f..cd855761e6 100644 --- a/tests/unit/test_routed_channel_selection.py +++ b/tests/unit/test_routed_channel_selection.py @@ -9,7 +9,7 @@ from objectstate import ObjectStateRegistry from objectstate.lazy_factory import ensure_global_config_context -from openhcs.constants import InputSource +from openhcs.constants.input_source import ReadPipelineStart, ReadPreviousStep from openhcs.core.config import ( GlobalPipelineConfig, LazyPathPlanningConfig, @@ -90,9 +90,9 @@ def test_grouped_consumer_requires_its_channel_in_previous_main_flow( func=identity, processing_config=LazyProcessingConfig( input_source=( - InputSource.PIPELINE_START + ReadPipelineStart if restart_from_source - else InputSource.PREVIOUS_STEP + else ReadPreviousStep ) ), ), diff --git a/tests/unit/test_routed_composite_execution.py b/tests/unit/test_routed_composite_execution.py index 5a31728f79..da447f8304 100644 --- a/tests/unit/test_routed_composite_execution.py +++ b/tests/unit/test_routed_composite_execution.py @@ -9,7 +9,7 @@ from objectstate import ObjectStateRegistry from objectstate.lazy_factory import ensure_global_config_context -from openhcs.constants import InputSource +from openhcs.constants.input_source import ReadPipelineStart from openhcs.core.config import ( GlobalPipelineConfig, LazyPathPlanningConfig, @@ -86,7 +86,7 @@ def test_reduced_channel_stack_is_the_next_steps_only_input( FunctionStep( func=identity, processing_config=LazyProcessingConfig( - input_source=InputSource.PIPELINE_START + input_source=ReadPipelineStart ), ), FunctionStep(func=assemble_stack_cpu), diff --git a/tests/unit/test_source_bindings.py b/tests/unit/test_source_bindings.py index 751225edce..22ef584190 100644 --- a/tests/unit/test_source_bindings.py +++ b/tests/unit/test_source_bindings.py @@ -12,7 +12,7 @@ from objectstate.object_state_registry import ObjectStateRegistry from polystore.virtual_workspace import SourcePixelRef -from openhcs.constants.input_source import InputSource +from openhcs.constants.input_source import ReadPipelineStart, ReadPreviousStep from openhcs.core import source_bindings as source_bindings_module from openhcs.core.artifacts import ( ArtifactInputPlan, @@ -537,9 +537,9 @@ def test_pipeline_start_binding_groups_do_not_mutate_resolved_enabled_state(): Microscopy.Channel, ) == ("1",) assert ( - source_bindings.for_input_source(InputSource.PIPELINE_START) is source_bindings + source_bindings.for_input_source(ReadPipelineStart) is source_bindings ) - assert source_bindings.for_input_source(InputSource.PREVIOUS_STEP).is_empty + assert source_bindings.for_input_source(ReadPreviousStep).is_empty def test_source_lineage_requires_its_declared_cursor_artifact() -> None: diff --git a/tests/unit/test_source_tile_geometry.py b/tests/unit/test_source_tile_geometry.py index 69f7db2b82..b643536d64 100644 --- a/tests/unit/test_source_tile_geometry.py +++ b/tests/unit/test_source_tile_geometry.py @@ -474,9 +474,10 @@ def test_synthetic_pipeline_compiles_and_executes_positions_artifact_then_paired from objectstate import ObjectStateRegistry from objectstate.lazy_factory import ensure_global_config_context + from openhcs.constants.input_source import ReadPipelineStart, ReadPreviousStep + from openhcs.core.config import ( GlobalPipelineConfig, - InputSource, LazyProcessingConfig, LazySourceBindingsConfig, LazyStepMaterializationConfig, @@ -572,7 +573,7 @@ def test_synthetic_pipeline_compiles_and_executes_positions_artifact_then_paired processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PREVIOUS_STEP, + input_source=ReadPreviousStep, ), ), ] @@ -684,7 +685,7 @@ def test_synthetic_pipeline_compiles_and_executes_positions_artifact_then_paired processing_config=LazyProcessingConfig( variable_components=[Microscopy.Channel], group_by=Ungrouped, - input_source=InputSource.PIPELINE_START, + input_source=ReadPipelineStart, ), ) ], From 3956bc8ad154844f65cdf964b38c84db4845a233 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:23:41 -0400 Subject: [PATCH 28/36] E1: materialized filename identity is a family Co-Authored-By: Claude Opus 5.5 --- openhcs/core/pipeline/artifact_planning.py | 8 +-- .../analysis/multi_template_matching.py | 6 +-- .../backends/analysis/neurite_outgrowth.py | 4 +- .../primary_object_diagnostics.py | 4 +- .../backends/cellprofiler/save_images.py | 7 +-- .../backends/enhance/basic_processor_jax.py | 4 +- .../pos_gen/tile_position_artifacts.py | 4 +- .../processing/materialization/__init__.py | 6 ++- openhcs/processing/materialization/core.py | 13 ++--- openhcs/processing/materialization/options.py | 50 +++++++++---------- ...artifact_output_materialization_planner.py | 8 +-- ...utomatic_image_role_materialization_435.py | 7 +-- tests/unit/test_cellprofiler_save_images.py | 4 +- ...test_collapsed_persisted_occurrence_435.py | 4 +- .../unit/test_derived_role_publication_435.py | 9 ++-- .../unit/test_file_materialization_options.py | 7 +-- .../test_function_artifact_materialization.py | 13 ++--- tests/unit/test_function_outputs.py | 6 +-- tests/unit/test_materialization_core.py | 15 +++--- tests/unit/test_multi_template_matching.py | 4 +- .../unit/test_named_checkpoint_publication.py | 5 +- tests/unit/test_primary_object_diagnostics.py | 4 +- tests/unit/test_tiff_output_compression.py | 4 +- 23 files changed, 103 insertions(+), 93 deletions(-) diff --git a/openhcs/core/pipeline/artifact_planning.py b/openhcs/core/pipeline/artifact_planning.py index c609b33847..42969bced9 100644 --- a/openhcs/core/pipeline/artifact_planning.py +++ b/openhcs/core/pipeline/artifact_planning.py @@ -41,7 +41,7 @@ from openhcs.processing.materialization import ( CsvOptions, ImageFileOptions, - MaterializedFilenameIdentity, + NameByArtifactName, ROIOptions, TerminalMaterializationSpec, ) @@ -102,11 +102,11 @@ def materialization(self) -> ArtifactMaterializationPayload: return TerminalMaterializationSpec( ROIOptions( min_area=1, - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ), ImageFileOptions( filename_suffix=".labels.tif", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ), ) @@ -124,7 +124,7 @@ class AutomaticMeasurementsArtifactOutputMaterializationStrategy( def materialization(self) -> ArtifactMaterializationPayload: return TerminalMaterializationSpec( CsvOptions( - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ) ) diff --git a/openhcs/processing/backends/analysis/multi_template_matching.py b/openhcs/processing/backends/analysis/multi_template_matching.py index eebac316f0..0ecfe50d67 100644 --- a/openhcs/processing/backends/analysis/multi_template_matching.py +++ b/openhcs/processing/backends/analysis/multi_template_matching.py @@ -32,7 +32,7 @@ MaterializationSpec, TiffStackOptions, ) -from openhcs.processing.materialization.options import MaterializedFilenameIdentity +from openhcs.processing.materialization.options import NameByArtifactName logger = logging.getLogger(__name__) @@ -135,7 +135,7 @@ class TemplateMatchResult: ImageArtifactType, materialization=MaterializationSpec( TiffStackOptions( - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME + filename_identity=NameByArtifactName ) ), ), @@ -145,7 +145,7 @@ class TemplateMatchResult: materialization=MaterializationSpec( CsvOptions( filename_suffix="_mtm_matches.csv", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, fields=[field.name for field in fields(TemplateMatchCsvRow)], row_unpacker=_mtm_row_unpacker, ) diff --git a/openhcs/processing/backends/analysis/neurite_outgrowth.py b/openhcs/processing/backends/analysis/neurite_outgrowth.py index 7f8e638841..7c8de3d1e6 100644 --- a/openhcs/processing/backends/analysis/neurite_outgrowth.py +++ b/openhcs/processing/backends/analysis/neurite_outgrowth.py @@ -62,7 +62,7 @@ CsvOptions, ImageFileOptions, MaterializationSpec, - MaterializedFilenameIdentity, + NameByArtifactName, ROIOptions, SpatialGraphROIOptions, SWCOptions, @@ -1088,7 +1088,7 @@ class NeuronalCellBodyResult: ROIOptions(), ImageFileOptions( filename_suffix=".labels.tif", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ), ) diff --git a/openhcs/processing/backends/cellprofiler/primary_object_diagnostics.py b/openhcs/processing/backends/cellprofiler/primary_object_diagnostics.py index 8c59ecf5b3..b03493ac01 100644 --- a/openhcs/processing/backends/cellprofiler/primary_object_diagnostics.py +++ b/openhcs/processing/backends/cellprofiler/primary_object_diagnostics.py @@ -33,7 +33,7 @@ from openhcs.processing.materialization import ( ImageFileOptions, TerminalMaterializationSpec, - MaterializedFilenameIdentity, + NameByArtifactName, ) @dataclass(frozen=True, slots=True) @@ -203,7 +203,7 @@ def artifact_specs( materialization=TerminalMaterializationSpec( ImageFileOptions( filename_suffix=".tif", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ) ), relations=( diff --git a/openhcs/processing/backends/cellprofiler/save_images.py b/openhcs/processing/backends/cellprofiler/save_images.py index a16e7776c2..e12d6a090d 100644 --- a/openhcs/processing/backends/cellprofiler/save_images.py +++ b/openhcs/processing/backends/cellprofiler/save_images.py @@ -66,7 +66,8 @@ ExecutionAxisMaterializationRelativePathScope, ImageFileOptions, MaterializationSpec, - MaterializedFilenameIdentity, + NameByArtifactName, + NameBySourceIdentity, WriteMode, ) from openhcs.core.runtime_image_values import ImagePayload @@ -886,9 +887,9 @@ def _materialization_spec( file_format=file_format, ) filename_identity = ( - MaterializedFilenameIdentity.SOURCE_IDENTITY + NameBySourceIdentity if filename_method is SaveImagesFilenameMethod.FROM_IMAGE_FILENAME - else MaterializedFilenameIdentity.ARTIFACT_NAME + else NameByArtifactName ) options = ImageFileOptions( filename_suffix=f"{suffix}{file_format.value}", diff --git a/openhcs/processing/backends/enhance/basic_processor_jax.py b/openhcs/processing/backends/enhance/basic_processor_jax.py index 6a747069ba..9a4103c01d 100644 --- a/openhcs/processing/backends/enhance/basic_processor_jax.py +++ b/openhcs/processing/backends/enhance/basic_processor_jax.py @@ -29,7 +29,7 @@ from openhcs.processing.materialization import ( ImageFileOptions, MaterializationSpec, - MaterializedFilenameIdentity, + NameByArtifactName, ) @@ -42,7 +42,7 @@ def _fitted_field_output(name: str) -> ArtifactSpec: materialization=MaterializationSpec( ImageFileOptions( filename_suffix=".tif", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ) ), viewer_streaming=ArtifactViewerStreaming.ON_DEMAND, diff --git a/openhcs/processing/backends/pos_gen/tile_position_artifacts.py b/openhcs/processing/backends/pos_gen/tile_position_artifacts.py index 370f74a56d..494072430a 100644 --- a/openhcs/processing/backends/pos_gen/tile_position_artifacts.py +++ b/openhcs/processing/backends/pos_gen/tile_position_artifacts.py @@ -9,13 +9,13 @@ from openhcs.processing.materialization import ( JsonOptions, MaterializationSpec, - MaterializedFilenameIdentity, + NameByArtifactName, ) TILE_POSITIONS_OUTPUT = ArtifactSpec.output( "positions", SpecialArtifactType, materialization=MaterializationSpec( - JsonOptions(filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME) + JsonOptions(filename_identity=NameByArtifactName) ), ) diff --git a/openhcs/processing/materialization/__init__.py b/openhcs/processing/materialization/__init__.py index ea0109b04e..2e41f35105 100644 --- a/openhcs/processing/materialization/__init__.py +++ b/openhcs/processing/materialization/__init__.py @@ -21,7 +21,8 @@ FileOutputOptions, ImageFileOptions, JsonOptions, - MaterializedFilenameIdentity, + NameByArtifactName, + NameBySourceIdentity, PointROIOptions, ROIOptions, SpatialGraphROIOptions, @@ -69,9 +70,10 @@ "FileOutputOptions", "ImageFileOptions", "FileBundleOptions", - "MaterializedFilenameIdentity", "CsvOptions", "JsonOptions", + "NameByArtifactName", + "NameBySourceIdentity", "PointROIOptions", "ROIOptions", "SpatialGraphROIOptions", diff --git a/openhcs/processing/materialization/core.py b/openhcs/processing/materialization/core.py index 80a45f9d4b..42d6a1727a 100644 --- a/openhcs/processing/materialization/core.py +++ b/openhcs/processing/materialization/core.py @@ -91,7 +91,8 @@ FileOutputOptions, ImageFileOptions, JsonOptions, - MaterializedFilenameIdentity, + NameByArtifactName, + NameBySourceIdentity, PointROIOptions, ROIOptions, SourceOptions, @@ -2075,7 +2076,7 @@ def source_identity_base_path(self, options: FileOutputOptions) -> Path | None: """Return the source-identity base path when it already names this output.""" suffix = options.primary_output_suffix if ( - options.filename_identity is MaterializedFilenameIdentity.SOURCE_IDENTITY + options.filename_identity is NameBySourceIdentity and suffix and self.base_path.name.endswith(suffix) ): @@ -3309,7 +3310,7 @@ def write_image_file( paths = context.paths(options) source_stem_authority = SourceStemAuthority.from_processing_context(context.context) preserves_planned_path = ( - options.filename_identity is MaterializedFilenameIdentity.ARTIFACT_NAME + options.filename_identity is NameByArtifactName or paths.source_identity_base_path(options) is not None ) outputs = [] @@ -4217,7 +4218,7 @@ def image_file_emits_variable_component_planes( data: MaterializationValue, ) -> bool: if not ( - options.filename_identity is MaterializedFilenameIdentity.SOURCE_IDENTITY + options.filename_identity is NameBySourceIdentity or _image_template_uses_source_metadata(options.relative_path_template) or _image_template_uses_sequence_index(options.relative_path_template) ): @@ -4261,7 +4262,7 @@ def image_file_uses_filename_source_identity( data: MaterializationValue, ) -> bool: del data - return options.filename_identity is MaterializedFilenameIdentity.SOURCE_IDENTITY + return options.filename_identity is NameBySourceIdentity @writer_for( @@ -4891,7 +4892,7 @@ def uses_source_identity_filename(self) -> bool: """Return whether this spec materializes files by source identity.""" return any( output_options.filename_identity - is MaterializedFilenameIdentity.SOURCE_IDENTITY + is NameBySourceIdentity for output_options in self.outputs ) diff --git a/openhcs/processing/materialization/options.py b/openhcs/processing/materialization/options.py index 4691e5a840..aad9451aca 100644 --- a/openhcs/processing/materialization/options.py +++ b/openhcs/processing/materialization/options.py @@ -10,9 +10,11 @@ import csv import io from dataclasses import dataclass, field -from enum import Enum +from abc import ABC from pathlib import Path -from typing import Any, Callable, Dict, List, Optional +from typing import Any, Callable, ClassVar, Dict, List, Optional + +from metaclass_registry import AutoRegisterMeta from openhcs.core.runtime_tabular_values import ColumnarRows from openhcs.core.runtime_measurements import RuntimeMeasurementFeature @@ -22,11 +24,26 @@ ) -class MaterializedFilenameIdentity(str, Enum): - """Semantic identity used to construct materialized artifact filenames.""" +class MaterializedFilenameIdentity(ABC, metaclass=AutoRegisterMeta): + """What a materialized artifact's filename is built from; ``name`` is its spelling.""" + + __registry_key__ = "name" + __skip_if_no_key__ = True + __registry__: ClassVar[dict[str, type["MaterializedFilenameIdentity"]]] = {} + + name: ClassVar[str | None] = None + + +class NameBySourceIdentity(MaterializedFilenameIdentity): + """The source image the artifact was computed from.""" + + name = "source_identity" - SOURCE_IDENTITY = "source_identity" - ARTIFACT_NAME = "artifact_name" + +class NameByArtifactName(MaterializedFilenameIdentity): + """The artifact's declared name.""" + + name = "artifact_name" @dataclass(frozen=True) @@ -36,20 +53,7 @@ class FileOutputOptions: filename_suffix: str = "" strip_roi_suffix: bool = False strip_pkl: bool = True - filename_identity: MaterializedFilenameIdentity = ( - MaterializedFilenameIdentity.SOURCE_IDENTITY - ) - - def __post_init__(self) -> None: - object.__setattr__( - self, - "filename_identity", - ( - self.filename_identity - if isinstance(self.filename_identity, MaterializedFilenameIdentity) - else MaterializedFilenameIdentity(self.filename_identity) - ), - ) + filename_identity: type[MaterializedFilenameIdentity] = NameBySourceIdentity @property def primary_output_suffix(self) -> str: @@ -151,7 +155,6 @@ class PointROIOptions(FileOutputOptions, SourceOptions): filename_suffix: str = "_points.roi.zip" def __post_init__(self) -> None: - super().__post_init__() features = (self.z_feature, self.y_feature, self.x_feature) if not all( isinstance(feature, RuntimeMeasurementFeature) for feature in features @@ -172,7 +175,6 @@ class SWCOptions(FileOutputOptions, SourceOptions): process_type: int = 2 def __post_init__(self) -> None: - super().__post_init__() if self.root_type < 0 or self.process_type < 0: raise ValueError("SWC node type codes must be nonnegative integers.") @@ -223,6 +225,4 @@ class ImageFileOptions(FileOutputOptions, SourceOptions): class FileBundleOptions(FileOutputOptions): """A validated mapping of relative output paths to bytes, text or typed outputs.""" - filename_identity: MaterializedFilenameIdentity = ( - MaterializedFilenameIdentity.ARTIFACT_NAME - ) + filename_identity: type[MaterializedFilenameIdentity] = NameByArtifactName diff --git a/tests/unit/test_artifact_output_materialization_planner.py b/tests/unit/test_artifact_output_materialization_planner.py index 2f1878fe87..6fbcd955c9 100644 --- a/tests/unit/test_artifact_output_materialization_planner.py +++ b/tests/unit/test_artifact_output_materialization_planner.py @@ -18,7 +18,7 @@ from openhcs.processing.materialization import ( CsvOptions, ImageFileOptions, - MaterializedFilenameIdentity, + NameByArtifactName, MaterializationSpec, ROIOptions, materialize, @@ -78,11 +78,11 @@ def test_object_label_output_gets_terminal_roi_and_label_materialization() -> No assert materialization.outputs == ( ROIOptions( min_area=1, - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ), ImageFileOptions( filename_suffix=".labels.tif", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ), ) assert materialization.participates_in_persistent_materialization() @@ -111,7 +111,7 @@ def test_terminal_measurement_output_gets_artifact_named_csv_materialization() - assert isinstance(materialization, TerminalMaterializationSpec) assert materialization.outputs == ( CsvOptions( - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ), ) diff --git a/tests/unit/test_automatic_image_role_materialization_435.py b/tests/unit/test_automatic_image_role_materialization_435.py index 392d0ebae8..3cd5763510 100644 --- a/tests/unit/test_automatic_image_role_materialization_435.py +++ b/tests/unit/test_automatic_image_role_materialization_435.py @@ -1,5 +1,6 @@ """Default retained image artifacts keep their declared output role identity.""" +from openhcs.processing.materialization.options import NameByArtifactName, NameBySourceIdentity from dataclasses import replace from pathlib import Path import json @@ -148,7 +149,7 @@ def test_automatic_two_role_images_publish_their_actual_saved_occurrences(tmp_pa )) def test_retained_role_policy_preserves_authored_and_unnamed_image_paths(tmp_path, purpose): from openhcs.processing.materialization import ( - ImageFileOptions, MaterializationSpec, MaterializedFilenameIdentity, + ImageFileOptions, MaterializationSpec, TerminalMaterializationSpec, materialization_outputs, ) @@ -156,8 +157,8 @@ def test_retained_role_policy_preserves_authored_and_unnamed_image_paths(tmp_pat filename_suffix=".tif", relative_path_template=("authored/plane_{index}.tif" if purpose.endswith("template") else None), filename_identity=( - MaterializedFilenameIdentity.ARTIFACT_NAME if purpose.endswith("artifact") - else MaterializedFilenameIdentity.SOURCE_IDENTITY + NameByArtifactName if purpose.endswith("artifact") + else NameBySourceIdentity ), ) spec = ( diff --git a/tests/unit/test_cellprofiler_save_images.py b/tests/unit/test_cellprofiler_save_images.py index 756d287762..2d22082af0 100644 --- a/tests/unit/test_cellprofiler_save_images.py +++ b/tests/unit/test_cellprofiler_save_images.py @@ -73,7 +73,7 @@ ExecutionAxisMaterializationRelativePathScope, ImageFileOptions, MaterializationSpec, - MaterializedFilenameIdentity, + NameBySourceIdentity, WriteMode, materialize, ) @@ -770,7 +770,7 @@ def test_save_images_from_image_filename_uses_source_identity_and_suffix() -> No options = _image_file_options(_contract(module)) - assert options.filename_identity is MaterializedFilenameIdentity.SOURCE_IDENTITY + assert options.filename_identity is NameBySourceIdentity assert options.filename_suffix == "_Overlay.png" assert options.relative_path_template is None diff --git a/tests/unit/test_collapsed_persisted_occurrence_435.py b/tests/unit/test_collapsed_persisted_occurrence_435.py index e513f32f86..e081022dbe 100644 --- a/tests/unit/test_collapsed_persisted_occurrence_435.py +++ b/tests/unit/test_collapsed_persisted_occurrence_435.py @@ -45,7 +45,7 @@ from openhcs.processing.materialization import ( ImageFileOptions, MaterializationSpec, - MaterializedFilenameIdentity, + NameByArtifactName, ) from tests.unit.test_function_outputs import ( @@ -90,7 +90,7 @@ def _actual_saved_occurrence(tmp_path, scenario): materialization=MaterializationSpec( ImageFileOptions( filename_suffix=".tif", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, relative_path_template=filename, ) ), diff --git a/tests/unit/test_derived_role_publication_435.py b/tests/unit/test_derived_role_publication_435.py index 25ccb1f56e..ae8fb6ebcf 100644 --- a/tests/unit/test_derived_role_publication_435.py +++ b/tests/unit/test_derived_role_publication_435.py @@ -1,5 +1,6 @@ """Issue435: real saved image occurrences projected through two existing owners.""" +from openhcs.processing.materialization.options import NameByArtifactName, NameBySourceIdentity from dataclasses import asdict, replace import json @@ -40,7 +41,7 @@ from openhcs.core.runtime_plane_projection import RuntimePlaneAxis, RuntimePlaneAxisValueProjection from python_introspect import to_jsonable from openhcs.processing.materialization import ( - ImageFileOptions, MaterializationSpec, MaterializedFilenameIdentity, + ImageFileOptions, MaterializationSpec, ) from openhcs.domains.microscopy.axes import Microscopy @@ -131,8 +132,8 @@ def test_saved_roles_publish_once_per_persisted_occurrence(tmp_path, scenario, a materialization=MaterializationSpec(ImageFileOptions( filename_suffix=".tif", filename_identity=( - MaterializedFilenameIdentity.ARTIFACT_NAME if aggregate - else MaterializedFilenameIdentity.SOURCE_IDENTITY + NameByArtifactName if aggregate + else NameBySourceIdentity ), relative_path_template=(filename if same_persisted_path else filename.replace(".tif", "_independent.tif")), )), @@ -282,7 +283,7 @@ def test_saved_roles_publish_once_per_persisted_occurrence(tmp_path, scenario, a "stored_key": repr(item.materialization.record.key)} for item in saved for output in item.outputs_for_backend(Backend.DISK.value)], "artifact_projections": [{"path": path, - "role": projection.projection_role.value, + "role": projection.projection_role.name, "address": None if projection.address is None else projection.address.as_component_metadata(), "alias": projection.source_alias, "kind": projection.artifact_kind.value} diff --git a/tests/unit/test_file_materialization_options.py b/tests/unit/test_file_materialization_options.py index d4ee275f12..e1122655f8 100644 --- a/tests/unit/test_file_materialization_options.py +++ b/tests/unit/test_file_materialization_options.py @@ -22,7 +22,8 @@ ) from openhcs.processing.materialization.options import ( FileOutputOptions, - MaterializedFilenameIdentity, + NameByArtifactName, + NameBySourceIdentity, SourceOptions, ) @@ -42,7 +43,7 @@ def test_image_and_bundle_options_extend_existing_writer_options() -> None: assert issubclass(image_options, SourceOptions) assert issubclass(bundle_options, FileOutputOptions) assert bundle_options().filename_identity is ( - MaterializedFilenameIdentity.ARTIFACT_NAME + NameByArtifactName ) @@ -69,7 +70,7 @@ def test_source_identity_image_file_projects_addressable_stack_planes() -> None: spec = MaterializationSpec( image_options( filename_suffix="_saved.npy", - filename_identity=MaterializedFilenameIdentity.SOURCE_IDENTITY, + filename_identity=NameBySourceIdentity, ) ) filemanager = FileManager({"memory": MemoryStorageBackend()}) diff --git a/tests/unit/test_function_artifact_materialization.py b/tests/unit/test_function_artifact_materialization.py index d89e94bdf5..c6970b81a8 100644 --- a/tests/unit/test_function_artifact_materialization.py +++ b/tests/unit/test_function_artifact_materialization.py @@ -163,7 +163,8 @@ ) from openhcs.processing.materialization.options import ( ImageFileOptions, - MaterializedFilenameIdentity, + NameByArtifactName, + NameBySourceIdentity, ) from openhcs.core.axes import AxisFamily, Ungrouped from openhcs.domains.microscopy.axes import Microscopy @@ -921,7 +922,7 @@ def test_materialize_artifact_outputs_uses_output_plan_axes_for_source_named_run materialization = MaterializationSpec( ImageFileOptions( filename_suffix="_saved.tif", - filename_identity=MaterializedFilenameIdentity.SOURCE_IDENTITY, + filename_identity=NameBySourceIdentity, ) ) output_plan = ArtifactOutputPlan( @@ -1930,7 +1931,7 @@ def test_terminal_persistence_is_reported_without_becoming_declared_export() -> ( TextOptions( filename_suffix=".csv", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ), "count,intensity\n2,1.25\n", ), @@ -2226,7 +2227,7 @@ def test_artifact_name_materialization_ignores_incomplete_source_identity() -> N materialization=MaterializationSpec( ImageFileOptions( filename_suffix=".tif", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ) ), ) @@ -4012,7 +4013,7 @@ def test_materialization_identity_replaces_provenance_not_payload_layout() -> No selected_image = ArtifactSpec.input("NucleiImage", ImageArtifactType) options = ImageFileOptions( filename_suffix="_NucleiLabels.tif", - filename_identity=MaterializedFilenameIdentity.SOURCE_IDENTITY, + filename_identity=NameBySourceIdentity, ) output_plan = ArtifactOutputPlan( name="SavedNuclei", @@ -4073,7 +4074,7 @@ def test_compiled_z_axis_reaches_source_named_image_materialization() -> None: materialization=MaterializationSpec( ImageFileOptions( filename_suffix="_NucleiLabels.tif", - filename_identity=MaterializedFilenameIdentity.SOURCE_IDENTITY, + filename_identity=NameBySourceIdentity, ) ), variable_components=(Microscopy.ZIndex,), diff --git a/tests/unit/test_function_outputs.py b/tests/unit/test_function_outputs.py index 164cfabe2d..40ec6d5079 100644 --- a/tests/unit/test_function_outputs.py +++ b/tests/unit/test_function_outputs.py @@ -105,7 +105,7 @@ from openhcs.processing.materialization import ( ImageFileOptions, MaterializationSpec, - MaterializedFilenameIdentity, + NameBySourceIdentity, ) from openhcs.core.axes import Axis, AxisFamily from openhcs.domains.microscopy.axes import Microscopy @@ -2334,7 +2334,7 @@ def test_runtime_image_metadata_target_requires_persisted_images(tmp_path, conte materialization=MaterializationSpec( ImageFileOptions( filename_suffix=".tif", - filename_identity=MaterializedFilenameIdentity.SOURCE_IDENTITY, + filename_identity=NameBySourceIdentity, ) ), ) @@ -2449,7 +2449,7 @@ def test_declared_image_destinations_publish_and_reconcile_after_value_cleanup( materialization=MaterializationSpec( ImageFileOptions( filename_suffix=".tif", - filename_identity=MaterializedFilenameIdentity.SOURCE_IDENTITY, + filename_identity=NameBySourceIdentity, relative_path_template=f"{directory}/A01_s001_w2_z001_t001.tif", ) ), diff --git a/tests/unit/test_materialization_core.py b/tests/unit/test_materialization_core.py index ee6fd4a87b..376ecddb82 100644 --- a/tests/unit/test_materialization_core.py +++ b/tests/unit/test_materialization_core.py @@ -1,3 +1,4 @@ +from openhcs.processing.materialization.options import NameByArtifactName, NameBySourceIdentity import json from pathlib import Path from functools import partial @@ -101,7 +102,8 @@ ImageFileOptions, JsonOptions, MaterializationSpec, - MaterializedFilenameIdentity, + NameByArtifactName, + NameBySourceIdentity, PointROIOptions, ROIOptions, TiffStackOptions, @@ -189,7 +191,7 @@ def test_retained_full_stack_stream_preserves_absolute_calibration_and_integer_p MaterializationSpec( ImageFileOptions( filename_suffix=".labels.tif", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ) ), payload, @@ -263,7 +265,7 @@ def test_image_file_materialization_densifies_sparse_named_object_labels() -> No MaterializationSpec( ImageFileOptions( filename_suffix=".labels.tif", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ) ), payload, @@ -535,7 +537,7 @@ def test_materialization_spec_candidate_paths_follow_registered_writers() -> Non def test_materialization_spec_declares_filename_identity() -> None: assert tiff_stack().uses_source_identity_filename() assert not tiff_stack( - TiffStackOptions(filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME) + TiffStackOptions(filename_identity=NameByArtifactName) ).uses_source_identity_filename() @@ -575,7 +577,7 @@ def test_indexed_image_materialization_streams_each_declared_component_plane( ) spec = MaterializationSpec( ImageFileOptions( - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, relative_path_template="frame-{index:04d}.png", ) ) @@ -2559,7 +2561,6 @@ def test_declared_volume_save_preserves_pixels_and_exact_plane_selection( tmp_path, intrinsic ): from openhcs.core.artifacts import ImageArtifactType - from openhcs.processing.materialization.options import MaterializedFilenameIdentity from openhcs.core.component_group_scope import RuntimeExecutionAxisScope from openhcs.core.image_file_serialization import TiffImageFileFormat from openhcs.core.runtime_slice_projection import RuntimeSliceProjection @@ -2609,7 +2610,7 @@ def test_declared_volume_save_preserves_pixels_and_exact_plane_selection( manager = FileManager({"disk": DiskStorageBackend()}) spec = MaterializationSpec(ImageFileOptions( filename_suffix=".tif", - filename_identity=MaterializedFilenameIdentity.SOURCE_IDENTITY, + filename_identity=NameBySourceIdentity, )) batch = prepare_materialization( spec, diff --git a/tests/unit/test_multi_template_matching.py b/tests/unit/test_multi_template_matching.py index 5572c5baad..cfb83d5d5d 100644 --- a/tests/unit/test_multi_template_matching.py +++ b/tests/unit/test_multi_template_matching.py @@ -27,7 +27,7 @@ multi_template_crop_subset, ) from openhcs.processing.materialization import materialization_outputs -from openhcs.processing.materialization.options import MaterializedFilenameIdentity +from openhcs.processing.materialization.options import NameByArtifactName @pytest.mark.parametrize( @@ -81,7 +81,7 @@ def test_template_match_artifact_materializes_every_match_column() -> None: assert not artifact_spec.materialization.uses_source_identity_filename() assert ( artifact_spec.materialization.outputs[0].filename_identity - is MaterializedFilenameIdentity.ARTIFACT_NAME + is NameByArtifactName ) results = [ TemplateMatchResult( diff --git a/tests/unit/test_named_checkpoint_publication.py b/tests/unit/test_named_checkpoint_publication.py index b8d29ba9a4..da75ecfa4b 100644 --- a/tests/unit/test_named_checkpoint_publication.py +++ b/tests/unit/test_named_checkpoint_publication.py @@ -4,6 +4,7 @@ compiler, runtime materialization, metadata reconciliation and reopen are used. """ +from openhcs.processing.materialization.options import NameByArtifactName from dataclasses import dataclass, replace import json @@ -38,7 +39,7 @@ Pure3DContract, ) from openhcs.processing.materialization import ( - ImageFileOptions, MaterializationSpec, MaterializedFilenameIdentity, + ImageFileOptions, MaterializationSpec, ) from openhcs.core.axes import TileAxis from openhcs.domains.microscopy.axes import Microscopy @@ -62,7 +63,7 @@ def _field(name): return MainFlowPlaneProjectionOutputSpec.output( name, ImageArtifactType, sidecar_role=QaCheckpoint, materialization=MaterializationSpec(ImageFileOptions( - filename_suffix=".tif", filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_suffix=".tif", filename_identity=NameByArtifactName, )), ) diff --git a/tests/unit/test_primary_object_diagnostics.py b/tests/unit/test_primary_object_diagnostics.py index 56bb53ca71..a6749d026e 100644 --- a/tests/unit/test_primary_object_diagnostics.py +++ b/tests/unit/test_primary_object_diagnostics.py @@ -32,7 +32,7 @@ ) from openhcs.processing.materialization import ( ImageFileOptions, - MaterializedFilenameIdentity, + NameByArtifactName, ) @@ -166,7 +166,7 @@ def test_diagnostic_contract_has_source_producer_stage_identity_not_new_objects( assert spec.materialization.outputs == ( ImageFileOptions( filename_suffix=".tif", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ), ) assert spec.source_context_sources() == (source.ref(),) diff --git a/tests/unit/test_tiff_output_compression.py b/tests/unit/test_tiff_output_compression.py index 47abc6832a..d6ecc6a848 100644 --- a/tests/unit/test_tiff_output_compression.py +++ b/tests/unit/test_tiff_output_compression.py @@ -16,7 +16,7 @@ from openhcs.processing.materialization import ( ImageFileOptions, MaterializationSpec, - MaterializedFilenameIdentity, + NameByArtifactName, materialize, ) @@ -55,7 +55,7 @@ def test_configured_named_artifact_tiff_output_is_lossless(tmp_path) -> None: MaterializationSpec( ImageFileOptions( filename_suffix=".tif", - filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + filename_identity=NameByArtifactName, ) ), pixels, From 1b4e3087aad86c980a03e07187fc4efbd2a961fc Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:24:33 -0400 Subject: [PATCH 29/36] E1: guard enum dispatch to listed boundaries; pin python-introspect at the 0.2.8 branch head The guard resolves each Enum.MEMBER through the module's imports and fails on a comparison, match case, dict key or EnumKeyedStrategyMixin key outside the table of boundary spellings and reassigned enums; a stale table entry fails too. It also keeps the deleted names deleted, the dataset identity modules free of resolve(), and numpy encodable through to_jsonable. python-introspect 0.2.8 (OpenHCSDev/python-introspect#22) lets pydantic accept a RegisteredClassChoices field's registry key, which the MCP tool schemas need for the family-valued config fields. Co-Authored-By: Claude Opus 5.5 --- external/python-introspect | 2 +- tests/unit/test_e1_enum_dispatch_guards.py | 468 +++++++++++++++++++++ 2 files changed, 469 insertions(+), 1 deletion(-) create mode 100644 tests/unit/test_e1_enum_dispatch_guards.py diff --git a/external/python-introspect b/external/python-introspect index 046a1f6be7..0d075a9e16 160000 --- a/external/python-introspect +++ b/external/python-introspect @@ -1 +1 @@ -Subproject commit 046a1f6be7713ac05efa7ccac4199250e5fa9ee3 +Subproject commit 0d075a9e1681c5385c119bb2f558d9121a7a2671 diff --git a/tests/unit/test_e1_enum_dispatch_guards.py b/tests/unit/test_e1_enum_dispatch_guards.py new file mode 100644 index 0000000000..9fa4fed56d --- /dev/null +++ b/tests/unit/test_e1_enum_dispatch_guards.py @@ -0,0 +1,468 @@ +"""E1 guards: our code does not dispatch on Enum members outside listed boundaries. + +A kind is a family whose members carry their behaviour (rule 1a). An ``Enum`` +stays only as a boundary spelling: an external contract's literal table (a +CellProfiler setting, a Qt/OS/MCP name) or a processing function's parameter +choices. Dispatching on its members (``is``/``==``/``in`` against a member, a +``match`` case, a dict keyed by members, an ``EnumKeyedStrategyMixin`` key) is +allowed only where the table below lists the enum and module, with its reason. +Entries leave the table when their enum is converted; a stale entry fails. +""" + +from __future__ import annotations + +import ast +import re +from collections import defaultdict +from functools import lru_cache +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +OPENHCS = ROOT / "openhcs" +ENUM_BASES = {"Enum", "IntEnum", "StrEnum", "Flag", "IntFlag"} + +PROCESSING_FUNCTIONS = "openhcs/processing/backends/" +"""Processing function modules: an enum declared here is a function's parameter +choice list (most spell a CellProfiler setting), dispatched only by that +function's module and its backend twins.""" + +CELLPROFILER_SPELLING = "CellProfiler setting or column literal table" + +ALLOWED: dict[tuple[str, str], str] = { + ('ArtifactPayloadShape', 'openhcs/core/artifacts.py'): 'K2: core/artifacts.py', + ('ArtifactPayloadShape', 'openhcs/core/runtime_exports.py'): 'K2: core/artifacts.py', + ('CPAPropertyName', 'openhcs/interop/cellprofiler/analyst_export.py'): CELLPROFILER_SPELLING, + ('CapabilityKind', 'openhcs/agent/capabilities.py'): 'U2: MCP tool-input filter choices; needs the MCP binder to project family fields (python-introspect 0.2.8)', + ('CapabilityRole', 'openhcs/agent/capabilities.py'): 'U2: MCP tool-input filter choices; needs the MCP binder to project family fields (python-introspect 0.2.8)', + ('CapabilityVisibility', 'openhcs/agent/capabilities.py'): 'U2: MCP tool-input filter choices; needs the MCP binder to project family fields (python-introspect 0.2.8)', + ('CellProfilerMeasurementTargetScope', 'openhcs/interop/cellprofiler/measurement_scope.py'): CELLPROFILER_SPELLING, + ('CellProfilerMeasurementTargetScope', 'openhcs/interop/cellprofiler/module_artifact_declarations.py'): CELLPROFILER_SPELLING, + ('CellProfilerObjectTableMode', 'openhcs/interop/cellprofiler/analyst_export.py'): CELLPROFILER_SPELLING, + ('CellProfilerSourceImageProjectionField', 'openhcs/interop/cellprofiler/analyst_export.py'): CELLPROFILER_SPELLING, + ('FileFormat', 'openhcs/core/plate_image_inventory.py'): "E2: restates polystore's format family (get_format_from_extension); key result actions by that family", + ('FileFormat', 'openhcs/pyqt_gui/widgets/image_browser.py'): "E2: restates polystore's format family (get_format_from_extension); key result actions by that family", + ('FunctionStepExecutionScope', 'openhcs/core/artifacts.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/core/callable_contract.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/core/function_patterns.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/core/function_reference.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/core/orchestrator/bundle_execution.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/core/orchestrator/worker_execution.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/core/pipeline/artifact_planning.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/core/pipeline/compilation_session.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/core/pipeline/compiler.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/core/pipeline/funcstep_contract_validator.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/core/pipeline/path_planner.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/interop/cellprofiler/compile_time_contracts.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/interop/cellprofiler/module_callable_abi.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/interop/cellprofiler/module_declarations.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/processing/backends/lib_registry/openhcs_registry.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/processing/backends/lib_registry/unified_registry.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/processing/custom_functions/validation.py'): 'K5: callable contract and function patterns', + ('ImagePayloadConsumption', 'openhcs/core/callable_contract.py'): 'K5: callable contract and function patterns', + ('ImagePayloadConsumption', 'openhcs/core/function_patterns.py'): 'K5: callable contract and function patterns', + ('ImagePayloadConsumption', 'openhcs/core/steps/function_execution.py'): 'K5: callable contract and function patterns', + ('ImagePayloadConsumption', 'openhcs/interop/cellprofiler/module_artifact_declarations.py'): 'K5: callable contract and function patterns', + ('ImagePayloadConsumption', 'openhcs/interop/cellprofiler/runtime/module_execution.py'): 'K5: callable contract and function patterns', + ('ImagePayloadMetadataCompositionMode', 'openhcs/core/aligned_image_payload.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ImagePayloadMetadataCompositionMode', 'openhcs/core/runtime_image_values.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ImagePayloadMetadataCompositionMode', 'openhcs/interop/cellprofiler/runtime/invocation.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('MainFlowInputProjection', 'openhcs/core/steps/function_runtime.py'): 'K5: callable contract and function patterns', + ('MainFlowInputProjection', 'openhcs/interop/cellprofiler/runtime/artifact_binding.py'): 'K5: callable contract and function patterns', + ('McpWireContentType', 'openhcs/mcp/dev_client_core.py'): 'MCP content-type names', + ('McpWireMethod', 'openhcs/mcp/dev_client_core.py'): 'MCP JSON-RPC method names', + ('MeasurementObjectRowIdentity', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_policies.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/core/measurement_feature_queries.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/core/measurement_row_materialization.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/core/runtime_artifact_queries.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/core/runtime_spatial_grid.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/module_artifact_declarations.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/measurement_recording.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/measurement_rows.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/object_measurement_vectors.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/relationship_measurement_rows.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/alignment.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/classification.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/colocalization.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/flagging.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/intensity.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/intensity_distribution.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/measurement_math.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/neighbors.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/relationships.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/spreadsheet_export.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/thresholding.py'): 'K3: measurement code held in #1199', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/tracking.py'): 'K3: measurement code held in #1199', + ('MeasurementRowValueField', 'openhcs/interop/cellprofiler/runtime/measurement_rows.py'): 'K3: measurement code held in #1199', + ('MeasurementRowValueField', 'openhcs/processing/backends/cellprofiler/measurement_math.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/core/equivalence/keys.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/core/equivalence/measurement_features.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/core/equivalence/measurement_rows.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/core/measurement_row_materialization.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/core/runtime_artifact_queries.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/core/runtime_measurements.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/interop/cellprofiler/analyst_export.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/interop/cellprofiler/database_column_dialect.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/interop/cellprofiler/image_set_numbering.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/interop/cellprofiler/measurement_dialect.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/interop/cellprofiler/measurement_scope.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/interop/cellprofiler/module_measurement_features.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/processing/backends/cellprofiler/calculate_statistics.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/processing/backends/cellprofiler/image_quality.py'): 'K3: measurement code held in #1199', + ('MeasurementScope', 'openhcs/processing/backends/cellprofiler/spreadsheet_export.py'): 'K3: measurement code held in #1199', + ('MeasurementStatistic', 'openhcs/core/equivalence/keys.py'): 'K3: measurement code held in #1199', + ('MeasurementStatistic', 'openhcs/core/equivalence/measurement_features.py'): 'K3: measurement code held in #1199', + ('MeasurementStatistic', 'openhcs/core/runtime_measurements.py'): 'K3: measurement code held in #1199', + ('MeasurementStatistic', 'openhcs/interop/cellprofiler/module_measurement_features.py'): 'K3: measurement code held in #1199', + ('MeasurementTableRowLayout', 'openhcs/interop/cellprofiler/analyst_export.py'): 'K3: measurement code held in #1199', + ('MissingObjectMeasurementValuePolicy', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_completion.py'): 'K3: measurement code held in #1199', + ('MultiprocessingStartMethod', 'openhcs/core/orchestrator/worker_execution.py'): 'Python multiprocessing start-method names; a class-valued top-level GlobalPipelineConfig field does not reach lazy PipelineConfig (ObjectState, L6)', + ('MultiprocessingStartMethod', 'openhcs/core/worker_start_policy.py'): 'Python multiprocessing start-method names; a class-valued top-level GlobalPipelineConfig field does not reach lazy PipelineConfig (ObjectState, L6)', + ('ObjectFeatureArrayDomain', 'openhcs/processing/backends/cellprofiler/shape.py'): 'K3: measurement code held in #1199', + ('ObjectLabelDomainScope', 'openhcs/core/measurement_image_alignment.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ObjectLabelDomainScope', 'openhcs/core/pipeline/function_contracts.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ObjectLabelDomainScope', 'openhcs/core/runtime_object_label_building.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ObjectLabelDomainScope', 'openhcs/core/runtime_object_label_domains.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ObjectLabelDomainScope', 'openhcs/core/runtime_object_labels.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ObjectLabelDomainScope', 'openhcs/core/runtime_relationships.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ObjectLabelDomainScope', 'openhcs/interop/cellprofiler/runtime/measurement_rows.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ObjectLabelDomainScope', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_completion.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ObjectLabelDomainScope', 'openhcs/interop/cellprofiler/runtime/relationship_measurement_rows.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ObjectLabelDomainScope', 'openhcs/processing/backends/cellprofiler/outlines.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ObjectLabelDomainScope', 'openhcs/processing/backends/cellprofiler/relationships.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ObjectLabelRepresentation', 'openhcs/core/runtime_object_label_aggregation.py'): 'K3: measurement code held in #1199', + ('ObjectLabelRepresentation', 'openhcs/core/runtime_object_labels.py'): 'K3: measurement code held in #1199', + ('ObjectLabelRepresentation', 'openhcs/processing/backends/cellprofiler/shape.py'): 'K3: measurement code held in #1199', + ('PlateFileKind', 'openhcs/agent/services/plate_inspection_service.py'): "U2: MCP tool-input choice with an 'all' literal; needs the MCP binder to project family fields (python-introspect 0.2.8)", + ('PlateFileKind', 'openhcs/agent/services/plate_streaming_service.py'): "U2: MCP tool-input choice with an 'all' literal; needs the MCP binder to project family fields (python-introspect 0.2.8)", + ('PlateFileKind', 'openhcs/agent/services/selected_plate_service.py'): "U2: MCP tool-input choice with an 'all' literal; needs the MCP binder to project family fields (python-introspect 0.2.8)", + ('PlateFileKind', 'openhcs/core/plate_image_inventory.py'): "U2: MCP tool-input choice with an 'all' literal; needs the MCP binder to project family fields (python-introspect 0.2.8)", + ('PlateFileKind', 'openhcs/mcp/installed_demo.py'): "U2: MCP tool-input choice with an 'all' literal; needs the MCP binder to project family fields (python-introspect 0.2.8)", + ('PlateFileKind', 'openhcs/pyqt_gui/widgets/image_browser.py'): "U2: MCP tool-input choice with an 'all' literal; needs the MCP binder to project family fields (python-introspect 0.2.8)", + ('PlateInspectionIssueCode', 'openhcs/mcp/dev_client_renderers/plate.py'): 'AgentError.code spellings read by the client', + ('QtPlatformName', 'openhcs/runtime/viewer_protocol.py'): 'Qt QPA plugin names (QT_QPA_PLATFORM)', + ('RuntimeCellValueKind', 'openhcs/core/equivalence/cells.py'): 'K3: measurement code held in #1199', + ('RuntimeCellValueKind', 'openhcs/core/equivalence/measurement_features.py'): 'K3: measurement code held in #1199', + ('RuntimeInvocationDomain', 'openhcs/core/function_patterns.py'): 'K5: callable contract and function patterns', + ('RuntimeInvocationDomain', 'openhcs/core/steps/function_execution.py'): 'K5: callable contract and function patterns', + ('RuntimeInvocationDomain', 'openhcs/core/steps/function_runtime.py'): 'K5: callable contract and function patterns', + ('RuntimeMeasurementFeatureNameMode', 'openhcs/core/equivalence/keys.py'): 'K3: measurement code held in #1199', + ('RuntimeMeasurementQualifierValueMode', 'openhcs/core/equivalence/measurement_rows.py'): 'K3: measurement code held in #1199', + ('RuntimeMeasurementQualifierValueMode', 'openhcs/core/measurement_dialect.py'): 'K3: measurement code held in #1199', + ('RuntimeMeasurementSourceNameEncoding', 'openhcs/core/equivalence/keys.py'): 'K3: measurement code held in #1199', + ('RuntimeMeasurementSourceNameEncoding', 'openhcs/core/measurement_dialect.py'): 'K3: measurement code held in #1199', + ('RuntimePlaneAxis', 'openhcs/core/aligned_image_payload.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/artifacts.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/debug.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/image_payload_execution_mode.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/measurement_image_alignment.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/runtime_image_values.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/runtime_measurements.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/runtime_object_labels.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/runtime_plane_projection.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/runtime_slice_alignment.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/runtime_sparse_labels.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/runtime_tabular_values.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/core/steps/function_runtime.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/interop/cellprofiler/runtime/adapter.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/interop/cellprofiler/runtime/artifact_binding.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_completion.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/area_occupied.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/colocalization.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/color.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/image_math.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/intensity.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/intensity_distribution.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('ViewerControlField', 'openhcs/agent/services/viewer_window_service.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', + ('ViewerControlField', 'openhcs/runtime/napari_viewer_server.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', + ('ViewerControlField', 'openhcs/runtime/viewer_control_actions.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', + ('ViewerDescriptorField', 'openhcs/runtime/napari_viewer_server.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', + ('ViewerIntensityWindowField', 'openhcs/runtime/napari_viewer_server.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', + ('ViewerLayerField', 'openhcs/runtime/napari_viewer_server.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', + ('ViewerPayloadField', 'openhcs/runtime/napari_viewer_server.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', + ('WriteMode', 'openhcs/serialization/pycodify_formatters.py'): 'K3: converting it edits spreadsheet_export.py, held in #1199', +} + + +@lru_cache(maxsize=1) +def _trees() -> dict[str, ast.Module]: + return { + path.relative_to(ROOT).as_posix(): ast.parse(path.read_text(encoding="utf-8")) + for path in sorted(OPENHCS.rglob("*.py")) + } + + +def _module_path(dotted: str) -> str | None: + base = dotted.replace(".", "/") + for candidate in (f"{base}.py", f"{base}/__init__.py"): + if candidate in _trees(): + return candidate + return None + + +def _absolute(module: str, node: ast.ImportFrom) -> str: + if not node.level: + return node.module or "" + package = module.removesuffix(".py").split("/")[: -node.level] + return ".".join([*package, *([node.module] if node.module else [])]) + + +@lru_cache(maxsize=None) +def _bindings(module: str) -> dict[str, tuple[str, str]]: + """Top-level and nested class names, and imported names, of one module. + + A class binds to (module, name); an import binds to (source module, name). + """ + + bound: dict[str, tuple[str, str]] = {} + for node in ast.walk(_trees()[module]): + if isinstance(node, ast.ClassDef): + bound.setdefault(node.name, (module, node.name)) + elif isinstance(node, ast.ImportFrom): + source = _module_path(_absolute(module, node)) + if source is None: + continue + for alias in node.names: + bound.setdefault(alias.asname or alias.name, (source, alias.name)) + return bound + + +@lru_cache(maxsize=None) +def _class(module: str, name: str, depth: int = 0) -> tuple[str, str] | None: + """The (module, class name) a name in ``module`` refers to, following imports.""" + + target = _bindings(module).get(name) + if target is None or depth > 8: + return None + if target == (module, name): + return target + return _class(*target, depth + 1) + + +@lru_cache(maxsize=1) +def _enums() -> frozenset[tuple[str, str]]: + """Every Enum class declared under openhcs/, as (module, name).""" + + declared: set[tuple[str, str]] = set() + changed = True + while changed: + changed = False + for module, tree in _trees().items(): + for node in ast.walk(tree): + if not isinstance(node, ast.ClassDef) or (module, node.name) in declared: + continue + for base in node.bases: + if isinstance(base, ast.Name | ast.Attribute): + name = base.id if isinstance(base, ast.Name) else base.attr + if name in ENUM_BASES or ( + isinstance(base, ast.Name) and _class(module, name) in declared + ): + declared.add((module, node.name)) + changed = True + break + return frozenset(declared) + + +def _member_enum(node: ast.expr, module: str) -> str | None: + """The enum a ``Enum.MEMBER`` (or ``.value``/``.name`` of one) expression names.""" + + if isinstance(node, ast.Attribute) and node.attr in {"value", "name"}: + node = node.value + if not isinstance(node, ast.Attribute) or not node.attr.isupper(): + return None + owner = node.value + if isinstance(owner, ast.Name): + found = _class(module, owner.id) + elif isinstance(owner, ast.Attribute): + found = next( + (enum for enum in _enums() if enum[1] == owner.attr and enum[0] == module), + None, + ) + else: + return None + return found[1] if found in _enums() else None + + +@lru_cache(maxsize=1) +def enum_dispatch_sites() -> dict[tuple[str, str], list[int]]: + """(enum name, module) -> lines where the module dispatches on its members.""" + + sites: dict[tuple[str, str], list[int]] = defaultdict(list) + compare_ops = (ast.Is, ast.IsNot, ast.Eq, ast.NotEq, ast.In, ast.NotIn) + for module, tree in _trees().items(): + for node in ast.walk(tree): + hits: list[str] = [] + if isinstance(node, ast.Compare) and any( + isinstance(op, compare_ops) for op in node.ops + ): + for side in (node.left, *node.comparators): + candidates = ( + side.elts if isinstance(side, ast.Tuple | ast.List | ast.Set) + else (side,) + ) + hits.extend( + name + for candidate in candidates + if (name := _member_enum(candidate, module)) + ) + elif isinstance(node, ast.match_case): + hits.extend( + name + for pattern in ast.walk(node.pattern) + if isinstance(pattern, ast.MatchValue) + and (name := _member_enum(pattern.value, module)) + ) + elif isinstance(node, ast.Dict): + hits.extend( + name + for key in node.keys + if key is not None and (name := _member_enum(key, module)) + ) + elif isinstance(node, ast.ClassDef): + keyed = any( + "EnumKeyedStrategyMixin" in ast.unparse(base) for base in node.bases + ) + for statement in node.body: + if not isinstance(statement, ast.Assign | ast.AnnAssign): + continue + if statement.value is None: + continue + targets = ( + statement.targets + if isinstance(statement, ast.Assign) + else [statement.target] + ) + names = [t.id for t in targets if isinstance(t, ast.Name)] + name = _member_enum(statement.value, module) + if name and (keyed or any("key" in target for target in names)): + hits.append(name) + for name in hits: + sites[(name, module)].append(getattr(node, "lineno", 0)) + return dict(sites) + + +def _allowed(enum_name: str, module: str) -> bool: + if (enum_name, module) in ALLOWED: + return True + return module.startswith(PROCESSING_FUNCTIONS) and any( + declaring.startswith(PROCESSING_FUNCTIONS) + for declaring, name in _enums() + if name == enum_name + ) + + +def test_enum_members_are_dispatched_only_at_listed_boundaries(): + offenders = sorted( + f"{module}:{lines[0]} dispatches on {enum_name}" + for (enum_name, module), lines in enum_dispatch_sites().items() + if not _allowed(enum_name, module) + ) + assert offenders == [] + + +def test_boundary_table_only_lists_live_dispatch(): + stale = sorted( + f"{enum_name} in {module}" + for enum_name, module in ALLOWED + if (enum_name, module) not in enum_dispatch_sites() + ) + assert stale == [] + + +DELETED_NAMES = ( + "SourceFilterMatcher", + "SourceFilterTargetResolver", + "SourceFilterMatchRequest", + "metadata_source_texts", + "filter_clause_matches", + "rule_filters_match", + "CellProfilerSourceFilterSubject", + "CellProfilerSourceFilterOperator", + "PipelineDebugSessionPhase", + "PipelineDebugSessionContext", + "PipelineDebugTargetState", + "PipelineDebugPauseBoundaryState", + "DebugSessionPhaseDeclarationBase", + "DebugActionPlacement", + "DebugToolbarAuxiliaryAction", + "CompiledArtifactInspectionControlPayload", + "ArtifactInspectionControlMessageType", + "_jsonable_metadata_value", + "_jsonable_array_values", + "_jsonable_scalar", + "_hashable_jsonable_metadata_value", +) +OLD_MEMBER_SPELLINGS = re.compile( + r"\b(SourceFilterMatchType|SourceFilterSubject|MetadataSource|SourceSetRole" + r"|SourceProjectionRole|SourceBindingOrigin)\.[A-Z_]+\b" +) +LIVE_TREES = ("openhcs", "tests", "scripts", "examples", "benchmark") + + +def _live_python_files(): + for tree in LIVE_TREES: + for path in (ROOT / tree).rglob("*.py"): + relative = path.relative_to(ROOT).as_posix() + if relative.startswith("benchmark/results/") or path == Path(__file__): + continue + yield path + + +def test_deleted_names_and_enum_member_spellings_stay_deleted(): + names = re.compile(r"\b(" + "|".join(DELETED_NAMES) + r")\b") + offenders = sorted( + path.relative_to(ROOT).as_posix() + for path in _live_python_files() + if names.search(text := path.read_text(encoding="utf-8")) + or OLD_MEMBER_SPELLINGS.search(text) + ) + assert offenders == [] + + +def test_source_family_kinds_are_not_enums(): + from enum import Enum + + from openhcs.core import debug_session_projection, source_bindings + + kinds = ( + source_bindings.SourceBindingOrigin, + source_bindings.SourceSetRole, + source_bindings.SourceProjectionRole, + source_bindings.MetadataSource, + source_bindings.SourceFilterSubject, + source_bindings.SourceFilterMatchType, + debug_session_projection.DebugSessionPhase, + ) + assert not [kind for kind in kinds if issubclass(kind, Enum)] + + +DATASET_IDENTITY_MODULES = ( + "openhcs/core/source_projection.py", + "openhcs/core/dataset_sources", + "openhcs/microscopes/bioformats_adapter.py", +) + + +def test_dataset_and_store_identity_never_resolve_symlinks(): + offenders = sorted( + f"{module}:{node.lineno}" + for module, tree in _trees().items() + if module.startswith(DATASET_IDENTITY_MODULES) + for node in ast.walk(tree) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr == "resolve" + and not node.args + and all(keyword.arg == "strict" for keyword in node.keywords) + ) + assert offenders == [] + + +def test_numpy_values_encode_through_to_jsonable(): + import numpy as np + + import openhcs.core.numpy_jsonable # noqa: F401 + from python_introspect import to_jsonable + + assert to_jsonable( + {"scalar": np.float32(1.5), "array": np.arange(4).reshape(2, 2), "n": np.int64(3)} + ) == {"scalar": 1.5, "array": [[0, 1], [2, 3]], "n": 3} From c501f82ab66785f9f9277dad9c4e4619159e82be Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:33:09 -0400 Subject: [PATCH 30/36] E1: zero structural-ratchet regressions Membership questions the converted families answer by flags become nominal identity or member behaviour: parameter sources split into agent- and runtime-supplied branches, the outcome export scope checks its own export path (one rule, written once), default step directories move onto the step-input dependency, viewers answer their own process liveness, and the semantic ObjectState field filter is the union of the four it names. Flags that restated a member's identity (writes_zarr, streams_from_source_context, keeps_every_field, completed, editable, collects_records, returns_every_record) are deleted. Forward references are unquoted. Co-Authored-By: Claude Opus 5.5 --- benchmark/cellprofiler_comparison.py | 3 +- openhcs/agent/dto/functions.py | 22 +++++++------ openhcs/agent/dto/plate.py | 5 +-- openhcs/agent/dto/ui_bridge.py | 14 +++------ openhcs/agent/runtime_platform.py | 2 +- .../services/function_catalog_service.py | 8 ++--- .../object_state_field_help_service.py | 8 +++-- .../agent/services/selected_plate_service.py | 3 +- openhcs/authoring/session/session.py | 3 +- openhcs/constants/constants.py | 2 +- openhcs/constants/input_source.py | 2 +- openhcs/core/config.py | 17 +++++----- openhcs/core/execution_state.py | 2 +- openhcs/core/measurement_image_alignment.py | 2 +- openhcs/core/orchestrator/bundle_execution.py | 3 +- openhcs/core/orchestrator/execution_result.py | 17 ++++------ openhcs/core/orchestrator/worker_execution.py | 2 +- openhcs/core/pipeline/compiler.py | 5 +-- openhcs/core/pipeline/path_planner.py | 22 ++++--------- openhcs/core/selection.py | 2 +- openhcs/core/source_binding_workspace.py | 6 ++-- openhcs/core/source_bindings.py | 14 ++++----- openhcs/core/step_dependencies.py | 31 ++++++++++++++++++- openhcs/core/steps/function_runtime.py | 7 ++--- .../cellprofiler/measurement_lookup.py | 2 +- .../interop/cellprofiler/module_settings.py | 6 ++-- .../interop/cellprofiler/pipeline_import.py | 2 +- .../interop/cellprofiler/plate_workspace.py | 12 ++++--- openhcs/mcp/dev_client_commands/ui.py | 6 ++-- openhcs/mcp/dev_client_core.py | 4 +-- openhcs/mcp/dev_client_renderers/knowledge.py | 6 ++-- .../mcp/dev_client_renderers/object_state.py | 3 +- openhcs/mcp/dev_client_rendering.py | 6 ++-- openhcs/mcp/http_auth.py | 4 +-- openhcs/mcp/lifecycle.py | 2 +- .../widgets/_roi_manager.py | 8 ++--- .../processing/backends/cellprofiler/grid.py | 14 ++++----- .../backends/cellprofiler/morphology.py | 4 +-- .../backends/cellprofiler/thresholding.py | 6 ++-- .../backends/cellprofiler/watershed.py | 2 +- openhcs/processing/materialization/options.py | 2 +- openhcs/pyqt_gui/app.py | 26 ++++++++++------ openhcs/pyqt_gui/main.py | 2 +- openhcs/pyqt_gui/services/logging_config.py | 8 ++--- .../services/ui_bridge_object_state.py | 3 +- .../shared/services/pipeline_debug_actions.py | 12 +++---- openhcs/runtime/napari_streaming_handlers.py | 2 +- openhcs/runtime/napari_viewer_server.py | 3 +- openhcs/runtime/viewer_display.py | 4 +-- openhcs/runtime/viewer_protocol.py | 29 ++++++++++++----- openhcs/runtime/zmq_execution_observation.py | 3 +- openhcs/runtime/zmq_execution_signature.py | 19 ++++++++---- 52 files changed, 222 insertions(+), 180 deletions(-) diff --git a/benchmark/cellprofiler_comparison.py b/benchmark/cellprofiler_comparison.py index 0e12fa430e..56bd930125 100644 --- a/benchmark/cellprofiler_comparison.py +++ b/benchmark/cellprofiler_comparison.py @@ -38,6 +38,7 @@ run_cellprofiler_cppipe_parity, ) from benchmark.timing import BenchmarkPhase, additive_phase_total_seconds +from openhcs.interop.cellprofiler.module_settings import SettingUnmapped from openhcs.core.config import GlobalPipelineConfig, PartitionFilterConfig from benchmark.equivalence.outputs import image_paths, table_paths @@ -271,7 +272,7 @@ def from_report( unmapped_cppipe_setting_row_count=sum( 1 for setting in report.cppipe_settings - if not setting.coverage.is_covered + if setting.coverage is SettingUnmapped ), supported_absorbed_processing_modules=( coverage.supported_absorbed_processing_modules diff --git a/openhcs/agent/dto/functions.py b/openhcs/agent/dto/functions.py index 01f3b11acd..dd599b732d 100644 --- a/openhcs/agent/dto/functions.py +++ b/openhcs/agent/dto/functions.py @@ -374,18 +374,22 @@ class FunctionParameterSource(ABC, metaclass=AutoRegisterMeta): __skip_if_no_key__ = True name: ClassVar[str | None] = None - agent_supplied: ClassVar[bool] = False - """The agent passes this parameter as a FunctionStep kwarg.""" - runtime_description: ClassVar[str | None] = None - """Why the agent must not pass a runtime-supplied parameter.""" + runtime_description: ClassVar[str] = "" + """Why the agent must not pass this parameter; empty when the agent passes it.""" class SuppliedByAgent(FunctionParameterSource): + """The agent passes this parameter as a FunctionStep kwarg.""" + name = "agent" - agent_supplied = True -class SuppliedFromPrimaryInput(FunctionParameterSource): +class SuppliedAtRuntime(FunctionParameterSource): + """OpenHCS supplies this parameter during execution.""" + + + +class SuppliedFromPrimaryInput(SuppliedAtRuntime): name = "runtime_primary_input" runtime_description = ( "Supplied by OpenHCS from the FunctionStep input image payload; do not " @@ -393,7 +397,7 @@ class SuppliedFromPrimaryInput(FunctionParameterSource): ) -class SuppliedFromArtifactInput(FunctionParameterSource): +class SuppliedFromArtifactInput(SuppliedAtRuntime): name = "runtime_artifact_input" runtime_description = ( "Supplied by OpenHCS from a declared artifact input during pipeline " @@ -401,7 +405,7 @@ class SuppliedFromArtifactInput(FunctionParameterSource): ) -class SuppliedByRuntime(FunctionParameterSource): +class SuppliedByRuntime(SuppliedAtRuntime): name = "runtime_parameter" runtime_description = ( "Supplied by OpenHCS runtime execution infrastructure; do not pass this " @@ -409,7 +413,7 @@ class SuppliedByRuntime(FunctionParameterSource): ) -class SuppliedAsRuntimeAdapter(FunctionParameterSource): +class SuppliedAsRuntimeAdapter(SuppliedAtRuntime): name = "runtime_adapter" runtime_description = ( "Supplied by OpenHCS as a runtime adapter object; do not pass this as a " diff --git a/openhcs/agent/dto/plate.py b/openhcs/agent/dto/plate.py index 39d1857acc..a4a288d82e 100644 --- a/openhcs/agent/dto/plate.py +++ b/openhcs/agent/dto/plate.py @@ -102,11 +102,9 @@ class SelectedPlateFileQueryTarget(ABC, metaclass=AutoRegisterMeta): name: ClassVar[str | None] = None is_selected_row_root: ClassVar[bool] = False """The query reads the selected row's own root, not a related plate.""" - streams_from_source_context: ClassVar[bool] = False - """Streamed results resolve against the row's source plate.""" @staticmethod - def named(name: str) -> type["SelectedPlateFileQueryTarget"]: + def named(name: str) -> type[SelectedPlateFileQueryTarget]: target = SelectedPlateFileQueryTarget.__registry__.get(name) if target is None: raise ValueError( @@ -137,7 +135,6 @@ def plate_root(cls, selected_row, plate_root, state_surface_id): class OutputPlateRoot(SelectedPlateFileQueryTarget): name = "output" - streams_from_source_context = True @classmethod def plate_root(cls, selected_row, plate_root, state_surface_id): diff --git a/openhcs/agent/dto/ui_bridge.py b/openhcs/agent/dto/ui_bridge.py index a30fd320fd..808abe0724 100644 --- a/openhcs/agent/dto/ui_bridge.py +++ b/openhcs/agent/dto/ui_bridge.py @@ -171,7 +171,7 @@ class UiBridgeOperationStatus(ABC, metaclass=AutoRegisterMeta): live_overview_severity: ClassVar[str] = UiLiveOverviewSeverity.ERROR.value @staticmethod - def named(name: str) -> type["UiBridgeOperationStatus"]: + def named(name: str) -> type[UiBridgeOperationStatus]: """Return the state spelled ``name``; ``ValueError`` when unknown.""" status = UiBridgeOperationStatus.__registry__.get(name) @@ -327,10 +327,9 @@ class UiObjectStateFieldFilter(ABC, metaclass=AutoRegisterMeta): __skip_if_no_key__ = True name: ClassVar[str | None] = None - keeps_every_field: ClassVar[bool] = False @staticmethod - def named(name: str) -> type["UiObjectStateFieldFilter"]: + def named(name: str) -> type[UiObjectStateFieldFilter]: field_filter = UiObjectStateFieldFilter.__registry__.get(name) if field_filter is None: raise ValueError( @@ -347,7 +346,6 @@ def matches(cls, field: "UiObjectStateFieldSummary") -> bool: class AllFields(UiObjectStateFieldFilter): name = "all" - keeps_every_field = True @classmethod def matches(cls, field: "UiObjectStateFieldSummary") -> bool: @@ -391,11 +389,9 @@ class SemanticFields(UiObjectStateFieldFilter): @classmethod def matches(cls, field: "UiObjectStateFieldSummary") -> bool: - return ( - field.dirty - or field.signature_diff - or field.inherited_value - or RawResolvedFields.matches(field) + return any( + part.matches(field) + for part in (DirtyFields, DefaultDiffFields, InheritedFields, RawResolvedFields) ) diff --git a/openhcs/agent/runtime_platform.py b/openhcs/agent/runtime_platform.py index a14141ebfc..b2386664e9 100644 --- a/openhcs/agent/runtime_platform.py +++ b/openhcs/agent/runtime_platform.py @@ -25,7 +25,7 @@ class AgentRuntimePlatform(ABC, metaclass=AutoRegisterMeta): name: ClassVar[str | None] = None @staticmethod - def current_type() -> type["AgentRuntimePlatform"]: + def current_type() -> type[AgentRuntimePlatform]: """Return the platform class of the running interpreter.""" if sys.platform.startswith("win"): return WindowsRuntimePlatform diff --git a/openhcs/agent/services/function_catalog_service.py b/openhcs/agent/services/function_catalog_service.py index 7e40b29e32..83de320225 100644 --- a/openhcs/agent/services/function_catalog_service.py +++ b/openhcs/agent/services/function_catalog_service.py @@ -466,7 +466,7 @@ def visible_signature( if self.should_document( name, parameter, skip_keyword_bag=True, hidden_names=hidden_names ) - and supplied_by[name].agent_supplied + and issubclass(supplied_by[name], SuppliedByAgent) ) ) return sig.replace(parameters=visible_parameters) @@ -488,7 +488,7 @@ def parameter_specs( name=name, annotation=_format_annotation(parameter.annotation), default_repr=_format_default(parameter.default), - required=supplier.agent_supplied + required=issubclass(supplier, SuppliedByAgent) and parameter.default is inspect.Parameter.empty, supplied_by=supplier, description=self.parameter_description( @@ -523,7 +523,7 @@ def agent_parameter_names( name for name, parameter in sig.parameters.items() if self.should_document(name, parameter, hidden_names=hidden_names) - and supplied_by[name].agent_supplied + and issubclass(supplied_by[name], SuppliedByAgent) ) ) runtime_names = ( @@ -595,7 +595,7 @@ def parameter_description( runtime_description = supplier.runtime_description if authored_description and runtime_description: return f"{authored_description.rstrip()} {runtime_description}" - return authored_description or runtime_description + return authored_description or runtime_description or None PARAMETER_DOCUMENTATION_POLICY = ParameterDocumentationPolicy() diff --git a/openhcs/agent/services/object_state_field_help_service.py b/openhcs/agent/services/object_state_field_help_service.py index ea399ecddb..b99ad03f5b 100644 --- a/openhcs/agent/services/object_state_field_help_service.py +++ b/openhcs/agent/services/object_state_field_help_service.py @@ -19,6 +19,8 @@ from openhcs.agent.dto.functions import ( FunctionArtifactSpec, FunctionDetail, + SuppliedAtRuntime, + SuppliedByAgent, ) from openhcs.agent.dto.ui_bridge import ( UiBridgeConnectionSpec, @@ -381,7 +383,7 @@ def _function_parameter_context( ), None, ) - if parameter is None or parameter.supplied_by.agent_supplied: + if parameter is None or issubclass(parameter.supplied_by, SuppliedByAgent): return None pieces = [ @@ -617,7 +619,7 @@ def _active_kwarg_context_lines( parameter_by_name = { parameter.name: parameter for parameter in detail.parameters - if parameter.supplied_by.agent_supplied + if issubclass(parameter.supplied_by, SuppliedByAgent) } lines = [" active kwargs:"] for name in active_kwarg_names: @@ -642,7 +644,7 @@ def _runtime_parameter_context_lines(detail: FunctionDetail) -> list[str]: return [ f"{parameter.name} ({parameter.supplied_by.name})" for parameter in detail.parameters - if not parameter.supplied_by.agent_supplied + if issubclass(parameter.supplied_by, SuppliedAtRuntime) ] diff --git a/openhcs/agent/services/selected_plate_service.py b/openhcs/agent/services/selected_plate_service.py index 4015fda611..46a6d98e64 100644 --- a/openhcs/agent/services/selected_plate_service.py +++ b/openhcs/agent/services/selected_plate_service.py @@ -9,6 +9,7 @@ from openhcs.agent.dto.common import AgentError, AgentWarning, SCHEMA_VERSION from python_introspect import JsonValue from openhcs.agent.dto.plate import ( + OutputPlateRoot, PlateInspectionDefaults, PlatePathInspectionRequest, SelectedPlateFileQueryRequest, @@ -476,7 +477,7 @@ def stream_context_plate_root( kind: PlateFileKind | None, ) -> str | None: if ( - not target.streams_from_source_context + target is not OutputPlateRoot or kind is not PlateFileKind.RESULT ): return None diff --git a/openhcs/authoring/session/session.py b/openhcs/authoring/session/session.py index f695ede04e..90db36c345 100644 --- a/openhcs/authoring/session/session.py +++ b/openhcs/authoring/session/session.py @@ -1209,9 +1209,8 @@ def observation_export_params( """Runtime observation export options for a run, under the access policy.""" scope = export_scope + scope.check_export_path(export_path) if export_path is None: - if not scope.retains_runtime_values: - raise ValueError("Outcome observation requires an export path.") return None path = self.dataset_access.require_writable(Path(export_path)) if path.exists(): diff --git a/openhcs/constants/constants.py b/openhcs/constants/constants.py index 46de57722e..f326a35fd9 100644 --- a/openhcs/constants/constants.py +++ b/openhcs/constants/constants.py @@ -67,7 +67,7 @@ class OrchestratorState(ABC, metaclass=AutoRegisterMeta): status_prefix: ClassVar[str] @staticmethod - def named(name: str) -> type["OrchestratorState"] | None: + def named(name: str) -> "type[OrchestratorState] | None": """The state spelled ``name``, or ``None`` for an unknown spelling.""" return OrchestratorState.__registry__.get(name) diff --git a/openhcs/constants/input_source.py b/openhcs/constants/input_source.py index fead3ed8a5..1142e29c74 100644 --- a/openhcs/constants/input_source.py +++ b/openhcs/constants/input_source.py @@ -28,7 +28,7 @@ class InputSource(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["InputSource"]]] = {} + __registry__: ClassVar[dict[str, type[InputSource]]] = {} name: ClassVar[str | None] = None reads_pipeline_start: ClassVar[bool] diff --git a/openhcs/core/config.py b/openhcs/core/config.py index 9aabbd42e7..e0fb03e962 100644 --- a/openhcs/core/config.py +++ b/openhcs/core/config.py @@ -134,12 +134,10 @@ class MaterializationBackend(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["MaterializationBackend"]]] = {} + __registry__: "ClassVar[dict[str, type[MaterializationBackend]]]" = {} name: ClassVar[str | None] = None backend: ClassVar[Backend] - writes_zarr: ClassVar[bool] = False - """Outputs are a Zarr store, which needs a Zarr config and converted inputs.""" class MaterializeAuto(MaterializationBackend): @@ -150,9 +148,10 @@ class MaterializeAuto(MaterializationBackend): class MaterializeToZarr(MaterializationBackend): + """Write a Zarr store, which needs a Zarr config and converted inputs.""" + name = "zarr" backend = Backend.ZARR - writes_zarr = True class MaterializeToDisk(MaterializationBackend): @@ -174,7 +173,7 @@ class PartitionFilterMode(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["PartitionFilterMode"]]] = {} + __registry__: "ClassVar[dict[str, type[PartitionFilterMode]]]" = {} name: ClassVar[str | None] = None preview_prefix: ClassVar[str] @@ -442,7 +441,7 @@ class PartitionFilterConfig(AnnotatedDataclassValidationMixin): """ @classmethod - def partition_filter_inheritance_branch(cls) -> type["PartitionFilterConfig"]: + def partition_filter_inheritance_branch(cls) -> "type[PartitionFilterConfig]": """Return the nominal MRO branch that may supply inherited filters.""" return cls @@ -549,7 +548,7 @@ def default_value(cls) -> "DtypeConfig": return cls() @classmethod - def annotation_type(cls) -> type["DtypeConfig"]: + def annotation_type(cls) -> "type[DtypeConfig]": """Expose the inheritable config type on callable signatures.""" return cls @@ -1296,13 +1295,13 @@ def supported_viewer_types(cls) -> tuple[ViewerType, ...]: def config_type_for_viewer( cls, viewer_type: ViewerType, - ) -> type["StreamingConfig"]: + ) -> "type[StreamingConfig]": """Return the registered config declaration for one viewer.""" return cls.__registry__[ViewerFamily.named(viewer_type)] @classmethod - def config_type_for_key(cls, config_key: str) -> type["StreamingConfig"]: + def config_type_for_key(cls, config_key: str) -> "type[StreamingConfig]": """Return the registered streaming config type for an ObjectState field key.""" return cls.__registry__[ViewerFamily.for_config_key(config_key)] diff --git a/openhcs/core/execution_state.py b/openhcs/core/execution_state.py index 282bbce038..94a6f5cf93 100644 --- a/openhcs/core/execution_state.py +++ b/openhcs/core/execution_state.py @@ -256,7 +256,7 @@ class TerminalExecutionStatus(ABC, metaclass=AutoRegisterMeta): serialized_aliases: ClassVar[frozenset[str]] = frozenset() @staticmethod - def named(value: str) -> type["TerminalExecutionStatus"]: + def named(value: str) -> type[TerminalExecutionStatus]: """The status spelled ``value`` or one of its aliases.""" for status in TerminalExecutionStatus.__registry__.values(): if value == status.name or value in status.serialized_aliases: diff --git a/openhcs/core/measurement_image_alignment.py b/openhcs/core/measurement_image_alignment.py index ac6bfcfd81..674eb976a6 100644 --- a/openhcs/core/measurement_image_alignment.py +++ b/openhcs/core/measurement_image_alignment.py @@ -740,7 +740,7 @@ class MeasurementImageReferenceDomain(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["MeasurementImageReferenceDomain"]]] = {} + __registry__: ClassVar[dict[str, type[MeasurementImageReferenceDomain]]] = {} name: ClassVar[str | None] = None labels_follow_image: ClassVar[bool] diff --git a/openhcs/core/orchestrator/bundle_execution.py b/openhcs/core/orchestrator/bundle_execution.py index ed63bc87c4..38edc4f616 100644 --- a/openhcs/core/orchestrator/bundle_execution.py +++ b/openhcs/core/orchestrator/bundle_execution.py @@ -46,6 +46,7 @@ RuntimeContextObservation, RuntimeExecutionObservation, MergeAllRuntimeRecords, + OmitRuntimeRecords, RuntimeObservationMode, ) from openhcs.core.orchestrator.worker_execution import ( @@ -396,7 +397,7 @@ def validate_bundle_execution( dataset_step_indexes = validate_dataset_scoped_contexts(compiled_contexts) if ( dataset_step_indexes - and not request.runtime_observation_mode.collects_records + and request.runtime_observation_mode is OmitRuntimeRecords ): raise ValueError( "Dataset-scoped FunctionSteps require a runtime observation mode " diff --git a/openhcs/core/orchestrator/execution_result.py b/openhcs/core/orchestrator/execution_result.py index d08fbd2523..f88e21e34e 100644 --- a/openhcs/core/orchestrator/execution_result.py +++ b/openhcs/core/orchestrator/execution_result.py @@ -130,12 +130,8 @@ def merge_into(self, execution_contexts: Mapping[str, ProcessingContext]) -> Non class RuntimeObservationMode(ABC): """Which worker runtime records are returned to the parent; each mode keeps its own.""" - collects_records: ClassVar[bool] = True - returns_every_record: ClassVar[bool] = False - """Workers keep each record to the end, so consumed inputs are never released.""" - @staticmethod - def from_parent_requirement(required: bool) -> type["RuntimeObservationMode"]: + def from_parent_requirement(required: bool) -> "type[RuntimeObservationMode]": """Select the mode implied by compiled parent-side execution needs.""" return MergeAllRuntimeRecords if required else OmitRuntimeRecords @@ -143,7 +139,7 @@ def from_parent_requirement(required: bool) -> type["RuntimeObservationMode"]: @staticmethod def for_compiled_bundle( bundle: "CompiledExecutionBundle", - ) -> type["RuntimeObservationMode"]: + ) -> "type[RuntimeObservationMode]": """Consolidation carries rendered tables; only dataset-scoped inputs need records.""" if bundle.requires_parent_runtime_observation: @@ -154,7 +150,7 @@ def for_compiled_bundle( def including_parent_requirement( cls, required: bool, - ) -> type["RuntimeObservationMode"]: + ) -> "type[RuntimeObservationMode]": """Return this mode strengthened by an additional retention requirement.""" return MergeAllRuntimeRecords if required else cls @@ -169,9 +165,10 @@ def retain_records( class MergeAllRuntimeRecords(RuntimeObservationMode): - """Return every worker record to the parent.""" + """Return every worker record to the parent. - returns_every_record = True + Workers keep each record to the end, so consumed inputs are never released. + """ @staticmethod def retain_records(records, context): @@ -216,8 +213,6 @@ def retain_records(records, context): class OmitRuntimeRecords(RuntimeObservationMode): """Return no worker records.""" - collects_records = False - @staticmethod def retain_records(records, context): return () diff --git a/openhcs/core/orchestrator/worker_execution.py b/openhcs/core/orchestrator/worker_execution.py index b55c419b8b..69409c8861 100644 --- a/openhcs/core/orchestrator/worker_execution.py +++ b/openhcs/core/orchestrator/worker_execution.py @@ -1488,7 +1488,7 @@ def _execute_single_axis_static( # original retention contract. Ordinary execution can retire dead inputs # only after the step's output/progress consumers have completed. release_consumed = ( - not runtime_observation_mode.returns_every_record + runtime_observation_mode is not MergeAllRuntimeRecords and isinstance(lane_context.debug_execution_policy, NoOpDebugExecutionPolicy) and not any( plan.visualize or plan.streaming_configs diff --git a/openhcs/core/pipeline/compiler.py b/openhcs/core/pipeline/compiler.py index d097ee052d..b3b3efd770 100644 --- a/openhcs/core/pipeline/compiler.py +++ b/openhcs/core/pipeline/compiler.py @@ -287,7 +287,7 @@ def _configure_input_conversion_if_needed( return vfs_config = session.global_config.vfs_config - if not vfs_config.materialization_backend.writes_zarr: + if vfs_config.materialization_backend is not MaterializeToZarr: return available_backends = context.microscope_handler.available_backends( @@ -507,7 +507,8 @@ def declare_zarr_stores(session: CompilationSession) -> None: session.plan(step_index).zarr_config = None if ( not session.step_count - or not session.global_config.vfs_config.materialization_backend.writes_zarr + or session.global_config.vfs_config.materialization_backend + is not MaterializeToZarr ): return last_step_index = session.step_count - 1 diff --git a/openhcs/core/pipeline/path_planner.py b/openhcs/core/pipeline/path_planner.py index 10c0a6b04c..fd21d5c280 100644 --- a/openhcs/core/pipeline/path_planner.py +++ b/openhcs/core/pipeline/path_planner.py @@ -71,10 +71,9 @@ source_binding_group_keys_for_group_by, ) from openhcs.core.step_dependencies import ( - NoMainFlowInput, - PipelineStartInput, StepInputDependency, StepOutputInput, + UnresolvedStepInput, ) from openhcs.core.steps.abstract import AbstractStep from openhcs.core.steps.function_step import FunctionStep @@ -1852,7 +1851,7 @@ def validate(self) -> None: for i in range(1, self.planner.session.step_count): curr = self.planner.session.pipeline.steps[i] dependency = self.planner.plans[i].main_input_dependency - if not dependency.is_resolved: + if isinstance(dependency, UnresolvedStepInput): raise ValueError(f"Step {curr.name} has unresolved main input dependency.") if not isinstance(dependency, StepOutputInput): continue @@ -2250,24 +2249,15 @@ def step_io_dirs( plan = self.planner.plans.get(step_index) if plan is not None and plan.input_dir is not None: input_dir = Path(plan.input_dir) - elif isinstance(main_input_dependency, PipelineStartInput): - input_dir = self.planner.initial_input - elif isinstance(main_input_dependency, NoMainFlowInput): - input_dir = self.planner.paths.build_output_path() else: - source_step_index = main_input_dependency.predecessor_step_index() - if source_step_index is None: - raise ValueError( - f"Step {step_index} main input dependency is unresolved." - ) - input_dir = Path(self.planner.plans[source_step_index].output_dir) + input_dir = main_input_dependency.default_input_dir(self.planner) if plan is not None and plan.output_dir is not None: output_dir = Path(plan.output_dir) - elif isinstance(main_input_dependency, PipelineStartInput | NoMainFlowInput): - output_dir = self.planner.paths.build_output_path() else: - output_dir = input_dir + output_dir = main_input_dependency.default_output_dir( + self.planner, input_dir + ) return input_dir, output_dir diff --git a/openhcs/core/selection.py b/openhcs/core/selection.py index f90c316e14..d05eb486a7 100644 --- a/openhcs/core/selection.py +++ b/openhcs/core/selection.py @@ -22,7 +22,7 @@ class SelectionScope(ABC, metaclass=AutoRegisterMeta): name: ClassVar[str | None] = None @staticmethod - def named(name: str) -> type["SelectionScope"]: + def named(name: str) -> type[SelectionScope]: scope = SelectionScope.__registry__.get(name) if scope is None: raise ValueError( diff --git a/openhcs/core/source_binding_workspace.py b/openhcs/core/source_binding_workspace.py index 0a85852221..9b50cfea91 100644 --- a/openhcs/core/source_binding_workspace.py +++ b/openhcs/core/source_binding_workspace.py @@ -508,7 +508,7 @@ class SourceBindingProjectionStrategy(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "role" __skip_if_no_key__ = True __registry__: ClassVar[ - dict[type[SourceProjectionRole], type["SourceBindingProjectionStrategy"]] + dict[type[SourceProjectionRole], type[SourceBindingProjectionStrategy]] ] = {} role: ClassVar[type[SourceProjectionRole] | None] = None @@ -1193,9 +1193,7 @@ def _source_set_projections( projections: list[SourceProjection] = [] for source_set in source_sets: well = self._source_set_well(source_set) - projection_indexes = { - role: 0 for role in SourceProjectionRole.__registry__.values() - } + projection_indexes = dict.fromkeys(SourceProjectionRole.__registry__.values(), 0) for binding in bindings: candidate = source_set.candidates_by_alias.get(binding.alias) if candidate is None: diff --git a/openhcs/core/source_bindings.py b/openhcs/core/source_bindings.py index 2a3c310950..5a3a815f99 100644 --- a/openhcs/core/source_bindings.py +++ b/openhcs/core/source_bindings.py @@ -64,7 +64,7 @@ class SourceBindingOrigin(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["SourceBindingOrigin"]]] = {} + __registry__: ClassVar[dict[str, type[SourceBindingOrigin]]] = {} name: ClassVar[str | None] = None """Spelling in forms and on the wire.""" @@ -87,7 +87,7 @@ class SourceSetRole(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["SourceSetRole"]]] = {} + __registry__: ClassVar[dict[str, type[SourceSetRole]]] = {} name: ClassVar[str | None] = None """Spelling in forms and on the wire.""" @@ -110,7 +110,7 @@ class SourceProjectionRole(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["SourceProjectionRole"]]] = {} + __registry__: ClassVar[dict[str, type[SourceProjectionRole]]] = {} name: ClassVar[str | None] = None """Spelling in forms and on the wire.""" @@ -145,7 +145,7 @@ class MetadataSource(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["MetadataSource"]]] = {} + __registry__: ClassVar[dict[str, type[MetadataSource]]] = {} name: ClassVar[str | None] = None """Spelling in forms and on the wire.""" @@ -181,7 +181,7 @@ class SourceFilterSubject(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["SourceFilterSubject"]]] = {} + __registry__: ClassVar[dict[str, type[SourceFilterSubject]]] = {} name: ClassVar[str | None] = None """Spelling in forms and on the wire.""" @@ -221,7 +221,7 @@ class SourceFilterMatchType(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["SourceFilterMatchType"]]] = {} + __registry__: ClassVar[dict[str, type[SourceFilterMatchType]]] = {} name: ClassVar[str | None] = None """Spelling in forms and on the wire.""" @@ -252,7 +252,7 @@ def matches(cls, file_path: str, text: str, value: str | None) -> bool: return cls.compare(text, value) @classmethod - def negation(cls) -> type["TextMatchType"]: + def negation(cls) -> type[TextMatchType]: """The comparison that holds exactly when this one does not.""" (negation,) = ( diff --git a/openhcs/core/step_dependencies.py b/openhcs/core/step_dependencies.py index de09a44ad9..d8cf951e67 100644 --- a/openhcs/core/step_dependencies.py +++ b/openhcs/core/step_dependencies.py @@ -4,7 +4,11 @@ from abc import ABC from dataclasses import dataclass -from typing import ClassVar +from pathlib import Path +from typing import TYPE_CHECKING, ClassVar + +if TYPE_CHECKING: + from openhcs.core.pipeline.path_planner import PathPlanner @dataclass(frozen=True, slots=True) @@ -32,6 +36,16 @@ def require_pipeline_start(self) -> None: f"{type(self).__name__}." ) + def default_input_dir(self, planner: PathPlanner) -> Path: + """The directory the step reads when its plan names none.""" + + raise ValueError(f"Step main input dependency is unresolved: {self!r}.") + + def default_output_dir(self, planner: PathPlanner, input_dir: Path) -> Path: + """The directory the step writes when its plan names none.""" + + return input_dir + @dataclass(frozen=True, slots=True) class UnresolvedStepInput(StepInputDependency): @@ -44,6 +58,12 @@ class UnresolvedStepInput(StepInputDependency): class NoMainFlowInput(StepInputDependency): """The step reads no main-flow image stack.""" + def default_input_dir(self, planner: PathPlanner) -> Path: + return planner.paths.build_output_path() + + def default_output_dir(self, planner: PathPlanner, input_dir: Path) -> Path: + return planner.paths.build_output_path() + @dataclass(frozen=True, slots=True) class PipelineStartInput(StepInputDependency): @@ -55,6 +75,12 @@ def uses_pipeline_start_anchors(self) -> bool: def require_pipeline_start(self) -> None: return None + def default_input_dir(self, planner: PathPlanner) -> Path: + return planner.initial_input + + def default_output_dir(self, planner: PathPlanner, input_dir: Path) -> Path: + return planner.paths.build_output_path() + @dataclass(frozen=True, slots=True) class StepOutputInput(StepInputDependency): @@ -69,3 +95,6 @@ def __post_init__(self) -> None: def predecessor_step_index(self) -> int: return self.source_step_index + + def default_input_dir(self, planner: PathPlanner) -> Path: + return Path(planner.plans[self.source_step_index].output_dir) diff --git a/openhcs/core/steps/function_runtime.py b/openhcs/core/steps/function_runtime.py index aa2c67291b..5fbc47987b 100644 --- a/openhcs/core/steps/function_runtime.py +++ b/openhcs/core/steps/function_runtime.py @@ -127,7 +127,7 @@ RuntimePlaneAxisValueProjection, RuntimePlaneProjection, ) -from openhcs.core.step_dependencies import PipelineStartInput, StepOutputInput +from openhcs.core.step_dependencies import StepOutputInput from openhcs.core.steps.function_output_manifest import ( NoStepOutputManifestMatch, ProducedOutputSemantics, @@ -1676,9 +1676,8 @@ def loaded_source_artifact_payload( if ( binding is None or self.invocation.key.position != 0 - or not isinstance( - self.group_data.execution_plan.main_input_dependency, PipelineStartInput - ) + or not self.group_data.execution_plan.main_input_dependency + .uses_pipeline_start_anchors() or binding.origin is not FromPipelineStart or request.context is not self.group_data.context ): diff --git a/openhcs/interop/cellprofiler/measurement_lookup.py b/openhcs/interop/cellprofiler/measurement_lookup.py index 24baec96b2..08c3222ced 100644 --- a/openhcs/interop/cellprofiler/measurement_lookup.py +++ b/openhcs/interop/cellprofiler/measurement_lookup.py @@ -90,7 +90,7 @@ class CellProfilerMeasurementFeatureKind(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["CellProfilerMeasurementFeatureKind"]]] = {} + __registry__: ClassVar[dict[str, type[CellProfilerMeasurementFeatureKind]]] = {} name: ClassVar[str | None] = None needs_prior_measurement_rows: ClassVar[bool] = True diff --git a/openhcs/interop/cellprofiler/module_settings.py b/openhcs/interop/cellprofiler/module_settings.py index 9aa19f5bd2..b2bf59d98e 100644 --- a/openhcs/interop/cellprofiler/module_settings.py +++ b/openhcs/interop/cellprofiler/module_settings.py @@ -56,7 +56,7 @@ class ModuleSettingRowRecord: class ModuleSettingCoverageRecord(ModuleSettingRowRecord): """Coverage status for one concrete CellProfiler setting row.""" - status: type["ModuleSettingCoverageStatus"] + status: type[ModuleSettingCoverageStatus] class ModuleSettingCoverageStatus(ABC, metaclass=AutoRegisterMeta): @@ -67,7 +67,7 @@ class ModuleSettingCoverageStatus(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["ModuleSettingCoverageStatus"]]] = {} + __registry__: ClassVar[dict[str, type[ModuleSettingCoverageStatus]]] = {} name: ClassVar[str | None] = None is_covered: ClassVar[bool] = True @@ -79,7 +79,7 @@ def for_setting( *, unmapped_kwargs: Mapping[str, Any], ignored_setting_names: frozenset[str], - ) -> type["ModuleSettingCoverageStatus"]: + ) -> type[ModuleSettingCoverageStatus]: """Return the status of one setting row after binding.""" if normalized_name not in unmapped_kwargs: return SettingBound diff --git a/openhcs/interop/cellprofiler/pipeline_import.py b/openhcs/interop/cellprofiler/pipeline_import.py index 40a35d3c18..cbc3a7db2e 100644 --- a/openhcs/interop/cellprofiler/pipeline_import.py +++ b/openhcs/interop/cellprofiler/pipeline_import.py @@ -964,7 +964,7 @@ def _lower_module_batch( ) == Counter(source_group_keys) has_unscoped_single_invocation = len(units) == 1 and not lineage_keys[0] follows_previous_main_flow = ( - len(units) == 1 and not processing_config.input_source.reads_pipeline_start + len(units) == 1 and processing_config.input_source is ReadPreviousStep ) public_pattern: FunctionPatternSyntax | None = None lowered_units = units diff --git a/openhcs/interop/cellprofiler/plate_workspace.py b/openhcs/interop/cellprofiler/plate_workspace.py index d48b146564..1ca26d4909 100644 --- a/openhcs/interop/cellprofiler/plate_workspace.py +++ b/openhcs/interop/cellprofiler/plate_workspace.py @@ -36,7 +36,7 @@ class CellProfilerPipelineStage(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["CellProfilerPipelineStage"]]] = {} + __registry__: ClassVar[dict[str, type[CellProfilerPipelineStage]]] = {} name: ClassVar[str | None] = None filename_token: ClassVar[str | None] = None @@ -45,7 +45,7 @@ class CellProfilerPipelineStage(ABC, metaclass=AutoRegisterMeta): """A folder's only pipeline of this stage is its default pipeline.""" @staticmethod - def of(cppipe_path: Path) -> type["CellProfilerPipelineStage"]: + def of(cppipe_path: Path) -> type[CellProfilerPipelineStage]: tokens = { token.casefold() for token in cppipe_path.stem.replace("-", "_").replace(" ", "_").split("_") @@ -201,9 +201,11 @@ def default_cppipe_path(self) -> Path | None: if same_named_cppipe in candidates: return same_named_cppipe - for stage in CellProfilerPipelineStage.__registry__.values(): - if not stage.chosen_when_unique: - continue + for stage in ( + stage + for stage in CellProfilerPipelineStage.__registry__.values() + if stage.chosen_when_unique + ): staged = tuple( path for path in candidates diff --git a/openhcs/mcp/dev_client_commands/ui.py b/openhcs/mcp/dev_client_commands/ui.py index bc49bc25f0..230946bfdb 100644 --- a/openhcs/mcp/dev_client_commands/ui.py +++ b/openhcs/mcp/dev_client_commands/ui.py @@ -20,6 +20,7 @@ ) from openhcs.agent.capabilities import agent_capabilities from openhcs.agent.dto.ui_bridge import ( + AllFields, UiActionInvokeRequest, UiCodeDocumentApplyRequest, UiCodeDocumentRequest, @@ -56,6 +57,7 @@ McpDevToolBatchResponse, McpDevToolCall, McpDevToolResult, + PollCompleted, PollFailed, PollTimedOut, WorkflowPollBaseline, @@ -433,7 +435,7 @@ async def await_events(after_sequence: int) -> int: status = terminal_status or PollTimedOut return ( status, - status.completed, + status is PollCompleted, poll_count, transient_poll_error_count, ) @@ -973,7 +975,7 @@ def calls_from_args( field_path_contains=tuple(args.contains), include_system_scopes=args.include_system_scopes, include_clean_fields=( - UiObjectStateFieldFilter.named(args.field_filter).keeps_every_field + UiObjectStateFieldFilter.named(args.field_filter) is AllFields ), include_container_fields=args.include_container_fields, field_filter=args.field_filter, diff --git a/openhcs/mcp/dev_client_core.py b/openhcs/mcp/dev_client_core.py index e4eb8fba83..7c82e2cc69 100644 --- a/openhcs/mcp/dev_client_core.py +++ b/openhcs/mcp/dev_client_core.py @@ -192,12 +192,10 @@ class WorkflowPollSummaryStatus(ABC, metaclass=AutoRegisterMeta): __skip_if_no_key__ = True name: ClassVar[str | None] = None - completed: ClassVar[bool] = False class PollCompleted(WorkflowPollSummaryStatus): name = "completed" - completed = True class PollFailed(WorkflowPollSummaryStatus): @@ -671,7 +669,7 @@ class WorkflowPollSummary: @property def mcp_error(self) -> bool: - return not self.status.completed + return self.status is not PollCompleted def as_payload(self) -> dict[str, JsonValue]: payload: dict[str, JsonValue] = { diff --git a/openhcs/mcp/dev_client_renderers/knowledge.py b/openhcs/mcp/dev_client_renderers/knowledge.py index a1e8211c03..07cd6cd0d0 100644 --- a/openhcs/mcp/dev_client_renderers/knowledge.py +++ b/openhcs/mcp/dev_client_renderers/knowledge.py @@ -18,6 +18,8 @@ FunctionCatalogPage, FunctionDetail, FunctionParameterSpec, + SuppliedAtRuntime, + SuppliedByAgent, ) from openhcs.agent.dto.knowledge import ( KnowledgeBaseCatalog, @@ -371,12 +373,12 @@ def render_payload( agent_parameters = tuple( parameter for parameter in payload.parameters - if parameter.supplied_by.agent_supplied + if issubclass(parameter.supplied_by, SuppliedByAgent) ) runtime_parameters = tuple( parameter for parameter in payload.parameters - if not parameter.supplied_by.agent_supplied + if issubclass(parameter.supplied_by, SuppliedAtRuntime) ) artifact_outputs = ( () diff --git a/openhcs/mcp/dev_client_renderers/object_state.py b/openhcs/mcp/dev_client_renderers/object_state.py index 3f77731329..5d986dd53e 100644 --- a/openhcs/mcp/dev_client_renderers/object_state.py +++ b/openhcs/mcp/dev_client_renderers/object_state.py @@ -3,6 +3,7 @@ from __future__ import annotations from openhcs.agent.dto.ui_bridge import ( + AllFields, UiCatalogPageMetadata, UiObjectStateFieldHelpResult, UiObjectStateFieldListResult, @@ -254,7 +255,7 @@ def _filter_lines(payload: UiObjectStateFieldListResult) -> tuple[str, ...]: filters.append("field_paths=" + ",".join(payload.field_paths)) if payload.field_path_contains: filters.append("contains=" + ",".join(payload.field_path_contains)) - if not payload.field_filter.keeps_every_field: + if payload.field_filter is not AllFields: filters.append(f"field_filter={payload.field_filter.name}") if payload.include_container_fields: filters.append("include_container_fields=True") diff --git a/openhcs/mcp/dev_client_rendering.py b/openhcs/mcp/dev_client_rendering.py index c2934f9492..d842d9947d 100644 --- a/openhcs/mcp/dev_client_rendering.py +++ b/openhcs/mcp/dev_client_rendering.py @@ -48,7 +48,7 @@ class WidgetTreeOutputFormat(ABC, metaclass=AutoRegisterMeta): """The request includes the nested tree and non-actionable branches.""" @staticmethod - def named(name: str) -> type["WidgetTreeOutputFormat"]: + def named(name: str) -> type[WidgetTreeOutputFormat]: return WidgetTreeOutputFormat.__registry__[name] @@ -270,7 +270,7 @@ class McpDevOutputRenderer(metaclass=AutoRegisterMeta): """ __registry__: ClassVar[ - dict[McpDevOutputRendererKey, type["McpDevOutputRenderer"]] + dict[McpDevOutputRendererKey, type[McpDevOutputRenderer]] ] = {} __registry_key__ = "renderer_key" __key_extractor__ = mcp_dev_output_renderer_key @@ -293,7 +293,7 @@ def __init_subclass__(cls, **kwargs): def for_output_contract( cls, output_contract: type | None, - ) -> type["McpDevOutputRenderer"] | None: + ) -> type[McpDevOutputRenderer] | None: if output_contract is None: return None from openhcs.mcp.dev_client_renderers import ( diff --git a/openhcs/mcp/http_auth.py b/openhcs/mcp/http_auth.py index 6d79cb5da1..455cb5c176 100644 --- a/openhcs/mcp/http_auth.py +++ b/openhcs/mcp/http_auth.py @@ -42,7 +42,7 @@ def tool_security_schemes( """Project this mode into MCP tool security-scheme metadata.""" @staticmethod - def named(name: str) -> type["McpHttpAuthenticationMode"]: + def named(name: str) -> type[McpHttpAuthenticationMode]: """Return the mode spelled ``name``.""" mode = McpHttpAuthenticationMode.__registry__.get(name) if mode is None: @@ -55,7 +55,7 @@ def named(name: str) -> type["McpHttpAuthenticationMode"]: @staticmethod def from_oauth_policy( oauth: "McpHttpOAuthSettings | None", - ) -> type["McpHttpAuthenticationMode"]: + ) -> type[McpHttpAuthenticationMode]: """Derive the mode from the materialized variant policy.""" if oauth is None: return PublicReadOnlyAccess diff --git a/openhcs/mcp/lifecycle.py b/openhcs/mcp/lifecycle.py index 966213e2bc..c8025a71c9 100644 --- a/openhcs/mcp/lifecycle.py +++ b/openhcs/mcp/lifecycle.py @@ -73,7 +73,7 @@ def from_drift( *, source_changed: bool, installation_changed: bool, - ) -> type["McpProcessRecoveryReason"]: + ) -> type[McpProcessRecoveryReason]: if source_changed and installation_changed: return SourceAndInstallationChanged if source_changed: diff --git a/openhcs/napari_roi_manager/widgets/_roi_manager.py b/openhcs/napari_roi_manager/widgets/_roi_manager.py index a33a2e2f33..22e618787c 100644 --- a/openhcs/napari_roi_manager/widgets/_roi_manager.py +++ b/openhcs/napari_roi_manager/widgets/_roi_manager.py @@ -100,10 +100,9 @@ class RoiTableColumn(ABC, metaclass=AutoRegisterMeta): index: ClassVar[int | None] = None header: ClassVar[str] - editable: ClassVar[bool] = False @classmethod - def at(cls, section: int) -> type["RoiTableColumn"]: + def at(cls, section: int) -> type[RoiTableColumn]: return cls.__registry__[section] @staticmethod @@ -119,7 +118,6 @@ def cell(cls, layer: Shapes, row: int) -> str: class RoiNameColumn(RoiTableColumn): index = 0 header = "name" - editable = True @staticmethod def values(layer: Shapes) -> list[str]: @@ -208,7 +206,7 @@ def headerData( def flags(self, index): flags = super().flags(index) - if index.isValid() and RoiTableColumn.at(index.column()).editable: + if index.isValid() and RoiTableColumn.at(index.column()) is RoiNameColumn: flags |= QtCore.Qt.ItemFlag.ItemIsEditable return flags @@ -217,7 +215,7 @@ def setData(self, index, value, role=QtCore.Qt.ItemDataRole.EditRole) -> bool: role != QtCore.Qt.ItemDataRole.EditRole or not index.isValid() or self._layer is None - or not RoiTableColumn.at(index.column()).editable + or RoiTableColumn.at(index.column()) is not RoiNameColumn ): return False names = RoiNameColumn.values(self._layer) diff --git a/openhcs/processing/backends/cellprofiler/grid.py b/openhcs/processing/backends/cellprofiler/grid.py index 5349a541a0..872d7d3fd1 100644 --- a/openhcs/processing/backends/cellprofiler/grid.py +++ b/openhcs/processing/backends/cellprofiler/grid.py @@ -131,14 +131,14 @@ class DefineGridVariant(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "function_name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["DefineGridVariant"]]] = {} + __registry__: ClassVar[dict[str, type[DefineGridVariant]]] = {} function_name: ClassVar[str | None] = None requires_object_input: ClassVar[bool] """The variant derives the grid from object labels.""" @staticmethod - def from_module(module: ModuleBlock) -> type["DefineGridVariant"]: + def from_module(module: ModuleBlock) -> type[DefineGridVariant]: method = DefineGridManualModule.DefinitionMethod.from_literal( _setting_value( module, @@ -153,7 +153,7 @@ def from_module(module: ModuleBlock) -> type["DefineGridVariant"]: ) @staticmethod - def for_function(function_name: str) -> type["DefineGridVariant"]: + def for_function(function_name: str) -> type[DefineGridVariant]: return DefineGridVariant.__registry__[function_name] @@ -204,7 +204,7 @@ def define_grid_bound_kwargs( ) ), } - if not DefineGridVariant.from_module(module).requires_object_input: + if DefineGridVariant.from_module(module) is ManualGrid: first_x, first_y = _coordinate_pair( _setting_value( module, @@ -494,7 +494,7 @@ def bind_settings(cls, module, *, binder): """Bind scalar rows and the two coordinate-pair compound rows.""" variant = DefineGridVariant.from_module(module) - if not variant.requires_object_input: + if variant is ManualGrid: manual_method = optional_setting_value( module, cls.manual_definition_method_setting ) @@ -863,7 +863,7 @@ class SpatialGridDefinitionBase(ABC, metaclass=AutoRegisterMeta): @classmethod def registered_definition_types( cls, - ) -> tuple[type["SpatialGridDefinitionBase"], ...]: + ) -> tuple[type[SpatialGridDefinitionBase], ...]: return tuple(cls.__registry__.values()) def canonical_row_index(self, row: int) -> int: @@ -1287,7 +1287,7 @@ class GridShapeContext(ABC, metaclass=AutoRegisterMeta): circle_diameter: int = 20 @classmethod - def registered_context_types(cls) -> tuple[type["GridShapeContext"], ...]: + def registered_context_types(cls) -> tuple[type[GridShapeContext], ...]: return tuple(cls.__registry__.values()) diff --git a/openhcs/processing/backends/cellprofiler/morphology.py b/openhcs/processing/backends/cellprofiler/morphology.py index ae97922906..f5cecff656 100644 --- a/openhcs/processing/backends/cellprofiler/morphology.py +++ b/openhcs/processing/backends/cellprofiler/morphology.py @@ -5886,7 +5886,7 @@ class SplitOrMergeInputTopology(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "function_name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["SplitOrMergeInputTopology"]]] = {} + __registry__: ClassVar[dict[str, type[SplitOrMergeInputTopology]]] = {} function_name: ClassVar[str | None] = None uses_guide_image: ClassVar[bool] = False @@ -5898,7 +5898,7 @@ def from_values( operation: SplitOrMergeOperation, merge_method: SplitOrMergeMergeMethod, use_guide_image: bool, - ) -> type["SplitOrMergeInputTopology"]: + ) -> type[SplitOrMergeInputTopology]: if operation is SplitOrMergeOperation.SPLIT: return SplitOrMergeLabelsOnly if merge_method is SplitOrMergeMergeMethod.PER_PARENT: diff --git a/openhcs/processing/backends/cellprofiler/thresholding.py b/openhcs/processing/backends/cellprofiler/thresholding.py index c45f93cb1f..7775565a92 100644 --- a/openhcs/processing/backends/cellprofiler/thresholding.py +++ b/openhcs/processing/backends/cellprofiler/thresholding.py @@ -915,7 +915,7 @@ class ThresholdDiagnosticRequest: """Typed inputs for one CellProfiler threshold diagnostic measurement.""" backend: "NumbaNumpyThresholdDiagnosticsBackendStrategy" - domain: type["ThresholdDiagnosticDomain"] + domain: type[ThresholdDiagnosticDomain] image: np.ndarray mask: np.ndarray | None binary_image: np.ndarray @@ -961,12 +961,12 @@ class ThresholdDiagnosticDomain(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["ThresholdDiagnosticDomain"]]] = {} + __registry__: ClassVar[dict[str, type[ThresholdDiagnosticDomain]]] = {} name: ClassVar[str | None] = None @staticmethod - def of_rank(ndim: int) -> type["ThresholdDiagnosticDomain"]: + def of_rank(ndim: int) -> type[ThresholdDiagnosticDomain]: """Planar images use the 2-D path; every other rank is one image.""" return PlanarThresholdDiagnostics if ndim == 2 else WholeImageThresholdDiagnostics diff --git a/openhcs/processing/backends/cellprofiler/watershed.py b/openhcs/processing/backends/cellprofiler/watershed.py index 7c102eae8d..bd9a31393a 100644 --- a/openhcs/processing/backends/cellprofiler/watershed.py +++ b/openhcs/processing/backends/cellprofiler/watershed.py @@ -659,7 +659,7 @@ def topology(self) -> WatershedInputTopology: def execute( self, - runtime: type["WatershedRuntimeStrategy"], + runtime: type[WatershedRuntimeStrategy], special_inputs: tuple[object, ...] = (), ) -> tuple[np.ndarray, DataclassMeasurementColumnarRows, ObjectLabelValue]: """Apply this request through its declaration-owned runtime family.""" diff --git a/openhcs/processing/materialization/options.py b/openhcs/processing/materialization/options.py index aad9451aca..d0a737c4a2 100644 --- a/openhcs/processing/materialization/options.py +++ b/openhcs/processing/materialization/options.py @@ -29,7 +29,7 @@ class MaterializedFilenameIdentity(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["MaterializedFilenameIdentity"]]] = {} + __registry__: ClassVar[dict[str, type[MaterializedFilenameIdentity]]] = {} name: ClassVar[str | None] = None diff --git a/openhcs/pyqt_gui/app.py b/openhcs/pyqt_gui/app.py index af3fb15cbe..549e5f7ec7 100644 --- a/openhcs/pyqt_gui/app.py +++ b/openhcs/pyqt_gui/app.py @@ -50,8 +50,7 @@ def report_painted_ready(readiness: "MainWindowStartupReadiness") -> None: @staticmethod def fail(readiness: "MainWindowStartupReadiness", error: BaseException) -> None: readiness._finish() - if readiness._on_failure is not None: - readiness._on_failure(error) + readiness._report_failure(error) class WaitingForDeferredInitialization(StartupReadinessState): @@ -73,14 +72,7 @@ class PaintCompletionQueued(StartupReadinessState): def report_painted_ready(readiness: "MainWindowStartupReadiness") -> None: readiness._finish() logger.info("OpenHCS main window painted and ready") - if readiness._on_ready is None: - return - try: - readiness._on_ready() - except Exception as error: - if readiness._on_failure is None: - raise - readiness._on_failure(error) + readiness._report_ready() class StartupReadinessFinished(StartupReadinessState): @@ -125,6 +117,20 @@ def eventFilter(self, watched, event) -> bool: # noqa: N802 def _report_painted_ready(self) -> None: self._state.report_painted_ready(self) + def _report_failure(self, error: BaseException) -> None: + if self._on_failure is not None: + self._on_failure(error) + + def _report_ready(self) -> None: + if self._on_ready is None: + return + try: + self._on_ready() + except Exception as error: + if self._on_failure is None: + raise + self._on_failure(error) + def _finish(self) -> None: self._state = StartupReadinessFinished self._main_window.removeEventFilter(self) diff --git a/openhcs/pyqt_gui/main.py b/openhcs/pyqt_gui/main.py index 8561d33b03..fc933bcc43 100644 --- a/openhcs/pyqt_gui/main.py +++ b/openhcs/pyqt_gui/main.py @@ -1439,7 +1439,7 @@ def _on_update_check_completed( self.check_for_updates_action.setEnabled(True) update = result.update if not update.update_available: - if not result.origin.quiet: + if result.origin is ExplicitUpdateCheck: self.status_message.emit("OpenHCS is up to date") self.desktop_update_presenter.show_up_to_date(update) return diff --git a/openhcs/pyqt_gui/services/logging_config.py b/openhcs/pyqt_gui/services/logging_config.py index f7d18fae55..3cd89eef3b 100644 --- a/openhcs/pyqt_gui/services/logging_config.py +++ b/openhcs/pyqt_gui/services/logging_config.py @@ -75,14 +75,14 @@ def configure_gui_logging( root_logger = logging.getLogger() previous_handlers = tuple(root_logger.handlers) root_logger.handlers = handlers - if not level.emits_records: - logging.disable(logging.CRITICAL) - root_logger.setLevel(level.logging_level) - else: + if level.emits_records: logging.disable(logging.NOTSET) root_logger.setLevel(level.logging_level) logging.getLogger("openhcs").setLevel(level.logging_level) logging.getLogger("PIL").setLevel(logging.WARNING) + else: + logging.disable(logging.CRITICAL) + root_logger.setLevel(level.logging_level) for handler in previous_handlers: if handler not in handlers and isinstance(handler, GuiLoggingHandler): diff --git a/openhcs/pyqt_gui/services/ui_bridge_object_state.py b/openhcs/pyqt_gui/services/ui_bridge_object_state.py index 8860cf92c7..178d982182 100644 --- a/openhcs/pyqt_gui/services/ui_bridge_object_state.py +++ b/openhcs/pyqt_gui/services/ui_bridge_object_state.py @@ -29,6 +29,7 @@ from openhcs.agent.dto.common import AgentError, SCHEMA_VERSION from openhcs.agent.dto.ui_bridge import ( + AllFields, UiCatalogPageMetadata, UiCodeDocument, UiCodeDocumentApplyRequest, @@ -421,7 +422,7 @@ def _field_page( ) else: all_field_paths = tuple(sorted(state.parameters.keys())) - if not request.field_filter.keeps_every_field: + if request.field_filter is not AllFields: all_field_paths = tuple( field_path for field_path in all_field_paths diff --git a/openhcs/pyqt_gui/widgets/shared/services/pipeline_debug_actions.py b/openhcs/pyqt_gui/widgets/shared/services/pipeline_debug_actions.py index be6e81479b..6c409065c7 100644 --- a/openhcs/pyqt_gui/widgets/shared/services/pipeline_debug_actions.py +++ b/openhcs/pyqt_gui/widgets/shared/services/pipeline_debug_actions.py @@ -48,7 +48,7 @@ class DebugToolbarPlacement(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["DebugToolbarPlacement"]]] = {} + __registry__: ClassVar[dict[str, type[DebugToolbarPlacement]]] = {} name: ClassVar[str | None] = None shown: ClassVar[bool] = True @@ -57,7 +57,7 @@ class DebugToolbarPlacement(ABC, metaclass=AutoRegisterMeta): """Position of this placement's button panel in the toolbar.""" @classmethod - def panels(cls) -> tuple[type["DebugToolbarPlacement"], ...]: + def panels(cls) -> tuple[type[DebugToolbarPlacement], ...]: """The placements the toolbar renders as button panels, in order.""" return tuple( @@ -96,7 +96,7 @@ class PipelineDebugActionDeclarationBase(ABC, metaclass=AutoRegisterMeta): __registry__: ClassVar[ dict[ type[DebugCommandType] | str, - type["PipelineDebugActionDeclarationBase"], + type[PipelineDebugActionDeclarationBase], ] ] = {} __registry_key__ = "identity" @@ -131,7 +131,7 @@ def is_toolbar_action(cls) -> bool: return cls.toolbar_placement.shown @classmethod - def toolbar_actions(cls) -> tuple[type["PipelineDebugActionDeclarationBase"], ...]: + def toolbar_actions(cls) -> tuple[type[PipelineDebugActionDeclarationBase], ...]: return tuple( sorted( ( @@ -150,7 +150,7 @@ def toolbar_actions(cls) -> tuple[type["PipelineDebugActionDeclarationBase"], .. @classmethod def for_action_id( cls, action_id: str - ) -> type["PipelineDebugActionDeclarationBase"]: + ) -> type[PipelineDebugActionDeclarationBase]: for declaration in cls.__registry__.values(): if declaration.action_id() == action_id: return declaration @@ -160,7 +160,7 @@ def for_action_id( def for_command_type( cls, command_type: type[DebugCommandType], - ) -> type["PipelineDebugCommandActionDeclaration"]: + ) -> type[PipelineDebugCommandActionDeclaration]: declaration = cls.__registry__.get(command_type) if declaration is None: raise ValueError(f"Unhandled debug command route: {command_type.name}") diff --git a/openhcs/runtime/napari_streaming_handlers.py b/openhcs/runtime/napari_streaming_handlers.py index 8cd957fc70..ac5e9a5f45 100644 --- a/openhcs/runtime/napari_streaming_handlers.py +++ b/openhcs/runtime/napari_streaming_handlers.py @@ -404,7 +404,7 @@ def begin_next( with self._lock: if self.updates is None: return None - if not self.phase.running: + if self.phase is not SettlementRunning: return None if self.active_route is not None: raise RuntimeError( diff --git a/openhcs/runtime/napari_viewer_server.py b/openhcs/runtime/napari_viewer_server.py index 2204fbd8c7..37a0f63598 100644 --- a/openhcs/runtime/napari_viewer_server.py +++ b/openhcs/runtime/napari_viewer_server.py @@ -4322,8 +4322,7 @@ def payload_summary( if isinstance(data, np.ndarray): summary.update(cls.array_summary(data)) elif isinstance(data, (list, tuple)): - summary["item_count"] = len(data) - summary["nonzero_count"] = len(data) + summary.update(item_count=len(data), nonzero_count=len(data)) summary.update(cls.shape_payload_summary(data)) return cls.payload_summary_type.from_wire_mapping(summary) diff --git a/openhcs/runtime/viewer_display.py b/openhcs/runtime/viewer_display.py index 876b25857e..856e41b63b 100644 --- a/openhcs/runtime/viewer_display.py +++ b/openhcs/runtime/viewer_display.py @@ -341,7 +341,7 @@ def __init_subclass__(cls, **kwargs: object) -> None: ViewerDisplaySettings._viewer_settings.append(cls) @classmethod - def settings_type(cls) -> type["ViewerDisplaySettings"]: + def settings_type(cls) -> "type[ViewerDisplaySettings]": """The viewer settings class this class is or inherits.""" return next( @@ -379,7 +379,7 @@ class NapariVariableSizeHandling(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["NapariVariableSizeHandling"]]] = {} + __registry__: "ClassVar[dict[str, type[NapariVariableSizeHandling]]]" = {} name: ClassVar[str | None] = None diff --git a/openhcs/runtime/viewer_protocol.py b/openhcs/runtime/viewer_protocol.py index ffcfce2118..ea64345903 100644 --- a/openhcs/runtime/viewer_protocol.py +++ b/openhcs/runtime/viewer_protocol.py @@ -129,7 +129,7 @@ class ViewerSettlePhase(ABC, metaclass=AutoRegisterMeta): __registry_key__ = "name" __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["ViewerSettlePhase"]]] = {} + __registry__: ClassVar[dict[str, type[ViewerSettlePhase]]] = {} name: ClassVar[str | None] = None running: ClassVar[bool] = False @@ -162,7 +162,7 @@ def validate(cls, progress: "ViewerSettleProgress") -> None: raise ValueError("Completed settlement cannot have active intake.") if ( progress.completed_update_count != progress.total_update_count - or progress.active_route is not None + or progress.routing ): raise ValueError( "Completed viewer settlement must have no active route and all " @@ -699,6 +699,11 @@ class ViewerSettleProgress(WireView): processed_intake_item_count: int = 0 intake_item_active: bool = False + @property + def routing(self) -> bool: + """An update is being routed to a layer.""" + return self.active_route is not None + def __post_init__(self) -> None: if ( type(self.processed_intake_item_count) is not int @@ -1578,7 +1583,7 @@ def rollback_failed_bootstrap(cls, viewer: "ManagedViewerLifecycleMixin") -> Non viewer.viewer_family.wire_value, port, stop=True, force=True, ) return - if cls.owns_process or viewer.process is not None: + if cls.owns_process or viewer.has_process: viewer.force_stop() return viewer.lifecycle = ViewerStopped @@ -1621,11 +1626,8 @@ class ViewerOwnedProcess(ViewerLifecycle): @classmethod def is_running(cls, viewer: "ManagedViewerLifecycleMixin") -> bool: log = logging.getLogger(type(viewer).__module__) - if viewer.process is None: - viewer.lifecycle = ViewerStopped - return False try: - alive = endpoint_process(viewer.process).is_alive() + alive = viewer.process_alive() except Exception as error: log.warning( "Error checking %s process status: %s", @@ -1775,7 +1777,7 @@ class ManagedViewerLifecycleMixin( __key_extractor__ = staticmethod(viewer_lifecycle_registry_key) __skip_if_no_key__ = True - __registry__: ClassVar[dict[str, type["ManagedViewerLifecycleMixin"]]] + __registry__: ClassVar[dict[str, type[ManagedViewerLifecycleMixin]]] viewer_registry_key: ClassVar[str | None] = None viewer_process_label: ClassVar[str] = "viewer" detached_server_entrypoint: ClassVar[DetachedViewerServerEntrypointSpec] @@ -1803,6 +1805,17 @@ def __init__( ) self.lifecycle: type[ViewerLifecycle] = ViewerStopped + @property + def has_process(self) -> bool: + """This manager holds a launched viewer process.""" + return self.process is not None + + def process_alive(self) -> bool: + """Whether this manager's launched viewer process is still alive.""" + if self.process is None: + return False + return endpoint_process(self.process).is_alive() + @property def required_port(self) -> int: port = self.port diff --git a/openhcs/runtime/zmq_execution_observation.py b/openhcs/runtime/zmq_execution_observation.py index 0fedf0abb6..e3b01be9a0 100644 --- a/openhcs/runtime/zmq_execution_observation.py +++ b/openhcs/runtime/zmq_execution_observation.py @@ -10,6 +10,7 @@ from openhcs.core.context.processing_context import ProcessingContext from openhcs.core.orchestrator.execution_result import ( + AxisSucceeded, ExecutionResult, RuntimeExecutionObservation, ExecutionStatus, @@ -142,7 +143,7 @@ def require_successful_axes(self) -> None: unsuccessful = { axis_id: outcome.status.name for axis_id, outcome in self.outcomes_by_axis.items() - if not outcome.status.succeeded + if outcome.status is not AxisSucceeded } failures = list(membership_failures) if unsuccessful: diff --git a/openhcs/runtime/zmq_execution_signature.py b/openhcs/runtime/zmq_execution_signature.py index d95fde7239..d44a56b943 100644 --- a/openhcs/runtime/zmq_execution_signature.py +++ b/openhcs/runtime/zmq_execution_signature.py @@ -55,8 +55,12 @@ class ZMQRuntimeObservationExportScope(ABC, metaclass=AutoRegisterMeta): retains_runtime_values: ClassVar[bool] """Workers return their full runtime values for the export.""" + @classmethod + def check_export_path(cls, export_path: object) -> None: + """Reject a request that lacks the export path this scope needs.""" + @staticmethod - def named(value: str) -> type["ZMQRuntimeObservationExportScope"]: + def named(value: str) -> type[ZMQRuntimeObservationExportScope]: scope = ZMQRuntimeObservationExportScope.__registry__.get(value) if scope is None: raise ValueError(f"Unknown runtime observation export scope: {value!r}.") @@ -105,6 +109,11 @@ class ExportRuntimeOutcomes(ZMQRuntimeObservationExportScope): name = "outcomes" retains_runtime_values = False + @classmethod + def check_export_path(cls, export_path: object) -> None: + if export_path is None: + raise ValueError("Outcome observation requires an export path.") + @staticmethod def export( *, execution_bundle, execution_results, output_roots, server_environment, execution_id @@ -138,11 +147,9 @@ class ZMQAuxiliaryExecutionParams: ) def __post_init__(self) -> None: - if ( - not self.runtime_observation_export_scope.retains_runtime_values - and self.runtime_observation_export_path is None - ): - raise ValueError("Outcome observation requires an export path.") + self.runtime_observation_export_scope.check_export_path( + self.runtime_observation_export_path + ) def runtime_observation_mode_for( self, From c3bbadb013d24d84e787c515f30693b696034ec8 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:43:04 -0400 Subject: [PATCH 31/36] E1: the thesis demo decodes the applied document through its assignments record Co-Authored-By: Claude Opus 5.5 --- scripts/mcp_thesis_demo_live.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/scripts/mcp_thesis_demo_live.py b/scripts/mcp_thesis_demo_live.py index 8676e262b5..bb9dd643e5 100755 --- a/scripts/mcp_thesis_demo_live.py +++ b/scripts/mcp_thesis_demo_live.py @@ -75,6 +75,7 @@ PlateManagerWidgetIdentity, ) from openhcs.core.config_cache import ConfigCacheSpec, save_config_sync +from openhcs.ui.shared.plate_manager_code_document import PlateManagerDocumentAssignments from openhcs.mcp.dev_client import McpDevClient from openhcs.mcp.dev_client_core import ( DEFAULT_CALL_TIMEOUT_SECONDS, @@ -1966,8 +1967,9 @@ def assert_applied_document_state( ) -> dict[str, Any]: namespace: dict[str, object] = {} exec(compile(source, str(ctx.source_path), "exec"), namespace) - per_plate_configs = namespace["per_plate_configs"] - pipeline_data = namespace["pipeline_data"] + assignments = PlateManagerDocumentAssignments.from_namespace(namespace) + per_plate_configs = assignments.per_plate_configs + pipeline_data = assignments.pipeline_data if not isinstance(per_plate_configs, dict) or not isinstance(pipeline_data, dict): raise RehearsalFailure( "Applied document did not reconstruct pipeline mappings." From 9a1476aed13719f50b7eaf46c842dc14f6b6f376 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:48:49 -0400 Subject: [PATCH 32/36] E1: tests ask the observation and parameter-source families directly Co-Authored-By: Claude Opus 5.5 --- tests/unit/test_function_catalog_zmq.py | 2 +- tests/unit/test_zmq_execution_server.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/unit/test_function_catalog_zmq.py b/tests/unit/test_function_catalog_zmq.py index f4830d5c6b..d1b514707f 100644 --- a/tests/unit/test_function_catalog_zmq.py +++ b/tests/unit/test_function_catalog_zmq.py @@ -66,7 +66,7 @@ def test_function_parameter_source_owns_runtime_presentation() -> None: - assert SuppliedByAgent.runtime_description is None + assert SuppliedByAgent.runtime_description == "" assert "FunctionStep input image payload" in ( SuppliedFromPrimaryInput.runtime_description or "" ) diff --git a/tests/unit/test_zmq_execution_server.py b/tests/unit/test_zmq_execution_server.py index 762dc552f9..feac85f2cd 100644 --- a/tests/unit/test_zmq_execution_server.py +++ b/tests/unit/test_zmq_execution_server.py @@ -256,7 +256,7 @@ def test_zmq_auxiliary_params_strengthen_compiled_observation_requirement( else None ) execution_bundle = SimpleNamespace( - requires_parent_runtime_observation=compiled_mode.collects_records, + requires_parent_runtime_observation=compiled_mode is not OmitRuntimeRecords, ) assert params.runtime_observation_mode_for(execution_bundle) is expected_mode From e1b1864606dcef98a66356487d1601a1d255ad07 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 11:50:07 -0400 Subject: [PATCH 33/36] E1: receipt records the census outcome, reassignments and corrections Co-Authored-By: Claude Opus 5.5 --- .../E1-leftover-families.md | 32 ++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/docs/refactor/consolidation-20261009/E1-leftover-families.md b/docs/refactor/consolidation-20261009/E1-leftover-families.md index b4e1d47b7d..80bca0a901 100644 --- a/docs/refactor/consolidation-20261009/E1-leftover-families.md +++ b/docs/refactor/consolidation-20261009/E1-leftover-families.md @@ -42,7 +42,7 @@ Saved `.py` pipelines and configs that name the six source-binding enums (`Sourc `tests/unit/test_e1_enum_dispatch_guards.py`: - No module under `openhcs/` dispatches on an `Enum` member (an `is`/`is not`/`==`/`!=`/`in` comparison with `Enum.MEMBER`, a `match` case on one, a dict display keyed by one, or an `EnumKeyedStrategyMixin` key) unless the enum is listed as a boundary spelling or reassigned in the guard's table, each entry naming its module and reason; the table only shrinks. -- The deleted names (the six source-binding enums, `SourceFilterMatcher`, `SourceFilterTargetResolver`, `DebugSessionPhase`, `PipelineDebugSessionPhase`, `DebugActionPlacement`, `DebugToolbarAuxiliaryAction`, `CompiledArtifactInspectionControlPayload`, `ArtifactInspectionControlMessageType`, `_json_value`, `_jsonable_metadata_value`, `_jsonable_scalar`, `_jsonable_array_values`) stay deleted in the live trees. +- The old member spellings of the six source-binding kinds (`SourceFilterSubject.FILE`, …) and the deleted names (`SourceFilterMatcher`, `SourceFilterTargetResolver`, `DebugSessionPhaseDeclarationBase`, `PipelineDebugSessionPhase`, `DebugActionPlacement`, `DebugToolbarAuxiliaryAction`, `CompiledArtifactInspectionControlPayload`, `ArtifactInspectionControlMessageType`, `_jsonable_metadata_value`, `_jsonable_scalar`, `_jsonable_array_values`) stay deleted in the live trees. - `core/source_projection.py`, `core/dataset_sources/` and `microscopes/bioformats_adapter.py` call no `.resolve(`. ## Tests @@ -59,3 +59,33 @@ Saved `.py` pipelines and configs that name the six source-binding enums (`Sourc ## Done when The brief's items are gone as named; every dispatched enum in the census is converted, justified or reassigned in the guard's table; the guards, the focused suites for touched code, the structural ratchet and nra-r1 pass with no regression. + +## Outcome and corrections + +Production (Python under `openhcs/`) −4095 +4010; tests −1517 +2120; benchmark and scripts −70 +63. Guards 6/6. Structural ratchet (openhcs, scripts, benchmark) and NRA R1 exit 0. Library PR: python-introspect #22 (0.2.8, `RegisteredClassChoices` validates registry keys in pydantic, so MCP tool schemas accept family-valued fields); the submodule points at its branch head and the `>=0.2.7` pin moves to `>=0.2.8` once it is released. + +The census ran as five batches (this worktree plus four in parallel, merged here). Every enum that `test_e1_enum_dispatch_guards.py` finds dispatched is converted, deleted, or listed in its `ALLOWED` table with a reason. + +**Converted to families or deleted** (members own the behaviour that was switched on; wire spellings are the old `.value` strings): +- Source bindings: `SourceBindingOrigin`, `SourceSetRole`, `SourceProjectionRole`, `MetadataSource`, `SourceFilterSubject`, `SourceFilterMatchType` (matcher and target-resolver families deleted; `CellProfilerSourceFilterSubject`/`CellProfilerSourceFilterOperator` replaced by the registry and one literal table), `SourceImageSetComponentRole`, `SourceAnchorSelectionStatus` (written, never read), `SourceBindingMatchResolutionStatus`, `StepInputDependencyKind` (the dependency record is now a family: `UnresolvedStepInput`, `NoMainFlowInput`, `PipelineStartInput`, `StepOutputInput`), `InputSource` (`ReadPreviousStep`, `ReadPipelineStart`). +- Debug and GUI: `DebugSessionPhase` (the phase declarations are the phases), `DebugActionPlacement`, `DebugToolbarAuxiliaryAction`, `ArtifactInspectionControlMessageType`, `MainWindowStartupReadinessState`, `_StartupWindowSignal`, `RoiTableColumn`, `SelectedAllSelectionMode`, `DesktopUpdateCheckOrigin`, `DesktopUpdateProgressAction`, `PlateManagerCodeNamespaceField` (a record). +- Runtime and viewers: `OrchestratorState`, `TerminalExecutionStatus`, `ExecutionStatus` (core), `RuntimeObservationMode`, `ZMQRuntimeObservationExportScope`, `MaterializationBackend`, `MaterializedFilenameIdentity`, `CustomFunctionLifetime`, `MfdPresetKey`/`MfdPresetFamily` (one definition per preset), `ViewerSettlePhase`, `ViewerSettleField`, `ViewerPayloadSummaryField`, `ViewerProcessLaunchField`, `ViewerLifecycleMode`, `ViewerLaunchContextMode`, `NapariLayerKind`, `NapariDimensionLabelRouteSource`, `NapariVariableSizeHandling`. +- Agent and MCP: `FunctionParameterSource` (agent- and runtime-supplied branches), `FunctionCatalogControlField`, `FunctionCatalogPreparationStatus`, `SelectedPlateFileQueryTarget`, `UiBridgeOperationStatus`, `UiObjectStateFieldFilter`, `UiActionInvocationStatus`, `UiWindowSemanticMarker`, `DescriptorSetCardinality`, `SummaryView`, `SignatureView`, `CatalogSearchRank`, `WorkflowPollSummaryStatus`, `WidgetTreeOutputFormat`, `McpProcessRecoveryReason`, `McpHttpAuthenticationMode`, `ClientRegistrationStatus`, `AgentRuntimePlatformKey`. +- CellProfiler and labels: `CellProfilerPipelineStage`, `CellProfilerPipelineFileKind`, `ModuleSettingCoverageStatus`, `CellProfilerMeasurementFeatureKind`, `MeasurementImageReferenceDomain`; internal processing kinds `ThresholdDiagnosticDomain`, `WatershedRuntimeFamily`, `ExperimentalAnalysisScope`, `DefineGridVariant`, `SplitOrMergeInputTopology`. + +**Boundary spellings kept** (guard table): `processing/backends/` function-parameter choices and CellProfiler setting tables, dispatched only by the declaring function's module and its backend twins (checked module by module: every one is a parameter or CP literal); `CPAPropertyName`, `CellProfilerObjectTableMode`, `CellProfilerSourceImageProjectionField`, `CellProfilerMeasurementTargetScope` (CellProfiler literal tables); `McpWireMethod`, `McpWireContentType` (MCP); `QtPlatformName` (Qt QPA names); `PlateInspectionIssueCode` (`AgentError.code` spellings); `MultiprocessingStartMethod` (stdlib names; converting it showed ObjectState does not pass a class-valued top-level `GlobalPipelineConfig` field down to lazy `PipelineConfig`, which is L6's). `CellProfilerSpatialGridMeasurementFeature` is only a CellProfiler feature-name table: its one reader is its own `render()`, nothing compares its members. + +**Reassigned** (guard table, by owner): +- K3 (#1199's measurement files): `MeasurementScope`, `MeasurementRowAxisField`, `MeasurementRowValueField`, `MeasurementStatistic`, `ObjectFeatureArrayDomain`, `RuntimeCellValueKind`, `RuntimeMeasurementFeatureNameMode`, `RuntimeMeasurementQualifierValueMode`, `RuntimeMeasurementSourceNameEncoding`, `MeasurementTableRowLayout`, `MeasurementObjectRowIdentity`, `MissingObjectMeasurementValuePolicy`, `ObjectLabelRepresentation`; `WriteMode` (its policy family restates it one-to-one; converting it edits `spreadsheet_export.py`). +- K5: `FunctionStepExecutionScope`, `ImagePayloadConsumption`, `MainFlowInputProjection`, `RuntimeInvocationDomain`. K2: `ArtifactPayloadShape`. +- Plane-mode family (flagged unowned by G3; needs `artifacts.py` and #1199): `RuntimePlaneAxis`, `ImagePayloadMetadataCompositionMode`, `ObjectLabelDomainScope`. +- L7/G6: `ViewerLayerField`, `ViewerControlField`, `ViewerPayloadField`, `ViewerIntensityWindowField`, `ViewerDescriptorField` (reply-dict keys; the server should emit the agent DTO records and the window service decode them). +- U2: `CapabilityKind`, `CapabilityRole`, `CapabilityVisibility`, `PlateFileKind` (MCP tool-input choices; converting needs the MCP binder to project family fields, which python-introspect 0.2.8 enables). +- No owner yet (proposed E2): `FileFormat` restates PolyStore's format family (`get_format_from_extension`) and keys the image browser's result actions. + +Corrections to the plan above: +- **Behaviour changes.** A plate reached through a symlink alias is a different dataset root (lexical identity, as K4 decided); the registration test now checks a lexically equal spelling. `SourceFilterClause` and the role fields no longer coerce strings in `__post_init__`; wire input decodes through `registered_class_from_key`. Catalog control replies carry their value under `result`. `DefineGrid` reads the first grid-definition row, not the last (differs only when a module repeats it). +- **Durable spellings that change** (E1-Q1, no tool): saved `.py` files naming the enums above (`SourceFilterSubject.FILE`, `InputSource.PIPELINE_START`, `MaterializationBackend.DISK`, `NapariVariableSizeHandling.PAD_TO_MAX`, …) fail on load and are re-saved from the GUI. +- **Ratchet fixes beyond the brief.** Flags that restated a member's identity (`writes_zarr`, `streams_from_source_context`, `keeps_every_field`, `completed`, `editable`, `collects_records`, `returns_every_record`, `agent_supplied`) are deleted for nominal identity; the outcome export scope checks its own export path (the rule was written twice); default step directories moved onto the step-input dependency. + +Found outside the brief: `processing/materialization/core.py` `json_payload_value` is a third value-to-JSON singledispatch (materialization; passes unknown values through and converts DataFrames, so not a drop-in for `to_jsonable`); `ZMQAuxiliaryParamField` is a wire-key table (L3); `viewer_protocol.py` still declares `ViewerPersistenceMode`, `FijiPayloadKind`, `ViewerProcessPlatform`, `OpenHCSViewerControlMessageType`, `ViewerLayerIsolationField`, `ViewerProtocolStatus` (no member dispatch the guard finds; G6/L7). From 257b757664024aa074b1449c4adcfdd68a39d644 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 13:56:40 -0400 Subject: [PATCH 34/36] E1: decode family fields at the config boundary; keep the MCP schema contract - ConfigService decodes a type[Family] patch value once, through the family registry (MCP JSON patches spell members by registry key). - MCP tool schemas: python-introspect's RegisteredClassChoices publishes a named enum definition, and the server's schema generator spells a family default by its registry key, so openhcs_stream_plate_files_to_viewer and openhcs_run_datasets match the published contract again; the run-datasets boundary enum keeps its published name. - python-introspect decodes registry keys without running the encoder. - benchmark CLI and the desktop update smoke script use the families. - Tests that built removed enum members (orchestrator READY, a string materialization backend) or could not populate family fields now use the member classes. Co-Authored-By: Claude Opus 5.5 --- benchmark/cellprofiler_benchmark_cli.py | 4 +- external/python-introspect | 2 +- openhcs/agent/dto/session.py | 8 +- openhcs/agent/services/config_service.py | 14 ++++ openhcs/mcp/server.py | 15 +++- openhcs/runtime/viewer_display.py | 84 +++++++++---------- scripts/smoke_staged_desktop_update.py | 7 +- .../agent/test_mcp_dev_client_renderers.py | 7 ++ .../test_plate_compile_connection_action.py | 5 +- ...st_compiler_auto_materialization_policy.py | 5 +- 10 files changed, 95 insertions(+), 56 deletions(-) diff --git a/benchmark/cellprofiler_benchmark_cli.py b/benchmark/cellprofiler_benchmark_cli.py index 30fc3e6eb3..10cc6df6bd 100644 --- a/benchmark/cellprofiler_benchmark_cli.py +++ b/benchmark/cellprofiler_benchmark_cli.py @@ -24,7 +24,7 @@ ) from benchmark.datasets.cppipe_case_catalog import official_cp3_case_category from benchmark.runtime_env import configure_headless_cpu_benchmark_runtime -from openhcs.core.source_matching import is_image_path +from openhcs.core.source_bindings import PathIsImage class BenchmarkCliCommand(ABC, metaclass=AutoRegisterMeta): @@ -1009,7 +1009,7 @@ def _official_cellprofiler3_source_root(dataset_wrapper_path: Path) -> Path: def _has_image_payloads(path: Path) -> bool: return any( - candidate.is_file() and is_image_path(str(candidate)) + candidate.is_file() and PathIsImage.holds_for(str(candidate)) for candidate in path.iterdir() ) diff --git a/external/python-introspect b/external/python-introspect index 0d075a9e16..47e33a2587 160000 --- a/external/python-introspect +++ b/external/python-introspect @@ -1 +1 @@ -Subproject commit 0d075a9e1681c5385c119bb2f558d9121a7a2671 +Subproject commit 47e33a2587bff0e193e4f4dd3cee6ae2034464df diff --git a/openhcs/agent/dto/session.py b/openhcs/agent/dto/session.py index 6a42bbb456..45e37b84cc 100644 --- a/openhcs/agent/dto/session.py +++ b/openhcs/agent/dto/session.py @@ -18,12 +18,16 @@ from openhcs.runtime.zmq_execution_signature import ZMQRuntimeObservationExportScope RuntimeObservationExportScopeName = Enum( - "RuntimeObservationExportScopeName", + "ZMQRuntimeObservationExportScope", [(name.upper(), name) for name in ZMQRuntimeObservationExportScope.__registry__], type=str, module=__name__, ) -"""Boundary view of the export-scope registry, for agent request fields.""" +"""Boundary view of the export-scope registry, for agent request fields. + +It keeps the MCP schema's published name and description for the field. +""" +RuntimeObservationExportScopeName.__doc__ = ZMQRuntimeObservationExportScope.__doc__.splitlines()[0] RuntimeObservationExportScopeName.scope = property( lambda member: ZMQRuntimeObservationExportScope.named(member.value) ) diff --git a/openhcs/agent/services/config_service.py b/openhcs/agent/services/config_service.py index 56da50d128..714e58524e 100644 --- a/openhcs/agent/services/config_service.py +++ b/openhcs/agent/services/config_service.py @@ -35,6 +35,7 @@ coerce_enum_member, declared_enum_type, enum_input_values, + registered_class_from_key, resolve_optional, ) @@ -528,6 +529,10 @@ def _coerce_patch_value(field_type, value: JsonValue) -> object: ): return AxisFamily.active().grouping_named(value) + family = _registered_family(resolve_optional(unwrapped_type)) + if family is not None and isinstance(value, str): + return registered_class_from_key(family, value) + path_type = _unwrap_path_type(unwrapped_type) if path_type is not None and isinstance(value, str): return path_type(value) @@ -588,6 +593,15 @@ def _coerce_collection_patch_value(field_type, value: JsonValue) -> tuple[bool, return False, value +def _registered_family(field_type) -> AutoRegisterMeta | None: + """The family a ``type[Family]`` field holds a member of, if it is registered.""" + + if get_origin(field_type) is not type: + return None + (bound,) = get_args(field_type) + return bound if isinstance(bound, AutoRegisterMeta) else None + + def _dataclass_type(field_type) -> type | None: field_type = _unwrap_annotated(field_type) if isinstance(field_type, type) and is_dataclass(field_type): diff --git a/openhcs/mcp/server.py b/openhcs/mcp/server.py index b133801f50..1ca97a4b91 100644 --- a/openhcs/mcp/server.py +++ b/openhcs/mcp/server.py @@ -34,6 +34,8 @@ from pydantic import Field as PydanticField from pydantic import WithJsonSchema +from pydantic.json_schema import GenerateJsonSchema +from metaclass_registry import AutoRegisterMeta from pyqt_reactive.services.window_snapshot import WindowSnapshotCaptureScope from python_introspect import JsonValue, dataclass_from_mapping, to_jsonable from zmqruntime.config import TransportMode @@ -92,6 +94,15 @@ MCP_SERVER_INSTRUCTIONS = CapabilityTransport.LOCAL_STDIO.server_instructions() +class RegisteredFamilyJsonSchema(GenerateJsonSchema): + """Tool schemas spell a family-valued default by its registry key.""" + + def encode_default(self, dft: object) -> object: + if isinstance(dft, AutoRegisterMeta): + return to_jsonable(dft) + return super().encode_default(dft) + + def _request_parameter_annotation( request_type: type, parameter_name: str, @@ -831,7 +842,9 @@ async def guarded_tool(*args, **kwargs): argument_model = registered_tool.fn_metadata.arg_model argument_model.model_config["extra"] = "forbid" argument_model.model_rebuild(force=True) - registered_tool.parameters = argument_model.model_json_schema(by_alias=True) + registered_tool.parameters = argument_model.model_json_schema( + by_alias=True, schema_generator=RegisteredFamilyJsonSchema + ) return guarded_tool return decorator diff --git a/openhcs/runtime/viewer_display.py b/openhcs/runtime/viewer_display.py index 856e41b63b..1c1caf30b1 100644 --- a/openhcs/runtime/viewer_display.py +++ b/openhcs/runtime/viewer_display.py @@ -372,9 +372,9 @@ def settings_wire_mapping(self) -> ViewerWireMapping: class NapariVariableSizeHandling(ABC, metaclass=AutoRegisterMeta): - """How Napari shows images of different sizes in one stream; each way is a subclass. + """How to handle images with different sizes in the same layer. - ``name`` is its spelling in forms and on the wire. + Each way is a subclass; ``name`` is its spelling in forms and on the wire. """ __registry_key__ = "name" @@ -395,46 +395,6 @@ def materialize_layer_items(cls, layer_items: list, *, route_key: str) -> list: """Return image items that can share one native Napari layer.""" -class PadToMaxSize(NapariVariableSizeHandling): - """Keep one component route and pad its images to a common shape.""" - - name = "pad_to_max" - - @classmethod - def materialize_layer_items(cls, layer_items: list, *, route_key: str) -> list: - import numpy as np - - shapes = {item.data.shape for item in layer_items} - if len(shapes) <= 1: - return layer_items - logger.info( - "🔬 NAPARI PROCESS: Images in layer %s have different shapes - " - "padding to max size", - route_key, - ) - max_shape = tuple( - max(shape[axis] for shape in shapes) - for axis in range(len(layer_items[0].data.shape)) - ) - for item_index, item in enumerate(layer_items): - if item.data.shape == max_shape: - continue - pad_width = tuple( - (0, maximum - current) - for current, maximum in zip(item.data.shape, max_shape, strict=True) - ) - layer_items[item_index] = replace( - item, - data=np.pad(item.data, pad_width, mode="constant", constant_values=0), - ) - logger.debug( - "🔬 NAPARI PROCESS: Padded image from %s to %s", - item.data.shape, - max_shape, - ) - return layer_items - - class SeparateLayers(NapariVariableSizeHandling): """Give each partition-axis value its own route and keep native shapes.""" @@ -484,6 +444,46 @@ def materialize_layer_items(cls, layer_items: list, *, route_key: str) -> list: return layer_items +class PadToMaxSize(NapariVariableSizeHandling): + """Keep one component route and pad its images to a common shape.""" + + name = "pad_to_max" + + @classmethod + def materialize_layer_items(cls, layer_items: list, *, route_key: str) -> list: + import numpy as np + + shapes = {item.data.shape for item in layer_items} + if len(shapes) <= 1: + return layer_items + logger.info( + "🔬 NAPARI PROCESS: Images in layer %s have different shapes - " + "padding to max size", + route_key, + ) + max_shape = tuple( + max(shape[axis] for shape in shapes) + for axis in range(len(layer_items[0].data.shape)) + ) + for item_index, item in enumerate(layer_items): + if item.data.shape == max_shape: + continue + pad_width = tuple( + (0, maximum - current) + for current, maximum in zip(item.data.shape, max_shape, strict=True) + ) + layer_items[item_index] = replace( + item, + data=np.pad(item.data, pad_width, mode="constant", constant_values=0), + ) + logger.debug( + "🔬 NAPARI PROCESS: Padded image from %s to %s", + item.data.shape, + max_shape, + ) + return layer_items + + @dataclass(frozen=True) class NapariDisplaySettings(ViewerDisplaySettings): slot_family = NapariSlots diff --git a/scripts/smoke_staged_desktop_update.py b/scripts/smoke_staged_desktop_update.py index 984a6c75d6..f141e0fb63 100644 --- a/scripts/smoke_staged_desktop_update.py +++ b/scripts/smoke_staged_desktop_update.py @@ -13,7 +13,8 @@ from openhcs.desktop.update_worker import ( DesktopUpdatePlan, - DesktopUpdateProgressAction, + DesktopUpdateFailureResponse, + ExitAfterFailedUpdate, DesktopUpdateProgressReporterABC, ResolvedProcessLaunchSpec, _run_update, @@ -27,9 +28,9 @@ def phase(self, phase) -> None: def output(self, message: str) -> None: print(message, file=sys.stderr, flush=True) - def failure(self, message: str) -> DesktopUpdateProgressAction: + def failure(self, message: str) -> type[DesktopUpdateFailureResponse]: print(f"FAILURE: {message}", file=sys.stderr, flush=True) - return DesktopUpdateProgressAction.EXIT + return ExitAfterFailedUpdate def complete(self) -> None: print("COMPLETE", file=sys.stderr, flush=True) diff --git a/tests/unit/agent/test_mcp_dev_client_renderers.py b/tests/unit/agent/test_mcp_dev_client_renderers.py index d45e9a5930..0ec6bdbc6b 100644 --- a/tests/unit/agent/test_mcp_dev_client_renderers.py +++ b/tests/unit/agent/test_mcp_dev_client_renderers.py @@ -14,6 +14,8 @@ import enum import types import typing + +from metaclass_registry import AutoRegisterMeta from dataclasses import MISSING, fields, is_dataclass import pytest @@ -42,6 +44,11 @@ def sample_value(hint, depth: int): if origin in (typing.Union, types.UnionType): members = [arg for arg in typing.get_args(hint) if arg is not type(None)] return sample_value(members[0], depth) + if origin is type: + (family,) = typing.get_args(hint) + if isinstance(family, AutoRegisterMeta): + # A family-valued field holds one of the family's member classes. + return next(iter(family.__registry__.values())) if origin is tuple: args = typing.get_args(hint) if depth >= MAX_DEPTH or not args: diff --git a/tests/unit/pyqt_gui/test_plate_compile_connection_action.py b/tests/unit/pyqt_gui/test_plate_compile_connection_action.py index 5a959f2e60..b7dd36a0c4 100644 --- a/tests/unit/pyqt_gui/test_plate_compile_connection_action.py +++ b/tests/unit/pyqt_gui/test_plate_compile_connection_action.py @@ -13,6 +13,7 @@ WaitForServer, ) from openhcs.authoring.session.views import DatasetListView +from openhcs.constants.constants import OrchestratorReady from openhcs.core.execution_state import ManagerExecutionState from openhcs.core.steps.function_step import FunctionStep from openhcs.pyqt_gui.config import get_default_ui_config @@ -59,9 +60,7 @@ def test_endpoint_phase_resolves_the_compile_slot(monkeypatch, phase, expected): def test_connect_needs_no_dataset_and_startup_is_never_available(tmp_path): with caller_session() as session: (scope_id,) = add_datasets(session, tmp_path, "plate") - session.orchestrator(scope_id)._state = ( - session.orchestrator(scope_id)._state.READY - ) + session.orchestrator(scope_id)._state = OrchestratorReady session.set_pipeline(scope_id, [FunctionStep(name="Step")]) assert ConnectServer.available(session, ConnectServer.request()) is None for selection in ((), (scope_id,)): diff --git a/tests/unit/test_compiler_auto_materialization_policy.py b/tests/unit/test_compiler_auto_materialization_policy.py index 0dfd5508ff..dac756a8a0 100644 --- a/tests/unit/test_compiler_auto_materialization_policy.py +++ b/tests/unit/test_compiler_auto_materialization_policy.py @@ -5,6 +5,7 @@ import pytest +from openhcs.core.config import MaterializeToDisk from openhcs.core.compiled_step_plan import RuntimeArtifactMaterializationPlan from openhcs.core.pipeline.compiler import PipelineCompiler from openhcs.core.pipeline.materialization_flag_planner import ( @@ -43,7 +44,7 @@ def backend(self, declaration): 1: SimpleNamespace(artifact_outputs={"export": declared_image}), } context = object() - vfs_config = SimpleNamespace(materialization_backend="disk") + vfs_config = SimpleNamespace(materialization_backend=MaterializeToDisk) session = SimpleNamespace( global_config=SimpleNamespace( materialize_runtime_artifacts=False, @@ -64,7 +65,7 @@ def backend(self, declaration): ) PipelineCompiler._compile_runtime_artifact_materialization_plans(session, planner) - assert backend_requests == [vfs_config.materialization_backend] + assert backend_requests == [vfs_config.materialization_backend.backend] assert not plans[0].runtime_artifact_materialization.has_persistent_target assert plans[1].runtime_artifact_materialization.has_persistent_target assert plans[1].runtime_artifact_materialization.require_persistent_backend() == ( From ba5f8518025848d0a965c8c93bc112956e61706e Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 14:34:38 -0400 Subject: [PATCH 35/36] E1: guard table regenerated after K2/K3/K5; receipt notes the remaining owners Co-Authored-By: Claude Opus 5.5 --- .../E1-leftover-families.md | 3 +- tests/unit/test_e1_enum_dispatch_guards.py | 236 +++++++++--------- 2 files changed, 116 insertions(+), 123 deletions(-) diff --git a/docs/refactor/consolidation-20261009/E1-leftover-families.md b/docs/refactor/consolidation-20261009/E1-leftover-families.md index 80bca0a901..5958801868 100644 --- a/docs/refactor/consolidation-20261009/E1-leftover-families.md +++ b/docs/refactor/consolidation-20261009/E1-leftover-families.md @@ -77,7 +77,8 @@ The census ran as five batches (this worktree plus four in parallel, merged here **Reassigned** (guard table, by owner): - K3 (#1199's measurement files): `MeasurementScope`, `MeasurementRowAxisField`, `MeasurementRowValueField`, `MeasurementStatistic`, `ObjectFeatureArrayDomain`, `RuntimeCellValueKind`, `RuntimeMeasurementFeatureNameMode`, `RuntimeMeasurementQualifierValueMode`, `RuntimeMeasurementSourceNameEncoding`, `MeasurementTableRowLayout`, `MeasurementObjectRowIdentity`, `MissingObjectMeasurementValuePolicy`, `ObjectLabelRepresentation`; `WriteMode` (its policy family restates it one-to-one; converting it edits `spreadsheet_export.py`). -- K5: `FunctionStepExecutionScope`, `ImagePayloadConsumption`, `MainFlowInputProjection`, `RuntimeInvocationDomain`. K2: `ArtifactPayloadShape`. +- K5: `FunctionStepExecutionScope`, `ImagePayloadConsumption`, `MainFlowInputProjection`, `RuntimeInvocationDomain`. K2 deleted `ArtifactPayloadShape`. +- After merging K2, K3 and K5 into E1, the K3, K5 and plane-mode entries above are still enums and no longer blocked by held files; they need an owner (proposed E2). - Plane-mode family (flagged unowned by G3; needs `artifacts.py` and #1199): `RuntimePlaneAxis`, `ImagePayloadMetadataCompositionMode`, `ObjectLabelDomainScope`. - L7/G6: `ViewerLayerField`, `ViewerControlField`, `ViewerPayloadField`, `ViewerIntensityWindowField`, `ViewerDescriptorField` (reply-dict keys; the server should emit the agent DTO records and the window service decode them). - U2: `CapabilityKind`, `CapabilityRole`, `CapabilityVisibility`, `PlateFileKind` (MCP tool-input choices; converting needs the MCP binder to project family fields, which python-introspect 0.2.8 enables). diff --git a/tests/unit/test_e1_enum_dispatch_guards.py b/tests/unit/test_e1_enum_dispatch_guards.py index 9fa4fed56d..6dd14222cb 100644 --- a/tests/unit/test_e1_enum_dispatch_guards.py +++ b/tests/unit/test_e1_enum_dispatch_guards.py @@ -29,8 +29,6 @@ CELLPROFILER_SPELLING = "CellProfiler setting or column literal table" ALLOWED: dict[tuple[str, str], str] = { - ('ArtifactPayloadShape', 'openhcs/core/artifacts.py'): 'K2: core/artifacts.py', - ('ArtifactPayloadShape', 'openhcs/core/runtime_exports.py'): 'K2: core/artifacts.py', ('CPAPropertyName', 'openhcs/interop/cellprofiler/analyst_export.py'): CELLPROFILER_SPELLING, ('CapabilityKind', 'openhcs/agent/capabilities.py'): 'U2: MCP tool-input filter choices; needs the MCP binder to project family fields (python-introspect 0.2.8)', ('CapabilityRole', 'openhcs/agent/capabilities.py'): 'U2: MCP tool-input filter choices; needs the MCP binder to project family fields (python-introspect 0.2.8)', @@ -41,97 +39,95 @@ ('CellProfilerSourceImageProjectionField', 'openhcs/interop/cellprofiler/analyst_export.py'): CELLPROFILER_SPELLING, ('FileFormat', 'openhcs/core/plate_image_inventory.py'): "E2: restates polystore's format family (get_format_from_extension); key result actions by that family", ('FileFormat', 'openhcs/pyqt_gui/widgets/image_browser.py'): "E2: restates polystore's format family (get_format_from_extension); key result actions by that family", - ('FunctionStepExecutionScope', 'openhcs/core/artifacts.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/core/callable_contract.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/core/function_patterns.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/core/function_reference.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/core/orchestrator/bundle_execution.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/core/orchestrator/worker_execution.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/core/pipeline/artifact_planning.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/core/pipeline/compilation_session.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/core/pipeline/compiler.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/core/pipeline/funcstep_contract_validator.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/core/pipeline/path_planner.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/interop/cellprofiler/compile_time_contracts.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/interop/cellprofiler/module_callable_abi.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/interop/cellprofiler/module_declarations.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/processing/backends/lib_registry/openhcs_registry.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/processing/backends/lib_registry/unified_registry.py'): 'K5: callable contract and function patterns', - ('FunctionStepExecutionScope', 'openhcs/processing/custom_functions/validation.py'): 'K5: callable contract and function patterns', - ('ImagePayloadConsumption', 'openhcs/core/callable_contract.py'): 'K5: callable contract and function patterns', - ('ImagePayloadConsumption', 'openhcs/core/function_patterns.py'): 'K5: callable contract and function patterns', - ('ImagePayloadConsumption', 'openhcs/core/steps/function_execution.py'): 'K5: callable contract and function patterns', - ('ImagePayloadConsumption', 'openhcs/interop/cellprofiler/module_artifact_declarations.py'): 'K5: callable contract and function patterns', - ('ImagePayloadConsumption', 'openhcs/interop/cellprofiler/runtime/module_execution.py'): 'K5: callable contract and function patterns', - ('ImagePayloadMetadataCompositionMode', 'openhcs/core/aligned_image_payload.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ImagePayloadMetadataCompositionMode', 'openhcs/core/runtime_image_values.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ImagePayloadMetadataCompositionMode', 'openhcs/interop/cellprofiler/runtime/invocation.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('MainFlowInputProjection', 'openhcs/core/steps/function_runtime.py'): 'K5: callable contract and function patterns', - ('MainFlowInputProjection', 'openhcs/interop/cellprofiler/runtime/artifact_binding.py'): 'K5: callable contract and function patterns', + ('FunctionStepExecutionScope', 'openhcs/core/artifacts/input_projection.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/core/callable_contract.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/core/function_patterns.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/core/function_reference.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/core/orchestrator/bundle_execution.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/core/orchestrator/worker_execution.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/core/pipeline/artifact_planning.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/core/pipeline/compilation_session.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/core/pipeline/compiler.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/core/pipeline/funcstep_contract_validator.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/core/pipeline/path_planner.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/interop/cellprofiler/compile_time_contracts.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/interop/cellprofiler/module_callable_abi.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/interop/cellprofiler/module_declarations.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/processing/backends/lib_registry/openhcs_registry.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/processing/backends/lib_registry/unified_registry.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('FunctionStepExecutionScope', 'openhcs/processing/custom_functions/validation.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('ImagePayloadConsumption', 'openhcs/core/callable_contract.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('ImagePayloadConsumption', 'openhcs/core/function_patterns.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('ImagePayloadConsumption', 'openhcs/core/steps/function_execution.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('ImagePayloadConsumption', 'openhcs/interop/cellprofiler/module_artifact_declarations.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('ImagePayloadConsumption', 'openhcs/interop/cellprofiler/runtime/module_execution.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('ImagePayloadMetadataCompositionMode', 'openhcs/core/aligned_image_payload.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ImagePayloadMetadataCompositionMode', 'openhcs/core/runtime_image_values.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ImagePayloadMetadataCompositionMode', 'openhcs/interop/cellprofiler/runtime/invocation.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('MainFlowInputProjection', 'openhcs/core/steps/function_runtime.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('MainFlowInputProjection', 'openhcs/interop/cellprofiler/runtime/artifact_binding.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', ('McpWireContentType', 'openhcs/mcp/dev_client_core.py'): 'MCP content-type names', ('McpWireMethod', 'openhcs/mcp/dev_client_core.py'): 'MCP JSON-RPC method names', - ('MeasurementObjectRowIdentity', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_policies.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/core/measurement_feature_queries.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/core/measurement_row_materialization.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/core/runtime_artifact_queries.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/core/runtime_spatial_grid.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/module_artifact_declarations.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/measurement_recording.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/measurement_rows.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/object_measurement_vectors.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/relationship_measurement_rows.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/alignment.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/classification.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/colocalization.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/flagging.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/intensity.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/intensity_distribution.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/measurement_math.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/neighbors.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/relationships.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/spreadsheet_export.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/thresholding.py'): 'K3: measurement code held in #1199', - ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/tracking.py'): 'K3: measurement code held in #1199', - ('MeasurementRowValueField', 'openhcs/interop/cellprofiler/runtime/measurement_rows.py'): 'K3: measurement code held in #1199', - ('MeasurementRowValueField', 'openhcs/processing/backends/cellprofiler/measurement_math.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/core/equivalence/keys.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/core/equivalence/measurement_features.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/core/equivalence/measurement_rows.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/core/measurement_row_materialization.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/core/runtime_artifact_queries.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/core/runtime_measurements.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/interop/cellprofiler/analyst_export.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/interop/cellprofiler/database_column_dialect.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/interop/cellprofiler/image_set_numbering.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/interop/cellprofiler/measurement_dialect.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/interop/cellprofiler/measurement_scope.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/interop/cellprofiler/module_measurement_features.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/processing/backends/cellprofiler/calculate_statistics.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/processing/backends/cellprofiler/image_quality.py'): 'K3: measurement code held in #1199', - ('MeasurementScope', 'openhcs/processing/backends/cellprofiler/spreadsheet_export.py'): 'K3: measurement code held in #1199', - ('MeasurementStatistic', 'openhcs/core/equivalence/keys.py'): 'K3: measurement code held in #1199', - ('MeasurementStatistic', 'openhcs/core/equivalence/measurement_features.py'): 'K3: measurement code held in #1199', - ('MeasurementStatistic', 'openhcs/core/runtime_measurements.py'): 'K3: measurement code held in #1199', - ('MeasurementStatistic', 'openhcs/interop/cellprofiler/module_measurement_features.py'): 'K3: measurement code held in #1199', - ('MeasurementTableRowLayout', 'openhcs/interop/cellprofiler/analyst_export.py'): 'K3: measurement code held in #1199', - ('MissingObjectMeasurementValuePolicy', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_completion.py'): 'K3: measurement code held in #1199', + ('MeasurementObjectRowIdentity', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_policies.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/core/measurement_feature_queries.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/core/measurement_row_materialization.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/core/runtime_artifact_queries.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/core/runtime_spatial_grid.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/module_artifact_declarations.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/measurement_recording.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/measurement_rows.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/object_measurement_vectors.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/interop/cellprofiler/runtime/relationship_measurement_rows.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/alignment.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/classification.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/colocalization.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/flagging.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/intensity.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/intensity_distribution.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/measurement_math.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/neighbors.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/relationships.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/spreadsheet_export.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/thresholding.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowAxisField', 'openhcs/processing/backends/cellprofiler/tracking.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowValueField', 'openhcs/interop/cellprofiler/runtime/measurement_rows.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementRowValueField', 'openhcs/processing/backends/cellprofiler/measurement_math.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/core/equivalence/keys.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/core/equivalence/measurement_features.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/core/measurement_row_materialization.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/core/runtime_artifact_queries.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/core/runtime_measurements.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/interop/cellprofiler/analyst_export.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/interop/cellprofiler/database_column_dialect.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/interop/cellprofiler/image_set_numbering.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/interop/cellprofiler/measurement_dialect.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/interop/cellprofiler/measurement_scope.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/interop/cellprofiler/module_measurement_features.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/processing/backends/cellprofiler/calculate_statistics.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/processing/backends/cellprofiler/image_quality.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementScope', 'openhcs/processing/backends/cellprofiler/spreadsheet_export.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementStatistic', 'openhcs/core/equivalence/keys.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementStatistic', 'openhcs/core/equivalence/measurement_features.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementStatistic', 'openhcs/core/runtime_measurements.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementStatistic', 'openhcs/interop/cellprofiler/module_measurement_features.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MeasurementTableRowLayout', 'openhcs/interop/cellprofiler/analyst_export.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('MissingObjectMeasurementValuePolicy', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_completion.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', ('MultiprocessingStartMethod', 'openhcs/core/orchestrator/worker_execution.py'): 'Python multiprocessing start-method names; a class-valued top-level GlobalPipelineConfig field does not reach lazy PipelineConfig (ObjectState, L6)', ('MultiprocessingStartMethod', 'openhcs/core/worker_start_policy.py'): 'Python multiprocessing start-method names; a class-valued top-level GlobalPipelineConfig field does not reach lazy PipelineConfig (ObjectState, L6)', - ('ObjectFeatureArrayDomain', 'openhcs/processing/backends/cellprofiler/shape.py'): 'K3: measurement code held in #1199', - ('ObjectLabelDomainScope', 'openhcs/core/measurement_image_alignment.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ObjectLabelDomainScope', 'openhcs/core/pipeline/function_contracts.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ObjectLabelDomainScope', 'openhcs/core/runtime_object_label_building.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ObjectLabelDomainScope', 'openhcs/core/runtime_object_label_domains.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ObjectLabelDomainScope', 'openhcs/core/runtime_object_labels.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ObjectLabelDomainScope', 'openhcs/core/runtime_relationships.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ObjectLabelDomainScope', 'openhcs/interop/cellprofiler/runtime/measurement_rows.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ObjectLabelDomainScope', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_completion.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ObjectLabelDomainScope', 'openhcs/interop/cellprofiler/runtime/relationship_measurement_rows.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ObjectLabelDomainScope', 'openhcs/processing/backends/cellprofiler/outlines.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ObjectLabelDomainScope', 'openhcs/processing/backends/cellprofiler/relationships.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('ObjectLabelRepresentation', 'openhcs/core/runtime_object_label_aggregation.py'): 'K3: measurement code held in #1199', - ('ObjectLabelRepresentation', 'openhcs/core/runtime_object_labels.py'): 'K3: measurement code held in #1199', - ('ObjectLabelRepresentation', 'openhcs/processing/backends/cellprofiler/shape.py'): 'K3: measurement code held in #1199', + ('ObjectLabelDomainScope', 'openhcs/core/measurement_image_alignment.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ObjectLabelDomainScope', 'openhcs/core/pipeline/function_contracts.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ObjectLabelDomainScope', 'openhcs/core/runtime_object_label_building.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ObjectLabelDomainScope', 'openhcs/core/runtime_object_label_domains.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ObjectLabelDomainScope', 'openhcs/core/runtime_object_labels.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ObjectLabelDomainScope', 'openhcs/core/runtime_relationships.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ObjectLabelDomainScope', 'openhcs/interop/cellprofiler/runtime/measurement_rows.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ObjectLabelDomainScope', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_completion.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ObjectLabelDomainScope', 'openhcs/interop/cellprofiler/runtime/relationship_measurement_rows.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ObjectLabelDomainScope', 'openhcs/processing/backends/cellprofiler/outlines.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ObjectLabelDomainScope', 'openhcs/processing/backends/cellprofiler/relationships.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('ObjectLabelRepresentation', 'openhcs/core/runtime_object_label_aggregation.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('ObjectLabelRepresentation', 'openhcs/core/runtime_object_labels.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('ObjectLabelRepresentation', 'openhcs/processing/backends/cellprofiler/shape.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', ('PlateFileKind', 'openhcs/agent/services/plate_inspection_service.py'): "U2: MCP tool-input choice with an 'all' literal; needs the MCP binder to project family fields (python-introspect 0.2.8)", ('PlateFileKind', 'openhcs/agent/services/plate_streaming_service.py'): "U2: MCP tool-input choice with an 'all' literal; needs the MCP binder to project family fields (python-introspect 0.2.8)", ('PlateFileKind', 'openhcs/agent/services/selected_plate_service.py'): "U2: MCP tool-input choice with an 'all' literal; needs the MCP binder to project family fields (python-introspect 0.2.8)", @@ -140,38 +136,34 @@ ('PlateFileKind', 'openhcs/pyqt_gui/widgets/image_browser.py'): "U2: MCP tool-input choice with an 'all' literal; needs the MCP binder to project family fields (python-introspect 0.2.8)", ('PlateInspectionIssueCode', 'openhcs/mcp/dev_client_renderers/plate.py'): 'AgentError.code spellings read by the client', ('QtPlatformName', 'openhcs/runtime/viewer_protocol.py'): 'Qt QPA plugin names (QT_QPA_PLATFORM)', - ('RuntimeCellValueKind', 'openhcs/core/equivalence/cells.py'): 'K3: measurement code held in #1199', - ('RuntimeCellValueKind', 'openhcs/core/equivalence/measurement_features.py'): 'K3: measurement code held in #1199', - ('RuntimeInvocationDomain', 'openhcs/core/function_patterns.py'): 'K5: callable contract and function patterns', - ('RuntimeInvocationDomain', 'openhcs/core/steps/function_execution.py'): 'K5: callable contract and function patterns', - ('RuntimeInvocationDomain', 'openhcs/core/steps/function_runtime.py'): 'K5: callable contract and function patterns', - ('RuntimeMeasurementFeatureNameMode', 'openhcs/core/equivalence/keys.py'): 'K3: measurement code held in #1199', - ('RuntimeMeasurementQualifierValueMode', 'openhcs/core/equivalence/measurement_rows.py'): 'K3: measurement code held in #1199', - ('RuntimeMeasurementQualifierValueMode', 'openhcs/core/measurement_dialect.py'): 'K3: measurement code held in #1199', - ('RuntimeMeasurementSourceNameEncoding', 'openhcs/core/equivalence/keys.py'): 'K3: measurement code held in #1199', - ('RuntimeMeasurementSourceNameEncoding', 'openhcs/core/measurement_dialect.py'): 'K3: measurement code held in #1199', - ('RuntimePlaneAxis', 'openhcs/core/aligned_image_payload.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/artifacts.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/debug.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/image_payload_execution_mode.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/measurement_image_alignment.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/runtime_image_values.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/runtime_measurements.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/runtime_object_labels.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/runtime_plane_projection.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/runtime_slice_alignment.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/runtime_sparse_labels.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/runtime_tabular_values.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/core/steps/function_runtime.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/interop/cellprofiler/runtime/adapter.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/interop/cellprofiler/runtime/artifact_binding.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_completion.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/area_occupied.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/colocalization.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/color.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/image_math.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/intensity.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', - ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/intensity_distribution.py'): 'plane-mode family (G3 unowned); needs artifacts.py (K2) and #1199 (K3)', + ('RuntimeCellValueKind', 'openhcs/core/equivalence/cells.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('RuntimeCellValueKind', 'openhcs/core/equivalence/measurement_features.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('RuntimeInvocationDomain', 'openhcs/core/function_patterns.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('RuntimeInvocationDomain', 'openhcs/core/steps/function_execution.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('RuntimeInvocationDomain', 'openhcs/core/steps/function_runtime.py'): 'K5 callable contract (#1202 merged, still an enum); needs an owner', + ('RuntimeMeasurementFeatureNameMode', 'openhcs/core/equivalence/keys.py'): 'K3 measurement vocabulary (#1199 merged, still an enum); needs an owner', + ('RuntimePlaneAxis', 'openhcs/core/aligned_image_payload.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/artifacts/kinds.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/debug.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/image_payload_execution_mode.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/measurement_image_alignment.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/runtime_image_values.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/runtime_measurements.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/runtime_object_labels.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/runtime_plane_projection.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/runtime_slice_alignment.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/runtime_sparse_labels.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/runtime_tabular_values.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/core/steps/function_runtime.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/interop/cellprofiler/runtime/adapter.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/interop/cellprofiler/runtime/artifact_binding.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/interop/cellprofiler/runtime/object_measurement_row_completion.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/area_occupied.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/colocalization.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/color.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/image_math.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/intensity.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', + ('RuntimePlaneAxis', 'openhcs/processing/backends/cellprofiler/intensity_distribution.py'): 'plane-mode family (G3: unowned); now unblocked by K2 and K3', ('ViewerControlField', 'openhcs/agent/services/viewer_window_service.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', ('ViewerControlField', 'openhcs/runtime/napari_viewer_server.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', ('ViewerControlField', 'openhcs/runtime/viewer_control_actions.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', @@ -179,7 +171,7 @@ ('ViewerIntensityWindowField', 'openhcs/runtime/napari_viewer_server.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', ('ViewerLayerField', 'openhcs/runtime/napari_viewer_server.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', ('ViewerPayloadField', 'openhcs/runtime/napari_viewer_server.py'): 'L7/G6: viewer reply keys; typed reply records replace the dicts', - ('WriteMode', 'openhcs/serialization/pycodify_formatters.py'): 'K3: converting it edits spreadsheet_export.py, held in #1199', + ('WriteMode', 'openhcs/serialization/pycodify_formatters.py'): 'K3 measurement writers restate it; unblocked now that #1199 merged', } From a052d83a163983533f93d758f35fa0663e9366ab Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 11 Oct 2026 15:34:40 -0400 Subject: [PATCH 36/36] E1: pin released python-introspect 0.2.8 Co-Authored-By: Claude Opus 5.5 --- external/python-introspect | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/external/python-introspect b/external/python-introspect index 47e33a2587..a88fe56f78 160000 --- a/external/python-introspect +++ b/external/python-introspect @@ -1 +1 @@ -Subproject commit 47e33a2587bff0e193e4f4dd3cee6ae2034464df +Subproject commit a88fe56f78b523ad5c791e3f868bb35d7d6b3f12