Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -914,6 +914,30 @@ _select:

Task IDs default from the left-hand variable name at the call site, converted to title case. If there is no simple left-hand variable, or if you want a stable explicit label, call `.named("Task Id")` before invoking the task. Use `.bind(...)` to pre-fill task arguments and `.with_annotations({...})` to add per-task annotations.

##### Default task annotations

`@pipeline(task_annotations=...)` sets default annotations for every task in that pipeline's body. Pass a mapping, or a function of the pipeline's resolved `cfg` when values differ per environment:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Run annotations do reach every task (the orchestrator merges defaults < run < task), and we use them for non-identity keys. For identity we want it to stay with the pipeline. Our MLCP annotations (cloud provider, impersonated account, project) are derived from the pipeline config's service_account, so the staging and prod compiles carry the right account whoever submits them. With run annotations every submit path has to pass the right values, and a wrong one silently runs as the wrong account.

If by "pipeline level" you mean @pipeline(annotations=...): as far as I can see it only lands in the root spec's metadata.annotations and isn't copied to the run or its tasks (create takes run annotations separately, and extra_submit_annotations ignores the spec). Am I missing a path?


```python
@pipeline(
"Daily",
config="config.yaml",
task_annotations=lambda cfg: {"example.com/account": cfg.service_account},
)
def daily(cfg=None) -> None:
extract = load() # the defaults
train = fit.with_annotations({"example.com/gpu": "a100"})() # the defaults plus its own key
subpipeline(report).with_task_annotations(
{"example.com/account": cfg.service_account}
)(wait_for=train)
```

- A function makes the config file a required compile-time input, as a `cfg` parameter does.
- A task's own annotations, from `.with_annotations(...)` or a typed helper such as `.with_position(...)`, win per key; the other defaults stay.
- Defaults do not reach a `subpipeline` boundary task or the tasks inside a child. `subpipeline(child).with_task_annotations({...})` pushes a mapping into every task of the child's subtree, except `subpipeline` boundary tasks. It takes a mapping of scalars, not a function: compute values from the parent's `cfg`.
- Precedence, lowest first: the child's own `task_annotations`, an outer call site, a nearer call site, the task's own annotations.
- Call-site defaults are part of the child's compile identity: calls pushing different defaults compile the child into separate sidecars, and identical calls share one.

##### Root pipeline annotations

`@pipeline(annotations={...})` writes the compiled pipeline's root `metadata.annotations` block. A caller that compiles programmatically can supply the block instead — typically from its own per-environment config file, so the values do not have to be hard-coded in source:
Expand Down
2 changes: 1 addition & 1 deletion packages/tangle-cli/src/tangle_cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,6 @@
try:
__version__ = metadata_version("tangle-cli")
except PackageNotFoundError:
__version__ = "0.1.35"
__version__ = "0.1.36"

__all__ = ["TangleDynamicDiscoveryClient", "__version__"]
45 changes: 40 additions & 5 deletions packages/tangle-cli/src/tangle_cli/pipeline_compiler.py
Original file line number Diff line number Diff line change
Expand Up @@ -706,7 +706,7 @@ def _resolve_cfg_for_pipeline(
missing config all still raise.
"""
cfg_path = _resolve_cfg_path_in_dir(pipeline_fn, base_dir)
uses_cfg = _pipeline_accepts_cfg(pipeline_fn)
uses_cfg = _pipeline_reads_cfg(pipeline_fn)
should_load_config = uses_cfg or pipeline_fn.propagate_config
if should_load_config:
_assert_config_output_path_is_separate(
Expand All @@ -722,7 +722,13 @@ def _resolve_cfg_for_pipeline(
overrides,
coerce=is_root,
pipeline_fn=pipeline_fn,
usage="cfg" if uses_cfg else "propagate_config",
usage=(
"cfg"
if _pipeline_accepts_cfg(pipeline_fn)
else "task_annotations"
if uses_cfg
else "propagate_config"
),
allow_override_only=allow_override_only,
suggest_override_only=implicit_root_config,
)
Expand Down Expand Up @@ -774,6 +780,7 @@ def _compile_pipeline_fn(
precomputed_key: PipelineCompileKey | None = None,
layout_policy: _LayoutPolicy = _NO_LAYOUT,
occurrence_path: tuple[str, ...] = (),
inherited_task_annotations: Mapping[str, Any] | None = None,
) -> SubgraphArtifact:
"""Trace + emit ``pipeline_fn`` into an in-memory :class:`SubgraphArtifact`.

Expand Down Expand Up @@ -831,7 +838,12 @@ def _compile_pipeline_fn(
# legitimate RUNTIME template placeholders the no-template-delimiter
# output guard must skip for THIS artifact's body.
with _temp_sys_path(base_dir):
builder = trace_pipeline(pipeline_fn, cfg=cfg, inputs={})
builder = trace_pipeline(
pipeline_fn,
cfg=cfg,
inputs={},
inherited_task_annotations=inherited_task_annotations,
)
# Caller-supplied root annotations: ROOT ONLY (a child keeps exactly what
# its own ``@pipeline`` declared), merged PER KEY with the caller winning
# on collision, before emit so they pass the same emit-time guards as
Expand Down Expand Up @@ -994,6 +1006,7 @@ def _compile_pipeline_fn(
parent_propagate_config=pipeline_fn.propagate_config,
layout_carry=layout_policy.carry,
occurrence_path=occurrence_path,
inherited_task_annotations=inherited_task_annotations or {},
)
finally:
ctx.active_stack.pop()
Expand Down Expand Up @@ -1022,6 +1035,7 @@ def _process_subpipeline_children(
parent_propagate_config: bool,
layout_carry: Layout | None = None,
occurrence_path: tuple[str, ...] = (),
inherited_task_annotations: Mapping[str, Any],
) -> None:
"""Compile each ``subpipeline(child)(...)`` recorded during ``builder``'s
trace and rewrite the parent task's ``componentRef`` to a pure
Expand Down Expand Up @@ -1085,7 +1099,7 @@ def _process_subpipeline_children(
# ``_select``/``_env``).
child_doc: CfgDocument | None = None
child_identity: str | None = None
if (_pipeline_accepts_cfg(child_fn) or child_fn.propagate_config) and child_cfg_path.exists():
if (_pipeline_reads_cfg(child_fn) or child_fn.propagate_config) and child_cfg_path.exists():
child_doc = read_cfg_document(child_cfg_path)
child_identity = _config_identity(child_doc)
if ctx.broadcast_stack or sub_ref.config_overrides:
Expand Down Expand Up @@ -1121,13 +1135,15 @@ def _process_subpipeline_children(
else:
effective = {}

child_task_annotations = {**inherited_task_annotations, **sub_ref.task_annotations}
child_key = _compile_key_for(
child_fn,
child_cfg_path,
effective,
fingerprint_context=f"on subpipeline task {task_id!r} -> child pipeline {child_fn.name!r}",
ambient_context=ambient_passthrough or None,
config_identity=child_identity,
task_annotations=child_task_annotations,
)

# Cycle detection (covers self-reference). active_stack holds the
Expand Down Expand Up @@ -1198,6 +1214,7 @@ def _process_subpipeline_children(
precomputed_key=child_key,
layout_policy=child_policy,
occurrence_path=(*occurrence_path, task_id),
inherited_task_annotations=child_task_annotations,
)
finally:
if pushed_edge_override:
Expand Down Expand Up @@ -1428,6 +1445,7 @@ def _compile_key_for(
fingerprint_context: str | None = None,
ambient_context: Mapping[str, Any] | None = None,
config_identity: str | None = None,
task_annotations: Mapping[str, Any] | None = None,
) -> PipelineCompileKey:
"""Build the canonical :class:`PipelineCompileKey` for ``pipeline_fn``.

Expand Down Expand Up @@ -1462,13 +1480,16 @@ def _compile_key_for(
see :func:`_config_identity`) -> the same envelope plus
``"\x00config"``, so different environment selections get distinct
keys while identical resolved configs keep identical keys.
* **``task_annotations``** (defaults pushed in by ancestor
``.with_task_annotations`` call sites) -> the same envelope plus
``"\x00task_annotations"``.
"""
src = _pipeline_source_path(pipeline_fn)
if src is not None:
source_canon = canonical_repo_path(src)
else:
source_canon = f"<dynamic:{pipeline_fn.fn.__qualname__}>"
if ambient_context or config_identity is not None:
if ambient_context or config_identity is not None or task_annotations:
# Reserved keys (NUL-prefixed) can never be real config keys, so an
# envelope never collides with an effective-only fingerprint over the
# same keys (e.g. effective={x:1},ambient={} vs effective={x:1},ambient={y:2}).
Expand All @@ -1477,6 +1498,8 @@ def _compile_key_for(
envelope["\x00ambient"] = dict(sorted(ambient_context.items()))
if config_identity is not None:
envelope["\x00config"] = config_identity
if task_annotations:
envelope["\x00task_annotations"] = dict(task_annotations)
fingerprint_input: Mapping[str, Any] = envelope
else:
# Byte-identical to today's key: the default-isolation guarantee.
Expand Down Expand Up @@ -2982,6 +3005,12 @@ def _pipeline_source_path(pipeline_fn: PipelineFn) -> Path | None:
return resolved


def _pipeline_reads_cfg(pipeline_fn: PipelineFn) -> bool:
"""Whether compile-time config is observable: a ``cfg`` parameter or a
``task_annotations`` function."""
return _pipeline_accepts_cfg(pipeline_fn) or callable(pipeline_fn.task_annotations)


def _pipeline_accepts_cfg(pipeline_fn: PipelineFn) -> bool:
"""Return whether the pipeline function has a real ``cfg`` parameter.

Expand Down Expand Up @@ -3100,6 +3129,12 @@ def _load_cfg_and_raw(
"descendant subpipelines. Create the file, remove `config=`, "
"or disable propagate_config if there is no config to broadcast."
)
elif usage == "task_annotations":
reason = (
"has a task_annotations function, so "
f"{source} is required as a compile-time input. Create the file, "
"or pass task_annotations a mapping instead."
)
else:
reason = (
"has a `cfg` parameter, so "
Expand Down
17 changes: 10 additions & 7 deletions packages/tangle-cli/src/tangle_cli/python_pipeline/pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from __future__ import annotations

import inspect
from collections.abc import Mapping
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Callable
Expand Down Expand Up @@ -45,7 +46,7 @@ class PipelineFn:
# ``metadata.labels`` — a separate block from annotations, and typed
# ``str -> str`` by both schemas.
labels: dict[str, str] = field(default_factory=dict)
task_annotations: dict[str, Any] = field(default_factory=dict)
task_annotations: Mapping[str, Any] | Callable[[Any], Mapping[str, Any]] = field(default_factory=dict)
caller_dir: Path | None = None
# Convention for the single Out[T] slot's name. Defaults to the PoC
# canonical sentinel ``wait_for_output``; overridable via
Expand Down Expand Up @@ -147,7 +148,7 @@ def pipeline(
config: str | None = None,
annotations: dict[str, Any] | None = None,
labels: dict[str, str] | None = None,
task_annotations: dict[str, Any] | None = None,
task_annotations: Mapping[str, Any] | Callable[[Any], Mapping[str, Any]] | None = None,
flow_direction: str | None = None,
output_name: str = "wait_for_output",
propagate_config: bool = False,
Expand Down Expand Up @@ -176,10 +177,12 @@ def pipeline(
``editor.flow-direction`` root annotation
(``"left-to-right"`` / ``"top-to-bottom"``). Sugar over
``annotations``; applied last, so it wins on that key.
task_annotations: Per-task default annotations applied to every
task in the pipeline. Accepted for API completeness but not
wired through in MVP — the PoC sets per-task annotations
explicitly via ``.with_annotations``.
task_annotations: Default annotations for every task in this
pipeline's body: a mapping, or a function of the pipeline's
resolved ``cfg`` returning one. A task's own annotations win per
key. ``subpipeline`` boundary tasks and the tasks inside a child
are not covered; push defaults into a child with
``subpipeline(child).with_task_annotations(...)``.
output_name: Name for the single ``Out[T]`` output slot.
Defaults to ``wait_for_output`` (matches the PoC).
propagate_config: Broadcasts this pipeline's own config.yaml deep
Expand Down Expand Up @@ -215,7 +218,7 @@ def decorator(fn: Callable[..., Any]) -> PipelineFn:
config_path=config,
annotations=merged_annotations,
labels=checked_labels,
task_annotations=dict(task_annotations or {}),
task_annotations=task_annotations or {},
caller_dir=caller_dir,
output_name=output_name,
propagate_config=propagate_config,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@
``subpipeline(child)`` returns a :class:`SubpipelineRef` — a task-like
handle that mirrors :class:`tangle_cli.python_pipeline.ref.CallableRef`
ergonomics (``.bind`` / ``.named`` / ``.with_annotations`` and call-site
kwargs). Calling the handle inside an active ``@pipeline`` trace records
kwargs), plus ``.with_task_annotations`` for defaults that reach the tasks
inside the child. Calling the handle inside an active ``@pipeline`` trace records
ONE parent task (never the child's internals) and returns a
:class:`tangle_cli.python_pipeline.placeholders.TaskOutputProxy`. Tangle only
supports conditional execution and execution options for container-component
Expand All @@ -33,6 +34,7 @@

from __future__ import annotations

from collections.abc import Mapping
from dataclasses import dataclass, field, replace
from typing import TYPE_CHECKING, Any

Expand Down Expand Up @@ -69,6 +71,7 @@ class SubpipelineRef:
task_id_hint: str | None = None
annotations: dict[str, str] | None = None
config_overrides: dict[str, Any] = field(default_factory=dict)
task_annotations: dict[str, Any] = field(default_factory=dict)

# ------------------------------------------------------------------
# Builders — all return a fresh SubpipelineRef (immutable composition)
Expand All @@ -92,6 +95,24 @@ def with_annotations(self, ann: dict[str, Any]) -> "SubpipelineRef":
merged[k] = v # type: ignore[assignment]
return replace(self, annotations=merged)

def with_task_annotations(self, ann: Mapping[str, Any]) -> "SubpipelineRef":
"""Default annotations for every task in the child's subtree except
boundary tasks; nearer call sites and a task's own annotations win.
Scalar mapping only: it is part of the child's compile key."""
if not isinstance(ann, Mapping):
raise CompileError(
"subpipeline(...).with_task_annotations() takes a mapping, got "
f"{type(ann).__name__}. Compute values from the parent's cfg, e.g. "
".with_task_annotations({'key': cfg.value})."
)
for key, value in ann.items():
if not (value is None or isinstance(value, (str, int, float, bool))):
raise CompileError(
f"subpipeline(...).with_task_annotations() value for {key!r} must be "
f"a string or another scalar, got {type(value).__name__}."
)
return replace(self, task_annotations={**self.task_annotations, **ann})

def with_position(
self,
x: float,
Expand Down Expand Up @@ -253,7 +274,8 @@ def subpipeline(child: "PipelineFn") -> SubpipelineRef:

Returns:
A :class:`SubpipelineRef` supporting ``.bind`` / ``.named`` /
``.with_annotations`` and trace-time ``__call__``.
``.with_annotations`` / ``.with_task_annotations`` and trace-time
``__call__``.

Raises:
CompileError: if ``child`` is not a :class:`PipelineFn` (e.g. a
Expand Down
25 changes: 25 additions & 0 deletions packages/tangle-cli/src/tangle_cli/python_pipeline/trace.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
import inspect
import textwrap
import typing
from collections.abc import Mapping
from contextvars import ContextVar
from typing import TYPE_CHECKING, Any

Expand Down Expand Up @@ -217,11 +218,17 @@ def trace_pipeline(
pipeline_fn: "PipelineFn",
cfg: Any,
inputs: dict[str, Any] | None = None,
*,
inherited_task_annotations: Mapping[str, Any] | None = None,
) -> GraphBuilder:
"""Trace ``pipeline_fn`` against ``cfg`` (+ optional runtime ``inputs``).

Returns the populated :class:`GraphBuilder` ready for emit.

Every task except a ``subpipeline`` boundary task gets the pipeline's own
``task_annotations``, overlaid by ``inherited_task_annotations`` (pushed in
by ancestor call sites), overlaid by the task's own annotations.

The function's parameters are inspected:
- ``cfg`` (no ``In`` annotation, named exactly ``cfg``) → bound to
the loaded :class:`Cfg` object.
Expand Down Expand Up @@ -309,6 +316,13 @@ def trace_pipeline(
finally:
_BUILDER.reset(token)

defaults = {**_own_task_annotations(pipeline_fn, cfg), **(inherited_task_annotations or {})}
if defaults:
boundary_task_ids = {task_id for task_id, _ref in builder.task_refs_for_subpipelines}
for node in builder.tasks:
if node.task_id not in boundary_task_ids:
node.annotations = {**defaults, **(node.annotations or {})}

# Capture the declared output(s). ``get_type_hints`` returns the
# resolved return annotation under the ``return`` key. Two shapes:
# * single ``-> Out[T]`` -> one output (legacy path, unchanged);
Expand Down Expand Up @@ -344,6 +358,17 @@ def trace_pipeline(
return builder


def _own_task_annotations(pipeline_fn: "PipelineFn", cfg: Any) -> dict[str, Any]:
declared = pipeline_fn.task_annotations
resolved = declared(cfg) if callable(declared) else declared
if not isinstance(resolved, Mapping):
raise CompileError(
f"@pipeline({pipeline_fn.name!r}) task_annotations must be a mapping or a "
f"function of cfg returning one, got {type(resolved).__name__}."
)
return dict(resolved)


def _trace_multi_output(
builder: GraphBuilder,
pipeline_fn: "PipelineFn",
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "tangle-cli"
version = "0.1.35"
version = "0.1.36"
description = "CLI for Tangle, the open-source ML pipeline orchestration platform"
readme = "README.md"
authors = [
Expand Down
2 changes: 1 addition & 1 deletion tests/test_packaging.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ def test_tangle_cli_wheel_supports_expert_no_deps_import_path_without_tangle_api
requires_dist = [line for line in metadata.splitlines() if line.startswith("Requires-Dist: ")]
assert not any(name.startswith("tangle_api/") for name in names)
assert "tangle_cli/openapi/openapi.json" not in names
assert "Version: 0.1.35" in metadata
assert "Version: 0.1.36" in metadata
assert "Requires-Dist: tangle-api==0.1.1" in requires_dist
assert not any("extra == 'native'" in line for line in requires_dist)
assert "Provides-Extra: native" in metadata
Expand Down
Loading
Loading