Skip to content

Apply @pipeline(task_annotations=...) and add subpipeline(...).with_task_annotations(...) (v0.1.36) - #85

Merged
Volv-G merged 2 commits into
TangleML:masterfrom
binazarov-shop:task-annotation-defaults
Nov 17, 2026
Merged

Volv-G merged 2 commits into
TangleML:masterfrom
binazarov-shop:task-annotation-defaults

Conversation

@binazarov-shop

@binazarov-shop binazarov-shop commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem

@pipeline(task_annotations=...) is accepted and stored, but nothing reads it: no task gets the annotations. A pipeline whose tasks all need the same annotations (for example the launcher's cloud-pipelines.net/orchestration/cloud_provider plus per-environment identity keys) has to repeat .with_annotations(...) on every task, and a missed task goes unnoticed until it runs. There is also no way to push such defaults into a subpipeline child: annotations on the subpipeline call land on the boundary task only.

Change

@pipeline(task_annotations=...) sets default annotations on every task in that pipeline's body.

  • Takes a mapping, or a function of the pipeline's resolved cfg (values often differ per environment). A function makes the config file a required compile-time input, as a cfg parameter does.
  • A task's own annotations (.with_annotations, .with_position, ...) win per key.
  • subpipeline boundary tasks are skipped, and a child does not inherit the parent's defaults.

subpipeline(child).with_task_annotations({...}) pushes defaults into every task of the child's subtree (boundary tasks skipped).

  • Precedence, lowest first: the child's own task_annotations, an outer call site, a nearer call site, the task's own annotations.
  • Mapping of scalars only, no function: the merged call-site defaults are folded into the child's PipelineCompileKey, and a function cannot be fingerprinted (two lambdas or closures would collide and silently share a sidecar). Values are computed from the parent's cfg.
  • Different defaults compile the child into separate sidecars; identical calls share one. Pipelines that don't use it keep byte-identical keys and sidecar names.

.with_annotations(...) on a subpipeline call keeps its current meaning (the boundary task), so .with_position(...) is unaffected. The README gets a "Default task annotations" section.

Bumps the version to 0.1.36.

Tests

tests/test_task_annotation_defaults.py compiles small pipelines and asserts on the emitted task annotations and sidecar paths:

  • own defaults, per-key task override, boundary tasks and children untouched;
  • a defaults function reads the resolved config, --override included, on a body without a cfg parameter;
  • precedence across root → child → grandchild;
  • sidecar sharing: the same child with different defaults splits, identical defaults share, and a plain grandchild under a shared child splits by the defaults pushed above it;
  • rejection of a non-mapping, of a function at the call site, and of a non-scalar call-site value.

All fail on master. Dropping the key fold, or keying a child on its own call site only, fails the sharing tests. Full suite: 2483 passed (Python 3.12).

…ask_annotations(...)

`@pipeline(task_annotations=...)` was stored and never applied. It now sets
default annotations on every task in the pipeline's body (a mapping, or a
function of the pipeline's resolved cfg); a task's own annotations win per
key, and subpipeline boundary tasks are skipped.

`subpipeline(child).with_task_annotations({...})` pushes defaults into every
task of the child's subtree. Precedence, lowest first: the child's own
task_annotations, an outer call site, a nearer call site, the task's own
annotations. The merged call-site defaults are part of the child's compile
key, so different defaults compile to separate sidecars; unused, keys are
unchanged.
@binazarov-shop
binazarov-shop marked this pull request as ready for review November 16, 2026 18:58
Comment thread README.md

##### Default task annotations

`@pipeline(task_annotations=...)` sets default annotations for every task in that pipeline's body. Pass a mapping, or a function of the pipeline's resolved `cfg` when values differ per environment:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Run annotations do reach every task (the orchestrator merges defaults < run < task), and we use them for non-identity keys. For identity we want it to stay with the pipeline. Our MLCP annotations (cloud provider, impersonated account, project) are derived from the pipeline config's service_account, so the staging and prod compiles carry the right account whoever submits them. With run annotations every submit path has to pass the right values, and a wrong one silently runs as the wrong account.

If by "pipeline level" you mean @pipeline(annotations=...): as far as I can see it only lands in the root spec's metadata.annotations and isn't copied to the run or its tasks (create takes run annotations separately, and extra_submit_annotations ignores the spec). Am I missing a path?

@Volv-G Volv-G left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, but don't forget to bump the patch version

@binazarov-shop binazarov-shop changed the title Apply @pipeline(task_annotations=...) and add subpipeline(...).with_task_annotations(...) Apply @pipeline(task_annotations=...) and add subpipeline(...).with_task_annotations(...) (v0.1.36) Nov 17, 2026
@Volv-G
Volv-G merged commit 078744a into TangleML:master Nov 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants