Repository navigation
Apply @pipeline(task_annotations=...) and add subpipeline(...).with_task_annotations(...) (v0.1.36) - #85
Conversation
…ask_annotations(...)
`@pipeline(task_annotations=...)` was stored and never applied. It now sets
default annotations on every task in the pipeline's body (a mapping, or a
function of the pipeline's resolved cfg); a task's own annotations win per
key, and subpipeline boundary tasks are skipped.
`subpipeline(child).with_task_annotations({...})` pushes defaults into every
task of the child's subtree. Precedence, lowest first: the child's own
task_annotations, an outer call site, a nearer call site, the task's own
annotations. The merged call-site defaults are part of the child's compile
key, so different defaults compile to separate sidecars; unused, keys are
unchanged.
|
|
||
| ##### Default task annotations | ||
|
|
||
| `@pipeline(task_annotations=...)` sets default annotations for every task in that pipeline's body. Pass a mapping, or a function of the pipeline's resolved `cfg` when values differ per environment: |
There was a problem hiding this comment.
Thanks! Run annotations do reach every task (the orchestrator merges defaults < run < task), and we use them for non-identity keys. For identity we want it to stay with the pipeline. Our MLCP annotations (cloud provider, impersonated account, project) are derived from the pipeline config's service_account, so the staging and prod compiles carry the right account whoever submits them. With run annotations every submit path has to pass the right values, and a wrong one silently runs as the wrong account.
If by "pipeline level" you mean @pipeline(annotations=...): as far as I can see it only lands in the root spec's metadata.annotations and isn't copied to the run or its tasks (create takes run annotations separately, and extra_submit_annotations ignores the spec). Am I missing a path?
Volv-G
left a comment
There was a problem hiding this comment.
lgtm, but don't forget to bump the patch version
Problem
@pipeline(task_annotations=...)is accepted and stored, but nothing reads it: no task gets the annotations. A pipeline whose tasks all need the same annotations (for example the launcher'scloud-pipelines.net/orchestration/cloud_providerplus per-environment identity keys) has to repeat.with_annotations(...)on every task, and a missed task goes unnoticed until it runs. There is also no way to push such defaults into asubpipelinechild: annotations on the subpipeline call land on the boundary task only.Change
@pipeline(task_annotations=...)sets default annotations on every task in that pipeline's body.cfg(values often differ per environment). A function makes the config file a required compile-time input, as acfgparameter does..with_annotations,.with_position, ...) win per key.subpipelineboundary tasks are skipped, and a child does not inherit the parent's defaults.subpipeline(child).with_task_annotations({...})pushes defaults into every task of the child's subtree (boundary tasks skipped).task_annotations, an outer call site, a nearer call site, the task's own annotations.PipelineCompileKey, and a function cannot be fingerprinted (two lambdas or closures would collide and silently share a sidecar). Values are computed from the parent'scfg..with_annotations(...)on a subpipeline call keeps its current meaning (the boundary task), so.with_position(...)is unaffected. The README gets a "Default task annotations" section.Bumps the version to 0.1.36.
Tests
tests/test_task_annotation_defaults.pycompiles small pipelines and asserts on the emitted task annotations and sidecar paths:--overrideincluded, on a body without acfgparameter;All fail on
master. Dropping the key fold, or keying a child on its own call site only, fails the sharing tests. Full suite: 2483 passed (Python 3.12).