Repository navigation
feat(orchestrator): Refresh a container execution as soon as its pod or job changes - #428
morgan-wowk wants to merge 1 commit into
Conversation
| ) -> _TLaunchedContainer: | ||
| raise NotImplementedError() | ||
|
|
||
| def pop_changed_container_execution_ids(self) -> list[str]: |
There was a problem hiding this comment.
Interface change
Only the launcher knows which containers changed (its pod and job informers already see every change), while only the orchestrator decides which container executions to refresh next, so a single method on the launcher interface is the smallest way to pass that signal from one to the other. It defaults to [], so launchers that can't tell (Davies, local and Docker launchers, routers that don't delegate) keep the current round-robin with no changes, and it doesn't commit the interface to anything Kubernetes-specific.
There was a problem hiding this comment.
Alternatives that leave ContainerTaskLauncher unchanged:
- Orchestrator kwarg: OrchestratorService_Sql takes an optional function that returns the changed ids, and the deployment passes in the Kubernetes launcher's method.
- Push callback: Kubernetes launchers take an on_container_execution_changed callback that feeds a set the orchestrator owns.
- Optional capability: a separate Protocol, or a getattr check, that only Kubernetes launchers implement.
- Launcher writes to the DB: the informer moves the row to the front itself, but that breaks layering because launchers have no DB session.
- Downstream patch: keep it as a deployment-side monkeypatch, which is fragile because it depends on informer internals.
Why an interface addition?
It's the only option where the launcher reports changes and the orchestrator consumes them through the same typed contract as the rest of the launcher API. The default [] means no other launcher or deployment needs any change, while the alternatives either spread the wiring across each deployment or hide the same method behind a looser contract.
There was a problem hiding this comment.
I gave this a critical review and propose a different approach.
Decision doc:
https://docs.google.com/document/d/1hH0iTcVH2g8biNEoEQX--OPL5A6y0iCv8xmwekoUnro/edit?usp=sharing
There was a problem hiding this comment.
Good analysis.
What do you think about the Optional capability option (protocol/abstract base class)?
There was a problem hiding this comment.
🤖 AI-assisted: As discussed, we went with the subscribe design from the doc instead. subscribe_to_container_execution_changes(*, listener) is on ContainerTaskLauncher with a no-op default, so launchers that can't tell need no changes (c7b30da).
There was a problem hiding this comment.
It's a good approach to consider. Thanks for mentioning it so I can think about it. I've found that a no-op default here gives the same opt-in without an isinstance branch in the code at runtime. I think we made a good call with the subscribe technique but both would certainly be feasible.
yuechao-qin
left a comment
There was a problem hiding this comment.
(AI Assisted)
Approve with changes. The annotation path is sound — the id is written onto the pod, inherited by the job's pod template, and read off either — and the front-of-queue update is filtered to PENDING|RUNNING, so a terminal row cannot be dragged back. Two behaviours differ from the version that ran in production and deserve an explicit answer: nothing is recorded until the first pop (m1), and a launcher nobody polls accumulates ids without bound (m2). One nit: the new OSS-side annotation write duplicates what oasis's own pod postprocessor already does (N1).
| # | Severity | Topic |
|---|---|---|
| m1 | 🔹 minor | nothing is recorded until the first pop, which the version tested in production did not do |
| m2 | 🔹 minor | a process that does not hold the orchestrator lock records changes nobody collects |
| N1 | ▫️ nit | the new annotation write duplicates what oasis's pod postprocessor already does |
| ) | ||
| self._pod_postprocessor = pod_postprocessor | ||
| self._create_volume_and_volume_mount = _create_volume_and_volume_mount | ||
| self._changed_container_execution_ids: set[str] | None = None |
There was a problem hiding this comment.
(AI Assisted)
m1 · 🔹 minor — nothing is recorded until the first pop, which the version tested in production did not do
🔹 Severity: minor
Issue. self._changed_container_execution_ids starts as None (kubernetes_launchers.py:206)
and _mark_container_execution_changed drops every event while it is None (:542). The set is
created by the first pop_changed_container_execution_ids() call (:527), so a launcher only
begins remembering changes once something has asked it once.
The version that ran in production (oasis-backend#843, patches/informer_dirty_refresh_patching.py)
recorded into a module-level set from the moment the informer started, so this window did not exist
there. The docstring says the behaviour is deliberate — "a launcher nobody asks keeps no ids" — so
this is a question about the trade rather than a bug report.
startup ──► informers start ──► events arrive ──► DROPPED (set is None)
│
first running sweep ─── pop() ──► set = {} ──► events recorded from here
What it costs. One sweep's worth of events per launcher, so roughly the first second after
start, plus any launcher built later. Those executions fall back to the ordinary 30-second
round-robin — the behaviour this PR exists to improve, during the window right after a deploy when
there is usually a backlog.
Worth an explicit answer: is the dropped window intended to cover a launcher that is
constructed but never polled, and if so, is a one-line set() initialiser plus a comment not
simpler than the sentinel?
Counter-argument, stated honestly. The sentinel costs almost nothing and protects a real case:
a long-lived launcher that nobody polls would otherwise grow a set for the life of the process
(see the non-leader case below). The window is about one sweep. This is a design question, not a
defect.
There was a problem hiding this comment.
🤖 AI-assisted: Fixed in b233833. The informers record from startup, as the production test did. New test: test_records_changes_before_the_first_call.
There was a problem hiding this comment.
🤖 AI-assisted: Still holds after c7b30da. The orchestrator subscribes at construction, so changes are recorded from startup.
| container_execution_id = _get_container_execution_id(obj) | ||
| if container_execution_id: | ||
| with self._changed_container_execution_ids_lock: | ||
| if self._changed_container_execution_ids is not None: |
There was a problem hiding this comment.
(AI Assisted)
m2 · 🔹 minor — a process that does not hold the orchestrator lock records changes nobody collects
🔹 Severity: minor
Issue. Recording is armed by the first pop and then never disarmed, while pop is only
reached from the running sweep — and in oasis the sweep is gated on the orchestrator lock
(oasis-backend orchestrator_main_oasis.py:202). A process that has been leader and then loses the lock keeps
its informers and keeps adding ids at kubernetes_launchers.py:544, with nothing draining them.
deploy overlap: pod A (was leader, now not) pod B (leader)
informers running sweeps, pops, drains
set grows, never popped ←── unbounded for as long as A lives
Fix. The cheap version is to clear the set when a pop has not happened for some interval, or
to cap it; the honest alternative is a comment saying growth is bounded by the process's lifetime
and that is acceptable.
Counter-argument, stated honestly. Each entry is a 36-character id in a set, a stalled
non-leader lives for minutes at most, and the whole structure dies with the process. This is the
smallest of the three leak-shaped observations in this set, and it is only visible at all because
the sentinel in m1 keeps the set alive once armed.
There was a problem hiding this comment.
🤖 AI-assisted: Fixed in b233833. The set is capped at _MAX_CHANGED_CONTAINER_EXECUTION_IDS = 10_000, and new ids are dropped past that. A dropped id is only a missed hint, so that execution waits its normal turn. New test: test_keeps_at_most_the_maximum_number_of_ids_between_calls.
There was a problem hiding this comment.
🤖 AI-assisted: In c7b30da the 10,000-id cap moved from the launcher to the orchestrator's set, so it still bounds an orchestrator that isn't running sweeps.
| common_annotations.CONTAINER_EXECUTION_ID_ANNOTATION_KEY | ||
| ) | ||
| if container_execution_id: | ||
| pod_annotations = (pod_annotations or {}) | { |
There was a problem hiding this comment.
(AI Assisted)
N1 · ▫️ nit — the new annotation write duplicates what oasis's pod postprocessor already does
▫️ Severity: nit — cosmetic, no behaviour change; ignore freely.
Issue. kubernetes_launchers.py:498 copies the container-execution id from the launch
annotations onto the pod's annotations. In oasis that annotation is already written, with the same
key and the same value, by the shared pod postprocessor at oasis-backend launchers/k8s.py:112 —
reached by both the GKE launchers (launchers/gke.py:41) and Nebius (launchers/nebius.py:210).
The duplication is harmless — same key, same value, merged with | — and it is the right call
upstream, where the postprocessor does not exist and the informer path would otherwise never fire.
Worth stating in the PR body so nobody later "cleans up" the oasis one and assumes the OSS write
has been there all along.
Counter-argument, stated honestly. Two writers of the same constant is better than a fast path
that silently depends on a downstream repository. Nit.
There was a problem hiding this comment.
🤖 AI-assisted: Kept on purpose, and now noted in the PR description. The two writes use the same key and value, so they merge. The OSS fast path shouldn't depend on an oasis-only postprocessor, and oasis's stamp predates this.
7b1fd53 to
b233833
Compare
| sql.update(bts.ContainerExecution) | ||
| .where( | ||
| bts.ContainerExecution.id.in_(container_execution_ids), | ||
| bts.ContainerExecution.status.in_( |
There was a problem hiding this comment.
Clever. This fixes a small race condition (caused by multi-threaded informers). Maybe we can leave a short comment about this.
The race condition is as follows:
- Informer catches pod X Revision A
- Orchestrator pops the list of changed IDs which includes X. The launcher's list is now empty.
- Informer catches pod X Revision B (Succeeded). It's added to the list of changed executions.
- Orchestrator refreshes the execution and sees revision B (Succeeded) updates the DB with new state of X.
- Orchestrator pops the list of changed IDs which includes X again. Orchestrator overwrites X's last_processed_at.
The case is really minor (incorrect last_processed_at is not a big deal), but I'm glad you avoided this issue.
There was a problem hiding this comment.
🤖 AI-assisted: Kept in c7b30da, and the case is now in the _move_changed_container_executions_to_front docstring. test_a_change_seen_during_a_refresh_moves_the_execution_forward_again covers it.
| DYNAMIC_DATA_SECRET_KEY = "secret" | ||
| DYNAMIC_DATA_SECRET_NAME_KEY = "name" | ||
|
|
||
| _FRONT_OF_RUNNING_QUEUE_LAST_PROCESSED_AT = datetime.datetime( |
There was a problem hiding this comment.
Good to see that you've avoided the gotcha with trying to put datetime.min into the DB, some of which do not support very old dates.
There was a problem hiding this comment.
🤖 AI-assisted: The front is now now − 2 minutes in c7b30da, so no very old dates at all. A fixed front let informer re-lists keep taking the front; the PR description has the details.
|
|
||
| def _get_container_execution_id(obj: Any) -> str | None: | ||
| """Reads the container execution id annotation off a pod, or off a job's pod template.""" | ||
| pod_template = getattr(getattr(obj, "spec", None), "template", None) |
There was a problem hiding this comment.
General comment: Let's not use the dynamic getattr anywhere (unless we absolutely need it).
I think you can use simple, typed expressions like this (it's a pity Python does not support the null coalescing operator):
metadata = None
if isinstance(obj, Pod):
metadata = pobj.metadata
elif isinstance(obj, Job):
pod_spec = obj.spec
if pod_spec:
pod_template = pod_spec.template
if pod_template:
metadata = pod_template.metadata
if not metadata:
return None
annotations = metadata.annotations or {}
container_execution_id = annotations.get(
common_annotations.CONTAINER_EXECUTION_ID_ANNOTATION_KEY
)
return container_execution_idThere was a problem hiding this comment.
🤖 AI-assisted: Done in c7b30da. No getattr; it reads obj.metadata and, for a V1Job, falls back to spec.template.metadata.
| ) | ||
|
|
||
|
|
||
| def _get_container_execution_id(obj: Any) -> str | None: |
There was a problem hiding this comment.
Let's make it typed (obj: Pod | Job) and avoid Any.
There was a problem hiding this comment.
🤖 AI-assisted: Done in c7b30da: obj: V1Pod | V1Job, and on_change is typed the same way.
| common_annotations.CONTAINER_EXECUTION_ID_ANNOTATION_KEY | ||
| ) | ||
| if container_execution_id: | ||
| pod_annotations = (pod_annotations or {}) | { |
There was a problem hiding this comment.
I agree with this approach. We should probably mark jobs as well.
There was a problem hiding this comment.
🤖 AI-assisted: Done in c7b30da. Jobs now carry the annotation on the job itself and on its pod template.
| ) -> _TLaunchedContainer: | ||
| raise NotImplementedError() | ||
|
|
||
| def pop_changed_container_execution_ids(self) -> list[str]: |
There was a problem hiding this comment.
def pop_changed_container_execution_ids(self) -> list[str]:
pop_changed_container_execution_ids
Maybe something like get_and_forget_recently_updated_container_executions? The "pop" term is correct, but to me it seems to put most emphasis on the "remove"/"forget" part (the result of pop is often unused).
list[str]
I suggest returning a list of objects rather than integers. Primitive types paint us into a corner (non-extensible).
@dataclasses.dataclass(kw_only=True)
class ContainerExecutionReference:
container_execution_id: strThere was a problem hiding this comment.
🤖 AI-assisted: Renamed in c7b30da. pop is gone; the launcher now pushes a ContainerExecutionChange to listeners passed to subscribe_to_container_execution_changes.
|
Thank you, @morgan-wowk .
Makes sense. I had a similar idea, but with returning just 1 execution ID to check ( My main concern regarding the general idea was about race conditions a potential for an execution to become stuck in limbo where the launcher thinks that it already gave the information to the Orchestrator, but the Orchestrator lost it (e.g. due to exception when processing). Another way an execution can be lost is due to race conditions for multiple updates for same execution. If we make a mistake with locking (and if we switch to multi-threaded orchestration the possibility increases), then it could be possible to miss an update to Succeeded state (Informer receives intermediate update, Orchestrator starts to check refresh the execution, then another update to Succeeded state comes, then Orchestrator marks that execution as processed, essentially dismissing the 2nd update. Fortunately, you had multiple clever ideas here that seem to solve my concerns.
One minor issue is that the |
Ark-kun
left a comment
There was a problem hiding this comment.
Thank you a lot, Morgan. Approved. Let's discuss the interface changes (new class vs new method, function naming and signature) before merging.
https://github.com/TangleML/tangle/pull/428/files#r4550563107
https://github.com/TangleML/tangle/pull/428/files#r4550604663
…or job changes The running queue visits container executions round-robin by last_processed_at, so a pod that finished waits for every other active container execution to be visited first. ContainerTaskLauncher gets subscribe_to_container_execution_changes(), which calls a listener with a ContainerExecutionChange whenever a container the launcher launched changes. It does nothing by default, so launchers that can't tell keep today's behaviour. Changes are hints: the orchestrator still does its normal refresh, and a dropped or repeated change only means that execution is checked at its normal turn, or once more. The Kubernetes launchers stamp the container execution id annotation on the pods and jobs they launch, and their pod and job informers send a change for each changed object to every listener. One listener failing doesn't stop the others. The pod-or-job launcher passes the subscription to both of the launchers it wraps. The orchestrator subscribes once, when it is created, and collects the ids in a set behind a lock, keeping at most 10,000. Before each pick from the running queue it swaps the set out and moves the executions that are still PENDING or RUNNING forward, in one UPDATE. Only the loop thread writes to the database. A moved execution goes back two minutes rather than to a fixed oldest time, and only if it isn't already further ahead. Every informer re-list (after a 410 Gone, a watch error or a resync) reports every cached pod and job as modified. With a fixed time those executions would keep taking the front and starve the executions of every other launcher. Now the front is served in the order changes were recorded, and an execution left unrefreshed for longer than two minutes still goes first. A change recorded while its execution is being refreshed, such as the container finishing, moves it forward again. Pods and jobs launched before this change carry no id and keep the round-robin.
b233833 to
c7b30da
Compare
| def _move_changed_container_executions_to_front(self, session: orm.Session) -> None: | ||
| """Moves each pending or running execution the launcher saw change to `_CHANGED_CONTAINER_EXECUTION_PRIORITY` ago, unless it's already further ahead. | ||
|
|
||
| Changes are served in the order recorded, an execution that keeps changing can't starve the rest, and a change |
There was a problem hiding this comment.
an execution that keeps changing can't starve the rest
Clever idea. A bit non-obvious to prove, but I've verified it's correct.
| if not container_execution_ids: | ||
| return | ||
| front_of_queue_last_processed_at = ( | ||
| _get_current_time() - _CHANGED_CONTAINER_EXECUTION_PRIORITY |
There was a problem hiding this comment.
datetime.now can be non-monotonic. Fortunately UTC timezone does not experience daylight saving time-travel. But time can still travel back when clock is adjusted.
Unlikely to be a big problem.
| self._changed_container_execution_ids.add(change.container_execution_id) | ||
|
|
||
| def _move_changed_container_executions_to_front(self, session: orm.Session) -> None: | ||
| """Moves each pending or running execution the launcher saw change to `_CHANGED_CONTAINER_EXECUTION_PRIORITY` ago, unless it's already further ahead. |
There was a problem hiding this comment.
This might behave differently from expectations in certain cases. For example if the queue is long and a normal full running execution queue sweep takes more than _CHANGED_CONTAINER_EXECUTION_PRIORITY, then the recently changed executions will be processed with arbitrary delay.
This seems to be the natural conflict between processing the "normal" queue and the listener-based queue. You can make the problem fixable by making _CHANGED_CONTAINER_EXECUTION_PRIORITY configurable.
Non-blocking.
Ark-kun
left a comment
There was a problem hiding this comment.
Thank you, Morgan. This is a great improvement.
Problem
The running queue is round-robin, so a pod that started or finished waits until every other active container execution is visited.
Changes
ContainerTaskLauncher.subscribe_to_container_execution_changes(*, listener): launchers report changed containers as aContainerExecutionChange. The default does nothing.last_processed_at = now − 2 min, unless they are already ahead.Why now − 2 minutes, not 1970
Informer re-lists report every pod as changed. With a fixed front, those rows starved other launchers' executions in production. A relative front keeps changes in order, and rows waiting longer still go first.
Impact
subscribe_to_container_execution_changesthrough.Production test
#426, #427 and #428 together, as patches (https://github.com/Shopify/oasis-backend/pull/843): completion lag p50 128s → 1.0s, start detection p50 81–250s → 5–14s.
Shares a few lines with #426, #427 and #419; the second to merge needs a small rebase.