Skip to content

feat(orchestrator): Refresh a container execution as soon as its pod or job changes - #428

Open
morgan-wowk wants to merge 1 commit into
masterfrom
orchestrator-informer-changed-first
Open

morgan-wowk wants to merge 1 commit into
masterfrom
orchestrator-informer-changed-first

Conversation

@morgan-wowk

@morgan-wowk morgan-wowk commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The running queue is round-robin, so a pod that started or finished waits until every other active container execution is visited.

Changes

  • ContainerTaskLauncher.subscribe_to_container_execution_changes(*, listener): launchers report changed containers as a ContainerExecutionChange. The default does nothing.
  • Kubernetes launchers annotate pods and jobs with the container execution id. Their informers notify listeners after the initial list.
  • The orchestrator subscribes once and collects ids in a locked set (max 10,000). Before each running-queue pick, it moves rows that are still PENDING/RUNNING to last_processed_at = now − 2 min, unless they are already ahead.

Why now − 2 minutes, not 1970

Informer re-lists report every pod as changed. With a fixed front, those rows starved other launchers' executions in production. A relative front keeps changes in order, and rows waiting longer still go first.

Impact

  • Changed pods and jobs are refreshed on the next sweep.
  • Wrapping launchers (routers, proxies) must pass subscribe_to_container_execution_changes through.
  • Pods launched before this change stay round-robin.

Production test

#426, #427 and #428 together, as patches (https://github.com/Shopify/oasis-backend/pull/843): completion lag p50 128s → 1.0s, start detection p50 81–250s → 5–14s.

Shares a few lines with #426, #427 and #419; the second to merge needs a small rebase.

) -> _TLaunchedContainer:
raise NotImplementedError()

def pop_changed_container_execution_ids(self) -> list[str]:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interface change

Only the launcher knows which containers changed (its pod and job informers already see every change), while only the orchestrator decides which container executions to refresh next, so a single method on the launcher interface is the smallest way to pass that signal from one to the other. It defaults to [], so launchers that can't tell (Davies, local and Docker launchers, routers that don't delegate) keep the current round-robin with no changes, and it doesn't commit the interface to anything Kubernetes-specific.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatives that leave ContainerTaskLauncher unchanged:

  • Orchestrator kwarg: OrchestratorService_Sql takes an optional function that returns the changed ids, and the deployment passes in the Kubernetes launcher's method.
  • Push callback: Kubernetes launchers take an on_container_execution_changed callback that feeds a set the orchestrator owns.
  • Optional capability: a separate Protocol, or a getattr check, that only Kubernetes launchers implement.
  • Launcher writes to the DB: the informer moves the row to the front itself, but that breaks layering because launchers have no DB session.
  • Downstream patch: keep it as a deployment-side monkeypatch, which is fragile because it depends on informer internals.

Why an interface addition?

It's the only option where the launcher reports changes and the orchestrator consumes them through the same typed contract as the rest of the launcher API. The default [] means no other launcher or deployment needs any change, while the alternatives either spread the wiring across each deployment or hide the same method behind a looser contract.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ark-kun Do you want any alternative?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave this a critical review and propose a different approach.

Decision doc:

https://docs.google.com/document/d/1hH0iTcVH2g8biNEoEQX--OPL5A6y0iCv8xmwekoUnro/edit?usp=sharing

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good analysis.
What do you think about the Optional capability option (protocol/abstract base class)?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: As discussed, we went with the subscribe design from the doc instead. subscribe_to_container_execution_changes(*, listener) is on ContainerTaskLauncher with a no-op default, so launchers that can't tell need no changes (c7b30da).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a good approach to consider. Thanks for mentioning it so I can think about it. I've found that a no-op default here gives the same opt-in without an isinstance branch in the code at runtime. I think we made a good call with the subscribe technique but both would certainly be feasible.

@yuechao-qin yuechao-qin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI Assisted)

Approve with changes. The annotation path is sound — the id is written onto the pod, inherited by the job's pod template, and read off either — and the front-of-queue update is filtered to PENDING|RUNNING, so a terminal row cannot be dragged back. Two behaviours differ from the version that ran in production and deserve an explicit answer: nothing is recorded until the first pop (m1), and a launcher nobody polls accumulates ids without bound (m2). One nit: the new OSS-side annotation write duplicates what oasis's own pod postprocessor already does (N1).

# Severity Topic
m1 🔹 minor nothing is recorded until the first pop, which the version tested in production did not do
m2 🔹 minor a process that does not hold the orchestrator lock records changes nobody collects
N1 ▫️ nit the new annotation write duplicates what oasis's pod postprocessor already does

)
self._pod_postprocessor = pod_postprocessor
self._create_volume_and_volume_mount = _create_volume_and_volume_mount
self._changed_container_execution_ids: set[str] | None = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI Assisted)

m1 · 🔹 minor — nothing is recorded until the first pop, which the version tested in production did not do

🔹 Severity: minor

Issue. self._changed_container_execution_ids starts as None (kubernetes_launchers.py:206)
and _mark_container_execution_changed drops every event while it is None (:542). The set is
created by the first pop_changed_container_execution_ids() call (:527), so a launcher only
begins remembering changes once something has asked it once.

The version that ran in production (oasis-backend#843, patches/informer_dirty_refresh_patching.py)
recorded into a module-level set from the moment the informer started, so this window did not exist
there. The docstring says the behaviour is deliberate — "a launcher nobody asks keeps no ids" — so
this is a question about the trade rather than a bug report.

startup ──► informers start ──► events arrive ──► DROPPED (set is None)
                                     │
            first running sweep ─── pop() ──► set = {} ──► events recorded from here

What it costs. One sweep's worth of events per launcher, so roughly the first second after
start, plus any launcher built later. Those executions fall back to the ordinary 30-second
round-robin — the behaviour this PR exists to improve, during the window right after a deploy when
there is usually a backlog.

Worth an explicit answer: is the dropped window intended to cover a launcher that is
constructed but never polled, and if so, is a one-line set() initialiser plus a comment not
simpler than the sentinel?

Counter-argument, stated honestly. The sentinel costs almost nothing and protects a real case:
a long-lived launcher that nobody polls would otherwise grow a set for the life of the process
(see the non-leader case below). The window is about one sweep. This is a design question, not a
defect.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Fixed in b233833. The informers record from startup, as the production test did. New test: test_records_changes_before_the_first_call.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Still holds after c7b30da. The orchestrator subscribes at construction, so changes are recorded from startup.

container_execution_id = _get_container_execution_id(obj)
if container_execution_id:
with self._changed_container_execution_ids_lock:
if self._changed_container_execution_ids is not None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI Assisted)

m2 · 🔹 minor — a process that does not hold the orchestrator lock records changes nobody collects

🔹 Severity: minor

Issue. Recording is armed by the first pop and then never disarmed, while pop is only
reached from the running sweep — and in oasis the sweep is gated on the orchestrator lock
(oasis-backend orchestrator_main_oasis.py:202). A process that has been leader and then loses the lock keeps
its informers and keeps adding ids at kubernetes_launchers.py:544, with nothing draining them.

deploy overlap:  pod A (was leader, now not)        pod B (leader)
                 informers running                  sweeps, pops, drains
                 set grows, never popped  ←── unbounded for as long as A lives

Fix. The cheap version is to clear the set when a pop has not happened for some interval, or
to cap it; the honest alternative is a comment saying growth is bounded by the process's lifetime
and that is acceptable.

Counter-argument, stated honestly. Each entry is a 36-character id in a set, a stalled
non-leader lives for minutes at most, and the whole structure dies with the process. This is the
smallest of the three leak-shaped observations in this set, and it is only visible at all because
the sentinel in m1 keeps the set alive once armed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Fixed in b233833. The set is capped at _MAX_CHANGED_CONTAINER_EXECUTION_IDS = 10_000, and new ids are dropped past that. A dropped id is only a missed hint, so that execution waits its normal turn. New test: test_keeps_at_most_the_maximum_number_of_ids_between_calls.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: In c7b30da the 10,000-id cap moved from the launcher to the orchestrator's set, so it still bounds an orchestrator that isn't running sweeps.

common_annotations.CONTAINER_EXECUTION_ID_ANNOTATION_KEY
)
if container_execution_id:
pod_annotations = (pod_annotations or {}) | {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI Assisted)

N1 · ▫️ nit — the new annotation write duplicates what oasis's pod postprocessor already does

▫️ Severity: nit — cosmetic, no behaviour change; ignore freely.

Issue. kubernetes_launchers.py:498 copies the container-execution id from the launch
annotations onto the pod's annotations. In oasis that annotation is already written, with the same
key and the same value, by the shared pod postprocessor at oasis-backend launchers/k8s.py:112 —
reached by both the GKE launchers (launchers/gke.py:41) and Nebius (launchers/nebius.py:210).

The duplication is harmless — same key, same value, merged with | — and it is the right call
upstream, where the postprocessor does not exist and the informer path would otherwise never fire.
Worth stating in the PR body so nobody later "cleans up" the oasis one and assumes the OSS write
has been there all along.

Counter-argument, stated honestly. Two writers of the same constant is better than a fast path
that silently depends on a downstream repository. Nit.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Kept on purpose, and now noted in the PR description. The two writes use the same key and value, so they merge. The OSS fast path shouldn't depend on an oasis-only postprocessor, and oasis's stamp predates this.

@morgan-wowk
morgan-wowk force-pushed the orchestrator-informer-changed-first branch from 7b1fd53 to b233833 Compare November 5, 2026 19:09
sql.update(bts.ContainerExecution)
.where(
bts.ContainerExecution.id.in_(container_execution_ids),
bts.ContainerExecution.status.in_(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clever. This fixes a small race condition (caused by multi-threaded informers). Maybe we can leave a short comment about this.

The race condition is as follows:

  • Informer catches pod X Revision A
  • Orchestrator pops the list of changed IDs which includes X. The launcher's list is now empty.
  • Informer catches pod X Revision B (Succeeded). It's added to the list of changed executions.
  • Orchestrator refreshes the execution and sees revision B (Succeeded) updates the DB with new state of X.
  • Orchestrator pops the list of changed IDs which includes X again. Orchestrator overwrites X's last_processed_at.

The case is really minor (incorrect last_processed_at is not a big deal), but I'm glad you avoided this issue.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Kept in c7b30da, and the case is now in the _move_changed_container_executions_to_front docstring. test_a_change_seen_during_a_refresh_moves_the_execution_forward_again covers it.

DYNAMIC_DATA_SECRET_KEY = "secret"
DYNAMIC_DATA_SECRET_NAME_KEY = "name"

_FRONT_OF_RUNNING_QUEUE_LAST_PROCESSED_AT = datetime.datetime(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to see that you've avoided the gotcha with trying to put datetime.min into the DB, some of which do not support very old dates.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: The front is now now − 2 minutes in c7b30da, so no very old dates at all. A fixed front let informer re-lists keep taking the front; the PR description has the details.


def _get_container_execution_id(obj: Any) -> str | None:
"""Reads the container execution id annotation off a pod, or off a job's pod template."""
pod_template = getattr(getattr(obj, "spec", None), "template", None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General comment: Let's not use the dynamic getattr anywhere (unless we absolutely need it).

I think you can use simple, typed expressions like this (it's a pity Python does not support the null coalescing operator):

metadata = None
if isinstance(obj, Pod):
    metadata = pobj.metadata
elif isinstance(obj, Job):
    pod_spec = obj.spec
    if pod_spec:
        pod_template = pod_spec.template
        if pod_template:
            metadata = pod_template.metadata
if not metadata:
    return None
annotations = metadata.annotations or {}
container_execution_id = annotations.get(
            common_annotations.CONTAINER_EXECUTION_ID_ANNOTATION_KEY
        )
return container_execution_id

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Done in c7b30da. No getattr; it reads obj.metadata and, for a V1Job, falls back to spec.template.metadata.

)


def _get_container_execution_id(obj: Any) -> str | None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make it typed (obj: Pod | Job) and avoid Any.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Done in c7b30da: obj: V1Pod | V1Job, and on_change is typed the same way.

common_annotations.CONTAINER_EXECUTION_ID_ANNOTATION_KEY
)
if container_execution_id:
pod_annotations = (pod_annotations or {}) | {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this approach. We should probably mark jobs as well.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Done in c7b30da. Jobs now carry the annotation on the job itself and on its pod template.

) -> _TLaunchedContainer:
raise NotImplementedError()

def pop_changed_container_execution_ids(self) -> list[str]:

@Ark-kun Ark-kun Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

def pop_changed_container_execution_ids(self) -> list[str]:

pop_changed_container_execution_ids

Maybe something like get_and_forget_recently_updated_container_executions? The "pop" term is correct, but to me it seems to put most emphasis on the "remove"/"forget" part (the result of pop is often unused).

list[str]

I suggest returning a list of objects rather than integers. Primitive types paint us into a corner (non-extensible).

@dataclasses.dataclass(kw_only=True)
class ContainerExecutionReference:
    container_execution_id: str

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Renamed in c7b30da. pop is gone; the launcher now pushes a ContainerExecutionChange to listeners passed to subscribe_to_container_execution_changes.

@Ark-kun

Ark-kun commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Thank you, @morgan-wowk .

Only the launcher knows which containers changed (its pod and job informers already see every change), while only the orchestrator decides which container executions to refresh next, so a single method on the launcher interface is the smallest way to pass that signal from one to the other.

Makes sense. I had a similar idea, but with returning just 1 execution ID to check (suggest_next_container_execution_to_refresh). #449

My main concern regarding the general idea was about race conditions a potential for an execution to become stuck in limbo where the launcher thinks that it already gave the information to the Orchestrator, but the Orchestrator lost it (e.g. due to exception when processing). Another way an execution can be lost is due to race conditions for multiple updates for same execution. If we make a mistake with locking (and if we switch to multi-threaded orchestration the possibility increases), then it could be possible to miss an update to Succeeded state (Informer receives intermediate update, Orchestrator starts to check refresh the execution, then another update to Succeeded state comes, then Orchestrator marks that execution as processed, essentially dismissing the 2nd update.
Yet another concern was that if we replace the normal running execution querying from the DB with a static list that is only refreshed once it's completes, then a single execution that somehow fails processing can stall the whole queue due to not requesting a new batch of changed executions.

Fortunately, you had multiple clever ideas here that seem to solve my concerns.
Namely:

  • Using the existing container_execution DB table to "store the list of recently updated executions". Integrating with the existing running queue rather than replacing or having two queues.
    • We no longer need to keep the list somewhere in memory.
    • The list does not replace the DB-based queue
  • The locked "get-and-remove" way of getting the changed items. Getting a whole list of changed executions at once, rather than a single execution. This has pros and cons, but you've avoided the cons.

One minor issue is that the last_updated_at is no longer correct for the items. But I think it's an OK price to pay.

@Ark-kun Ark-kun left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you a lot, Morgan. Approved. Let's discuss the interface changes (new class vs new method, function naming and signature) before merging.
https://github.com/TangleML/tangle/pull/428/files#r4550563107
https://github.com/TangleML/tangle/pull/428/files#r4550604663

…or job changes

The running queue visits container executions round-robin by
last_processed_at, so a pod that finished waits for every other active
container execution to be visited first.

ContainerTaskLauncher gets subscribe_to_container_execution_changes(),
which calls a listener with a ContainerExecutionChange whenever a
container the launcher launched changes. It does nothing by default, so
launchers that can't tell keep today's behaviour. Changes are hints: the
orchestrator still does its normal refresh, and a dropped or repeated
change only means that execution is checked at its normal turn, or once
more.

The Kubernetes launchers stamp the container execution id annotation on
the pods and jobs they launch, and their pod and job informers send a
change for each changed object to every listener. One listener failing
doesn't stop the others. The pod-or-job launcher passes the subscription
to both of the launchers it wraps.

The orchestrator subscribes once, when it is created, and collects the
ids in a set behind a lock, keeping at most 10,000. Before each pick from
the running queue it swaps the set out and moves the executions that are
still PENDING or RUNNING forward, in one UPDATE. Only the loop thread
writes to the database.

A moved execution goes back two minutes rather than to a fixed oldest
time, and only if it isn't already further ahead. Every informer re-list
(after a 410 Gone, a watch error or a resync) reports every cached pod
and job as modified. With a fixed time those executions would keep
taking the front and starve the executions of every other launcher. Now
the front is served in the order changes were recorded, and an execution
left unrefreshed for longer than two minutes still goes first. A change
recorded while its execution is being refreshed, such as the container
finishing, moves it forward again.

Pods and jobs launched before this change carry no id and keep the
round-robin.
@morgan-wowk
morgan-wowk force-pushed the orchestrator-informer-changed-first branch from b233833 to c7b30da Compare November 16, 2026 19:51
def _move_changed_container_executions_to_front(self, session: orm.Session) -> None:
"""Moves each pending or running execution the launcher saw change to `_CHANGED_CONTAINER_EXECUTION_PRIORITY` ago, unless it's already further ahead.

Changes are served in the order recorded, an execution that keeps changing can't starve the rest, and a change

@Ark-kun Ark-kun Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an execution that keeps changing can't starve the rest

Clever idea. A bit non-obvious to prove, but I've verified it's correct.

if not container_execution_ids:
return
front_of_queue_last_processed_at = (
_get_current_time() - _CHANGED_CONTAINER_EXECUTION_PRIORITY

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

datetime.now can be non-monotonic. Fortunately UTC timezone does not experience daylight saving time-travel. But time can still travel back when clock is adjusted.

Unlikely to be a big problem.

self._changed_container_execution_ids.add(change.container_execution_id)

def _move_changed_container_executions_to_front(self, session: orm.Session) -> None:
"""Moves each pending or running execution the launcher saw change to `_CHANGED_CONTAINER_EXECUTION_PRIORITY` ago, unless it's already further ahead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might behave differently from expectations in certain cases. For example if the queue is long and a normal full running execution queue sweep takes more than _CHANGED_CONTAINER_EXECUTION_PRIORITY, then the recently changed executions will be processed with arbitrary delay.

This seems to be the natural conflict between processing the "normal" queue and the listener-based queue. You can make the problem fixable by making _CHANGED_CONTAINER_EXECUTION_PRIORITY configurable.

Non-blocking.

@Ark-kun Ark-kun left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, Morgan. This is a great improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants