Skip to content

feat(orchestrator): Sweep again without waiting while work is overdue - #426

Open
morgan-wowk wants to merge 1 commit into
masterfrom
orchestrator-sweep-when-overdue
Open

morgan-wowk wants to merge 1 commit into
masterfrom
orchestrator-sweep-when-overdue

Conversation

@morgan-wowk

@morgan-wowk morgan-wowk commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

run_loop sleeps after every sweep. Each sweep refreshes up to _max_queue_batch_size running container executions, and stops early at _max_queue_batch_duration (oasis-backend uses 20 and 200ms). When refreshes are slow API calls, the time cap ends each batch after a few visits, so a backlog clears slowly and the loop spends most of that time asleep.

Changes

  • process_each_queue_once() now returns whether to sweep again right away. It does when a queued execution was processed, or when the running row it picked was last refreshed more than _running_execution_refresh_interval ago (default 30s).
  • run_loop skips the sleep in that case. Idle, caught-up and failing sweeps still sleep.
  • The batch loop moved into _process_queue_batch().

Impact

A backlog is refreshed back to back, then the loop returns to its normal pace. No extra queries.

Production test

#426, #427 and #428 together, as patches (https://github.com/Shopify/oasis-backend/pull/843): completion lag p50 128s → 1.0s, start detection p50 81–250s → 5–14s.

Shares a few lines with #428 and #419; the second to merge needs a small rebase.

@yuechao-qin yuechao-qin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI Assisted)

Approve with two small changes. The pacing decision is in the right place and fails safe: sweep_again_now starts False, so an exception in the sweep still sleeps, and the overdue predicate rides the ix_container_execution_status_last_processed_at index rather than scanning. The five new tests pin the four interesting answers. Two things are worth fixing before merge: a queue that did find work reports empty when the last batch iteration raises (m1), and the overdue query re-asks the database something the running sweep already knew (m2).

# Severity Topic
m1 🔹 minor a queue that did find work reports empty when its last batch iteration raises
m2 🔹 minor the overdue query re-asks the database what the running sweep just selected
N1 ▫️ nit the two-queue unpack breaks the day a third queue handler is added

start_time_seconds = time.monotonic()
queue_has_items = False
for _ in range(self._max_queue_batch_size):
queue_has_items = False

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI Assisted)

m1 · 🔹 minor — a queue that did find work reports empty when its last batch iteration raises

🔹 Severity: minor

Issue. queue_has_items is reset to False at the top of every batch iteration
(orchestrator_sql.py:133) and only assigned on a successful handler call (:136). The except
at :139 logs and does not break, so the value that survives the loop is the one from the
last iteration that ran — and if that iteration raised, it is False.

_max_queue_batch_size = 20   (oasis-backend orchestrator_main_oasis.py:300, production)

iter 1   handler → True      launched an execution
iter 2   handler → True      launched another
iter 3   raises              queue_has_items = False, no break
         ↓
queues_have_items = [False, …]  →  process_each_queue_once() → False  →  sleep 1s

So the sweep that just launched two executions and then hit one transient database error decides
there is nothing to do and waits a full second — at exactly the moment the system is already
struggling. The same reset also makes the pre-loop assignment at :131 dead code for every
_max_queue_batch_size above zero.

Fix. Keep the answer monotonic within the batch:

queue_has_items = False
for _ in range(self._max_queue_batch_size):
    try:
        with self._session_factory() as session:
            if not queue_handler(session=session):
                break
            queue_has_items = True
    except Exception as exc:
        ...

Counter-argument, stated honestly. test_waits_after_a_queue_handler_raises
(tests/test_orchestrator_sweep_pacing.py:135) shows the sleep-on-error behaviour is wanted, and
on an error the cautious choice is to slow down rather than spin. The objection is narrower: this
does not slow down on error, it slows down on error after success, and it loses work the sweep
already proved was there. One extra second only, and only on a mixed batch.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Fixed in d8a2367. The batch loop now sets queue_has_items = True right after a handler returns True, so a batch that found work and then raised still reports work. New test: test_sweeps_again_without_waiting_when_a_batch_raises_after_finding_work.

running_queue_has_items and self._has_overdue_running_container_execution()
)

def _has_overdue_running_container_execution(self) -> bool:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI Assisted)

m2 · 🔹 minor — the overdue query re-asks the database what the running sweep just selected

🔹 Severity: minor

Issue. The running queue selects the single oldest PENDING|RUNNING row, ordered by
last_processed_at (orchestrator_sql.py:234). _has_overdue_running_container_execution then
opens a second session and asks whether any such row is older than the interval — which, given
that ordering, is the same question about the same row the sweep has just handled.

running sweep   SELECT id … ORDER BY last_processed_at LIMIT 1     ← the oldest row
                  (and stamps it `now` at orchestrator_sql.py:892)
                           │
overdue check   SELECT EXISTS(… last_processed_at < now − 30s)     ← one more round trip
                           │
                  answer was already in hand: was the row we just
                  took older than now − 30s when we took it?

One extra round trip per sweep, on a loop whose entire purpose is to sweep more often, in a PR
whose entire purpose is latency.

Fix. Record the selected row's previous last_processed_at in the running handler and compare
it there, dropping _has_overdue_running_container_execution entirely. If the second query is
preferred for clarity, it is worth a comment saying so, because the cheaper answer is right next
to it.

Counter-argument, stated honestly. The query is a covering EXISTS on
ix_container_execution_status_last_processed_at (backend_types_sql.py:513), so it is a
sub-millisecond index probe, not a scan — and the 11-hour production run showed no new errors with
it in place. Keeping the pacing decision out of the handler also keeps the handler's job single.
This is efficiency and clarity, not correctness.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Fixed in d8a2367. The EXISTS query is gone. The running handler records whether the row it just selected was overdue (_is_overdue). The queue selects the oldest row first, so that answers the same question without another query.

):
break
queues_have_items.append(queue_has_items)
queued_queue_has_items, running_queue_has_items = queues_have_items

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI Assisted)

N1 · ▫️ nit — the two-queue unpack breaks the day a third queue handler is added

▫️ Severity: nit — cosmetic, no behaviour change; ignore freely.

Issue. queued_queue_has_items, running_queue_has_items = queues_have_items
(orchestrator_sql.py:150) is positional over a list built 30 lines earlier, so adding a third
handler to queue_handlers (:116) turns the sweep into a ValueError on every iteration.

Fix. Either zip the results back to their handlers, or name the two explicitly:

queued_queue_has_items = self._run_queue_batch(self.internal_process_queued_executions_queue)
running_queue_has_items = self._run_queue_batch(self.internal_process_running_executions_queue)

Counter-argument, stated honestly. The list is immediately above the unpack, there have been
two handlers for the lifetime of this class, and a third would fail loudly on the first test run
rather than silently in production. Nit, raised only because the loop body is now the thing that
decides pacing.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted: Done in d8a2367. The batch loop moved into _process_queue_batch(queue_handler), and process_each_queue_once names each queue's result, so there's no positional unpack.

@morgan-wowk
morgan-wowk force-pushed the orchestrator-sweep-when-overdue branch from 7df65cd to d8a2367 Compare November 5, 2026 19:09
process_each_queue_once now returns whether the next sweep should start
right away: a queued execution was launched, or a pending or running
container execution was last refreshed longer ago than the new internal
_running_execution_refresh_interval (30s). run_loop skips its sleep in
that case, so a backlog of running executions is refreshed back to back
instead of one per second. An idle or caught-up orchestrator still
sleeps between sweeps, and so does one whose queue handler raised.
@Ark-kun

Ark-kun commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

run_loop sleeps after every sweep, and each sweep refreshes at most one running container execution.

Are you sure this is correct? Each sweep refreshes up to _max_queue_batch_size executions which is 20 in our internal Tangle deployment. Which corresponds to up to 20*60=1200 executions per minute.

Sweep again without waiting while work is overdue
A backlog takes minutes to clear, mostly spent asleep.

The main reason for the sleep is throttling. Just because the Orchestrator can send 1000 requests for refresh in 1 second does not mean that the underlying cloud service (GKE, GCP CustomJobs, HF Jobs, etc) will allow that.

A proper solution would be to introduce proper throttling (e.g. 60 new launches per minute, 600 refreshes per minute, 30 seconds between refreshes for same execution (unless launcher has signaled that it was updated)).
With throttlers, an important part is to balance the queued execution queue and the running executions queue. Both need to flow.

@morgan-wowk

Copy link
Copy Markdown
Collaborator Author

🤖 AI-assisted reply to #426 (comment)

  • Batch size: right, it's up to _max_queue_batch_size per sweep; I've fixed the description. In practice the 200ms _max_queue_batch_duration cap ends a batch after a few visits when each refresh is an API call.
  • Throttling: the sleep still applies whenever nothing is more than 30s overdue.
    • Kubernetes refreshes read the informer cache (get_refreshed), so skipping the sleep adds no GKE API calls.
    • In our deployment, ML CP makes one GET per visit, and visits are serial. The ML Infra team has confirmed the rate this produces is fine for their API.
  • Both queues flow: every sweep runs the queued batch before the running one, and queued work also skips the sleep, so launches keep moving during a running backlog.
  • Proper throttles: agreed, as a follow-up. "30s per execution unless the launcher signaled" is what feat(orchestrator): Refresh a container execution as soon as its pod or job changes #428 makes possible: a freshness filter on the running-queue select, plus a per-minute refresh cap.

@morgan-wowk

Copy link
Copy Markdown
Collaborator Author

The main reason for the sleep is throttling. Just because the Orchestrator can send 1000 requests for refresh in 1 second does not mean that the underlying cloud service (GKE, GCP CustomJobs, HF Jobs, etc) will allow that.

A proper solution would be to introduce proper throttling (e.g. 60 new launches per minute, 600 refreshes per minute, 30 seconds between refreshes for same execution (unless launcher has signaled that it was updated)).

🤖 AI-assisted: The per-execution throttle is being introduced in #473, stacked on this PR. Each running container execution is refreshed at most once every 60s unless the launcher signalled a change or the run was cancelled.

This PR's skip-the-sleep behavior has already been running in our production deployment since https://github.com/Shopify/oasis-backend/pull/843, and it hasn't overwhelmed Kubernetes, because:

  • Kubernetes refreshes read the informer cache, not the API server (get_refreshed); an API read only happens on a cache miss.
  • The informers keep one long-lived watch per namespace, so the API load doesn't depend on how often we sweep.
  • Visits are serial, one at a time, so even without the sleep the refresh rate is bounded by visit time.
  • The only per-task Kubernetes API calls are launch and terminate, and this PR doesn't change how often those happen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants