Repository navigation
feat(orchestrator): Sweep again without waiting while work is overdue - #426
morgan-wowk wants to merge 1 commit into
Conversation
yuechao-qin
left a comment
There was a problem hiding this comment.
(AI Assisted)
Approve with two small changes. The pacing decision is in the right place and fails safe: sweep_again_now starts False, so an exception in the sweep still sleeps, and the overdue predicate rides the ix_container_execution_status_last_processed_at index rather than scanning. The five new tests pin the four interesting answers. Two things are worth fixing before merge: a queue that did find work reports empty when the last batch iteration raises (m1), and the overdue query re-asks the database something the running sweep already knew (m2).
| # | Severity | Topic |
|---|---|---|
| m1 | 🔹 minor | a queue that did find work reports empty when its last batch iteration raises |
| m2 | 🔹 minor | the overdue query re-asks the database what the running sweep just selected |
| N1 | ▫️ nit | the two-queue unpack breaks the day a third queue handler is added |
| start_time_seconds = time.monotonic() | ||
| queue_has_items = False | ||
| for _ in range(self._max_queue_batch_size): | ||
| queue_has_items = False |
There was a problem hiding this comment.
(AI Assisted)
m1 · 🔹 minor — a queue that did find work reports empty when its last batch iteration raises
🔹 Severity: minor
Issue. queue_has_items is reset to False at the top of every batch iteration
(orchestrator_sql.py:133) and only assigned on a successful handler call (:136). The except
at :139 logs and does not break, so the value that survives the loop is the one from the
last iteration that ran — and if that iteration raised, it is False.
_max_queue_batch_size = 20 (oasis-backend orchestrator_main_oasis.py:300, production)
iter 1 handler → True launched an execution
iter 2 handler → True launched another
iter 3 raises queue_has_items = False, no break
↓
queues_have_items = [False, …] → process_each_queue_once() → False → sleep 1s
So the sweep that just launched two executions and then hit one transient database error decides
there is nothing to do and waits a full second — at exactly the moment the system is already
struggling. The same reset also makes the pre-loop assignment at :131 dead code for every
_max_queue_batch_size above zero.
Fix. Keep the answer monotonic within the batch:
queue_has_items = False
for _ in range(self._max_queue_batch_size):
try:
with self._session_factory() as session:
if not queue_handler(session=session):
break
queue_has_items = True
except Exception as exc:
...Counter-argument, stated honestly. test_waits_after_a_queue_handler_raises
(tests/test_orchestrator_sweep_pacing.py:135) shows the sleep-on-error behaviour is wanted, and
on an error the cautious choice is to slow down rather than spin. The objection is narrower: this
does not slow down on error, it slows down on error after success, and it loses work the sweep
already proved was there. One extra second only, and only on a mixed batch.
There was a problem hiding this comment.
🤖 AI-assisted: Fixed in d8a2367. The batch loop now sets queue_has_items = True right after a handler returns True, so a batch that found work and then raised still reports work. New test: test_sweeps_again_without_waiting_when_a_batch_raises_after_finding_work.
| running_queue_has_items and self._has_overdue_running_container_execution() | ||
| ) | ||
|
|
||
| def _has_overdue_running_container_execution(self) -> bool: |
There was a problem hiding this comment.
(AI Assisted)
m2 · 🔹 minor — the overdue query re-asks the database what the running sweep just selected
🔹 Severity: minor
Issue. The running queue selects the single oldest PENDING|RUNNING row, ordered by
last_processed_at (orchestrator_sql.py:234). _has_overdue_running_container_execution then
opens a second session and asks whether any such row is older than the interval — which, given
that ordering, is the same question about the same row the sweep has just handled.
running sweep SELECT id … ORDER BY last_processed_at LIMIT 1 ← the oldest row
(and stamps it `now` at orchestrator_sql.py:892)
│
overdue check SELECT EXISTS(… last_processed_at < now − 30s) ← one more round trip
│
answer was already in hand: was the row we just
took older than now − 30s when we took it?
One extra round trip per sweep, on a loop whose entire purpose is to sweep more often, in a PR
whose entire purpose is latency.
Fix. Record the selected row's previous last_processed_at in the running handler and compare
it there, dropping _has_overdue_running_container_execution entirely. If the second query is
preferred for clarity, it is worth a comment saying so, because the cheaper answer is right next
to it.
Counter-argument, stated honestly. The query is a covering EXISTS on
ix_container_execution_status_last_processed_at (backend_types_sql.py:513), so it is a
sub-millisecond index probe, not a scan — and the 11-hour production run showed no new errors with
it in place. Keeping the pacing decision out of the handler also keeps the handler's job single.
This is efficiency and clarity, not correctness.
There was a problem hiding this comment.
🤖 AI-assisted: Fixed in d8a2367. The EXISTS query is gone. The running handler records whether the row it just selected was overdue (_is_overdue). The queue selects the oldest row first, so that answers the same question without another query.
| ): | ||
| break | ||
| queues_have_items.append(queue_has_items) | ||
| queued_queue_has_items, running_queue_has_items = queues_have_items |
There was a problem hiding this comment.
(AI Assisted)
N1 · ▫️ nit — the two-queue unpack breaks the day a third queue handler is added
▫️ Severity: nit — cosmetic, no behaviour change; ignore freely.
Issue. queued_queue_has_items, running_queue_has_items = queues_have_items
(orchestrator_sql.py:150) is positional over a list built 30 lines earlier, so adding a third
handler to queue_handlers (:116) turns the sweep into a ValueError on every iteration.
Fix. Either zip the results back to their handlers, or name the two explicitly:
queued_queue_has_items = self._run_queue_batch(self.internal_process_queued_executions_queue)
running_queue_has_items = self._run_queue_batch(self.internal_process_running_executions_queue)Counter-argument, stated honestly. The list is immediately above the unpack, there have been
two handlers for the lifetime of this class, and a third would fail loudly on the first test run
rather than silently in production. Nit, raised only because the loop body is now the thing that
decides pacing.
There was a problem hiding this comment.
🤖 AI-assisted: Done in d8a2367. The batch loop moved into _process_queue_batch(queue_handler), and process_each_queue_once names each queue's result, so there's no positional unpack.
7df65cd to
d8a2367
Compare
process_each_queue_once now returns whether the next sweep should start right away: a queued execution was launched, or a pending or running container execution was last refreshed longer ago than the new internal _running_execution_refresh_interval (30s). run_loop skips its sleep in that case, so a backlog of running executions is refreshed back to back instead of one per second. An idle or caught-up orchestrator still sleeps between sweeps, and so does one whose queue handler raised.
d8a2367 to
63c7a02
Compare
Are you sure this is correct? Each sweep refreshes up to
The main reason for the sleep is throttling. Just because the Orchestrator can send 1000 requests for refresh in 1 second does not mean that the underlying cloud service (GKE, GCP CustomJobs, HF Jobs, etc) will allow that. A proper solution would be to introduce proper throttling (e.g. 60 new launches per minute, 600 refreshes per minute, 30 seconds between refreshes for same execution (unless launcher has signaled that it was updated)). |
|
🤖 AI-assisted reply to #426 (comment)
|
🤖 AI-assisted: The per-execution throttle is being introduced in #473, stacked on this PR. Each running container execution is refreshed at most once every 60s unless the launcher signalled a change or the run was cancelled. This PR's skip-the-sleep behavior has already been running in our production deployment since https://github.com/Shopify/oasis-backend/pull/843, and it hasn't overwhelmed Kubernetes, because:
|
Problem
run_loopsleeps after every sweep. Each sweep refreshes up to_max_queue_batch_sizerunning container executions, and stops early at_max_queue_batch_duration(oasis-backend uses 20 and 200ms). When refreshes are slow API calls, the time cap ends each batch after a few visits, so a backlog clears slowly and the loop spends most of that time asleep.Changes
process_each_queue_once()now returns whether to sweep again right away. It does when a queued execution was processed, or when the running row it picked was last refreshed more than_running_execution_refresh_intervalago (default 30s).run_loopskips the sleep in that case. Idle, caught-up and failing sweeps still sleep._process_queue_batch().Impact
A backlog is refreshed back to back, then the loop returns to its normal pace. No extra queries.
Production test
#426, #427 and #428 together, as patches (https://github.com/Shopify/oasis-backend/pull/843): completion lag p50 128s → 1.0s, start detection p50 81–250s → 5–14s.
Shares a few lines with #428 and #419; the second to merge needs a small rebase.