Skip to content

fix: keep workers from restarting nodes the planner already skipped - #1414

Closed
kvlonge wants to merge 4 commits into
mainfrom
fix/worker-start-after-skip
Closed

kvlonge wants to merge 4 commits into
mainfrom
fix/worker-start-after-skip

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Why

When a source fails its freshness check, a scheduled build already reports the blocked models as skipped before execution. The executor still calls the node-start hook for those planner skips, so the worker then reported them as started. The control API rejects a start for a node whose outcome is final ("node change is inconsistent with its current outcome"). The worker stopped, and every node it had not reached ended skipped, including branches independent of the stale source.

Changes

  • A worker never reports a start for a node whose outcome it has already reported, such as a planner skip, a skip inherited from a failed sibling, or an unchanged node.
  • no_change planner skips need no separate test. The planner never emits a model SKIP with reason no_change: its only model skip is disabled, and blocked models get source_freshness_error or external_upstream_failed. Unchanged models are rebuilt or refreshed. If such an entry ever reached the worker, it would take the same path, completed first and then refused a start.
  • New worker integration test, using a new DuckShop edge case worker_freshness_skip.yml. One stale source has two dependent models; an independent model and a model on a fresh source come after them in execution order.

Verification

  • On main the new test fails: the worker crashes on the rejected start, and the model on the fresh source ends skipped. With this change it ends success, and both stale-source models end skipped and never started.
  • All 29 worker integration tests pass.
  • make check-ci passes.

A scheduled build reported a source-freshness skip as finished and then
reported the same node as started when the executor reached it. The control
API rejects that change, so the worker stopped and every node it had not
reached yet ended skipped, including branches independent of the stale source.

Workers now never report a start for a node whose outcome is already final.
@kvlonge
kvlonge enabled auto-merge (squash) November 11, 2026 11:14
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 11, 2026 — with Amp for GitHub

kvlonge commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Folded into #1413 so the fixes share one CI run. The branch is kept.

@kvlonge kvlonge closed this Nov 11, 2026
auto-merge was automatically disabled November 11, 2026 16:50

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant