Skip to content

fix(incremental): reject cursor models with no inputs and allow empty cursor inputs - #965

Merged
kvlonge merged 2 commits into
mainfrom
fix/incremental-cursor-window-inputs
Oct 6, 2026
Merged

kvlonge merged 2 commits into
mainfrom
fix/incremental-cursor-window-inputs

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Why

Cursor-based incremental models had no defined behaviour with no inputs or empty cursor inputs. Empty inputs could fail late after staging work, or leave an existing table empty on a rebuild.

Changes

  • Breaking: a cursor-based incremental model that reads no __ref()/__source()/__seed() and lists no cursor_inputs (for example one reading a hard-coded table) is now compile error P011, for every incremental strategy (delete_insert, append, merge). Such models previously re-read everything on every run.
  • Empty cursor inputs on an incremental run: the model succeeds without changing its destination, its audits still run, and plan/build show no input rows (<input>.<column>). JSON: empty_cursor_inputs (build), resolution_status: "no_input_rows" and empty_inputs (plan).
  • When only some inputs are empty and others have new rows, plan and build warn waiting on empty input <input>.<column>; other inputs have new rows, counted in WARN= and the JSON warnings, with waiting_on_empty_inputs: true. Exit code and status are unchanged.
  • A first build over empty inputs creates the destination empty.
  • A rebuild that would replace an existing table with nothing (replay_on_change full, aliased models included, or watermark microbatch --full-refresh in any watermark mode) fails with S302 and keeps the rows.
  • Models whose inputs are built earlier in the same run follow the same rules. Previously they could advance past an empty input and skip rows it delivered later.
  • Inputs are named by project name, including inputs built earlier in the same run.

Verification

  • New E2E tests cover empty first builds, emptied incremental runs, the waiting warning (planned and same-run), aliased and microbatch rebuild refusal, project-name labels, P011, and Dagster check results on a no-op run. Text and --json outputs are both checked.
  • One independent review plus follow-up; make test, make check-ci.

…e missing or empty

Breaking: a cursor-based incremental model that reads no __ref(),
__source() or __seed() and lists no cursor_inputs (for example one that
reads a hard-coded table) is now compile error P011, pointing at its
cursor declaration. This applies to every incremental strategy
(delete_insert, append and merge). Such models previously re-read everything on every
run because no input could bound their window.

When cursor inputs have no rows:
- An incremental run succeeds without changing the destination, still
  runs the model's audits against it, and reports
  "no input rows (<input>.<column>)" in plan and build output. Build
  JSON lists empty_cursor_inputs; plan JSON reports resolution_status
  "no_input_rows" with empty_inputs.
- When only some inputs are empty and others have new rows, the model
  makes no progress and plan and build show a warning such as
  "waiting on empty input stg_refunds.refunded_at; other inputs have
  new rows". It counts toward the build summary's WARN= total and the
  asset's JSON warnings, and JSON adds "waiting_on_empty_inputs": true.
  Exit code and status are unchanged.
- A first build creates the destination empty.
- A rebuild that would replace an existing destination with nothing
  (replay_on_change full after a query change, including aliased
  models, or a watermark microbatch --full-refresh in any watermark
  mode) fails with S302 and keeps the existing rows.

Models whose inputs are built earlier in the same run follow the same
rules; previously such a model could advance past an empty input and
skip rows that input later delivered. Inputs are named by their project
names, including inputs built earlier in the same run. The window is decided before any delta or
staging relation is created, and other window failures name the model,
its inputs and the --start-cursor/--end-cursor flags.
@kvlonge
kvlonge enabled auto-merge (squash) October 6, 2026 08:40
@kvlonge
kvlonge merged commit a4b86c6 into main Oct 6, 2026
32 checks passed
@kvlonge
kvlonge deleted the fix/incremental-cursor-window-inputs branch October 6, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant