Skip to content

Commit e382490

Browse files
authored
fix(planner)!: decide each model replay from its own change only (#895)
* fix(planner)!: decide each model's replay from its own change only Remove upstream backfill cascades. A model's backfill now comes only from its own change and its own replay_on_change, so a first-run, renamed or replayed upstream no longer forces a full or bounded replay of downstream incremental models. A changed function counts as an effective query change of the models that call it directly. Those models plan as function_changed with the cause named and apply their own replay_on_change; models further downstream continue forward. The function-level replay_on_change header is removed. A model that sets full_refresh false now refuses (S203) any planned full rebuild other than its own first run or its own query change with replay_on_change full. A FULL backfill on a non-microbatch model with cursor intrinsics reports the FULL backfill and its cause instead of an unresolved-bounds error. Plan text drops the Upstream changed group and adds a Function changed group; plan JSON drops the cascade field and upstream_changed reason and adds changed_functions to direct callers. BREAKING CHANGE: backfills no longer cascade from upstream models; each model replays only for its own change per its own replay_on_change. The function-level replay_on_change header is removed. Plan JSON drops the cascade field and the upstream_changed reason, and adds changed_functions on direct callers of changed functions and query_changed on renamed models whose query changed. New planner error S203 refuses full rebuilds of full_refresh false models not requested by their own configuration. * fix(migrations): plan renames with changed SQL as renamed query changes A recognised migrate_from rename of a table or view whose handed-over definition differs from its new query now keeps the handed-over identity and plans as renamed with a query change, instead of a first run. Its backfill comes from the model's own replay_on_change (forward-only by default). Once the renamed model has built after the recorded rename, its own fingerprint is used, so later plans no longer report the query change. * test(e2e): cover own-change replay for new, renamed and function-changed upstreams * docs: describe per-model replay decisions, function callers and full_refresh protection * fix(functions): reject unknown FUNCTION header and @udf keys SQL FUNCTION() headers and Python @udf(...) decorators now fail discovery with D002 when they use a key that functions do not support, naming the key and its file and line, the same way HOOK() and AUDIT() headers reject unsupported keys. Previously such keys were silently ignored. * fix(executor): report called-function changes to custom materializations Custom materializations now receive query_changed for function_changed models as well as query_changed ones, since a changed directly called function is the model's own query change. * fix(planner): check full_refresh protection only for models that execute S203 is no longer evaluated for models the plan skips because a source freshness check or an external upstream blocked them. * fix(planner): ignore rename-rewritten references and inferred upstream columns for replay A downstream model whose only difference is __ref and cursor_inputs names rewritten for an upstream renamed with migrate_from in this plan is no longer a query or config change: its migration fingerprint with the renamed names mapped back to their origins must equal its stored one. Views are still recreated and incremental models continue forward; a real edit alongside the rewrite is a normal query change. Schema findings inferred from a model's unchanged SQL, such as a column an upstream added through SELECT *, no longer trigger replay_on_change; they evolve forward through on_schema_change. Changes to the model's own declared columns still follow its replay_on_change. * fix(planner): match rename-rewritten references by query hash and recorded renames A downstream model is no longer a query change when its compiled SQL, with __ref names of renamed models substituted back to their origin names, hashes to its stored definition hash. Substitution touches only reference calls outside comments and string literals, so it works for any SQL, including models the migration fingerprint parser cannot normalize. The rename map combines this plan's handovers with renames recorded in migration history after the downstream's last build, so renames built in an earlier run or whose migrate_from was removed are recognized too. Migration events are read only when a selected model's query hash changed and the migration table was already listed by the warehouse snapshot. A column removed from the warehouse is attributed to the model's declared columns only when no inferred columns exist, so an upstream dropping a column a partial columns block does not declare evolves forward instead of following replay_on_change. * fix(planner): replay schema differences only when the model's own definition changed A schema finding follows replay_on_change only when the model's own SQL, header config, contract, or called functions changed in this plan, with rename-rewritten references not counting as a change. When the model is unchanged, every schema difference comes from upstream, whether added, removed or retyped, declared or inferred; it evolves through on_schema_change and contract enforcement as before, so an enforced contract whose declared column disappeared upstream still fails the build instead of replaying. The interim removed-column source heuristic is reverted. * fix(planner): record declared columns so their own edits replay schema changes Model fingerprint metadata now carries a declared_columns_hash key covering the model's columns block or schema YAML columns (name, type, nullable), dynamic column families, type enforcement and contract setting. The key is additive, sits outside version identity, and is ignored by config-change comparison. A schema finding follows replay_on_change when this hash differs from the recorded one, alongside SQL, config, contract and function changes. Rows written before the key existed have no hash, so their declared columns count as unchanged and the model continues forward once. * fix(planner): keep migration fingerprint work at one computation per model The migration fingerprint cache keyed entries on the raw fingerprint metadata JSON, so the declared_columns_hash key made build-time entries miss the value computed during planning. The cache now strips that key before keying, as the key never contributes to the fingerprint.
1 parent f7c8faf commit e382490

96 files changed

Lines changed: 3390 additions & 1610 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,7 @@ E2E_DUCKDB_BUILD_CORE_PATHS := \
9494
tests/e2e/src/sqlbuild/cli/commands/main/build/test_validation_failures.py
9595

9696
E2E_DUCKDB_BUILD_INCREMENTAL_PATHS := \
97+
tests/e2e/src/sqlbuild/cli/commands/main/build/own_change_replay \
9798
tests/e2e/src/sqlbuild/cli/commands/main/build/test_append_cursor_build.py \
9899
tests/e2e/src/sqlbuild/cli/commands/main/build/test_capped_microbatch_build.py \
99100
tests/e2e/src/sqlbuild/cli/commands/main/build/test_concurrent_microbatch_build.py \

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in
5555
- [Functions](concepts/functions.md) (`concepts/functions`) - User-defined functions and table functions managed as part of your project.
5656
- [Incremental Models](concepts/incremental.md) (`concepts/incremental`) - Cursor-based incremental strategies, microbatch execution, and backfill policies.
5757
- [Planning and change detection](concepts/planning.md) (`concepts/planning`) - How SQLBuild explains build work, detects changes, and chooses safe model actions.
58-
- [Cascade propagation](concepts/planning/cascade-propagation.md) (`concepts/planning/cascade-propagation`) - How a change signal propagates downstream through the DAG, and how each materialization type responds.
58+
- [Replay decisions](concepts/planning/replay-decisions.md) (`concepts/planning/replay-decisions`) - How much history each model reprocesses: decided by its own change and its own replay_on_change, never inherited from upstream.
5959
- [Source freshness](concepts/planning/source-freshness.md) (`concepts/planning/source-freshness`) - Observe external source changes and propagate them through planning.
6060
- [Selection and staleness](concepts/planning/selection-and-staleness.md) (`concepts/planning/selection-and-staleness`) - How selection interacts with change detection, and the stale warnings that stop silent partial rebuilds.
6161
- [Snapshots (SCD Type 2)](concepts/snapshots.md) (`concepts/snapshots`) - Preserve row history over time using SCD Type 2 semantics with timestamp or check-based change detection.

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,9 @@ Seeds (1)
8585
✓ Plan complete 13 selected
8686
```
8787

88-
When query or schema changes are detected, the plan shows the affected models with backfill actions and cascade information.
88+
When query, function, or schema changes are detected, the plan lists each affected model under its own change with its backfill action. A model that calls a changed function directly appears under `Function changed` with `cause: function <name> changed`; models further downstream continue forward. See [Replay decisions](../concepts/planning/replay-decisions.md).
89+
90+
In `sqb plan --json`, each model's `reason` is its own change. A direct caller of a changed function has `"reason": "function_changed"` and a `changed_functions` list, and a renamed model whose query also changed has `"reason": "renamed"` and `"query_changed": true`.
8991

9092
Warnings and maintenance notices, such as `migrate_from can be removed`, are listed in a `Warnings` section before the final `Plan complete` line. `sqb plan --json` output is unchanged.
9193

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/functions.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,15 +16,15 @@ Online: https://sqlbuild.com/docs/concepts/functions/
1616
- Python UDF options
1717
- Adapter support
1818

19-
Functions are SQL or Python definitions under `functions/` that SQLBuild compiles and deploys to the warehouse alongside your models. They participate in the DAG - if a function definition changes, every model that uses it is rebuilt.
19+
Functions are SQL or Python definitions under `functions/` that SQLBuild compiles and deploys to the warehouse alongside your models. They participate in the DAG: if a function definition changes, SQLBuild redeploys it, and every model that calls it directly treats the change as a change to its own query.
2020

2121
## Scalar UDFs
2222

2323
Scalar UDFs return a single value per row. They can be written in SQL or Python.
2424

2525
### SQL UDFs
2626

27-
Place SQL function files under `functions/sql/`. Each file has a `FUNCTION()` header declaring arguments and return type, followed by a SQL expression body:
27+
Place SQL function files under `functions/sql/`. Each file has a `FUNCTION()` header declaring arguments and return type, followed by a SQL expression body. The header accepts `arguments`, `returns`, `database`, `schema`, and `tags`; any other key fails compilation with `D002`, naming the key and its file and line:
2828

2929
```sql
3030
-- functions/sql/udf__is_completed_order.sql
@@ -144,7 +144,9 @@ function folder or to that folder and its children.
144144

145145
## Change propagation
146146

147-
Functions participate in fingerprint-based change detection. Their identity includes dependencies and declared return contracts in addition to the function body and runtime metadata. If a function changes, SQLBuild redeploys it and marks dependent models as changed.
147+
Functions participate in fingerprint-based change detection. Their identity includes dependencies and declared return contracts in addition to the function body and runtime metadata. If a function changes, SQLBuild redeploys it.
148+
149+
A model that calls a changed function directly plans as `Function changed`, with `cause: function <name> changed`. The function change counts as a change to that model's own query, so the model's own `replay_on_change` decides how much history it reprocesses (forward-only by default). Models further downstream are not affected by the function change and continue forward. Functions have no replay setting of their own; to reprocess history when a function changes, set `replay_on_change` on the models that call it. See [Replay decisions](planning/replay-decisions.md).
148150

149151
## Project layout
150152

@@ -169,6 +171,8 @@ The `@udf` decorator accepts these keyword arguments:
169171
| `entry_point` | No | Function name to use as the entry point (defaults to the decorated function name) |
170172
| `packages` | No | List of Python packages required at runtime |
171173

174+
`database`, `schema`, and `tags` are also accepted. Any other keyword fails compilation with `D002`, naming the keyword and its file and line.
175+
172176
## Adapter support
173177

174178
All four supported adapters implement SQL UDFs, Python UDFs, and table functions:

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/incremental.md‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,15 @@ MODEL (
114114
);
115115
```
116116

117-
The override is evaluated per model, so one selection can contain incrementally executed opt-outs and full-refreshed models. It controls execution mode rather than acting as a safety rejection: an opted-out model does not abort the rest of the build.
117+
The override is evaluated per model, so one selection can contain incrementally executed opt-outs and full-refreshed models. With `--full-refresh`, an opted-out model simply runs incrementally; it does not abort the rest of the build.
118+
119+
`full_refresh false` also protects the model from every other full rebuild. Only the model's own explicit configuration can fully rebuild it: its own first run, or its own query change with `replay_on_change full`. If anything else would plan a full rebuild, for example a changed function that the model calls while it sets `replay_on_change full`, the plan fails with `S203`, naming the model and the cause, instead of silently replaying its history:
120+
121+
```
122+
error[S203]: model 'order_history' sets full_refresh false, but the plan would fully rebuild it (function normalize_status changed with replay_on_change full)
123+
```
124+
125+
Set `replay_on_change` to `forward` or `bounded-<duration>` to keep the existing history, or remove `full_refresh false` to allow the rebuild. Upstream changes never cause a full rebuild of a downstream model; see [Replay decisions](planning/replay-decisions.md).
118126

119127
This does not skip initial loading. If the destination relation does not exist, an incremental or microbatch model still builds the history required by its cursor policy. Cloning an existing destination before the build can provide a current watermark and avoid a first-run historical replay.
120128

@@ -173,7 +181,7 @@ WHERE ordered_at >= __cursor_start()
173181
AND ordered_at < __cursor_end()
174182
```
175183

176-
`__cursor_start()` is the effective inclusive start and `__cursor_end()` is the effective exclusive end after cursor floors, lookback, replay policy, and command-line overrides have been applied. The intrinsics accept no arguments and are only valid in built-in cursor incremental model query SQL. They are rejected in functions, hooks, audits, SQL tests and scenarios, source expressions, non-incremental or cursorless models, custom materializations, and non-microbatch full refreshes.
184+
`__cursor_start()` is the effective inclusive start and `__cursor_end()` is the effective exclusive end after cursor floors, lookback, replay policy, and command-line overrides have been applied. The intrinsics accept no arguments and are only valid in built-in cursor incremental model query SQL. They are rejected in functions, hooks, audits, SQL tests and scenarios, source expressions, non-incremental or cursorless models, custom materializations, and non-microbatch full refreshes. A non-microbatch model given a FULL backfill by `--full-refresh` fails with a message naming that cause; pass an explicit interval with `--start-cursor-ts` and `--end-cursor-ts` (or the `-int` forms) to rebuild it.
177185

178186
In microbatch mode, the intrinsics resolve to each batch's concrete bounds. A microbatch full refresh discovers its range from current inputs while ignoring the old destination watermark.
179187

@@ -324,7 +332,7 @@ The legacy scalar `max_microbatches` model field remains supported as a fail/war
324332

325333
### Mixed-grain chains
326334

327-
When a downstream microbatch model reads from an upstream model with a coarser time grain, SQLBuild aligns the replay to the coarsest participating grain (the model's own grain and its cursor-input grains). This happens on every run that resolves cursor bounds from upstream models, not only when something changes. It is independent of the `replay_on_change` cascade behavior described below.
335+
When a downstream microbatch model reads from an upstream model with a coarser time grain, SQLBuild aligns the replay to the coarsest participating grain (the model's own grain and its cursor-input grains). This happens on every run that resolves cursor bounds from upstream models, not only when something changes. It is independent of the `replay_on_change` policy described below.
328336

329337
Alignment does two things: it floors the replay window edges to the coarsest grain, and it coarsens the batch size to that grain. For example, an hourly model downstream of a daily model processes in day-sized batches, so each batch lines up with a unit of upstream data that actually advances instead of producing empty or boundary-straddling windows:
330338

@@ -360,7 +368,7 @@ MODEL (
360368

361369
## Replay on change
362370

363-
When a model's version identity changes (query, config, upstream cascade, or any other change reason), `replay_on_change` is the explicit, per-model policy for how much data to reprocess. Reprocessing is a policy you set, not an automatic forced rebuild, so a definition change does not silently trigger a full rebuild of large downstream tables. You choose the cost per model:
371+
When a model's own query changes, a function it calls directly changes, or its own header, contract, or declared columns change, `replay_on_change` is the explicit, per-model policy for how much data to reprocess. Reprocessing is a policy you set, not an automatic forced rebuild, so a definition change does not silently trigger a full rebuild of large tables. Upstream changes never replay a downstream model. You choose the cost per model:
364372

365373
| Value | Effect |
366374
|-------|--------|
@@ -381,7 +389,7 @@ MODEL (
381389
);
382390
```
383391

384-
See [Cascade propagation](planning/cascade-propagation.md) for how replay policies propagate through the DAG and how downstream models can override inherited replay behavior.
392+
See [Replay decisions](planning/replay-decisions.md) for what counts as a model's own change and how materialization types respond to upstream changes.
385393

386394
### on_schema_change
387395

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,8 @@ Use `schema.name` when the old relation is in a different schema of the same dat
4040

4141
On the next build, SQLBuild copies the old relation's data to the new name through a staging table (see [How a move runs](#how-a-move-runs)), records the move, and builds the model incrementally from the migrated state. There is no first-run rebuild, and no `replay_on_change` replay caused by the rename. After the model builds, the old relation is archived and a compatibility view takes its name (see [Old names](#old-names)).
4242

43+
A rename is usually made together with other edits, for example a changed filter or a renamed upstream reference. If the renamed model's query also differs from the old model's, the plan still recognises the rename: the model is listed as `Renamed` with its query diff, never as a first run. The query change follows the model's own `replay_on_change`, exactly as it would without the rename (forward-only by default). Models downstream of the renamed model are not replayed because of the rename; see [Replay decisions](../planning/replay-decisions.md).
44+
4345
After the move is recorded, `migrate_from` has no further effect, and SQLBuild tells you it can be removed. Keep it until every target that needs the move has built.
4446

4547
## Decisions
@@ -66,7 +68,7 @@ Table and view models only ever see `migrate`, `done`, and `origin missing`: not
6668

6769
If a selected model has never been built, SQLBuild compares it with models that were removed from the project but whose relations still exist. When exactly one removed model has equivalent logic, it is migrated automatically. Equivalent means the same query and configuration, ignoring the model's own name, CTE and table alias names, comments, formatting, and storage-only settings. Renamed upstream models are matched first, so a renamed chain of models is migrated together. Removed models whose data was already moved on by a recorded migration are not candidates, so a model renamed several times matches its latest table.
6870

69-
Renamed tables and views are matched the same way. They are rebuilt under the new name, since they hold no history, but they keep their identity for change detection, so incremental models downstream of a renamed view are not replayed or rebuilt. These renames are recorded as `renamed` events.
71+
Renamed tables and views are matched the same way. They are rebuilt under the new name, since they hold no history, but they keep their identity for change detection. These renames are recorded as `renamed` events.
7072

7173
Automatic discovery never guesses. If a match is ambiguous, SQLBuild warns (`M107`) and builds from scratch; declare `migrate_from` to choose. An explicit `migrate_from` always wins. Automatic discovery covers renames within the project's schemas; use `migrate_from` to move a model to another schema. Relations last built before this feature carry no stored fingerprint and aren't matched automatically.
7274

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning.md‎

Lines changed: 12 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Online: https://sqlbuild.com/docs/concepts/planning/
1111
- Reading a plan
1212
- What SQLBuild compares
1313
- Change reasons
14-
- Cascades and incremental actions
14+
- Replay and incremental actions
1515
- Selection and coherence
1616
- Recorded state
1717
- Related topics
@@ -47,8 +47,8 @@ A plan answers four questions:
4747
3. Why does each selected model need work?
4848
4. What materialization or incremental action will the build take?
4949

50-
Plans may include query diffs, schema changes, backfill ranges, source freshness signals, cascade
51-
reasons, and warnings about stale dependencies. Planning does not modify model relations.
50+
Plans may include query diffs, schema changes, backfill ranges, source freshness signals, change
51+
causes, and warnings about stale dependencies. Planning does not modify model relations.
5252

5353
## What SQLBuild compares
5454

@@ -94,26 +94,25 @@ The plan assigns reasons to explain detected work:
9494
| Query changed / checksum changed | Compiled SQL or seed content differs from the recorded identity |
9595
| Config changed | Version-relevant configuration differs |
9696
| Schema changed | A relevant warehouse or upstream schema changed |
97-
| Function changed | A referenced function has a different identity |
98-
| Upstream changed | A dependency change propagated to this model |
97+
| Function changed | A function the model calls directly has a different definition |
9998
| Run despite unchanged | A table's explicit policy requires another run despite an unchanged identity |
10099

101100
These reasons explain the plan; they are not a promise that a build will skip entries labelled
102101
current. Builds execute the selected scope.
103102

104-
## Cascades and incremental actions
103+
## Replay and incremental actions
105104

106-
Changes propagate through the DAG in dependency order. The resulting action depends on the
105+
Each model's action comes from its own change. The resulting action depends on the
107106
materialization:
108107

109108
- views are recreated when selected;
110109
- tables rebuild when selected;
111-
- incremental models use their cursor state, backfill policy, and `replay_on_change` setting to
112-
determine the affected range; and
113-
- a changed upstream can alter the action of downstream models even when their own SQL is unchanged.
110+
- incremental models use their cursor state and their own `replay_on_change` setting to determine
111+
the affected range; and
112+
- a changed, new, renamed, or rebuilt upstream never replays a downstream incremental model, which
113+
continues forward from its cursor.
114114

115-
See [Cascade propagation](planning/cascade-propagation.md) for the detailed materialization
116-
rules.
115+
See [Replay decisions](planning/replay-decisions.md) for the detailed rules.
117116

118117
## Selection and coherence
119118

@@ -136,6 +135,6 @@ The planner reads the latest applicable facts and appends new facts after succes
136135

137136
## Related topics
138137

139-
- [Cascade propagation](planning/cascade-propagation.md)
138+
- [Replay decisions](planning/replay-decisions.md)
140139
- [Source freshness](planning/source-freshness.md)
141140
- [Selection and staleness](planning/selection-and-staleness.md)

0 commit comments

Comments
 (0)