You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e382490
Browse filesBrowse the repository at this point in the historyBrowse files
fix(planner)!: decide each model replay from its own change only (#895)
* fix(planner)!: decide each model's replay from its own change only
Remove upstream backfill cascades. A model's backfill now comes only from
its own change and its own replay_on_change, so a first-run, renamed or
replayed upstream no longer forces a full or bounded replay of downstream
incremental models.
A changed function counts as an effective query change of the models that
call it directly. Those models plan as function_changed with the cause
named and apply their own replay_on_change; models further downstream
continue forward. The function-level replay_on_change header is removed.
A model that sets full_refresh false now refuses (S203) any planned full
rebuild other than its own first run or its own query change with
replay_on_change full. A FULL backfill on a non-microbatch model with
cursor intrinsics reports the FULL backfill and its cause instead of an
unresolved-bounds error.
Plan text drops the Upstream changed group and adds a Function changed
group; plan JSON drops the cascade field and upstream_changed reason and
adds changed_functions to direct callers.
BREAKING CHANGE: backfills no longer cascade from upstream models; each
model replays only for its own change per its own replay_on_change. The
function-level replay_on_change header is removed. Plan JSON drops the
cascade field and the upstream_changed reason, and adds changed_functions
on direct callers of changed functions and query_changed on renamed models
whose query changed. New planner error S203 refuses full rebuilds of
full_refresh false models not requested by their own configuration.
* fix(migrations): plan renames with changed SQL as renamed query changes
A recognised migrate_from rename of a table or view whose handed-over
definition differs from its new query now keeps the handed-over identity
and plans as renamed with a query change, instead of a first run. Its
backfill comes from the model's own replay_on_change (forward-only by
default). Once the renamed model has built after the recorded rename, its
own fingerprint is used, so later plans no longer report the query change.
* test(e2e): cover own-change replay for new, renamed and function-changed upstreams
* docs: describe per-model replay decisions, function callers and full_refresh protection
* fix(functions): reject unknown FUNCTION header and @udf keys
SQL FUNCTION() headers and Python @udf(...) decorators now fail discovery with D002 when they use a key that functions do not support, naming the key and its file and line, the same way HOOK() and AUDIT() headers reject unsupported keys. Previously such keys were silently ignored.
* fix(executor): report called-function changes to custom materializations
Custom materializations now receive query_changed for function_changed models as well as query_changed ones, since a changed directly called function is the model's own query change.
* fix(planner): check full_refresh protection only for models that execute
S203 is no longer evaluated for models the plan skips because a source freshness check or an external upstream blocked them.
* fix(planner): ignore rename-rewritten references and inferred upstream columns for replay
A downstream model whose only difference is __ref and cursor_inputs names
rewritten for an upstream renamed with migrate_from in this plan is no
longer a query or config change: its migration fingerprint with the
renamed names mapped back to their origins must equal its stored one.
Views are still recreated and incremental models continue forward; a real
edit alongside the rewrite is a normal query change.
Schema findings inferred from a model's unchanged SQL, such as a column an
upstream added through SELECT *, no longer trigger replay_on_change; they
evolve forward through on_schema_change. Changes to the model's own
declared columns still follow its replay_on_change.
* fix(planner): match rename-rewritten references by query hash and recorded renames
A downstream model is no longer a query change when its compiled SQL, with
__ref names of renamed models substituted back to their origin names,
hashes to its stored definition hash. Substitution touches only reference
calls outside comments and string literals, so it works for any SQL,
including models the migration fingerprint parser cannot normalize. The
rename map combines this plan's handovers with renames recorded in
migration history after the downstream's last build, so renames built in
an earlier run or whose migrate_from was removed are recognized too.
Migration events are read only when a selected model's query hash changed
and the migration table was already listed by the warehouse snapshot.
A column removed from the warehouse is attributed to the model's declared
columns only when no inferred columns exist, so an upstream dropping a
column a partial columns block does not declare evolves forward instead of
following replay_on_change.
* fix(planner): replay schema differences only when the model's own definition changed
A schema finding follows replay_on_change only when the model's own SQL,
header config, contract, or called functions changed in this plan, with
rename-rewritten references not counting as a change. When the model is
unchanged, every schema difference comes from upstream, whether added,
removed or retyped, declared or inferred; it evolves through
on_schema_change and contract enforcement as before, so an enforced
contract whose declared column disappeared upstream still fails the build
instead of replaying. The interim removed-column source heuristic is
reverted.
* fix(planner): record declared columns so their own edits replay schema changes
Model fingerprint metadata now carries a declared_columns_hash key covering
the model's columns block or schema YAML columns (name, type, nullable),
dynamic column families, type enforcement and contract setting. The key is
additive, sits outside version identity, and is ignored by config-change
comparison. A schema finding follows replay_on_change when this hash
differs from the recorded one, alongside SQL, config, contract and
function changes. Rows written before the key existed have no hash, so
their declared columns count as unchanged and the model continues forward
once.
* fix(planner): keep migration fingerprint work at one computation per model
The migration fingerprint cache keyed entries on the raw fingerprint metadata JSON, so the declared_columns_hash key made build-time entries miss the value computed during planning. The cache now strips that key before keying, as the key never contributes to the fingerprint.
-[Planning and change detection](concepts/planning.md) (`concepts/planning`) - How SQLBuild explains build work, detects changes, and chooses safe model actions.
58
-
-[Cascade propagation](concepts/planning/cascade-propagation.md) (`concepts/planning/cascade-propagation`) - How a change signal propagates downstream through the DAG, and how each materialization type responds.
58
+
-[Replay decisions](concepts/planning/replay-decisions.md) (`concepts/planning/replay-decisions`) - How much history each model reprocesses: decided by its own change and its own replay_on_change, never inherited from upstream.
59
59
-[Source freshness](concepts/planning/source-freshness.md) (`concepts/planning/source-freshness`) - Observe external source changes and propagate them through planning.
60
60
-[Selection and staleness](concepts/planning/selection-and-staleness.md) (`concepts/planning/selection-and-staleness`) - How selection interacts with change detection, and the stale warnings that stop silent partial rebuilds.
61
61
-[Snapshots (SCD Type 2)](concepts/snapshots.md) (`concepts/snapshots`) - Preserve row history over time using SCD Type 2 semantics with timestamp or check-based change detection.
Copy file name to clipboardExpand all lines: src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -85,7 +85,9 @@ Seeds (1)
85
85
✓ Plan complete 13 selected
86
86
```
87
87
88
-
When query or schema changes are detected, the plan shows the affected models with backfill actions and cascade information.
88
+
When query, function, or schema changes are detected, the plan lists each affected model under its own change with its backfill action. A model that calls a changed function directly appears under `Function changed` with `cause: function <name> changed`; models further downstream continue forward. See [Replay decisions](../concepts/planning/replay-decisions.md).
89
+
90
+
In `sqb plan --json`, each model's `reason` is its own change. A direct caller of a changed function has `"reason": "function_changed"` and a `changed_functions` list, and a renamed model whose query also changed has `"reason": "renamed"` and `"query_changed": true`.
89
91
90
92
Warnings and maintenance notices, such as `migrate_from can be removed`, are listed in a `Warnings` section before the final `Plan complete` line. `sqb plan --json` output is unchanged.
Functions are SQL or Python definitions under `functions/` that SQLBuild compiles and deploys to the warehouse alongside your models. They participate in the DAG - if a function definition changes, every model that uses it is rebuilt.
19
+
Functions are SQL or Python definitions under `functions/` that SQLBuild compiles and deploys to the warehouse alongside your models. They participate in the DAG: if a function definition changes, SQLBuild redeploys it, and every model that calls it directly treats the change as a change to its own query.
20
20
21
21
## Scalar UDFs
22
22
23
23
Scalar UDFs return a single value per row. They can be written in SQL or Python.
24
24
25
25
### SQL UDFs
26
26
27
-
Place SQL function files under `functions/sql/`. Each file has a `FUNCTION()` header declaring arguments and return type, followed by a SQL expression body:
27
+
Place SQL function files under `functions/sql/`. Each file has a `FUNCTION()` header declaring arguments and return type, followed by a SQL expression body. The header accepts `arguments`, `returns`, `database`, `schema`, and `tags`; any other key fails compilation with `D002`, naming the key and its file and line:
28
28
29
29
```sql
30
30
-- functions/sql/udf__is_completed_order.sql
@@ -144,7 +144,9 @@ function folder or to that folder and its children.
144
144
145
145
## Change propagation
146
146
147
-
Functions participate in fingerprint-based change detection. Their identity includes dependencies and declared return contracts in addition to the function body and runtime metadata. If a function changes, SQLBuild redeploys it and marks dependent models as changed.
147
+
Functions participate in fingerprint-based change detection. Their identity includes dependencies and declared return contracts in addition to the function body and runtime metadata. If a function changes, SQLBuild redeploys it.
148
+
149
+
A model that calls a changed function directly plans as `Function changed`, with `cause: function <name> changed`. The function change counts as a change to that model's own query, so the model's own `replay_on_change` decides how much history it reprocesses (forward-only by default). Models further downstream are not affected by the function change and continue forward. Functions have no replay setting of their own; to reprocess history when a function changes, set `replay_on_change` on the models that call it. See [Replay decisions](planning/replay-decisions.md).
148
150
149
151
## Project layout
150
152
@@ -169,6 +171,8 @@ The `@udf` decorator accepts these keyword arguments:
169
171
|`entry_point`| No | Function name to use as the entry point (defaults to the decorated function name) |
170
172
|`packages`| No | List of Python packages required at runtime |
171
173
174
+
`database`, `schema`, and `tags` are also accepted. Any other keyword fails compilation with `D002`, naming the keyword and its file and line.
175
+
172
176
## Adapter support
173
177
174
178
All four supported adapters implement SQL UDFs, Python UDFs, and table functions:
Copy file name to clipboardExpand all lines: src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/incremental.md
+13-5Lines changed: 13 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -114,7 +114,15 @@ MODEL (
114
114
);
115
115
```
116
116
117
-
The override is evaluated per model, so one selection can contain incrementally executed opt-outs and full-refreshed models. It controls execution mode rather than acting as a safety rejection: an opted-out model does not abort the rest of the build.
117
+
The override is evaluated per model, so one selection can contain incrementally executed opt-outs and full-refreshed models. With `--full-refresh`, an opted-out model simply runs incrementally; it does not abort the rest of the build.
118
+
119
+
`full_refresh false` also protects the model from every other full rebuild. Only the model's own explicit configuration can fully rebuild it: its own first run, or its own query change with `replay_on_change full`. If anything else would plan a full rebuild, for example a changed function that the model calls while it sets `replay_on_change full`, the plan fails with `S203`, naming the model and the cause, instead of silently replaying its history:
120
+
121
+
```
122
+
error[S203]: model 'order_history' sets full_refresh false, but the plan would fully rebuild it (function normalize_status changed with replay_on_change full)
123
+
```
124
+
125
+
Set `replay_on_change` to `forward` or `bounded-<duration>` to keep the existing history, or remove `full_refresh false` to allow the rebuild. Upstream changes never cause a full rebuild of a downstream model; see [Replay decisions](planning/replay-decisions.md).
118
126
119
127
This does not skip initial loading. If the destination relation does not exist, an incremental or microbatch model still builds the history required by its cursor policy. Cloning an existing destination before the build can provide a current watermark and avoid a first-run historical replay.
120
128
@@ -173,7 +181,7 @@ WHERE ordered_at >= __cursor_start()
173
181
AND ordered_at < __cursor_end()
174
182
```
175
183
176
-
`__cursor_start()` is the effective inclusive start and `__cursor_end()` is the effective exclusive end after cursor floors, lookback, replay policy, and command-line overrides have been applied. The intrinsics accept no arguments and are only valid in built-in cursor incremental model query SQL. They are rejected in functions, hooks, audits, SQL tests and scenarios, source expressions, non-incremental or cursorless models, custom materializations, and non-microbatch full refreshes.
184
+
`__cursor_start()` is the effective inclusive start and `__cursor_end()` is the effective exclusive end after cursor floors, lookback, replay policy, and command-line overrides have been applied. The intrinsics accept no arguments and are only valid in built-in cursor incremental model query SQL. They are rejected in functions, hooks, audits, SQL tests and scenarios, source expressions, non-incremental or cursorless models, custom materializations, and non-microbatch full refreshes. A non-microbatch model given a FULL backfill by `--full-refresh` fails with a message naming that cause; pass an explicit interval with `--start-cursor-ts` and `--end-cursor-ts` (or the `-int` forms) to rebuild it.
177
185
178
186
In microbatch mode, the intrinsics resolve to each batch's concrete bounds. A microbatch full refresh discovers its range from current inputs while ignoring the old destination watermark.
179
187
@@ -324,7 +332,7 @@ The legacy scalar `max_microbatches` model field remains supported as a fail/war
324
332
325
333
### Mixed-grain chains
326
334
327
-
When a downstream microbatch model reads from an upstream model with a coarser time grain, SQLBuild aligns the replay to the coarsest participating grain (the model's own grain and its cursor-input grains). This happens on every run that resolves cursor bounds from upstream models, not only when something changes. It is independent of the `replay_on_change`cascade behavior described below.
335
+
When a downstream microbatch model reads from an upstream model with a coarser time grain, SQLBuild aligns the replay to the coarsest participating grain (the model's own grain and its cursor-input grains). This happens on every run that resolves cursor bounds from upstream models, not only when something changes. It is independent of the `replay_on_change`policy described below.
328
336
329
337
Alignment does two things: it floors the replay window edges to the coarsest grain, and it coarsens the batch size to that grain. For example, an hourly model downstream of a daily model processes in day-sized batches, so each batch lines up with a unit of upstream data that actually advances instead of producing empty or boundary-straddling windows:
330
338
@@ -360,7 +368,7 @@ MODEL (
360
368
361
369
## Replay on change
362
370
363
-
When a model's version identity changes (query, config, upstream cascade, or any other change reason), `replay_on_change` is the explicit, per-model policy for how much data to reprocess. Reprocessing is a policy you set, not an automatic forced rebuild, so a definition change does not silently trigger a full rebuild of large downstream tables. You choose the cost per model:
371
+
When a model's own query changes, a function it calls directly changes, or its own header, contract, or declared columns change, `replay_on_change` is the explicit, per-model policy for how much data to reprocess. Reprocessing is a policy you set, not an automatic forced rebuild, so a definition change does not silently trigger a full rebuild of large tables. Upstream changes never replay a downstream model. You choose the cost per model:
364
372
365
373
| Value | Effect |
366
374
|-------|--------|
@@ -381,7 +389,7 @@ MODEL (
381
389
);
382
390
```
383
391
384
-
See [Cascade propagation](planning/cascade-propagation.md) for how replay policies propagate through the DAG and how downstream models can override inherited replay behavior.
392
+
See [Replay decisions](planning/replay-decisions.md) for what counts as a model's own change and how materialization types respond to upstream changes.
Copy file name to clipboardExpand all lines: src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -40,6 +40,8 @@ Use `schema.name` when the old relation is in a different schema of the same dat
40
40
41
41
On the next build, SQLBuild copies the old relation's data to the new name through a staging table (see [How a move runs](#how-a-move-runs)), records the move, and builds the model incrementally from the migrated state. There is no first-run rebuild, and no `replay_on_change` replay caused by the rename. After the model builds, the old relation is archived and a compatibility view takes its name (see [Old names](#old-names)).
42
42
43
+
A rename is usually made together with other edits, for example a changed filter or a renamed upstream reference. If the renamed model's query also differs from the old model's, the plan still recognises the rename: the model is listed as `Renamed` with its query diff, never as a first run. The query change follows the model's own `replay_on_change`, exactly as it would without the rename (forward-only by default). Models downstream of the renamed model are not replayed because of the rename; see [Replay decisions](../planning/replay-decisions.md).
44
+
43
45
After the move is recorded, `migrate_from` has no further effect, and SQLBuild tells you it can be removed. Keep it until every target that needs the move has built.
44
46
45
47
## Decisions
@@ -66,7 +68,7 @@ Table and view models only ever see `migrate`, `done`, and `origin missing`: not
66
68
67
69
If a selected model has never been built, SQLBuild compares it with models that were removed from the project but whose relations still exist. When exactly one removed model has equivalent logic, it is migrated automatically. Equivalent means the same query and configuration, ignoring the model's own name, CTE and table alias names, comments, formatting, and storage-only settings. Renamed upstream models are matched first, so a renamed chain of models is migrated together. Removed models whose data was already moved on by a recorded migration are not candidates, so a model renamed several times matches its latest table.
68
70
69
-
Renamed tables and views are matched the same way. They are rebuilt under the new name, since they hold no history, but they keep their identity for change detection, so incremental models downstream of a renamed view are not replayed or rebuilt. These renames are recorded as `renamed` events.
71
+
Renamed tables and views are matched the same way. They are rebuilt under the new name, since they hold no history, but they keep their identity for change detection. These renames are recorded as `renamed` events.
70
72
71
73
Automatic discovery never guesses. If a match is ambiguous, SQLBuild warns (`M107`) and builds from scratch; declare `migrate_from` to choose. An explicit `migrate_from` always wins. Automatic discovery covers renames within the project's schemas; use `migrate_from` to move a model to another schema. Relations last built before this feature carry no stored fingerprint and aren't matched automatically.
0 commit comments