Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ E2E_DUCKDB_BUILD_CORE_PATHS := \
tests/e2e/src/sqlbuild/cli/commands/main/build/test_validation_failures.py

E2E_DUCKDB_BUILD_INCREMENTAL_PATHS := \
tests/e2e/src/sqlbuild/cli/commands/main/build/own_change_replay \
tests/e2e/src/sqlbuild/cli/commands/main/build/test_append_cursor_build.py \
tests/e2e/src/sqlbuild/cli/commands/main/build/test_capped_microbatch_build.py \
tests/e2e/src/sqlbuild/cli/commands/main/build/test_concurrent_microbatch_build.py \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in
- [Functions](concepts/functions.md) (`concepts/functions`) - User-defined functions and table functions managed as part of your project.
- [Incremental Models](concepts/incremental.md) (`concepts/incremental`) - Cursor-based incremental strategies, microbatch execution, and backfill policies.
- [Planning and change detection](concepts/planning.md) (`concepts/planning`) - How SQLBuild explains build work, detects changes, and chooses safe model actions.
- [Cascade propagation](concepts/planning/cascade-propagation.md) (`concepts/planning/cascade-propagation`) - How a change signal propagates downstream through the DAG, and how each materialization type responds.
- [Replay decisions](concepts/planning/replay-decisions.md) (`concepts/planning/replay-decisions`) - How much history each model reprocesses: decided by its own change and its own replay_on_change, never inherited from upstream.
- [Source freshness](concepts/planning/source-freshness.md) (`concepts/planning/source-freshness`) - Observe external source changes and propagate them through planning.
- [Selection and staleness](concepts/planning/selection-and-staleness.md) (`concepts/planning/selection-and-staleness`) - How selection interacts with change detection, and the stale warnings that stop silent partial rebuilds.
- [Snapshots (SCD Type 2)](concepts/snapshots.md) (`concepts/snapshots`) - Preserve row history over time using SCD Type 2 semantics with timestamp or check-based change detection.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ Seeds (1)
✓ Plan complete 13 selected
```

When query or schema changes are detected, the plan shows the affected models with backfill actions and cascade information.
When query, function, or schema changes are detected, the plan lists each affected model under its own change with its backfill action. A model that calls a changed function directly appears under `Function changed` with `cause: function <name> changed`; models further downstream continue forward. See [Replay decisions](../concepts/planning/replay-decisions.md).

In `sqb plan --json`, each model's `reason` is its own change. A direct caller of a changed function has `"reason": "function_changed"` and a `changed_functions` list, and a renamed model whose query also changed has `"reason": "renamed"` and `"query_changed": true`.

Warnings and maintenance notices, such as `migrate_from can be removed`, are listed in a `Warnings` section before the final `Plan complete` line. `sqb plan --json` output is unchanged.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,15 @@ Online: https://sqlbuild.com/docs/concepts/functions/
- Python UDF options
- Adapter support

Functions are SQL or Python definitions under `functions/` that SQLBuild compiles and deploys to the warehouse alongside your models. They participate in the DAG - if a function definition changes, every model that uses it is rebuilt.
Functions are SQL or Python definitions under `functions/` that SQLBuild compiles and deploys to the warehouse alongside your models. They participate in the DAG: if a function definition changes, SQLBuild redeploys it, and every model that calls it directly treats the change as a change to its own query.

## Scalar UDFs

Scalar UDFs return a single value per row. They can be written in SQL or Python.

### SQL UDFs

Place SQL function files under `functions/sql/`. Each file has a `FUNCTION()` header declaring arguments and return type, followed by a SQL expression body:
Place SQL function files under `functions/sql/`. Each file has a `FUNCTION()` header declaring arguments and return type, followed by a SQL expression body. The header accepts `arguments`, `returns`, `database`, `schema`, and `tags`; any other key fails compilation with `D002`, naming the key and its file and line:

```sql
-- functions/sql/udf__is_completed_order.sql
Expand Down Expand Up @@ -144,7 +144,9 @@ function folder or to that folder and its children.

## Change propagation

Functions participate in fingerprint-based change detection. Their identity includes dependencies and declared return contracts in addition to the function body and runtime metadata. If a function changes, SQLBuild redeploys it and marks dependent models as changed.
Functions participate in fingerprint-based change detection. Their identity includes dependencies and declared return contracts in addition to the function body and runtime metadata. If a function changes, SQLBuild redeploys it.

A model that calls a changed function directly plans as `Function changed`, with `cause: function <name> changed`. The function change counts as a change to that model's own query, so the model's own `replay_on_change` decides how much history it reprocesses (forward-only by default). Models further downstream are not affected by the function change and continue forward. Functions have no replay setting of their own; to reprocess history when a function changes, set `replay_on_change` on the models that call it. See [Replay decisions](planning/replay-decisions.md).

## Project layout

Expand All @@ -169,6 +171,8 @@ The `@udf` decorator accepts these keyword arguments:
| `entry_point` | No | Function name to use as the entry point (defaults to the decorated function name) |
| `packages` | No | List of Python packages required at runtime |

`database`, `schema`, and `tags` are also accepted. Any other keyword fails compilation with `D002`, naming the keyword and its file and line.

## Adapter support

All four supported adapters implement SQL UDFs, Python UDFs, and table functions:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,15 @@ MODEL (
);
```

The override is evaluated per model, so one selection can contain incrementally executed opt-outs and full-refreshed models. It controls execution mode rather than acting as a safety rejection: an opted-out model does not abort the rest of the build.
The override is evaluated per model, so one selection can contain incrementally executed opt-outs and full-refreshed models. With `--full-refresh`, an opted-out model simply runs incrementally; it does not abort the rest of the build.

`full_refresh false` also protects the model from every other full rebuild. Only the model's own explicit configuration can fully rebuild it: its own first run, or its own query change with `replay_on_change full`. If anything else would plan a full rebuild, for example a changed function that the model calls while it sets `replay_on_change full`, the plan fails with `S203`, naming the model and the cause, instead of silently replaying its history:

```
error[S203]: model 'order_history' sets full_refresh false, but the plan would fully rebuild it (function normalize_status changed with replay_on_change full)
```

Set `replay_on_change` to `forward` or `bounded-<duration>` to keep the existing history, or remove `full_refresh false` to allow the rebuild. Upstream changes never cause a full rebuild of a downstream model; see [Replay decisions](planning/replay-decisions.md).

This does not skip initial loading. If the destination relation does not exist, an incremental or microbatch model still builds the history required by its cursor policy. Cloning an existing destination before the build can provide a current watermark and avoid a first-run historical replay.

Expand Down Expand Up @@ -173,7 +181,7 @@ WHERE ordered_at >= __cursor_start()
AND ordered_at < __cursor_end()
```

`__cursor_start()` is the effective inclusive start and `__cursor_end()` is the effective exclusive end after cursor floors, lookback, replay policy, and command-line overrides have been applied. The intrinsics accept no arguments and are only valid in built-in cursor incremental model query SQL. They are rejected in functions, hooks, audits, SQL tests and scenarios, source expressions, non-incremental or cursorless models, custom materializations, and non-microbatch full refreshes.
`__cursor_start()` is the effective inclusive start and `__cursor_end()` is the effective exclusive end after cursor floors, lookback, replay policy, and command-line overrides have been applied. The intrinsics accept no arguments and are only valid in built-in cursor incremental model query SQL. They are rejected in functions, hooks, audits, SQL tests and scenarios, source expressions, non-incremental or cursorless models, custom materializations, and non-microbatch full refreshes. A non-microbatch model given a FULL backfill by `--full-refresh` fails with a message naming that cause; pass an explicit interval with `--start-cursor-ts` and `--end-cursor-ts` (or the `-int` forms) to rebuild it.

In microbatch mode, the intrinsics resolve to each batch's concrete bounds. A microbatch full refresh discovers its range from current inputs while ignoring the old destination watermark.

Expand Down Expand Up @@ -324,7 +332,7 @@ The legacy scalar `max_microbatches` model field remains supported as a fail/war

### Mixed-grain chains

When a downstream microbatch model reads from an upstream model with a coarser time grain, SQLBuild aligns the replay to the coarsest participating grain (the model's own grain and its cursor-input grains). This happens on every run that resolves cursor bounds from upstream models, not only when something changes. It is independent of the `replay_on_change` cascade behavior described below.
When a downstream microbatch model reads from an upstream model with a coarser time grain, SQLBuild aligns the replay to the coarsest participating grain (the model's own grain and its cursor-input grains). This happens on every run that resolves cursor bounds from upstream models, not only when something changes. It is independent of the `replay_on_change` policy described below.

Alignment does two things: it floors the replay window edges to the coarsest grain, and it coarsens the batch size to that grain. For example, an hourly model downstream of a daily model processes in day-sized batches, so each batch lines up with a unit of upstream data that actually advances instead of producing empty or boundary-straddling windows:

Expand Down Expand Up @@ -360,7 +368,7 @@ MODEL (

## Replay on change

When a model's version identity changes (query, config, upstream cascade, or any other change reason), `replay_on_change` is the explicit, per-model policy for how much data to reprocess. Reprocessing is a policy you set, not an automatic forced rebuild, so a definition change does not silently trigger a full rebuild of large downstream tables. You choose the cost per model:
When a model's own query changes, a function it calls directly changes, or its own header, contract, or declared columns change, `replay_on_change` is the explicit, per-model policy for how much data to reprocess. Reprocessing is a policy you set, not an automatic forced rebuild, so a definition change does not silently trigger a full rebuild of large tables. Upstream changes never replay a downstream model. You choose the cost per model:

| Value | Effect |
|-------|--------|
Expand All @@ -381,7 +389,7 @@ MODEL (
);
```

See [Cascade propagation](planning/cascade-propagation.md) for how replay policies propagate through the DAG and how downstream models can override inherited replay behavior.
See [Replay decisions](planning/replay-decisions.md) for what counts as a model's own change and how materialization types respond to upstream changes.

### on_schema_change

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ Use `schema.name` when the old relation is in a different schema of the same dat

On the next build, SQLBuild copies the old relation's data to the new name through a staging table (see [How a move runs](#how-a-move-runs)), records the move, and builds the model incrementally from the migrated state. There is no first-run rebuild, and no `replay_on_change` replay caused by the rename. After the model builds, the old relation is archived and a compatibility view takes its name (see [Old names](#old-names)).

A rename is usually made together with other edits, for example a changed filter or a renamed upstream reference. If the renamed model's query also differs from the old model's, the plan still recognises the rename: the model is listed as `Renamed` with its query diff, never as a first run. The query change follows the model's own `replay_on_change`, exactly as it would without the rename (forward-only by default). Models downstream of the renamed model are not replayed because of the rename; see [Replay decisions](../planning/replay-decisions.md).

After the move is recorded, `migrate_from` has no further effect, and SQLBuild tells you it can be removed. Keep it until every target that needs the move has built.

## Decisions
Expand All @@ -66,7 +68,7 @@ Table and view models only ever see `migrate`, `done`, and `origin missing`: not

If a selected model has never been built, SQLBuild compares it with models that were removed from the project but whose relations still exist. When exactly one removed model has equivalent logic, it is migrated automatically. Equivalent means the same query and configuration, ignoring the model's own name, CTE and table alias names, comments, formatting, and storage-only settings. Renamed upstream models are matched first, so a renamed chain of models is migrated together. Removed models whose data was already moved on by a recorded migration are not candidates, so a model renamed several times matches its latest table.

Renamed tables and views are matched the same way. They are rebuilt under the new name, since they hold no history, but they keep their identity for change detection, so incremental models downstream of a renamed view are not replayed or rebuilt. These renames are recorded as `renamed` events.
Renamed tables and views are matched the same way. They are rebuilt under the new name, since they hold no history, but they keep their identity for change detection. These renames are recorded as `renamed` events.

Automatic discovery never guesses. If a match is ambiguous, SQLBuild warns (`M107`) and builds from scratch; declare `migrate_from` to choose. An explicit `migrate_from` always wins. Automatic discovery covers renames within the project's schemas; use `migrate_from` to move a model to another schema. Relations last built before this feature carry no stored fingerprint and aren't matched automatically.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Online: https://sqlbuild.com/docs/concepts/planning/
- Reading a plan
- What SQLBuild compares
- Change reasons
- Cascades and incremental actions
- Replay and incremental actions
- Selection and coherence
- Recorded state
- Related topics
Expand Down Expand Up @@ -47,8 +47,8 @@ A plan answers four questions:
3. Why does each selected model need work?
4. What materialization or incremental action will the build take?

Plans may include query diffs, schema changes, backfill ranges, source freshness signals, cascade
reasons, and warnings about stale dependencies. Planning does not modify model relations.
Plans may include query diffs, schema changes, backfill ranges, source freshness signals, change
causes, and warnings about stale dependencies. Planning does not modify model relations.

## What SQLBuild compares

Expand Down Expand Up @@ -94,26 +94,25 @@ The plan assigns reasons to explain detected work:
| Query changed / checksum changed | Compiled SQL or seed content differs from the recorded identity |
| Config changed | Version-relevant configuration differs |
| Schema changed | A relevant warehouse or upstream schema changed |
| Function changed | A referenced function has a different identity |
| Upstream changed | A dependency change propagated to this model |
| Function changed | A function the model calls directly has a different definition |
| Run despite unchanged | A table's explicit policy requires another run despite an unchanged identity |

These reasons explain the plan; they are not a promise that a build will skip entries labelled
current. Builds execute the selected scope.

## Cascades and incremental actions
## Replay and incremental actions

Changes propagate through the DAG in dependency order. The resulting action depends on the
Each model's action comes from its own change. The resulting action depends on the
materialization:

- views are recreated when selected;
- tables rebuild when selected;
- incremental models use their cursor state, backfill policy, and `replay_on_change` setting to
determine the affected range; and
- a changed upstream can alter the action of downstream models even when their own SQL is unchanged.
- incremental models use their cursor state and their own `replay_on_change` setting to determine
the affected range; and
- a changed, new, renamed, or rebuilt upstream never replays a downstream incremental model, which
continues forward from its cursor.

See [Cascade propagation](planning/cascade-propagation.md) for the detailed materialization
rules.
See [Replay decisions](planning/replay-decisions.md) for the detailed rules.

## Selection and coherence

Expand All @@ -136,6 +135,6 @@ The planner reads the latest applicable facts and appends new facts after succes

## Related topics

- [Cascade propagation](planning/cascade-propagation.md)
- [Replay decisions](planning/replay-decisions.md)
- [Source freshness](planning/source-freshness.md)
- [Selection and staleness](planning/selection-and-staleness.md)
Loading
Loading