From a25568c7802cad6f9ca7bc0c262174369843de40 Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Tue, 27 Oct 2026 08:52:47 +0000 Subject: [PATCH 01/10] docs: restructure the semantic compilation page around what users check and fix --- .../docs/concepts/semantic-compilation.md | 340 ++++++++---------- .../docs/concepts/semantic-compilation.mdx | 332 +++++++---------- 2 files changed, 277 insertions(+), 395 deletions(-) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/semantic-compilation.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/semantic-compilation.md index 6dc5b1d9ee..a9519eafde 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/semantic-compilation.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/semantic-compilation.md @@ -8,127 +8,15 @@ Online: https://sqlbuild.com/docs/concepts/semantic-compilation/ ## Contents -- Authoritative column names -- Checks and limits -- Reading diagnostics +- Closed and open inputs +- What gets checked +- What isn't checked +- Reading a diagnostic +- Turning analysis off - Diagnostic codes -`sqb compile` stays offline. It validates SQL references against each input separately, including -models with no relation inputs. A join to an open source does not disable checks on a closed input. -An unqualified name that could belong to an open input cannot be rejected offline. - -## Authoritative column names - -- Enforced model and source contracts, seeds, and declared table-function outputs are closed shapes. -- Expression sources have the column names inferred from their SQL expression. -- An explicit model projection defines its output names even over an open input. Types propagate - through dependencies where they can be inferred; an unknown type does not make the names unknown. -- A star over closed inputs expands; a star over an open input remains open. -- Declaring some columns on a table source does **not** declare its complete shape. Use - `contract: enforced` when those declarations are complete. - -Adopt a physical source schema with: - -```bash -sqb contract generate --from --select source:raw_orders --write -``` - -`sqb plan` and `sqb build` recheck source readers against inspected physical columns before model -execution. Existing bulk metadata is reused. Selected table sources omitted from that metadata -are inspected in database-grouped batches: one relation-list call and one bulk-column call per -database, using the adapter's normal metadata batching. Missing managed sources and ambiguous -unqualified physical names stay open. - -## Checks and limits - -Binding checks apply to SQL clauses, CTEs, subqueries, and expanded macros. Semantic grouping, -aggregate and window-placement diagnostics are errors. Project UDF arity and known argument types -are checked separately; declared return types contribute to inference. SQL and Python UDFs and -table functions are registered with the native catalogue, along with types named in project -contracts and function signatures. Registration does not replace SQLBuild's signature checks. - -Known output shapes also validate `unique_key`, `cursor`, `cursor_type`, `cursor_inputs`, custom -`config.partition_column`, `row_diff_exclude_columns`, `row_diff_tolerances.by_column`, source -`cursor_column`, audit expressions/relationships/accepted values, and SQL-test mock/expected names. - -The same binding, function-name and type checks run on every SQL body a project owns, not only -model queries: singular and generic audits (including measurement and evidence queries), SQL unit -tests (mock, helper, expected and assertion CTEs), and inline and named SQL hooks. An audit reading -`__ref("orders")` is checked against the closed `orders` output shape, so `WHERE o.shipped_at IS -NULL` on a model without that column fails compile instead of the warehouse. Each diagnostic names -its owner: `SQL test 'orders_are_open': ...`, `audit 'recent_rows' on model 'orders': ...` (with the -attached column when there is one), or `post_hooks[1] SQL hook: ...` at the hook file. Type -mismatches show the explicit conversion to write, such as `CAST('1' AS NUMBER)` in a test's expected -rows or `TO_DATE(@date_column)` in a generic audit. An -audit or SQL-test body the SQL parser cannot read is a P001 syntax error, never a silent pass. -Relations without a closed shape, such as physical tables written in hooks, stay open. - -Expression type checks are enabled for **DuckDB (including MotherDuck), PostgreSQL, Snowflake, -and BigQuery**. Other adapters retain reference and semantic checks without expression type checks. - -Proven bind-time rejections and implicit conversions are both errors. For example, DuckDB rejects -`ordered_at > 5` for a TIMESTAMP column. Snowflake accepts `order_id = customer_code` for an -INTEGER and a VARCHAR column, but the conversion fails on non-numeric values, so SQLBuild reports -it as a blocking B217 error; write the conversion explicitly. Typed string literals such as -`ordered_at > '2026-01-01'` remain valid. Columns from `SELECT *` are type-checked through their -upstream shapes in every query. Unknown input types never cause type errors. Dialect coercions -differ: numeric predicates are accepted by DuckDB but rejected by PostgreSQL, Snowflake, and -BigQuery. -Snowflake's function catalogue is partial and deliberately does not reject unknown function names. -Function signatures and overload coverage remain partial; compilation is not a replacement for -executing SQL unit tests. - -Built-in function spellings are checked for Snowflake and DuckDB. The SQL parser treats alias -spellings such as `STARTS_WITH` and `STARTSWITH` as one built-in, so a spelling the warehouse -rejects would otherwise compile. SQLBuild compares each call's authored name, case-insensitively, -with the dialect's accepted spellings and reports a blocking B101 error naming the dialect's -spelling, for example `Unknown function 'STARTS_WITH' for dialect Snowflake; did you mean -STARTSWITH?`. Snowflake's accepted spellings come from its function reference and include -documented synonyms such as `SUBSTR` and `SUBSTRING`, `DATEDIFF` and `TIMESTAMPDIFF`, and -`TRY_TO_DECIMAL` and `TRY_TO_NUMBER`. DuckDB's come from its function catalogue. A name is -rejected only when the parser maps it to the same built-in as an accepted spelling, so project -UDFs called through `__udf`, schema-qualified and method calls, quoted names, and unknown -functions are not affected. There is no per-model suppression: use the suggested spelling, or -schema-qualify a call to an external UDF that shares an alias name. Other dialects are not checked. - -Truth-value tests (`IS [NOT] TRUE` and `IS [NOT] FALSE`) are rejected for Snowflake and SQL -Server, whose warehouses do not accept them although the SQL parser does. SQLBuild reports a -blocking B103 error at the `IS` keyword with a null-safe rewrite, for example -`'IS NOT TRUE' is not supported for dialect Snowflake; write NOT COALESCE(, FALSE) or -IS DISTINCT FROM TRUE`. SQL Server suggestions compare a BIT value, such as -`COALESCE(, 0) = 1`. DuckDB (including MotherDuck), PostgreSQL, BigQuery and Databricks -accept truth-value tests. - -Derived-table column aliases, UNPIVOT, and Snowflake GROUPING SETS are checked. DuckDB and Snowflake -reject excess relation aliases; fewer aliases than output columns remain valid. Missing UNPIVOT -input columns are errors. - -Project function registration prevents unknown-function errors but does not declare return types to -the validator: some expressions consuming a project function result remain unchecked. Raw external -relations and scope-ordering cases also remain -partially checked. A cast to a type SQLBuild does not know is a B218 error. Types named in project -contracts and function signatures are registered, so declare an extension-defined type there. - -One notice summarizes partial semantic checks. `sqb compile --json` includes per-model reasons in -`semantic_checks_partial`; progress and notices go to stderr. The notice does not increment the -diagnostic warning count. - -Escape hatches are `MODEL (sql_analysis false)`, the corresponding path default, and -`--no-sql-analysis`. Disabling SQL analysis also disables inference and lineage for that model. -`[settings] require_sql_analysis = true` limits the MODEL and path-default opt-outs to SQL the -parser cannot read; see [Requiring SQL analysis](models/configuration.md#requiring-sql-analysis). -Help that names a setting always shows the exact file, section and `key = value` to write. - -## Reading diagnostics - -Diagnostics use one layout: headline, resource and location, source snippet with a span underline, -notes, then code-specific help. SQL-test diagnostics include their authored fixture snippet and -use `sql test: name`, just as model diagnostics use `model: name`. JSON retains the location's -exclusive `end_line`/`end_column` boundary, notes, and help. - -An unknown column identifies its input and authored alias. Suggestions use edit distance with a -bounded typo cutoff and recognize short abbreviations such as `qty`. No suggestion is printed when -none is close. The available-column note puts the closest names first and shows at most ten names. +`sqb compile` checks your SQL before anything runs in the warehouse: unknown columns, wrong types, +misspelled functions, invalid grouping, and more. It works offline, from your project alone. ```text error[B002]: Unknown column 'qty' in stg_orders (as o) @@ -142,91 +30,145 @@ error[B002]: Unknown column 'qty' in stg_orders (as o) = help: did you mean 'quantity'? ``` -Type errors report uppercase types and available operand evidence, for example -`o.ordered_at is TIMESTAMP, 5 is INTEGER`. Comparison help recommends a literal appropriate to the -type family and dialect. Native spans are mapped back through normalization and macro expansion; -simple binary expressions are underlined in full. Diagnostics without usable spans keep the -authored query-start fallback. - -Error recovery treats poisoned output names and their downstream lineage as provisional. A -downstream error traceable only to that root is suppressed and counted in a root-error note; -independent errors on other columns and other inputs remain errors. Recovery does not silence a -whole dependent query or make its unchecked uses valid. Affected models remain in -`semantic_checks_partial`, including on warm/cache-hit runs. - -Located binding and type errors also make the affected projection's type unknown. That uncertainty -propagates through column lineage. Failed dependants are checked again with only those input types -unknown; an independent error on another column or input remains visible. Root notes count unchecked -downstream output uses, including transitive uses. Contract type checks do not emit a secondary -unknown-type warning for those same poisoned outputs. If native evidence cannot locate the failing -projection, recovery retains the diagnostic rather than opening the entire input. - -Temporal cursor metadata accepts DATE, DATETIME, and TIMESTAMP families interchangeably, including -Snowflake timestamp variants. A timestamp cursor on a string, numeric, or Boolean column still -produces B301. - -Snowflake binding honours `QUOTED_IDENTIFIERS_IGNORE_CASE = true` from the effective target's -connection `session_parameters`, for both offline declared schemas and warehouse-backed rebinding. -False or absent settings preserve the dialect's existing case rules. For example: - -```toml -[targets.analytics] -schema = "preserve" - -[targets.analytics.connection] -session_parameters = { QUOTED_IDENTIFIERS_IGNORE_CASE = true } +## Closed and open inputs + +A check can only reject a column when SQLBuild knows the input's complete set of columns. Such an +input is **closed**; otherwise it is **open**. + +| Input | Shape | +|---|---| +| Model or source with `contract: enforced`, seed, declared table-function output | Closed | +| Expression source | Closed: names come from its expression | +| Model with an explicit `SELECT` list | Closed names, even over open inputs | +| `SELECT *` over closed inputs | Closed | +| `SELECT *` over an open input, or a table source without an enforced contract | Open | + +Each input is checked separately, so joining an open source doesn't switch off checks on a closed +one. An unqualified column that could come from an open input can't be rejected offline. + +Declaring some columns on a source doesn't close it. To close a source, adopt its physical schema: + +```bash +sqb contract generate --from --select source:raw_orders --write ``` -The setting participates in the analysis-cache identity. Identifier folding is token-based and -does not alter string literals, comments, or authored SQL. Diagnostic offset mapping is lazy and -uses a bounded cache of token alignments, shared by all diagnostics on the same normalized query. +`sqb plan` and `sqb build` also recheck source readers against the warehouse's real columns before +running models. + +## What gets checked + +Checks run on every SQL body a project owns: models (including CTEs, subqueries and expanded +macros), audits, SQL unit tests and SQL hooks. A problem in an audit or test is reported against it, +for example `audit 'recent_rows' on model 'orders': ...`, and fails compile instead of the +warehouse. + +| Check | Example | Code | +|---|---|---| +| Unknown or ambiguous columns | `o.qty` when the input has `quantity` | B002, B003 | +| Columns named in configuration | `unique_key`, `cursor`, audit columns, SQL-test fixture columns | B300, B302 | +| Types of configured columns | A timestamp `cursor` on a VARCHAR column | B301 | +| Types that the warehouse rejects | `ordered_at > 5` on a TIMESTAMP in DuckDB | B210–B219 | +| Implicit conversions that can fail at runtime | `order_id = customer_code` (INTEGER and VARCHAR) in Snowflake | B211–B218 | +| Function names and spellings | `STARTS_WITH` in Snowflake, which only accepts `STARTSWITH` | B101 | +| Function arity | Too many arguments to a built-in or project UDF | B102 | +| Truth-value tests the warehouse rejects | `x IS NOT TRUE` in Snowflake or SQL Server | B103 | +| Grouping, aggregates and windows | A non-grouped column next to an aggregate | B230–B232 | +| Set operations | A VARCHAR branch combined with a numeric branch | B215, B216 | +| Unknown types | `CAST(x AS MYTYPE)` | B218 | -Snowflake set-operation checks are directional: each branch is compared with the accumulated -result type, following the query's operator precedence (`INTERSECT` binds more tightly than -`UNION` and `EXCEPT`). Proven incompatibilities and implicit conversions whose success depends on -runtime values both produce blocking B215 errors. For example, combining a VARCHAR branch with a -numeric branch requires a value-dependent numeric conversion; cast the branch column explicitly. +Derived-table column aliases, `UNPIVOT` and Snowflake `GROUPING SETS` are checked too. -An unmodelled Snowflake function has an unknown result type rather than inheriting its first -argument's type. SQLBuild retains the output name and lineage, reports partial type coverage, and -uses explicit declared function signatures where available. This prevents a guessed return type -from creating downstream type errors. +Every check except B104 is a blocking error. The fix is always explicit SQL. For example, write +`order_id = CAST(customer_code AS INTEGER)`, or use the spelling the error suggests. There is no +per-model suppression. -The single human partial-check notice groups model names by reason, gives the relevant repair or -rerun command, and truncates long lists. `sqb compile --json` contains the full selected-model -reason map. Unknown output types and unresolved stars are reported as explicitly as open sources; -the notice never reduces them to a count alone. +### Type checks by dialect + +| Dialect | Reference and semantic checks | Expression type checks | Function spellings | +|---|---|---|---| +| DuckDB (including MotherDuck) | Yes | Yes | Yes | +| Snowflake | Yes | Yes | Yes | +| PostgreSQL | Yes | Yes | No | +| BigQuery | Yes | Yes | No | +| Other adapters | Yes | No | No | + +Coercion rules differ between dialects: DuckDB accepts a number as a predicate, while PostgreSQL, +Snowflake and BigQuery reject it. Typed string literals such as `ordered_at > '2026-01-01'` remain +valid. + +### Snowflake details + +- Unknown function names are not rejected, because Snowflake's built-in catalogue is incomplete. An + unmodelled function's result type is unknown rather than guessed. +- `QUOTED_IDENTIFIERS_IGNORE_CASE = true` in the target's session parameters is honoured: + + ```toml + [targets.analytics.connection] + session_parameters = { QUOTED_IDENTIFIERS_IGNORE_CASE = true } + ``` + +- Set operations are checked in operator order (`INTERSECT` before `UNION` and `EXCEPT`). + +## What isn't checked + +Compilation doesn't replace running your SQL unit tests. These remain partly unchecked: + +- **Expressions that use a project function's result.** Registering a function prevents + unknown-function errors but doesn't declare its return type. +- **Raw relations SQLBuild doesn't manage,** such as tables written by hooks. +- **Function overloads.** Signature coverage is partial for every dialect. + +Inputs with unknown types never cause type errors. Instead, `sqb compile` prints one notice listing +the affected models by reason, with the command to fix or rerun each. `sqb compile --json` gives +the full list in `semantic_checks_partial`. + +## Reading a diagnostic + +Each diagnostic shows, in order: + +1. the code and headline; +2. the model, test, audit or hook it belongs to, with the file and position; +3. the source line, with the problem underlined; +4. notes, such as the input's available columns, closest first; +5. help: a suggested fix, such as the column you probably meant or the explicit cast to write. + +When one error makes downstream columns uncertain, SQLBuild doesn't repeat it for each use. A note +counts the downstream uses that weren't checked. Independent errors elsewhere are still reported. + +`sqb compile --json` carries the same fields, with exclusive `end_line` and `end_column`. + +## Turning analysis off + +To skip SQL analysis for one model, use `MODEL (sql_analysis false)`, a path default, or +`--no-sql-analysis` for a run. This also turns off type inference and lineage for that model. +To allow those opt-outs only for SQL the parser can't read, set +`[settings] require_sql_analysis = true`; see +[Requiring SQL analysis](models/configuration.md#requiring-sql-analysis). ## Diagnostic codes -| SQLBuild | Native | Meaning | -|---|---|---| -| B002 | E201 | Unknown column | -| B003 | E221 | Ambiguous column | -| B004–B005 | E222–E223 | Reference/scope errors | -| B101 | E202 or SQLBuild | Unknown function, or a built-in spelling the dialect does not accept | -| B102 | E203 or project declaration | Invalid function arity | -| B103 | SQLBuild | Truth-value test the dialect does not accept | -| B104 | SQLBuild | Warning: a SQLite `CAST` to a date or time type, which keeps only the leading number | -| B210–B219 | E210–E219 | Proven native type/shape errors on enabled dialects | -| B211–B218 | W210–W216 | Implicit conversions; blocking, fixed by an explicit `CAST` or comparison | -| B216 | E216 | Set-operation, subquery, or row arity | -| B230–B232 | E230–E232 | Grouping, aggregate, and window semantics | -| B233 | E233 | Duplicate scope names, according to dialect rules | -| B234 | E234 | Invalid LIMIT/OFFSET bounds | -| B300 | SQLBuild | Unknown metadata/audit column | -| B301 | SQLBuild | Incompatible declared metadata, audit, or UDF argument type | -| B302 | SQLBuild | Unknown SQL-test fixture/expected column | - -Implicit conversions are enforced rather than reported as warnings. Each native W210–W216 -finding maps to the SQLBuild code of the incompatibility it risks: comparisons to B217, arithmetic -to B212, assignments to B214, casts and coercible literals to B218, set-operation branches to B215, -predicates to B211, and function arguments to B213. The remedy is always an explicit conversion, -for example `order_id = CAST(customer_code AS INTEGER)`, or an explicit comparison such as -`quantity <> 0`. There is no per-model suppression. - -B002–B005 predate these changes and overlap the executor's separately phased build codes. New -semantic codes use distinct ranges. Authored model diagnostics retain source paths and positions. -Cast, coercible-literal and set-operation findings point at the offending cast, literal or branch -projection, and every finding is reported: two failing casts in one query are two diagnostics at -their own locations, never one merged diagnostic at the statement start. +| Code | Meaning | +|---|---| +| B002 | Unknown column | +| B003 | Ambiguous column | +| B004–B005 | Reference and scope errors | +| B101 | Unknown function, or a spelling the dialect doesn't accept | +| B102 | Wrong number of function arguments | +| B103 | Truth-value test the dialect doesn't accept | +| B104 | Warning: a SQLite `CAST` to a date or time type, which keeps only the leading number | +| B210–B219 | Type and shape errors the warehouse would raise | +| B211 | Implicit conversion in a predicate | +| B212 | Implicit conversion in arithmetic | +| B213 | Implicit conversion in a function argument | +| B214 | Implicit conversion in an assignment | +| B215 | Incompatible set-operation branches | +| B216 | Set-operation, subquery or row arity | +| B217 | Implicit conversion in a comparison | +| B218 | Implicit conversion in a cast or literal, or an unknown type | +| B230–B232 | Grouping, aggregate and window errors | +| B233 | Duplicate names in one scope | +| B234 | Invalid `LIMIT` or `OFFSET` | +| B300 | Unknown column in metadata or an audit | +| B301 | Incompatible declared metadata, audit or UDF argument type | +| B302 | Unknown SQL-test fixture or expected column | +| P001 | SQL the parser can't read, including in audits and tests | diff --git a/website/src/content/docs/docs/concepts/semantic-compilation.mdx b/website/src/content/docs/docs/concepts/semantic-compilation.mdx index 1387a963ec..8095d04aff 100644 --- a/website/src/content/docs/docs/concepts/semantic-compilation.mdx +++ b/website/src/content/docs/docs/concepts/semantic-compilation.mdx @@ -3,122 +3,8 @@ title: "Semantic Compilation" description: "Offline reference, semantic and type checks that sqb compile runs before any warehouse work." --- -`sqb compile` stays offline. It validates SQL references against each input separately, including -models with no relation inputs. A join to an open source does not disable checks on a closed input. -An unqualified name that could belong to an open input cannot be rejected offline. - -## Authoritative column names - -- Enforced model and source contracts, seeds, and declared table-function outputs are closed shapes. -- Expression sources have the column names inferred from their SQL expression. -- An explicit model projection defines its output names even over an open input. Types propagate - through dependencies where they can be inferred; an unknown type does not make the names unknown. -- A star over closed inputs expands; a star over an open input remains open. -- Declaring some columns on a table source does **not** declare its complete shape. Use - `contract: enforced` when those declarations are complete. - -Adopt a physical source schema with: - -```bash -sqb contract generate --from --select source:raw_orders --write -``` - -`sqb plan` and `sqb build` recheck source readers against inspected physical columns before model -execution. Existing bulk metadata is reused. Selected table sources omitted from that metadata -are inspected in database-grouped batches: one relation-list call and one bulk-column call per -database, using the adapter's normal metadata batching. Missing managed sources and ambiguous -unqualified physical names stay open. - -## Checks and limits - -Binding checks apply to SQL clauses, CTEs, subqueries, and expanded macros. Semantic grouping, -aggregate and window-placement diagnostics are errors. Project UDF arity and known argument types -are checked separately; declared return types contribute to inference. SQL and Python UDFs and -table functions are registered with the native catalogue, along with types named in project -contracts and function signatures. Registration does not replace SQLBuild's signature checks. - -Known output shapes also validate `unique_key`, `cursor`, `cursor_type`, `cursor_inputs`, custom -`config.partition_column`, `row_diff_exclude_columns`, `row_diff_tolerances.by_column`, source -`cursor_column`, audit expressions/relationships/accepted values, and SQL-test mock/expected names. - -The same binding, function-name and type checks run on every SQL body a project owns, not only -model queries: singular and generic audits (including measurement and evidence queries), SQL unit -tests (mock, helper, expected and assertion CTEs), and inline and named SQL hooks. An audit reading -`__ref("orders")` is checked against the closed `orders` output shape, so `WHERE o.shipped_at IS -NULL` on a model without that column fails compile instead of the warehouse. Each diagnostic names -its owner: `SQL test 'orders_are_open': ...`, `audit 'recent_rows' on model 'orders': ...` (with the -attached column when there is one), or `post_hooks[1] SQL hook: ...` at the hook file. Type -mismatches show the explicit conversion to write, such as `CAST('1' AS NUMBER)` in a test's expected -rows or `TO_DATE(@date_column)` in a generic audit. An -audit or SQL-test body the SQL parser cannot read is a P001 syntax error, never a silent pass. -Relations without a closed shape, such as physical tables written in hooks, stay open. - -Expression type checks are enabled for **DuckDB (including MotherDuck), PostgreSQL, Snowflake, -and BigQuery**. Other adapters retain reference and semantic checks without expression type checks. - -Proven bind-time rejections and implicit conversions are both errors. For example, DuckDB rejects -`ordered_at > 5` for a TIMESTAMP column. Snowflake accepts `order_id = customer_code` for an -INTEGER and a VARCHAR column, but the conversion fails on non-numeric values, so SQLBuild reports -it as a blocking B217 error; write the conversion explicitly. Typed string literals such as -`ordered_at > '2026-01-01'` remain valid. Columns from `SELECT *` are type-checked through their -upstream shapes in every query. Unknown input types never cause type errors. Dialect coercions -differ: numeric predicates are accepted by DuckDB but rejected by PostgreSQL, Snowflake, and -BigQuery. -Snowflake's function catalogue is partial and deliberately does not reject unknown function names. -Function signatures and overload coverage remain partial; compilation is not a replacement for -executing SQL unit tests. - -Built-in function spellings are checked for Snowflake and DuckDB. The SQL parser treats alias -spellings such as `STARTS_WITH` and `STARTSWITH` as one built-in, so a spelling the warehouse -rejects would otherwise compile. SQLBuild compares each call's authored name, case-insensitively, -with the dialect's accepted spellings and reports a blocking B101 error naming the dialect's -spelling, for example `Unknown function 'STARTS_WITH' for dialect Snowflake; did you mean -STARTSWITH?`. Snowflake's accepted spellings come from its function reference and include -documented synonyms such as `SUBSTR` and `SUBSTRING`, `DATEDIFF` and `TIMESTAMPDIFF`, and -`TRY_TO_DECIMAL` and `TRY_TO_NUMBER`. DuckDB's come from its function catalogue. A name is -rejected only when the parser maps it to the same built-in as an accepted spelling, so project -UDFs called through `__udf`, schema-qualified and method calls, quoted names, and unknown -functions are not affected. There is no per-model suppression: use the suggested spelling, or -schema-qualify a call to an external UDF that shares an alias name. Other dialects are not checked. - -Truth-value tests (`IS [NOT] TRUE` and `IS [NOT] FALSE`) are rejected for Snowflake and SQL -Server, whose warehouses do not accept them although the SQL parser does. SQLBuild reports a -blocking B103 error at the `IS` keyword with a null-safe rewrite, for example -`'IS NOT TRUE' is not supported for dialect Snowflake; write NOT COALESCE(, FALSE) or -IS DISTINCT FROM TRUE`. SQL Server suggestions compare a BIT value, such as -`COALESCE(, 0) = 1`. DuckDB (including MotherDuck), PostgreSQL, BigQuery and Databricks -accept truth-value tests. - -Derived-table column aliases, UNPIVOT, and Snowflake GROUPING SETS are checked. DuckDB and Snowflake -reject excess relation aliases; fewer aliases than output columns remain valid. Missing UNPIVOT -input columns are errors. - -Project function registration prevents unknown-function errors but does not declare return types to -the validator: some expressions consuming a project function result remain unchecked. Raw external -relations and scope-ordering cases also remain -partially checked. A cast to a type SQLBuild does not know is a B218 error. Types named in project -contracts and function signatures are registered, so declare an extension-defined type there. - -One notice summarizes partial semantic checks. `sqb compile --json` includes per-model reasons in -`semantic_checks_partial`; progress and notices go to stderr. The notice does not increment the -diagnostic warning count. - -Escape hatches are `MODEL (sql_analysis false)`, the corresponding path default, and -`--no-sql-analysis`. Disabling SQL analysis also disables inference and lineage for that model. -`[settings] require_sql_analysis = true` limits the MODEL and path-default opt-outs to SQL the -parser cannot read; see [Requiring SQL analysis](/docs/concepts/models/configuration#requiring-sql-analysis). -Help that names a setting always shows the exact file, section and `key = value` to write. - -## Reading diagnostics - -Diagnostics use one layout: headline, resource and location, source snippet with a span underline, -notes, then code-specific help. SQL-test diagnostics include their authored fixture snippet and -use `sql test: name`, just as model diagnostics use `model: name`. JSON retains the location's -exclusive `end_line`/`end_column` boundary, notes, and help. - -An unknown column identifies its input and authored alias. Suggestions use edit distance with a -bounded typo cutoff and recognize short abbreviations such as `qty`. No suggestion is printed when -none is close. The available-column note puts the closest names first and shows at most ten names. +`sqb compile` checks your SQL before anything runs in the warehouse: unknown columns, wrong types, +misspelled functions, invalid grouping, and more. It works offline, from your project alone. ```text error[B002]: Unknown column 'qty' in stg_orders (as o) @@ -132,91 +18,145 @@ error[B002]: Unknown column 'qty' in stg_orders (as o) = help: did you mean 'quantity'? ``` -Type errors report uppercase types and available operand evidence, for example -`o.ordered_at is TIMESTAMP, 5 is INTEGER`. Comparison help recommends a literal appropriate to the -type family and dialect. Native spans are mapped back through normalization and macro expansion; -simple binary expressions are underlined in full. Diagnostics without usable spans keep the -authored query-start fallback. - -Error recovery treats poisoned output names and their downstream lineage as provisional. A -downstream error traceable only to that root is suppressed and counted in a root-error note; -independent errors on other columns and other inputs remain errors. Recovery does not silence a -whole dependent query or make its unchecked uses valid. Affected models remain in -`semantic_checks_partial`, including on warm/cache-hit runs. - -Located binding and type errors also make the affected projection's type unknown. That uncertainty -propagates through column lineage. Failed dependants are checked again with only those input types -unknown; an independent error on another column or input remains visible. Root notes count unchecked -downstream output uses, including transitive uses. Contract type checks do not emit a secondary -unknown-type warning for those same poisoned outputs. If native evidence cannot locate the failing -projection, recovery retains the diagnostic rather than opening the entire input. - -Temporal cursor metadata accepts DATE, DATETIME, and TIMESTAMP families interchangeably, including -Snowflake timestamp variants. A timestamp cursor on a string, numeric, or Boolean column still -produces B301. - -Snowflake binding honours `QUOTED_IDENTIFIERS_IGNORE_CASE = true` from the effective target's -connection `session_parameters`, for both offline declared schemas and warehouse-backed rebinding. -False or absent settings preserve the dialect's existing case rules. For example: - -```toml -[targets.analytics] -schema = "preserve" - -[targets.analytics.connection] -session_parameters = { QUOTED_IDENTIFIERS_IGNORE_CASE = true } +## Closed and open inputs + +A check can only reject a column when SQLBuild knows the input's complete set of columns. Such an +input is **closed**; otherwise it is **open**. + +| Input | Shape | +|---|---| +| Model or source with `contract: enforced`, seed, declared table-function output | Closed | +| Expression source | Closed: names come from its expression | +| Model with an explicit `SELECT` list | Closed names, even over open inputs | +| `SELECT *` over closed inputs | Closed | +| `SELECT *` over an open input, or a table source without an enforced contract | Open | + +Each input is checked separately, so joining an open source doesn't switch off checks on a closed +one. An unqualified column that could come from an open input can't be rejected offline. + +Declaring some columns on a source doesn't close it. To close a source, adopt its physical schema: + +```bash +sqb contract generate --from --select source:raw_orders --write ``` -The setting participates in the analysis-cache identity. Identifier folding is token-based and -does not alter string literals, comments, or authored SQL. Diagnostic offset mapping is lazy and -uses a bounded cache of token alignments, shared by all diagnostics on the same normalized query. +`sqb plan` and `sqb build` also recheck source readers against the warehouse's real columns before +running models. + +## What gets checked + +Checks run on every SQL body a project owns: models (including CTEs, subqueries and expanded +macros), audits, SQL unit tests and SQL hooks. A problem in an audit or test is reported against it, +for example `audit 'recent_rows' on model 'orders': ...`, and fails compile instead of the +warehouse. + +| Check | Example | Code | +|---|---|---| +| Unknown or ambiguous columns | `o.qty` when the input has `quantity` | B002, B003 | +| Columns named in configuration | `unique_key`, `cursor`, audit columns, SQL-test fixture columns | B300, B302 | +| Types of configured columns | A timestamp `cursor` on a VARCHAR column | B301 | +| Types that the warehouse rejects | `ordered_at > 5` on a TIMESTAMP in DuckDB | B210–B219 | +| Implicit conversions that can fail at runtime | `order_id = customer_code` (INTEGER and VARCHAR) in Snowflake | B211–B218 | +| Function names and spellings | `STARTS_WITH` in Snowflake, which only accepts `STARTSWITH` | B101 | +| Function arity | Too many arguments to a built-in or project UDF | B102 | +| Truth-value tests the warehouse rejects | `x IS NOT TRUE` in Snowflake or SQL Server | B103 | +| Grouping, aggregates and windows | A non-grouped column next to an aggregate | B230–B232 | +| Set operations | A VARCHAR branch combined with a numeric branch | B215, B216 | +| Unknown types | `CAST(x AS MYTYPE)` | B218 | -Snowflake set-operation checks are directional: each branch is compared with the accumulated -result type, following the query's operator precedence (`INTERSECT` binds more tightly than -`UNION` and `EXCEPT`). Proven incompatibilities and implicit conversions whose success depends on -runtime values both produce blocking B215 errors. For example, combining a VARCHAR branch with a -numeric branch requires a value-dependent numeric conversion; cast the branch column explicitly. +Derived-table column aliases, `UNPIVOT` and Snowflake `GROUPING SETS` are checked too. -An unmodelled Snowflake function has an unknown result type rather than inheriting its first -argument's type. SQLBuild retains the output name and lineage, reports partial type coverage, and -uses explicit declared function signatures where available. This prevents a guessed return type -from creating downstream type errors. +Every check except B104 is a blocking error. The fix is always explicit SQL. For example, write +`order_id = CAST(customer_code AS INTEGER)`, or use the spelling the error suggests. There is no +per-model suppression. -The single human partial-check notice groups model names by reason, gives the relevant repair or -rerun command, and truncates long lists. `sqb compile --json` contains the full selected-model -reason map. Unknown output types and unresolved stars are reported as explicitly as open sources; -the notice never reduces them to a count alone. +### Type checks by dialect + +| Dialect | Reference and semantic checks | Expression type checks | Function spellings | +|---|---|---|---| +| DuckDB (including MotherDuck) | Yes | Yes | Yes | +| Snowflake | Yes | Yes | Yes | +| PostgreSQL | Yes | Yes | No | +| BigQuery | Yes | Yes | No | +| Other adapters | Yes | No | No | + +Coercion rules differ between dialects: DuckDB accepts a number as a predicate, while PostgreSQL, +Snowflake and BigQuery reject it. Typed string literals such as `ordered_at > '2026-01-01'` remain +valid. + +### Snowflake details + +- Unknown function names are not rejected, because Snowflake's built-in catalogue is incomplete. An + unmodelled function's result type is unknown rather than guessed. +- `QUOTED_IDENTIFIERS_IGNORE_CASE = true` in the target's session parameters is honoured: + + ```toml + [targets.analytics.connection] + session_parameters = { QUOTED_IDENTIFIERS_IGNORE_CASE = true } + ``` + +- Set operations are checked in operator order (`INTERSECT` before `UNION` and `EXCEPT`). + +## What isn't checked + +Compilation doesn't replace running your SQL unit tests. These remain partly unchecked: + +- **Expressions that use a project function's result.** Registering a function prevents + unknown-function errors but doesn't declare its return type. +- **Raw relations SQLBuild doesn't manage,** such as tables written by hooks. +- **Function overloads.** Signature coverage is partial for every dialect. + +Inputs with unknown types never cause type errors. Instead, `sqb compile` prints one notice listing +the affected models by reason, with the command to fix or rerun each. `sqb compile --json` gives +the full list in `semantic_checks_partial`. + +## Reading a diagnostic + +Each diagnostic shows, in order: + +1. the code and headline; +2. the model, test, audit or hook it belongs to, with the file and position; +3. the source line, with the problem underlined; +4. notes, such as the input's available columns, closest first; +5. help: a suggested fix, such as the column you probably meant or the explicit cast to write. + +When one error makes downstream columns uncertain, SQLBuild doesn't repeat it for each use. A note +counts the downstream uses that weren't checked. Independent errors elsewhere are still reported. + +`sqb compile --json` carries the same fields, with exclusive `end_line` and `end_column`. + +## Turning analysis off + +To skip SQL analysis for one model, use `MODEL (sql_analysis false)`, a path default, or +`--no-sql-analysis` for a run. This also turns off type inference and lineage for that model. +To allow those opt-outs only for SQL the parser can't read, set +`[settings] require_sql_analysis = true`; see +[Requiring SQL analysis](/docs/concepts/models/configuration#requiring-sql-analysis). ## Diagnostic codes -| SQLBuild | Native | Meaning | -|---|---|---| -| B002 | E201 | Unknown column | -| B003 | E221 | Ambiguous column | -| B004–B005 | E222–E223 | Reference/scope errors | -| B101 | E202 or SQLBuild | Unknown function, or a built-in spelling the dialect does not accept | -| B102 | E203 or project declaration | Invalid function arity | -| B103 | SQLBuild | Truth-value test the dialect does not accept | -| B104 | SQLBuild | Warning: a SQLite `CAST` to a date or time type, which keeps only the leading number | -| B210–B219 | E210–E219 | Proven native type/shape errors on enabled dialects | -| B211–B218 | W210–W216 | Implicit conversions; blocking, fixed by an explicit `CAST` or comparison | -| B216 | E216 | Set-operation, subquery, or row arity | -| B230–B232 | E230–E232 | Grouping, aggregate, and window semantics | -| B233 | E233 | Duplicate scope names, according to dialect rules | -| B234 | E234 | Invalid LIMIT/OFFSET bounds | -| B300 | SQLBuild | Unknown metadata/audit column | -| B301 | SQLBuild | Incompatible declared metadata, audit, or UDF argument type | -| B302 | SQLBuild | Unknown SQL-test fixture/expected column | - -Implicit conversions are enforced rather than reported as warnings. Each native W210–W216 -finding maps to the SQLBuild code of the incompatibility it risks: comparisons to B217, arithmetic -to B212, assignments to B214, casts and coercible literals to B218, set-operation branches to B215, -predicates to B211, and function arguments to B213. The remedy is always an explicit conversion, -for example `order_id = CAST(customer_code AS INTEGER)`, or an explicit comparison such as -`quantity <> 0`. There is no per-model suppression. - -B002–B005 predate these changes and overlap the executor's separately phased build codes. New -semantic codes use distinct ranges. Authored model diagnostics retain source paths and positions. -Cast, coercible-literal and set-operation findings point at the offending cast, literal or branch -projection, and every finding is reported: two failing casts in one query are two diagnostics at -their own locations, never one merged diagnostic at the statement start. +| Code | Meaning | +|---|---| +| B002 | Unknown column | +| B003 | Ambiguous column | +| B004–B005 | Reference and scope errors | +| B101 | Unknown function, or a spelling the dialect doesn't accept | +| B102 | Wrong number of function arguments | +| B103 | Truth-value test the dialect doesn't accept | +| B104 | Warning: a SQLite `CAST` to a date or time type, which keeps only the leading number | +| B210–B219 | Type and shape errors the warehouse would raise | +| B211 | Implicit conversion in a predicate | +| B212 | Implicit conversion in arithmetic | +| B213 | Implicit conversion in a function argument | +| B214 | Implicit conversion in an assignment | +| B215 | Incompatible set-operation branches | +| B216 | Set-operation, subquery or row arity | +| B217 | Implicit conversion in a comparison | +| B218 | Implicit conversion in a cast or literal, or an unknown type | +| B230–B232 | Grouping, aggregate and window errors | +| B233 | Duplicate names in one scope | +| B234 | Invalid `LIMIT` or `OFFSET` | +| B300 | Unknown column in metadata or an audit | +| B301 | Incompatible declared metadata, audit or UDF argument type | +| B302 | Unknown SQL-test fixture or expected column | +| P001 | SQL the parser can't read, including in audits and tests | From dbb567fb247f279fe6897fa3dc02a8f58c2d7fa5 Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Tue, 27 Oct 2026 09:04:42 +0000 Subject: [PATCH 02/10] docs: restructure the compile page around what users run and when to force a fresh compile --- .../sqlbuild/references/docs/cli/compile.md | 506 +++++++++--------- website/src/content/docs/docs/cli/compile.mdx | 504 +++++++++-------- 2 files changed, 490 insertions(+), 520 deletions(-) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md index ca2647b029..80441d5df0 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md @@ -11,15 +11,41 @@ Online: https://sqlbuild.com/docs/cli/compile/ - Usage - Flags - What compile does +- Compiling a selection - Compiled artifacts -- Focused compilation -- Reusing an unchanged compile -- Static analysis +- Contract diagnostics - Output - Column lineage modes +- Reusing an unchanged compile +- Compiling after an edit +- Forcing a fresh compile - Examples -Discovers the complete project, resolves references, expands macros, validates SQL, checks column contracts, computes column lineage, and writes compiled artifacts to `target/`. The compile command is fully offline—it does not connect to the warehouse. +`sqb compile` turns your project into the SQL it will run. It resolves references, expands +macros, checks SQL and column contracts, computes column lineage, runs your Rules, and writes the +compiled SQL to `target/compiled/`. It works offline and never connects to the warehouse. + +```bash +sqb compile +``` + +``` +Compile ready 12 models + +├── daily_revenue OK 6 columns +├── dim_customers OK 7 columns +├── fact_orders OK 13 columns +├── stg_customers OK 5 columns +├── stg_orders OK 6 columns +└── stg_payments OK 6 columns +... + +✓ Project compiled 12 models, 1 seed, 2 functions, 0 errors, 0 warnings + Wrote: target/compiled/ +``` + +Each model shows its status (`OK` or `FAIL`) and column count. Compile exits with `1` when it +finds any error, so it can gate CI. ## Usage @@ -27,57 +53,48 @@ Discovers the complete project, resolves references, expands macros, validates S sqb compile [SELECTOR ...] [flags] ``` -Positional `SELECTOR` arguments work like `--select`, and both forms can be combined. `--project-dir `, `--no-color`, and `--debug` can go before or after the command name. +Positional `SELECTOR` arguments work like `--select`, and both forms can be combined. +`--project-dir `, `--no-color` and `--debug` can go before or after the command name. ## Flags | Flag | Description | |------|-------------| -| `--no-sql-analysis` | Disable SQL syntax, binding, type inference, and semantic validation (`--no-sql-validation` remains an alias) | -| `--no-cache` | Bypass the reusable compile-analysis cache and the Rules cache for this invocation | -| `--defer-to` | Resolve unselected model references against another target | -| `--json` | Output the full compile report as JSON | -| `--manifest` | Generate `target/manifest.json` with project metadata | -| `--lineage-mode` | Column lineage mode: `fast` (default), `rich` (slower, more detail), or `none` | | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Analyze and report selected resources | | `--select-file` | Read selectors from a file, one selector per line | | `--exclude` | Remove resources from the selected scope | +| `--no-sql-analysis` | Turn off SQL syntax, binding, type inference and semantic checks (`--no-sql-validation` is an alias) | +| `--no-cache` | Compile in full, without reusing earlier compiles or the Rules cache; see [Forcing a fresh compile](#forcing-a-fresh-compile) | +| `--defer-to` | Resolve unselected model references against another target | +| `--json` | Print the full compile report as JSON | +| `--manifest` | Also write `target/manifest.json` with project metadata | +| `--lineage-mode` | Column lineage mode: `fast` (default), `rich` or `none` | ## What compile does -1. **Discovery** - finds `sqlbuild_project.toml`, scans for models, sources, seeds, functions, audits, tests, and macros -2. **Graph resolution** - resolves `__ref()`, `__source()`, and `__seed()` references, expands macros, orders models by dependency -3. **SQL validation** - validates SQL syntax (when SQL analysis is enabled) -4. **Column lineage** - analyzes column-level dependencies across models (fast mode by default) -5. **Contract validation** - checks declared column contracts against inferred query output -6. **Rules** - evaluates selected native built-ins, then selected custom Python Rules. Rules that - need inferred type proofs, such as `SQBRCONTRACT105`, are skipped when SQL analysis is disabled, - and compile prints one note naming them -7. **Artifact write** - writes compiled SQL to `target/compiled/` when Rules pass - -## Compiled artifacts - -`target/compiled/` mirrors what each resource will run, with every macro, variable and audit -argument already rendered: +1. **Discovery.** Finds `sqlbuild_project.toml` and the project's models, sources, seeds, + functions, audits, tests and macros. +2. **Graph.** Resolves `__ref()`, `__source()` and `__seed()`, expands macros, and orders models + by dependency. +3. **SQL checks.** Parses and checks SQL, when SQL analysis is on. See + [Semantic Compilation](../concepts/semantic-compilation.md). +4. **Column lineage.** Traces column-level dependencies across models. +5. **Contracts.** Checks declared columns against the inferred query output. +6. **Rules.** Runs the selected built-in Rules, then the selected custom Rules. +7. **Write.** Writes compiled SQL to `target/compiled/` when the Rules pass. -| Path | Contents | -|------|----------| -| `models/.sql` | Each model's query | -| `functions//.sql` | Each function's `CREATE FUNCTION` statements | -| `audits/generic//__.sql` | Each attached generic audit, under the model, source or seed it checks; column audits end in `__`, and an audit's `name` replaces the audit name | -| `audits/singular/.sql` | Each singular audit | -| `tests/...` | Each SQL unit test's comparison query | +Rules that need inferred types, such as `SQBRCONTRACT105`, are skipped when SQL analysis is off, +and compile prints one note naming them. -Read an audit's file to see how its arguments were spliced into the SQL without running it or -looking up the warehouse's query history. Files are rewritten on every compile, and files for -resources that no longer exist are removed. References such as `__ref("orders")` stay as written, -as in compiled models. +`sqb compile` is authoritative for correctness, contracts, lineage and configured Rules. Rule +findings block the artifacts. To check one Rule code or family without changing project +configuration, use [`sqb rules run`](rules.md). -## Focused compilation +## Compiling a selection -Use normal selectors to limit expensive analysis and reporting while retaining complete project -discovery and reference integrity: +Selectors limit analysis and reporting. Discovery still covers the whole project, so every +reference is still checked. ```bash sqb compile fact_orders daily_revenue @@ -85,161 +102,30 @@ sqb compile --select-file selected-models.txt sqb compile --exclude deprecated_model ``` -SQLBuild deeply analyzes the selected resources and the upstream closure needed to understand them. -The text report contains only the selected scope. Project totals remain available in JSON, together -with selected model, seed, and function counts. +SQLBuild analyses the selected resources and the upstream models needed to understand them. The +text report shows only the selection. JSON also has the project totals and the selected model, seed +and function counts. The manifest and the DAG always cover the whole project. -Manifest and DAG outputs remain full-project artifacts. Focused compilation changes analysis and -reporting scope; it does not produce a partial project graph. +## Compiled artifacts -## Reusing an unchanged compile +`target/compiled/` holds what each resource will run, with every macro, variable and audit +argument rendered: -When nothing that can change the result has changed since the last compile, `sqb compile` replays -that result instead of compiling again. It prints one line to stderr, for example -`Inputs unchanged; reused the previous compile (0.1 s)`, then the same report and exit code as the -previous run. Failed compiles are replayed too, with the same diagnostics. - -A compile is reused only when all of these match the stored run: - -- Every file and directory under the project, compared by type, size, modification and change - times, and inode. When those differ, SQLBuild compares content only for files whose content - digest it already knows; otherwise the compile runs in full. Digests are recorded for files that - changed within two seconds of a compile and for files whose timestamps moved since the previous - compile, so large unchanged files are never read. After a checkout that only moves timestamps, - the next compile runs in full once, and later timestamp-only changes are reused. -- The command line: project directory, working directory, flags, selectors, `--vars`, target, and - whether color output is active. -- Environment variables that SQLBuild reads: every `SQLBUILD_*` and `SQB_*` variable, every - variable that a template or `@@ENV` read during the stored compile, and every variable a - provider's settings can read, including `env_prefix`, aliases, and nested-delimiter variables. - Values are stored only as hashes. -- The contents of each provider settings `env_file` and `secrets_dir`. -- The Python interpreter, the SQLBuild version and native extension, `sys.path` entries, and every - loaded Python module file outside the project. -- Every artifact the stored compile wrote under `target/compiled/`, and the `--dag` file. If one is - missing or changed, compile runs in full and rewrites it. A compile is stored only when these - artifacts still hold the bytes it wrote, so a concurrent compile cannot leave another compile's - artifacts behind a reused result. - -These project paths are not inputs: `target/`, `logs/`, `venv/`, `.git/` and `__pycache__/` -anywhere, and these folders at the project root: `.cache`, `.fensu`, `.hg`, `.idea`, -`.mypy_cache`, `.nox`, `.pytest_cache`, `.ruff_cache`, `.sqlbuild`, `.svn`, `.tox`, `.venv`, and -`.vscode`. Other hidden folders are inputs, because custom rules can read them. DuckDB database -files count only by presence. - -Reuse assumes macros, hooks, providers, adapters, and custom rules are deterministic: the same inputs -must produce the same output. Macro results are also reused within a compile and may be kept between -compiles; see [Macros must be deterministic](../concepts/macros/python-macros.md#macros-must-be-deterministic). A -compile whose templates read `run.id` is never reused. +| Path | Contents | +|------|----------| +| `models/.sql` | Each model's query | +| `functions//.sql` | Each function's `CREATE FUNCTION` statements | +| `audits/generic//__.sql` | Each attached generic audit, under the model, source or seed it checks; column audits end in `__`, and an audit's `name` replaces the audit name | +| `audits/singular/.sql` | Each singular audit | +| `tests/...` | Each SQL unit test's comparison query | -Compile always runs in full with `--no-cache`, `--manifest`, `--debug`, or a profiling flag, when -`SQLBUILD_DISABLE_COMPILE_CACHE=1` or `SQLBUILD_DISABLE_COMPILE_REUSE=1` is set, when the project has -no `sqlbuild_project.toml` or `sqlbuild_project.yml`, when the target sets `compile_cache = false`, -and when a provider's settings use sources SQLBuild cannot list exactly, such as a custom -`settings_customise_sources` or command-line parsing. The reason is logged at debug level. - -SQLBuild keeps one stored result per project and target under -`target/cache/compiler-native-v1/project-reuse-v3/`: a small `.store` file holding what the result -depends on, and a `.output` file beside it holding the report. A check reads only the `.store` file; -the report is read only when nothing changed. Each file is replaced atomically; a damaged, missing or -unreadable file is ignored and compile runs in full. Storing a result normally takes a fraction of a second after the -report is ready. When it has to hash a lot of data or the project has very many files, compile -prints `Recording compile for reuse...` and a completion line to stderr before the report. - -### Compiling after an edit - -When only model SQL files and SQL test files changed since the stored compile, `sqb compile` -compiles just those files against it. It renders, analyses and checks the changed models and SQL -tests, plus any model whose inputs' columns, types or nullability the edit changed, and takes -everything else from a project image stored beside the stored compile. It prints one line to -stderr, for example `Compiled 1 changed file(s) against the previous compile (1 model(s), 2.1s)`, -then the same report and exit code a full compile would produce, including any errors the edit -introduces or fixes. Project-wide rules still run over the whole project. - -The compile runs in full instead, decided before any compile work starts, when: - -- A file was added, removed or renamed, or any file other than a model or SQL test changed, such - as `sqlbuild_project.toml`, a source or schema YAML file, a macro, a seed or a function. -- An edited model changes what it references, or changes a header setting other than its - `description`, `columns`, `dynamic_columns`, `model_schema`, `audits`, `contract`, or a switch - between `view` and `table` materialization. -- A project-subject custom rule reads something the edit changed, or a model-subject custom rule - must run again for a model the edit did not change, for example because it reads the text of an - edited upstream file. -- The stored compile reported errors or warnings other than rule findings in model or SQL test - files the edit did not change. -- The stored compile used the other output format (`--json` or text), or has no project image. - -A few edits are found to need a full compile only part-way through, for example when a macro's -arguments change which models or sources it reads, when the edit changes a model's -`dynamic_columns` families, or when the edit changes the shape of so many models that a full -compile is cheaper. A model-subject custom rule that must run again for an edited model runs in the -delta compile when it reads only that model's SQL, columns, contract and grain, project files, and -the source list; if it reads anything else, such as the model's dependencies, the compile runs in -full. - -Otherwise, an edit runs the whole compile again, and a few caches under `target/cache/` shorten -it: every model, test, and audit is rendered again, and contracts and semantic validation run for -the whole project. Every model's SQL references are scanned again too. - -- An unchanged model takes its column analysis from `model-analyses.bin`. Entries are keyed by - the model's query and settings, the inferred columns, types and nullability of every relation - it reads, the project's analysis settings, and the SQLBuild build, so a model whose upstream - output changed is analysed again; a damaged or unreadable entry is analysed again too. -- An unchanged SQL test file reuses its stored test scan, the CTEs and expected models read from - it, from `sql-test-scans.bin`. Entries are keyed by the file's expanded text, the adapter's SQL - lexical rules, and the SQLBuild build; a damaged or unreadable entry is scanned again. -- Rule results are reused for unchanged inputs. -- Macro call results are reused as described in - [Macros must be deterministic](../concepts/macros/python-macros.md#macros-must-be-deterministic). - -Either way, the output is byte-identical to `sqb compile --no-cache`. To turn compiling only the -changed files off, set `delta_compile = false` under `[settings]` or set -`SQLBUILD_DISABLE_DELTA_COMPILE=1`; compiles then no longer store a project image. - -Known limits: - -- Python packages outside the project are checked by the stamps of the module files the compile - loaded. A newly added submodule or package data file that the compile reads without importing it - first is not tracked. -- On network filesystems that cache file attributes, or when clocks differ between machines, - timestamps may not reflect a recent edit. -- On Windows, a file's change time is its creation time. An in-place rewrite that keeps the same - size and restores the previous modification time, such as `touch -r`, `rsync -t`, or - `robocopy /COPY:T`, is not detected unless the file changed within two seconds of the stored - compile. The same applies to filesystems without a real change time, such as some FUSE, SMB, and - exFAT mounts. -- In these cases, or whenever a result looks stale, run `sqb compile --no-cache` or set - `SQLBUILD_DISABLE_COMPILE_REUSE=1`. - -`--no-cache` also turns the Rules cache off for that compile: built-in and custom Rules evaluate -every subject again, and the compile neither reads nor writes stored Rule results, as with -`[rules.cache] enabled = false`. A `--no-cache` compile reports `rule_cache_hits` as `0`. The next -compile without the flag reuses whatever the cache held before. - -A replayed JSON report is byte-identical to the previous one except for `compile_timings`, which -holds `project_reuse_hits`, `project_reuse_misses`, `project_reuse_bypasses`, -`project_reuse_check_ms`, and `total_ms`. Full compiles report the same reuse counters alongside -the phase timings and cache counters such as `sql_test_scan_cache_hits` and -`sql_test_scan_cache_misses`. In text mode, a replayed compile does not print the per-phase progress lines. -Lifecycle sinks receive only the invocation events for a replayed compile; per-phase `operation` -events are not emitted because no compile phase runs. - -## Static analysis - -When SQL analysis is enabled (default), compile performs static analysis on your models without connecting to the warehouse: - -- **Column inference**: Infers output columns from each model's SQL, including through CTEs, subqueries, and JOINs -- **Column contract validation**: Under the default `settings.column_contract_mode = "implicit"`, a model with declared columns and no model-level `contract` declaration checks that every declared column exists in the statically inferred query output. `explicit` mode requires `contract enforced` to activate shape checks. Explicit type enforcement remains independent and verifies inferred types when possible -- **Column lineage**: Traces which source columns flow into each output column, including transform classification. See [Column Lineage](../concepts/column-lineage.md) for details - -`sqb compile` is authoritative for mandatory correctness, contracts, lineage, and configured Rules. -Rules findings block artifact completion. Use [`sqb rules run`](rules.md) to focus on one exact -code or family without changing project configuration. - -### Contract diagnostics - -When a contract violation is found, compile reports it with source-annotated diagnostics: +Read an audit's file to see how its arguments were spliced into the SQL, without running it. +References such as `__ref("orders")` stay as written. Files are rewritten on every compile, and +files for resources that no longer exist are removed. + +## Contract diagnostics + +A model's declared columns are checked against the columns its SQL produces: ``` error[K001]: declared column 'total_cents' was not found in statically inferred output for model 'fact_orders' @@ -250,43 +136,31 @@ error[K001]: declared column 'total_cents' was not found in statically inferred = help: add total_cents to the SELECT list or correct/remove MODEL(columns (...)); MODEL(columns (...)) is validated using static SQL analysis because settings.column_contract_mode is "implicit" (the default). If this project intentionally uses columns only for metadata and audits, set [settings] column_contract_mode = "explicit"; models with contract enforced remain validated ``` -The configuration guidance is an intentional project-policy choice, not a general error suppression. Fix the query or declaration when the model is intended to have a column contract. Diagnostics for `contract enforced` models do not recommend changing the project mode because explicit model contracts remain authoritative. - -Diagnostic codes: - | Code | Meaning | |------|---------| -| `K001` | A declared column is missing from the model's query output | -| `K002` | A column's inferred type does not match the declared type | -| `K003` | A column's type could not be proven (with `type_enforcement` enabled) | -| `K013` | A string literal fills an enum-typed column of an enforced contract but is not a member of the enum (not checked on SQL Server) | - -Compile returns exit code `1` when any error-severity diagnostic is found, making it suitable for CI checks. - -## Output +| `K001` | A declared column is missing from the query output | +| `K002` | A column's inferred type doesn't match the declared type | +| `K003` | A column's type can't be proven (with `type_enforcement` on) | +| `K013` | A string literal fills an enum-typed column of an enforced contract but isn't a member of the enum (not checked on SQL Server) | -### Text output (default) +Which models are checked depends on `[settings] column_contract_mode`: -```bash -sqb compile -``` +- `implicit` (default): a model with declared columns and no `contract` setting must output every + declared column. +- `explicit`: only models with `contract enforced` are shape-checked; declared columns elsewhere + are metadata and audit targets. -``` -Compile ready 12 models +Explicit type enforcement works in both modes. Switching to `explicit` is a project policy, not a +way to silence an error: if a model is meant to have a contract, fix the query or the +declaration. Errors on `contract enforced` models don't suggest changing the mode. See +[Contracts](../concepts/models/contracts.md). -├── daily_revenue OK 6 columns -├── dim_customers OK 7 columns -├── fact_orders OK 13 columns -├── stg_customers OK 5 columns -├── stg_orders OK 6 columns -└── stg_payments OK 6 columns -... +## Output -✓ Project compiled 12 models, 1 seed, 2 functions, 0 errors, 0 warnings - Wrote: target/compiled/ -``` +### Text output -Each model shows its name, status (OK or FAIL), and column count. Models with contract errors are marked FAIL. +The default output is the tree shown at the top of this page, followed by any diagnostics. Models +with contract errors are marked `FAIL`. ### JSON output @@ -294,74 +168,190 @@ Each model shows its name, status (OK or FAIL), and column count. Models with co sqb compile --json ``` -Returns a structured report including: +| Field | Contents | +|-------|----------| +| `summary` | Model, seed, function, audit, test, error and warning counts | +| `resources` | Per model: column count, dependencies, lineage summary and compiled SQL | +| `diagnostics` | Every diagnostic, with its source location | +| `compile_timings` | Timings and cache counters; see below | +| `lineage_mode` | The lineage mode used | +| `artifacts` | Paths of the files written | -- `summary` - model, seed, function, audit, test, error, and warning counts -- `resources` - per-model details including column count, dependencies, lineage summary, and compiled SQL -- `diagnostics` - all contract violations with source locations -- `compile_timings` - timing breakdown for discovery, graph, lineage, contracts, and write phases -- `lineage_mode` - which lineage mode was used -- `artifacts` - paths to written files +If compile stops before it can build the report, for example on a model header that doesn't +parse, `--json` still prints one document with `command`, `has_errors: true`, `stopped: true` and +a `diagnostics` list holding the error's `code`, `message` and `help`. The exit code is `1`, and +progress stays on stderr. -Every `compile_timings` value is a non-negative integer. Besides the phase timings, it attributes -custom Rule cost: +`compile_timings` is one level deep, and every value is a non-negative integer. Use it to see +where a slow compile spends its time: | Key | Meaning | |-----|---------| -| `custom_rules_ms`, `custom_rules_cpu_ms` | Wall and CPU time of the custom Rules phase, including its host processes | -| `custom_rules_invocations` | Custom Rule invocations the hosts ran, one per Rule and subject | +| `discover_ms`, `graph_ms`, `lineage_ms`, `contracts_ms`, `write_ms` | Wall time of each phase | +| `built_in_rules_ms` | Wall time of the built-in Rules | +| `custom_rules_ms`, `custom_rules_cpu_ms` | Wall and CPU time of the custom Rules, including their host processes | +| `custom_rules_invocations` | Custom Rule invocations, one per Rule and subject | | `custom_rules_monitor_cpu_ms` | Host CPU spent checking Rule modules for state one subject could leave for another | -| `custom_rules_guard_cpu_ms` | Host CPU spent by the runtime guard deciding whether an audited operation, such as opening a file or importing a module, came from Rule code | +| `custom_rules_guard_cpu_ms` | Host CPU spent deciding whether an audited operation, such as opening a file, came from Rule code | | `custom_rules..invocations` | Invocations of one custom Rule | -| `custom_rules..cpu_ms` | Host CPU of those invocations, excluding the guard's checks | +| `custom_rules..cpu_ms` | Host CPU of those invocations, without the guard's checks | | `custom_rules..cache_hits` | Subjects whose stored result was reused | -| `custom_rules..cache_misses` | Subjects evaluated because no stored result applied; every subject when the Rules cache is off | - -Each selected custom Rule has the four `custom_rules..*` keys; a Rule the cache answered in -full reports `0` invocations. The keys are flat so that `compile_timings` stays one level deep, for -example `jq '.compile_timings["custom_rules.XORDERS001.cpu_ms"]'`. Invocations and CPU include a -Rule SQLBuild evaluates again in one process because its module state changed, so they can exceed -the Rule's subject count. Monitor and guard time are separate from each Rule's CPU; the hosts' -remaining CPU goes to starting Python, loading Rule modules, and reading the project. - -If compile stops on an error before it can build the report, such as a model header that does not -parse, `--json` still prints one JSON document. It has `command`, `has_errors: true`, -`stopped: true`, and a `diagnostics` list with that error's `code`, `message`, and `help`, the same -text the error shows on stderr. The exit code is still 1, and progress stays on stderr. +| `custom_rules..cache_misses` | Subjects evaluated again; every subject when the Rules cache is off | +| `rule_cache_hits`, `rule_cache_misses` | Rule results reused or evaluated again | +| `project_reuse_hits`, `project_reuse_misses`, `project_reuse_bypasses` | Whether the compile [reused an unchanged compile](#reusing-an-unchanged-compile), compiled because something changed, or was not eligible | +| `project_reuse_check_ms` | Time spent checking whether the previous compile can be reused | +| `total_ms` | Wall time of the whole compile | + +Other keys are finer-grained phase timings and cache counters. Each selected custom Rule has all +four `custom_rules..*` keys; read one with +`jq '.compile_timings["custom_rules.XORDERS001.cpu_ms"]'`. A Rule answered fully from the cache +reports `0` invocations. Invocations and CPU can exceed a Rule's subject count, because a Rule +whose module state changed is evaluated again. ## Column lineage modes -The `--lineage-mode` flag controls how column lineage is computed during compile: - | Mode | Description | |------|-------------| -| `fast` | Default. Lightweight column extraction using SQL model metadata. | -| `rich` | Full SQL analysis with transform classification and deeper tracing. Slower on large projects. | -| `none` | Skip column lineage entirely. | +| `fast` | Default. Lightweight column extraction from model metadata | +| `rich` | Full SQL analysis with transform classification and deeper tracing; slower on large projects | +| `none` | No column lineage | + +The JSON report holds a lineage summary per model, not the full column graph. Use +`sqb lineage [.]` to inspect lineage. See +[Column Lineage](../concepts/column-lineage.md). + +## Reusing an unchanged compile + +When nothing that affects the result has changed, `sqb compile` replays the previous result. It +prints one line to stderr, then the same report and exit code as before, including failed compiles +and their diagnostics: + +``` +Inputs unchanged; reused the previous compile (0.1 s) +``` + +A compile is reused only when all of these match the previous one: + +- every file and directory in the project, apart from the paths listed below; +- the command line: project and working directory, flags, selectors, `--vars`, target, and + whether color output is on; +- every `SQLBUILD_*` and `SQB_*` environment variable, every variable a template or `@@ENV` read, + and every variable a provider's settings can read, including `env_prefix`, aliases and + nested-delimiter variables (values are stored only as hashes); +- the contents of each provider settings `env_file` and `secrets_dir`; +- the Python interpreter, the SQLBuild version, `sys.path`, and every loaded Python module file + outside the project; +- the files the previous compile wrote under `target/compiled/` and the `--dag` file. If one is + missing or edited, compile runs in full and rewrites it. + +These paths are not inputs: `target/`, `logs/`, `venv/`, `.git/` and `__pycache__/` anywhere, and +`.cache`, `.fensu`, `.hg`, `.idea`, `.mypy_cache`, `.nox`, `.pytest_cache`, `.ruff_cache`, +`.sqlbuild`, `.svn`, `.tox`, `.venv` and `.vscode` at the project root. Other hidden folders are +inputs, because custom Rules can read them. DuckDB database files count only by whether they exist. + +Reuse assumes that macros, hooks, providers, adapters and custom Rules are deterministic: the same +inputs give the same output. See +[Macros must be deterministic](../concepts/macros/python-macros.md#macros-must-be-deterministic). A +compile whose templates read `run.id` is never reused. + +A replay differs from a full compile in three ways: -The JSON compile report includes a lineage summary for each model, not the full column graph. Use `sqb lineage [.]` to inspect lineage as a tree, edge list, or JSON. See [Column Lineage](../concepts/column-lineage.md) for details on analysis modes and transform types. +- In JSON, only `compile_timings` differs. It holds `project_reuse_hits`, `project_reuse_misses`, + `project_reuse_bypasses`, `project_reuse_check_ms` and `total_ms`. +- In text mode, the per-phase progress lines are not printed. +- Lifecycle sinks receive only the invocation events, because no compile phase runs. + +When storing a result for reuse takes noticeable time, for example in a project with very many +files, compile prints `Recording compile for reuse...` and a completion line on stderr before the +report. + +## Compiling after an edit + +When only model SQL files and SQL test files changed since the previous compile, `sqb compile` +compiles just those files. It checks the changed models and tests, and every model whose inputs' +columns, types or nullability the edit changed, and takes everything else from the previous +compile. Project-wide Rules still run over the whole project. It prints one line to stderr, then +the report a full compile would give: + +``` +Compiled 1 changed file(s) against the previous compile (1 model(s), 2.1s) +``` + +Compile runs in full instead when: + +- a file was added, removed or renamed, or any other file changed, such as + `sqlbuild_project.toml`, a source or schema YAML file, a macro, a seed or a function; +- an edited model changes what it references, or changes a header setting other than + `description`, `columns`, `dynamic_columns`, `model_schema`, `audits`, `contract`, or a switch + between `view` and `table`; +- a custom Rule needs to see the edit: a project-subject Rule reads something the edit changed, or + a model-subject Rule must run again for a model that was not edited (for example because it reads + an edited upstream file), or reads more for an edited model than that model's SQL, columns, + contract and grain, project files and the source list; +- the previous compile reported errors or warnings, other than Rule findings, in files the edit + didn't change; +- the previous compile used the other output format (`--json` or text), or ran with delta compiles + off; +- the edit changes which models or sources a macro reads through its arguments, changes a model's + `dynamic_columns` families, or changes the shape of so many models that a full compile is cheaper. + +A full compile after an edit still reuses work for unchanged inputs: the column analysis of models +whose query, settings and upstream columns are unchanged, the scan of unchanged SQL test files, +Rule results, and macro results. + +Either way, the output is identical to `sqb compile --no-cache`. To turn delta compiles off, set +`delta_compile = false` under `[settings]` or set `SQLBUILD_DISABLE_DELTA_COMPILE=1`. + +## Forcing a fresh compile + +Reuse notices edits from file metadata, such as size and timestamps, without reading unchanged +files. An edit can go unnoticed in these cases: + +| Situation | Why | +|-----------|-----| +| A Python package outside the project gains a submodule or data file that compile reads without importing first | Packages are tracked by the module files compile loaded | +| A network filesystem that caches file attributes, or clocks that differ between machines | Timestamps may not show a recent edit | +| Windows, or a FUSE, SMB or exFAT mount, and a rewrite that keeps the size and restores the modification time (`touch -r`, `rsync -t`, `robocopy /COPY:T`) | These filesystems have no real change time | + +In these cases, or whenever a result looks stale, compile in full: + +| To | Use | +|----|-----| +| Compile in full once | `sqb compile --no-cache` | +| Always compile in full | `SQLBUILD_DISABLE_COMPILE_REUSE=1`, `SQLBUILD_DISABLE_COMPILE_CACHE=1`, or `compile_cache = false` on the target | +| Never compile only the changed files | `[settings] delta_compile = false` or `SQLBUILD_DISABLE_DELTA_COMPILE=1` | + +`--no-cache` also turns the Rules cache off for that compile, as `[rules.cache] enabled = false` +does: every Rule evaluates every subject again, and `rule_cache_hits` is `0`. The next compile +without the flag reuses what the cache held before. + +Compile also always runs in full with `--manifest`, `--debug` or a profiling flag, in a project +without `sqlbuild_project.toml` or `sqlbuild_project.yml`, and when a provider's settings use +sources SQLBuild can't list exactly, such as a custom `settings_customise_sources` or +command-line parsing. The reason is logged at debug level. ## Examples ```bash -# Basic compile +# Compile the project sqb compile -# Compile with full JSON report +# Print the full report as JSON sqb compile --json -# Compile with rich column lineage +# Compile with rich column lineage, or none sqb compile --lineage-mode rich - -# Skip column lineage sqb compile --lineage-mode none -# Generate manifest +# Also write target/manifest.json sqb compile --manifest # Compile only selected models sqb compile fact_orders daily_revenue -# Disable SQL analysis +# Turn off SQL analysis sqb compile --no-sql-analysis + +# Ignore earlier compiles and caches +sqb compile --no-cache ``` diff --git a/website/src/content/docs/docs/cli/compile.mdx b/website/src/content/docs/docs/cli/compile.mdx index 9322d31b43..ed640b0ebb 100644 --- a/website/src/content/docs/docs/cli/compile.mdx +++ b/website/src/content/docs/docs/cli/compile.mdx @@ -5,9 +5,31 @@ sidebar: label: "compile" --- -import Note from '../../../../components/mintlify/Note.astro'; +`sqb compile` turns your project into the SQL it will run. It resolves references, expands +macros, checks SQL and column contracts, computes column lineage, runs your Rules, and writes the +compiled SQL to `target/compiled/`. It works offline and never connects to the warehouse. -Discovers the complete project, resolves references, expands macros, validates SQL, checks column contracts, computes column lineage, and writes compiled artifacts to `target/`. The compile command is fully offline—it does not connect to the warehouse. +```bash +sqb compile +``` + +``` +Compile ready 12 models + +├── daily_revenue OK 6 columns +├── dim_customers OK 7 columns +├── fact_orders OK 13 columns +├── stg_customers OK 5 columns +├── stg_orders OK 6 columns +└── stg_payments OK 6 columns +... + +✓ Project compiled 12 models, 1 seed, 2 functions, 0 errors, 0 warnings + Wrote: target/compiled/ +``` + +Each model shows its status (`OK` or `FAIL`) and column count. Compile exits with `1` when it +finds any error, so it can gate CI. ## Usage @@ -15,57 +37,48 @@ Discovers the complete project, resolves references, expands macros, validates S sqb compile [SELECTOR ...] [flags] ``` -Positional `SELECTOR` arguments work like `--select`, and both forms can be combined. `--project-dir `, `--no-color`, and `--debug` can go before or after the command name. +Positional `SELECTOR` arguments work like `--select`, and both forms can be combined. +`--project-dir `, `--no-color` and `--debug` can go before or after the command name. ## Flags | Flag | Description | |------|-------------| -| `--no-sql-analysis` | Disable SQL syntax, binding, type inference, and semantic validation (`--no-sql-validation` remains an alias) | -| `--no-cache` | Bypass the reusable compile-analysis cache and the Rules cache for this invocation | -| `--defer-to` | Resolve unselected model references against another target | -| `--json` | Output the full compile report as JSON | -| `--manifest` | Generate `target/manifest.json` with project metadata | -| `--lineage-mode` | Column lineage mode: `fast` (default), `rich` (slower, more detail), or `none` | | `SELECTOR ...` | Positional selectors, the same as `--select` | | `--select`, `-s` | Analyze and report selected resources | | `--select-file` | Read selectors from a file, one selector per line | | `--exclude` | Remove resources from the selected scope | +| `--no-sql-analysis` | Turn off SQL syntax, binding, type inference and semantic checks (`--no-sql-validation` is an alias) | +| `--no-cache` | Compile in full, without reusing earlier compiles or the Rules cache; see [Forcing a fresh compile](#forcing-a-fresh-compile) | +| `--defer-to` | Resolve unselected model references against another target | +| `--json` | Print the full compile report as JSON | +| `--manifest` | Also write `target/manifest.json` with project metadata | +| `--lineage-mode` | Column lineage mode: `fast` (default), `rich` or `none` | ## What compile does -1. **Discovery** - finds `sqlbuild_project.toml`, scans for models, sources, seeds, functions, audits, tests, and macros -2. **Graph resolution** - resolves `__ref()`, `__source()`, and `__seed()` references, expands macros, orders models by dependency -3. **SQL validation** - validates SQL syntax (when SQL analysis is enabled) -4. **Column lineage** - analyzes column-level dependencies across models (fast mode by default) -5. **Contract validation** - checks declared column contracts against inferred query output -6. **Rules** - evaluates selected native built-ins, then selected custom Python Rules. Rules that - need inferred type proofs, such as `SQBRCONTRACT105`, are skipped when SQL analysis is disabled, - and compile prints one note naming them -7. **Artifact write** - writes compiled SQL to `target/compiled/` when Rules pass +1. **Discovery.** Finds `sqlbuild_project.toml` and the project's models, sources, seeds, + functions, audits, tests and macros. +2. **Graph.** Resolves `__ref()`, `__source()` and `__seed()`, expands macros, and orders models + by dependency. +3. **SQL checks.** Parses and checks SQL, when SQL analysis is on. See + [Semantic Compilation](/docs/concepts/semantic-compilation). +4. **Column lineage.** Traces column-level dependencies across models. +5. **Contracts.** Checks declared columns against the inferred query output. +6. **Rules.** Runs the selected built-in Rules, then the selected custom Rules. +7. **Write.** Writes compiled SQL to `target/compiled/` when the Rules pass. -## Compiled artifacts +Rules that need inferred types, such as `SQBRCONTRACT105`, are skipped when SQL analysis is off, +and compile prints one note naming them. -`target/compiled/` mirrors what each resource will run, with every macro, variable and audit -argument already rendered: +`sqb compile` is authoritative for correctness, contracts, lineage and configured Rules. Rule +findings block the artifacts. To check one Rule code or family without changing project +configuration, use [`sqb rules run`](/docs/cli/rules). -| Path | Contents | -|------|----------| -| `models/.sql` | Each model's query | -| `functions//.sql` | Each function's `CREATE FUNCTION` statements | -| `audits/generic//__.sql` | Each attached generic audit, under the model, source or seed it checks; column audits end in `__`, and an audit's `name` replaces the audit name | -| `audits/singular/.sql` | Each singular audit | -| `tests/...` | Each SQL unit test's comparison query | +## Compiling a selection -Read an audit's file to see how its arguments were spliced into the SQL without running it or -looking up the warehouse's query history. Files are rewritten on every compile, and files for -resources that no longer exist are removed. References such as `__ref("orders")` stay as written, -as in compiled models. - -## Focused compilation - -Use normal selectors to limit expensive analysis and reporting while retaining complete project -discovery and reference integrity: +Selectors limit analysis and reporting. Discovery still covers the whole project, so every +reference is still checked. ```bash sqb compile fact_orders daily_revenue @@ -73,167 +86,30 @@ sqb compile --select-file selected-models.txt sqb compile --exclude deprecated_model ``` -SQLBuild deeply analyzes the selected resources and the upstream closure needed to understand them. -The text report contains only the selected scope. Project totals remain available in JSON, together -with selected model, seed, and function counts. +SQLBuild analyses the selected resources and the upstream models needed to understand them. The +text report shows only the selection. JSON also has the project totals and the selected model, seed +and function counts. The manifest and the DAG always cover the whole project. - -Manifest and DAG outputs remain full-project artifacts. Focused compilation changes analysis and -reporting scope; it does not produce a partial project graph. - +## Compiled artifacts -## Reusing an unchanged compile +`target/compiled/` holds what each resource will run, with every macro, variable and audit +argument rendered: -When nothing that can change the result has changed since the last compile, `sqb compile` replays -that result instead of compiling again. It prints one line to stderr, for example -`Inputs unchanged; reused the previous compile (0.1 s)`, then the same report and exit code as the -previous run. Failed compiles are replayed too, with the same diagnostics. - -A compile is reused only when all of these match the stored run: - -- Every file and directory under the project, compared by type, size, modification and change - times, and inode. When those differ, SQLBuild compares content only for files whose content - digest it already knows; otherwise the compile runs in full. Digests are recorded for files that - changed within two seconds of a compile and for files whose timestamps moved since the previous - compile, so large unchanged files are never read. After a checkout that only moves timestamps, - the next compile runs in full once, and later timestamp-only changes are reused. -- The command line: project directory, working directory, flags, selectors, `--vars`, target, and - whether color output is active. -- Environment variables that SQLBuild reads: every `SQLBUILD_*` and `SQB_*` variable, every - variable that a template or `@@ENV` read during the stored compile, and every variable a - provider's settings can read, including `env_prefix`, aliases, and nested-delimiter variables. - Values are stored only as hashes. -- The contents of each provider settings `env_file` and `secrets_dir`. -- The Python interpreter, the SQLBuild version and native extension, `sys.path` entries, and every - loaded Python module file outside the project. -- Every artifact the stored compile wrote under `target/compiled/`, and the `--dag` file. If one is - missing or changed, compile runs in full and rewrites it. A compile is stored only when these - artifacts still hold the bytes it wrote, so a concurrent compile cannot leave another compile's - artifacts behind a reused result. - -These project paths are not inputs: `target/`, `logs/`, `venv/`, `.git/` and `__pycache__/` -anywhere, and these folders at the project root: `.cache`, `.fensu`, `.hg`, `.idea`, -`.mypy_cache`, `.nox`, `.pytest_cache`, `.ruff_cache`, `.sqlbuild`, `.svn`, `.tox`, `.venv`, and -`.vscode`. Other hidden folders are inputs, because custom rules can read them. DuckDB database -files count only by presence. - -Reuse assumes macros, hooks, providers, adapters, and custom rules are deterministic: the same inputs -must produce the same output. Macro results are also reused within a compile and may be kept between -compiles; see [Macros must be deterministic](/docs/concepts/macros/python-macros#macros-must-be-deterministic). A -compile whose templates read `run.id` is never reused. +| Path | Contents | +|------|----------| +| `models/.sql` | Each model's query | +| `functions//.sql` | Each function's `CREATE FUNCTION` statements | +| `audits/generic//__.sql` | Each attached generic audit, under the model, source or seed it checks; column audits end in `__`, and an audit's `name` replaces the audit name | +| `audits/singular/.sql` | Each singular audit | +| `tests/...` | Each SQL unit test's comparison query | -Compile always runs in full with `--no-cache`, `--manifest`, `--debug`, or a profiling flag, when -`SQLBUILD_DISABLE_COMPILE_CACHE=1` or `SQLBUILD_DISABLE_COMPILE_REUSE=1` is set, when the project has -no `sqlbuild_project.toml` or `sqlbuild_project.yml`, when the target sets `compile_cache = false`, -and when a provider's settings use sources SQLBuild cannot list exactly, such as a custom -`settings_customise_sources` or command-line parsing. The reason is logged at debug level. - -SQLBuild keeps one stored result per project and target under -`target/cache/compiler-native-v1/project-reuse-v3/`: a small `.store` file holding what the result -depends on, and a `.output` file beside it holding the report. A check reads only the `.store` file; -the report is read only when nothing changed. Each file is replaced atomically; a damaged, missing or -unreadable file is ignored and compile runs in full. Storing a result normally takes a fraction of a second after the -report is ready. When it has to hash a lot of data or the project has very many files, compile -prints `Recording compile for reuse...` and a completion line to stderr before the report. - -### Compiling after an edit - -When only model SQL files and SQL test files changed since the stored compile, `sqb compile` -compiles just those files against it. It renders, analyses and checks the changed models and SQL -tests, plus any model whose inputs' columns, types or nullability the edit changed, and takes -everything else from a project image stored beside the stored compile. It prints one line to -stderr, for example `Compiled 1 changed file(s) against the previous compile (1 model(s), 2.1s)`, -then the same report and exit code a full compile would produce, including any errors the edit -introduces or fixes. Project-wide rules still run over the whole project. - -The compile runs in full instead, decided before any compile work starts, when: - -- A file was added, removed or renamed, or any file other than a model or SQL test changed, such - as `sqlbuild_project.toml`, a source or schema YAML file, a macro, a seed or a function. -- An edited model changes what it references, or changes a header setting other than its - `description`, `columns`, `dynamic_columns`, `model_schema`, `audits`, `contract`, or a switch - between `view` and `table` materialization. -- A project-subject custom rule reads something the edit changed, or a model-subject custom rule - must run again for a model the edit did not change, for example because it reads the text of an - edited upstream file. -- The stored compile reported errors or warnings other than rule findings in model or SQL test - files the edit did not change. -- The stored compile used the other output format (`--json` or text), or has no project image. - -A few edits are found to need a full compile only part-way through, for example when a macro's -arguments change which models or sources it reads, when the edit changes a model's -`dynamic_columns` families, or when the edit changes the shape of so many models that a full -compile is cheaper. A model-subject custom rule that must run again for an edited model runs in the -delta compile when it reads only that model's SQL, columns, contract and grain, project files, and -the source list; if it reads anything else, such as the model's dependencies, the compile runs in -full. - -Otherwise, an edit runs the whole compile again, and a few caches under `target/cache/` shorten -it: every model, test, and audit is rendered again, and contracts and semantic validation run for -the whole project. Every model's SQL references are scanned again too. - -- An unchanged model takes its column analysis from `model-analyses.bin`. Entries are keyed by - the model's query and settings, the inferred columns, types and nullability of every relation - it reads, the project's analysis settings, and the SQLBuild build, so a model whose upstream - output changed is analysed again; a damaged or unreadable entry is analysed again too. -- An unchanged SQL test file reuses its stored test scan, the CTEs and expected models read from - it, from `sql-test-scans.bin`. Entries are keyed by the file's expanded text, the adapter's SQL - lexical rules, and the SQLBuild build; a damaged or unreadable entry is scanned again. -- Rule results are reused for unchanged inputs. -- Macro call results are reused as described in - [Macros must be deterministic](/docs/concepts/macros/python-macros#macros-must-be-deterministic). - -Either way, the output is byte-identical to `sqb compile --no-cache`. To turn compiling only the -changed files off, set `delta_compile = false` under `[settings]` or set -`SQLBUILD_DISABLE_DELTA_COMPILE=1`; compiles then no longer store a project image. - -Known limits: - -- Python packages outside the project are checked by the stamps of the module files the compile - loaded. A newly added submodule or package data file that the compile reads without importing it - first is not tracked. -- On network filesystems that cache file attributes, or when clocks differ between machines, - timestamps may not reflect a recent edit. -- On Windows, a file's change time is its creation time. An in-place rewrite that keeps the same - size and restores the previous modification time, such as `touch -r`, `rsync -t`, or - `robocopy /COPY:T`, is not detected unless the file changed within two seconds of the stored - compile. The same applies to filesystems without a real change time, such as some FUSE, SMB, and - exFAT mounts. -- In these cases, or whenever a result looks stale, run `sqb compile --no-cache` or set - `SQLBUILD_DISABLE_COMPILE_REUSE=1`. - -`--no-cache` also turns the Rules cache off for that compile: built-in and custom Rules evaluate -every subject again, and the compile neither reads nor writes stored Rule results, as with -`[rules.cache] enabled = false`. A `--no-cache` compile reports `rule_cache_hits` as `0`. The next -compile without the flag reuses whatever the cache held before. - - -A replayed JSON report is byte-identical to the previous one except for `compile_timings`, which -holds `project_reuse_hits`, `project_reuse_misses`, `project_reuse_bypasses`, -`project_reuse_check_ms`, and `total_ms`. Full compiles report the same reuse counters alongside -the phase timings and cache counters such as `sql_test_scan_cache_hits` and -`sql_test_scan_cache_misses`. In text mode, a replayed compile does not print the per-phase progress lines. -Lifecycle sinks receive only the invocation events for a replayed compile; per-phase `operation` -events are not emitted because no compile phase runs. - - -## Static analysis - -When SQL analysis is enabled (default), compile performs static analysis on your models without connecting to the warehouse: - -- **Column inference**: Infers output columns from each model's SQL, including through CTEs, subqueries, and JOINs -- **Column contract validation**: Under the default `settings.column_contract_mode = "implicit"`, a model with declared columns and no model-level `contract` declaration checks that every declared column exists in the statically inferred query output. `explicit` mode requires `contract enforced` to activate shape checks. Explicit type enforcement remains independent and verifies inferred types when possible -- **Column lineage**: Traces which source columns flow into each output column, including transform classification. See [Column Lineage](/docs/concepts/column-lineage) for details - - -`sqb compile` is authoritative for mandatory correctness, contracts, lineage, and configured Rules. -Rules findings block artifact completion. Use [`sqb rules run`](/docs/cli/rules) to focus on one exact -code or family without changing project configuration. - - -### Contract diagnostics - -When a contract violation is found, compile reports it with source-annotated diagnostics: +Read an audit's file to see how its arguments were spliced into the SQL, without running it. +References such as `__ref("orders")` stay as written. Files are rewritten on every compile, and +files for resources that no longer exist are removed. + +## Contract diagnostics + +A model's declared columns are checked against the columns its SQL produces: ``` error[K001]: declared column 'total_cents' was not found in statically inferred output for model 'fact_orders' @@ -244,43 +120,31 @@ error[K001]: declared column 'total_cents' was not found in statically inferred = help: add total_cents to the SELECT list or correct/remove MODEL(columns (...)); MODEL(columns (...)) is validated using static SQL analysis because settings.column_contract_mode is "implicit" (the default). If this project intentionally uses columns only for metadata and audits, set [settings] column_contract_mode = "explicit"; models with contract enforced remain validated ``` -The configuration guidance is an intentional project-policy choice, not a general error suppression. Fix the query or declaration when the model is intended to have a column contract. Diagnostics for `contract enforced` models do not recommend changing the project mode because explicit model contracts remain authoritative. - -Diagnostic codes: - | Code | Meaning | |------|---------| -| `K001` | A declared column is missing from the model's query output | -| `K002` | A column's inferred type does not match the declared type | -| `K003` | A column's type could not be proven (with `type_enforcement` enabled) | -| `K013` | A string literal fills an enum-typed column of an enforced contract but is not a member of the enum (not checked on SQL Server) | - -Compile returns exit code `1` when any error-severity diagnostic is found, making it suitable for CI checks. +| `K001` | A declared column is missing from the query output | +| `K002` | A column's inferred type doesn't match the declared type | +| `K003` | A column's type can't be proven (with `type_enforcement` on) | +| `K013` | A string literal fills an enum-typed column of an enforced contract but isn't a member of the enum (not checked on SQL Server) | -## Output +Which models are checked depends on `[settings] column_contract_mode`: -### Text output (default) +- `implicit` (default): a model with declared columns and no `contract` setting must output every + declared column. +- `explicit`: only models with `contract enforced` are shape-checked; declared columns elsewhere + are metadata and audit targets. -```bash -sqb compile -``` +Explicit type enforcement works in both modes. Switching to `explicit` is a project policy, not a +way to silence an error: if a model is meant to have a contract, fix the query or the +declaration. Errors on `contract enforced` models don't suggest changing the mode. See +[Contracts](/docs/concepts/models/contracts). -``` -Compile ready 12 models - -├── daily_revenue OK 6 columns -├── dim_customers OK 7 columns -├── fact_orders OK 13 columns -├── stg_customers OK 5 columns -├── stg_orders OK 6 columns -└── stg_payments OK 6 columns -... +## Output -✓ Project compiled 12 models, 1 seed, 2 functions, 0 errors, 0 warnings - Wrote: target/compiled/ -``` +### Text output -Each model shows its name, status (OK or FAIL), and column count. Models with contract errors are marked FAIL. +The default output is the tree shown at the top of this page, followed by any diagnostics. Models +with contract errors are marked `FAIL`. ### JSON output @@ -288,74 +152,190 @@ Each model shows its name, status (OK or FAIL), and column count. Models with co sqb compile --json ``` -Returns a structured report including: +| Field | Contents | +|-------|----------| +| `summary` | Model, seed, function, audit, test, error and warning counts | +| `resources` | Per model: column count, dependencies, lineage summary and compiled SQL | +| `diagnostics` | Every diagnostic, with its source location | +| `compile_timings` | Timings and cache counters; see below | +| `lineage_mode` | The lineage mode used | +| `artifacts` | Paths of the files written | -- `summary` - model, seed, function, audit, test, error, and warning counts -- `resources` - per-model details including column count, dependencies, lineage summary, and compiled SQL -- `diagnostics` - all contract violations with source locations -- `compile_timings` - timing breakdown for discovery, graph, lineage, contracts, and write phases -- `lineage_mode` - which lineage mode was used -- `artifacts` - paths to written files +If compile stops before it can build the report, for example on a model header that doesn't +parse, `--json` still prints one document with `command`, `has_errors: true`, `stopped: true` and +a `diagnostics` list holding the error's `code`, `message` and `help`. The exit code is `1`, and +progress stays on stderr. -Every `compile_timings` value is a non-negative integer. Besides the phase timings, it attributes -custom Rule cost: +`compile_timings` is one level deep, and every value is a non-negative integer. Use it to see +where a slow compile spends its time: | Key | Meaning | |-----|---------| -| `custom_rules_ms`, `custom_rules_cpu_ms` | Wall and CPU time of the custom Rules phase, including its host processes | -| `custom_rules_invocations` | Custom Rule invocations the hosts ran, one per Rule and subject | +| `discover_ms`, `graph_ms`, `lineage_ms`, `contracts_ms`, `write_ms` | Wall time of each phase | +| `built_in_rules_ms` | Wall time of the built-in Rules | +| `custom_rules_ms`, `custom_rules_cpu_ms` | Wall and CPU time of the custom Rules, including their host processes | +| `custom_rules_invocations` | Custom Rule invocations, one per Rule and subject | | `custom_rules_monitor_cpu_ms` | Host CPU spent checking Rule modules for state one subject could leave for another | -| `custom_rules_guard_cpu_ms` | Host CPU spent by the runtime guard deciding whether an audited operation, such as opening a file or importing a module, came from Rule code | +| `custom_rules_guard_cpu_ms` | Host CPU spent deciding whether an audited operation, such as opening a file, came from Rule code | | `custom_rules..invocations` | Invocations of one custom Rule | -| `custom_rules..cpu_ms` | Host CPU of those invocations, excluding the guard's checks | +| `custom_rules..cpu_ms` | Host CPU of those invocations, without the guard's checks | | `custom_rules..cache_hits` | Subjects whose stored result was reused | -| `custom_rules..cache_misses` | Subjects evaluated because no stored result applied; every subject when the Rules cache is off | - -Each selected custom Rule has the four `custom_rules..*` keys; a Rule the cache answered in -full reports `0` invocations. The keys are flat so that `compile_timings` stays one level deep, for -example `jq '.compile_timings["custom_rules.XORDERS001.cpu_ms"]'`. Invocations and CPU include a -Rule SQLBuild evaluates again in one process because its module state changed, so they can exceed -the Rule's subject count. Monitor and guard time are separate from each Rule's CPU; the hosts' -remaining CPU goes to starting Python, loading Rule modules, and reading the project. - -If compile stops on an error before it can build the report, such as a model header that does not -parse, `--json` still prints one JSON document. It has `command`, `has_errors: true`, -`stopped: true`, and a `diagnostics` list with that error's `code`, `message`, and `help`, the same -text the error shows on stderr. The exit code is still 1, and progress stays on stderr. +| `custom_rules..cache_misses` | Subjects evaluated again; every subject when the Rules cache is off | +| `rule_cache_hits`, `rule_cache_misses` | Rule results reused or evaluated again | +| `project_reuse_hits`, `project_reuse_misses`, `project_reuse_bypasses` | Whether the compile [reused an unchanged compile](#reusing-an-unchanged-compile), compiled because something changed, or was not eligible | +| `project_reuse_check_ms` | Time spent checking whether the previous compile can be reused | +| `total_ms` | Wall time of the whole compile | + +Other keys are finer-grained phase timings and cache counters. Each selected custom Rule has all +four `custom_rules..*` keys; read one with +`jq '.compile_timings["custom_rules.XORDERS001.cpu_ms"]'`. A Rule answered fully from the cache +reports `0` invocations. Invocations and CPU can exceed a Rule's subject count, because a Rule +whose module state changed is evaluated again. ## Column lineage modes -The `--lineage-mode` flag controls how column lineage is computed during compile: - | Mode | Description | |------|-------------| -| `fast` | Default. Lightweight column extraction using SQL model metadata. | -| `rich` | Full SQL analysis with transform classification and deeper tracing. Slower on large projects. | -| `none` | Skip column lineage entirely. | +| `fast` | Default. Lightweight column extraction from model metadata | +| `rich` | Full SQL analysis with transform classification and deeper tracing; slower on large projects | +| `none` | No column lineage | + +The JSON report holds a lineage summary per model, not the full column graph. Use +`sqb lineage [.]` to inspect lineage. See +[Column Lineage](/docs/concepts/column-lineage). + +## Reusing an unchanged compile + +When nothing that affects the result has changed, `sqb compile` replays the previous result. It +prints one line to stderr, then the same report and exit code as before, including failed compiles +and their diagnostics: + +``` +Inputs unchanged; reused the previous compile (0.1 s) +``` + +A compile is reused only when all of these match the previous one: + +- every file and directory in the project, apart from the paths listed below; +- the command line: project and working directory, flags, selectors, `--vars`, target, and + whether color output is on; +- every `SQLBUILD_*` and `SQB_*` environment variable, every variable a template or `@@ENV` read, + and every variable a provider's settings can read, including `env_prefix`, aliases and + nested-delimiter variables (values are stored only as hashes); +- the contents of each provider settings `env_file` and `secrets_dir`; +- the Python interpreter, the SQLBuild version, `sys.path`, and every loaded Python module file + outside the project; +- the files the previous compile wrote under `target/compiled/` and the `--dag` file. If one is + missing or edited, compile runs in full and rewrites it. + +These paths are not inputs: `target/`, `logs/`, `venv/`, `.git/` and `__pycache__/` anywhere, and +`.cache`, `.fensu`, `.hg`, `.idea`, `.mypy_cache`, `.nox`, `.pytest_cache`, `.ruff_cache`, +`.sqlbuild`, `.svn`, `.tox`, `.venv` and `.vscode` at the project root. Other hidden folders are +inputs, because custom Rules can read them. DuckDB database files count only by whether they exist. + +Reuse assumes that macros, hooks, providers, adapters and custom Rules are deterministic: the same +inputs give the same output. See +[Macros must be deterministic](/docs/concepts/macros/python-macros#macros-must-be-deterministic). A +compile whose templates read `run.id` is never reused. + +A replay differs from a full compile in three ways: -The JSON compile report includes a lineage summary for each model, not the full column graph. Use `sqb lineage [.]` to inspect lineage as a tree, edge list, or JSON. See [Column Lineage](/docs/concepts/column-lineage) for details on analysis modes and transform types. +- In JSON, only `compile_timings` differs. It holds `project_reuse_hits`, `project_reuse_misses`, + `project_reuse_bypasses`, `project_reuse_check_ms` and `total_ms`. +- In text mode, the per-phase progress lines are not printed. +- Lifecycle sinks receive only the invocation events, because no compile phase runs. + +When storing a result for reuse takes noticeable time, for example in a project with very many +files, compile prints `Recording compile for reuse...` and a completion line on stderr before the +report. + +## Compiling after an edit + +When only model SQL files and SQL test files changed since the previous compile, `sqb compile` +compiles just those files. It checks the changed models and tests, and every model whose inputs' +columns, types or nullability the edit changed, and takes everything else from the previous +compile. Project-wide Rules still run over the whole project. It prints one line to stderr, then +the report a full compile would give: + +``` +Compiled 1 changed file(s) against the previous compile (1 model(s), 2.1s) +``` + +Compile runs in full instead when: + +- a file was added, removed or renamed, or any other file changed, such as + `sqlbuild_project.toml`, a source or schema YAML file, a macro, a seed or a function; +- an edited model changes what it references, or changes a header setting other than + `description`, `columns`, `dynamic_columns`, `model_schema`, `audits`, `contract`, or a switch + between `view` and `table`; +- a custom Rule needs to see the edit: a project-subject Rule reads something the edit changed, or + a model-subject Rule must run again for a model that was not edited (for example because it reads + an edited upstream file), or reads more for an edited model than that model's SQL, columns, + contract and grain, project files and the source list; +- the previous compile reported errors or warnings, other than Rule findings, in files the edit + didn't change; +- the previous compile used the other output format (`--json` or text), or ran with delta compiles + off; +- the edit changes which models or sources a macro reads through its arguments, changes a model's + `dynamic_columns` families, or changes the shape of so many models that a full compile is cheaper. + +A full compile after an edit still reuses work for unchanged inputs: the column analysis of models +whose query, settings and upstream columns are unchanged, the scan of unchanged SQL test files, +Rule results, and macro results. + +Either way, the output is identical to `sqb compile --no-cache`. To turn delta compiles off, set +`delta_compile = false` under `[settings]` or set `SQLBUILD_DISABLE_DELTA_COMPILE=1`. + +## Forcing a fresh compile + +Reuse notices edits from file metadata, such as size and timestamps, without reading unchanged +files. An edit can go unnoticed in these cases: + +| Situation | Why | +|-----------|-----| +| A Python package outside the project gains a submodule or data file that compile reads without importing first | Packages are tracked by the module files compile loaded | +| A network filesystem that caches file attributes, or clocks that differ between machines | Timestamps may not show a recent edit | +| Windows, or a FUSE, SMB or exFAT mount, and a rewrite that keeps the size and restores the modification time (`touch -r`, `rsync -t`, `robocopy /COPY:T`) | These filesystems have no real change time | + +In these cases, or whenever a result looks stale, compile in full: + +| To | Use | +|----|-----| +| Compile in full once | `sqb compile --no-cache` | +| Always compile in full | `SQLBUILD_DISABLE_COMPILE_REUSE=1`, `SQLBUILD_DISABLE_COMPILE_CACHE=1`, or `compile_cache = false` on the target | +| Never compile only the changed files | `[settings] delta_compile = false` or `SQLBUILD_DISABLE_DELTA_COMPILE=1` | + +`--no-cache` also turns the Rules cache off for that compile, as `[rules.cache] enabled = false` +does: every Rule evaluates every subject again, and `rule_cache_hits` is `0`. The next compile +without the flag reuses what the cache held before. + +Compile also always runs in full with `--manifest`, `--debug` or a profiling flag, in a project +without `sqlbuild_project.toml` or `sqlbuild_project.yml`, and when a provider's settings use +sources SQLBuild can't list exactly, such as a custom `settings_customise_sources` or +command-line parsing. The reason is logged at debug level. ## Examples ```bash -# Basic compile +# Compile the project sqb compile -# Compile with full JSON report +# Print the full report as JSON sqb compile --json -# Compile with rich column lineage +# Compile with rich column lineage, or none sqb compile --lineage-mode rich - -# Skip column lineage sqb compile --lineage-mode none -# Generate manifest +# Also write target/manifest.json sqb compile --manifest # Compile only selected models sqb compile fact_orders daily_revenue -# Disable SQL analysis +# Turn off SQL analysis sqb compile --no-sql-analysis + +# Ignore earlier compiles and caches +sqb compile --no-cache ``` From 16967b65811a51e5b9654bddbf3b4522a2dfa9e9 Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Tue, 27 Oct 2026 09:07:18 +0000 Subject: [PATCH 03/10] docs: split how unit tests work and drop history and internals from the testing page --- .../references/docs/concepts/adapters.md | 2 +- .../references/docs/concepts/testing.md | 793 +++++++++--------- .../content/docs/docs/concepts/adapters.mdx | 2 +- .../content/docs/docs/concepts/testing.mdx | 785 ++++++++--------- 4 files changed, 750 insertions(+), 832 deletions(-) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md index 1496fb99b0..beced17e68 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md @@ -102,7 +102,7 @@ Set `sql_analysis_dialect_name` to the SQL dialect name that matches your engine For full control with no inherited defaults, subclass `StrictAdapter` instead. Every method is abstract and must be implemented explicitly. SQLBuild raises a clear error listing any unimplemented methods. -A few `StrictAdapter` methods are optional and have a documented default. `create_sql_test_output` and `drop_sql_test_output` let an adapter store a unit test's actual output once so every comparison reads it (see [Single-read comparison](testing.md#single-read-comparison)). By default `create_sql_test_output` returns `None`, and SQLBuild runs the test's inline comparison query instead, so adapters that do not implement them keep working unchanged. `render_create_temporary_function` returns the call target and DDL for a session-scoped copy of a project function, which `sqb test` calls instead of the built function (see [Functions in `sqb test`](testing.md#functions-in-sqb-test)). By default it returns `None`, and tests call the built function. `render_drop_schema` returns the statements that drop a schema and everything in it, which [`sqb review`](../cli/review.md) uses to remove the schema it built into. By default it raises an error, and `sqb review` is unavailable for that adapter. +A few `StrictAdapter` methods are optional and have a documented default. `create_sql_test_output` and `drop_sql_test_output` let an adapter store a unit test's actual output once, in a temporary relation, so every comparison of that test reads it instead of recomputing the model chain. By default `create_sql_test_output` returns `None`, and SQLBuild runs the test's inline comparison query instead, so adapters that do not implement them keep working unchanged. `render_create_temporary_function` returns the call target and DDL for a session-scoped copy of a project function, which `sqb test` calls instead of the built function (see [Functions in `sqb test`](testing.md#functions-in-sqb-test)). By default it returns `None`, and tests call the built function. `render_drop_schema` returns the statements that drop a schema and everything in it, which [`sqb review`](../cli/review.md) uses to remove the schema it built into. By default it raises an error, and `sqb review` is unavailable for that adapter. ### Discovery rules diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md index 48d3b583b6..06ba631f48 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md @@ -9,32 +9,28 @@ Online: https://sqlbuild.com/docs/concepts/testing/ ## Contents - How tests work +- Test files - CTE conventions - Multi-model tests - Mocking refs and seeds - Fixture columns - Mocking table functions in model tests - Multiple expected models -- Macro-powered mocks +- Macros, enums and constants in tests - Macro mocking - Assertions - Cursor windows in tests - Empty-input tests - Test modes -- Multiple tests per file - Repeating test logic -- Test placement +- Running tests - Cost guards - Skipping SQL analysis -- Running tests - -SQLBuild supports SQL-native unit tests that validate model logic by comparing actual query results against expected values. Tests can chain across multiple models (multi-model tests), use macros for reusable mock data, and include zero-row assertions. - -For end-to-end testing across many models with physical warehouse relations, see [Scenarios](scenarios.md). - -## How tests work +- Details -A test file defines mock inputs and expected outputs using CTEs. SQLBuild substitutes the mock data into the real model SQL, executes it, and compares the result against the expected CTE using `EXCEPT` queries. Zero mismatched rows means the test passes. +A unit test checks a model's logic on rows you write. You mock the model's inputs with CTEs, +SQLBuild runs the model's real SQL over them, and the test passes when the output matches the rows +you expect: ```sql -- tests/unit/test_stg_orders.sql @@ -61,105 +57,133 @@ __expected__stg_orders AS ( ) ``` -The test: -1. Mocks the `raw__orders` source with the `__source__raw__orders` CTE -2. Runs the real `stg_orders` model SQL with the mock substituted in -3. Compares the output against `__expected__stg_orders` -4. Passes if row counts match and there are zero mismatched rows +Tests can chain several models, mock models, seeds, macros and table functions, assert properties +that must hold for every row, and test macros, UDFs and table functions directly. For end-to-end +tests over physical warehouse relations, see [Scenarios](scenarios.md). + +## How tests work + +The test above: + +1. mocks the `raw__orders` source with the `__source__raw__orders` CTE; +2. runs the real `stg_orders` SQL with the mock in place of the source; +3. compares the output with `__expected__stg_orders` in both directions; +4. passes when the row counts match and no row differs. -When an expected-output comparison fails, SQLBuild reports the unexpected and missing row counts -and best-effort samples from each direction. Samples are deliberately bounded to three rows, 12 -columns, and 120 characters per value, and pass through diagnostic redaction. They are also present -as structured `unexpected_samples` and `missing_samples` in JSON output. Sampling failure never -replaces the known test failure. +### The `TEST` header -When exactly one row exists in each difference direction and the bounded samples have the same -columns, SQLBuild also aligns the rows and reports only changed columns with their redacted actual -and expected values. It does not guess an alignment when several rows differ or sampling is -incomplete. +Every test starts with a `TEST()` header. All of its fields are optional: `TEST();` is a complete +header, and a test without a `name` is named after its file. Fields name the test, set its +[mode](#test-modes) or [cursor window](#cursor-windows-in-tests), declare +[parameters](#native-independent-cases), set a [time limit](#time-limits), or +[skip SQL analysis](#skipping-sql-analysis). -Before opening a warehouse connection, SQLBuild statically checks fixture shapes when SQL analysis -is enabled. Missing columns are reported together with the test path, fixture resource, and models -that read them. Statically provable collection-versus-scalar type conflicts identify the affected -column and suggest explicit `CAST`, `ARRAY_CONSTRUCT`, or `PARSE_JSON` expressions. SQLBuild never -invents missing values. +A test ends after its last CTE. A trailing `SELECT 1` is accepted; nothing else may follow the last +CTE. -Every test file starts with a `TEST()` header. All of its fields are optional: `TEST();` is a -complete header, and a test without a `name` is named after its file. Add fields to name the test, -set a cursor window, declare parameters, or turn off SQL analysis. +### Failure output -A test ends after its last CTE. The trailing `SELECT 1` that earlier releases required is still -accepted but no longer needed, so this file is equivalent to the first example: +When the output doesn't match, SQLBuild reports how many rows were unexpected and how many were +missing, with samples of each. Samples are limited to three rows, 12 columns and 120 characters per +value, and are redacted like other diagnostics. JSON output has them as `unexpected_samples` and +`missing_samples`. If sampling fails, the test still fails with its counts. + +When exactly one row differs in each direction and both samples have the same columns, SQLBuild +lines the two rows up and shows only the changed columns, with their actual and expected values. +With several differing rows it doesn't guess which rows belong together. + +`target/run/tests/` keeps the complete comparison query of each test, which you can run by hand. + +### Fixture checks + +When SQL analysis is on, SQLBuild checks fixture shapes before it connects to the warehouse: + +- Missing columns are reported together, with the test path, the fixture and the models that read + them. +- A collection value where a scalar is expected, or the reverse, names the column and suggests an + explicit `CAST`, `ARRAY_CONSTRUCT` or `PARSE_JSON`. + +SQLBuild never invents missing values, except the typed nulls described in +[Fixture columns](#fixture-columns). + +## Test files + +Put unit tests under `tests/unit/`. SQLBuild finds every `.sql` file there, recursively. + +``` +tests/ + unit/ + test_stg_orders.sql # model test + test_fact_orders.sql # model test with assertion + test_daily_revenue_chain.sql # multi-model test + test_line_total_cents_macro.sql # macro test + test_is_completed_order_udf.sql # UDF test + test_customer_orders_table_fn.sql # table function test + scenarios/ + ... +``` + +### Multiple tests per file + +A file can hold several `TEST()` blocks. Each block in such a file needs its own `name`; a file with +one test can leave it out. ```sql -TEST(); +TEST (name "completed_orders_only"); WITH __source__raw__orders AS ( - SELECT 1 AS id, 100 AS customer_id, 2 AS waffle_type_id, 3 AS quantity, - '2026-04-01 10:00:00' AS ordered_at, 'completed' AS status + SELECT 1 AS id, 100 AS customer_id, 'completed' AS status ), __expected__stg_orders AS ( - SELECT 1 AS order_id, 100 AS customer_id, 2 AS waffle_type_id, 3 AS quantity, - '2026-04-01 10:00:00' AS ordered_at, 'completed' AS status + SELECT 1 AS order_id, 100 AS customer_id, 'completed' AS status +) + +TEST (name "cancelled_orders_excluded"); + +WITH +__source__raw__orders AS ( + SELECT 1 AS id, 100 AS customer_id, 'cancelled' AS status +), +__expected__stg_orders AS ( + SELECT 1 AS order_id, 100 AS customer_id, 'cancelled' AS status ) -SELECT 1 ``` -Nothing other than `SELECT 1` may follow the last CTE. +Test names are unique across the project: a test can't share a name with another test, a scenario, +a model, a source, a seed, a function or a Python node. Name a test after the behaviour it checks, +such as `orders_status_rules`, rather than after the model. The cases of one +[parameterized test](#native-independent-cases) share its name. ## CTE conventions +A CTE's prefix says what it does: + | Prefix | Purpose | |--------|---------| -| `__source__` | Mock a source. Replaces `__source("")` in the model SQL. | -| `__ref__` | Mock a model. Replaces `__ref("")` in the model SQL. | -| `__seed__` | Mock a seed. Replaces `__seed("")` in the model SQL. | -| `__table_fn__` | Mock a table function in model mode. Replaces the complete `__table_fn("")(...)` invocation with the fixture relation. | -| `__expected__` | Define expected output for a model. SQLBuild resolves the model's real SQL and compares both ways on the listed columns. | -| `__assert__` | Zero-row assertion. Passes if the query returns no rows; fails with the returned rows as diagnostics. | -| `__macro__` | Mock a macro. Replaces every `@(...)` call with the mock value. | -| any other name | Helper CTE, shared by the test's mocks, helpers, `__expected__` and `__assert__` CTEs. Models never see it, so it can share a name with a model's CTE or a table. It is emitted as `__helper__`, so `__helper__` names are reserved. | - -Any CTE without one of these prefixes is treated as a helper CTE, available to mock, helper, -`__expected__`, and `__assert__` SQL in the test. Models never see helpers: a model's own CTEs and -the tables it reads keep their meaning even when a helper shares their name. A CTE inside another -test CTE must not redefine a helper's name, and a model CTE must not use a name SQLBuild generates -for the test query, such as `__ref__orders` or `__helper__expected_rows`. - -SQLBuild splits test files into CTEs and finds `__ref`, `__source`, `__seed`, `__dbt_ref`, -`__udf` and `__table_fn` calls outside comments and quoted text, in tests, scenarios, models, -functions, audits and hooks alike. Quoted text and comments follow the project adapter's rules: -backslash escapes on Snowflake, BigQuery and Databricks, `E'...'` strings on DuckDB and Postgres, -nested block comments on DuckDB, Postgres, Databricks and SQL Server, `#` line comments on BigQuery, -and `//` line comments on Snowflake. - -An `__expected__` CTE compares only the columns it lists, matched by name, so column order -does not matter and unlisted model columns are ignored. Row counts are always compared. Listing a -column the model does not output is a test error. +| `__source__` | Mock a source. Replaces `__source("")` in the model SQL | +| `__ref__` | Mock a model. Replaces `__ref("")` in the model SQL | +| `__seed__` | Mock a seed. Replaces `__seed("")` in the model SQL | +| `__table_fn__` | Mock a table function. Replaces the whole `__table_fn("")(...)` call | +| `__expected__` | Expected output of a model, compared in both directions on the listed columns | +| `__assert__` | Assertion. Passes when the query returns no rows; returned rows are reported | +| `__macro__` | Mock a macro. Replaces every `@(...)` call with the mock value | +| any other name | Helper CTE, shared by the test's mocks, helpers, `__expected__` and `__assert__` CTEs | + +An `__expected__` CTE compares only the columns it lists, matched by name. Column order +doesn't matter and unlisted model columns are ignored, but row counts are always compared. Listing +a column the model doesn't output is an error. [Scenario](scenarios.md#scenario-format) expected output is compared the same way. -SQLBuild reads test CTEs strictly and rejects other forms with a compile error that names the CTE: - -- A CTE name is an unquoted identifier of ASCII letters, digits and underscores that starts with a - letter or underscore. Quoted names such as `"recent orders"` or `` `recent orders` ``, and names - with `$` or non-ASCII characters, are errors that point at the name. -- A CTE body follows `AS` directly. Materialization hints such as `AS MATERIALIZED (...)` or - `AS NOT MATERIALIZED (...)` are not supported in test CTEs and are errors that point at the hint. -- `WITH`, `AS`, `SELECT` and the other keywords match ASCII letters in any case. -- Every column of an expected CTE (`__expected__`, `__macro_expected__`, `__udf_expected__` - or `__table_fn_expected__`) needs a name. A bare or qualified column reference such as `order_id` - or `o.order_id` names itself; any other expression, including `amount IS NULL` or - `CASE ... END`, needs an alias (` AS `). An alias without `AS`, such as - `1 order_id`, also names the column. -- An expected CTE is a plain `SELECT`, optionally combined with `UNION`, `INTERSECT` or `EXCEPT`. - Branches wrapped in parentheses and a `WITH` inside the CTE are errors; put shared rows in a - helper CTE instead. `SELECT *`, `SELECT DISTINCT *` and `t.*` are errors too, except - `SELECT * FROM __empty_fixture()` in a contracted `__expected__` CTE. - -Helper CTEs can read other helpers and mocks by CTE name, such as `__ref__stg_orders`. Shared -expected rows can therefore live in one helper that both an `__expected__` CTE and an assertion -read: +### Helper CTEs + +Models never see helpers, so a helper can share a name with a model's CTE or a table. Helpers are +emitted as `__helper__`, so `__helper__` names are reserved. A CTE inside another test CTE +must not redefine a helper's name, and a model CTE must not use a name SQLBuild generates for the +test query, such as `__ref__orders` or `__helper__expected_rows`. + +Helpers can read other helpers and mocks by CTE name, such as `__ref__stg_orders`. Shared expected +rows can therefore live in one helper that an `__expected__` CTE and an assertion both read: ```sql TEST(); @@ -181,21 +205,25 @@ __assert__no_unexpected_order_ids AS ( ) ``` -SQLBuild emits each helper and mock a query needs once, in dependency order, including mocks that -a helper reaches through another mock. +SQLBuild emits each helper and mock once, in dependency order, after the mocks and models it reads. +A helper that no `__expected__` or `__assert__` CTE reaches, directly or through other helpers and +mocks, is left out of the test query, and its references are neither checked nor run. + +### References in test CTEs -`__expected__` and `__assert__` CTEs, and the helpers they read, can call `__ref()`, -`__source()`, `__seed()` and `__dbt_ref()`. They resolve the same way in every test CTE: +`__expected__` and `__assert__` CTEs, and the helpers they read, can call `__ref()`, `__source()`, +`__seed()` and `__dbt_ref()`. They resolve the same way everywhere in a test: -- A mocked name always means the mock. `__ref("stg_orders")` reads `__ref__stg_orders` when the - test defines it, and a mocked model is neither run nor counted as a model the test covers, so - `sqb test --select stg_orders` does not select the test because of it. -- An unmocked `__ref("")` reads the model's real SQL, so the test runs that model and - `sqb test --select ` selects the test, even when only a helper or expected rows read it. -- `__source()`, `__seed()` and `__dbt_ref()` must be mocked, for example with +- **A mocked name means the mock.** `__ref("stg_orders")` reads `__ref__stg_orders` when the test + defines it. A mocked model isn't run, and `sqb test --select stg_orders` doesn't select the test + because of it. +- **An unmocked `__ref("")` runs the model's real SQL,** and `sqb test --select ` + selects the test, even when only a helper reads it. +- **`__source()`, `__seed()` and `__dbt_ref()` must be mocked,** for example with `__source__raw__orders AS (SELECT ...)`. -SQLBuild emits a helper after the mocks and models it reads: +Mocks are defined before the models a test runs, so a mock reads another mock by CTE name, such as +`FROM __ref__stg_orders`, not through `__ref("stg_orders")`: ```sql TEST(); @@ -212,18 +240,40 @@ __assert__quantities_are_positive AS ( ) ``` -Only helpers the test reads count: those its `__expected__` and `__assert__` CTEs reach, directly -or through other helpers and mocks. A helper nothing reads is left out of the test query, and its -references are neither checked nor run. Compile rejects references the test query cannot resolve -with `P013`, at the CTE or call that causes it: a `__ref()` to a model that does not exist, an -unmocked `__source()`, `__seed()` or `__dbt_ref()`, helpers or mocks that read each other in a -cycle, and a mock or `__table_fn__` fixture the test uses that reads a helper calling a reference. Mocks are defined before the models a test runs, so -a mock reads other mocks by CTE name, such as `FROM __ref__stg_orders`, rather than through -`__ref("stg_orders")`. +Compile rejects references the test query can't resolve with `P013`, at the CTE or call that causes +it: + +- a `__ref()` to a model that doesn't exist; +- an unmocked `__source()`, `__seed()` or `__dbt_ref()`; +- helpers or mocks that read each other in a cycle; +- a mock or `__table_fn__` fixture the test uses that reads a helper calling a reference. + +A malformed reference call in a helper or expected CTE, such as `__source("raw", "orders")`, is a +`P012` error at the call. Reference calls are found outside comments and quoted text, following the +project adapter's quoting and comment rules. + +### CTE syntax + +SQLBuild reads test CTEs strictly. Other forms are compile errors that name the CTE: + +| Rule | Error example | +|------|---------------| +| A CTE name is an unquoted ASCII identifier: letters, digits and underscores, not starting with a digit | `"recent orders"`, `` `recent orders` ``, names with `$` or non-ASCII characters | +| The CTE body follows `AS` directly | `AS MATERIALIZED (...)`, `AS NOT MATERIALIZED (...)` | +| Every column of an expected CTE has a name | `amount IS NULL` or `CASE ... END` without an alias | +| An expected CTE is a plain `SELECT`, optionally combined with `UNION`, `INTERSECT` or `EXCEPT` | Parenthesized branches, a `WITH` inside the CTE | +| An expected CTE lists its columns | `SELECT *`, `SELECT DISTINCT *`, `t.*` | + +Keywords such as `WITH`, `AS` and `SELECT` match in any case. Expected CTEs are +`__expected__`, `__macro_expected__`, `__udf_expected__` and `__table_fn_expected__`. A +column reference such as `order_id` or `o.order_id` names itself, and an alias without `AS`, such as +`1 order_id`, names the column too. For shared rows, use a helper CTE instead of a nested `WITH`. +The one `SELECT *` allowed is `SELECT * FROM __empty_fixture()` in a contracted `__expected__` CTE. ## Multi-model tests -Tests can span multiple models in a single file. Mock your sources, define an expected output for the model you care about, and SQLBuild automatically resolves every intermediate model using its real SQL. +A test can span several models. Mock the sources, give the expected output of the model you care +about, and SQLBuild runs every model in between from its real SQL: ```sql -- Mock two sources, assert on the final mart. @@ -245,22 +295,22 @@ __expected__fact_orders AS ( ) ``` -SQLBuild topologically sorts the expected models, resolves each intermediate model's real SQL with mocks substituted, and chains the outputs forward. Every model between the mocked sources and the expected model is computed automatically. - -Inspect that boundary without opening a warehouse connection: +Inspect what a test will run without connecting to the warehouse: ```bash sqb test --select fact_orders --inspect ``` -The resolved plan lists mocked sources, refs, seeds, real models in execution order, expected -models, and any unsatisfied leaf dependencies. A mocked ref is called out explicitly because its -real model SQL will not execute. Inspection exits non-zero when the plan contains unresolved -dependency errors. +The plan lists the mocked sources, refs and seeds, the real models in execution order, the expected +models, and any unsatisfied leaf dependencies. A mocked ref is called out, because its real SQL +won't run. Inspection exits non-zero when the plan has unresolved dependencies. + +Each real model adds to the cost of the test; see [Cost guards](#cost-guards). ## Mocking refs and seeds -You can mock models directly with `__ref__` and seeds with `__seed__`, not just sources. This skips the model's real SQL (or the seed's real CSV data) and provides controlled data instead: +Mock a model with `__ref__` or a seed with `__seed__`. The model's SQL, or the seed's +CSV data, is then replaced by your rows: ```sql TEST(); @@ -309,28 +359,32 @@ __expected__fact_orders AS ( ) ``` -You can mix `__source__`, `__ref__`, and `__seed__` mocks in the same test. As long as every leaf dependency is satisfied (either by a source mock, a ref mock, a seed mock, or by being in the expected chain), the test resolves. +`__source__`, `__ref__` and `__seed__` mocks can be mixed. The test resolves as long as every leaf +dependency is mocked or is in the expected chain. ## Fixture columns -A relation fixture can omit a column when the compiled test or scenario closure requires it and -SQLBuild knows its adapter type, unless the column is explicitly non-nullable. SQLBuild completes -that test-only fixture column with a typed null such as `CAST(NULL AS VARCHAR)`; it never changes -model SQL or warehouse defaults. Required columns declared with `nullable false` and columns with -unknown types must be supplied explicitly. When the relation's complete column set is -authoritative, misspelled or unknown supplied fixture columns are rejected. +A fixture can leave out a column the test needs when SQLBuild knows the column's type and the column +isn't declared `nullable false`. SQLBuild fills it with a typed null, such as +`CAST(NULL AS VARCHAR)`, in the test only; it never changes model SQL or warehouse defaults. +Supply these columns yourself: -A direct `NULL AS column_name` projection receives the authoritative relation type in compiled test -SQL, including expected-output CTEs for contracted models. To represent a contracted upstream with -no rows, use `SELECT * FROM __empty_fixture()` as the complete body of a `__ref__`, `__source__`, or -`__seed__` fixture CTE, or a contracted `__expected__` CTE. SQLBuild expands it to the relation's -full typed schema with a false filter. +- columns declared `nullable false`; +- columns whose type is unknown. + +When a relation's full column list is known, a misspelled or unknown fixture column is rejected. + +A `NULL AS column_name` projection gets the relation's declared type in the compiled test SQL, +including in expected CTEs for contracted models. To mock a contracted relation with no rows, write +`SELECT * FROM __empty_fixture()` as the whole body of a `__ref__`, `__source__` or `__seed__` +fixture, or of a contracted `__expected__` CTE. SQLBuild expands it to the relation's typed columns +with a false filter. ## Mocking table functions in model tests -Use `__table_fn__` when the model under test reads a managed table function but the test -should provide rows directly. SQLBuild replaces the complete invocation, including its argument -expressions, with the fixture relation. The mocked function is not deployed as a test dependency. +Use `__table_fn__` when the model under test reads a managed table function and the test +should supply the rows. SQLBuild replaces the whole call, arguments included, with the fixture. The +mocked function isn't deployed for the test. ```sql -- models/customer_order_totals.sql reads: @@ -349,17 +403,13 @@ __expected__customer_order_totals AS ( ) ``` -The fixture name must match a declared table function. Helper CTEs are available inside the fixture, -just as they are for source, ref, and seed mocks. A table-function fixture can also be used directly -inside a zero-row assertion. - -This is separate from `TEST (mode table_fn)`, which tests the table function implementation itself. -Model-mode fixtures isolate a model from that implementation; table-function mode executes and -compares the real function. +The fixture name must match a declared table function. Fixtures can read helper CTEs, and an +assertion can read a table-function fixture directly. To test the table function itself, use a +[table function test](#table-function-tests). ## Multiple expected models -A single test can assert on multiple models. SQLBuild resolves and compares each one independently: +One test can check several models. Each is compared on its own: ```sql TEST(); @@ -386,17 +436,12 @@ __expected__dim_customers AS ( ) ``` -If the expected models form a chain (e.g. `stg_orders` feeds into `fact_orders` which feeds into `dim_customers`), SQLBuild resolves them in dependency order, using the output of earlier steps as input to later ones. - -When a test defines `__expected__model_name` output, it may also use the public enums and constants -available to that model. A test that checks several models may use the public enums and constants -available to each of them. Public names are unique, so those values cannot conflict. - -Only explicit expected CTEs create grants. A matching test filename, `__ref__` mock, or nearby model path does not. Model-private declarations and macros are never granted through expected models. +When the expected models form a chain, such as `stg_orders` to `fact_orders` to `dim_customers`, +they run in dependency order, each reading the output of the one before. -## Macro-powered mocks +## Macros, enums and constants in tests -Because unit tests are written in SQL, they support macro calls. This lets you write reusable mock generators instead of copy-pasting mock data across test files: +Test SQL can call macros, so mock data can come from a reusable generator: ```sql TEST(); @@ -414,22 +459,27 @@ __expected__fact_orders AS ( ) ``` -The `@py.mock_orders()` call expands at compile time to whatever SQL the Python macro function returns. +`@py.mock_orders()` expands at compile time to the SQL the Python macro returns. -Test SQL uses macros, enums, and constants available from the test file's directory under -`tests/unit/`. Public enums and constants available to a model are also available when the test -defines `__expected__model_name` output for that model. Model-private values are not available to -tests. See [How Visibility Works](declaration-scopes/visibility.md#tests-and-expected-output). +A test can use: -A macro used only by tests must be private to them. Put it in `_sqlbuild/_macros/` in the tests' -folder, such as `tests/unit/orders/_sqlbuild/_macros/`, or in the `_sqlbuild/_macros/` of a model -the tests define `__expected__` output for. The project-wide `macros/` directory rejects a -test-only macro with `S024`, which names the folder to use, and `tests/unit/_sqlbuild/` is rejected -with `D013`. +- the macros, enums and constants visible from its directory under `tests/unit/`; +- the public enums and constants visible to each model it defines `__expected__` output for. + +Only an `__expected__` CTE grants that access. A matching file name, a `__ref__` mock or a nearby +model path doesn't. Model-private values and macros are never available to tests. See +[How Visibility Works](declaration-scopes/visibility.md#tests-and-expected-output). + +A macro used only by tests must be private to them. Put it in a `_sqlbuild/_macros/` folder among +the tests, such as `tests/unit/orders/_sqlbuild/_macros/`, or in the `_sqlbuild/_macros/` of a model +the tests define `__expected__` output for. The project `macros/` directory rejects a test-only +macro with `S024`, which names the folder to use, and `tests/unit/_sqlbuild/` is rejected with +`D013`. ## Macro mocking -When a model uses macros that you want to control in tests (e.g. target-specific logic, dynamic SQL generation), you can override their output with `__macro__` CTEs: +To control a macro's output in a test, such as target-specific logic or generated SQL, mock it with +a `__macro__` CTE: ```sql TEST(); @@ -446,23 +496,20 @@ __expected__stg_orders AS ( ) ``` -When a `__macro__` mock is defined, every call to `@py.country_filter(...)` in any model SQL resolved by the test is replaced with the mock value (`country_code = 'US'`). The macro's actual Python function is not called, and the arguments are ignored. - -Mocks of [SQL macros](macros/sql-macros.md) work the same way: `__macro__net_amount` -replaces every `@sql.net_amount(...)` call written in the model SQL with the mock value. A call -inside another SQL macro's body is expanded as usual. Macro names are unique across both kinds, so -the mock name never needs the `sql.` or `py.` prefix. +Every `@py.country_filter(...)` call in the model SQL the test runs is replaced with +`country_code = 'US'`. The macro isn't called, and its arguments are ignored. -The mock value must be a single `SELECT` with one string literal, which may be named with `AS`, such as `SELECT 'country_code = ''US''' AS country_filter`. Use doubled single quotes for quotes within the value (standard SQL escaping). - -This is useful for: -- Testing models that use target-specific macros without depending on target config -- Controlling dynamic SQL generation to produce predictable test inputs -- Isolating model logic from macro implementation details +- The mock is a single `SELECT` of one string literal, optionally named with `AS`. Escape quotes by + doubling them. +- [SQL macros](macros/sql-macros.md) are mocked the same way: `__macro__net_amount` + replaces each `@sql.net_amount(...)` call written in model SQL. A call inside another SQL macro's + body is expanded as usual. +- Macro names are unique across both kinds, so the mock name has no `sql.` or `py.` prefix. ## Assertions -Unit tests can include `__assert__` CTEs for property-based checks. An assertion passes if the query returns zero rows - any returned rows are failing examples. +An `__assert__` CTE checks a property. It passes when it returns no rows; each returned row is +a failing example. ```sql TEST(); @@ -477,48 +524,50 @@ __assert__order_ids_are_not_null AS ( ) ``` -An assertion must read a model the test runs. A mocked name always means the mock, so a test whose -`__expected__` and `__assert__` CTEs read only mocks, such as `__ref("stg_orders")` in a test that -also defines `__ref__stg_orders`, has no model to run against. Compile rejects it with `P013` -instead of skipping it. Mock the model's inputs instead, such as `__source__raw__orders` above, and -keep `__ref("stg_orders")` in the assertion, or remove the test. Earlier SQLBuild versions ran the -model's real SQL for an assertion calling `__ref()` on a mocked model, so such tests used to run. - -Assertions can be mixed with `__expected__` CTEs in the same test, or used on their own. They are useful when the natural check is "no rows should violate this rule" rather than "the output should exactly equal these rows" - for example, duplicate checks, negative-value constraints, or conditional business rules. +Use assertions when the natural check is "no row breaks this rule", such as duplicates, negative +values or conditional business rules. They can be mixed with `__expected__` CTEs or used alone. +`sqb test` and `sqb build` show each assertion as a nested check row. -During `sqb test` and `sqb build`, assertion results appear as nested check rows alongside expected comparisons. +An assertion must read a model the test runs. A test whose checks read only mocks, such as +`__ref("stg_orders")` in a test that also defines `__ref__stg_orders`, has nothing to run, and +compile rejects it with `P013`. Mock the model's inputs instead, as above, or remove the test. ## Cursor windows in tests -Models that call `__cursor_start()` and `__cursor_end()` run in a test over one wide window, never -the model's configured `cursor_start` and `cursor_end`: `1900-01-01 00:00:00` to -`2999-12-31 00:00:00` for timestamp cursors, and `-1000000000000000` to `1000000000000000` for -integer cursors. The bounds render as the same typed literals as a real run, so date arithmetic -such as `DATEADD('day', -7, __cursor_start()::DATE)` keeps working and mocked rows are not filtered -out. Microbatch models run once over the whole window, not once per batch. +Models that call `__cursor_start()` and `__cursor_end()` run in a test over one wide window, +never the model's own `cursor_start` and `cursor_end`: -To test windowing itself, declare the window in the header. Either key is optional. As in model -configuration, `cursor_start` is inclusive and `cursor_end` is exclusive: +| Cursor type | Window | +|-------------|--------| +| Timestamp | `1900-01-01 00:00:00` to `2999-12-31 00:00:00` | +| Integer | `-1000000000000000` to `1000000000000000` | + +The bounds render as the same typed literals as in a real run, so date arithmetic such as +`DATEADD('day', -7, __cursor_start()::DATE)` works and mocked rows aren't filtered out. +Microbatch models run once over the whole window. + +To test windowing itself, set the window in the header. Either key is optional; `cursor_start` is +inclusive and `cursor_end` exclusive: ```sql TEST (name "daily_orders_skips_rows_outside_window", cursor_start "2026-02-01", cursor_end "2026-02-03"); ``` -The window applies to every model the test evaluates that uses the intrinsics. Values are checked -against each model's `cursor_type` and `cursor_grain` at compile time. Invalid, misaligned, or -inverted bounds, and a window on a test whose models do not use the intrinsics, are compile errors. -The compiled test SQL shows the rendered window. +The window applies to every model in the test that uses the cursor functions, and is checked +against each model's `cursor_type` and `cursor_grain`. Invalid, misaligned or inverted bounds, and a +window on a test whose models don't use the cursor functions, are compile errors. The compiled test +SQL shows the rendered window. ## Empty-input tests -Do not write a test whose mocks are all empty (`WHERE FALSE`, `WHERE 1 = 0`, `LIMIT 0`, or -`__EMPTY_FIXTURE()`) and whose only checks are an empty `__expected__` CTE or a bare -`SELECT ... FROM __ref("")` assertion. It cannot fail for a model whose rows come from its -inputs. Rule `SQBRTEST203` rejects it, and `SQBRTEST202` does not count it toward -`min_tests_per_model`. Mock representative rows and assert concrete output instead. +A test whose mocks are all empty (`WHERE FALSE`, `WHERE 1 = 0`, `LIMIT 0` or `__EMPTY_FIXTURE()`) +and whose only checks are an empty `__expected__` CTE or a bare `SELECT ... FROM __ref("")` +assertion can't fail for a model whose rows come from its inputs. Rule `SQBRTEST203` rejects it, +and `SQBRTEST202` doesn't count it toward `min_tests_per_model`. Mock representative rows and +assert concrete output instead. -An empty-input test is legitimate when it asserts concrete output, for example a global aggregate -that must return one zero-valued summary row: +An empty-input test is fine when it asserts concrete output, such as a global aggregate that must +return one zero row: ```sql TEST (name "order_summary__empty_inputs_return_zero_row"); @@ -532,22 +581,33 @@ __expected__order_summary AS ( ) ``` -Assertions that filter for invalid rows, compare against expected rows, aggregate, or limit their -result are not treated as bare existence checks. Reviewed exceptions are listed by test name in the -project file; see [Rule options](rules/configuration-and-selection.md#rule-options). +Assertions that filter for invalid rows, compare with expected rows, aggregate or limit their result +are not bare existence checks. List reviewed exceptions by test name in the project file; see +[Rule options](rules/configuration-and-selection.md#rule-options). ## Test modes -Model tests mock sources and refs and compare model outputs. Three more modes test reusable logic -directly, without a model chain. SQLBuild infers the mode from the test's CTEs: a test that defines -`__macro_actual__` is a macro test, `__udf_actual__` a UDF test, `__table_fn_actual__` a table -function test, and any other test a model test, so the header's `mode` field is optional. -`TEST (mode ...)` states the mode explicitly; an explicit mode that contradicts the CTEs is a -compile error. +Model tests mock inputs and compare model output. Three more modes test reusable logic directly, +without a model chain: + +| Mode | Actual CTE | Expected CTE | Also allowed | +|------|-----------|--------------|--------------| +| `model` | none: the model chain | `__expected__` | `__source__`, `__ref__`, `__seed__`, `__assert__`, `__macro__` | +| `macro` | `__macro_actual__` | `__macro_expected__` | Helper CTEs, `@py.name()` and `@sql.name()` calls in actual, CTE-list `@sql.name()` calls in the `WITH` list | +| `udf` | `__udf_actual__` | `__udf_expected__` | Helper CTEs, `__udf()` calls in actual | +| `table_fn` | `__table_fn_actual__` | `__table_fn_expected__` | Helper CTEs, `__table_fn()` calls in actual | + +SQLBuild infers the mode from the actual CTE, so `TEST (mode ...)` is optional. An explicit mode +that contradicts the CTEs is a compile error, and so is a prefix from another mode, such as +`__source__` in a macro test. + +Only the actual CTE calls the macro, UDF or table function under test. Helpers don't call it +either, so a helper of a macro test must not call `__udf()` or `__table_fn()`. Expected CTEs must +not call macros, UDFs or table functions, so the expected data stays independent and easy to read. ### Macro tests -Test macro output by calling the macro in `__macro_actual__` and comparing against `__macro_expected__`: +Call the macro in `__macro_actual__` and give the result in `__macro_expected__`: ```sql TEST (mode macro, name "calculates_line_total_cents"); @@ -565,7 +625,8 @@ __macro_expected__ AS ( ) ``` -Macros are compile-time code, so macro tests expand the macro at compile time and compare the results. During `sqb build`, macro tests run before any model that uses the tested macro. +Macros expand at compile time. During `sqb build`, a macro test runs before any model that uses the +macro. [SQL macros](macros/sql-macros.md) are tested the same way. Call an expression macro in `__macro_actual__`: @@ -581,8 +642,8 @@ __macro_actual__ AS ( __macro_expected__ AS (SELECT CAST(12.5 AS DECIMAL(18, 2)) AS dollars) ``` -Call a column-list macro as an item of the `__macro_actual__` select list, and pass a helper CTE -as each `RELATION` argument: +Call a column-list macro as an item of the `__macro_actual__` select list, and pass a helper CTE as +each `RELATION` argument: ```sql TEST (mode macro, name "audit_columns_flag_updates"); @@ -598,8 +659,8 @@ __macro_expected__ AS ( ) ``` -A CTE-list macro is called in the test's `WITH` list, the one place outside `__macro_actual__` -where a macro test may call a macro, and `__macro_actual__` selects from one of its CTEs: +Call a CTE-list macro in the test's `WITH` list, and select from one of its CTEs in +`__macro_actual__`: ```sql TEST (mode macro, name "clean_orders_keeps_latest_live"); @@ -615,13 +676,13 @@ __macro_actual__ AS (SELECT order_id, updated_at FROM latest), __macro_expected__ AS (SELECT 1 AS order_id, TIMESTAMP '2026-01-02' AS updated_at) ``` -These calls may pass `__cte := name`, as in a model, to call the macro more than once or to give -its CTEs names of the test's own. Other test modes may not call a CTE-list macro in their `WITH` -list. +This is the only place outside `__macro_actual__` where a macro test may call a macro. The call may +pass `__cte := name`, as in a model, to call the macro more than once or to rename its CTEs. Other +modes can't call a CTE-list macro in their `WITH` list. ### UDF tests -Test scalar UDFs by calling them in `__udf_actual__` and comparing against `__udf_expected__`: +Call the UDF in `__udf_actual__` and give the result in `__udf_expected__`: ```sql TEST (name "detects_completed_orders"); @@ -645,46 +706,42 @@ __udf_expected__ AS ( ) ``` -UDFs are warehouse objects, so the function is created before the test runs. During `sqb build`, UDF tests run after the function is created but before any model that uses it. +During `sqb build`, a UDF test runs after the function is created and before any model that uses +it. ### Functions in `sqb test` -`sqb test` does not need a build to test the current version of a function. Before the tests run, -SQLBuild creates a session-scoped temporary copy of every project UDF and table function the -selected tests call, including functions those functions call, from the definitions in your -project. The tests call the copies, so an edited function is tested without `sqb build`, and the -built function in the target schema is never created, replaced or dropped. The copies disappear -when the test connections close. +`sqb test` tests the current version of your functions without a build. Before the tests run, +SQLBuild creates a temporary copy of every project UDF and table function the selected tests call, +and of the functions those call. The tests call the copies, so an edited function is tested without +`sqb build`, and the built function in the target schema is never created, replaced or dropped. The +copies disappear when the test connections close. | Adapter | Temporary copy | |---|---| -| DuckDB, MotherDuck | `CREATE TEMP MACRO`, called as `temp.main.`. Python UDFs are registered on each test connection. | +| DuckDB, MotherDuck | `CREATE TEMP MACRO`, called as `temp.main.`. Python UDFs are registered on each test connection | | PostgreSQL | SQL function in the session's `pg_temp` schema, called as `pg_temp.` | | Snowflake | `CREATE TEMPORARY FUNCTION __sqb_test__` in the session's current schema, for SQL and Python UDFs and table functions | | Databricks | `CREATE TEMPORARY FUNCTION __sqb_test__` for SQL functions | -| BigQuery, SQL Server, ClickHouse | Not supported; tests call the built function (ClickHouse functions are server-wide) | +| BigQuery, SQL Server, ClickHouse | No copy; tests call the built function (ClickHouse functions are server-wide) | | SQLite | Not applicable; SQLite has no SQL functions | -When an adapter cannot create a copy (a Databricks Python UDF, for example), or the warehouse -refuses to create it (the role lacks permission, or the Snowflake session has no current schema), -the test calls the built function instead. If that function has not been built either, the test -fails with an error that names the function and the reason no copy was created, and asks you to run -`sqb build` first. -A function whose callee has no copy calls the built functions too, so one test never mixes copies -and built functions. +When a copy can't be made, because the adapter doesn't support it (a Databricks Python UDF, for +example) or the warehouse refuses (the role lacks permission, or the Snowflake session has no +current schema), the test calls the built function. If that function isn't built either, the test +fails with an error naming the function and the reason, and asks you to run `sqb build` first. A +function whose callee has no copy calls the built functions too, so a test never mixes copies and +built functions. -SQLBuild decides which copies exist on the first test connection and then creates the same copies -on every other test connection, including a connection that replaces one a timeout left unusable. -If one of those connections cannot create a copy, `sqb test` stops with error `T006`, which names -the function and the connection. +Every test connection gets the same copies as the first one. If a later connection can't create a +copy, `sqb test` stops with `T006`, naming the function and the connection. -PostgreSQL temporary functions belong to one server session. Behind a transaction-mode connection -pooler, a test's statements can reach a different server session that does not hold the copies, so -run `sqb test` for projects with functions through a direct connection or a session-mode pool. +On PostgreSQL, the copies live in one server session. Run `sqb test` for projects with functions +through a direct connection or a session-mode pool, not a transaction-mode connection pooler. ### Table function tests -Test table functions by calling them in `__table_fn_actual__` and comparing against `__table_fn_expected__`: +Call the table function in `__table_fn_actual__` and give its rows in `__table_fn_expected__`: ```sql TEST (name "returns_customer_orders"); @@ -701,71 +758,20 @@ __table_fn_expected__ AS ( ) ``` -Table function tests run after the function is created. Since table functions are terminal (models cannot depend on them), these tests validate the function independently. - -### Mode rules - -Each mode has strict CTE validation: - -| Mode | Actual CTE | Expected CTE | Allowed | -|------|-----------|--------------|---------| -| `model` (no actual CTE) | existing model-chain syntax | `__expected__` | `__source__`, `__ref__`, `__seed__`, `__assert__`, `__macro__` | -| `macro` | `__macro_actual__` | `__macro_expected__` | Helper CTEs, `@py.name()` and `@sql.name()` calls in actual, CTE-list `@sql.name()` calls in the `WITH` list | -| `udf` | `__udf_actual__` | `__udf_expected__` | Helper CTEs, `__udf()` calls in actual | -| `table_fn` | `__table_fn_actual__` | `__table_fn_expected__` | Helper CTEs, `__table_fn()` calls in actual | - -CTE prefixes from other modes are not allowed. For example, `__source__` in a macro test or `__macro_actual__` in a test declared `TEST (mode model)` will produce a clear error pointing you to the right mode. Expected CTEs must not call macros, UDFs, or table functions - they should be independent, inspectable expected data. - -Helper CTEs follow the same rule in every mode: only the actual CTE calls the macro, UDF or table -function under test, so a helper of a macro test must not call `__udf()` or `__table_fn()` either. -A malformed reference call in a helper or expected CTE, such as `__source("raw", "orders")`, is a -P012 error at the call. - -## Multiple tests per file - -A single test file can contain multiple `TEST()` blocks. A test's name is its `name`, or the file -stem when `name` is omitted. Test names are globally unique: a test cannot share a name with another -test, a scenario, a model, a source, a seed, a function, or a Python node. Name the file or the block -after the behavior it checks, such as `orders_status_rules`, rather than after the model. The cases -of one parameterized test share its name. Each block in a file must have its own `name`: - -```sql -TEST (name "completed_orders_only"); - -WITH -__source__raw__orders AS ( - SELECT 1 AS id, 100 AS customer_id, 'completed' AS status -), -__expected__stg_orders AS ( - SELECT 1 AS order_id, 100 AS customer_id, 'completed' AS status -) - -TEST (name "cancelled_orders_excluded"); - -WITH -__source__raw__orders AS ( - SELECT 1 AS id, 100 AS customer_id, 'cancelled' AS status -), -__expected__stg_orders AS ( - SELECT 1 AS order_id, 100 AS customer_id, 'cancelled' AS status -) -``` - -A file with a single test can omit the `name` field. Files with multiple tests require names on every block. +Table function tests run after the function is created. Models can't depend on table functions, so +these tests check the function on its own. ## Repeating test logic -SQLBuild supports three repetition patterns: - -| Pattern | Use when | Result identity | -|---------|----------|-----------------| -| Multiple `TEST` blocks | Cases need different SQL shapes | One result per block | -| Macro or `VALUES` case table | One aggregate comparison is sufficient | One result for the table | -| Native parameters and cases | Cases share a template but need independent status | One result per named case | +| Pattern | Use when | Results | +|---------|----------|---------| +| Several `TEST` blocks | Cases need different SQL | One per block | +| A `VALUES` case table or macro | One comparison over all cases is enough | One for the table | +| Parameters and cases | Cases share a template but each needs its own result | One per named case | ### Native independent cases -Declare a typed schema and ordered named cases in one header: +Declare typed parameters and ordered, named cases in the header: ```sql TEST ( @@ -790,22 +796,20 @@ __expected__stg_orders AS ( ) ``` -The cases report independently as `order status: maps source states [completed]`, `[cancelled]`, -and `[pending]`. Authored order controls display order. Selecting `stg_orders` selects every case by -default. Add `--case` to run one named case without changing the test file: +Each case reports on its own, in the order written: `order status: maps source states [completed]`, +`[cancelled]` and `[pending]`. Selecting `stg_orders` runs every case. Run one case with `--case`: ```bash sqb test --select stg_orders --case cancelled ``` -If the case does not exist within the parent/model selection, SQLBuild reports the available case -names. +An unknown case name lists the available ones. -Supported scalar types are `string`, `integer`, `boolean`, `float`, and exact `decimal`. Decimal -values are quoted, such as `tax_rate "0.2000"`, so they never pass through binary float. Boolean -and integer remain distinct. Raw SQL and collection parameters are rejected. +Parameter types are `string`, `integer`, `boolean`, `float` and exact `decimal`. Quote decimal +values, such as `tax_rate "0.2000"`, so they never pass through binary floating point. Booleans and +integers stay distinct. Raw SQL and collection parameters are rejected. -Nullable parameters use an expanded declaration and are valid where SQL can infer the null type: +A nullable parameter uses the expanded form, and works where SQL can infer the null's type: ```sql parameters ( @@ -816,35 +820,24 @@ cases ( ) ``` -Every case provides exactly the declared parameters, and every declaration must appear as -`@param("name")` in the template. SQLBuild rejects missing, extra, incompatible, undeclared, and -unused values before execution. - -Expansion is deterministic: - -```text -parse TEST template and cases - -> render typed @param values with the active adapter - -> expand @const and @enum references - -> expand macros - -> validate SQL and compile expected models/assertions -``` +Every case gives exactly the declared parameters, and every parameter appears as `@param("name")` +in the test. Missing, extra, incompatible, undeclared and unused values are rejected before +anything runs. -Values can therefore feed macro arguments, for example -`@py.order_fixture(@param("source_status"))`. Parameters work in model, macro, UDF, and table-function -modes and preserve every expected model in multi-model tests. +Parameters are rendered first, as typed literals for the active adapter, then `@const` and `@enum` +references, then macros. A parameter can therefore be a macro argument, as in +`@py.order_fixture(@param("source_status"))`. Parameters work in every mode and keep every expected +model of a multi-model test. -Text and JSON output include parent/case identity and safe typed values. Compile JSON, manifests, -DAG checks, and compiled/runtime SQL artifacts retain source, block, case, and content-fingerprint -provenance. Changing one case changes that case's fingerprint without reassigning another case's -stable identity. +Text and JSON output show the test, the case and its typed values. Compile JSON, manifests, DAG +checks and compiled SQL record each case's source, block and fingerprint. Changing one case changes +only that case's fingerprint. -Scenarios are intentionally not parameterized. Keep one scenario per coherent business world so -capture and replay identity remains explicit. +Scenarios can't be parameterized: keep one scenario per coherent business world. ### Aggregate SQL case tables -When independent status is unnecessary, a normal SQL case table remains concise: +When cases don't need their own results, a `VALUES` table is shorter: ```sql TEST(); @@ -859,41 +852,31 @@ __expected__status_mapping AS ( ) ``` -This produces one aggregate test result. Use native cases when each row must pass or fail -independently. - -Quoted strings in `VALUES` rows may contain commas, brackets, and parentheses. SQLBuild preserves -these literals while extracting test CTEs; use normal SQL quote escaping for embedded quotes. +This gives one test result. Quoted strings in `VALUES` rows may contain commas, brackets and +parentheses; escape embedded quotes as usual in SQL. -## Test placement +## Running tests -Place unit test files under `tests/unit/` in your project directory. SQLBuild discovers all `.sql` files in this directory recursively. +Tests run during `sqb build` in DAG order, before the model they test is built. Run them on their +own with `sqb test`: -``` -tests/ - unit/ - test_stg_orders.sql # model test - test_fact_orders.sql # model test with assertion - test_daily_revenue_chain.sql # multi-model test - test_line_total_cents_macro.sql # macro test - test_is_completed_order_udf.sql # UDF test - test_customer_orders_table_fn.sql # table function test - scenarios/ - ... +```bash +sqb test +sqb test --select stg_orders ``` ## Cost guards A unit test runs the real SQL of every unmocked model between its mocks and the model under test as -one warehouse statement. That is cheap for one or two models, but the warehouse compiles and -optimises the whole chain at once, and that cost can grow much faster than the query's size. -SQLBuild therefore bounds each unit test in two ways. +one warehouse statement. The warehouse compiles and optimises that whole chain at once, and the cost +can grow much faster than the query's size. SQLBuild bounds each test in two ways. ### Real-model limit -`[settings] unit_test_max_real_models` (default `10`, also accepted in `sqlbuild_local.toml`) caps -how many real models one unit test may execute, counting the model under test. A test over the limit -fails compilation before any warehouse connection opens: +`[settings] unit_test_max_real_models` (default `10`) caps how many real models one test may run, +counting the model under test. It can also be set in `sqlbuild_local.toml`. A test over the limit +fails compilation, in `sqb test` and in `sqb build` when it runs tests, before any warehouse +connection opens: ```text error[S000]: Invalid SQL test fixtures: @@ -918,28 +901,31 @@ Options: unit_test_max_real_models = 3 ``` -Upgrade note: the limit is new, so after upgrading, a selection containing a unit test that executes -more than 10 real models fails to compile (`sqb test`, and `sqb build` when it runs tests) until the -test is mocked or the limit is raised. - -`sqb test` shows each test's real-model count on its row, such as `3 real models`, so a chain that -is growing towards the limit is visible early. Direct macro, UDF and table function tests execute -no models and are not counted. +`sqb test` shows each test's count on its row, such as `3 real models`, so a growing chain is +visible early. Macro, UDF and table function tests run no models and aren't counted. ### Time limits -Each unit test has a time limit, `30s` by default, measured from its first warehouse statement. -Rendering, warehouse queueing where the warehouse reports it, and cleanup after the comparison do not -count; see [per-item time limits](project-configuration.md#per-item-time-limits) for -each adapter. -Set it, from highest precedence, with `--test-timeout` on `sqb test` or `sqb build`, with -`TEST(timeout "45s")` in the test file, with `[targets..execution_limits] test_timeout`, or -with `[settings] test_timeout`. Each accepts a positive fixed duration or `"disabled"`. +Each unit test has a time limit, `30s` by default. The first of these that is set applies: + +1. `--test-timeout` on `sqb test` or `sqb build` +2. `TEST(timeout "45s")` in the test file +3. `[targets..execution_limits] test_timeout` +4. `[settings] test_timeout`, in `sqlbuild_local.toml` or `sqlbuild_project.toml` +5. The `30s` default -At the limit SQLBuild cancels the test's running statement, reports the test as `TIMEOUT` with the -elapsed time, the limit, where it was set and the cancellation outcome, and continues with the -remaining tests. When the test inlines real models beyond the model under test, the message also -lists them and the same options as the real-model limit: +Each takes a positive fixed duration, such as `"45s"` or `"10m"`, or `"disabled"`. + +The clock starts at the test's first warehouse statement. Rendering, warehouse queueing where the +warehouse reports it, and cleanup after the comparison don't count. See +[Per-item time limits](project-configuration.md#per-item-time-limits) for how each +adapter enforces the limit, and [Duration limit](project-configuration.md#duration-limit) +for how each adapter cancels a running statement. + +At the limit, SQLBuild cancels the running statement, reports the test as `TIMEOUT`, and continues +with the other tests. The report gives the elapsed time, the limit, where it was set, the +cancellation outcome and the query IDs it saw. When the test runs real models beyond the model +under test, it also lists them, with the same options as the real-model limit: ```text SQL unit test 'order_scans_kept' timed out after 1.0s (limit 1s). @@ -960,49 +946,22 @@ Options: in sqlbuild_project.toml (or sqlbuild_local.toml), or --test-timeout for one run. ``` -A test that runs only the model under test reports the timeout, the limit, its source and how to -change it, without the scenario advice. See -[Duration limit](project-configuration.md#duration-limit) for how each adapter cancels -a running statement. - -### Single-read comparison - -A test compares its actual output with the expected rows several ways: both row counts, rows -missing from each side, the expected-column check and, on failure, row samples. Where the warehouse -supports it, SQLBuild evaluates each compared model's actual output once into a temporary relation -and runs every comparison against it, so the chain is not recomputed for each read. Results, -messages and samples are unchanged. DuckDB and MotherDuck use a session temporary table that -SQLBuild drops after the test, including after a failure or timeout. PostgreSQL runs the test in -one transaction with an `ON COMMIT DROP` temporary table that the transaction's end removes, so the -reads stay on one server session behind a transaction-mode connection pooler. Snowflake reads the -first query's persisted result through `RESULT_SCAN`. If a stored output can no longer be read, -the test falls back to the inline comparison. If storing an output fails, SQLBuild runs the output -query on its own: when that query fails too, for example because a fixture value cannot be -converted in a column the expected rows leave out, the test errors with the warehouse's message -instead of passing on a comparison of the other columns; otherwise it compares inline. On DuckDB and -MotherDuck, an internal engine error while storing an output is a DuckDB defect rather than a -problem with the model, so the test compares inline. Other adapters compare the inline query as before. `target/run/tests/` keeps the self-contained inline comparison query for each test. +A test that runs only the model under test reports the timeout, the limit, where it was set and how +to change it, without the scenario advice. -## Skipping SQL analysis +Scenarios have their own limit, `scenario_timeout`; see +[Scenario timeouts](project-configuration.md#unit-test-and-scenario-limits). -Compile checks SQL test fixtures and assertions like model SQL. For SQL the parser cannot read, -add `sql_analysis false` to the `TEST` header; the test still runs. Tests of a model with -`sql_analysis false`, and every test in a `--no-sql-analysis` run, are not analysed. With -`[settings] require_sql_analysis = true`, the opt-out is accepted only on test SQL that cannot be -parsed (see [Requiring SQL analysis](models/configuration.md#requiring-sql-analysis)). - -## Running tests - -Tests run automatically during `sqb build` in DAG order, before their target model is materialized. +## Skipping SQL analysis -Run tests standalone: +Compile checks test fixtures and assertions like model SQL. For SQL the parser can't read, add +`sql_analysis false` to the `TEST` header; the test still runs. Tests of a model with +`sql_analysis false`, and every test in a `--no-sql-analysis` run, aren't analysed. With +`[settings] require_sql_analysis = true`, the opt-out is accepted only on test SQL that can't be +parsed; see [Requiring SQL analysis](models/configuration.md#requiring-sql-analysis). -```bash -sqb test -``` +## Details -Scope to specific models: - -```bash -sqb test --select stg_orders -``` +- On DuckDB, MotherDuck, PostgreSQL and Snowflake, a test evaluates each compared model's full + output once. If that output fails, for example because a fixture value can't be converted in a + column the expected rows leave out, the test errors with the warehouse's message. diff --git a/website/src/content/docs/docs/concepts/adapters.mdx b/website/src/content/docs/docs/concepts/adapters.mdx index 3d65adb756..b1d4a094cb 100644 --- a/website/src/content/docs/docs/concepts/adapters.mdx +++ b/website/src/content/docs/docs/concepts/adapters.mdx @@ -95,7 +95,7 @@ Set `sql_analysis_dialect_name` to the SQL dialect name that matches your engine For full control with no inherited defaults, subclass `StrictAdapter` instead. Every method is abstract and must be implemented explicitly. SQLBuild raises a clear error listing any unimplemented methods. -A few `StrictAdapter` methods are optional and have a documented default. `create_sql_test_output` and `drop_sql_test_output` let an adapter store a unit test's actual output once so every comparison reads it (see [Single-read comparison](/docs/concepts/testing#single-read-comparison)). By default `create_sql_test_output` returns `None`, and SQLBuild runs the test's inline comparison query instead, so adapters that do not implement them keep working unchanged. `render_create_temporary_function` returns the call target and DDL for a session-scoped copy of a project function, which `sqb test` calls instead of the built function (see [Functions in `sqb test`](/docs/concepts/testing#functions-in-sqb-test)). By default it returns `None`, and tests call the built function. `render_drop_schema` returns the statements that drop a schema and everything in it, which [`sqb review`](/docs/cli/review) uses to remove the schema it built into. By default it raises an error, and `sqb review` is unavailable for that adapter. +A few `StrictAdapter` methods are optional and have a documented default. `create_sql_test_output` and `drop_sql_test_output` let an adapter store a unit test's actual output once, in a temporary relation, so every comparison of that test reads it instead of recomputing the model chain. By default `create_sql_test_output` returns `None`, and SQLBuild runs the test's inline comparison query instead, so adapters that do not implement them keep working unchanged. `render_create_temporary_function` returns the call target and DDL for a session-scoped copy of a project function, which `sqb test` calls instead of the built function (see [Functions in `sqb test`](/docs/concepts/testing#functions-in-sqb-test)). By default it returns `None`, and tests call the built function. `render_drop_schema` returns the statements that drop a schema and everything in it, which [`sqb review`](/docs/cli/review) uses to remove the schema it built into. By default it raises an error, and `sqb review` is unavailable for that adapter. ### Discovery rules diff --git a/website/src/content/docs/docs/concepts/testing.mdx b/website/src/content/docs/docs/concepts/testing.mdx index cf142a25c7..0be31dc194 100644 --- a/website/src/content/docs/docs/concepts/testing.mdx +++ b/website/src/content/docs/docs/concepts/testing.mdx @@ -3,13 +3,9 @@ title: "Testing" description: "SQL unit tests and multi-model tests with macro support, assertions, and model chaining." --- -SQLBuild supports SQL-native unit tests that validate model logic by comparing actual query results against expected values. Tests can chain across multiple models (multi-model tests), use macros for reusable mock data, and include zero-row assertions. - -For end-to-end testing across many models with physical warehouse relations, see [Scenarios](/docs/concepts/scenarios). - -## How tests work - -A test file defines mock inputs and expected outputs using CTEs. SQLBuild substitutes the mock data into the real model SQL, executes it, and compares the result against the expected CTE using `EXCEPT` queries. Zero mismatched rows means the test passes. +A unit test checks a model's logic on rows you write. You mock the model's inputs with CTEs, +SQLBuild runs the model's real SQL over them, and the test passes when the output matches the rows +you expect: ```sql -- tests/unit/test_stg_orders.sql @@ -36,105 +32,133 @@ __expected__stg_orders AS ( ) ``` -The test: -1. Mocks the `raw__orders` source with the `__source__raw__orders` CTE -2. Runs the real `stg_orders` model SQL with the mock substituted in -3. Compares the output against `__expected__stg_orders` -4. Passes if row counts match and there are zero mismatched rows +Tests can chain several models, mock models, seeds, macros and table functions, assert properties +that must hold for every row, and test macros, UDFs and table functions directly. For end-to-end +tests over physical warehouse relations, see [Scenarios](/docs/concepts/scenarios). + +## How tests work + +The test above: + +1. mocks the `raw__orders` source with the `__source__raw__orders` CTE; +2. runs the real `stg_orders` SQL with the mock in place of the source; +3. compares the output with `__expected__stg_orders` in both directions; +4. passes when the row counts match and no row differs. + +### The `TEST` header + +Every test starts with a `TEST()` header. All of its fields are optional: `TEST();` is a complete +header, and a test without a `name` is named after its file. Fields name the test, set its +[mode](#test-modes) or [cursor window](#cursor-windows-in-tests), declare +[parameters](#native-independent-cases), set a [time limit](#time-limits), or +[skip SQL analysis](#skipping-sql-analysis). + +A test ends after its last CTE. A trailing `SELECT 1` is accepted; nothing else may follow the last +CTE. + +### Failure output + +When the output doesn't match, SQLBuild reports how many rows were unexpected and how many were +missing, with samples of each. Samples are limited to three rows, 12 columns and 120 characters per +value, and are redacted like other diagnostics. JSON output has them as `unexpected_samples` and +`missing_samples`. If sampling fails, the test still fails with its counts. + +When exactly one row differs in each direction and both samples have the same columns, SQLBuild +lines the two rows up and shows only the changed columns, with their actual and expected values. +With several differing rows it doesn't guess which rows belong together. + +`target/run/tests/` keeps the complete comparison query of each test, which you can run by hand. + +### Fixture checks + +When SQL analysis is on, SQLBuild checks fixture shapes before it connects to the warehouse: + +- Missing columns are reported together, with the test path, the fixture and the models that read + them. +- A collection value where a scalar is expected, or the reverse, names the column and suggests an + explicit `CAST`, `ARRAY_CONSTRUCT` or `PARSE_JSON`. + +SQLBuild never invents missing values, except the typed nulls described in +[Fixture columns](#fixture-columns). -When an expected-output comparison fails, SQLBuild reports the unexpected and missing row counts -and best-effort samples from each direction. Samples are deliberately bounded to three rows, 12 -columns, and 120 characters per value, and pass through diagnostic redaction. They are also present -as structured `unexpected_samples` and `missing_samples` in JSON output. Sampling failure never -replaces the known test failure. +## Test files -When exactly one row exists in each difference direction and the bounded samples have the same -columns, SQLBuild also aligns the rows and reports only changed columns with their redacted actual -and expected values. It does not guess an alignment when several rows differ or sampling is -incomplete. +Put unit tests under `tests/unit/`. SQLBuild finds every `.sql` file there, recursively. -Before opening a warehouse connection, SQLBuild statically checks fixture shapes when SQL analysis -is enabled. Missing columns are reported together with the test path, fixture resource, and models -that read them. Statically provable collection-versus-scalar type conflicts identify the affected -column and suggest explicit `CAST`, `ARRAY_CONSTRUCT`, or `PARSE_JSON` expressions. SQLBuild never -invents missing values. +``` +tests/ + unit/ + test_stg_orders.sql # model test + test_fact_orders.sql # model test with assertion + test_daily_revenue_chain.sql # multi-model test + test_line_total_cents_macro.sql # macro test + test_is_completed_order_udf.sql # UDF test + test_customer_orders_table_fn.sql # table function test + scenarios/ + ... +``` -Every test file starts with a `TEST()` header. All of its fields are optional: `TEST();` is a -complete header, and a test without a `name` is named after its file. Add fields to name the test, -set a cursor window, declare parameters, or turn off SQL analysis. +### Multiple tests per file -A test ends after its last CTE. The trailing `SELECT 1` that earlier releases required is still -accepted but no longer needed, so this file is equivalent to the first example: +A file can hold several `TEST()` blocks. Each block in such a file needs its own `name`; a file with +one test can leave it out. ```sql -TEST(); +TEST (name "completed_orders_only"); WITH __source__raw__orders AS ( - SELECT 1 AS id, 100 AS customer_id, 2 AS waffle_type_id, 3 AS quantity, - '2026-04-01 10:00:00' AS ordered_at, 'completed' AS status + SELECT 1 AS id, 100 AS customer_id, 'completed' AS status ), __expected__stg_orders AS ( - SELECT 1 AS order_id, 100 AS customer_id, 2 AS waffle_type_id, 3 AS quantity, - '2026-04-01 10:00:00' AS ordered_at, 'completed' AS status + SELECT 1 AS order_id, 100 AS customer_id, 'completed' AS status +) + +TEST (name "cancelled_orders_excluded"); + +WITH +__source__raw__orders AS ( + SELECT 1 AS id, 100 AS customer_id, 'cancelled' AS status +), +__expected__stg_orders AS ( + SELECT 1 AS order_id, 100 AS customer_id, 'cancelled' AS status ) -SELECT 1 ``` -Nothing other than `SELECT 1` may follow the last CTE. +Test names are unique across the project: a test can't share a name with another test, a scenario, +a model, a source, a seed, a function or a Python node. Name a test after the behaviour it checks, +such as `orders_status_rules`, rather than after the model. The cases of one +[parameterized test](#native-independent-cases) share its name. ## CTE conventions +A CTE's prefix says what it does: + | Prefix | Purpose | |--------|---------| -| `__source__` | Mock a source. Replaces `__source("")` in the model SQL. | -| `__ref__` | Mock a model. Replaces `__ref("")` in the model SQL. | -| `__seed__` | Mock a seed. Replaces `__seed("")` in the model SQL. | -| `__table_fn__` | Mock a table function in model mode. Replaces the complete `__table_fn("")(...)` invocation with the fixture relation. | -| `__expected__` | Define expected output for a model. SQLBuild resolves the model's real SQL and compares both ways on the listed columns. | -| `__assert__` | Zero-row assertion. Passes if the query returns no rows; fails with the returned rows as diagnostics. | -| `__macro__` | Mock a macro. Replaces every `@(...)` call with the mock value. | -| any other name | Helper CTE, shared by the test's mocks, helpers, `__expected__` and `__assert__` CTEs. Models never see it, so it can share a name with a model's CTE or a table. It is emitted as `__helper__`, so `__helper__` names are reserved. | - -Any CTE without one of these prefixes is treated as a helper CTE, available to mock, helper, -`__expected__`, and `__assert__` SQL in the test. Models never see helpers: a model's own CTEs and -the tables it reads keep their meaning even when a helper shares their name. A CTE inside another -test CTE must not redefine a helper's name, and a model CTE must not use a name SQLBuild generates -for the test query, such as `__ref__orders` or `__helper__expected_rows`. - -SQLBuild splits test files into CTEs and finds `__ref`, `__source`, `__seed`, `__dbt_ref`, -`__udf` and `__table_fn` calls outside comments and quoted text, in tests, scenarios, models, -functions, audits and hooks alike. Quoted text and comments follow the project adapter's rules: -backslash escapes on Snowflake, BigQuery and Databricks, `E'...'` strings on DuckDB and Postgres, -nested block comments on DuckDB, Postgres, Databricks and SQL Server, `#` line comments on BigQuery, -and `//` line comments on Snowflake. - -An `__expected__` CTE compares only the columns it lists, matched by name, so column order -does not matter and unlisted model columns are ignored. Row counts are always compared. Listing a -column the model does not output is a test error. +| `__source__` | Mock a source. Replaces `__source("")` in the model SQL | +| `__ref__` | Mock a model. Replaces `__ref("")` in the model SQL | +| `__seed__` | Mock a seed. Replaces `__seed("")` in the model SQL | +| `__table_fn__` | Mock a table function. Replaces the whole `__table_fn("")(...)` call | +| `__expected__` | Expected output of a model, compared in both directions on the listed columns | +| `__assert__` | Assertion. Passes when the query returns no rows; returned rows are reported | +| `__macro__` | Mock a macro. Replaces every `@(...)` call with the mock value | +| any other name | Helper CTE, shared by the test's mocks, helpers, `__expected__` and `__assert__` CTEs | + +An `__expected__` CTE compares only the columns it lists, matched by name. Column order +doesn't matter and unlisted model columns are ignored, but row counts are always compared. Listing +a column the model doesn't output is an error. [Scenario](/docs/concepts/scenarios#scenario-format) expected output is compared the same way. -SQLBuild reads test CTEs strictly and rejects other forms with a compile error that names the CTE: - -- A CTE name is an unquoted identifier of ASCII letters, digits and underscores that starts with a - letter or underscore. Quoted names such as `"recent orders"` or `` `recent orders` ``, and names - with `$` or non-ASCII characters, are errors that point at the name. -- A CTE body follows `AS` directly. Materialization hints such as `AS MATERIALIZED (...)` or - `AS NOT MATERIALIZED (...)` are not supported in test CTEs and are errors that point at the hint. -- `WITH`, `AS`, `SELECT` and the other keywords match ASCII letters in any case. -- Every column of an expected CTE (`__expected__`, `__macro_expected__`, `__udf_expected__` - or `__table_fn_expected__`) needs a name. A bare or qualified column reference such as `order_id` - or `o.order_id` names itself; any other expression, including `amount IS NULL` or - `CASE ... END`, needs an alias (` AS `). An alias without `AS`, such as - `1 order_id`, also names the column. -- An expected CTE is a plain `SELECT`, optionally combined with `UNION`, `INTERSECT` or `EXCEPT`. - Branches wrapped in parentheses and a `WITH` inside the CTE are errors; put shared rows in a - helper CTE instead. `SELECT *`, `SELECT DISTINCT *` and `t.*` are errors too, except - `SELECT * FROM __empty_fixture()` in a contracted `__expected__` CTE. - -Helper CTEs can read other helpers and mocks by CTE name, such as `__ref__stg_orders`. Shared -expected rows can therefore live in one helper that both an `__expected__` CTE and an assertion -read: +### Helper CTEs + +Models never see helpers, so a helper can share a name with a model's CTE or a table. Helpers are +emitted as `__helper__`, so `__helper__` names are reserved. A CTE inside another test CTE +must not redefine a helper's name, and a model CTE must not use a name SQLBuild generates for the +test query, such as `__ref__orders` or `__helper__expected_rows`. + +Helpers can read other helpers and mocks by CTE name, such as `__ref__stg_orders`. Shared expected +rows can therefore live in one helper that an `__expected__` CTE and an assertion both read: ```sql TEST(); @@ -156,21 +180,25 @@ __assert__no_unexpected_order_ids AS ( ) ``` -SQLBuild emits each helper and mock a query needs once, in dependency order, including mocks that -a helper reaches through another mock. +SQLBuild emits each helper and mock once, in dependency order, after the mocks and models it reads. +A helper that no `__expected__` or `__assert__` CTE reaches, directly or through other helpers and +mocks, is left out of the test query, and its references are neither checked nor run. -`__expected__` and `__assert__` CTEs, and the helpers they read, can call `__ref()`, -`__source()`, `__seed()` and `__dbt_ref()`. They resolve the same way in every test CTE: +### References in test CTEs -- A mocked name always means the mock. `__ref("stg_orders")` reads `__ref__stg_orders` when the - test defines it, and a mocked model is neither run nor counted as a model the test covers, so - `sqb test --select stg_orders` does not select the test because of it. -- An unmocked `__ref("")` reads the model's real SQL, so the test runs that model and - `sqb test --select ` selects the test, even when only a helper or expected rows read it. -- `__source()`, `__seed()` and `__dbt_ref()` must be mocked, for example with +`__expected__` and `__assert__` CTEs, and the helpers they read, can call `__ref()`, `__source()`, +`__seed()` and `__dbt_ref()`. They resolve the same way everywhere in a test: + +- **A mocked name means the mock.** `__ref("stg_orders")` reads `__ref__stg_orders` when the test + defines it. A mocked model isn't run, and `sqb test --select stg_orders` doesn't select the test + because of it. +- **An unmocked `__ref("")` runs the model's real SQL,** and `sqb test --select ` + selects the test, even when only a helper reads it. +- **`__source()`, `__seed()` and `__dbt_ref()` must be mocked,** for example with `__source__raw__orders AS (SELECT ...)`. -SQLBuild emits a helper after the mocks and models it reads: +Mocks are defined before the models a test runs, so a mock reads another mock by CTE name, such as +`FROM __ref__stg_orders`, not through `__ref("stg_orders")`: ```sql TEST(); @@ -187,18 +215,40 @@ __assert__quantities_are_positive AS ( ) ``` -Only helpers the test reads count: those its `__expected__` and `__assert__` CTEs reach, directly -or through other helpers and mocks. A helper nothing reads is left out of the test query, and its -references are neither checked nor run. Compile rejects references the test query cannot resolve -with `P013`, at the CTE or call that causes it: a `__ref()` to a model that does not exist, an -unmocked `__source()`, `__seed()` or `__dbt_ref()`, helpers or mocks that read each other in a -cycle, and a mock or `__table_fn__` fixture the test uses that reads a helper calling a reference. Mocks are defined before the models a test runs, so -a mock reads other mocks by CTE name, such as `FROM __ref__stg_orders`, rather than through -`__ref("stg_orders")`. +Compile rejects references the test query can't resolve with `P013`, at the CTE or call that causes +it: + +- a `__ref()` to a model that doesn't exist; +- an unmocked `__source()`, `__seed()` or `__dbt_ref()`; +- helpers or mocks that read each other in a cycle; +- a mock or `__table_fn__` fixture the test uses that reads a helper calling a reference. + +A malformed reference call in a helper or expected CTE, such as `__source("raw", "orders")`, is a +`P012` error at the call. Reference calls are found outside comments and quoted text, following the +project adapter's quoting and comment rules. + +### CTE syntax + +SQLBuild reads test CTEs strictly. Other forms are compile errors that name the CTE: + +| Rule | Error example | +|------|---------------| +| A CTE name is an unquoted ASCII identifier: letters, digits and underscores, not starting with a digit | `"recent orders"`, `` `recent orders` ``, names with `$` or non-ASCII characters | +| The CTE body follows `AS` directly | `AS MATERIALIZED (...)`, `AS NOT MATERIALIZED (...)` | +| Every column of an expected CTE has a name | `amount IS NULL` or `CASE ... END` without an alias | +| An expected CTE is a plain `SELECT`, optionally combined with `UNION`, `INTERSECT` or `EXCEPT` | Parenthesized branches, a `WITH` inside the CTE | +| An expected CTE lists its columns | `SELECT *`, `SELECT DISTINCT *`, `t.*` | + +Keywords such as `WITH`, `AS` and `SELECT` match in any case. Expected CTEs are +`__expected__`, `__macro_expected__`, `__udf_expected__` and `__table_fn_expected__`. A +column reference such as `order_id` or `o.order_id` names itself, and an alias without `AS`, such as +`1 order_id`, names the column too. For shared rows, use a helper CTE instead of a nested `WITH`. +The one `SELECT *` allowed is `SELECT * FROM __empty_fixture()` in a contracted `__expected__` CTE. ## Multi-model tests -Tests can span multiple models in a single file. Mock your sources, define an expected output for the model you care about, and SQLBuild automatically resolves every intermediate model using its real SQL. +A test can span several models. Mock the sources, give the expected output of the model you care +about, and SQLBuild runs every model in between from its real SQL: ```sql -- Mock two sources, assert on the final mart. @@ -220,22 +270,22 @@ __expected__fact_orders AS ( ) ``` -SQLBuild topologically sorts the expected models, resolves each intermediate model's real SQL with mocks substituted, and chains the outputs forward. Every model between the mocked sources and the expected model is computed automatically. - -Inspect that boundary without opening a warehouse connection: +Inspect what a test will run without connecting to the warehouse: ```bash sqb test --select fact_orders --inspect ``` -The resolved plan lists mocked sources, refs, seeds, real models in execution order, expected -models, and any unsatisfied leaf dependencies. A mocked ref is called out explicitly because its -real model SQL will not execute. Inspection exits non-zero when the plan contains unresolved -dependency errors. +The plan lists the mocked sources, refs and seeds, the real models in execution order, the expected +models, and any unsatisfied leaf dependencies. A mocked ref is called out, because its real SQL +won't run. Inspection exits non-zero when the plan has unresolved dependencies. + +Each real model adds to the cost of the test; see [Cost guards](#cost-guards). ## Mocking refs and seeds -You can mock models directly with `__ref__` and seeds with `__seed__`, not just sources. This skips the model's real SQL (or the seed's real CSV data) and provides controlled data instead: +Mock a model with `__ref__` or a seed with `__seed__`. The model's SQL, or the seed's +CSV data, is then replaced by your rows: ```sql TEST(); @@ -284,28 +334,32 @@ __expected__fact_orders AS ( ) ``` -You can mix `__source__`, `__ref__`, and `__seed__` mocks in the same test. As long as every leaf dependency is satisfied (either by a source mock, a ref mock, a seed mock, or by being in the expected chain), the test resolves. +`__source__`, `__ref__` and `__seed__` mocks can be mixed. The test resolves as long as every leaf +dependency is mocked or is in the expected chain. ## Fixture columns -A relation fixture can omit a column when the compiled test or scenario closure requires it and -SQLBuild knows its adapter type, unless the column is explicitly non-nullable. SQLBuild completes -that test-only fixture column with a typed null such as `CAST(NULL AS VARCHAR)`; it never changes -model SQL or warehouse defaults. Required columns declared with `nullable false` and columns with -unknown types must be supplied explicitly. When the relation's complete column set is -authoritative, misspelled or unknown supplied fixture columns are rejected. +A fixture can leave out a column the test needs when SQLBuild knows the column's type and the column +isn't declared `nullable false`. SQLBuild fills it with a typed null, such as +`CAST(NULL AS VARCHAR)`, in the test only; it never changes model SQL or warehouse defaults. +Supply these columns yourself: + +- columns declared `nullable false`; +- columns whose type is unknown. -A direct `NULL AS column_name` projection receives the authoritative relation type in compiled test -SQL, including expected-output CTEs for contracted models. To represent a contracted upstream with -no rows, use `SELECT * FROM __empty_fixture()` as the complete body of a `__ref__`, `__source__`, or -`__seed__` fixture CTE, or a contracted `__expected__` CTE. SQLBuild expands it to the relation's -full typed schema with a false filter. +When a relation's full column list is known, a misspelled or unknown fixture column is rejected. + +A `NULL AS column_name` projection gets the relation's declared type in the compiled test SQL, +including in expected CTEs for contracted models. To mock a contracted relation with no rows, write +`SELECT * FROM __empty_fixture()` as the whole body of a `__ref__`, `__source__` or `__seed__` +fixture, or of a contracted `__expected__` CTE. SQLBuild expands it to the relation's typed columns +with a false filter. ## Mocking table functions in model tests -Use `__table_fn__` when the model under test reads a managed table function but the test -should provide rows directly. SQLBuild replaces the complete invocation, including its argument -expressions, with the fixture relation. The mocked function is not deployed as a test dependency. +Use `__table_fn__` when the model under test reads a managed table function and the test +should supply the rows. SQLBuild replaces the whole call, arguments included, with the fixture. The +mocked function isn't deployed for the test. ```sql -- models/customer_order_totals.sql reads: @@ -324,17 +378,13 @@ __expected__customer_order_totals AS ( ) ``` -The fixture name must match a declared table function. Helper CTEs are available inside the fixture, -just as they are for source, ref, and seed mocks. A table-function fixture can also be used directly -inside a zero-row assertion. - -This is separate from `TEST (mode table_fn)`, which tests the table function implementation itself. -Model-mode fixtures isolate a model from that implementation; table-function mode executes and -compares the real function. +The fixture name must match a declared table function. Fixtures can read helper CTEs, and an +assertion can read a table-function fixture directly. To test the table function itself, use a +[table function test](#table-function-tests). ## Multiple expected models -A single test can assert on multiple models. SQLBuild resolves and compares each one independently: +One test can check several models. Each is compared on its own: ```sql TEST(); @@ -361,17 +411,12 @@ __expected__dim_customers AS ( ) ``` -If the expected models form a chain (e.g. `stg_orders` feeds into `fact_orders` which feeds into `dim_customers`), SQLBuild resolves them in dependency order, using the output of earlier steps as input to later ones. - -When a test defines `__expected__model_name` output, it may also use the public enums and constants -available to that model. A test that checks several models may use the public enums and constants -available to each of them. Public names are unique, so those values cannot conflict. +When the expected models form a chain, such as `stg_orders` to `fact_orders` to `dim_customers`, +they run in dependency order, each reading the output of the one before. -Only explicit expected CTEs create grants. A matching test filename, `__ref__` mock, or nearby model path does not. Model-private declarations and macros are never granted through expected models. +## Macros, enums and constants in tests -## Macro-powered mocks - -Because unit tests are written in SQL, they support macro calls. This lets you write reusable mock generators instead of copy-pasting mock data across test files: +Test SQL can call macros, so mock data can come from a reusable generator: ```sql TEST(); @@ -389,22 +434,27 @@ __expected__fact_orders AS ( ) ``` -The `@py.mock_orders()` call expands at compile time to whatever SQL the Python macro function returns. +`@py.mock_orders()` expands at compile time to the SQL the Python macro returns. + +A test can use: -Test SQL uses macros, enums, and constants available from the test file's directory under -`tests/unit/`. Public enums and constants available to a model are also available when the test -defines `__expected__model_name` output for that model. Model-private values are not available to -tests. See [How Visibility Works](/docs/concepts/declaration-scopes/visibility#tests-and-expected-output). +- the macros, enums and constants visible from its directory under `tests/unit/`; +- the public enums and constants visible to each model it defines `__expected__` output for. -A macro used only by tests must be private to them. Put it in `_sqlbuild/_macros/` in the tests' -folder, such as `tests/unit/orders/_sqlbuild/_macros/`, or in the `_sqlbuild/_macros/` of a model -the tests define `__expected__` output for. The project-wide `macros/` directory rejects a -test-only macro with `S024`, which names the folder to use, and `tests/unit/_sqlbuild/` is rejected -with `D013`. +Only an `__expected__` CTE grants that access. A matching file name, a `__ref__` mock or a nearby +model path doesn't. Model-private values and macros are never available to tests. See +[How Visibility Works](/docs/concepts/declaration-scopes/visibility#tests-and-expected-output). + +A macro used only by tests must be private to them. Put it in a `_sqlbuild/_macros/` folder among +the tests, such as `tests/unit/orders/_sqlbuild/_macros/`, or in the `_sqlbuild/_macros/` of a model +the tests define `__expected__` output for. The project `macros/` directory rejects a test-only +macro with `S024`, which names the folder to use, and `tests/unit/_sqlbuild/` is rejected with +`D013`. ## Macro mocking -When a model uses macros that you want to control in tests (e.g. target-specific logic, dynamic SQL generation), you can override their output with `__macro__` CTEs: +To control a macro's output in a test, such as target-specific logic or generated SQL, mock it with +a `__macro__` CTE: ```sql TEST(); @@ -421,23 +471,20 @@ __expected__stg_orders AS ( ) ``` -When a `__macro__` mock is defined, every call to `@py.country_filter(...)` in any model SQL resolved by the test is replaced with the mock value (`country_code = 'US'`). The macro's actual Python function is not called, and the arguments are ignored. - -Mocks of [SQL macros](/docs/concepts/macros/sql-macros) work the same way: `__macro__net_amount` -replaces every `@sql.net_amount(...)` call written in the model SQL with the mock value. A call -inside another SQL macro's body is expanded as usual. Macro names are unique across both kinds, so -the mock name never needs the `sql.` or `py.` prefix. - -The mock value must be a single `SELECT` with one string literal, which may be named with `AS`, such as `SELECT 'country_code = ''US''' AS country_filter`. Use doubled single quotes for quotes within the value (standard SQL escaping). +Every `@py.country_filter(...)` call in the model SQL the test runs is replaced with +`country_code = 'US'`. The macro isn't called, and its arguments are ignored. -This is useful for: -- Testing models that use target-specific macros without depending on target config -- Controlling dynamic SQL generation to produce predictable test inputs -- Isolating model logic from macro implementation details +- The mock is a single `SELECT` of one string literal, optionally named with `AS`. Escape quotes by + doubling them. +- [SQL macros](/docs/concepts/macros/sql-macros) are mocked the same way: `__macro__net_amount` + replaces each `@sql.net_amount(...)` call written in model SQL. A call inside another SQL macro's + body is expanded as usual. +- Macro names are unique across both kinds, so the mock name has no `sql.` or `py.` prefix. ## Assertions -Unit tests can include `__assert__` CTEs for property-based checks. An assertion passes if the query returns zero rows - any returned rows are failing examples. +An `__assert__` CTE checks a property. It passes when it returns no rows; each returned row is +a failing example. ```sql TEST(); @@ -452,48 +499,50 @@ __assert__order_ids_are_not_null AS ( ) ``` -An assertion must read a model the test runs. A mocked name always means the mock, so a test whose -`__expected__` and `__assert__` CTEs read only mocks, such as `__ref("stg_orders")` in a test that -also defines `__ref__stg_orders`, has no model to run against. Compile rejects it with `P013` -instead of skipping it. Mock the model's inputs instead, such as `__source__raw__orders` above, and -keep `__ref("stg_orders")` in the assertion, or remove the test. Earlier SQLBuild versions ran the -model's real SQL for an assertion calling `__ref()` on a mocked model, so such tests used to run. +Use assertions when the natural check is "no row breaks this rule", such as duplicates, negative +values or conditional business rules. They can be mixed with `__expected__` CTEs or used alone. +`sqb test` and `sqb build` show each assertion as a nested check row. -Assertions can be mixed with `__expected__` CTEs in the same test, or used on their own. They are useful when the natural check is "no rows should violate this rule" rather than "the output should exactly equal these rows" - for example, duplicate checks, negative-value constraints, or conditional business rules. - -During `sqb test` and `sqb build`, assertion results appear as nested check rows alongside expected comparisons. +An assertion must read a model the test runs. A test whose checks read only mocks, such as +`__ref("stg_orders")` in a test that also defines `__ref__stg_orders`, has nothing to run, and +compile rejects it with `P013`. Mock the model's inputs instead, as above, or remove the test. ## Cursor windows in tests -Models that call `__cursor_start()` and `__cursor_end()` run in a test over one wide window, never -the model's configured `cursor_start` and `cursor_end`: `1900-01-01 00:00:00` to -`2999-12-31 00:00:00` for timestamp cursors, and `-1000000000000000` to `1000000000000000` for -integer cursors. The bounds render as the same typed literals as a real run, so date arithmetic -such as `DATEADD('day', -7, __cursor_start()::DATE)` keeps working and mocked rows are not filtered -out. Microbatch models run once over the whole window, not once per batch. +Models that call `__cursor_start()` and `__cursor_end()` run in a test over one wide window, +never the model's own `cursor_start` and `cursor_end`: + +| Cursor type | Window | +|-------------|--------| +| Timestamp | `1900-01-01 00:00:00` to `2999-12-31 00:00:00` | +| Integer | `-1000000000000000` to `1000000000000000` | -To test windowing itself, declare the window in the header. Either key is optional. As in model -configuration, `cursor_start` is inclusive and `cursor_end` is exclusive: +The bounds render as the same typed literals as in a real run, so date arithmetic such as +`DATEADD('day', -7, __cursor_start()::DATE)` works and mocked rows aren't filtered out. +Microbatch models run once over the whole window. + +To test windowing itself, set the window in the header. Either key is optional; `cursor_start` is +inclusive and `cursor_end` exclusive: ```sql TEST (name "daily_orders_skips_rows_outside_window", cursor_start "2026-02-01", cursor_end "2026-02-03"); ``` -The window applies to every model the test evaluates that uses the intrinsics. Values are checked -against each model's `cursor_type` and `cursor_grain` at compile time. Invalid, misaligned, or -inverted bounds, and a window on a test whose models do not use the intrinsics, are compile errors. -The compiled test SQL shows the rendered window. +The window applies to every model in the test that uses the cursor functions, and is checked +against each model's `cursor_type` and `cursor_grain`. Invalid, misaligned or inverted bounds, and a +window on a test whose models don't use the cursor functions, are compile errors. The compiled test +SQL shows the rendered window. ## Empty-input tests -Do not write a test whose mocks are all empty (`WHERE FALSE`, `WHERE 1 = 0`, `LIMIT 0`, or -`__EMPTY_FIXTURE()`) and whose only checks are an empty `__expected__` CTE or a bare -`SELECT ... FROM __ref("")` assertion. It cannot fail for a model whose rows come from its -inputs. Rule `SQBRTEST203` rejects it, and `SQBRTEST202` does not count it toward -`min_tests_per_model`. Mock representative rows and assert concrete output instead. +A test whose mocks are all empty (`WHERE FALSE`, `WHERE 1 = 0`, `LIMIT 0` or `__EMPTY_FIXTURE()`) +and whose only checks are an empty `__expected__` CTE or a bare `SELECT ... FROM __ref("")` +assertion can't fail for a model whose rows come from its inputs. Rule `SQBRTEST203` rejects it, +and `SQBRTEST202` doesn't count it toward `min_tests_per_model`. Mock representative rows and +assert concrete output instead. -An empty-input test is legitimate when it asserts concrete output, for example a global aggregate -that must return one zero-valued summary row: +An empty-input test is fine when it asserts concrete output, such as a global aggregate that must +return one zero row: ```sql TEST (name "order_summary__empty_inputs_return_zero_row"); @@ -507,22 +556,33 @@ __expected__order_summary AS ( ) ``` -Assertions that filter for invalid rows, compare against expected rows, aggregate, or limit their -result are not treated as bare existence checks. Reviewed exceptions are listed by test name in the -project file; see [Rule options](/docs/concepts/rules/configuration-and-selection#rule-options). +Assertions that filter for invalid rows, compare with expected rows, aggregate or limit their result +are not bare existence checks. List reviewed exceptions by test name in the project file; see +[Rule options](/docs/concepts/rules/configuration-and-selection#rule-options). ## Test modes -Model tests mock sources and refs and compare model outputs. Three more modes test reusable logic -directly, without a model chain. SQLBuild infers the mode from the test's CTEs: a test that defines -`__macro_actual__` is a macro test, `__udf_actual__` a UDF test, `__table_fn_actual__` a table -function test, and any other test a model test, so the header's `mode` field is optional. -`TEST (mode ...)` states the mode explicitly; an explicit mode that contradicts the CTEs is a -compile error. +Model tests mock inputs and compare model output. Three more modes test reusable logic directly, +without a model chain: + +| Mode | Actual CTE | Expected CTE | Also allowed | +|------|-----------|--------------|--------------| +| `model` | none: the model chain | `__expected__` | `__source__`, `__ref__`, `__seed__`, `__assert__`, `__macro__` | +| `macro` | `__macro_actual__` | `__macro_expected__` | Helper CTEs, `@py.name()` and `@sql.name()` calls in actual, CTE-list `@sql.name()` calls in the `WITH` list | +| `udf` | `__udf_actual__` | `__udf_expected__` | Helper CTEs, `__udf()` calls in actual | +| `table_fn` | `__table_fn_actual__` | `__table_fn_expected__` | Helper CTEs, `__table_fn()` calls in actual | + +SQLBuild infers the mode from the actual CTE, so `TEST (mode ...)` is optional. An explicit mode +that contradicts the CTEs is a compile error, and so is a prefix from another mode, such as +`__source__` in a macro test. + +Only the actual CTE calls the macro, UDF or table function under test. Helpers don't call it +either, so a helper of a macro test must not call `__udf()` or `__table_fn()`. Expected CTEs must +not call macros, UDFs or table functions, so the expected data stays independent and easy to read. ### Macro tests -Test macro output by calling the macro in `__macro_actual__` and comparing against `__macro_expected__`: +Call the macro in `__macro_actual__` and give the result in `__macro_expected__`: ```sql TEST (mode macro, name "calculates_line_total_cents"); @@ -540,7 +600,8 @@ __macro_expected__ AS ( ) ``` -Macros are compile-time code, so macro tests expand the macro at compile time and compare the results. During `sqb build`, macro tests run before any model that uses the tested macro. +Macros expand at compile time. During `sqb build`, a macro test runs before any model that uses the +macro. [SQL macros](/docs/concepts/macros/sql-macros) are tested the same way. Call an expression macro in `__macro_actual__`: @@ -556,8 +617,8 @@ __macro_actual__ AS ( __macro_expected__ AS (SELECT CAST(12.5 AS DECIMAL(18, 2)) AS dollars) ``` -Call a column-list macro as an item of the `__macro_actual__` select list, and pass a helper CTE -as each `RELATION` argument: +Call a column-list macro as an item of the `__macro_actual__` select list, and pass a helper CTE as +each `RELATION` argument: ```sql TEST (mode macro, name "audit_columns_flag_updates"); @@ -573,8 +634,8 @@ __macro_expected__ AS ( ) ``` -A CTE-list macro is called in the test's `WITH` list, the one place outside `__macro_actual__` -where a macro test may call a macro, and `__macro_actual__` selects from one of its CTEs: +Call a CTE-list macro in the test's `WITH` list, and select from one of its CTEs in +`__macro_actual__`: ```sql TEST (mode macro, name "clean_orders_keeps_latest_live"); @@ -590,13 +651,13 @@ __macro_actual__ AS (SELECT order_id, updated_at FROM latest), __macro_expected__ AS (SELECT 1 AS order_id, TIMESTAMP '2026-01-02' AS updated_at) ``` -These calls may pass `__cte := name`, as in a model, to call the macro more than once or to give -its CTEs names of the test's own. Other test modes may not call a CTE-list macro in their `WITH` -list. +This is the only place outside `__macro_actual__` where a macro test may call a macro. The call may +pass `__cte := name`, as in a model, to call the macro more than once or to rename its CTEs. Other +modes can't call a CTE-list macro in their `WITH` list. ### UDF tests -Test scalar UDFs by calling them in `__udf_actual__` and comparing against `__udf_expected__`: +Call the UDF in `__udf_actual__` and give the result in `__udf_expected__`: ```sql TEST (name "detects_completed_orders"); @@ -620,46 +681,42 @@ __udf_expected__ AS ( ) ``` -UDFs are warehouse objects, so the function is created before the test runs. During `sqb build`, UDF tests run after the function is created but before any model that uses it. +During `sqb build`, a UDF test runs after the function is created and before any model that uses +it. ### Functions in `sqb test` -`sqb test` does not need a build to test the current version of a function. Before the tests run, -SQLBuild creates a session-scoped temporary copy of every project UDF and table function the -selected tests call, including functions those functions call, from the definitions in your -project. The tests call the copies, so an edited function is tested without `sqb build`, and the -built function in the target schema is never created, replaced or dropped. The copies disappear -when the test connections close. +`sqb test` tests the current version of your functions without a build. Before the tests run, +SQLBuild creates a temporary copy of every project UDF and table function the selected tests call, +and of the functions those call. The tests call the copies, so an edited function is tested without +`sqb build`, and the built function in the target schema is never created, replaced or dropped. The +copies disappear when the test connections close. | Adapter | Temporary copy | |---|---| -| DuckDB, MotherDuck | `CREATE TEMP MACRO`, called as `temp.main.`. Python UDFs are registered on each test connection. | +| DuckDB, MotherDuck | `CREATE TEMP MACRO`, called as `temp.main.`. Python UDFs are registered on each test connection | | PostgreSQL | SQL function in the session's `pg_temp` schema, called as `pg_temp.` | | Snowflake | `CREATE TEMPORARY FUNCTION __sqb_test__` in the session's current schema, for SQL and Python UDFs and table functions | | Databricks | `CREATE TEMPORARY FUNCTION __sqb_test__` for SQL functions | -| BigQuery, SQL Server, ClickHouse | Not supported; tests call the built function (ClickHouse functions are server-wide) | +| BigQuery, SQL Server, ClickHouse | No copy; tests call the built function (ClickHouse functions are server-wide) | | SQLite | Not applicable; SQLite has no SQL functions | -When an adapter cannot create a copy (a Databricks Python UDF, for example), or the warehouse -refuses to create it (the role lacks permission, or the Snowflake session has no current schema), -the test calls the built function instead. If that function has not been built either, the test -fails with an error that names the function and the reason no copy was created, and asks you to run -`sqb build` first. -A function whose callee has no copy calls the built functions too, so one test never mixes copies -and built functions. +When a copy can't be made, because the adapter doesn't support it (a Databricks Python UDF, for +example) or the warehouse refuses (the role lacks permission, or the Snowflake session has no +current schema), the test calls the built function. If that function isn't built either, the test +fails with an error naming the function and the reason, and asks you to run `sqb build` first. A +function whose callee has no copy calls the built functions too, so a test never mixes copies and +built functions. -SQLBuild decides which copies exist on the first test connection and then creates the same copies -on every other test connection, including a connection that replaces one a timeout left unusable. -If one of those connections cannot create a copy, `sqb test` stops with error `T006`, which names -the function and the connection. +Every test connection gets the same copies as the first one. If a later connection can't create a +copy, `sqb test` stops with `T006`, naming the function and the connection. -PostgreSQL temporary functions belong to one server session. Behind a transaction-mode connection -pooler, a test's statements can reach a different server session that does not hold the copies, so -run `sqb test` for projects with functions through a direct connection or a session-mode pool. +On PostgreSQL, the copies live in one server session. Run `sqb test` for projects with functions +through a direct connection or a session-mode pool, not a transaction-mode connection pooler. ### Table function tests -Test table functions by calling them in `__table_fn_actual__` and comparing against `__table_fn_expected__`: +Call the table function in `__table_fn_actual__` and give its rows in `__table_fn_expected__`: ```sql TEST (name "returns_customer_orders"); @@ -676,71 +733,20 @@ __table_fn_expected__ AS ( ) ``` -Table function tests run after the function is created. Since table functions are terminal (models cannot depend on them), these tests validate the function independently. - -### Mode rules - -Each mode has strict CTE validation: - -| Mode | Actual CTE | Expected CTE | Allowed | -|------|-----------|--------------|---------| -| `model` (no actual CTE) | existing model-chain syntax | `__expected__` | `__source__`, `__ref__`, `__seed__`, `__assert__`, `__macro__` | -| `macro` | `__macro_actual__` | `__macro_expected__` | Helper CTEs, `@py.name()` and `@sql.name()` calls in actual, CTE-list `@sql.name()` calls in the `WITH` list | -| `udf` | `__udf_actual__` | `__udf_expected__` | Helper CTEs, `__udf()` calls in actual | -| `table_fn` | `__table_fn_actual__` | `__table_fn_expected__` | Helper CTEs, `__table_fn()` calls in actual | - -CTE prefixes from other modes are not allowed. For example, `__source__` in a macro test or `__macro_actual__` in a test declared `TEST (mode model)` will produce a clear error pointing you to the right mode. Expected CTEs must not call macros, UDFs, or table functions - they should be independent, inspectable expected data. - -Helper CTEs follow the same rule in every mode: only the actual CTE calls the macro, UDF or table -function under test, so a helper of a macro test must not call `__udf()` or `__table_fn()` either. -A malformed reference call in a helper or expected CTE, such as `__source("raw", "orders")`, is a -P012 error at the call. - -## Multiple tests per file - -A single test file can contain multiple `TEST()` blocks. A test's name is its `name`, or the file -stem when `name` is omitted. Test names are globally unique: a test cannot share a name with another -test, a scenario, a model, a source, a seed, a function, or a Python node. Name the file or the block -after the behavior it checks, such as `orders_status_rules`, rather than after the model. The cases -of one parameterized test share its name. Each block in a file must have its own `name`: - -```sql -TEST (name "completed_orders_only"); - -WITH -__source__raw__orders AS ( - SELECT 1 AS id, 100 AS customer_id, 'completed' AS status -), -__expected__stg_orders AS ( - SELECT 1 AS order_id, 100 AS customer_id, 'completed' AS status -) - -TEST (name "cancelled_orders_excluded"); - -WITH -__source__raw__orders AS ( - SELECT 1 AS id, 100 AS customer_id, 'cancelled' AS status -), -__expected__stg_orders AS ( - SELECT 1 AS order_id, 100 AS customer_id, 'cancelled' AS status -) -``` - -A file with a single test can omit the `name` field. Files with multiple tests require names on every block. +Table function tests run after the function is created. Models can't depend on table functions, so +these tests check the function on its own. ## Repeating test logic -SQLBuild supports three repetition patterns: - -| Pattern | Use when | Result identity | -|---------|----------|-----------------| -| Multiple `TEST` blocks | Cases need different SQL shapes | One result per block | -| Macro or `VALUES` case table | One aggregate comparison is sufficient | One result for the table | -| Native parameters and cases | Cases share a template but need independent status | One result per named case | +| Pattern | Use when | Results | +|---------|----------|---------| +| Several `TEST` blocks | Cases need different SQL | One per block | +| A `VALUES` case table or macro | One comparison over all cases is enough | One for the table | +| Parameters and cases | Cases share a template but each needs its own result | One per named case | ### Native independent cases -Declare a typed schema and ordered named cases in one header: +Declare typed parameters and ordered, named cases in the header: ```sql TEST ( @@ -765,22 +771,20 @@ __expected__stg_orders AS ( ) ``` -The cases report independently as `order status: maps source states [completed]`, `[cancelled]`, -and `[pending]`. Authored order controls display order. Selecting `stg_orders` selects every case by -default. Add `--case` to run one named case without changing the test file: +Each case reports on its own, in the order written: `order status: maps source states [completed]`, +`[cancelled]` and `[pending]`. Selecting `stg_orders` runs every case. Run one case with `--case`: ```bash sqb test --select stg_orders --case cancelled ``` -If the case does not exist within the parent/model selection, SQLBuild reports the available case -names. +An unknown case name lists the available ones. -Supported scalar types are `string`, `integer`, `boolean`, `float`, and exact `decimal`. Decimal -values are quoted, such as `tax_rate "0.2000"`, so they never pass through binary float. Boolean -and integer remain distinct. Raw SQL and collection parameters are rejected. +Parameter types are `string`, `integer`, `boolean`, `float` and exact `decimal`. Quote decimal +values, such as `tax_rate "0.2000"`, so they never pass through binary floating point. Booleans and +integers stay distinct. Raw SQL and collection parameters are rejected. -Nullable parameters use an expanded declaration and are valid where SQL can infer the null type: +A nullable parameter uses the expanded form, and works where SQL can infer the null's type: ```sql parameters ( @@ -791,35 +795,24 @@ cases ( ) ``` -Every case provides exactly the declared parameters, and every declaration must appear as -`@param("name")` in the template. SQLBuild rejects missing, extra, incompatible, undeclared, and -unused values before execution. - -Expansion is deterministic: - -```text -parse TEST template and cases - -> render typed @param values with the active adapter - -> expand @const and @enum references - -> expand macros - -> validate SQL and compile expected models/assertions -``` +Every case gives exactly the declared parameters, and every parameter appears as `@param("name")` +in the test. Missing, extra, incompatible, undeclared and unused values are rejected before +anything runs. -Values can therefore feed macro arguments, for example -`@py.order_fixture(@param("source_status"))`. Parameters work in model, macro, UDF, and table-function -modes and preserve every expected model in multi-model tests. +Parameters are rendered first, as typed literals for the active adapter, then `@const` and `@enum` +references, then macros. A parameter can therefore be a macro argument, as in +`@py.order_fixture(@param("source_status"))`. Parameters work in every mode and keep every expected +model of a multi-model test. -Text and JSON output include parent/case identity and safe typed values. Compile JSON, manifests, -DAG checks, and compiled/runtime SQL artifacts retain source, block, case, and content-fingerprint -provenance. Changing one case changes that case's fingerprint without reassigning another case's -stable identity. +Text and JSON output show the test, the case and its typed values. Compile JSON, manifests, DAG +checks and compiled SQL record each case's source, block and fingerprint. Changing one case changes +only that case's fingerprint. -Scenarios are intentionally not parameterized. Keep one scenario per coherent business world so -capture and replay identity remains explicit. +Scenarios can't be parameterized: keep one scenario per coherent business world. ### Aggregate SQL case tables -When independent status is unnecessary, a normal SQL case table remains concise: +When cases don't need their own results, a `VALUES` table is shorter: ```sql TEST(); @@ -834,41 +827,31 @@ __expected__status_mapping AS ( ) ``` -This produces one aggregate test result. Use native cases when each row must pass or fail -independently. +This gives one test result. Quoted strings in `VALUES` rows may contain commas, brackets and +parentheses; escape embedded quotes as usual in SQL. -Quoted strings in `VALUES` rows may contain commas, brackets, and parentheses. SQLBuild preserves -these literals while extracting test CTEs; use normal SQL quote escaping for embedded quotes. - -## Test placement +## Running tests -Place unit test files under `tests/unit/` in your project directory. SQLBuild discovers all `.sql` files in this directory recursively. +Tests run during `sqb build` in DAG order, before the model they test is built. Run them on their +own with `sqb test`: -``` -tests/ - unit/ - test_stg_orders.sql # model test - test_fact_orders.sql # model test with assertion - test_daily_revenue_chain.sql # multi-model test - test_line_total_cents_macro.sql # macro test - test_is_completed_order_udf.sql # UDF test - test_customer_orders_table_fn.sql # table function test - scenarios/ - ... +```bash +sqb test +sqb test --select stg_orders ``` ## Cost guards A unit test runs the real SQL of every unmocked model between its mocks and the model under test as -one warehouse statement. That is cheap for one or two models, but the warehouse compiles and -optimises the whole chain at once, and that cost can grow much faster than the query's size. -SQLBuild therefore bounds each unit test in two ways. +one warehouse statement. The warehouse compiles and optimises that whole chain at once, and the cost +can grow much faster than the query's size. SQLBuild bounds each test in two ways. ### Real-model limit -`[settings] unit_test_max_real_models` (default `10`, also accepted in `sqlbuild_local.toml`) caps -how many real models one unit test may execute, counting the model under test. A test over the limit -fails compilation before any warehouse connection opens: +`[settings] unit_test_max_real_models` (default `10`) caps how many real models one test may run, +counting the model under test. It can also be set in `sqlbuild_local.toml`. A test over the limit +fails compilation, in `sqb test` and in `sqb build` when it runs tests, before any warehouse +connection opens: ```text error[S000]: Invalid SQL test fixtures: @@ -893,28 +876,31 @@ Options: unit_test_max_real_models = 3 ``` -Upgrade note: the limit is new, so after upgrading, a selection containing a unit test that executes -more than 10 real models fails to compile (`sqb test`, and `sqb build` when it runs tests) until the -test is mocked or the limit is raised. - -`sqb test` shows each test's real-model count on its row, such as `3 real models`, so a chain that -is growing towards the limit is visible early. Direct macro, UDF and table function tests execute -no models and are not counted. +`sqb test` shows each test's count on its row, such as `3 real models`, so a growing chain is +visible early. Macro, UDF and table function tests run no models and aren't counted. ### Time limits -Each unit test has a time limit, `30s` by default, measured from its first warehouse statement. -Rendering, warehouse queueing where the warehouse reports it, and cleanup after the comparison do not -count; see [per-item time limits](/docs/concepts/project-configuration#per-item-time-limits) for -each adapter. -Set it, from highest precedence, with `--test-timeout` on `sqb test` or `sqb build`, with -`TEST(timeout "45s")` in the test file, with `[targets..execution_limits] test_timeout`, or -with `[settings] test_timeout`. Each accepts a positive fixed duration or `"disabled"`. +Each unit test has a time limit, `30s` by default. The first of these that is set applies: + +1. `--test-timeout` on `sqb test` or `sqb build` +2. `TEST(timeout "45s")` in the test file +3. `[targets..execution_limits] test_timeout` +4. `[settings] test_timeout`, in `sqlbuild_local.toml` or `sqlbuild_project.toml` +5. The `30s` default + +Each takes a positive fixed duration, such as `"45s"` or `"10m"`, or `"disabled"`. -At the limit SQLBuild cancels the test's running statement, reports the test as `TIMEOUT` with the -elapsed time, the limit, where it was set and the cancellation outcome, and continues with the -remaining tests. When the test inlines real models beyond the model under test, the message also -lists them and the same options as the real-model limit: +The clock starts at the test's first warehouse statement. Rendering, warehouse queueing where the +warehouse reports it, and cleanup after the comparison don't count. See +[Per-item time limits](/docs/concepts/project-configuration#per-item-time-limits) for how each +adapter enforces the limit, and [Duration limit](/docs/concepts/project-configuration#duration-limit) +for how each adapter cancels a running statement. + +At the limit, SQLBuild cancels the running statement, reports the test as `TIMEOUT`, and continues +with the other tests. The report gives the elapsed time, the limit, where it was set, the +cancellation outcome and the query IDs it saw. When the test runs real models beyond the model +under test, it also lists them, with the same options as the real-model limit: ```text SQL unit test 'order_scans_kept' timed out after 1.0s (limit 1s). @@ -935,49 +921,22 @@ Options: in sqlbuild_project.toml (or sqlbuild_local.toml), or --test-timeout for one run. ``` -A test that runs only the model under test reports the timeout, the limit, its source and how to -change it, without the scenario advice. See -[Duration limit](/docs/concepts/project-configuration#duration-limit) for how each adapter cancels -a running statement. - -### Single-read comparison - -A test compares its actual output with the expected rows several ways: both row counts, rows -missing from each side, the expected-column check and, on failure, row samples. Where the warehouse -supports it, SQLBuild evaluates each compared model's actual output once into a temporary relation -and runs every comparison against it, so the chain is not recomputed for each read. Results, -messages and samples are unchanged. DuckDB and MotherDuck use a session temporary table that -SQLBuild drops after the test, including after a failure or timeout. PostgreSQL runs the test in -one transaction with an `ON COMMIT DROP` temporary table that the transaction's end removes, so the -reads stay on one server session behind a transaction-mode connection pooler. Snowflake reads the -first query's persisted result through `RESULT_SCAN`. If a stored output can no longer be read, -the test falls back to the inline comparison. If storing an output fails, SQLBuild runs the output -query on its own: when that query fails too, for example because a fixture value cannot be -converted in a column the expected rows leave out, the test errors with the warehouse's message -instead of passing on a comparison of the other columns; otherwise it compares inline. On DuckDB and -MotherDuck, an internal engine error while storing an output is a DuckDB defect rather than a -problem with the model, so the test compares inline. Other adapters compare the inline query as before. `target/run/tests/` keeps the self-contained inline comparison query for each test. - -## Skipping SQL analysis - -Compile checks SQL test fixtures and assertions like model SQL. For SQL the parser cannot read, -add `sql_analysis false` to the `TEST` header; the test still runs. Tests of a model with -`sql_analysis false`, and every test in a `--no-sql-analysis` run, are not analysed. With -`[settings] require_sql_analysis = true`, the opt-out is accepted only on test SQL that cannot be -parsed (see [Requiring SQL analysis](/docs/concepts/models/configuration#requiring-sql-analysis)). +A test that runs only the model under test reports the timeout, the limit, where it was set and how +to change it, without the scenario advice. -## Running tests +Scenarios have their own limit, `scenario_timeout`; see +[Scenario timeouts](/docs/concepts/project-configuration#unit-test-and-scenario-limits). -Tests run automatically during `sqb build` in DAG order, before their target model is materialized. +## Skipping SQL analysis -Run tests standalone: +Compile checks test fixtures and assertions like model SQL. For SQL the parser can't read, add +`sql_analysis false` to the `TEST` header; the test still runs. Tests of a model with +`sql_analysis false`, and every test in a `--no-sql-analysis` run, aren't analysed. With +`[settings] require_sql_analysis = true`, the opt-out is accepted only on test SQL that can't be +parsed; see [Requiring SQL analysis](/docs/concepts/models/configuration#requiring-sql-analysis). -```bash -sqb test -``` +## Details -Scope to specific models: - -```bash -sqb test --select stg_orders -``` +- On DuckDB, MotherDuck, PostgreSQL and Snowflake, a test evaluates each compared model's full + output once. If that output fails, for example because a fixture value can't be converted in a + column the expected rows leave out, the test errors with the warehouse's message. From ea0593ebb6d276a762d935a50f725ca0d2c07c18 Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Tue, 27 Oct 2026 09:08:53 +0000 Subject: [PATCH 04/10] docs: reorder the janitor page around its lifecycle and safety, with per-adapter usage tables --- .../sqlbuild/references/docs/cli/janitor.md | 443 +++++++++++------- website/src/content/docs/docs/cli/janitor.mdx | 427 +++++++++++------ 2 files changed, 558 insertions(+), 312 deletions(-) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md index 8e21ca3e63..0c4c715458 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md @@ -10,49 +10,22 @@ Online: https://sqlbuild.com/docs/cli/janitor/ - Usage - Flags -- Output -- Relations in use -- Speed -- Targets - Lifecycle -- Review schemas -- Compatibility views -- Pending migration origins -- Relation age -- Audit record -- Restoring an archive +- What else the janitor cleans +- Safety +- Targets - Configuration +- Output - Examples -- State history pruning -- Safety - -Finds stale warehouse relations that are no longer part of the project, archives them, and deletes archives once they expire. Requires `janitor.enabled = true` in `sqlbuild_project.toml`. -## Usage +`sqb janitor` cleans up relations your project no longer builds. It never drops a live relation +directly: it first renames a stale relation to an archive name, and drops the archive only once it +expires. It shows the plan and asks before changing anything. ```bash -sqb --project-dir janitor [flags] +sqb janitor ``` -## Flags - -| Flag | Description | -|------|-------------| -| `--target ` | Clean up a configured target instead of the active/default target, using that target's connection, schemas and state | -| `--as ` | Preview the janitor plan for another configured target's schemas through the active target's connection. Inspection only: nothing is archived, dropped or pruned, and there is no prompt. Cannot be combined with `--auto-approve` | -| `--auto-approve` | Skip the confirmation prompt and run the planned actions immediately | -| `--retention-days` | Override the configured retention period (days) before a stale relation is archived | -| `--direct-state-history-versions` | Override how many state-history versions are kept per identity | -| `--verbose`, `-v` | List every entry of each preview section and every skipped object with its reason. Without it, each section shows its first 10 entries and skipped objects are counted by reason | -| `--json` | Print the plan and the result as one JSON document on stdout; the preview, prompt and progress go to stderr | -| `--keep-in-use` | Keep relations that other queries read within `janitor.in_use_window` instead of archiving them, for this run. Same as `janitor.keep_in_use = true`. See [Relations in use](#relations-in-use) | -| `--drop-old-name-view ` | Drop the [compatibility view](../concepts/models/migrations.md#old-names) at a renamed model's old name before it expires. Repeat for several views. Fails with `C503` if no compatibility view is recorded at the name | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | - -## Output - -After the preview and confirmation, each phase with work announces how many actions it runs, prints one line per action as it finishes, and closes with its duration. The run ends with one line of totals: - ``` Archiving 3 relations... 1/3 archived analytics.old_orders -> analytics._sqb_archive__20261022t101500z__old_orders OK 0.21s @@ -66,183 +39,296 @@ Deleted 1 expired archive. (0.12s) ✓ Janitor completed archived 3, deleted 1, pruned 0 (0.81s) ``` -The phases are archiving stale relations, deleting expired archives, dropping old-name views, dropping expired query artifacts, dropping leftover review schemas and pruning direct state, in that order. If an action fails, its line shows `FAIL` and the error, no further action starts, and the run ends with `✗ Janitor failed` and the totals completed before the failure, exiting with `1`. +The janitor is off until you set `enabled = true` under [`[janitor]`](#configuration). -Long preview sections show their first 10 entries and then `... and N more (use --verbose to show all)`. Skipped objects are counted by reason: +## Usage +```bash +sqb --project-dir janitor [flags] ``` -Skipped objects (385) - 210 relation is newer than 14 days - 175 relation is not tracked by SQLBuild -``` - -With `--json`, stdout holds one document with `status` (`completed`, `failed`, `cancelled`, `blocked`, `nothing_to_do`, `previewed` or `disabled`), the full `plan`, the `usage_check`, the `result` with counts and every action, and `error`. Errors before the cleanup starts, such as a refused confirmation, are reported on stderr only. -## Relations in use +## Flags -Archiving renames a relation, so anything that still reads it fails from then on. Before archiving, the janitor looks up which candidates other queries read within `in_use_window` (default `7d`) and shows them, with their readers, in an `In use` section: +| Flag | Description | +|------|-------------| +| `--target ` | Clean up another configured target, with that target's connection, schemas and state | +| `--as ` | Preview the plan for another target's schemas through the active target's connection. Changes nothing and doesn't prompt. Can't be combined with `--auto-approve` | +| `--auto-approve` | Run the plan without the confirmation prompt | +| `--retention-days` | Override `retention_days` for this run | +| `--direct-state-history-versions` | Override how many state-history versions are kept per identity | +| `--keep-in-use` | Keep relations read within `in_use_window` instead of archiving them, for this run. See [Relations in use](#relations-in-use) | +| `--drop-old-name-view ` | Drop the [compatibility view](../concepts/models/migrations.md#old-names) at a renamed model's old name before it expires. Repeat for several views. Fails with `C503` if no compatibility view is recorded at the name | +| `--verbose`, `-v` | List every entry of each preview section and every skipped object with its reason | +| `--json` | Print the plan and the result as one JSON document on stdout; the preview, prompt and progress go to stderr | +| `--warehouse ` | Snowflake warehouse for this run; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | -``` -In use - analytics.orders_daily queried 2026-10-21 by svc_reporting, alice (14 queries in 7 days) - analytics.old_orders queried 2026-10-20 by svc_reporting, alice, bob +2 more (40 queries in 7 days); frozen since 2026-09-01, still in use - analytics.old_customers queried 2026-10-21 by alice via analytics.customer_report (3 queries in 7 days); frozen since 2026-09-01, still in use -``` +## Lifecycle -Each line names the latest read, up to three readers (most queries first) and how many queries read the relation. Reading a view also reads what it is built on, so those relations are in use too. When a relation was only read that way, the line says `via` and names the views, or says `through a view` when the warehouse doesn't record which. A stale table that is still read is also flagged `frozen since , still in use`: its data stopped changing on that date, so its readers are getting old data. Views are not flagged, because their reads stay live. The confirmation prompt states the count, for example `14 relations will be archived; 3 of them are in use.` +A relation is **stale** when all of these are true: -By default in-use relations are still archived, which is also how their readers find out. To keep them instead, set `keep_in_use = true` under `[janitor]` or pass `--keep-in-use`. In-use relations are then listed under `Kept (in use)` with their readers and are left out of the archive list and the counts. A later run archives them once nothing reads them within the window. +- it is in a schema the project writes to; +- the current, complete project doesn't produce it; +- SQLBuild tracked it, when `delete_tracked_only = true` (the default); +- it matches no `exclude_patterns` entry, ignoring case. -Only relations to archive are checked. Expired archives and state-table pruning are not. +A stale relation goes through two stages: -Usage comes from the warehouse's query history: +1. **Archive.** Once it has been stale for `retention_days` (default `14`), the janitor renames it, + in the same schema, to `_SQB_ARCHIVE____`. For example, + `stg_orders` becomes `_SQB_ARCHIVE__20260924T101500Z__stg_orders`. A rename copies no data and + keeps the relation's table type, grants and time travel. +2. **Delete.** Once the archive is older than `archive_retention_days` (default `14`), the janitor + drops it. -| Adapter | Usage source | -|---------|--------------| -| Snowflake | `SNOWFLAKE.ACCOUNT_USAGE.ACCESS_HISTORY` (`direct_objects_accessed`, `base_objects_accessed`) joined with `QUERY_HISTORY`, and `OBJECT_DEPENDENCIES` | -| BigQuery | `` `.region-`.INFORMATION_SCHEMA.JOBS `` (`referenced_tables`, `referenced_logical_views`) | -| Databricks | `system.access.table_lineage` (`source_table_full_name`, `direct_access`) | -| ClickHouse | `system.query_log` (`tables`) on the connected server | -| Other adapters | Not available | +Setting both to `0` archives and deletes stale relations in the same run. -On Snowflake, one query reads the access history of all candidates (up to 1,000 names per query). A candidate listed in a query's base objects but not its direct objects was read through the objects the query named, which the line names after `via`. When something was read, a second query reads the view dependencies in `OBJECT_DEPENDENCIES` for the databases of the candidates and of the objects queries read directly. Anything those directly read objects are built on, through any number of views, is in use: an unread candidate reached that way is in use `via` the directly read object, with its readers. So a candidate view between a non-candidate view someone reads and a candidate table is kept too. Links through views in other databases are not followed. Dependencies created by `CREATE TABLE ... AS SELECT` and similar statements are not recorded there. Queries SQLBuild tags as its own (`QUERY_TAG` with `"app":"sqlbuild"`, which builds, audits, loads and hooks set) and `CREATE ... CLONE` statements of tables, dynamic tables, schemas and databases don't count as reads. A statement counts as a clone only when it starts with `CREATE`, after any comments, and names its source after `CLONE`; a column, string or comment containing the word doesn't make it one. SQLBuild's own reads don't need this check: schemas holding the project's sources are skipped, so the project graph already protects what the project reads. Access history needs Enterprise edition. The janitor's role needs `IMPORTED PRIVILEGES` on the `SNOWFLAKE` database, or the `SNOWFLAKE.GOVERNANCE_VIEWER` database role for access history and the `SNOWFLAKE.OBJECT_VIEWER` database role for object dependencies: +Archive names are written as unquoted lowercase identifiers, so warehouses that fold unquoted names +show them in their own case; on Snowflake the example above appears as +`_SQB_ARCHIVE__20260924T101500Z__STG_ORDERS`. Long names are shortened with a short hash to fit the +adapter's identifier limit. The prefix and timestamp are never shortened. -```sql -GRANT DATABASE ROLE SNOWFLAKE.GOVERNANCE_VIEWER TO ROLE janitor_role; -GRANT DATABASE ROLE SNOWFLAKE.OBJECT_VIEWER TO ROLE janitor_role; -``` +An archive's age comes from the timestamp in its name, so deleting needs no other state. -Without `OBJECT_VIEWER`, reads are still checked, but view chains are not followed and the preview says `view chains not followed: grant SNOWFLAKE.OBJECT_VIEWER`. Access history and object dependencies lag by up to about 3 hours, so reads in the last few hours may not show yet. +### Relation age -On BigQuery, the janitor reads the `INFORMATION_SCHEMA.JOBS` view of each candidate's project, in the location of its dataset, and counts successful query jobs that reference the relation, by `user_email`. A job lists the views it reads as well as the tables behind them, so both count. Whether views nested inside other views are listed is not documented, so a candidate view that is only read through another view may not show as in use. Jobs answered from cached results record no tables or views, so with an `in_use_window` under about a day a reader whose queries were all cached can be missed. Jobs that run in another project are not in that view, so their reads are not seen. Script parent jobs, `CREATE SNAPSHOT TABLE` and `CREATE TABLE ... CLONE` don't count. SQLBuild sets no job labels on BigQuery, so its own jobs count like any other. The janitor's role needs `bigquery.jobs.listAll` and `bigquery.jobs.create` on each project, for example through the BigQuery Resource Viewer and BigQuery User roles. The view keeps 180 days of jobs, so a longer `in_use_window` sees only the last 180 days. +`retention_days` counts from the last time the relation was written, as the warehouse reports it. +For a relation the project no longer builds, that is roughly when it stopped being built. -On Databricks, the janitor counts read events in Unity Catalog's table lineage, by `created_by`. Lineage records the view a query reads as a direct read and the tables behind it as reads through the view, which the line names when lineage records the view for the same statement. It needs Unity Catalog with the `access` system schema enabled, and `USE CATALOG` on `system`, `USE SCHEMA` on `system.access` and `SELECT` on `system.access.table_lineage`: +| Adapter | Age source | +|---------|------------| +| Snowflake | `INFORMATION_SCHEMA.TABLES` `CREATED` and `LAST_ALTERED`, which change on every write | +| Databricks | `DESCRIBE DETAIL` `lastModified` for tables, and `INFORMATION_SCHEMA.TABLES` for views | +| BigQuery | `__TABLES__` `creation_time` and `last_modified_time` | +| PostgreSQL, SQL Server, DuckDB, MotherDuck | Not available | -```sql -GRANT USE CATALOG ON CATALOG system TO `janitor-principal`; -GRANT USE SCHEMA ON SCHEMA system.access TO `janitor-principal`; -GRANT SELECT ON TABLE system.access.table_lineage TO `janitor-principal`; -``` +When the warehouse can't report an age, the janitor skips the relation unless `retention_days` is +`0`. SQL Server is in this group because its modification date ignores inserts and updates, so a +recently written table could look old. -Lineage records only reads Databricks can infer lineage for, and can arrive with a delay, so "not in use" is a weaker signal than on Snowflake. Lineage has no query text, so clones and SQLBuild's own reads can't be told apart and count as reads. The table keeps one year of events. +### Restoring an archive -On ClickHouse, the janitor counts finished `SELECT` and `INSERT` queries in `system.query_log` of the server it connects to, by the user who started them. A query's `tables` lists every table and view it touched, including the views and tables a view is built on, so a whole view chain shows as in use, without `via`; queries a query starts count once, for its initial query. Only the connected server's `query_log` is read: queries on other replicas or shards are logged on those servers and are not seen, which matters on ClickHouse Cloud and behind load-balanced replicas. When `system.clusters` shows other replicas of the connected server's shard, the preview says `usage read from this server only; other replicas not checked`. Writes into a relation count as use, since renaming it breaks the writer too. ClickHouse answers some queries, such as a plain `count()` of a table, from metadata without logging the table, so those reads are not seen. SQLBuild doesn't mark its ClickHouse queries, so its own reads count. The janitor's user needs `SELECT` on `system.query_log`: +Rename the archive back to its original name before it expires. For example, on Snowflake: ```sql -GRANT SELECT ON system.query_log TO janitor_user; +ALTER TABLE analytics._SQB_ARCHIVE__20260924T101500Z__stg_orders RENAME TO analytics.stg_orders; ``` -Queries are logged when `log_queries` is on, which is the default; when it is off for the janitor's connection, usage is not checked. Readers whose settings profile turns `log_queries` off, or logs only a sample with `log_queries_probability`, are missed. The log is flushed about every 7.5 seconds, and ClickHouse keeps it until a TTL configured on the server removes it. - -When usage can't be checked, because the adapter has no usage source or the role can't read its query history, the preview and the prompt show one notice, such as `usage not checked: unsupported adapter`, and the janitor archives as usual. With `keep_in_use` on, a second notice, `nothing kept in use`, says that relations in use are archived too. +If the archive's name was shortened, look up the original name in the +[audit record](#audit-record). After an archive is deleted, your warehouse's time travel or undrop +feature may still recover it within its own retention window. -In the `--json` plan, each entry of `relations_to_archive` and `kept_in_use` has `in_use`: `null` when usage was not checked, `false` when nothing read the relation within the window, or an object with `last_read`, `readers`, `query_count` and `via` (`null` when the relation was read directly, otherwise the names of the views it was read through, empty when unknown). Entries of `kept_in_use` have only `name`, `age_days` and `in_use`, since they are not archived. The top-level `usage_check` has `status` (`checked` or `not_checked`), `reason`, `window`, `keep_in_use` and `notices` (what a check that ran left out, such as view chains it could not follow), or is `null` when there was nothing to check. +## What else the janitor cleans -## Speed +Each run works through these phases, in order: -The janitor reads each schema's relations once per run, and the definitions of compatibility views with one read per schema. On warehouses that run DDL concurrently (Snowflake, BigQuery and Databricks), the independent actions of each phase run on up to the target's [`concurrency`](../concepts/project-configuration.md#target-concurrency) connections at once; phases still run one after another. DuckDB, SQLite and the other adapters apply one action at a time. - -## Targets +1. archive stale relations; +2. delete expired archives; +3. drop expired [compatibility views](#compatibility-views); +4. drop expired query artifacts; +5. drop leftover [review schemas](#review-schemas); +6. [prune state history](#state-history-pruning). -By default the janitor cleans up the active target. `--target ` selects another configured target, exactly like `sqb build --target`: the janitor connects with that target's connection and inspects and cleans that target's schemas and state tables. Other targets are never touched. +### Compatibility views -`--as ` previews what the janitor would do in another target without changing anything, like [`sqb plan --as`](plan.md). The project compiles for that target, and its schemas are inspected through the active target's connection, which can be the default target or the one given with `--target`. The janitor lists the relations it would archive, the archives it would delete, the compatibility views it would drop and the state tables it would prune (all of them with `--verbose`), then exits. It drops nothing, writes no audit or state rows, prunes nothing and does not prompt. `--drop-old-name-view` is allowed with `--as` and only previews the drop. +When a model is renamed, SQLBuild keeps a [compatibility view](../concepts/models/migrations.md#old-names) +at its old name. The janitor lists each one with the model it reads and its expiry: ``` -Previewing janitor as target 'prod' through the connection of target 'dev' (inspection only). -... -Relations to archive - prod.customers -> prod._sqb_archive__20261001t101500z__customers age unknown, delete after 2026-10-15 10:15:00 UTC - -Archives to delete - prod._sqb_archive__20200101t000000z__old_products archived 2020-01-01 00:00:00 UTC, age 2465d, expired 2020-01-15 00:00:00 UTC - -Previewed janitor as target 'prod' through the connection of target 'dev'. Nothing was changed. -Rerun with `--target prod` instead of `--as prod` to apply it. +Old name views (1) +└── prod.revenue -> model:daily_revenue + └── expires 2026-10-28 (live; drop early with --drop-old-name-view prod.revenue) ``` -An unknown target name fails before anything connects. A preview exits with `0`, or with `1` when a schema is blocked, as a normal run would. - -## Lifecycle - -The janitor never drops a live relation directly. Each stale relation goes through two stages: - -1. **Archive.** Once a relation has been stale for `retention_days`, the janitor renames it in place, in the same schema, to: +A compatibility view is never archived as a stale relation. Once it expires, or when you name it +with `--drop-old-name-view`, the janitor drops it and records the drop in +`_sqlbuild_old_name_views`. - ``` - _SQB_ARCHIVE____ - ``` +The janitor drops the view only if the relation at the old name is still the view SQLBuild created. +Any other relation there, such as a view a project model built after an interrupted early drop, is +left alone. The janitor then records the compatibility view as dropped and lists it as +`record dropped (name now used by another relation, model:revenue)`, naming the model when the +name belongs to one. A compatibility view that no longer exists is recorded as dropped too. - For example, `stg_orders` becomes `_SQB_ARCHIVE__20260924T101500Z__stg_orders`. A rename copies no data, and keeps the relation's table type, grants and time travel. -2. **Delete.** Once an archive is older than `archive_retention_days`, the janitor drops it. +### Review schemas -Stale relations are identified exactly as before. A relation is a candidate only if it is in a schema the project writes to, it is not produced by the current complete project, it was tracked by SQLBuild (when `delete_tracked_only = true`), it matches no exclude pattern, and it is older than the retention period. +[`sqb review`](review.md) builds into a schema it creates and drops for one run. When a +review couldn't drop its schema, the janitor drops it once the timestamp in its name is more than 24 +hours old, so a running review is never touched. Only schemas with a review ownership marker in the +active target schema's `_sqlbuild_fingerprints` table are dropped; a schema whose name only looks +like a review schema is left alone. -The age of an archive comes from the timestamp in its name, so deletion needs no other state. The janitor only deletes relations whose names **strictly** match the archive format and that are in a schema the project writes to. Names that only look similar are reported and never touched. The janitor also only acts on relations whose names are plain lowercase identifiers (letters, digits and underscores), and that don't collide case-insensitively with another relation in the same schema. Anything else is reported and left alone. +### State history pruning -Archive names are written as unquoted lowercase identifiers, so warehouses that fold unquoted names show them in their own case. On Snowflake, for example, the archive appears as `_SQB_ARCHIVE__20260924T101500Z__STG_ORDERS`. +The janitor prunes old rows from `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness`, keeping +the latest `direct_state_history_versions` rows per identity (default `20`). This is separate from +relation retention. -Long names are shortened with a short hash so they fit the adapter's identifier limit. The prefix and timestamp are never shortened. - -Setting both `retention_days` and `archive_retention_days` to `0` archives and deletes stale relations in the same run. - -## Review schemas - -[`sqb review`](review.md) builds into a schema it creates for one run and drops when the run ends. Before creating the schema it writes an ownership marker to the active target schema's `_sqlbuild_fingerprints` table. When a review could not drop its schema, the janitor drops it once the timestamp in its name is more than 24 hours old, so a review that is still running is never touched. Only schemas with a marker in the active target schema are dropped; a schema whose name merely looks like a review schema is left alone. - -## Compatibility views +## Safety -When a model is renamed, SQLBuild keeps a [compatibility view](../concepts/models/migrations.md#old-names) at its old name. The janitor lists each one with the model it reads and its expiry: +- **You confirm the plan.** The preview lists the planned archives and deletes with exact names, + ages and expiry, then asks you to type an exact confirmation phrase. Use `--auto-approve` only in + automation, or when you're certain. +- **Only exact archive names are deleted.** An archive is deleted only when its name strictly + matches the archive format and it is in a schema the project writes to. Names that only look + similar are reported and never touched. +- **Unusual names are left alone.** The janitor acts only on plain lowercase identifiers (letters, + digits and underscores) that don't collide, ignoring case, with another relation in the same + schema. Anything else is reported and skipped. +- **Pending migration origins are kept.** While a model declares `migrate_from` and its own + relation doesn't exist yet in the target, the origin isn't archived and is listed as + `pending migration origin for `. A bare `migrate_from` name is matched in each schema the + project builds into; a qualified name only at that location. Once the model has built, the old + name holds a compatibility view, which expires as above. +- **Schemas with sources are protected.** A schema that holds the project's configured sources is + skipped and listed under `Skipped schemas`, so what the project reads is never archived. When such a schema is one SQLBuild manages, the + preview shows `Janitor blocked`, lists the actions it suppressed, and the run exits `1`. +- **Run the same SQLBuild version as your builds.** An older janitor doesn't know compatibility + views and archives them as stale relations. + +### Relations in use + +Archiving renames a relation, so anything that still reads it fails from then on. Before +archiving, the janitor checks which candidates other queries read within `in_use_window` (default +`7d`) and lists them, with their readers: ``` -Old name views (1) -└── prod.revenue -> model:daily_revenue - └── expires 2026-10-28 (live; drop early with --drop-old-name-view prod.revenue) +In use + analytics.orders_daily queried 2026-10-21 by svc_reporting, alice (14 queries in 7 days) + analytics.old_orders queried 2026-10-20 by svc_reporting, alice, bob +2 more (40 queries in 7 days); frozen since 2026-09-01, still in use + analytics.old_customers queried 2026-10-21 by alice via analytics.customer_report (3 queries in 7 days); frozen since 2026-09-01, still in use ``` -A compatibility view is never archived as a stale relation. Once it expires, or when it is named with `--drop-old-name-view`, the janitor drops it and records the drop in `_sqlbuild_old_name_views`. Before dropping, the janitor checks that the relation at the old name is still the view SQLBuild created there, by comparing its stored definition with the view's SQL. Any other relation at that name, such as a view a project model built after an interrupted early drop, is left alone: the janitor records the compatibility view as dropped and lists it as `record dropped (name now used by another relation, model:revenue)`, naming the model when the name is a project destination. A compatibility view that no longer exists is recorded as dropped too. +Each line shows the latest read, up to three readers (most queries first) and the query count. -Run the same SQLBuild version for `sqb janitor` as for builds. An older janitor does not know compatibility views and archives them as stale relations. +- Reading a view also reads what it's built on, so those relations are in use too. A relation read + only that way shows `via` and the views, or `through a view` when the warehouse doesn't record + which. +- A stale table that is still read is flagged `frozen since , still in use`: its data stopped + changing on that date, so its readers get old data. Views aren't flagged, because their reads stay + live. +- The prompt states the count, for example + `14 relations will be archived; 3 of them are in use.` -## Pending migration origins +By default, in-use relations are still archived, which is also how their readers find out. To keep +them, set `keep_in_use = true` or pass `--keep-in-use`. They are then listed under `Kept (in use)` +and left out of the archive list and the counts. A later run archives them once nothing reads them +within the window. Only relations to archive are checked, not expired archives or state pruning. -A relation that a project model still has to migrate from is never archived. While a model declares `migrate_from` and its own relation does not exist yet in the target, the janitor skips the origin and lists it as `pending migration origin for `. A bare `migrate_from` name is matched in each schema the project builds into; a qualified name only at that location. Once the model has built, the origin is handled as usual: after a migration, the old name holds a compatibility view, which expires as described above. +Usage comes from the warehouse's query history: -## Relation age +| Adapter | Source | Grants | Caveats | +|---------|--------|--------|---------| +| Snowflake | `ACCOUNT_USAGE.ACCESS_HISTORY` with `QUERY_HISTORY`, and `OBJECT_DEPENDENCIES` for view chains | `IMPORTED PRIVILEGES` on the `SNOWFLAKE` database, or the `SNOWFLAKE.GOVERNANCE_VIEWER` and `SNOWFLAKE.OBJECT_VIEWER` database roles | Needs Enterprise edition. Lags up to about 3 hours | +| BigQuery | `` `.region-`.INFORMATION_SCHEMA.JOBS `` of each candidate's project, in its dataset's location | `bigquery.jobs.listAll` and `bigquery.jobs.create` on each project, for example BigQuery Resource Viewer and BigQuery User | Keeps 180 days of jobs | +| Databricks | `system.access.table_lineage` | `USE CATALOG` on `system`, `USE SCHEMA` on `system.access`, `SELECT` on `system.access.table_lineage` | Needs Unity Catalog with the `access` system schema. Keeps one year of events | +| ClickHouse | `system.query_log` of the connected server | `SELECT` on `system.query_log` | Only the connected server's log is read | +| Other adapters | Not available | | | -`retention_days` is measured from the last time a relation was written, as reported by the warehouse. For a relation the project no longer builds, that is roughly when it stopped being built. +```sql +-- Snowflake +GRANT DATABASE ROLE SNOWFLAKE.GOVERNANCE_VIEWER TO ROLE janitor_role; +GRANT DATABASE ROLE SNOWFLAKE.OBJECT_VIEWER TO ROLE janitor_role; -| Adapter | Age source | -|---------|------------| -| Snowflake | `INFORMATION_SCHEMA.TABLES` `CREATED` and `LAST_ALTERED`, which change on every write | -| Databricks | `DESCRIBE DETAIL` `lastModified` for tables, and `INFORMATION_SCHEMA.TABLES` for views | -| BigQuery | `__TABLES__` `creation_time` and `last_modified_time` | -| PostgreSQL, SQL Server, DuckDB, MotherDuck | Not available | +-- Databricks +GRANT USE CATALOG ON CATALOG system TO `janitor-principal`; +GRANT USE SCHEMA ON SCHEMA system.access TO `janitor-principal`; +GRANT SELECT ON TABLE system.access.table_lineage TO `janitor-principal`; -When the warehouse can't report a relation's age, the janitor skips that relation whenever `retention_days` is greater than `0`. SQL Server is in this group because its modification date ignores inserts and updates, so a recently written table could look old. +-- ClickHouse +GRANT SELECT ON system.query_log TO janitor_user; +``` -Ages are only looked up for relations that are otherwise candidates, and not at all when `retention_days` is `0`. +When usage can't be checked, because the adapter has no source or the role can't read it, the +preview and the prompt show one notice, such as `usage not checked: unsupported adapter`, and the +janitor archives as usual. With `keep_in_use` on, a second notice, `nothing kept in use`, says that +in-use relations are archived too. + +#### What counts as a read + +**Snowflake.** A relation read through the objects a query named shows `via` those objects. +Everything a directly read object is built on, through any number of views, is in use, so a +candidate view between a view someone reads and a candidate table is kept too. Links through views +in other databases aren't followed, and dependencies created by `CREATE TABLE ... AS SELECT` and +similar statements aren't recorded. These don't count as reads: + +- queries SQLBuild tags as its own (`QUERY_TAG` with `"app":"sqlbuild"`, set by builds, audits, + loads and hooks); +- `CREATE ... CLONE` of tables, dynamic tables, schemas and databases. A statement counts as a clone + only when it starts with `CREATE`, after any comments, and names its source after `CLONE`. + +Without `OBJECT_VIEWER`, reads are still checked but view chains aren't followed, and the preview +says `view chains not followed: grant SNOWFLAKE.OBJECT_VIEWER`. + +**BigQuery.** Successful query jobs that reference the relation count, by `user_email`. A job lists +the views it reads and the tables behind them, so both count. These are missed or don't count: + +- a candidate view read only through another view may not show, because BigQuery doesn't document + whether nested views are listed; +- jobs answered from cached results record no tables, so with an `in_use_window` under about a day + a reader whose queries were all cached can be missed; +- jobs that run in another project aren't seen; +- script parent jobs, `CREATE SNAPSHOT TABLE` and `CREATE TABLE ... CLONE` don't count; +- SQLBuild's own jobs count like any other. + +**Databricks.** Read events in Unity Catalog's table lineage count, by `created_by`. A view a query +reads is a direct read, and the tables behind it are reads through the view. Lineage covers only +reads Databricks can infer lineage for, and can arrive late, so "not in use" is a weaker signal than +on Snowflake. Clones and SQLBuild's own reads count as reads. + +**ClickHouse.** Finished `SELECT` and `INSERT` queries count, by the user who started them, once per +initial query. A query lists every table and view it touched, so a whole view chain shows as in +use, without `via`. Writes count, since renaming the relation breaks the writer too. These are +missed or don't count: + +- queries on other replicas or shards, which matters on ClickHouse Cloud and behind load-balanced + replicas. When `system.clusters` shows other replicas of the connected server's shard, the preview + says `usage read from this server only; other replicas not checked`; +- queries ClickHouse answers from metadata without logging the table, such as a plain `count()`; +- readers whose settings profile turns `log_queries` off or samples it with + `log_queries_probability`. When `log_queries` is off for the janitor's own connection, usage isn't + checked; +- queries in the last few seconds: the log is flushed about every 7.5 seconds; +- queries older than the server's `query_log` TTL; +- SQLBuild's own reads count. + +### Audit record + +Every archive and delete is recorded in an append-only `_sqlbuild_janitor_events` table in the +affected schema: the original name, the archive name, the time and the invocation. The janitor +never reads it to decide what to do; it is for auditing, and for restoring archives whose names +were shortened, which are recorded right after their rename. Other events are written in batches +of up to 100 per schema, so if the process is killed, up to 100 actions in a schema may have no +event. Those archives still expire by the timestamp in their names. -## Audit record +## Targets -Every archive and every delete is recorded in an append-only `_sqlbuild_janitor_events` table in the affected schema. It records the original name, the archive name, the time and the invocation. The janitor never reads this table to decide what to do. It exists for auditing, and for restoring archives whose names were shortened. +By default the janitor cleans the active target. `--target ` selects another configured +target, exactly like `sqb build --target`. Other targets are never touched. -Events are written in batches of up to 100 per schema, after each batch of actions and after a failure. An archive whose name was shortened gets its event right after its rename. If the process is killed, at most one batch of actions in a schema can lack events; those archives still expire by the timestamp in their names. +`--as ` previews another target without changing anything, like +[`sqb plan --as`](plan.md). The project compiles for that target, and its schemas are +inspected through the active target's connection. The preview lists what would be archived, +deleted, dropped and pruned, then exits. It drops nothing, writes no audit or state rows and doesn't +prompt. `--drop-old-name-view` with `--as` only previews the drop. -## Restoring an archive +``` +Previewing janitor as target 'prod' through the connection of target 'dev' (inspection only). +... +Relations to archive + prod.customers -> prod._sqb_archive__20261001t101500z__customers age unknown, delete after 2026-10-15 10:15:00 UTC -Rename the archive back to its original name before it expires, using your warehouse's rename statement. For example, on Snowflake: +Archives to delete + prod._sqb_archive__20200101t000000z__old_products archived 2020-01-01 00:00:00 UTC, age 2465d, expired 2020-01-15 00:00:00 UTC -```sql -ALTER TABLE analytics._SQB_ARCHIVE__20260924T101500Z__stg_orders RENAME TO analytics.stg_orders; +Previewed janitor as target 'prod' through the connection of target 'dev'. Nothing was changed. +Rerun with `--target prod` instead of `--as prod` to apply it. ``` -If the archive's name was shortened, look up the original name in `_sqlbuild_janitor_events`. After an archive has been deleted, your warehouse's own time travel or undrop feature may still be able to recover it within its retention window. +An unknown target name fails before anything connects. A preview exits `0`, or `1` when a schema +is blocked, as a normal run would. ## Configuration -Configure janitor behavior in `sqlbuild_project.toml`: - ```toml [janitor] enabled = true @@ -254,18 +340,59 @@ in_use_window = "7d" keep_in_use = false ``` -When `retention_days` is not set, it defaults to `14`. See [Project Configuration](../concepts/project-configuration.md#janitor) for every janitor setting. +See [Project Configuration](../concepts/project-configuration.md#janitor) for every key and default. +On Snowflake, BigQuery and Databricks, the actions of each phase run on up to the target's +[`concurrency`](../concepts/project-configuration.md#target-concurrency) connections at once. + +## Output + +After the preview and confirmation, each phase with work prints how many actions it runs, one line +per action, and its duration, as in the example at the top. The run ends with one line of totals. + +If an action fails, its line shows `FAIL` and the error, no further action starts, and the run ends +with `✗ Janitor failed` and the totals completed so far. The exit code is `1`. + +Without `--verbose`, each preview section shows its first 10 entries and then +`... and N more (use --verbose to show all)`, and skipped objects are counted by reason: + +``` +Skipped objects (385) + 210 relation is newer than 14 days + 175 relation is not tracked by SQLBuild +``` + +### JSON output + +With `--json`, stdout holds one document: + +| Field | Contents | +|-------|----------| +| `status` | `completed`, `failed`, `cancelled`, `blocked`, `nothing_to_do`, `previewed` or `disabled` | +| `plan` | The full plan | +| `usage_check` | `status` (`checked` or `not_checked`), `reason`, `window`, `keep_in_use` and `notices` (what a check left out, such as view chains it couldn't follow); `null` when there was nothing to check | +| `result` | Counts and every action | +| `error` | The error, if any | + +Each entry of `plan.relations_to_archive` and `plan.kept_in_use` has `in_use`: + +- `null` when usage wasn't checked; +- `false` when nothing read the relation within the window; +- otherwise an object with `last_read`, `readers`, `query_count` and `via` (`null` for a direct + read, otherwise the views it was read through, empty when unknown). + +Entries of `kept_in_use` have only `name`, `age_days` and `in_use`. Errors before the cleanup +starts, such as a refused confirmation, go to stderr only. ## Examples ```bash -# Interactive mode (shows the plan and prompts for confirmation) +# Show the plan and prompt for confirmation sqb janitor -# Run the planned archives and deletes without prompting +# Run the plan without prompting sqb janitor --auto-approve -# Override retention to 7 days +# Archive relations stale for 7 days instead of the configured retention sqb janitor --retention-days 7 # Preview the prod cleanup through the current connection, changing nothing @@ -277,11 +404,3 @@ sqb janitor --target prod # Keep relations that are still read instead of archiving them sqb janitor --keep-in-use ``` - -## State history pruning - -In addition to archiving stale relations, the janitor prunes old rows from `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness`, keeping the latest `direct_state_history_versions` rows per identity (default `20`). This is configured separately from relation retention. - -## Safety - -The janitor shows the planned archives and deletes, with exact names, ages and expiry (every one of them with `--verbose`, or in the `--json` plan), then asks you to type an exact confirmation phrase. Relations whose names differ only by case from another relation in the same schema are reported and never dropped, and exclude patterns match regardless of case. Use `--auto-approve` only in automation, or when you're certain. diff --git a/website/src/content/docs/docs/cli/janitor.mdx b/website/src/content/docs/docs/cli/janitor.mdx index 246441459e..3c9ba2279e 100644 --- a/website/src/content/docs/docs/cli/janitor.mdx +++ b/website/src/content/docs/docs/cli/janitor.mdx @@ -5,33 +5,14 @@ sidebar: label: "janitor" --- -Finds stale warehouse relations that are no longer part of the project, archives them, and deletes archives once they expire. Requires `janitor.enabled = true` in `sqlbuild_project.toml`. - -## Usage +`sqb janitor` cleans up relations your project no longer builds. It never drops a live relation +directly: it first renames a stale relation to an archive name, and drops the archive only once it +expires. It shows the plan and asks before changing anything. ```bash -sqb --project-dir janitor [flags] +sqb janitor ``` -## Flags - -| Flag | Description | -|------|-------------| -| `--target ` | Clean up a configured target instead of the active/default target, using that target's connection, schemas and state | -| `--as ` | Preview the janitor plan for another configured target's schemas through the active target's connection. Inspection only: nothing is archived, dropped or pruned, and there is no prompt. Cannot be combined with `--auto-approve` | -| `--auto-approve` | Skip the confirmation prompt and run the planned actions immediately | -| `--retention-days` | Override the configured retention period (days) before a stale relation is archived | -| `--direct-state-history-versions` | Override how many state-history versions are kept per identity | -| `--verbose`, `-v` | List every entry of each preview section and every skipped object with its reason. Without it, each section shows its first 10 entries and skipped objects are counted by reason | -| `--json` | Print the plan and the result as one JSON document on stdout; the preview, prompt and progress go to stderr | -| `--keep-in-use` | Keep relations that other queries read within `janitor.in_use_window` instead of archiving them, for this run. Same as `janitor.keep_in_use = true`. See [Relations in use](#relations-in-use) | -| `--drop-old-name-view ` | Drop the [compatibility view](/docs/concepts/models/migrations#old-names) at a renamed model's old name before it expires. Repeat for several views. Fails with `C503` if no compatibility view is recorded at the name | -| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | - -## Output - -After the preview and confirmation, each phase with work announces how many actions it runs, prints one line per action as it finishes, and closes with its duration. The run ends with one line of totals: - ``` Archiving 3 relations... 1/3 archived analytics.old_orders -> analytics._sqb_archive__20261022t101500z__old_orders OK 0.21s @@ -45,183 +26,296 @@ Deleted 1 expired archive. (0.12s) ✓ Janitor completed archived 3, deleted 1, pruned 0 (0.81s) ``` -The phases are archiving stale relations, deleting expired archives, dropping old-name views, dropping expired query artifacts, dropping leftover review schemas and pruning direct state, in that order. If an action fails, its line shows `FAIL` and the error, no further action starts, and the run ends with `✗ Janitor failed` and the totals completed before the failure, exiting with `1`. +The janitor is off until you set `enabled = true` under [`[janitor]`](#configuration). -Long preview sections show their first 10 entries and then `... and N more (use --verbose to show all)`. Skipped objects are counted by reason: +## Usage -``` -Skipped objects (385) - 210 relation is newer than 14 days - 175 relation is not tracked by SQLBuild +```bash +sqb --project-dir janitor [flags] ``` -With `--json`, stdout holds one document with `status` (`completed`, `failed`, `cancelled`, `blocked`, `nothing_to_do`, `previewed` or `disabled`), the full `plan`, the `usage_check`, the `result` with counts and every action, and `error`. Errors before the cleanup starts, such as a refused confirmation, are reported on stderr only. - -## Relations in use +## Flags -Archiving renames a relation, so anything that still reads it fails from then on. Before archiving, the janitor looks up which candidates other queries read within `in_use_window` (default `7d`) and shows them, with their readers, in an `In use` section: +| Flag | Description | +|------|-------------| +| `--target ` | Clean up another configured target, with that target's connection, schemas and state | +| `--as ` | Preview the plan for another target's schemas through the active target's connection. Changes nothing and doesn't prompt. Can't be combined with `--auto-approve` | +| `--auto-approve` | Run the plan without the confirmation prompt | +| `--retention-days` | Override `retention_days` for this run | +| `--direct-state-history-versions` | Override how many state-history versions are kept per identity | +| `--keep-in-use` | Keep relations read within `in_use_window` instead of archiving them, for this run. See [Relations in use](#relations-in-use) | +| `--drop-old-name-view ` | Drop the [compatibility view](/docs/concepts/models/migrations#old-names) at a renamed model's old name before it expires. Repeat for several views. Fails with `C503` if no compatibility view is recorded at the name | +| `--verbose`, `-v` | List every entry of each preview section and every skipped object with its reason | +| `--json` | Print the plan and the result as one JSON document on stdout; the preview, prompt and progress go to stderr | +| `--warehouse ` | Snowflake warehouse for this run; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | -``` -In use - analytics.orders_daily queried 2026-10-21 by svc_reporting, alice (14 queries in 7 days) - analytics.old_orders queried 2026-10-20 by svc_reporting, alice, bob +2 more (40 queries in 7 days); frozen since 2026-09-01, still in use - analytics.old_customers queried 2026-10-21 by alice via analytics.customer_report (3 queries in 7 days); frozen since 2026-09-01, still in use -``` +## Lifecycle -Each line names the latest read, up to three readers (most queries first) and how many queries read the relation. Reading a view also reads what it is built on, so those relations are in use too. When a relation was only read that way, the line says `via` and names the views, or says `through a view` when the warehouse doesn't record which. A stale table that is still read is also flagged `frozen since , still in use`: its data stopped changing on that date, so its readers are getting old data. Views are not flagged, because their reads stay live. The confirmation prompt states the count, for example `14 relations will be archived; 3 of them are in use.` +A relation is **stale** when all of these are true: -By default in-use relations are still archived, which is also how their readers find out. To keep them instead, set `keep_in_use = true` under `[janitor]` or pass `--keep-in-use`. In-use relations are then listed under `Kept (in use)` with their readers and are left out of the archive list and the counts. A later run archives them once nothing reads them within the window. +- it is in a schema the project writes to; +- the current, complete project doesn't produce it; +- SQLBuild tracked it, when `delete_tracked_only = true` (the default); +- it matches no `exclude_patterns` entry, ignoring case. -Only relations to archive are checked. Expired archives and state-table pruning are not. +A stale relation goes through two stages: -Usage comes from the warehouse's query history: +1. **Archive.** Once it has been stale for `retention_days` (default `14`), the janitor renames it, + in the same schema, to `_SQB_ARCHIVE____`. For example, + `stg_orders` becomes `_SQB_ARCHIVE__20260924T101500Z__stg_orders`. A rename copies no data and + keeps the relation's table type, grants and time travel. +2. **Delete.** Once the archive is older than `archive_retention_days` (default `14`), the janitor + drops it. -| Adapter | Usage source | -|---------|--------------| -| Snowflake | `SNOWFLAKE.ACCOUNT_USAGE.ACCESS_HISTORY` (`direct_objects_accessed`, `base_objects_accessed`) joined with `QUERY_HISTORY`, and `OBJECT_DEPENDENCIES` | -| BigQuery | `` `.region-`.INFORMATION_SCHEMA.JOBS `` (`referenced_tables`, `referenced_logical_views`) | -| Databricks | `system.access.table_lineage` (`source_table_full_name`, `direct_access`) | -| ClickHouse | `system.query_log` (`tables`) on the connected server | -| Other adapters | Not available | +Setting both to `0` archives and deletes stale relations in the same run. -On Snowflake, one query reads the access history of all candidates (up to 1,000 names per query). A candidate listed in a query's base objects but not its direct objects was read through the objects the query named, which the line names after `via`. When something was read, a second query reads the view dependencies in `OBJECT_DEPENDENCIES` for the databases of the candidates and of the objects queries read directly. Anything those directly read objects are built on, through any number of views, is in use: an unread candidate reached that way is in use `via` the directly read object, with its readers. So a candidate view between a non-candidate view someone reads and a candidate table is kept too. Links through views in other databases are not followed. Dependencies created by `CREATE TABLE ... AS SELECT` and similar statements are not recorded there. Queries SQLBuild tags as its own (`QUERY_TAG` with `"app":"sqlbuild"`, which builds, audits, loads and hooks set) and `CREATE ... CLONE` statements of tables, dynamic tables, schemas and databases don't count as reads. A statement counts as a clone only when it starts with `CREATE`, after any comments, and names its source after `CLONE`; a column, string or comment containing the word doesn't make it one. SQLBuild's own reads don't need this check: schemas holding the project's sources are skipped, so the project graph already protects what the project reads. Access history needs Enterprise edition. The janitor's role needs `IMPORTED PRIVILEGES` on the `SNOWFLAKE` database, or the `SNOWFLAKE.GOVERNANCE_VIEWER` database role for access history and the `SNOWFLAKE.OBJECT_VIEWER` database role for object dependencies: +Archive names are written as unquoted lowercase identifiers, so warehouses that fold unquoted names +show them in their own case; on Snowflake the example above appears as +`_SQB_ARCHIVE__20260924T101500Z__STG_ORDERS`. Long names are shortened with a short hash to fit the +adapter's identifier limit. The prefix and timestamp are never shortened. -```sql -GRANT DATABASE ROLE SNOWFLAKE.GOVERNANCE_VIEWER TO ROLE janitor_role; -GRANT DATABASE ROLE SNOWFLAKE.OBJECT_VIEWER TO ROLE janitor_role; -``` +An archive's age comes from the timestamp in its name, so deleting needs no other state. -Without `OBJECT_VIEWER`, reads are still checked, but view chains are not followed and the preview says `view chains not followed: grant SNOWFLAKE.OBJECT_VIEWER`. Access history and object dependencies lag by up to about 3 hours, so reads in the last few hours may not show yet. +### Relation age -On BigQuery, the janitor reads the `INFORMATION_SCHEMA.JOBS` view of each candidate's project, in the location of its dataset, and counts successful query jobs that reference the relation, by `user_email`. A job lists the views it reads as well as the tables behind them, so both count. Whether views nested inside other views are listed is not documented, so a candidate view that is only read through another view may not show as in use. Jobs answered from cached results record no tables or views, so with an `in_use_window` under about a day a reader whose queries were all cached can be missed. Jobs that run in another project are not in that view, so their reads are not seen. Script parent jobs, `CREATE SNAPSHOT TABLE` and `CREATE TABLE ... CLONE` don't count. SQLBuild sets no job labels on BigQuery, so its own jobs count like any other. The janitor's role needs `bigquery.jobs.listAll` and `bigquery.jobs.create` on each project, for example through the BigQuery Resource Viewer and BigQuery User roles. The view keeps 180 days of jobs, so a longer `in_use_window` sees only the last 180 days. +`retention_days` counts from the last time the relation was written, as the warehouse reports it. +For a relation the project no longer builds, that is roughly when it stopped being built. -On Databricks, the janitor counts read events in Unity Catalog's table lineage, by `created_by`. Lineage records the view a query reads as a direct read and the tables behind it as reads through the view, which the line names when lineage records the view for the same statement. It needs Unity Catalog with the `access` system schema enabled, and `USE CATALOG` on `system`, `USE SCHEMA` on `system.access` and `SELECT` on `system.access.table_lineage`: +| Adapter | Age source | +|---------|------------| +| Snowflake | `INFORMATION_SCHEMA.TABLES` `CREATED` and `LAST_ALTERED`, which change on every write | +| Databricks | `DESCRIBE DETAIL` `lastModified` for tables, and `INFORMATION_SCHEMA.TABLES` for views | +| BigQuery | `__TABLES__` `creation_time` and `last_modified_time` | +| PostgreSQL, SQL Server, DuckDB, MotherDuck | Not available | -```sql -GRANT USE CATALOG ON CATALOG system TO `janitor-principal`; -GRANT USE SCHEMA ON SCHEMA system.access TO `janitor-principal`; -GRANT SELECT ON TABLE system.access.table_lineage TO `janitor-principal`; -``` +When the warehouse can't report an age, the janitor skips the relation unless `retention_days` is +`0`. SQL Server is in this group because its modification date ignores inserts and updates, so a +recently written table could look old. -Lineage records only reads Databricks can infer lineage for, and can arrive with a delay, so "not in use" is a weaker signal than on Snowflake. Lineage has no query text, so clones and SQLBuild's own reads can't be told apart and count as reads. The table keeps one year of events. +### Restoring an archive -On ClickHouse, the janitor counts finished `SELECT` and `INSERT` queries in `system.query_log` of the server it connects to, by the user who started them. A query's `tables` lists every table and view it touched, including the views and tables a view is built on, so a whole view chain shows as in use, without `via`; queries a query starts count once, for its initial query. Only the connected server's `query_log` is read: queries on other replicas or shards are logged on those servers and are not seen, which matters on ClickHouse Cloud and behind load-balanced replicas. When `system.clusters` shows other replicas of the connected server's shard, the preview says `usage read from this server only; other replicas not checked`. Writes into a relation count as use, since renaming it breaks the writer too. ClickHouse answers some queries, such as a plain `count()` of a table, from metadata without logging the table, so those reads are not seen. SQLBuild doesn't mark its ClickHouse queries, so its own reads count. The janitor's user needs `SELECT` on `system.query_log`: +Rename the archive back to its original name before it expires. For example, on Snowflake: ```sql -GRANT SELECT ON system.query_log TO janitor_user; +ALTER TABLE analytics._SQB_ARCHIVE__20260924T101500Z__stg_orders RENAME TO analytics.stg_orders; ``` -Queries are logged when `log_queries` is on, which is the default; when it is off for the janitor's connection, usage is not checked. Readers whose settings profile turns `log_queries` off, or logs only a sample with `log_queries_probability`, are missed. The log is flushed about every 7.5 seconds, and ClickHouse keeps it until a TTL configured on the server removes it. +If the archive's name was shortened, look up the original name in the +[audit record](#audit-record). After an archive is deleted, your warehouse's time travel or undrop +feature may still recover it within its own retention window. -When usage can't be checked, because the adapter has no usage source or the role can't read its query history, the preview and the prompt show one notice, such as `usage not checked: unsupported adapter`, and the janitor archives as usual. With `keep_in_use` on, a second notice, `nothing kept in use`, says that relations in use are archived too. +## What else the janitor cleans -In the `--json` plan, each entry of `relations_to_archive` and `kept_in_use` has `in_use`: `null` when usage was not checked, `false` when nothing read the relation within the window, or an object with `last_read`, `readers`, `query_count` and `via` (`null` when the relation was read directly, otherwise the names of the views it was read through, empty when unknown). Entries of `kept_in_use` have only `name`, `age_days` and `in_use`, since they are not archived. The top-level `usage_check` has `status` (`checked` or `not_checked`), `reason`, `window`, `keep_in_use` and `notices` (what a check that ran left out, such as view chains it could not follow), or is `null` when there was nothing to check. +Each run works through these phases, in order: -## Speed +1. archive stale relations; +2. delete expired archives; +3. drop expired [compatibility views](#compatibility-views); +4. drop expired query artifacts; +5. drop leftover [review schemas](#review-schemas); +6. [prune state history](#state-history-pruning). -The janitor reads each schema's relations once per run, and the definitions of compatibility views with one read per schema. On warehouses that run DDL concurrently (Snowflake, BigQuery and Databricks), the independent actions of each phase run on up to the target's [`concurrency`](/docs/concepts/project-configuration#target-concurrency) connections at once; phases still run one after another. DuckDB, SQLite and the other adapters apply one action at a time. - -## Targets +### Compatibility views -By default the janitor cleans up the active target. `--target ` selects another configured target, exactly like `sqb build --target`: the janitor connects with that target's connection and inspects and cleans that target's schemas and state tables. Other targets are never touched. - -`--as ` previews what the janitor would do in another target without changing anything, like [`sqb plan --as`](/docs/cli/plan). The project compiles for that target, and its schemas are inspected through the active target's connection, which can be the default target or the one given with `--target`. The janitor lists the relations it would archive, the archives it would delete, the compatibility views it would drop and the state tables it would prune (all of them with `--verbose`), then exits. It drops nothing, writes no audit or state rows, prunes nothing and does not prompt. `--drop-old-name-view` is allowed with `--as` and only previews the drop. +When a model is renamed, SQLBuild keeps a [compatibility view](/docs/concepts/models/migrations#old-names) +at its old name. The janitor lists each one with the model it reads and its expiry: ``` -Previewing janitor as target 'prod' through the connection of target 'dev' (inspection only). -... -Relations to archive - prod.customers -> prod._sqb_archive__20261001t101500z__customers age unknown, delete after 2026-10-15 10:15:00 UTC - -Archives to delete - prod._sqb_archive__20200101t000000z__old_products archived 2020-01-01 00:00:00 UTC, age 2465d, expired 2020-01-15 00:00:00 UTC - -Previewed janitor as target 'prod' through the connection of target 'dev'. Nothing was changed. -Rerun with `--target prod` instead of `--as prod` to apply it. +Old name views (1) +└── prod.revenue -> model:daily_revenue + └── expires 2026-10-28 (live; drop early with --drop-old-name-view prod.revenue) ``` -An unknown target name fails before anything connects. A preview exits with `0`, or with `1` when a schema is blocked, as a normal run would. - -## Lifecycle - -The janitor never drops a live relation directly. Each stale relation goes through two stages: - -1. **Archive.** Once a relation has been stale for `retention_days`, the janitor renames it in place, in the same schema, to: - - ``` - _SQB_ARCHIVE____ - ``` - - For example, `stg_orders` becomes `_SQB_ARCHIVE__20260924T101500Z__stg_orders`. A rename copies no data, and keeps the relation's table type, grants and time travel. -2. **Delete.** Once an archive is older than `archive_retention_days`, the janitor drops it. - -Stale relations are identified exactly as before. A relation is a candidate only if it is in a schema the project writes to, it is not produced by the current complete project, it was tracked by SQLBuild (when `delete_tracked_only = true`), it matches no exclude pattern, and it is older than the retention period. +A compatibility view is never archived as a stale relation. Once it expires, or when you name it +with `--drop-old-name-view`, the janitor drops it and records the drop in +`_sqlbuild_old_name_views`. -The age of an archive comes from the timestamp in its name, so deletion needs no other state. The janitor only deletes relations whose names **strictly** match the archive format and that are in a schema the project writes to. Names that only look similar are reported and never touched. The janitor also only acts on relations whose names are plain lowercase identifiers (letters, digits and underscores), and that don't collide case-insensitively with another relation in the same schema. Anything else is reported and left alone. +The janitor drops the view only if the relation at the old name is still the view SQLBuild created. +Any other relation there, such as a view a project model built after an interrupted early drop, is +left alone. The janitor then records the compatibility view as dropped and lists it as +`record dropped (name now used by another relation, model:revenue)`, naming the model when the +name belongs to one. A compatibility view that no longer exists is recorded as dropped too. -Archive names are written as unquoted lowercase identifiers, so warehouses that fold unquoted names show them in their own case. On Snowflake, for example, the archive appears as `_SQB_ARCHIVE__20260924T101500Z__STG_ORDERS`. +### Review schemas -Long names are shortened with a short hash so they fit the adapter's identifier limit. The prefix and timestamp are never shortened. +[`sqb review`](/docs/cli/review) builds into a schema it creates and drops for one run. When a +review couldn't drop its schema, the janitor drops it once the timestamp in its name is more than 24 +hours old, so a running review is never touched. Only schemas with a review ownership marker in the +active target schema's `_sqlbuild_fingerprints` table are dropped; a schema whose name only looks +like a review schema is left alone. -Setting both `retention_days` and `archive_retention_days` to `0` archives and deletes stale relations in the same run. +### State history pruning -## Review schemas +The janitor prunes old rows from `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness`, keeping +the latest `direct_state_history_versions` rows per identity (default `20`). This is separate from +relation retention. -[`sqb review`](/docs/cli/review) builds into a schema it creates for one run and drops when the run ends. Before creating the schema it writes an ownership marker to the active target schema's `_sqlbuild_fingerprints` table. When a review could not drop its schema, the janitor drops it once the timestamp in its name is more than 24 hours old, so a review that is still running is never touched. Only schemas with a marker in the active target schema are dropped; a schema whose name merely looks like a review schema is left alone. - -## Compatibility views +## Safety -When a model is renamed, SQLBuild keeps a [compatibility view](/docs/concepts/models/migrations#old-names) at its old name. The janitor lists each one with the model it reads and its expiry: +- **You confirm the plan.** The preview lists the planned archives and deletes with exact names, + ages and expiry, then asks you to type an exact confirmation phrase. Use `--auto-approve` only in + automation, or when you're certain. +- **Only exact archive names are deleted.** An archive is deleted only when its name strictly + matches the archive format and it is in a schema the project writes to. Names that only look + similar are reported and never touched. +- **Unusual names are left alone.** The janitor acts only on plain lowercase identifiers (letters, + digits and underscores) that don't collide, ignoring case, with another relation in the same + schema. Anything else is reported and skipped. +- **Pending migration origins are kept.** While a model declares `migrate_from` and its own + relation doesn't exist yet in the target, the origin isn't archived and is listed as + `pending migration origin for `. A bare `migrate_from` name is matched in each schema the + project builds into; a qualified name only at that location. Once the model has built, the old + name holds a compatibility view, which expires as above. +- **Schemas with sources are protected.** A schema that holds the project's configured sources is + skipped and listed under `Skipped schemas`, so what the project reads is never archived. When such a schema is one SQLBuild manages, the + preview shows `Janitor blocked`, lists the actions it suppressed, and the run exits `1`. +- **Run the same SQLBuild version as your builds.** An older janitor doesn't know compatibility + views and archives them as stale relations. + +### Relations in use + +Archiving renames a relation, so anything that still reads it fails from then on. Before +archiving, the janitor checks which candidates other queries read within `in_use_window` (default +`7d`) and lists them, with their readers: ``` -Old name views (1) -└── prod.revenue -> model:daily_revenue - └── expires 2026-10-28 (live; drop early with --drop-old-name-view prod.revenue) +In use + analytics.orders_daily queried 2026-10-21 by svc_reporting, alice (14 queries in 7 days) + analytics.old_orders queried 2026-10-20 by svc_reporting, alice, bob +2 more (40 queries in 7 days); frozen since 2026-09-01, still in use + analytics.old_customers queried 2026-10-21 by alice via analytics.customer_report (3 queries in 7 days); frozen since 2026-09-01, still in use ``` -A compatibility view is never archived as a stale relation. Once it expires, or when it is named with `--drop-old-name-view`, the janitor drops it and records the drop in `_sqlbuild_old_name_views`. Before dropping, the janitor checks that the relation at the old name is still the view SQLBuild created there, by comparing its stored definition with the view's SQL. Any other relation at that name, such as a view a project model built after an interrupted early drop, is left alone: the janitor records the compatibility view as dropped and lists it as `record dropped (name now used by another relation, model:revenue)`, naming the model when the name is a project destination. A compatibility view that no longer exists is recorded as dropped too. +Each line shows the latest read, up to three readers (most queries first) and the query count. -Run the same SQLBuild version for `sqb janitor` as for builds. An older janitor does not know compatibility views and archives them as stale relations. +- Reading a view also reads what it's built on, so those relations are in use too. A relation read + only that way shows `via` and the views, or `through a view` when the warehouse doesn't record + which. +- A stale table that is still read is flagged `frozen since , still in use`: its data stopped + changing on that date, so its readers get old data. Views aren't flagged, because their reads stay + live. +- The prompt states the count, for example + `14 relations will be archived; 3 of them are in use.` -## Pending migration origins +By default, in-use relations are still archived, which is also how their readers find out. To keep +them, set `keep_in_use = true` or pass `--keep-in-use`. They are then listed under `Kept (in use)` +and left out of the archive list and the counts. A later run archives them once nothing reads them +within the window. Only relations to archive are checked, not expired archives or state pruning. -A relation that a project model still has to migrate from is never archived. While a model declares `migrate_from` and its own relation does not exist yet in the target, the janitor skips the origin and lists it as `pending migration origin for `. A bare `migrate_from` name is matched in each schema the project builds into; a qualified name only at that location. Once the model has built, the origin is handled as usual: after a migration, the old name holds a compatibility view, which expires as described above. +Usage comes from the warehouse's query history: -## Relation age +| Adapter | Source | Grants | Caveats | +|---------|--------|--------|---------| +| Snowflake | `ACCOUNT_USAGE.ACCESS_HISTORY` with `QUERY_HISTORY`, and `OBJECT_DEPENDENCIES` for view chains | `IMPORTED PRIVILEGES` on the `SNOWFLAKE` database, or the `SNOWFLAKE.GOVERNANCE_VIEWER` and `SNOWFLAKE.OBJECT_VIEWER` database roles | Needs Enterprise edition. Lags up to about 3 hours | +| BigQuery | `` `.region-`.INFORMATION_SCHEMA.JOBS `` of each candidate's project, in its dataset's location | `bigquery.jobs.listAll` and `bigquery.jobs.create` on each project, for example BigQuery Resource Viewer and BigQuery User | Keeps 180 days of jobs | +| Databricks | `system.access.table_lineage` | `USE CATALOG` on `system`, `USE SCHEMA` on `system.access`, `SELECT` on `system.access.table_lineage` | Needs Unity Catalog with the `access` system schema. Keeps one year of events | +| ClickHouse | `system.query_log` of the connected server | `SELECT` on `system.query_log` | Only the connected server's log is read | +| Other adapters | Not available | | | -`retention_days` is measured from the last time a relation was written, as reported by the warehouse. For a relation the project no longer builds, that is roughly when it stopped being built. +```sql +-- Snowflake +GRANT DATABASE ROLE SNOWFLAKE.GOVERNANCE_VIEWER TO ROLE janitor_role; +GRANT DATABASE ROLE SNOWFLAKE.OBJECT_VIEWER TO ROLE janitor_role; -| Adapter | Age source | -|---------|------------| -| Snowflake | `INFORMATION_SCHEMA.TABLES` `CREATED` and `LAST_ALTERED`, which change on every write | -| Databricks | `DESCRIBE DETAIL` `lastModified` for tables, and `INFORMATION_SCHEMA.TABLES` for views | -| BigQuery | `__TABLES__` `creation_time` and `last_modified_time` | -| PostgreSQL, SQL Server, DuckDB, MotherDuck | Not available | +-- Databricks +GRANT USE CATALOG ON CATALOG system TO `janitor-principal`; +GRANT USE SCHEMA ON SCHEMA system.access TO `janitor-principal`; +GRANT SELECT ON TABLE system.access.table_lineage TO `janitor-principal`; -When the warehouse can't report a relation's age, the janitor skips that relation whenever `retention_days` is greater than `0`. SQL Server is in this group because its modification date ignores inserts and updates, so a recently written table could look old. +-- ClickHouse +GRANT SELECT ON system.query_log TO janitor_user; +``` -Ages are only looked up for relations that are otherwise candidates, and not at all when `retention_days` is `0`. +When usage can't be checked, because the adapter has no source or the role can't read it, the +preview and the prompt show one notice, such as `usage not checked: unsupported adapter`, and the +janitor archives as usual. With `keep_in_use` on, a second notice, `nothing kept in use`, says that +in-use relations are archived too. + +#### What counts as a read + +**Snowflake.** A relation read through the objects a query named shows `via` those objects. +Everything a directly read object is built on, through any number of views, is in use, so a +candidate view between a view someone reads and a candidate table is kept too. Links through views +in other databases aren't followed, and dependencies created by `CREATE TABLE ... AS SELECT` and +similar statements aren't recorded. These don't count as reads: + +- queries SQLBuild tags as its own (`QUERY_TAG` with `"app":"sqlbuild"`, set by builds, audits, + loads and hooks); +- `CREATE ... CLONE` of tables, dynamic tables, schemas and databases. A statement counts as a clone + only when it starts with `CREATE`, after any comments, and names its source after `CLONE`. + +Without `OBJECT_VIEWER`, reads are still checked but view chains aren't followed, and the preview +says `view chains not followed: grant SNOWFLAKE.OBJECT_VIEWER`. + +**BigQuery.** Successful query jobs that reference the relation count, by `user_email`. A job lists +the views it reads and the tables behind them, so both count. These are missed or don't count: + +- a candidate view read only through another view may not show, because BigQuery doesn't document + whether nested views are listed; +- jobs answered from cached results record no tables, so with an `in_use_window` under about a day + a reader whose queries were all cached can be missed; +- jobs that run in another project aren't seen; +- script parent jobs, `CREATE SNAPSHOT TABLE` and `CREATE TABLE ... CLONE` don't count; +- SQLBuild's own jobs count like any other. + +**Databricks.** Read events in Unity Catalog's table lineage count, by `created_by`. A view a query +reads is a direct read, and the tables behind it are reads through the view. Lineage covers only +reads Databricks can infer lineage for, and can arrive late, so "not in use" is a weaker signal than +on Snowflake. Clones and SQLBuild's own reads count as reads. + +**ClickHouse.** Finished `SELECT` and `INSERT` queries count, by the user who started them, once per +initial query. A query lists every table and view it touched, so a whole view chain shows as in +use, without `via`. Writes count, since renaming the relation breaks the writer too. These are +missed or don't count: + +- queries on other replicas or shards, which matters on ClickHouse Cloud and behind load-balanced + replicas. When `system.clusters` shows other replicas of the connected server's shard, the preview + says `usage read from this server only; other replicas not checked`; +- queries ClickHouse answers from metadata without logging the table, such as a plain `count()`; +- readers whose settings profile turns `log_queries` off or samples it with + `log_queries_probability`. When `log_queries` is off for the janitor's own connection, usage isn't + checked; +- queries in the last few seconds: the log is flushed about every 7.5 seconds; +- queries older than the server's `query_log` TTL; +- SQLBuild's own reads count. + +### Audit record + +Every archive and delete is recorded in an append-only `_sqlbuild_janitor_events` table in the +affected schema: the original name, the archive name, the time and the invocation. The janitor +never reads it to decide what to do; it is for auditing, and for restoring archives whose names +were shortened, which are recorded right after their rename. Other events are written in batches +of up to 100 per schema, so if the process is killed, up to 100 actions in a schema may have no +event. Those archives still expire by the timestamp in their names. -## Audit record +## Targets -Every archive and every delete is recorded in an append-only `_sqlbuild_janitor_events` table in the affected schema. It records the original name, the archive name, the time and the invocation. The janitor never reads this table to decide what to do. It exists for auditing, and for restoring archives whose names were shortened. +By default the janitor cleans the active target. `--target ` selects another configured +target, exactly like `sqb build --target`. Other targets are never touched. -Events are written in batches of up to 100 per schema, after each batch of actions and after a failure. An archive whose name was shortened gets its event right after its rename. If the process is killed, at most one batch of actions in a schema can lack events; those archives still expire by the timestamp in their names. +`--as ` previews another target without changing anything, like +[`sqb plan --as`](/docs/cli/plan). The project compiles for that target, and its schemas are +inspected through the active target's connection. The preview lists what would be archived, +deleted, dropped and pruned, then exits. It drops nothing, writes no audit or state rows and doesn't +prompt. `--drop-old-name-view` with `--as` only previews the drop. -## Restoring an archive +``` +Previewing janitor as target 'prod' through the connection of target 'dev' (inspection only). +... +Relations to archive + prod.customers -> prod._sqb_archive__20261001t101500z__customers age unknown, delete after 2026-10-15 10:15:00 UTC -Rename the archive back to its original name before it expires, using your warehouse's rename statement. For example, on Snowflake: +Archives to delete + prod._sqb_archive__20200101t000000z__old_products archived 2020-01-01 00:00:00 UTC, age 2465d, expired 2020-01-15 00:00:00 UTC -```sql -ALTER TABLE analytics._SQB_ARCHIVE__20260924T101500Z__stg_orders RENAME TO analytics.stg_orders; +Previewed janitor as target 'prod' through the connection of target 'dev'. Nothing was changed. +Rerun with `--target prod` instead of `--as prod` to apply it. ``` -If the archive's name was shortened, look up the original name in `_sqlbuild_janitor_events`. After an archive has been deleted, your warehouse's own time travel or undrop feature may still be able to recover it within its retention window. +An unknown target name fails before anything connects. A preview exits `0`, or `1` when a schema +is blocked, as a normal run would. ## Configuration -Configure janitor behavior in `sqlbuild_project.toml`: - ```toml [janitor] enabled = true @@ -233,18 +327,59 @@ in_use_window = "7d" keep_in_use = false ``` -When `retention_days` is not set, it defaults to `14`. See [Project Configuration](/docs/concepts/project-configuration#janitor) for every janitor setting. +See [Project Configuration](/docs/concepts/project-configuration#janitor) for every key and default. +On Snowflake, BigQuery and Databricks, the actions of each phase run on up to the target's +[`concurrency`](/docs/concepts/project-configuration#target-concurrency) connections at once. + +## Output + +After the preview and confirmation, each phase with work prints how many actions it runs, one line +per action, and its duration, as in the example at the top. The run ends with one line of totals. + +If an action fails, its line shows `FAIL` and the error, no further action starts, and the run ends +with `✗ Janitor failed` and the totals completed so far. The exit code is `1`. + +Without `--verbose`, each preview section shows its first 10 entries and then +`... and N more (use --verbose to show all)`, and skipped objects are counted by reason: + +``` +Skipped objects (385) + 210 relation is newer than 14 days + 175 relation is not tracked by SQLBuild +``` + +### JSON output + +With `--json`, stdout holds one document: + +| Field | Contents | +|-------|----------| +| `status` | `completed`, `failed`, `cancelled`, `blocked`, `nothing_to_do`, `previewed` or `disabled` | +| `plan` | The full plan | +| `usage_check` | `status` (`checked` or `not_checked`), `reason`, `window`, `keep_in_use` and `notices` (what a check left out, such as view chains it couldn't follow); `null` when there was nothing to check | +| `result` | Counts and every action | +| `error` | The error, if any | + +Each entry of `plan.relations_to_archive` and `plan.kept_in_use` has `in_use`: + +- `null` when usage wasn't checked; +- `false` when nothing read the relation within the window; +- otherwise an object with `last_read`, `readers`, `query_count` and `via` (`null` for a direct + read, otherwise the views it was read through, empty when unknown). + +Entries of `kept_in_use` have only `name`, `age_days` and `in_use`. Errors before the cleanup +starts, such as a refused confirmation, go to stderr only. ## Examples ```bash -# Interactive mode (shows the plan and prompts for confirmation) +# Show the plan and prompt for confirmation sqb janitor -# Run the planned archives and deletes without prompting +# Run the plan without prompting sqb janitor --auto-approve -# Override retention to 7 days +# Archive relations stale for 7 days instead of the configured retention sqb janitor --retention-days 7 # Preview the prod cleanup through the current connection, changing nothing @@ -256,11 +391,3 @@ sqb janitor --target prod # Keep relations that are still read instead of archiving them sqb janitor --keep-in-use ``` - -## State history pruning - -In addition to archiving stale relations, the janitor prunes old rows from `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness`, keeping the latest `direct_state_history_versions` rows per identity (default `20`). This is configured separately from relation retention. - -## Safety - -The janitor shows the planned archives and deletes, with exact names, ages and expiry (every one of them with `--verbose`, or in the `--json` plan), then asks you to type an exact confirmation phrase. Relations whose names differ only by case from another relation in the same schema are reported and never dropped, and exclude patterns match regardless of case. Use `--auto-approve` only in automation, or when you're certain. From 07ad94839f5e1be0d059912fe094e3680a88f1d8 Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Tue, 27 Oct 2026 09:09:26 +0000 Subject: [PATCH 05/10] docs: cut ClickHouse storage internals and history notes from the recall page --- .../docs/concepts/observability/recall.md | 120 ++++++++---------- .../docs/concepts/observability/recall.mdx | 120 ++++++++---------- 2 files changed, 108 insertions(+), 132 deletions(-) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/recall.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/recall.md index aa69e1b5b4..d4ee10a7ed 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/recall.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/recall.md @@ -168,10 +168,6 @@ from, so a local check reads differently from a failed scheduled build: `integration_run_id` and `integration_retry`. The Dagster integration also records `step_key` and `partition_key`, which you read from `external_context` with the store's JSON functions. -Invocations recorded by an older SQLBuild version have `origin` and `command_mode` set to `NULL`. -Older versions also stored `external_context` in a form that is not JSON; for those runs the -`integration_*` columns are `NULL` and the query does not fail. - ## Settings Configure Recall in `[recall]` in `sqlbuild_project.toml`. A `[targets..recall]` table @@ -246,10 +242,6 @@ find the same corrupt file coordinate through `recall.sqlite3.lock`, so only one A `write_connection` on any adapter other than `sqlite` or `clickhouse` is not supported; Recall prints a notice and records nothing. -`write_connection` and `read_connection` replaced the single `connection` key. A configuration that -still sets `connection` fails with an error naming the replacements: set both keys to the old -connection to keep the previous behaviour. - ## Sharing history in ClickHouse Point `write_connection` and `read_connection` at ClickHouse connections to keep one history for a @@ -294,7 +286,7 @@ writer's password on a reader's laptop, need not be set. ### Users and grants -SQLBuild does not enforce grants; it only lets the roles be separated. Give the writer the rights to +SQLBuild doesn't enforce grants; it lets you separate the roles. Give the writer the rights to create and upgrade the store in its database, so the first command sets it up and later SQLBuild versions upgrade it without anyone running DDL by hand: @@ -307,64 +299,60 @@ GRANT CREATE DATABASE, CREATE TABLE, CREATE VIEW, DROP TABLE, ALTER ADD COLUMN, GRANT SELECT ON observability.* TO recall_reader; ``` -`DROP TABLE` is there because ClickHouse replaces a view through a temporary table; Recall itself -never drops a table. +`DROP TABLE` is needed because ClickHouse replaces a view through a temporary table; Recall never +drops a table itself. + +| User | Rights | What happens | +|------|--------|--------------| +| Writer with the grants above | Create and upgrade the store | The first command creates the store; later versions upgrade it | +| Writer with only `SELECT` and `INSERT` | Write history | Works once the store exists. A missing or older store makes Recall print the statements an administrator should run | +| Reader with `SELECT` | Read history | `sqb recall query`, `schema` and `output` work, and can't change the store | -The reader may be a standard read-only user (`SETTINGS readonly = 1`, or a settings profile with -it), which may not change any setting. Such a user's queries run with the server's own settings; -every other reader gets the +A reader may be a standard read-only user (`SETTINGS readonly = 1`, or a settings profile with +it). Its queries run with the server's own settings; every other reader gets the [settings SQLBuild pins](../adapters/clickhouse.md#settings-sqlbuild-pins). The views -return the same rows either way. `sqb recall query` runs with -`readonly = 2`, which refuses writes but still allows query settings, unless the user is already -read-only; a user that may not set the row limit gets the same truncated result, cut off while -reading. A setting that Recall needs and the user may not change fails with an error naming it; set -it in the user's profile, or give the user `readonly = 2`, which keeps it read-only. - -A writer limited to `SELECT` and `INSERT` also works, once the store exists: it writes history but -cannot create or upgrade the store. When such a writer finds the store missing or behind, Recall -prints the exact statements an administrator should run, and `sqb recall schema --ddl` prints them -again. - -- **Setup.** On first use the writer creates the database if it is missing, printing - ``Recall: created ClickHouse database `observability` for the store.`` once, then the `recall_*` - MergeTree tables and the views above. An existing database is used as it is, whatever its engine. - The database is created on the server the writer connects to, never `ON CLUSTER`: on a - self-managed cluster whose default database engine is not `Replicated`, create the database - yourself first with `ENGINE = Replicated(...)`, and Recall leaves it alone. - ClickHouse Cloud needs nothing. - If the user may not create the store, the command still succeeds and prints the statements, - starting with `CREATE DATABASE IF NOT EXISTS` when the database is missing; a store created from - them accepts writers limited to `SELECT` and `INSERT`. -- **Upgrades.** When a store is behind this SQLBuild version and the writer may not upgrade it, the - command still runs and exits as it otherwise would. History is still written, to the columns - that exist, and Recall prints one warning (at most once a day) with the exact statements an - administrator should run; `sqb recall schema --ddl` prints them again. -- **Schema version.** The store's version is a `recall_meta` row, which retention expires like any - other. Writers append it again once it is older than half the retention, with the highest version - recorded. When it has expired anyway, a writer compares the stored tables' and views' columns - with its own: it upgrades an older store, records its version on a current one, and never - replaces the views of a store a newer SQLBuild version created. Views whose definitions show an - older version created them, such as views that set `join_use_nulls`, count as older whatever the - version row says. -- **Writes.** History is sent in batches of up to 5,000 rows about every five seconds, one insert - per table, so a long command adds few parts; everything still queued is sent when it ends, with - the same queue, shutdown budget and exit-code guarantees as the local store. Many concurrent - commands can write one store. -- **Duplicates.** Every row has a deterministic key. A retried batch is deduplicated by ClickHouse, - and the views read each key once, so a repeated delivery is counted once: views that list or count - rows read the `ReplacingMergeTree` tables with `FINAL`, and `invocations` and `statements`, which - only take the first, last, smallest or largest value of a key's copies, read them without it. Before writing, Recall looks up the batch's keys: a key already stored with - different content keeps the stored row, and the rejected copy is listed in `conflicts`. -- **Retention.** Every table carries a TTL of `retention` (SQL text a week longer, and refreshed - while it is still referenced), and the views hide rows past their TTL at once. ClickHouse drops - expired daily parts in the background, so rows may stay on disk for a while after the views stop - showing them. Retention belongs to the store: when a project's `retention` differs from the - tables' TTL, Recall warns once a day with the `ALTER TABLE … MODIFY TTL` and view statements that - would apply it, and never changes the TTL itself. -- **Reading.** `sqb recall query`, `schema` and `output` read the ClickHouse store, so queries use - ClickHouse SQL. `--json` output includes the server's `rows_read`, `bytes_read` and `query_id`. - -Use a single ClickHouse server or ClickHouse Cloud; `ON CLUSTER` stores are not supported yet. +return the same rows either way. `sqb recall query` runs with `readonly = 2`, which refuses writes +but allows query settings, unless the user is already read-only. A user that may not set the row +limit gets the same truncated result, cut off while reading. A setting Recall needs that the user +may not change fails with an error naming it: set it in the user's profile, or give the user +`readonly = 2`, which keeps it read-only. + +### Setting up and upgrading the store + +- **Setup.** On first use, the writer creates the database if it's missing, printing + ``Recall: created ClickHouse database `observability` for the store.`` once, then the store's + tables and views. An existing database is used as it is, whatever its engine. +- **Clusters.** The database is created on the server the writer connects to, never `ON CLUSTER`. + On a self-managed cluster whose default database engine isn't `Replicated`, create the database + yourself first with `ENGINE = Replicated(...)`; Recall leaves it alone. ClickHouse Cloud needs + nothing. `ON CLUSTER` stores aren't supported: use a single ClickHouse server or ClickHouse Cloud. +- **Without DDL rights.** If the writer may not create or upgrade the store, the command still + runs and exits as it otherwise would. Recall prints the statements an administrator should run, at + most once a day, starting with `CREATE DATABASE IF NOT EXISTS` when the database is missing. + `sqb recall schema --ddl` prints them again. A store behind the current version still receives + history, in the columns that exist. +- **Mixed versions.** A writer never replaces the views of a store that a newer SQLBuild version + created. +- **Many writers.** Many concurrent commands can write one store. A repeated delivery is counted + once. A record written again with different content keeps the stored row, and the rejected copy + is listed in the `conflicts` view. +- **Writes.** History is sent about every five seconds and when the command ends, with the same shutdown + budget and exit-code guarantees as the local store. + +### Retention in ClickHouse + +Retention belongs to the store. Every table has a TTL of `retention`; SQL text is kept a week +longer, and for as long as a kept invocation references it. The views hide expired rows at once, +but ClickHouse drops expired parts in the background, so rows may stay on disk for a while. + +When a project's `retention` differs from the tables' TTL, Recall warns once a day with the +`ALTER TABLE … MODIFY TTL` and view statements that would apply it. It never changes the TTL +itself. + +### Reading a ClickHouse store + +`sqb recall query`, `schema` and `output` read the ClickHouse store, so queries use ClickHouse SQL. +`--json` output includes the server's `rows_read`, `bytes_read` and `query_id`. ## Filtering what Recall stores @@ -489,5 +477,5 @@ statement. It is `NULL` on DuckDB and BigQuery; a BigQuery statement's job ID is ## Relationship to compute logs Compute logs (`logs///stdout.log`, `stderr.log`, and `diagnostics.jsonl`) are -still written alongside Recall. Recall stores the same output bytes, captured at the same point, so +written alongside Recall. Recall stores the same output bytes, captured at the same point, so `sqb recall output` matches the compute log files. diff --git a/website/src/content/docs/docs/concepts/observability/recall.mdx b/website/src/content/docs/docs/concepts/observability/recall.mdx index c6d88f50dd..401c4f772a 100644 --- a/website/src/content/docs/docs/concepts/observability/recall.mdx +++ b/website/src/content/docs/docs/concepts/observability/recall.mdx @@ -150,10 +150,6 @@ from, so a local check reads differently from a failed scheduled build: `integration_run_id` and `integration_retry`. The Dagster integration also records `step_key` and `partition_key`, which you read from `external_context` with the store's JSON functions. -Invocations recorded by an older SQLBuild version have `origin` and `command_mode` set to `NULL`. -Older versions also stored `external_context` in a form that is not JSON; for those runs the -`integration_*` columns are `NULL` and the query does not fail. - ## Settings Configure Recall in `[recall]` in `sqlbuild_project.toml`. A `[targets..recall]` table @@ -228,10 +224,6 @@ find the same corrupt file coordinate through `recall.sqlite3.lock`, so only one A `write_connection` on any adapter other than `sqlite` or `clickhouse` is not supported; Recall prints a notice and records nothing. -`write_connection` and `read_connection` replaced the single `connection` key. A configuration that -still sets `connection` fails with an error naming the replacements: set both keys to the old -connection to keep the previous behaviour. - ## Sharing history in ClickHouse Point `write_connection` and `read_connection` at ClickHouse connections to keep one history for a @@ -276,7 +268,7 @@ writer's password on a reader's laptop, need not be set. ### Users and grants -SQLBuild does not enforce grants; it only lets the roles be separated. Give the writer the rights to +SQLBuild doesn't enforce grants; it lets you separate the roles. Give the writer the rights to create and upgrade the store in its database, so the first command sets it up and later SQLBuild versions upgrade it without anyone running DDL by hand: @@ -289,64 +281,60 @@ GRANT CREATE DATABASE, CREATE TABLE, CREATE VIEW, DROP TABLE, ALTER ADD COLUMN, GRANT SELECT ON observability.* TO recall_reader; ``` -`DROP TABLE` is there because ClickHouse replaces a view through a temporary table; Recall itself -never drops a table. +`DROP TABLE` is needed because ClickHouse replaces a view through a temporary table; Recall never +drops a table itself. + +| User | Rights | What happens | +|------|--------|--------------| +| Writer with the grants above | Create and upgrade the store | The first command creates the store; later versions upgrade it | +| Writer with only `SELECT` and `INSERT` | Write history | Works once the store exists. A missing or older store makes Recall print the statements an administrator should run | +| Reader with `SELECT` | Read history | `sqb recall query`, `schema` and `output` work, and can't change the store | -The reader may be a standard read-only user (`SETTINGS readonly = 1`, or a settings profile with -it), which may not change any setting. Such a user's queries run with the server's own settings; -every other reader gets the +A reader may be a standard read-only user (`SETTINGS readonly = 1`, or a settings profile with +it). Its queries run with the server's own settings; every other reader gets the [settings SQLBuild pins](/docs/concepts/adapters/clickhouse#settings-sqlbuild-pins). The views -return the same rows either way. `sqb recall query` runs with -`readonly = 2`, which refuses writes but still allows query settings, unless the user is already -read-only; a user that may not set the row limit gets the same truncated result, cut off while -reading. A setting that Recall needs and the user may not change fails with an error naming it; set -it in the user's profile, or give the user `readonly = 2`, which keeps it read-only. - -A writer limited to `SELECT` and `INSERT` also works, once the store exists: it writes history but -cannot create or upgrade the store. When such a writer finds the store missing or behind, Recall -prints the exact statements an administrator should run, and `sqb recall schema --ddl` prints them -again. - -- **Setup.** On first use the writer creates the database if it is missing, printing - ``Recall: created ClickHouse database `observability` for the store.`` once, then the `recall_*` - MergeTree tables and the views above. An existing database is used as it is, whatever its engine. - The database is created on the server the writer connects to, never `ON CLUSTER`: on a - self-managed cluster whose default database engine is not `Replicated`, create the database - yourself first with `ENGINE = Replicated(...)`, and Recall leaves it alone. - ClickHouse Cloud needs nothing. - If the user may not create the store, the command still succeeds and prints the statements, - starting with `CREATE DATABASE IF NOT EXISTS` when the database is missing; a store created from - them accepts writers limited to `SELECT` and `INSERT`. -- **Upgrades.** When a store is behind this SQLBuild version and the writer may not upgrade it, the - command still runs and exits as it otherwise would. History is still written, to the columns - that exist, and Recall prints one warning (at most once a day) with the exact statements an - administrator should run; `sqb recall schema --ddl` prints them again. -- **Schema version.** The store's version is a `recall_meta` row, which retention expires like any - other. Writers append it again once it is older than half the retention, with the highest version - recorded. When it has expired anyway, a writer compares the stored tables' and views' columns - with its own: it upgrades an older store, records its version on a current one, and never - replaces the views of a store a newer SQLBuild version created. Views whose definitions show an - older version created them, such as views that set `join_use_nulls`, count as older whatever the - version row says. -- **Writes.** History is sent in batches of up to 5,000 rows about every five seconds, one insert - per table, so a long command adds few parts; everything still queued is sent when it ends, with - the same queue, shutdown budget and exit-code guarantees as the local store. Many concurrent - commands can write one store. -- **Duplicates.** Every row has a deterministic key. A retried batch is deduplicated by ClickHouse, - and the views read each key once, so a repeated delivery is counted once: views that list or count - rows read the `ReplacingMergeTree` tables with `FINAL`, and `invocations` and `statements`, which - only take the first, last, smallest or largest value of a key's copies, read them without it. Before writing, Recall looks up the batch's keys: a key already stored with - different content keeps the stored row, and the rejected copy is listed in `conflicts`. -- **Retention.** Every table carries a TTL of `retention` (SQL text a week longer, and refreshed - while it is still referenced), and the views hide rows past their TTL at once. ClickHouse drops - expired daily parts in the background, so rows may stay on disk for a while after the views stop - showing them. Retention belongs to the store: when a project's `retention` differs from the - tables' TTL, Recall warns once a day with the `ALTER TABLE … MODIFY TTL` and view statements that - would apply it, and never changes the TTL itself. -- **Reading.** `sqb recall query`, `schema` and `output` read the ClickHouse store, so queries use - ClickHouse SQL. `--json` output includes the server's `rows_read`, `bytes_read` and `query_id`. - -Use a single ClickHouse server or ClickHouse Cloud; `ON CLUSTER` stores are not supported yet. +return the same rows either way. `sqb recall query` runs with `readonly = 2`, which refuses writes +but allows query settings, unless the user is already read-only. A user that may not set the row +limit gets the same truncated result, cut off while reading. A setting Recall needs that the user +may not change fails with an error naming it: set it in the user's profile, or give the user +`readonly = 2`, which keeps it read-only. + +### Setting up and upgrading the store + +- **Setup.** On first use, the writer creates the database if it's missing, printing + ``Recall: created ClickHouse database `observability` for the store.`` once, then the store's + tables and views. An existing database is used as it is, whatever its engine. +- **Clusters.** The database is created on the server the writer connects to, never `ON CLUSTER`. + On a self-managed cluster whose default database engine isn't `Replicated`, create the database + yourself first with `ENGINE = Replicated(...)`; Recall leaves it alone. ClickHouse Cloud needs + nothing. `ON CLUSTER` stores aren't supported: use a single ClickHouse server or ClickHouse Cloud. +- **Without DDL rights.** If the writer may not create or upgrade the store, the command still + runs and exits as it otherwise would. Recall prints the statements an administrator should run, at + most once a day, starting with `CREATE DATABASE IF NOT EXISTS` when the database is missing. + `sqb recall schema --ddl` prints them again. A store behind the current version still receives + history, in the columns that exist. +- **Mixed versions.** A writer never replaces the views of a store that a newer SQLBuild version + created. +- **Many writers.** Many concurrent commands can write one store. A repeated delivery is counted + once. A record written again with different content keeps the stored row, and the rejected copy + is listed in the `conflicts` view. +- **Writes.** History is sent about every five seconds and when the command ends, with the same shutdown + budget and exit-code guarantees as the local store. + +### Retention in ClickHouse + +Retention belongs to the store. Every table has a TTL of `retention`; SQL text is kept a week +longer, and for as long as a kept invocation references it. The views hide expired rows at once, +but ClickHouse drops expired parts in the background, so rows may stay on disk for a while. + +When a project's `retention` differs from the tables' TTL, Recall warns once a day with the +`ALTER TABLE … MODIFY TTL` and view statements that would apply it. It never changes the TTL +itself. + +### Reading a ClickHouse store + +`sqb recall query`, `schema` and `output` read the ClickHouse store, so queries use ClickHouse SQL. +`--json` output includes the server's `rows_read`, `bytes_read` and `query_id`. ## Filtering what Recall stores @@ -472,5 +460,5 @@ statement. It is `NULL` on DuckDB and BigQuery; a BigQuery statement's job ID is ## Relationship to compute logs Compute logs (`logs///stdout.log`, `stderr.log`, and `diagnostics.jsonl`) are -still written alongside Recall. Recall stores the same output bytes, captured at the same point, so +written alongside Recall. Recall stores the same output bytes, captured at the same point, so `sqb recall output` matches the compute log files. From 531c0c95f8ac7afdf34c50dff08090bbda6d4d3a Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Tue, 27 Oct 2026 09:10:04 +0000 Subject: [PATCH 06/10] docs: drop legacy notes and duplicated timeout rules from the project configuration reference --- .../docs/concepts/project-configuration.md | 108 +++++++----------- .../docs/concepts/project-configuration.mdx | 108 +++++++----------- 2 files changed, 80 insertions(+), 136 deletions(-) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md index 17ab16b1a2..19740dcb30 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md @@ -137,10 +137,9 @@ schema = "DEV_ALICE" ``` When the project defines exactly one named connection, across `sqlbuild_project.toml` and -`sqlbuild_local.toml`, a target that does not set `connection` uses it. This default does not -apply when either file has a legacy top-level `[connection]` block or the target sets an inline -`[targets..connection]` block. With several named connections, every target must choose -one; SQLBuild rejects a target without `connection` while loading configuration. +`sqlbuild_local.toml`, a target that doesn't set `connection` uses it. With several named +connections, every target must choose one; SQLBuild rejects a target without `connection` while +loading configuration. ```toml [connections.local] @@ -151,8 +150,7 @@ schema = "dev" # uses connections.local, the only named connection ``` A command that needs the warehouse, such as `build`, `plan`, `query`, `diff`, or `debug`, fails -with `D001` before connecting when no connection is configured at all: no named connection, no -legacy `[connection]` block, and no inline target connection. Commands and modes that do not +with `D001` before connecting when no connection is configured. Commands and modes that don't connect, such as `compile`, `format`, `lineage`, `scope`, `rename`, `mv`, `test --inspect`, and `scenario test --local`, keep working without one. SQLBuild never falls back to an implicit in-memory DuckDB database; to use one, set it @@ -169,10 +167,6 @@ authoritative database and schema on `[targets.]`. SQLBuild validates conn references while loading configuration, without opening a warehouse connection, and reports an unknown `targets..connection` name as an offline configuration error. -For migration only, SQLBuild still maps legacy `[connection]` to an implicit connection and -legacy `[targets..connection]` blocks to target-specific implicit connections. These -forms are compatibility syntax, not the canonical format for new or updated projects. - ## Targets A target is a named build context - the database and schema you build into, plus execution policy (for example `dev` and `prod`). Each target references a named connection (or uses the only one defined; see [Named connections](#named-connections)) and can configure: @@ -232,9 +226,8 @@ The active target is determined by (in order of precedence): 1. `--target` on the command line (highest priority) 2. `sqlbuild_local.toml` `target` field 3. `default_target` in `sqlbuild_project.toml` -4. No target (models build to the default schema). Named connections are chosen through - targets, so warehouse commands fail with `D001` unless a target is selected or a legacy - top-level `[connection]` block exists. +4. No target. Named connections are chosen through targets, so warehouse commands fail with + `D001` unless a target is selected. A typical developer keeps shared target definitions in `sqlbuild_project.toml` and selects their normal target once in the optional local file: @@ -292,7 +285,7 @@ default warehouse for each command group, so `sqb build` runs on a large warehou ```toml [connections.warehouse] account = "my_org-my_account" -warehouse = "ANALYTICS_WH" # fallback for every command, as before +warehouse = "ANALYTICS_WH" # fallback for every command [targets.dev] connection = "warehouse" @@ -311,7 +304,7 @@ Each command connects with the first warehouse set in this order: 3. The connection's `warehouse` 4. The Snowflake user's default warehouse -Both groups are optional. A target without `warehouses` behaves exactly as before. A group +Both groups are optional; a target without `warehouses` uses the connection's `warehouse`. A group warehouse takes precedence over the connection `warehouse` in every file. In particular, a shared group overrides a connection `warehouse` you set in `sqlbuild_local.toml`; to change a group locally, set the group itself, or pass `--warehouse` for one invocation. Group values @@ -419,8 +412,8 @@ Automated tools must request approval before changing this policy. """ ``` -All fields are optional. A target without an `execution_limits` section retains unlimited existing -behavior, so production or orchestrated targets do not need placeholder values. +All fields are optional. A target without an `execution_limits` section has no limits, so +production or orchestrated targets need no placeholder values. #### Model limit @@ -443,31 +436,20 @@ Every built-in adapter enforces the limit: | Adapter | Warehouse statement limit | Cancellation at the deadline | Longest accepted duration | |---------|---------------------------|------------------------------|---------------------------| | Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement | Driver cancel request for the running query | 7 days | -| DuckDB, MotherDuck | none | Connection interrupt | about 24 days | -| PostgreSQL | `SET LOCAL statement_timeout` inside transactions | Driver cancel (`cancel_safe`) | about 24 days | -| BigQuery | Job timeout (`job_timeout_ms`) on each query job | `jobs.cancel` (asynchronous) | about 24 days | -| Databricks | `STATEMENT_TIMEOUT` session setting, capped at 48 hours | Cursor cancel (asynchronous) | about 24 days | -| SQL Server | none | Driver cancel | about 24 days | -| SQLite | none | Connection interrupt | about 24 days | - -BigQuery and Databricks stop cancelled work asynchronously, so SQLBuild reports their cancellation -as requested but unconfirmed. SQL Server has no per-statement limit; the driver's query timeout is -process-wide, so SQLBuild relies on cancelling the running statement instead. A project adapter that -cannot cancel a running statement is rejected for `max_duration` rather than presenting a limit that -may leave warehouse work running. Under a per-item timeout, such an adapter's items stop before their -next statement and report the cancellation as not supported. - -PostgreSQL never sets `statement_timeout` for the whole session, so a limit cannot leak to another -client through a transaction-mode connection pooler; statements outside a transaction are bounded by -cancellation alone. - -A duration failure may occur after earlier models have completed. Its error therefore distinguishes -the timeout from a pre-execution model-limit failure and states that warehouse changes may already -have occurred. - -Upgrade note: earlier releases accepted `max_duration` only on Snowflake and rejected it on other -adapters. It is now accepted on every built-in adapter, so a target that sets it for a non-Snowflake -warehouse is enforced instead of failing at startup. +| DuckDB, MotherDuck | None | Connection interrupt | About 24 days | +| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; statements outside a transaction rely on cancellation | Driver cancel (`cancel_safe`) | About 24 days | +| BigQuery | Job timeout (`job_timeout_ms`) on each query job | `jobs.cancel`; reported as requested, not confirmed | About 24 days | +| Databricks | `STATEMENT_TIMEOUT` session setting, capped at 48 hours | Cursor cancel; reported as requested, not confirmed | About 24 days | +| SQL Server | None; the driver's query timeout is process-wide | Driver cancel | About 24 days | +| SQLite | None | Connection interrupt | About 24 days | + +A project adapter that can't cancel a running statement is rejected for `max_duration`, rather than +showing a limit that may leave warehouse work running. Under a per-item timeout, its items stop +before their next statement and report the cancellation as not supported. + +A duration failure may come after earlier models completed. Its error therefore says it is a +timeout, unlike a model-limit failure before execution, and that warehouse changes may already +have happened. #### Per-item time limits @@ -494,9 +476,8 @@ connection before giving it to the next item and replaces it if it is no longer #### Unit test and scenario timeouts `test_timeout` and `scenario_timeout` set the per-test and per-scenario time limit on this target, -replacing `[settings] test_timeout` and `[settings] scenario_timeout`. They follow the -[timeout precedence](#unit-test-and-scenario-limits) described under Settings and accept a positive -fixed duration or `"disabled"`. +replacing `[settings] test_timeout` and `[settings] scenario_timeout`. Each takes a positive fixed +duration or `"disabled"`. See [Unit test and scenario limits](#unit-test-and-scenario-limits). #### Custom remediation @@ -704,27 +685,18 @@ delta_compile = true ### Unit test and scenario limits -A SQL unit test runs the real SQL of every unmocked model between its mocks and the model under -test as one warehouse statement. `unit_test_max_real_models` stops that statement from silently -growing into a whole-chain integration query: a test that executes more real models fails -compilation and names every real model, the dependency path that pulls it in, and the inputs to -mock. See [Testing](testing.md#cost-guards) for an example. - -`test_timeout` bounds each test's warehouse work, measured from the test's first statement. The -limit that applies to a test is, from highest precedence: - -1. `--test-timeout` on `sqb test` or `sqb build` -2. `TEST(timeout "...")` in the test file -3. `[targets..execution_limits] test_timeout` -4. `[settings] test_timeout`, then the 30-second default +| Setting | Limits | Details | +|---------|--------|---------| +| `unit_test_max_real_models` | Real (unmocked) models one SQL unit test may run; a larger test fails compilation and names each model, the path that pulls it in, and the inputs to mock | [Real-model limit](testing.md#real-model-limit) | +| `test_timeout` | Each unit test's warehouse time, from its first statement | [Time limits](testing.md#time-limits), including precedence | +| `scenario_timeout` | Each scenario's warehouse time; no default | Below | -`scenario_timeout` works the same way with `--scenario-timeout`, `SCENARIO(timeout "...")`, -`[targets..execution_limits] scenario_timeout` and `[settings] scenario_timeout`; it has -no default. At the limit, SQLBuild cancels the item's running statement, reports the item as -`TIMEOUT` with the elapsed time, the limit, where the limit was set, the cancellation outcome and -the query IDs it saw, and continues with independent items. Every value accepts a positive fixed -duration such as `"45s"` or `"10m"`, or `"disabled"`. `sqlbuild_local.toml` accepts all three -settings. +The scenario limit that applies is the first one set of `--scenario-timeout`, +`SCENARIO(timeout "...")`, `[targets..execution_limits] scenario_timeout`, and +`[settings] scenario_timeout`. At the limit, SQLBuild cancels the scenario's running statement, +reports it as `TIMEOUT`, and continues with independent items, as for +[unit tests](testing.md#time-limits). Every value is a positive fixed duration such as +`"45s"` or `"10m"`, or `"disabled"`. `sqlbuild_local.toml` accepts all three settings. ### Table promotion mode @@ -752,7 +724,7 @@ codes begin with `SQBR`; repository-defined codes begin with `XSQBR`. See | `[[rules.rule_ignores]]` | Rules turned off for `paths` or `selectors`, with a `reason`. | Rule exceptions live only in this file. Inline `-- sqb: ignore` comments are `SQBRSQL000` errors -that print the entry to add, and the former `allow_exceptions` setting is a configuration error. +that print the entry to add. ## Scopes @@ -853,7 +825,7 @@ old_name_views = "30d" | Field | Default | Description | |-------|---------|-------------| -| `old_name_views` | `"30d"` | How long a renamed model's old name keeps working through a [compatibility view](models/migrations.md#old-names) before janitor drops it. `false` keeps the old relation in place instead, as before. A model's `old_name_view` header overrides it. | +| `old_name_views` | `"30d"` | How long a renamed model's old name keeps working through a [compatibility view](models/migrations.md#old-names) before janitor drops it. `false` keeps the old relation in place instead. A model's `old_name_view` header overrides it. | ## Scenario @@ -861,7 +833,7 @@ Configuration for scenario run isolation, snapshot capture safety limits, and lo ```toml [scenario] -run_namespace = "orders-job-123" # Optional; omit to preserve existing scenario relation names +run_namespace = "orders-job-123" # Optional [scenario.snapshot_limits] max_rows_per_relation = 10000 diff --git a/website/src/content/docs/docs/concepts/project-configuration.mdx b/website/src/content/docs/docs/concepts/project-configuration.mdx index 956cc6c2ed..6f428fd607 100644 --- a/website/src/content/docs/docs/concepts/project-configuration.mdx +++ b/website/src/content/docs/docs/concepts/project-configuration.mdx @@ -112,10 +112,9 @@ schema = "DEV_ALICE" ``` When the project defines exactly one named connection, across `sqlbuild_project.toml` and -`sqlbuild_local.toml`, a target that does not set `connection` uses it. This default does not -apply when either file has a legacy top-level `[connection]` block or the target sets an inline -`[targets..connection]` block. With several named connections, every target must choose -one; SQLBuild rejects a target without `connection` while loading configuration. +`sqlbuild_local.toml`, a target that doesn't set `connection` uses it. With several named +connections, every target must choose one; SQLBuild rejects a target without `connection` while +loading configuration. ```toml [connections.local] @@ -126,8 +125,7 @@ schema = "dev" # uses connections.local, the only named connection ``` A command that needs the warehouse, such as `build`, `plan`, `query`, `diff`, or `debug`, fails -with `D001` before connecting when no connection is configured at all: no named connection, no -legacy `[connection]` block, and no inline target connection. Commands and modes that do not +with `D001` before connecting when no connection is configured. Commands and modes that don't connect, such as `compile`, `format`, `lineage`, `scope`, `rename`, `mv`, `test --inspect`, and `scenario test --local`, keep working without one. SQLBuild never falls back to an implicit in-memory DuckDB database; to use one, set it @@ -144,10 +142,6 @@ authoritative database and schema on `[targets.]`. SQLBuild validates conn references while loading configuration, without opening a warehouse connection, and reports an unknown `targets..connection` name as an offline configuration error. -For migration only, SQLBuild still maps legacy `[connection]` to an implicit connection and -legacy `[targets..connection]` blocks to target-specific implicit connections. These -forms are compatibility syntax, not the canonical format for new or updated projects. - ## Targets A target is a named build context - the database and schema you build into, plus execution policy (for example `dev` and `prod`). Each target references a named connection (or uses the only one defined; see [Named connections](#named-connections)) and can configure: @@ -207,9 +201,8 @@ The active target is determined by (in order of precedence): 1. `--target` on the command line (highest priority) 2. `sqlbuild_local.toml` `target` field 3. `default_target` in `sqlbuild_project.toml` -4. No target (models build to the default schema). Named connections are chosen through - targets, so warehouse commands fail with `D001` unless a target is selected or a legacy - top-level `[connection]` block exists. +4. No target. Named connections are chosen through targets, so warehouse commands fail with + `D001` unless a target is selected. A typical developer keeps shared target definitions in `sqlbuild_project.toml` and selects their normal target once in the optional local file: @@ -267,7 +260,7 @@ default warehouse for each command group, so `sqb build` runs on a large warehou ```toml [connections.warehouse] account = "my_org-my_account" -warehouse = "ANALYTICS_WH" # fallback for every command, as before +warehouse = "ANALYTICS_WH" # fallback for every command [targets.dev] connection = "warehouse" @@ -286,7 +279,7 @@ Each command connects with the first warehouse set in this order: 3. The connection's `warehouse` 4. The Snowflake user's default warehouse -Both groups are optional. A target without `warehouses` behaves exactly as before. A group +Both groups are optional; a target without `warehouses` uses the connection's `warehouse`. A group warehouse takes precedence over the connection `warehouse` in every file. In particular, a shared group overrides a connection `warehouse` you set in `sqlbuild_local.toml`; to change a group locally, set the group itself, or pass `--warehouse` for one invocation. Group values @@ -394,8 +387,8 @@ Automated tools must request approval before changing this policy. """ ``` -All fields are optional. A target without an `execution_limits` section retains unlimited existing -behavior, so production or orchestrated targets do not need placeholder values. +All fields are optional. A target without an `execution_limits` section has no limits, so +production or orchestrated targets need no placeholder values. #### Model limit @@ -418,31 +411,20 @@ Every built-in adapter enforces the limit: | Adapter | Warehouse statement limit | Cancellation at the deadline | Longest accepted duration | |---------|---------------------------|------------------------------|---------------------------| | Snowflake | `STATEMENT_TIMEOUT_IN_SECONDS` per statement | Driver cancel request for the running query | 7 days | -| DuckDB, MotherDuck | none | Connection interrupt | about 24 days | -| PostgreSQL | `SET LOCAL statement_timeout` inside transactions | Driver cancel (`cancel_safe`) | about 24 days | -| BigQuery | Job timeout (`job_timeout_ms`) on each query job | `jobs.cancel` (asynchronous) | about 24 days | -| Databricks | `STATEMENT_TIMEOUT` session setting, capped at 48 hours | Cursor cancel (asynchronous) | about 24 days | -| SQL Server | none | Driver cancel | about 24 days | -| SQLite | none | Connection interrupt | about 24 days | - -BigQuery and Databricks stop cancelled work asynchronously, so SQLBuild reports their cancellation -as requested but unconfirmed. SQL Server has no per-statement limit; the driver's query timeout is -process-wide, so SQLBuild relies on cancelling the running statement instead. A project adapter that -cannot cancel a running statement is rejected for `max_duration` rather than presenting a limit that -may leave warehouse work running. Under a per-item timeout, such an adapter's items stop before their -next statement and report the cancellation as not supported. - -PostgreSQL never sets `statement_timeout` for the whole session, so a limit cannot leak to another -client through a transaction-mode connection pooler; statements outside a transaction are bounded by -cancellation alone. - -A duration failure may occur after earlier models have completed. Its error therefore distinguishes -the timeout from a pre-execution model-limit failure and states that warehouse changes may already -have occurred. - -Upgrade note: earlier releases accepted `max_duration` only on Snowflake and rejected it on other -adapters. It is now accepted on every built-in adapter, so a target that sets it for a non-Snowflake -warehouse is enforced instead of failing at startup. +| DuckDB, MotherDuck | None | Connection interrupt | About 24 days | +| PostgreSQL | `SET LOCAL statement_timeout` inside transactions; statements outside a transaction rely on cancellation | Driver cancel (`cancel_safe`) | About 24 days | +| BigQuery | Job timeout (`job_timeout_ms`) on each query job | `jobs.cancel`; reported as requested, not confirmed | About 24 days | +| Databricks | `STATEMENT_TIMEOUT` session setting, capped at 48 hours | Cursor cancel; reported as requested, not confirmed | About 24 days | +| SQL Server | None; the driver's query timeout is process-wide | Driver cancel | About 24 days | +| SQLite | None | Connection interrupt | About 24 days | + +A project adapter that can't cancel a running statement is rejected for `max_duration`, rather than +showing a limit that may leave warehouse work running. Under a per-item timeout, its items stop +before their next statement and report the cancellation as not supported. + +A duration failure may come after earlier models completed. Its error therefore says it is a +timeout, unlike a model-limit failure before execution, and that warehouse changes may already +have happened. #### Per-item time limits @@ -469,9 +451,8 @@ connection before giving it to the next item and replaces it if it is no longer #### Unit test and scenario timeouts `test_timeout` and `scenario_timeout` set the per-test and per-scenario time limit on this target, -replacing `[settings] test_timeout` and `[settings] scenario_timeout`. They follow the -[timeout precedence](#unit-test-and-scenario-limits) described under Settings and accept a positive -fixed duration or `"disabled"`. +replacing `[settings] test_timeout` and `[settings] scenario_timeout`. Each takes a positive fixed +duration or `"disabled"`. See [Unit test and scenario limits](#unit-test-and-scenario-limits). #### Custom remediation @@ -679,27 +660,18 @@ delta_compile = true ### Unit test and scenario limits -A SQL unit test runs the real SQL of every unmocked model between its mocks and the model under -test as one warehouse statement. `unit_test_max_real_models` stops that statement from silently -growing into a whole-chain integration query: a test that executes more real models fails -compilation and names every real model, the dependency path that pulls it in, and the inputs to -mock. See [Testing](/docs/concepts/testing#cost-guards) for an example. - -`test_timeout` bounds each test's warehouse work, measured from the test's first statement. The -limit that applies to a test is, from highest precedence: - -1. `--test-timeout` on `sqb test` or `sqb build` -2. `TEST(timeout "...")` in the test file -3. `[targets..execution_limits] test_timeout` -4. `[settings] test_timeout`, then the 30-second default +| Setting | Limits | Details | +|---------|--------|---------| +| `unit_test_max_real_models` | Real (unmocked) models one SQL unit test may run; a larger test fails compilation and names each model, the path that pulls it in, and the inputs to mock | [Real-model limit](/docs/concepts/testing#real-model-limit) | +| `test_timeout` | Each unit test's warehouse time, from its first statement | [Time limits](/docs/concepts/testing#time-limits), including precedence | +| `scenario_timeout` | Each scenario's warehouse time; no default | Below | -`scenario_timeout` works the same way with `--scenario-timeout`, `SCENARIO(timeout "...")`, -`[targets..execution_limits] scenario_timeout` and `[settings] scenario_timeout`; it has -no default. At the limit, SQLBuild cancels the item's running statement, reports the item as -`TIMEOUT` with the elapsed time, the limit, where the limit was set, the cancellation outcome and -the query IDs it saw, and continues with independent items. Every value accepts a positive fixed -duration such as `"45s"` or `"10m"`, or `"disabled"`. `sqlbuild_local.toml` accepts all three -settings. +The scenario limit that applies is the first one set of `--scenario-timeout`, +`SCENARIO(timeout "...")`, `[targets..execution_limits] scenario_timeout`, and +`[settings] scenario_timeout`. At the limit, SQLBuild cancels the scenario's running statement, +reports it as `TIMEOUT`, and continues with independent items, as for +[unit tests](/docs/concepts/testing#time-limits). Every value is a positive fixed duration such as +`"45s"` or `"10m"`, or `"disabled"`. `sqlbuild_local.toml` accepts all three settings. ### Table promotion mode @@ -727,7 +699,7 @@ codes begin with `SQBR`; repository-defined codes begin with `XSQBR`. See | `[[rules.rule_ignores]]` | Rules turned off for `paths` or `selectors`, with a `reason`. | Rule exceptions live only in this file. Inline `-- sqb: ignore` comments are `SQBRSQL000` errors -that print the entry to add, and the former `allow_exceptions` setting is a configuration error. +that print the entry to add. ## Scopes @@ -828,7 +800,7 @@ old_name_views = "30d" | Field | Default | Description | |-------|---------|-------------| -| `old_name_views` | `"30d"` | How long a renamed model's old name keeps working through a [compatibility view](/docs/concepts/models/migrations#old-names) before janitor drops it. `false` keeps the old relation in place instead, as before. A model's `old_name_view` header overrides it. | +| `old_name_views` | `"30d"` | How long a renamed model's old name keeps working through a [compatibility view](/docs/concepts/models/migrations#old-names) before janitor drops it. `false` keeps the old relation in place instead. A model's `old_name_view` header overrides it. | ## Scenario @@ -836,7 +808,7 @@ Configuration for scenario run isolation, snapshot capture safety limits, and lo ```toml [scenario] -run_namespace = "orders-job-123" # Optional; omit to preserve existing scenario relation names +run_namespace = "orders-job-123" # Optional [scenario.snapshot_limits] max_rows_per_relation = 10000 From 292dd756823e66434590cae5b92276916cffc1ad Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Tue, 27 Oct 2026 09:10:19 +0000 Subject: [PATCH 07/10] docs: add a style guide for the documentation site --- website/STYLE.md | 56 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 website/STYLE.md diff --git a/website/STYLE.md b/website/STYLE.md new file mode 100644 index 0000000000..105b407dc8 --- /dev/null +++ b/website/STYLE.md @@ -0,0 +1,56 @@ +# Docs style guide + +These rules apply to every page under `website/src/content/docs/`. Pages that follow them well: +`concepts/semantic-compilation.mdx`, `cli/review.mdx`, `concepts/audits.mdx`, +`concepts/models/column-migrations.mdx` and `benchmarks.mdx`. + +## Write for users + +- Document what a user does, sees and must decide: commands, flags, config keys, defaults, limits, + error codes and messages, output, and the fix for each problem. +- Leave out how SQLBuild works inside: cache files and their formats, batching, hashing, internal + tables or structures, and timing internals. Include a mechanism only when the user must act on + it, and then say what to do (for example, which grant to give or which flag forces a fresh run). +- Never document behaviour you haven't checked on the existing page or in the source. + +## Structure + +- Lead with what the feature does and a short example: a command, a config block or the output a + user sees. +- Organise sections around the user's tasks ("Declaring a rename", "Forcing a fresh compile"), + not around the implementation. +- One topic per section. Put flags, config keys and codes in reference tables near the top or + bottom, and details that rarely matter in a short "Details" or "Limitations" list at the end. +- Keep headings that other pages link to. Before renaming or removing one, run + `grep -rn "#" website/src` and update the links. + +## Tables + +- Use a table for anything that varies by adapter or dialect: support, grants, sources, limits and + caveats. Don't write a paragraph per adapter. +- Use tables for flags, config keys with defaults, exit codes, diagnostic codes and checks. +- Keep cells short. Put a long explanation in the text below the table. + +## Sentences + +- Write short, plain sentences in the present tense. One idea per sentence. +- Give every rule an example where it matters, such as the SQL that triggers an error and the SQL + that fixes it. +- Use neutral orders-style examples (orders, customers, products, payments). See `AGENTS.md`. + +## No history + +- Describe the product as it is. Don't write "now", "no longer", "used to", "earlier releases", + "as before", "upgrade note" or "previously". History belongs in `CHANGELOG.md`. +- Drop notes about legacy syntax and migrations from old versions once the error message itself + tells the user what to do. + +## Changing a page + +- When a feature changes a page, edit the page into shape: put the new fact in the section it + belongs to, update tables and examples, and remove what is no longer true. Don't append a + paragraph to the end of a section or page. +- Don't repeat a rule that another page owns. Link to it instead, for example to + `concepts/testing#time-limits` for unit test timeout precedence. +- After editing a page, regenerate the skill reference copies with `scripts.generate_docs_skill` and + commit them with the page. From 8421479443ad83843aef878e4314ab596c7ce76b Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Tue, 27 Oct 2026 09:10:28 +0000 Subject: [PATCH 08/10] docs: point agents at the docs style guide --- AGENTS.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 30e75b1806..7edbc2d8fb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -141,6 +141,14 @@ - Use `sqb debug` as the first diagnostic when distinguishing authentication, connection, configuration, and command-specific failures. +## User Documentation + +- Every change to the docs site (`website/src/content/docs/`) must follow `website/STYLE.md`: + user-facing content only, task-first structure, tables for per-adapter facts, and no history + wording or internal mechanisms. +- When a change affects a docs page, edit the page into shape: update the section the fact belongs + to and remove what is no longer true. Never append a paragraph to a page or section. + ## Subagent Verification - Read-only review subagents are standing-authorized at meaningful review boundaries and do not From fff4e129f5e5b51fbc716b1e788d7b098232f6af Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Tue, 27 Oct 2026 09:12:43 +0000 Subject: [PATCH 09/10] docs: say that a janitor block on source schemas applies in direct mode --- .../.agents/skills/sqlbuild/references/docs/cli/janitor.md | 5 +++-- website/src/content/docs/docs/cli/janitor.mdx | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md index 0c4c715458..0f416c4451 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md @@ -180,8 +180,9 @@ relation retention. project builds into; a qualified name only at that location. Once the model has built, the old name holds a compatibility view, which expires as above. - **Schemas with sources are protected.** A schema that holds the project's configured sources is - skipped and listed under `Skipped schemas`, so what the project reads is never archived. When such a schema is one SQLBuild manages, the - preview shows `Janitor blocked`, lists the actions it suppressed, and the run exits `1`. + skipped and listed under `Skipped schemas`, so what the project reads is never archived. In + direct mode, when such a schema is also one SQLBuild builds into, the preview shows + `Janitor blocked`, lists the actions it suppressed, and the run exits `1`. - **Run the same SQLBuild version as your builds.** An older janitor doesn't know compatibility views and archives them as stale relations. diff --git a/website/src/content/docs/docs/cli/janitor.mdx b/website/src/content/docs/docs/cli/janitor.mdx index 3c9ba2279e..9314df64f4 100644 --- a/website/src/content/docs/docs/cli/janitor.mdx +++ b/website/src/content/docs/docs/cli/janitor.mdx @@ -167,8 +167,9 @@ relation retention. project builds into; a qualified name only at that location. Once the model has built, the old name holds a compatibility view, which expires as above. - **Schemas with sources are protected.** A schema that holds the project's configured sources is - skipped and listed under `Skipped schemas`, so what the project reads is never archived. When such a schema is one SQLBuild manages, the - preview shows `Janitor blocked`, lists the actions it suppressed, and the run exits `1`. + skipped and listed under `Skipped schemas`, so what the project reads is never archived. In + direct mode, when such a schema is also one SQLBuild builds into, the preview shows + `Janitor blocked`, lists the actions it suppressed, and the run exits `1`. - **Run the same SQLBuild version as your builds.** An older janitor doesn't know compatibility views and archives them as stale relations. From a114f367bbcb5fb349294f83affd40ff3cf066da Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Tue, 27 Oct 2026 09:18:52 +0000 Subject: [PATCH 10/10] docs: keep the compile timings table to keys users read --- .../.agents/skills/sqlbuild/references/docs/cli/compile.md | 7 +++---- website/src/content/docs/docs/cli/compile.mdx | 7 +++---- 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md index 80441d5df0..2a7c01186c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md @@ -191,10 +191,8 @@ where a slow compile spends its time: | `built_in_rules_ms` | Wall time of the built-in Rules | | `custom_rules_ms`, `custom_rules_cpu_ms` | Wall and CPU time of the custom Rules, including their host processes | | `custom_rules_invocations` | Custom Rule invocations, one per Rule and subject | -| `custom_rules_monitor_cpu_ms` | Host CPU spent checking Rule modules for state one subject could leave for another | -| `custom_rules_guard_cpu_ms` | Host CPU spent deciding whether an audited operation, such as opening a file, came from Rule code | | `custom_rules..invocations` | Invocations of one custom Rule | -| `custom_rules..cpu_ms` | Host CPU of those invocations, without the guard's checks | +| `custom_rules..cpu_ms` | CPU time of those invocations | | `custom_rules..cache_hits` | Subjects whose stored result was reused | | `custom_rules..cache_misses` | Subjects evaluated again; every subject when the Rules cache is off | | `rule_cache_hits`, `rule_cache_misses` | Rule results reused or evaluated again | @@ -202,7 +200,8 @@ where a slow compile spends its time: | `project_reuse_check_ms` | Time spent checking whether the previous compile can be reused | | `total_ms` | Wall time of the whole compile | -Other keys are finer-grained phase timings and cache counters. Each selected custom Rule has all +Other keys are finer-grained phase timings, cache counters and the custom Rules host's own +overhead (`custom_rules_monitor_cpu_ms`, `custom_rules_guard_cpu_ms`). Each selected custom Rule has all four `custom_rules..*` keys; read one with `jq '.compile_timings["custom_rules.XORDERS001.cpu_ms"]'`. A Rule answered fully from the cache reports `0` invocations. Invocations and CPU can exceed a Rule's subject count, because a Rule diff --git a/website/src/content/docs/docs/cli/compile.mdx b/website/src/content/docs/docs/cli/compile.mdx index ed640b0ebb..0bb775514f 100644 --- a/website/src/content/docs/docs/cli/compile.mdx +++ b/website/src/content/docs/docs/cli/compile.mdx @@ -175,10 +175,8 @@ where a slow compile spends its time: | `built_in_rules_ms` | Wall time of the built-in Rules | | `custom_rules_ms`, `custom_rules_cpu_ms` | Wall and CPU time of the custom Rules, including their host processes | | `custom_rules_invocations` | Custom Rule invocations, one per Rule and subject | -| `custom_rules_monitor_cpu_ms` | Host CPU spent checking Rule modules for state one subject could leave for another | -| `custom_rules_guard_cpu_ms` | Host CPU spent deciding whether an audited operation, such as opening a file, came from Rule code | | `custom_rules..invocations` | Invocations of one custom Rule | -| `custom_rules..cpu_ms` | Host CPU of those invocations, without the guard's checks | +| `custom_rules..cpu_ms` | CPU time of those invocations | | `custom_rules..cache_hits` | Subjects whose stored result was reused | | `custom_rules..cache_misses` | Subjects evaluated again; every subject when the Rules cache is off | | `rule_cache_hits`, `rule_cache_misses` | Rule results reused or evaluated again | @@ -186,7 +184,8 @@ where a slow compile spends its time: | `project_reuse_check_ms` | Time spent checking whether the previous compile can be reused | | `total_ms` | Wall time of the whole compile | -Other keys are finer-grained phase timings and cache counters. Each selected custom Rule has all +Other keys are finer-grained phase timings, cache counters and the custom Rules host's own +overhead (`custom_rules_monitor_cpu_ms`, `custom_rules_guard_cpu_ms`). Each selected custom Rule has all four `custom_rules..*` keys; read one with `jq '.compile_timings["custom_rules.XORDERS001.cpu_ms"]'`. A Rule answered fully from the cache reports `0` invocations. Invocations and CPU can exceed a Rule's subject count, because a Rule