Skip to content
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,9 @@ sqb --project-dir <path> compile [flags]
3. **SQL validation** - validates SQL syntax (when SQL analysis is enabled)
4. **Column lineage** - analyzes column-level dependencies across models (fast mode by default)
5. **Contract validation** - checks declared column contracts against inferred query output
6. **Rules** - evaluates selected native built-ins, then selected custom Python Rules
6. **Rules** - evaluates selected native built-ins, then selected custom Python Rules. Rules that
need inferred type proofs, such as `SQBRCONTRACT105`, are skipped when SQL analysis is disabled,
and compile prints one note naming them
7. **Artifact write** - writes compiled SQL to `target/compiled/` when Rules pass

## Focused compilation
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,7 @@ my-project/
schemas/
sources/
seeds/
loaders/
tasks/
assets/
checks/
python/
hooks/
sql/
python/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ The positional argument is the directory to create (default `sqlbuild-playground
| Template | Description |
|----------|-------------|
| `waffle_shop` | Default. DuckDB-backed project with models, tests, scenarios, and macros. |
| `dagster` | Waffle shop project plus a `dagster/` directory with a ready-to-run `definitions.py`. |
| `rivers` | Waffle shop project plus a `rivers_pipeline/` directory with a Rivers repository definition. |
| `dagster` | A Dagster code location: a ready-to-run `definitions.py` next to the waffle shop SQLBuild project in `waffle_shop/`. |
| `rivers` | A Rivers code repository: a `definitions.py` repository definition next to the waffle shop SQLBuild project in `waffle_shop/`. |
| `python_nodes` | A small DuckDB project demonstrating [Python nodes](../concepts/python-nodes/overview.md): a task feeding a loader, a model read by a Python asset through `ctx.relation(model(...))`, a soft-skip fan-in, `materialized=False`, and a Python check. |

## What it creates
Expand All @@ -31,21 +31,28 @@ A complete DuckDB-backed project with:

- Staging views, fact/dimension tables, and incremental models
- Sources with inline expression data (no external setup)
- Python source loaders in `python/loaders/`
- Seeds, SQL functions, and a custom materialization
- Built-in and custom audits
- SQL unit tests and multi-model tests
- E2E scenario tests
- Python macros
- AI agent skill files (auto-installed for OpenCode, Claude Code, and other agents)

The `dagster` template adds:
The `dagster` and `rivers` templates keep orchestrator code outside the SQLBuild project:

- `dagster/definitions.py` - Dagster definitions with `sqlbuild_assets`, `sqlbuild_scenario_checks`, and `SqlBuildCliResource`
- `dagster/README.md` - Setup instructions
```text
sqlbuild-playground/
definitions.py # Dagster or Rivers definitions pointing at waffle_shop/
README.md # setup instructions
waffle_shop/ # the waffle shop SQLBuild project, including agent skill files
```

The Dagster `definitions.py` uses `sqlbuild_assets`, `sqlbuild_scenario_checks`, and `SqlBuildCliResource`; the Rivers `definitions.py` defines a `waffle_shop` job. Run `sqb` commands from `waffle_shop/` and the orchestrator from the top-level directory. Agent skill files are installed in `waffle_shop/`, where `sqb skills` and stale-skill checks maintain them.

The `python_nodes` template instead creates a focused Python-nodes project:

- `tasks/orders.py`, `loaders/orders.py`, `assets/orders_export.py`, `checks/orders_export.py`
- `python/tasks/orders.py`, `python/loaders/orders.py`, `python/assets/orders_export.py`, `python/checks/orders_export.py`
- A `fact_orders` SQL model over a managed `raw_orders` source
- Examples of the SQL boundary, `ctx.relation(model(...))`, soft-skip fan-in, and a Python check

Expand All @@ -61,13 +68,13 @@ sqb build
sqb playground --template dagster
cd sqlbuild-playground
uv pip install 'sqlbuild[dagster]'
dagster dev -f dagster/definitions.py
dagster dev -f definitions.py

# With Rivers integration
sqb playground --template rivers
cd sqlbuild-playground
uv pip install 'sqlbuild[rivers]'
uv run rivers dev rivers_pipeline.definitions
uv run rivers dev definitions

# With Python nodes
sqb playground --template python_nodes
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,13 @@ order_id,customer_id,status
2,200,completed
```

## Errors

When the warehouse rejects the SQL, `sqb query` prints the warehouse message as
`error[C108]: ...` on stderr and exits with code `2`. Nothing is written to stdout, so
redirected `json` or `csv` output is never partial. Invalid command input, such as a missing
query file, exits with code `1`.

## Examples

```bash
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -531,16 +531,32 @@ See [skills CLI reference](../cli/skills.md) for usage details.

## Project Python

Project-owned Python must live in a supported extension location such as `factories/`, `libs/`,
`macros/`, `providers/`, or another documented Python resource root. Factory locations contain
normal Python: constants, classes, undecorated helper functions, and modules such as `_helpers.py`
are allowed, while decorators determine which functions become SQLBuild resources. Compilation
rejects Python under unsupported project roots, so indirectly importable modules cannot create an
unofficial project structure.
Project-owned Python lives in a small set of roots. Anything that runs as part of the pipeline graph
goes in `python/`: loaders, tasks, assets, checks, factories, and the helper modules they import.
Modules under `python/` are normal Python: constants, classes, undecorated helper functions, and
helper-only modules such as `python/_helpers.py` are allowed, while decorators determine which
functions become graph nodes. See [Python nodes](python-nodes/overview.md#the-python-folder).

Code that extends SQLBuild itself keeps its own root:

| Location | Holds |
|----------|-------|
| `rules/` | Custom project rules |
| `materializations/` | Custom materializations |
| `adapters/`, `adapter.py` | Custom adapters |
| `providers/` | Runtime providers |
| `sinks/` | Lifecycle event and command output sinks |
| `hooks/python/` | Python model lifecycle hooks |
| `functions/python/` | Python UDFs |
| `macros/`, `enums/`, `constants/` and scoped `_macros/`, `_enums/`, `_constants/` | Python macros and declarations |

Compilation rejects Python anywhere else with `D016`, so indirectly importable modules cannot create
an unofficial project structure.

Keep repository pytest tests outside the SQLBuild project's `tests/` directory, which is reserved for
SQLBuild SQL tests and scenarios. Documented integration paths such as `dagster/`,
`rivers_pipeline/`, and their `definitions.py` modules are also supported.
SQLBuild SQL tests and scenarios. Orchestrator code such as Dagster or Rivers definitions lives outside
the SQLBuild project directory and points at it; see [Dagster](../integrations/dagster.md) and
[Rivers](../integrations/rivers.md).

## sqlbuild_local.toml

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ Assets are Python nodes that produce or observe an external artifact - an export

## Defining an asset

Place Python files under `assets/` and decorate functions with `@asset`:
Place Python files under `python/` (the playgrounds use `python/assets/`) and decorate functions with `@asset`:

```python
# assets/exports.py
# python/assets/exports.py
from sqlbuild.assets import asset, AssetContext

@asset
Expand Down Expand Up @@ -57,7 +57,7 @@ Assets declare dependencies with `depends_on` (a single function, tuple, or list

```python
from sqlbuild.assets import asset
from tasks.orders import export_orders
from python.tasks.orders import export_orders

@asset(depends_on=export_orders)
def orders_dashboard(ctx):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,12 @@ Checks are separate graph nodes, not callbacks embedded in a task or asset. A ch

## Defining a check

Place Python files under `checks/` and decorate functions with `@check`. `depends_on` is required:
Place Python files under `python/` (the playgrounds use `python/checks/`) and decorate functions with `@check`. `depends_on` is required:

```python
# checks/orders.py
# python/checks/orders.py
from sqlbuild.checks import check, CheckContext
from tasks.orders import export_orders
from python.tasks.orders import export_orders

@check(depends_on=export_orders)
def check_orders_exported(ctx: CheckContext):
Expand Down Expand Up @@ -121,7 +121,7 @@ sqb check --select tag:exports
| | Checks | Audits |
|---|--------|--------|
| Validates | Python tasks, assets, loaders | SQL relations |
| Authored in | `checks/` (Python) | `MODEL()` headers / `audits/` (SQL) |
| Authored in | `python/` (Python) | `MODEL()` headers / `audits/` (SQL) |
| Run by | `sqb build`, `sqb check` | `sqb build`, `sqb audit` |
| Severity | `error`, `warn` | `error`, `warn` |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Factories are an advanced feature. Most projects author nodes directly; reach fo
A `@factory` function takes **no arguments** and returns one or more decorated node functions:

```python
# factories/exports.py
# python/factories/exports.py
from sqlbuild.factories import factory
from sqlbuild.assets import asset

Expand All @@ -36,38 +36,18 @@ The returned nodes are discovered and added to the graph exactly as if you had w

A factory may return a single node or a list, tuple, or set of nodes. Every returned item must be a decorated `@task`, `@asset`, `@loader`, or `@check` function.

## Folder rules
## Where factories live

Where a factory lives determines what it is allowed to emit:
Factories live anywhere under `python/`, like every other Python node. The decorators on the returned functions decide each node's kind, so a factory may emit a single kind or a mix, and it can sit next to hand-written nodes in the same file. The playgrounds keep factories in `python/factories/`.

| Location | May emit |
|----------|----------|
| `loaders/` | loaders only |
| `tasks/` | tasks only |
| `assets/` | assets only |
| `checks/` | checks only |
| `factories/` | any kind, including a mix |

- A **single-kind** factory can live in that kind's folder (e.g. an asset-only factory in `assets/`), keeping it next to the nodes it generates. It can also live in `factories/`.
- A factory that emits **more than one kind** must live in `factories/`.

This keeps each kind folder honest: everything in `assets/`, whether hand-written or factory-generated, is an asset.

SQLBuild enforces this at discovery time. A factory in a kind folder that returns a foreign kind raises an error pointing you to `factories/`:

```
Factory export_pipeline in assets/ returned a loader 'raw_orders';
mixed-kind factories must live in factories/.
```

The `factories/` folder is not created by `sqb init` - add it when you need it.
Factory modules are normal Python: constants, classes, and undecorated helpers are allowed, and shared helpers can live in their own modules, for example `python/factories/_helpers.py` imported as `from python.factories._helpers import TABLES`.

## Mixed-kind factories

A factory in `factories/` can generate a whole related pipeline at once - a loader, the asset that reads it, and a check on the result:
A single factory can generate a whole related pipeline at once - a loader, the asset that reads it, and a check on the result:

```python
# factories/orders.py
# python/factories/orders.py
from sqlbuild.factories import factory
from sqlbuild.loaders import loader
from sqlbuild.assets import asset
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,18 +26,18 @@ Loaders are Python functions that load data into source tables. They replace exp

## How it works

1. Write a Python function under `loaders/` decorated with `@loader`
1. Write a Python function under `python/` decorated with `@loader`
2. Declare a managed source in `sources/*.yml` with `managed: true` and **the same name as the loader function**
3. SQLBuild calls the function, writes returned rows to a staging table, then applies the configured write strategy to the target

Loaders participate in the build lifecycle. When `sqb build` runs, managed sources are loaded before any dependent model is materialized.

## Defining a loader

Place Python files under `loaders/` in your project directory. Each file can contain one or more loader functions:
Place Python files under `python/` in your project directory (the playgrounds use `python/loaders/`). Each file can contain one or more loader functions:

```python
# loaders/raw_sources.py
# python/loaders/raw_sources.py
from sqlbuild.loaders import loader
from sqlbuild.executor.load.models import LoaderContext

Expand Down Expand Up @@ -357,17 +357,18 @@ When a loader returns rows with columns not present in the existing target table

```
my-project/
loaders/
raw_sources.py # loader functions
api_sources.py # more loader functions
python/
loaders/
raw_sources.py # loader functions
api_sources.py # more loader functions
sources/
raw.yml # managed source declarations (managed: true)
models/
staging/
stg_customers.sql # __source("raw_customers")
```

SQLBuild discovers all `.py` files under `loaders/` recursively (excluding `__init__.py` and files starting with `_`). Each file is scanned for functions decorated with `@loader`.
SQLBuild imports all `.py` files under `python/` recursively (excluding `__init__.py`). Functions decorated with `@loader` become loaders; undecorated helpers do not become nodes.

## Config reference

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Online: https://sqlbuild.com/docs/concepts/python-nodes/overview/

## Contents

- The `python/` folder
- The SQL boundary
- Decorators
- Identity tracking
Expand All @@ -20,14 +21,14 @@ Online: https://sqlbuild.com/docs/concepts/python-nodes/overview/

Python nodes let your project grow beyond warehouse-only SQL while keeping the SQL graph clean. They are ordinary Python functions, decorated to become nodes in the same DAG as your SQL models, and they run as part of `sqb build`.

There are four kinds, authored in dedicated top-level folders:
There are four kinds. The decorator says what each function is, so all of them live under one `python/` folder:

| Kind | Decorator | Folder | Purpose |
|------|-----------|--------|---------|
| Loader | `@loader` | `loaders/` | Load external data into a managed source |
| Task | `@task` | `tasks/` | Run Python computation or side effects |
| Asset | `@asset` | `assets/` | Produce or observe an external artifact |
| Check | `@check` | `checks/` | Validate tasks, assets, and loaders |
| Kind | Decorator | Purpose |
|------|-----------|---------|
| Loader | `@loader` | Load external data into a managed source |
| Task | `@task` | Run Python computation or side effects |
| Asset | `@asset` | Produce or observe an external artifact |
| Check | `@check` | Validate tasks, assets, and loaders |

All four share the same decorator conventions, dependency model, selection syntax, and runtime context helpers. The pages for each kind cover their specifics:

Expand All @@ -40,6 +41,26 @@ All four share the same decorator conventions, dependency model, selection synta

Nodes can also be generated programmatically with [`@factory`](factories.md) instead of authored one at a time.

## The `python/` folder

Anything that runs as part of the pipeline graph goes in `python/`: loaders, tasks, assets, checks, [factories](factories.md), and the helper modules they import. SQLBuild imports every `.py` file under `python/` recursively (except `__init__.py`) and registers each decorated function by its decorator. Undecorated functions and helper-only modules are allowed and never become nodes.

How you organize files inside `python/` is up to you. The playgrounds group nodes by kind:

```
python/
_helpers.py # shared helpers, not nodes
loaders/orders.py
tasks/orders.py
assets/orders_export.py
checks/orders_export.py
factories/regional.py
```

Grouping by domain (`python/orders/`, `python/customers/`) works just as well, and a single file may mix kinds. Import other project modules through the `python` package, for example `from python.tasks.orders import export_orders` or `from python._helpers import normalize_status`.

Code that extends SQLBuild itself keeps its own folder, such as `rules/`, `materializations/`, `adapters/`, `providers/`, `sinks/`, `hooks/python/`, `functions/python/`, and `macros/`. Python outside the supported locations fails compilation with `D016`; see [Project Python](../project-configuration.md#project-python) for the full list.

## The SQL boundary

The most important rule: **SQL models never depend on Python nodes.** The dependency direction is strictly one way.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -202,8 +202,9 @@ my-project/
record_notification.sql
python/
notify.py
loaders/
load_orders.py
tasks/
export_orders.py
python/
loaders/
load_orders.py
tasks/
export_orders.py
```
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ Tasks are Python functions that run as part of the DAG. Use them for computation

## Defining a task

Place Python files under `tasks/` and decorate functions with `@task`:
Place Python files under `python/` (the playgrounds use `python/tasks/`) and decorate functions with `@task`:

```python
# tasks/orders.py
# python/tasks/orders.py
from sqlbuild.tasks import task, TaskContext

@task
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ sqb build --select models/marts
sqb build --select path:models/intermediate
```

Any name containing `/` is treated as a path selector, so `path:models/marts` and a bare `models/marts` work the same way. Path selectors require an explicit root directory: `models/`, `tasks/`, `assets/`, `checks/`, or `loaders/`. Nested paths work too: `models/staging/orders`.
Any name containing `/` is treated as a path selector, so `path:models/marts` and a bare `models/marts` work the same way. Path selectors require an explicit root directory: `models/` for SQL models or `python/` for Python nodes. Nested paths work too: `models/staging/orders` or `python/tasks`.

### Seed and source

Expand Down Expand Up @@ -167,5 +167,5 @@ path selector 'fact_orders~' requires names on both sides of '~'
If a path selector omits the root directory, SQLBuild asks for the explicit form:

```
path selectors require an explicit root: use 'models/', 'tasks/', 'assets/', 'checks/', or 'loaders/'
path selectors require an explicit root: use 'models/' or 'python/'
```
Loading
Loading