Skip to content

Commit cbaa787

Browse files
authored
feat!: merge Python node folders into python/ (#850)
* feat!: merge Python node folders into python/ Discover loaders, tasks, assets, checks, factories, and audit factories anywhere under a single python/ root. The decorator defines each node kind, so folder-based kind validation is gone and undecorated helper modules under python/ are importable without becoming nodes. Remove the tasks/, assets/, checks/, loaders/, factories/, libs/, dagster/, and rivers_pipeline/ Python roots and the root definitions.py allowance. The Dagster and Rivers playgrounds now generate an orchestrator directory with definitions.py next to a nested waffle_shop/ SQLBuild project. Path selectors for Python nodes use the python/ root. BREAKING CHANGE: Python nodes and their helper modules must live under python/ (for example python/tasks/orders.py, imported as python.tasks.orders). Python under tasks/, assets/, checks/, loaders/, factories/, libs/, dagster/, rivers_pipeline/, or a root definitions.py now fails compilation with D016. Python path selectors use python/... instead of tasks/, assets/, checks/, or loaders/. * docs: document the python/ node root and external orchestrator layout Update the Python nodes, project Python, selector, init, playground, quickstart, dlt, Dagster, and Rivers pages for the single python/ root and for orchestrator definitions that live outside the SQLBuild project, then regenerate the bundled skill references. * docs: regenerate bundled skill references * fix: import python/ modules under their package names Discovery imports every python/ module as python.<dotted.path> through the normal import system, so helper packages can use relative imports and each module initializes once per discovery pass. python.* entries are evicted at the start of each pass so sequential projects and edited files are imported afresh. * fix: register factory-returned module nodes once A factory may return a decorated node that direct discovery already registers from its own python/ module. Skip that identical callable in factory output while keeping the duplicate-name error for distinct nodes. * fix: install wrapper playground skills in the SQLBuild project Dagster and Rivers playgrounds nest the SQLBuild project under waffle_shop/. Install the scaffolded agent skills there, where sqb skills and stale-skill maintenance resolve them, and document the location. * test: align python/ discovery regressions with fensu rules Check node decorators directly when de-duplicating factory output, split the factory E2E into success and conflict tests, and use expected_ fields and the shared repo-file fixture in the new regressions. * fix: bind python/ to the project and de-duplicate factory nodes by identity Discovery installs an explicit python package whose __path__ is the project's python/ folder, so project imports never resolve to an unrelated python package elsewhere on sys.path. Unrelated python.* modules displaced during discovery are restored afterwards. Discovery now imports every python/ module and registers direct nodes before running factories, then skips factory results that are the same callable direct discovery registered. A factory that binds a generated node to a module global registers it once instead of dropping it. * test: resolve the project root from python/ node fixtures
1 parent ffe0dcf commit cbaa787

123 files changed

Lines changed: 2634 additions & 1187 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,9 @@ sqb --project-dir <path> compile [flags]
4646
3. **SQL validation** - validates SQL syntax (when SQL analysis is enabled)
4747
4. **Column lineage** - analyzes column-level dependencies across models (fast mode by default)
4848
5. **Contract validation** - checks declared column contracts against inferred query output
49-
6. **Rules** - evaluates selected native built-ins, then selected custom Python Rules
49+
6. **Rules** - evaluates selected native built-ins, then selected custom Python Rules. Rules that
50+
need inferred type proofs, such as `SQBRCONTRACT105`, are skipped when SQL analysis is disabled,
51+
and compile prints one note naming them
5052
7. **Artifact write** - writes compiled SQL to `target/compiled/` when Rules pass
5153

5254
## Focused compilation

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/init.md‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -30,10 +30,7 @@ my-project/
3030
schemas/
3131
sources/
3232
seeds/
33-
loaders/
34-
tasks/
35-
assets/
36-
checks/
33+
python/
3734
hooks/
3835
sql/
3936
python/

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/playground.md‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,8 @@ The positional argument is the directory to create (default `sqlbuild-playground
2121
| Template | Description |
2222
|----------|-------------|
2323
| `waffle_shop` | Default. DuckDB-backed project with models, tests, scenarios, and macros. |
24-
| `dagster` | Waffle shop project plus a `dagster/` directory with a ready-to-run `definitions.py`. |
25-
| `rivers` | Waffle shop project plus a `rivers_pipeline/` directory with a Rivers repository definition. |
24+
| `dagster` | A Dagster code location: a ready-to-run `definitions.py` next to the waffle shop SQLBuild project in `waffle_shop/`. |
25+
| `rivers` | A Rivers code repository: a `definitions.py` repository definition next to the waffle shop SQLBuild project in `waffle_shop/`. |
2626
| `python_nodes` | A small DuckDB project demonstrating [Python nodes](../concepts/python-nodes/overview.md): a task feeding a loader, a model read by a Python asset through `ctx.relation(model(...))`, a soft-skip fan-in, `materialized=False`, and a Python check. |
2727

2828
## What it creates
@@ -31,21 +31,28 @@ A complete DuckDB-backed project with:
3131

3232
- Staging views, fact/dimension tables, and incremental models
3333
- Sources with inline expression data (no external setup)
34+
- Python source loaders in `python/loaders/`
3435
- Seeds, SQL functions, and a custom materialization
3536
- Built-in and custom audits
3637
- SQL unit tests and multi-model tests
3738
- E2E scenario tests
3839
- Python macros
3940
- AI agent skill files (auto-installed for OpenCode, Claude Code, and other agents)
4041

41-
The `dagster` template adds:
42+
The `dagster` and `rivers` templates keep orchestrator code outside the SQLBuild project:
4243

43-
- `dagster/definitions.py` - Dagster definitions with `sqlbuild_assets`, `sqlbuild_scenario_checks`, and `SqlBuildCliResource`
44-
- `dagster/README.md` - Setup instructions
44+
```text
45+
sqlbuild-playground/
46+
definitions.py # Dagster or Rivers definitions pointing at waffle_shop/
47+
README.md # setup instructions
48+
waffle_shop/ # the waffle shop SQLBuild project, including agent skill files
49+
```
50+
51+
The Dagster `definitions.py` uses `sqlbuild_assets`, `sqlbuild_scenario_checks`, and `SqlBuildCliResource`; the Rivers `definitions.py` defines a `waffle_shop` job. Run `sqb` commands from `waffle_shop/` and the orchestrator from the top-level directory. Agent skill files are installed in `waffle_shop/`, where `sqb skills` and stale-skill checks maintain them.
4552

4653
The `python_nodes` template instead creates a focused Python-nodes project:
4754

48-
- `tasks/orders.py`, `loaders/orders.py`, `assets/orders_export.py`, `checks/orders_export.py`
55+
- `python/tasks/orders.py`, `python/loaders/orders.py`, `python/assets/orders_export.py`, `python/checks/orders_export.py`
4956
- A `fact_orders` SQL model over a managed `raw_orders` source
5057
- Examples of the SQL boundary, `ctx.relation(model(...))`, soft-skip fan-in, and a Python check
5158

@@ -61,13 +68,13 @@ sqb build
6168
sqb playground --template dagster
6269
cd sqlbuild-playground
6370
uv pip install 'sqlbuild[dagster]'
64-
dagster dev -f dagster/definitions.py
71+
dagster dev -f definitions.py
6572

6673
# With Rivers integration
6774
sqb playground --template rivers
6875
cd sqlbuild-playground
6976
uv pip install 'sqlbuild[rivers]'
70-
uv run rivers dev rivers_pipeline.definitions
77+
uv run rivers dev definitions
7178

7279
# With Python nodes
7380
sqb playground --template python_nodes

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,13 @@ order_id,customer_id,status
8383
2,200,completed
8484
```
8585

86+
## Errors
87+
88+
When the warehouse rejects the SQL, `sqb query` prints the warehouse message as
89+
`error[C108]: ...` on stderr and exits with code `2`. Nothing is written to stdout, so
90+
redirected `json` or `csv` output is never partial. Invalid command input, such as a missing
91+
query file, exits with code `1`.
92+
8693
## Examples
8794

8895
```bash

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md‎

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -531,16 +531,32 @@ See [skills CLI reference](../cli/skills.md) for usage details.
531531

532532
## Project Python
533533

534-
Project-owned Python must live in a supported extension location such as `factories/`, `libs/`,
535-
`macros/`, `providers/`, or another documented Python resource root. Factory locations contain
536-
normal Python: constants, classes, undecorated helper functions, and modules such as `_helpers.py`
537-
are allowed, while decorators determine which functions become SQLBuild resources. Compilation
538-
rejects Python under unsupported project roots, so indirectly importable modules cannot create an
539-
unofficial project structure.
534+
Project-owned Python lives in a small set of roots. Anything that runs as part of the pipeline graph
535+
goes in `python/`: loaders, tasks, assets, checks, factories, and the helper modules they import.
536+
Modules under `python/` are normal Python: constants, classes, undecorated helper functions, and
537+
helper-only modules such as `python/_helpers.py` are allowed, while decorators determine which
538+
functions become graph nodes. See [Python nodes](python-nodes/overview.md#the-python-folder).
539+
540+
Code that extends SQLBuild itself keeps its own root:
541+
542+
| Location | Holds |
543+
|----------|-------|
544+
| `rules/` | Custom project rules |
545+
| `materializations/` | Custom materializations |
546+
| `adapters/`, `adapter.py` | Custom adapters |
547+
| `providers/` | Runtime providers |
548+
| `sinks/` | Lifecycle event and command output sinks |
549+
| `hooks/python/` | Python model lifecycle hooks |
550+
| `functions/python/` | Python UDFs |
551+
| `macros/`, `enums/`, `constants/` and scoped `_macros/`, `_enums/`, `_constants/` | Python macros and declarations |
552+
553+
Compilation rejects Python anywhere else with `D016`, so indirectly importable modules cannot create
554+
an unofficial project structure.
540555

541556
Keep repository pytest tests outside the SQLBuild project's `tests/` directory, which is reserved for
542-
SQLBuild SQL tests and scenarios. Documented integration paths such as `dagster/`,
543-
`rivers_pipeline/`, and their `definitions.py` modules are also supported.
557+
SQLBuild SQL tests and scenarios. Orchestrator code such as Dagster or Rivers definitions lives outside
558+
the SQLBuild project directory and points at it; see [Dagster](../integrations/dagster.md) and
559+
[Rivers](../integrations/rivers.md).
544560

545561
## sqlbuild_local.toml
546562

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/assets.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,10 +20,10 @@ Assets are Python nodes that produce or observe an external artifact - an export
2020

2121
## Defining an asset
2222

23-
Place Python files under `assets/` and decorate functions with `@asset`:
23+
Place Python files under `python/` (the playgrounds use `python/assets/`) and decorate functions with `@asset`:
2424

2525
```python
26-
# assets/exports.py
26+
# python/assets/exports.py
2727
from sqlbuild.assets import asset, AssetContext
2828

2929
@asset
@@ -57,7 +57,7 @@ Assets declare dependencies with `depends_on` (a single function, tuple, or list
5757

5858
```python
5959
from sqlbuild.assets import asset
60-
from tasks.orders import export_orders
60+
from python.tasks.orders import export_orders
6161

6262
@asset(depends_on=export_orders)
6363
def orders_dashboard(ctx):

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/checks.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,12 @@ Checks are separate graph nodes, not callbacks embedded in a task or asset. A ch
2222

2323
## Defining a check
2424

25-
Place Python files under `checks/` and decorate functions with `@check`. `depends_on` is required:
25+
Place Python files under `python/` (the playgrounds use `python/checks/`) and decorate functions with `@check`. `depends_on` is required:
2626

2727
```python
28-
# checks/orders.py
28+
# python/checks/orders.py
2929
from sqlbuild.checks import check, CheckContext
30-
from tasks.orders import export_orders
30+
from python.tasks.orders import export_orders
3131

3232
@check(depends_on=export_orders)
3333
def check_orders_exported(ctx: CheckContext):
@@ -121,7 +121,7 @@ sqb check --select tag:exports
121121
| | Checks | Audits |
122122
|---|--------|--------|
123123
| Validates | Python tasks, assets, loaders | SQL relations |
124-
| Authored in | `checks/` (Python) | `MODEL()` headers / `audits/` (SQL) |
124+
| Authored in | `python/` (Python) | `MODEL()` headers / `audits/` (SQL) |
125125
| Run by | `sqb build`, `sqb check` | `sqb build`, `sqb audit` |
126126
| Severity | `error`, `warn` | `error`, `warn` |
127127

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/factories.md‎

Lines changed: 6 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Factories are an advanced feature. Most projects author nodes directly; reach fo
1515
A `@factory` function takes **no arguments** and returns one or more decorated node functions:
1616

1717
```python
18-
# factories/exports.py
18+
# python/factories/exports.py
1919
from sqlbuild.factories import factory
2020
from sqlbuild.assets import asset
2121

@@ -36,38 +36,18 @@ The returned nodes are discovered and added to the graph exactly as if you had w
3636

3737
A factory may return a single node or a list, tuple, or set of nodes. Every returned item must be a decorated `@task`, `@asset`, `@loader`, or `@check` function.
3838

39-
## Folder rules
39+
## Where factories live
4040

41-
Where a factory lives determines what it is allowed to emit:
41+
Factories live anywhere under `python/`, like every other Python node. The decorators on the returned functions decide each node's kind, so a factory may emit a single kind or a mix, and it can sit next to hand-written nodes in the same file. The playgrounds keep factories in `python/factories/`.
4242

43-
| Location | May emit |
44-
|----------|----------|
45-
| `loaders/` | loaders only |
46-
| `tasks/` | tasks only |
47-
| `assets/` | assets only |
48-
| `checks/` | checks only |
49-
| `factories/` | any kind, including a mix |
50-
51-
- A **single-kind** factory can live in that kind's folder (e.g. an asset-only factory in `assets/`), keeping it next to the nodes it generates. It can also live in `factories/`.
52-
- A factory that emits **more than one kind** must live in `factories/`.
53-
54-
This keeps each kind folder honest: everything in `assets/`, whether hand-written or factory-generated, is an asset.
55-
56-
SQLBuild enforces this at discovery time. A factory in a kind folder that returns a foreign kind raises an error pointing you to `factories/`:
57-
58-
```
59-
Factory export_pipeline in assets/ returned a loader 'raw_orders';
60-
mixed-kind factories must live in factories/.
61-
```
62-
63-
The `factories/` folder is not created by `sqb init` - add it when you need it.
43+
Factory modules are normal Python: constants, classes, and undecorated helpers are allowed, and shared helpers can live in their own modules, for example `python/factories/_helpers.py` imported as `from python.factories._helpers import TABLES`.
6444

6545
## Mixed-kind factories
6646

67-
A factory in `factories/` can generate a whole related pipeline at once - a loader, the asset that reads it, and a check on the result:
47+
A single factory can generate a whole related pipeline at once - a loader, the asset that reads it, and a check on the result:
6848

6949
```python
70-
# factories/orders.py
50+
# python/factories/orders.py
7151
from sqlbuild.factories import factory
7252
from sqlbuild.loaders import loader
7353
from sqlbuild.assets import asset

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -26,18 +26,18 @@ Loaders are Python functions that load data into source tables. They replace exp
2626

2727
## How it works
2828

29-
1. Write a Python function under `loaders/` decorated with `@loader`
29+
1. Write a Python function under `python/` decorated with `@loader`
3030
2. Declare a managed source in `sources/*.yml` with `managed: true` and **the same name as the loader function**
3131
3. SQLBuild calls the function, writes returned rows to a staging table, then applies the configured write strategy to the target
3232

3333
Loaders participate in the build lifecycle. When `sqb build` runs, managed sources are loaded before any dependent model is materialized.
3434

3535
## Defining a loader
3636

37-
Place Python files under `loaders/` in your project directory. Each file can contain one or more loader functions:
37+
Place Python files under `python/` in your project directory (the playgrounds use `python/loaders/`). Each file can contain one or more loader functions:
3838

3939
```python
40-
# loaders/raw_sources.py
40+
# python/loaders/raw_sources.py
4141
from sqlbuild.loaders import loader
4242
from sqlbuild.executor.load.models import LoaderContext
4343

@@ -357,17 +357,18 @@ When a loader returns rows with columns not present in the existing target table
357357

358358
```
359359
my-project/
360-
loaders/
361-
raw_sources.py # loader functions
362-
api_sources.py # more loader functions
360+
python/
361+
loaders/
362+
raw_sources.py # loader functions
363+
api_sources.py # more loader functions
363364
sources/
364365
raw.yml # managed source declarations (managed: true)
365366
models/
366367
staging/
367368
stg_customers.sql # __source("raw_customers")
368369
```
369370
370-
SQLBuild discovers all `.py` files under `loaders/` recursively (excluding `__init__.py` and files starting with `_`). Each file is scanned for functions decorated with `@loader`.
371+
SQLBuild imports all `.py` files under `python/` recursively (excluding `__init__.py`). Functions decorated with `@loader` become loaders; undecorated helpers do not become nodes.
371372
372373
## Config reference
373374

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/overview.md‎

Lines changed: 28 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Online: https://sqlbuild.com/docs/concepts/python-nodes/overview/
88

99
## Contents
1010

11+
- The `python/` folder
1112
- The SQL boundary
1213
- Decorators
1314
- Identity tracking
@@ -20,14 +21,14 @@ Online: https://sqlbuild.com/docs/concepts/python-nodes/overview/
2021

2122
Python nodes let your project grow beyond warehouse-only SQL while keeping the SQL graph clean. They are ordinary Python functions, decorated to become nodes in the same DAG as your SQL models, and they run as part of `sqb build`.
2223

23-
There are four kinds, authored in dedicated top-level folders:
24+
There are four kinds. The decorator says what each function is, so all of them live under one `python/` folder:
2425

25-
| Kind | Decorator | Folder | Purpose |
26-
|------|-----------|--------|---------|
27-
| Loader | `@loader` | `loaders/` | Load external data into a managed source |
28-
| Task | `@task` | `tasks/` | Run Python computation or side effects |
29-
| Asset | `@asset` | `assets/` | Produce or observe an external artifact |
30-
| Check | `@check` | `checks/` | Validate tasks, assets, and loaders |
26+
| Kind | Decorator | Purpose |
27+
|------|-----------|---------|
28+
| Loader | `@loader` | Load external data into a managed source |
29+
| Task | `@task` | Run Python computation or side effects |
30+
| Asset | `@asset` | Produce or observe an external artifact |
31+
| Check | `@check` | Validate tasks, assets, and loaders |
3132

3233
All four share the same decorator conventions, dependency model, selection syntax, and runtime context helpers. The pages for each kind cover their specifics:
3334

@@ -40,6 +41,26 @@ All four share the same decorator conventions, dependency model, selection synta
4041

4142
Nodes can also be generated programmatically with [`@factory`](factories.md) instead of authored one at a time.
4243

44+
## The `python/` folder
45+
46+
Anything that runs as part of the pipeline graph goes in `python/`: loaders, tasks, assets, checks, [factories](factories.md), and the helper modules they import. SQLBuild imports every `.py` file under `python/` recursively (except `__init__.py`) and registers each decorated function by its decorator. Undecorated functions and helper-only modules are allowed and never become nodes.
47+
48+
How you organize files inside `python/` is up to you. The playgrounds group nodes by kind:
49+
50+
```
51+
python/
52+
_helpers.py # shared helpers, not nodes
53+
loaders/orders.py
54+
tasks/orders.py
55+
assets/orders_export.py
56+
checks/orders_export.py
57+
factories/regional.py
58+
```
59+
60+
Grouping by domain (`python/orders/`, `python/customers/`) works just as well, and a single file may mix kinds. Import other project modules through the `python` package, for example `from python.tasks.orders import export_orders` or `from python._helpers import normalize_status`.
61+
62+
Code that extends SQLBuild itself keeps its own folder, such as `rules/`, `materializations/`, `adapters/`, `providers/`, `sinks/`, `hooks/python/`, `functions/python/`, and `macros/`. Python outside the supported locations fails compilation with `D016`; see [Project Python](../project-configuration.md#project-python) for the full list.
63+
4364
## The SQL boundary
4465

4566
The most important rule: **SQL models never depend on Python nodes.** The dependency direction is strictly one way.

0 commit comments

Comments
 (0)