From d5d0044f61df3b8cefc58d0c241474cc294fa6de Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Fri, 23 Oct 2026 10:06:46 +0100 Subject: [PATCH] docs: give macros an overview of SQL and Python macros, and a Python macros page --- .../sqlbuild/references/docs/CONTENTS.md | 5 +- .../sqlbuild/references/docs/cli/compile.md | 4 +- .../docs/concepts/declaration-scopes.md | 2 +- .../references/docs/concepts/macros.md | 286 ++++------------- .../macros/composition-and-context.md | 10 +- .../docs/concepts/macros/python-macros.md | 249 +++++++++++++++ .../docs/concepts/macros/sql-macros.md | 16 +- .../docs/concepts/project-configuration.md | 2 +- website/src/content/docs/docs/cli/compile.mdx | 4 +- .../docs/docs/concepts/declaration-scopes.mdx | 2 +- .../src/content/docs/docs/concepts/macros.mdx | 294 ++++-------------- .../macros/composition-and-context.mdx | 10 +- .../docs/concepts/macros/python-macros.mdx | 246 +++++++++++++++ .../docs/docs/concepts/macros/sql-macros.mdx | 15 +- .../docs/concepts/project-configuration.mdx | 2 +- website/src/sidebar.json | 5 +- 16 files changed, 645 insertions(+), 507 deletions(-) create mode 100644 src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/python-macros.md create mode 100644 website/src/content/docs/docs/concepts/macros/python-macros.mdx diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md index 4643399fce..651fb4314a 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md @@ -52,9 +52,10 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in - [Collections and Rendering](concepts/constants/collections-and-rendering.md) (`concepts/constants/collections-and-rendering`) - Define list, set, and object constants and control their adapter-specific SQL rendering. - [Enum Model Contracts](concepts/enums/model-contracts.md) (`concepts/enums/model-contracts`) - Use an enum as a portable model-column domain with generated accepted-value validation. - [Model-Private Values](concepts/model-private-values.md) (`concepts/model-private-values`) - Keep enums and constants inside one model when no other resource should use them. -- [Writing Macros](concepts/macros.md) (`concepts/macros`) - Write Python functions that generate reusable SQL fragments at compile time. -- [Composition and Context](concepts/macros/composition-and-context.md) (`concepts/macros/composition-and-context`) - Compose macros through Python, use compile context, and understand scoped imports. +- [Macros](concepts/macros.md) (`concepts/macros`) - Reuse SQL with typed SQL macros, or generate it with Python macros when it needs logic. - [SQL Macros](concepts/macros/sql-macros.md) (`concepts/macros/sql-macros`) - Write typed, logic-free SQL fragments that the compiler checks and sees inside. +- [Python Macros](concepts/macros/python-macros.md) (`concepts/macros/python-macros`) - Write Python functions that generate SQL at compile time when the SQL needs logic. +- [Python Macro Composition and Context](concepts/macros/composition-and-context.md) (`concepts/macros/composition-and-context`) - Compose Python macros through imports, and use adapter, target, constants and enums. - [Interpolation](concepts/interpolation.md) (`concepts/interpolation`) - How SQLBuild processes variables, context, and dynamic content in SQL and config. - [Functions](concepts/functions.md) (`concepts/functions`) - User-defined functions and table functions managed as part of your project. - [Incremental Models](concepts/incremental.md) (`concepts/incremental`) - Cursor-based incremental strategies, microbatch execution, and backfill policies. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md index e567a22d60..86e3b3251d 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md @@ -129,7 +129,7 @@ files count only by presence. Reuse assumes macros, hooks, providers, adapters, and custom rules are deterministic: the same inputs must produce the same output. Macro results are also reused within a compile and may be kept between -compiles; see [Macros must be deterministic](../concepts/macros.md#macros-must-be-deterministic). A +compiles; see [Macros must be deterministic](../concepts/macros/python-macros.md#macros-must-be-deterministic). A compile whose templates read `run.id` is never reused. Compile always runs in full with `--no-cache`, `--manifest`, `--debug`, or a profiling flag, when @@ -191,7 +191,7 @@ the whole project. Every model's SQL references are scanned again too. lexical rules, and the SQLBuild build; a damaged or unreadable entry is scanned again. - Rule results are reused for unchanged inputs. - Macro call results are reused as described in - [Macros must be deterministic](../concepts/macros.md#macros-must-be-deterministic). + [Macros must be deterministic](../concepts/macros/python-macros.md#macros-must-be-deterministic). Either way, the output is byte-identical to `sqb compile --no-cache`. To turn compiling only the changed files off, set `delta_compile = false` under `[settings]` or set diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md index 05602fc504..ff91087e21 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md @@ -140,6 +140,6 @@ top-level roles as a genuine project API instead of a neutral dumping ground. Ask SQLBuild what a file can use and preview how moving it would change that answer. Learn the features themselves in [Enums](enums.md), [Constants](constants.md), -[Writing Macros](macros.md), [Audits](audits.md), +[Macros](macros.md), [Audits](audits.md), [Reusable Schemas](models/schemas.md), and [Hooks](models/hooks.md). See [Interpolation](interpolation.md) for project, environment, and runtime context values, which are separate from declarations. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros.md index 9a07c6b32b..dc52f461d9 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros.md @@ -1,266 +1,77 @@ -# Writing Macros +# Macros -> Write Python functions that generate reusable SQL fragments at compile time. +> Reuse SQL with typed SQL macros, or generate it with Python macros when it needs logic. Online: https://sqlbuild.com/docs/concepts/macros/ -## Contents +A macro is a reusable piece of SQL that a model, test, hook or audit calls by name. SQLBuild +expands each call at compile time, so the warehouse only ever sees plain SQL. -- Create a macro -- Call a macro from SQL -- Arguments -- Typed reference arguments -- Keep implementation details private -- Use macros in tests -- Use macros in hooks -- Macros must be deterministic -- Where macros work -- Next steps +There are two kinds: -Python macros are Python functions that generate SQL during compilation. They provide reusable, -parameterized SQL without introducing a separate template language. When the SQL needs no logic, -a typed [SQL macro](macros/sql-macros.md) is simpler and the compiler can see inside -it. +| | SQL macro | Python macro | +|---|---|---| +| Written as | A `.sql` file: a `MACRO` header and a SQL body | A Python function that returns SQL | +| Called as | `@sql.name(...)` | `@py.name(...)` | +| Lives in | `macros/sql/` | `macros/python/` | +| Arguments | Typed SQL expressions, checked against the caller's column types | Python values: strings, numbers, lists, dictionaries, references | +| Logic | None: the body is SQL | Any Python: loops, conditions, adapter and target context | +| Checks and tools | Each macro is checked on its own; `sqb format`, Rules and lineage see inside every call | Rules and lineage see the SQL each call returns | -## Create a macro +Prefer a SQL macro whenever no logic is needed: its calls stay readable, typed and checked, and +every tool sees the SQL it produces. -Put project-wide Python macros in Python files under `macros/python/`. SQL macros live next to -them in `macros/sql/`: + Typed, logic-free expressions, CTE lists and column lists. + Python functions that generate SQL when it needs logic. + +## Where macros live + +Each kind has its own folder: ```text my_project/ ├── macros/ │ ├── python/ -│ │ └── currency.py +│ │ └── labels.py │ └── sql/ +│ └── cents_to_dollars.sql ├── models/ └── sqlbuild_project.toml ``` A file directly under a `macros/` folder fails with `D013`, which names the subfolder to move it -to. Folder-scoped macros use the same layout, such as `models/marts/_sqlbuild/_macros/python/`. - -Every public function defined by a macro file becomes callable from SQL: - -```python -# macros/python/currency.py -def cents_to_dollars(expression: str) -> str: - """Convert a cents expression to dollars.""" - return f"ROUND(CAST({expression} AS DOUBLE) / 100, 2)" -``` +to. A macro used by only part of the project can live in that folder instead, as +`_sqlbuild/macros/sql/` or `_sqlbuild/_macros/python/`, and is then visible only there. See +[Declarations and Scopes](declaration-scopes.md). -## Call a macro from SQL +## Call a macro -Use `@py.macro_name(...)` in a model query. Every macro call says its kind: `@py.` calls a Python -macro and `@sql.` a SQL macro. A bare `@macro_name(...)` fails with `P014`, which names both -spellings, and calling a macro with the other kind's prefix fails with `P020`. +Every call names the kind of macro it calls: ```sql -MODEL ( - description "One row per order", - materialized table, - tags [marts], -); - SELECT order_id, - @py.cents_to_dollars("amount_cents") AS amount_dollars + @sql.cents_to_dollars(amount_cents) AS amount_dollars, + @py.status_label("status") AS status_label FROM __ref("stg_orders") ``` -During compilation, SQLBuild replaces the call with the function's returned SQL: - -```sql -ROUND(CAST(amount_cents AS DOUBLE) / 100, 2) -``` - -A macro used directly in SQL must return a string. - -## Arguments - -Macro calls accept Python literal values: - -- Strings: `"value"` or `'value'` -- Numbers: `42`, `3.14`, `-1` -- Booleans: `True`, `False` -- Lists: `[1, 2, 3]` -- Dictionaries: `{"key": "value"}` -- `None` -- The result of another macro call -- `__ref("model")`, `__source("source")`, and `__seed("seed")` references, alone or inside lists - and dictionaries - -These inputs are rejected with an error. In a model body the error names the file line and -column; elsewhere it names the position within the macro's arguments: - -- bytes (`b"text"`), complex numbers (`2j`), f-strings, and `...` -- a keyword argument passed twice, such as `@py.lit(a=1, a=2)`, which Python also rejects -- lone surrogate escapes such as `"\ud800"`; write a character outside the Basic Multilingual - Plane as one escape, such as `"\U0001F600"`, rather than as a surrogate pair -- `\N{...}` naming a multi-character named sequence; `\N` escapes name one character -- two values with no comma between them - -Positional and keyword arguments are supported: - -```sql -@py.mock_orders(count=5, status="completed") -``` - -Use quoted strings when passing SQL expressions such as column names. The macro decides how to -place that text into its returned SQL. - -## Typed reference arguments - -Pass the models, sources, and seeds a macro reads as reference arguments: - -```sql -SELECT * FROM @py.union_all([__ref("orders_eu"), __ref("orders_us")]) -``` - -The macro receives each reference as a typed `SqlResourceRef` from `sqlbuild.refs`, the same type -Python nodes use. It has `kind` and `name` attributes, and formatting it into SQL renders the -relation with the adapter's quoting: - -```python -# models/marts/_sqlbuild/_macros/python/unions.py -from sqlbuild.refs import SqlResourceRef - -def union_all(relations: list[SqlResourceRef]) -> str: - return " UNION ALL ".join(f"SELECT order_id FROM {relation}" for relation in relations) -``` - -The dependency belongs to the model that makes the call. The graph, column lineage, and -declaration scopes treat `@py.union_all([__ref("orders_eu"), __ref("orders_us")])` exactly like a -model that wrote `__ref("orders_eu")` and `__ref("orders_us")` itself. - -### References in macro output - -A macro must not produce references on its own. Any `__ref()`, `__source()`, or `__seed()` in a -macro's expanded output fails to compile with `P006`, whether the macro's Python code wrote it or -it arrived as a quoted string argument such as `@py.wrap('__ref("stg_orders")')`: - -``` -error[P006]: model:order_summary depends on model:stg_orders through macro orders_base() - model: order_summary - --> models/marts/_sqlbuild/_macros/python/orders.py - = help: write the reference in the model, or pass it in: @py.orders_base(__ref("stg_orders")) - = help: the current value is [references] enforce_explicit = true (from sqlbuild_project.toml or its default); while migrating a project, to allow macro-generated references, set this in sqlbuild_project.toml: - [references] - enforce_explicit = false -``` - -A dependency hidden inside a macro is invisible to reviewers of the model file and bypasses -declaration scopes. For projects migrating from other tools, -[`[references] enforce_explicit = false`](project-configuration.md#references) allows -macro-generated references project-wide. - -## Keep implementation details private - -Only public functions owned by the file are exported as macros. Prefix helpers, constants, classes, -and type aliases with `_`: - -```python -# macros/python/orders.py -_DEFAULT_STATUS = "completed" - -def _status_filter(status: str) -> str: - return f"status = '{status}'" - -def completed_orders() -> str: - return _status_filter(_DEFAULT_STATUS) -``` +A bare `@name(...)` fails with `P014`, which names both spellings. Calling a macro with the other +kind's prefix fails with `P020`, and a SQL macro and a Python macro may not share a visible name. -Here, SQL may call `@py.completed_orders()`. `_status_filter` and `_DEFAULT_STATUS` remain ordinary -Python implementation details. +## Use the two kinds together -Imported functions are not re-exported as new macros from the importing file. - -## Use macros in tests - -Tests are SQL, so they can use macros as reusable fixture generators. A macro that only tests use -follows the usual [placement](declaration-scopes/placement.md) rule: it must be private -to the narrowest folder that contains its uses, so the project-wide `macros/` directory rejects it -with `S024`. Put it in `_sqlbuild/_macros/python/` in the folder of the tests that call it, such as -`tests/unit/orders/_sqlbuild/_macros/python/`, or, when those tests define `__expected__` output -for a model, in that model's `_sqlbuild/_macros/python/`. The `S024` error names the folder to use. -`tests/unit/_sqlbuild/` itself is rejected with `D013`. - -```python -# tests/unit/orders/_sqlbuild/_macros/python/order_fixtures.py -def mock_orders(count: int = 1) -> str: - rows = [ - f"SELECT {i} AS id, {i * 100} AS customer_id, 'completed' AS status" - for i in range(1, count + 1) - ] - return " UNION ALL ".join(rows) -``` +A SQL macro calls only SQL macros, and a Python macro composes other Python macros through Python +imports. To combine the two, pass one call's result to the other in the calling SQL: ```sql -TEST(); - -WITH -__source__raw__orders AS ( - @py.mock_orders(3) -), -__expected__stg_orders AS ( - SELECT 1 AS order_id, 100 AS customer_id, 'completed' AS status - UNION ALL - SELECT 2 AS order_id, 200 AS customer_id, 'completed' AS status - UNION ALL - SELECT 3 AS order_id, 300 AS customer_id, 'completed' AS status -) +SELECT @py.bracket(@sql.cents_to_dollars(o.amount_cents)) AS amount_band +FROM __ref("stg_orders") AS o ``` -## Use macros in hooks - -Macros work inside inline and named SQL hooks: - -```python -# macros/python/permissions.py -def grant_target(target: str) -> str: - return f"GRANT SELECT ON {target} TO analyst_role" -``` - -```sql -MODEL ( - description "Example model", - post_hooks [inline_sql('@py.grant_target(@@CTX:destination.qualified)')], -); - -SELECT 1 AS id -``` - -See [SQL Hooks](models/hooks/sql.md) for hook lifecycle and context syntax. - -## Macros must be deterministic - -A macro must return the same result whenever it is called with the same arguments and sees the same -context. Avoid module-level state such as counters, random values, the current time, and reads of -files or environment variables that the call does not receive. - -SQLBuild relies on this contract to reuse macro results: - -- [Compile reuse](../cli/compile.md) replays an unchanged compile without running macros again. -- A compile may run each distinct macro call once and reuse its result for identical calls - elsewhere in the project. -- A compile that fails stops at its first error. It does not run the project's macros a second - time to report that error. -- Macro call results are kept between compiles under `target/cache/` by default, so editing a - model does not run its unchanged macro calls again. They are recomputed when - macro or other Python files, configuration, declarations, seeds, sources, variables, the target, - the adapter, the installed packages, or the SQLBuild version change. Editing only model SQL keeps - them, and a changed call is always run again. Only the first compile in a process uses and saves - them: later compiles in the same process, such as in a notebook or a long-running server, run their - macros again. A compile that has loaded Python code it cannot read from a plain file, such as a - module imported from a zip archive, neither uses nor saves them. - -Pass configuration through what SQLBuild compares: macro arguments (including `@@ENV:NAME` and -`@@var` values written in the calling SQL), `ctx.vars`, `ctx.target_name`, `ctx.adapter_name`, -and visible constants and enums. Environment variables a macro reads directly, for example with -`os.environ`, are compared only when their names start with `SQLBUILD_` or `SQB_`, the same set -compile reuse tracks. A macro whose result changes for any other reason may return a stale result. +The SQL macro expands first, and the Python macro receives its SQL as a string. ## Where macros work @@ -275,8 +86,21 @@ Macros are supported in: Macros are not accepted in ordinary `MODEL()` configuration fields. SQL hook entries are the exception because their contents are SQL. -## Next steps +### Macros used only by tests + +A macro that only tests use follows the usual [placement](declaration-scopes/placement.md) +rule: it must be private to the narrowest folder that contains its uses, so the project-wide +`macros/` directory rejects it with `S024`. Put it in `_sqlbuild/_macros/python/` or +`_sqlbuild/_macros/sql/` in the folder of the tests that call it, such as +`tests/unit/orders/_sqlbuild/_macros/python/`, or, when those tests define `__expected__` output +for a model, in that model's `_sqlbuild/_macros/`. The `S024` error names the folder to use. +`tests/unit/_sqlbuild/` itself is rejected with `D013`. + +## Choose the right tool - Write typed, logic-free SQL fragments the compiler checks and sees inside. - Compose macros through Python imports and use adapter or target context. - Limit a macro to one folder, or to that folder and its child folders. +| Use | When | +|---|---| +| [SQL macro](macros/sql-macros.md) | The fragment is plain SQL with typed inputs: a calculation, a filter, a set of CTEs or columns you repeat. The compiler sees inside it. | +| [Python macro](macros/python-macros.md) | Generating the SQL needs logic: loops, conditions on arguments, the adapter or target, or reading project context. | +| [SQL function](functions.md) | The logic belongs in the warehouse, to be called by other tools or evaluated per row there, or it is too large to inline everywhere. | +| [Generic audit](audits.md) | The SQL checks data quality on a model rather than producing part of its query. | diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/composition-and-context.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/composition-and-context.md index b4e8eb3688..b58436f520 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/composition-and-context.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/composition-and-context.md @@ -1,8 +1,8 @@ -# Composition and Context +# Python Macro Composition and Context -> Compose macros through Python, use compile context, and understand scoped imports. +> Compose Python macros through imports, and use adapter, target, constants and enums. Online: https://sqlbuild.com/docs/concepts/macros/composition-and-context/ @@ -14,8 +14,10 @@ Online: https://sqlbuild.com/docs/concepts/macros/composition-and-context/ - Nested calls written in SQL - Macro context -Macros are Python functions, so reusable macros should compose through ordinary Python calls. A -macro returns final SQL; SQLBuild does not treat its output as another layer of macro source. +[Python macros](python-macros.md) are Python functions, so they compose through +ordinary Python calls. A macro returns final SQL; SQLBuild does not treat its output as another +layer of macro source. [SQL macros](sql-macros.md#compose-macros) compose by +calling each other in their bodies. ## Compose helpers in one file diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/python-macros.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/python-macros.md new file mode 100644 index 0000000000..bf3eb2b3fa --- /dev/null +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/python-macros.md @@ -0,0 +1,249 @@ + + +# Python Macros + +> Write Python functions that generate SQL at compile time when the SQL needs logic. + +Online: https://sqlbuild.com/docs/concepts/macros/python-macros/ + +## Contents + +- Create a Python macro +- Call a Python macro +- Arguments +- Typed reference arguments +- Keep implementation details private +- Use Python macros in tests +- Use Python macros in hooks +- Macros must be deterministic +- Next steps + +A Python macro is a Python function that returns SQL during compilation. Use one when generating +the SQL needs logic: loops, conditions on arguments, the adapter or target, or project context. +When the SQL needs no logic, a [SQL macro](sql-macros.md) is simpler and the +compiler can see inside it. See [Macros](../macros.md) for how the two kinds compare. + +## Create a Python macro + +Put project-wide Python macros in Python files under `macros/python/`. Folder-scoped macros use the +same layout, such as `models/marts/_sqlbuild/_macros/python/`. + +Every public function defined by a macro file becomes callable from SQL: + +```python +# macros/python/currency.py +def cents_to_dollars(expression: str) -> str: + """Convert a cents expression to dollars.""" + return f"ROUND(CAST({expression} AS DOUBLE) / 100, 2)" +``` + +## Call a Python macro + +Call it with `@py.name(...)`: + +```sql +MODEL ( + description "One row per order", + materialized table, + tags [marts], +); + +SELECT + order_id, + @py.cents_to_dollars("amount_cents") AS amount_dollars +FROM __ref("stg_orders") +``` + +During compilation, SQLBuild replaces the call with the function's returned SQL: + +```sql +ROUND(CAST(amount_cents AS DOUBLE) / 100, 2) +``` + +A macro used directly in SQL must return a string. + +## Arguments + +Macro calls accept Python literal values: + +- Strings: `"value"` or `'value'` +- Numbers: `42`, `3.14`, `-1` +- Booleans: `True`, `False` +- Lists: `[1, 2, 3]` +- Dictionaries: `{"key": "value"}` +- `None` +- The result of another macro call, including a SQL macro, as in + `@py.bracket(@sql.cents_to_dollars(o.amount_cents))` +- `__ref("model")`, `__source("source")`, and `__seed("seed")` references, alone or inside lists + and dictionaries + +These inputs are rejected with an error. In a model body the error names the file line and +column; elsewhere it names the position within the macro's arguments: + +- bytes (`b"text"`), complex numbers (`2j`), f-strings, and `...` +- a keyword argument passed twice, such as `@py.lit(a=1, a=2)`, which Python also rejects +- lone surrogate escapes such as `"\ud800"`; write a character outside the Basic Multilingual + Plane as one escape, such as `"\U0001F600"`, rather than as a surrogate pair +- `\N{...}` naming a multi-character named sequence; `\N` escapes name one character +- two values with no comma between them + +Positional and keyword arguments are supported: + +```sql +@py.mock_orders(count=5, status="completed") +``` + +Use quoted strings when passing SQL expressions such as column names. The macro decides how to +place that text into its returned SQL. + +## Typed reference arguments + +Pass the models, sources, and seeds a macro reads as reference arguments: + +```sql +SELECT * FROM @py.union_all([__ref("orders_eu"), __ref("orders_us")]) +``` + +The macro receives each reference as a typed `SqlResourceRef` from `sqlbuild.refs`, the same type +Python nodes use. It has `kind` and `name` attributes, and formatting it into SQL renders the +relation with the adapter's quoting: + +```python +# models/marts/_sqlbuild/_macros/python/unions.py +from sqlbuild.refs import SqlResourceRef + +def union_all(relations: list[SqlResourceRef]) -> str: + return " UNION ALL ".join(f"SELECT order_id FROM {relation}" for relation in relations) +``` + +The dependency belongs to the model that makes the call. The graph, column lineage, and +declaration scopes treat `@py.union_all([__ref("orders_eu"), __ref("orders_us")])` exactly like a +model that wrote `__ref("orders_eu")` and `__ref("orders_us")` itself. + +### References in macro output + +A macro must not produce references on its own. Any `__ref()`, `__source()`, or `__seed()` in a +macro's expanded output fails to compile with `P006`, whether the macro's Python code wrote it or +it arrived as a quoted string argument such as `@py.wrap('__ref("stg_orders")')`: + +``` +error[P006]: model:order_summary depends on model:stg_orders through macro orders_base() + model: order_summary + --> models/marts/_sqlbuild/_macros/python/orders.py + = help: write the reference in the model, or pass it in: @py.orders_base(__ref("stg_orders")) + = help: the current value is [references] enforce_explicit = true (from sqlbuild_project.toml or its default); while migrating a project, to allow macro-generated references, set this in sqlbuild_project.toml: + [references] + enforce_explicit = false +``` + +A dependency hidden inside a macro is invisible to reviewers of the model file and bypasses +declaration scopes. For projects migrating from other tools, +[`[references] enforce_explicit = false`](../project-configuration.md#references) allows +macro-generated references project-wide. + +## Keep implementation details private + +Only public functions owned by the file are exported as macros. Prefix helpers, constants, classes, +and type aliases with `_`: + +```python +# macros/python/orders.py +_DEFAULT_STATUS = "completed" + +def _status_filter(status: str) -> str: + return f"status = '{status}'" + +def completed_orders() -> str: + return _status_filter(_DEFAULT_STATUS) +``` + +Here, SQL may call `@py.completed_orders()`. `_status_filter` and `_DEFAULT_STATUS` remain ordinary +Python implementation details. + +Imported functions are not re-exported as new macros from the importing file. + +## Use Python macros in tests + +Tests are SQL, so they can use macros as reusable fixture generators: + +```python +# tests/unit/orders/_sqlbuild/_macros/python/order_fixtures.py +def mock_orders(count: int = 1) -> str: + rows = [ + f"SELECT {i} AS id, {i * 100} AS customer_id, 'completed' AS status" + for i in range(1, count + 1) + ] + return " UNION ALL ".join(rows) +``` + +```sql +TEST(); + +WITH +__source__raw__orders AS ( + @py.mock_orders(3) +), +__expected__stg_orders AS ( + SELECT 1 AS order_id, 100 AS customer_id, 'completed' AS status + UNION ALL + SELECT 2 AS order_id, 200 AS customer_id, 'completed' AS status + UNION ALL + SELECT 3 AS order_id, 300 AS customer_id, 'completed' AS status +) +``` + +See [where a test-only macro lives](../macros.md#macros-used-only-by-tests). + +## Use Python macros in hooks + +Macros work inside inline and named SQL hooks: + +```python +# macros/python/permissions.py +def grant_target(target: str) -> str: + return f"GRANT SELECT ON {target} TO analyst_role" +``` + +```sql +MODEL ( + description "Example model", + post_hooks [inline_sql('@py.grant_target(@@CTX:destination.qualified)')], +); + +SELECT 1 AS id +``` + +See [SQL Hooks](../models/hooks/sql.md) for hook lifecycle and context syntax. + +## Macros must be deterministic + +A macro must return the same result whenever it is called with the same arguments and sees the same +context. Avoid module-level state such as counters, random values, the current time, and reads of +files or environment variables that the call does not receive. + +SQLBuild relies on this contract to reuse macro results: + +- [Compile reuse](../../cli/compile.md) replays an unchanged compile without running macros again. +- A compile may run each distinct macro call once and reuse its result for identical calls + elsewhere in the project. +- A compile that fails stops at its first error. It does not run the project's macros a second + time to report that error. +- Macro call results are kept between compiles under `target/cache/` by default, so editing a + model does not run its unchanged macro calls again. They are recomputed when + macro or other Python files, configuration, declarations, seeds, sources, variables, the target, + the adapter, the installed packages, or the SQLBuild version change. Editing only model SQL keeps + them, and a changed call is always run again. Only the first compile in a process uses and saves + them: later compiles in the same process, such as in a notebook or a long-running server, run their + macros again. A compile that has loaded Python code it cannot read from a plain file, such as a + module imported from a zip archive, neither uses nor saves them. + +Pass configuration through what SQLBuild compares: macro arguments (including `@@ENV:NAME` and +`@@var` values written in the calling SQL), `ctx.vars`, `ctx.target_name`, `ctx.adapter_name`, +and visible constants and enums. Environment variables a macro reads directly, for example with +`os.environ`, are compared only when their names start with `SQLBUILD_` or `SQB_`, the same set +compile reuse tracks. A macro whose result changes for any other reason may return a stale result. + +## Next steps + + Compose Python macros through imports, and use adapter, target, constants and enums. + Write typed, logic-free SQL fragments the compiler checks and sees inside. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md index 3db8c8c4a1..c0e2718f54 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md @@ -14,10 +14,9 @@ Online: https://sqlbuild.com/docs/concepts/macros/sql-macros/ - Parameters in the body - Compose macros - Checks and tools -- Choose the right tool A SQL macro is a reusable piece of SQL with typed parameters. Unlike a -[Python macro](../macros.md), it holds no logic: its body is SQL, so the compiler checks +[Python macro](python-macros.md), it holds no logic: its body is SQL, so the compiler checks it on its own, expands it in place, and formatting, rules and column lineage see through every call. @@ -275,14 +274,5 @@ macro file. Test SQL macros with [macro-mode unit tests](../testing.md#macro-tests), and replace one in a model test with a [`__macro__` mock](../testing.md#macro-mocking). -## Choose the right tool - -| Use | When | -|---|---| -| SQL macro | The fragment is plain SQL with typed inputs: a calculation, a filter, a set of CTEs or columns you repeat. The compiler sees inside it. | -| [Python macro](../macros.md) | Generating the SQL needs logic: loops, conditions on arguments, the adapter or target, or reading project context. | -| [SQL function](../functions.md) | The logic belongs in the warehouse, to be called by other tools or evaluated per row there, or it is too large to inline everywhere. | -| [Generic audit](../audits.md) | The SQL checks data quality on a model rather than producing part of its query. | - -Prefer a SQL macro over a Python macro whenever no logic is needed: its calls stay readable, -typed and checked, and every tool sees the SQL it produces. +See [Macros](../macros.md#choose-the-right-tool) for when to use a SQL macro, a Python +macro, a SQL function or a generic audit. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md index 00ba7a33df..b733f82084 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md @@ -772,7 +772,7 @@ Every reference to a model, source, or seed must be explicit and typed. By defau - A macro whose expanded output contains `__ref()`, `__source()`, or `__seed()` fails to compile with `P006`. Pass the reference to the macro as an argument instead; see - [Typed reference arguments](macros.md#typed-reference-arguments). + [Typed reference arguments](macros/python-macros.md#typed-reference-arguments). - Literal SQL passed to `ctx.query()` or `ctx.execute_sql()` in a task, asset, check, loader, or Python hook that names a project relation fails to compile with `P008`. - The same check runs on SQL sent at run time and reports a `P008` warning; the query still runs. diff --git a/website/src/content/docs/docs/cli/compile.mdx b/website/src/content/docs/docs/cli/compile.mdx index a0ac52bf6b..42f65754c8 100644 --- a/website/src/content/docs/docs/cli/compile.mdx +++ b/website/src/content/docs/docs/cli/compile.mdx @@ -119,7 +119,7 @@ files count only by presence. Reuse assumes macros, hooks, providers, adapters, and custom rules are deterministic: the same inputs must produce the same output. Macro results are also reused within a compile and may be kept between -compiles; see [Macros must be deterministic](/docs/concepts/macros#macros-must-be-deterministic). A +compiles; see [Macros must be deterministic](/docs/concepts/macros/python-macros#macros-must-be-deterministic). A compile whose templates read `run.id` is never reused. Compile always runs in full with `--no-cache`, `--manifest`, `--debug`, or a profiling flag, when @@ -181,7 +181,7 @@ the whole project. Every model's SQL references are scanned again too. lexical rules, and the SQLBuild build; a damaged or unreadable entry is scanned again. - Rule results are reused for unchanged inputs. - Macro call results are reused as described in - [Macros must be deterministic](/docs/concepts/macros#macros-must-be-deterministic). + [Macros must be deterministic](/docs/concepts/macros/python-macros#macros-must-be-deterministic). Either way, the output is byte-identical to `sqb compile --no-cache`. To turn compiling only the changed files off, set `delta_compile = false` under `[settings]` or set diff --git a/website/src/content/docs/docs/concepts/declaration-scopes.mdx b/website/src/content/docs/docs/concepts/declaration-scopes.mdx index dceb1e04d3..e4d905a62f 100644 --- a/website/src/content/docs/docs/concepts/declaration-scopes.mdx +++ b/website/src/content/docs/docs/concepts/declaration-scopes.mdx @@ -141,6 +141,6 @@ top-level roles as a genuine project API instead of a neutral dumping ground. Learn the features themselves in [Enums](/docs/concepts/enums), [Constants](/docs/concepts/constants), -[Writing Macros](/docs/concepts/macros), [Audits](/docs/concepts/audits), +[Macros](/docs/concepts/macros), [Audits](/docs/concepts/audits), [Reusable Schemas](/docs/concepts/models/schemas), and [Hooks](/docs/concepts/models/hooks). See [Interpolation](/docs/concepts/interpolation) for project, environment, and runtime context values, which are separate from declarations. diff --git a/website/src/content/docs/docs/concepts/macros.mdx b/website/src/content/docs/docs/concepts/macros.mdx index 21e661d40d..0cc14583d2 100644 --- a/website/src/content/docs/docs/concepts/macros.mdx +++ b/website/src/content/docs/docs/concepts/macros.mdx @@ -1,256 +1,83 @@ --- -title: "Writing Macros" -description: "Write Python functions that generate reusable SQL fragments at compile time." +title: "Macros" +description: "Reuse SQL with typed SQL macros, or generate it with Python macros when it needs logic." --- import Card from '../../../../components/mintlify/Card.astro'; import CardGroup from '../../../../components/mintlify/CardGroup.astro'; -Python macros are Python functions that generate SQL during compilation. They provide reusable, -parameterized SQL without introducing a separate template language. When the SQL needs no logic, -a typed [SQL macro](/docs/concepts/macros/sql-macros) is simpler and the compiler can see inside -it. +A macro is a reusable piece of SQL that a model, test, hook or audit calls by name. SQLBuild +expands each call at compile time, so the warehouse only ever sees plain SQL. -## Create a macro +There are two kinds: -Put project-wide Python macros in Python files under `macros/python/`. SQL macros live next to -them in `macros/sql/`: +| | SQL macro | Python macro | +|---|---|---| +| Written as | A `.sql` file: a `MACRO` header and a SQL body | A Python function that returns SQL | +| Called as | `@sql.name(...)` | `@py.name(...)` | +| Lives in | `macros/sql/` | `macros/python/` | +| Arguments | Typed SQL expressions, checked against the caller's column types | Python values: strings, numbers, lists, dictionaries, references | +| Logic | None: the body is SQL | Any Python: loops, conditions, adapter and target context | +| Checks and tools | Each macro is checked on its own; `sqb format`, Rules and lineage see inside every call | Rules and lineage see the SQL each call returns | + +Prefer a SQL macro whenever no logic is needed: its calls stay readable, typed and checked, and +every tool sees the SQL it produces. + + + + Typed, logic-free expressions, CTE lists and column lists. + + + Python functions that generate SQL when it needs logic. + + + +## Where macros live + +Each kind has its own folder: ```text my_project/ ├── macros/ │ ├── python/ -│ │ └── currency.py +│ │ └── labels.py │ └── sql/ +│ └── cents_to_dollars.sql ├── models/ └── sqlbuild_project.toml ``` A file directly under a `macros/` folder fails with `D013`, which names the subfolder to move it -to. Folder-scoped macros use the same layout, such as `models/marts/_sqlbuild/_macros/python/`. +to. A macro used by only part of the project can live in that folder instead, as +`_sqlbuild/macros/sql/` or `_sqlbuild/_macros/python/`, and is then visible only there. See +[Declarations and Scopes](/docs/concepts/declaration-scopes). -Every public function defined by a macro file becomes callable from SQL: - -```python -# macros/python/currency.py -def cents_to_dollars(expression: str) -> str: - """Convert a cents expression to dollars.""" - return f"ROUND(CAST({expression} AS DOUBLE) / 100, 2)" -``` +## Call a macro -## Call a macro from SQL - -Use `@py.macro_name(...)` in a model query. Every macro call says its kind: `@py.` calls a Python -macro and `@sql.` a SQL macro. A bare `@macro_name(...)` fails with `P014`, which names both -spellings, and calling a macro with the other kind's prefix fails with `P020`. +Every call names the kind of macro it calls: ```sql -MODEL ( - description "One row per order", - materialized table, - tags [marts], -); - SELECT order_id, - @py.cents_to_dollars("amount_cents") AS amount_dollars + @sql.cents_to_dollars(amount_cents) AS amount_dollars, + @py.status_label("status") AS status_label FROM __ref("stg_orders") ``` -During compilation, SQLBuild replaces the call with the function's returned SQL: - -```sql -ROUND(CAST(amount_cents AS DOUBLE) / 100, 2) -``` - -A macro used directly in SQL must return a string. - -## Arguments - -Macro calls accept Python literal values: - -- Strings: `"value"` or `'value'` -- Numbers: `42`, `3.14`, `-1` -- Booleans: `True`, `False` -- Lists: `[1, 2, 3]` -- Dictionaries: `{"key": "value"}` -- `None` -- The result of another macro call -- `__ref("model")`, `__source("source")`, and `__seed("seed")` references, alone or inside lists - and dictionaries - -These inputs are rejected with an error. In a model body the error names the file line and -column; elsewhere it names the position within the macro's arguments: - -- bytes (`b"text"`), complex numbers (`2j`), f-strings, and `...` -- a keyword argument passed twice, such as `@py.lit(a=1, a=2)`, which Python also rejects -- lone surrogate escapes such as `"\ud800"`; write a character outside the Basic Multilingual - Plane as one escape, such as `"\U0001F600"`, rather than as a surrogate pair -- `\N{...}` naming a multi-character named sequence; `\N` escapes name one character -- two values with no comma between them - -Positional and keyword arguments are supported: - -```sql -@py.mock_orders(count=5, status="completed") -``` - -Use quoted strings when passing SQL expressions such as column names. The macro decides how to -place that text into its returned SQL. - -## Typed reference arguments - -Pass the models, sources, and seeds a macro reads as reference arguments: - -```sql -SELECT * FROM @py.union_all([__ref("orders_eu"), __ref("orders_us")]) -``` - -The macro receives each reference as a typed `SqlResourceRef` from `sqlbuild.refs`, the same type -Python nodes use. It has `kind` and `name` attributes, and formatting it into SQL renders the -relation with the adapter's quoting: - -```python -# models/marts/_sqlbuild/_macros/python/unions.py -from sqlbuild.refs import SqlResourceRef - - -def union_all(relations: list[SqlResourceRef]) -> str: - return " UNION ALL ".join(f"SELECT order_id FROM {relation}" for relation in relations) -``` - -The dependency belongs to the model that makes the call. The graph, column lineage, and -declaration scopes treat `@py.union_all([__ref("orders_eu"), __ref("orders_us")])` exactly like a -model that wrote `__ref("orders_eu")` and `__ref("orders_us")` itself. - -### References in macro output - -A macro must not produce references on its own. Any `__ref()`, `__source()`, or `__seed()` in a -macro's expanded output fails to compile with `P006`, whether the macro's Python code wrote it or -it arrived as a quoted string argument such as `@py.wrap('__ref("stg_orders")')`: - -``` -error[P006]: model:order_summary depends on model:stg_orders through macro orders_base() - model: order_summary - --> models/marts/_sqlbuild/_macros/python/orders.py - = help: write the reference in the model, or pass it in: @py.orders_base(__ref("stg_orders")) - = help: the current value is [references] enforce_explicit = true (from sqlbuild_project.toml or its default); while migrating a project, to allow macro-generated references, set this in sqlbuild_project.toml: - [references] - enforce_explicit = false -``` - -A dependency hidden inside a macro is invisible to reviewers of the model file and bypasses -declaration scopes. For projects migrating from other tools, -[`[references] enforce_explicit = false`](/docs/concepts/project-configuration#references) allows -macro-generated references project-wide. - -## Keep implementation details private - -Only public functions owned by the file are exported as macros. Prefix helpers, constants, classes, -and type aliases with `_`: - -```python -# macros/python/orders.py -_DEFAULT_STATUS = "completed" - - -def _status_filter(status: str) -> str: - return f"status = '{status}'" - - -def completed_orders() -> str: - return _status_filter(_DEFAULT_STATUS) -``` +A bare `@name(...)` fails with `P014`, which names both spellings. Calling a macro with the other +kind's prefix fails with `P020`, and a SQL macro and a Python macro may not share a visible name. -Here, SQL may call `@py.completed_orders()`. `_status_filter` and `_DEFAULT_STATUS` remain ordinary -Python implementation details. +## Use the two kinds together -Imported functions are not re-exported as new macros from the importing file. - -## Use macros in tests - -Tests are SQL, so they can use macros as reusable fixture generators. A macro that only tests use -follows the usual [placement](/docs/concepts/declaration-scopes/placement) rule: it must be private -to the narrowest folder that contains its uses, so the project-wide `macros/` directory rejects it -with `S024`. Put it in `_sqlbuild/_macros/python/` in the folder of the tests that call it, such as -`tests/unit/orders/_sqlbuild/_macros/python/`, or, when those tests define `__expected__` output -for a model, in that model's `_sqlbuild/_macros/python/`. The `S024` error names the folder to use. -`tests/unit/_sqlbuild/` itself is rejected with `D013`. - -```python -# tests/unit/orders/_sqlbuild/_macros/python/order_fixtures.py -def mock_orders(count: int = 1) -> str: - rows = [ - f"SELECT {i} AS id, {i * 100} AS customer_id, 'completed' AS status" - for i in range(1, count + 1) - ] - return " UNION ALL ".join(rows) -``` +A SQL macro calls only SQL macros, and a Python macro composes other Python macros through Python +imports. To combine the two, pass one call's result to the other in the calling SQL: ```sql -TEST(); - -WITH -__source__raw__orders AS ( - @py.mock_orders(3) -), -__expected__stg_orders AS ( - SELECT 1 AS order_id, 100 AS customer_id, 'completed' AS status - UNION ALL - SELECT 2 AS order_id, 200 AS customer_id, 'completed' AS status - UNION ALL - SELECT 3 AS order_id, 300 AS customer_id, 'completed' AS status -) +SELECT @py.bracket(@sql.cents_to_dollars(o.amount_cents)) AS amount_band +FROM __ref("stg_orders") AS o ``` -## Use macros in hooks - -Macros work inside inline and named SQL hooks: - -```python -# macros/python/permissions.py -def grant_target(target: str) -> str: - return f"GRANT SELECT ON {target} TO analyst_role" -``` - -```sql -MODEL ( - description "Example model", - post_hooks [inline_sql('@py.grant_target(@@CTX:destination.qualified)')], -); - -SELECT 1 AS id -``` - -See [SQL Hooks](/docs/concepts/models/hooks/sql) for hook lifecycle and context syntax. - -## Macros must be deterministic - -A macro must return the same result whenever it is called with the same arguments and sees the same -context. Avoid module-level state such as counters, random values, the current time, and reads of -files or environment variables that the call does not receive. - -SQLBuild relies on this contract to reuse macro results: - -- [Compile reuse](/docs/cli/compile) replays an unchanged compile without running macros again. -- A compile may run each distinct macro call once and reuse its result for identical calls - elsewhere in the project. -- A compile that fails stops at its first error. It does not run the project's macros a second - time to report that error. -- Macro call results are kept between compiles under `target/cache/` by default, so editing a - model does not run its unchanged macro calls again. They are recomputed when - macro or other Python files, configuration, declarations, seeds, sources, variables, the target, - the adapter, the installed packages, or the SQLBuild version change. Editing only model SQL keeps - them, and a changed call is always run again. Only the first compile in a process uses and saves - them: later compiles in the same process, such as in a notebook or a long-running server, run their - macros again. A compile that has loaded Python code it cannot read from a plain file, such as a - module imported from a zip archive, neither uses nor saves them. - -Pass configuration through what SQLBuild compares: macro arguments (including `@@ENV:NAME` and -`@@var` values written in the calling SQL), `ctx.vars`, `ctx.target_name`, `ctx.adapter_name`, -and visible constants and enums. Environment variables a macro reads directly, for example with -`os.environ`, are compared only when their names start with `SQLBUILD_` or `SQB_`, the same set -compile reuse tracks. A macro whose result changes for any other reason may return a stale result. +The SQL macro expands first, and the Python macro receives its SQL as a string. ## Where macros work @@ -265,16 +92,21 @@ Macros are supported in: Macros are not accepted in ordinary `MODEL()` configuration fields. SQL hook entries are the exception because their contents are SQL. -## Next steps +### Macros used only by tests - - - Write typed, logic-free SQL fragments the compiler checks and sees inside. - - - Compose macros through Python imports and use adapter or target context. - - - Limit a macro to one folder, or to that folder and its child folders. - - +A macro that only tests use follows the usual [placement](/docs/concepts/declaration-scopes/placement) +rule: it must be private to the narrowest folder that contains its uses, so the project-wide +`macros/` directory rejects it with `S024`. Put it in `_sqlbuild/_macros/python/` or +`_sqlbuild/_macros/sql/` in the folder of the tests that call it, such as +`tests/unit/orders/_sqlbuild/_macros/python/`, or, when those tests define `__expected__` output +for a model, in that model's `_sqlbuild/_macros/`. The `S024` error names the folder to use. +`tests/unit/_sqlbuild/` itself is rejected with `D013`. + +## Choose the right tool + +| Use | When | +|---|---| +| [SQL macro](/docs/concepts/macros/sql-macros) | The fragment is plain SQL with typed inputs: a calculation, a filter, a set of CTEs or columns you repeat. The compiler sees inside it. | +| [Python macro](/docs/concepts/macros/python-macros) | Generating the SQL needs logic: loops, conditions on arguments, the adapter or target, or reading project context. | +| [SQL function](/docs/concepts/functions) | The logic belongs in the warehouse, to be called by other tools or evaluated per row there, or it is too large to inline everywhere. | +| [Generic audit](/docs/concepts/audits) | The SQL checks data quality on a model rather than producing part of its query. | diff --git a/website/src/content/docs/docs/concepts/macros/composition-and-context.mdx b/website/src/content/docs/docs/concepts/macros/composition-and-context.mdx index 4b2a00d308..54f9091665 100644 --- a/website/src/content/docs/docs/concepts/macros/composition-and-context.mdx +++ b/website/src/content/docs/docs/concepts/macros/composition-and-context.mdx @@ -1,10 +1,12 @@ --- -title: "Composition and Context" -description: "Compose macros through Python, use compile context, and understand scoped imports." +title: "Python Macro Composition and Context" +description: "Compose Python macros through imports, and use adapter, target, constants and enums." --- -Macros are Python functions, so reusable macros should compose through ordinary Python calls. A -macro returns final SQL; SQLBuild does not treat its output as another layer of macro source. +[Python macros](/docs/concepts/macros/python-macros) are Python functions, so they compose through +ordinary Python calls. A macro returns final SQL; SQLBuild does not treat its output as another +layer of macro source. [SQL macros](/docs/concepts/macros/sql-macros#compose-macros) compose by +calling each other in their bodies. ## Compose helpers in one file diff --git a/website/src/content/docs/docs/concepts/macros/python-macros.mdx b/website/src/content/docs/docs/concepts/macros/python-macros.mdx new file mode 100644 index 0000000000..8df5d4f850 --- /dev/null +++ b/website/src/content/docs/docs/concepts/macros/python-macros.mdx @@ -0,0 +1,246 @@ +--- +title: "Python Macros" +description: "Write Python functions that generate SQL at compile time when the SQL needs logic." +--- + +import Card from '../../../../../components/mintlify/Card.astro'; +import CardGroup from '../../../../../components/mintlify/CardGroup.astro'; + +A Python macro is a Python function that returns SQL during compilation. Use one when generating +the SQL needs logic: loops, conditions on arguments, the adapter or target, or project context. +When the SQL needs no logic, a [SQL macro](/docs/concepts/macros/sql-macros) is simpler and the +compiler can see inside it. See [Macros](/docs/concepts/macros) for how the two kinds compare. + +## Create a Python macro + +Put project-wide Python macros in Python files under `macros/python/`. Folder-scoped macros use the +same layout, such as `models/marts/_sqlbuild/_macros/python/`. + +Every public function defined by a macro file becomes callable from SQL: + +```python +# macros/python/currency.py +def cents_to_dollars(expression: str) -> str: + """Convert a cents expression to dollars.""" + return f"ROUND(CAST({expression} AS DOUBLE) / 100, 2)" +``` + +## Call a Python macro + +Call it with `@py.name(...)`: + +```sql +MODEL ( + description "One row per order", + materialized table, + tags [marts], +); + +SELECT + order_id, + @py.cents_to_dollars("amount_cents") AS amount_dollars +FROM __ref("stg_orders") +``` + +During compilation, SQLBuild replaces the call with the function's returned SQL: + +```sql +ROUND(CAST(amount_cents AS DOUBLE) / 100, 2) +``` + +A macro used directly in SQL must return a string. + +## Arguments + +Macro calls accept Python literal values: + +- Strings: `"value"` or `'value'` +- Numbers: `42`, `3.14`, `-1` +- Booleans: `True`, `False` +- Lists: `[1, 2, 3]` +- Dictionaries: `{"key": "value"}` +- `None` +- The result of another macro call, including a SQL macro, as in + `@py.bracket(@sql.cents_to_dollars(o.amount_cents))` +- `__ref("model")`, `__source("source")`, and `__seed("seed")` references, alone or inside lists + and dictionaries + +These inputs are rejected with an error. In a model body the error names the file line and +column; elsewhere it names the position within the macro's arguments: + +- bytes (`b"text"`), complex numbers (`2j`), f-strings, and `...` +- a keyword argument passed twice, such as `@py.lit(a=1, a=2)`, which Python also rejects +- lone surrogate escapes such as `"\ud800"`; write a character outside the Basic Multilingual + Plane as one escape, such as `"\U0001F600"`, rather than as a surrogate pair +- `\N{...}` naming a multi-character named sequence; `\N` escapes name one character +- two values with no comma between them + +Positional and keyword arguments are supported: + +```sql +@py.mock_orders(count=5, status="completed") +``` + +Use quoted strings when passing SQL expressions such as column names. The macro decides how to +place that text into its returned SQL. + +## Typed reference arguments + +Pass the models, sources, and seeds a macro reads as reference arguments: + +```sql +SELECT * FROM @py.union_all([__ref("orders_eu"), __ref("orders_us")]) +``` + +The macro receives each reference as a typed `SqlResourceRef` from `sqlbuild.refs`, the same type +Python nodes use. It has `kind` and `name` attributes, and formatting it into SQL renders the +relation with the adapter's quoting: + +```python +# models/marts/_sqlbuild/_macros/python/unions.py +from sqlbuild.refs import SqlResourceRef + + +def union_all(relations: list[SqlResourceRef]) -> str: + return " UNION ALL ".join(f"SELECT order_id FROM {relation}" for relation in relations) +``` + +The dependency belongs to the model that makes the call. The graph, column lineage, and +declaration scopes treat `@py.union_all([__ref("orders_eu"), __ref("orders_us")])` exactly like a +model that wrote `__ref("orders_eu")` and `__ref("orders_us")` itself. + +### References in macro output + +A macro must not produce references on its own. Any `__ref()`, `__source()`, or `__seed()` in a +macro's expanded output fails to compile with `P006`, whether the macro's Python code wrote it or +it arrived as a quoted string argument such as `@py.wrap('__ref("stg_orders")')`: + +``` +error[P006]: model:order_summary depends on model:stg_orders through macro orders_base() + model: order_summary + --> models/marts/_sqlbuild/_macros/python/orders.py + = help: write the reference in the model, or pass it in: @py.orders_base(__ref("stg_orders")) + = help: the current value is [references] enforce_explicit = true (from sqlbuild_project.toml or its default); while migrating a project, to allow macro-generated references, set this in sqlbuild_project.toml: + [references] + enforce_explicit = false +``` + +A dependency hidden inside a macro is invisible to reviewers of the model file and bypasses +declaration scopes. For projects migrating from other tools, +[`[references] enforce_explicit = false`](/docs/concepts/project-configuration#references) allows +macro-generated references project-wide. + +## Keep implementation details private + +Only public functions owned by the file are exported as macros. Prefix helpers, constants, classes, +and type aliases with `_`: + +```python +# macros/python/orders.py +_DEFAULT_STATUS = "completed" + + +def _status_filter(status: str) -> str: + return f"status = '{status}'" + + +def completed_orders() -> str: + return _status_filter(_DEFAULT_STATUS) +``` + +Here, SQL may call `@py.completed_orders()`. `_status_filter` and `_DEFAULT_STATUS` remain ordinary +Python implementation details. + +Imported functions are not re-exported as new macros from the importing file. + +## Use Python macros in tests + +Tests are SQL, so they can use macros as reusable fixture generators: + +```python +# tests/unit/orders/_sqlbuild/_macros/python/order_fixtures.py +def mock_orders(count: int = 1) -> str: + rows = [ + f"SELECT {i} AS id, {i * 100} AS customer_id, 'completed' AS status" + for i in range(1, count + 1) + ] + return " UNION ALL ".join(rows) +``` + +```sql +TEST(); + +WITH +__source__raw__orders AS ( + @py.mock_orders(3) +), +__expected__stg_orders AS ( + SELECT 1 AS order_id, 100 AS customer_id, 'completed' AS status + UNION ALL + SELECT 2 AS order_id, 200 AS customer_id, 'completed' AS status + UNION ALL + SELECT 3 AS order_id, 300 AS customer_id, 'completed' AS status +) +``` + +See [where a test-only macro lives](/docs/concepts/macros#macros-used-only-by-tests). + +## Use Python macros in hooks + +Macros work inside inline and named SQL hooks: + +```python +# macros/python/permissions.py +def grant_target(target: str) -> str: + return f"GRANT SELECT ON {target} TO analyst_role" +``` + +```sql +MODEL ( + description "Example model", + post_hooks [inline_sql('@py.grant_target(@@CTX:destination.qualified)')], +); + +SELECT 1 AS id +``` + +See [SQL Hooks](/docs/concepts/models/hooks/sql) for hook lifecycle and context syntax. + +## Macros must be deterministic + +A macro must return the same result whenever it is called with the same arguments and sees the same +context. Avoid module-level state such as counters, random values, the current time, and reads of +files or environment variables that the call does not receive. + +SQLBuild relies on this contract to reuse macro results: + +- [Compile reuse](/docs/cli/compile) replays an unchanged compile without running macros again. +- A compile may run each distinct macro call once and reuse its result for identical calls + elsewhere in the project. +- A compile that fails stops at its first error. It does not run the project's macros a second + time to report that error. +- Macro call results are kept between compiles under `target/cache/` by default, so editing a + model does not run its unchanged macro calls again. They are recomputed when + macro or other Python files, configuration, declarations, seeds, sources, variables, the target, + the adapter, the installed packages, or the SQLBuild version change. Editing only model SQL keeps + them, and a changed call is always run again. Only the first compile in a process uses and saves + them: later compiles in the same process, such as in a notebook or a long-running server, run their + macros again. A compile that has loaded Python code it cannot read from a plain file, such as a + module imported from a zip archive, neither uses nor saves them. + +Pass configuration through what SQLBuild compares: macro arguments (including `@@ENV:NAME` and +`@@var` values written in the calling SQL), `ctx.vars`, `ctx.target_name`, `ctx.adapter_name`, +and visible constants and enums. Environment variables a macro reads directly, for example with +`os.environ`, are compared only when their names start with `SQLBUILD_` or `SQB_`, the same set +compile reuse tracks. A macro whose result changes for any other reason may return a stale result. + +## Next steps + + + + Compose Python macros through imports, and use adapter, target, constants and enums. + + + Write typed, logic-free SQL fragments the compiler checks and sees inside. + + diff --git a/website/src/content/docs/docs/concepts/macros/sql-macros.mdx b/website/src/content/docs/docs/concepts/macros/sql-macros.mdx index 3b0bf46801..f9b16676e7 100644 --- a/website/src/content/docs/docs/concepts/macros/sql-macros.mdx +++ b/website/src/content/docs/docs/concepts/macros/sql-macros.mdx @@ -4,7 +4,7 @@ description: "Write typed, logic-free SQL fragments that the compiler checks and --- A SQL macro is a reusable piece of SQL with typed parameters. Unlike a -[Python macro](/docs/concepts/macros), it holds no logic: its body is SQL, so the compiler checks +[Python macro](/docs/concepts/macros/python-macros), it holds no logic: its body is SQL, so the compiler checks it on its own, expands it in place, and formatting, rules and column lineage see through every call. @@ -262,14 +262,5 @@ macro file. Test SQL macros with [macro-mode unit tests](/docs/concepts/testing#macro-tests), and replace one in a model test with a [`__macro__` mock](/docs/concepts/testing#macro-mocking). -## Choose the right tool - -| Use | When | -|---|---| -| SQL macro | The fragment is plain SQL with typed inputs: a calculation, a filter, a set of CTEs or columns you repeat. The compiler sees inside it. | -| [Python macro](/docs/concepts/macros) | Generating the SQL needs logic: loops, conditions on arguments, the adapter or target, or reading project context. | -| [SQL function](/docs/concepts/functions) | The logic belongs in the warehouse, to be called by other tools or evaluated per row there, or it is too large to inline everywhere. | -| [Generic audit](/docs/concepts/audits) | The SQL checks data quality on a model rather than producing part of its query. | - -Prefer a SQL macro over a Python macro whenever no logic is needed: its calls stay readable, -typed and checked, and every tool sees the SQL it produces. +See [Macros](/docs/concepts/macros#choose-the-right-tool) for when to use a SQL macro, a Python +macro, a SQL function or a generic audit. diff --git a/website/src/content/docs/docs/concepts/project-configuration.mdx b/website/src/content/docs/docs/concepts/project-configuration.mdx index da42b449b6..349dd292a1 100644 --- a/website/src/content/docs/docs/concepts/project-configuration.mdx +++ b/website/src/content/docs/docs/concepts/project-configuration.mdx @@ -748,7 +748,7 @@ Every reference to a model, source, or seed must be explicit and typed. By defau - A macro whose expanded output contains `__ref()`, `__source()`, or `__seed()` fails to compile with `P006`. Pass the reference to the macro as an argument instead; see - [Typed reference arguments](/docs/concepts/macros#typed-reference-arguments). + [Typed reference arguments](/docs/concepts/macros/python-macros#typed-reference-arguments). - Literal SQL passed to `ctx.query()` or `ctx.execute_sql()` in a task, asset, check, loader, or Python hook that names a project relation fails to compile with `P008`. - The same check runs on SQL sent at run time and reports a `P008` warning; the query still runs. diff --git a/website/src/sidebar.json b/website/src/sidebar.json index 9ab1bad3c1..c8f1f25bdf 100644 --- a/website/src/sidebar.json +++ b/website/src/sidebar.json @@ -93,8 +93,9 @@ "collapsed": true, "items": [ "docs/concepts/macros", - "docs/concepts/macros/composition-and-context", - "docs/concepts/macros/sql-macros" + "docs/concepts/macros/sql-macros", + "docs/concepts/macros/python-macros", + "docs/concepts/macros/composition-and-context" ] }, "docs/concepts/interpolation",