Repository navigation
feat!: require explicit, typed references in macros, hooks and Python nodes - #858
Merged
Merged
Conversation
…nerated references Macros now accept __ref(), __source() and __seed() as arguments, including inside lists and dictionaries, and receive sqlbuild.refs typed values. The reference renders at the call site, so the graph, lineage and scopes see it as written in the model. References that appear in a macro's expanded output fail compile with P006 unless [references] enforce_explicit = false. BREAKING CHANGE: macros that emit __ref(), __source() or __seed() in their output now fail compile; pass the reference in as a typed argument.
…ames @hook(reads=...) declares the models, sources and seeds a Python hook reads; ctx.relation(...) resolves them and raises for undeclared refs. Declared reads order the hooked model after the resources it reads, and a read that depends on the hooked model fails compile with P007. Literal SQL passed to ctx.query or ctx.execute_sql in tasks, assets and hooks that names a project relation fails compile with P008. At run time the same check parses SQL in the adapter dialect and records a P008 warning in run output, the summary and the JSON run results; the query still runs. [references] enforce_explicit = false disables both. Build summaries now print warning details before the completion line and count a node that succeeded with warnings as WARN. BREAKING CHANGE: Python tasks, assets and hooks that hard-code project relation names in literal SQL now fail compile; declare the dependency and use ctx.relation(...).
…nd check SQL Tasks, assets and checks may declare model(), source() and seed() in depends_on and resolve them with ctx.relation(...). Seed dependencies order read-side tasks after the seed, appear as edges in sqb dag --json and Dagster, and are validated by sqb check. A check with SQL dependencies runs at the end of sqb build when those resources ran in the build, and its checked assets include them. Selector expansion now crosses the SQL/Python boundary: +task:x also selects the models, sources and seeds x declares and their upstream, and x+ on a SQL resource also selects the tasks and assets that read it. Literal SQL in checks and loaders that names a project relation now fails compile with P008, and dynamic SQL produces a P008 run-time warning, as for tasks and assets. Loader help points to ctx.source(...) or ctx.loader(...), and a model or seed named in loader SQL is reported as unreadable because loaders run before models and seeds. ctx.source() and ctx.loader() count as resolved. Unqualified matches add a help to qualify an external table that shares the name. Loader warnings appear in the build summary and in the JSON run results. BREAKING CHANGE: Python checks and loaders that hard-code project relation names in literal SQL now fail compile; use ctx.relation(...) in checks and ctx.source(...) or ctx.loader(...) in loaders. BREAKING CHANGE: +<python node> selectors now include the SQL resources the node declares, and <sql resource>+ selectors include the tasks and assets that declare it.
… rules Checks may depend on models, sources and seeds; tasks and assets may depend on seeds; selector expansion follows declared SQL references; loaders read upstream relations through ctx.source() and ctx.loader(); Dagster does not add hook reads to its asset graph.
Macro-generated references, hook reads of a hook's own dependant, and hard-coded relation names in Python SQL now become compile diagnostics instead of stopping at the first one. One compile reports each violation once (per consumer, macro and reference for P006; per hook read for P007; per node, line and relation for P008) in deterministic order, in the terminal and in sqb compile --json with has_errors. Commands that plan or build still fail before planning and list every error. Re-expanding model SQL with mocked macros for SQL tests no longer checks macro-generated references again; the model's own expansion reports them.
…pile Single-model singular audits (P004), attached audits reading a resource built from their target (P005), declarations not visible from their consumer (S006), and declaration placement errors (S007, S008, S009, S010, S024) now become individual compile diagnostics instead of stopping at the first one or folding every scope finding into one P001. Each carries its file path and, where it has one, the model or audit name, so the terminal marks the right rows FAIL and sqb compile --json reports every error with has_errors. Commands that plan or build still fail before planning and list every error. Project-loading failures stay fatal: audits/ and _sqlbuild/ layout errors, unknown seed YAML keys, and declaration-index parse, name and duplicate errors.
…m the selected roots +x+ selected the downstream of x's upstream, so a task or model selector also ran sibling tasks that read x's upstream. Upstream and downstream expansion now both start from the selected resources, so +x+ is x, its ancestors and its descendants, and SQL-only selections are unchanged.
Python hooks resolved ctx.relation(source(...)) from the plan's load map, so under defer_sources_to a hook read the local source while the model it ran on read the deferred one. Tasks, assets and checks only deferred a source when a selected SQL model also read it, and sqb check never deferred. The plan now carries the source entries Python reads resolve, each as a selected SQL read of that source sees it. Hooks, tasks, assets, checks, the check existence guard and the hard-coded relation guard use it, and sqb check plans with source deferral like sqb build.
Every SqlResourceRef formatted while a macro ran rendered as an approved
call-site placeholder, so a macro that built model("orders") itself
compiled without P006. Only references written as arguments at the call
site, including inside nested macro arguments, render as approved
placeholders now; any other reference renders as its reference call and
is reported as macro-generated, or becomes an ordinary reference with
[references] enforce_explicit = false.
A reference passed to a macro became a dependency only if the macro rendered it, so @unused(__ref("orders")) or an unchosen dictionary or list entry left the caller without that dependency. Every __ref(), __source() and __seed() written as a macro argument, including inside nested macro arguments, is now a dependency of the calling model, audit or function, so it orders, selects and appears in the DAG like a rendered reference.
…word
Compile-time P008 only inspected the first positional argument, so
ctx.query(sql="...") and ctx.execute_sql(sql="...") skipped the
hard-coded project relation check. The sql keyword and a literal
**{"sql": ...} unpack are now checked like the positional form.
__ref(), __source() and __seed() in inline and named SQL hook SQL, written directly or passed as typed macro arguments, previously compiled without being recorded and reached the warehouse unresolved. They now resolve at plan time exactly like references in model SQL, including adapter quoting, source deferral, --defer and target overrides. Each reference is also a hook read with the same rules as Python @hook(reads=...): the hooked model runs after the resources it reads when both are in the build, reads never change selection, a read of a resource built from the hooked model fails with P007, and an unknown reference fails to compile. Macro-generated references in hook SQL still fail with P006.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Dependencies hidden inside macros, Python SQL and hooks were invisible to reviewers, scopes, selection and future renames, and hard-coded relation names broke under deferral.
Changes
__ref()/__source()/__seed()as typed arguments; every call-site reference is a dependency. References a macro emits or builds itself areP006.@hook(reads=...)and usectx.relation. SQL hooks resolve__ref/__source/__seedlike model SQL. Hook reads order the model after what they read; reading something built from the hooked model isP007.model()/source()/seed()independs_on; reads follow source deferral.P008at compile time and a warning at run time.[references] enforce_explicit = falsedisablesP006/P008for migrations.P004–P008and scope finding, also in--json.+x+follows declared data reads across SQL and Python nodes, from the selected roots.BREAKING CHANGE: macro-generated references, hard-coded relation names in Python SQL and hook reads of their own dependants fail compile, and
+/x+selectors include Python nodes that declare SQL reads.Verification
make testandmake check-cipass; hygiene scan clean.