Skip to content

Commit 21a802b

Browse files
authored
feat!: add typed SQL macros and require @py./@SQL. macro calls (#1224)
* feat!: add typed SQL macros and require @py./@SQL. macro calls Add SQL macros: logic-free SQL fragments in macros/sql/ with a MACRO header and typed arguments, in three forms decided by the body (an expression, a WITH list of CTEs, or a SELECT list of columns). Calls are written @sql.name(...), expand in place in the declaration pass, and are checked on their own and again in every caller. Parameters are written @name in bodies; types follow schema declaration types, with CONST for compile-time scalars, LIST/SET, ENUM, RELATION and COLUMN OF arguments. Runtime parameters in literal-only positions are rejected. Formatting, rules and column lineage see through calls, and macro-mode unit tests and __macro__ mocks work for SQL macros. BREAKING CHANGE: Python macros now live in macros/python/ (and the python/ folder of scoped macro roots) and are called as @py.name(...). A macro file directly under a macros/ folder fails with D013, and a bare @name(...) call fails with P014; move files into macros/python/ and replace @name( with @py.name( in project SQL. * fix: check SQL macro arguments against caller types and harden call placement Report RELATION columns, COLUMN OF names and typed arguments that do not fit at the call, with the parameter's declaration. Accept every comma spelling around CTE-list and column-list calls, reject aliases on column-list calls, resolve macro bodies in the macro file's own scope, keep valid SQL macros when another fails inventory, use real call spans, read backslash-escaped strings, parenthesise before AT, OVER and FILTER, keep separators in mocks, record hook-only SQL macro dependencies, and format expression bodies and MACRO headers. * refactor: resolve SQL macro bodies through native scope contexts Check cache-hit models' SQL macro arguments too, resolve macro body declarations through the native per-file declaration context, build related locations as native records, and keep the SQL macro helpers flat. * fix: skip SQL macro argument checks the probe cannot place Treat typed arguments in subqueries or set-operation scopes as unknown, resolve a FROM alias before a same-named CTE, keep DISTINCT ON calls rejected as before, and keep string literal lines when unwrapping formatted expression bodies. * perf: infer SQL macro argument types without full query analysis Merge a scope's typed arguments into one probe, infer probe output types directly instead of running full analysis per probe, check a wave's models together, and locate calls without re-scanning the whole file. * fix: keep macro kind folders when moves carry declaration layouts Apply group-folder layout rules within a macro's python/ or sql/ folder, migrate the layout and move tests to kind folders and @py. calls, and teach the agent skill SQL macros and kind-prefixed calls. * docs: say which SQL macro calls the argument check skips * refactor: keep SQL macro probe helpers to plain loops * fix: pass SQL macro arguments read through untyped sources Treat a probe as unknown when any FROM or JOIN source in it cannot be typed, so stars over unnest() or table functions no longer report missing columns. Teach kind-prefixed macro calls in the remaining docs. * test: import the compile-reuse rounding helper from macros/python * refactor: take the code SQL macro findings supersede from analysis The render-stage view filtered unknown-column findings inside macro calls by redeclaring B002, which the render code inventory reads as a code it emits. Name the analysis-owned constant instead. * perf: find @ with memchr when looking for bare macro calls The byte-by-byte scan cost about 4 instructions per byte of every authored SQL text, 0.15% of a 10,000-model compile without SQL macros. * test: regenerate goldens for kind-prefixed macro calls * test: move --since fixtures to macro kind folders and @py. calls * feat: compare SQL macros by their definitions for --since Fingerprint a SQL macro declaration by its form, typed arguments, result type and normalised body, ignoring layout, comments and its description, and keep the edges to the SQL macros it calls. * test: call semantic corpus and layered project macros with @py. * test: move the rename and move batch fixtures to macro kind folders
1 parent d46ee4b commit 21a802b

392 files changed

Lines changed: 16418 additions & 4691 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -167,15 +167,16 @@ SELECT
167167
o.order_id,
168168
o.customer_id,
169169
p.amount_cents AS total_cents,
170-
@cents_to_dollars("p.amount_cents") AS total_dollars
170+
@py.cents_to_dollars("p.amount_cents") AS total_dollars
171171
FROM __ref("stg_orders") o
172172
JOIN __ref("stg_payments") p USING (order_id)
173173
```
174174

175-
Macros are plain Python functions that return SQL, kept next to the models that use them:
175+
Python macros are plain functions that return SQL, kept next to the models that use them; SQL
176+
macros in `macros/sql/` cover logic-free fragments with typed, checked arguments:
176177

177178
```python
178-
# models/marts/_sqlbuild/_macros/currency.py
179+
# models/marts/_sqlbuild/_macros/python/currency.py
179180
def cents_to_dollars(column: str) -> str:
180181
"""Convert a cents integer column to a dollars decimal with two decimal places."""
181182
return f"ROUND(({column}) / 100.0, 2)"

‎benchmarks/dbt-compare/write_dbt_dense.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@
4343
COLUMN_RE = re.compile(
4444
r'^\s+(\w+) \(type (\w+), nullable (true|false), description "([^"]*)"(, audits \[not_null\])?\)'
4545
)
46-
MACRO_CALL_RE = re.compile(r'@(macro_\d{3}_\d{5})\("([^"]*)"\)')
46+
MACRO_CALL_RE = re.compile(r'@py\.(macro_\d{3}_\d{5})\("([^"]*)"\)')
4747
CAST_RE = re.compile(r"CAST\(([^ ]+) AS (\w+)\) AS (\w+)")
4848

4949

@@ -201,7 +201,7 @@ def write_dbt(sqb: Path, out: Path, n: int) -> Path:
201201
base_re = re.compile(
202202
r'def (base_offset_\d{3})\(expression: str\) -> str:\n return f"\(\{expression\} \+ 1\)"'
203203
)
204-
for macro in sorted(sqb.glob("models/*/intermediate/clean/_sqlbuild/macros/*/*.py")):
204+
for macro in sorted(sqb.glob("models/*/intermediate/clean/_sqlbuild/macros/python/*/*.py")):
205205
text = macro.read_text()
206206
name, ret = macro_def_re.search(text).groups()
207207
offset = offset_re.search(text)[1]

‎crates/sqlbuild-analysis/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ default = []
1313
test-support = []
1414

1515
[dependencies]
16+
memchr.workspace = true
1617
polyglot-sql.workspace = true
1718
polyglot-sql-function-catalogs.workspace = true
1819
rayon.workspace = true

‎crates/sqlbuild-analysis/src/assembly/analysis_session/_helpers/analysis_cache.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -551,6 +551,7 @@ pub(crate) fn decode_outcome(
551551
let binding_validated: bool = decoder.flag()?;
552552
decoder.finish()?;
553553
let outcome = ModelOutcome {
554+
sql_macro_findings: Vec::new(),
554555
analysis: ModelAnalysis {
555556
analysis_succeeded,
556557
columns,

‎crates/sqlbuild-analysis/src/assembly/analysis_session/_helpers/session.rs‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ use crate::assembly::analysis_session::_helpers::mappings::{
3030
};
3131
use crate::assembly::analysis_session::_helpers::mappings::{producers, waves};
3232
use crate::assembly::analysis_session::constants::{UNKNOWN_NULLABILITY, UNKNOWN_TYPE};
33+
use crate::assembly::analysis_session::main::resolved_cleaned_sql::resolved_cleaned_sql;
3334
use crate::assembly::analysis_session::models::{
3435
AnalysisSession, ColumnFact, ContractProof, DeferredCleaning, FinishedSession, LineageFacts,
3536
LineageRow, ModelAnalysis, ModelOutcome, ModelRequest, Phase, PivotOutcome, PivotTables,
@@ -38,6 +39,9 @@ use crate::assembly::analysis_session::models::{
3839
use crate::assembly::analysis_session::types::{Pairs, Shapes};
3940
use crate::semantic_validation::models::DiagnosticRow;
4041
use crate::semantic_validation::models::ProjectCatalog;
42+
use crate::sql_macros::main::check_sql_macro_arguments::check_sql_macro_arguments;
43+
use crate::sql_macros::models::SqlMacroFinding;
44+
use crate::sql_macros::types::ArgumentCheckInput;
4145

4246
/// What an internal native failure of the session names.
4347
const SESSION_CONTEXT: &str = "native model analysis";
@@ -279,13 +283,56 @@ impl AnalysisSession {
279283
}
280284
};
281285
self.outcomes[*model] = Some(ModelOutcome {
286+
sql_macro_findings: Vec::new(),
282287
analysis,
283288
cleaned_sql: result.cleaned_sql,
284289
deferred_cleaning: None,
285290
validated_schema: schema,
286291
fused_binding_validated: true,
287292
});
288293
}
294+
self.check_sql_macros(models)
295+
}
296+
297+
/// Each model's SQL macro call arguments checked against its analysed types, cache hits too.
298+
fn check_sql_macros(&mut self, models: &[usize]) -> Result<(), String> {
299+
let checked: Vec<usize> = models
300+
.iter()
301+
.copied()
302+
.filter(|model| {
303+
let request: &ModelRequest = &self.request.models[*model];
304+
!request.sql_macro_checks.is_empty() && request.placeholders.is_empty()
305+
})
306+
.collect();
307+
self.ensure_cleaned(&checked)?;
308+
let mut inputs: Vec<(usize, String, Shapes)> = Vec::with_capacity(checked.len());
309+
for model in checked {
310+
let Some(outcome) = self.outcomes[model].as_ref() else {
311+
continue;
312+
};
313+
let Ok(cleaned) = resolved_cleaned_sql(outcome) else {
314+
continue;
315+
};
316+
let schema: Shapes = self.binding_schema(&self.request.models[model]);
317+
inputs.push((model, cleaned.into_owned(), schema));
318+
}
319+
let batch: Vec<ArgumentCheckInput<'_>> = inputs
320+
.iter()
321+
.map(|(model, cleaned, schema)| {
322+
(
323+
cleaned.as_str(),
324+
schema.as_slice(),
325+
self.request.models[*model].sql_macro_checks.as_slice(),
326+
)
327+
})
328+
.collect();
329+
let findings: Vec<Vec<SqlMacroFinding>> =
330+
check_sql_macro_arguments(&self.request.dialect, &batch);
331+
for ((model, _, _), found) in inputs.iter().zip(findings) {
332+
if let Some(outcome) = self.outcomes[*model].as_mut() {
333+
outcome.sql_macro_findings = found;
334+
}
335+
}
289336
Ok(())
290337
}
291338

‎crates/sqlbuild-analysis/src/assembly/analysis_session/models.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ use crate::assembly::analysis_session::_helpers::catalog_state::SessionCatalog;
1212
use crate::assembly::analysis_session::_helpers::mappings::ShapeTable;
1313
use crate::assembly::analysis_session::types::{NullabilityRuleFn, OutputSources, Pairs, Shapes};
1414
use crate::semantic_validation::models::DiagnosticRow;
15+
use crate::sql_macros::models::{SqlMacroCheck, SqlMacroFinding};
1516

1617
/// One `ref`, `source`, `seed`, `table_fn` or `udf` call a model's SQL makes.
1718
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -42,6 +43,8 @@ pub struct ModelRequest {
4243
/// The SQL Python's dynamic pivot proof parses: placeholders replaced by their defaults.
4344
pub pivot_sql: String,
4445
pub dynamic_families: Vec<DynamicFamily>,
46+
/// What the model's analysed SQL must show for its SQL macro calls' arguments to fit.
47+
pub sql_macro_checks: Vec<SqlMacroCheck>,
4548
}
4649

4750
/// The reference-derived fields of a `ModelRequest`.
@@ -258,6 +261,8 @@ pub struct ModelOutcome {
258261
pub deferred_cleaning: Option<DeferredCleaning>,
259262
pub validated_schema: Shapes,
260263
pub fused_binding_validated: bool,
264+
/// The SQL macro call arguments the analysed SQL shows do not fit; never cached.
265+
pub sql_macro_findings: Vec<SqlMacroFinding>,
261266
}
262267

263268
/// The finished run of the session: what Python records from it.

‎crates/sqlbuild-analysis/src/assembly/analysis_session/tests/helpers.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,7 @@ pub(crate) fn model(name: &str, sql: &str, sources: &[&str], refs: &[&str]) -> M
108108
snapshot_columns: None,
109109
pivot_sql: sql.to_owned(),
110110
dynamic_families: Vec::new(),
111+
sql_macro_checks: Vec::new(),
111112
}
112113
}
113114

‎crates/sqlbuild-analysis/src/binding_locations/main/locate_binding_diagnostics.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,8 @@ pub fn locate_binding_diagnostics(
6767
line,
6868
column,
6969
location,
70+
help: None,
71+
related: None,
7072
});
7173
}
7274
Ok(located)

‎crates/sqlbuild-analysis/src/binding_locations/models.rs‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,4 +42,15 @@ pub struct LocatedDiagnostic {
4242
pub line: Option<usize>,
4343
pub column: Option<usize>,
4444
pub location: Option<AuthoredSpan>,
45+
pub help: Option<String>,
46+
pub related: Option<RelatedSpot>,
47+
}
48+
49+
/// A second authored location that adds context to a diagnostic.
50+
#[derive(Clone, Debug, PartialEq, Eq)]
51+
pub struct RelatedSpot {
52+
pub label: String,
53+
pub path: String,
54+
pub line: usize,
55+
pub column: usize,
4556
}

0 commit comments

Comments
 (0)