Skip to content

Commit d2b4ce2

Browse files
authored
fix: Snowflake, scenario, rule and type correctness batch (#1250)
* fix(snowflake): bind listed source columns as unquoted identifiers Planning rebinds models against the source columns it lists from the warehouse. The Snowflake adapter lists column names lowercased, and the rebinding stored them quoted ("bundle_sku"), so an ordinary unquoted reference, which Snowflake resolves as BUNDLE_SKU, failed with B002 unless the session set QUOTED_IDENTIFIERS_IGNORE_CASE. Plain single-case names are now stored unquoted on Snowflake, so they bind the way Snowflake resolves unquoted names. Mixed-case or special names stay quoted. BREAKING CHANGE: on Snowflake without QUOTED_IDENTIFIERS_IGNORE_CASE, a quoted lowercase reference such as "bundle_sku" to a listed source column now reports B002 during planning, as Snowflake would reject it against an upper-case column. The listing cannot tell a column really created with a quoted lowercase name apart, so such columns now bind as unquoted names. * fix(snowflake): parse loader JSON in the select list, not inside VALUES Row-returning managed-source loaders rendered VARIANT values as PARSE_JSON(...) inside the VALUES list, but Snowflake allows only constants there, so those loads failed. JSON values are now written as string constants and parsed with PARSE_JSON in the projection over the VALUES rows. Snowflake string literals treat backslashes as escapes, so the adapter now doubles them when quoting loader strings. Without that, JSON escapes such as \" and any backslash in a loaded string were altered. * fix(scenario): fingerprint local snapshots from the authored fixture SQL The local snapshot fingerprint hashed fixture SQL after the running adapter rendered it. Capture on Snowflake and replay on DuckDB render a partial fixture's wrapper alias in different case, and capture can add casts from warehouse column types that offline planning does not, so a snapshot was reported stale (X602) right after it was captured, and capturing again did not help. Fixture plans now carry the authored fixture SQL, and the fingerprint uses it, so capture and local replay agree. BREAKING CHANGE: local scenario snapshots captured by earlier versions report as stale once. Run `sqb scenario capture` to refresh them. * fix(scenario): stop treating lambda parameters as mock columns The fixture checker's column-read walk counted lambda parameters, such as i in FILTER(items, i -> ...) and acc and q in (acc, q) -> acc + q, as unqualified column reads. When one relation in scope was read through a star, those names resolved to it and the scenario failed with S510 asking the mock to supply them. Unqualified references to a lambda's own parameters are now skipped in projections and in clauses. Compile lineage and column-reference rewriting already scoped lambda parameters. DuckShop's cart funnel scenario drops its placeholder acc, i and q mock columns. * fix(scenario): treat contract-enforced models as authoritative fixture schemas A contract-enforced model whose query selected a star was treated as having a non-authoritative schema, so `__empty_fixture()` mocks of it were rejected (S510), and so were the `__empty_fixture()` fixtures `sqb format` writes for typed `WHERE FALSE` fixtures of contracted models. An enforced contract fixes the model's output columns, so its declared schema is now authoritative whatever the query looks like. * fix(rules): do not flag LIMIT 0 as nondeterministic row selection SQBRSQL004 flagged `LIMIT 0` without ORDER BY, though it selects no rows in any order. Empty fixtures written as `SELECT ... LIMIT 0` tripped it. * fix(test): cast bare NULL columns of set-operation fixtures to declared types A fixture's bare NULL columns are cast to the mocked relation's declared types, but only when the fixture was a single SELECT. A fixture written as a UNION ALL of rows kept untyped NULLs, so typed functions over them failed, for example YEAR(NULL) on Snowflake or DuckDB. Set operations now take their NULL columns from the first branch, which names the output. * fix(build): skip tests and end-of-build audits that read unbuilt resources When a model failed, an end-of-build audit that read it still ran and failed with "object does not exist". A SQL test that called a table function skipped because of the same failure also ran: failure blocking only reached selected models and functions, not tests in the execution order. Both are now skipped, like the audits attached to a skipped model. * fix(scenario): build cursor-intrinsic incremental models over a wide window A scenario rebuilds every model it runs from scratch, and a non-microbatch full rebuild has no cursor interval, so a scenario that ran an incremental model calling __cursor_start() or __cursor_end() failed. Such models now run over the same wide window SQL tests use (1900-01-01 to 2999-12-31, or the integer equivalent), so mocked rows are not filtered out. Microbatch models keep discovering their range from their inputs. * fix(audits): require a relation reference as the relationships target A built-in relationships audit written as `to "fct_orders"` compiled without complaint, but it queried a bare, unqualified table name and did not depend on the model, so it could run before that table existed, or read another one. The documentation showed that form. The `to` argument must now be a `__ref(...)`, `__source(...)`, `__seed(...)` or `__dbt_ref(...)` call, and compile says which reference to write. BREAKING CHANGE: a built-in `relationships` audit whose `to` is a plain or quoted relation name is now a compile error (P001). Write `to __ref("name")` (or `__source` / `__seed`) instead. * fix(build): do not reload managed sources when loading is off Without --select, a build's default selection included every managed source and its Python loader, so `sqb build --no-load` and `auto_load_sources = false` still ran the loaders. With loading off, the default selection now leaves managed sources and their loaders out. A source selected by name still loads. BREAKING CHANGE: `sqb build --no-load`, or a build with `auto_load_sources = false`, no longer runs managed-source loaders unless the source is selected explicitly. * fix(types): compare sized TEXT and STRING as VARCHAR A sized TEXT(n) or STRING(n) normalised as an unrelated type, so a contract declaring VARCHAR(n) rejected a column cast to TEXT(n). Sized variable-length text now normalises as VARCHAR(n) outside BigQuery, and BigQuery spells every variable-length text type STRING. The generated-type golden is re-recorded for the new spellings. * fix(headers): read a meta description as an ordinary string Header scanners and the tokenizer applied the prose quoting rules to any string after a description key, so a meta value such as meta (description "say \", then") ended at the escaped quote and the header failed to parse. Prose rules now apply only to description keys outside a meta value. * fix(rules): compare contract casts as the dialect does and point at the cast SQBRCONTRACT105 compared an outer cast's type with the declared contract type by spelling, so on Snowflake CAST(... AS VARCHAR(16777216)) or CAST(... AS STRING) was rejected against a VARCHAR contract. Both types are now normalized for the target dialect before comparing. A mismatched outer cast now names the cast's type and the declared type, points at the CAST itself, and asks to change the cast rather than wrap the expression in another one. Model rule locations were relative to the query below the header, so they pointed at header lines. They are now lines of the model file. * fix(format): lay out column and audit descriptions like the model's sqb format laid out only a header's own description: one line when it fits, otherwise an indented triple-quoted block. Column and audit descriptions now get the same layout at their own nesting depth. Formatting is idempotent and stored text is unchanged. * fix(rules): lint parameterised SQL tests instead of failing with L005 SQL rules expand a test file before linting it, and @PARAM("name") is not a macro, so every parameterised test failed with L005 while a SQBRSQL rule was selected. Test parameter references now stand as string sentinels for linting, as audit parameters already do. * fix(scenario): stop asking fixtures for CONNECT BY pseudo-columns The fixture checker read LEVEL, CONNECT_BY_ISLEAF and CONNECT_BY_ISCYCLE in a CONNECT BY query as unqualified input columns. When a relation in scope was read through a star, they resolved to it and the scenario failed with S510 asking the mock for them. They now read no input in a SELECT that has a CONNECT BY clause. * fix(test): let a macro mock name its value A macro mock had to be a bare SELECT of one string literal, which SQBRSQL022 rejects as a calculated projection with no stable name, so every macro mock needed a rule exception. The literal may now carry an AS alias, which is ignored. DuckShop's mock uses one and drops its exception. * fix(rules): accept a test file of several TEST blocks in SQBRTEST102 SQBRTEST102 asked a test file to be named after each of its blocks' behaviours, which no file holding two differently named blocks can satisfy. A file of several blocks now needs only the test_<subject>__<behavior>.sql grammar. DuckShop drops its exception. * fix(macros): check SQL macro argument types for every adapter's dialect SQL macro argument types were parsed only for four adapters, so a Postgres argument of type FOO BAR or a MotherDuck INTEGER NOT NULL compiled silently, and argument decimals were never range-checked. Argument types now go through the same per-dialect check as returns. BigQuery NUMERIC and BIGNUMERIC types beyond their integer-digit and scale limits, such as NUMERIC(40, 2), are now reported too. Adds an E2E for BigQuery STRING contracts filled by sized VARCHAR casts. BREAKING CHANGE: SQL macros whose argument types the adapter's dialect cannot parse or hold, and BigQuery NUMERIC or BIGNUMERIC types wider than BigQuery allows, now fail to compile with P015. * docs(duckshop): drop known limitations that are now fixed Removes the limitations this batch fixes: unquoted Snowflake column references after rebinding, JSON loader rows, managed sources reloaded with loading off, cursor-intrinsic incremental models and empty fixtures in scenarios, stale snapshots, untyped NULL source fixtures, tests and audits over skipped resources, and macro tests of folder-scoped macros. * fix(test): cast only set-operation fixture columns that are NULL in every branch Casting a set-operation fixture's bare NULL columns to the declared types also cast columns that a later branch already typed, changing that branch's values. Only columns that are a bare NULL in every branch are now cast; a typed value in any branch types the column as before. * test: update expectations for model rule lines, nested description layout and BigQuery STRING Model rule findings now carry file lines, column descriptions are laid out like the model's, BigQuery compares VARCHAR and STRING as one type, and builds that should run loaders opt in to loading sources. * test: run the new build and scenario E2E files in their DuckDB shards * refactor: share the description layout and first-select walks with their new siblings * fix(rules): stop exempting typed SQL macro calls from SQBRCONTRACT105 A select-list item that was one call to a macro with returns passed SQBRCONTRACT105 without a cast. returns is analysis-only and never reaches the generated SQL, so a body such as 'ACTIVE' created a VARCHAR(6) column under a VARCHAR contract with no error. A call now establishes the contract type only when the generated SQL casts it to that type: the macro body ends in CAST(... AS type) or ...::type, or the call is cast. Otherwise the rule names the macro and its returns type and asks for a cast in the macro or at the call. returns TYPE OF calls follow the same rule. BREAKING CHANGE: a contracted output that is one call to a SQL macro with returns, whose body is not cast to the contract type, now fails SQBRCONTRACT105 until the body or the call is cast. * fix(scenario): create local replay databases at the latest DuckDB storage version Local scenario replay created its DuckDB file at DuckDB's default storage version, v1.0.0, which cannot hold VARIANT columns, so a captured snapshot with a VARIANT column, such as a PARSE_JSON fixture on Snowflake, failed with X699. The local database is sqlbuild's own and recreated on every run, so it is now opened with storage_compatibility_version = latest. The DuckDB adapter passes that connection option through; existing database files keep their own storage version and still open. * fix(scenario): fingerprint the helper CTEs and references behind a fixture The local snapshot fingerprint hashed only a fixture's own authored text, so editing a helper CTE the fixture reads, such as changing helper_ids from SELECT 1 to SELECT 2, left a stale snapshot looking fresh. The fingerprint now also covers the authored body of every helper CTE the fixture reads, transitively, and whether each relation reference in them reads a scenario fixture or the warehouse. All of it is authored, adapter-neutral text, so a snapshot captured on one adapter stays fresh when replayed on another. BREAKING CHANGE: local scenario snapshots captured by earlier versions report as stale once. Run sqb scenario capture to refresh them. * fix(build): audit managed sources instead of dropping them when loading is off With loading off, the previous fix dropped managed sources from the default selection along with their loaders, so their audits stopped running and a build over bad source data passed. Loading off now means only that no loader runs: managed sources stay in the plan without a load step, and their audits check the data they already hold. This holds for --no-load and auto_load_sources = false, and now also with --select, where naming a managed source used to run its loader anyway. Audits of a selected source that loads nothing and that no model read during the build run at the end of the build. BREAKING CHANGE: with --no-load or auto_load_sources = false, no managed-source loader runs, even for a source selected by name. Selected managed sources are audited against their existing data, so a build over failing source data now fails. * fix(build): report end-of-build audits over unbuilt resources as skipped An end-of-build audit that read a resource the build did not build was dropped from the results without a trace. It is now recorded with a new skipped audit outcome and the reason, such as "reads broken_orders, which was not built", shown on its progress row, in the skipped rollup, in the JSON checks and in the SKIP count. Skipped audits are not written to audit result history. A model counted as unbuilt whenever it failed or was skipped, including a model a Python hook skipped on purpose, whose relation still holds earlier data. Only a model that was not built and has no existing relation now counts as unbuilt, so the end-of-build audits over a hook-skipped model run. * docs(loaders): note that Snowflake loader strings keep their backslashes Loader string values on Snowflake now have every backslash escaped, so values are stored exactly as returned. Documents the effect on keys loaded by earlier releases. BREAKING CHANGE: Snowflake loader string values containing backslashes are now stored literally. Earlier releases wrote them unescaped, so merge, delete_insert or append deduplication keys containing backslashes will not match rows those releases loaded. * fix(duckdb): keep the replay storage version out of user connection config The DuckDB adapter read storage_compatibility_version from the connection config, which made it a user setting for project warehouses. The latest storage version now applies only to the local scenario replay database, through an internal adapter method, and a user config naming the key ignores it like any other unknown key. * fix(build): audit unread sources only with loading off or when selected End-of-build source audits ran for every source no model read, on every build, so a plain build over an unmanaged source with a missing table failed. They now run only when loading is off (--no-load or auto_load_sources = false), or for a source selected by name. With loading on and no such selection, a source is audited only before a model that reads it, as before. An unbuilt model is now one that was not built and whose relation the inspected warehouse does not hold, rather than one planned as a first run, so a full refresh on a fresh warehouse skips the end audits over a failed model. * fix(dagster): emit no check result for an audit that did not run An end-of-build audit skipped because it reads a resource the build did not build has passed: false in the JSON checks, so Dagster reported it as a failed asset check, even in a green build. The Dagster integration now emits no check result for a skipped audit, which Dagster shows as not evaluated, and logs the audit and its skip reason instead. The JSON shape is unchanged.
1 parent 9a7d605 commit d2b4ce2

149 files changed

Lines changed: 4106 additions & 642 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,10 @@ E2E_DUCKDB_BUILD_CORE_PATHS := \
110110
tests/e2e/src/sqlbuild/cli/commands/main/build/test_target_concurrency.py \
111111
tests/e2e/src/sqlbuild/cli/commands/main/build/test_template_expressions.py \
112112
tests/e2e/src/sqlbuild/cli/commands/main/build/test_terminal_failure.py \
113+
tests/e2e/src/sqlbuild/cli/commands/main/build/test_no_load_sources.py \
114+
tests/e2e/src/sqlbuild/cli/commands/main/build/test_duckdb_storage_key.py \
115+
tests/e2e/src/sqlbuild/cli/commands/main/build/test_source_end_audits.py \
116+
tests/e2e/src/sqlbuild/cli/commands/main/build/test_unbuilt_dependencies.py \
113117
tests/e2e/src/sqlbuild/cli/commands/main/build/test_validation_failures.py
114118

115119
E2E_DUCKDB_BUILD_INCREMENTAL_PATHS := \
@@ -147,6 +151,9 @@ E2E_DUCKDB_CLI_DATA_PATHS := \
147151
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_empty_fixture.py \
148152
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_expected_columns.py \
149153
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_helper_ctes.py \
154+
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_fixture_columns.py \
155+
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_local_variant.py \
156+
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_snapshot_staleness.py \
150157
tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_unmocked_inputs.py
151158

152159
E2E_DUCKDB_CLI_PATHS := \

‎benchmarks/duckshop/README.md‎

Lines changed: 2 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,8 @@ failures are the baseline that better translation should remove:
129129
The other three scenarios (`campaign_roas__attributes_orders_to_channel_spend`,
130130
`carrier_performance__weekly_rates_from_shipment_facts` and
131131
`payments__fully_refunded_orders_collect_nothing`) never reach translation: their snapshots are
132-
reported stale straight after capture, a SQLBuild issue described under known limitations.
132+
reported stale straight after capture. The snapshot fingerprint now hashes the authored fixture
133+
SQL, so these results predate that fix; recapture the snapshots to replay them.
133134

134135
## Expected divergences
135136

@@ -148,53 +149,15 @@ them as expected differences or refusals instead of errors:
148149
These SQLBuild limitations shape parts of the project; each workaround is a reviewed exception or
149150
noted here:
150151

151-
- Fixture checking asks for `CONNECT BY` pseudo-columns and lambda parameters as mock columns, so
152-
`int_category_paths` and `int_cart_funnel` have no unit test. Scenarios cover both, and the
153-
category scenario builds the hierarchy from a source fixture for the same reason.
154-
- Parameterised tests (`parameters` and `cases`) fail with `L005` while `SQBRSQL` is selected, so
155-
repeated cases use multi-row fixtures or `VALUES` case tables instead.
156-
- Macro tests can only reach project-wide macros, so folder-scoped staging macros such as
157-
`cdc_latest` are covered through the models that call them.
158-
- Scenarios cannot build incremental models that use cursor intrinsics, so scenarios that read
159-
them provide the incremental model as a fixture.
160-
- `sqb format` rewrites a typed `WHERE FALSE` fixture into `__empty_fixture()` even where the
161-
relation's schema is not authoritative, so the empty fixtures here use `WHERE 1 = 0`.
162-
- A macro mock must be an unaliased `SELECT` of one string literal, which `SQBRSQL022` rejects.
163-
- A file with several `TEST` blocks cannot satisfy `SQBRTEST102`, which names the file after each
164-
block.
165-
- Loaders that return rows write JSON values as `PARSE_JSON(...)` inside a `VALUES` list, which
166-
Snowflake rejects because `VALUES` accepts only constants. The DuckShop loaders therefore write
167-
their own tables, parsing JSON in the select list of each `INSERT` (`python/loaders/_writer.py`).
168-
- Once a source table exists, Snowflake planning rebinds models to the table's inspected columns.
169-
SQLBuild reads those names in lower case and then quotes them, so they only match quoted
170-
lower-case references, and every unquoted column reference fails with `B002`. The connection
171-
therefore sets `QUOTED_IDENTIFIERS_IGNORE_CASE = true` in `session_parameters`, which the binder
172-
honours. DuckShop quotes no identifier whose case matters, so the setting changes no results.
173152
- Snowflake does not support concurrent microbatch `delete_insert` batches (`P001`), so
174153
`int_web_sessions` runs its batches one at a time.
175-
- `sqb build --no-load` and `auto_load_sources = false` still load managed sources when the build
176-
has no `--select`, because the default selection includes the sources.
177154
- Self-managed loaders always report `rows=0`, since SQLBuild does not see the rows they insert.
178155
- Compilation accepts SQL that Snowflake rejects: `CONNECT_BY_ISLEAF`, which Snowflake does not
179156
have, and a CTE whose body is a bare `VALUES` list. `int_category_paths` finds leaf categories
180157
with a join instead, and test case tables use `SELECT ... FROM VALUES ... AS v(...)`.
181158
- `SQBRSQL018` can only take proof from single-column keys. Append-only feeds redeliver rows, so
182159
their id is unique only together with `_loaded_at` (the `unique_per_delivery` audit), and the
183160
staging models that keep the latest delivery carry reviewed exceptions.
184-
- Source fixtures are not cast to the source's declared types, so a typed function over an
185-
untyped `NULL` fixture column (such as `YEAR(NULL)`) fails on Snowflake; such fixtures cast it.
186-
- The scenario fixture checker treats lambda parameters (`i -> ...`, `(acc, q) -> ...`) as columns
187-
the model reads, so the `cart_funnel` scenario mocks carry placeholder `acc`, `i` and `q` columns.
188-
- A `relationships` audit with a quoted relation name, `to "fct_orders"`, compiles cleanly but
189-
queries an unqualified table and does not wait for it; the relation must be `to __ref(...)`.
190-
- A local scenario snapshot is reported stale (`X602`) straight after capture when a fixture has a
191-
bare `NULL` column, which SQLBuild wraps in a typed `__sqlbuild_partial_fixture` subquery. The
192-
snapshot fingerprint hashes that adapter-rendered SQL: capture renders the alias in Snowflake's
193-
upper case and local replay in DuckDB's lower case, and capture can also cast columns whose types
194-
it only knows from the warehouse. The fingerprint can never match, so the snapshot cannot be
195-
replayed. Three of the eight scenarios are affected.
196-
- A unit test of a table function, and an end-of-build audit, still run when the function or the
197-
relation they read was skipped, and then fail with a missing-object error.
198161

199162
## Compile
200163

‎benchmarks/duckshop/project/sqlbuild_project.toml‎

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -235,18 +235,3 @@ reason = """
235235
The KPI grid is every reporting day for every region and metric pair that has ever had a value, so
236236
days without activity report zero instead of disappearing. Both sides are small and bounded.
237237
"""
238-
239-
[[rules.rule_exceptions]]
240-
rule = "SQBRSQL022"
241-
path = "tests/unit/staging/storefront/test_stg_wishlist_items__uses_the_mocked_sku_normaliser.sql"
242-
reason = """
243-
A macro mock must be a bare SELECT of one string literal, so its projection cannot carry an alias.
244-
"""
245-
246-
[[rules.rule_exceptions]]
247-
rule = "SQBRTEST102"
248-
path = "tests/unit/marts/catalogue/test_rpt_wishlist_conversion__edge_cases.sql"
249-
reason = """
250-
This file keeps two related edge cases for one report together as separate TEST blocks. Each block
251-
has its own name, so the file cannot be named after both.
252-
"""

‎benchmarks/duckshop/project/tests/scenarios/clickstream/cart_funnel__assigns_furthest_stage_per_session.sql‎

Lines changed: 7 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,7 @@ WITH __ref__int_web_sessions AS (
2222
NULL AS landing_page_url,
2323
FALSE AS is_bounce,
2424
TRUE AS has_add_to_cart,
25-
FALSE AS has_order,
26-
-- Not real columns: the fixture checker asks for int_cart_funnel's lambda parameters
27-
-- (i, q, acc) as mock columns. Remove these once that is fixed.
28-
NULL AS acc,
29-
NULL AS i,
30-
NULL AS q
25+
FALSE AS has_order
3126
UNION ALL
3227
SELECT
3328
'S2',
@@ -46,10 +41,7 @@ WITH __ref__int_web_sessions AS (
4641
NULL,
4742
FALSE,
4843
FALSE,
49-
TRUE,
50-
NULL,
51-
NULL,
52-
NULL
44+
TRUE
5345
UNION ALL
5446
SELECT
5547
'S3',
@@ -68,10 +60,7 @@ WITH __ref__int_web_sessions AS (
6860
NULL,
6961
FALSE,
7062
FALSE,
71-
FALSE,
72-
NULL,
73-
NULL,
74-
NULL
63+
FALSE
7564
),
7665

7766
__ref__stg_web_events AS (
@@ -103,9 +92,7 @@ __ref__stg_web_events AS (
10392
OBJECT_CONSTRUCT('sku', 'DUCK-2', 'price_minor', 900),
10493
OBJECT_CONSTRUCT('quantity', 1, 'price_minor', 100),
10594
OBJECT_CONSTRUCT('sku', 'duck-3', 'quantity', 0)
106-
)::ARRAY AS items,
107-
-- See the note on int_web_sessions: i is a lambda parameter, not a column.
108-
NULL AS i
95+
)::ARRAY AS items
10996
UNION ALL
11097
SELECT
11198
'e2',
@@ -125,8 +112,7 @@ __ref__stg_web_events AS (
125112
NULL,
126113
ARRAY_CONSTRUCT(
127114
OBJECT_CONSTRUCT('sku', 'duck-1', 'quantity', 1)
128-
)::ARRAY,
129-
NULL
115+
)::ARRAY
130116
UNION ALL
131117
SELECT
132118
'e3',
@@ -153,8 +139,7 @@ __ref__stg_web_events AS (
153139
'price_minor',
154140
500
155141
)
156-
)::ARRAY,
157-
NULL
142+
)::ARRAY
158143
UNION ALL
159144
SELECT
160145
'e4',
@@ -174,8 +159,7 @@ __ref__stg_web_events AS (
174159
NULL,
175160
ARRAY_CONSTRUCT(
176161
OBJECT_CONSTRUCT('sku', 'DUCK-9', 'price_minor', 9900)
177-
)::ARRAY,
178-
NULL
162+
)::ARRAY
179163
),
180164

181165
__ref__stg_web_checkout_steps AS (

‎benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlist_items__uses_the_mocked_sku_normaliser.sql‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
TEST (name "stg_wishlist_items__uses_the_mocked_sku_normaliser");
22

33
WITH __macro__normalise_sku AS (
4-
SELECT '''MOCKED-SKU'''
4+
SELECT '''MOCKED-SKU''' AS normalised_sku
55
),
66

77
__ref__stg_wishlists AS (

‎crates/sqlbuild-analysis/src/assembly/analysis_session/_helpers/dict_walk.rs‎

Lines changed: 78 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,9 @@ use crate::assembly::analysis_session::constants::{
1212
COLUMN_AST_KIND, CTE_ALIAS_PAYLOAD_KEY, CTE_LIST_PAYLOAD_KEY, DEPENDENCY_FUNCTION_NAMES,
1313
FUNCTION_AST_KIND, FUNCTION_FIXTURE_TYPES, FUNCTION_PAYLOAD_KIND, HELPER_READS_DIALECT,
1414
LITERAL_FIXTURE_TYPES, LITERAL_PAYLOAD_KIND, MAX_FUNCTION_CALL_DEPTH, NAME_PAYLOAD_KEY,
15-
NULL_PAYLOAD_KIND, RENDERED_TYPE_NAMES, SCHEMA_TABLE_ARG, SELECT_PAYLOAD_KEY, STAR_AST_KIND,
16-
STRUCTURED_PAYLOAD_KINDS, TABLE_AST_KIND, VARIANT_FIXTURE_TYPE, WILDCARD,
15+
NULL_PAYLOAD_KIND, RENDERED_TYPE_NAMES, SCHEMA_TABLE_ARG, SELECT_PAYLOAD_KEY,
16+
SET_OPERATION_PAYLOAD_KEYS, STAR_AST_KIND, STRUCTURED_PAYLOAD_KINDS, TABLE_AST_KIND,
17+
VARIANT_FIXTURE_TYPE, WILDCARD,
1718
};
1819

1920
/// Python's `bool(value)`.
@@ -452,34 +453,101 @@ pub(crate) fn fixture_projection_facts(
452453
return Ok(ProjectionFacts::default());
453454
};
454455
let mut facts: ProjectionFacts = collected_facts(select);
456+
let top: Option<&Map<String, Value>> = payload.as_object();
455457
let direct: bool = matches!(
456-
payload
457-
.as_object()
458-
.and_then(|map| map.get(SELECT_PAYLOAD_KEY)),
458+
top.and_then(|map| map.get(SELECT_PAYLOAD_KEY)),
459459
Some(Value::Object(_))
460460
);
461-
if !direct {
461+
let set_operation: bool = top.is_some_and(|map| {
462+
SET_OPERATION_PAYLOAD_KEYS
463+
.iter()
464+
.any(|key| matches!(map.get(*key), Some(Value::Object(_))))
465+
});
466+
if !direct && !set_operation {
462467
facts.null_literal_names.clear();
463468
}
469+
if set_operation {
470+
let branches: Vec<&Map<String, Value>> = branch_selects(&payload);
471+
let null_in_every_branch = |index: usize| {
472+
branches
473+
.iter()
474+
.all(|branch| null_positions(branch).get(index) == Some(&true))
475+
};
476+
let kept: Vec<String> = null_names_by_position(select)
477+
.into_iter()
478+
.filter(|(index, _)| null_in_every_branch(*index))
479+
.map(|(_, name)| name)
480+
.collect();
481+
facts.null_literal_names.retain(|name| kept.contains(name));
482+
}
464483
Ok(facts)
465484
}
466485

467-
/// Python's recursive `_first_select`, depth first in payload order, without recursion.
468-
fn first_select(payload: &Value) -> Option<&Map<String, Value>> {
486+
/// Every branch `SELECT` of a set operation, without the selects nested inside them.
487+
fn branch_selects(payload: &Value) -> Vec<&Map<String, Value>> {
488+
let mut branches: Vec<&Map<String, Value>> = Vec::new();
469489
let mut pending: Vec<&Value> = vec![payload];
470490
while let Some(node) = pending.pop() {
471491
match node {
472492
Value::Object(map) => {
473493
if let Some(Value::Object(select)) = map.get(SELECT_PAYLOAD_KEY) {
474-
return Some(select);
494+
branches.push(select);
495+
continue;
475496
}
476497
pending.extend(map.values().rev());
477498
}
478499
Value::Array(items) => pending.extend(items.iter().rev()),
479500
_ => {}
480501
}
481502
}
482-
None
503+
branches
504+
}
505+
506+
/// The projection payloads of a select, each with any alias removed.
507+
fn projected_values(select: &Map<String, Value>) -> Vec<&Map<String, Value>> {
508+
select
509+
.get("expressions")
510+
.and_then(Value::as_array)
511+
.map_or(&[][..], Vec::as_slice)
512+
.iter()
513+
.filter_map(|expression| {
514+
match expression.get("alias").and_then(|alias| alias.get("this")) {
515+
Some(Value::Object(value)) => Some(value),
516+
_ => expression.as_object(),
517+
}
518+
})
519+
.collect()
520+
}
521+
522+
/// Whether each projection of a select is a bare `NULL`.
523+
fn null_positions(select: &Map<String, Value>) -> Vec<bool> {
524+
projected_values(select)
525+
.into_iter()
526+
.map(|value| value.keys().next().map(String::as_str) == Some(NULL_PAYLOAD_KIND))
527+
.collect()
528+
}
529+
530+
/// The folded alias of each bare `NULL` projection of a select, with its position.
531+
fn null_names_by_position(select: &Map<String, Value>) -> Vec<(usize, String)> {
532+
let expressions: &[Value] = select
533+
.get("expressions")
534+
.and_then(Value::as_array)
535+
.map_or(&[], Vec::as_slice);
536+
expressions
537+
.iter()
538+
.enumerate()
539+
.filter_map(|(index, expression)| {
540+
let alias_payload = expression.get("alias")?;
541+
let name: &str = alias_payload.get("alias")?.get("name")?.as_str()?;
542+
let kind: &str = alias_payload.get("this")?.as_object()?.keys().next()?;
543+
(kind == NULL_PAYLOAD_KIND).then(|| (index, str::to_lowercase(name)))
544+
})
545+
.collect()
546+
}
547+
548+
/// Python's recursive `_first_select`, depth first in payload order, without recursion.
549+
fn first_select(payload: &Value) -> Option<&Map<String, Value>> {
550+
branch_selects(payload).into_iter().next()
483551
}
484552

485553
fn collected_facts(select: &Map<String, Value>) -> ProjectionFacts {

‎crates/sqlbuild-analysis/src/assembly/analysis_session/constants.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,8 @@ pub(crate) const ARRAY_PAYLOAD_KIND: &str = "array_func";
212212
pub(crate) const STRUCTURED_PAYLOAD_KINDS: [&str; 2] = ["parse_json", "struct"];
213213
pub(crate) const FUNCTION_PAYLOAD_KIND: &str = "function";
214214
pub(crate) const SELECT_PAYLOAD_KEY: &str = "select";
215+
/// Set operations whose output columns are named and typed by their first branch.
216+
pub(crate) const SET_OPERATION_PAYLOAD_KEYS: [&str; 3] = ["union", "intersect", "except"];
215217
/// `{literal_type: type}` Python's `_simple_expression_type` maps a literal to.
216218
pub(crate) const LITERAL_FIXTURE_TYPES: [(&str, &str); 3] = [
217219
("string", "VARCHAR"),

‎crates/sqlbuild-analysis/src/compiler/_helpers/sql_tests/extraction.rs‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -683,6 +683,14 @@ fn macro_mock_value(
683683
}
684684
let (value, end) = read_string(body, index)?;
685685
index = skip_ignorable(body, end, syntax)?;
686+
if let Some(alias_start) = consume_keyword(body, index, "AS") {
687+
let alias: usize = skip_ignorable(body, alias_start, syntax)?;
688+
let alias_end: usize = identifier_end(body, alias);
689+
if alias_end == alias {
690+
return Err(macro_mock_shape(file, &cte.0, true));
691+
}
692+
index = skip_ignorable(body, alias_end, syntax)?;
693+
}
686694
if byte_at(body, index) == Some(b';') {
687695
index = skip_ignorable(body, index + 1, syntax)?;
688696
}
@@ -692,6 +700,20 @@ fn macro_mock_value(
692700
Ok(value)
693701
}
694702

703+
/// The end of the plain or double-quoted identifier at `start`, or `start` when there is none.
704+
fn identifier_end(sql: &str, start: usize) -> usize {
705+
if byte_at(sql, start) == Some(b'"') {
706+
return sql[start + 1..]
707+
.find('"')
708+
.map_or(start, |close| start + close + 2);
709+
}
710+
let mut end: usize = start;
711+
while char_at(sql, end).is_some_and(is_identifier_continue) {
712+
end += char_len(sql, end);
713+
}
714+
end
715+
}
716+
695717
fn macro_mock_shape(file: &str, name: &str, trailing: bool) -> String {
696718
if trailing {
697719
format!(

‎crates/sqlbuild-analysis/src/planner_scope/main/source_selectors.rs‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,12 @@ use crate::graph::models::ParsedSelector;
77
use crate::planner_scope::constants::{NAME, SOURCE};
88
use std::collections::HashSet;
99

10-
/// Find selected sources explicitly named by upstream source or bare-name selectors.
10+
/// Selected sources named by a `name` or `source` selector, only `+`-prefixed ones if `upstream_only`.
1111
pub fn source_selectors(
1212
select: &[String],
1313
names: &[(String, ObjectKey)],
1414
selected: &[ObjectKey],
15+
upstream_only: bool,
1516
) -> Result<HashSet<ObjectKey>, SelectorError> {
1617
let mut keys: HashSet<ObjectKey> = HashSet::new();
1718
for part in select
@@ -22,12 +23,15 @@ pub fn source_selectors(
2223
let ParsedSelector::Kind {
2324
kind,
2425
value,
25-
upstream: true,
26+
upstream,
2627
..
2728
} = parse_selector(part)?
2829
else {
2930
continue;
3031
};
32+
if upstream_only && !upstream {
33+
continue;
34+
}
3135
if kind != NAME && kind != SOURCE {
3236
continue;
3337
}

0 commit comments

Comments
 (0)