diff --git a/Makefile b/Makefile index 54dfc049b2..b9bf0cd60c 100644 --- a/Makefile +++ b/Makefile @@ -110,6 +110,10 @@ E2E_DUCKDB_BUILD_CORE_PATHS := \ tests/e2e/src/sqlbuild/cli/commands/main/build/test_target_concurrency.py \ tests/e2e/src/sqlbuild/cli/commands/main/build/test_template_expressions.py \ tests/e2e/src/sqlbuild/cli/commands/main/build/test_terminal_failure.py \ + tests/e2e/src/sqlbuild/cli/commands/main/build/test_no_load_sources.py \ + tests/e2e/src/sqlbuild/cli/commands/main/build/test_duckdb_storage_key.py \ + tests/e2e/src/sqlbuild/cli/commands/main/build/test_source_end_audits.py \ + tests/e2e/src/sqlbuild/cli/commands/main/build/test_unbuilt_dependencies.py \ tests/e2e/src/sqlbuild/cli/commands/main/build/test_validation_failures.py E2E_DUCKDB_BUILD_INCREMENTAL_PATHS := \ @@ -147,6 +151,9 @@ E2E_DUCKDB_CLI_DATA_PATHS := \ tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_empty_fixture.py \ tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_expected_columns.py \ tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_helper_ctes.py \ + tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_fixture_columns.py \ + tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_local_variant.py \ + tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_snapshot_staleness.py \ tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_unmocked_inputs.py E2E_DUCKDB_CLI_PATHS := \ diff --git a/benchmarks/duckshop/README.md b/benchmarks/duckshop/README.md index a932874863..9d1705c3b0 100644 --- a/benchmarks/duckshop/README.md +++ b/benchmarks/duckshop/README.md @@ -129,7 +129,8 @@ failures are the baseline that better translation should remove: The other three scenarios (`campaign_roas__attributes_orders_to_channel_spend`, `carrier_performance__weekly_rates_from_shipment_facts` and `payments__fully_refunded_orders_collect_nothing`) never reach translation: their snapshots are -reported stale straight after capture, a SQLBuild issue described under known limitations. +reported stale straight after capture. The snapshot fingerprint now hashes the authored fixture +SQL, so these results predate that fix; recapture the snapshots to replay them. ## Expected divergences @@ -148,32 +149,8 @@ them as expected differences or refusals instead of errors: These SQLBuild limitations shape parts of the project; each workaround is a reviewed exception or noted here: -- Fixture checking asks for `CONNECT BY` pseudo-columns and lambda parameters as mock columns, so - `int_category_paths` and `int_cart_funnel` have no unit test. Scenarios cover both, and the - category scenario builds the hierarchy from a source fixture for the same reason. -- Parameterised tests (`parameters` and `cases`) fail with `L005` while `SQBRSQL` is selected, so - repeated cases use multi-row fixtures or `VALUES` case tables instead. -- Macro tests can only reach project-wide macros, so folder-scoped staging macros such as - `cdc_latest` are covered through the models that call them. -- Scenarios cannot build incremental models that use cursor intrinsics, so scenarios that read - them provide the incremental model as a fixture. -- `sqb format` rewrites a typed `WHERE FALSE` fixture into `__empty_fixture()` even where the - relation's schema is not authoritative, so the empty fixtures here use `WHERE 1 = 0`. -- A macro mock must be an unaliased `SELECT` of one string literal, which `SQBRSQL022` rejects. -- A file with several `TEST` blocks cannot satisfy `SQBRTEST102`, which names the file after each - block. -- Loaders that return rows write JSON values as `PARSE_JSON(...)` inside a `VALUES` list, which - Snowflake rejects because `VALUES` accepts only constants. The DuckShop loaders therefore write - their own tables, parsing JSON in the select list of each `INSERT` (`python/loaders/_writer.py`). -- Once a source table exists, Snowflake planning rebinds models to the table's inspected columns. - SQLBuild reads those names in lower case and then quotes them, so they only match quoted - lower-case references, and every unquoted column reference fails with `B002`. The connection - therefore sets `QUOTED_IDENTIFIERS_IGNORE_CASE = true` in `session_parameters`, which the binder - honours. DuckShop quotes no identifier whose case matters, so the setting changes no results. - Snowflake does not support concurrent microbatch `delete_insert` batches (`P001`), so `int_web_sessions` runs its batches one at a time. -- `sqb build --no-load` and `auto_load_sources = false` still load managed sources when the build - has no `--select`, because the default selection includes the sources. - Self-managed loaders always report `rows=0`, since SQLBuild does not see the rows they insert. - Compilation accepts SQL that Snowflake rejects: `CONNECT_BY_ISLEAF`, which Snowflake does not have, and a CTE whose body is a bare `VALUES` list. `int_category_paths` finds leaf categories @@ -181,20 +158,6 @@ noted here: - `SQBRSQL018` can only take proof from single-column keys. Append-only feeds redeliver rows, so their id is unique only together with `_loaded_at` (the `unique_per_delivery` audit), and the staging models that keep the latest delivery carry reviewed exceptions. -- Source fixtures are not cast to the source's declared types, so a typed function over an - untyped `NULL` fixture column (such as `YEAR(NULL)`) fails on Snowflake; such fixtures cast it. -- The scenario fixture checker treats lambda parameters (`i -> ...`, `(acc, q) -> ...`) as columns - the model reads, so the `cart_funnel` scenario mocks carry placeholder `acc`, `i` and `q` columns. -- A `relationships` audit with a quoted relation name, `to "fct_orders"`, compiles cleanly but - queries an unqualified table and does not wait for it; the relation must be `to __ref(...)`. -- A local scenario snapshot is reported stale (`X602`) straight after capture when a fixture has a - bare `NULL` column, which SQLBuild wraps in a typed `__sqlbuild_partial_fixture` subquery. The - snapshot fingerprint hashes that adapter-rendered SQL: capture renders the alias in Snowflake's - upper case and local replay in DuckDB's lower case, and capture can also cast columns whose types - it only knows from the warehouse. The fingerprint can never match, so the snapshot cannot be - replayed. Three of the eight scenarios are affected. -- A unit test of a table function, and an end-of-build audit, still run when the function or the - relation they read was skipped, and then fail with a missing-object error. ## Compile diff --git a/benchmarks/duckshop/project/sqlbuild_project.toml b/benchmarks/duckshop/project/sqlbuild_project.toml index 70b314918e..2c18525d70 100644 --- a/benchmarks/duckshop/project/sqlbuild_project.toml +++ b/benchmarks/duckshop/project/sqlbuild_project.toml @@ -235,18 +235,3 @@ reason = """ The KPI grid is every reporting day for every region and metric pair that has ever had a value, so days without activity report zero instead of disappearing. Both sides are small and bounded. """ - -[[rules.rule_exceptions]] -rule = "SQBRSQL022" -path = "tests/unit/staging/storefront/test_stg_wishlist_items__uses_the_mocked_sku_normaliser.sql" -reason = """ -A macro mock must be a bare SELECT of one string literal, so its projection cannot carry an alias. -""" - -[[rules.rule_exceptions]] -rule = "SQBRTEST102" -path = "tests/unit/marts/catalogue/test_rpt_wishlist_conversion__edge_cases.sql" -reason = """ -This file keeps two related edge cases for one report together as separate TEST blocks. Each block -has its own name, so the file cannot be named after both. -""" diff --git a/benchmarks/duckshop/project/tests/scenarios/clickstream/cart_funnel__assigns_furthest_stage_per_session.sql b/benchmarks/duckshop/project/tests/scenarios/clickstream/cart_funnel__assigns_furthest_stage_per_session.sql index 03439d4ab1..6780a8144c 100644 --- a/benchmarks/duckshop/project/tests/scenarios/clickstream/cart_funnel__assigns_furthest_stage_per_session.sql +++ b/benchmarks/duckshop/project/tests/scenarios/clickstream/cart_funnel__assigns_furthest_stage_per_session.sql @@ -22,12 +22,7 @@ WITH __ref__int_web_sessions AS ( NULL AS landing_page_url, FALSE AS is_bounce, TRUE AS has_add_to_cart, - FALSE AS has_order, - -- Not real columns: the fixture checker asks for int_cart_funnel's lambda parameters - -- (i, q, acc) as mock columns. Remove these once that is fixed. - NULL AS acc, - NULL AS i, - NULL AS q + FALSE AS has_order UNION ALL SELECT 'S2', @@ -46,10 +41,7 @@ WITH __ref__int_web_sessions AS ( NULL, FALSE, FALSE, - TRUE, - NULL, - NULL, - NULL + TRUE UNION ALL SELECT 'S3', @@ -68,10 +60,7 @@ WITH __ref__int_web_sessions AS ( NULL, FALSE, FALSE, - FALSE, - NULL, - NULL, - NULL + FALSE ), __ref__stg_web_events AS ( @@ -103,9 +92,7 @@ __ref__stg_web_events AS ( OBJECT_CONSTRUCT('sku', 'DUCK-2', 'price_minor', 900), OBJECT_CONSTRUCT('quantity', 1, 'price_minor', 100), OBJECT_CONSTRUCT('sku', 'duck-3', 'quantity', 0) - )::ARRAY AS items, - -- See the note on int_web_sessions: i is a lambda parameter, not a column. - NULL AS i + )::ARRAY AS items UNION ALL SELECT 'e2', @@ -125,8 +112,7 @@ __ref__stg_web_events AS ( NULL, ARRAY_CONSTRUCT( OBJECT_CONSTRUCT('sku', 'duck-1', 'quantity', 1) - )::ARRAY, - NULL + )::ARRAY UNION ALL SELECT 'e3', @@ -153,8 +139,7 @@ __ref__stg_web_events AS ( 'price_minor', 500 ) - )::ARRAY, - NULL + )::ARRAY UNION ALL SELECT 'e4', @@ -174,8 +159,7 @@ __ref__stg_web_events AS ( NULL, ARRAY_CONSTRUCT( OBJECT_CONSTRUCT('sku', 'DUCK-9', 'price_minor', 9900) - )::ARRAY, - NULL + )::ARRAY ), __ref__stg_web_checkout_steps AS ( diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlist_items__uses_the_mocked_sku_normaliser.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlist_items__uses_the_mocked_sku_normaliser.sql index ff09a45546..1354cc2b4b 100644 --- a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlist_items__uses_the_mocked_sku_normaliser.sql +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlist_items__uses_the_mocked_sku_normaliser.sql @@ -1,7 +1,7 @@ TEST (name "stg_wishlist_items__uses_the_mocked_sku_normaliser"); WITH __macro__normalise_sku AS ( - SELECT '''MOCKED-SKU''' + SELECT '''MOCKED-SKU''' AS normalised_sku ), __ref__stg_wishlists AS ( diff --git a/crates/sqlbuild-analysis/src/assembly/analysis_session/_helpers/dict_walk.rs b/crates/sqlbuild-analysis/src/assembly/analysis_session/_helpers/dict_walk.rs index bbdf207c06..3daa93527a 100644 --- a/crates/sqlbuild-analysis/src/assembly/analysis_session/_helpers/dict_walk.rs +++ b/crates/sqlbuild-analysis/src/assembly/analysis_session/_helpers/dict_walk.rs @@ -12,8 +12,9 @@ use crate::assembly::analysis_session::constants::{ COLUMN_AST_KIND, CTE_ALIAS_PAYLOAD_KEY, CTE_LIST_PAYLOAD_KEY, DEPENDENCY_FUNCTION_NAMES, FUNCTION_AST_KIND, FUNCTION_FIXTURE_TYPES, FUNCTION_PAYLOAD_KIND, HELPER_READS_DIALECT, LITERAL_FIXTURE_TYPES, LITERAL_PAYLOAD_KIND, MAX_FUNCTION_CALL_DEPTH, NAME_PAYLOAD_KEY, - NULL_PAYLOAD_KIND, RENDERED_TYPE_NAMES, SCHEMA_TABLE_ARG, SELECT_PAYLOAD_KEY, STAR_AST_KIND, - STRUCTURED_PAYLOAD_KINDS, TABLE_AST_KIND, VARIANT_FIXTURE_TYPE, WILDCARD, + NULL_PAYLOAD_KIND, RENDERED_TYPE_NAMES, SCHEMA_TABLE_ARG, SELECT_PAYLOAD_KEY, + SET_OPERATION_PAYLOAD_KEYS, STAR_AST_KIND, STRUCTURED_PAYLOAD_KINDS, TABLE_AST_KIND, + VARIANT_FIXTURE_TYPE, WILDCARD, }; /// Python's `bool(value)`. @@ -452,26 +453,46 @@ pub(crate) fn fixture_projection_facts( return Ok(ProjectionFacts::default()); }; let mut facts: ProjectionFacts = collected_facts(select); + let top: Option<&Map> = payload.as_object(); let direct: bool = matches!( - payload - .as_object() - .and_then(|map| map.get(SELECT_PAYLOAD_KEY)), + top.and_then(|map| map.get(SELECT_PAYLOAD_KEY)), Some(Value::Object(_)) ); - if !direct { + let set_operation: bool = top.is_some_and(|map| { + SET_OPERATION_PAYLOAD_KEYS + .iter() + .any(|key| matches!(map.get(*key), Some(Value::Object(_)))) + }); + if !direct && !set_operation { facts.null_literal_names.clear(); } + if set_operation { + let branches: Vec<&Map> = branch_selects(&payload); + let null_in_every_branch = |index: usize| { + branches + .iter() + .all(|branch| null_positions(branch).get(index) == Some(&true)) + }; + let kept: Vec = null_names_by_position(select) + .into_iter() + .filter(|(index, _)| null_in_every_branch(*index)) + .map(|(_, name)| name) + .collect(); + facts.null_literal_names.retain(|name| kept.contains(name)); + } Ok(facts) } -/// Python's recursive `_first_select`, depth first in payload order, without recursion. -fn first_select(payload: &Value) -> Option<&Map> { +/// Every branch `SELECT` of a set operation, without the selects nested inside them. +fn branch_selects(payload: &Value) -> Vec<&Map> { + let mut branches: Vec<&Map> = Vec::new(); let mut pending: Vec<&Value> = vec![payload]; while let Some(node) = pending.pop() { match node { Value::Object(map) => { if let Some(Value::Object(select)) = map.get(SELECT_PAYLOAD_KEY) { - return Some(select); + branches.push(select); + continue; } pending.extend(map.values().rev()); } @@ -479,7 +500,54 @@ fn first_select(payload: &Value) -> Option<&Map> { _ => {} } } - None + branches +} + +/// The projection payloads of a select, each with any alias removed. +fn projected_values(select: &Map) -> Vec<&Map> { + select + .get("expressions") + .and_then(Value::as_array) + .map_or(&[][..], Vec::as_slice) + .iter() + .filter_map(|expression| { + match expression.get("alias").and_then(|alias| alias.get("this")) { + Some(Value::Object(value)) => Some(value), + _ => expression.as_object(), + } + }) + .collect() +} + +/// Whether each projection of a select is a bare `NULL`. +fn null_positions(select: &Map) -> Vec { + projected_values(select) + .into_iter() + .map(|value| value.keys().next().map(String::as_str) == Some(NULL_PAYLOAD_KIND)) + .collect() +} + +/// The folded alias of each bare `NULL` projection of a select, with its position. +fn null_names_by_position(select: &Map) -> Vec<(usize, String)> { + let expressions: &[Value] = select + .get("expressions") + .and_then(Value::as_array) + .map_or(&[], Vec::as_slice); + expressions + .iter() + .enumerate() + .filter_map(|(index, expression)| { + let alias_payload = expression.get("alias")?; + let name: &str = alias_payload.get("alias")?.get("name")?.as_str()?; + let kind: &str = alias_payload.get("this")?.as_object()?.keys().next()?; + (kind == NULL_PAYLOAD_KIND).then(|| (index, str::to_lowercase(name))) + }) + .collect() +} + +/// Python's recursive `_first_select`, depth first in payload order, without recursion. +fn first_select(payload: &Value) -> Option<&Map> { + branch_selects(payload).into_iter().next() } fn collected_facts(select: &Map) -> ProjectionFacts { diff --git a/crates/sqlbuild-analysis/src/assembly/analysis_session/constants.rs b/crates/sqlbuild-analysis/src/assembly/analysis_session/constants.rs index 0cd68c4162..5da96a56e2 100644 --- a/crates/sqlbuild-analysis/src/assembly/analysis_session/constants.rs +++ b/crates/sqlbuild-analysis/src/assembly/analysis_session/constants.rs @@ -212,6 +212,8 @@ pub(crate) const ARRAY_PAYLOAD_KIND: &str = "array_func"; pub(crate) const STRUCTURED_PAYLOAD_KINDS: [&str; 2] = ["parse_json", "struct"]; pub(crate) const FUNCTION_PAYLOAD_KIND: &str = "function"; pub(crate) const SELECT_PAYLOAD_KEY: &str = "select"; +/// Set operations whose output columns are named and typed by their first branch. +pub(crate) const SET_OPERATION_PAYLOAD_KEYS: [&str; 3] = ["union", "intersect", "except"]; /// `{literal_type: type}` Python's `_simple_expression_type` maps a literal to. pub(crate) const LITERAL_FIXTURE_TYPES: [(&str, &str); 3] = [ ("string", "VARCHAR"), diff --git a/crates/sqlbuild-analysis/src/compiler/_helpers/sql_tests/extraction.rs b/crates/sqlbuild-analysis/src/compiler/_helpers/sql_tests/extraction.rs index 243271e3df..d97708ac75 100644 --- a/crates/sqlbuild-analysis/src/compiler/_helpers/sql_tests/extraction.rs +++ b/crates/sqlbuild-analysis/src/compiler/_helpers/sql_tests/extraction.rs @@ -683,6 +683,14 @@ fn macro_mock_value( } let (value, end) = read_string(body, index)?; index = skip_ignorable(body, end, syntax)?; + if let Some(alias_start) = consume_keyword(body, index, "AS") { + let alias: usize = skip_ignorable(body, alias_start, syntax)?; + let alias_end: usize = identifier_end(body, alias); + if alias_end == alias { + return Err(macro_mock_shape(file, &cte.0, true)); + } + index = skip_ignorable(body, alias_end, syntax)?; + } if byte_at(body, index) == Some(b';') { index = skip_ignorable(body, index + 1, syntax)?; } @@ -692,6 +700,20 @@ fn macro_mock_value( Ok(value) } +/// The end of the plain or double-quoted identifier at `start`, or `start` when there is none. +fn identifier_end(sql: &str, start: usize) -> usize { + if byte_at(sql, start) == Some(b'"') { + return sql[start + 1..] + .find('"') + .map_or(start, |close| start + close + 2); + } + let mut end: usize = start; + while char_at(sql, end).is_some_and(is_identifier_continue) { + end += char_len(sql, end); + } + end +} + fn macro_mock_shape(file: &str, name: &str, trailing: bool) -> String { if trailing { format!( diff --git a/crates/sqlbuild-analysis/src/planner_scope/main/source_selectors.rs b/crates/sqlbuild-analysis/src/planner_scope/main/source_selectors.rs index 3b66fcebc4..edcad43bb1 100644 --- a/crates/sqlbuild-analysis/src/planner_scope/main/source_selectors.rs +++ b/crates/sqlbuild-analysis/src/planner_scope/main/source_selectors.rs @@ -7,11 +7,12 @@ use crate::graph::models::ParsedSelector; use crate::planner_scope::constants::{NAME, SOURCE}; use std::collections::HashSet; -/// Find selected sources explicitly named by upstream source or bare-name selectors. +/// Selected sources named by a `name` or `source` selector, only `+`-prefixed ones if `upstream_only`. pub fn source_selectors( select: &[String], names: &[(String, ObjectKey)], selected: &[ObjectKey], + upstream_only: bool, ) -> Result, SelectorError> { let mut keys: HashSet = HashSet::new(); for part in select @@ -22,12 +23,15 @@ pub fn source_selectors( let ParsedSelector::Kind { kind, value, - upstream: true, + upstream, .. } = parse_selector(part)? else { continue; }; + if upstream_only && !upstream { + continue; + } if kind != NAME && kind != SOURCE { continue; } diff --git a/crates/sqlbuild-analysis/src/required_columns/_helpers/payload.rs b/crates/sqlbuild-analysis/src/required_columns/_helpers/payload.rs index 464d72d2ff..bb55a71d11 100644 --- a/crates/sqlbuild-analysis/src/required_columns/_helpers/payload.rs +++ b/crates/sqlbuild-analysis/src/required_columns/_helpers/payload.rs @@ -2,10 +2,11 @@ use polyglot_sql::{Expression, ast_json}; use serde_json::{Map, Value}; +use sqlbuild_core::text::main::python_casefold::python_casefold; use crate::required_columns::_helpers::tree::payload; use crate::required_columns::constants::{ - KIND_COLUMN, KIND_SELECT, KIND_SUBQUERY, SET_OPERATION_KINDS, + KIND_COLUMN, KIND_LAMBDA, KIND_SELECT, KIND_SUBQUERY, SET_OPERATION_KINDS, }; /// `expression.args.get(key)`, unless the wheel exposed it as expression objects. @@ -52,6 +53,16 @@ fn object_column_refs(map: &Map) -> Vec<(String, String)> { if nested_query { return Vec::new(); } + if let Some(Value::Object(lambda)) = map.get(KIND_LAMBDA) { + let parameters: Vec = match lambda.get("parameters") { + Some(Value::Array(parameters)) => parameters + .iter() + .map(|parameter| python_casefold(&payload_name(Some(parameter)))) + .collect(), + _ => Vec::new(), + }; + return without_lambda_parameters(column_refs_in_payload(lambda.get("body")), ¶meters); + } if let Some(Value::Object(column)) = map.get(KIND_COLUMN) { return vec![( payload_name(column.get("name")), @@ -64,3 +75,13 @@ fn object_column_refs(map: &Map) -> Vec<(String, String)> { .flat_map(|key| column_refs_in_payload(map.get(key))) .collect() } + +/// Column references that are not a lambda's own unqualified parameters. +pub(crate) fn without_lambda_parameters( + refs: Vec<(String, String)>, + parameters: &[String], +) -> Vec<(String, String)> { + refs.into_iter() + .filter(|(name, table)| !table.is_empty() || !parameters.contains(&python_casefold(name))) + .collect() +} diff --git a/crates/sqlbuild-analysis/src/required_columns/_helpers/queries.rs b/crates/sqlbuild-analysis/src/required_columns/_helpers/queries.rs index 03ce96d702..e1a2df1c53 100644 --- a/crates/sqlbuild-analysis/src/required_columns/_helpers/queries.rs +++ b/crates/sqlbuild-analysis/src/required_columns/_helpers/queries.rs @@ -7,7 +7,7 @@ use serde_json::{Map, Value}; use sqlbuild_core::text::main::python_casefold::python_casefold; use crate::required_columns::_helpers::payload::{ - arg, column_refs_in_payload, payload_name, plain_value, + arg, column_refs_in_payload, payload_name, plain_value, without_lambda_parameters, }; use crate::required_columns::_helpers::relations::{ RelationMap, RelationReads, either, unique_reads, unique_resources, @@ -17,7 +17,7 @@ use crate::required_columns::_helpers::tree::{ unwrap_annotations, }; use crate::required_columns::constants::{ - CLAUSE_NAMES, KIND_COLUMN, KIND_SELECT, KIND_SUBQUERY, KIND_TABLE, + CLAUSE_NAMES, HIERARCHY_PSEUDO_COLUMNS, KIND_COLUMN, KIND_SELECT, KIND_SUBQUERY, KIND_TABLE, OUTPUT_ALIAS_FALLBACK_CLAUSES, OUTPUT_ALIAS_FIRST_CLAUSES, SET_OPERATION_KINDS, }; use crate::required_columns::models::ColumnRead; @@ -32,6 +32,8 @@ struct Scope<'a> { local: &'a RelationMap, outer: &'a RelationMap, shared: &'a SharedReads, + /// Whether the `SELECT` walks a `CONNECT BY` hierarchy, whose pseudo-columns read no input. + hierarchical: bool, } fn shared_get<'a>(shared: &'a SharedReads, name: &str) -> Option<&'a Vec> { @@ -85,6 +87,7 @@ pub(crate) fn query_reads( local: &local, outer, shared: &shared, + hierarchical: matches!(query, Expression::Select(select) if select.connect.is_some()), }; let (slots, passthrough, projection_required) = projection_slots(query, &scope); required.extend(clause_reads(query, &slots, &scope)); @@ -320,6 +323,13 @@ fn star_modifier_slots(projection: &Expression, scope: &Scope<'_>) -> Slots { /// `_resolve_unqualified_column_with_presence`. fn unqualified_column(column_name: &str, scope: &Scope<'_>) -> (bool, Vec) { + if scope.hierarchical + && HIERARCHY_PSEUDO_COLUMNS + .iter() + .any(|pseudo| pseudo.eq_ignore_ascii_case(column_name)) + { + return (true, Vec::new()); + } if let Some(reads) = shared_get(scope.shared, &python_casefold(column_name)) { return (true, reads.clone()); } @@ -455,6 +465,14 @@ fn direct_column_refs(expression: &Expression) -> Vec<(String, String)> { { return Vec::new(); } + if let Expression::Lambda(lambda) = expression { + let parameters: Vec = lambda + .parameters + .iter() + .map(|parameter| python_casefold(¶meter.name)) + .collect(); + return without_lambda_parameters(direct_column_refs(&lambda.body), ¶meters); + } children(expression) .into_iter() .flat_map(direct_column_refs) diff --git a/crates/sqlbuild-analysis/src/required_columns/constants.rs b/crates/sqlbuild-analysis/src/required_columns/constants.rs index 92f19c845f..150f1b7fbc 100644 --- a/crates/sqlbuild-analysis/src/required_columns/constants.rs +++ b/crates/sqlbuild-analysis/src/required_columns/constants.rs @@ -2,6 +2,7 @@ pub(crate) const KIND_ANNOTATED: &str = "annotated"; pub(crate) const KIND_COLUMN: &str = "column"; +pub(crate) const KIND_LAMBDA: &str = "lambda"; pub(crate) const KIND_SELECT: &str = "select"; pub(crate) const KIND_SUBQUERY: &str = "subquery"; pub(crate) const KIND_TABLE: &str = "table"; @@ -32,3 +33,6 @@ pub(crate) const TABLE_FUNCTION_STUB_PREFIX: &str = "__sqlbuild_table_function_" /// The column name a star projection parses as. pub(crate) const STAR: &str = "*"; pub(crate) const GENERIC_DIALECT: &str = "generic"; +/// Columns a `CONNECT BY` query computes rather than reads from its inputs. +pub(crate) const HIERARCHY_PSEUDO_COLUMNS: [&str; 3] = + ["LEVEL", "CONNECT_BY_ISLEAF", "CONNECT_BY_ISCYCLE"]; diff --git a/crates/sqlbuild-analysis/src/sql_macros/_helpers/rendering/expansion.rs b/crates/sqlbuild-analysis/src/sql_macros/_helpers/rendering/expansion.rs index 96c8c22834..b9524aa6ae 100644 --- a/crates/sqlbuild-analysis/src/sql_macros/_helpers/rendering/expansion.rs +++ b/crates/sqlbuild-analysis/src/sql_macros/_helpers/rendering/expansion.rs @@ -2,7 +2,6 @@ use std::sync::Arc; -use crate::compiler::main::function_type_validation::function_type_error; use crate::sql_macros::_helpers::parsing::body::{expand_body, with_line_comment_closed}; use crate::sql_macros::_helpers::parsing::lexing::{AtConstruct, at_constructs, outer_sql_calls}; use crate::sql_macros::_helpers::parsing::lexing::{Token, line_column, tokens}; @@ -607,13 +606,13 @@ fn check_adapter_types( adapter: &str, ) -> Result<(), SqlMacroError> { let invalid = |data_type: &str, context: String| -> Option { - match function_type_error(data_type, adapter, &context) { - Ok(Some(message)) => Some( - SqlMacroError::new(INVALID_DEFINITION_CODE, message) - .helped("use a SQL type the warehouse accepts, as in schema declarations"), - ), - Ok(None) | Err(_) => None, + if adapter.is_empty() { + return None; } + returns_type_error(data_type, adapter, &context).map(|message| { + SqlMacroError::new(INVALID_DEFINITION_CODE, message) + .helped("use a SQL type the warehouse accepts, as in schema declarations") + }) }; for parameter in &definition.parameters { let types: Vec<&str> = match ¶meter.kind { diff --git a/crates/sqlbuild-analysis/src/sql_macros/_helpers/typing/return_types.rs b/crates/sqlbuild-analysis/src/sql_macros/_helpers/typing/return_types.rs index 469ac054ee..ff191cdacd 100644 --- a/crates/sqlbuild-analysis/src/sql_macros/_helpers/typing/return_types.rs +++ b/crates/sqlbuild-analysis/src/sql_macros/_helpers/typing/return_types.rs @@ -4,6 +4,9 @@ use polyglot_sql::expressions::DataType; use polyglot_sql::{Dialect, DialectType, Expression}; use crate::sql_macros::_helpers::typing::returns_check::adapter_dialect; +use crate::sql_macros::constants::{ + BIGQUERY_BIG_DECIMAL_NAMES, BIGQUERY_BIGNUMERIC_LIMITS, BIGQUERY_NUMERIC_LIMITS, +}; /// Why `returns` is not a type the adapter's SQL analysis dialect parses and can hold. pub(crate) fn returns_type_error(returns: &str, adapter: &str, context: &str) -> Option { @@ -12,7 +15,8 @@ pub(crate) fn returns_type_error(returns: &str, adapter: &str, context: &str) -> return None; }; let reason: String = match polyglot_sql::parse_data_type(returns, dialect) { - Ok(parsed) => decimal_error(&parsed, dialect)?, + Ok(parsed) => decimal_error(&parsed, dialect) + .or_else(|| bigquery_numeric_error(returns, &parsed, dialect))?, Err(error) => error.to_string(), }; Some(format!( @@ -21,6 +25,40 @@ pub(crate) fn returns_type_error(returns: &str, adapter: &str, context: &str) -> )) } +/// Why a BigQuery `NUMERIC(P, S)` or `BIGNUMERIC(P, S)` has more digits than the type holds. +fn bigquery_numeric_error( + type_sql: &str, + data_type: &DataType, + dialect: DialectType, +) -> Option { + let DataType::Decimal { + precision: Some(precision), + scale, + } = data_type + else { + return None; + }; + if dialect != DialectType::BigQuery { + return None; + } + let scale: u32 = scale.unwrap_or(0); + let big: bool = BIGQUERY_BIG_DECIMAL_NAMES + .iter() + .any(|name| type_sql.trim_start().to_ascii_uppercase().starts_with(name)); + let (max_integer_digits, max_scale) = if big { + BIGQUERY_BIGNUMERIC_LIMITS + } else { + BIGQUERY_NUMERIC_LIMITS + }; + let integer_digits: u32 = precision.saturating_sub(scale); + (integer_digits > max_integer_digits || scale > max_scale).then(|| { + format!( + "precision {precision} and scale {scale} exceed BigQuery's limits of {max_integer_digits} \ + integer digits and scale {max_scale}" + ) + }) +} + /// Why a decimal within `data_type` has a precision or scale `dialect` cannot hold. fn decimal_error(data_type: &DataType, dialect: DialectType) -> Option { match data_type { diff --git a/crates/sqlbuild-analysis/src/sql_macros/constants.rs b/crates/sqlbuild-analysis/src/sql_macros/constants.rs index f63bb88d55..6d7397f960 100644 --- a/crates/sqlbuild-analysis/src/sql_macros/constants.rs +++ b/crates/sqlbuild-analysis/src/sql_macros/constants.rs @@ -134,3 +134,9 @@ pub(crate) const MARKER_PARSE_DIALECTS: [polyglot_sql::DialectType; 4] = [ ]; /// The literal a typed marker still types, though it is one bare word. pub(crate) const NULL_KEYWORD: &str = "NULL"; +/// BigQuery's most integer digits and scale of a `NUMERIC`. +pub(crate) const BIGQUERY_NUMERIC_LIMITS: (u32, u32) = (29, 9); +/// BigQuery's most integer digits and scale of a `BIGNUMERIC`. +pub(crate) const BIGQUERY_BIGNUMERIC_LIMITS: (u32, u32) = (38, 38); +/// BigQuery's spellings of `BIGNUMERIC`. +pub(crate) const BIGQUERY_BIG_DECIMAL_NAMES: [&str; 2] = ["BIGNUMERIC", "BIGDECIMAL"]; diff --git a/crates/sqlbuild-analysis/src/sql_macros/tests/test_expansion.rs b/crates/sqlbuild-analysis/src/sql_macros/tests/test_expansion.rs index c15ba68538..d1cc4bf0cc 100644 --- a/crates/sqlbuild-analysis/src/sql_macros/tests/test_expansion.rs +++ b/crates/sqlbuild-analysis/src/sql_macros/tests/test_expansion.rs @@ -150,6 +150,19 @@ const ODD: (&str, &str) = ( "MACRO (description \"Odd\", arguments (v FOO BAR), returns BIGINT);\n\n@v + 1\n", ); +const WIDE: (&str, &str) = ( + "wide", + "MACRO (description \"Wide\", arguments (v DECIMAL(50, 2)), returns BIGINT);\n\n@v + 1\n", +); +const NOT_NULL_ARGUMENT: (&str, &str) = ( + "not_null_argument", + "MACRO (description \"Not null\", arguments (v INTEGER NOT NULL), returns BIGINT);\n\n@v + 1\n", +); +const WIDE_NUMERIC_RETURNS: (&str, &str) = ( + "wide_numeric", + "MACRO (description \"Wide\", arguments (v INT64), returns NUMERIC(40, 2));\n\n@v + 1\n", +); + #[test] fn given_calls_of_each_form_when_expanding_then_the_ticket_examples_compile() { let test_cases = [ @@ -695,6 +708,38 @@ fn given_invalid_calls_when_expanding_then_each_error_names_the_problem() { expected_code: "P015", expected_message: "argument v of macro odd() type 'FOO BAR' is not valid for adapter 'duckdb'", }, + ExpansionErrorTestCase { + description: "an argument type Postgres cannot parse", + adapter: "postgres", + macros: &[ODD], + sql: "SELECT @sql.odd(1) AS a", + expected_code: "P015", + expected_message: "argument v of macro odd() type 'FOO BAR' is not valid for adapter 'postgres'", + }, + ExpansionErrorTestCase { + description: "an argument type MotherDuck reads as DuckDB", + adapter: "motherduck", + macros: &[NOT_NULL_ARGUMENT], + sql: "SELECT @sql.not_null_argument(1) AS a", + expected_code: "P015", + expected_message: "argument v of macro not_null_argument() type 'INTEGER NOT NULL' is not valid for adapter 'motherduck'", + }, + ExpansionErrorTestCase { + description: "a decimal argument wider than DuckDB holds", + adapter: "duckdb", + macros: &[WIDE], + sql: "SELECT @sql.wide(1) AS a", + expected_code: "P015", + expected_message: "argument v of macro wide() type 'DECIMAL(50, 2)' is not valid for adapter 'duckdb'", + }, + ExpansionErrorTestCase { + description: "a NUMERIC returns wider than BigQuery holds", + adapter: "bigquery", + macros: &[WIDE_NUMERIC_RETURNS], + sql: "SELECT @sql.wide_numeric(1) AS a", + expected_code: "P015", + expected_message: "returns of macro wide_numeric() type 'NUMERIC(40, 2)' is not valid for adapter 'bigquery'", + }, ]; for test_case in test_cases { diff --git a/crates/sqlbuild-analysis/src/sql_test_attachment/tests/test_sql_test_classification_cases.rs b/crates/sqlbuild-analysis/src/sql_test_attachment/tests/test_sql_test_classification_cases.rs index 76a5369b34..c3eceb9112 100644 --- a/crates/sqlbuild-analysis/src/sql_test_attachment/tests/test_sql_test_classification_cases.rs +++ b/crates/sqlbuild-analysis/src/sql_test_attachment/tests/test_sql_test_classification_cases.rs @@ -161,6 +161,16 @@ fn given_sql_test_blocks_when_classifying_then_outcomes_are_exact() { request: r####"{"syntax": {"backslashEscapeQuotes": [], "escapeStringPrefix": false, "rawStringPrefix": false, "tripleQuotedStrings": false, "nestedBlockComments": false, "lineCommentPrefixes": ["--"]}, "tests": [{"sql": "WITH\n __source__raw_orders AS (SELECT 1 AS order_id),\n __expected__orders AS (SELECT 1 AS order_id),\n expected_helper AS (SELECT * FROM __expected__orders),\n __assert__expected_is_nonempty AS (SELECT * FROM expected_helper)\n SELECT 1", "fileLabel": "tests/unit/orders.sql", "mode": "model", "raw": true}]}"####, expected_response: r####"{"error":{"index":0,"message":"SQL test 'tests/unit/orders.sql' check CTE '__assert__expected_is_nonempty' must not depend on '__expected__orders' through 'expected_helper'; expected results and assertions must be independent","help":null,"token":null,"tokenOffset":null}}"####, }, + BatchClassificationTestCase { + description: "model roles: a macro mock may name its one string literal", + request: r####"{"syntax": {"backslashEscapeQuotes": [], "escapeStringPrefix": false, "rawStringPrefix": false, "tripleQuotedStrings": false, "nestedBlockComments": false, "lineCommentPrefixes": ["--"]}, "tests": [{"sql": "WITH\n __macro__country AS (SELECT '''US''' AS country_filter),\n __macro__region AS (SELECT 'EU' AS \"region\"),\n __source__raw_orders AS (SELECT 1 AS order_id),\n __expected__orders AS (SELECT 1 AS order_id)\n SELECT 1", "fileLabel": "tests/unit/orders.sql", "mode": "model", "raw": true}]}"####, + expected_response: r####"{"tests":[{"kind":"model","mode":"model","authored":[["__source__raw_orders","SELECT 1 AS order_id"]],"macroMocks":[["country","'US'"],["region","EU"]],"mockModels":[],"mockSources":["raw_orders"],"mockSeeds":[],"mockDbtRefs":[],"mockTableFunctions":[],"expected":[["__expected__orders","SELECT 1 AS order_id"]],"expectedModels":["orders"],"assertions":[],"assertionNames":[]}]}"####, + }, + BatchClassificationTestCase { + description: "invalid: raises when a macro mock's alias is missing after AS", + request: r####"{"syntax": {"backslashEscapeQuotes": [], "escapeStringPrefix": false, "rawStringPrefix": false, "tripleQuotedStrings": false, "nestedBlockComments": false, "lineCommentPrefixes": ["--"]}, "tests": [{"sql": "WITH __macro__country AS (SELECT 'US' AS),\n __source__raw_orders AS (SELECT 1 AS order_id),\n __expected__orders AS (SELECT 1 AS order_id)\n SELECT 1", "fileLabel": "tests/unit/orders.sql", "mode": "model", "raw": true}]}"####, + expected_response: r####"{"error":{"index":0,"message":"SQL test 'tests/unit/orders.sql' macro mock '__macro__country' must be a single SELECT string literal with no FROM, UNION, or additional columns","help":null,"token":null,"tokenOffset":null}}"####, + }, BatchClassificationTestCase { description: "invalid: raises when macro mock returns multiple columns", request: r####"{"syntax": {"backslashEscapeQuotes": [], "escapeStringPrefix": false, "rawStringPrefix": false, "tripleQuotedStrings": false, "nestedBlockComments": false, "lineCommentPrefixes": ["--"]}, "tests": [{"sql": "WITH __macro__country AS (SELECT 'US', 'CA'),\n __source__raw_orders AS (SELECT 1 AS order_id),\n __expected__orders AS (SELECT 1 AS order_id)\n SELECT 1", "fileLabel": "tests/unit/orders.sql", "mode": "model", "raw": true}]}"####, diff --git a/crates/sqlbuild-analysis/src/type_system/_helpers/fallback.rs b/crates/sqlbuild-analysis/src/type_system/_helpers/fallback.rs index c561cc2176..6ee0ba4bf1 100644 --- a/crates/sqlbuild-analysis/src/type_system/_helpers/fallback.rs +++ b/crates/sqlbuild-analysis/src/type_system/_helpers/fallback.rs @@ -7,8 +7,8 @@ use crate::type_system::constants::{ BIGNUMERIC_TYPE_NAME, BOOLEAN_TYPE_NAMES, DATE_TYPE_NAME, DATETIME_TYPE_NAME, DECIMAL_TYPE_NAMES, DEFAULT_TEXT_LENGTH, FLOAT_TYPE_NAMES, INTEGER_PRECISION, INTEGER_SCALE, INTEGER_TYPE_NAMES, NORMALIZED_LTZ_INPUT_TYPE_NAME, NORMALIZED_NTZ_INPUT_TYPE_NAMES, - NORMALIZED_TZ_INPUT_TYPE_NAME, STRING_TYPE_NAMES, TEXT_TYPE_NAME, TIMESTAMP_TYPE_TOKEN, - UNBOUNDED_TEXT_TYPE_NAMES, + NORMALIZED_TZ_INPUT_TYPE_NAME, SIZED_TEXT_TYPE_NAMES, STRING_TYPE_NAMES, TEXT_TYPE_NAME, + TIMESTAMP_TYPE_TOKEN, UNBOUNDED_TEXT_TYPE_NAMES, VARCHAR_TYPE_NAME, }; use crate::type_system::models::{NormalizedType, PythonInteger, TypeDialect, TypeFamily}; @@ -69,8 +69,10 @@ pub(crate) fn normalize_with_fallback( let mut length: Option = first; let base_name: &str = if bigquery { "STRING" - } else if snowflake && base_type == TEXT_TYPE_NAME { - "VARCHAR" + } else if snowflake && base_type == TEXT_TYPE_NAME + || length.is_some() && SIZED_TEXT_TYPE_NAMES.contains(&base_type) + { + VARCHAR_TYPE_NAME } else { base_type }; diff --git a/crates/sqlbuild-analysis/src/type_system/_helpers/parsed.rs b/crates/sqlbuild-analysis/src/type_system/_helpers/parsed.rs index 1b4007841f..1ee482d2c8 100644 --- a/crates/sqlbuild-analysis/src/type_system/_helpers/parsed.rs +++ b/crates/sqlbuild-analysis/src/type_system/_helpers/parsed.rs @@ -12,11 +12,12 @@ use crate::type_system::_helpers::fallback::{ use crate::type_system::_helpers::python_text::{python_int, split_type_and_params, upper}; use crate::type_system::_helpers::shallow_args::shallow_args; use crate::type_system::constants::{ - BIGNUMERIC_TYPE_NAME, BOOLEAN_TYPE_NAMES, CUSTOM_NORMALIZATION_TYPE_NAMES, DATE_TYPE_NAME, - DATETIME_TYPE_NAME, DECIMAL_TYPE_NAMES, FLOAT_TYPE_NAMES, FLOAT_WIRE_TYPE_NAME, - INTEGER_PARSE_TYPE_NAMES, INTEGER_PRECISION, INTEGER_SCALE, INTEGER_TYPE_NAMES, - MAX_FUNCTION_CALL_DEPTH, POLYGLOT_CUSTOM_TYPE_NAME, POLYGLOT_TYPE_NAME_ALIASES, - STRING_TYPE_NAMES, TIMESTAMP_TYPE_NAMES, UNBOUNDED_TEXT_TYPE_NAMES, + BIGNUMERIC_TYPE_NAME, BIGQUERY_STRING_TYPE_NAME, BOOLEAN_TYPE_NAMES, + CUSTOM_NORMALIZATION_TYPE_NAMES, DATE_TYPE_NAME, DATETIME_TYPE_NAME, DECIMAL_TYPE_NAMES, + FLOAT_TYPE_NAMES, FLOAT_WIRE_TYPE_NAME, INTEGER_PARSE_TYPE_NAMES, INTEGER_PRECISION, + INTEGER_SCALE, INTEGER_TYPE_NAMES, MAX_FUNCTION_CALL_DEPTH, POLYGLOT_CUSTOM_TYPE_NAME, + POLYGLOT_TYPE_NAME_ALIASES, SIZED_TEXT_TYPE_NAMES, STRING_TYPE_NAMES, TIMESTAMP_TYPE_NAMES, + UNBOUNDED_TEXT_TYPE_NAMES, VARCHAR_TYPE_NAME, }; use crate::type_system::errors::TypeNormalizationError; use crate::type_system::models::{NormalizedType, PythonInteger, TypeDialect, TypeFamily}; @@ -120,6 +121,10 @@ pub(crate) fn normalize_parsed( let bounded: PythonInteger = python_or_default(length); name = format!("VARCHAR({bounded})"); length = Some(bounded); + } else if bigquery && UNBOUNDED_TEXT_TYPE_NAMES.contains(&base_name) { + name = name.replacen(base_name, BIGQUERY_STRING_TYPE_NAME, 1); + } else if length.is_some() && SIZED_TEXT_TYPE_NAMES.contains(&base_name) { + name = name.replacen(base_name, VARCHAR_TYPE_NAME, 1); } return Ok(NormalizedType { normalized_name: name, diff --git a/crates/sqlbuild-analysis/src/type_system/constants.rs b/crates/sqlbuild-analysis/src/type_system/constants.rs index 39f858f52d..ef06dd32cb 100644 --- a/crates/sqlbuild-analysis/src/type_system/constants.rs +++ b/crates/sqlbuild-analysis/src/type_system/constants.rs @@ -26,10 +26,16 @@ pub(crate) const NORMALIZED_NTZ_INPUT_TYPE_NAMES: &[&str] = &["TIMESTAMP", "TIME pub(crate) const NORMALIZED_TZ_INPUT_TYPE_NAME: &str = "TIMESTAMPTZ"; pub(crate) const TEXT_TYPE_NAME: &str = "TEXT"; pub(crate) const UNBOUNDED_TEXT_TYPE_NAMES: &[&str] = &["STRING", "TEXT", "VARCHAR"]; +/// Text types spelled `VARCHAR` when they carry a length. +pub(crate) const SIZED_TEXT_TYPE_NAMES: &[&str] = &["STRING", "TEXT"]; +pub(crate) const VARCHAR_TYPE_NAME: &str = "VARCHAR"; +/// BigQuery's one spelling of every variable-length text type. +pub(crate) const BIGQUERY_STRING_TYPE_NAME: &str = "STRING"; /// Polyglot `data_type` tags whose Python spelling drops the word separator. pub(crate) const POLYGLOT_TYPE_NAME_ALIASES: &[(&str, &str)] = &[ ("BIG_INT", "BIGINT"), ("SMALL_INT", "SMALLINT"), + ("TEXT_WITH_LENGTH", "TEXT"), ("TINY_INT", "TINYINT"), ("VAR_CHAR", "VARCHAR"), ]; diff --git a/crates/sqlbuild-analysis/src/type_system/tests/test_normalize_type.rs b/crates/sqlbuild-analysis/src/type_system/tests/test_normalize_type.rs index 207359e441..afee3fc479 100644 --- a/crates/sqlbuild-analysis/src/type_system/tests/test_normalize_type.rs +++ b/crates/sqlbuild-analysis/src/type_system/tests/test_normalize_type.rs @@ -42,6 +42,24 @@ fn given_type_strings_when_normalizing_then_python_normalization_is_returned() { dialect: "snowflake", expected_normalization: Ok("VARCHAR(16777216) | string | - | - | 16777216 | -"), }, + NormalizeTypeTestCase { + description: "a sized TEXT is a VARCHAR of that length", + type_sql: "TEXT(5)", + dialect: "duckdb", + expected_normalization: Ok("VARCHAR(5) | string | - | - | 5 | -"), + }, + NormalizeTypeTestCase { + description: "a sized STRING is a VARCHAR of that length on Snowflake", + type_sql: "STRING(5)", + dialect: "snowflake", + expected_normalization: Ok("VARCHAR(5) | string | - | - | 5 | -"), + }, + NormalizeTypeTestCase { + description: "BigQuery spells every variable-length text type STRING", + type_sql: "VARCHAR(5)", + dialect: "bigquery", + expected_normalization: Ok("STRING(5) | string | - | - | 5 | -"), + }, NormalizeTypeTestCase { description: "Snowflake timestamp aliases", type_sql: "TIMESTAMP_NTZ", diff --git a/crates/sqlbuild-attachments/src/audits/_helpers/rendering.rs b/crates/sqlbuild-attachments/src/audits/_helpers/rendering.rs index c60ebc63a2..228febd8cc 100644 --- a/crates/sqlbuild-attachments/src/audits/_helpers/rendering.rs +++ b/crates/sqlbuild-attachments/src/audits/_helpers/rendering.rs @@ -4,7 +4,10 @@ use crate::audits::_helpers::parameters::{ RenderStop, parameter_matches, render_parameterized_sql, }; use crate::audits::_helpers::policies::{resolve_run_scope, resolve_severity}; -use crate::audits::constants::{MODEL_RESOURCE, SEED_RESOURCE, SOURCE_RESOURCE}; +use crate::audits::constants::{ + MODEL_RESOURCE, RELATION_REFERENCE_CALLS, RELATIONSHIPS_AUDIT, RELATIONSHIPS_TARGET_ARGUMENT, + SEED_RESOURCE, SOURCE_RESOURCE, +}; use crate::audits::models::{ArgumentValue, AuditAttachment, PolicySource}; /// Severity first, then run scope, as Python resolves them after expanding the SQL. @@ -129,3 +132,33 @@ pub(crate) fn unused_argument(attachment: &AuditAttachment) -> Option { attachment.owner_label, attachment.definition_name )) } + +/// The error for a built-in `relationships` audit whose `to` is not a relation reference call. +pub(crate) fn unreferenced_relationship_target(attachment: &AuditAttachment) -> Option { + if attachment.definition_path.is_some() || attachment.definition_name != RELATIONSHIPS_AUDIT { + return None; + } + let target: &str = + attachment + .explicit_arguments + .iter() + .find_map(|(name, value)| match value { + ArgumentValue::Text(text) if name == RELATIONSHIPS_TARGET_ARGUMENT => { + Some(text.as_str()) + } + _ => None, + })?; + if RELATION_REFERENCE_CALLS + .iter() + .any(|call| target.contains(call)) + { + return None; + } + let name: &str = target.trim().trim_matches(['"', '\'', '`']); + Some(format!( + "{} audit 'relationships' argument 'to' is {target}, not a relation reference; write \ + to __ref(\"{name}\") (or __source or __seed) so the audit reads the built relation and \ + runs after it", + attachment.owner_label + )) +} diff --git a/crates/sqlbuild-attachments/src/audits/constants.rs b/crates/sqlbuild-attachments/src/audits/constants.rs index 657b260741..6d94ba6c52 100644 --- a/crates/sqlbuild-attachments/src/audits/constants.rs +++ b/crates/sqlbuild-attachments/src/audits/constants.rs @@ -9,6 +9,11 @@ pub(crate) const MODEL_RESOURCE: &str = "model"; pub(crate) const SOURCE_RESOURCE: &str = "source"; pub(crate) const SEED_RESOURCE: &str = "seed"; pub(crate) const FUNCTION_RESOURCE: &str = "function"; +pub(crate) const RELATIONSHIPS_AUDIT: &str = "relationships"; +pub(crate) const RELATIONSHIPS_TARGET_ARGUMENT: &str = "to"; +/// The calls that make a `relationships` target a dependency on a built relation. +pub(crate) const RELATION_REFERENCE_CALLS: [&str; 4] = + ["__ref(", "__source(", "__seed(", "__dbt_ref("]; /// Singular audits reading fewer models must also read a companion resource. pub(crate) const CROSS_MODEL_MINIMUM: usize = 2; /// The run scope Python falls back to without an instance or project value. diff --git a/crates/sqlbuild-attachments/src/audits/main/render_attached_audit.rs b/crates/sqlbuild-attachments/src/audits/main/render_attached_audit.rs index 99894aabc3..885e4d9b4b 100644 --- a/crates/sqlbuild-attachments/src/audits/main/render_attached_audit.rs +++ b/crates/sqlbuild-attachments/src/audits/main/render_attached_audit.rs @@ -1,6 +1,8 @@ //! Attached generic audit rendering: argument merge, SQL, severity and run scope. -use crate::audits::_helpers::rendering::{merged_arguments, policies, rendered, unused_argument}; +use crate::audits::_helpers::rendering::{ + merged_arguments, policies, rendered, unreferenced_relationship_target, unused_argument, +}; use crate::audits::models::{ArgumentValue, AuditAttachment, AuditRendering, RenderedAudit}; /// Render one attachment, or the error rendering raises. @@ -14,7 +16,9 @@ pub fn render_attached_audit(attachment: &AuditAttachment) -> AuditRendering { .as_deref() .map(|evidence| rendered(attachment, evidence, &arguments)) .transpose()?; - if let Some(message) = unused_argument(attachment) { + if let Some(message) = + unused_argument(attachment).or_else(|| unreferenced_relationship_target(attachment)) + { return Err(message); } Ok(RenderedAudit { diff --git a/crates/sqlbuild-attachments/src/audits/tests/test_render_attached_audit.rs b/crates/sqlbuild-attachments/src/audits/tests/test_render_attached_audit.rs index 0a5ec0c20f..b639d058d8 100644 --- a/crates/sqlbuild-attachments/src/audits/tests/test_render_attached_audit.rs +++ b/crates/sqlbuild-attachments/src/audits/tests/test_render_attached_audit.rs @@ -102,6 +102,48 @@ fn given_attached_audits_when_rendering_then_python_rendering_is_returned() { 'where_condition' from the attachment", ), }, + AttachedAuditTestCase { + description: "a relationships target named as text is told to use a reference", + attachment: AuditAttachment { + definition_name: "relationships".to_owned(), + sql_body: "SELECT * FROM t WHERE status NOT IN (SELECT @field FROM @to)".to_owned(), + ..attachment( + vec![ + ("to".to_owned(), ArgumentValue::Text("customers".to_owned())), + ("field".to_owned(), ArgumentValue::Text("id".to_owned())), + ], + None, + None, + ) + }, + expected_rendering: failed( + "models/schema.yml audit 'relationships' argument 'to' is customers, not a \ + relation reference; write to __ref(\"customers\") (or __source or __seed) so \ + the audit reads the built relation and runs after it", + ), + }, + AttachedAuditTestCase { + description: "a relationships target written as a reference renders", + attachment: AuditAttachment { + definition_name: "relationships".to_owned(), + sql_body: "SELECT * FROM t WHERE status NOT IN (SELECT @field FROM @to)".to_owned(), + ..attachment( + vec![ + ( + "to".to_owned(), + ArgumentValue::Text("__ref(\"customers\")".to_owned()), + ), + ("field".to_owned(), ArgumentValue::Text("id".to_owned())), + ], + None, + None, + ) + }, + expected_rendering: rendered( + "SELECT * FROM t WHERE status NOT IN (SELECT id FROM __ref(\"customers\"))", + Ok(("error", PolicySource::Fallback)), + ), + }, AttachedAuditTestCase { description: "a missing argument is Python's error before the policies", attachment: attachment(Vec::new(), Some("fatal"), None), diff --git a/crates/sqlbuild-core/src/build_aggregate/constants.rs b/crates/sqlbuild-core/src/build_aggregate/constants.rs index ec787e5d19..b08b839743 100644 --- a/crates/sqlbuild-core/src/build_aggregate/constants.rs +++ b/crates/sqlbuild-core/src/build_aggregate/constants.rs @@ -7,6 +7,7 @@ pub(crate) const TEST_PASS: &str = "pass"; pub(crate) const AUDIT_WARN: &str = "warn"; pub(crate) const AUDIT_ERROR: &str = "error"; pub(crate) const AUDIT_INSUFFICIENT: &str = "insufficient"; +pub(crate) const AUDIT_SKIPPED: &str = "skipped"; pub(crate) const SOURCE_FRESHNESS_BLOCKED_CODE: &str = "B005"; pub(crate) const TEST_TIMEOUT: &str = "timeout"; pub(crate) const ITEM_TIMEOUT_CODE: &str = "C417"; diff --git a/crates/sqlbuild-core/src/build_aggregate/main/aggregate_build_counts.rs b/crates/sqlbuild-core/src/build_aggregate/main/aggregate_build_counts.rs index 68db45ec51..24c823dd06 100644 --- a/crates/sqlbuild-core/src/build_aggregate/main/aggregate_build_counts.rs +++ b/crates/sqlbuild-core/src/build_aggregate/main/aggregate_build_counts.rs @@ -1,7 +1,7 @@ //! Aggregate counts and overall status of one build. use crate::build_aggregate::constants::{ - AUDIT_ERROR, AUDIT_INSUFFICIENT, AUDIT_WARN, FAILED, ITEM_TIMEOUT_CODE, SKIPPED, + AUDIT_ERROR, AUDIT_INSUFFICIENT, AUDIT_SKIPPED, AUDIT_WARN, FAILED, ITEM_TIMEOUT_CODE, SKIPPED, SOURCE_FRESHNESS_BLOCKED_CODE, SUCCESS, TEST_PASS, TEST_TIMEOUT, }; use crate::build_aggregate::models::{BuildCounts, BuildResultFacts}; @@ -60,6 +60,8 @@ pub fn aggregate_build_counts(facts: &BuildResultFacts) -> BuildCounts { .chain(model_audits) .filter(|outcome| *outcome == AUDIT_INSUFFICIENT) .count(); + counts.skipped += matching(&facts.source_audit_outcomes, AUDIT_SKIPPED) + + matching(&facts.end_audit_outcomes, AUDIT_SKIPPED); counts.failure += matching(&facts.source_audit_outcomes, AUDIT_ERROR); counts.warning += matching(&facts.source_audit_outcomes, AUDIT_WARN) + matching(&facts.end_audit_outcomes, AUDIT_WARN); diff --git a/crates/sqlbuild-core/src/build_progress/_helpers/footer.rs b/crates/sqlbuild-core/src/build_progress/_helpers/footer.rs index a9ac63ec88..b11ab9d35f 100644 --- a/crates/sqlbuild-core/src/build_progress/_helpers/footer.rs +++ b/crates/sqlbuild-core/src/build_progress/_helpers/footer.rs @@ -54,6 +54,7 @@ pub(crate) fn count_results(result: &BuildResultText, python: &[PythonNodeText]) PASS => counts.pass += 1, WARN => counts.warn += 1, ERROR => counts.fail += 1, + SKIPPED => counts.skip += 1, _ => {} } } @@ -295,6 +296,14 @@ pub(crate) fn python_failure_details(python: &[PythonNodeText], style: CliStyle) lines } +/// A skipped audit's rollup entry: its name and why it did not run. +fn skipped_audit_label(audit: &AuditResultText) -> String { + match &audit.skip_reason { + Some(reason) => format!("{} (audit, {reason})", audit.audit_name), + None => format!("{} (audit)", audit.audit_name), + } +} + pub(crate) fn skipped_rollup( result: &BuildResultText, python: &[PythonNodeText], @@ -333,10 +342,25 @@ pub(crate) fn skipped_rollup( .map(|node| node.name.as_str()), ) .collect(); + let names: Vec = names + .into_iter() + .map(str::to_owned) + .chain( + result + .end_audits + .iter() + .filter(|audit| audit.outcome == SKIPPED) + .map(skipped_audit_label), + ) + .collect(); if names.is_empty() { return Vec::new(); } - let visible: Vec<&str> = names.iter().take(SKIPPED_ROLLUP_CAP).copied().collect(); + let visible: Vec<&str> = names + .iter() + .take(SKIPPED_ROLLUP_CAP) + .map(String::as_str) + .collect(); let remaining: usize = names.len() - visible.len(); let mut rendered: String = visible.join(", "); if remaining > 0 { diff --git a/crates/sqlbuild-core/src/build_progress/_helpers/text.rs b/crates/sqlbuild-core/src/build_progress/_helpers/text.rs index 67ec55d183..7c087f7b07 100644 --- a/crates/sqlbuild-core/src/build_progress/_helpers/text.rs +++ b/crates/sqlbuild-core/src/build_progress/_helpers/text.rs @@ -90,6 +90,7 @@ pub(crate) fn audit_outcome_display(outcome: &str) -> String { WARN => "WARN".to_owned(), ERROR => "FAIL".to_owned(), INSUFFICIENT => "INSUFFICIENT".to_owned(), + SKIPPED => "SKIP".to_owned(), other => other.to_owned(), } } diff --git a/crates/sqlbuild-core/src/build_progress/models.rs b/crates/sqlbuild-core/src/build_progress/models.rs index 80a8ba1452..00ab94fc7e 100644 --- a/crates/sqlbuild-core/src/build_progress/models.rs +++ b/crates/sqlbuild-core/src/build_progress/models.rs @@ -38,6 +38,8 @@ pub struct AuditResultText { /// `str(measured_value)`. pub measured_value: String, pub execution_error: Option, + /// Why the audit did not run, when its outcome is `skipped`. + pub skip_reason: Option, } /// One pre- or post-hook result. diff --git a/crates/sqlbuild-core/src/integration_results/constants.rs b/crates/sqlbuild-core/src/integration_results/constants.rs index 6139ea82ef..9bdf314da0 100644 --- a/crates/sqlbuild-core/src/integration_results/constants.rs +++ b/crates/sqlbuild-core/src/integration_results/constants.rs @@ -44,7 +44,8 @@ pub(crate) const FAILED_PHASES: [&str; 12] = [ "staging", "type_enforcement", ]; -pub(crate) const CHECK_STATUSES: [&str; 5] = ["error", "fail", "insufficient", "pass", "warn"]; +pub(crate) const CHECK_STATUSES: [&str; 6] = + ["error", "fail", "insufficient", "pass", "skipped", "warn"]; pub(crate) const CHECK_NON_FAIL_STATUSES: [&str; 2] = ["insufficient", "pass"]; /// `AuditSeverity` values, in declaration order. pub(crate) const AUDIT_SEVERITIES: [&str; 2] = ["warn", "error"]; diff --git a/crates/sqlbuild-core/src/progress/constants.rs b/crates/sqlbuild-core/src/progress/constants.rs index f5fa3267e7..96c15345fc 100644 --- a/crates/sqlbuild-core/src/progress/constants.rs +++ b/crates/sqlbuild-core/src/progress/constants.rs @@ -120,9 +120,10 @@ pub(crate) const STATUS_ELLIPSIS: &str = "..."; pub(crate) const AUDIT_RESOURCE_KIND: &str = "audit"; pub(crate) const AUDIT_RESOURCE_PREFIX: &str = "audit:"; /// Audit outcomes and the status words their rows show. -pub(crate) const AUDIT_OUTCOME_STATUSES: [(&str, &str); 4] = [ +pub(crate) const AUDIT_OUTCOME_STATUSES: [(&str, &str); 5] = [ ("pass", "PASS"), ("warn", "WARN"), ("error", "FAIL"), ("insufficient", "INSUFFICIENT"), + ("skipped", "SKIP"), ]; diff --git a/crates/sqlbuild-host/src/models.rs b/crates/sqlbuild-host/src/models.rs index f397ffa72e..ee79ea246c 100644 --- a/crates/sqlbuild-host/src/models.rs +++ b/crates/sqlbuild-host/src/models.rs @@ -317,10 +317,17 @@ pub enum AuditOutcome { Warn, Error, Insufficient, + Skipped, } impl ResultEnum for AuditOutcome { - const ALL: &'static [Self] = &[Self::Pass, Self::Warn, Self::Error, Self::Insufficient]; + const ALL: &'static [Self] = &[ + Self::Pass, + Self::Warn, + Self::Error, + Self::Insufficient, + Self::Skipped, + ]; fn value(self) -> &'static str { match self { @@ -328,6 +335,7 @@ impl ResultEnum for AuditOutcome { Self::Warn => "warn", Self::Error => "error", Self::Insufficient => "insufficient", + Self::Skipped => "skipped", } } } @@ -474,6 +482,7 @@ pub struct AuditExecutionResult { pub audit_description: Option, pub execution_error: Option, pub audit_meta: Option>, + pub skip_reason: Option, } /// One executed model. Mirrors `sqlbuild.executor.run.models.ModelExecutionResult`; `V` is a host value. diff --git a/crates/sqlbuild-host/src/tests/test_enum_values.rs b/crates/sqlbuild-host/src/tests/test_enum_values.rs index 00773a54d5..1a96bdac08 100644 --- a/crates/sqlbuild-host/src/tests/test_enum_values.rs +++ b/crates/sqlbuild-host/src/tests/test_enum_values.rs @@ -108,7 +108,7 @@ fn given_result_enums_when_read_by_value_then_they_match_the_python_members() { description: "AuditOutcome round-trips its Python values", values: values::(), parsed: parsed::(), - expected_values: vec!["pass", "warn", "error", "insufficient"], + expected_values: vec!["pass", "warn", "error", "insufficient", "skipped"], }, EnumValueTestCase { description: "AuditRunScope round-trips its Python values", diff --git a/crates/sqlbuild-python/src/bindings/_helpers/build_progress/result_text.rs b/crates/sqlbuild-python/src/bindings/_helpers/build_progress/result_text.rs index 049d1d71db..31bbc2b8c5 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/build_progress/result_text.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/build_progress/result_text.rs @@ -94,6 +94,7 @@ pub(crate) fn audit_text(audit: &Bound<'_, PyAny>) -> PyResult measurement: text(audit, "evaluation_mode")? == MEASUREMENT, measured_value: text(audit, "measured_value")?, execution_error: optional(audit, "execution_error")?, + skip_reason: optional(audit, "skip_reason")?, }) } diff --git a/crates/sqlbuild-python/src/bindings/_helpers/compile_driver/plan_result.rs b/crates/sqlbuild-python/src/bindings/_helpers/compile_driver/plan_result.rs index 3dfb61f1cc..ec633262cd 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/compile_driver/plan_result.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/compile_driver/plan_result.rs @@ -10,9 +10,14 @@ use crate::bindings::_helpers::compile_driver::pipeline_entries::build_project_g use crate::bindings::_helpers::compile_driver::python_plan_entries::{ build_python_run_plan_outputs, python_read_source_names_for, }; +use crate::bindings::_helpers::compiled_views::{project_reads, source_reads}; +use crate::bindings::_helpers::graph::keys::{KeyCache, native_key}; use crate::bindings::_helpers::planner_entries::execution_plan::build_execution_plan; use crate::bindings::_helpers::planner_phases::objects::object; use crate::bindings::_helpers::records::construct::new_record_with; +use sqlbuild_analysis::planner_scope::constants::SOURCE; +use std::collections::HashSet; + use pyo3::exceptions::PyException; use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyModule, PyModuleMethods, PyResult, Python}; use pyo3::types::{PyDict, PyDictMethods, PyFrozenSet}; @@ -38,6 +43,50 @@ pub(crate) struct PlanningInputs<'a, 'py> { pub(crate) connection: &'a Bound<'py, PyAny>, } +/// The selected Python nodes and SQL keys when loading is off: loaders out, their sources in. +fn without_loaders<'py>( + (names, sql_keys): (&Bound<'py, PyAny>, &Bound<'py, PyAny>), + (options, discovered, project): (&Bound<'py, PyAny>, &Bound<'py, PyAny>, &Bound<'py, PyAny>), +) -> PyResult<(Bound<'py, PyAny>, Bound<'py, PyAny>)> { + let py: Python<'py> = names.py(); + if native_field(options, "auto_load_sources")?.is_truthy()? { + return Ok((names.clone(), sql_keys.clone())); + } + let mut loaders: HashSet = HashSet::new(); + for loader in native_field(discovered, "loader_functions")?.try_iter()? { + loaders.insert(native_field(&loader?, "name")?.extract()?); + } + let mut runnable: Vec = Vec::new(); + let mut dropped: HashSet = HashSet::new(); + for name in names.try_iter()? { + let name: String = name?.extract()?; + if loaders.contains(&name) { + dropped.insert(name); + } else { + runnable.push(name); + } + } + let mut sources: Vec<(String, String)> = Vec::new(); + for source in project_reads::sources(project)?.try_iter()? { + let source = source?; + let loader: Option = + native_field(&source_reads::source_entry(&source)?, "loader")?.extract()?; + if loader.is_some_and(|loader| dropped.contains(&loader)) { + sources.push((SOURCE.to_owned(), source_reads::name(&source)?.extract()?)); + } + } + let mut selected: Vec<(String, String)> = sql_keys + .try_iter()? + .map(|key| native_key(&key?)) + .collect::>()?; + selected.extend(sources); + let selected_keys = PyFrozenSet::new(py, KeyCache::new(py)?.keys(&selected)?)?; + Ok(( + PyFrozenSet::new(py, runnable)?.into_any(), + selected_keys.into_any(), + )) +} + /// Plan the compiled project over an open connection and attach Python node plans. pub(crate) fn pipeline_result<'py>( inputs: &PlanningInputs<'_, 'py>, @@ -66,8 +115,13 @@ pub(crate) fn pipeline_result<'py>( )?; run_selection = run_selection_from_inputs((select.extract()?, exclude.extract()?), &graph, discovered)?; - selected_sql_keys = native_field(&run_selection, "sql_keys")?; - selected_python_node_names = native_field(&run_selection, "python_node_names")?; + (selected_python_node_names, selected_sql_keys) = without_loaders( + ( + &native_field(&run_selection, "python_node_names")?, + &native_field(&run_selection, "sql_keys")?, + ), + (options, discovered, project), + )?; python_read_source_names = python_read_source_names_for( py, Some(keywords( diff --git a/crates/sqlbuild-python/src/bindings/_helpers/dagster_invocation/results.rs b/crates/sqlbuild-python/src/bindings/_helpers/dagster_invocation/results.rs index 3668e3300e..0ccbc4bfee 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/dagster_invocation/results.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/dagster_invocation/results.rs @@ -28,6 +28,8 @@ const MATERIALIZE: &str = "materialize"; const MATERIALIZATION: &str = "materialization"; const CHECK: &str = "check"; const AUDIT_CHECK_KIND: &str = "audit"; +/// The status of an audit that did not run; Dagster gets no check result for it. +const SKIPPED_CHECK_STATUS: &str = "skipped"; /// DAG nodes upstream first, or in artifact order when edges form a cycle. pub(crate) fn sorted_nodes<'py>(dag: &Bound<'py, PyAny>) -> PyResult>> { @@ -283,6 +285,9 @@ fn check_plans<'py>( ) -> PyResult<(Vec>, HashSet)> { let mut seen: HashSet = seen; let py: Python<'py> = dag.py(); + if get(check, "status")?.eq(SKIPPED_CHECK_STATUS)? { + return Ok((Vec::new(), seen)); + } let dag_check: Option> = dag_check_for(dag, check)?; let (asset_ids, check_name): (Vec, String) = match &dag_check { Some(dag_check) if !check_enabled(dag_check)? => return Ok((Vec::new(), seen)), diff --git a/crates/sqlbuild-python/src/bindings/_helpers/planner_entries/plan_output.rs b/crates/sqlbuild-python/src/bindings/_helpers/planner_entries/plan_output.rs index 9d15d3127d..422562bd55 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/planner_entries/plan_output.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/planner_entries/plan_output.rs @@ -552,6 +552,21 @@ fn planned_sql_tests<'py>( ) } +/// The names of the planned models whose relation the inspected warehouse already holds. +fn existing_model_names<'py>( + model_entries: &Bound<'py, PyAny>, + existing_relations: &Bound<'py, PyAny>, +) -> PyResult> { + let mut names: Vec> = Vec::new(); + for entry in model_entries.try_iter()? { + let name = native_field(&entry?, "name")?; + if existing_relations.contains(&name)? { + names.push(name); + } + } + Ok(PyFrozenSet::new(model_entries.py(), names)?.into_any()) +} + /// The plan output's fields from planned entries and resolved relation inputs. pub(crate) fn plan_output_fields<'py>( inputs: PlanOutputInputs<'py>, @@ -578,15 +593,19 @@ pub(crate) fn plan_output_fields<'py>( let execution_order = native_field(&scope, "execution_order")?; let source_map = native_field(&relations, "source_map")?; let seeds = seed_entries(&project, &snapshot, &selected_keys, &extras)?; - let source_load_entries = object( - py, - build_source_load_entries( - &execution_order, - &selected_keys, - source_map.downcast::()?, - reload_sources.is_truthy()?, - )?, - )?; + let source_load_entries = if native_field(&scope, "load_sources")?.is_truthy()? { + object( + py, + build_source_load_entries( + &execution_order, + &selected_keys, + source_map.downcast::()?, + reload_sources.is_truthy()?, + )?, + )? + } else { + PyTuple::empty(py).into_any() + }; require_intermediate_loader_targets(&project, &snapshot, &source_map, &source_load_entries)?; let functions = function_entries( (&project, &adapter, &snapshot), @@ -665,6 +684,17 @@ pub(crate) fn plan_output_fields<'py>( ), ("selected_keys", selected_keys), ("warnings", tuple_of(py, warnings)?), + ( + "existing_model_names", + existing_model_names( + &model_entries, + &native_field(&snapshot, "existing_relations")?, + )?, + ), + ( + "end_audit_source_keys", + native_field(&scope, "end_audit_source_keys")?, + ), ("upstream_deps", native_field(&scope, "upstream_deps")?), ("downstream_deps", native_field(&scope, "downstream_deps")?), ( diff --git a/crates/sqlbuild-python/src/bindings/_helpers/planner_records/constants.rs b/crates/sqlbuild-python/src/bindings/_helpers/planner_records/constants.rs index 6982428912..1f7d0a8eed 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/planner_records/constants.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/planner_records/constants.rs @@ -51,6 +51,8 @@ const R_PLANNER_SCOPE: RecordSpec = RecordSpec { "sql_test_selection", FieldDefault::NativeCall(RecordId::SqlTestSelection), ), + FieldSpec::new("load_sources", FieldDefault::True), + FieldSpec::new("end_audit_source_keys", FieldDefault::EmptyFrozenset), ], }; const R_SQL_TEST_ASSERTION_STEP: RecordSpec = RecordSpec { @@ -855,6 +857,7 @@ const R_SCENARIO_FIXTURE_PLAN: RecordSpec = RecordSpec { FieldSpec::required("logical_name"), FieldSpec::required("destination"), FieldSpec::required("sql"), + FieldSpec::new("fingerprint_sql", FieldDefault::Text("")), ], }; const R_SCENARIO_EXPECTED_EXPECTATION_PLAN: RecordSpec = RecordSpec { @@ -924,6 +927,8 @@ const R_PLAN_OUTPUT: RecordSpec = RecordSpec { FieldSpec::new("python_identity_fingerprints", FieldDefault::EmptyDict), FieldSpec::new("metadata", FieldDefault::EmptyDict), FieldSpec::new("cursor_overrides", FieldDefault::None), + FieldSpec::new("existing_model_names", FieldDefault::EmptyFrozenset), + FieldSpec::new("end_audit_source_keys", FieldDefault::EmptyFrozenset), ], }; diff --git a/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/constants.rs b/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/constants.rs index 5bbce82b02..230067769f 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/constants.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/constants.rs @@ -16,3 +16,9 @@ pub(super) const SCENARIO_PLAN_UNRESOLVED_RELATION_MARKER: &str = "S511"; pub(super) const SCENARIO_PLAN_EXPECTED_COLUMNS: &str = "S512"; pub(super) const SCENARIO_EXPECTED_COLUMNS_HELP: &str = "list every column the model outputs in the expected CTE, and remove columns it does not output"; pub(super) const SCENARIO_PLAN_INTERNAL: &str = "S599"; +/// The zero-argument cursor intrinsics a model query can call. +pub(super) const CURSOR_INTRINSIC_NAMES: [&str; 2] = ["__cursor_start", "__cursor_end"]; +pub(super) const INCREMENTAL_MODE_KEY: &str = "incremental_mode"; +pub(super) const MICROBATCH_INCREMENTAL_MODE: &str = "microbatch"; +pub(super) const CURSOR_TYPE_KEY: &str = "cursor_type"; +pub(super) const INTEGER_CURSOR_TYPE: &str = "integer"; diff --git a/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/execution.rs b/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/execution.rs index 7328fe60e0..4397d1a1fa 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/execution.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/execution.rs @@ -5,6 +5,7 @@ use crate::bindings::_helpers::boundary::python_calls::keywords; use crate::bindings::_helpers::compiled_views::model_reads; use crate::bindings::_helpers::compiled_views::project_reads; use crate::bindings::_helpers::compiled_views::records::compiled_model_record; +use crate::bindings::_helpers::model_config::typed_config::{config_value, model_values}; use crate::bindings::_helpers::planner_entries::model_entries::plan_model; use crate::bindings::_helpers::planner_entries::models::ModelPlanContext; use crate::bindings::_helpers::planner_identity::graph::build_execution_upstream_deps; @@ -12,7 +13,9 @@ use crate::bindings::_helpers::planner_identity::graph::topologically_order_keys use crate::bindings::_helpers::planner_phases::objects::object; use crate::bindings::_helpers::planner_resolution::locations::build_function_locations; use crate::bindings::_helpers::planner_scenario::constants::{ - SCENARIO_EXPECTED_COLUMNS_HELP, SCENARIO_PLAN_EXPECTED_COLUMNS, SCENARIO_PLAN_INVALID_FIXTURE, + CURSOR_INTRINSIC_NAMES, CURSOR_TYPE_KEY, INCREMENTAL_MODE_KEY, INTEGER_CURSOR_TYPE, + MICROBATCH_INCREMENTAL_MODE, SCENARIO_EXPECTED_COLUMNS_HELP, SCENARIO_PLAN_EXPECTED_COLUMNS, + SCENARIO_PLAN_INVALID_FIXTURE, }; use crate::bindings::_helpers::planner_scenario::helper_scope::scenario_helper_scope; use crate::bindings::_helpers::planner_scenario::{ @@ -30,6 +33,9 @@ use crate::bindings::_helpers::records::fields::field as native_field; use crate::bindings::_helpers::records::fields::mapping_get; use crate::bindings::_helpers::records::fields::replaced_keywords; use crate::bindings::_helpers::records::ids::RecordId; +use crate::bindings::_helpers::sql_test_glue::constants::{ + DEFAULT_INTEGER_END, DEFAULT_INTEGER_START, DEFAULT_TIMESTAMP_END, DEFAULT_TIMESTAMP_START, +}; use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult, Python, pyfunction}; use pyo3::types::{PyBool, PyDict, PyDictMethods, PyFrozenSet, PyTuple}; use sqlbuild_analysis::compiler::main::scenario_expected_column_message::scenario_expected_column_message; @@ -136,6 +142,7 @@ pub(crate) fn build_scenario_execution_plan<'py>( &model_reads::query_sql(&model)?.extract::()?, &context.relations, )?; + let cursor_window = scenario_cursor_window(&model)?; let record = compiled_model_record(&model)?; let changes = PyDict::new(py); changes.set_item("destination", destination)?; @@ -162,10 +169,7 @@ pub(crate) fn build_scenario_execution_plan<'py>( PyBool::new(py, false).to_owned().into_any(), ), ("full_refresh", PyBool::new(py, true).to_owned().into_any()), - ( - "cursor_overrides", - new_record(py, RecordId::CursorOverridePair, (), None)?, - ), + ("cursor_overrides", cursor_window), ], )?), )?, @@ -392,3 +396,33 @@ fn expectations<'py>( PyTuple::new(py, assertions)?.into_any(), )) } + +/// The SQL-test wide window for a non-microbatch model whose SQL reads the cursor intrinsics. +fn scenario_cursor_window<'py>(model: &Bound<'py, PyAny>) -> PyResult> { + let py = model.py(); + let query: String = model_reads::query_sql(model)?.extract()?; + let values = model_values(model)?; + let microbatch: bool = + config_value(py, &values, INCREMENTAL_MODE_KEY)?.eq(MICROBATCH_INCREMENTAL_MODE)?; + if microbatch + || !CURSOR_INTRINSIC_NAMES + .iter() + .any(|name| query.contains(name)) + { + return new_record(py, RecordId::CursorOverridePair, (), None); + } + let integer: bool = config_value(py, &values, CURSOR_TYPE_KEY)?.eq(INTEGER_CURSOR_TYPE)?; + let (start, end) = if integer { + (DEFAULT_INTEGER_START, DEFAULT_INTEGER_END) + } else { + (DEFAULT_TIMESTAMP_START, DEFAULT_TIMESTAMP_END) + }; + new_record_with( + py, + RecordId::CursorOverridePair, + &[ + ("start_cursor_override", string(py, start)), + ("end_cursor_override", string(py, end)), + ], + ) +} diff --git a/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/fixtures.rs b/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/fixtures.rs index d4342bc52e..88c31ae3be 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/fixtures.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/planner_scenario/fixtures.rs @@ -12,9 +12,16 @@ use crate::bindings::_helpers::records::construct::{enum_from_value, new_record_ use crate::bindings::_helpers::records::fields::field as native_field; use crate::bindings::_helpers::records::fields::mapping_get_or; use crate::bindings::_helpers::records::ids::RecordId; +use crate::bindings::_helpers::sqltext::lexical_syntax::LexicalSyntaxInput; use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult, Python, pyfunction}; use pyo3::types::{PyDict, PyDictMethods, PyTuple}; +use sqlbuild_analysis::compiler::main::scenario_reader_ctes::scenario_reader_ctes; use sqlbuild_analysis::compiler::main::scenario_reader_sql::scenario_reader_sql; +use sqlbuild_analysis::compiler::models::ScenarioHelperScope; +use sqlbuild_sqltext::sql_references::main::extract_sql_references::extract_sql_references; +use sqlbuild_sqltext::sql_references::models::ReferenceExtraction; +use sqlbuild_sqltext::sql_scan::models::LexicalSyntax; +use std::collections::{BTreeMap, BTreeSet}; pub(super) fn extract_fixtures<'py>( scenario: &Bound<'py, PyAny>, @@ -60,7 +67,7 @@ pub(super) fn fixture_plans<'py>( for name in native_field(graph, names)?.try_iter()? { let name: String = name?.extract()?; let target = required_target(&context.relations, locations, &name, kind)?; - let Some(mut sql) = fixtures.get_item(&name)? else { + let Some(authored) = fixtures.get_item(&name)? else { return Err(error( py, &format!("Scenario is missing {kind} fixture SQL '{name}'"), @@ -68,6 +75,7 @@ pub(super) fn fixture_plans<'py>( None, )?); }; + let mut sql = authored.clone(); if kind != super::constants::DBT_REF && !overrides.is_none() { sql = mapping_get_or(overrides, (resource, &name), sql)?; } @@ -76,6 +84,8 @@ pub(super) fn fixture_plans<'py>( &scenario_reader_sql(&scope, &sql.extract::()?, false), true, )?; + let fingerprint: String = + fingerprint_sql(scenario, (&scope, context), &authored.extract::()?)?; result.push(new_record_with( py, RecordId::ScenarioFixturePlan, @@ -87,6 +97,7 @@ pub(super) fn fixture_plans<'py>( ("logical_name", string(py, &name)), ("destination", target), ("sql", string(py, &sql)), + ("fingerprint_sql", string(py, &fingerprint)), ], )?); } @@ -94,6 +105,55 @@ pub(super) fn fixture_plans<'py>( PyTuple::new(py, result) } +/// The authored fixture, its helper CTEs, transitively, and each reference's fixture-or-relation target. +fn fingerprint_sql( + scenario: &Bound<'_, PyAny>, + (scope, context): (&ScenarioHelperScope, &SqlContext<'_>), + authored: &str, +) -> PyResult { + let mut bodies: BTreeMap = BTreeMap::new(); + for cte in native_field(scenario, "authored_ctes")?.try_iter()? { + let cte = cte?; + bodies.insert( + native_field(&cte, "name")?.extract()?, + native_field(&cte, "sql_body")?.extract()?, + ); + } + let mut text: String = authored.to_owned(); + let mut read: Vec = vec![authored.to_owned()]; + for (name, _) in scenario_reader_ctes(scope, authored, false) { + let body: &str = bodies.get(&name).map_or("", String::as_str); + text.push_str(&format!("\n-- helper {name}\n{body}")); + read.push(body.to_owned()); + } + let syntax: LexicalSyntax = native_field(&context.syntax, "native_mapping")? + .extract::()? + .into(); + let mut references: BTreeSet = BTreeSet::new(); + for sql in &read { + let ReferenceExtraction::Extracted(scan) = extract_sql_references(sql, &syntax) else { + continue; + }; + for reference in scan.references { + let package: &str = reference.package.as_deref().unwrap_or(""); + let target: &str = + if bodies.contains_key(&format!("__{}__{}", reference.kind, reference.name)) { + "fixture" + } else { + "relation" + }; + references.insert(format!( + "{}:{package}:{} -> {target}", + reference.kind, reference.name + )); + } + } + for reference in references { + text.push_str(&format!("\n-- reads {reference}")); + } + Ok(text) +} + #[pyfunction] #[pyo3(signature = (**kwargs))] pub(crate) fn build_scenario_fixture_plans<'py>( diff --git a/crates/sqlbuild-python/src/bindings/_helpers/planner_scope/scope.rs b/crates/sqlbuild-python/src/bindings/_helpers/planner_scope/scope.rs index 83438799cc..16797c3c0b 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/planner_scope/scope.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/planner_scope/scope.rs @@ -21,7 +21,7 @@ use crate::bindings::_helpers::records::fields::mapping_items; use crate::bindings::_helpers::records::ids::RecordId; use pyo3::prelude::{Bound, PyAny, PyAnyMethods, PyResult, Python}; use pyo3::types::PyDictMethods; -use pyo3::types::{PyDict, PyFrozenSet, PyTuple}; +use pyo3::types::{PyBool, PyDict, PyFrozenSet, PyTuple}; use sqlbuild_analysis::planner_scope::constants::SOURCE; use sqlbuild_analysis::planner_scope::main::{ managed_sources::managed_sources, source_selectors::source_selectors, @@ -94,14 +94,18 @@ pub(crate) fn build_planner_scope<'py>( let carrier = Carrier::new(py, &texts, || { Ok(names.iter().map(|(name, _)| name.clone()).collect()) })?; - let source_keys = source_selectors(&carrier.send_all(&texts)?, &names, &keys(&selected)?) + let sent = carrier.send_all(&texts)?; + let selected_names = keys(&selected)?; + let source_keys = source_selectors(&sent, &names, &selected_names, true) + .map_err(|error| carrier.error(error))?; + let named_sources = source_selectors(&sent, &names, &selected_names, false) .map_err(|error| carrier.error(error))?; let cache = KeyCache::new(py)?; - let mut executable = PyFrozenSet::new( - py, - cache.keys(&source_keys.into_iter().collect::>())?, - )? - .into_any(); + let explicit_sources: Vec<(String, String)> = source_keys.into_iter().collect(); + let mut executable = PyFrozenSet::new(py, cache.keys(&explicit_sources)?)?.into_any(); + if !auto_load { + executable = PyFrozenSet::empty(py)?.into_any(); + } if auto_load { let loaded = managed_source_upstream_keys(&selected, &upstream, &project)?; selected = selected.call_method1("union", (&loaded,))?; @@ -115,6 +119,11 @@ pub(crate) fn build_planner_scope<'py>( upstream = deps.into_any(); downstream = build_downstream_deps(&upstream)?.into_any(); } + let end_audit_sources: Vec<(String, String)> = keys(&selected)? + .into_iter() + .filter(|key| key.0 == SOURCE && (!auto_load || named_sources.contains(key))) + .collect(); + let end_audit_sources = PyFrozenSet::new(py, cache.keys(&end_audit_sources)?)?.into_any(); let origins = build_execution_edge_origins(&project)?.into_any(); let order = topologically_order_keys(&upstream, Some(origins))?.into_any(); let models = PyDict::new(py); @@ -143,6 +152,11 @@ pub(crate) fn build_planner_scope<'py>( ("user_selected_keys", selected), ("python_read_source_names", reads), ("sql_test_selection", tests), + ( + "load_sources", + PyBool::new(py, auto_load).to_owned().into_any(), + ), + ("end_audit_source_keys", end_audit_sources), ], ) } diff --git a/crates/sqlbuild-python/src/bindings/_helpers/planner_strategy/fixture_metadata.rs b/crates/sqlbuild-python/src/bindings/_helpers/planner_strategy/fixture_metadata.rs index c488412173..3d0bf2cf0c 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/planner_strategy/fixture_metadata.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/planner_strategy/fixture_metadata.rs @@ -34,10 +34,9 @@ pub(crate) fn model_fixture_relation(model: &Bound<'_, PyAny>) -> PyResult( )); entries.push(("evidence_count", evidence_count)); entries.push(("evidence_truncated", evidence_truncated)); - entries.push(("execution_error", native_field(&result, "execution_error")?)); + entries.extend(fields(&result, &["execution_error", "skip_reason"])?); checks.push(without_none(py, entries)?); } Ok(checks) diff --git a/crates/sqlbuild-python/src/bindings/_helpers/rules/lint_pipeline.rs b/crates/sqlbuild-python/src/bindings/_helpers/rules/lint_pipeline.rs index 1eac80974b..e3317161d0 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/rules/lint_pipeline.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/rules/lint_pipeline.rs @@ -29,6 +29,7 @@ use sqlbuild_rules::lint_text::main::named_declaration_role_kind::named_declarat use sqlbuild_rules::lint_text::main::neutralize_audit_parameters::neutralize_audit_parameters; use sqlbuild_rules::lint_text::main::neutralize_context_interpolation::neutralize_context_interpolation; use sqlbuild_rules::lint_text::main::neutralize_sql_macro_parameters::neutralize_sql_macro_parameters; +use sqlbuild_rules::lint_text::main::neutralize_test_parameters::neutralize_test_parameters; use sqlbuild_rules::lint_text::main::reject_inline_suppressions::reject_inline_suppressions; use sqlbuild_rules::lint_text::models::{ AuthoredDiagnostic, ExpansionSpan, FramedExpression, InterpolationSite, LintSqlBody, @@ -64,6 +65,7 @@ const SINGULAR_AUDIT_KIND: &str = "singular_audit"; const SQL_HOOK_KIND: &str = "sql_hook"; const CEREMONIAL_KINDS: [&str; 2] = ["TEST", "SCENARIO"]; const SQL_MACRO_KIND: &str = "MACRO"; +const SQL_TEST_KIND: &str = "TEST"; const INTERPOLATION_MARKERS: [&str; 2] = ["@", "${"]; const FAULT_SEVERITY: &str = "FAULT"; const WARNING_SEVERITY: &str = "WARNING"; @@ -752,8 +754,11 @@ fn plan_bodies( if ranges.is_empty() { ranges = sql_body_ranges(text, headers); } + let sql_test: bool = headers.iter().any(|header| header.kind == SQL_TEST_KIND); let neutralize: Option Neutralized> = if sql_macro { Some(neutralize_sql_macro_parameters) + } else if sql_test { + Some(neutralize_test_parameters) } else { neutralizer(&BodyRole { kind, diff --git a/crates/sqlbuild-python/src/bindings/_helpers/semantic_shapes/source_bindings.rs b/crates/sqlbuild-python/src/bindings/_helpers/semantic_shapes/source_bindings.rs index bd4ad69b31..327c4bb470 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/semantic_shapes/source_bindings.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/semantic_shapes/source_bindings.rs @@ -37,6 +37,8 @@ const UNKNOWN: &str = "UNKNOWN"; const GENERIC_DIALECT: &str = "generic"; /// Dialects whose unquoted identifiers already bind with their exact case. const EXACT_UNQUOTED_DIALECTS: [&str; 1] = ["clickhouse"]; +/// The dialect whose unquoted identifiers resolve as upper case; the adapter lists them lowercased. +const UPPERCASE_FOLDING_DIALECT: &str = "snowflake"; type Object<'py> = Bound<'py, PyAny>; @@ -84,6 +86,8 @@ pub(crate) fn source_binding_diagnostics<'py>( && !ignore_case.is_truthy()?; let exact_unquoted: bool = !dialect.is_none() && EXACT_UNQUOTED_DIALECTS.contains(&dialect.extract::()?.as_str()); + let folds_upper: bool = + !dialect.is_none() && dialect.extract::()?.as_str() == UPPERCASE_FOLDING_DIALECT; let mut changed: HashSet = HashSet::new(); for item in mapping_items(&columns)?.try_iter()? { let (name, value): (String, Object<'py>) = item?.extract()?; @@ -105,7 +109,7 @@ pub(crate) fn source_binding_diagnostics<'py>( } let quoted: Bound<'py, PyDict> = PyDict::new(py); for (column, column_type) in physical { - let name: String = if exact_unquoted { + let name: String = if exact_unquoted || (folds_upper && is_folded_identifier(&column)) { column } else { format!("\"{}\"", column.replace('"', "\"\"")) @@ -451,6 +455,17 @@ fn placeholders<'py>(model: &Object<'py>) -> PyResult> { Ok(placeholders.into_any()) } +/// Whether a listed column is a plain identifier in one case, so it binds as an unquoted name. +fn is_folded_identifier(name: &str) -> bool { + let mut chars = name.chars(); + chars + .next() + .is_some_and(|first| first.is_ascii_alphabetic() || first == '_') + && chars.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '$') + && (!name.chars().any(|c| c.is_ascii_uppercase()) + || !name.chars().any(|c| c.is_ascii_lowercase())) +} + fn ignores_quoted_case<'py>(project: &Object<'py>, dialect: &Object<'py>) -> PyResult> { let ignores: bool = binding_catalog::ignores_quoted_case( &native_field(project, "effective_connection")?, diff --git a/crates/sqlbuild-python/src/bindings/_helpers/sql_test_glue/constants.rs b/crates/sqlbuild-python/src/bindings/_helpers/sql_test_glue/constants.rs index 019c313c3b..7921a369a4 100644 --- a/crates/sqlbuild-python/src/bindings/_helpers/sql_test_glue/constants.rs +++ b/crates/sqlbuild-python/src/bindings/_helpers/sql_test_glue/constants.rs @@ -12,10 +12,10 @@ pub(super) const FIXTURE_PREFIXES: [(&str, &str); 4] = [ pub(super) const CURSOR_ALGEBRA_EXCEPTIONS: &str = "sqlbuild.cursor_algebra.exceptions"; pub(super) const INTEGER_CURSOR_TYPE: &str = "integer"; pub(super) const TIMESTAMP_CURSOR_TYPE: &str = "timestamp"; -pub(super) const DEFAULT_TIMESTAMP_START: &str = "1900-01-01 00:00:00"; -pub(super) const DEFAULT_TIMESTAMP_END: &str = "2999-12-31 00:00:00"; -pub(super) const DEFAULT_INTEGER_START: &str = "-1000000000000000"; -pub(super) const DEFAULT_INTEGER_END: &str = "1000000000000000"; +pub(crate) const DEFAULT_TIMESTAMP_START: &str = "1900-01-01 00:00:00"; +pub(crate) const DEFAULT_TIMESTAMP_END: &str = "2999-12-31 00:00:00"; +pub(crate) const DEFAULT_INTEGER_START: &str = "-1000000000000000"; +pub(crate) const DEFAULT_INTEGER_END: &str = "1000000000000000"; pub(super) const CURSOR_START_KEY: &str = "cursor_start"; pub(super) const CURSOR_END_KEY: &str = "cursor_end"; pub(super) const SENTINEL: &str = "__SQLBUILD_CALL_SUFFIX__"; diff --git a/crates/sqlbuild-rules/src/dsl_headers/_helpers/formatting.rs b/crates/sqlbuild-rules/src/dsl_headers/_helpers/formatting.rs index 253cc699bb..ffe374ebda 100644 --- a/crates/sqlbuild-rules/src/dsl_headers/_helpers/formatting.rs +++ b/crates/sqlbuild-rules/src/dsl_headers/_helpers/formatting.rs @@ -6,13 +6,14 @@ use sqlbuild_core::text::main::python_strip::python_strip; use crate::dsl_headers::_helpers::audits::wrap_header; use sqlbuild_sqltext::header_strings::constants::TRIPLE_QUOTE; use sqlbuild_sqltext::header_strings::main::fold_prose::fold_prose; +use sqlbuild_sqltext::header_strings::main::follows_description_key::follows_description_key; use sqlbuild_sqltext::header_strings::main::prose_string::prose_string; use crate::dsl_headers::_helpers::macro_header::same_macro; use crate::dsl_headers::_helpers::prose_layout::{ canonical_prose, closing_width, description_string_span, }; -use crate::dsl_headers::_helpers::quoted::split_outside_quotes; +use crate::dsl_headers::_helpers::quoted::{description_value_spans, split_outside_quotes}; use crate::dsl_headers::_helpers::rules::violation_at; use crate::dsl_headers::_helpers::text::{ chars, find, line_count, line_starts, lstrip, rfind_before, rstrip, slice, string, tail, @@ -184,19 +185,55 @@ pub(crate) fn wrap_descriptions( Ok(updated) } -/// `header_text` with its description laid out canonically, or `None` when nothing changes. -fn laid_out_description( +/// Every column and audit description of `headers` of the `kinds` in its canonical layout. +pub(crate) fn wrap_nested_descriptions( + text: &[char], + headers: &[HeaderSpan], + (kinds, line_width): (&[&str], i64), + parser: &P, +) -> Result, P::Error> { + let mut updated: Vec = text.to_vec(); + for header in headers.iter().rev() { + if !kinds.contains(&header.kind.as_str()) { + continue; + } + let mut header_text: Vec = slice(&updated, header.start, header.end).to_vec(); + let nested: Vec = description_value_spans(&header_text) + .into_iter() + .filter(|(_, depth)| *depth > 1) + .map(|((value_start, _), _)| value_start - 1) + .collect(); + for start in nested.into_iter().rev() { + if let Some(rewritten) = + laid_out_string(&header.kind, &header_text, start, (line_width, parser))? + { + header_text = rewritten; + } + } + updated = [ + slice(&updated, 0, header.start), + &header_text, + tail(&updated, header.end), + ] + .concat(); + } + Ok(updated) +} + +/// `header_text` with the prose string at `start` laid out canonically, or `None` if unchanged. +fn laid_out_string( kind: &str, header_text: &[char], - line_width: i64, - parser: &P, + start: usize, + (line_width, parser): (i64, &P), ) -> Result>, P::Error> { - let Some((start, end)) = description_string_span(header_text) else { + if !follows_description_key(header_text, start) { return Ok(None); - }; + } let Ok(prose) = prose_string(header_text, start) else { return Ok(None); }; + let end: usize = prose.end; if header_text[start] == '\'' && keeps_single_quotes(header_text, (start, end), line_width) { return Ok(None); } @@ -219,6 +256,19 @@ fn laid_out_description( Ok(Some(rewritten)) } +/// `header_text` with its description laid out canonically, or `None` when nothing changes. +fn laid_out_description( + kind: &str, + header_text: &[char], + line_width: i64, + parser: &P, +) -> Result>, P::Error> { + let Some((start, _)) = description_string_span(header_text) else { + return Ok(None); + }; + laid_out_string(kind, header_text, start, (line_width, parser)) +} + /// Whether a `'...'` description stays as written: on one line that fits, with no double quote. fn keeps_single_quotes( header_text: &[char], diff --git a/crates/sqlbuild-rules/src/dsl_headers/_helpers/quoted.rs b/crates/sqlbuild-rules/src/dsl_headers/_helpers/quoted.rs index 6a17299ee4..0f12b9d3a2 100644 --- a/crates/sqlbuild-rules/src/dsl_headers/_helpers/quoted.rs +++ b/crates/sqlbuild-rules/src/dsl_headers/_helpers/quoted.rs @@ -15,6 +15,15 @@ pub(crate) fn quoted_value_end(text: &[char], start: usize) -> usize { /// The span of the top-level `description` string value inside the header parentheses. pub(crate) fn description_value_span(text: &[char]) -> Option<(usize, usize)> { + description_value_spans(text) + .into_iter() + .find(|(_, depth)| *depth == 1) + .map(|(span, _)| span) +} + +/// The span and bracket depth of every `description` string value, the header's own at depth 1. +pub(crate) fn description_value_spans(text: &[char]) -> Vec<((usize, usize), usize)> { + let mut spans: Vec<((usize, usize), usize)> = Vec::new(); let mut delimiters: Vec = Vec::new(); let mut index: usize = 0; while index < text.len() { @@ -39,7 +48,7 @@ pub(crate) fn description_value_span(text: &[char]) -> Option<(usize, usize)> { index += 1; continue; } - if delimiters == [')'] && (is_alpha(character) || character == '_') { + if !delimiters.is_empty() && (is_alpha(character) || character == '_') { let mut word_end: usize = index + 1; while word_end < text.len() && (python_isalnum(text[word_end]) || text[word_end] == '_') { @@ -52,7 +61,9 @@ pub(crate) fn description_value_span(text: &[char]) -> Option<(usize, usize)> { } if quote_start < text.len() && QUOTES.contains(&text[quote_start]) { let quote_end: usize = quoted_value_end(text, quote_start); - return Some((quote_start + 1, quote_end - 1)); + spans.push(((quote_start + 1, quote_end - 1), delimiters.len())); + index = quote_end; + continue; } } index = word_end; @@ -60,7 +71,7 @@ pub(crate) fn description_value_span(text: &[char]) -> Option<(usize, usize)> { } index += 1; } - None + spans } /// `text` split on newlines that are outside quoted values. diff --git a/crates/sqlbuild-rules/src/dsl_headers/main/format_headers.rs b/crates/sqlbuild-rules/src/dsl_headers/main/format_headers.rs index 00fa69e00a..7226b3cd42 100644 --- a/crates/sqlbuild-rules/src/dsl_headers/main/format_headers.rs +++ b/crates/sqlbuild-rules/src/dsl_headers/main/format_headers.rs @@ -2,6 +2,7 @@ use crate::dsl_headers::_helpers::formatting::{ relocate_leading_comment, without_trailing_whitespace, wrap_audits, wrap_descriptions, + wrap_nested_descriptions, }; use crate::dsl_headers::_helpers::macro_header::changes_macro; use crate::dsl_headers::_helpers::macro_header::layout_macro_headers; @@ -30,7 +31,14 @@ pub fn format_headers( (&PROSE_LAYOUT_HEADER_KINDS, config.line_width), parser, )?; - let audited: Vec = wrap_audits(&wrapped, config.line_width, parser)?; + let audits_wrapped: Vec = wrap_audits(&wrapped, config.line_width, parser)?; + let audits_wrapped_headers: Vec = scan_headers(&audits_wrapped, false); + let audited: Vec = wrap_nested_descriptions( + &audits_wrapped, + &audits_wrapped_headers, + (&PROSE_LAYOUT_HEADER_KINDS, config.line_width), + parser, + )?; let macro_headers: Vec = scan_headers(&audited, false); let mut laid_out: Vec = layout_macro_headers(&audited, ¯o_headers, config.line_width); let defines_macro: bool = macro_headers diff --git a/crates/sqlbuild-rules/src/engine/_helpers/project_rows.rs b/crates/sqlbuild-rules/src/engine/_helpers/project_rows.rs index afe6dbf8d4..2370e9df16 100644 --- a/crates/sqlbuild-rules/src/engine/_helpers/project_rows.rs +++ b/crates/sqlbuild-rules/src/engine/_helpers/project_rows.rs @@ -24,6 +24,11 @@ pub(crate) fn model_row(facts: &CompiledModelFacts) -> Result } else { facts.authored_query_sql.to_string() }, + query_line_offset: query_line_offset( + &facts.authored_sql, + &facts.authored_query_sql, + &facts.query_sql, + ), config: facts.config.values.clone(), authored_config_keys: facts.config.header_keys.clone(), logical_schema: facts.config.layer_schema.clone(), @@ -101,3 +106,18 @@ pub(crate) fn declaration(fact: &DeclarationFact) -> Declaration { render_as: fact.render_as.clone(), } } + +/// The lines of `file` above `query`, the rendered form of its `authored_query`; 0 if unknown. +fn query_line_offset(file: &str, authored_query: &str, query: &str) -> u64 { + let leading_lines = |text: &str| { + let body: &str = text.trim_start(); + text[..text.len() - body.len()].matches('\n').count() + }; + if authored_query.is_empty() { + return 0; + } + file.rfind(authored_query).map_or(0, |start| { + (file[..start].matches('\n').count() + leading_lines(authored_query)) + .saturating_sub(leading_lines(query)) as u64 + }) +} diff --git a/crates/sqlbuild-rules/src/engine/_helpers/retained.rs b/crates/sqlbuild-rules/src/engine/_helpers/retained.rs index f0b485612c..bb56b7b76c 100644 --- a/crates/sqlbuild-rules/src/engine/_helpers/retained.rs +++ b/crates/sqlbuild-rules/src/engine/_helpers/retained.rs @@ -132,6 +132,7 @@ impl RowWriter { relative_path, query_sql: _, authored_sql: _, + query_line_offset: _, typed_calls: _, config, authored_config_keys, @@ -349,6 +350,7 @@ impl RowReader<'_> { relative_path: self.text()?, query_sql: String::new(), authored_sql: String::new(), + query_line_offset: 0, typed_calls: Vec::new(), config: { let count: usize = self.count()?; diff --git a/crates/sqlbuild-rules/src/engine/_helpers/rows.rs b/crates/sqlbuild-rules/src/engine/_helpers/rows.rs index 4a94856b59..8cf1544beb 100644 --- a/crates/sqlbuild-rules/src/engine/_helpers/rows.rs +++ b/crates/sqlbuild-rules/src/engine/_helpers/rows.rs @@ -83,6 +83,7 @@ fn build_model(row: ModelRow, inputs: &ModelInputs<'_, '_>) -> Result Model { enum_declarations: vec![declaration.clone()], constant_declarations: vec![declaration], typed_calls: Vec::new(), + query_line_offset: 0, declared_audit_count: 3, targeting_test_count: 4, empty_input_only_test_count: 0, diff --git a/crates/sqlbuild-rules/src/engine/tests/test_evaluation.rs b/crates/sqlbuild-rules/src/engine/tests/test_evaluation.rs index f974029eee..cebbf4fb43 100644 --- a/crates/sqlbuild-rules/src/engine/tests/test_evaluation.rs +++ b/crates/sqlbuild-rules/src/engine/tests/test_evaluation.rs @@ -715,7 +715,7 @@ fn given_enforced_contract_outputs_when_evaluating_explicit_type_rule_then_retur columns: json!([{"name": "total", "type": "INTEGER", "type_proven": true}]), contract: "enforced", expected_fault_count: 1, - expected_message_fragment: "outer cast that does not target", + expected_message_fragment: "which is not its declared contract type", }, test_types::ExplicitOutputTypeTestCase { description: "unproven direct passthrough", @@ -1227,7 +1227,7 @@ fn given_model_sql_when_stubbing_shape_then_offsets_map_back_to_source_positions .sql .find(test_case.expected_fragment) .ok_or("fragment missing from the model SQL")?; - let source = ModelSource::new(test_case.sql, &shape.offsets); + let source = ModelSource::new(test_case.sql, &shape.offsets, 0); let end = normalized_start + test_case.expected_fragment.chars().count(); assert_eq!( source.text(normalized_start, end), diff --git a/crates/sqlbuild-rules/src/engine/tests/test_query_facts.rs b/crates/sqlbuild-rules/src/engine/tests/test_query_facts.rs index 9c73c0ffc0..b03abed424 100644 --- a/crates/sqlbuild-rules/src/engine/tests/test_query_facts.rs +++ b/crates/sqlbuild-rules/src/engine/tests/test_query_facts.rs @@ -131,7 +131,12 @@ fn given_dialect_sql_when_reading_query_facts_then_findings_match_the_authored_s select: OUTPUT_TYPE_RULE, enum_columns: &[], columns: json!([{"name": "amount", "type": "INTEGER", "type_proven": true}]), - expected_faults: &[("SQBRCONTRACT105", 3, 13, "outer cast that does not target")], + expected_faults: &[( + "SQBRCONTRACT105", + 3, + 8, + "which is not its declared contract type", + )], }, QueryFactsTestCase { description: "nested parenthesized set operations keep each branch position", @@ -154,7 +159,12 @@ fn given_dialect_sql_when_reading_query_facts_then_findings_match_the_authored_s select: OUTPUT_TYPE_RULE, enum_columns: &[], columns: json!([{"name": "amount", "type": "INTEGER", "type_proven": true}]), - expected_faults: &[("SQBRCONTRACT105", 3, 13, "outer cast that does not target")], + expected_faults: &[( + "SQBRCONTRACT105", + 3, + 8, + "which is not its declared contract type", + )], }, QueryFactsTestCase { description: "unaliased union branches inside a final CTE keep their positions", diff --git a/crates/sqlbuild-rules/src/engine/tests/test_sql_test_rules.rs b/crates/sqlbuild-rules/src/engine/tests/test_sql_test_rules.rs index ac3598f03d..963d696cc3 100644 --- a/crates/sqlbuild-rules/src/engine/tests/test_sql_test_rules.rs +++ b/crates/sqlbuild-rules/src/engine/tests/test_sql_test_rules.rs @@ -75,6 +75,17 @@ fn given_sql_test_facts_when_evaluating_project_rules_then_returns_expected_faul "source_digest": "digest" }} })); + let first_edge_case = helpers::sql_test_fact( + "tests/unit/test_stg_orders__edge_cases.sql", + Some("stg_orders__keeps_paid_orders"), + json!(["stg_orders"]), + ); + let mut second_edge_case = helpers::sql_test_fact( + "tests/unit/test_stg_orders__edge_cases.sql", + Some("stg_orders__excludes_cancelled_orders"), + json!(["stg_orders"]), + ); + second_edge_case["block_index"] = json!(2); let test_cases = [ test_types::SqlTestRulesTestCase { description: "canonical roots run without a model anchor", @@ -125,6 +136,17 @@ fn given_sql_test_facts_when_evaluating_project_rules_then_returns_expected_faul expected_evaluated_models: 0, expected_paths: &["tests/scenarios/daily_revenue.sql", "tests/unit/orders.sql"], }, + test_types::SqlTestRulesTestCase { + description: "a file of several test blocks needs only the filename grammar", + code: "SQBRTEST102", + tests: json!([first_edge_case, second_edge_case]), + scenarios: json!([]), + scope_index: scope.clone(), + config: json!({}), + expected_fault_count: 0, + expected_evaluated_models: 0, + expected_paths: &[], + }, test_types::SqlTestRulesTestCase { description: "resolved ownership uses LCA, pipeline, and direct resource paths", code: "SQBRTEST103", diff --git a/crates/sqlbuild-rules/src/lint_text/constants.rs b/crates/sqlbuild-rules/src/lint_text/constants.rs index 47fc5ec436..6bb802f943 100644 --- a/crates/sqlbuild-rules/src/lint_text/constants.rs +++ b/crates/sqlbuild-rules/src/lint_text/constants.rs @@ -7,6 +7,8 @@ pub(crate) const AUDIT_PARAMETER_SENTINEL_PREFIX: &str = "__sqlbuild_audit_param pub(crate) const CONTEXT_SENTINEL_PREFIX: &str = "__sqlbuild_context_parameter_"; pub(crate) const SQL_MACRO_PARAMETER_SENTINEL_PREFIX: &str = "__sqlbuild_macro_parameter_"; pub(crate) const CONTEXT_INTERPOLATION_TOKEN: &str = "@@CTX:"; +pub(crate) const TEST_PARAMETER_SENTINEL_PREFIX: &str = "__sqlbuild_test_parameter_"; +pub(crate) const TEST_PARAMETER_TOKEN: &str = "@param"; /// Every SQLBuild intrinsic call protected while formatting. pub(crate) const FUNCTION_NAMES: [&str; 9] = [ diff --git a/crates/sqlbuild-rules/src/lint_text/main/mod.rs b/crates/sqlbuild-rules/src/lint_text/main/mod.rs index 38b4423133..61981d45cf 100644 --- a/crates/sqlbuild-rules/src/lint_text/main/mod.rs +++ b/crates/sqlbuild-rules/src/lint_text/main/mod.rs @@ -9,6 +9,7 @@ pub mod neutralize_audit_parameters; pub mod neutralize_context_interpolation; pub mod neutralize_interpolation; pub mod neutralize_sql_macro_parameters; +pub mod neutralize_test_parameters; pub mod reject_inline_suppressions; pub mod resolve_format_settings; pub mod restore_interpolation; diff --git a/crates/sqlbuild-rules/src/lint_text/main/neutralize_test_parameters.rs b/crates/sqlbuild-rules/src/lint_text/main/neutralize_test_parameters.rs new file mode 100644 index 0000000000..fdea44930a --- /dev/null +++ b/crates/sqlbuild-rules/src/lint_text/main/neutralize_test_parameters.rs @@ -0,0 +1,40 @@ +//! `neutralize_test_parameters`: SQL test `@param("name")` references replaced by sentinels. + +use crate::lint_text::_helpers::sites::{ByteSite, assemble, sentinel}; +use crate::lint_text::constants::{TEST_PARAMETER_SENTINEL_PREFIX, TEST_PARAMETER_TOKEN}; +use crate::lint_text::models::Neutralized; + +/// Replace each `@param("name")` with a string sentinel, as every case supplies a literal there. +pub fn neutralize_test_parameters(body: &str) -> Neutralized { + let mut byte_sites: Vec = Vec::new(); + let mut searched_to: usize = 0; + while let Some(found) = body[searched_to..].find(TEST_PARAMETER_TOKEN) { + let start: usize = searched_to + found; + searched_to = start + TEST_PARAMETER_TOKEN.len(); + let Some(end) = reference_end(body, searched_to) else { + continue; + }; + let identifier: String = sentinel(TEST_PARAMETER_SENTINEL_PREFIX, byte_sites.len()); + byte_sites.push((start, end, format!("'{identifier}'"))); + searched_to = end; + } + assemble(body, byte_sites) +} + +/// The end of `("name")` after `@param` at `start`, spaces allowed around its parts. +fn reference_end(body: &str, start: usize) -> Option { + let bytes: &[u8] = body.as_bytes(); + let skip_spaces = |mut index: usize| { + while bytes.get(index).is_some_and(u8::is_ascii_whitespace) { + index += 1; + } + index + }; + let mut index: usize = skip_spaces(start); + (bytes.get(index) == Some(&b'(')).then_some(())?; + index = skip_spaces(index + 1); + (bytes.get(index) == Some(&b'"')).then_some(())?; + index += 1 + body[index + 1..].find('"')?; + index = skip_spaces(index + 1); + (bytes.get(index) == Some(&b')')).then_some(index + 1) +} diff --git a/crates/sqlbuild-rules/src/lint_text/tests/test_lint_text.rs b/crates/sqlbuild-rules/src/lint_text/tests/test_lint_text.rs index 0b16c25b2e..ebadb2145c 100644 --- a/crates/sqlbuild-rules/src/lint_text/tests/test_lint_text.rs +++ b/crates/sqlbuild-rules/src/lint_text/tests/test_lint_text.rs @@ -10,6 +10,7 @@ use crate::lint_text::main::neutralize_audit_parameters::neutralize_audit_parame use crate::lint_text::main::neutralize_context_interpolation::neutralize_context_interpolation; use crate::lint_text::main::neutralize_interpolation::neutralize_interpolation; use crate::lint_text::main::neutralize_sql_macro_parameters::neutralize_sql_macro_parameters; +use crate::lint_text::main::neutralize_test_parameters::neutralize_test_parameters; use crate::lint_text::main::reject_inline_suppressions::reject_inline_suppressions; use crate::lint_text::main::restore_interpolation::restore_interpolation; use crate::lint_text::models::{ExpansionSpan, FormatFile, LintSqlBody, LintSqlOptions}; @@ -158,6 +159,12 @@ fn given_generated_parameters_when_neutralizing_then_only_bare_parameters_are_re neutralize: neutralize_sql_macro_parameters, expected_text: "@sql.other(__sqlbuild_macro_parameter_0__) + @enum(\"s\").A + LENGTH('@v') + @@x", }, + ParameterTestCase { + description: "SQL test parameters, spaced or not, but not a malformed reference", + body: "SELECT @param(\"a\") AS a, @param ( \"b\" ) AS b, @param(c) AS c", + neutralize: neutralize_test_parameters, + expected_text: "SELECT '__sqlbuild_test_parameter_0__' AS a, '__sqlbuild_test_parameter_1__' AS b, @param(c) AS c", + }, ParameterTestCase { description: "context interpolation in hooks", body: "analyze @@CTX:relation; select @@CTX:a.b", diff --git a/crates/sqlbuild-rules/src/models.rs b/crates/sqlbuild-rules/src/models.rs index f045e25276..8f3a910df5 100644 --- a/crates/sqlbuild-rules/src/models.rs +++ b/crates/sqlbuild-rules/src/models.rs @@ -559,6 +559,9 @@ pub struct Model { pub relative_path: String, pub query_sql: String, pub authored_sql: String, + /// The lines of the model file above `query_sql`, which locations inside it skip. + #[serde(default)] + pub query_line_offset: u64, pub config: BTreeMap, pub authored_config_keys: Vec, pub logical_schema: Option, @@ -870,6 +873,8 @@ pub struct ModelRow { pub relative_path: String, pub query_sql: String, pub authored_sql: String, + /// The lines of the model file above `query_sql`. + pub query_line_offset: u64, pub config: BTreeMap, pub authored_config_keys: Vec, pub logical_schema: Option, diff --git a/crates/sqlbuild-rules/src/query_facts/models.rs b/crates/sqlbuild-rules/src/query_facts/models.rs index db8a95d7d2..ace5ffb435 100644 --- a/crates/sqlbuild-rules/src/query_facts/models.rs +++ b/crates/sqlbuild-rules/src/query_facts/models.rs @@ -225,10 +225,12 @@ pub(crate) struct ModelSource<'a> { bytes: Vec, lines: Vec, offsets: &'a [usize], + /// The lines of the model file above this SQL. + line_offset: u64, } impl<'a> ModelSource<'a> { - pub(crate) fn new(sql: &'a str, offsets: &'a [usize]) -> Self { + pub(crate) fn new(sql: &'a str, offsets: &'a [usize], line_offset: u64) -> Self { let mut bytes: Vec = Vec::with_capacity(sql.len() + 1); let mut lines: Vec = vec![0]; for (character, (byte, value)) in sql.char_indices().enumerate() { @@ -243,6 +245,7 @@ impl<'a> ModelSource<'a> { bytes, lines, offsets, + line_offset, } } @@ -266,7 +269,7 @@ impl<'a> ModelSource<'a> { let character = self.original(offset); let line = self.lines.partition_point(|start| *start <= character); let column = character - self.lines[line - 1] + 1; - (line as u64, column as u64) + (line as u64 + self.line_offset, column as u64) } /// The model's own text between normalized characters `start` and `end`. diff --git a/crates/sqlbuild-rules/src/rules/_helpers/evaluation.rs b/crates/sqlbuild-rules/src/rules/_helpers/evaluation.rs index 8c9135be8d..373f72499a 100644 --- a/crates/sqlbuild-rules/src/rules/_helpers/evaluation.rs +++ b/crates/sqlbuild-rules/src/rules/_helpers/evaluation.rs @@ -144,7 +144,7 @@ fn evaluate_model_inner(request: ModelEvaluationRequest<'_>) -> Result) -> Result { model: &'a Model, rule: &'a RuleMetadata, faults: &'a FaultCollector, + dialect: &'a str, } -pub(crate) fn evaluate(parsed: &ParsedModel<'_>, rule: &RuleMetadata, faults: &FaultCollector) { +pub(crate) fn evaluate( + parsed: &ParsedModel<'_>, + rule: &RuleMetadata, + faults: &FaultCollector, + dialect: &str, +) { let model = parsed.model; let query = parsed.facts; if model @@ -46,6 +53,7 @@ pub(crate) fn evaluate(parsed: &ParsedModel<'_>, rule: &RuleMetadata, faults: &F model, rule, faults, + dialect, }; if let Some((boundary, columns)) = final_cte_output_boundary(query, &model.columns) { evaluator.evaluate_set( @@ -182,24 +190,49 @@ impl OutputRule<'_> { continue; }; let direct = expression.direct_column.is_some(); - let explicit_cast = expression.cast_type.is_some(); let cast_matches_contract = expression .cast_type .as_ref() - .is_some_and(|data_type| type_spelling_equal(data_type, &column.data_type)); + .is_some_and(|data_type| same_type(data_type, &column.data_type, self.dialect)); let call: Option<&TypedCall> = self.whole_call(expression); - let passes = cast_matches_contract - || column.type_proven && direct && !scope.set_branch - || column.type_proven - && call - .is_some_and(|call| call.returns.is_some() || call.returns_like.is_some()); + let passes = cast_matches_contract || column.type_proven && direct && !scope.set_branch; if passes { continue; } let name = output_name.unwrap_or_else(|| column.name.clone()); - let reason = if explicit_cast && !cast_matches_contract { - "has an outer cast that does not target its declared contract type" - } else if direct && scope.set_branch { + if let Some(cast_type) = expression + .cast_type + .as_ref() + .filter(|_| !cast_matches_contract) + { + self.push( + expression.span.map(|(start, _)| start).or(expression.start), + format!( + "output {name:?} has an outer cast to {cast_type}, which is not its declared contract type {}", + column.data_type + ), + format!( + "Change the outer cast to CAST(... AS {}).", + column.data_type + ), + ); + continue; + } + if let Some(call) = + call.filter(|call| call.returns.is_some() || call.returns_like.is_some()) + { + self.push( + expression.start, + typed_call_message(&name, call, &column.data_type), + format!( + "Cast the result in the macro or at the call, as CAST(... AS {}); returns \ + types the call for checks but does not change the column's physical type.", + call.returns.as_deref().unwrap_or(&column.data_type) + ), + ); + continue; + } + let reason = if direct && scope.set_branch { "is in a set-operation branch that must cast each output explicitly" } else if direct { "is a passthrough whose type is not proven against the declared contract" @@ -252,6 +285,22 @@ impl OutputRule<'_> { } } +/// Why one call to a typed SQL macro does not establish its output's contract type. +fn typed_call_message(name: &str, call: &TypedCall, declared: &str) -> String { + let macro_name: &str = &call.macro_name; + match call.returns.as_deref() { + Some(returns) => format!( + "output {name:?} is a call to SQL macro {macro_name}(), which declares returns \ + {returns}, but its body isn't cast to it, so the column's physical type may differ; \ + declared type is {declared}" + ), + None => format!( + "output {name:?} is a call to SQL macro {macro_name}(), whose returns TYPE OF does \ + not set the column's physical type; declared type is {declared}" + ), + } +} + /// Whether `text` holds only whitespace and comments. fn only_trivia(mut text: &str) -> bool { loop { @@ -426,6 +475,24 @@ fn set_expression_output_names_with_missing(set_expr: &SetFacts) -> Option bool { + if type_spelling_equal(left, right) { + return true; + } + match ( + normalize_type(left, dialect), + normalize_type(right, dialect), + ) { + (Ok(left), Ok(right)) => { + left.parse_error.is_none() + && right.parse_error.is_none() + && left.normalized == right.normalized + } + _ => false, + } +} + fn type_spelling_equal(left: &str, right: &str) -> bool { let normalize = |value: &str| { value diff --git a/crates/sqlbuild-rules/src/rules/_helpers/sql_test_rules.rs b/crates/sqlbuild-rules/src/rules/_helpers/sql_test_rules.rs index d9673c87f1..ae4c1d8d29 100644 --- a/crates/sqlbuild-rules/src/rules/_helpers/sql_test_rules.rs +++ b/crates/sqlbuild-rules/src/rules/_helpers/sql_test_rules.rs @@ -159,11 +159,15 @@ fn filenames( let stem = file_stem(&test.source_path); let allowed = allowed_subjects(evaluation, test); let valid = stem.strip_prefix("test_").is_some_and(valid_unit_filename); - let behavior_matches = test.explicit_name.as_deref().is_none_or(|name| { - filename_behavior(stem, &allowed) - .zip(structured_name(name, &allowed).map(|(_, behavior)| behavior)) - .is_none_or(|(slug, behavior)| behavior_slug_matches(slug, behavior)) + let shares_file: bool = tests.iter().any(|other| { + other.source_path == test.source_path && other.block_index != test.block_index }); + let behavior_matches = shares_file + || test.explicit_name.as_deref().is_none_or(|name| { + filename_behavior(stem, &allowed) + .zip(structured_name(name, &allowed).map(|(_, behavior)| behavior)) + .is_none_or(|(slug, behavior)| behavior_slug_matches(slug, behavior)) + }); if !valid || !behavior_matches { faults.push(path_fault( rule, diff --git a/crates/sqlbuild-rules/src/sql_lint/_helpers/engine.rs b/crates/sqlbuild-rules/src/sql_lint/_helpers/engine.rs index 567376e688..34c1ce5695 100644 --- a/crates/sqlbuild-rules/src/sql_lint/_helpers/engine.rs +++ b/crates/sqlbuild-rules/src/sql_lint/_helpers/engine.rs @@ -548,6 +548,7 @@ fn collect_token_query_facts( let limit = first_row_selection_clause(tokens, &direct); if order.is_none() && let Some(index) = limit + && !limits_to_no_rows(tokens, index) { facts.unordered_limits.push(tokens[index].span); } @@ -732,6 +733,14 @@ fn first_row_selection_clause(tokens: &[Token], direct: &[usize]) -> Option bool { + tokens[index].token_type == TokenType::Limit + && tokens.get(index + 1).is_some_and(|count| { + count.token_type == TokenType::Number && count.text.parse::() == Ok(0) + }) +} + pub(super) fn is_query_from(tokens: &[Token], index: usize) -> bool { tokens[index].token_type == TokenType::From && significant_before(tokens, index) diff --git a/crates/sqlbuild-rules/src/sql_lint/tests/test_sql_lint.rs b/crates/sqlbuild-rules/src/sql_lint/tests/test_sql_lint.rs index 35f1297fbb..141ff9c8b3 100644 --- a/crates/sqlbuild-rules/src/sql_lint/tests/test_sql_lint.rs +++ b/crates/sqlbuild-rules/src/sql_lint/tests/test_sql_lint.rs @@ -84,6 +84,18 @@ fn given_sql_cases_when_linting_then_diagnostics_match() -> Result<(), String> { expected_codes: &["SQBRSQL004"], expected_anchors: &[("SQBRSQL004", "LIMIT")], }, + test_types::LintTestCase { + description: "limit zero selects no rows in any order", + sql: "SELECT id FROM items LIMIT 0", + expected_codes: &[], + expected_anchors: &[], + }, + test_types::LintTestCase { + description: "limit zero with an offset still selects no rows", + sql: "SELECT id FROM items LIMIT 0 OFFSET 5", + expected_codes: &[], + expected_anchors: &[], + }, test_types::LintTestCase { description: "semi-structured limit and offset keys are not row selection clauses", sql: "SELECT payload:Limit::VARCHAR AS order_limit, payload:Offset::NUMBER AS order_offset FROM orders", diff --git a/crates/sqlbuild-sqltext/src/header_strings/_helpers/description_keys.rs b/crates/sqlbuild-sqltext/src/header_strings/_helpers/description_keys.rs new file mode 100644 index 0000000000..a21ad31449 --- /dev/null +++ b/crates/sqlbuild-sqltext/src/header_strings/_helpers/description_keys.rs @@ -0,0 +1,52 @@ +//! Header keys around a string: whole-word key matches and enclosing `meta` values. + +use crate::header_strings::constants::META_KEY; + +/// Whether `unit` can be part of a header key. +fn is_key_unit>(unit: T) -> bool { + char::from_u32(unit.into()) + .is_some_and(|character| character.is_alphanumeric() || character == '_') +} + +/// Whether `text` holds `key`, in any letter case, as a whole word ending at `end`. +pub(crate) fn key_ends_at + Into>( + text: &[T], + end: usize, + key: &str, +) -> bool { + let Some(start) = end.checked_sub(key.len()) else { + return false; + }; + let matches_key: bool = key.bytes().enumerate().all(|(offset, byte)| { + let unit: T = text[start + offset]; + unit == T::from(byte) || unit == T::from(byte.to_ascii_uppercase()) + }); + matches_key && (start == 0 || !is_key_unit(text[start - 1])) +} + +/// Whether the key ending before `end` sits inside a `meta (...)` value, which holds no prose. +pub(crate) fn inside_meta + Into>( + text: &[T], + end: usize, + is_space: impl Fn(T) -> bool, +) -> bool { + let mut depth: usize = 0; + for index in (0..end).rev() { + if text[index] == T::from(b')') { + depth += 1; + } else if text[index] == T::from(b'(') { + if depth > 0 { + depth -= 1; + continue; + } + let mut key_end: usize = index; + while key_end > 0 && is_space(text[key_end - 1]) { + key_end -= 1; + } + if key_ends_at(text, key_end, META_KEY) { + return true; + } + } + } + false +} diff --git a/crates/sqlbuild-sqltext/src/header_strings/_helpers/mod.rs b/crates/sqlbuild-sqltext/src/header_strings/_helpers/mod.rs index 6def37b961..2388e517c6 100644 --- a/crates/sqlbuild-sqltext/src/header_strings/_helpers/mod.rs +++ b/crates/sqlbuild-sqltext/src/header_strings/_helpers/mod.rs @@ -1,3 +1,4 @@ +pub(crate) mod description_keys; pub(crate) mod locations; pub(crate) mod prose_ends; pub(crate) mod separators; diff --git a/crates/sqlbuild-sqltext/src/header_strings/constants.rs b/crates/sqlbuild-sqltext/src/header_strings/constants.rs index 9ee43c62ab..f50ce137c8 100644 --- a/crates/sqlbuild-sqltext/src/header_strings/constants.rs +++ b/crates/sqlbuild-sqltext/src/header_strings/constants.rs @@ -8,6 +8,8 @@ pub const TRIPLE_QUOTE_LENGTH: usize = 3; pub const LONGEST_CLOSING_RUN: usize = 5; /// The header key whose string values are folded prose. pub const DESCRIPTION_KEY: &str = "description"; +/// The header key whose free-form values are never prose, even under a `description` key. +pub const META_KEY: &str = "meta"; /// The header key whose list items are audits, whose `description` option is prose. pub const AUDITS_KEY: &str = "audits"; /// What a prose value stores for a blank line: a paragraph break. diff --git a/crates/sqlbuild-sqltext/src/header_strings/main/follows_description_key.rs b/crates/sqlbuild-sqltext/src/header_strings/main/follows_description_key.rs index 64750bc975..23fbdaeac2 100644 --- a/crates/sqlbuild-sqltext/src/header_strings/main/follows_description_key.rs +++ b/crates/sqlbuild-sqltext/src/header_strings/main/follows_description_key.rs @@ -1,13 +1,8 @@ //! Whether a header string is the value of a `description` key, which holds prose. +use crate::header_strings::_helpers::description_keys::{inside_meta, key_ends_at}; use crate::header_strings::constants::DESCRIPTION_KEY; -/// Whether `unit` can be part of a header key. -fn is_key_unit>(unit: T) -> bool { - char::from_u32(unit.into()) - .is_some_and(|character| character.is_alphanumeric() || character == '_') -} - /// Whether the string at `start` directly follows a `description` key, in any letter case. pub fn follows_description_key + Into>( text: &[T], @@ -18,12 +13,7 @@ pub fn follows_description_key + Into>( while key_end > 0 && is_space(text[key_end - 1]) { key_end -= 1; } - let Some(key_start) = key_end.checked_sub(DESCRIPTION_KEY.len()) else { - return false; - }; - let matches_key: bool = DESCRIPTION_KEY.bytes().enumerate().all(|(offset, byte)| { - let unit: T = text[key_start + offset]; - unit == T::from(byte) || unit == T::from(byte.to_ascii_uppercase()) - }); - matches_key && key_end < start && (key_start == 0 || !is_key_unit(text[key_start - 1])) + key_end < start + && key_ends_at(text, key_end, DESCRIPTION_KEY) + && !inside_meta(text, key_end - DESCRIPTION_KEY.len(), is_space) } diff --git a/crates/sqlbuild-sqltext/src/header_strings/tests/test_header_strings.rs b/crates/sqlbuild-sqltext/src/header_strings/tests/test_header_strings.rs index f22ef69e2f..42c14f6191 100644 --- a/crates/sqlbuild-sqltext/src/header_strings/tests/test_header_strings.rs +++ b/crates/sqlbuild-sqltext/src/header_strings/tests/test_header_strings.rs @@ -184,6 +184,18 @@ fn given_description_string_when_finding_its_end_then_prose_rules_decide_it() { start: 12, expected_end: Some(19), }, + DescriptionEndTestCase { + description: "a description inside meta reads the backslash as an escape", + text: r#"meta (owner "ops", description "say \", then"), x"#, + start: 31, + expected_end: Some(45), + }, + DescriptionEndTestCase { + description: "a description after a closed meta value is prose again", + text: r#"meta (owner "ops"), description "C:\", x"#, + start: 32, + expected_end: Some(37), + }, DescriptionEndTestCase { description: "a key that only ends in description is not one", text: r#"redescription "C:\", x""#, diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md index 9e7e4e3d7e..b034a137af 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md @@ -68,7 +68,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`: | `--start-cursor-int` | Override start cursor for integer incremental models | | `--end-cursor-int` | Override end cursor for integer incremental models | | `--load` | Explicitly load managed sources before building | -| `--no-load` | Skip automatic source loading | +| `--no-load` | Run no source loaders; selected managed sources are still audited | | `--reload` | Reload managed sources (passes `is_reload=True` to loaders) | | `--selection-diagnostics` | Warn when a selected model will build on changed upstream models that are not selected; see [Selection and staleness](../concepts/planning/selection-and-staleness.md) | | `--manifest` | Generate `target/manifest.json` with plan-aware project metadata | @@ -89,9 +89,9 @@ This replaces the former `sqb run` command. The full lifecycle (tests + audits) ## Execution order -1. Managed sources are loaded (unless `--no-load`) +1. Managed sources are loaded (unless `--no-load`, which audits their existing data instead) 2. Selected seeds are loaded -3. Source audits run before their dependent models (unless `--no-audits`) +3. Source audits run before their dependent models (unless `--no-audits`); with `--no-load`, or for a source selected by name, a source no model reads is audited at the end 4. SQL unit tests run before their target model (unless `--no-tests`) 5. Models are materialized in DAG topological order 6. Error-severity audits run against the staging table before promotion to the target (unless `--no-audits`) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md index a991f7df01..c4229a95fe 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md @@ -93,4 +93,9 @@ sqb build --no-load sqb build --reload ``` +With loading off, through `--no-load` or `auto_load_sources = false`, no loader runs, even for a +source selected by name such as `sqb build --no-load --select raw_orders`. Managed sources in the +selection stay in the plan, and their audits check the data they already hold, at the end of the +build if no model reads them. A source outside the selection is neither loaded nor audited. + See [Loaders](../concepts/python-nodes/loaders.md) for full documentation on write strategies, the loader context API, and source deferral. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md index 6f64c8f1b7..9fe8b1386d 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md @@ -154,7 +154,7 @@ SQLBuild includes four generic audits out of the box. You do not need to define | `not_null` | Fails if any row has a NULL value in the column | Column-level only | | `unique` | Fails if any non-NULL value appears more than once | Column-level only | | `accepted_values` | Fails if any non-NULL value is not in the allowed list | `values` - list of allowed values | -| `relationships` | Fails if any non-NULL value does not exist in the referenced column | `to` - target relation, `field` - target column | +| `relationships` | Fails if any non-NULL value does not exist in the referenced column | `to` - target relation as `__ref(...)`, `__source(...)` or `__seed(...)`, `field` - target column | ### Using built-in audits @@ -175,7 +175,7 @@ MODEL ( ), payment_method ( audits [ - relationships (to "stg_payments", field "method"), + relationships (to __ref("stg_payments"), field "method"), ], ), ), @@ -379,6 +379,11 @@ sources: If a source audit with `error` severity fails, all downstream models that depend on that source are blocked. This lets you catch data quality issues at the source before any transformations run. +A source's audits run before the first model in the build that reads it. A source no selected model +reads is audited at the end of the build only when loading is off (`--no-load` or +`auto_load_sources = false`) or when the source is selected by name, such as +`sqb build --select raw_orders`; otherwise its audits do not run. + A source audit that reads other resources waits for them before any model that depends on the source is built. It may read a model that is not built from the source; reading one that is fails with `P005`. See [Audits that read other resources](#audits-that-read-other-resources). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md index b00e5ad7b5..f8dc16a189 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/sql-macros.md @@ -339,9 +339,15 @@ bar(LENGTH(@value), 0, 10) `returns VARCHAR(5)`, and contract checks report a mismatch with `K002`. - The type flows on as an inferred type would: through CTEs, `SELECT *`, subqueries, joins and each branch of a set operation, through operators and functions around the call, to contract checks - (`K002`, `K003`), `SQBRCONTRACT105` and models that read the column. A select-list item that is - one call to a macro with `returns` needs no `CAST` for `SQBRCONTRACT105`; a call inside a larger - expression still does. + (`K002`, `K003`), `SQBRCONTRACT105` and models that read the column. +- `returns` is used only for analysis. It doesn't appear in the generated SQL, so on its own it doesn't + change the column's physical type. A body of `'ACTIVE'` still creates `VARCHAR(6)` on Snowflake. + For `SQBRCONTRACT105`, a select-list item that is one call needs no `CAST` at the call only when + the macro body's outermost expression is a `CAST(... AS type)` or `...::type` to the column's + contract type, compared as the dialect does, such as `CAST(LEFT(@value, 3) AS VARCHAR)` for a + `VARCHAR` column. Otherwise the rule asks for a cast in the macro or at the call, whether the + macro declares a fixed `returns` or `returns TYPE OF`. A call inside a larger expression always + needs an outer cast. - Without `returns`, the call is typed by inference alone. When an output is one call whose type is unknown, `K003` and `SQBRCONTRACT105` suggest declaring `returns` on the macro. - `returns TYPE OF ` declares the type of that argument at each call, under the same @@ -353,8 +359,9 @@ bar(LENGTH(@value), 0, 10) `returns BIGINT`, fails with `P024`, as does a partly unknown body of another shape, such as `[]` with `returns BIGINT`. A body whose type cannot be inferred, for example because it reads `ANY` arguments or calls a function whose type is unknown or only guessed, trusts `returns`. -- `returns` must be a type the adapter's SQL dialect accepts, including a decimal precision and - scale it can hold, such as at most `DECIMAL(38, s)` on DuckDB (`P015`). +- `returns` and every argument type must be a type the adapter's SQL dialect accepts, including a + decimal precision and scale it can hold, such as at most `DECIMAL(38, s)` on DuckDB or + `NUMERIC(38, 9)` on BigQuery (`P015`). - A call where only a keyword can stand, such as the date part of `EXTRACT`, is typed by inference alone. - A call inside a Python macro argument, as in `@py.bracket(@sql.status_bar(o.status))`, is typed diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md index 0cd4e2c50f..8d2415eb3c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md @@ -171,6 +171,15 @@ sources: Existing rows matching the unique key are updated; new rows are inserted. +### String values on Snowflake + +Snowflake reads a backslash inside a string literal as an escape, so SQLBuild doubles every +backslash in the values a loader returns. A value such as `C:\orders\2026` is stored exactly as +returned. Earlier releases wrote those backslashes unescaped, so Snowflake dropped them or read them +as escapes. If a `merge` or `delete_insert` unique key, or a value an `append` load is +deduplicated by, contains a backslash, rows loaded by an earlier release will not match newly +loaded rows. Rewrite or delete those rows before relying on the key. + ## Self-managed loaders If a loader returns `None`, SQLBuild skips its row-writing pipeline. The loader is responsible for writing data to the target itself, using whatever approach makes sense - `ctx.execute_sql()`, an external library, a subprocess, or anything else: @@ -329,6 +338,11 @@ sqb build --no-load sqb build --reload ``` +With loading off, through `--no-load` or `auto_load_sources = false`, no loader runs, even for a +source selected by name such as `sqb build --no-load --select raw_orders`. Managed sources in the +selection stay in the plan, and their audits check the data they already hold, at the end of the +build if no model reads them. A source outside the selection is neither loaded nor audited. + When `--reload` is passed, `ctx.is_reload` is `True` in the loader function. This lets loaders implement different behavior for full reloads versus normal incremental loads. ## Source deferral diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md index 347d21adf8..97d0318f70 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md @@ -76,6 +76,11 @@ sources: If a source audit with `error` severity fails, all downstream models that depend on that source are blocked. +A source's audits run before the first model in the build that reads it. A source no selected model +reads is audited at the end of the build only when loading is off (`--no-load` or +`auto_load_sources = false`) or when the source is selected by name, such as +`sqb build --select raw_orders`; otherwise its audits do not run. + A source audit may also read other resources; SQLBuild builds them first. See [Audits that read other resources](audits.md#audits-that-read-other-resources). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md index 6ef7862a67..48d3b583b6 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md @@ -453,7 +453,7 @@ replaces every `@sql.net_amount(...)` call written in the model SQL with the moc inside another SQL macro's body is expanded as usual. Macro names are unique across both kinds, so the mock name never needs the `sql.` or `py.` prefix. -The mock value must be a single `SELECT` with one string literal. Use doubled single quotes for quotes within the value (standard SQL escaping). +The mock value must be a single `SELECT` with one string literal, which may be named with `AS`, such as `SELECT 'country_code = ''US''' AS country_filter`. Use doubled single quotes for quotes within the value (standard SQL escaping). This is useful for: - Testing models that use target-specific macros without depending on target config diff --git a/src/sqlbuild/_native.pyi b/src/sqlbuild/_native.pyi index fa8e53d51f..8c941e3403 100644 --- a/src/sqlbuild/_native.pyi +++ b/src/sqlbuild/_native.pyi @@ -10172,6 +10172,8 @@ class PlannerScope(NativeRecord): user_selected_keys: frozenset[CompiledObjectKey] python_read_source_names: frozenset[str] sql_test_selection: SqlTestSelection + load_sources: bool + end_audit_source_keys: frozenset[CompiledObjectKey] def __init__( self, upstream_deps: dict[CompiledObjectKey, tuple[CompiledObjectKey, ...]], @@ -10183,6 +10185,8 @@ class PlannerScope(NativeRecord): user_selected_keys: frozenset[CompiledObjectKey] = ..., python_read_source_names: frozenset[str] = ..., sql_test_selection: SqlTestSelection = ..., + load_sources: bool = ..., + end_audit_source_keys: frozenset[CompiledObjectKey] = ..., ) -> None: ... class PlannerRelationsContext: @@ -14198,12 +14202,14 @@ class ScenarioFixturePlan(NativeRecord): logical_name: str destination: CompiledRelationLocation sql: str + fingerprint_sql: str def __init__( self, kind: ScenarioArtifactKind, logical_name: str, destination: CompiledRelationLocation, sql: str, + fingerprint_sql: str = ..., ) -> None: ... class ScenarioExpectedExpectationPlan(NativeRecord): @@ -14295,6 +14301,8 @@ class PlanOutput(NativeRecord): python_identity_fingerprints: dict[tuple[str, str], Fingerprint] metadata: dict[str, object] cursor_overrides: CursorOverrides | None + existing_model_names: frozenset[str] + end_audit_source_keys: frozenset[CompiledObjectKey] def __init__( self, execution_order: tuple[CompiledObjectKey, ...] = ..., @@ -14328,6 +14336,8 @@ class PlanOutput(NativeRecord): python_identity_fingerprints: dict[tuple[str, str], Fingerprint] = ..., metadata: dict[str, object] = ..., cursor_overrides: CursorOverrides | None = ..., + existing_model_names: frozenset[str] = ..., + end_audit_source_keys: frozenset[CompiledObjectKey] = ..., ) -> None: ... @property def python_source_entries(self) -> dict[str, SourceEntry]: ... diff --git a/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py b/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py index b6907e80e1..32829b302b 100644 --- a/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py +++ b/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py @@ -43,6 +43,8 @@ DUCKDB_INTERNAL_ERROR_CLASS_NAME, DUCKDB_MISSING_OBJECT_MARKER, DUCKDB_NOT_FOUND_ERROR_CLASS_NAMES, + DUCKDB_STORAGE_VERSION_KEY, + LATEST_DUCKDB_STORAGE_VERSION, QUALIFIED_NAME_SEPARATOR, ) from sqlbuild.adapter.contract.exceptions import AdapterUserError @@ -1071,6 +1073,22 @@ def connect(self, config: dict[str, Any]) -> Any: database: str = str(config.get("database", ":memory:")) raw_connection: duckdb.DuckDBPyConnection = duckdb.connect(database=database) + return self._configured_connection(raw_connection=raw_connection, config=config) + + def connect_local_replay(self, database: Path) -> Any: + """Open sqlbuild's own local replay database at the latest DuckDB storage version.""" + + import duckdb + + raw_connection: duckdb.DuckDBPyConnection = duckdb.connect( + database=str(database), + config={DUCKDB_STORAGE_VERSION_KEY: LATEST_DUCKDB_STORAGE_VERSION}, + ) + return self._configured_connection(raw_connection=raw_connection, config={}) + + def _configured_connection(self, *, raw_connection: Any, config: dict[str, Any]) -> Any: + """Wrap a DuckDB connection and apply the config's extensions, settings and attachments.""" + connection: ObservedConnection = ObservedConnection( raw_connection=raw_connection, adapter=self.adapter_name ) diff --git a/src/sqlbuild/adapter/contract/constants.py b/src/sqlbuild/adapter/contract/constants.py index 9c94083584..e37c2b2c11 100644 --- a/src/sqlbuild/adapter/contract/constants.py +++ b/src/sqlbuild/adapter/contract/constants.py @@ -20,3 +20,6 @@ DUCKDB_INTERNAL_ERROR_CLASS_NAME: str = "InternalException" CANCELLABLE_DURATION_LIMIT_SECONDS: int = 2_147_483 UNSUPPORTED_RELATION_USAGE_REASON: str = "unsupported adapter" + +DUCKDB_STORAGE_VERSION_KEY: str = "storage_compatibility_version" +LATEST_DUCKDB_STORAGE_VERSION: str = "latest" diff --git a/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py b/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py index f15e059afd..5122de425b 100644 --- a/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py +++ b/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py @@ -1514,14 +1514,21 @@ def render_loader_rows_select( for column_name in column_names ) return f"SELECT {projections} WHERE 1 = 0" + json_columns: frozenset[str] = frozenset( + column_name + for column_name in column_names + if inferred_types.get(column_name) == LoaderLogicalType.JSON + ) value_rows: list[str] = [] for row in rows: row_values: list[str] = [] for column_name in column_names: + value: object = row.get(column_name) row_values.append( - self.render_loader_value_literal( - value=row.get(column_name), - logical_type=inferred_types.get(column_name), + self._quote_sql_string(json.dumps(value, sort_keys=True)) + if column_name in json_columns and value is not None + else self.render_loader_value_literal( + value=value, logical_type=inferred_types.get(column_name) ) ) value_rows.append(f"({', '.join(row_values)})") @@ -1529,20 +1536,24 @@ def render_loader_rows_select( column_sql: str = ", ".join( self.render_identifier(column_name) for column_name in column_names ) - select_sql: str = ", ".join( - ( - self.render_identifier(column_name) - if column_name not in column_sql_types - else "CAST(" - f"{self.render_identifier(column_name)} AS {column_sql_types[column_name]}) " - f"AS {self.render_identifier(column_name)}" + projections: list[str] = [] + for column_name in column_names: + identifier: str = self.render_identifier(column_name) + value_sql: str = ( + f"PARSE_JSON({identifier})" if column_name in json_columns else identifier ) - for column_name in column_names - ) + projections.append( + f"CAST({value_sql} AS {column_sql_types[column_name]}) AS {identifier}" + if column_name in column_sql_types + else f"{value_sql} AS {identifier}" + if column_name in json_columns + else identifier + ) + select_sql: str = ", ".join(projections) return f"SELECT {select_sql} FROM (VALUES {values_sql}) AS __loader_rows({column_sql})" def _quote_sql_string(self, value: str) -> str: - return "'" + value.replace("'", "''") + "'" + return "'" + value.replace("\\", "\\\\").replace("'", "''") + "'" def render_create_initial_snapshot_destination( self, diff --git a/src/sqlbuild/compiler/auditing/types.py b/src/sqlbuild/compiler/auditing/types.py index 8ca6370158..16eced37fc 100644 --- a/src/sqlbuild/compiler/auditing/types.py +++ b/src/sqlbuild/compiler/auditing/types.py @@ -38,3 +38,4 @@ class AuditOutcome(StrEnum): WARN = "warn" ERROR = "error" INSUFFICIENT = "insufficient" + SKIPPED = "skipped" diff --git a/src/sqlbuild/executor/auditing/models.py b/src/sqlbuild/executor/auditing/models.py index 6c07bef9a4..39458e8b8b 100644 --- a/src/sqlbuild/executor/auditing/models.py +++ b/src/sqlbuild/executor/auditing/models.py @@ -73,3 +73,4 @@ class AuditExecutionResult: audit_description: str | None = None execution_error: str | None = None audit_meta: Mapping[str, object] | None = None + skip_reason: str | None = None diff --git a/src/sqlbuild/executor/build/_helpers/source_audits.py b/src/sqlbuild/executor/build/_helpers/source_audits.py index 0942ce4e50..1170fbe316 100644 --- a/src/sqlbuild/executor/build/_helpers/source_audits.py +++ b/src/sqlbuild/executor/build/_helpers/source_audits.py @@ -32,7 +32,7 @@ def run_pending_source_audits( *, - model_key: CompiledObjectKey, + source_keys: tuple[CompiledObjectKey, ...], plan: PlanOutput, source_audits_by_source: dict[str, tuple[AuditPlanEntry, ...]], executed_source_audits: frozenset[str], @@ -50,7 +50,7 @@ def run_pending_source_audits( newly_blocked: set[CompiledObjectKey] = set() audit_results: list[AuditExecutionResult] = [] dep_key: CompiledObjectKey - for dep_key in plan.upstream_deps.get(model_key, ()): + for dep_key in source_keys: if dep_key.resource_type != CompiledResourceType.SOURCE: continue source_name: str = dep_key.name diff --git a/src/sqlbuild/executor/build/classes/build_scheduler.py b/src/sqlbuild/executor/build/classes/build_scheduler.py index 7170b1956e..c4d3805ffc 100644 --- a/src/sqlbuild/executor/build/classes/build_scheduler.py +++ b/src/sqlbuild/executor/build/classes/build_scheduler.py @@ -32,6 +32,7 @@ SourceEntry, SqlTestPlanEntry, TableTypePlanEntry, + audit_resource_id, sql_test_resource_id, ) from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter @@ -43,6 +44,9 @@ from sqlbuild.diagnostics.main.diagnostics_context import diagnostics_context from sqlbuild.diagnostics.main.log_debug_event import log_debug_event from sqlbuild.errors.contracts.exceptions import ExecutorInputError +from sqlbuild.executor.auditing.main._record_audit_progress_outcome import ( + record_audit_progress_outcome, +) from sqlbuild.executor.auditing.main.publish_completed_audit_results import ( publish_completed_audit_results, ) @@ -233,6 +237,7 @@ def __init__( ) self._max_concurrency: int = len(connections) self._blocked_keys: set[CompiledObjectKey] = set() + self._unbuilt_keys: set[CompiledObjectKey] = set(initial_state.initial_failed_keys) self._completed_keys: set[CompiledObjectKey] = set(initial_state.precompleted_keys) self._initial_failed_keys: frozenset[CompiledObjectKey] = initial_state.initial_failed_keys self._in_flight: set[CompiledObjectKey] = set() @@ -297,11 +302,25 @@ def run( enforce_execution_deadline() self._skip_remaining() + if self._run_audits and not self._stop: + _ = self._run_source_audits(source_keys=self._unloaded_selected_sources()) end_audit_results: tuple[AuditExecutionResult, ...] = () - if self._run_audits and self._indexes.end_audits: + runnable_end_audits: tuple[AuditPlanEntry, ...] = tuple( + audit + for audit in self._indexes.end_audits + if self._unbuilt_keys.isdisjoint(audit.scope_deps) + ) + skipped_end_audits: tuple[AuditExecutionResult, ...] = () + if self._run_audits: + skipped_end_audits = tuple( + self._skipped_end_audit(audit=audit) + for audit in self._indexes.end_audits + if not self._unbuilt_keys.isdisjoint(audit.scope_deps) + ) + if self._run_audits and runnable_end_audits: end_audit_results = run_end_audits( - end_audits=self._indexes.end_audits, + end_audits=runnable_end_audits, adapter=self._adapter, connection=self._scheduler_connection, model_locations=self._plan.model_locations, @@ -317,7 +336,7 @@ def run( tuple(self._load_results), tuple(sorted(self._test_results, key=self._test_result_order)), tuple(self._source_audit_results), - end_audit_results, + (*end_audit_results, *skipped_end_audits), ) def _test_result_order(self, result: SqlTestExecutionResult) -> int: @@ -386,7 +405,7 @@ def _block_initial_failed_keys(self) -> None: downstream_blocked_keys( failed_key=key, downstream_deps=self._plan.downstream_deps, - selected_keys=self._plan.selected_keys, + selected_keys=self._selected_execution_keys, ) ) if self._fail_fast and self._initial_failed_keys: @@ -729,22 +748,7 @@ def _pre_dispatch(self, key: CompiledObjectKey) -> bool: model_entry: ModelPlanEntry | None = self._indexes.model_entries_by_key.get(key) if model_entry is None: return False - audit_run: SourceAuditRunResult = run_pending_source_audits( - model_key=key, - plan=self._plan, - source_audits_by_source=self._indexes.source_audits_by_source, - executed_source_audits=frozenset(self._executed_source_audits), - failed_sources=frozenset(self._failed_sources), - adapter=self._adapter, - connection=self._scheduler_connection, - fail_fast=self._fail_fast, - run_id=self._run_id, - ) - self._executed_source_audits.update(audit_run.executed_source_names) - self._failed_sources.update(audit_run.failed_source_names) - self._blocked_keys.update(audit_run.newly_blocked_keys) - self._source_audit_results.extend(audit_run.audit_results) - if audit_run.blocked: + if self._run_source_audits(source_keys=self._plan.upstream_deps.get(key, ())): self._blocked_keys.add(key) self._record_skipped_result( key=key, @@ -758,6 +762,85 @@ def _pre_dispatch(self, key: CompiledObjectKey) -> bool: return True + def _mark_unbuilt(self, *, key: CompiledObjectKey) -> None: + """Record that `key` was not built, unless it is a model whose relation already exists.""" + + if ( + key.resource_type == CompiledResourceType.MODEL + and key.name in self._plan.existing_model_names + ): + return + self._unbuilt_keys.add(key) + + def _skipped_end_audit(self, *, audit: AuditPlanEntry) -> AuditExecutionResult: + """The SKIPPED result of an end-of-build audit that reads a resource not built this run.""" + + unbuilt: CompiledObjectKey = next( + key for key in audit.scope_deps if key in self._unbuilt_keys + ) + result: AuditExecutionResult = AuditExecutionResult( + audit_name=audit.name, + audit_definition_name=audit.definition_name, + attachment_kind=audit.attachment_kind, + severity=audit.severity, + outcome=AuditOutcome.SKIPPED, + row_count=0, + executed_sql="", + attached_target_kind=audit.attached_target_kind, + attached_target_name=audit.attached_target_name, + attached_column_name=audit.attached_column_name, + audit_description=audit.description, + audit_meta=audit.meta, + skip_reason=f"reads {unbuilt.name}, which was not built", + ) + with ResourceAttemptLifecycle( + resource_id=audit_resource_id( + audit_name=audit.name, + attachment_kind=audit.attachment_kind, + attached_target_kind=audit.attached_target_kind, + attached_target_name=audit.attached_target_name, + attached_column_name=audit.attached_column_name, + ), + resource_kind="audit", + resource_name=audit.name, + run_id=self._run_id, + ) as lifecycle: + record_audit_progress_outcome( + resource_attempt_id=lifecycle.resource_attempt_id, result=result + ) + lifecycle.skipped(skip_code="scheduler") + return result + + def _unloaded_selected_sources(self) -> tuple[CompiledObjectKey, ...]: + """Sources audited at the end: all of them with loading off, else only explicit ones.""" + + return tuple( + key + for key in self._plan.execution_order + if key in self._plan.end_audit_source_keys + and key not in self._indexes.source_load_entries_by_key + ) + + def _run_source_audits(self, *, source_keys: tuple[CompiledObjectKey, ...]) -> bool: + """Run the audits of sources that have not run them yet; whether one blocks its readers.""" + + audit_run: SourceAuditRunResult = run_pending_source_audits( + source_keys=source_keys, + plan=self._plan, + source_audits_by_source=self._indexes.source_audits_by_source, + executed_source_audits=frozenset(self._executed_source_audits), + failed_sources=frozenset(self._failed_sources), + adapter=self._adapter, + connection=self._scheduler_connection, + fail_fast=self._fail_fast, + run_id=self._run_id, + ) + self._executed_source_audits.update(audit_run.executed_source_names) + self._failed_sources.update(audit_run.failed_source_names) + self._blocked_keys.update(audit_run.newly_blocked_keys) + self._source_audit_results.extend(audit_run.audit_results) + return audit_run.blocked + def _execute_node( self, *, key: CompiledObjectKey, connection: Any ) -> ( @@ -1271,7 +1354,7 @@ def _handle_completion( downstream_blocked_keys( failed_key=dep_key, downstream_deps=self._plan.downstream_deps, - selected_keys=self._plan.selected_keys, + selected_keys=self._selected_execution_keys, ) ) @@ -1295,11 +1378,12 @@ def _handle_completion( failed = result.status in {ExecutionStatus.FAILED, ExecutionStatus.SKIPPED} if failed: + self._mark_unbuilt(key=key) self._blocked_keys.update( downstream_blocked_keys( failed_key=key, downstream_deps=self._plan.downstream_deps, - selected_keys=self._plan.selected_keys, + selected_keys=self._selected_execution_keys, ) ) if self._fail_fast: @@ -1400,6 +1484,7 @@ def _record_skipped_result( run_id=self._run_id, ) as lifecycle: skip_mode: object = getattr(result, "skip_mode", None) + self._mark_unbuilt(key=key) lifecycle.skipped( skip_code="scheduler", skip_mode=getattr(skip_mode, "value", None), diff --git a/src/sqlbuild/executor/build/main/_execute.py b/src/sqlbuild/executor/build/main/_execute.py index d67d05dcec..87f8f4bdc1 100644 --- a/src/sqlbuild/executor/build/main/_execute.py +++ b/src/sqlbuild/executor/build/main/_execute.py @@ -10,6 +10,7 @@ from sqlbuild.adapter.relations.main.relation_metadata_cache_scope import ( relation_metadata_cache_scope, ) +from sqlbuild.compiler.auditing.types import AuditOutcome from sqlbuild.executor.auditing.main._project_results import ( AuditResultProjection, project_audit_result_batch, @@ -186,4 +187,4 @@ def _all_audit_results( results: list[AuditExecutionResult] = [*source_audit_results, *end_audit_results] for model_result in model_results: results.extend(model_result.audit_results) - return tuple(results) + return tuple(result for result in results if result.outcome != AuditOutcome.SKIPPED) diff --git a/src/sqlbuild/executor/scenario/_helpers/local/execution.py b/src/sqlbuild/executor/scenario/_helpers/local/execution.py index a702cae017..a29675dd13 100644 --- a/src/sqlbuild/executor/scenario/_helpers/local/execution.py +++ b/src/sqlbuild/executor/scenario/_helpers/local/execution.py @@ -21,6 +21,7 @@ error_message, ) from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapter.contract.classes.duckdb_backed_adapter import DuckDbBackedAdapter from sqlbuild.adapter.contract.classes.statement_recorder import StatementRecorder from sqlbuild.adapter.contract.types import TablePromotionMode from sqlbuild.compiler.compile.types import FunctionLanguage @@ -133,7 +134,11 @@ def execute_local_scenario_load_only_run( run_dir.mkdir(parents=True, exist_ok=True) duckdb_path: Path = run_dir / "local.duckdb" _remove_local_duckdb_files(duckdb_path) - connection: Any = adapter.connect({"database": str(duckdb_path)}) + connection: Any = ( + adapter.connect_local_replay(duckdb_path) + if isinstance(adapter, DuckDbBackedAdapter) + else adapter.connect({"database": str(duckdb_path)}) + ) try: input_fingerprint: str = build_scenario_snapshot_input_fingerprint( scenario_name=scenario_plan.name, diff --git a/src/sqlbuild/executor/scenario/_helpers/snapshots/core.py b/src/sqlbuild/executor/scenario/_helpers/snapshots/core.py index 5f0fa53810..875f16e9c6 100644 --- a/src/sqlbuild/executor/scenario/_helpers/snapshots/core.py +++ b/src/sqlbuild/executor/scenario/_helpers/snapshots/core.py @@ -97,6 +97,7 @@ def build_scenario_snapshot_input_specs( logical_name=fixture_plan.logical_name, ), capture_sql=fixture_plan.sql, + fingerprint_sql=fixture_plan.fingerprint_sql or None, ) ) @@ -135,7 +136,7 @@ def build_scenario_snapshot_input_fingerprint( "kind": spec.kind.value, "logical_name": spec.logical_name, "file_path": spec.file_path.as_posix(), - "capture_sql": _normalize_sql(spec.capture_sql), + "capture_sql": _normalize_sql(spec.fingerprint_sql or spec.capture_sql), } for spec in sorted(input_specs, key=_input_spec_sort_key) ], diff --git a/src/sqlbuild/executor/scenario/models.py b/src/sqlbuild/executor/scenario/models.py index ec386b1e9d..80563767e6 100644 --- a/src/sqlbuild/executor/scenario/models.py +++ b/src/sqlbuild/executor/scenario/models.py @@ -107,6 +107,7 @@ class ScenarioSnapshotInputSpec: logical_name: str file_path: Path capture_sql: str + fingerprint_sql: str | None = None @dataclass(frozen=True) diff --git a/src/sqlbuild/integrations/dagster/_helpers/invocation.py b/src/sqlbuild/integrations/dagster/_helpers/invocation.py index bd8a689c70..c250b4e2f7 100644 --- a/src/sqlbuild/integrations/dagster/_helpers/invocation.py +++ b/src/sqlbuild/integrations/dagster/_helpers/invocation.py @@ -23,6 +23,7 @@ _CHECK_RESULT: str = "check" _ASSET_MATERIALIZATION: str = "materialization" +_SKIPPED_CHECK_STATUS: str = "skipped" def _log_invocation(*, context: Any, invocation: SqlBuildCliInvocation) -> None: @@ -42,6 +43,21 @@ def _log_invocation(*, context: Any, invocation: SqlBuildCliInvocation) -> None: logger.info(" %s", line) +def _log_skipped_checks(*, context: Any, payload: Mapping[str, Any]) -> None: + """Log each audit that did not run and why; Dagster gets no check result for it.""" + + logger: Any = getattr(context, "log", None) + if logger is None: + return + for check in payload.get("checks", ()): + if check.get("status") == _SKIPPED_CHECK_STATUS: + logger.info( + "SQLBuild audit %s did not run: %s", + check.get("name"), + check.get("skip_reason") or "skipped", + ) + + def _start_stream_future( *, executor: ThreadPoolExecutor, @@ -178,6 +194,7 @@ def _build_results_from_execution_payload( command: tuple[str, ...], context: Any, ) -> tuple[Any, ...]: + _log_skipped_checks(context=context, payload=payload) return _dagster_events( dg=dg, plans=dagster_payload_results( diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/build/_test_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/build/_test_types.py index 262e84be58..d065846d30 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/build/_test_types.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/build/_test_types.py @@ -1409,3 +1409,55 @@ class DeferredMigrationOriginE2ETestCase: prod_sql: str expected_order_ids: tuple[int, ...] expect_note: bool + + +@dataclass(frozen=True) +class UnbuiltDependencyBuildE2ETestCase: + """A build whose failed model leaves a reader that must be skipped, not run.""" + + description: str + repo_files: dict[str, str] + expected_stdout_fragments: tuple[str, ...] + unexpected_stdout_fragments: tuple[str, ...] + build_arguments: tuple[str, ...] = () + + +@dataclass(frozen=True) +class NoLoadBuildE2ETestCase: + """A build with loading off over a managed source holding a NULL, and what it must do.""" + + description: str + second_build: tuple[str, ...] + project_settings: str + expected_exit_code: int + expected_audit_reported: bool + expected_order_rows: int = 2 + + +@dataclass(frozen=True) +class HookSkippedAuditBuildE2ETestCase: + """A rebuild where a hook skips a model that already exists, and its end audit's outcome.""" + + description: str + hooked_model: str + expected_exit_code: int + expected_stdout_fragments: tuple[str, ...] + + +@dataclass(frozen=True) +class ConnectionStorageKeyBuildE2ETestCase: + """A DuckDB connection key in the project config, and the storage the warehouse file gets.""" + + description: str + connection_key: str + expected_storage_version: str + + +@dataclass(frozen=True) +class SourceEndAuditBuildE2ETestCase: + """A build over an unread source whose table is missing, and whether its audit runs.""" + + description: str + build_arguments: tuple[str, ...] + expected_exit_code: int + expected_audit_reported: bool diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/build/test_duckdb_storage_key.py b/tests/e2e/src/sqlbuild/cli/commands/main/build/test_duckdb_storage_key.py new file mode 100644 index 0000000000..44ff171421 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/build/test_duckdb_storage_key.py @@ -0,0 +1,60 @@ +"""E2E coverage that a DuckDB storage version in user config is not applied to the warehouse.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.build._test_types import ( + ConnectionStorageKeyBuildE2ETestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import ( + prepare_inline_project, + query_duckdb, + run_sqb, +) + + +@pytest.mark.parametrize( + "test_case", + [ + ConnectionStorageKeyBuildE2ETestCase( + description="a storage version key is ignored like any unknown connection key", + connection_key="storage_compatibility_version", + expected_storage_version="v1.0.0+", + ), + ConnectionStorageKeyBuildE2ETestCase( + description="an unknown connection key is ignored", + connection_key="unknown_option", + expected_storage_version="v1.0.0+", + ), + ], + ids=lambda case: case.description, +) +def test_given_storage_key_in_connection_when_building_then_warehouse_keeps_default_storage( + test_case: ConnectionStorageKeyBuildE2ETestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="storage_key", + repo_files={ + "sqlbuild_project.toml": ( + 'name = "storage_key"\nadapter = "duckdb"\n\n' + f'[connection]\ndatabase = "warehouse.duckdb"\n{test_case.connection_key} = "latest"\n' + ), + "models/orders.sql": "MODEL (description 'Test model orders.');\nSELECT 1 AS order_id\n", + }, + ) + + built: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "build"), project_dir=project_dir + ) + + assert built.returncode == 0, built.stdout + built.stderr + tags: list[tuple[object, ...]] = query_duckdb( + db_path=project_dir / "warehouse.duckdb", + sql="SELECT tags FROM duckdb_databases() WHERE database_name = 'warehouse'", + ) + assert tags == [({"storage_version": test_case.expected_storage_version},)] diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/build/test_no_load_sources.py b/tests/e2e/src/sqlbuild/cli/commands/main/build/test_no_load_sources.py new file mode 100644 index 0000000000..50529a3f9f --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/build/test_no_load_sources.py @@ -0,0 +1,114 @@ +"""E2E coverage for builds with loading off: managed sources are audited, never reloaded.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.build._test_types import NoLoadBuildE2ETestCase +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import ( + execute_duckdb, + prepare_inline_project, + query_duckdb, + run_sqb, +) + +_AUTO_LOAD_OFF: str = "\n[settings]\nauto_load_sources = false\n" + + +@pytest.mark.parametrize( + "test_case", + [ + NoLoadBuildE2ETestCase( + description="no-load flag without a selection audits the existing data", + second_build=("--no-color", "build", "--no-load"), + project_settings="", + expected_exit_code=1, + expected_audit_reported=True, + ), + NoLoadBuildE2ETestCase( + description="auto_load_sources disabled audits the existing data", + second_build=("--no-color", "build"), + project_settings=_AUTO_LOAD_OFF, + expected_exit_code=1, + expected_audit_reported=True, + ), + NoLoadBuildE2ETestCase( + description="a source selected with the no-load flag is audited, not loaded", + second_build=("--no-color", "build", "--no-load", "--select", "raw_orders"), + project_settings="", + expected_exit_code=1, + expected_audit_reported=True, + ), + NoLoadBuildE2ETestCase( + description="a source selected with auto_load_sources disabled is audited, not loaded", + second_build=("--no-color", "build", "--select", "raw_orders"), + project_settings=_AUTO_LOAD_OFF, + expected_exit_code=1, + expected_audit_reported=True, + ), + NoLoadBuildE2ETestCase( + description="a source outside the selection is neither loaded nor audited", + second_build=("--no-color", "build", "--no-load", "--select", "orders"), + project_settings="", + expected_exit_code=0, + expected_audit_reported=False, + ), + ], + ids=lambda case: case.description, +) +def test_given_loaded_source_when_building_without_loading_then_only_its_audits_run( + test_case: NoLoadBuildE2ETestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="no_load_sources", + repo_files={ + "sqlbuild_project.toml": ( + 'name = "no_load_sources"\nadapter = "duckdb"\n\n' + '[connection]\ndatabase = "warehouse.duckdb"\n\n' + '[defaults]\nmaterialized = "table"\n' + ), + "sources/raw.yml": ( + "sources:\n - name: raw_orders\n description: Test source raw_orders.\n" + " managed: true\n write_strategy: append\n columns:\n" + " - name: order_id\n type: INTEGER\n audits: [not_null]\n" + ), + "python/loaders/orders.py": ( + "from sqlbuild.loaders import loader\n\n\n" + "@loader()\ndef raw_orders(ctx):\n" + " '''Test loader raw_orders.'''\n return [{'order_id': 1}]\n" + ), + "models/orders.sql": ( + "MODEL (description 'Test model orders.');\n" + 'SELECT r.order_id FROM __source("raw_orders") AS r\n' + ), + }, + ) + first: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "build"), project_dir=project_dir + ) + assert first.returncode == 0, first.stdout + first.stderr + execute_duckdb( + db_path=project_dir / "warehouse.duckdb", + sql="INSERT INTO raw_orders (order_id) VALUES (NULL)", + ) + settings: Path = project_dir / "sqlbuild_project.toml" + _ = settings.write_text(settings.read_text() + test_case.project_settings) + + second: subprocess.CompletedProcess[str] = run_sqb( + command=test_case.second_build, project_dir=project_dir + ) + + assert second.returncode == test_case.expected_exit_code, second.stdout + second.stderr + assert ("not_null (raw_orders.order_id)" in second.stdout) == test_case.expected_audit_reported + rows: list[tuple[object, ...]] = query_duckdb( + db_path=project_dir / "warehouse.duckdb", sql="SELECT COUNT(*) FROM raw_orders" + ) + assert rows == [(test_case.expected_order_rows,)] + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/build/test_source_end_audits.py b/tests/e2e/src/sqlbuild/cli/commands/main/build/test_source_end_audits.py new file mode 100644 index 0000000000..025baf3948 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/build/test_source_end_audits.py @@ -0,0 +1,65 @@ +"""E2E coverage of audits of sources no model reads: run with loading off or when selected.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.build._test_types import ( + SourceEndAuditBuildE2ETestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + +_PROJECT_FILES: dict[str, str] = { + "sqlbuild_project.toml": ( + 'name = "source_end_audits"\nadapter = "duckdb"\n\n' + '[connection]\ndatabase = "warehouse.duckdb"\n\n[defaults]\nmaterialized = "table"\n' + ), + "sources/raw.yml": ( + "sources:\n - name: raw_refunds\n description: Test source raw_refunds.\n" + " schema: main\n table: raw_refunds\n columns:\n" + " - name: refund_id\n type: INTEGER\n audits: [not_null]\n" + ), + "models/orders.sql": "MODEL (description 'Test model orders.');\nSELECT 1 AS order_id\n", +} + + +@pytest.mark.parametrize( + "test_case", + [ + SourceEndAuditBuildE2ETestCase( + description="with loading on an unread source is not audited", + build_arguments=(), + expected_exit_code=0, + expected_audit_reported=False, + ), + SourceEndAuditBuildE2ETestCase( + description="with loading off an unread source is audited", + build_arguments=("--no-load",), + expected_exit_code=1, + expected_audit_reported=True, + ), + SourceEndAuditBuildE2ETestCase( + description="a source selected by name is audited", + build_arguments=("--select", "raw_refunds"), + expected_exit_code=1, + expected_audit_reported=True, + ), + ], + ids=lambda case: case.description, +) +def test_given_unread_source_with_missing_table_when_building_then_audit_runs_only_when_asked( + test_case: SourceEndAuditBuildE2ETestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, project_name="source_end_audits", repo_files=_PROJECT_FILES + ) + + built: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "build", *test_case.build_arguments), project_dir=project_dir + ) + + assert built.returncode == test_case.expected_exit_code, built.stdout + built.stderr + assert ("not_null on raw_refunds" in built.stdout) == test_case.expected_audit_reported diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/build/test_unbuilt_dependencies.py b/tests/e2e/src/sqlbuild/cli/commands/main/build/test_unbuilt_dependencies.py new file mode 100644 index 0000000000..eea6167c71 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/build/test_unbuilt_dependencies.py @@ -0,0 +1,177 @@ +"""E2E coverage for readers of resources a build failed to build.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.build._test_types import ( + HookSkippedAuditBuildE2ETestCase, + UnbuiltDependencyBuildE2ETestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + +_BASE_FILES: dict[str, str] = { + "sqlbuild_project.toml": ( + 'name = "unbuilt_dependencies"\nadapter = "duckdb"\n\n' + '[connection]\ndatabase = "warehouse.duckdb"\n\n[defaults]\nmaterialized = "table"\n' + ), + "models/broken_orders.sql": ( + "MODEL (description 'Test model broken_orders.');\n" + "SELECT CAST('not a number' AS INTEGER) AS order_id\n" + ), + "models/customers.sql": ( + "MODEL (description 'Test model customers.');\nSELECT 1 AS customer_id\n" + ), +} + +_HOOK_FILE: str = ( + "from sqlbuild.hooks import hook\n\n\n" + "@hook\ndef skip_model(ctx):\n" + " '''Test hook skip_model.'''\n" + " return ctx.skip(reason='paused')\n" +) +_CUSTOMERS_SQL: str = "SELECT 1 AS customer_id\n" + + +@pytest.mark.parametrize( + "test_case", + [ + UnbuiltDependencyBuildE2ETestCase( + description="end-of-build audit over a failed model is skipped", + repo_files={ + **_BASE_FILES, + "audits/singular/orders_have_customers.sql": ( + "AUDIT ();\n" + 'SELECT o.order_id FROM __ref("broken_orders") AS o\n' + 'LEFT JOIN __ref("customers") AS c ON c.customer_id = o.order_id\n' + "WHERE c.customer_id IS NULL\n" + ), + }, + expected_stdout_fragments=( + "broken_orders", + "Could not convert string", + "audit orders_have_customers SKIP", + "Skipped (1): orders_have_customers (audit, reads broken_orders, which was not " + "built)", + "SKIP=1", + ), + unexpected_stdout_fragments=("does not exist",), + ), + UnbuiltDependencyBuildE2ETestCase( + description="full refresh on a fresh warehouse skips the audit over a failed model", + repo_files={ + **_BASE_FILES, + "audits/singular/orders_have_customers.sql": ( + "AUDIT ();\n" + 'SELECT o.order_id FROM __ref("broken_orders") AS o\n' + 'LEFT JOIN __ref("customers") AS c ON c.customer_id = o.order_id\n' + "WHERE c.customer_id IS NULL\n" + ), + }, + expected_stdout_fragments=( + "audit orders_have_customers SKIP", + "Skipped (1): orders_have_customers (audit, reads broken_orders, which was not " + "built)", + ), + unexpected_stdout_fragments=("does not exist",), + build_arguments=("--full-refresh",), + ), + UnbuiltDependencyBuildE2ETestCase( + description="test calling a skipped table function is skipped", + repo_files={ + **_BASE_FILES, + "functions/sql/customer_orders.sql": ( + 'FUNCTION (description "Test function customer_orders.",\n' + " arguments (p_customer_id INTEGER), returns table (order_id INTEGER));\n" + 'SELECT o.order_id FROM __ref("broken_orders") AS o\n' + "WHERE o.order_id = p_customer_id\n" + ), + "models/summary.sql": ( + "MODEL (description 'Test model summary.');\n" + "SELECT c.customer_id, f.order_id\n" + 'FROM __ref("customers") AS c,\n' + ' __table_fn("customer_orders")(c.customer_id) AS f\n' + ), + "tests/unit/test_summary.sql": ( + "TEST();\nWITH\n__ref__customers AS (SELECT 1 AS customer_id),\n" + "__expected__summary AS (SELECT 1 AS customer_id, 1 AS order_id WHERE FALSE)\n" + "SELECT 1\n" + ), + }, + expected_stdout_fragments=("Skipped (2): summary, customer_orders",), + unexpected_stdout_fragments=("test_summary", "does not exist"), + ), + ], + ids=lambda case: case.description, +) +def test_given_failed_model_when_building_then_skips_readers_of_unbuilt_resources( + test_case: UnbuiltDependencyBuildE2ETestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, project_name="unbuilt_dependencies", repo_files=test_case.repo_files + ) + + result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "build", *test_case.build_arguments), project_dir=project_dir + ) + + assert result.returncode == 1, result.stdout + result.stderr + assert all(fragment in result.stdout for fragment in test_case.expected_stdout_fragments) + assert not any(fragment in result.stdout for fragment in test_case.unexpected_stdout_fragments) + + +@pytest.mark.parametrize( + "test_case", + [ + HookSkippedAuditBuildE2ETestCase( + description="end-of-build audit over a model a hook skipped still runs", + hooked_model=( + "MODEL (description 'Test model customers.', " + 'pre_hooks [python("skip_model")]);\n' + _CUSTOMERS_SQL + ), + expected_exit_code=0, + expected_stdout_fragments=("audit customers_exist PASS",), + ), + ], + ids=lambda case: case.description, +) +def test_given_hook_skipped_existing_model_when_rebuilding_then_its_end_audits_run( + test_case: HookSkippedAuditBuildE2ETestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="unbuilt_dependencies", + repo_files={ + "sqlbuild_project.toml": _BASE_FILES["sqlbuild_project.toml"], + "models/customers.sql": "MODEL (description 'Test model customers.');\n" + + _CUSTOMERS_SQL, + "models/orders.sql": "MODEL (description 'Test model orders.');\nSELECT 1 AS order_id\n", + "audits/singular/customers_exist.sql": ( + "AUDIT ();\n" + 'SELECT o.order_id FROM __ref("orders") AS o\n' + 'LEFT JOIN __ref("customers") AS c ON c.customer_id = o.order_id\n' + "WHERE c.customer_id IS NULL\n" + ), + }, + ) + first: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "build"), project_dir=project_dir + ) + assert first.returncode == 0, first.stdout + first.stderr + (project_dir / "hooks/python").mkdir(parents=True) + _ = (project_dir / "hooks/python/skips.py").write_text(_HOOK_FILE) + _ = (project_dir / "models/customers.sql").write_text(test_case.hooked_model) + + rebuilt: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "build"), project_dir=project_dir + ) + + assert rebuilt.returncode == test_case.expected_exit_code, rebuilt.stdout + rebuilt.stderr + assert all(fragment in rebuilt.stdout for fragment in test_case.expected_stdout_fragments) + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/_test_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/_test_types.py index 8ea2bac6dd..f9f228811d 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/compile/_test_types.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/_test_types.py @@ -1260,3 +1260,55 @@ class GeneratedBenchmarkCompileTestCase: expected_min_models: int expected_max_wall_seconds: float expected_max_rss_bytes: int + + +@dataclass(frozen=True) +class RelationshipsTargetCompileTestCase: + """A relationships audit target and how compiling it must end.""" + + description: str + target: str + expected_exit_code: int + expected_output_fragment: str + + +@dataclass(frozen=True) +class ContractCastTypeTestCase: + """An outer cast on a contracted Snowflake output, compile's exit code and report fragments.""" + + description: str + projection: str + expected_exit_code: int + expected_fragments: tuple[str, ...] + + +@dataclass(frozen=True) +class ParameterisedTestLintTestCase: + """A parameterised SQL test linted by SQL rules, and the commands that must succeed on it.""" + + description: str + selected_rules: str + command: tuple[str, ...] + expected_fragment: str + + +@dataclass(frozen=True) +class BigQueryStringContractTestCase: + """A BigQuery contract column of a STRING type and the cast that fills it.""" + + description: str + declared_type: str + cast_type: str + expected_exit_code: int + + +@dataclass(frozen=True) +class TypedCallContractTestCase: + """A typed SQL macro filling a Snowflake contract column, and how SQBRCONTRACT105 judges it.""" + + description: str + returns: str + body: str + projection: str + expected_exit_code: int + expected_fragments: tuple[str, ...] diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/header_strings/_test_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/header_strings/_test_types.py index 386cfb5d07..8a2651e38f 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/compile/header_strings/_test_types.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/header_strings/_test_types.py @@ -52,3 +52,12 @@ class MacroDescriptionTestCase: description: str header: str expected_exit_code: int + + +@dataclass(frozen=True) +class MetaDescriptionTestCase: + """An audit `meta` value under a `description` key, and the meta the build records.""" + + description: str + meta: str + expected_meta: dict[str, str] diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/header_strings/test_header_strings.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/header_strings/test_header_strings.py index 0ee70b2310..e84f2ca867 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/compile/header_strings/test_header_strings.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/header_strings/test_header_strings.py @@ -15,6 +15,7 @@ HeaderQuoteErrorTestCase, ListStringTestCase, MacroDescriptionTestCase, + MetaDescriptionTestCase, ) from tests.e2e.src.sqlbuild.cli.commands.main.compile.header_strings.helpers import ( header_strings_project, @@ -366,3 +367,39 @@ def test_given_adjacent_strings_outside_descriptions_when_compiling_then_they_re assert compiled.returncode == 0, compiled.stdout + compiled.stderr manifest: dict[str, Any] = json.loads((project_dir / "target/manifest.json").read_text()) assert manifest["nodes"]["model.shop.customers"]["tags"] == test_case.expected_tags + + +@pytest.mark.parametrize( + "test_case", + [ + MetaDescriptionTestCase( + description="escaped quote then a separator inside a meta description", + meta='meta (description "say \\", then", owner "ops")', + expected_meta={"description": 'say ", then', "owner": "ops"}, + ), + ], + ids=lambda case: case.description, +) +def test_given_meta_description_with_escaped_quote_when_building_then_meta_keeps_its_text( + test_case: MetaDescriptionTestCase, tmp_path: Path +) -> None: + project_dir: Path = header_strings_project( + tmp_path, + { + "models/customers.sql": ( + 'MODEL (\n description "Customers.",\n' + f" audits [not_null (column customer_id, {test_case.meta})],\n);\n\n" + "SELECT 1 AS customer_id\n" + ) + }, + ) + + built: subprocess.CompletedProcess[str] = run_sqb( + project_dir=project_dir, command=("--no-color", "build", "--json") + ) + + assert built.returncode == 0, built.stdout + built.stderr + audit_meta: list[object] = [ + check.get("audit_meta") for check in json.loads(built.stdout)["checks"] + ] + assert audit_meta == [test_case.expected_meta] diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/sql_macros/test_sql_macro_returns.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/sql_macros/test_sql_macro_returns.py index 370e4570ec..490781b16d 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/compile/sql_macros/test_sql_macro_returns.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/sql_macros/test_sql_macro_returns.py @@ -276,6 +276,51 @@ "range 1..=38", ), ), + SqlMacroCompileTestCase( + description="an_invalid_argument_type_is_reported_on_postgres", + files={ + **adapter_project("postgres"), + **text_macro("m", "BIGINT", "@value", argument="FOO BAR"), + **orders_model( + 'SELECT @sql.m(o.amount_cents) AS v FROM __ref("stg_orders") AS o\n' + ), + }, + expected_returncode=1, + expected_fragments=( + "error[P015]: argument value of macro m() type 'FOO BAR' is not valid for adapter " + "'postgres' SQL analysis dialect 'postgres'", + ), + ), + SqlMacroCompileTestCase( + description="an_argument_decimal_wider_than_the_dialect_holds_is_reported", + files={ + **text_macro("m", "BIGINT", "@value", argument="DECIMAL(50, 2)"), + **orders_model( + 'SELECT @sql.m(o.amount_cents) AS v FROM __ref("stg_orders") AS o\n' + ), + }, + expected_returncode=1, + expected_fragments=( + "error[P015]: argument value of macro m() type 'DECIMAL(50, 2)' is not valid for " + "adapter 'duckdb' SQL analysis dialect 'duckdb'", + ), + ), + SqlMacroCompileTestCase( + description="a_returns_numeric_wider_than_bigquery_holds_is_reported", + files={ + **adapter_project("bigquery"), + **text_macro("m", "NUMERIC(40, 2)", "@value", argument="ANY"), + **orders_model( + 'SELECT @sql.m(o.amount_cents) AS v FROM __ref("stg_orders") AS o\n' + ), + }, + expected_returncode=1, + expected_fragments=( + "error[P015]: returns of macro m() type 'NUMERIC(40, 2)' is not valid for adapter " + "'bigquery' SQL analysis dialect 'bigquery': precision 40 and scale 2 exceed " + "BigQuery's limits of 29 integer digits and scale 9", + ), + ), SqlMacroCompileTestCase( description="a_returns_decimal_scale_above_its_precision_is_reported", files={ @@ -363,7 +408,6 @@ def test_given_optional_returns_when_compiling_then_calls_are_typed_and_checked( SqlMacroCompileTestCase( description="returns_type_of_types_a_later_union_branch", files={ - **with_rules("SQBRCONTRACT105"), **_FIRST_OF, **contract_model("orders", "day_value (type DATE)", _TYPE_OF_UNION), }, diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/sql_macros/test_sql_macro_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/sql_macros/test_sql_macro_types.py index 6d837cd889..0e2c510d29 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/compile/sql_macros/test_sql_macro_types.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/sql_macros/test_sql_macro_types.py @@ -37,8 +37,8 @@ _TYPED_COLUMNS: str = "order_id (type BIGINT), shouted (type VARCHAR), status_bar (type VARCHAR)" _TYPED_RULES_AND_MACROS: dict[str, str] = { **with_rules("SQBRCONTRACT105"), - **text_macro("shouted", "VARCHAR", "UPPER(@value)"), - **text_macro("status_bar", "VARCHAR", "bar(LENGTH(@value), 0, 10)"), + **text_macro("shouted", "VARCHAR", "UPPER(@value)::VARCHAR"), + **text_macro("status_bar", "VARCHAR", "CAST(bar(LENGTH(@value), 0, 10) AS VARCHAR)"), } _TYPED_CALLS: dict[str, str] = { **_TYPED_RULES_AND_MACROS, @@ -428,7 +428,7 @@ def test_given_an_any_argument_misused_when_compiling_then_it_fails( description="a_call_in_a_later_main_query_branch_matching_its_sibling_is_typed", files={ **with_rules("SQBRCONTRACT105"), - **text_macro("m", "BIGINT", "NULL"), + **text_macro("m", "BIGINT", "CAST(NULL AS BIGINT)"), **contract_model( "orders", "v (type BIGINT)", diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_bigquery_string_contracts.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_bigquery_string_contracts.py new file mode 100644 index 0000000000..fb3fe33fb2 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_bigquery_string_contracts.py @@ -0,0 +1,69 @@ +"""E2E coverage of BigQuery contracts, where every variable-length text type is STRING.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.compile._test_types import ( + BigQueryStringContractTestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + +_PROJECT_TOML: str = ( + 'name = "shop"\nadapter = "bigquery"\ndefault_target = "dev"\n\n' + '[targets.dev]\ndatabase = "warehouse"\nschema = "analytics"\n\n' + "[scopes]\nenforce_placement = false\n" +) +_MODEL: str = ( + 'MODEL (\n description "Order notes.",\n contract enforced,\n' + " columns (note (type {declared_type})),\n);\n\n" + "SELECT CAST('gift' AS {cast_type}) AS note\n" +) + + +@pytest.mark.parametrize( + "test_case", + [ + BigQueryStringContractTestCase( + description="a sized VARCHAR fills a STRING of that length", + declared_type="STRING(10)", + cast_type="VARCHAR(10)", + expected_exit_code=0, + ), + BigQueryStringContractTestCase( + description="a STRING fills a STRING", + declared_type="STRING", + cast_type="STRING", + expected_exit_code=0, + ), + BigQueryStringContractTestCase( + description="a different length is still a mismatch", + declared_type="STRING(10)", + cast_type="STRING(5)", + expected_exit_code=1, + ), + ], + ids=lambda case: case.description, +) +def test_given_bigquery_string_contract_when_compiling_then_text_types_compare_as_string( + test_case: BigQueryStringContractTestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="shop", + repo_files={ + "sqlbuild_project.toml": _PROJECT_TOML, + "models/order_notes.sql": _MODEL.format( + declared_type=test_case.declared_type, cast_type=test_case.cast_type + ), + }, + ) + + compiled: subprocess.CompletedProcess[str] = run_sqb( + project_dir=project_dir, command=("--no-color", "compile") + ) + + assert compiled.returncode == test_case.expected_exit_code, compiled.stdout + compiled.stderr diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_compile_early_builtin_rules.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_compile_early_builtin_rules.py index 3f62424a1f..c86a335b70 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_compile_early_builtin_rules.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_compile_early_builtin_rules.py @@ -40,7 +40,7 @@ " return []\n" " return [ctx.finding(subject=model)]\n" ) -_STAR_FINDING: tuple[str, str, int, int] = ("SQBRMODEL102", "models/order_totals.sql", 1, 8) +_STAR_FINDING: tuple[str, str, int, int] = ("SQBRMODEL102", "models/order_totals.sql", 2, 8) @pytest.mark.parametrize( diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_contract_cast_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_contract_cast_types.py new file mode 100644 index 0000000000..9492bf3f4e --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_contract_cast_types.py @@ -0,0 +1,199 @@ +"""E2E coverage of outer casts that establish a contract type, compared as the dialect does.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.compile._test_types import ( + ContractCastTypeTestCase, + TypedCallContractTestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + +_PROJECT_TOML: str = ( + 'name = "shop"\nadapter = "snowflake"\ndefault_target = "dev"\n\n' + '[targets.dev]\ndatabase = "warehouse"\nschema = "analytics"\n\n' + '[scopes]\nenforce_placement = false\n\n[rules]\nselect = ["SQBRCONTRACT105"]\n' +) +_STATUS_LABEL_MACRO: str = ( + "MACRO (\n" + ' description "Map a status code to a label.",\n' + " arguments (code VARCHAR),\n" + " returns VARCHAR,\n" + ");\n\n" + "CASE @code WHEN 'a' THEN 'ACTIVE' WHEN 'c' THEN 'CLOSED' END\n" +) +_ORDERS_MODEL: str = ( + 'MODEL (description "Orders.");\n\n' + "SELECT CAST(1 AS INTEGER) AS order_id, CAST('a' AS VARCHAR) AS code\n" +) +_LABELS_MODEL: str = ( + "MODEL (\n" + ' description "Order status labels.",\n' + " contract enforced,\n" + " columns (\n" + " order_id (type INTEGER),\n" + " status_label (type VARCHAR),\n" + " ),\n" + ");\n\n" + "SELECT\n" + " o.order_id,\n" + " {projection} AS status_label\n" + 'FROM __ref("orders") AS o\n' + "WHERE o.order_id > 0\n" +) + +_TYPED_MACRO: str = ( + "MACRO (\n" + ' description "Label a status code.",\n' + " arguments (code VARCHAR),\n" + " returns {returns},\n" + ");\n\n" + "{body}\n" +) + + +@pytest.mark.parametrize( + "test_case", + [ + ContractCastTypeTestCase( + description="a cast to the maximum VARCHAR length is the declared VARCHAR", + projection="CAST(@sql.status_label(o.code) AS VARCHAR(16777216))", + expected_exit_code=0, + expected_fragments=("0 errors",), + ), + ContractCastTypeTestCase( + description="a cast to STRING is the declared VARCHAR", + projection="CAST(@sql.status_label(o.code) AS STRING)", + expected_exit_code=0, + expected_fragments=("0 errors",), + ), + ContractCastTypeTestCase( + description="a narrower cast names both types, points at the cast and asks to change it", + projection="CAST(@sql.status_label(o.code) AS VARCHAR(10))", + expected_exit_code=1, + expected_fragments=( + 'error[SQBRCONTRACT105]: output "status_label" has an outer cast to VARCHAR(10),' + " which is not its declared contract type VARCHAR", + "--> models/labels.sql:12:3", + "help: Change the outer cast to CAST(... AS VARCHAR).", + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_outer_cast_on_contract_output_when_compiling_then_types_compare_as_snowflake_does( + test_case: ContractCastTypeTestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="shop", + repo_files={ + "sqlbuild_project.toml": _PROJECT_TOML, + "macros/sql/status_label.sql": _STATUS_LABEL_MACRO, + "models/orders.sql": _ORDERS_MODEL, + "models/labels.sql": _LABELS_MODEL.format(projection=test_case.projection), + }, + ) + + compiled: subprocess.CompletedProcess[str] = run_sqb( + project_dir=project_dir, command=("--no-color", "compile") + ) + report: str = compiled.stdout + compiled.stderr + + assert compiled.returncode == test_case.expected_exit_code, report + assert [fragment in report for fragment in test_case.expected_fragments] == [ + True for _ in test_case.expected_fragments + ], report + + +@pytest.mark.parametrize( + "test_case", + [ + TypedCallContractTestCase( + description="a body cast to its returns type establishes the contract type", + returns="VARCHAR", + body="CAST(UPPER(@code) AS STRING)", + projection="@sql.status_label(o.code)", + expected_exit_code=0, + expected_fragments=("0 errors",), + ), + TypedCallContractTestCase( + description="a double-colon body cast establishes the contract type", + returns="VARCHAR", + body="LEFT(@code, 3)::VARCHAR", + projection="@sql.status_label(o.code)", + expected_exit_code=0, + expected_fragments=("0 errors",), + ), + TypedCallContractTestCase( + description="a body without a cast is flagged", + returns="VARCHAR", + body="'ACTIVE'", + projection="@sql.status_label(o.code)", + expected_exit_code=1, + expected_fragments=( + 'error[SQBRCONTRACT105]: output "status_label" is a call to SQL macro ' + "status_label(), which declares returns VARCHAR, but its body isn't cast to it", + "--> models/labels.sql:12:3", + "help: Cast the result in the macro or at the call", + ), + ), + TypedCallContractTestCase( + description="a body cast to another type than the contract is flagged", + returns="VARCHAR", + body="CAST(@code AS VARCHAR(6))", + projection="@sql.status_label(o.code)", + expected_exit_code=1, + expected_fragments=( + 'output "status_label" has an outer cast to VARCHAR(6), which is not its declared ' + "contract type VARCHAR", + ), + ), + TypedCallContractTestCase( + description="a returns TYPE OF call is flagged", + returns="TYPE OF code", + body="LEFT(@code, 3)", + projection="@sql.status_label(o.code)", + expected_exit_code=1, + expected_fragments=("whose returns TYPE OF does not set the column's physical type",), + ), + TypedCallContractTestCase( + description="a returns TYPE OF call with an outer cast is accepted", + returns="TYPE OF code", + body="LEFT(@code, 3)", + projection="CAST(@sql.status_label(o.code) AS VARCHAR)", + expected_exit_code=0, + expected_fragments=("0 errors",), + ), + ], + ids=lambda case: case.description, +) +def test_given_typed_macro_call_on_contract_output_when_compiling_then_only_a_body_cast_counts( + test_case: TypedCallContractTestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="shop", + repo_files={ + "sqlbuild_project.toml": _PROJECT_TOML, + "macros/sql/status_label.sql": _TYPED_MACRO.format( + returns=test_case.returns, body=test_case.body + ), + "models/orders.sql": _ORDERS_MODEL, + "models/labels.sql": _LABELS_MODEL.format(projection=test_case.projection), + }, + ) + + compiled: subprocess.CompletedProcess[str] = run_sqb( + project_dir=project_dir, command=("--no-color", "compile") + ) + report: str = compiled.stdout + compiled.stderr + + assert compiled.returncode == test_case.expected_exit_code, report + assert [fragment in report for fragment in test_case.expected_fragments] == [ + True for _ in test_case.expected_fragments + ], report diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_native_type_system.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_native_type_system.py index b6516a8ce7..08c286aa81 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_native_type_system.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_native_type_system.py @@ -42,6 +42,22 @@ SELECT order_id, amount AS total FROM __ref("orders") """ +_SIZED_TEXT_ORDERS: str = """MODEL ( + description "Orders whose status is cast to a sized TEXT.", + materialized table, +); + +SELECT CAST(1 AS INT) AS order_id, CAST('new' AS TEXT(5)) AS status +""" +_SIZED_VARCHAR_STATUSES: str = """MODEL ( + description "Order statuses contracted as the VARCHAR that a sized TEXT is.", + materialized table, + contract enforced, + columns (status (type VARCHAR(5))), +); + +SELECT status FROM __ref("orders") +""" _IMMEDIATE_PROMOTION_TOML: str = ( _PROJECT_TOML + '\n[settings]\ntable_promotion_mode = "immediate"\n' ) @@ -126,6 +142,16 @@ expected_exit_code=1, expected_report_text="inferred as numeric(10,2) but declared type is DECIMAL(12,2)", ), + NativeTypeSystemTestCase( + description="a sized TEXT satisfies a VARCHAR contract of the same length", + project_files={ + "sqlbuild_project.toml": _PROJECT_TOML, + "models/orders.sql": _SIZED_TEXT_ORDERS, + "models/order_statuses.sql": _SIZED_VARCHAR_STATUSES, + }, + expected_exit_code=0, + expected_report_text='"has_errors": false', + ), NativeTypeSystemTestCase( description="every contract family fails alike under immediate promotion", project_files={ diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_parameterised_test_lint.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_parameterised_test_lint.py new file mode 100644 index 0000000000..f1d374bb3a --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_parameterised_test_lint.py @@ -0,0 +1,81 @@ +"""E2E coverage of SQL rules over parameterised SQL tests, which they lint like any test.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.compile._test_types import ( + ParameterisedTestLintTestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + +_PROJECT_TOML: str = ( + 'name = "shop"\nadapter = "duckdb"\n\n[connection]\ndatabase = "shop.duckdb"\n\n' + "[scopes]\nenforce_placement = false\n\n[rules]\nselect = [{rules}]\n" +) +_ORDERS_MODEL: str = ( + "MODEL (description \"Orders.\");\n\nSELECT 1 AS order_id, 'completed' AS status\n" +) +_STAGED_ORDERS_MODEL: str = ( + 'MODEL (description "Staged orders.");\n\n' + "SELECT o.order_id, CASE o.status WHEN 'completed' THEN 'completed' ELSE 'open' END AS status\n" + 'FROM __ref("orders") AS o\n' +) +_STATUS_TEST: str = ( + "TEST (\n" + ' name "order_status_maps",\n' + " parameters (source_status string, expected_status string),\n" + " cases (\n" + ' completed (source_status "completed", expected_status "completed"),\n' + ' pending (source_status "pending", expected_status "open"),\n' + " ),\n" + ");\n\n" + "WITH\n" + '__ref__orders AS (SELECT 1 AS order_id, @param("source_status") AS status),\n' + '__expected__stg_orders AS (SELECT 1 AS order_id, @param("expected_status") AS status)\n' +) + + +@pytest.mark.parametrize( + "test_case", + [ + ParameterisedTestLintTestCase( + description="compile lints the parameterised test instead of failing with L005", + selected_rules='"SQBRSQL004"', + command=("--no-color", "compile"), + expected_fragment="0 errors", + ), + ParameterisedTestLintTestCase( + description="the cases still run once the test file is linted", + selected_rules='"SQBRSQL004"', + command=("--no-color", "test"), + expected_fragment="PASS=2 FAIL=0 TOTAL=2", + ), + ], + ids=lambda case: case.description, +) +def test_given_parameterised_test_and_sql_rules_when_compiling_then_the_test_is_linted( + test_case: ParameterisedTestLintTestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="shop", + repo_files={ + "sqlbuild_project.toml": _PROJECT_TOML.format(rules=test_case.selected_rules), + "models/orders.sql": _ORDERS_MODEL, + "models/stg_orders.sql": _STAGED_ORDERS_MODEL, + "tests/unit/test_stg_orders_status.sql": _STATUS_TEST, + }, + ) + + ran: subprocess.CompletedProcess[str] = run_sqb( + project_dir=project_dir, command=test_case.command + ) + report: str = ran.stdout + ran.stderr + + assert ran.returncode == 0, report + assert "L005" not in report + assert test_case.expected_fragment in report diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_relationships_target.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_relationships_target.py new file mode 100644 index 0000000000..dc1415f9a0 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_relationships_target.py @@ -0,0 +1,62 @@ +"""E2E coverage for the relation a relationships audit reads.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.compile._test_types import ( + RelationshipsTargetCompileTestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + + +@pytest.mark.parametrize( + "test_case", + [ + RelationshipsTargetCompileTestCase( + description="quoted relation name is rejected", + target='"customers"', + expected_exit_code=1, + expected_output_fragment='write to __ref("customers")', + ), + RelationshipsTargetCompileTestCase( + description="reference call compiles", + target='__ref("customers")', + expected_exit_code=0, + expected_output_fragment="Project compiled", + ), + ], + ids=lambda case: case.description, +) +def test_given_relationships_audit_when_compiling_then_target_must_be_a_reference( + test_case: RelationshipsTargetCompileTestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="relationships_target", + repo_files={ + "sqlbuild_project.toml": 'name = "relationships_target"\nadapter = "duckdb"\n', + "models/customers.sql": ( + "MODEL (description 'Test model customers.');\nSELECT 1 AS id\n" + ), + "models/orders.sql": ( + "MODEL (description 'Test model orders.', columns (customer_id (\n" + f" audits [relationships (to {test_case.target}, field id)])));\n" + "SELECT 1 AS order_id, 1 AS customer_id\n" + ), + }, + ) + + result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "compile"), project_dir=project_dir + ) + + assert result.returncode == test_case.expected_exit_code, result.stdout + result.stderr + assert test_case.expected_output_fragment in result.stdout + result.stderr + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/format/helpers.py b/tests/e2e/src/sqlbuild/cli/commands/main/format/helpers.py index 186a587394..a456988544 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/format/helpers.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/format/helpers.py @@ -8,6 +8,7 @@ import sys import time from pathlib import Path +from typing import Any from scripts.format_performance._helpers.model_project import write_model_format_project from sqlbuild._native import compute_query_hash @@ -188,3 +189,25 @@ def stored_descriptions(project_dir: Path) -> dict[str, str]: ) assert dag.returncode == 0, dag.stderr return {node["name"]: node["description"] for node in json.loads(dag.stdout)["nodes"]} + + +def stored_column_descriptions(project_dir: Path) -> dict[str, str]: + """The description `sqb dag` reports for each column of the project's models, by node.column.""" + + dag: subprocess.CompletedProcess[str] = run_sqb( + project_dir=project_dir, arguments=("--no-color", "dag", "--json") + ) + assert dag.returncode == 0, dag.stderr + descriptions: dict[str, str] = {} + for node in json.loads(dag.stdout)["nodes"]: + descriptions.update(_node_column_descriptions(node)) + return descriptions + + +def _node_column_descriptions(node: dict[str, Any]) -> dict[str, str]: + """One `sqb dag` node's column descriptions, by node.column.""" + + return { + f"{node['name']}.{column['name']}": column.get("description", "") + for column in node["columns"] + } diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/format/test_description_layout.py b/tests/e2e/src/sqlbuild/cli/commands/main/format/test_description_layout.py index c8e0052e2f..7a56152225 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/format/test_description_layout.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/format/test_description_layout.py @@ -15,7 +15,10 @@ QuoteFixTestCase, RefusedQuoteFixTestCase, ) -from tests.e2e.src.sqlbuild.cli.commands.main.format.helpers import stored_descriptions +from tests.e2e.src.sqlbuild.cli.commands.main.format.helpers import ( + stored_column_descriptions, + stored_descriptions, +) from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb _PROJECT_FILE: str = ( @@ -75,6 +78,69 @@ ");\n" ), ), + DescriptionLayoutTestCase( + description="a_long_column_description_becomes_a_block_at_its_depth", + header=( + "MODEL (\n" + ' description "Customers.",\n' + " columns (\n" + ' customer_id (type INTEGER, description "The customer key, unique across ' + 'every channel the shop sells through, assigned at sign-up."),\n' + " ),\n" + ");\n" + ), + expected_header=( + "MODEL (\n" + ' description "Customers.",\n' + " columns (\n" + ' customer_id (type INTEGER, description """\n' + " The customer key, unique across every channel the shop sells through, " + "assigned at sign-up.\n" + ' """),\n' + " ),\n" + ");\n" + ), + ), + DescriptionLayoutTestCase( + description="a_short_wrapped_column_description_is_joined", + header=( + "MODEL (\n" + ' description "Customers.",\n' + ' columns (customer_id (type INTEGER, description """The customer\n' + ' key.""")),\n' + ");\n" + ), + expected_header=( + "MODEL (\n" + ' description "Customers.",\n' + ' columns (customer_id (type INTEGER, description "The customer key.")),\n' + ");\n" + ), + ), + DescriptionLayoutTestCase( + description="a_long_audit_description_becomes_a_block_at_its_depth", + header=( + "MODEL (\n" + ' description "Customers.",\n' + ' audits [not_null (column customer_id, description "Every customer must have ' + 'a key, otherwise downstream joins silently drop rows from reports.")],\n' + ");\n" + ), + expected_header=( + "MODEL (\n" + ' description "Customers.",\n' + " audits [\n" + " not_null (\n" + " column customer_id,\n" + ' description """\n' + " Every customer must have a key, otherwise downstream joins silently drop " + "rows from reports.\n" + ' """,\n' + " ),\n" + " ],\n" + ");\n" + ), + ), ], ids=lambda case: case.description, ) @@ -90,6 +156,7 @@ def test_given_model_description_when_formatting_then_layout_is_canonical_and_te }, ) before: dict[str, str] = stored_descriptions(project_dir) + columns_before: dict[str, str] = stored_column_descriptions(project_dir) formatted: subprocess.CompletedProcess[str] = run_sqb( project_dir=project_dir, command=("--no-color", "format") @@ -104,6 +171,7 @@ def test_given_model_description_when_formatting_then_layout_is_canonical_and_te model: str = (project_dir / "models/customers.sql").read_text() assert model == test_case.expected_header + _BODY assert stored_descriptions(project_dir) == before + assert stored_column_descriptions(project_dir) == columns_before @pytest.mark.parametrize( diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/rules/test_unevaluated.py b/tests/e2e/src/sqlbuild/cli/commands/main/rules/test_unevaluated.py index 13c28705b1..ddb2405d1e 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/rules/test_unevaluated.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/rules/test_unevaluated.py @@ -113,8 +113,8 @@ def test_given_guarded_resource_when_running_rules_then_reports_failure_and_cove "FROM items AS i WHERE i.quantities[1] > 7", ("SQBRMODEL101", "SQBRMODEL102", "SQBRDECLARATION102"), "SQBRMODEL102", - (("SQBRMODEL102", 2, 8),), - (("SQBRMODEL102", 2, 8), ("SQBRDECLARATION102", 3, 23)), + (("SQBRMODEL102", 3, 8),), + (("SQBRMODEL102", 3, 8), ("SQBRDECLARATION102", 4, 23)), ), ], ids=lambda case: case.description, diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/scenario/_test_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/_test_types.py index 2841c03b13..067698c193 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/scenario/_test_types.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/_test_types.py @@ -305,3 +305,38 @@ class UnmockedInputScenarioE2ETestCase: description: str expected_fragments: tuple[str, ...] unexpected_fragment: str + + +@dataclass(frozen=True) +class ScenarioInlineProjectE2ETestCase: + """One scenario command over an inline project and the output it must print.""" + + description: str + repo_files: dict[str, str] + command: tuple[str, ...] + expected_exit_code: int + expected_stdout_fragments: tuple[str, ...] + + +@dataclass(frozen=True) +class LocalVariantScenarioTestCase: + """A captured scenario whose fixture holds a VARIANT, replayed locally in strict mode.""" + + description: str + repo_files: dict[str, str] + scenario_name: str + expected_stdout_fragment: str + + +@dataclass(frozen=True) +class SnapshotHelperEditTestCase: + """A captured scenario, one edit to its files, and what strict local replay reports.""" + + description: str + repo_files: dict[str, str] + scenario_name: str + edited_path: str + edited_from: str + edited_to: str + expected_exit_code: int + expected_stdout_fragment: str diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/scenario/helpers.py b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/helpers.py index 94f157084d..a0f92daa25 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/scenario/helpers.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/helpers.py @@ -990,3 +990,205 @@ def build_unmocked_input_scenario_project_files() -> dict[str, str]: ), } return files + + +def build_lambda_scenario_project_files() -> dict[str, str]: + """Build a scenario whose model reads list items through lambdas beside a star CTE.""" + + return { + "sqlbuild_project.toml": ( + 'name = "lambda_scenario"\nadapter = "duckdb"\n\n' + '[connection]\ndatabase = "lambda_scenario.duckdb"\n' + ), + "sources/raw.yml": ( + "sources:\n - name: raw_sessions\n description: Test source raw_sessions.\n" + " schema: main\n table: raw_sessions\n" + ), + "models/sessions.sql": ( + "MODEL (description 'Test model sessions.');\n" + 'SELECT * FROM __source("raw_sessions")\n' + ), + "models/carts.sql": ( + "MODEL (description 'Test model carts.');\n" + "SELECT 1 AS session_id, [1, 2, 3] AS quantities\n" + ), + "models/cart_totals.sql": ( + "MODEL (description 'Test model cart_totals.');\n" + 'WITH sess AS (SELECT * FROM __ref("sessions")),\n' + 'crt AS (SELECT c.session_id, c.quantities FROM __ref("carts") AS c)\n' + "SELECT\n s.session_id,\n" + " list_reduce(list_transform(c.quantities, q -> q * 2), (acc, x) -> acc + x)" + " AS total,\n" + " len(list_filter(c.quantities, i -> i > 1)) AS big\n" + "FROM sess AS s\nINNER JOIN crt AS c ON s.session_id = c.session_id\n" + "WHERE len(list_filter(c.quantities, w -> w < 0)) = 0\n" + ), + "tests/scenarios/cart_totals_doubled.sql": ( + "SCENARIO (description 'Test scenario cart_totals_doubled.');\n" + "WITH\n__ref__sessions AS (SELECT 1 AS session_id),\n" + "__ref__carts AS (SELECT 1 AS session_id, [1, 2, 3] AS quantities),\n" + "__expected__cart_totals AS (SELECT 1 AS session_id, 12 AS total, 2 AS big)\n" + "SELECT 1\n" + ), + } + + +def build_contracted_star_empty_fixture_project_files() -> dict[str, str]: + """Build a scenario that empties a contract-enforced model whose query selects a star.""" + + return { + "sqlbuild_project.toml": ( + 'name = "contracted_star"\nadapter = "duckdb"\n\n' + '[connection]\ndatabase = "contracted_star.duckdb"\n' + ), + "sources/raw.yml": ( + "sources:\n - name: raw_customers\n description: Test source raw_customers.\n" + " schema: main\n table: raw_customers\n" + ), + "models/customers.sql": ( + "MODEL (description 'Test model customers.', contract enforced,\n" + " columns (id (type INTEGER), name (type VARCHAR)));\n" + 'SELECT * FROM __source("raw_customers")\n' + ), + "models/orders.sql": ( + "MODEL (description 'Test model orders.');\nSELECT 1 AS id, 7 AS customer_id\n" + ), + "models/customer_orders.sql": ( + "MODEL (description 'Test model customer_orders.');\n" + "SELECT o.id AS order_id, c.name AS customer_name\n" + 'FROM __ref("orders") AS o\nJOIN __ref("customers") AS c ON c.id = o.customer_id\n' + ), + "tests/scenarios/orders_without_customers.sql": ( + "SCENARIO (description 'Test scenario orders_without_customers.');\n" + "WITH\n__ref__orders AS (SELECT 1 AS id, 7 AS customer_id),\n" + "__ref__customers AS (SELECT * FROM __empty_fixture()),\n" + "__expected__customer_orders AS (\n" + " SELECT CAST(NULL AS INTEGER) AS order_id, CAST(NULL AS VARCHAR) AS customer_name\n" + " WHERE FALSE\n" + ")\n" + "SELECT 1\n" + ), + } + + +def build_cursor_intrinsic_scenario_project_files() -> dict[str, str]: + """Build a scenario over a non-microbatch incremental model that reads cursor intrinsics.""" + + return { + "sqlbuild_project.toml": ( + 'name = "cursor_scenario"\nadapter = "duckdb"\n\n' + '[connection]\ndatabase = "cursor_scenario.duckdb"\n' + ), + "models/orders.sql": ( + "MODEL (description 'Test model orders.');\n" + "SELECT 1 AS order_id, TIMESTAMP '2026-01-02 10:00:00' AS ordered_at\n" + ), + "models/hourly_orders.sql": ( + "MODEL (\n description 'Test model hourly_orders.',\n materialized incremental,\n" + " incremental_strategy delete_insert,\n cursor ordered_at,\n" + " cursor_type timestamp,\n cursor_grain hour,\n cursor_inputs (orders ordered_at),\n" + ");\n" + 'SELECT o.order_id, o.ordered_at FROM __ref("orders") AS o\n' + "WHERE o.ordered_at >= __cursor_start() AND o.ordered_at < __cursor_end()\n" + ), + "tests/scenarios/hourly_orders_window.sql": ( + "SCENARIO (description 'Test scenario hourly_orders_window.');\n" + "WITH\n__ref__orders AS (\n" + " SELECT 1 AS order_id, TIMESTAMP '2026-01-02 10:00:00' AS ordered_at\n),\n" + "__expected__hourly_orders AS (\n" + " SELECT 1 AS order_id, TIMESTAMP '2026-01-02 10:00:00' AS ordered_at\n)\n" + "SELECT 1\n" + ), + } + + +def build_connect_by_scenario_project_files() -> dict[str, str]: + """Build a Snowflake scenario whose model walks a hierarchy beside a star CTE.""" + + return { + "sqlbuild_project.toml": ( + 'name = "connect_by_scenario"\nadapter = "snowflake"\ndefault_target = "dev"\n\n' + '[targets.dev]\ndatabase = "warehouse"\nschema = "analytics"\n\n' + "[scopes]\nenforce_placement = false\n" + ), + "models/categories.sql": ( + "MODEL (description 'Test model categories.');\n" + "SELECT CAST(1 AS INTEGER) AS category_id, CAST(NULL AS INTEGER) AS parent_id,\n" + " CAST('root' AS VARCHAR) AS name\n" + ), + "models/category_paths.sql": ( + "MODEL (description 'Test model category_paths.');\n" + 'WITH categories AS (SELECT * FROM __ref("categories")),\n' + "tree AS (\n" + " SELECT c.category_id, LEVEL AS depth, SYS_CONNECT_BY_PATH(c.name, '/') AS path\n" + " FROM categories AS c\n" + " START WITH c.parent_id IS NULL\n" + " CONNECT BY PRIOR c.category_id = c.parent_id\n" + ")\n" + "SELECT category_id, depth, path FROM tree\n" + ), + "tests/scenarios/category_paths_walk.sql": ( + "SCENARIO (description 'Test scenario category_paths_walk.');\n" + "WITH\n__ref__categories AS (\n" + " SELECT 1 AS category_id, CAST(NULL AS INTEGER) AS parent_id, 'root' AS name\n),\n" + "__expected__category_paths AS (SELECT 1 AS category_id, 1 AS depth, '/root' AS path)\n" + "SELECT 1\n" + ), + } + + +def build_local_variant_scenario_project_files() -> dict[str, str]: + """Build an in-memory DuckDB scenario whose source fixture holds a VARIANT column.""" + + return { + "sqlbuild_project.toml": ( + 'name = "local_variant"\nadapter = "duckdb"\n\n[connection]\ndatabase = ":memory:"\n\n' + "[scopes]\nenforce_placement = false\n" + ), + "sources/raw.yml": ( + "sources:\n - name: raw_orders\n description: Test source raw_orders.\n" + " schema: main\n table: raw_orders\n columns:\n" + " - {name: order_id, type: INTEGER}\n - {name: payload, type: VARIANT}\n" + ), + "models/stg_orders.sql": ( + "MODEL (description 'Test model stg_orders.');\n" + "SELECT o.order_id, o.payload IS NOT NULL AS has_payload\n" + 'FROM __source("raw_orders") AS o\n' + ), + "tests/scenarios/orders__keep_payload.sql": ( + "SCENARIO (description 'Test scenario orders__keep_payload.');\n" + "WITH\n" + "__source__raw_orders AS (\n" + " SELECT 1 AS order_id, '{\"qty\": 2}'::JSON::VARIANT AS payload\n),\n" + "__expected__stg_orders AS (SELECT 1 AS order_id, TRUE AS has_payload)\n" + "SELECT 1\n" + ), + } + + +def build_helper_fixture_scenario_project_files() -> dict[str, str]: + """Build an in-memory DuckDB scenario whose source fixture reads a helper CTE.""" + + return { + "sqlbuild_project.toml": ( + 'name = "helper_fixture"\nadapter = "duckdb"\n\n[connection]\ndatabase = ":memory:"\n\n' + "[scopes]\nenforce_placement = false\n" + ), + "sources/raw.yml": ( + "sources:\n - name: raw_orders\n description: Test source raw_orders.\n" + " schema: main\n table: raw_orders\n columns:\n" + " - {name: order_id, type: INTEGER}\n" + ), + "models/stg_orders.sql": ( + "MODEL (description 'Test model stg_orders.');\n" + 'SELECT o.order_id FROM __source("raw_orders") AS o\n' + ), + "tests/scenarios/orders__from_helper.sql": ( + "SCENARIO (description 'Test scenario orders__from_helper.');\n" + "WITH\n" + "helper_ids AS (SELECT 1 AS order_id),\n" + "__source__raw_orders AS (SELECT h.order_id FROM helper_ids AS h),\n" + "__expected__stg_orders AS (SELECT 1 AS order_id)\n" + "SELECT 1\n" + ), + } diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_fixture_columns.py b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_fixture_columns.py new file mode 100644 index 0000000000..b7e719d6aa --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_fixture_columns.py @@ -0,0 +1,72 @@ +"""E2E tests for scenario fixtures and the models scenarios build from them.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.scenario._test_types import ( + ScenarioInlineProjectE2ETestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.main.scenario.helpers import ( + build_connect_by_scenario_project_files, + build_contracted_star_empty_fixture_project_files, + build_cursor_intrinsic_scenario_project_files, + build_lambda_scenario_project_files, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + + +@pytest.mark.parametrize( + "test_case", + ( + ScenarioInlineProjectE2ETestCase( + description="lambda parameters are not mock columns", + repo_files=build_lambda_scenario_project_files(), + command=("--no-color", "scenario", "test"), + expected_exit_code=0, + expected_stdout_fragments=("PASS=1 FAIL=0 TOTAL=1",), + ), + ScenarioInlineProjectE2ETestCase( + description="contract enforced star model accepts an empty fixture", + repo_files=build_contracted_star_empty_fixture_project_files(), + command=("--no-color", "scenario", "test"), + expected_exit_code=0, + expected_stdout_fragments=("PASS=1 FAIL=0 TOTAL=1",), + ), + ScenarioInlineProjectE2ETestCase( + description="incremental model with cursor intrinsics builds over a wide window", + repo_files=build_cursor_intrinsic_scenario_project_files(), + command=("--no-color", "scenario", "test"), + expected_exit_code=0, + expected_stdout_fragments=("PASS=1 FAIL=0 TOTAL=1",), + ), + ScenarioInlineProjectE2ETestCase( + description="CONNECT BY pseudo-columns are not mock columns", + repo_files=build_connect_by_scenario_project_files(), + command=("--no-color", "scenario", "test", "--local"), + expected_exit_code=0, + expected_stdout_fragments=("missing local snapshot manifest",), + ), + ), + ids=lambda case: case.description, +) +def test_given_scenario_fixture_when_running_cli_then_requires_only_real_columns( + test_case: ScenarioInlineProjectE2ETestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, project_name="fixture_columns", repo_files=test_case.repo_files + ) + + result: subprocess.CompletedProcess[str] = run_sqb( + command=test_case.command, project_dir=project_dir + ) + + assert result.returncode == test_case.expected_exit_code, result.stdout + result.stderr + assert all(fragment in result.stdout for fragment in test_case.expected_stdout_fragments) + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_local_variant.py b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_local_variant.py new file mode 100644 index 0000000000..82ab7d6f4a --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_local_variant.py @@ -0,0 +1,57 @@ +"""E2E coverage of local scenario replay over VARIANT fixtures, which need current DuckDB storage.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.scenario._test_types import ( + LocalVariantScenarioTestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.main.scenario.helpers import ( + build_local_variant_scenario_project_files, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + + +@pytest.mark.parametrize( + "test_case", + [ + LocalVariantScenarioTestCase( + description="a captured VARIANT fixture replays locally in strict mode", + repo_files=build_local_variant_scenario_project_files(), + scenario_name="orders__keep_payload", + expected_stdout_fragment="PASS=1 FAIL=0 ERROR=0 SKIP=0 TOTAL=1", + ), + ], + ids=lambda case: case.description, +) +def test_given_captured_variant_fixture_when_replaying_locally_then_the_scenario_passes( + test_case: LocalVariantScenarioTestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, project_name="local_variant", repo_files=test_case.repo_files + ) + captured: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "scenario", "capture", test_case.scenario_name), + project_dir=project_dir, + ) + assert captured.returncode == 0, captured.stdout + captured.stderr + + replayed: subprocess.CompletedProcess[str] = run_sqb( + command=( + "--no-color", + "scenario", + "test", + "--local", + "--strict", + test_case.scenario_name, + ), + project_dir=project_dir, + ) + + assert replayed.returncode == 0, replayed.stdout + replayed.stderr + assert "X699" not in replayed.stdout + assert test_case.expected_stdout_fragment in replayed.stdout diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_snapshot_staleness.py b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_snapshot_staleness.py new file mode 100644 index 0000000000..588da673b1 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_snapshot_staleness.py @@ -0,0 +1,70 @@ +"""E2E coverage of local snapshot staleness when the SQL behind a fixture changes.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.scenario._test_types import ( + SnapshotHelperEditTestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.main.scenario.helpers import ( + build_helper_fixture_scenario_project_files, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + +_SCENARIO_PATH: str = "tests/scenarios/orders__from_helper.sql" + + +@pytest.mark.parametrize( + "test_case", + [ + SnapshotHelperEditTestCase( + description="editing a helper CTE the fixture reads makes the snapshot stale", + repo_files=build_helper_fixture_scenario_project_files(), + scenario_name="orders__from_helper", + edited_path=_SCENARIO_PATH, + edited_from="helper_ids AS (SELECT 1 AS order_id)", + edited_to="helper_ids AS (SELECT 2 AS order_id)", + expected_exit_code=1, + expected_stdout_fragment="X602", + ), + SnapshotHelperEditTestCase( + description="editing the expected output alone keeps the snapshot fresh", + repo_files=build_helper_fixture_scenario_project_files(), + scenario_name="orders__from_helper", + edited_path=_SCENARIO_PATH, + edited_from="__expected__stg_orders AS (SELECT 1 AS order_id)", + edited_to="__expected__stg_orders AS (SELECT 1 AS order_id\n)", + expected_exit_code=0, + expected_stdout_fragment="PASS=1 FAIL=0 ERROR=0 SKIP=0 TOTAL=1", + ), + ], + ids=lambda case: case.description, +) +def test_given_captured_snapshot_when_fixture_sql_changes_then_local_replay_reports_staleness( + test_case: SnapshotHelperEditTestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, project_name="helper_fixture", repo_files=test_case.repo_files + ) + captured: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "scenario", "capture", test_case.scenario_name), + project_dir=project_dir, + ) + assert captured.returncode == 0, captured.stdout + captured.stderr + edited: Path = project_dir / test_case.edited_path + _ = edited.write_text( + edited.read_text(encoding="utf-8").replace(test_case.edited_from, test_case.edited_to), + encoding="utf-8", + ) + + replayed: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "scenario", "test", "--local", "--strict", test_case.scenario_name), + project_dir=project_dir, + ) + + assert replayed.returncode == test_case.expected_exit_code, replayed.stdout + replayed.stderr + assert test_case.expected_stdout_fragment in replayed.stdout diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/test/_test_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/test/_test_types.py index 2a96d1fdbe..e48ffb8404 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/test/_test_types.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/test/_test_types.py @@ -571,3 +571,25 @@ class SinceUnwritableCacheE2ETestCase: command: tuple[str, ...] expected_stderr_fragments: tuple[str, ...] expected_test_names: tuple[str, ...] + + +@dataclass(frozen=True) +class FixtureDeclaredTypeE2ETestCase: + """A source fixture whose bare NULL column must take the source's declared type.""" + + description: str + fixture_sql: str + expected_sql: str + expected_artifact_fragment: str + expected_stdout_fragment: str = "PASS=1 FAIL=0 TOTAL=1" + + +@dataclass(frozen=True) +class MacroMockAliasTestCase: + """A macro mock's select item, with SQL rules selected, and what one command makes of it.""" + + description: str + mock_select: str + command: tuple[str, ...] + expected_exit_code: int + expected_fragment: str diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/test/test_fixture_declared_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/test/test_fixture_declared_types.py new file mode 100644 index 0000000000..c83952d766 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/test/test_fixture_declared_types.py @@ -0,0 +1,84 @@ +"""E2E coverage for casting bare NULL fixture columns to declared source types.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.test._test_types import ( + FixtureDeclaredTypeE2ETestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + +_PROJECT_FILES: dict[str, str] = { + "sqlbuild_project.toml": ( + 'name = "fixture_types"\nadapter = "duckdb"\n\n' + '[connection]\ndatabase = "fixture_types.duckdb"\n' + ), + "sources/raw_orders.yml": ( + "sources:\n - name: raw_orders\n description: Test source raw_orders.\n" + " schema: main\n table: raw_orders\n columns:\n" + " - name: order_id\n type: INTEGER\n" + " - name: ordered_on\n type: DATE\n" + ), + "models/order_years.sql": ( + "MODEL (description 'Test model order_years.');\n" + 'SELECT o.order_id, YEAR(o.ordered_on) AS order_year FROM __source("raw_orders") AS o\n' + ), +} + + +@pytest.mark.parametrize( + "test_case", + [ + FixtureDeclaredTypeE2ETestCase( + description="single select fixture", + fixture_sql="SELECT 1 AS order_id, NULL AS ordered_on", + expected_sql="SELECT 1 AS order_id, CAST(NULL AS BIGINT) AS order_year", + expected_artifact_fragment='CAST("__sqlbuild_partial_fixture".ordered_on AS DATE)', + ), + FixtureDeclaredTypeE2ETestCase( + description="union all fixture", + fixture_sql=( + "SELECT 1 AS order_id, NULL AS ordered_on\n" + " UNION ALL SELECT 2 AS order_id, NULL AS ordered_on" + ), + expected_sql=( + "SELECT 1 AS order_id, CAST(NULL AS BIGINT) AS order_year\n" + " UNION ALL SELECT 2 AS order_id, CAST(NULL AS BIGINT) AS order_year" + ), + expected_artifact_fragment='CAST("__sqlbuild_partial_fixture".ordered_on AS DATE)', + ), + ], + ids=lambda case: case.description, +) +def test_given_bare_null_source_fixture_when_testing_then_casts_to_declared_type( + test_case: FixtureDeclaredTypeE2ETestCase, tmp_path: Path +) -> None: + files: dict[str, str] = { + **_PROJECT_FILES, + "tests/unit/test_order_years.sql": ( + "TEST();\nWITH\n" + f"__source__raw_orders AS (\n {test_case.fixture_sql}\n),\n" + f"__expected__order_years AS (\n {test_case.expected_sql}\n)\nSELECT 1\n" + ), + } + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, project_name="fixture_types", repo_files=files + ) + + result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "test"), project_dir=project_dir + ) + + artifact: str = (project_dir / "target/run/tests/order_years/test_order_years.sql").read_text( + encoding="utf-8" + ) + assert test_case.expected_stdout_fragment in result.stdout, result.stdout + result.stderr + assert test_case.expected_artifact_fragment in artifact + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/test/test_macro_mock_alias.py b/tests/e2e/src/sqlbuild/cli/commands/main/test/test_macro_mock_alias.py new file mode 100644 index 0000000000..39d0fbf094 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/test/test_macro_mock_alias.py @@ -0,0 +1,85 @@ +"""E2E coverage of macro mocks that name their value, as SQBRSQL022 asks.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.test._test_types import MacroMockAliasTestCase +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + +_PROJECT_TOML: str = ( + 'name = "shop"\nadapter = "duckdb"\n\n[connection]\ndatabase = "shop.duckdb"\n\n' + '[scopes]\nenforce_placement = false\n\n[rules]\nselect = ["SQBRSQL022"]\n' +) +_REGION_MACRO: str = ( + "MACRO (\n description \"The sales region filter\",\n returns BOOLEAN,\n);\n\nregion = 'NA'\n" +) +_ORDERS_MODEL: str = ( + 'MODEL (description "Orders.");\n\n' + "SELECT 1 AS order_id, 'EU' AS region UNION ALL SELECT 2 AS order_id, 'NA' AS region\n" +) +_REGIONAL_ORDERS_MODEL: str = ( + 'MODEL (description "Orders of one region.");\n\n' + 'SELECT o.order_id FROM __ref("orders") AS o WHERE @sql.region_filter()\n' +) +_TEST: str = ( + "TEST ();\n\n" + "WITH\n" + "__macro__region_filter AS ({mock_select}),\n" + "__ref__orders AS (SELECT 1 AS order_id, 'EU' AS region),\n" + "__expected__regional_orders AS (SELECT 1 AS order_id)\n" +) + + +@pytest.mark.parametrize( + "test_case", + [ + MacroMockAliasTestCase( + description="an aliased mock satisfies SQBRSQL022", + mock_select="SELECT 'region = ''EU''' AS region_filter", + command=("--no-color", "compile"), + expected_exit_code=0, + expected_fragment="0 errors", + ), + MacroMockAliasTestCase( + description="an aliased mock still replaces the macro", + mock_select="SELECT 'region = ''EU''' AS region_filter", + command=("--no-color", "test"), + expected_exit_code=0, + expected_fragment="PASS=1 FAIL=0", + ), + MacroMockAliasTestCase( + description="an unaliased mock is reported by SQBRSQL022", + mock_select="SELECT 'region = ''EU'''", + command=("--no-color", "compile"), + expected_exit_code=1, + expected_fragment="SQBRSQL022", + ), + ], + ids=lambda case: case.description, +) +def test_given_macro_mock_with_alias_when_testing_then_the_mock_value_is_used( + test_case: MacroMockAliasTestCase, tmp_path: Path +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="shop", + repo_files={ + "sqlbuild_project.toml": _PROJECT_TOML, + "macros/sql/region_filter.sql": _REGION_MACRO, + "models/orders.sql": _ORDERS_MODEL, + "models/regional_orders.sql": _REGIONAL_ORDERS_MODEL, + "tests/unit/test_regional_orders.sql": _TEST.format(mock_select=test_case.mock_select), + }, + ) + + ran: subprocess.CompletedProcess[str] = run_sqb( + project_dir=project_dir, command=test_case.command + ) + report: str = ran.stdout + ran.stderr + + assert ran.returncode == test_case.expected_exit_code, report + assert test_case.expected_fragment in report diff --git a/tests/e2e/src/sqlbuild/integrations/dagster/_test_types.py b/tests/e2e/src/sqlbuild/integrations/dagster/_test_types.py index a9ac76f605..7a3d6c2bdf 100644 --- a/tests/e2e/src/sqlbuild/integrations/dagster/_test_types.py +++ b/tests/e2e/src/sqlbuild/integrations/dagster/_test_types.py @@ -152,3 +152,12 @@ class DagsterAuditAttachmentChecksE2ETestCase: description: str command: tuple[str, ...] expected_checked_assets: tuple[tuple[str, ...], ...] + + +@dataclass(frozen=True) +class DagsterSkippedAuditCheckE2ETestCase: + """A green build whose end audit is skipped, and the check Dagster must not report as failed.""" + + description: str + skipped_check_name: str + expected_log_line: str diff --git a/tests/e2e/src/sqlbuild/integrations/dagster/test_audit_attachment_checks.py b/tests/e2e/src/sqlbuild/integrations/dagster/test_audit_attachment_checks.py index 4b3cc97628..70655e79cc 100644 --- a/tests/e2e/src/sqlbuild/integrations/dagster/test_audit_attachment_checks.py +++ b/tests/e2e/src/sqlbuild/integrations/dagster/test_audit_attachment_checks.py @@ -19,6 +19,7 @@ ) from tests.e2e.src.sqlbuild.integrations.dagster._test_types import ( DagsterAuditAttachmentChecksE2ETestCase, + DagsterSkippedAuditCheckE2ETestCase, ) from tests.e2e.src.sqlbuild.integrations.dagster.helpers import ( ATTACHMENT_CHECK_PROJECT_FILES, @@ -26,6 +27,30 @@ audit_check_outcome, ) +_SKIPPED_AUDIT_PROJECT_FILES: dict[str, str] = { + "sqlbuild_project.toml": ( + 'name = "skipped_audit_checks"\nadapter = "duckdb"\n\n' + '[connection]\ndatabase = "warehouse.duckdb"\n\n[defaults]\nmaterialized = "table"\n' + ), + "hooks/python/skips.py": ( + "from sqlbuild.hooks import hook\n\n\n" + "@hook\ndef skip_model(ctx):\n" + " '''Test hook skip_model.'''\n" + " return ctx.skip(reason='paused')\n" + ), + "models/customers.sql": ( + "MODEL (description 'Test model customers.', pre_hooks [python(\"skip_model\")]);\n" + "SELECT 1 AS customer_id\n" + ), + "models/orders.sql": "MODEL (description 'Test model orders.');\nSELECT 1 AS order_id\n", + "audits/singular/orders_have_customers.sql": ( + "AUDIT ();\n" + 'SELECT o.order_id FROM __ref("orders") AS o\n' + 'LEFT JOIN __ref("customers") AS c ON c.customer_id = o.order_id\n' + "WHERE c.customer_id IS NULL\n" + ), +} + @pytest.mark.parametrize( "test_case", @@ -137,3 +162,49 @@ def sqlbuild_attachment_checks(context: AssetExecutionContext) -> Iterator[objec assert checks.declared <= checks.evaluated assert checks.all_passed is True assert "integration output degraded" not in capsys.readouterr().err + + +@pytest.mark.parametrize( + "test_case", + [ + DagsterSkippedAuditCheckE2ETestCase( + description="an end audit over a hook-skipped first-run model is not a failed check", + skipped_check_name="audit__orders_have_customers", + expected_log_line=( + "SQLBuild audit orders_have_customers did not run: reads customers, which was not " + "built" + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_skipped_end_audit_when_building_through_dagster_then_no_failed_check_is_reported( + test_case: DagsterSkippedAuditCheckE2ETestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="skipped_audit_checks", + repo_files=_SKIPPED_AUDIT_PROJECT_FILES, + ) + sqlbuild_project: SqlBuildProject = SqlBuildProject( + project_dir=project_dir, sqb_command=(str(REPO_ROOT / ".venv" / "bin" / "sqb"),) + ) + sqlbuild_project.prepare() + + @sqlbuild_assets(project=sqlbuild_project, required_resource_keys={"sqb"}) + def sqlbuild_skipped_audit(context: AssetExecutionContext) -> Iterator[object]: + yield from context.resources.sqb.cli(["build"], context=context).stream() + + result: ExecuteInProcessResult = materialize( + [sqlbuild_skipped_audit], + resources={"sqb": SqlBuildCliResource(project_dir=sqlbuild_project)}, + ) + + checks: AuditCheckOutcome = audit_check_outcome(result=result, assets=sqlbuild_skipped_audit) + assert checks.success is True + assert test_case.skipped_check_name in {name for _, name in checks.declared} + assert test_case.skipped_check_name not in {name for _, name in checks.evaluated} + assert checks.all_passed is True + assert test_case.expected_log_line in capsys.readouterr().err diff --git a/tests/goldens/python_oracle/contracts_generated_enforced_implicit_typed_nullability_and_dynamic_contracts.json b/tests/goldens/python_oracle/contracts_generated_enforced_implicit_typed_nullability_and_dynamic_contracts.json index 443217bb53..06b2d6254c 100644 --- a/tests/goldens/python_oracle/contracts_generated_enforced_implicit_typed_nullability_and_dynamic_contracts.json +++ b/tests/goldens/python_oracle/contracts_generated_enforced_implicit_typed_nullability_and_dynamic_contracts.json @@ -447,7 +447,7 @@ "contracts", { "None": "afb26acda81b184518d6d6f5", -"bigquery": "afb26acda81b184518d6d6f5", +"bigquery": "f6dafe38c2aa2436ad17d2e6", "databricks": "f6dafe38c2aa2436ad17d2e6", "duckdb": "f6dafe38c2aa2436ad17d2e6", "postgres": "afb26acda81b184518d6d6f5", diff --git a/tests/goldens/python_oracle/types_generated_seeded_scalar_parameterized_nested_padded_and_invalid_types.json b/tests/goldens/python_oracle/types_generated_seeded_scalar_parameterized_nested_padded_and_invalid_types.json index 30eded6455..1729e43aef 100644 --- a/tests/goldens/python_oracle/types_generated_seeded_scalar_parameterized_nested_padded_and_invalid_types.json +++ b/tests/goldens/python_oracle/types_generated_seeded_scalar_parameterized_nested_padded_and_invalid_types.json @@ -240,18 +240,18 @@ [ "type", { -"DuckDB": "e1c36a5cdd2427ff2b089ac9", -"None": "e1c36a5cdd2427ff2b089ac9", +"DuckDB": "18db5afe2c1e80d1ee652b54", +"None": "18db5afe2c1e80d1ee652b54", "bigquery": "e1c36a5cdd2427ff2b089ac9", -"databricks": "e1c36a5cdd2427ff2b089ac9", -"duckdb": "e1c36a5cdd2427ff2b089ac9", -"generic": "e1c36a5cdd2427ff2b089ac9", +"databricks": "18db5afe2c1e80d1ee652b54", +"duckdb": "18db5afe2c1e80d1ee652b54", +"generic": "18db5afe2c1e80d1ee652b54", "motherduck": "901c5133ed47d4769286e860", -"mysql": "e1c36a5cdd2427ff2b089ac9", -"postgres": "e1c36a5cdd2427ff2b089ac9", -"postgresql": "e1c36a5cdd2427ff2b089ac9", +"mysql": "18db5afe2c1e80d1ee652b54", +"postgres": "18db5afe2c1e80d1ee652b54", +"postgresql": "18db5afe2c1e80d1ee652b54", "snowflake": "18db5afe2c1e80d1ee652b54", -"tsql": "e1c36a5cdd2427ff2b089ac9" +"tsql": "18db5afe2c1e80d1ee652b54" } ], [ @@ -378,7 +378,7 @@ { "DuckDB": "19bdd92ea0cc5fb028657510", "None": "78b2cf275c640af54feeca31", -"bigquery": "78b2cf275c640af54feeca31", +"bigquery": "1e75e5227bea0ae294ac7af1", "databricks": "1e75e5227bea0ae294ac7af1", "duckdb": "19bdd92ea0cc5fb028657510", "generic": "78b2cf275c640af54feeca31", @@ -1668,18 +1668,18 @@ [ "type", { -"DuckDB": "4c4563a9f306a1bb11e116d9", -"None": "4c4563a9f306a1bb11e116d9", +"DuckDB": "5d25d5e13f81700741695ed4", +"None": "5d25d5e13f81700741695ed4", "bigquery": "23ee196b7bd6039855a1460f", -"databricks": "4c4563a9f306a1bb11e116d9", -"duckdb": "4c4563a9f306a1bb11e116d9", -"generic": "4c4563a9f306a1bb11e116d9", +"databricks": "5d25d5e13f81700741695ed4", +"duckdb": "5d25d5e13f81700741695ed4", +"generic": "5d25d5e13f81700741695ed4", "motherduck": "901c5133ed47d4769286e860", -"mysql": "4c4563a9f306a1bb11e116d9", -"postgres": "4c4563a9f306a1bb11e116d9", -"postgresql": "4c4563a9f306a1bb11e116d9", +"mysql": "5d25d5e13f81700741695ed4", +"postgres": "5d25d5e13f81700741695ed4", +"postgresql": "5d25d5e13f81700741695ed4", "snowflake": "5d25d5e13f81700741695ed4", -"tsql": "4c4563a9f306a1bb11e116d9" +"tsql": "5d25d5e13f81700741695ed4" } ], [ @@ -1753,18 +1753,18 @@ [ "type", { -"DuckDB": "768fc4c8dd514e3bc66f6184", -"None": "768fc4c8dd514e3bc66f6184", -"bigquery": "1b5e4ba111b3a98453dd1b5c", -"databricks": "28beac12e88e437078a0c62a", -"duckdb": "768fc4c8dd514e3bc66f6184", -"generic": "768fc4c8dd514e3bc66f6184", +"DuckDB": "f2ed52f6bf46b4db9343551c", +"None": "f2ed52f6bf46b4db9343551c", +"bigquery": "f10bdd4d40eb495cbcecd31d", +"databricks": "f2ed52f6bf46b4db9343551c", +"duckdb": "f2ed52f6bf46b4db9343551c", +"generic": "f2ed52f6bf46b4db9343551c", "motherduck": "901c5133ed47d4769286e860", -"mysql": "768fc4c8dd514e3bc66f6184", -"postgres": "768fc4c8dd514e3bc66f6184", -"postgresql": "768fc4c8dd514e3bc66f6184", -"snowflake": "28beac12e88e437078a0c62a", -"tsql": "28beac12e88e437078a0c62a" +"mysql": "f2ed52f6bf46b4db9343551c", +"postgres": "f2ed52f6bf46b4db9343551c", +"postgresql": "f2ed52f6bf46b4db9343551c", +"snowflake": "f2ed52f6bf46b4db9343551c", +"tsql": "f2ed52f6bf46b4db9343551c" } ], [ @@ -1804,18 +1804,18 @@ [ "type", { -"DuckDB": "2a0918be53051fe629d9ff8f", -"None": "2a0918be53051fe629d9ff8f", +"DuckDB": "3e611643f0beab4953c4e09b", +"None": "3e611643f0beab4953c4e09b", "bigquery": "2a0918be53051fe629d9ff8f", -"databricks": "2a0918be53051fe629d9ff8f", -"duckdb": "2a0918be53051fe629d9ff8f", -"generic": "2a0918be53051fe629d9ff8f", +"databricks": "3e611643f0beab4953c4e09b", +"duckdb": "3e611643f0beab4953c4e09b", +"generic": "3e611643f0beab4953c4e09b", "motherduck": "901c5133ed47d4769286e860", -"mysql": "2a0918be53051fe629d9ff8f", -"postgres": "2a0918be53051fe629d9ff8f", -"postgresql": "2a0918be53051fe629d9ff8f", +"mysql": "3e611643f0beab4953c4e09b", +"postgres": "3e611643f0beab4953c4e09b", +"postgresql": "3e611643f0beab4953c4e09b", "snowflake": "3e611643f0beab4953c4e09b", -"tsql": "2a0918be53051fe629d9ff8f" +"tsql": "3e611643f0beab4953c4e09b" } ], [ @@ -2467,11 +2467,11 @@ [ "type", { -"DuckDB": "638ca9212b62ada31a448c62", +"DuckDB": "d81d8c19074af0e39239164d", "None": "1fec266134381c760a254d16", "bigquery": "1fec266134381c760a254d16", "databricks": "1fec266134381c760a254d16", -"duckdb": "638ca9212b62ada31a448c62", +"duckdb": "d81d8c19074af0e39239164d", "generic": "1fec266134381c760a254d16", "motherduck": "901c5133ed47d4769286e860", "mysql": "1fec266134381c760a254d16", @@ -2535,11 +2535,11 @@ [ "type", { -"DuckDB": "6226bc03436ecf8fccd39252", +"DuckDB": "5aa51d84ac5e6c7a9306f348", "None": "5aa51d84ac5e6c7a9306f348", -"bigquery": "5aa51d84ac5e6c7a9306f348", +"bigquery": "46487f079f1382836c406de7", "databricks": "5aa51d84ac5e6c7a9306f348", -"duckdb": "6226bc03436ecf8fccd39252", +"duckdb": "5aa51d84ac5e6c7a9306f348", "generic": "5aa51d84ac5e6c7a9306f348", "motherduck": "901c5133ed47d4769286e860", "mysql": "5aa51d84ac5e6c7a9306f348", @@ -3130,18 +3130,18 @@ [ "type", { -"DuckDB": "787213ce6b6e210ded1d752c", -"None": "787213ce6b6e210ded1d752c", +"DuckDB": "b561884132dbe7e42b719110", +"None": "b561884132dbe7e42b719110", "bigquery": "b1b23029bedaa16e2a6f8516", -"databricks": "787213ce6b6e210ded1d752c", -"duckdb": "787213ce6b6e210ded1d752c", -"generic": "787213ce6b6e210ded1d752c", +"databricks": "b561884132dbe7e42b719110", +"duckdb": "b561884132dbe7e42b719110", +"generic": "b561884132dbe7e42b719110", "motherduck": "901c5133ed47d4769286e860", -"mysql": "787213ce6b6e210ded1d752c", -"postgres": "787213ce6b6e210ded1d752c", -"postgresql": "787213ce6b6e210ded1d752c", +"mysql": "b561884132dbe7e42b719110", +"postgres": "b561884132dbe7e42b719110", +"postgresql": "b561884132dbe7e42b719110", "snowflake": "b561884132dbe7e42b719110", -"tsql": "787213ce6b6e210ded1d752c" +"tsql": "b561884132dbe7e42b719110" } ], [ @@ -3181,11 +3181,11 @@ [ "type", { -"DuckDB": "638ca9212b62ada31a448c62", +"DuckDB": "d81d8c19074af0e39239164d", "None": "1fec266134381c760a254d16", "bigquery": "1fec266134381c760a254d16", "databricks": "1fec266134381c760a254d16", -"duckdb": "638ca9212b62ada31a448c62", +"duckdb": "d81d8c19074af0e39239164d", "generic": "1fec266134381c760a254d16", "motherduck": "901c5133ed47d4769286e860", "mysql": "1fec266134381c760a254d16", @@ -3402,11 +3402,11 @@ [ "type", { -"DuckDB": "72bd5e38b32893397eb51579", +"DuckDB": "1bdf5908ca249b2331a0427c", "None": "1bdf5908ca249b2331a0427c", -"bigquery": "1bdf5908ca249b2331a0427c", +"bigquery": "730d7cf7510ba0934828d971", "databricks": "1bdf5908ca249b2331a0427c", -"duckdb": "72bd5e38b32893397eb51579", +"duckdb": "1bdf5908ca249b2331a0427c", "generic": "1bdf5908ca249b2331a0427c", "motherduck": "901c5133ed47d4769286e860", "mysql": "1bdf5908ca249b2331a0427c", @@ -4031,18 +4031,18 @@ [ "type", { -"DuckDB": "b1b23029bedaa16e2a6f8516", -"None": "b1b23029bedaa16e2a6f8516", +"DuckDB": "b561884132dbe7e42b719110", +"None": "b561884132dbe7e42b719110", "bigquery": "b1b23029bedaa16e2a6f8516", -"databricks": "b1b23029bedaa16e2a6f8516", -"duckdb": "b1b23029bedaa16e2a6f8516", -"generic": "b1b23029bedaa16e2a6f8516", +"databricks": "b561884132dbe7e42b719110", +"duckdb": "b561884132dbe7e42b719110", +"generic": "b561884132dbe7e42b719110", "motherduck": "901c5133ed47d4769286e860", -"mysql": "b1b23029bedaa16e2a6f8516", -"postgres": "b1b23029bedaa16e2a6f8516", -"postgresql": "b1b23029bedaa16e2a6f8516", +"mysql": "b561884132dbe7e42b719110", +"postgres": "b561884132dbe7e42b719110", +"postgresql": "b561884132dbe7e42b719110", "snowflake": "b561884132dbe7e42b719110", -"tsql": "b1b23029bedaa16e2a6f8516" +"tsql": "b561884132dbe7e42b719110" } ], [ @@ -4388,18 +4388,18 @@ [ "type", { -"DuckDB": "4062ec81199547543ab068fb", -"None": "4062ec81199547543ab068fb", -"bigquery": "7009fc15fd2c35608ab16caf", -"databricks": "0739784ba4c53d23afd45a6b", -"duckdb": "4062ec81199547543ab068fb", -"generic": "4062ec81199547543ab068fb", +"DuckDB": "5aa51d84ac5e6c7a9306f348", +"None": "5aa51d84ac5e6c7a9306f348", +"bigquery": "46487f079f1382836c406de7", +"databricks": "5aa51d84ac5e6c7a9306f348", +"duckdb": "5aa51d84ac5e6c7a9306f348", +"generic": "5aa51d84ac5e6c7a9306f348", "motherduck": "901c5133ed47d4769286e860", -"mysql": "4062ec81199547543ab068fb", -"postgres": "4062ec81199547543ab068fb", -"postgresql": "4062ec81199547543ab068fb", -"snowflake": "0739784ba4c53d23afd45a6b", -"tsql": "0739784ba4c53d23afd45a6b" +"mysql": "5aa51d84ac5e6c7a9306f348", +"postgres": "5aa51d84ac5e6c7a9306f348", +"postgresql": "5aa51d84ac5e6c7a9306f348", +"snowflake": "5aa51d84ac5e6c7a9306f348", +"tsql": "5aa51d84ac5e6c7a9306f348" } ], [ @@ -4932,18 +4932,18 @@ [ "type", { -"DuckDB": "0ca1be8f7d61ca627e620450", -"None": "0ca1be8f7d61ca627e620450", +"DuckDB": "18db5afe2c1e80d1ee652b54", +"None": "18db5afe2c1e80d1ee652b54", "bigquery": "e1c36a5cdd2427ff2b089ac9", -"databricks": "0ca1be8f7d61ca627e620450", -"duckdb": "0ca1be8f7d61ca627e620450", -"generic": "0ca1be8f7d61ca627e620450", +"databricks": "18db5afe2c1e80d1ee652b54", +"duckdb": "18db5afe2c1e80d1ee652b54", +"generic": "18db5afe2c1e80d1ee652b54", "motherduck": "901c5133ed47d4769286e860", -"mysql": "0ca1be8f7d61ca627e620450", -"postgres": "0ca1be8f7d61ca627e620450", -"postgresql": "0ca1be8f7d61ca627e620450", +"mysql": "18db5afe2c1e80d1ee652b54", +"postgres": "18db5afe2c1e80d1ee652b54", +"postgresql": "18db5afe2c1e80d1ee652b54", "snowflake": "18db5afe2c1e80d1ee652b54", -"tsql": "0ca1be8f7d61ca627e620450" +"tsql": "18db5afe2c1e80d1ee652b54" } ], [ @@ -5595,11 +5595,11 @@ [ "type", { -"DuckDB": "72bd5e38b32893397eb51579", +"DuckDB": "1bdf5908ca249b2331a0427c", "None": "1bdf5908ca249b2331a0427c", -"bigquery": "1bdf5908ca249b2331a0427c", +"bigquery": "730d7cf7510ba0934828d971", "databricks": "1bdf5908ca249b2331a0427c", -"duckdb": "72bd5e38b32893397eb51579", +"duckdb": "1bdf5908ca249b2331a0427c", "generic": "1bdf5908ca249b2331a0427c", "motherduck": "901c5133ed47d4769286e860", "mysql": "1bdf5908ca249b2331a0427c", @@ -5731,11 +5731,11 @@ [ "type", { -"DuckDB": "6226bc03436ecf8fccd39252", +"DuckDB": "5aa51d84ac5e6c7a9306f348", "None": "5aa51d84ac5e6c7a9306f348", -"bigquery": "5aa51d84ac5e6c7a9306f348", +"bigquery": "46487f079f1382836c406de7", "databricks": "5aa51d84ac5e6c7a9306f348", -"duckdb": "6226bc03436ecf8fccd39252", +"duckdb": "5aa51d84ac5e6c7a9306f348", "generic": "5aa51d84ac5e6c7a9306f348", "motherduck": "901c5133ed47d4769286e860", "mysql": "5aa51d84ac5e6c7a9306f348", @@ -5867,11 +5867,11 @@ [ "type", { -"DuckDB": "638ca9212b62ada31a448c62", +"DuckDB": "d81d8c19074af0e39239164d", "None": "1fec266134381c760a254d16", "bigquery": "1fec266134381c760a254d16", "databricks": "1fec266134381c760a254d16", -"duckdb": "638ca9212b62ada31a448c62", +"duckdb": "d81d8c19074af0e39239164d", "generic": "1fec266134381c760a254d16", "motherduck": "901c5133ed47d4769286e860", "mysql": "1fec266134381c760a254d16", @@ -6241,18 +6241,18 @@ [ "type", { -"DuckDB": "4062ec81199547543ab068fb", -"None": "4062ec81199547543ab068fb", -"bigquery": "7009fc15fd2c35608ab16caf", -"databricks": "0739784ba4c53d23afd45a6b", -"duckdb": "4062ec81199547543ab068fb", -"generic": "4062ec81199547543ab068fb", +"DuckDB": "5aa51d84ac5e6c7a9306f348", +"None": "5aa51d84ac5e6c7a9306f348", +"bigquery": "46487f079f1382836c406de7", +"databricks": "5aa51d84ac5e6c7a9306f348", +"duckdb": "5aa51d84ac5e6c7a9306f348", +"generic": "5aa51d84ac5e6c7a9306f348", "motherduck": "901c5133ed47d4769286e860", -"mysql": "4062ec81199547543ab068fb", -"postgres": "4062ec81199547543ab068fb", -"postgresql": "4062ec81199547543ab068fb", -"snowflake": "0739784ba4c53d23afd45a6b", -"tsql": "0739784ba4c53d23afd45a6b" +"mysql": "5aa51d84ac5e6c7a9306f348", +"postgres": "5aa51d84ac5e6c7a9306f348", +"postgresql": "5aa51d84ac5e6c7a9306f348", +"snowflake": "5aa51d84ac5e6c7a9306f348", +"tsql": "5aa51d84ac5e6c7a9306f348" } ], [ @@ -6292,11 +6292,11 @@ [ "type", { -"DuckDB": "f6c3d80bfee0ec0b286b82b5", +"DuckDB": "27f973a6b2da27a79f5c0074", "None": "b137c998bdaa2555d912e8fa", "bigquery": "b137c998bdaa2555d912e8fa", "databricks": "b137c998bdaa2555d912e8fa", -"duckdb": "f6c3d80bfee0ec0b286b82b5", +"duckdb": "27f973a6b2da27a79f5c0074", "generic": "b137c998bdaa2555d912e8fa", "motherduck": "901c5133ed47d4769286e860", "mysql": "b137c998bdaa2555d912e8fa", @@ -6853,11 +6853,11 @@ [ "type", { -"DuckDB": "05ebf79d18b2ae4eaa41f450", +"DuckDB": "0baaca265927140414f32b37", "None": "336fd1b7a1f7d8105c29280c", "bigquery": "aa04c66b4fed5190c90308fd", "databricks": "336fd1b7a1f7d8105c29280c", -"duckdb": "05ebf79d18b2ae4eaa41f450", +"duckdb": "0baaca265927140414f32b37", "generic": "336fd1b7a1f7d8105c29280c", "motherduck": "901c5133ed47d4769286e860", "mysql": "336fd1b7a1f7d8105c29280c", @@ -7074,18 +7074,18 @@ [ "type", { -"DuckDB": "23ee196b7bd6039855a1460f", -"None": "23ee196b7bd6039855a1460f", +"DuckDB": "5d25d5e13f81700741695ed4", +"None": "5d25d5e13f81700741695ed4", "bigquery": "23ee196b7bd6039855a1460f", -"databricks": "23ee196b7bd6039855a1460f", -"duckdb": "23ee196b7bd6039855a1460f", -"generic": "23ee196b7bd6039855a1460f", +"databricks": "5d25d5e13f81700741695ed4", +"duckdb": "5d25d5e13f81700741695ed4", +"generic": "5d25d5e13f81700741695ed4", "motherduck": "901c5133ed47d4769286e860", -"mysql": "23ee196b7bd6039855a1460f", -"postgres": "23ee196b7bd6039855a1460f", -"postgresql": "23ee196b7bd6039855a1460f", +"mysql": "5d25d5e13f81700741695ed4", +"postgres": "5d25d5e13f81700741695ed4", +"postgresql": "5d25d5e13f81700741695ed4", "snowflake": "5d25d5e13f81700741695ed4", -"tsql": "23ee196b7bd6039855a1460f" +"tsql": "5d25d5e13f81700741695ed4" } ], [ @@ -7635,12 +7635,12 @@ [ "type", { -"DuckDB": "d9315b336b73c3d8d4b6e9a3", -"None": "f10bdd4d40eb495cbcecd31d", +"DuckDB": "f2ed52f6bf46b4db9343551c", +"None": "f2ed52f6bf46b4db9343551c", "bigquery": "f10bdd4d40eb495cbcecd31d", -"databricks": "f10bdd4d40eb495cbcecd31d", -"duckdb": "d9315b336b73c3d8d4b6e9a3", -"generic": "f10bdd4d40eb495cbcecd31d", +"databricks": "f2ed52f6bf46b4db9343551c", +"duckdb": "f2ed52f6bf46b4db9343551c", +"generic": "f2ed52f6bf46b4db9343551c", "motherduck": "901c5133ed47d4769286e860", "mysql": "f2ed52f6bf46b4db9343551c", "postgres": "f2ed52f6bf46b4db9343551c", @@ -8876,18 +8876,18 @@ [ "type", { -"DuckDB": "0ca1be8f7d61ca627e620450", -"None": "0ca1be8f7d61ca627e620450", +"DuckDB": "18db5afe2c1e80d1ee652b54", +"None": "18db5afe2c1e80d1ee652b54", "bigquery": "e1c36a5cdd2427ff2b089ac9", -"databricks": "0ca1be8f7d61ca627e620450", -"duckdb": "0ca1be8f7d61ca627e620450", -"generic": "0ca1be8f7d61ca627e620450", +"databricks": "18db5afe2c1e80d1ee652b54", +"duckdb": "18db5afe2c1e80d1ee652b54", +"generic": "18db5afe2c1e80d1ee652b54", "motherduck": "901c5133ed47d4769286e860", -"mysql": "0ca1be8f7d61ca627e620450", -"postgres": "0ca1be8f7d61ca627e620450", -"postgresql": "0ca1be8f7d61ca627e620450", +"mysql": "18db5afe2c1e80d1ee652b54", +"postgres": "18db5afe2c1e80d1ee652b54", +"postgresql": "18db5afe2c1e80d1ee652b54", "snowflake": "18db5afe2c1e80d1ee652b54", -"tsql": "0ca1be8f7d61ca627e620450" +"tsql": "18db5afe2c1e80d1ee652b54" } ], [ @@ -9267,11 +9267,11 @@ [ "type", { -"DuckDB": "05ebf79d18b2ae4eaa41f450", +"DuckDB": "0baaca265927140414f32b37", "None": "336fd1b7a1f7d8105c29280c", "bigquery": "aa04c66b4fed5190c90308fd", "databricks": "336fd1b7a1f7d8105c29280c", -"duckdb": "05ebf79d18b2ae4eaa41f450", +"duckdb": "0baaca265927140414f32b37", "generic": "336fd1b7a1f7d8105c29280c", "motherduck": "901c5133ed47d4769286e860", "mysql": "336fd1b7a1f7d8105c29280c", @@ -9318,11 +9318,11 @@ [ "type", { -"DuckDB": "6226bc03436ecf8fccd39252", +"DuckDB": "5aa51d84ac5e6c7a9306f348", "None": "5aa51d84ac5e6c7a9306f348", -"bigquery": "5aa51d84ac5e6c7a9306f348", +"bigquery": "46487f079f1382836c406de7", "databricks": "5aa51d84ac5e6c7a9306f348", -"duckdb": "6226bc03436ecf8fccd39252", +"duckdb": "5aa51d84ac5e6c7a9306f348", "generic": "5aa51d84ac5e6c7a9306f348", "motherduck": "901c5133ed47d4769286e860", "mysql": "5aa51d84ac5e6c7a9306f348", @@ -9760,18 +9760,18 @@ [ "type", { -"DuckDB": "35d19e3415493d51f36f89b1", -"None": "35d19e3415493d51f36f89b1", +"DuckDB": "3e611643f0beab4953c4e09b", +"None": "3e611643f0beab4953c4e09b", "bigquery": "2a0918be53051fe629d9ff8f", -"databricks": "35d19e3415493d51f36f89b1", -"duckdb": "35d19e3415493d51f36f89b1", -"generic": "35d19e3415493d51f36f89b1", +"databricks": "3e611643f0beab4953c4e09b", +"duckdb": "3e611643f0beab4953c4e09b", +"generic": "3e611643f0beab4953c4e09b", "motherduck": "901c5133ed47d4769286e860", -"mysql": "35d19e3415493d51f36f89b1", -"postgres": "35d19e3415493d51f36f89b1", -"postgresql": "35d19e3415493d51f36f89b1", +"mysql": "3e611643f0beab4953c4e09b", +"postgres": "3e611643f0beab4953c4e09b", +"postgresql": "3e611643f0beab4953c4e09b", "snowflake": "3e611643f0beab4953c4e09b", -"tsql": "35d19e3415493d51f36f89b1" +"tsql": "3e611643f0beab4953c4e09b" } ], [ @@ -9998,18 +9998,18 @@ [ "type", { -"DuckDB": "63c80b5d8e861216f0c45bfd", -"None": "aa04c66b4fed5190c90308fd", +"DuckDB": "35a9c5c1419bff507cf65ae2", +"None": "d227b99a82e505d599c2ed8a", "bigquery": "aa04c66b4fed5190c90308fd", -"databricks": "aa04c66b4fed5190c90308fd", -"duckdb": "63c80b5d8e861216f0c45bfd", -"generic": "aa04c66b4fed5190c90308fd", +"databricks": "d227b99a82e505d599c2ed8a", +"duckdb": "35a9c5c1419bff507cf65ae2", +"generic": "d227b99a82e505d599c2ed8a", "motherduck": "901c5133ed47d4769286e860", -"mysql": "aa04c66b4fed5190c90308fd", -"postgres": "aa04c66b4fed5190c90308fd", -"postgresql": "aa04c66b4fed5190c90308fd", +"mysql": "d227b99a82e505d599c2ed8a", +"postgres": "d227b99a82e505d599c2ed8a", +"postgresql": "d227b99a82e505d599c2ed8a", "snowflake": "d227b99a82e505d599c2ed8a", -"tsql": "aa04c66b4fed5190c90308fd" +"tsql": "d227b99a82e505d599c2ed8a" } ], [ @@ -10661,18 +10661,18 @@ [ "type", { -"DuckDB": "bfce65957f9ed1017daedf21", -"None": "bfce65957f9ed1017daedf21", -"bigquery": "a82a2bde913bf78a8d3f9332", -"databricks": "2d20c9c50b78d6e633cf256a", -"duckdb": "bfce65957f9ed1017daedf21", -"generic": "bfce65957f9ed1017daedf21", +"DuckDB": "1bdf5908ca249b2331a0427c", +"None": "1bdf5908ca249b2331a0427c", +"bigquery": "730d7cf7510ba0934828d971", +"databricks": "1bdf5908ca249b2331a0427c", +"duckdb": "1bdf5908ca249b2331a0427c", +"generic": "1bdf5908ca249b2331a0427c", "motherduck": "901c5133ed47d4769286e860", -"mysql": "bfce65957f9ed1017daedf21", -"postgres": "bfce65957f9ed1017daedf21", -"postgresql": "bfce65957f9ed1017daedf21", -"snowflake": "2d20c9c50b78d6e633cf256a", -"tsql": "2d20c9c50b78d6e633cf256a" +"mysql": "1bdf5908ca249b2331a0427c", +"postgres": "1bdf5908ca249b2331a0427c", +"postgresql": "1bdf5908ca249b2331a0427c", +"snowflake": "8ff614b7306c0edbe36ce00b", +"tsql": "1bdf5908ca249b2331a0427c" } ], [ @@ -11273,11 +11273,11 @@ [ "type", { -"DuckDB": "d9315b336b73c3d8d4b6e9a3", +"DuckDB": "f2ed52f6bf46b4db9343551c", "None": "f2ed52f6bf46b4db9343551c", -"bigquery": "f2ed52f6bf46b4db9343551c", +"bigquery": "f10bdd4d40eb495cbcecd31d", "databricks": "f2ed52f6bf46b4db9343551c", -"duckdb": "d9315b336b73c3d8d4b6e9a3", +"duckdb": "f2ed52f6bf46b4db9343551c", "generic": "f2ed52f6bf46b4db9343551c", "motherduck": "901c5133ed47d4769286e860", "mysql": "f2ed52f6bf46b4db9343551c", @@ -11358,11 +11358,11 @@ [ "type", { -"DuckDB": "6226bc03436ecf8fccd39252", +"DuckDB": "5aa51d84ac5e6c7a9306f348", "None": "5aa51d84ac5e6c7a9306f348", -"bigquery": "5aa51d84ac5e6c7a9306f348", +"bigquery": "46487f079f1382836c406de7", "databricks": "5aa51d84ac5e6c7a9306f348", -"duckdb": "6226bc03436ecf8fccd39252", +"duckdb": "5aa51d84ac5e6c7a9306f348", "generic": "5aa51d84ac5e6c7a9306f348", "motherduck": "901c5133ed47d4769286e860", "mysql": "5aa51d84ac5e6c7a9306f348", @@ -11749,18 +11749,18 @@ [ "type", { -"DuckDB": "787213ce6b6e210ded1d752c", -"None": "787213ce6b6e210ded1d752c", +"DuckDB": "b561884132dbe7e42b719110", +"None": "b561884132dbe7e42b719110", "bigquery": "b1b23029bedaa16e2a6f8516", -"databricks": "787213ce6b6e210ded1d752c", -"duckdb": "787213ce6b6e210ded1d752c", -"generic": "787213ce6b6e210ded1d752c", +"databricks": "b561884132dbe7e42b719110", +"duckdb": "b561884132dbe7e42b719110", +"generic": "b561884132dbe7e42b719110", "motherduck": "901c5133ed47d4769286e860", -"mysql": "787213ce6b6e210ded1d752c", -"postgres": "787213ce6b6e210ded1d752c", -"postgresql": "787213ce6b6e210ded1d752c", +"mysql": "b561884132dbe7e42b719110", +"postgres": "b561884132dbe7e42b719110", +"postgresql": "b561884132dbe7e42b719110", "snowflake": "b561884132dbe7e42b719110", -"tsql": "787213ce6b6e210ded1d752c" +"tsql": "b561884132dbe7e42b719110" } ], [ @@ -12633,11 +12633,11 @@ [ "type", { -"DuckDB": "63c80b5d8e861216f0c45bfd", +"DuckDB": "35a9c5c1419bff507cf65ae2", "None": "d227b99a82e505d599c2ed8a", "bigquery": "aa04c66b4fed5190c90308fd", "databricks": "d227b99a82e505d599c2ed8a", -"duckdb": "63c80b5d8e861216f0c45bfd", +"duckdb": "35a9c5c1419bff507cf65ae2", "generic": "d227b99a82e505d599c2ed8a", "motherduck": "901c5133ed47d4769286e860", "mysql": "d227b99a82e505d599c2ed8a", @@ -12820,11 +12820,11 @@ [ "type", { -"DuckDB": "f6c3d80bfee0ec0b286b82b5", +"DuckDB": "27f973a6b2da27a79f5c0074", "None": "b137c998bdaa2555d912e8fa", "bigquery": "b137c998bdaa2555d912e8fa", "databricks": "b137c998bdaa2555d912e8fa", -"duckdb": "f6c3d80bfee0ec0b286b82b5", +"duckdb": "27f973a6b2da27a79f5c0074", "generic": "b137c998bdaa2555d912e8fa", "motherduck": "901c5133ed47d4769286e860", "mysql": "b137c998bdaa2555d912e8fa", @@ -12854,11 +12854,11 @@ [ "type", { -"DuckDB": "05ebf79d18b2ae4eaa41f450", +"DuckDB": "0baaca265927140414f32b37", "None": "336fd1b7a1f7d8105c29280c", "bigquery": "aa04c66b4fed5190c90308fd", "databricks": "336fd1b7a1f7d8105c29280c", -"duckdb": "05ebf79d18b2ae4eaa41f450", +"duckdb": "0baaca265927140414f32b37", "generic": "336fd1b7a1f7d8105c29280c", "motherduck": "901c5133ed47d4769286e860", "mysql": "336fd1b7a1f7d8105c29280c", @@ -13262,18 +13262,18 @@ [ "type", { -"DuckDB": "e1c36a5cdd2427ff2b089ac9", -"None": "e1c36a5cdd2427ff2b089ac9", +"DuckDB": "18db5afe2c1e80d1ee652b54", +"None": "18db5afe2c1e80d1ee652b54", "bigquery": "e1c36a5cdd2427ff2b089ac9", -"databricks": "e1c36a5cdd2427ff2b089ac9", -"duckdb": "e1c36a5cdd2427ff2b089ac9", -"generic": "e1c36a5cdd2427ff2b089ac9", +"databricks": "18db5afe2c1e80d1ee652b54", +"duckdb": "18db5afe2c1e80d1ee652b54", +"generic": "18db5afe2c1e80d1ee652b54", "motherduck": "901c5133ed47d4769286e860", -"mysql": "e1c36a5cdd2427ff2b089ac9", -"postgres": "e1c36a5cdd2427ff2b089ac9", -"postgresql": "e1c36a5cdd2427ff2b089ac9", +"mysql": "18db5afe2c1e80d1ee652b54", +"postgres": "18db5afe2c1e80d1ee652b54", +"postgresql": "18db5afe2c1e80d1ee652b54", "snowflake": "18db5afe2c1e80d1ee652b54", -"tsql": "e1c36a5cdd2427ff2b089ac9" +"tsql": "18db5afe2c1e80d1ee652b54" } ], [ @@ -14044,11 +14044,11 @@ [ "type", { -"DuckDB": "63c80b5d8e861216f0c45bfd", +"DuckDB": "35a9c5c1419bff507cf65ae2", "None": "d227b99a82e505d599c2ed8a", "bigquery": "aa04c66b4fed5190c90308fd", "databricks": "d227b99a82e505d599c2ed8a", -"duckdb": "63c80b5d8e861216f0c45bfd", +"duckdb": "35a9c5c1419bff507cf65ae2", "generic": "d227b99a82e505d599c2ed8a", "motherduck": "901c5133ed47d4769286e860", "mysql": "d227b99a82e505d599c2ed8a", @@ -14214,11 +14214,11 @@ [ "type", { -"DuckDB": "d9315b336b73c3d8d4b6e9a3", +"DuckDB": "f2ed52f6bf46b4db9343551c", "None": "f2ed52f6bf46b4db9343551c", -"bigquery": "f2ed52f6bf46b4db9343551c", +"bigquery": "f10bdd4d40eb495cbcecd31d", "databricks": "f2ed52f6bf46b4db9343551c", -"duckdb": "d9315b336b73c3d8d4b6e9a3", +"duckdb": "f2ed52f6bf46b4db9343551c", "generic": "f2ed52f6bf46b4db9343551c", "motherduck": "901c5133ed47d4769286e860", "mysql": "f2ed52f6bf46b4db9343551c", @@ -14350,18 +14350,18 @@ [ "type", { -"DuckDB": "23ee196b7bd6039855a1460f", -"None": "23ee196b7bd6039855a1460f", +"DuckDB": "5d25d5e13f81700741695ed4", +"None": "5d25d5e13f81700741695ed4", "bigquery": "23ee196b7bd6039855a1460f", -"databricks": "23ee196b7bd6039855a1460f", -"duckdb": "23ee196b7bd6039855a1460f", -"generic": "23ee196b7bd6039855a1460f", +"databricks": "5d25d5e13f81700741695ed4", +"duckdb": "5d25d5e13f81700741695ed4", +"generic": "5d25d5e13f81700741695ed4", "motherduck": "901c5133ed47d4769286e860", -"mysql": "23ee196b7bd6039855a1460f", -"postgres": "23ee196b7bd6039855a1460f", -"postgresql": "23ee196b7bd6039855a1460f", +"mysql": "5d25d5e13f81700741695ed4", +"postgres": "5d25d5e13f81700741695ed4", +"postgresql": "5d25d5e13f81700741695ed4", "snowflake": "5d25d5e13f81700741695ed4", -"tsql": "23ee196b7bd6039855a1460f" +"tsql": "5d25d5e13f81700741695ed4" } ], [ @@ -14996,18 +14996,18 @@ [ "type", { -"DuckDB": "4c4563a9f306a1bb11e116d9", -"None": "4c4563a9f306a1bb11e116d9", +"DuckDB": "5d25d5e13f81700741695ed4", +"None": "5d25d5e13f81700741695ed4", "bigquery": "23ee196b7bd6039855a1460f", -"databricks": "4c4563a9f306a1bb11e116d9", -"duckdb": "4c4563a9f306a1bb11e116d9", -"generic": "4c4563a9f306a1bb11e116d9", +"databricks": "5d25d5e13f81700741695ed4", +"duckdb": "5d25d5e13f81700741695ed4", +"generic": "5d25d5e13f81700741695ed4", "motherduck": "901c5133ed47d4769286e860", -"mysql": "4c4563a9f306a1bb11e116d9", -"postgres": "4c4563a9f306a1bb11e116d9", -"postgresql": "4c4563a9f306a1bb11e116d9", +"mysql": "5d25d5e13f81700741695ed4", +"postgres": "5d25d5e13f81700741695ed4", +"postgresql": "5d25d5e13f81700741695ed4", "snowflake": "5d25d5e13f81700741695ed4", -"tsql": "4c4563a9f306a1bb11e116d9" +"tsql": "5d25d5e13f81700741695ed4" } ], [ @@ -15030,18 +15030,18 @@ [ "type", { -"DuckDB": "787213ce6b6e210ded1d752c", -"None": "787213ce6b6e210ded1d752c", +"DuckDB": "b561884132dbe7e42b719110", +"None": "b561884132dbe7e42b719110", "bigquery": "b1b23029bedaa16e2a6f8516", -"databricks": "787213ce6b6e210ded1d752c", -"duckdb": "787213ce6b6e210ded1d752c", -"generic": "787213ce6b6e210ded1d752c", +"databricks": "b561884132dbe7e42b719110", +"duckdb": "b561884132dbe7e42b719110", +"generic": "b561884132dbe7e42b719110", "motherduck": "901c5133ed47d4769286e860", -"mysql": "787213ce6b6e210ded1d752c", -"postgres": "787213ce6b6e210ded1d752c", -"postgresql": "787213ce6b6e210ded1d752c", +"mysql": "b561884132dbe7e42b719110", +"postgres": "b561884132dbe7e42b719110", +"postgresql": "b561884132dbe7e42b719110", "snowflake": "b561884132dbe7e42b719110", -"tsql": "787213ce6b6e210ded1d752c" +"tsql": "b561884132dbe7e42b719110" } ], [ @@ -15149,11 +15149,11 @@ [ "type", { -"DuckDB": "05ebf79d18b2ae4eaa41f450", +"DuckDB": "0baaca265927140414f32b37", "None": "ab62f43260f32e2cf62666fc", "bigquery": "aa04c66b4fed5190c90308fd", "databricks": "ab62f43260f32e2cf62666fc", -"duckdb": "05ebf79d18b2ae4eaa41f450", +"duckdb": "0baaca265927140414f32b37", "generic": "ab62f43260f32e2cf62666fc", "motherduck": "901c5133ed47d4769286e860", "mysql": "ab62f43260f32e2cf62666fc", @@ -15270,7 +15270,7 @@ { "DuckDB": "19bdd92ea0cc5fb028657510", "None": "78b2cf275c640af54feeca31", -"bigquery": "78b2cf275c640af54feeca31", +"bigquery": "1e75e5227bea0ae294ac7af1", "databricks": "1e75e5227bea0ae294ac7af1", "duckdb": "19bdd92ea0cc5fb028657510", "generic": "78b2cf275c640af54feeca31", @@ -16339,18 +16339,18 @@ [ "type", { -"DuckDB": "b1b23029bedaa16e2a6f8516", -"None": "b1b23029bedaa16e2a6f8516", +"DuckDB": "b561884132dbe7e42b719110", +"None": "b561884132dbe7e42b719110", "bigquery": "b1b23029bedaa16e2a6f8516", -"databricks": "b1b23029bedaa16e2a6f8516", -"duckdb": "b1b23029bedaa16e2a6f8516", -"generic": "b1b23029bedaa16e2a6f8516", +"databricks": "b561884132dbe7e42b719110", +"duckdb": "b561884132dbe7e42b719110", +"generic": "b561884132dbe7e42b719110", "motherduck": "901c5133ed47d4769286e860", -"mysql": "b1b23029bedaa16e2a6f8516", -"postgres": "b1b23029bedaa16e2a6f8516", -"postgresql": "b1b23029bedaa16e2a6f8516", +"mysql": "b561884132dbe7e42b719110", +"postgres": "b561884132dbe7e42b719110", +"postgresql": "b561884132dbe7e42b719110", "snowflake": "b561884132dbe7e42b719110", -"tsql": "b1b23029bedaa16e2a6f8516" +"tsql": "b561884132dbe7e42b719110" } ], [ @@ -16424,18 +16424,18 @@ [ "type", { -"DuckDB": "23ee196b7bd6039855a1460f", -"None": "23ee196b7bd6039855a1460f", +"DuckDB": "5d25d5e13f81700741695ed4", +"None": "5d25d5e13f81700741695ed4", "bigquery": "23ee196b7bd6039855a1460f", -"databricks": "23ee196b7bd6039855a1460f", -"duckdb": "23ee196b7bd6039855a1460f", -"generic": "23ee196b7bd6039855a1460f", +"databricks": "5d25d5e13f81700741695ed4", +"duckdb": "5d25d5e13f81700741695ed4", +"generic": "5d25d5e13f81700741695ed4", "motherduck": "901c5133ed47d4769286e860", -"mysql": "23ee196b7bd6039855a1460f", -"postgres": "23ee196b7bd6039855a1460f", -"postgresql": "23ee196b7bd6039855a1460f", +"mysql": "5d25d5e13f81700741695ed4", +"postgres": "5d25d5e13f81700741695ed4", +"postgresql": "5d25d5e13f81700741695ed4", "snowflake": "5d25d5e13f81700741695ed4", -"tsql": "23ee196b7bd6039855a1460f" +"tsql": "5d25d5e13f81700741695ed4" } ], [ @@ -16560,18 +16560,18 @@ [ "type", { -"DuckDB": "23ee196b7bd6039855a1460f", -"None": "23ee196b7bd6039855a1460f", +"DuckDB": "5d25d5e13f81700741695ed4", +"None": "5d25d5e13f81700741695ed4", "bigquery": "23ee196b7bd6039855a1460f", -"databricks": "23ee196b7bd6039855a1460f", -"duckdb": "23ee196b7bd6039855a1460f", -"generic": "23ee196b7bd6039855a1460f", +"databricks": "5d25d5e13f81700741695ed4", +"duckdb": "5d25d5e13f81700741695ed4", +"generic": "5d25d5e13f81700741695ed4", "motherduck": "901c5133ed47d4769286e860", -"mysql": "23ee196b7bd6039855a1460f", -"postgres": "23ee196b7bd6039855a1460f", -"postgresql": "23ee196b7bd6039855a1460f", +"mysql": "5d25d5e13f81700741695ed4", +"postgres": "5d25d5e13f81700741695ed4", +"postgresql": "5d25d5e13f81700741695ed4", "snowflake": "5d25d5e13f81700741695ed4", -"tsql": "23ee196b7bd6039855a1460f" +"tsql": "5d25d5e13f81700741695ed4" } ], [ @@ -16730,18 +16730,18 @@ [ "type", { -"DuckDB": "4062ec81199547543ab068fb", -"None": "4062ec81199547543ab068fb", -"bigquery": "7009fc15fd2c35608ab16caf", -"databricks": "0739784ba4c53d23afd45a6b", -"duckdb": "4062ec81199547543ab068fb", -"generic": "4062ec81199547543ab068fb", +"DuckDB": "5aa51d84ac5e6c7a9306f348", +"None": "5aa51d84ac5e6c7a9306f348", +"bigquery": "46487f079f1382836c406de7", +"databricks": "5aa51d84ac5e6c7a9306f348", +"duckdb": "5aa51d84ac5e6c7a9306f348", +"generic": "5aa51d84ac5e6c7a9306f348", "motherduck": "901c5133ed47d4769286e860", -"mysql": "4062ec81199547543ab068fb", -"postgres": "4062ec81199547543ab068fb", -"postgresql": "4062ec81199547543ab068fb", -"snowflake": "0739784ba4c53d23afd45a6b", -"tsql": "0739784ba4c53d23afd45a6b" +"mysql": "5aa51d84ac5e6c7a9306f348", +"postgres": "5aa51d84ac5e6c7a9306f348", +"postgresql": "5aa51d84ac5e6c7a9306f348", +"snowflake": "5aa51d84ac5e6c7a9306f348", +"tsql": "5aa51d84ac5e6c7a9306f348" } ], [ @@ -17087,12 +17087,12 @@ [ "type", { -"DuckDB": "72bd5e38b32893397eb51579", -"None": "730d7cf7510ba0934828d971", +"DuckDB": "1bdf5908ca249b2331a0427c", +"None": "1bdf5908ca249b2331a0427c", "bigquery": "730d7cf7510ba0934828d971", -"databricks": "730d7cf7510ba0934828d971", -"duckdb": "72bd5e38b32893397eb51579", -"generic": "730d7cf7510ba0934828d971", +"databricks": "1bdf5908ca249b2331a0427c", +"duckdb": "1bdf5908ca249b2331a0427c", +"generic": "1bdf5908ca249b2331a0427c", "motherduck": "901c5133ed47d4769286e860", "mysql": "1bdf5908ca249b2331a0427c", "postgres": "1bdf5908ca249b2331a0427c", @@ -17529,18 +17529,18 @@ [ "type", { -"DuckDB": "0ca1be8f7d61ca627e620450", -"None": "0ca1be8f7d61ca627e620450", +"DuckDB": "18db5afe2c1e80d1ee652b54", +"None": "18db5afe2c1e80d1ee652b54", "bigquery": "e1c36a5cdd2427ff2b089ac9", -"databricks": "0ca1be8f7d61ca627e620450", -"duckdb": "0ca1be8f7d61ca627e620450", -"generic": "0ca1be8f7d61ca627e620450", +"databricks": "18db5afe2c1e80d1ee652b54", +"duckdb": "18db5afe2c1e80d1ee652b54", +"generic": "18db5afe2c1e80d1ee652b54", "motherduck": "901c5133ed47d4769286e860", -"mysql": "0ca1be8f7d61ca627e620450", -"postgres": "0ca1be8f7d61ca627e620450", -"postgresql": "0ca1be8f7d61ca627e620450", +"mysql": "18db5afe2c1e80d1ee652b54", +"postgres": "18db5afe2c1e80d1ee652b54", +"postgresql": "18db5afe2c1e80d1ee652b54", "snowflake": "18db5afe2c1e80d1ee652b54", -"tsql": "0ca1be8f7d61ca627e620450" +"tsql": "18db5afe2c1e80d1ee652b54" } ], [ @@ -17886,11 +17886,11 @@ [ "type", { -"DuckDB": "638ca9212b62ada31a448c62", +"DuckDB": "d81d8c19074af0e39239164d", "None": "1fec266134381c760a254d16", "bigquery": "1fec266134381c760a254d16", "databricks": "1fec266134381c760a254d16", -"duckdb": "638ca9212b62ada31a448c62", +"duckdb": "d81d8c19074af0e39239164d", "generic": "1fec266134381c760a254d16", "motherduck": "901c5133ed47d4769286e860", "mysql": "1fec266134381c760a254d16", @@ -17920,18 +17920,18 @@ [ "type", { -"DuckDB": "b1b23029bedaa16e2a6f8516", -"None": "b1b23029bedaa16e2a6f8516", +"DuckDB": "b561884132dbe7e42b719110", +"None": "b561884132dbe7e42b719110", "bigquery": "b1b23029bedaa16e2a6f8516", -"databricks": "b1b23029bedaa16e2a6f8516", -"duckdb": "b1b23029bedaa16e2a6f8516", -"generic": "b1b23029bedaa16e2a6f8516", +"databricks": "b561884132dbe7e42b719110", +"duckdb": "b561884132dbe7e42b719110", +"generic": "b561884132dbe7e42b719110", "motherduck": "901c5133ed47d4769286e860", -"mysql": "b1b23029bedaa16e2a6f8516", -"postgres": "b1b23029bedaa16e2a6f8516", -"postgresql": "b1b23029bedaa16e2a6f8516", +"mysql": "b561884132dbe7e42b719110", +"postgres": "b561884132dbe7e42b719110", +"postgresql": "b561884132dbe7e42b719110", "snowflake": "b561884132dbe7e42b719110", -"tsql": "b1b23029bedaa16e2a6f8516" +"tsql": "b561884132dbe7e42b719110" } ], [ @@ -18772,7 +18772,7 @@ { "DuckDB": "19bdd92ea0cc5fb028657510", "None": "78b2cf275c640af54feeca31", -"bigquery": "78b2cf275c640af54feeca31", +"bigquery": "1e75e5227bea0ae294ac7af1", "databricks": "1e75e5227bea0ae294ac7af1", "duckdb": "19bdd92ea0cc5fb028657510", "generic": "78b2cf275c640af54feeca31", @@ -19263,11 +19263,11 @@ [ "type", { -"DuckDB": "05ebf79d18b2ae4eaa41f450", +"DuckDB": "0baaca265927140414f32b37", "None": "336fd1b7a1f7d8105c29280c", "bigquery": "aa04c66b4fed5190c90308fd", "databricks": "336fd1b7a1f7d8105c29280c", -"duckdb": "05ebf79d18b2ae4eaa41f450", +"duckdb": "0baaca265927140414f32b37", "generic": "336fd1b7a1f7d8105c29280c", "motherduck": "901c5133ed47d4769286e860", "mysql": "336fd1b7a1f7d8105c29280c", @@ -20606,12 +20606,12 @@ [ "type", { -"DuckDB": "72bd5e38b32893397eb51579", -"None": "730d7cf7510ba0934828d971", +"DuckDB": "1bdf5908ca249b2331a0427c", +"None": "1bdf5908ca249b2331a0427c", "bigquery": "730d7cf7510ba0934828d971", -"databricks": "730d7cf7510ba0934828d971", -"duckdb": "72bd5e38b32893397eb51579", -"generic": "730d7cf7510ba0934828d971", +"databricks": "1bdf5908ca249b2331a0427c", +"duckdb": "1bdf5908ca249b2331a0427c", +"generic": "1bdf5908ca249b2331a0427c", "motherduck": "901c5133ed47d4769286e860", "mysql": "1bdf5908ca249b2331a0427c", "postgres": "1bdf5908ca249b2331a0427c", @@ -21371,12 +21371,12 @@ [ "type", { -"DuckDB": "6226bc03436ecf8fccd39252", -"None": "46487f079f1382836c406de7", +"DuckDB": "5aa51d84ac5e6c7a9306f348", +"None": "5aa51d84ac5e6c7a9306f348", "bigquery": "46487f079f1382836c406de7", -"databricks": "46487f079f1382836c406de7", -"duckdb": "6226bc03436ecf8fccd39252", -"generic": "46487f079f1382836c406de7", +"databricks": "5aa51d84ac5e6c7a9306f348", +"duckdb": "5aa51d84ac5e6c7a9306f348", +"generic": "5aa51d84ac5e6c7a9306f348", "motherduck": "901c5133ed47d4769286e860", "mysql": "5aa51d84ac5e6c7a9306f348", "postgres": "5aa51d84ac5e6c7a9306f348", @@ -21524,12 +21524,12 @@ [ "type", { -"DuckDB": "72bd5e38b32893397eb51579", -"None": "730d7cf7510ba0934828d971", +"DuckDB": "1bdf5908ca249b2331a0427c", +"None": "1bdf5908ca249b2331a0427c", "bigquery": "730d7cf7510ba0934828d971", -"databricks": "730d7cf7510ba0934828d971", -"duckdb": "72bd5e38b32893397eb51579", -"generic": "730d7cf7510ba0934828d971", +"databricks": "1bdf5908ca249b2331a0427c", +"duckdb": "1bdf5908ca249b2331a0427c", +"generic": "1bdf5908ca249b2331a0427c", "motherduck": "901c5133ed47d4769286e860", "mysql": "1bdf5908ca249b2331a0427c", "postgres": "1bdf5908ca249b2331a0427c", @@ -22289,11 +22289,11 @@ [ "type", { -"DuckDB": "6226bc03436ecf8fccd39252", +"DuckDB": "5aa51d84ac5e6c7a9306f348", "None": "5aa51d84ac5e6c7a9306f348", -"bigquery": "5aa51d84ac5e6c7a9306f348", +"bigquery": "46487f079f1382836c406de7", "databricks": "5aa51d84ac5e6c7a9306f348", -"duckdb": "6226bc03436ecf8fccd39252", +"duckdb": "5aa51d84ac5e6c7a9306f348", "generic": "5aa51d84ac5e6c7a9306f348", "motherduck": "901c5133ed47d4769286e860", "mysql": "5aa51d84ac5e6c7a9306f348", @@ -22527,18 +22527,18 @@ [ "type", { -"DuckDB": "787213ce6b6e210ded1d752c", -"None": "787213ce6b6e210ded1d752c", +"DuckDB": "b561884132dbe7e42b719110", +"None": "b561884132dbe7e42b719110", "bigquery": "b1b23029bedaa16e2a6f8516", -"databricks": "787213ce6b6e210ded1d752c", -"duckdb": "787213ce6b6e210ded1d752c", -"generic": "787213ce6b6e210ded1d752c", +"databricks": "b561884132dbe7e42b719110", +"duckdb": "b561884132dbe7e42b719110", +"generic": "b561884132dbe7e42b719110", "motherduck": "901c5133ed47d4769286e860", -"mysql": "787213ce6b6e210ded1d752c", -"postgres": "787213ce6b6e210ded1d752c", -"postgresql": "787213ce6b6e210ded1d752c", +"mysql": "b561884132dbe7e42b719110", +"postgres": "b561884132dbe7e42b719110", +"postgresql": "b561884132dbe7e42b719110", "snowflake": "b561884132dbe7e42b719110", -"tsql": "787213ce6b6e210ded1d752c" +"tsql": "b561884132dbe7e42b719110" } ], [ @@ -22714,18 +22714,18 @@ [ "type", { -"DuckDB": "2a0918be53051fe629d9ff8f", -"None": "2a0918be53051fe629d9ff8f", +"DuckDB": "3e611643f0beab4953c4e09b", +"None": "3e611643f0beab4953c4e09b", "bigquery": "2a0918be53051fe629d9ff8f", -"databricks": "2a0918be53051fe629d9ff8f", -"duckdb": "2a0918be53051fe629d9ff8f", -"generic": "2a0918be53051fe629d9ff8f", +"databricks": "3e611643f0beab4953c4e09b", +"duckdb": "3e611643f0beab4953c4e09b", +"generic": "3e611643f0beab4953c4e09b", "motherduck": "901c5133ed47d4769286e860", -"mysql": "2a0918be53051fe629d9ff8f", -"postgres": "2a0918be53051fe629d9ff8f", -"postgresql": "2a0918be53051fe629d9ff8f", +"mysql": "3e611643f0beab4953c4e09b", +"postgres": "3e611643f0beab4953c4e09b", +"postgresql": "3e611643f0beab4953c4e09b", "snowflake": "3e611643f0beab4953c4e09b", -"tsql": "2a0918be53051fe629d9ff8f" +"tsql": "3e611643f0beab4953c4e09b" } ], [ @@ -23105,11 +23105,11 @@ [ "type", { -"DuckDB": "d9315b336b73c3d8d4b6e9a3", +"DuckDB": "f2ed52f6bf46b4db9343551c", "None": "f2ed52f6bf46b4db9343551c", -"bigquery": "f2ed52f6bf46b4db9343551c", +"bigquery": "f10bdd4d40eb495cbcecd31d", "databricks": "f2ed52f6bf46b4db9343551c", -"duckdb": "d9315b336b73c3d8d4b6e9a3", +"duckdb": "f2ed52f6bf46b4db9343551c", "generic": "f2ed52f6bf46b4db9343551c", "motherduck": "901c5133ed47d4769286e860", "mysql": "f2ed52f6bf46b4db9343551c", @@ -23190,18 +23190,18 @@ [ "type", { -"DuckDB": "0ca1be8f7d61ca627e620450", -"None": "0ca1be8f7d61ca627e620450", +"DuckDB": "18db5afe2c1e80d1ee652b54", +"None": "18db5afe2c1e80d1ee652b54", "bigquery": "e1c36a5cdd2427ff2b089ac9", -"databricks": "0ca1be8f7d61ca627e620450", -"duckdb": "0ca1be8f7d61ca627e620450", -"generic": "0ca1be8f7d61ca627e620450", +"databricks": "18db5afe2c1e80d1ee652b54", +"duckdb": "18db5afe2c1e80d1ee652b54", +"generic": "18db5afe2c1e80d1ee652b54", "motherduck": "901c5133ed47d4769286e860", -"mysql": "0ca1be8f7d61ca627e620450", -"postgres": "0ca1be8f7d61ca627e620450", -"postgresql": "0ca1be8f7d61ca627e620450", +"mysql": "18db5afe2c1e80d1ee652b54", +"postgres": "18db5afe2c1e80d1ee652b54", +"postgresql": "18db5afe2c1e80d1ee652b54", "snowflake": "18db5afe2c1e80d1ee652b54", -"tsql": "0ca1be8f7d61ca627e620450" +"tsql": "18db5afe2c1e80d1ee652b54" } ], [ @@ -23564,28 +23564,28 @@ [ "type", { -"DuckDB": "787213ce6b6e210ded1d752c", -"None": "787213ce6b6e210ded1d752c", +"DuckDB": "b561884132dbe7e42b719110", +"None": "b561884132dbe7e42b719110", "bigquery": "b1b23029bedaa16e2a6f8516", -"databricks": "787213ce6b6e210ded1d752c", -"duckdb": "787213ce6b6e210ded1d752c", -"generic": "787213ce6b6e210ded1d752c", +"databricks": "b561884132dbe7e42b719110", +"duckdb": "b561884132dbe7e42b719110", +"generic": "b561884132dbe7e42b719110", "motherduck": "901c5133ed47d4769286e860", -"mysql": "787213ce6b6e210ded1d752c", -"postgres": "787213ce6b6e210ded1d752c", -"postgresql": "787213ce6b6e210ded1d752c", +"mysql": "b561884132dbe7e42b719110", +"postgres": "b561884132dbe7e42b719110", +"postgresql": "b561884132dbe7e42b719110", "snowflake": "b561884132dbe7e42b719110", -"tsql": "787213ce6b6e210ded1d752c" +"tsql": "b561884132dbe7e42b719110" } ], [ "type", { -"DuckDB": "63c80b5d8e861216f0c45bfd", +"DuckDB": "35a9c5c1419bff507cf65ae2", "None": "f8219c960720043f0662a603", "bigquery": "f8219c960720043f0662a603", "databricks": "f8219c960720043f0662a603", -"duckdb": "63c80b5d8e861216f0c45bfd", +"duckdb": "35a9c5c1419bff507cf65ae2", "generic": "f8219c960720043f0662a603", "motherduck": "901c5133ed47d4769286e860", "mysql": "f8219c960720043f0662a603", @@ -24669,11 +24669,11 @@ [ "type", { -"DuckDB": "f6c3d80bfee0ec0b286b82b5", +"DuckDB": "27f973a6b2da27a79f5c0074", "None": "b137c998bdaa2555d912e8fa", "bigquery": "b137c998bdaa2555d912e8fa", "databricks": "b137c998bdaa2555d912e8fa", -"duckdb": "f6c3d80bfee0ec0b286b82b5", +"duckdb": "27f973a6b2da27a79f5c0074", "generic": "b137c998bdaa2555d912e8fa", "motherduck": "901c5133ed47d4769286e860", "mysql": "b137c998bdaa2555d912e8fa", diff --git a/tests/integration/src/sqlbuild/adapters/duckdb/_test_types.py b/tests/integration/src/sqlbuild/adapters/duckdb/_test_types.py index 09db89e6db..c9a9638a43 100644 --- a/tests/integration/src/sqlbuild/adapters/duckdb/_test_types.py +++ b/tests/integration/src/sqlbuild/adapters/duckdb/_test_types.py @@ -242,3 +242,20 @@ class SnapshotTransactionRollbackTestCase: description: str expected_error_fragment: str expected_rows_after_failure: tuple[tuple[object, ...], ...] + + +@dataclass(frozen=True) +class StorageVersionConnectTestCase: + """A database file written by an earlier storage version, reopened for local replay.""" + + description: str + expected_rows: list[tuple[int]] + + +@dataclass(frozen=True) +class UserStorageVersionKeyTestCase: + """A user connection config naming a DuckDB storage version, and the storage files get.""" + + description: str + config_key: str + expected_storage_version: str diff --git a/tests/integration/src/sqlbuild/adapters/duckdb/test_client.py b/tests/integration/src/sqlbuild/adapters/duckdb/test_client.py index 60e9669bbb..4d7a50849e 100644 --- a/tests/integration/src/sqlbuild/adapters/duckdb/test_client.py +++ b/tests/integration/src/sqlbuild/adapters/duckdb/test_client.py @@ -44,8 +44,10 @@ RecorderWriteTestCase, RelationExistsTestCase, RenameTestCase, + StorageVersionConnectTestCase, SwapTestCase, TransactionalAtomicityTestCase, + UserStorageVersionKeyTestCase, ) from tests.integration.src.sqlbuild.adapters.duckdb.helpers import count_relation_rows @@ -161,6 +163,65 @@ def test_given_config_when_connecting_then_returns_usable_connection( adapter.close(connection) +@pytest.mark.parametrize( + "test_case", + [ + StorageVersionConnectTestCase( + description="an existing default-storage database opens for local replay", + expected_rows=[(1,)], + ), + ], + ids=lambda case: case.description, +) +def test_given_existing_database_when_opening_for_local_replay_then_its_tables_read( + test_case: StorageVersionConnectTestCase, + adapter: DuckDbAdapter, + tmp_path: Path, +) -> None: + database: Path = tmp_path / "existing.duckdb" + written: duckdb.DuckDBPyConnection = duckdb.connect(str(database)) + written.execute("CREATE TABLE orders AS SELECT 1 AS order_id") + written.close() + + connection: Any = adapter.connect_local_replay(database) + rows: list[tuple[int]] = connection.execute("SELECT order_id FROM orders").fetchall() + adapter.close(connection) + + assert rows == test_case.expected_rows + + +@pytest.mark.parametrize( + "test_case", + [ + UserStorageVersionKeyTestCase( + description="a storage version in user config is ignored like any unknown key", + config_key="storage_compatibility_version", + expected_storage_version="v1.0.0+", + ), + UserStorageVersionKeyTestCase( + description="an unknown connection key is ignored", + config_key="unknown_option", + expected_storage_version="v1.0.0+", + ), + ], + ids=lambda case: case.description, +) +def test_given_user_config_with_storage_version_when_connecting_then_it_is_ignored( + test_case: UserStorageVersionKeyTestCase, + adapter: DuckDbAdapter, + tmp_path: Path, +) -> None: + database: Path = tmp_path / "warehouse.duckdb" + + connection: Any = adapter.connect({"database": str(database), test_case.config_key: "latest"}) + tags: list[tuple[dict[str, str]]] = connection.execute( + "SELECT tags FROM duckdb_databases() WHERE database_name = 'warehouse'" + ).fetchall() + adapter.close(connection) + + assert tags == [({"storage_version": test_case.expected_storage_version},)] + + @pytest.mark.parametrize( "test_case", [ diff --git a/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py b/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py index 598e87a79b..07377d7931 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py @@ -323,6 +323,10 @@ class FormatCompileIntegrationTestCase: description: str expected_literal: str + upstream_sql: str = ( + "SELECT CAST(1 AS INTEGER) AS order_id, CAST('new' AS VARCHAR) AS order_label\n" + ) + selected_rules: str = '["SQBRSQL021"]' @dataclass(frozen=True) diff --git a/tests/integration/src/sqlbuild/cli/commands/main/test_format.py b/tests/integration/src/sqlbuild/cli/commands/main/test_format.py index 153087da93..d15e674dcd 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/test_format.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/test_format.py @@ -151,7 +151,18 @@ def test_given_commented_lateral_alias_when_formatting_then_canonical_sql_is_wri FormatCompileIntegrationTestCase( description="canonical empty input fixture remains compiler-compatible", expected_literal="SELECT * FROM __empty_fixture()", - ) + ), + FormatCompileIntegrationTestCase( + description="contracted star model accepts the canonical empty fixture", + expected_literal="SELECT * FROM __empty_fixture()", + upstream_sql=( + "WITH staged AS (\n" + " SELECT CAST(1 AS INTEGER) AS order_id, CAST('new' AS VARCHAR) AS order_label\n" + ")\n" + "SELECT * FROM staged\n" + ), + selected_rules="[]", + ), ], ids=lambda case: case.description, ) @@ -161,7 +172,8 @@ def test_given_typed_null_input_when_fixture_formatting_then_canonical_empty_fix ) -> None: """Prove formatter output remains valid with uncontrolled-star lint selected.""" (tmp_path / "sqlbuild_project.toml").write_text( - 'name = "orders"\nadapter = "duckdb"\n\n[rules]\nselect = ["SQBRSQL021"]\n', + f'name = "orders"\nadapter = "duckdb"\n\n[rules]\nselect = {test_case.selected_rules}\n' + '\n[connection]\ndatabase = "orders.duckdb"\n', encoding="utf-8", ) models: Path = tmp_path / "models" @@ -174,8 +186,7 @@ def test_given_typed_null_input_when_fixture_formatting_then_canonical_empty_fix " order_id (type INTEGER, nullable true),\n" " order_label (type VARCHAR, nullable true),\n" " ),\n" - ");\n\n" - "SELECT CAST(1 AS INTEGER) AS order_id, CAST('new' AS VARCHAR) AS order_label\n", + ");\n\n" + test_case.upstream_sql, encoding="utf-8", ) (models / "orders.sql").write_text( @@ -193,16 +204,21 @@ def test_given_typed_null_input_when_fixture_formatting_then_canonical_empty_fix " CAST(NULL AS VARCHAR) AS order_label\n" " WHERE FALSE\n" "),\n" - "__expected__orders AS (SELECT 1 AS order_id, 'new' AS order_label)\n" + "__expected__orders AS (\n" + " SELECT CAST(NULL AS INTEGER) AS order_id, CAST(NULL AS VARCHAR) AS order_label\n" + " WHERE FALSE\n" + ")\n" "SELECT 1\n", encoding="utf-8", ) format_exit: int = main(["--project-dir", str(tmp_path), "format", "--fixtures-only"]) compile_exit: int = main(["--project-dir", str(tmp_path), "compile", "--no-cache"]) + test_exit: int = main(["--project-dir", str(tmp_path), "test"]) assert format_exit == 0 assert compile_exit == 0 + assert test_exit == 0 assert test_case.expected_literal in test_file.read_text(encoding="utf-8") diff --git a/tests/integration/src/sqlbuild/compiler/pipeline/test_native_binding_catalog.py b/tests/integration/src/sqlbuild/compiler/pipeline/test_native_binding_catalog.py index ad43c91fb6..0b17af264c 100644 --- a/tests/integration/src/sqlbuild/compiler/pipeline/test_native_binding_catalog.py +++ b/tests/integration/src/sqlbuild/compiler/pipeline/test_native_binding_catalog.py @@ -150,6 +150,50 @@ def test_given_catalog_when_rebinding_physical_shapes_then_does_not_reuse_stale_ assert repeated == missing +@pytest.mark.parametrize( + "test_case", + [ + NativeCatalogCase("unquoted reference", 'SELECT bundle_sku FROM __source("orders")'), + NativeCatalogCase("uppercase reference", 'SELECT BUNDLE_SKU FROM __source("orders")'), + NativeCatalogCase( + "quoted uppercase reference", 'SELECT "BUNDLE_SKU" FROM __source("orders")' + ), + NativeCatalogCase( + "quoted lowercase reference", 'SELECT "bundle_sku" FROM __source("orders")', ("B002",) + ), + NativeCatalogCase( + "unknown column", 'SELECT missing_sku FROM __source("orders")', ("B002",) + ), + ], + ids=lambda case: case.description, +) +def test_given_case_sensitive_snowflake_when_rebinding_listed_columns_then_unquoted_names_bind( + test_case: NativeCatalogCase, tmp_path: Path +) -> None: + (tmp_path / "sqlbuild_project.toml").write_text( + 'name = "orders"\nadapter = "snowflake"\n[rules]\nselect = []\n' + ) + (tmp_path / "models").mkdir() + (tmp_path / "sources").mkdir() + (tmp_path / "sources/orders.yml").write_text( + "sources:\n - name: orders\n description: Test source orders.\n table: orders\n" + ) + (tmp_path / "models/report.sql").write_text( + "MODEL (description 'Test model report.', database warehouse, schema analytics);\n" + + test_case.sql + ) + project: CompiledProjectView = build_compiled_project( + discovered_inputs=discover_project_inputs(project_dir=tmp_path), adapter=SnowflakeAdapter() + ) + diagnostics: tuple[CompilerDiagnostic, ...] = source_binding_diagnostics( + project=project, + columns={"orders": (ColumnInfo("bundle_sku", "VARCHAR"),)}, + profile=ExpressionInferenceProfile(sql_analysis_dialect="snowflake"), + selected_keys=frozenset(model.key for model in project.models), + ) + assert tuple(item.code for item in diagnostics) == test_case.expected_codes + + @pytest.mark.parametrize( "test_case", [ diff --git a/tests/integration/src/sqlbuild/compiler/pipeline/test_semantic_triage.py b/tests/integration/src/sqlbuild/compiler/pipeline/test_semantic_triage.py index 25e8701f84..6bc4077592 100644 --- a/tests/integration/src/sqlbuild/compiler/pipeline/test_semantic_triage.py +++ b/tests/integration/src/sqlbuild/compiler/pipeline/test_semantic_triage.py @@ -107,7 +107,7 @@ def test_given_effective_target_when_binding_quotes_then_honours_session_setting profile=ExpressionInferenceProfile(sql_analysis_dialect="snowflake"), selected_keys=frozenset(model.key for model in project.models), ) - assert physical == () + assert tuple(item.code for item in physical) == test_case.expected_codes @pytest.mark.parametrize( diff --git a/tests/integration/src/sqlbuild/executor/build/concurrent/helpers.py b/tests/integration/src/sqlbuild/executor/build/concurrent/helpers.py index 09608926ef..47b3934a58 100644 --- a/tests/integration/src/sqlbuild/executor/build/concurrent/helpers.py +++ b/tests/integration/src/sqlbuild/executor/build/concurrent/helpers.py @@ -106,7 +106,9 @@ def run_concurrent_build( pipeline_result: CompilePipelineResult = run_compile_pipeline( discovered_inputs=discovered, adapter=adapter, - options=CompilePipelineOptions(no_sql_validation=True, connection_config=config), + options=CompilePipelineOptions( + no_sql_validation=True, connection_config=config, auto_load_sources=True + ), ) plan: PlanOutput = pipeline_result.plan_output diff --git a/tests/integration/src/sqlbuild/executor/build/snowflake_metadata/_test_types.py b/tests/integration/src/sqlbuild/executor/build/snowflake_metadata/_test_types.py index 3b9efaa8cb..460df0d0ad 100644 --- a/tests/integration/src/sqlbuild/executor/build/snowflake_metadata/_test_types.py +++ b/tests/integration/src/sqlbuild/executor/build/snowflake_metadata/_test_types.py @@ -31,3 +31,13 @@ class StateTableRetentionTestCase: max_retention_days: int expected_first_retention_days: str expected_later_retention_days: frozenset[str] + + +@dataclass(frozen=True) +class SourceBindingCaseTestCase: + """A select list over an existing source and the binding error planning should report.""" + + description: str + select: str + expected_error: str | None = None + expected_models: tuple[str, ...] = () diff --git a/tests/integration/src/sqlbuild/executor/build/snowflake_metadata/helpers.py b/tests/integration/src/sqlbuild/executor/build/snowflake_metadata/helpers.py index cdce00d9fa..d4484ccc70 100644 --- a/tests/integration/src/sqlbuild/executor/build/snowflake_metadata/helpers.py +++ b/tests/integration/src/sqlbuild/executor/build/snowflake_metadata/helpers.py @@ -522,6 +522,66 @@ def _show_schema_identity(match: re.Match[str]) -> _Identity: } +def write_bundles_project(*, project_dir: Path, select: str) -> None: + """Write one table model selecting ``select`` from a source without declared columns.""" + + files: dict[str, str] = { + "sqlbuild_project.toml": ( + 'name = "catalog"\nadapter = "snowflake"\n\n[connection]\n' + 'account = "example"\nuser = "example"\ndatabase = "analytics"\nschema = "marts"\n\n' + '[defaults]\ndatabase = "analytics"\nschema = "marts"\nmaterialized = "table"\n' + ), + "sources/raw.yml": ( + "sources:\n - name: raw_bundles\n description: Test source raw_bundles.\n" + " database: analytics\n schema: raw\n table: bundles\n" + ), + "models/bundle_report.sql": ( + "MODEL (description 'Test model bundle_report.');\n" + f'SELECT {select} FROM __source("raw_bundles")\n' + ), + } + relative: str + content: str + for relative, content in files.items(): + path: Path = project_dir / relative + path.parent.mkdir(parents=True, exist_ok=True) + _ = path.write_text(content, encoding="utf-8") + + +def bundles_warehouse() -> SimulatedSnowflakeWarehouse: + """Return a simulated warehouse holding the raw bundles source with upper-case columns.""" + + return SimulatedSnowflakeWarehouse( + relations=( + FakeRelation( + database="ANALYTICS", + schema="RAW", + name="BUNDLES", + columns=( + FakeColumn(name="BUNDLE_SKU", data_type="TEXT"), + FakeColumn(name="QUANTITY", data_type="NUMBER"), + ), + ), + ) + ) + + +def plan_offline_snowflake_project( + *, project_dir: Path, warehouse: SimulatedSnowflakeWarehouse +) -> CompilePipelineResult: + """Compile and plan ``project_dir`` with SQL validation against the offline warehouse.""" + + inputs: DiscoveredProjectInputs = discover_project_inputs(project_dir=project_dir) + return run_compile_pipeline( + discovered_inputs=inputs, + adapter=OfflineSnowflakeAdapter(warehouse=warehouse), + options=CompilePipelineOptions( + no_cache=True, connection_config=dict(inputs.project_config.connection) + ), + hooks=ConnectionHooks(), + ) + + def write_orders_project(*, project_dir: Path) -> None: """Write a table, a view, a merge incremental, and a three-day microbatch model.""" diff --git a/tests/integration/src/sqlbuild/executor/build/snowflake_metadata/test_source_binding_case.py b/tests/integration/src/sqlbuild/executor/build/snowflake_metadata/test_source_binding_case.py new file mode 100644 index 0000000000..134699d7f0 --- /dev/null +++ b/tests/integration/src/sqlbuild/executor/build/snowflake_metadata/test_source_binding_case.py @@ -0,0 +1,63 @@ +"""Planning binds models to listed Snowflake source columns with Snowflake's identifier case rules.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from sqlbuild._native import CompilePipelineResult +from tests.integration.src.sqlbuild.executor.build.snowflake_metadata._test_types import ( + SourceBindingCaseTestCase, +) +from tests.integration.src.sqlbuild.executor.build.snowflake_metadata.helpers import ( + bundles_warehouse, + plan_offline_snowflake_project, + write_bundles_project, +) + + +@pytest.mark.parametrize( + "test_case", + [ + SourceBindingCaseTestCase( + "unquoted", "bundle_sku, quantity", expected_models=("bundle_report",) + ), + SourceBindingCaseTestCase("uppercase", "BUNDLE_SKU", expected_models=("bundle_report",)), + SourceBindingCaseTestCase( + "quoted uppercase", '"BUNDLE_SKU"', expected_models=("bundle_report",) + ), + ], + ids=lambda case: case.description, +) +def test_given_existing_source_without_quoted_case_setting_when_planning_then_names_bind( + test_case: SourceBindingCaseTestCase, tmp_path: Path +) -> None: + write_bundles_project(project_dir=tmp_path, select=test_case.select) + planned: CompilePipelineResult = plan_offline_snowflake_project( + project_dir=tmp_path, warehouse=bundles_warehouse() + ) + assert ( + tuple(entry.name for entry in planned.plan_output.model_entries) + == test_case.expected_models + ) + + +@pytest.mark.parametrize( + "test_case", + [ + SourceBindingCaseTestCase("quoted lowercase", '"bundle_sku"', "B002"), + SourceBindingCaseTestCase("unknown", "missing_sku", "B002"), + ], + ids=lambda case: case.description, +) +def test_given_existing_source_without_quoted_case_setting_when_planning_then_rejects_unknown( + test_case: SourceBindingCaseTestCase, tmp_path: Path +) -> None: + write_bundles_project(project_dir=tmp_path, select=test_case.select) + with pytest.raises(Exception, match=test_case.expected_error): + _ = plan_offline_snowflake_project(project_dir=tmp_path, warehouse=bundles_warehouse()) + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/integration/src/sqlbuild/executor/scenario/_test_types.py b/tests/integration/src/sqlbuild/executor/scenario/_test_types.py index 534fd78c43..7c401d2b80 100644 --- a/tests/integration/src/sqlbuild/executor/scenario/_test_types.py +++ b/tests/integration/src/sqlbuild/executor/scenario/_test_types.py @@ -64,3 +64,12 @@ class ScenarioSnapshotCaptureTypesIntegrationTestCase: description: str expected_local_types: dict[str, str] expected_replay_summary_rows: tuple[tuple[object, ...], ...] + + +@dataclass(frozen=True) +class ScenarioSnapshotFingerprintIntegrationTestCase: + description: str + fixture_sql: str + helper_ctes: str + expected_fixture_renderings: int + expected_fingerprints: int diff --git a/tests/integration/src/sqlbuild/executor/scenario/helpers.py b/tests/integration/src/sqlbuild/executor/scenario/helpers.py index afe96ebcd5..9944e9cf00 100644 --- a/tests/integration/src/sqlbuild/executor/scenario/helpers.py +++ b/tests/integration/src/sqlbuild/executor/scenario/helpers.py @@ -7,7 +7,10 @@ CompiledObjectKey, CompiledRelationLocation, CompiledResourceType, + CompilePipelineOptions, + CompilePipelineResult, DiscoveredHookFunction, + DiscoveredProjectInputs, MaterializationType, ModelPlanEntry, PlanAction, @@ -20,8 +23,19 @@ ScenarioGraphPlan, ScenarioRelationMap, ScenarioRelationPlan, + discover_project_inputs, + run_compile_pipeline, ) from sqlbuild.executor.run.models import HookContext +from sqlbuild.runtime.contracts.models import ConnectionHooks +from tests.integration.src.sqlbuild.executor.build.snowflake_metadata.helpers import ( + SimulatedSnowflakeWarehouse, +) +from tests.unit.src.sqlbuild.adapters.snowflake.inspection.helpers import ( + FakeColumn, + FakeRelation, + OfflineSnowflakeAdapter, +) SCENARIO_NAME: str = "revenue__customer_refund" SCHEMA_NAME: str = "scenario_schema" @@ -317,3 +331,62 @@ def _model_entry( pre_hooks=pre_hooks, post_hooks=post_hooks, ) + + +def write_partial_fixture_snowflake_project( + *, project_dir: Path, fixture_sql: str, helper_ctes: str = "" +) -> None: + """Write a Snowflake project whose scenario mocks a contracted model with ``fixture_sql``.""" + + files: dict[str, str] = { + "sqlbuild_project.toml": ( + 'name = "shop"\nadapter = "snowflake"\n\n[connection]\naccount = "example"\n' + 'user = "example"\ndatabase = "analytics"\nschema = "marts"\n\n' + '[defaults]\ndatabase = "analytics"\nschema = "marts"\nmaterialized = "table"\n' + ), + "models/upstream.sql": ( + "MODEL (description 'Upstream.', contract enforced,\n" + " columns (id (type INT), label (type VARCHAR)));\n" + "SELECT 1 AS id, CAST('a' AS VARCHAR) AS label\n" + ), + "models/downstream.sql": ( + "MODEL (description 'Downstream.');\n" + 'SELECT u.id, u.label FROM __ref("upstream") AS u\n' + ), + "tests/scenarios/downstream_labels.sql": ( + "SCENARIO (description 'Downstream labels.');\n" + f"WITH\n{helper_ctes}__ref__upstream AS ({fixture_sql}),\n" + "__expected__downstream AS (SELECT 1 AS id, CAST(NULL AS VARCHAR) AS label)\n" + "SELECT 1\n" + ), + } + relative: str + content: str + for relative, content in files.items(): + path: Path = project_dir / relative + path.parent.mkdir(parents=True, exist_ok=True) + _ = path.write_text(content, encoding="utf-8") + + +def compile_offline_snowflake_project(*, project_dir: Path) -> CompilePipelineResult: + """Compile and plan a Snowflake project against an offline warehouse.""" + + inputs: DiscoveredProjectInputs = discover_project_inputs(project_dir=project_dir) + warehouse: SimulatedSnowflakeWarehouse = SimulatedSnowflakeWarehouse( + relations=( + FakeRelation( + database="ANALYTICS", + schema="MARTS", + name="UNRELATED", + columns=(FakeColumn(name="ID", data_type="NUMBER"),), + ), + ) + ) + return run_compile_pipeline( + discovered_inputs=inputs, + adapter=OfflineSnowflakeAdapter(warehouse=warehouse), + options=CompilePipelineOptions( + no_cache=True, connection_config=dict(inputs.project_config.connection) + ), + hooks=ConnectionHooks(), + ) diff --git a/tests/integration/src/sqlbuild/executor/scenario/test_snapshot_fingerprint.py b/tests/integration/src/sqlbuild/executor/scenario/test_snapshot_fingerprint.py new file mode 100644 index 0000000000..fb90c504ae --- /dev/null +++ b/tests/integration/src/sqlbuild/executor/scenario/test_snapshot_fingerprint.py @@ -0,0 +1,86 @@ +"""Local snapshot fingerprints do not depend on the adapter that renders fixture SQL.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from sqlbuild._native import ( + CompiledSqlScenario, + CompilePipelineResult, + ScenarioExecutionPlan, + build_cli_scenario_plan, +) +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapter.discovery.main.builtins import builtin_adapter_classes +from sqlbuild.executor.scenario._helpers.snapshots.core import ( + build_scenario_snapshot_input_fingerprint, + build_scenario_snapshot_input_specs, +) +from tests.integration.src.sqlbuild.executor.scenario._test_types import ( + ScenarioSnapshotFingerprintIntegrationTestCase, +) +from tests.integration.src.sqlbuild.executor.scenario.helpers import ( + compile_offline_snowflake_project, + write_partial_fixture_snowflake_project, +) + + +@pytest.mark.parametrize( + "test_case", + [ + ScenarioSnapshotFingerprintIntegrationTestCase( + description="partial fixture rendered by capture and local adapters", + fixture_sql="SELECT 1 AS id, NULL AS label", + helper_ctes="", + expected_fixture_renderings=2, + expected_fingerprints=1, + ), + ScenarioSnapshotFingerprintIntegrationTestCase( + description="fixture reading a helper CTE, rendered by capture and local adapters", + fixture_sql="SELECT h.id, NULL AS label FROM helper_ids AS h", + helper_ctes="helper_ids AS (SELECT 1 AS id),\n", + expected_fixture_renderings=2, + expected_fingerprints=1, + ), + ], + ids=lambda case: case.description, +) +def test_given_capture_and_local_adapters_when_fingerprinting_snapshot_then_matches( + test_case: ScenarioSnapshotFingerprintIntegrationTestCase, tmp_path: Path +) -> None: + write_partial_fixture_snowflake_project( + project_dir=tmp_path, fixture_sql=test_case.fixture_sql, helper_ctes=test_case.helper_ctes + ) + pipeline: CompilePipelineResult = compile_offline_snowflake_project(project_dir=tmp_path) + scenario: CompiledSqlScenario = pipeline.project.sql_scenarios[0] + capture_adapter: BaseAdapter = builtin_adapter_classes()["snowflake"]() + adapters: tuple[BaseAdapter, ...] = (capture_adapter, builtin_adapter_classes()["duckdb"]()) + plans: tuple[ScenarioExecutionPlan, ...] = tuple( + build_cli_scenario_plan( + scenario=scenario, + pipeline_result=pipeline, + adapter=adapter, + project_name="shop", + source_lexical_syntax=capture_adapter.sql_lexical_syntax, + ) + for adapter in adapters + ) + + renderings: set[str] = {plan.fixture_plans[0].sql for plan in plans} + fingerprints: set[str] = { + build_scenario_snapshot_input_fingerprint( + scenario_name=plan.name, + input_specs=build_scenario_snapshot_input_specs(scenario_plan=plan), + capture_adapter="snowflake", + capture_dialect="snowflake", + ) + for plan in plans + } + assert len(renderings) == test_case.expected_fixture_renderings + assert len(fingerprints) == test_case.expected_fingerprints + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/integration/src/sqlbuild/rule_engine/native_rows/test_native_rules_request.py b/tests/integration/src/sqlbuild/rule_engine/native_rows/test_native_rules_request.py index 28c6958ff5..5e6a0297a7 100644 --- a/tests/integration/src/sqlbuild/rule_engine/native_rows/test_native_rules_request.py +++ b/tests/integration/src/sqlbuild/rule_engine/native_rows/test_native_rules_request.py @@ -31,7 +31,7 @@ ( "SQBRCONTRACT105", "models/orders.sql", - 1, + 5, 18, 'output "amount" is a passthrough whose type is not proven against the declared ' "contract; declared type is BIGINT", diff --git a/tests/unit/src/sqlbuild/adapter/contract/_test_types.py b/tests/unit/src/sqlbuild/adapter/contract/_test_types.py index 62f3f3cfcd..4f6c12ef80 100644 --- a/tests/unit/src/sqlbuild/adapter/contract/_test_types.py +++ b/tests/unit/src/sqlbuild/adapter/contract/_test_types.py @@ -71,3 +71,11 @@ class AdapterSourceExpressionRenderingTestCase: expected_relation: str expected_cast_subquery: str expected_relation_cast_subquery: str + + +@dataclass(frozen=True) +class AdapterLoaderJsonRowsTestCase: + description: str + adapter_name: str + rows: tuple[dict[str, object], ...] + expected_sql: str diff --git a/tests/unit/src/sqlbuild/adapter/contract/test_loader_types.py b/tests/unit/src/sqlbuild/adapter/contract/test_loader_types.py index e013217292..df7520904e 100644 --- a/tests/unit/src/sqlbuild/adapter/contract/test_loader_types.py +++ b/tests/unit/src/sqlbuild/adapter/contract/test_loader_types.py @@ -9,6 +9,7 @@ from sqlbuild.adapter.discovery.main.builtins import builtin_adapter_classes from tests.unit.src.sqlbuild.adapter.contract._test_types import ( AdapterIdentifierRenderingTestCase, + AdapterLoaderJsonRowsTestCase, AdapterLoaderRowsEmptySelectTestCase, AdapterLoaderRowsSelectTestCase, AdapterLoaderTypeMappingTestCase, @@ -274,6 +275,52 @@ def test_given_builtin_adapter_when_rendering_loader_rows_then_returns_adapter_s assert not any(fragment in sql for fragment in test_case.forbidden_fragments) +@pytest.mark.parametrize( + "test_case", + [ + AdapterLoaderJsonRowsTestCase( + description="snowflake parses json in the select list, not inside values", + adapter_name=BuiltinAdapter.SNOWFLAKE.value, + rows=({"id": 1, "payload": {"sku": "a-1"}}, {"id": 2, "payload": None}), + expected_sql=( + 'SELECT CAST("ID" AS BIGINT) AS "ID", ' + 'CAST(PARSE_JSON("PAYLOAD") AS VARIANT) AS "PAYLOAD" ' + 'FROM (VALUES (1, \'{"sku": "a-1"}\'), (2, NULL)) ' + 'AS __loader_rows("ID", "PAYLOAD")' + ), + ), + AdapterLoaderJsonRowsTestCase( + description="snowflake keeps json escapes and backslashes in string constants", + adapter_name=BuiltinAdapter.SNOWFLAKE.value, + rows=({"id": 1, "payload": {"note": 'say "hi"\\now'}},), + expected_sql=( + 'SELECT CAST("ID" AS BIGINT) AS "ID", ' + 'CAST(PARSE_JSON("PAYLOAD") AS VARIANT) AS "PAYLOAD" ' + 'FROM (VALUES (1, \'{"note": "say \\\\"hi\\\\"\\\\\\\\now"}\')) ' + 'AS __loader_rows("ID", "PAYLOAD")' + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_json_rows_when_rendering_loader_rows_then_values_hold_only_constants( + test_case: AdapterLoaderJsonRowsTestCase, +) -> None: + adapter: BaseAdapter = builtin_adapter_classes()[test_case.adapter_name]() + + sql: str = adapter.render_loader_rows_select( + rows=test_case.rows, + column_names=("id", "payload"), + column_sql_types={ + "id": adapter.render_loader_logical_type(LoaderLogicalType.INTEGER), + "payload": adapter.render_loader_logical_type(LoaderLogicalType.JSON), + }, + inferred_types={"id": LoaderLogicalType.INTEGER, "payload": LoaderLogicalType.JSON}, + ) + + assert sql == test_case.expected_sql + + @pytest.mark.parametrize( "test_case", [ diff --git a/tests/unit/src/sqlbuild/lint/_helpers/test_native.py b/tests/unit/src/sqlbuild/lint/_helpers/test_native.py index 47364d79b1..f3c067bb19 100644 --- a/tests/unit/src/sqlbuild/lint/_helpers/test_native.py +++ b/tests/unit/src/sqlbuild/lint/_helpers/test_native.py @@ -313,7 +313,7 @@ def test_given_contents_when_formatting_then_contents_match_expected( expected_contents='MODEL (description "Canonical customer records.");\nSELECT 1\n', ), FormatDescriptionTestCase( - description="nested column description is not treated as the model description", + description="nested column description is laid out at its own depth", contents=( 'MODEL (\n columns (\n customer_id (type BIGINT, description "A deliberately ' 'long column description that remains authored text.")\n ),\n description "Builds ' @@ -321,8 +321,9 @@ def test_given_contents_when_formatting_then_contents_match_expected( ), line_width=60, expected_contents=( - 'MODEL (\n columns (\n customer_id (type BIGINT, description "A deliberately ' - 'long column description that remains authored text.")\n ),\n description """\n' + 'MODEL (\n columns (\n customer_id (type BIGINT, description """\n' + " A deliberately long column description that remains\n" + ' authored text.\n """)\n ),\n description """\n' " Builds canonical customer records from every available\n" ' source.\n """\n);\nSELECT 1\n' ), diff --git a/website/src/content/docs/docs/cli/build.mdx b/website/src/content/docs/docs/cli/build.mdx index 9826556dc6..47e5cad318 100644 --- a/website/src/content/docs/docs/cli/build.mdx +++ b/website/src/content/docs/docs/cli/build.mdx @@ -54,7 +54,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`: | `--start-cursor-int` | Override start cursor for integer incremental models | | `--end-cursor-int` | Override end cursor for integer incremental models | | `--load` | Explicitly load managed sources before building | -| `--no-load` | Skip automatic source loading | +| `--no-load` | Run no source loaders; selected managed sources are still audited | | `--reload` | Reload managed sources (passes `is_reload=True` to loaders) | | `--selection-diagnostics` | Warn when a selected model will build on changed upstream models that are not selected; see [Selection and staleness](/docs/concepts/planning/selection-and-staleness) | | `--manifest` | Generate `target/manifest.json` with plan-aware project metadata | @@ -75,9 +75,9 @@ This replaces the former `sqb run` command. The full lifecycle (tests + audits) ## Execution order -1. Managed sources are loaded (unless `--no-load`) +1. Managed sources are loaded (unless `--no-load`, which audits their existing data instead) 2. Selected seeds are loaded -3. Source audits run before their dependent models (unless `--no-audits`) +3. Source audits run before their dependent models (unless `--no-audits`); with `--no-load`, or for a source selected by name, a source no model reads is audited at the end 4. SQL unit tests run before their target model (unless `--no-tests`) 5. Models are materialized in DAG topological order 6. Error-severity audits run against the staging table before promotion to the target (unless `--no-audits`) diff --git a/website/src/content/docs/docs/cli/load.mdx b/website/src/content/docs/docs/cli/load.mdx index c04bde9282..46e2515106 100644 --- a/website/src/content/docs/docs/cli/load.mdx +++ b/website/src/content/docs/docs/cli/load.mdx @@ -92,4 +92,9 @@ sqb build --no-load sqb build --reload ``` +With loading off, through `--no-load` or `auto_load_sources = false`, no loader runs, even for a +source selected by name such as `sqb build --no-load --select raw_orders`. Managed sources in the +selection stay in the plan, and their audits check the data they already hold, at the end of the +build if no model reads them. A source outside the selection is neither loaded nor audited. + See [Loaders](/docs/concepts/python-nodes/loaders) for full documentation on write strategies, the loader context API, and source deferral. diff --git a/website/src/content/docs/docs/concepts/audits.mdx b/website/src/content/docs/docs/concepts/audits.mdx index 3e0f006fc6..06db321a40 100644 --- a/website/src/content/docs/docs/concepts/audits.mdx +++ b/website/src/content/docs/docs/concepts/audits.mdx @@ -135,7 +135,7 @@ SQLBuild includes four generic audits out of the box. You do not need to define | `not_null` | Fails if any row has a NULL value in the column | Column-level only | | `unique` | Fails if any non-NULL value appears more than once | Column-level only | | `accepted_values` | Fails if any non-NULL value is not in the allowed list | `values` - list of allowed values | -| `relationships` | Fails if any non-NULL value does not exist in the referenced column | `to` - target relation, `field` - target column | +| `relationships` | Fails if any non-NULL value does not exist in the referenced column | `to` - target relation as `__ref(...)`, `__source(...)` or `__seed(...)`, `field` - target column | ### Using built-in audits @@ -156,7 +156,7 @@ MODEL ( ), payment_method ( audits [ - relationships (to "stg_payments", field "method"), + relationships (to __ref("stg_payments"), field "method"), ], ), ), @@ -360,6 +360,11 @@ sources: If a source audit with `error` severity fails, all downstream models that depend on that source are blocked. This lets you catch data quality issues at the source before any transformations run. +A source's audits run before the first model in the build that reads it. A source no selected model +reads is audited at the end of the build only when loading is off (`--no-load` or +`auto_load_sources = false`) or when the source is selected by name, such as +`sqb build --select raw_orders`; otherwise its audits do not run. + A source audit that reads other resources waits for them before any model that depends on the source is built. It may read a model that is not built from the source; reading one that is fails with `P005`. See [Audits that read other resources](#audits-that-read-other-resources). diff --git a/website/src/content/docs/docs/concepts/macros/sql-macros.mdx b/website/src/content/docs/docs/concepts/macros/sql-macros.mdx index c87be57cd4..70cba17614 100644 --- a/website/src/content/docs/docs/concepts/macros/sql-macros.mdx +++ b/website/src/content/docs/docs/concepts/macros/sql-macros.mdx @@ -325,9 +325,15 @@ bar(LENGTH(@value), 0, 10) `returns VARCHAR(5)`, and contract checks report a mismatch with `K002`. - The type flows on as an inferred type would: through CTEs, `SELECT *`, subqueries, joins and each branch of a set operation, through operators and functions around the call, to contract checks - (`K002`, `K003`), `SQBRCONTRACT105` and models that read the column. A select-list item that is - one call to a macro with `returns` needs no `CAST` for `SQBRCONTRACT105`; a call inside a larger - expression still does. + (`K002`, `K003`), `SQBRCONTRACT105` and models that read the column. +- `returns` is used only for analysis. It doesn't appear in the generated SQL, so on its own it doesn't + change the column's physical type. A body of `'ACTIVE'` still creates `VARCHAR(6)` on Snowflake. + For `SQBRCONTRACT105`, a select-list item that is one call needs no `CAST` at the call only when + the macro body's outermost expression is a `CAST(... AS type)` or `...::type` to the column's + contract type, compared as the dialect does, such as `CAST(LEFT(@value, 3) AS VARCHAR)` for a + `VARCHAR` column. Otherwise the rule asks for a cast in the macro or at the call, whether the + macro declares a fixed `returns` or `returns TYPE OF`. A call inside a larger expression always + needs an outer cast. - Without `returns`, the call is typed by inference alone. When an output is one call whose type is unknown, `K003` and `SQBRCONTRACT105` suggest declaring `returns` on the macro. - `returns TYPE OF ` declares the type of that argument at each call, under the same @@ -339,8 +345,9 @@ bar(LENGTH(@value), 0, 10) `returns BIGINT`, fails with `P024`, as does a partly unknown body of another shape, such as `[]` with `returns BIGINT`. A body whose type cannot be inferred, for example because it reads `ANY` arguments or calls a function whose type is unknown or only guessed, trusts `returns`. -- `returns` must be a type the adapter's SQL dialect accepts, including a decimal precision and - scale it can hold, such as at most `DECIMAL(38, s)` on DuckDB (`P015`). +- `returns` and every argument type must be a type the adapter's SQL dialect accepts, including a + decimal precision and scale it can hold, such as at most `DECIMAL(38, s)` on DuckDB or + `NUMERIC(38, 9)` on BigQuery (`P015`). - A call where only a keyword can stand, such as the date part of `EXTRACT`, is typed by inference alone. - A call inside a Python macro argument, as in `@py.bracket(@sql.status_bar(o.status))`, is typed diff --git a/website/src/content/docs/docs/concepts/python-nodes/loaders.mdx b/website/src/content/docs/docs/concepts/python-nodes/loaders.mdx index 8f8da8d450..6c538061ad 100644 --- a/website/src/content/docs/docs/concepts/python-nodes/loaders.mdx +++ b/website/src/content/docs/docs/concepts/python-nodes/loaders.mdx @@ -152,6 +152,15 @@ sources: Existing rows matching the unique key are updated; new rows are inserted. +### String values on Snowflake + +Snowflake reads a backslash inside a string literal as an escape, so SQLBuild doubles every +backslash in the values a loader returns. A value such as `C:\orders\2026` is stored exactly as +returned. Earlier releases wrote those backslashes unescaped, so Snowflake dropped them or read them +as escapes. If a `merge` or `delete_insert` unique key, or a value an `append` load is +deduplicated by, contains a backslash, rows loaded by an earlier release will not match newly +loaded rows. Rewrite or delete those rows before relying on the key. + ## Self-managed loaders If a loader returns `None`, SQLBuild skips its row-writing pipeline. The loader is responsible for writing data to the target itself, using whatever approach makes sense - `ctx.execute_sql()`, an external library, a subprocess, or anything else: @@ -310,6 +319,11 @@ sqb build --no-load sqb build --reload ``` +With loading off, through `--no-load` or `auto_load_sources = false`, no loader runs, even for a +source selected by name such as `sqb build --no-load --select raw_orders`. Managed sources in the +selection stay in the plan, and their audits check the data they already hold, at the end of the +build if no model reads them. A source outside the selection is neither loaded nor audited. + When `--reload` is passed, `ctx.is_reload` is `True` in the loader function. This lets loaders implement different behavior for full reloads versus normal incremental loads. ## Source deferral diff --git a/website/src/content/docs/docs/concepts/sources.mdx b/website/src/content/docs/docs/concepts/sources.mdx index cca96b7b90..bc0c640b8e 100644 --- a/website/src/content/docs/docs/concepts/sources.mdx +++ b/website/src/content/docs/docs/concepts/sources.mdx @@ -62,6 +62,11 @@ sources: If a source audit with `error` severity fails, all downstream models that depend on that source are blocked. +A source's audits run before the first model in the build that reads it. A source no selected model +reads is audited at the end of the build only when loading is off (`--no-load` or +`auto_load_sources = false`) or when the source is selected by name, such as +`sqb build --select raw_orders`; otherwise its audits do not run. + A source audit may also read other resources; SQLBuild builds them first. See [Audits that read other resources](/docs/concepts/audits#audits-that-read-other-resources). diff --git a/website/src/content/docs/docs/concepts/testing.mdx b/website/src/content/docs/docs/concepts/testing.mdx index b51229cdf8..cf142a25c7 100644 --- a/website/src/content/docs/docs/concepts/testing.mdx +++ b/website/src/content/docs/docs/concepts/testing.mdx @@ -428,7 +428,7 @@ replaces every `@sql.net_amount(...)` call written in the model SQL with the moc inside another SQL macro's body is expanded as usual. Macro names are unique across both kinds, so the mock name never needs the `sql.` or `py.` prefix. -The mock value must be a single `SELECT` with one string literal. Use doubled single quotes for quotes within the value (standard SQL escaping). +The mock value must be a single `SELECT` with one string literal, which may be named with `AS`, such as `SELECT 'country_code = ''US''' AS country_filter`. Use doubled single quotes for quotes within the value (standard SQL escaping). This is useful for: - Testing models that use target-specific macros without depending on target config