diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md index 16adc5a1ee..964152ccd2 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md @@ -26,6 +26,7 @@ sqb --project-dir audit [flags] | `--concurrency` | Maximum number of audits to run concurrently (default: `1`) | | `--json` | Print structured audit results | | `--json-output` | Write structured audit results to a file | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ## Concurrency diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md index e8beb16eaf..635b22ac6b 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md @@ -54,6 +54,7 @@ sqb --project-dir build [flags] | `--manifest` | Generate `target/manifest.json` with plan-aware project metadata | | `--select`, `-s` | Select specific models | | `--exclude` | Exclude specific models | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ## Fast iteration diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md index 8fef3284f9..befd47345e 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md @@ -24,6 +24,7 @@ sqb --project-dir check [flags] | `--json` | Print check results as JSON | | `--json-output` | Write check results JSON to a file path | | `--vars` | Override project variables | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | Selecting a non-check node is rejected; use [`sqb build`](build.md) to run tasks and assets. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md index d51a437337..1d2a61c9f6 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md @@ -36,6 +36,7 @@ sqb --project-dir clone --from [--to ] [flags] | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | | `--select`, `-s` | Select specific models to clone | | `--exclude` | Exclude specific models from cloning | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md index b8d9463b8c..ecb0141d99 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md @@ -37,6 +37,7 @@ replace conflicts rather than preserving authored declarations. | `--vars` | both | Override project variables | | `--write` | generate | Apply generated repository changes | | `--overwrite` | generate | Replace conflicting declarations; requires `--write` | +| `--warehouse ` | both | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | See [Contracts](../concepts/models/contracts.md#adopting-an-existing-schema) for ownership and safety guarantees. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md index e645276c60..fa539c7253 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md @@ -91,3 +91,8 @@ target_path = "../dbt_project/target" ``` See [Project Configuration](../concepts/project-configuration.md#dbt) for all fields. + +SQLBuild-owned work in `sqb dbt plan`, `run` and `build` connects with the target's `build` +[warehouse group](../concepts/project-configuration.md#command-group-warehouses). dbt itself +keeps using the warehouse from its profile. These commands do not accept `--warehouse`, because +their remaining arguments pass through to dbt. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md index f53dbe7dde..1cbd7b53a0 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md @@ -20,9 +20,9 @@ The command runs three groups of checks: **Runtime** - SQLBuild version, Python version, Python path, OS info. -**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter and active target. +**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter and active target. For Snowflake it also shows the warehouse each [command group](../concepts/project-configuration.md#command-group-warehouses) connects with and where that value came from. -**Connection** - Displays connection settings (secrets are masked), attempts to connect to the warehouse, and runs `SELECT 1` to verify query execution. +**Connection** - Displays connection settings as `sqb debug` itself uses them, with the `query` warehouse group applied (secrets are masked), attempts to connect to the warehouse, and runs `SELECT 1` to verify query execution. ## Flags @@ -30,6 +30,7 @@ The command runs three groups of checks: |------|-------------| | `--no-connection` | Skip the connection and query tests. Useful for validating config without warehouse access. | | `--json` | Output results as JSON instead of formatted text. | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ## Example @@ -52,6 +53,8 @@ Configuration: project: waffle_shop [OK loaded] adapter: snowflake [OK found] target: dev [OK resolved] + build warehouse: BUILD_WH [OK sqlbuild_project.toml [targets.dev.warehouses] build] + query warehouse: SQB_WH [OK connection warehouse] Connection: account: FJMQFQV-OJ66172 diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md index 9f03bdc4a1..c340299bd1 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md @@ -60,6 +60,7 @@ In query mode, omit `FROM:TO` and model selectors; each query identifies its own | `--json` | Print one JSON document to stdout | | `--select`, `-s` | Select specific models to diff (required in model mode) | | `--exclude` | Exclude specific models from diffing | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md index 25ce7ecc98..b46c677134 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md @@ -37,6 +37,7 @@ sqb --project-dir freshness [flags] | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | | `--select`, `-s` | Select specific sources or models (sources upstream of selected models are included) | | `--exclude` | Exclude specific sources or models | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ## Source selection diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md index 17a1d2c630..d7fdcca9ed 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md @@ -40,6 +40,7 @@ sqb --project-dir janitor [flags] | `--retention-days` | Override the configured retention period (days) before a stale relation is archived | | `--direct-state-history-versions` | Override how many state-history versions are kept per identity | | `--drop-old-name-view ` | Drop the [compatibility view](../concepts/models/migrations.md#old-names) at a renamed model's old name before it expires. Repeat for several views. Fails with `C503` if no compatibility view is recorded at the name | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ## Targets diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md index 28cd952723..770802380a 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md @@ -29,6 +29,7 @@ sqb --project-dir load [flags] | `--json` | Output results as JSON | | `--json-output` | Write JSON results to a file path | | `--var` | Set project variables (`--var key=value`) | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md index 8c9867e347..0e2c4cf31a 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md @@ -39,6 +39,7 @@ sqb --project-dir plan [flags] | `--end-cursor-int` | Override end cursor for integer incremental models | | `--select`, `-s` | Select specific models | | `--exclude` | Exclude specific models | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ## Output diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md index 1d9110dc78..ce42aee3e7 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md @@ -31,6 +31,7 @@ File paths are resolved from the current working directory. | `--format` | Output format: `long` (default), `table`, `json`, or `csv` | | `--limit` | Maximum rows to return (default: 20) | | `--no-limit` | Disable the row limit | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ## Output formats diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md index 401f7e9aef..457750e029 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md @@ -40,6 +40,7 @@ sqb scenario test [flags] | `--max-snapshot-bytes` | Override per-relation byte limit for capture | | `--max-snapshot-total-bytes` | Override total byte limit for capture | | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ### Selectors @@ -130,6 +131,7 @@ sqb scenario capture [flags] | `--max-snapshot-bytes` | Override per-relation byte limit | | `--max-snapshot-total-bytes` | Override total byte limit | | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ### Examples diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md index 43947809f0..b06457fd5a 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md @@ -20,6 +20,7 @@ sqb --project-dir seed [flags] |------|-------------| | `--select`, `-s` | Select specific seeds by name | | `--exclude` | Exclude specific seeds | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md index 23b6ca4bca..5e4d1839cc 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md @@ -22,6 +22,7 @@ sqb --project-dir test [flags] | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | | `--select`, `-s` | Select tests targeting specific models | | `--exclude` | Exclude tests targeting specific models | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | Parameterized cases use parent-level selection. Selecting a target model includes every case in each matching `TEST` template; case names are not model selectors. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md index 9c7f1f0bd0..3bea08f0ab 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md @@ -69,6 +69,12 @@ class DuckDbPlusAdapter(DuckDbAdapter): adapter = "duckdb_plus" ``` +A subclass inherits the parent's capabilities. For example, an adapter extending the Snowflake +adapter keeps `supports_session_warehouse = True`, so it can use +[command-group warehouses](project-configuration.md#command-group-warehouses) and +`--warehouse`. An adapter written from scratch that sets this attribute receives the selected +warehouse as the connection's `warehouse` key. + ### Writing an adapter from scratch For a database engine with no built-in support, subclass `BaseAdapter`. It provides ANSI SQL defaults for most methods - you only need to implement `connect`, `execute`, `close`, and any methods where your engine differs from standard SQL: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md index 4789033457..ebc4156195 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md @@ -50,6 +50,11 @@ On connect, SQLBuild applies the connection's role and warehouse, then applies t target's authoritative database and schema. These ensure the session context is set correctly regardless of the user's default settings. +The warehouse can differ by command. A target's `[targets..warehouses]` section selects +one warehouse for build-style commands and another for ad-hoc commands such as `sqb query` and +`sqb diff`, and `--warehouse` overrides both for one invocation. See +[command-group warehouses](../project-configuration.md#command-group-warehouses). + ## Planning metadata Planning reads relation and column metadata with `SHOW TABLES`, `SHOW VIEWS`, and `SHOW COLUMNS` diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md index 044ebcf639..722b2e4f1a 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md @@ -153,6 +153,7 @@ A target is a named build context - the database and schema you build into, plus | `defer_sources_to` | Target name to read managed source data from (see [Loaders](python-nodes/loaders.md#source-deferral)) | | `clone` | Clone policy (see below) | | `missing_migration_origin` | What a build does when a declared `migrate_from` origin does not exist in this target: `allow` (default), `require_confirmation`, or `deny` (see below) | +| `warehouses` | Default Snowflake warehouse per command group, `build` and `query` (see below) | ```toml [connections.warehouse] @@ -218,6 +219,76 @@ password = "${ENV:SNOWFLAKE_PASSWORD}" Commands then use `dev` automatically. An explicit command such as `sqb build --target prod` still takes precedence for that invocation. +### Command-group warehouses + +Builds and ad-hoc inspection usually need different compute. A Snowflake target can choose a +default warehouse for each command group, so `sqb build` runs on a large warehouse while +`sqb query` and `sqb diff` stay on a small one: + +```toml +[connections.warehouse] +account = "my_org-my_account" +warehouse = "ANALYTICS_WH" # fallback for every command, as before + +[targets.dev] +connection = "warehouse" +database = "ANALYTICS" +schema = "DEV" + +[targets.dev.warehouses] +build = "BUILD_WH" # commands in the build group +query = "ADHOC_WH" # commands in the query group +``` + +Each command connects with the first warehouse set in this order: + +1. `--warehouse ` on the command line, for that invocation only +2. The target's group warehouse, `[targets..warehouses] build` or `query` +3. The connection's `warehouse` +4. The Snowflake user's default warehouse + +Both groups are optional. A target without `warehouses` behaves exactly as before. A group +warehouse takes precedence over the connection `warehouse` in every file. In particular, a +shared group overrides a connection `warehouse` you set in `sqlbuild_local.toml`; to change a +group locally, set the group itself, or pass `--warehouse` for one invocation. Group values +accept the same `${ENV:...}` and project-variable (`${my_var}`) templates as connection values. +Group values and `--warehouse` must be Snowflake identifiers: unquoted (`ADHOC_WH`) or +double-quoted (`"Adhoc WH"`). + +Every connection a command opens uses its group, so one invocation never mixes warehouses: +`sqb build` plans, inspects, and executes on the `build` warehouse, and `sqb diff` creates its +temporary comparison tables on the `query` warehouse. The exception is `sqb dbt`: it applies the +`build` group to SQLBuild-owned work, while dbt nodes keep the warehouse from their dbt profile. + +| Group | Commands | +|-------|----------| +| `build` | `audit`, `build`, `check`, `clone`, `dbt` (SQLBuild-owned work), `janitor`, `load`, `plan`, `scenario`, `seed`, `test` | +| `query` | `contract`, `cost`, `debug`, `diff`, `freshness`, `query` | + +Other commands, such as `compile`, `lineage`, and `format`, never connect to a warehouse. +Integrations that run builds, such as Dagster, invoke `sqb build` and use the `build` group. + +Each developer can point ad-hoc commands at their own warehouse without editing shared +configuration: + +```toml +# sqlbuild_local.toml +[targets.dev.warehouses] +query = "ALICE_XS_WH" +``` + +Local group values override the project value for the same group only; the other group keeps +its project value. `sqb debug` shows the resolved warehouse for each group and where it came +from, and `sqb build --verbose` shows it in the execution header. + +Groups need an adapter with a session warehouse: the Snowflake adapter, or a custom adapter +that extends it. `warehouses` in `sqlbuild_project.toml` is checked against the project's +adapter, and `warehouses` in `sqlbuild_local.toml` against the adapter you actually run. Setting +them for an adapter without a session warehouse, or passing `--warehouse`, is an error rather +than a silently ignored setting. The one exception is a local `adapter` override, for example a +developer running a Snowflake project on DuckDB: shared groups then don't apply, and +`sqb debug` reports them as `not used (adapter duckdb)`. + ### Missing migration origins A `migrate_from` declaration, on a model or a column, names a relation or column that exists in the targets where the rename is pending. A target that never built the old name, such as a fresh development schema, has nothing to migrate. `missing_migration_origin` decides what happens there: diff --git a/src/sqlbuild/adapter/contract/classes/base_adapter.py b/src/sqlbuild/adapter/contract/classes/base_adapter.py index 29fd64a56e..b9ea630cf2 100644 --- a/src/sqlbuild/adapter/contract/classes/base_adapter.py +++ b/src/sqlbuild/adapter/contract/classes/base_adapter.py @@ -112,6 +112,7 @@ class BaseAdapter(RetentionAdapterMixin, StrictAdapter): allows_implicit_managed_write_schema: ClassVar[bool] = False execution_duration_limit_seconds: ClassVar[int | None] = None metadata_inspection_concurrency: ClassVar[int] = 1 + supports_session_warehouse: ClassVar[bool] = False _snapshot_sql_dialect: ClassVar[SnapshotSqlDialect] = SnapshotSqlDialect( timestamp_type="TIMESTAMP", distinct_condition=render_is_distinct_from, diff --git a/src/sqlbuild/adapter/discovery/main/session_warehouse_support.py b/src/sqlbuild/adapter/discovery/main/session_warehouse_support.py new file mode 100644 index 0000000000..44465ae76c --- /dev/null +++ b/src/sqlbuild/adapter/discovery/main/session_warehouse_support.py @@ -0,0 +1,15 @@ +"""Public check for whether a project's adapter selects a session warehouse.""" + +from __future__ import annotations + +from pathlib import Path + +from sqlbuild.adapter.discovery.main.resolve_adapter import resolve_adapter + + +def adapter_supports_session_warehouse(*, adapter_name: str, project_dir: Path | None) -> bool: + """Return whether the adapter, resolved like any project adapter, has a session warehouse.""" + + return resolve_adapter( + adapter_name=adapter_name, project_dir=project_dir + ).supports_session_warehouse diff --git a/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py b/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py index 9a24d754e3..1afd84cb3a 100644 --- a/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py +++ b/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py @@ -180,6 +180,7 @@ class SnowflakeAdapter(MicrobatchMixin, UnkeyedDiffMixin, BaseAdapter): max_identifier_length: ClassVar[int] = 255 state_tables_transient: ClassVar[bool] = True + supports_session_warehouse: ClassVar[bool] = True connection_routing_keys: ClassVar[frozenset[str]] = frozenset( {"source", "profile", "target", "project_dir", "profiles_dir"} ) diff --git a/src/sqlbuild/cli/commands/_helpers/build_execution/run_context.py b/src/sqlbuild/cli/commands/_helpers/build_execution/run_context.py index c2062b0a6f..77a4f2975c 100644 --- a/src/sqlbuild/cli/commands/_helpers/build_execution/run_context.py +++ b/src/sqlbuild/cli/commands/_helpers/build_execution/run_context.py @@ -6,12 +6,13 @@ from pathlib import Path from typing import TextIO +from sqlbuild.cli.commands._helpers.runtime.warehouses import describe_connection_warehouse from sqlbuild.cli.commands.constants import ( C0_CONTROL_CODE_LIMIT, C1_CONTROL_CODE_LIMIT, C1_CONTROL_CODE_START, ) -from sqlbuild.cli.commands.models import BuildRunContext +from sqlbuild.cli.commands.models import BuildRunContext, ResolvedWarehouse from sqlbuild.presentation.classes.cli_document import CliDocument from sqlbuild.presentation.classes.cli_style import CliStyle @@ -54,7 +55,7 @@ def _format_build_run_context(*, context: BuildRunContext, use_color: bool) -> s ("target", _display_value(context.project.effective_target_name)), ("database", _display_value(context.project.effective_target_database)), *placement_rows, - ("warehouse", _display_value(context.connection_config.get("warehouse"))), + ("warehouse", _warehouse_display(context)), ("concurrency", f"{context.concurrency} configured limit"), ("full_refresh", str(context.full_refresh).lower()), ("selected", f"{selected_count:,} of {total_count:,} build resources"), @@ -81,6 +82,17 @@ def _format_build_run_context(*, context: BuildRunContext, use_color: bool) -> s return document.render() +def _warehouse_display(context: BuildRunContext) -> str: + resolved: ResolvedWarehouse = describe_connection_warehouse( + discovered_inputs=context.discovered_inputs, + selected_target=context.project.effective_target_name, + connection_config=context.connection_config, + ) + if resolved.warehouse is None: + return _display_value(None) + return f"{_display_value(resolved.warehouse)} ({resolved.source})" + + def _display_value(value: object) -> str: display_value: str | None = _safe_display_value(value) return display_value if display_value is not None else "not set" diff --git a/src/sqlbuild/cli/commands/_helpers/debug/checks.py b/src/sqlbuild/cli/commands/_helpers/debug/checks.py index f0d5bdc70d..fe78f4c85e 100644 --- a/src/sqlbuild/cli/commands/_helpers/debug/checks.py +++ b/src/sqlbuild/cli/commands/_helpers/debug/checks.py @@ -12,8 +12,23 @@ from sqlbuild.cli.commands._helpers.runtime.connection import ( resolve_project_connection_config, ) -from sqlbuild.cli.commands.models import DebugLine, DebugResult +from sqlbuild.cli.commands._helpers.runtime.warehouses import ( + apply_command_warehouse, + command_warehouse_scope, + current_command_warehouse_scope, + find_target_group_warehouse, + resolve_command_warehouse, +) +from sqlbuild.cli.commands.exceptions import CliUserError +from sqlbuild.cli.commands.models import ( + AuthoredTargetWarehouse, + CommandWarehouseScope, + DebugLine, + DebugResult, + ResolvedWarehouse, +) from sqlbuild.cli.commands.types import DebugCheckStatus +from sqlbuild.compiler.compile.exceptions import CompileInputError from sqlbuild.compiler.compile.main.effective_target_namespace import ( build_effective_target_namespace, ) @@ -29,6 +44,7 @@ resolve_effective_adapter_name, ) from sqlbuild.spec.contracts.models import TargetConfig +from sqlbuild.spec.contracts.types import WarehouseGroup _SECRET_CONNECTION_KEYS: frozenset[str] = frozenset( { @@ -60,11 +76,24 @@ def build_debug_result( local_config=discovered_inputs.local_config, ) adapter: BaseAdapter = resolve_adapter(adapter_name=adapter_name, project_dir=project_dir) - connection_config: dict[str, object] = resolve_project_connection_config( - discovered_inputs=discovered_inputs, - project_dir=project_dir, - selected_target=selected_target, - ) + command_scope: CommandWarehouseScope = current_command_warehouse_scope() + with command_warehouse_scope(command=None, cli_warehouse=None): + base_connection_config: dict[str, object] = resolve_project_connection_config( + discovered_inputs=discovered_inputs, + project_dir=project_dir, + selected_target=selected_target, + ) + connection_config: dict[str, object] + try: + connection_config = apply_command_warehouse( + config=base_connection_config, + adapter_name=adapter_name, + project_dir=project_dir, + discovered_inputs=discovered_inputs, + selected_target=selected_target, + ) + except (CompileInputError, CliUserError): + connection_config = base_connection_config target_name: str | None target_config: TargetConfig | None target_database: str | None @@ -131,6 +160,13 @@ def build_debug_result( status=DebugCheckStatus.OK, status_message="resolved", ), + *_build_warehouse_lines( + adapter=adapter, + discovered_inputs=discovered_inputs, + selected_target=selected_target, + command_scope=command_scope, + base_connection_config=base_connection_config, + ), ] connection: list[DebugLine] = _build_connection_config_lines(connection_config) connection = _append_connection_checks( @@ -147,6 +183,75 @@ def build_debug_result( ) +def _build_warehouse_lines( + *, + adapter: BaseAdapter, + discovered_inputs: DiscoveredProjectInputs, + selected_target: str | None, + command_scope: CommandWarehouseScope, + base_connection_config: dict[str, object], +) -> list[DebugLine]: + lines: list[DebugLine] = [] + group: WarehouseGroup + for group in WarehouseGroup: + authored: AuthoredTargetWarehouse | None = find_target_group_warehouse( + discovered_inputs=discovered_inputs, selected_target=selected_target, group=group + ) + if not adapter.supports_session_warehouse: + if authored is not None: + lines.append( + DebugLine( + label=f"{group.value} warehouse", + message=f"not used (adapter {adapter.adapter_name})", + status=DebugCheckStatus.SKIP, + status_message=authored.source, + ) + ) + continue + lines.append( + _build_warehouse_line( + group=group, + discovered_inputs=discovered_inputs, + selected_target=selected_target, + command_scope=command_scope, + base_connection_config=base_connection_config, + ) + ) + return lines + + +def _build_warehouse_line( + *, + group: WarehouseGroup, + discovered_inputs: DiscoveredProjectInputs, + selected_target: str | None, + command_scope: CommandWarehouseScope, + base_connection_config: dict[str, object], +) -> DebugLine: + try: + resolved: ResolvedWarehouse = resolve_command_warehouse( + discovered_inputs=discovered_inputs, + selected_target=selected_target, + group=group, + cli_warehouse=command_scope.cli_warehouse, + connection_config=base_connection_config, + cli_vars=command_scope.cli_vars, + ) + except (CompileInputError, CliUserError) as error: + return DebugLine( + label=f"{group.value} warehouse", + message=f"error: {error.message if isinstance(error, CliUserError) else error}", + status=DebugCheckStatus.ERROR, + status_message="unresolved", + ) + return DebugLine( + label=f"{group.value} warehouse", + message=resolved.warehouse or "not set", + status=DebugCheckStatus.OK, + status_message=resolved.source, + ) + + def _resolve_existing_path( *, project_dir: Path, preferred_filename: str, legacy_filename: str ) -> Path: diff --git a/src/sqlbuild/cli/commands/_helpers/entry/dispatch.py b/src/sqlbuild/cli/commands/_helpers/entry/dispatch.py index 74ccc3cf23..2ac93f75b8 100644 --- a/src/sqlbuild/cli/commands/_helpers/entry/dispatch.py +++ b/src/sqlbuild/cli/commands/_helpers/entry/dispatch.py @@ -7,6 +7,10 @@ from sqlbuild.cli.commands._helpers.diff.validation import parse_diff_name_range from sqlbuild.cli.commands._helpers.entry.parsing import read_selector_file_inputs from sqlbuild.cli.commands._helpers.lint.positional_paths import format_path_selectors +from sqlbuild.cli.commands._helpers.runtime.warehouses import ( + command_warehouse_scope, + validate_cli_warehouse, +) from sqlbuild.cli.commands.classes.cli_namespace import CliNamespace from sqlbuild.cli.commands.constants import ( DBT_INIT_COMMAND, @@ -33,6 +37,17 @@ def dispatch_cli_command(*, args: CliNamespace, handlers: CliEntrypointHandlers) -> int: """Route a parsed CLI namespace to its command handler and return its exit code.""" + validate_cli_warehouse( + cli_warehouse=args.warehouse, + project_dir=Path.cwd() if args.project_dir is None else Path(args.project_dir).resolve(), + ) + with command_warehouse_scope( + command=args.command, cli_warehouse=args.warehouse, cli_vars=args.vars + ): + return _dispatch_cli_command(args=args, handlers=handlers) + + +def _dispatch_cli_command(*, args: CliNamespace, handlers: CliEntrypointHandlers) -> int: project_dir: Path | None = ( None if args.project_dir is None else Path(args.project_dir).resolve() ) diff --git a/src/sqlbuild/cli/commands/_helpers/entry/parser.py b/src/sqlbuild/cli/commands/_helpers/entry/parser.py index 0440bdde45..4ea29c6640 100644 --- a/src/sqlbuild/cli/commands/_helpers/entry/parser.py +++ b/src/sqlbuild/cli/commands/_helpers/entry/parser.py @@ -5,6 +5,7 @@ import argparse from importlib.metadata import PackageNotFoundError, version from pathlib import Path +from typing import cast from sqlbuild.cli.commands._helpers.entry.errors import build_argument_parser_class from sqlbuild.cli.commands._helpers.entry.parser_arguments import ( @@ -20,6 +21,7 @@ from sqlbuild.cli.commands.classes.sqlbuild_argument_parser import SqlbuildArgumentParser from sqlbuild.cli.commands.constants import ( COLUMN_LINEAGE_MODE_VALUES, + COMMAND_WAREHOUSE_GROUPS, COMPILE_LINEAGE_MODE_VALUES, PLAYGROUND_TEMPLATE_VALUES, ) @@ -74,9 +76,46 @@ def build_cli_parser(*, use_color: bool = False) -> argparse.ArgumentParser: _add_dbt_parsers(subparsers) _add_skills_parsers(subparsers) _add_rules_parser(subparsers) + _add_warehouse_args(subparsers) return parser +def _add_warehouse_args( + subparsers: argparse._SubParsersAction[argparse.ArgumentParser], +) -> None: + command_name: str + command_parser: argparse.ArgumentParser + for command_name, command_parser in subparsers.choices.items(): + if command_name == CliCommand.DBT: + continue + if COMMAND_WAREHOUSE_GROUPS.get(CliCommand(command_name)) is None: + continue + _add_warehouse_arg_to_leaves(command_parser) + + +def _add_warehouse_arg_to_leaves(parser: argparse.ArgumentParser) -> None: + nested: list[argparse._SubParsersAction[argparse.ArgumentParser]] = [ + cast("argparse._SubParsersAction[argparse.ArgumentParser]", action) + for action in parser._actions + if isinstance(action, argparse._SubParsersAction) + ] + if not nested: + parser.add_argument( + "--warehouse", + default=None, + help=( + "warehouse for this invocation; overrides the target's command-group " + "warehouse and the connection warehouse" + ), + ) + return + action: argparse._SubParsersAction[argparse.ArgumentParser] + for action in nested: + child: argparse.ArgumentParser + for child in action.choices.values(): + _add_warehouse_arg_to_leaves(child) + + def _add_sql_analysis_override(parser: argparse.ArgumentParser) -> None: parser.add_argument( "--no-sql-analysis", diff --git a/src/sqlbuild/cli/commands/_helpers/runtime/connection.py b/src/sqlbuild/cli/commands/_helpers/runtime/connection.py index 8a8a9e20f0..8735832dac 100644 --- a/src/sqlbuild/cli/commands/_helpers/runtime/connection.py +++ b/src/sqlbuild/cli/commands/_helpers/runtime/connection.py @@ -6,6 +6,7 @@ from pathlib import Path from sqlbuild.adapter.contract.types import BuiltinAdapter +from sqlbuild.cli.commands._helpers.runtime.warehouses import apply_command_warehouse from sqlbuild.compiler.compile.main.effective_config import build_effective_connection_config from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.integrations.dbt.exceptions import DbtProfileError @@ -36,8 +37,9 @@ def resolve_connection_config( adapter_name: str, discovered_inputs: DiscoveredProjectInputs | None = None, cli_vars: dict[str, object] | None = None, + selected_target: str | None = None, ) -> dict[str, object]: - """Resolve relative database paths in connection config against the project directory.""" + """Resolve dbt profiles, relative DuckDB paths, and the command group's warehouse.""" config: dict[str, object] = dict(raw_config) if discovered_inputs is not None: @@ -70,7 +72,13 @@ def resolve_connection_config( and database != _DUCKDB_MEMORY_DATABASE ): config[_DATABASE_KEY] = str(project_dir / database) - return config + return apply_command_warehouse( + config=config, + adapter_name=adapter_name, + project_dir=project_dir, + discovered_inputs=discovered_inputs, + selected_target=selected_target, + ) def _warn_if_duckdb_has_snowflake_like_keys( @@ -114,6 +122,7 @@ def resolve_project_connection_config( ), discovered_inputs=discovered_inputs, cli_vars=cli_vars, + selected_target=selected_target, ) @@ -139,4 +148,5 @@ def resolve_target_connection_config( ), discovered_inputs=discovered_inputs, cli_vars=cli_vars, + selected_target=target_name, ) diff --git a/src/sqlbuild/cli/commands/_helpers/runtime/warehouses.py b/src/sqlbuild/cli/commands/_helpers/runtime/warehouses.py new file mode 100644 index 0000000000..174f146c58 --- /dev/null +++ b/src/sqlbuild/cli/commands/_helpers/runtime/warehouses.py @@ -0,0 +1,327 @@ +"""Command-group warehouse selection for CLI warehouse connections.""" + +from __future__ import annotations + +import re +from collections.abc import Iterator +from contextlib import contextmanager +from contextvars import ContextVar, Token +from pathlib import Path + +from sqlbuild.adapter.discovery.main.session_warehouse_support import ( + adapter_supports_session_warehouse, +) +from sqlbuild.cli.commands.constants import ( + CLI_WAREHOUSE_SOURCE, + COMMAND_WAREHOUSE_GROUPS, + CONNECTION_DEFAULT_WAREHOUSE_SOURCE, + CONNECTION_WAREHOUSE_SOURCE, +) +from sqlbuild.cli.commands.exceptions import CliUserError +from sqlbuild.cli.commands.models import ( + AuthoredTargetWarehouse, + CommandWarehouseScope, + ResolvedWarehouse, +) +from sqlbuild.cli.commands.types import CliCommand +from sqlbuild.compiler.compile.exceptions import CompileInputError +from sqlbuild.compiler.compile.main.effective_runtime import build_effective_runtime_config +from sqlbuild.compiler.compile.main.expand_template_data import expand_template_data +from sqlbuild.compiler.discovery.constants import LOCAL_CONFIG_FILENAME, PROJECT_CONFIG_FILENAME +from sqlbuild.compiler.discovery.main.discover_configuration import ( + discover_project_configuration, +) +from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.errors.setting_help.main.setting_help import setting_help +from sqlbuild.spec.contracts.constants import CONNECTION_WAREHOUSE_KEY +from sqlbuild.spec.contracts.main.resolve_effective_adapter_name import ( + resolve_effective_adapter_name, +) +from sqlbuild.spec.contracts.main.resolve_target_name import resolve_target_name +from sqlbuild.spec.contracts.models import LocalTargetConfig, TargetConfig +from sqlbuild.spec.contracts.types import WarehouseGroup + +_SNOWFLAKE_IDENTIFIER_PATTERN: re.Pattern[str] = re.compile( + r'[A-Za-z_][A-Za-z0-9_$]*|"(?:[^"]|"")+"' +) +_CONNECTION_SOURCES: frozenset[str] = frozenset( + {CONNECTION_WAREHOUSE_SOURCE, CONNECTION_DEFAULT_WAREHOUSE_SOURCE} +) +_CURRENT_SCOPE: ContextVar[CommandWarehouseScope | None] = ContextVar( + "sqlbuild_command_warehouse_scope", default=None +) + + +@contextmanager +def command_warehouse_scope( + *, + command: CliCommand | str | None, + cli_warehouse: str | None, + cli_vars: dict[str, object] | None = None, +) -> Iterator[CommandWarehouseScope]: + """Install the warehouse group, CLI override, and CLI vars for one command invocation.""" + + scope: CommandWarehouseScope = CommandWarehouseScope( + group=None if command is None else COMMAND_WAREHOUSE_GROUPS.get(CliCommand(command)), + cli_warehouse=cli_warehouse, + cli_vars={} if cli_vars is None else dict(cli_vars), + ) + token: Token[CommandWarehouseScope | None] = _CURRENT_SCOPE.set(scope) + try: + yield scope + finally: + _CURRENT_SCOPE.reset(token) + + +def current_command_warehouse_scope() -> CommandWarehouseScope: + """Return the active command's warehouse scope, or an empty one outside the CLI.""" + + scope: CommandWarehouseScope | None = _CURRENT_SCOPE.get() + return scope if scope is not None else CommandWarehouseScope(group=None) + + +def validate_cli_warehouse(*, cli_warehouse: str | None, project_dir: Path) -> None: + """Reject an empty, malformed, or unsupported `--warehouse` before the command runs.""" + + if cli_warehouse is None: + return + _validate_cli_warehouse_identifier(cli_warehouse) + discovered: DiscoveredProjectInputs = discover_project_configuration(project_dir=project_dir) + adapter_name: str = resolve_effective_adapter_name( + project_config=discovered.project_config, local_config=discovered.local_config + ) + if not adapter_supports_session_warehouse(adapter_name=adapter_name, project_dir=project_dir): + _raise_unsupported_cli_warehouse(cli_warehouse=cli_warehouse, adapter_name=adapter_name) + + +def find_target_group_warehouse( + *, + discovered_inputs: DiscoveredProjectInputs, + selected_target: str | None, + group: WarehouseGroup, +) -> AuthoredTargetWarehouse | None: + """Find the active target's authored, unexpanded warehouse for one command group.""" + + target_name: str | None = resolve_target_name( + project_config=discovered_inputs.project_config, + local_config=discovered_inputs.local_config, + selected_target=selected_target, + ) + if target_name is None: + return None + local_target: LocalTargetConfig | None = discovered_inputs.local_config.targets.get(target_name) + project_target: TargetConfig | None = discovered_inputs.project_config.targets.get(target_name) + local_value: str | None = ( + None if local_target is None else local_target.warehouses.for_group(group) + ) + project_value: str | None = ( + None if project_target is None else project_target.warehouses.for_group(group) + ) + authored: str | None = local_value if local_value is not None else project_value + if authored is None: + return None + return AuthoredTargetWarehouse( + warehouse=authored, + target_name=target_name, + group=group, + file_name=LOCAL_CONFIG_FILENAME if local_value is not None else PROJECT_CONFIG_FILENAME, + ) + + +def resolve_command_warehouse( + *, + discovered_inputs: DiscoveredProjectInputs | None, + selected_target: str | None, + group: WarehouseGroup | None, + cli_warehouse: str | None, + connection_config: dict[str, object], + cli_vars: dict[str, object] | None = None, +) -> ResolvedWarehouse: + """Resolve one group's warehouse: CLI, then target group, then connection, then default.""" + + if cli_warehouse is not None: + _validate_cli_warehouse_identifier(cli_warehouse) + return ResolvedWarehouse(warehouse=cli_warehouse, source=CLI_WAREHOUSE_SOURCE) + authored: AuthoredTargetWarehouse | None = ( + None + if group is None or discovered_inputs is None + else find_target_group_warehouse( + discovered_inputs=discovered_inputs, selected_target=selected_target, group=group + ) + ) + if authored is not None and discovered_inputs is not None: + return _expand_target_group_warehouse( + authored=authored, + discovered_inputs=discovered_inputs, + selected_target=selected_target, + cli_vars=cli_vars, + ) + connection_warehouse: object | None = connection_config.get(CONNECTION_WAREHOUSE_KEY) + if isinstance(connection_warehouse, str) and connection_warehouse.strip(): + return ResolvedWarehouse(warehouse=connection_warehouse, source=CONNECTION_WAREHOUSE_SOURCE) + return ResolvedWarehouse(warehouse=None, source=CONNECTION_DEFAULT_WAREHOUSE_SOURCE) + + +def apply_command_warehouse( + *, + config: dict[str, object], + adapter_name: str, + project_dir: Path | None, + discovered_inputs: DiscoveredProjectInputs | None, + selected_target: str | None, +) -> dict[str, object]: + """Return connection config whose warehouse follows the active command's group.""" + + scope: CommandWarehouseScope = current_command_warehouse_scope() + authored: AuthoredTargetWarehouse | None = ( + None + if scope.group is None or discovered_inputs is None + else find_target_group_warehouse( + discovered_inputs=discovered_inputs, + selected_target=selected_target, + group=scope.group, + ) + ) + if scope.cli_warehouse is None and authored is None: + return config + if not adapter_supports_session_warehouse(adapter_name=adapter_name, project_dir=project_dir): + if scope.cli_warehouse is not None: + _raise_unsupported_cli_warehouse( + cli_warehouse=scope.cli_warehouse, adapter_name=adapter_name + ) + return config + resolved: ResolvedWarehouse = resolve_command_warehouse( + discovered_inputs=discovered_inputs, + selected_target=selected_target, + group=scope.group, + cli_warehouse=scope.cli_warehouse, + connection_config=config, + cli_vars=scope.cli_vars, + ) + if resolved.source in _CONNECTION_SOURCES: + return config + return {**config, CONNECTION_WAREHOUSE_KEY: resolved.warehouse} + + +def describe_connection_warehouse( + *, + discovered_inputs: DiscoveredProjectInputs, + selected_target: str | None, + connection_config: dict[str, object], +) -> ResolvedWarehouse: + """Describe the warehouse an already resolved connection uses and where it came from.""" + + scope: CommandWarehouseScope = current_command_warehouse_scope() + connection_warehouse: object | None = connection_config.get(CONNECTION_WAREHOUSE_KEY) + if not isinstance(connection_warehouse, str) or not connection_warehouse.strip(): + return ResolvedWarehouse(warehouse=None, source=CONNECTION_DEFAULT_WAREHOUSE_SOURCE) + if scope.cli_warehouse == connection_warehouse: + return ResolvedWarehouse(warehouse=connection_warehouse, source=CLI_WAREHOUSE_SOURCE) + authored: AuthoredTargetWarehouse | None = ( + None + if scope.group is None + else find_target_group_warehouse( + discovered_inputs=discovered_inputs, + selected_target=selected_target, + group=scope.group, + ) + ) + if authored is not None and _expands_to( + authored=authored, + warehouse=connection_warehouse, + discovered_inputs=discovered_inputs, + selected_target=selected_target, + cli_vars=scope.cli_vars, + ): + return ResolvedWarehouse(warehouse=connection_warehouse, source=authored.source) + return ResolvedWarehouse(warehouse=connection_warehouse, source=CONNECTION_WAREHOUSE_SOURCE) + + +def _expands_to( + *, + authored: AuthoredTargetWarehouse, + warehouse: str, + discovered_inputs: DiscoveredProjectInputs, + selected_target: str | None, + cli_vars: dict[str, object], +) -> bool: + try: + resolved: ResolvedWarehouse = _expand_target_group_warehouse( + authored=authored, + discovered_inputs=discovered_inputs, + selected_target=selected_target, + cli_vars=cli_vars, + ) + except (CompileInputError, CliUserError): + return False + return resolved.warehouse == warehouse + + +def _expand_target_group_warehouse( + *, + authored: AuthoredTargetWarehouse, + discovered_inputs: DiscoveredProjectInputs, + selected_target: str | None, + cli_vars: dict[str, object] | None, +) -> ResolvedWarehouse: + effective_vars: dict[str, object] = build_effective_runtime_config( + discovered_inputs=discovered_inputs, + selected_target=selected_target, + cli_vars=cli_vars, + )[1] + warehouse: str = str( + expand_template_data( + value=authored.warehouse, + variables=effective_vars, + context_values={}, + context_label=f"targets.{authored.target_name}.warehouses.{authored.group.value}", + allow_context=False, + preserve_context_tokens=False, + preserve_unknown_context=False, + ) + ) + if _SNOWFLAKE_IDENTIFIER_PATTERN.fullmatch(warehouse) is None: + section: str = f"targets.{authored.target_name}.warehouses" + raise CliUserError( + f"{authored.file_name} [{section}] {authored.group.value} resolves to " + f"{warehouse!r}, which is not a valid Snowflake warehouse identifier", + code="C261", + help=setting_help( + purpose='to use an unquoted identifier (or a double-quoted one such as "Adhoc WH")', + file_name=authored.file_name, + section=section, + key=authored.group.value, + value="ADHOC_WH", + ), + ) + return ResolvedWarehouse(warehouse=warehouse, source=authored.source) + + +def _validate_cli_warehouse_identifier(cli_warehouse: str) -> None: + if not cli_warehouse.strip(): + raise CliUserError( + "--warehouse requires a warehouse name", + code="C261", + help="pass a name such as `--warehouse ADHOC_WH`, or omit the flag", + ) + if _SNOWFLAKE_IDENTIFIER_PATTERN.fullmatch(cli_warehouse) is None: + raise CliUserError( + f"--warehouse {cli_warehouse!r} is not a valid Snowflake warehouse identifier", + code="C261", + help=( + "pass an unquoted identifier such as `--warehouse ADHOC_WH`, or a double-quoted " + """identifier such as `--warehouse '"Adhoc WH"'`""" + ), + ) + + +def _raise_unsupported_cli_warehouse(*, cli_warehouse: str, adapter_name: str) -> None: + raise CliUserError( + f"--warehouse {cli_warehouse} was given, but the '{adapter_name}' adapter has no " + "warehouse to select", + code="C260", + help=( + "remove --warehouse; it needs an adapter with a session warehouse, such as " + "snowflake or an adapter extending it" + ), + ) diff --git a/src/sqlbuild/cli/commands/classes/cli_namespace.py b/src/sqlbuild/cli/commands/classes/cli_namespace.py index 11530e5d84..929137fb5a 100644 --- a/src/sqlbuild/cli/commands/classes/cli_namespace.py +++ b/src/sqlbuild/cli/commands/classes/cli_namespace.py @@ -24,6 +24,7 @@ "defer_clone_from": None, "defer_sources_to": None, "target": None, + "warehouse": None, "case": None, "inspect": False, "target_range": None, @@ -188,6 +189,7 @@ class CliNamespace: defer_clone_from: str | None defer_sources_to: str | None target: str | None + warehouse: str | None case: str | None inspect: bool target_range: str | None diff --git a/src/sqlbuild/cli/commands/constants.py b/src/sqlbuild/cli/commands/constants.py index 9aab032ea6..9ff371fc19 100644 --- a/src/sqlbuild/cli/commands/constants.py +++ b/src/sqlbuild/cli/commands/constants.py @@ -1,8 +1,11 @@ """CLI command decision constants.""" -from sqlbuild.cli.commands.types import CompileLineageMode, PlaygroundTemplate +from types import MappingProxyType + +from sqlbuild.cli.commands.types import CliCommand, CompileLineageMode, PlaygroundTemplate from sqlbuild.compiler.lineage.types import ColumnLineageMode from sqlbuild.compiler.planner.types import SelectorKind +from sqlbuild.spec.contracts.types import WarehouseGroup C0_CONTROL_CODE_LIMIT: int = 32 C1_CONTROL_CODE_START: int = 127 @@ -138,3 +141,39 @@ EXECUTION_LIMIT_BUILD_NOTE: str = "No warehouse changes were made." EXECUTION_LIMIT_DBT_NOTE: str = "dbt work has already run; no SQLBuild models were built." QUERY_DIFF_INCOMPLETE_EXECUTION_CODES: frozenset[str] = frozenset({"C237", "C238", "C239"}) +CLI_WAREHOUSE_SOURCE: str = "--warehouse" +CONNECTION_WAREHOUSE_SOURCE: str = "connection warehouse" +CONNECTION_DEFAULT_WAREHOUSE_SOURCE: str = "connection default" +COMMAND_WAREHOUSE_GROUPS: MappingProxyType[CliCommand, WarehouseGroup | None] = MappingProxyType( + { + CliCommand.AUDIT: WarehouseGroup.BUILD, + CliCommand.BUILD: WarehouseGroup.BUILD, + CliCommand.CHECK: WarehouseGroup.BUILD, + CliCommand.CLONE: WarehouseGroup.BUILD, + CliCommand.DBT: WarehouseGroup.BUILD, + CliCommand.JANITOR: WarehouseGroup.BUILD, + CliCommand.LOAD: WarehouseGroup.BUILD, + CliCommand.PLAN: WarehouseGroup.BUILD, + CliCommand.SCENARIO: WarehouseGroup.BUILD, + CliCommand.SEED: WarehouseGroup.BUILD, + CliCommand.TEST: WarehouseGroup.BUILD, + CliCommand.CONTRACT: WarehouseGroup.QUERY, + CliCommand.COST: WarehouseGroup.QUERY, + CliCommand.DEBUG: WarehouseGroup.QUERY, + CliCommand.DIFF: WarehouseGroup.QUERY, + CliCommand.FRESHNESS: WarehouseGroup.QUERY, + CliCommand.QUERY: WarehouseGroup.QUERY, + CliCommand.CLEAN: None, + CliCommand.COMPILE: None, + CliCommand.DAG: None, + CliCommand.FORMAT: None, + CliCommand.INIT: None, + CliCommand.LINEAGE: None, + CliCommand.MV: None, + CliCommand.PLAYGROUND: None, + CliCommand.RENAME: None, + CliCommand.RULES: None, + CliCommand.SCOPE: None, + CliCommand.SKILLS: None, + } +) diff --git a/src/sqlbuild/cli/commands/main/execution/command_warehouse.py b/src/sqlbuild/cli/commands/main/execution/command_warehouse.py new file mode 100644 index 0000000000..27fdc6755f --- /dev/null +++ b/src/sqlbuild/cli/commands/main/execution/command_warehouse.py @@ -0,0 +1,29 @@ +"""Command-group warehouse public entry for cross-domain orchestration.""" + +from __future__ import annotations + +from pathlib import Path + +from sqlbuild.cli.commands._helpers.runtime.warehouses import ( + apply_command_warehouse as _apply_command_warehouse, +) +from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs + + +def apply_command_warehouse( + *, + config: dict[str, object], + adapter_name: str, + project_dir: Path | None, + discovered_inputs: DiscoveredProjectInputs | None, + selected_target: str | None, +) -> dict[str, object]: + """Return connection config whose warehouse follows the active command's group.""" + + return _apply_command_warehouse( + config=config, + adapter_name=adapter_name, + project_dir=project_dir, + discovered_inputs=discovered_inputs, + selected_target=selected_target, + ) diff --git a/src/sqlbuild/cli/commands/models.py b/src/sqlbuild/cli/commands/models.py index 906928cc66..7d222bcb26 100644 --- a/src/sqlbuild/cli/commands/models.py +++ b/src/sqlbuild/cli/commands/models.py @@ -89,6 +89,38 @@ SnapshotsConfig, SourceEntry, ) +from sqlbuild.spec.contracts.types import WarehouseGroup + + +@dataclass(frozen=True) +class CommandWarehouseScope: + """The warehouse group and CLI override active for one command invocation.""" + + group: WarehouseGroup | None + cli_warehouse: str | None = None + cli_vars: dict[str, object] = field(default_factory=dict) + + +@dataclass(frozen=True) +class AuthoredTargetWarehouse: + """One target's authored, unexpanded warehouse for a command group and its file.""" + + warehouse: str + target_name: str + group: WarehouseGroup + file_name: str + + @property + def source(self) -> str: + return f"{self.file_name} [targets.{self.target_name}.warehouses] {self.group.value}" + + +@dataclass(frozen=True) +class ResolvedWarehouse: + """The warehouse one command connects with and where that value came from.""" + + warehouse: str | None + source: str @dataclass(frozen=True) diff --git a/src/sqlbuild/compiler/discovery/_helpers/yml/project.py b/src/sqlbuild/compiler/discovery/_helpers/yml/project.py index f05cf7df1f..d2de618858 100644 --- a/src/sqlbuild/compiler/discovery/_helpers/yml/project.py +++ b/src/sqlbuild/compiler/discovery/_helpers/yml/project.py @@ -20,6 +20,10 @@ ) from sqlbuild.compiler.compile.constants import MAX_MICROBATCHES_CONFIG_KEY, TEMPLATE_OPEN_TOKEN from sqlbuild.compiler.discovery._helpers.validation.supported_keys import unsupported_keys_help +from sqlbuild.compiler.discovery._helpers.yml.target_warehouses import ( + load_target_warehouses, + reject_unsupported_target_warehouses, +) from sqlbuild.compiler.discovery.constants import ( CONFIG_CONCURRENCY_KEY, DBT_DEFER_CLONE_CONFIG_KEY, @@ -482,6 +486,20 @@ def validate_local_sql_analysis_policy( ) +def validate_target_warehouses_adapter( + *, project_dir: Path, project_config: ProjectConfig, local_config: LocalConfig +) -> None: + """Reject command-group warehouses for adapters without a session warehouse.""" + + reject_unsupported_target_warehouses( + project_dir=project_dir, + project_config=project_config, + local_config=local_config, + project_config_path=_resolve_project_config_path(project_dir=project_dir), + local_config_path=_resolve_local_config_path(project_dir=project_dir), + ) + + def _resolve_project_config_path(*, project_dir: Path) -> Path: toml_path: Path = project_dir / PROJECT_CONFIG_FILENAME if toml_path.exists(): @@ -1314,6 +1332,7 @@ def _project_target(local: LocalTargetConfig) -> TargetConfig: missing_migration_origin=local.missing_migration_origin or defaults.missing_migration_origin, execution_limits=local.execution_limits, + warehouses=local.warehouses, ) @@ -1401,6 +1420,11 @@ def _load_local_targets(*, payload: object, file_path: Path) -> dict[str, LocalT policy=MissingMigrationOriginPolicy, ), execution_limits=execution_limits, + warehouses=load_target_warehouses( + payload=target_mapping.get("warehouses"), + target_name=target_name, + file_path=file_path, + ), clone=LocalClonePolicy( allow_as_clone_origin=_optional_nullable_bool( mapping=clone_mapping, @@ -1473,6 +1497,7 @@ def _validate_target_keys( "missing_migration_origin", "clone", "execution_limits", + "warehouses", } ), label=f"targets.{target_name}", diff --git a/src/sqlbuild/compiler/discovery/_helpers/yml/target_warehouses.py b/src/sqlbuild/compiler/discovery/_helpers/yml/target_warehouses.py new file mode 100644 index 0000000000..63cc5bda42 --- /dev/null +++ b/src/sqlbuild/compiler/discovery/_helpers/yml/target_warehouses.py @@ -0,0 +1,128 @@ +"""Loading and adapter validation for per-target command-group warehouses.""" + +from __future__ import annotations + +from pathlib import Path +from typing import cast + +from sqlbuild.adapter.discovery.main.session_warehouse_support import ( + adapter_supports_session_warehouse, +) +from sqlbuild.compiler.discovery._helpers.validation.supported_keys import ( + reject_unknown_mapping_keys, +) +from sqlbuild.compiler.discovery.constants import LOCAL_CONFIG_FILENAME, PROJECT_CONFIG_FILENAME +from sqlbuild.compiler.discovery.exceptions import ProjectConfigError +from sqlbuild.errors.setting_help.main.setting_note import setting_note +from sqlbuild.spec.contracts.main.resolve_effective_adapter_name import ( + resolve_effective_adapter_name, +) +from sqlbuild.spec.contracts.models import LocalConfig, ProjectConfig, TargetWarehousesConfig +from sqlbuild.spec.contracts.types import WarehouseGroup + + +def load_target_warehouses( + *, payload: object, target_name: str, file_path: Path +) -> TargetWarehousesConfig: + """Parse one target's optional `[targets..warehouses]` section.""" + + label: str = f"targets.{target_name}.warehouses" + if payload is None: + return TargetWarehousesConfig() + if not isinstance(payload, dict): + raise ProjectConfigError(f"{file_path} {label} must be a mapping") + mapping: dict[str, object] = cast(dict[str, object], payload) + reject_unknown_mapping_keys( + mapping=mapping, + allowed=frozenset(group.value for group in WarehouseGroup), + file_path=file_path, + label=label, + error_class=ProjectConfigError, + ) + values: dict[WarehouseGroup, str | None] = {} + group: WarehouseGroup + for group in WarehouseGroup: + value: object | None = mapping.get(group.value) + if value is not None and (not isinstance(value, str) or not value.strip()): + raise ProjectConfigError( + f"{file_path} {label}.{group.value} must be a non-empty warehouse name" + ) + values[group] = value.strip() if isinstance(value, str) else None + return TargetWarehousesConfig( + build=values[WarehouseGroup.BUILD], query=values[WarehouseGroup.QUERY] + ) + + +def reject_unsupported_target_warehouses( + *, + project_dir: Path, + project_config: ProjectConfig, + local_config: LocalConfig, + project_config_path: Path, + local_config_path: Path, +) -> None: + """Check shared groups against the shared adapter and local groups against the effective one.""" + + project_group: tuple[str, WarehouseGroup, str] | None = _first_configured_group( + {name: target.warehouses for name, target in project_config.targets.items()} + ) + if project_group is not None and not adapter_supports_session_warehouse( + adapter_name=project_config.adapter, project_dir=project_dir + ): + _raise_unsupported_target_warehouse( + adapter_name=project_config.adapter, + file_path=project_config_path, + file_name=PROJECT_CONFIG_FILENAME, + configured=project_group, + ) + local_group: tuple[str, WarehouseGroup, str] | None = _first_configured_group( + {name: target.warehouses for name, target in local_config.targets.items()} + ) + if local_group is None: + return + effective_adapter: str = resolve_effective_adapter_name( + project_config=project_config, local_config=local_config + ) + if not adapter_supports_session_warehouse( + adapter_name=effective_adapter, project_dir=project_dir + ): + _raise_unsupported_target_warehouse( + adapter_name=effective_adapter, + file_path=local_config_path, + file_name=LOCAL_CONFIG_FILENAME, + configured=local_group, + ) + + +def _first_configured_group( + warehouses_by_target: dict[str, TargetWarehousesConfig], +) -> tuple[str, WarehouseGroup, str] | None: + target_name: str + warehouses: TargetWarehousesConfig + for target_name, warehouses in sorted(warehouses_by_target.items()): + group: WarehouseGroup + for group in WarehouseGroup: + value: str | None = warehouses.for_group(group) + if value is not None: + return target_name, group, value + return None + + +def _raise_unsupported_target_warehouse( + *, + adapter_name: str, + file_path: Path, + file_name: str, + configured: tuple[str, WarehouseGroup, str], +) -> None: + target_name, group, value = configured + section: str = f"targets.{target_name}.warehouses" + raise ProjectConfigError( + f"{file_path} [{section}] selects a {group.value} warehouse, but the " + f"'{adapter_name}' adapter has no warehouse to select; " + + setting_note(file_name=file_name, section=section, key=group.value, value=value), + help=( + f"remove the [{section}] section from {file_name}; command-group warehouses need an " + "adapter with a session warehouse, such as snowflake or an adapter extending it" + ), + ) diff --git a/src/sqlbuild/compiler/discovery/main/discover.py b/src/sqlbuild/compiler/discovery/main/discover.py index 7d749b96f7..bd5915c9eb 100644 --- a/src/sqlbuild/compiler/discovery/main/discover.py +++ b/src/sqlbuild/compiler/discovery/main/discover.py @@ -16,6 +16,7 @@ load_local_config, load_project_config, validate_local_sql_analysis_policy, + validate_target_warehouses_adapter, ) from sqlbuild.compiler.discovery.constants import ( DISCOVERY_FACT_CACHE_ALGORITHM, @@ -74,6 +75,9 @@ def _assemble_discovered_project_inputs( validate_local_sql_analysis_policy( project_dir=project_dir, project_config=project_config, local_config=local_config ) + validate_target_warehouses_adapter( + project_dir=project_dir, project_config=project_config, local_config=local_config + ) validate_project_python_paths(project_dir=project_dir) sql_analysis_enabled: bool = ( sql_analysis_enabled_override diff --git a/src/sqlbuild/integrations/dbt/_helpers/pipeline/execution_phases.py b/src/sqlbuild/integrations/dbt/_helpers/pipeline/execution_phases.py index 5c1ab0793e..94e1dece9a 100644 --- a/src/sqlbuild/integrations/dbt/_helpers/pipeline/execution_phases.py +++ b/src/sqlbuild/integrations/dbt/_helpers/pipeline/execution_phases.py @@ -5,6 +5,7 @@ from dataclasses import replace from sqlbuild.cli.commands.main.dbt.dbt_sqlbuild_work import execute_dbt_sqlbuild_work +from sqlbuild.cli.commands.main.execution.command_warehouse import apply_command_warehouse from sqlbuild.cli.commands.main.execution.connection_progress import ( build_connection_progress_reporter, ) @@ -47,13 +48,19 @@ def resolve_dbt_connection_config( ) -> dict[str, object]: """Resolve the connection used only by selected SQLBuild work.""" - return resolve_connection_config( - raw_config=build_effective_connection_config( - discovered_inputs=invocation.discovered_inputs + return apply_command_warehouse( + config=resolve_connection_config( + raw_config=build_effective_connection_config( + discovered_inputs=invocation.discovered_inputs + ), + project_dir=request.project_dir, + adapter_name=compiled.adapter_name, + discovered_inputs=invocation.discovered_inputs, ), - project_dir=request.project_dir, adapter_name=compiled.adapter_name, discovered_inputs=invocation.discovered_inputs, + project_dir=request.project_dir, + selected_target=None, ) diff --git a/src/sqlbuild/integrations/dbt/_helpers/pipeline/plan_output.py b/src/sqlbuild/integrations/dbt/_helpers/pipeline/plan_output.py index 5ddec9b493..b081fbe88e 100644 --- a/src/sqlbuild/integrations/dbt/_helpers/pipeline/plan_output.py +++ b/src/sqlbuild/integrations/dbt/_helpers/pipeline/plan_output.py @@ -9,6 +9,7 @@ from typing import Any from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.cli.commands.main.execution.command_warehouse import apply_command_warehouse from sqlbuild.compiler.compile.main.effective_config import build_effective_connection_config from sqlbuild.compiler.compile.models import CompiledProject from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs @@ -57,13 +58,19 @@ def build_sqlbuild_plan_output( if not request.selected_model_names: return None planning_project: CompiledProject = environment.project - connection_config: dict[str, object] = resolve_connection_config( - raw_config=build_effective_connection_config( - discovered_inputs=environment.discovered_inputs + connection_config: dict[str, object] = apply_command_warehouse( + config=resolve_connection_config( + raw_config=build_effective_connection_config( + discovered_inputs=environment.discovered_inputs + ), + project_dir=environment.project_dir, + adapter_name=environment.adapter_name, + discovered_inputs=environment.discovered_inputs, ), - project_dir=environment.project_dir, adapter_name=environment.adapter_name, discovered_inputs=environment.discovered_inputs, + project_dir=environment.project_dir, + selected_target=None, ) connection: Any = _connect_for_plan( adapter=environment.adapter, diff --git a/src/sqlbuild/spec/contracts/_helpers/targets.py b/src/sqlbuild/spec/contracts/_helpers/targets.py index aa62d4c54c..0150104fdf 100644 --- a/src/sqlbuild/spec/contracts/_helpers/targets.py +++ b/src/sqlbuild/spec/contracts/_helpers/targets.py @@ -11,6 +11,7 @@ LocalTargetConfig, ProjectConfig, TargetConfig, + TargetWarehousesConfig, ) @@ -125,6 +126,18 @@ def resolve_target_config( project_clone=project_target.clone, local_clone=local_target.clone, ), + warehouses=TargetWarehousesConfig( + build=( + local_target.warehouses.build + if local_target.warehouses.build is not None + else project_target.warehouses.build + ), + query=( + local_target.warehouses.query + if local_target.warehouses.query is not None + else project_target.warehouses.query + ), + ), ) return target_config diff --git a/src/sqlbuild/spec/contracts/constants.py b/src/sqlbuild/spec/contracts/constants.py index 3214e623c5..9ad47f86c4 100644 --- a/src/sqlbuild/spec/contracts/constants.py +++ b/src/sqlbuild/spec/contracts/constants.py @@ -16,3 +16,4 @@ "incremental", "snapshot", ) +CONNECTION_WAREHOUSE_KEY: str = "warehouse" diff --git a/src/sqlbuild/spec/contracts/models.py b/src/sqlbuild/spec/contracts/models.py index 853a180eb0..09d524ad73 100644 --- a/src/sqlbuild/spec/contracts/models.py +++ b/src/sqlbuild/spec/contracts/models.py @@ -23,6 +23,7 @@ TableTypeDowngradePolicy, TableTypeSource, TimeTravelRetentionSource, + WarehouseGroup, ) from sqlbuild.sql_values.types import CollectionRendering @@ -102,6 +103,21 @@ class ExecutionLimitsConfig: remediation: str | None = None +@dataclass(frozen=True) +class TargetWarehousesConfig: + """Optional per-command-group default warehouses for one target.""" + + build: str | None = None + query: str | None = None + + def for_group(self, group: WarehouseGroup) -> str | None: + return self.build if group == WarehouseGroup.BUILD else self.query + + @property + def configured(self) -> bool: + return self.build is not None or self.query is not None + + @dataclass(frozen=True) class TargetConfig: """One named target configuration.""" @@ -126,6 +142,7 @@ class TargetConfig: time_travel_retention_decrease: RetentionDecreasePolicy = RetentionDecreasePolicy.DENY missing_migration_origin: MissingMigrationOriginPolicy = MissingMigrationOriginPolicy.ALLOW execution_limits: ExecutionLimitsConfig = field(default_factory=ExecutionLimitsConfig) + warehouses: TargetWarehousesConfig = field(default_factory=TargetWarehousesConfig) @dataclass(frozen=True) @@ -150,6 +167,7 @@ class LocalTargetConfig: time_travel_retention_decrease: RetentionDecreasePolicy | None = None missing_migration_origin: MissingMigrationOriginPolicy | None = None execution_limits: ExecutionLimitsConfig = field(default_factory=ExecutionLimitsConfig) + warehouses: TargetWarehousesConfig = field(default_factory=TargetWarehousesConfig) @dataclass(frozen=True) diff --git a/src/sqlbuild/spec/contracts/types.py b/src/sqlbuild/spec/contracts/types.py index 58c0ea6d92..b505530fe8 100644 --- a/src/sqlbuild/spec/contracts/types.py +++ b/src/sqlbuild/spec/contracts/types.py @@ -14,6 +14,13 @@ class EventExportSeverity(StrEnum): ERROR = "error" +class WarehouseGroup(StrEnum): + """Command group that selects a target's default warehouse.""" + + BUILD = "build" + QUERY = "query" + + class SourceWriteStrategy(StrEnum): APPEND = "append" DELETE_INSERT = "delete_insert" diff --git a/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py b/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py index a1cf537f81..f10cda5377 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py @@ -964,3 +964,47 @@ class RepeatedJsonParseTestCase: expected_locations: tuple[tuple[int, int], ...] expected_detail: str = "" extra_files: tuple[tuple[str, str], ...] = () + + +@dataclass(frozen=True) +class CommandWarehouseCliTestCase: + """One real CLI invocation and the only warehouse its Snowflake sessions may use.""" + + description: str + argv: tuple[str, ...] + expected_warehouse: str + warehouses_section: str = '[targets.dev.warehouses]\nbuild = "BUILD_WH"\nquery = "ADHOC_WH"\n' + local_config: str = "" + + +@dataclass(frozen=True) +class CommandWarehouseOutputTestCase: + """One real CLI invocation and the warehouse report it must print.""" + + description: str + argv: tuple[str, ...] + expected_fragments: tuple[str, ...] + warehouses_section: str = '[targets.dev.warehouses]\nbuild = "BUILD_WH"\nquery = "ADHOC_WH"\n' + local_config: str = "" + + +@dataclass(frozen=True) +class CommandWarehouseDebugJsonTestCase: + """One `sqb debug --json` invocation and the warehouse lines it must report.""" + + description: str + argv: tuple[str, ...] + expected_lines: tuple[tuple[str, str, str], ...] + + +@dataclass(frozen=True) +class CommandWarehouseExitTestCase: + """One real CLI invocation, its exit code, and the output it must print.""" + + description: str + argv: tuple[str, ...] + expected_exit_code: int + expected_fragments: tuple[str, ...] + expected_connect_calls: int = 0 + warehouses_section: str = '[targets.dev.warehouses]\nbuild = "BUILD_WH"\nquery = "ADHOC_WH"\n' + local_config: str = "" diff --git a/tests/integration/src/sqlbuild/cli/commands/main/helpers.py b/tests/integration/src/sqlbuild/cli/commands/main/helpers.py index 8e6533315b..56aaa5a5bf 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/helpers.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/helpers.py @@ -2,15 +2,18 @@ import json import os +import sys import tempfile import threading from collections.abc import Callable from pathlib import Path +from types import ModuleType from typing import Any, cast import duckdb import pytest +from sqlbuild.adapter.contract.exceptions import AdapterUserError from sqlbuild.adapter.contract.models import ( RenderedRetentionChange, RetentionRequest, @@ -683,3 +686,97 @@ def write_repeated_json_parse_project( path: Path = tmp_path / relative_path path.parent.mkdir(parents=True, exist_ok=True) path.write_text(contents, encoding="utf-8") + + +class OfflineSnowflakeRawConnection: + """Snowflake connector double that answers reads with no rows and refuses writes.""" + + description: None = None + rowcount: int = 0 + sfqid: str = "offline" + + def __init__(self, executed_sql: list[str]) -> None: + self._executed_sql: list[str] = executed_sql + + def cursor(self, *args: object, **kwargs: object) -> OfflineSnowflakeRawConnection: + del args, kwargs + return self + + def execute(self, sql: str, *args: object, **kwargs: object) -> OfflineSnowflakeRawConnection: + del args, kwargs + self._executed_sql.append(sql) + _OFFLINE_SNOWFLAKE_STATEMENT_RESPONSES.get( + sql.lstrip().split(None, 1)[0].upper(), _refuse_offline_snowflake_write + )() + return self + + def fetchall(self) -> list[tuple[object, ...]]: + return [] + + def fetchone(self) -> None: + return None + + def close(self) -> None: + pass + + +def _accept_offline_snowflake_read() -> None: + return None + + +def _refuse_offline_snowflake_write() -> None: + raise AdapterUserError(message="offline Snowflake double refuses writes") + + +_OFFLINE_SNOWFLAKE_STATEMENT_RESPONSES: dict[str, Callable[[], None]] = dict.fromkeys( + ("USE", "SHOW", "SELECT", "DESC", "DESCRIBE", "WITH"), _accept_offline_snowflake_read +) + + +def install_offline_snowflake_connector( + monkeypatch: pytest.MonkeyPatch, +) -> tuple[list[dict[str, object]], list[str]]: + """Install an offline snowflake.connector and capture every connect call and statement.""" + + connect_calls: list[dict[str, object]] = [] + executed_sql: list[str] = [] + snowflake_module: ModuleType = ModuleType("snowflake") + connector_module: ModuleType = ModuleType("snowflake.connector") + + def connect(**kwargs: object) -> OfflineSnowflakeRawConnection: + connect_calls.append(kwargs) + return OfflineSnowflakeRawConnection(executed_sql) + + connector_module.__dict__["connect"] = connect + snowflake_module.__dict__["connector"] = connector_module + monkeypatch.setitem(sys.modules, "snowflake", snowflake_module) + monkeypatch.setitem(sys.modules, "snowflake.connector", connector_module) + return connect_calls, executed_sql + + +def write_snowflake_warehouse_project( + *, project_dir: Path, warehouses_section: str, local_config: str +) -> None: + """Write a one-model Snowflake project whose connection names a fallback warehouse.""" + + (project_dir / "sqlbuild_project.toml").write_text( + 'name = "orders"\n' + 'adapter = "snowflake"\n' + 'default_target = "dev"\n\n' + "[connections.main]\n" + 'account = "example-account"\n' + 'user = "builder"\n' + 'password = "example-password"\n' + 'warehouse = "ANALYTICS_WH"\n\n' + "[targets.dev]\n" + 'connection = "main"\n' + 'database = "ANALYTICS"\n' + 'schema = "DEV"\n\n' + warehouses_section, + encoding="utf-8", + ) + (project_dir / "sqlbuild_local.toml").write_text(local_config, encoding="utf-8") + (project_dir / "models").mkdir() + (project_dir / "models" / "orders.sql").write_text( + "MODEL (description 'Orders.', materialized table);\nSELECT 1 AS order_id\n", + encoding="utf-8", + ) diff --git a/tests/integration/src/sqlbuild/cli/commands/main/test_command_warehouses.py b/tests/integration/src/sqlbuild/cli/commands/main/test_command_warehouses.py new file mode 100644 index 0000000000..7e2cd2d285 --- /dev/null +++ b/tests/integration/src/sqlbuild/cli/commands/main/test_command_warehouses.py @@ -0,0 +1,339 @@ +"""Real CLI coverage for the warehouse each command group's Snowflake sessions select.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +from _pytest.capture import CaptureResult + +from sqlbuild.cli.commands.main.entrypoint.entry import main +from tests.integration.src.sqlbuild.cli.commands.main._test_types import ( + CommandWarehouseCliTestCase, + CommandWarehouseDebugJsonTestCase, + CommandWarehouseExitTestCase, + CommandWarehouseOutputTestCase, +) +from tests.integration.src.sqlbuild.cli.commands.main.helpers import ( + install_offline_snowflake_connector, + write_snowflake_warehouse_project, +) + +_RAW_QUERY_DIFF_ARGS: tuple[str, ...] = ( + "diff", + "--left-query", + "SELECT 1 AS order_id", + "--right-query", + "SELECT 1 AS order_id", + "--key", + "order_id", +) +_LOCAL_QUERY_OVERRIDE: str = '[targets.dev.warehouses]\nquery = "LOCAL_ADHOC_WH"\n' +_LOCAL_DUCKDB_OVERRIDE: str = ( + 'adapter = "duckdb"\n\n' + "[connections.local]\n" + 'database = "dev.duckdb"\n\n' + "[targets.dev]\n" + 'connection = "local"\n' + 'database = "dev"\n' +) +_MISSING_ENV_QUERY_GROUP: str = ( + '[targets.dev.warehouses]\nbuild = "BUILD_WH"\nquery = "${ENV:SQB_EXAMPLE_MISSING_QWH}"\n' +) + + +@pytest.mark.parametrize( + "test_case", + [ + CommandWarehouseCliTestCase( + description="build plans and executes on the build warehouse", + argv=("build",), + expected_warehouse="BUILD_WH", + ), + CommandWarehouseCliTestCase( + description="plan inspects on the build warehouse", + argv=("plan",), + expected_warehouse="BUILD_WH", + ), + CommandWarehouseCliTestCase( + description="query runs on the query warehouse", + argv=("query", "SELECT 1"), + expected_warehouse="ADHOC_WH", + ), + CommandWarehouseCliTestCase( + description="raw query diff runs on the query warehouse", + argv=_RAW_QUERY_DIFF_ARGS, + expected_warehouse="ADHOC_WH", + ), + CommandWarehouseCliTestCase( + description="cli warehouse overrides the build group", + argv=("build", "--warehouse", "CLI_WH"), + expected_warehouse="CLI_WH", + ), + CommandWarehouseCliTestCase( + description="cli warehouse overrides the query group", + argv=("query", "SELECT 1", "--warehouse", "CLI_WH"), + expected_warehouse="CLI_WH", + ), + CommandWarehouseCliTestCase( + description="local query group overrides the project query group", + argv=("query", "SELECT 1"), + local_config=_LOCAL_QUERY_OVERRIDE, + expected_warehouse="LOCAL_ADHOC_WH", + ), + CommandWarehouseCliTestCase( + description="local query group leaves the project build group", + argv=("plan",), + local_config=_LOCAL_QUERY_OVERRIDE, + expected_warehouse="BUILD_WH", + ), + CommandWarehouseCliTestCase( + description="quoted cli warehouse is passed through", + argv=("query", "SELECT 1", "--warehouse", '"Adhoc WH"'), + expected_warehouse='"Adhoc WH"', + ), + CommandWarehouseCliTestCase( + description="missing env var in the query group does not affect build", + argv=("build",), + warehouses_section=_MISSING_ENV_QUERY_GROUP, + expected_warehouse="BUILD_WH", + ), + CommandWarehouseCliTestCase( + description="unset groups keep the connection warehouse for build", + argv=("build",), + warehouses_section="", + expected_warehouse="ANALYTICS_WH", + ), + CommandWarehouseCliTestCase( + description="unset groups keep the connection warehouse for diff", + argv=_RAW_QUERY_DIFF_ARGS, + warehouses_section="", + expected_warehouse="ANALYTICS_WH", + ), + ], + ids=lambda case: case.description, +) +def test_given_command_group_when_running_cli_then_every_session_uses_one_warehouse( + test_case: CommandWarehouseCliTestCase, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + write_snowflake_warehouse_project( + project_dir=tmp_path, + warehouses_section=test_case.warehouses_section, + local_config=test_case.local_config, + ) + connect_calls, executed_sql = install_offline_snowflake_connector(monkeypatch) + + main(["--no-color", "--project-dir", str(tmp_path), *test_case.argv]) + + capsys.readouterr() + assert connect_calls + assert {call.get("warehouse") for call in connect_calls} == {test_case.expected_warehouse} + assert set(filter(lambda sql: sql.startswith("USE WAREHOUSE"), executed_sql)) == { + f"USE WAREHOUSE {test_case.expected_warehouse}" + } + + +@pytest.mark.parametrize( + "test_case", + [ + CommandWarehouseOutputTestCase( + description="verbose build header names the warehouse and its source", + argv=("build", "--verbose"), + expected_fragments=( + "warehouse BUILD_WH (sqlbuild_project.toml [targets.dev.warehouses] build)", + ), + ), + CommandWarehouseOutputTestCase( + description="debug text reports each group and its fallback", + argv=("debug", "--no-connection"), + warehouses_section="", + local_config='[targets.dev.warehouses]\nquery = "ADHOC_WH"\n', + expected_fragments=( + "build warehouse: ANALYTICS_WH [OK connection warehouse]", + "query warehouse: ADHOC_WH [OK sqlbuild_local.toml [targets.dev.warehouses] query]", + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_warehouse_groups_when_running_cli_then_output_reports_warehouse_source( + test_case: CommandWarehouseOutputTestCase, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + write_snowflake_warehouse_project( + project_dir=tmp_path, + warehouses_section=test_case.warehouses_section, + local_config=test_case.local_config, + ) + install_offline_snowflake_connector(monkeypatch) + + main(["--no-color", "--project-dir", str(tmp_path), *test_case.argv]) + + output: str = capsys.readouterr().out + assert all(fragment in output for fragment in test_case.expected_fragments) + + +@pytest.mark.parametrize( + "test_case", + [ + CommandWarehouseDebugJsonTestCase( + description="configured groups report their target section", + argv=("debug", "--json", "--no-connection"), + expected_lines=( + ( + "build warehouse", + "BUILD_WH", + "sqlbuild_project.toml [targets.dev.warehouses] build", + ), + ( + "query warehouse", + "ADHOC_WH", + "sqlbuild_project.toml [targets.dev.warehouses] query", + ), + ("warehouse", "ADHOC_WH", ""), + ), + ), + CommandWarehouseDebugJsonTestCase( + description="cli warehouse reports the flag for both groups", + argv=("debug", "--json", "--no-connection", "--warehouse", "CLI_WH"), + expected_lines=( + ("build warehouse", "CLI_WH", "--warehouse"), + ("query warehouse", "CLI_WH", "--warehouse"), + ("warehouse", "CLI_WH", ""), + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_warehouse_groups_when_running_debug_json_then_each_group_reports_its_source( + test_case: CommandWarehouseDebugJsonTestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + write_snowflake_warehouse_project( + project_dir=tmp_path, + warehouses_section='[targets.dev.warehouses]\nbuild = "BUILD_WH"\nquery = "ADHOC_WH"\n', + local_config="", + ) + + exit_code: int = main(["--no-color", "--project-dir", str(tmp_path), *test_case.argv]) + + payload: dict[str, list[dict[str, str]]] = json.loads(capsys.readouterr().out) + reported: set[tuple[str, str, str]] = { + (line["label"], line["message"], line.get("status_message", "")) + for line in (*payload["configuration"], *payload["connection"]) + } + assert exit_code == 0 + assert set(test_case.expected_lines) <= reported + + +@pytest.mark.parametrize( + "test_case", + [ + CommandWarehouseExitTestCase( + description="local duckdb override compiles with shared groups", + argv=("compile",), + local_config=_LOCAL_DUCKDB_OVERRIDE, + expected_exit_code=0, + expected_fragments=("Project compiled",), + ), + CommandWarehouseExitTestCase( + description="local duckdb override builds with shared groups", + argv=("build", "--verbose"), + warehouses_section=_MISSING_ENV_QUERY_GROUP, + local_config=_LOCAL_DUCKDB_OVERRIDE, + expected_exit_code=0, + expected_fragments=("warehouse not set", "Completed successfully"), + ), + CommandWarehouseExitTestCase( + description="local duckdb override reports shared groups as unused", + argv=("debug", "--no-connection"), + local_config=_LOCAL_DUCKDB_OVERRIDE, + expected_exit_code=0, + expected_fragments=( + "build warehouse: not used (adapter duckdb)", + "query warehouse: not used (adapter duckdb)", + ), + ), + CommandWarehouseExitTestCase( + description="local duckdb override rejects a cli warehouse before running", + argv=("test", "--warehouse", "ADHOC_WH"), + local_config=_LOCAL_DUCKDB_OVERRIDE, + expected_exit_code=1, + expected_fragments=( + "error[C260]: --warehouse ADHOC_WH was given, but the 'duckdb' adapter", + ), + ), + CommandWarehouseExitTestCase( + description="blank cli warehouse is rejected before connecting", + argv=("plan", "--warehouse", " "), + expected_exit_code=1, + expected_fragments=("error[C261]: --warehouse requires a warehouse name",), + ), + CommandWarehouseExitTestCase( + description="injected cli warehouse is rejected before connecting", + argv=("query", "SELECT 1", "--warehouse", "x; drop table t"), + expected_exit_code=1, + expected_fragments=( + "error[C261]: --warehouse 'x; drop table t' is not a valid Snowflake warehouse", + ), + ), + CommandWarehouseExitTestCase( + description="injected group warehouse is rejected before connecting", + argv=("query", "SELECT 1"), + warehouses_section='[targets.dev.warehouses]\nquery = "x; drop table t"\n', + expected_exit_code=1, + expected_fragments=( + "error[C261]: sqlbuild_project.toml [targets.dev.warehouses] query resolves to " + "'x; drop table t', which is not a valid Snowflake warehouse identifier", + "[targets.dev.warehouses]", + 'query = "ADHOC_WH"', + ), + ), + CommandWarehouseExitTestCase( + description="debug reports a broken group and keeps checking", + argv=("debug", "--no-connection"), + warehouses_section=_MISSING_ENV_QUERY_GROUP, + expected_exit_code=1, + expected_fragments=( + "build warehouse: BUILD_WH [OK sqlbuild_project.toml [targets.dev.warehouses] " + "build]", + "query warehouse: error: targets.dev.warehouses.query references missing ENV " + "variable 'SQB_EXAMPLE_MISSING_QWH'", + "connection test: [SKIP skipped by --no-connection]", + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_warehouse_inputs_when_running_cli_then_exit_and_output_match( + test_case: CommandWarehouseExitTestCase, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + monkeypatch.delenv("SQB_EXAMPLE_MISSING_QWH", raising=False) + write_snowflake_warehouse_project( + project_dir=tmp_path, + warehouses_section=test_case.warehouses_section, + local_config=test_case.local_config, + ) + connect_calls, _ = install_offline_snowflake_connector(monkeypatch) + + exit_code: int = main(["--no-color", "--project-dir", str(tmp_path), *test_case.argv]) + + captured: CaptureResult[str] = capsys.readouterr() + output: str = captured.out + captured.err + assert exit_code == test_case.expected_exit_code + assert all(fragment in output for fragment in test_case.expected_fragments) + assert len(connect_calls) == test_case.expected_connect_calls + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/unit/src/sqlbuild/cli/commands/_helpers/debug/_test_types.py b/tests/unit/src/sqlbuild/cli/commands/_helpers/debug/_test_types.py index 4ddb0a205d..e05cd444e6 100644 --- a/tests/unit/src/sqlbuild/cli/commands/_helpers/debug/_test_types.py +++ b/tests/unit/src/sqlbuild/cli/commands/_helpers/debug/_test_types.py @@ -12,3 +12,11 @@ class DebugOutputTestCase: expected_text: str expected_json_fragment: str expected_color_fragments: tuple[str, ...] = () + + +@dataclass(frozen=True) +class DebugWarehouseVarsTestCase: + description: str + project_contents: str + cli_vars: dict[str, object] + expected_lines: tuple[tuple[str, str, str], ...] diff --git a/tests/unit/src/sqlbuild/cli/commands/_helpers/debug/test_checks.py b/tests/unit/src/sqlbuild/cli/commands/_helpers/debug/test_checks.py new file mode 100644 index 0000000000..79321259e8 --- /dev/null +++ b/tests/unit/src/sqlbuild/cli/commands/_helpers/debug/test_checks.py @@ -0,0 +1,67 @@ +"""Debug diagnostics for command-group warehouses.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from sqlbuild.cli.commands._helpers.debug.checks import build_debug_result +from sqlbuild.cli.commands._helpers.runtime.warehouses import command_warehouse_scope +from sqlbuild.cli.commands.models import DebugResult +from tests.unit.src.sqlbuild.cli.commands._helpers.debug._test_types import ( + DebugWarehouseVarsTestCase, +) + + +@pytest.mark.parametrize( + "test_case", + [ + DebugWarehouseVarsTestCase( + description="cli vars expand the group exactly as the connection does", + project_contents=""" +name = "orders" +adapter = "snowflake" +default_target = "dev" + +[connections.main] +account = "example-account" +warehouse = "ANALYTICS_WH" + +[targets.dev] +connection = "main" +schema = "DEV" + +[targets.dev.warehouses] +query = "${adhoc_wh}" +""".strip(), + cli_vars={"adhoc_wh": "VAR_ADHOC_WH"}, + expected_lines=( + ( + "query warehouse", + "VAR_ADHOC_WH", + "sqlbuild_project.toml [targets.dev.warehouses] query", + ), + ("build warehouse", "ANALYTICS_WH", "connection warehouse"), + ("warehouse", "VAR_ADHOC_WH", ""), + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_cli_vars_when_building_debug_result_then_groups_use_invocation_vars( + test_case: DebugWarehouseVarsTestCase, tmp_path: Path +) -> None: + (tmp_path / "sqlbuild_project.toml").write_text(test_case.project_contents, encoding="utf-8") + + with command_warehouse_scope(command="debug", cli_warehouse=None, cli_vars=test_case.cli_vars): + result: DebugResult = build_debug_result(project_dir=tmp_path, check_connection=False) + + reported: set[tuple[str, str, str]] = { + (line.label, line.message, line.status_message or "") for line in result.lines + } + assert set(test_case.expected_lines) <= reported + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/_test_types.py b/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/_test_types.py index f847700a7d..b61ca160e2 100644 --- a/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/_test_types.py +++ b/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/_test_types.py @@ -1,4 +1,7 @@ -from dataclasses import dataclass +from dataclasses import dataclass, field + +from sqlbuild.cli.commands.types import CliCommand +from sqlbuild.spec.contracts.models import LocalTargetConfig, TargetWarehousesConfig @dataclass(frozen=True) @@ -38,3 +41,50 @@ class NamedConnectionBehaviorTestCase: description: str expected_connection: dict[str, object] expected_error_fragment: str | None = None + + +@dataclass(frozen=True) +class CommandWarehouseResolutionTestCase: + description: str + command: str | None + expected_warehouse: str | None + expected_source: str + project_warehouses: TargetWarehousesConfig = field(default_factory=TargetWarehousesConfig) + local_targets: dict[str, LocalTargetConfig] = field(default_factory=dict) + connection: dict[str, object] = field( + default_factory=lambda: {"account": "example-account", "warehouse": "ANALYTICS_WH"} + ) + cli_warehouse: str | None = None + selected_target: str | None = None + adapter: str = "snowflake" + local_adapter: str | None = None + adapter_file_contents: str = "" + + +@dataclass(frozen=True) +class CommandWarehouseErrorTestCase: + description: str + resolution: CommandWarehouseResolutionTestCase + expected_code: str + expected_message_fragment: str + + +@dataclass(frozen=True) +class WarehouseFlagParseTestCase: + description: str + argv: tuple[str, ...] + expected_warehouse: str + + +@dataclass(frozen=True) +class WarehouseFlagRejectedTestCase: + description: str + argv: tuple[str, ...] + expected_error_fragment: str = "unrecognized arguments: --warehouse" + + +@dataclass(frozen=True) +class CommandWarehouseClassificationTestCase: + description: str + command: CliCommand + expected_classified: bool = True diff --git a/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/helpers.py b/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/helpers.py new file mode 100644 index 0000000000..bd00e4aa67 --- /dev/null +++ b/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/helpers.py @@ -0,0 +1,50 @@ +"""Shared builders for command-group warehouse resolution tests.""" + +from __future__ import annotations + +from pathlib import Path + +from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.spec.contracts.models import ( + LocalConfig, + ProjectConfig, + TargetConfig, + TargetWarehousesConfig, +) +from tests.unit.src.sqlbuild.cli.commands._helpers.runtime._test_types import ( + CommandWarehouseResolutionTestCase, +) + + +def build_warehouse_discovered_inputs( + test_case: CommandWarehouseResolutionTestCase, +) -> DiscoveredProjectInputs: + """Build a `dev`/`prod` project whose `dev` target uses the case's warehouse groups.""" + + return DiscoveredProjectInputs( + project_config=ProjectConfig( + name="shop", + adapter=test_case.adapter, + default_target="dev", + connections={"main": dict(test_case.connection)}, + targets={ + "dev": TargetConfig( + connection_name="main", warehouses=test_case.project_warehouses + ), + "prod": TargetConfig( + connection_name="main", + warehouses=TargetWarehousesConfig(build="PROD_BUILD_WH"), + ), + }, + ), + local_config=LocalConfig( + adapter=test_case.local_adapter, targets=dict(test_case.local_targets) + ), + ) + + +def write_case_adapter_file(*, project_dir: Path, contents: str) -> None: + """Write the case's project-local adapter module, which may be empty.""" + + (project_dir / "adapters").mkdir(exist_ok=True) + (project_dir / "adapters" / "custom.py").write_text(contents, encoding="utf-8") diff --git a/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/test_warehouses.py b/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/test_warehouses.py new file mode 100644 index 0000000000..954c0a516e --- /dev/null +++ b/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/test_warehouses.py @@ -0,0 +1,343 @@ +"""Command-group warehouse precedence for CLI connection resolution.""" + +from __future__ import annotations + +import argparse +from pathlib import Path + +import pytest + +from sqlbuild.cli.commands._helpers.entry.parser import build_cli_parser +from sqlbuild.cli.commands._helpers.runtime.connection import resolve_project_connection_config +from sqlbuild.cli.commands._helpers.runtime.warehouses import ( + command_warehouse_scope, + describe_connection_warehouse, +) +from sqlbuild.cli.commands.constants import COMMAND_WAREHOUSE_GROUPS +from sqlbuild.cli.commands.exceptions import CliUserError +from sqlbuild.cli.commands.models import ResolvedWarehouse +from sqlbuild.cli.commands.types import CliCommand +from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.spec.contracts.models import LocalTargetConfig, TargetWarehousesConfig +from tests.unit.src.sqlbuild.cli.commands._helpers.runtime._test_types import ( + CommandWarehouseClassificationTestCase, + CommandWarehouseErrorTestCase, + CommandWarehouseResolutionTestCase, + WarehouseFlagParseTestCase, + WarehouseFlagRejectedTestCase, +) +from tests.unit.src.sqlbuild.cli.commands._helpers.runtime.helpers import ( + build_warehouse_discovered_inputs, + write_case_adapter_file, +) + +_PROJECT_GROUPS: TargetWarehousesConfig = TargetWarehousesConfig(build="BUILD_WH", query="ADHOC_WH") +_SNOWFLAKE_SUBCLASS_ADAPTER: str = ( + "from sqlbuild.adapters.snowflake.classes.snowflake_adapter import SnowflakeAdapter\n\n\n" + "class SnowflakePlusAdapter(SnowflakeAdapter):\n" + ' adapter_name = "snowflake_plus"\n' +) +_LOCAL_QUERY_OVERRIDE: dict[str, LocalTargetConfig] = { + "dev": LocalTargetConfig(warehouses=TargetWarehousesConfig(query="LOCAL_ADHOC_WH")) +} + + +@pytest.mark.parametrize( + "test_case", + [ + CommandWarehouseResolutionTestCase( + description="build uses the project build group", + command="build", + project_warehouses=_PROJECT_GROUPS, + expected_warehouse="BUILD_WH", + expected_source="sqlbuild_project.toml [targets.dev.warehouses] build", + ), + CommandWarehouseResolutionTestCase( + description="query uses the project query group", + command="query", + project_warehouses=_PROJECT_GROUPS, + expected_warehouse="ADHOC_WH", + expected_source="sqlbuild_project.toml [targets.dev.warehouses] query", + ), + CommandWarehouseResolutionTestCase( + description="local query group overrides the project for diff", + command="diff", + project_warehouses=_PROJECT_GROUPS, + local_targets=_LOCAL_QUERY_OVERRIDE, + expected_warehouse="LOCAL_ADHOC_WH", + expected_source="sqlbuild_local.toml [targets.dev.warehouses] query", + ), + CommandWarehouseResolutionTestCase( + description="local query override leaves the project build group for plan", + command="plan", + project_warehouses=_PROJECT_GROUPS, + local_targets=_LOCAL_QUERY_OVERRIDE, + expected_warehouse="BUILD_WH", + expected_source="sqlbuild_project.toml [targets.dev.warehouses] build", + ), + CommandWarehouseResolutionTestCase( + description="cli warehouse overrides the group", + command="build", + project_warehouses=_PROJECT_GROUPS, + cli_warehouse="CLI_WH", + expected_warehouse="CLI_WH", + expected_source="--warehouse", + ), + CommandWarehouseResolutionTestCase( + description="unset group falls back to the connection warehouse", + command="build", + project_warehouses=TargetWarehousesConfig(query="ADHOC_WH"), + expected_warehouse="ANALYTICS_WH", + expected_source="connection warehouse", + ), + CommandWarehouseResolutionTestCase( + description="no configured warehouse keeps the connection default", + command="query", + connection={"account": "example-account"}, + expected_warehouse=None, + expected_source="connection default", + ), + CommandWarehouseResolutionTestCase( + description="command without a group keeps the connection warehouse", + command="compile", + project_warehouses=_PROJECT_GROUPS, + expected_warehouse="ANALYTICS_WH", + expected_source="connection warehouse", + ), + CommandWarehouseResolutionTestCase( + description="resolution outside a cli command keeps the connection warehouse", + command=None, + project_warehouses=_PROJECT_GROUPS, + expected_warehouse="ANALYTICS_WH", + expected_source="connection warehouse", + ), + CommandWarehouseResolutionTestCase( + description="selected target uses its own groups", + command="build", + project_warehouses=_PROJECT_GROUPS, + selected_target="prod", + expected_warehouse="PROD_BUILD_WH", + expected_source="sqlbuild_project.toml [targets.prod.warehouses] build", + ), + CommandWarehouseResolutionTestCase( + description="custom adapter extending snowflake applies groups", + command="build", + adapter="snowflake_plus", + adapter_file_contents=_SNOWFLAKE_SUBCLASS_ADAPTER, + project_warehouses=_PROJECT_GROUPS, + expected_warehouse="BUILD_WH", + expected_source="sqlbuild_project.toml [targets.dev.warehouses] build", + ), + CommandWarehouseResolutionTestCase( + description="local duckdb override ignores shared groups without error", + command="query", + local_adapter="duckdb", + project_warehouses=TargetWarehousesConfig( + build="BUILD_WH", query="${ENV:SQB_EXAMPLE_MISSING_QWH}" + ), + expected_warehouse="ANALYTICS_WH", + expected_source="connection warehouse", + ), + CommandWarehouseResolutionTestCase( + description="double-quoted group identifier is kept", + command="query", + project_warehouses=TargetWarehousesConfig(query='"Adhoc WH"'), + expected_warehouse='"Adhoc WH"', + expected_source="sqlbuild_project.toml [targets.dev.warehouses] query", + ), + CommandWarehouseResolutionTestCase( + description="environment template in a group is expanded", + command="query", + project_warehouses=TargetWarehousesConfig(query="${ENV:SQB_EXAMPLE_ADHOC_WH}"), + expected_warehouse="ENV_ADHOC_WH", + expected_source="sqlbuild_project.toml [targets.dev.warehouses] query", + ), + ], + ids=lambda case: case.description, +) +def test_given_command_group_when_resolving_connection_then_warehouse_follows_precedence( + test_case: CommandWarehouseResolutionTestCase, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("SQB_EXAMPLE_ADHOC_WH", "ENV_ADHOC_WH") + monkeypatch.delenv("SQB_EXAMPLE_MISSING_QWH", raising=False) + write_case_adapter_file(project_dir=tmp_path, contents=test_case.adapter_file_contents) + discovered_inputs: DiscoveredProjectInputs = build_warehouse_discovered_inputs(test_case) + + with command_warehouse_scope(command=test_case.command, cli_warehouse=test_case.cli_warehouse): + connection: dict[str, object] = resolve_project_connection_config( + discovered_inputs=discovered_inputs, + project_dir=tmp_path, + selected_target=test_case.selected_target, + ) + described: ResolvedWarehouse = describe_connection_warehouse( + discovered_inputs=discovered_inputs, + selected_target=test_case.selected_target, + connection_config=connection, + ) + + assert connection.get("warehouse") == test_case.expected_warehouse + assert described == ResolvedWarehouse( + warehouse=test_case.expected_warehouse, source=test_case.expected_source + ) + + +@pytest.mark.parametrize( + "test_case", + [ + CommandWarehouseErrorTestCase( + description="duckdb rejects a cli warehouse", + resolution=CommandWarehouseResolutionTestCase( + description="duckdb", + command="query", + adapter="duckdb", + connection={"database": ":memory:"}, + cli_warehouse="ADHOC_WH", + expected_warehouse=None, + expected_source="connection default", + ), + expected_code="C260", + expected_message_fragment="'duckdb' adapter has no warehouse to select", + ), + CommandWarehouseErrorTestCase( + description="injected group warehouse is rejected", + resolution=CommandWarehouseResolutionTestCase( + description="injected group", + command="query", + project_warehouses=TargetWarehousesConfig(query="x; drop table t"), + expected_warehouse=None, + expected_source="", + ), + expected_code="C261", + expected_message_fragment=( + "[targets.dev.warehouses] query resolves to 'x; drop table t', which is not a " + "valid Snowflake warehouse identifier" + ), + ), + CommandWarehouseErrorTestCase( + description="injected cli warehouse is rejected", + resolution=CommandWarehouseResolutionTestCase( + description="injected cli", + command="build", + cli_warehouse="x; drop table t", + expected_warehouse=None, + expected_source="", + ), + expected_code="C261", + expected_message_fragment="--warehouse 'x; drop table t' is not a valid Snowflake", + ), + CommandWarehouseErrorTestCase( + description="unterminated quoted cli warehouse is rejected", + resolution=CommandWarehouseResolutionTestCase( + description="unterminated quote", + command="build", + cli_warehouse='"ADHOC', + expected_warehouse=None, + expected_source="", + ), + expected_code="C261", + expected_message_fragment="is not a valid Snowflake warehouse identifier", + ), + ], + ids=lambda case: case.description, +) +def test_given_adapter_without_warehouses_when_cli_warehouse_is_given_then_it_fails( + test_case: CommandWarehouseErrorTestCase, tmp_path: Path +) -> None: + discovered_inputs: DiscoveredProjectInputs = build_warehouse_discovered_inputs( + test_case.resolution + ) + + with ( + command_warehouse_scope( + command=test_case.resolution.command, + cli_warehouse=test_case.resolution.cli_warehouse, + ), + pytest.raises(CliUserError) as error, + ): + resolve_project_connection_config(discovered_inputs=discovered_inputs, project_dir=tmp_path) + + assert error.value.code == test_case.expected_code + assert test_case.expected_message_fragment in error.value.message + + +@pytest.mark.parametrize( + "test_case", + [ + CommandWarehouseClassificationTestCase(description=command.value, command=command) + for command in CliCommand + ], + ids=lambda case: case.description, +) +def test_given_cli_command_when_checking_warehouse_groups_then_it_is_classified( + test_case: CommandWarehouseClassificationTestCase, +) -> None: + assert (test_case.command in COMMAND_WAREHOUSE_GROUPS) == test_case.expected_classified + + +@pytest.mark.parametrize( + "test_case", + [ + WarehouseFlagParseTestCase( + description="build", argv=("build", "--warehouse", "W"), expected_warehouse="W" + ), + WarehouseFlagParseTestCase( + description="plan", argv=("plan", "--warehouse", "W"), expected_warehouse="W" + ), + WarehouseFlagParseTestCase( + description="query", + argv=("query", "SELECT 1", "--warehouse", "W"), + expected_warehouse="W", + ), + WarehouseFlagParseTestCase( + description="diff", argv=("diff", "--warehouse", "W"), expected_warehouse="W" + ), + WarehouseFlagParseTestCase( + description="debug", argv=("debug", "--warehouse", "W"), expected_warehouse="W" + ), + WarehouseFlagParseTestCase( + description="contract subcommand", + argv=("contract", "diff", "--from", "prod", "--warehouse", "W"), + expected_warehouse="W", + ), + WarehouseFlagParseTestCase( + description="scenario subcommand", + argv=("scenario", "test", "--warehouse", "W"), + expected_warehouse="W", + ), + ], + ids=lambda case: case.description, +) +def test_given_grouped_command_when_parsing_warehouse_flag_then_value_is_kept( + test_case: WarehouseFlagParseTestCase, +) -> None: + parsed: argparse.Namespace = build_cli_parser().parse_args(list(test_case.argv)) + + assert parsed.warehouse == test_case.expected_warehouse + + +@pytest.mark.parametrize( + "test_case", + [ + WarehouseFlagRejectedTestCase( + description="compile never connects", argv=("compile", "--warehouse", "W") + ), + WarehouseFlagRejectedTestCase( + description="lineage never connects", argv=("lineage", "--warehouse", "W") + ), + ], + ids=lambda case: case.description, +) +def test_given_command_without_group_when_parsing_warehouse_flag_then_it_is_rejected( + test_case: WarehouseFlagRejectedTestCase, + capsys: pytest.CaptureFixture[str], +) -> None: + with pytest.raises(SystemExit): + build_cli_parser().parse_args(list(test_case.argv)) + + assert test_case.expected_error_fragment in capsys.readouterr().err + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/unit/src/sqlbuild/compiler/discovery/_helpers/_test_types.py b/tests/unit/src/sqlbuild/compiler/discovery/_helpers/_test_types.py index c85fe6b18e..5929a72e83 100644 --- a/tests/unit/src/sqlbuild/compiler/discovery/_helpers/_test_types.py +++ b/tests/unit/src/sqlbuild/compiler/discovery/_helpers/_test_types.py @@ -2,7 +2,7 @@ from decimal import Decimal from pathlib import Path -from sqlbuild.spec.contracts.models import AuthoredTimeTravelRetention +from sqlbuild.spec.contracts.models import AuthoredTimeTravelRetention, TargetWarehousesConfig from sqlbuild.spec.contracts.types import MissingMigrationOriginPolicy, RetentionDecreasePolicy from sqlbuild.sql_values.types import CollectionRendering @@ -979,3 +979,29 @@ class NativeHeaderMatchTestCase: seed: int count: int expected_mismatches: tuple[str, ...] = () + + +@dataclass(frozen=True) +class TargetWarehousesConfigTestCase: + description: str + contents: str + expected_warehouses: TargetWarehousesConfig + + +@dataclass(frozen=True) +class TargetWarehousesConfigErrorTestCase: + description: str + project_contents: str + expected_error_fragments: tuple[str, ...] + local_contents: str = "" + expected_help_fragments: tuple[str, ...] = () + adapter_file_contents: str = "" + + +@dataclass(frozen=True) +class TargetWarehousesDiscoveryTestCase: + description: str + project_contents: str + expected_build: str | None + local_contents: str = "" + adapter_file_contents: str = "" diff --git a/tests/unit/src/sqlbuild/compiler/discovery/_helpers/test_target_warehouses.py b/tests/unit/src/sqlbuild/compiler/discovery/_helpers/test_target_warehouses.py new file mode 100644 index 0000000000..a753905b74 --- /dev/null +++ b/tests/unit/src/sqlbuild/compiler/discovery/_helpers/test_target_warehouses.py @@ -0,0 +1,245 @@ +"""Loading and validation of per-target command-group warehouses.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from sqlbuild.compiler.discovery._helpers.yml.project import ( + load_local_config, + load_project_config, +) +from sqlbuild.compiler.discovery.exceptions import ProjectConfigError +from sqlbuild.compiler.discovery.main.discover import discover_project_inputs +from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.spec.contracts.models import TargetWarehousesConfig +from tests.unit.src.sqlbuild.compiler.discovery._helpers._test_types import ( + TargetWarehousesConfigErrorTestCase, + TargetWarehousesConfigTestCase, + TargetWarehousesDiscoveryTestCase, +) + +_SNOWFLAKE_PROJECT: str = """ +name = "shop" +adapter = "snowflake" + +[connections.main] +warehouse = "ANALYTICS_WH" + +[targets.dev] +connection = "main" +schema = "DEV" +""".strip() +_SHARED_GROUPS: str = '\n\n[targets.dev.warehouses]\nbuild = "BUILD_WH"\nquery = "ADHOC_WH"\n' +_LOCAL_DUCKDB_OVERRIDE: str = ( + 'adapter = "duckdb"\n\n[connections.local]\ndatabase = "dev.duckdb"\n\n' + '[targets.dev]\nconnection = "local"\n' +) +_SNOWFLAKE_SUBCLASS_ADAPTER: str = """ +from sqlbuild.adapters.snowflake.classes.snowflake_adapter import SnowflakeAdapter + + +class SnowflakePlusAdapter(SnowflakeAdapter): + adapter_name = "snowflake_plus" +""".lstrip() +_DUCKDB_SUBCLASS_ADAPTER: str = """ +from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter + + +class DuckDbPlusAdapter(DuckDbAdapter): + adapter_name = "duckdb_plus" +""".lstrip() + + +@pytest.mark.parametrize( + "test_case", + [ + TargetWarehousesConfigTestCase( + description="project target sets both groups", + contents=_SNOWFLAKE_PROJECT + + '\n\n[targets.dev.warehouses]\nbuild = "BUILD_WH"\nquery = "ADHOC_WH"\n', + expected_warehouses=TargetWarehousesConfig(build="BUILD_WH", query="ADHOC_WH"), + ), + TargetWarehousesConfigTestCase( + description="target without warehouses keeps both groups unset", + contents=_SNOWFLAKE_PROJECT, + expected_warehouses=TargetWarehousesConfig(), + ), + ], + ids=lambda case: case.description, +) +def test_given_project_target_warehouses_when_loading_then_groups_are_typed( + test_case: TargetWarehousesConfigTestCase, tmp_path: Path +) -> None: + (tmp_path / "sqlbuild_project.toml").write_text(test_case.contents, encoding="utf-8") + + config_warehouses: TargetWarehousesConfig = ( + load_project_config(project_dir=tmp_path).targets["dev"].warehouses + ) + + assert config_warehouses == test_case.expected_warehouses + + +@pytest.mark.parametrize( + "test_case", + [ + TargetWarehousesConfigTestCase( + description="local target sets only the query group", + contents='[targets.dev.warehouses]\nquery = "ADHOC_WH"\n', + expected_warehouses=TargetWarehousesConfig(query="ADHOC_WH"), + ), + ], + ids=lambda case: case.description, +) +def test_given_local_target_warehouses_when_loading_then_groups_are_typed( + test_case: TargetWarehousesConfigTestCase, tmp_path: Path +) -> None: + (tmp_path / "sqlbuild_local.toml").write_text(test_case.contents, encoding="utf-8") + + config_warehouses: TargetWarehousesConfig = ( + load_local_config(project_dir=tmp_path).targets["dev"].warehouses + ) + + assert config_warehouses == test_case.expected_warehouses + + +@pytest.mark.parametrize( + "test_case", + [ + TargetWarehousesConfigErrorTestCase( + description="unknown group key is rejected with a suggestion", + project_contents=_SNOWFLAKE_PROJECT + + '\n\n[targets.dev.warehouses]\nbiuld = "BUILD_WH"\n', + expected_error_fragments=( + "targets.dev.warehouses has unknown keys: biuld; allowed keys: build, query", + ), + expected_help_fragments=("did you mean 'build'?",), + ), + TargetWarehousesConfigErrorTestCase( + description="unknown group key in the local file is rejected", + project_contents=_SNOWFLAKE_PROJECT, + local_contents='[targets.dev.warehouses]\nadhoc = "ADHOC_WH"\n', + expected_error_fragments=("targets.dev.warehouses has unknown keys: adhoc",), + ), + TargetWarehousesConfigErrorTestCase( + description="blank warehouse name is rejected", + project_contents=_SNOWFLAKE_PROJECT + '\n\n[targets.dev.warehouses]\nquery = " "\n', + expected_error_fragments=( + "targets.dev.warehouses.query must be a non-empty warehouse name", + ), + ), + TargetWarehousesConfigErrorTestCase( + description="adapter without warehouses rejects project groups", + project_contents=""" +name = "shop" +adapter = "duckdb" + +[targets.dev] +schema = "dev" + +[targets.dev.warehouses] +build = "BUILD_WH" +""".strip(), + expected_error_fragments=( + "[targets.dev.warehouses] selects a build warehouse, but the 'duckdb' adapter " + "has no warehouse to select", + 'sqlbuild_project.toml sets [targets.dev.warehouses] build = "BUILD_WH"', + ), + expected_help_fragments=( + "remove the [targets.dev.warehouses] section from sqlbuild_project.toml", + "need an adapter with a session warehouse", + ), + ), + TargetWarehousesConfigErrorTestCase( + description="adapter without warehouses rejects local groups", + project_contents=( + 'name = "shop"\nadapter = "postgres"\n\n[targets.dev]\nschema = "dev"\n' + ), + local_contents='[targets.dev.warehouses]\nquery = "ADHOC_WH"\n', + expected_error_fragments=( + "the 'postgres' adapter has no warehouse to select", + 'sqlbuild_local.toml sets [targets.dev.warehouses] query = "ADHOC_WH"', + ), + expected_help_fragments=( + "remove the [targets.dev.warehouses] section from sqlbuild_local.toml", + ), + ), + TargetWarehousesConfigErrorTestCase( + description="local adapter override rejects local groups", + project_contents=_SNOWFLAKE_PROJECT, + local_contents=_LOCAL_DUCKDB_OVERRIDE + + '\n[targets.dev.warehouses]\nquery = "ALICE_WH"\n', + expected_error_fragments=( + "the 'duckdb' adapter has no warehouse to select", + 'sqlbuild_local.toml sets [targets.dev.warehouses] query = "ALICE_WH"', + ), + expected_help_fragments=( + "remove the [targets.dev.warehouses] section from sqlbuild_local.toml", + ), + ), + TargetWarehousesConfigErrorTestCase( + description="custom adapter extending duckdb rejects shared groups", + project_contents=_SNOWFLAKE_PROJECT.replace('"snowflake"', '"duckdb_plus"') + + _SHARED_GROUPS, + adapter_file_contents=_DUCKDB_SUBCLASS_ADAPTER, + expected_error_fragments=("the 'duckdb_plus' adapter has no warehouse to select",), + ), + ], + ids=lambda case: case.description, +) +def test_given_invalid_target_warehouses_when_discovering_then_config_error_explains_fix( + test_case: TargetWarehousesConfigErrorTestCase, tmp_path: Path +) -> None: + (tmp_path / "sqlbuild_project.toml").write_text(test_case.project_contents, encoding="utf-8") + (tmp_path / "sqlbuild_local.toml").write_text(test_case.local_contents, encoding="utf-8") + (tmp_path / "adapters").mkdir() + (tmp_path / "adapters" / "custom.py").write_text( + test_case.adapter_file_contents, encoding="utf-8" + ) + + with pytest.raises(ProjectConfigError) as error: + discover_project_inputs(project_dir=tmp_path) + + assert all(fragment in str(error.value) for fragment in test_case.expected_error_fragments) + assert all( + fragment in (error.value.help or "") for fragment in test_case.expected_help_fragments + ) + + +@pytest.mark.parametrize( + "test_case", + [ + TargetWarehousesDiscoveryTestCase( + description="local duckdb override leaves shared groups unused", + project_contents=_SNOWFLAKE_PROJECT + _SHARED_GROUPS, + local_contents=_LOCAL_DUCKDB_OVERRIDE, + expected_build="BUILD_WH", + ), + TargetWarehousesDiscoveryTestCase( + description="custom adapter extending snowflake accepts shared groups", + project_contents=_SNOWFLAKE_PROJECT.replace('"snowflake"', '"snowflake_plus"') + + _SHARED_GROUPS, + adapter_file_contents=_SNOWFLAKE_SUBCLASS_ADAPTER, + expected_build="BUILD_WH", + ), + ], + ids=lambda case: case.description, +) +def test_given_adapter_layers_when_discovering_target_warehouses_then_config_is_accepted( + test_case: TargetWarehousesDiscoveryTestCase, tmp_path: Path +) -> None: + (tmp_path / "sqlbuild_project.toml").write_text(test_case.project_contents, encoding="utf-8") + (tmp_path / "sqlbuild_local.toml").write_text(test_case.local_contents, encoding="utf-8") + (tmp_path / "adapters").mkdir() + (tmp_path / "adapters" / "custom.py").write_text( + test_case.adapter_file_contents, encoding="utf-8" + ) + + discovered: DiscoveredProjectInputs = discover_project_inputs(project_dir=tmp_path) + + assert discovered.project_config.targets["dev"].warehouses.build == test_case.expected_build + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/unit/src/sqlbuild/spec/contracts/main/_test_types.py b/tests/unit/src/sqlbuild/spec/contracts/main/_test_types.py index bd195c7ca4..906d7a16ce 100644 --- a/tests/unit/src/sqlbuild/spec/contracts/main/_test_types.py +++ b/tests/unit/src/sqlbuild/spec/contracts/main/_test_types.py @@ -8,6 +8,7 @@ LoaderDestinationParts, LocalConfig, ProjectConfig, + TargetWarehousesConfig, ) from sqlbuild.spec.contracts.types import MissingMigrationOriginPolicy from sqlbuild.sql_values.types import CollectionRendering @@ -65,3 +66,11 @@ class MissingOriginPolicyResolutionTestCase: local_config: LocalConfig target_name: str expected_policy: MissingMigrationOriginPolicy + + +@dataclass(frozen=True) +class TargetWarehousesResolutionTestCase: + description: str + project_warehouses: TargetWarehousesConfig + local_warehouses: TargetWarehousesConfig + expected_warehouses: TargetWarehousesConfig diff --git a/tests/unit/src/sqlbuild/spec/contracts/main/test_targets.py b/tests/unit/src/sqlbuild/spec/contracts/main/test_targets.py index a01f23747d..5e566ea9b2 100644 --- a/tests/unit/src/sqlbuild/spec/contracts/main/test_targets.py +++ b/tests/unit/src/sqlbuild/spec/contracts/main/test_targets.py @@ -10,12 +10,14 @@ LocalTargetConfig, ProjectConfig, TargetConfig, + TargetWarehousesConfig, ) from sqlbuild.spec.contracts.types import MissingMigrationOriginPolicy from tests.unit.src.sqlbuild.spec.contracts.main._test_types import ( ExecutionLimitsResolutionTestCase, MissingOriginPolicyResolutionTestCase, TargetRetentionResolutionTestCase, + TargetWarehousesResolutionTestCase, ) @@ -188,3 +190,39 @@ def test_given_missing_origin_policy_when_resolving_then_local_overrides_project ) assert target_config.missing_migration_origin is test_case.expected_policy + + +@pytest.mark.parametrize( + "test_case", + [ + TargetWarehousesResolutionTestCase( + description="local query group overrides only the query group", + project_warehouses=TargetWarehousesConfig(build="BUILD_WH", query="ADHOC_WH"), + local_warehouses=TargetWarehousesConfig(query="LOCAL_ADHOC_WH"), + expected_warehouses=TargetWarehousesConfig(build="BUILD_WH", query="LOCAL_ADHOC_WH"), + ), + TargetWarehousesResolutionTestCase( + description="local groups apply when the project sets none", + project_warehouses=TargetWarehousesConfig(), + local_warehouses=TargetWarehousesConfig(build="LOCAL_BUILD_WH"), + expected_warehouses=TargetWarehousesConfig(build="LOCAL_BUILD_WH"), + ), + ], + ids=lambda case: case.description, +) +def test_given_local_target_warehouses_when_resolving_then_each_group_overrides_project( + test_case: TargetWarehousesResolutionTestCase, +) -> None: + target_config: TargetConfig = resolve_target_config( + project_config=ProjectConfig( + name="shop", + adapter="snowflake", + targets={"dev": TargetConfig(warehouses=test_case.project_warehouses)}, + ), + local_config=LocalConfig( + targets={"dev": LocalTargetConfig(warehouses=test_case.local_warehouses)} + ), + target_name="dev", + ) + + assert target_config.warehouses == test_case.expected_warehouses diff --git a/website/src/content/docs/docs/cli/audit.mdx b/website/src/content/docs/docs/cli/audit.mdx index ef530c0f20..02b3102f8a 100644 --- a/website/src/content/docs/docs/cli/audit.mdx +++ b/website/src/content/docs/docs/cli/audit.mdx @@ -25,6 +25,7 @@ sqb --project-dir audit [flags] | `--concurrency` | Maximum number of audits to run concurrently (default: `1`) | | `--json` | Print structured audit results | | `--json-output` | Write structured audit results to a file | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ## Concurrency diff --git a/website/src/content/docs/docs/cli/build.mdx b/website/src/content/docs/docs/cli/build.mdx index b58447633a..57db524614 100644 --- a/website/src/content/docs/docs/cli/build.mdx +++ b/website/src/content/docs/docs/cli/build.mdx @@ -41,6 +41,7 @@ sqb --project-dir build [flags] | `--manifest` | Generate `target/manifest.json` with plan-aware project metadata | | `--select`, `-s` | Select specific models | | `--exclude` | Exclude specific models | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ## Fast iteration diff --git a/website/src/content/docs/docs/cli/check.mdx b/website/src/content/docs/docs/cli/check.mdx index 3707cbe444..9ff49c8cae 100644 --- a/website/src/content/docs/docs/cli/check.mdx +++ b/website/src/content/docs/docs/cli/check.mdx @@ -23,6 +23,7 @@ sqb --project-dir check [flags] | `--json` | Print check results as JSON | | `--json-output` | Write check results JSON to a file path | | `--vars` | Override project variables | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | Selecting a non-check node is rejected; use [`sqb build`](/docs/cli/build) to run tasks and assets. diff --git a/website/src/content/docs/docs/cli/clone.mdx b/website/src/content/docs/docs/cli/clone.mdx index d7bc526222..1819bf15d1 100644 --- a/website/src/content/docs/docs/cli/clone.mdx +++ b/website/src/content/docs/docs/cli/clone.mdx @@ -35,6 +35,7 @@ sqb --project-dir clone --from [--to ] [flags] | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | | `--select`, `-s` | Select specific models to clone | | `--exclude` | Exclude specific models from cloning | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/website/src/content/docs/docs/cli/contract.mdx b/website/src/content/docs/docs/cli/contract.mdx index d228191976..e1eee35090 100644 --- a/website/src/content/docs/docs/cli/contract.mdx +++ b/website/src/content/docs/docs/cli/contract.mdx @@ -36,6 +36,7 @@ replace conflicts rather than preserving authored declarations. | `--vars` | both | Override project variables | | `--write` | generate | Apply generated repository changes | | `--overwrite` | generate | Replace conflicting declarations; requires `--write` | +| `--warehouse ` | both | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | See [Contracts](/docs/concepts/models/contracts#adopting-an-existing-schema) for ownership and safety guarantees. diff --git a/website/src/content/docs/docs/cli/dbt.mdx b/website/src/content/docs/docs/cli/dbt.mdx index b04633f72b..49e6e0a693 100644 --- a/website/src/content/docs/docs/cli/dbt.mdx +++ b/website/src/content/docs/docs/cli/dbt.mdx @@ -90,3 +90,8 @@ target_path = "../dbt_project/target" ``` See [Project Configuration](/docs/concepts/project-configuration#dbt) for all fields. + +SQLBuild-owned work in `sqb dbt plan`, `run` and `build` connects with the target's `build` +[warehouse group](/docs/concepts/project-configuration#command-group-warehouses). dbt itself +keeps using the warehouse from its profile. These commands do not accept `--warehouse`, because +their remaining arguments pass through to dbt. diff --git a/website/src/content/docs/docs/cli/debug.mdx b/website/src/content/docs/docs/cli/debug.mdx index 1c83543315..d240abffd2 100644 --- a/website/src/content/docs/docs/cli/debug.mdx +++ b/website/src/content/docs/docs/cli/debug.mdx @@ -17,9 +17,9 @@ The command runs three groups of checks: **Runtime** - SQLBuild version, Python version, Python path, OS info. -**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter and active target. +**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter and active target. For Snowflake it also shows the warehouse each [command group](/docs/concepts/project-configuration#command-group-warehouses) connects with and where that value came from. -**Connection** - Displays connection settings (secrets are masked), attempts to connect to the warehouse, and runs `SELECT 1` to verify query execution. +**Connection** - Displays connection settings as `sqb debug` itself uses them, with the `query` warehouse group applied (secrets are masked), attempts to connect to the warehouse, and runs `SELECT 1` to verify query execution. ## Flags @@ -27,6 +27,7 @@ The command runs three groups of checks: |------|-------------| | `--no-connection` | Skip the connection and query tests. Useful for validating config without warehouse access. | | `--json` | Output results as JSON instead of formatted text. | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ## Example @@ -49,6 +50,8 @@ Configuration: project: waffle_shop [OK loaded] adapter: snowflake [OK found] target: dev [OK resolved] + build warehouse: BUILD_WH [OK sqlbuild_project.toml [targets.dev.warehouses] build] + query warehouse: SQB_WH [OK connection warehouse] Connection: account: FJMQFQV-OJ66172 diff --git a/website/src/content/docs/docs/cli/diff.mdx b/website/src/content/docs/docs/cli/diff.mdx index d30b094389..4c2768d785 100644 --- a/website/src/content/docs/docs/cli/diff.mdx +++ b/website/src/content/docs/docs/cli/diff.mdx @@ -57,6 +57,7 @@ In query mode, omit `FROM:TO` and model selectors; each query identifies its own | `--json` | Print one JSON document to stdout | | `--select`, `-s` | Select specific models to diff (required in model mode) | | `--exclude` | Exclude specific models from diffing | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/website/src/content/docs/docs/cli/freshness.mdx b/website/src/content/docs/docs/cli/freshness.mdx index 9938ad5f72..36325597da 100644 --- a/website/src/content/docs/docs/cli/freshness.mdx +++ b/website/src/content/docs/docs/cli/freshness.mdx @@ -25,6 +25,7 @@ sqb --project-dir freshness [flags] | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | | `--select`, `-s` | Select specific sources or models (sources upstream of selected models are included) | | `--exclude` | Exclude specific sources or models | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ## Source selection diff --git a/website/src/content/docs/docs/cli/janitor.mdx b/website/src/content/docs/docs/cli/janitor.mdx index 70df8b633b..83d87ad7b2 100644 --- a/website/src/content/docs/docs/cli/janitor.mdx +++ b/website/src/content/docs/docs/cli/janitor.mdx @@ -23,6 +23,7 @@ sqb --project-dir janitor [flags] | `--retention-days` | Override the configured retention period (days) before a stale relation is archived | | `--direct-state-history-versions` | Override how many state-history versions are kept per identity | | `--drop-old-name-view ` | Drop the [compatibility view](/docs/concepts/models/migrations#old-names) at a renamed model's old name before it expires. Repeat for several views. Fails with `C503` if no compatibility view is recorded at the name | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ## Targets diff --git a/website/src/content/docs/docs/cli/load.mdx b/website/src/content/docs/docs/cli/load.mdx index 9b2c6bdbd2..b2303c114e 100644 --- a/website/src/content/docs/docs/cli/load.mdx +++ b/website/src/content/docs/docs/cli/load.mdx @@ -28,6 +28,7 @@ sqb --project-dir load [flags] | `--json` | Output results as JSON | | `--json-output` | Write JSON results to a file path | | `--var` | Set project variables (`--var key=value`) | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/website/src/content/docs/docs/cli/plan.mdx b/website/src/content/docs/docs/cli/plan.mdx index e7f024217d..b2321f692b 100644 --- a/website/src/content/docs/docs/cli/plan.mdx +++ b/website/src/content/docs/docs/cli/plan.mdx @@ -30,6 +30,7 @@ sqb --project-dir plan [flags] | `--end-cursor-int` | Override end cursor for integer incremental models | | `--select`, `-s` | Select specific models | | `--exclude` | Exclude specific models | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ## Output diff --git a/website/src/content/docs/docs/cli/query.mdx b/website/src/content/docs/docs/cli/query.mdx index 77334b72d9..5a4a68f7d9 100644 --- a/website/src/content/docs/docs/cli/query.mdx +++ b/website/src/content/docs/docs/cli/query.mdx @@ -30,6 +30,7 @@ File paths are resolved from the current working directory. | `--format` | Output format: `long` (default), `table`, `json`, or `csv` | | `--limit` | Maximum rows to return (default: 20) | | `--no-limit` | Disable the row limit | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ## Output formats diff --git a/website/src/content/docs/docs/cli/scenario.mdx b/website/src/content/docs/docs/cli/scenario.mdx index 46a717ec17..0cc83eb374 100644 --- a/website/src/content/docs/docs/cli/scenario.mdx +++ b/website/src/content/docs/docs/cli/scenario.mdx @@ -33,6 +33,7 @@ sqb scenario test [flags] | `--max-snapshot-bytes` | Override per-relation byte limit for capture | | `--max-snapshot-total-bytes` | Override total byte limit for capture | | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ### Selectors @@ -123,6 +124,7 @@ sqb scenario capture [flags] | `--max-snapshot-bytes` | Override per-relation byte limit | | `--max-snapshot-total-bytes` | Override total byte limit | | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ### Examples diff --git a/website/src/content/docs/docs/cli/seed.mdx b/website/src/content/docs/docs/cli/seed.mdx index 739fc20003..9094c1258a 100644 --- a/website/src/content/docs/docs/cli/seed.mdx +++ b/website/src/content/docs/docs/cli/seed.mdx @@ -19,6 +19,7 @@ sqb --project-dir seed [flags] |------|-------------| | `--select`, `-s` | Select specific seeds by name | | `--exclude` | Exclude specific seeds | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | ## Examples diff --git a/website/src/content/docs/docs/cli/test.mdx b/website/src/content/docs/docs/cli/test.mdx index 0d1e73d882..560114ae8c 100644 --- a/website/src/content/docs/docs/cli/test.mdx +++ b/website/src/content/docs/docs/cli/test.mdx @@ -21,6 +21,7 @@ sqb --project-dir test [flags] | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | | `--select`, `-s` | Select tests targeting specific models | | `--exclude` | Exclude tests targeting specific models | +| `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | Parameterized cases use parent-level selection. Selecting a target model includes every case in each matching `TEST` template; case names are not model selectors. diff --git a/website/src/content/docs/docs/concepts/adapters.mdx b/website/src/content/docs/docs/concepts/adapters.mdx index 82a506cbae..53da54eb6f 100644 --- a/website/src/content/docs/docs/concepts/adapters.mdx +++ b/website/src/content/docs/docs/concepts/adapters.mdx @@ -62,6 +62,12 @@ class DuckDbPlusAdapter(DuckDbAdapter): adapter = "duckdb_plus" ``` +A subclass inherits the parent's capabilities. For example, an adapter extending the Snowflake +adapter keeps `supports_session_warehouse = True`, so it can use +[command-group warehouses](/docs/concepts/project-configuration#command-group-warehouses) and +`--warehouse`. An adapter written from scratch that sets this attribute receives the selected +warehouse as the connection's `warehouse` key. + ### Writing an adapter from scratch For a database engine with no built-in support, subclass `BaseAdapter`. It provides ANSI SQL defaults for most methods - you only need to implement `connect`, `execute`, `close`, and any methods where your engine differs from standard SQL: diff --git a/website/src/content/docs/docs/concepts/adapters/snowflake.mdx b/website/src/content/docs/docs/concepts/adapters/snowflake.mdx index 389c7149cb..83d4814dae 100644 --- a/website/src/content/docs/docs/concepts/adapters/snowflake.mdx +++ b/website/src/content/docs/docs/concepts/adapters/snowflake.mdx @@ -38,6 +38,11 @@ On connect, SQLBuild applies the connection's role and warehouse, then applies t target's authoritative database and schema. These ensure the session context is set correctly regardless of the user's default settings. +The warehouse can differ by command. A target's `[targets..warehouses]` section selects +one warehouse for build-style commands and another for ad-hoc commands such as `sqb query` and +`sqb diff`, and `--warehouse` overrides both for one invocation. See +[command-group warehouses](/docs/concepts/project-configuration#command-group-warehouses). + ## Planning metadata Planning reads relation and column metadata with `SHOW TABLES`, `SHOW VIEWS`, and `SHOW COLUMNS` diff --git a/website/src/content/docs/docs/concepts/project-configuration.mdx b/website/src/content/docs/docs/concepts/project-configuration.mdx index 59942120a3..1af7573666 100644 --- a/website/src/content/docs/docs/concepts/project-configuration.mdx +++ b/website/src/content/docs/docs/concepts/project-configuration.mdx @@ -130,6 +130,7 @@ A target is a named build context - the database and schema you build into, plus | `defer_sources_to` | Target name to read managed source data from (see [Loaders](/docs/concepts/python-nodes/loaders#source-deferral)) | | `clone` | Clone policy (see below) | | `missing_migration_origin` | What a build does when a declared `migrate_from` origin does not exist in this target: `allow` (default), `require_confirmation`, or `deny` (see below) | +| `warehouses` | Default Snowflake warehouse per command group, `build` and `query` (see below) | ```toml [connections.warehouse] @@ -195,6 +196,76 @@ password = "${ENV:SNOWFLAKE_PASSWORD}" Commands then use `dev` automatically. An explicit command such as `sqb build --target prod` still takes precedence for that invocation. +### Command-group warehouses + +Builds and ad-hoc inspection usually need different compute. A Snowflake target can choose a +default warehouse for each command group, so `sqb build` runs on a large warehouse while +`sqb query` and `sqb diff` stay on a small one: + +```toml +[connections.warehouse] +account = "my_org-my_account" +warehouse = "ANALYTICS_WH" # fallback for every command, as before + +[targets.dev] +connection = "warehouse" +database = "ANALYTICS" +schema = "DEV" + +[targets.dev.warehouses] +build = "BUILD_WH" # commands in the build group +query = "ADHOC_WH" # commands in the query group +``` + +Each command connects with the first warehouse set in this order: + +1. `--warehouse ` on the command line, for that invocation only +2. The target's group warehouse, `[targets..warehouses] build` or `query` +3. The connection's `warehouse` +4. The Snowflake user's default warehouse + +Both groups are optional. A target without `warehouses` behaves exactly as before. A group +warehouse takes precedence over the connection `warehouse` in every file. In particular, a +shared group overrides a connection `warehouse` you set in `sqlbuild_local.toml`; to change a +group locally, set the group itself, or pass `--warehouse` for one invocation. Group values +accept the same `${ENV:...}` and project-variable (`${my_var}`) templates as connection values. +Group values and `--warehouse` must be Snowflake identifiers: unquoted (`ADHOC_WH`) or +double-quoted (`"Adhoc WH"`). + +Every connection a command opens uses its group, so one invocation never mixes warehouses: +`sqb build` plans, inspects, and executes on the `build` warehouse, and `sqb diff` creates its +temporary comparison tables on the `query` warehouse. The exception is `sqb dbt`: it applies the +`build` group to SQLBuild-owned work, while dbt nodes keep the warehouse from their dbt profile. + +| Group | Commands | +|-------|----------| +| `build` | `audit`, `build`, `check`, `clone`, `dbt` (SQLBuild-owned work), `janitor`, `load`, `plan`, `scenario`, `seed`, `test` | +| `query` | `contract`, `cost`, `debug`, `diff`, `freshness`, `query` | + +Other commands, such as `compile`, `lineage`, and `format`, never connect to a warehouse. +Integrations that run builds, such as Dagster, invoke `sqb build` and use the `build` group. + +Each developer can point ad-hoc commands at their own warehouse without editing shared +configuration: + +```toml +# sqlbuild_local.toml +[targets.dev.warehouses] +query = "ALICE_XS_WH" +``` + +Local group values override the project value for the same group only; the other group keeps +its project value. `sqb debug` shows the resolved warehouse for each group and where it came +from, and `sqb build --verbose` shows it in the execution header. + +Groups need an adapter with a session warehouse: the Snowflake adapter, or a custom adapter +that extends it. `warehouses` in `sqlbuild_project.toml` is checked against the project's +adapter, and `warehouses` in `sqlbuild_local.toml` against the adapter you actually run. Setting +them for an adapter without a session warehouse, or passing `--warehouse`, is an error rather +than a silently ignored setting. The one exception is a local `adapter` override, for example a +developer running a Snowflake project on DuckDB: shared groups then don't apply, and +`sqb debug` reports them as `not used (adapter duckdb)`. + ### Missing migration origins A `migrate_from` declaration, on a model or a column, names a relation or column that exists in the targets where the rename is pending. A target that never built the old name, such as a fresh development schema, has nothing to migrate. `missing_migration_origin` decides what happens there: