Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ sqb --project-dir <path> audit [flags]
| `--concurrency` | Maximum number of audits to run concurrently (default: `1`) |
| `--json` | Print structured audit results |
| `--json-output` | Write structured audit results to a file |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

## Concurrency

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ sqb --project-dir <path> build [flags]
| `--manifest` | Generate `target/manifest.json` with plan-aware project metadata |
| `--select`, `-s` | Select specific models |
| `--exclude` | Exclude specific models |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

## Fast iteration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ sqb --project-dir <path> check [flags]
| `--json` | Print check results as JSON |
| `--json-output` | Write check results JSON to a file path |
| `--vars` | Override project variables |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

Selecting a non-check node is rejected; use [`sqb build`](build.md) to run tasks and assets.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ sqb --project-dir <path> clone --from <target> [--to <target>] [flags]
| `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) |
| `--select`, `-s` | Select specific models to clone |
| `--exclude` | Exclude specific models from cloning |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

## Examples

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ replace conflicts rather than preserving authored declarations.
| `--vars` | both | Override project variables |
| `--write` | generate | Apply generated repository changes |
| `--overwrite` | generate | Replace conflicting declarations; requires `--write` |
| `--warehouse <name>` | both | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

See [Contracts](../concepts/models/contracts.md#adopting-an-existing-schema) for ownership and safety
guarantees.
Original file line number Diff line number Diff line change
Expand Up @@ -91,3 +91,8 @@ target_path = "../dbt_project/target"
```

See [Project Configuration](../concepts/project-configuration.md#dbt) for all fields.

SQLBuild-owned work in `sqb dbt plan`, `run` and `build` connects with the target's `build`
[warehouse group](../concepts/project-configuration.md#command-group-warehouses). dbt itself
keeps using the warehouse from its profile. These commands do not accept `--warehouse`, because
their remaining arguments pass through to dbt.
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,17 @@ The command runs three groups of checks:

**Runtime** - SQLBuild version, Python version, Python path, OS info.

**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter and active target.
**Configuration** - Finds and validates `sqlbuild_project.toml`, loads `sqlbuild_local.toml` if present, resolves the adapter and active target. For Snowflake it also shows the warehouse each [command group](../concepts/project-configuration.md#command-group-warehouses) connects with and where that value came from.

**Connection** - Displays connection settings (secrets are masked), attempts to connect to the warehouse, and runs `SELECT 1` to verify query execution.
**Connection** - Displays connection settings as `sqb debug` itself uses them, with the `query` warehouse group applied (secrets are masked), attempts to connect to the warehouse, and runs `SELECT 1` to verify query execution.

## Flags

| Flag | Description |
|------|-------------|
| `--no-connection` | Skip the connection and query tests. Useful for validating config without warehouse access. |
| `--json` | Output results as JSON instead of formatted text. |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

## Example

Expand All @@ -52,6 +53,8 @@ Configuration:
project: waffle_shop [OK loaded]
adapter: snowflake [OK found]
target: dev [OK resolved]
build warehouse: BUILD_WH [OK sqlbuild_project.toml [targets.dev.warehouses] build]
query warehouse: SQB_WH [OK connection warehouse]

Connection:
account: FJMQFQV-OJ66172
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ In query mode, omit `FROM:TO` and model selectors; each query identifies its own
| `--json` | Print one JSON document to stdout |
| `--select`, `-s` | Select specific models to diff (required in model mode) |
| `--exclude` | Exclude specific models from diffing |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

## Examples

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ sqb --project-dir <path> freshness [flags]
| `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) |
| `--select`, `-s` | Select specific sources or models (sources upstream of selected models are included) |
| `--exclude` | Exclude specific sources or models |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

## Source selection

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ sqb --project-dir <path> janitor [flags]
| `--retention-days` | Override the configured retention period (days) before a stale relation is archived |
| `--direct-state-history-versions` | Override how many state-history versions are kept per identity |
| `--drop-old-name-view <name>` | Drop the [compatibility view](../concepts/models/migrations.md#old-names) at a renamed model's old name before it expires. Repeat for several views. Fails with `C503` if no compatibility view is recorded at the name |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

## Targets

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ sqb --project-dir <path> load [flags]
| `--json` | Output results as JSON |
| `--json-output` | Write JSON results to a file path |
| `--var` | Set project variables (`--var key=value`) |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

## Examples

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ sqb --project-dir <path> plan [flags]
| `--end-cursor-int` | Override end cursor for integer incremental models |
| `--select`, `-s` | Select specific models |
| `--exclude` | Exclude specific models |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

## Output

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ File paths are resolved from the current working directory.
| `--format` | Output format: `long` (default), `table`, `json`, or `csv` |
| `--limit` | Maximum rows to return (default: 20) |
| `--no-limit` | Disable the row limit |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

## Output formats

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ sqb scenario test [flags]
| `--max-snapshot-bytes` | Override per-relation byte limit for capture |
| `--max-snapshot-total-bytes` | Override total byte limit for capture |
| `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

### Selectors

Expand Down Expand Up @@ -130,6 +131,7 @@ sqb scenario capture [flags]
| `--max-snapshot-bytes` | Override per-relation byte limit |
| `--max-snapshot-total-bytes` | Override total byte limit |
| `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

### Examples

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ sqb --project-dir <path> seed [flags]
|------|-------------|
| `--select`, `-s` | Select specific seeds by name |
| `--exclude` | Exclude specific seeds |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

## Examples

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ sqb --project-dir <path> test [flags]
| `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) |
| `--select`, `-s` | Select tests targeting specific models |
| `--exclude` | Exclude tests targeting specific models |
| `--warehouse <name>` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse |

Parameterized cases use parent-level selection. Selecting a target model includes every case in
each matching `TEST` template; case names are not model selectors.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,12 @@ class DuckDbPlusAdapter(DuckDbAdapter):
adapter = "duckdb_plus"
```

A subclass inherits the parent's capabilities. For example, an adapter extending the Snowflake
adapter keeps `supports_session_warehouse = True`, so it can use
[command-group warehouses](project-configuration.md#command-group-warehouses) and
`--warehouse`. An adapter written from scratch that sets this attribute receives the selected
warehouse as the connection's `warehouse` key.

### Writing an adapter from scratch

For a database engine with no built-in support, subclass `BaseAdapter`. It provides ANSI SQL defaults for most methods - you only need to implement `connect`, `execute`, `close`, and any methods where your engine differs from standard SQL:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ On connect, SQLBuild applies the connection's role and warehouse, then applies t
target's authoritative database and schema. These ensure the session context is set correctly
regardless of the user's default settings.

The warehouse can differ by command. A target's `[targets.<name>.warehouses]` section selects
one warehouse for build-style commands and another for ad-hoc commands such as `sqb query` and
`sqb diff`, and `--warehouse` overrides both for one invocation. See
[command-group warehouses](../project-configuration.md#command-group-warehouses).

## Planning metadata

Planning reads relation and column metadata with `SHOW TABLES`, `SHOW VIEWS`, and `SHOW COLUMNS`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ A target is a named build context - the database and schema you build into, plus
| `defer_sources_to` | Target name to read managed source data from (see [Loaders](python-nodes/loaders.md#source-deferral)) |
| `clone` | Clone policy (see below) |
| `missing_migration_origin` | What a build does when a declared `migrate_from` origin does not exist in this target: `allow` (default), `require_confirmation`, or `deny` (see below) |
| `warehouses` | Default Snowflake warehouse per command group, `build` and `query` (see below) |

```toml
[connections.warehouse]
Expand Down Expand Up @@ -218,6 +219,76 @@ password = "${ENV:SNOWFLAKE_PASSWORD}"
Commands then use `dev` automatically. An explicit command such as `sqb build --target prod`
still takes precedence for that invocation.

### Command-group warehouses

Builds and ad-hoc inspection usually need different compute. A Snowflake target can choose a
default warehouse for each command group, so `sqb build` runs on a large warehouse while
`sqb query` and `sqb diff` stay on a small one:

```toml
[connections.warehouse]
account = "my_org-my_account"
warehouse = "ANALYTICS_WH" # fallback for every command, as before

[targets.dev]
connection = "warehouse"
database = "ANALYTICS"
schema = "DEV"

[targets.dev.warehouses]
build = "BUILD_WH" # commands in the build group
query = "ADHOC_WH" # commands in the query group
```

Each command connects with the first warehouse set in this order:

1. `--warehouse <name>` on the command line, for that invocation only
2. The target's group warehouse, `[targets.<name>.warehouses] build` or `query`
3. The connection's `warehouse`
4. The Snowflake user's default warehouse

Both groups are optional. A target without `warehouses` behaves exactly as before. A group
warehouse takes precedence over the connection `warehouse` in every file. In particular, a
shared group overrides a connection `warehouse` you set in `sqlbuild_local.toml`; to change a
group locally, set the group itself, or pass `--warehouse` for one invocation. Group values
accept the same `${ENV:...}` and project-variable (`${my_var}`) templates as connection values.
Group values and `--warehouse` must be Snowflake identifiers: unquoted (`ADHOC_WH`) or
double-quoted (`"Adhoc WH"`).

Every connection a command opens uses its group, so one invocation never mixes warehouses:
`sqb build` plans, inspects, and executes on the `build` warehouse, and `sqb diff` creates its
temporary comparison tables on the `query` warehouse. The exception is `sqb dbt`: it applies the
`build` group to SQLBuild-owned work, while dbt nodes keep the warehouse from their dbt profile.

| Group | Commands |
|-------|----------|
| `build` | `audit`, `build`, `check`, `clone`, `dbt` (SQLBuild-owned work), `janitor`, `load`, `plan`, `scenario`, `seed`, `test` |
| `query` | `contract`, `cost`, `debug`, `diff`, `freshness`, `query` |

Other commands, such as `compile`, `lineage`, and `format`, never connect to a warehouse.
Integrations that run builds, such as Dagster, invoke `sqb build` and use the `build` group.

Each developer can point ad-hoc commands at their own warehouse without editing shared
configuration:

```toml
# sqlbuild_local.toml
[targets.dev.warehouses]
query = "ALICE_XS_WH"
```

Local group values override the project value for the same group only; the other group keeps
its project value. `sqb debug` shows the resolved warehouse for each group and where it came
from, and `sqb build --verbose` shows it in the execution header.

Groups need an adapter with a session warehouse: the Snowflake adapter, or a custom adapter
that extends it. `warehouses` in `sqlbuild_project.toml` is checked against the project's
adapter, and `warehouses` in `sqlbuild_local.toml` against the adapter you actually run. Setting
them for an adapter without a session warehouse, or passing `--warehouse`, is an error rather
than a silently ignored setting. The one exception is a local `adapter` override, for example a
developer running a Snowflake project on DuckDB: shared groups then don't apply, and
`sqb debug` reports them as `not used (adapter duckdb)`.

### Missing migration origins

A `migrate_from` declaration, on a model or a column, names a relation or column that exists in the targets where the rename is pending. A target that never built the old name, such as a fresh development schema, has nothing to migrate. `missing_migration_origin` decides what happens there:
Expand Down
1 change: 1 addition & 0 deletions src/sqlbuild/adapter/contract/classes/base_adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ class BaseAdapter(RetentionAdapterMixin, StrictAdapter):
allows_implicit_managed_write_schema: ClassVar[bool] = False
execution_duration_limit_seconds: ClassVar[int | None] = None
metadata_inspection_concurrency: ClassVar[int] = 1
supports_session_warehouse: ClassVar[bool] = False
_snapshot_sql_dialect: ClassVar[SnapshotSqlDialect] = SnapshotSqlDialect(
timestamp_type="TIMESTAMP",
distinct_condition=render_is_distinct_from,
Expand Down
15 changes: 15 additions & 0 deletions src/sqlbuild/adapter/discovery/main/session_warehouse_support.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
"""Public check for whether a project's adapter selects a session warehouse."""

from __future__ import annotations

from pathlib import Path

from sqlbuild.adapter.discovery.main.resolve_adapter import resolve_adapter


def adapter_supports_session_warehouse(*, adapter_name: str, project_dir: Path | None) -> bool:
"""Return whether the adapter, resolved like any project adapter, has a session warehouse."""

return resolve_adapter(
adapter_name=adapter_name, project_dir=project_dir
).supports_session_warehouse
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,7 @@ class SnowflakeAdapter(MicrobatchMixin, UnkeyedDiffMixin, BaseAdapter):

max_identifier_length: ClassVar[int] = 255
state_tables_transient: ClassVar[bool] = True
supports_session_warehouse: ClassVar[bool] = True
connection_routing_keys: ClassVar[frozenset[str]] = frozenset(
{"source", "profile", "target", "project_dir", "profiles_dir"}
)
Expand Down
16 changes: 14 additions & 2 deletions src/sqlbuild/cli/commands/_helpers/build_execution/run_context.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,13 @@
from pathlib import Path
from typing import TextIO

from sqlbuild.cli.commands._helpers.runtime.warehouses import describe_connection_warehouse
from sqlbuild.cli.commands.constants import (
C0_CONTROL_CODE_LIMIT,
C1_CONTROL_CODE_LIMIT,
C1_CONTROL_CODE_START,
)
from sqlbuild.cli.commands.models import BuildRunContext
from sqlbuild.cli.commands.models import BuildRunContext, ResolvedWarehouse
from sqlbuild.presentation.classes.cli_document import CliDocument
from sqlbuild.presentation.classes.cli_style import CliStyle

Expand Down Expand Up @@ -54,7 +55,7 @@ def _format_build_run_context(*, context: BuildRunContext, use_color: bool) -> s
("target", _display_value(context.project.effective_target_name)),
("database", _display_value(context.project.effective_target_database)),
*placement_rows,
("warehouse", _display_value(context.connection_config.get("warehouse"))),
("warehouse", _warehouse_display(context)),
("concurrency", f"{context.concurrency} configured limit"),
("full_refresh", str(context.full_refresh).lower()),
("selected", f"{selected_count:,} of {total_count:,} build resources"),
Expand All @@ -81,6 +82,17 @@ def _format_build_run_context(*, context: BuildRunContext, use_color: bool) -> s
return document.render()


def _warehouse_display(context: BuildRunContext) -> str:
resolved: ResolvedWarehouse = describe_connection_warehouse(
discovered_inputs=context.discovered_inputs,
selected_target=context.project.effective_target_name,
connection_config=context.connection_config,
)
if resolved.warehouse is None:
return _display_value(None)
return f"{_display_value(resolved.warehouse)} ({resolved.source})"


def _display_value(value: object) -> str:
display_value: str | None = _safe_display_value(value)
return display_value if display_value is not None else "not set"
Expand Down
Loading
Loading