diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md index 2f6070241d..d425a95333 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md @@ -92,7 +92,12 @@ materialized = "view" |-------|-------------| | `name` | Project name. Used in fingerprint tracking and manifest generation. | | `adapter` | Database adapter: `duckdb`, `motherduck`, `snowflake`, `bigquery`, `databricks`, `postgres`, or `sqlserver`. See [Adapters](adapters.md). | -| `default_target` | Name of the target to build against when none is selected (see [Targets](#targets)). | + +### Optional fields + +| Field | Description | +|-------|-------------| +| `default_target` | Name of the target to build against when none is selected (see [Selecting a target](#selecting-a-target)). | ### Named connections @@ -129,6 +134,33 @@ database = "ANALYTICS" schema = "DEV_ALICE" ``` +When the project defines exactly one named connection, across `sqlbuild_project.toml` and +`sqlbuild_local.toml`, a target that does not set `connection` uses it. This default does not +apply when either file has a legacy top-level `[connection]` block or the target sets an inline +`[targets..connection]` block. With several named connections, every target must choose +one; SQLBuild rejects a target without `connection` while loading configuration. + +```toml +[connections.local] +database = "my_project.duckdb" + +[targets.dev] +schema = "dev" # uses connections.local, the only named connection +``` + +A command that needs the warehouse, such as `build`, `plan`, `query`, `diff`, or `debug`, fails +with `D001` before connecting when no connection is configured at all: no named connection, no +legacy `[connection]` block, and no inline target connection. Commands and modes that do not +connect, such as `compile`, `format`, `lineage`, `scope`, `rename`, `mv`, `test --inspect`, and +`scenario test --local`, keep working without one. +SQLBuild never falls back to an implicit in-memory DuckDB database; to use one, set it +explicitly: + +```toml +[connections.memory] +database = ":memory:" +``` + A `database` or `schema` present in a named connection is connection/session metadata only; it does not satisfy the mandatory namespace strategy for a named target. Put the target's authoritative database and schema on `[targets.]`. SQLBuild validates connection @@ -141,14 +173,14 @@ forms are compatibility syntax, not the canonical format for new or updated proj ## Targets -A target is a named build context - the database and schema you build into, plus execution policy (for example `dev` and `prod`). Each target references a named connection and can configure: +A target is a named build context - the database and schema you build into, plus execution policy (for example `dev` and `prod`). Each target references a named connection (or uses the only one defined; see [Named connections](#named-connections)) and can configure: | Field | Description | |-------|-------------| | `schema` | Schema for all models in this target; required for named targets | | `loader_schema` | Default write schema for managed source loaders; falls back to `schema` | | `database` | Database for all models in this target | -| `connection` | Name from `[connections.]` used for endpoint, authentication, and compute | +| `connection` | Name from `[connections.]` used for endpoint, authentication, and compute; optional when exactly one named connection is defined | | `vars` | Target-specific project variables | | `defer_sources_to` | Target name to read managed source data from (see [Loaders](python-nodes/loaders.md#source-deferral)) | | `clone` | Clone policy (see below) | @@ -197,7 +229,9 @@ The active target is determined by (in order of precedence): 1. `--target` on the command line (highest priority) 2. `sqlbuild_local.toml` `target` field 3. `default_target` in `sqlbuild_project.toml` -4. No target (models build to the default schema) +4. No target (models build to the default schema). Named connections are chosen through + targets, so warehouse commands fail with `D001` unless a target is selected or a legacy + top-level `[connection]` block exists. A typical developer keeps shared target definitions in `sqlbuild_project.toml` and selects their normal target once in the optional local file: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md index 9d087b5068..5e21ef3e48 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md @@ -150,14 +150,35 @@ sources: lag_tolerance: 15m ``` +When the column is declared with a type under `columns:`, `strategy` and `type` can be left out: + +```yaml +sources: + - name: raw_events + schema: raw + table: events + columns: + - name: updated_at + type: timestamp + freshness: + column: updated_at # strategy column, type timestamp + lag_tolerance: 15m +``` + ### Strategies | Strategy | Description | Required fields | |----------|-------------|-----------------| | `adapter` | Uses warehouse metadata (e.g. Snowflake `LAST_ALTERED`). No query against the source table. | None | -| `column` | Reads `MAX(column)` from the source table. | `column`, `type` | +| `column` | Reads `MAX(column)` from the source table. | `column`, `type` (inferred from a typed column declaration) | | `sql` | Runs a custom query that returns a single scalar value. | `query`, `type` | +`strategy` is optional. Without it, SQLBuild infers the strategy from the keys you set: `column` +means `column`, `query` means `sql`, and neither means `adapter`, so `freshness: {}` observes +warehouse metadata. Setting both `column` and `query` without a `strategy` is an error, as is +setting `type` or `filter` without either. An explicit `strategy` keeps working and must agree +with the keys present. + #### adapter ```yaml @@ -176,7 +197,7 @@ freshness: type: timestamp ``` -Queries `MAX(column)` from the source table. The column must be a plain column name (no expressions; use `sql` strategy for those). Requires `type`. +Queries `MAX(column)` from the source table. The column must be a plain column name (no expressions; use `sql` strategy for those). Requires `type`, unless the source declares the column with a type that SQLBuild can map (see [Type](#type)). #### sql @@ -199,6 +220,19 @@ The `type` field declares the value kind for comparison: | `integer` | Integer value. Change detected by exact comparison. | | `string` | String value. Change detected by exact comparison. | +For the `column` strategy, SQLBuild takes `type` from the column's declared type in the source's +`columns:` when it can map it: + +| Declared column type | Freshness `type` | +|----------------------|------------------| +| `TIMESTAMP`, `TIMESTAMP_NTZ`, `TIMESTAMP_LTZ`, `TIMESTAMP_TZ`, `TIMESTAMPTZ`, `TIMESTAMP WITH TIME ZONE`, `DATETIME`, `DATETIME2`, `DATETIMEOFFSET`, `SMALLDATETIME` | `timestamp` | +| `INTEGER`, `INT`, `BIGINT`, `SMALLINT`, `TINYINT`, `INT2`, `INT4`, `INT8`, `INT64` | `integer` | +| `VARCHAR`, `STRING`, `TEXT`, `CHAR`, `CHARACTER`, `CHARACTER VARYING`, `NCHAR`, `NVARCHAR` | `string` | + +Matching ignores case and length or precision arguments. Any other declared type, such as +`DATE` or `NUMBER(38,0)`, needs an explicit `type`; a column declared with an unmapped type and no +`type` is an error. An explicit `type` that contradicts a mapped declared type is also an error. + ### Lag tolerance `lag_tolerance` is optional and only valid with `type: timestamp`. It declares how much the observed timestamp can drift from the previous observation before being treated as a change: diff --git a/src/sqlbuild/cli/commands/_helpers/build_planning/invocation.py b/src/sqlbuild/cli/commands/_helpers/build_planning/invocation.py index 781c337405..f37e4ee86c 100644 --- a/src/sqlbuild/cli/commands/_helpers/build_planning/invocation.py +++ b/src/sqlbuild/cli/commands/_helpers/build_planning/invocation.py @@ -6,12 +6,11 @@ from pathlib import Path from typing import TextIO -from sqlbuild.cli.commands._helpers.runtime.adapter_context import ( - resolve_adapter_connection_context, -) +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.cli.commands._helpers.runtime.adapters import resolve_adapter +from sqlbuild.cli.commands._helpers.runtime.connection import resolve_project_connection_config from sqlbuild.cli.commands.exceptions import CliUserError from sqlbuild.cli.commands.models import ( - AdapterConnectionContext, BuildCommandRequest, BuildInvocation, ) @@ -23,6 +22,9 @@ from sqlbuild.compiler.discovery.main.discover import discover_project_inputs from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.presentation.main.supports_color import supports_color +from sqlbuild.spec.contracts.main.resolve_effective_adapter_name import ( + resolve_effective_adapter_name, +) from sqlbuild.spec.contracts.main.resolve_target_config import resolve_target_config from sqlbuild.spec.contracts.main.resolve_target_name import resolve_target_name from sqlbuild.spec.contracts.models import ExecutionLimitsConfig, TargetConfig @@ -44,9 +46,27 @@ def resolve_build_invocation(*, request: BuildCommandRequest) -> BuildInvocation ) if request.defer_to is not None and effective_defer_clone_from is not None: raise CliUserError("--defer-clone-from cannot be used with --defer-to", code="C408") - adapter_context: AdapterConnectionContext = resolve_adapter_connection_context( + adapter_name: str = resolve_effective_adapter_name( + project_config=discovered_inputs.project_config, + local_config=discovered_inputs.local_config, + ) + adapter: BaseAdapter = resolve_adapter( + adapter_name=adapter_name, project_dir=effective_project_dir + ) + effective_target_name: str | None = resolve_target_name( + project_config=discovered_inputs.project_config, + local_config=discovered_inputs.local_config, + selected_target=request.selected_target, + ) + execution_limits: ExecutionLimitsConfig = _resolve_execution_limits( discovered_inputs=discovered_inputs, - effective_project_dir=effective_project_dir, + target_name=effective_target_name, + adapter_name=adapter_name, + adapter=adapter, + ) + connection_config: dict[str, object] = resolve_project_connection_config( + discovered_inputs=discovered_inputs, + project_dir=effective_project_dir, selected_target=request.selected_target, cli_vars=request.cli_vars, ) @@ -54,7 +74,7 @@ def resolve_build_invocation(*, request: BuildCommandRequest) -> BuildInvocation use_color: bool = not request.no_color and not machine_output and supports_color() progress_stream: TextIO = sys.stderr if request.debug or machine_output else sys.stdout reporters: CommandProgressReporters = build_command_progress_reporters( - adapter_name=adapter_context.adapter_name, + adapter_name=adapter_name, stream=progress_stream, use_color=use_color, ) @@ -63,59 +83,65 @@ def resolve_build_invocation(*, request: BuildCommandRequest) -> BuildInvocation if request.load_sources is not None else build_effective_settings_config(discovered_inputs=discovered_inputs).auto_load_sources ) - effective_target_name: str | None = resolve_target_name( - project_config=discovered_inputs.project_config, - local_config=discovered_inputs.local_config, - selected_target=request.selected_target, + return BuildInvocation( + effective_project_dir=effective_project_dir, + discovered_inputs=discovered_inputs, + effective_defer_clone_from=effective_defer_clone_from, + adapter_name=adapter_name, + adapter=adapter, + connection_config=connection_config, + use_color=use_color, + progress_stream=progress_stream, + connection_progress=reporters.connection, + planning_progress=reporters.planning, + should_load_sources=should_load_sources, + effective_target_name=effective_target_name, + execution_limits=execution_limits, ) + + +def _resolve_execution_limits( + *, + discovered_inputs: DiscoveredProjectInputs, + target_name: str | None, + adapter_name: str, + adapter: BaseAdapter, +) -> ExecutionLimitsConfig: + """Resolve the target's execution limits and validate them against the project adapter.""" + execution_limits: ExecutionLimitsConfig = ( resolve_target_config( project_config=discovered_inputs.project_config, local_config=discovered_inputs.local_config, - target_name=effective_target_name, + target_name=target_name, ).execution_limits - if effective_target_name is not None + if target_name is not None else ExecutionLimitsConfig() ) + target_label: str = target_name or "default" if ( execution_limits.max_duration is not None - and adapter_context.adapter.execution_duration_limit_seconds is None + and adapter.execution_duration_limit_seconds is None ): - target_label: str = effective_target_name or "default" raise CliUserError( f"Target '{target_label}' configures max_duration, but adapter " - f"'{adapter_context.adapter_name}' cannot cancel active statements safely", + f"'{adapter_name}' cannot cancel active statements safely", code="C415", help=execution_limits.remediation, ) - maximum_duration_seconds: int | None = adapter_context.adapter.execution_duration_limit_seconds + maximum_duration_seconds: int | None = adapter.execution_duration_limit_seconds if ( execution_limits.max_duration_seconds is not None and maximum_duration_seconds is not None and execution_limits.max_duration_seconds > maximum_duration_seconds ): - target_label = effective_target_name or "default" raise CliUserError( f"Target '{target_label}' max_duration exceeds the largest duration supported by " - f"adapter '{adapter_context.adapter_name}' ({maximum_duration_seconds} seconds)", + f"adapter '{adapter_name}' ({maximum_duration_seconds} seconds)", code="C416", help=execution_limits.remediation, ) - return BuildInvocation( - effective_project_dir=effective_project_dir, - discovered_inputs=discovered_inputs, - effective_defer_clone_from=effective_defer_clone_from, - adapter_name=adapter_context.adapter_name, - adapter=adapter_context.adapter, - connection_config=adapter_context.connection_config, - use_color=use_color, - progress_stream=progress_stream, - connection_progress=reporters.connection, - planning_progress=reporters.planning, - should_load_sources=should_load_sources, - effective_target_name=effective_target_name, - execution_limits=execution_limits, - ) + return execution_limits def _resolve_defer_clone_from( diff --git a/src/sqlbuild/cli/commands/_helpers/runtime/adapter_context.py b/src/sqlbuild/cli/commands/_helpers/runtime/adapter_context.py index d4b13fada9..9127cfd8a9 100644 --- a/src/sqlbuild/cli/commands/_helpers/runtime/adapter_context.py +++ b/src/sqlbuild/cli/commands/_helpers/runtime/adapter_context.py @@ -20,6 +20,7 @@ def resolve_adapter_connection_context( effective_project_dir: Path, selected_target: str | None, cli_vars: dict[str, object] | None, + require_connection: bool = True, ) -> AdapterConnectionContext: """Resolve the effective adapter and project connection configuration.""" @@ -36,6 +37,7 @@ def resolve_adapter_connection_context( project_dir=effective_project_dir, selected_target=selected_target, cli_vars=cli_vars, + require_connection=require_connection, ) return AdapterConnectionContext( adapter_name=adapter_name, diff --git a/src/sqlbuild/cli/commands/_helpers/runtime/command_invocation.py b/src/sqlbuild/cli/commands/_helpers/runtime/command_invocation.py index 62e105d459..768ff9be5f 100644 --- a/src/sqlbuild/cli/commands/_helpers/runtime/command_invocation.py +++ b/src/sqlbuild/cli/commands/_helpers/runtime/command_invocation.py @@ -54,7 +54,9 @@ def resolve_command_invocation[InvocationT]( ) -> InvocationT: """Resolve discovery, adapter, connection, and output context for one command.""" - context: _CommandInvocationContext = _resolve_command_invocation_context(request=request) + context: _CommandInvocationContext = _resolve_command_invocation_context( + request=request, require_connection=True + ) return invocation_type( effective_project_dir=context.effective_project_dir, discovered_inputs=context.discovered_inputs, @@ -67,11 +69,16 @@ def resolve_command_invocation[InvocationT]( def resolve_reported_command_invocation[InvocationT]( - *, request: _CommandInvocationRequest, invocation_type: Callable[..., InvocationT] + *, + request: _CommandInvocationRequest, + invocation_type: Callable[..., InvocationT], + require_connection: bool = True, ) -> InvocationT: """Resolve command context plus shared connection and planning progress reporters.""" - context: _CommandInvocationContext = _resolve_command_invocation_context(request=request) + context: _CommandInvocationContext = _resolve_command_invocation_context( + request=request, require_connection=require_connection + ) reporters: CommandProgressReporters = build_command_progress_reporters( adapter_name=context.adapter_context.adapter_name, stream=context.progress_stream, @@ -91,7 +98,7 @@ def resolve_reported_command_invocation[InvocationT]( def _resolve_command_invocation_context( - *, request: _CommandInvocationRequest + *, request: _CommandInvocationRequest, require_connection: bool ) -> _CommandInvocationContext: effective_project_dir: Path = ( request.project_dir if request.project_dir is not None else Path.cwd() @@ -104,6 +111,7 @@ def _resolve_command_invocation_context( effective_project_dir=effective_project_dir, selected_target=request.selected_target, cli_vars=request.cli_vars, + require_connection=require_connection, ) machine_output: bool = request.json_output use_color: bool = not request.no_color and not machine_output and supports_color() diff --git a/src/sqlbuild/cli/commands/_helpers/runtime/connection.py b/src/sqlbuild/cli/commands/_helpers/runtime/connection.py index 8735832dac..1e9ed21167 100644 --- a/src/sqlbuild/cli/commands/_helpers/runtime/connection.py +++ b/src/sqlbuild/cli/commands/_helpers/runtime/connection.py @@ -8,6 +8,8 @@ from sqlbuild.adapter.contract.types import BuiltinAdapter from sqlbuild.cli.commands._helpers.runtime.warehouses import apply_command_warehouse from sqlbuild.compiler.compile.main.effective_config import build_effective_connection_config +from sqlbuild.compiler.discovery.constants import PROJECT_CONFIG_FILENAME +from sqlbuild.compiler.discovery.exceptions import ProjectConfigError from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.integrations.dbt.exceptions import DbtProfileError from sqlbuild.integrations.dbt.main.profile.profile_connection import ( @@ -17,6 +19,9 @@ from sqlbuild.spec.contracts.main.resolve_effective_adapter_name import ( resolve_effective_adapter_name, ) +from sqlbuild.spec.contracts.main.resolve_target_config import resolve_target_config +from sqlbuild.spec.contracts.main.resolve_target_name import resolve_target_name +from sqlbuild.spec.contracts.models import LocalConfig, ProjectConfig, TargetConfig _DUCKDB_SNOWFLAKE_LIKE_WARNING_KEYS: frozenset[str] = frozenset( {"account", "authenticator", "role", "token", "warehouse"} @@ -106,9 +111,16 @@ def resolve_project_connection_config( project_dir: Path, selected_target: str | None = None, cli_vars: dict[str, object] | None = None, + require_connection: bool = True, ) -> dict[str, object]: - """Resolve the effective project connection config for CLI command execution.""" + """Resolve the project connection; offline modes pass ``require_connection=False``.""" + if require_connection: + _require_configured_connection( + discovered_inputs=discovered_inputs, + project_dir=project_dir, + selected_target=selected_target, + ) return resolve_connection_config( raw_config=build_effective_connection_config( discovered_inputs=discovered_inputs, @@ -135,18 +147,46 @@ def resolve_target_connection_config( ) -> dict[str, object]: """Resolve the effective connection config for one named target.""" - return resolve_connection_config( - raw_config=build_effective_connection_config( - discovered_inputs=discovered_inputs, - selected_target=target_name, - cli_vars=cli_vars, - ), - project_dir=project_dir, - adapter_name=resolve_effective_adapter_name( - project_config=discovered_inputs.project_config, - local_config=discovered_inputs.local_config, - ), + return resolve_project_connection_config( discovered_inputs=discovered_inputs, - cli_vars=cli_vars, + project_dir=project_dir, selected_target=target_name, + cli_vars=cli_vars, + ) + + +def _require_configured_connection( + *, + discovered_inputs: DiscoveredProjectInputs, + project_dir: Path, + selected_target: str | None, +) -> None: + """Reject warehouse commands whose connection would come from no configuration at all.""" + + project_config: ProjectConfig = discovered_inputs.project_config + local_config: LocalConfig = discovered_inputs.local_config + if project_config.connection or local_config.connection: + return + target_name: str | None = resolve_target_name( + project_config=project_config, + local_config=local_config, + selected_target=selected_target, + ) + if target_name is None: + raise ProjectConfigError( + f"{project_dir / PROJECT_CONFIG_FILENAME} no target is selected and no top-level " + "[connection] is configured; add [connections.] and a [targets.] that " + "uses it, then set default_target or pass --target" + ) + target_config: TargetConfig = resolve_target_config( + project_config=project_config, + local_config=local_config, + target_name=target_name, + ) + if target_config.connection_name is not None or target_config.connection: + return + raise ProjectConfigError( + f"{project_dir / PROJECT_CONFIG_FILENAME} target {target_name} has no connection; add " + f'[connections.] (or set connection = "" on [targets.{target_name}]) in ' + "sqlbuild_project.toml or sqlbuild_local.toml" ) diff --git a/src/sqlbuild/cli/commands/_helpers/test/invocation.py b/src/sqlbuild/cli/commands/_helpers/test/invocation.py index 6572d84451..562d137d69 100644 --- a/src/sqlbuild/cli/commands/_helpers/test/invocation.py +++ b/src/sqlbuild/cli/commands/_helpers/test/invocation.py @@ -9,6 +9,10 @@ def resolve_test_invocation(*, request: TestCommandRequest) -> TestInvocation: - """Resolve discovery, adapter, connection, and reporters for test.""" + """Resolve test context; ``--inspect`` never connects, so it needs no connection.""" - return resolve_reported_command_invocation(request=request, invocation_type=TestInvocation) + return resolve_reported_command_invocation( + request=request, + invocation_type=TestInvocation, + require_connection=not request.inspect, + ) diff --git a/src/sqlbuild/cli/commands/main/execution/_freshness.py b/src/sqlbuild/cli/commands/main/execution/_freshness.py index d5d57c28f6..af1df800ba 100644 --- a/src/sqlbuild/cli/commands/main/execution/_freshness.py +++ b/src/sqlbuild/cli/commands/main/execution/_freshness.py @@ -67,12 +67,6 @@ def run_freshness(request: FreshnessCommandRequest) -> int: adapter: BaseAdapter = resolve_adapter( adapter_name=adapter_name, project_dir=effective_project_dir ) - connection_config: dict[str, object] = resolve_project_connection_config( - discovered_inputs=discovered_inputs, - project_dir=effective_project_dir, - selected_target=selected_target, - cli_vars=cli_vars, - ) graph: ProjectGraph = build_project_graph( discovered_inputs=discovered_inputs, adapter=adapter, @@ -91,6 +85,12 @@ def run_freshness(request: FreshnessCommandRequest) -> int: if not selected_source_names: result: FreshnessCommandResult = FreshnessCommandResult() else: + connection_config: dict[str, object] = resolve_project_connection_config( + discovered_inputs=discovered_inputs, + project_dir=effective_project_dir, + selected_target=selected_target, + cli_vars=cli_vars, + ) connection: Any = adapter.connect(connection_config) try: previous_records_by_source_name: dict[str, SourceFreshnessRecord] | None = None diff --git a/src/sqlbuild/compiler/discovery/_helpers/validation/target_connections.py b/src/sqlbuild/compiler/discovery/_helpers/validation/target_connections.py new file mode 100644 index 0000000000..fd6a20692e --- /dev/null +++ b/src/sqlbuild/compiler/discovery/_helpers/validation/target_connections.py @@ -0,0 +1,28 @@ +"""Offline validation of each target's connection selection.""" + +from __future__ import annotations + +from pathlib import Path + +from sqlbuild.compiler.discovery.constants import PROJECT_CONFIG_FILENAME +from sqlbuild.compiler.discovery.exceptions import ProjectConfigError +from sqlbuild.spec.contracts.exceptions import SpecConfigError +from sqlbuild.spec.contracts.main.resolve_target_config import resolve_target_config +from sqlbuild.spec.contracts.models import LocalConfig, ProjectConfig + + +def validate_target_connections( + *, project_dir: Path, project_config: ProjectConfig, local_config: LocalConfig +) -> None: + """Reject targets whose connection cannot be chosen among several named connections.""" + + target_name: str + for target_name in sorted({*project_config.targets, *local_config.targets}): + try: + _ = resolve_target_config( + project_config=project_config, + local_config=local_config, + target_name=target_name, + ) + except SpecConfigError as error: + raise ProjectConfigError(f"{project_dir / PROJECT_CONFIG_FILENAME} {error}") from error diff --git a/src/sqlbuild/compiler/discovery/_helpers/yml/sources.py b/src/sqlbuild/compiler/discovery/_helpers/yml/sources.py index 2a6bcd2c68..3bdd09915a 100644 --- a/src/sqlbuild/compiler/discovery/_helpers/yml/sources.py +++ b/src/sqlbuild/compiler/discovery/_helpers/yml/sources.py @@ -87,6 +87,29 @@ _SOURCE_FRESHNESS_VALUE_KINDS: frozenset[str] = frozenset( value_kind.value for value_kind in SourceFreshnessValueKind ) +_FRESHNESS_TIMESTAMP_TYPE_NAMES: frozenset[str] = frozenset( + { + "DATETIME", + "DATETIME2", + "DATETIMEOFFSET", + "SMALLDATETIME", + "TIMESTAMP", + "TIMESTAMPTZ", + "TIMESTAMP_LTZ", + "TIMESTAMP_NTZ", + "TIMESTAMP_TZ", + } +) +_FRESHNESS_TIMESTAMP_WITH_ZONE_PREFIXES: tuple[str, ...] = ( + "TIMESTAMP WITH TIME ZONE", + "TIMESTAMP WITHOUT TIME ZONE", +) +_FRESHNESS_INTEGER_TYPE_NAMES: frozenset[str] = frozenset( + {"BIGINT", "INT", "INT2", "INT4", "INT8", "INT64", "INTEGER", "SMALLINT", "TINYINT"} +) +_FRESHNESS_STRING_TYPE_NAMES: frozenset[str] = frozenset( + {"CHAR", "CHARACTER", "CHARACTER VARYING", "NCHAR", "NVARCHAR", "STRING", "TEXT", "VARCHAR"} +) def parse_sources_yml(*, contents: str, file_path: Path) -> tuple[SourceEntry, ...]: @@ -227,7 +250,7 @@ def _parse_source_entry(*, entry: dict[str, object], file_path: Path) -> SourceE managed=managed, loader=resolved_loader, integration_loader=integration_loader, - freshness=_optional_freshness_config(entry=entry, file_path=file_path), + freshness=_optional_freshness_config(entry=entry, columns=columns, file_path=file_path), write_strategy=_optional_write_strategy(entry=entry, file_path=file_path), load_batch_size=_optional_positive_int( entry=entry, @@ -400,7 +423,7 @@ def _optional_write_strategy( def _optional_freshness_config( - *, entry: dict[str, object], file_path: Path + *, entry: dict[str, object], columns: tuple[SourceColumnEntry, ...], file_path: Path ) -> SourceFreshnessConfig | None: raw_freshness: object | None = entry.get("freshness") if raw_freshness is None: @@ -414,19 +437,18 @@ def _optional_freshness_config( file_path=file_path, label="source freshness", ) - raw_strategy: str = require_non_empty_string( + raw_strategy: str | None = optional_non_empty_string( entry=freshness, key="strategy", file_path=file_path, label="source freshness", error_class=SourceParseError, ) - if raw_strategy not in _SOURCE_FRESHNESS_STRATEGIES: + if raw_strategy is not None and raw_strategy not in _SOURCE_FRESHNESS_STRATEGIES: strategies: str = ", ".join(sorted(_SOURCE_FRESHNESS_STRATEGIES)) raise SourceParseError( f"{file_path} source freshness 'strategy' must be one of: {strategies}" ) - strategy: SourceFreshnessStrategy = SourceFreshnessStrategy(raw_strategy) raw_value_kind: str | None = optional_non_empty_string( entry=freshness, key="type", @@ -474,6 +496,21 @@ def _optional_freshness_config( freshness=freshness, file_path=file_path, ) + strategy: SourceFreshnessStrategy = ( + SourceFreshnessStrategy(raw_strategy) + if raw_strategy is not None + else _implied_freshness_strategy( + column=column, + query=query, + has_type=value_kind is not None, + has_filter=freshness_filter is not None, + file_path=file_path, + ) + ) + if strategy == SourceFreshnessStrategy.COLUMN and column is not None: + value_kind = _resolve_column_freshness_value_kind( + column=column, explicit=value_kind, columns=columns, file_path=file_path + ) config: SourceFreshnessConfig = SourceFreshnessConfig( strategy=strategy, value_kind=value_kind, @@ -487,6 +524,80 @@ def _optional_freshness_config( return config +def _implied_freshness_strategy( + *, column: str | None, query: str | None, has_type: bool, has_filter: bool, file_path: Path +) -> SourceFreshnessStrategy: + if column is not None and query is not None: + raise SourceParseError( + f"{file_path} source freshness cannot set both column and query; set strategy " + "or remove one of them" + ) + if column is not None: + return SourceFreshnessStrategy.COLUMN + if query is not None: + return SourceFreshnessStrategy.SQL + if has_type or has_filter: + raise SourceParseError( + f"{file_path} source freshness type and filter require column or query" + ) + return SourceFreshnessStrategy.ADAPTER + + +def _resolve_column_freshness_value_kind( + *, + column: str, + explicit: SourceFreshnessValueKind | None, + columns: tuple[SourceColumnEntry, ...], + file_path: Path, +) -> SourceFreshnessValueKind | None: + declared: SourceColumnEntry | None = _find_source_column(columns=columns, name=column) + declared_type: str | None = None if declared is None else declared.type + inferred: SourceFreshnessValueKind | None = ( + None if declared_type is None else _freshness_value_kind_for_type(declared_type) + ) + if explicit is None: + if inferred is None and declared_type is not None: + raise SourceParseError( + f"{file_path} source freshness cannot infer type from column '{column}' " + f"declared as {declared_type}; set type to timestamp, integer, or string" + ) + return inferred + if inferred is not None and inferred != explicit: + raise SourceParseError( + f"{file_path} source freshness type {explicit.value} contradicts column " + f"'{column}' declared as {declared_type}; remove type or set it to {inferred.value}" + ) + return explicit + + +def _find_source_column( + *, columns: tuple[SourceColumnEntry, ...], name: str +) -> SourceColumnEntry | None: + exact: SourceColumnEntry | None = next( + (column for column in columns if column.name == name), None + ) + if exact is not None: + return exact + folded: list[SourceColumnEntry] = [ + column for column in columns if column.name.casefold() == name.casefold() + ] + return folded[0] if len(folded) == 1 else None + + +def _freshness_value_kind_for_type(type_sql: str) -> SourceFreshnessValueKind | None: + normalized: str = " ".join(type_sql.upper().split()) + base_name: str = normalized.partition("(")[0].strip() + if base_name in _FRESHNESS_TIMESTAMP_TYPE_NAMES or normalized.startswith( + _FRESHNESS_TIMESTAMP_WITH_ZONE_PREFIXES + ): + return SourceFreshnessValueKind.TIMESTAMP + if base_name in _FRESHNESS_INTEGER_TYPE_NAMES: + return SourceFreshnessValueKind.INTEGER + if base_name in _FRESHNESS_STRING_TYPE_NAMES: + return SourceFreshnessValueKind.STRING + return None + + def _validate_freshness_config(*, config: SourceFreshnessConfig, file_path: Path) -> None: if config.lag_tolerance is not None: _validate_source_freshness_lag_tolerance(config=config, file_path=file_path) @@ -513,7 +624,10 @@ def _validate_freshness_config(*, config: SourceFreshnessConfig, file_path: Path "use strategy sql for expressions" ) if config.value_kind is None: - raise SourceParseError(f"{file_path} source freshness strategy column requires type") + raise SourceParseError( + f"{file_path} source freshness strategy column requires type; set type or " + "declare the column with a type under columns" + ) if config.query is not None: raise SourceParseError( f"{file_path} source freshness strategy column does not support query" diff --git a/src/sqlbuild/compiler/discovery/main/discover.py b/src/sqlbuild/compiler/discovery/main/discover.py index af91ecf729..e298c16593 100644 --- a/src/sqlbuild/compiler/discovery/main/discover.py +++ b/src/sqlbuild/compiler/discovery/main/discover.py @@ -13,6 +13,9 @@ validate_project_python_paths, ) from sqlbuild.compiler.discovery._helpers.validation.discovery import validate_discovered_inputs +from sqlbuild.compiler.discovery._helpers.validation.target_connections import ( + validate_target_connections, +) from sqlbuild.compiler.discovery._helpers.yml.project import ( load_local_config, load_project_config, @@ -104,6 +107,9 @@ def _assemble_discovered_project_inputs( validate_target_warehouses_adapter( project_dir=project_dir, project_config=project_config, local_config=local_config ) + validate_target_connections( + project_dir=project_dir, project_config=project_config, local_config=local_config + ) validate_project_python_paths(project_dir=project_dir) sql_analysis_enabled: bool = ( sql_analysis_enabled_override diff --git a/src/sqlbuild/spec/contracts/_helpers/targets.py b/src/sqlbuild/spec/contracts/_helpers/targets.py index 8a5ae5395d..0705dd3d2e 100644 --- a/src/sqlbuild/spec/contracts/_helpers/targets.py +++ b/src/sqlbuild/spec/contracts/_helpers/targets.py @@ -2,6 +2,8 @@ from __future__ import annotations +from dataclasses import replace + from sqlbuild.spec.contracts.exceptions import SpecConfigError from sqlbuild.spec.contracts.models import ( ClonePolicy, @@ -44,6 +46,53 @@ def resolve_target_config( ) -> TargetConfig: """Merge project target config with local developer overrides.""" + return _apply_default_connection( + project_config=project_config, + local_config=local_config, + target_name=target_name, + target_config=_merge_target_config( + project_config=project_config, + local_config=local_config, + target_name=target_name, + ), + ) + + +def _apply_default_connection( + *, + project_config: ProjectConfig, + local_config: LocalConfig, + target_name: str, + target_config: TargetConfig, +) -> TargetConfig: + """Select the only named connection for a target that does not choose one.""" + + if ( + target_config.connection_name is not None + or target_config.connection + or project_config.connection + or local_config.connection + ): + return target_config + connection_names: list[str] = sorted({*project_config.connections, *local_config.connections}) + if not connection_names: + return target_config + if len(connection_names) > 1: + names: str = ", ".join(connection_names) + raise SpecConfigError( + f"targets.{target_name} does not set connection and several named connections " + f'are defined ({names}); set connection = "" on [targets.{target_name}] in ' + "sqlbuild_project.toml or sqlbuild_local.toml" + ) + return replace(target_config, connection_name=connection_names[0]) + + +def _merge_target_config( + *, + project_config: ProjectConfig, + local_config: LocalConfig, + target_name: str, +) -> TargetConfig: project_target: TargetConfig = project_config.targets.get(target_name, TargetConfig()) local_target: LocalTargetConfig | None = local_config.targets.get(target_name) if local_target is None: diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/helpers.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/helpers.py index dd23c1fc04..40101e5a8f 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/compile/helpers.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/helpers.py @@ -1460,6 +1460,9 @@ def _layered_write_project_config(*, project_dir: Path) -> None: adapter = "duckdb" default_target = "dev" +[connections.local] +database = ":memory:" + [settings] column_contract_mode = "explicit" diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_compile_type_proof_rules.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_compile_type_proof_rules.py index 6cc5f4990d..27675056a5 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_compile_type_proof_rules.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_compile_type_proof_rules.py @@ -18,7 +18,8 @@ "note: skipped type-proof rules (SQBRCONTRACT105) because SQL analysis is disabled" ) _PROJECT_TOML: str = ( - 'name = "orders"\nadapter = "duckdb"\n\n[rules]\nselect = ["SQBRCONTRACT105"]\n' + 'name = "orders"\nadapter = "duckdb"\n\n[connection]\ndatabase = ":memory:"\n\n' + '[rules]\nselect = ["SQBRCONTRACT105"]\n' ) _ORDERS_MODEL: str = ( "MODEL (\n" diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_python_project_layout.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_python_project_layout.py index 1df9bffb70..6fc4af0c65 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_python_project_layout.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_python_project_layout.py @@ -316,7 +316,9 @@ def test_given_python_root_project_when_compiling_and_building_then_nodes_run_wi FactoryModuleNodePlanTestCase( description="factory returning its module-level task registers one node", repo_files={ - "sqlbuild_project.toml": 'name = "factory_layout"\nadapter = "duckdb"\n', + "sqlbuild_project.toml": ( + 'name = "factory_layout"\nadapter = "duckdb"\n\n[connection]\ndatabase = ":memory:"\n' + ), "python/factories/orders.py": ( "from sqlbuild.factories import factory\n" "from sqlbuild.tasks import task\n\n\n" @@ -333,7 +335,9 @@ def test_given_python_root_project_when_compiling_and_building_then_nodes_run_wi FactoryModuleNodePlanTestCase( description="factory binding its generated task to a module global registers it", repo_files={ - "sqlbuild_project.toml": 'name = "factory_layout"\nadapter = "duckdb"\n', + "sqlbuild_project.toml": ( + 'name = "factory_layout"\nadapter = "duckdb"\n\n[connection]\ndatabase = ":memory:"\n' + ), "python/factories/orders.py": ( "from sqlbuild.factories import factory\n" "from sqlbuild.tasks import task\n\n\n" @@ -380,7 +384,9 @@ def test_given_factory_returning_module_node_when_planning_then_node_registers_o PythonProjectLayoutCompileTestCase( description="factory returning a distinct same-name task still conflicts", repo_files={ - "sqlbuild_project.toml": 'name = "factory_layout"\nadapter = "duckdb"\n', + "sqlbuild_project.toml": ( + 'name = "factory_layout"\nadapter = "duckdb"\n\n[connection]\ndatabase = ":memory:"\n' + ), "python/factories/orders.py": ( "from sqlbuild.factories import factory\n" "from sqlbuild.tasks import task\n\n\n" @@ -430,7 +436,9 @@ def test_given_factory_returning_distinct_same_name_node_when_planning_then_conf "python/helpers.py": "raise RuntimeError('unrelated python package imported')\n", }, repo_files={ - "sqlbuild_project.toml": 'name = "python_collision"\nadapter = "duckdb"\n', + "sqlbuild_project.toml": ( + 'name = "python_collision"\nadapter = "duckdb"\n\n[connection]\ndatabase = ":memory:"\n' + ), "python/helpers/__init__.py": "", "python/helpers/values.py": "STATUS = 'shipped'\n", "python/helpers/clean.py": ( diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/debug/test_debug.py b/tests/e2e/src/sqlbuild/cli/commands/main/debug/test_debug.py index ef2e0c5869..d323f908b4 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/debug/test_debug.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/debug/test_debug.py @@ -145,6 +145,9 @@ def test_given_local_default_and_explicit_target_when_debugging_text_then_report adapter = "duckdb" default_target = "dev" + [connections.local] + database = "debug.duckdb" + [targets.dev] database = "dev.duckdb" schema = "dev_orders" @@ -178,7 +181,7 @@ def test_given_local_default_and_explicit_target_when_debugging_text_then_report assert ( " adapter: duckdb [OK found]\n" " target: test [OK resolved]\n" - " connection: inline [OK resolved]\n" + " connection: local [OK resolved]\n" " database: local-test.duckdb [OK resolved]\n" " schema: local_test_orders [OK resolved]\n" ) in result.stdout @@ -209,6 +212,9 @@ def test_given_local_default_and_explicit_target_when_debugging_json_then_report adapter = "duckdb" default_target = "dev" + [connections.local] + database = "debug.duckdb" + [targets.dev] database = "dev.duckdb" schema = "dev_orders" @@ -248,7 +254,7 @@ def test_given_local_default_and_explicit_target_when_debugging_json_then_report "project": "debug_target_json_project", "adapter": "duckdb", "target": "test", - "connection": "inline", + "connection": "local", "database": "local-test.duckdb", "schema": "local_test_orders", } diff --git a/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py b/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py index 99cfcd2840..337cf7677d 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/_test_types.py @@ -1029,3 +1029,24 @@ class ReusedCompileImportFootprintTestCase: expected_exit_code: int expected_reuse_message: str forbidden_modules: tuple[str, ...] + + +@dataclass(frozen=True) +class DefaultConnectionCliTestCase: + """One CLI command against a project whose target does not set connection.""" + + description: str + project_toml: str + argv: tuple[str, ...] + expected_exit_code: int + expected_output_fragments: tuple[str, ...] + expected_order_rows: tuple[tuple[object, ...], ...] | None = None + + +@dataclass(frozen=True) +class InferredFreshnessCliTestCase: + """One source whose freshness strategy and type come from its declaration.""" + + description: str + sources_yaml: str + expected_sources: tuple[tuple[object, ...], ...] diff --git a/tests/integration/src/sqlbuild/cli/commands/main/helpers.py b/tests/integration/src/sqlbuild/cli/commands/main/helpers.py index 56aaa5a5bf..575c9be04b 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/helpers.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/helpers.py @@ -780,3 +780,32 @@ def write_snowflake_warehouse_project( "MODEL (description 'Orders.', materialized table);\nSELECT 1 AS order_id\n", encoding="utf-8", ) + + +def write_default_connection_project(*, project_dir: Path, project_toml: str) -> None: + """Write a one-model DuckDB project with the given project config.""" + + (project_dir / "models").mkdir() + _ = (project_dir / "sqlbuild_project.toml").write_text(project_toml, encoding="utf-8") + _ = (project_dir / "models" / "orders.sql").write_text( + 'MODEL (description "Orders.", materialized table);\n\nSELECT 1 AS order_id\n', + encoding="utf-8", + ) + + +def write_default_connection_sql_test(*, project_dir: Path) -> None: + """Add a downstream model and a SQL test that mocks its orders dependency.""" + + _ = (project_dir / "models" / "order_totals.sql").write_text( + 'MODEL (description "Order totals.", materialized table);\n\n' + 'SELECT order_id FROM __ref("orders")\n', + encoding="utf-8", + ) + (project_dir / "tests" / "unit").mkdir(parents=True) + _ = (project_dir / "tests" / "unit" / "test_order_totals.sql").write_text( + 'TEST (name "order_totals_value");\n\n' + "WITH __ref__orders AS (SELECT 1 AS order_id),\n" + "__expected__order_totals AS (SELECT 1 AS order_id)\n" + "SELECT 1\n", + encoding="utf-8", + ) diff --git a/tests/integration/src/sqlbuild/cli/commands/main/test_default_connection.py b/tests/integration/src/sqlbuild/cli/commands/main/test_default_connection.py new file mode 100644 index 0000000000..dc753f24b6 --- /dev/null +++ b/tests/integration/src/sqlbuild/cli/commands/main/test_default_connection.py @@ -0,0 +1,253 @@ +"""Real CLI coverage for targets that rely on the only named connection.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from sqlbuild.cli.commands.main.entrypoint.entry import main +from tests.integration.src.sqlbuild.cli.commands.main._test_types import ( + DefaultConnectionCliTestCase, + InferredFreshnessCliTestCase, +) +from tests.integration.src.sqlbuild.cli.commands.main.helpers import ( + execute_duckdb_sql, + query_duckdb_rows, + write_default_connection_project, + write_default_connection_sql_test, +) + +_ONE_CONNECTION_PROJECT: str = ( + 'name = "shop"\n' + 'adapter = "duckdb"\n' + 'default_target = "dev"\n\n' + "[connections.local]\n" + 'database = "shop.duckdb"\n\n' + "[targets.dev]\n" + 'schema = "dev"\n' +) +_SEVERAL_CONNECTIONS_PROJECT: str = ( + 'name = "shop"\n' + 'adapter = "duckdb"\n' + 'default_target = "dev"\n\n' + "[connections.local]\n" + 'database = "shop.duckdb"\n\n' + "[connections.archive]\n" + 'database = "archive.duckdb"\n\n' + "[targets.dev]\n" + 'schema = "dev"\n' +) +_NO_CONNECTION_PROJECT: str = ( + 'name = "shop"\nadapter = "duckdb"\ndefault_target = "dev"\n\n[targets.dev]\nschema = "dev"\n' +) +_MEMORY_CONNECTION_PROJECT: str = ( + 'name = "shop"\n' + 'adapter = "duckdb"\n' + 'default_target = "dev"\n\n' + "[connections.memory]\n" + 'database = ":memory:"\n\n' + "[targets.dev]\n" + 'schema = "main"\n' +) + + +@pytest.mark.parametrize( + "test_case", + [ + DefaultConnectionCliTestCase( + description="build writes into the only named connection", + project_toml=_ONE_CONNECTION_PROJECT, + argv=("build",), + expected_exit_code=0, + expected_output_fragments=("Completed successfully",), + expected_order_rows=((1,),), + ), + ], + ids=lambda case: case.description, +) +def test_given_target_without_connection_when_building_then_only_named_connection_is_used( + test_case: DefaultConnectionCliTestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + write_default_connection_project(project_dir=tmp_path, project_toml=test_case.project_toml) + + exit_code: int = main(["--no-color", "--project-dir", str(tmp_path), *test_case.argv]) + + output: str = "".join(capsys.readouterr()) + assert exit_code == test_case.expected_exit_code, output + assert all(fragment in output for fragment in test_case.expected_output_fragments), output + assert ( + query_duckdb_rows(db_path=tmp_path / "shop.duckdb", sql="SELECT order_id FROM dev.orders") + == test_case.expected_order_rows + ) + + +@pytest.mark.parametrize( + "test_case", + [ + DefaultConnectionCliTestCase( + description="debug reports the only named connection", + project_toml=_ONE_CONNECTION_PROJECT, + argv=("debug", "--no-connection"), + expected_exit_code=0, + expected_output_fragments=("connection: local", "shop.duckdb"), + ), + DefaultConnectionCliTestCase( + description="build fails at config load among several named connections", + project_toml=_SEVERAL_CONNECTIONS_PROJECT, + argv=("build",), + expected_exit_code=1, + expected_output_fragments=( + "error[D001]", + "targets.dev does not set connection and several named connections are " + "defined (archive, local)", + ), + ), + DefaultConnectionCliTestCase( + description="build fails before connecting without any connection", + project_toml=_NO_CONNECTION_PROJECT, + argv=("build",), + expected_exit_code=1, + expected_output_fragments=( + "error[D001]", + "target dev has no connection; add [connections.] (or set connection = " + '"" on [targets.dev]) in sqlbuild_project.toml or sqlbuild_local.toml', + ), + ), + DefaultConnectionCliTestCase( + description="compile works without any connection", + project_toml=_NO_CONNECTION_PROJECT, + argv=("compile",), + expected_exit_code=0, + expected_output_fragments=("Project compiled",), + ), + DefaultConnectionCliTestCase( + description="freshness without selected sources works without any connection", + project_toml=_NO_CONNECTION_PROJECT, + argv=("freshness",), + expected_exit_code=0, + expected_output_fragments=("OBSERVED=0",), + ), + DefaultConnectionCliTestCase( + description="build uses an explicit in-memory connection", + project_toml=_MEMORY_CONNECTION_PROJECT, + argv=("build",), + expected_exit_code=0, + expected_output_fragments=("Completed successfully",), + ), + ], + ids=lambda case: case.description, +) +def test_given_target_without_connection_when_resolving_offline_then_connection_is_reported( + test_case: DefaultConnectionCliTestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + write_default_connection_project(project_dir=tmp_path, project_toml=test_case.project_toml) + + exit_code: int = main(["--no-color", "--project-dir", str(tmp_path), *test_case.argv]) + + output: str = "".join(capsys.readouterr()) + assert exit_code == test_case.expected_exit_code, output + assert all(fragment in output for fragment in test_case.expected_output_fragments), output + assert not (tmp_path / "shop.duckdb").exists() + + +@pytest.mark.parametrize( + "test_case", + [ + DefaultConnectionCliTestCase( + description="test inspection works without any connection", + project_toml=_NO_CONNECTION_PROJECT, + argv=("test", "--inspect"), + expected_exit_code=0, + expected_output_fragments=( + "boundary: orders is replaced by __ref__orders", + "Test plan inspection complete: 1 selected, 0 errors.", + ), + ), + DefaultConnectionCliTestCase( + description="test execution fails before connecting without any connection", + project_toml=_NO_CONNECTION_PROJECT, + argv=("test",), + expected_exit_code=1, + expected_output_fragments=("error[D001]", "target dev has no connection"), + ), + ], + ids=lambda case: case.description, +) +def test_given_no_connection_when_running_sql_tests_then_only_inspection_succeeds( + test_case: DefaultConnectionCliTestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + write_default_connection_project(project_dir=tmp_path, project_toml=test_case.project_toml) + write_default_connection_sql_test(project_dir=tmp_path) + + exit_code: int = main(["--no-color", "--project-dir", str(tmp_path), *test_case.argv]) + + output: str = "".join(capsys.readouterr()) + assert exit_code == test_case.expected_exit_code, output + assert all(fragment in output for fragment in test_case.expected_output_fragments), output + assert "Connecting to" not in output + assert not (tmp_path / "shop.duckdb").exists() + + +@pytest.mark.parametrize( + "test_case", + [ + InferredFreshnessCliTestCase( + description="column key and declared timestamp column imply column timestamp", + sources_yaml=( + "sources:\n" + " - name: raw_orders\n" + " description: Raw orders.\n" + " schema: raw\n" + " table: orders\n" + " columns:\n" + " - name: updated_at\n" + " type: timestamp\n" + " freshness:\n" + " column: updated_at\n" + ), + expected_sources=(("raw_orders", "column", "timestamp", "2026-01-02T03:04:05+00:00"),), + ), + ], + ids=lambda case: case.description, +) +def test_given_freshness_without_strategy_or_type_when_observing_then_both_are_inferred( + test_case: InferredFreshnessCliTestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + write_default_connection_project(project_dir=tmp_path, project_toml=_ONE_CONNECTION_PROJECT) + (tmp_path / "sources").mkdir() + _ = (tmp_path / "sources" / "raw.yml").write_text(test_case.sources_yaml, encoding="utf-8") + execute_duckdb_sql( + db_path=tmp_path / "shop.duckdb", + sql=( + "CREATE SCHEMA raw; " + "CREATE TABLE raw.orders AS SELECT TIMESTAMP '2026-01-02 03:04:05' AS updated_at" + ), + ) + + exit_code: int = main(["--no-color", "--project-dir", str(tmp_path), "freshness", "--json"]) + + captured: str = capsys.readouterr().out + assert exit_code == 0, captured + sources: list[dict[str, object]] = json.loads(captured)["sources"] + assert ( + tuple( + ( + source["name"], + source["strategy"], + source["value_kind"], + source["current_data_version"], + ) + for source in sources + ) + == test_case.expected_sources + ) diff --git a/tests/integration/src/sqlbuild/cli/commands/main/test_rules.py b/tests/integration/src/sqlbuild/cli/commands/main/test_rules.py index a4551570e5..ee2a0d9085 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/test_rules.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/test_rules.py @@ -2513,7 +2513,8 @@ def test_given_typed_unselected_model_when_planning_selection_then_rules_use_sel capsys: pytest.CaptureFixture[str], ) -> None: (tmp_path / "sqlbuild_project.toml").write_text( - 'name = "orders"\nadapter = "duckdb"\n\n[rules]\nselect = ["SQBRCONTRACT105"]\n', + 'name = "orders"\nadapter = "duckdb"\n\n[connection]\ndatabase = ":memory:"\n\n' + '[rules]\nselect = ["SQBRCONTRACT105"]\n', encoding="utf-8", ) sources: Path = tmp_path / "sources" diff --git a/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/_test_types.py b/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/_test_types.py index 26f3acc986..31d2463627 100644 --- a/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/_test_types.py +++ b/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/_test_types.py @@ -1,7 +1,12 @@ from dataclasses import dataclass, field from sqlbuild.cli.commands.types import CliCommand -from sqlbuild.spec.contracts.models import LocalTargetConfig, TargetWarehousesConfig +from sqlbuild.spec.contracts.models import ( + LocalConfig, + LocalTargetConfig, + ProjectConfig, + TargetWarehousesConfig, +) from sqlbuild.spec.contracts.types import WarehouseGroup @@ -96,3 +101,28 @@ class CommandWarehouseGroupTestCase: description: str command: CliCommand expected_group: WarehouseGroup + + +@dataclass(frozen=True) +class MissingConnectionTestCase: + description: str + project_config: ProjectConfig + local_config: LocalConfig + selected_target: str | None + expected_error_fragment: str + + +@dataclass(frozen=True) +class ExplicitMemoryConnectionTestCase: + description: str + project_config: ProjectConfig + local_config: LocalConfig + expected_connection: dict[str, object] + + +@dataclass(frozen=True) +class OfflineConnectionResolutionTestCase: + description: str + project_config: ProjectConfig + local_config: LocalConfig + expected_connection: dict[str, object] diff --git a/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/test_connection.py b/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/test_connection.py index 9fc55c0c80..8ff4420324 100644 --- a/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/test_connection.py +++ b/tests/unit/src/sqlbuild/cli/commands/_helpers/runtime/test_connection.py @@ -12,6 +12,7 @@ resolve_target_connection_config, ) from sqlbuild.compiler.compile.exceptions import CompileInputError +from sqlbuild.compiler.discovery.exceptions import ProjectConfigError from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.integrations.dbt.models import NormalizedDbtProfileConnection from sqlbuild.spec.contracts.models import ( @@ -21,7 +22,10 @@ TargetConfig, ) from tests.unit.src.sqlbuild.cli.commands._helpers.runtime._test_types import ( + ExplicitMemoryConnectionTestCase, + MissingConnectionTestCase, NamedConnectionBehaviorTestCase, + OfflineConnectionResolutionTestCase, ResolveConnectionConfigWarningTestCase, ResolveDbtProfileConnectionConfigTestCase, ResolveEnvironmentConnectionConfigTestCase, @@ -389,3 +393,195 @@ def test_given_target_connection_when_resolving_then_it_expands_effective_config ) assert connection == test_case.expected_connection + + +@pytest.mark.parametrize( + "test_case", + [ + MissingConnectionTestCase( + description="default target without any connection", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + default_target="dev", + targets={"dev": TargetConfig(schema="dev")}, + ), + local_config=LocalConfig(), + selected_target=None, + expected_error_fragment=( + "target dev has no connection; add [connections.] (or set connection = " + '"" on [targets.dev]) in sqlbuild_project.toml or sqlbuild_local.toml' + ), + ), + MissingConnectionTestCase( + description="selected local target without any connection", + project_config=ProjectConfig(name="shop", adapter="duckdb"), + local_config=LocalConfig(targets={"scratch": LocalTargetConfig(schema="scratch")}), + selected_target="scratch", + expected_error_fragment="target scratch has no connection", + ), + MissingConnectionTestCase( + description="no target and no connection", + project_config=ProjectConfig(name="shop", adapter="duckdb"), + local_config=LocalConfig(), + selected_target=None, + expected_error_fragment="no target is selected and no top-level [connection] is configured", + ), + MissingConnectionTestCase( + description="no target selected while only named connections exist", + project_config=ProjectConfig( + name="shop", adapter="duckdb", connections={"local": {"database": "shop.duckdb"}} + ), + local_config=LocalConfig(), + selected_target=None, + expected_error_fragment="no target is selected and no top-level [connection] is configured", + ), + ], + ids=lambda case: case.description, +) +def test_given_no_configured_connection_when_resolving_then_it_fails_before_connecting( + test_case: MissingConnectionTestCase, tmp_path: Path +) -> None: + discovered_inputs: DiscoveredProjectInputs = DiscoveredProjectInputs( + project_config=test_case.project_config, local_config=test_case.local_config + ) + + with pytest.raises(ProjectConfigError) as error_info: + resolve_project_connection_config( + discovered_inputs=discovered_inputs, + project_dir=tmp_path, + selected_target=test_case.selected_target, + ) + + assert error_info.value.code == "D001" + assert test_case.expected_error_fragment in str(error_info.value) + assert str(tmp_path / "sqlbuild_project.toml") in str(error_info.value) + + +@pytest.mark.parametrize( + "test_case", + [ + MissingConnectionTestCase( + description="named target without any connection", + project_config=ProjectConfig( + name="shop", adapter="duckdb", targets={"prod": TargetConfig(schema="prod")} + ), + local_config=LocalConfig(), + selected_target="prod", + expected_error_fragment="target prod has no connection", + ), + ], + ids=lambda case: case.description, +) +def test_given_target_without_connection_when_resolving_target_then_it_fails_before_connecting( + test_case: MissingConnectionTestCase, tmp_path: Path +) -> None: + discovered_inputs: DiscoveredProjectInputs = DiscoveredProjectInputs( + project_config=test_case.project_config, local_config=test_case.local_config + ) + + with pytest.raises(ProjectConfigError, match=test_case.expected_error_fragment): + resolve_target_connection_config( + discovered_inputs=discovered_inputs, + project_dir=tmp_path, + target_name=str(test_case.selected_target), + ) + + +@pytest.mark.parametrize( + "test_case", + [ + ExplicitMemoryConnectionTestCase( + description="named in-memory connection", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + default_target="dev", + connections={"memory": {"database": ":memory:"}}, + targets={"dev": TargetConfig(schema="main")}, + ), + local_config=LocalConfig(), + expected_connection={"database": ":memory:"}, + ), + ExplicitMemoryConnectionTestCase( + description="top level in-memory connection without a target", + project_config=ProjectConfig( + name="shop", adapter="duckdb", connection={"database": ":memory:"} + ), + local_config=LocalConfig(), + expected_connection={"database": ":memory:"}, + ), + ExplicitMemoryConnectionTestCase( + description="local top level in-memory connection", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + default_target="dev", + targets={"dev": TargetConfig(schema="main")}, + ), + local_config=LocalConfig(connection={"database": ":memory:"}), + expected_connection={"database": ":memory:"}, + ), + ExplicitMemoryConnectionTestCase( + description="inline target in-memory connection", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + default_target="dev", + targets={"dev": TargetConfig(schema="main", connection={"database": ":memory:"})}, + ), + local_config=LocalConfig(), + expected_connection={"database": ":memory:"}, + ), + ], + ids=lambda case: case.description, +) +def test_given_explicit_memory_connection_when_resolving_then_it_is_used( + test_case: ExplicitMemoryConnectionTestCase, tmp_path: Path +) -> None: + discovered_inputs: DiscoveredProjectInputs = DiscoveredProjectInputs( + project_config=test_case.project_config, local_config=test_case.local_config + ) + + connection: dict[str, object] = resolve_project_connection_config( + discovered_inputs=discovered_inputs, project_dir=tmp_path + ) + + assert connection == test_case.expected_connection + + +@pytest.mark.parametrize( + "test_case", + [ + OfflineConnectionResolutionTestCase( + description="target without any connection resolves empty for offline modes", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + default_target="dev", + targets={"dev": TargetConfig(schema="dev")}, + ), + local_config=LocalConfig(), + expected_connection={}, + ), + OfflineConnectionResolutionTestCase( + description="no target and no connection resolves empty for offline modes", + project_config=ProjectConfig(name="shop", adapter="duckdb"), + local_config=LocalConfig(), + expected_connection={}, + ), + ], + ids=lambda case: case.description, +) +def test_given_no_configured_connection_when_resolving_for_offline_mode_then_it_is_not_required( + test_case: OfflineConnectionResolutionTestCase, tmp_path: Path +) -> None: + discovered_inputs: DiscoveredProjectInputs = DiscoveredProjectInputs( + project_config=test_case.project_config, local_config=test_case.local_config + ) + + connection: dict[str, object] = resolve_project_connection_config( + discovered_inputs=discovered_inputs, project_dir=tmp_path, require_connection=False + ) + + assert connection == test_case.expected_connection diff --git a/tests/unit/src/sqlbuild/compiler/discovery/_helpers/test_yml_sources.py b/tests/unit/src/sqlbuild/compiler/discovery/_helpers/test_yml_sources.py index e68bbc6dc7..6ee1b29865 100644 --- a/tests/unit/src/sqlbuild/compiler/discovery/_helpers/test_yml_sources.py +++ b/tests/unit/src/sqlbuild/compiler/discovery/_helpers/test_yml_sources.py @@ -286,6 +286,117 @@ expected_freshness_age_policy_warn_afters=(None, "12h", None, "6h", None), expected_freshness_age_policy_error_afters=(None, "24h", None, None, "1d"), ), + ParseSourcesYamlTestCase( + description="infers source freshness strategy and type from keys and columns", + contents=""" + sources: + - name: raw_orders + schema: public + table: orders + freshness: {} + - name: raw_events + schema: public + table: events + columns: + - name: Updated_At + type: timestamp_ntz(9) + freshness: + column: updated_at + lag_tolerance: 15m + - name: raw_customers + schema: public + table: customers + freshness: + query: SELECT MAX(version) FROM raw.public.customers + type: integer + - name: raw_clicks + schema: public + table: clicks + columns: + - name: batch_id + type: BIGINT + freshness: + strategy: column + column: batch_id + - name: raw_tags + schema: public + table: tags + columns: + - name: tag_version + type: character varying(32) + freshness: + column: tag_version + - name: raw_shipments + schema: public + table: shipments + columns: + - name: shipped_on + type: date + freshness: + column: loaded_at + type: timestamp + - name: raw_returns + schema: public + table: returns + columns: + - name: returned_at + type: timestamp with time zone + freshness: + strategy: column + column: returned_at + type: timestamp + """, + expected_source_names=( + "raw_orders", + "raw_events", + "raw_customers", + "raw_clicks", + "raw_tags", + "raw_shipments", + "raw_returns", + ), + expected_column_names=( + (), + ("Updated_At",), + (), + ("batch_id",), + ("tag_version",), + ("shipped_on",), + ("returned_at",), + ), + expected_type_enforcement_values=(None, True, None, True, True, True, True), + expected_contract_values=(None,) * 7, + expected_expressions=(None,) * 7, + expected_source_audit_names=((),) * 7, + expected_column_audit_names=((), ((),), (), ((),), ((),), ((),), ((),)), + expected_freshness_strategies=( + "adapter", + "column", + "sql", + "column", + "column", + "column", + "column", + ), + expected_freshness_value_kinds=( + None, + "timestamp", + "integer", + "integer", + "string", + "timestamp", + "timestamp", + ), + expected_freshness_columns=( + None, + "updated_at", + None, + "batch_id", + "tag_version", + "loaded_at", + "returned_at", + ), + ), ParseSourcesYamlTestCase( description="allows empty sources files with no declarations", contents="{}\n", @@ -1054,6 +1165,84 @@ def test_given_dlt_sources_yaml_when_parsing_then_expands_managed_sources( """, expected_error_fragment="source freshness strategy column requires type", ), + ParseSourcesYamlErrorTestCase( + description="raises when implied column freshness has an untyped column", + contents=""" + sources: + - name: raw_orders + columns: + - name: updated_at + freshness: + column: updated_at + """, + expected_error_fragment="source freshness strategy column requires type", + ), + ParseSourcesYamlErrorTestCase( + description="raises when freshness column type cannot be inferred", + contents=""" + sources: + - name: raw_orders + columns: + - name: ordered_on + type: date + freshness: + column: ordered_on + """, + expected_error_fragment=( + "source freshness cannot infer type from column 'ordered_on' declared as date" + ), + ), + ParseSourcesYamlErrorTestCase( + description="raises when explicit freshness type contradicts the declared column", + contents=""" + sources: + - name: raw_orders + columns: + - name: updated_at + type: timestamp + freshness: + column: updated_at + type: integer + """, + expected_error_fragment=( + "source freshness type integer contradicts column 'updated_at' declared as " + "timestamp; remove type or set it to timestamp" + ), + ), + ParseSourcesYamlErrorTestCase( + description="raises when freshness sets column and query without a strategy", + contents=""" + sources: + - name: raw_orders + freshness: + column: updated_at + query: SELECT MAX(updated_at) FROM raw.orders + type: timestamp + """, + expected_error_fragment="source freshness cannot set both column and query", + ), + ParseSourcesYamlErrorTestCase( + description="raises when freshness sets type without column or query", + contents=""" + sources: + - name: raw_orders + freshness: + type: timestamp + """, + expected_error_fragment="source freshness type and filter require column or query", + ), + ParseSourcesYamlErrorTestCase( + description="raises when explicit freshness strategy contradicts implied keys", + contents=""" + sources: + - name: raw_orders + freshness: + strategy: column + query: SELECT MAX(updated_at) FROM raw.orders + type: timestamp + """, + expected_error_fragment="source freshness strategy column requires column", + ), ParseSourcesYamlErrorTestCase( description="raises when sql freshness has no query", contents=""" diff --git a/tests/unit/src/sqlbuild/compiler/discovery/test_main.py b/tests/unit/src/sqlbuild/compiler/discovery/test_main.py index 85eea8b12a..71d18333b1 100644 --- a/tests/unit/src/sqlbuild/compiler/discovery/test_main.py +++ b/tests/unit/src/sqlbuild/compiler/discovery/test_main.py @@ -6,7 +6,7 @@ import pytest -from sqlbuild.compiler.discovery.exceptions import ProjectPythonPathError +from sqlbuild.compiler.discovery.exceptions import ProjectConfigError, ProjectPythonPathError from sqlbuild.compiler.discovery.main.discover import discover_project_inputs from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs, DiscoveredSqlTestBlock from sqlbuild.observability import EventDispatcher, LifecycleEvent, dispatcher_scope @@ -702,6 +702,55 @@ def test_given_python_in_removed_root_when_discovering_inputs_then_d016_is_raise ) +@pytest.mark.parametrize( + "test_case", + ( + DiscoverProjectInputsErrorTestCase( + description="unselected project target without connection among several", + repo_files={ + "sqlbuild_project.toml": ( + 'name = "demo"\nadapter = "duckdb"\ndefault_target = "dev"\n\n' + '[connections.local]\ndatabase = "demo.duckdb"\n\n' + '[connections.archive]\ndatabase = "archive.duckdb"\n\n' + '[targets.dev]\nconnection = "local"\nschema = "dev"\n\n' + '[targets.prod]\nschema = "prod"\n' + ) + }, + expected_error_fragment=( + "targets.prod does not set connection and several named connections " + "are defined (archive, local)" + ), + ), + DiscoverProjectInputsErrorTestCase( + description="local connection makes a project target ambiguous", + repo_files={ + "sqlbuild_project.toml": ( + 'name = "demo"\nadapter = "duckdb"\n\n' + '[connections.local]\ndatabase = "demo.duckdb"\n\n' + '[targets.dev]\nschema = "dev"\n' + ), + "sqlbuild_local.toml": '[connections.scratch]\ndatabase = "scratch.duckdb"\n', + }, + expected_error_fragment="several named connections are defined (local, scratch)", + ), + ), + ids=lambda case: case.description, +) +def test_given_ambiguous_target_connection_when_discovering_then_config_error_is_raised( + test_case: DiscoverProjectInputsErrorTestCase, + tmp_path: Path, + write_repo_files: Callable[[Path, dict[str, str]], None], +) -> None: + write_repo_files(tmp_path, test_case.repo_files) + + with pytest.raises(ProjectConfigError) as error_info: + discover_project_inputs(project_dir=tmp_path) + + assert error_info.value.code == "D001" + assert test_case.expected_error_fragment in str(error_info.value) + assert str(tmp_path / "sqlbuild_project.toml") in str(error_info.value) + + @pytest.mark.parametrize( "test_case", ( diff --git a/tests/unit/src/sqlbuild/spec/contracts/main/_test_types.py b/tests/unit/src/sqlbuild/spec/contracts/main/_test_types.py index 906d7a16ce..caa388cd75 100644 --- a/tests/unit/src/sqlbuild/spec/contracts/main/_test_types.py +++ b/tests/unit/src/sqlbuild/spec/contracts/main/_test_types.py @@ -74,3 +74,20 @@ class TargetWarehousesResolutionTestCase: project_warehouses: TargetWarehousesConfig local_warehouses: TargetWarehousesConfig expected_warehouses: TargetWarehousesConfig + + +@dataclass(frozen=True) +class TargetConnectionDefaultTestCase: + description: str + project_config: ProjectConfig + local_config: LocalConfig + expected_connection_name: str | None + expected_inline_connection: dict[str, object] = field(default_factory=dict) + + +@dataclass(frozen=True) +class AmbiguousTargetConnectionTestCase: + description: str + project_config: ProjectConfig + local_config: LocalConfig + expected_error_fragment: str diff --git a/tests/unit/src/sqlbuild/spec/contracts/main/test_targets.py b/tests/unit/src/sqlbuild/spec/contracts/main/test_targets.py index 5e566ea9b2..00ee47ce0c 100644 --- a/tests/unit/src/sqlbuild/spec/contracts/main/test_targets.py +++ b/tests/unit/src/sqlbuild/spec/contracts/main/test_targets.py @@ -2,6 +2,7 @@ import pytest +from sqlbuild.spec.contracts.exceptions import SpecConfigError from sqlbuild.spec.contracts.main.resolve_target_config import resolve_target_config from sqlbuild.spec.contracts.models import ( AuthoredTimeTravelRetention, @@ -14,8 +15,10 @@ ) from sqlbuild.spec.contracts.types import MissingMigrationOriginPolicy from tests.unit.src.sqlbuild.spec.contracts.main._test_types import ( + AmbiguousTargetConnectionTestCase, ExecutionLimitsResolutionTestCase, MissingOriginPolicyResolutionTestCase, + TargetConnectionDefaultTestCase, TargetRetentionResolutionTestCase, TargetWarehousesResolutionTestCase, ) @@ -226,3 +229,163 @@ def test_given_local_target_warehouses_when_resolving_then_each_group_overrides_ ) assert target_config.warehouses == test_case.expected_warehouses + + +@pytest.mark.parametrize( + "test_case", + [ + TargetConnectionDefaultTestCase( + description="only project connection is used by a target without one", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + connections={"local": {"database": "shop.duckdb"}}, + targets={"dev": TargetConfig(schema="dev")}, + ), + local_config=LocalConfig(), + expected_connection_name="local", + ), + TargetConnectionDefaultTestCase( + description="local override of the only connection keeps it the default", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + connections={"local": {"database": "shop.duckdb"}}, + targets={"dev": TargetConfig(schema="dev")}, + ), + local_config=LocalConfig(connections={"local": {"database": "mine.duckdb"}}), + expected_connection_name="local", + ), + TargetConnectionDefaultTestCase( + description="only connection defined locally is used by a project target", + project_config=ProjectConfig( + name="shop", adapter="duckdb", targets={"dev": TargetConfig(schema="dev")} + ), + local_config=LocalConfig(connections={"local": {"database": "mine.duckdb"}}), + expected_connection_name="local", + ), + TargetConnectionDefaultTestCase( + description="no named connections keeps the target without one", + project_config=ProjectConfig( + name="shop", adapter="duckdb", targets={"dev": TargetConfig(schema="dev")} + ), + local_config=LocalConfig(), + expected_connection_name=None, + ), + TargetConnectionDefaultTestCase( + description="top level project connection block disables the default", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + connection={"database": "shop.duckdb"}, + connections={"local": {"database": "other.duckdb"}}, + targets={"dev": TargetConfig(schema="dev")}, + ), + local_config=LocalConfig(), + expected_connection_name=None, + ), + TargetConnectionDefaultTestCase( + description="top level local connection block disables the default", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + connections={"local": {"database": "shop.duckdb"}}, + targets={"dev": TargetConfig(schema="dev")}, + ), + local_config=LocalConfig(connection={"database": "mine.duckdb"}), + expected_connection_name=None, + ), + TargetConnectionDefaultTestCase( + description="inline target connection disables the default", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + connections={"local": {"database": "shop.duckdb"}}, + targets={"dev": TargetConfig(schema="dev")}, + ), + local_config=LocalConfig( + targets={"dev": LocalTargetConfig(connection={"database": ":memory:"})} + ), + expected_connection_name=None, + expected_inline_connection={"database": ":memory:"}, + ), + TargetConnectionDefaultTestCase( + description="explicit target connection is kept among several", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + connections={"local": {}, "shared": {}}, + targets={"dev": TargetConfig(schema="dev", connection_name="shared")}, + ), + local_config=LocalConfig(), + expected_connection_name="shared", + ), + TargetConnectionDefaultTestCase( + description="local target connection selects among several", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + connections={"local": {}, "shared": {}}, + targets={"dev": TargetConfig(schema="dev")}, + ), + local_config=LocalConfig(targets={"dev": LocalTargetConfig(connection_name="local")}), + expected_connection_name="local", + ), + ], + ids=lambda case: case.description, +) +def test_given_target_without_connection_when_resolving_then_only_named_connection_is_default( + test_case: TargetConnectionDefaultTestCase, +) -> None: + target_config: TargetConfig = resolve_target_config( + project_config=test_case.project_config, + local_config=test_case.local_config, + target_name="dev", + ) + + assert target_config.connection_name == test_case.expected_connection_name + assert target_config.connection == test_case.expected_inline_connection + + +@pytest.mark.parametrize( + "test_case", + [ + AmbiguousTargetConnectionTestCase( + description="several project connections", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + connections={"shared": {}, "local": {}}, + targets={"dev": TargetConfig(schema="dev")}, + ), + local_config=LocalConfig(), + expected_error_fragment=( + "targets.dev does not set connection and several named connections are defined " + "(local, shared)" + ), + ), + AmbiguousTargetConnectionTestCase( + description="project and local connections together", + project_config=ProjectConfig( + name="shop", + adapter="duckdb", + connections={"shared": {}}, + targets={"dev": TargetConfig(schema="dev")}, + ), + local_config=LocalConfig(connections={"local": {}}), + expected_error_fragment="several named connections are defined (local, shared)", + ), + ], + ids=lambda case: case.description, +) +def test_given_several_connections_and_target_without_one_when_resolving_then_raises( + test_case: AmbiguousTargetConnectionTestCase, +) -> None: + with pytest.raises(SpecConfigError) as error: + _ = resolve_target_config( + project_config=test_case.project_config, + local_config=test_case.local_config, + target_name="dev", + ) + + assert test_case.expected_error_fragment in str(error.value) diff --git a/website/src/content/docs/docs/concepts/project-configuration.mdx b/website/src/content/docs/docs/concepts/project-configuration.mdx index 26ac1afd54..14b25205ec 100644 --- a/website/src/content/docs/docs/concepts/project-configuration.mdx +++ b/website/src/content/docs/docs/concepts/project-configuration.mdx @@ -69,7 +69,12 @@ materialized = "view" |-------|-------------| | `name` | Project name. Used in fingerprint tracking and manifest generation. | | `adapter` | Database adapter: `duckdb`, `motherduck`, `snowflake`, `bigquery`, `databricks`, `postgres`, or `sqlserver`. See [Adapters](/docs/concepts/adapters). | -| `default_target` | Name of the target to build against when none is selected (see [Targets](#targets)). | + +### Optional fields + +| Field | Description | +|-------|-------------| +| `default_target` | Name of the target to build against when none is selected (see [Selecting a target](#selecting-a-target)). | ### Named connections @@ -106,6 +111,33 @@ database = "ANALYTICS" schema = "DEV_ALICE" ``` +When the project defines exactly one named connection, across `sqlbuild_project.toml` and +`sqlbuild_local.toml`, a target that does not set `connection` uses it. This default does not +apply when either file has a legacy top-level `[connection]` block or the target sets an inline +`[targets..connection]` block. With several named connections, every target must choose +one; SQLBuild rejects a target without `connection` while loading configuration. + +```toml +[connections.local] +database = "my_project.duckdb" + +[targets.dev] +schema = "dev" # uses connections.local, the only named connection +``` + +A command that needs the warehouse, such as `build`, `plan`, `query`, `diff`, or `debug`, fails +with `D001` before connecting when no connection is configured at all: no named connection, no +legacy `[connection]` block, and no inline target connection. Commands and modes that do not +connect, such as `compile`, `format`, `lineage`, `scope`, `rename`, `mv`, `test --inspect`, and +`scenario test --local`, keep working without one. +SQLBuild never falls back to an implicit in-memory DuckDB database; to use one, set it +explicitly: + +```toml +[connections.memory] +database = ":memory:" +``` + A `database` or `schema` present in a named connection is connection/session metadata only; it does not satisfy the mandatory namespace strategy for a named target. Put the target's authoritative database and schema on `[targets.]`. SQLBuild validates connection @@ -118,14 +150,14 @@ forms are compatibility syntax, not the canonical format for new or updated proj ## Targets -A target is a named build context - the database and schema you build into, plus execution policy (for example `dev` and `prod`). Each target references a named connection and can configure: +A target is a named build context - the database and schema you build into, plus execution policy (for example `dev` and `prod`). Each target references a named connection (or uses the only one defined; see [Named connections](#named-connections)) and can configure: | Field | Description | |-------|-------------| | `schema` | Schema for all models in this target; required for named targets | | `loader_schema` | Default write schema for managed source loaders; falls back to `schema` | | `database` | Database for all models in this target | -| `connection` | Name from `[connections.]` used for endpoint, authentication, and compute | +| `connection` | Name from `[connections.]` used for endpoint, authentication, and compute; optional when exactly one named connection is defined | | `vars` | Target-specific project variables | | `defer_sources_to` | Target name to read managed source data from (see [Loaders](/docs/concepts/python-nodes/loaders#source-deferral)) | | `clone` | Clone policy (see below) | @@ -174,7 +206,9 @@ The active target is determined by (in order of precedence): 1. `--target` on the command line (highest priority) 2. `sqlbuild_local.toml` `target` field 3. `default_target` in `sqlbuild_project.toml` -4. No target (models build to the default schema) +4. No target (models build to the default schema). Named connections are chosen through + targets, so warehouse commands fail with `D001` unless a target is selected or a legacy + top-level `[connection]` block exists. A typical developer keeps shared target definitions in `sqlbuild_project.toml` and selects their normal target once in the optional local file: diff --git a/website/src/content/docs/docs/concepts/sources.mdx b/website/src/content/docs/docs/concepts/sources.mdx index 1c34ec0595..efd123df59 100644 --- a/website/src/content/docs/docs/concepts/sources.mdx +++ b/website/src/content/docs/docs/concepts/sources.mdx @@ -137,14 +137,35 @@ sources: lag_tolerance: 15m ``` +When the column is declared with a type under `columns:`, `strategy` and `type` can be left out: + +```yaml +sources: + - name: raw_events + schema: raw + table: events + columns: + - name: updated_at + type: timestamp + freshness: + column: updated_at # strategy column, type timestamp + lag_tolerance: 15m +``` + ### Strategies | Strategy | Description | Required fields | |----------|-------------|-----------------| | `adapter` | Uses warehouse metadata (e.g. Snowflake `LAST_ALTERED`). No query against the source table. | None | -| `column` | Reads `MAX(column)` from the source table. | `column`, `type` | +| `column` | Reads `MAX(column)` from the source table. | `column`, `type` (inferred from a typed column declaration) | | `sql` | Runs a custom query that returns a single scalar value. | `query`, `type` | +`strategy` is optional. Without it, SQLBuild infers the strategy from the keys you set: `column` +means `column`, `query` means `sql`, and neither means `adapter`, so `freshness: {}` observes +warehouse metadata. Setting both `column` and `query` without a `strategy` is an error, as is +setting `type` or `filter` without either. An explicit `strategy` keeps working and must agree +with the keys present. + #### adapter ```yaml @@ -163,7 +184,7 @@ freshness: type: timestamp ``` -Queries `MAX(column)` from the source table. The column must be a plain column name (no expressions; use `sql` strategy for those). Requires `type`. +Queries `MAX(column)` from the source table. The column must be a plain column name (no expressions; use `sql` strategy for those). Requires `type`, unless the source declares the column with a type that SQLBuild can map (see [Type](#type)). #### sql @@ -186,6 +207,19 @@ The `type` field declares the value kind for comparison: | `integer` | Integer value. Change detected by exact comparison. | | `string` | String value. Change detected by exact comparison. | +For the `column` strategy, SQLBuild takes `type` from the column's declared type in the source's +`columns:` when it can map it: + +| Declared column type | Freshness `type` | +|----------------------|------------------| +| `TIMESTAMP`, `TIMESTAMP_NTZ`, `TIMESTAMP_LTZ`, `TIMESTAMP_TZ`, `TIMESTAMPTZ`, `TIMESTAMP WITH TIME ZONE`, `DATETIME`, `DATETIME2`, `DATETIMEOFFSET`, `SMALLDATETIME` | `timestamp` | +| `INTEGER`, `INT`, `BIGINT`, `SMALLINT`, `TINYINT`, `INT2`, `INT4`, `INT8`, `INT64` | `integer` | +| `VARCHAR`, `STRING`, `TEXT`, `CHAR`, `CHARACTER`, `CHARACTER VARYING`, `NCHAR`, `NVARCHAR` | `string` | + +Matching ignores case and length or precision arguments. Any other declared type, such as +`DATE` or `NUMBER(38,0)`, needs an explicit `type`; a column declared with an unmapped type and no +`type` is an error. An explicit `type` that contradicts a mapped declared type is also an error. + ### Lag tolerance `lag_tolerance` is optional and only valid with `type: timestamp`. It declares how much the observed timestamp can drift from the previous observation before being treated as a change: