Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,12 @@ materialized = "view"
|-------|-------------|
| `name` | Project name. Used in fingerprint tracking and manifest generation. |
| `adapter` | Database adapter: `duckdb`, `motherduck`, `snowflake`, `bigquery`, `databricks`, `postgres`, or `sqlserver`. See [Adapters](adapters.md). |
| `default_target` | Name of the target to build against when none is selected (see [Targets](#targets)). |

### Optional fields

| Field | Description |
|-------|-------------|
| `default_target` | Name of the target to build against when none is selected (see [Selecting a target](#selecting-a-target)). |

### Named connections

Expand Down Expand Up @@ -129,6 +134,33 @@ database = "ANALYTICS"
schema = "DEV_ALICE"
```

When the project defines exactly one named connection, across `sqlbuild_project.toml` and
`sqlbuild_local.toml`, a target that does not set `connection` uses it. This default does not
apply when either file has a legacy top-level `[connection]` block or the target sets an inline
`[targets.<name>.connection]` block. With several named connections, every target must choose
one; SQLBuild rejects a target without `connection` while loading configuration.

```toml
[connections.local]
database = "my_project.duckdb"

[targets.dev]
schema = "dev" # uses connections.local, the only named connection
```

A command that needs the warehouse, such as `build`, `plan`, `query`, `diff`, or `debug`, fails
with `D001` before connecting when no connection is configured at all: no named connection, no
legacy `[connection]` block, and no inline target connection. Commands and modes that do not
connect, such as `compile`, `format`, `lineage`, `scope`, `rename`, `mv`, `test --inspect`, and
`scenario test --local`, keep working without one.
SQLBuild never falls back to an implicit in-memory DuckDB database; to use one, set it
explicitly:

```toml
[connections.memory]
database = ":memory:"
```

A `database` or `schema` present in a named connection is connection/session metadata only;
it does not satisfy the mandatory namespace strategy for a named target. Put the target's
authoritative database and schema on `[targets.<name>]`. SQLBuild validates connection
Expand All @@ -141,14 +173,14 @@ forms are compatibility syntax, not the canonical format for new or updated proj

## Targets

A target is a named build context - the database and schema you build into, plus execution policy (for example `dev` and `prod`). Each target references a named connection and can configure:
A target is a named build context - the database and schema you build into, plus execution policy (for example `dev` and `prod`). Each target references a named connection (or uses the only one defined; see [Named connections](#named-connections)) and can configure:

| Field | Description |
|-------|-------------|
| `schema` | Schema for all models in this target; required for named targets |
| `loader_schema` | Default write schema for managed source loaders; falls back to `schema` |
| `database` | Database for all models in this target |
| `connection` | Name from `[connections.<name>]` used for endpoint, authentication, and compute |
| `connection` | Name from `[connections.<name>]` used for endpoint, authentication, and compute; optional when exactly one named connection is defined |
| `vars` | Target-specific project variables |
| `defer_sources_to` | Target name to read managed source data from (see [Loaders](python-nodes/loaders.md#source-deferral)) |
| `clone` | Clone policy (see below) |
Expand Down Expand Up @@ -197,7 +229,9 @@ The active target is determined by (in order of precedence):
1. `--target` on the command line (highest priority)
2. `sqlbuild_local.toml` `target` field
3. `default_target` in `sqlbuild_project.toml`
4. No target (models build to the default schema)
4. No target (models build to the default schema). Named connections are chosen through
targets, so warehouse commands fail with `D001` unless a target is selected or a legacy
top-level `[connection]` block exists.

A typical developer keeps shared target definitions in `sqlbuild_project.toml` and selects
their normal target once in the optional local file:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -150,14 +150,35 @@ sources:
lag_tolerance: 15m
```

When the column is declared with a type under `columns:`, `strategy` and `type` can be left out:

```yaml
sources:
- name: raw_events
schema: raw
table: events
columns:
- name: updated_at
type: timestamp
freshness:
column: updated_at # strategy column, type timestamp
lag_tolerance: 15m
```

### Strategies

| Strategy | Description | Required fields |
|----------|-------------|-----------------|
| `adapter` | Uses warehouse metadata (e.g. Snowflake `LAST_ALTERED`). No query against the source table. | None |
| `column` | Reads `MAX(column)` from the source table. | `column`, `type` |
| `column` | Reads `MAX(column)` from the source table. | `column`, `type` (inferred from a typed column declaration) |
| `sql` | Runs a custom query that returns a single scalar value. | `query`, `type` |

`strategy` is optional. Without it, SQLBuild infers the strategy from the keys you set: `column`
means `column`, `query` means `sql`, and neither means `adapter`, so `freshness: {}` observes
warehouse metadata. Setting both `column` and `query` without a `strategy` is an error, as is
setting `type` or `filter` without either. An explicit `strategy` keeps working and must agree
with the keys present.

#### adapter

```yaml
Expand All @@ -176,7 +197,7 @@ freshness:
type: timestamp
```

Queries `MAX(column)` from the source table. The column must be a plain column name (no expressions; use `sql` strategy for those). Requires `type`.
Queries `MAX(column)` from the source table. The column must be a plain column name (no expressions; use `sql` strategy for those). Requires `type`, unless the source declares the column with a type that SQLBuild can map (see [Type](#type)).

#### sql

Expand All @@ -199,6 +220,19 @@ The `type` field declares the value kind for comparison:
| `integer` | Integer value. Change detected by exact comparison. |
| `string` | String value. Change detected by exact comparison. |

For the `column` strategy, SQLBuild takes `type` from the column's declared type in the source's
`columns:` when it can map it:

| Declared column type | Freshness `type` |
|----------------------|------------------|
| `TIMESTAMP`, `TIMESTAMP_NTZ`, `TIMESTAMP_LTZ`, `TIMESTAMP_TZ`, `TIMESTAMPTZ`, `TIMESTAMP WITH TIME ZONE`, `DATETIME`, `DATETIME2`, `DATETIMEOFFSET`, `SMALLDATETIME` | `timestamp` |
| `INTEGER`, `INT`, `BIGINT`, `SMALLINT`, `TINYINT`, `INT2`, `INT4`, `INT8`, `INT64` | `integer` |
| `VARCHAR`, `STRING`, `TEXT`, `CHAR`, `CHARACTER`, `CHARACTER VARYING`, `NCHAR`, `NVARCHAR` | `string` |

Matching ignores case and length or precision arguments. Any other declared type, such as
`DATE` or `NUMBER(38,0)`, needs an explicit `type`; a column declared with an unmapped type and no
`type` is an error. An explicit `type` that contradicts a mapped declared type is also an error.

### Lag tolerance

`lag_tolerance` is optional and only valid with `type: timestamp`. It declares how much the observed timestamp can drift from the previous observation before being treated as a change:
Expand Down
94 changes: 60 additions & 34 deletions src/sqlbuild/cli/commands/_helpers/build_planning/invocation.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,11 @@
from pathlib import Path
from typing import TextIO

from sqlbuild.cli.commands._helpers.runtime.adapter_context import (
resolve_adapter_connection_context,
)
from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter
from sqlbuild.cli.commands._helpers.runtime.adapters import resolve_adapter
from sqlbuild.cli.commands._helpers.runtime.connection import resolve_project_connection_config
from sqlbuild.cli.commands.exceptions import CliUserError
from sqlbuild.cli.commands.models import (
AdapterConnectionContext,
BuildCommandRequest,
BuildInvocation,
)
Expand All @@ -23,6 +22,9 @@
from sqlbuild.compiler.discovery.main.discover import discover_project_inputs
from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs
from sqlbuild.presentation.main.supports_color import supports_color
from sqlbuild.spec.contracts.main.resolve_effective_adapter_name import (
resolve_effective_adapter_name,
)
from sqlbuild.spec.contracts.main.resolve_target_config import resolve_target_config
from sqlbuild.spec.contracts.main.resolve_target_name import resolve_target_name
from sqlbuild.spec.contracts.models import ExecutionLimitsConfig, TargetConfig
Expand All @@ -44,17 +46,35 @@ def resolve_build_invocation(*, request: BuildCommandRequest) -> BuildInvocation
)
if request.defer_to is not None and effective_defer_clone_from is not None:
raise CliUserError("--defer-clone-from cannot be used with --defer-to", code="C408")
adapter_context: AdapterConnectionContext = resolve_adapter_connection_context(
adapter_name: str = resolve_effective_adapter_name(
project_config=discovered_inputs.project_config,
local_config=discovered_inputs.local_config,
)
adapter: BaseAdapter = resolve_adapter(
adapter_name=adapter_name, project_dir=effective_project_dir
)
effective_target_name: str | None = resolve_target_name(
project_config=discovered_inputs.project_config,
local_config=discovered_inputs.local_config,
selected_target=request.selected_target,
)
execution_limits: ExecutionLimitsConfig = _resolve_execution_limits(
discovered_inputs=discovered_inputs,
effective_project_dir=effective_project_dir,
target_name=effective_target_name,
adapter_name=adapter_name,
adapter=adapter,
)
connection_config: dict[str, object] = resolve_project_connection_config(
discovered_inputs=discovered_inputs,
project_dir=effective_project_dir,
selected_target=request.selected_target,
cli_vars=request.cli_vars,
)
machine_output: bool = request.json_output
use_color: bool = not request.no_color and not machine_output and supports_color()
progress_stream: TextIO = sys.stderr if request.debug or machine_output else sys.stdout
reporters: CommandProgressReporters = build_command_progress_reporters(
adapter_name=adapter_context.adapter_name,
adapter_name=adapter_name,
stream=progress_stream,
use_color=use_color,
)
Expand All @@ -63,59 +83,65 @@ def resolve_build_invocation(*, request: BuildCommandRequest) -> BuildInvocation
if request.load_sources is not None
else build_effective_settings_config(discovered_inputs=discovered_inputs).auto_load_sources
)
effective_target_name: str | None = resolve_target_name(
project_config=discovered_inputs.project_config,
local_config=discovered_inputs.local_config,
selected_target=request.selected_target,
return BuildInvocation(
effective_project_dir=effective_project_dir,
discovered_inputs=discovered_inputs,
effective_defer_clone_from=effective_defer_clone_from,
adapter_name=adapter_name,
adapter=adapter,
connection_config=connection_config,
use_color=use_color,
progress_stream=progress_stream,
connection_progress=reporters.connection,
planning_progress=reporters.planning,
should_load_sources=should_load_sources,
effective_target_name=effective_target_name,
execution_limits=execution_limits,
)


def _resolve_execution_limits(
*,
discovered_inputs: DiscoveredProjectInputs,
target_name: str | None,
adapter_name: str,
adapter: BaseAdapter,
) -> ExecutionLimitsConfig:
"""Resolve the target's execution limits and validate them against the project adapter."""

execution_limits: ExecutionLimitsConfig = (
resolve_target_config(
project_config=discovered_inputs.project_config,
local_config=discovered_inputs.local_config,
target_name=effective_target_name,
target_name=target_name,
).execution_limits
if effective_target_name is not None
if target_name is not None
else ExecutionLimitsConfig()
)
target_label: str = target_name or "default"
if (
execution_limits.max_duration is not None
and adapter_context.adapter.execution_duration_limit_seconds is None
and adapter.execution_duration_limit_seconds is None
):
target_label: str = effective_target_name or "default"
raise CliUserError(
f"Target '{target_label}' configures max_duration, but adapter "
f"'{adapter_context.adapter_name}' cannot cancel active statements safely",
f"'{adapter_name}' cannot cancel active statements safely",
code="C415",
help=execution_limits.remediation,
)
maximum_duration_seconds: int | None = adapter_context.adapter.execution_duration_limit_seconds
maximum_duration_seconds: int | None = adapter.execution_duration_limit_seconds
if (
execution_limits.max_duration_seconds is not None
and maximum_duration_seconds is not None
and execution_limits.max_duration_seconds > maximum_duration_seconds
):
target_label = effective_target_name or "default"
raise CliUserError(
f"Target '{target_label}' max_duration exceeds the largest duration supported by "
f"adapter '{adapter_context.adapter_name}' ({maximum_duration_seconds} seconds)",
f"adapter '{adapter_name}' ({maximum_duration_seconds} seconds)",
code="C416",
help=execution_limits.remediation,
)
return BuildInvocation(
effective_project_dir=effective_project_dir,
discovered_inputs=discovered_inputs,
effective_defer_clone_from=effective_defer_clone_from,
adapter_name=adapter_context.adapter_name,
adapter=adapter_context.adapter,
connection_config=adapter_context.connection_config,
use_color=use_color,
progress_stream=progress_stream,
connection_progress=reporters.connection,
planning_progress=reporters.planning,
should_load_sources=should_load_sources,
effective_target_name=effective_target_name,
execution_limits=execution_limits,
)
return execution_limits


def _resolve_defer_clone_from(
Expand Down
2 changes: 2 additions & 0 deletions src/sqlbuild/cli/commands/_helpers/runtime/adapter_context.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ def resolve_adapter_connection_context(
effective_project_dir: Path,
selected_target: str | None,
cli_vars: dict[str, object] | None,
require_connection: bool = True,
) -> AdapterConnectionContext:
"""Resolve the effective adapter and project connection configuration."""

Expand All @@ -36,6 +37,7 @@ def resolve_adapter_connection_context(
project_dir=effective_project_dir,
selected_target=selected_target,
cli_vars=cli_vars,
require_connection=require_connection,
)
return AdapterConnectionContext(
adapter_name=adapter_name,
Expand Down
16 changes: 12 additions & 4 deletions src/sqlbuild/cli/commands/_helpers/runtime/command_invocation.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,9 @@ def resolve_command_invocation[InvocationT](
) -> InvocationT:
"""Resolve discovery, adapter, connection, and output context for one command."""

context: _CommandInvocationContext = _resolve_command_invocation_context(request=request)
context: _CommandInvocationContext = _resolve_command_invocation_context(
request=request, require_connection=True
)
return invocation_type(
effective_project_dir=context.effective_project_dir,
discovered_inputs=context.discovered_inputs,
Expand All @@ -67,11 +69,16 @@ def resolve_command_invocation[InvocationT](


def resolve_reported_command_invocation[InvocationT](
*, request: _CommandInvocationRequest, invocation_type: Callable[..., InvocationT]
*,
request: _CommandInvocationRequest,
invocation_type: Callable[..., InvocationT],
require_connection: bool = True,
) -> InvocationT:
"""Resolve command context plus shared connection and planning progress reporters."""

context: _CommandInvocationContext = _resolve_command_invocation_context(request=request)
context: _CommandInvocationContext = _resolve_command_invocation_context(
request=request, require_connection=require_connection
)
reporters: CommandProgressReporters = build_command_progress_reporters(
adapter_name=context.adapter_context.adapter_name,
stream=context.progress_stream,
Expand All @@ -91,7 +98,7 @@ def resolve_reported_command_invocation[InvocationT](


def _resolve_command_invocation_context(
*, request: _CommandInvocationRequest
*, request: _CommandInvocationRequest, require_connection: bool
) -> _CommandInvocationContext:
effective_project_dir: Path = (
request.project_dir if request.project_dir is not None else Path.cwd()
Expand All @@ -104,6 +111,7 @@ def _resolve_command_invocation_context(
effective_project_dir=effective_project_dir,
selected_target=request.selected_target,
cli_vars=request.cli_vars,
require_connection=require_connection,
)
machine_output: bool = request.json_output
use_color: bool = not request.no_color and not machine_output and supports_color()
Expand Down
Loading
Loading