diff --git a/fensu.toml b/fensu.toml index 55dc1cd71d..8394307b0a 100644 --- a/fensu.toml +++ b/fensu.toml @@ -93,6 +93,12 @@ rule = "FFR601" path = "src/sqlbuild/adapters/sqlserver/classes/sqlserver_adapter.py" reason = "The SQL Server adapter intentionally keeps overrideable backend behavior on one class." +[[targets.python.rule_exceptions]] +rule = "XSB051" +path = "src/sqlbuild/compiler/planner/_helpers/warehouse/source_columns.py" +symbols = ["gather_source_inspection"] +reason = "Each type-enforced expression source is its own query whose output columns can only be read by running it; table sources are listed in one batch per database." + [[targets.python.rule_exceptions]] rule = "XSB051" path = "src/sqlbuild/executor/migrations/_helpers/columns.py" diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md index 514e82f8a8..25ce7ecc98 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md @@ -74,6 +74,12 @@ Summary: observed=3 changed=0 unchanged=0 tolerated=0 unknown=0 errors=0 Sources without explicit `freshness:` config are auto-observed using the `adapter` strategy if the adapter supports table metadata. Sources that can't be observed (expression sources, managed sources without freshness config on unsupported adapters) show as `unknown`. +Each source is reported on its own. Adapter metadata is fetched in one batched lookup, but a +source the metadata can't describe affects only that source: it shows as `unknown` with the +reason (for example a view, an external table, a table with no last-modified time, or a table the +metadata lookup did not find). A column or SQL freshness query that fails shows as `error`. The +other sources are still observed. + ## Comparing against state Use `--state` to compare current observations against the freshness state stored from the last successful build: @@ -105,7 +111,7 @@ Summary: observed=0 changed=1 unchanged=1 tolerated=1 unknown=0 errors=0 | `changed` | Data version differs from the stored state | | `unchanged` | Data version matches the stored state exactly | | `tolerated` | Data version differs but is within the `lag_tolerance` threshold | -| `unknown` | No freshness config and adapter metadata unavailable, or no previous state to compare against | +| `unknown` | No freshness config and adapter metadata unavailable, metadata unavailable for an existing table, or no previous state to compare against | | `error` | Observation failed (e.g. source table does not exist, query error) | ## CI integration diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md index ba85022841..8c9867e347 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md @@ -12,6 +12,7 @@ Online: https://sqlbuild.com/docs/cli/plan/ - Flags - Output - Missing upstream dependencies +- Missing source tables Shows the execution plan without making any changes. Useful for inspecting change detection, backfill policies, and selector scope before building. @@ -122,3 +123,10 @@ error[S301]: cannot build selected scope: audit 'relationships' on 'stg_orders' Load or build the resource first (for a seed, `sqb seed`), or add it to the selection. When both kinds are missing, the error lists the missing upstream dependencies and then the audit reads. + +## Missing source tables + +When a selected model reads an external source whose table does not exist in the warehouse, +planning fails with `S405` before anything runs. The error names each missing source, its table, +the selected models that read it, and the source declaration file and line. See +[Source freshness: Missing source tables](../concepts/planning/source-freshness.md#missing-source-tables). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md index 58f3efcb22..8c6d8a9161 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md @@ -24,6 +24,52 @@ During planning, SQLBuild observes the current data version of each source that Sources without explicit `freshness:` config are auto-observed using the `adapter` strategy if the adapter supports table metadata and the source has a physical table (not an expression source, not a managed source). +Adapter metadata for all sources is fetched in one batched lookup, but each source gets its own +result. When one source can't be observed, only that source is unknown; the others keep their +freshness. The `Freshness metadata` progress line reports the outcome, for example +`Freshness metadata OK (257 sources, 3 unknown)`. The plan adds one warning per reason, listing up +to 10 sources, such as `source freshness unknown (unavailable) for raw_events, raw_sessions: ...`. +Sources without freshness configuration that the adapter can't observe are not warned about. +Every unknown source is listed with its reason and message under `unknown_source_details` in the +`direct_source_freshness` metadata of `sqb plan --json`; run `sqb freshness` for the same detail +in text form. + +## Missing source tables + +A source table that a selected model reads must exist before anything runs. If it doesn't, +planning fails with `S405`, naming each missing source, its table, and the selected models that +read it, and pointing at the source declaration: + +``` +error[S405]: cannot build selected scope: 1 source table read by selected resources does not exist in the warehouse: + - source 'raw_payments' (raw.payments), read by payments + = help: create or load the table, correct its database, schema, or table in the source declaration, or stop reading it from the selected models; declared 'raw_payments' at sources/raw.yml:16 +``` + +Create the table, fix the declaration, or leave the reading models out of the selection. Managed +sources and sources with a loader are exempt, because SQLBuild creates their tables when it loads +them. Expression sources, sources without a schema, and wildcard tables are also exempt. A missing +table that only unselected models read does not fail the plan; its freshness is reported as unknown. + +The check first looks for each source in the warehouse listing that planning already reads for +source columns, so it normally issues no extra queries. A source the listing does not find (for +example because of identifier case, a stream, a materialized view, or a table in another database) +is then probed directly with `SELECT 1 FROM WHERE 1=0`, using the same relation name the +model SQL reads. Probes for several such sources run concurrently, within the same limit as other +warehouse inspection reads. Only a not-found error from that probe counts as missing, including a +missing schema, catalog, or attached database; any other error (for example a connection error) +fails planning with that error. On Snowflake the probe may need a running warehouse, but it only +runs for sources the listing did not find. + +Snowflake reports a missing table and a table the current role can't read with the same error, and +its listing hides tables the role can't access. For those sources `S405` says the table "does not +exist or is not readable by" the role, names the role in use when the connection reports it, and the help suggests both fixes: +create or load the table, or grant `SELECT` on it to that role. + +A table that a model's own `pre_hook` creates does not exist yet when the plan is made, so planning +fails with `S405`. Declare such a table as a managed source or a source with a loader, or create it +before planning. + ## Lag tolerance For timestamp-based freshness, `lag_tolerance` controls how much the observed value can drift before being considered a real change. If the current timestamp is within the tolerance of the previous observation, the source is treated as unchanged. This is useful for sources where the freshness timestamp moves by seconds or minutes on every query but the underlying data hasn't meaningfully changed. diff --git a/src/sqlbuild/adapter/contract/classes/base_adapter.py b/src/sqlbuild/adapter/contract/classes/base_adapter.py index 1880fed65f..29fd64a56e 100644 --- a/src/sqlbuild/adapter/contract/classes/base_adapter.py +++ b/src/sqlbuild/adapter/contract/classes/base_adapter.py @@ -30,6 +30,7 @@ AdapterUserError, UnsupportedTypedSqlRenderingError, ) +from sqlbuild.adapter.contract.main.run_relation_read_probe import run_relation_read_probe from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, @@ -41,6 +42,7 @@ QueryResult, RelationGrant, RelationInfo, + RelationReadProbe, RowDiffCoverage, RowDiffPreparedRelations, RowDiffResult, @@ -271,6 +273,16 @@ def describe_relation(self, *, connection: Any, relation: str) -> tuple[ColumnIn cursor: Any = self.execute(connection=connection, sql=f"DESCRIBE {relation}") return tuple(ColumnInfo(name=row[0], type=row[1]) for row in cursor.fetchall()) + def probe_relation_read(self, *, connection: Any, relation: str) -> RelationReadProbe: + """Probe a rendered relation; only the adapter's not-found errors become missing.""" + + return run_relation_read_probe( + execute=self.execute, + connection=connection, + relation=relation, + classify_not_found=lambda error: None, + ) + def get_table_freshness_metadata( self, *, @@ -292,12 +304,15 @@ def get_tables_freshness_metadata( results: dict[TableFreshnessRequest, TableFreshnessMetadata] = {} request: TableFreshnessRequest for request in requests: - results[request] = self.get_table_freshness_metadata( - connection=connection, - database=request.database, - schema=request.schema, - name=request.name, - ) + try: + results[request] = self.get_table_freshness_metadata( + connection=connection, + database=request.database, + schema=request.schema, + name=request.name, + ) + except AdapterUserError as error: + results[request] = TableFreshnessMetadata.unavailable(message=error.message) return results def query_column_names(self, *, connection: Any, sql: str) -> tuple[str, ...]: diff --git a/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py b/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py index 14958f616c..c7a5c1ad76 100644 --- a/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py +++ b/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py @@ -32,9 +32,12 @@ from sqlbuild.adapter.contract.constants import ( DIFF_LEFT_SIDE, DIFF_RIGHT_SIDE, + DUCKDB_MISSING_OBJECT_MARKER, + DUCKDB_NOT_FOUND_ERROR_CLASS_NAMES, QUALIFIED_NAME_SEPARATOR, ) from sqlbuild.adapter.contract.exceptions import AdapterUserError +from sqlbuild.adapter.contract.main.run_relation_read_probe import run_relation_read_probe from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, @@ -46,6 +49,7 @@ QueryResult, RelationGrant, RelationInfo, + RelationReadProbe, RowDiffColumnResult, RowDiffCoverage, RowDiffPreparedRelations, @@ -66,6 +70,7 @@ LoaderLogicalType, MigrationTransfer, PromotionStrategy, + RelationReadStatus, StatementSizeLimit, TablePromotionMode, ) @@ -120,6 +125,23 @@ def supports_relation_age_metadata(self) -> bool: def supports_table_freshness_metadata(self) -> bool: return False + def probe_relation_read(self, *, connection: Any, relation: str) -> RelationReadProbe: + return run_relation_read_probe( + execute=self.execute, + connection=connection, + relation=relation, + classify_not_found=self._classify_relation_not_found, + ) + + @staticmethod + def _classify_relation_not_found(error: BaseException) -> RelationReadStatus | None: + return ( + RelationReadStatus.MISSING + if type(error).__name__ in DUCKDB_NOT_FOUND_ERROR_CLASS_NAMES + and DUCKDB_MISSING_OBJECT_MARKER in str(error) + else None + ) + def get_table_freshness_metadata( self, *, diff --git a/src/sqlbuild/adapter/contract/classes/strict_adapter.py b/src/sqlbuild/adapter/contract/classes/strict_adapter.py index e57a17b2af..71f3bcf5ae 100644 --- a/src/sqlbuild/adapter/contract/classes/strict_adapter.py +++ b/src/sqlbuild/adapter/contract/classes/strict_adapter.py @@ -17,6 +17,7 @@ ExpressionInferenceProfile, MigrationStagePlan, RelationGrant, + RelationReadProbe, RowDiffTolerance, RowDiffTolerances, SnapshotChangeTarget, @@ -134,6 +135,11 @@ def describe_relation(self, *, connection: Any, relation: str) -> tuple[ColumnIn """Return relation column metadata.""" ... + @abstractmethod + def probe_relation_read(self, *, connection: Any, relation: str) -> RelationReadProbe: + """Probe a rendered relation; only the adapter's not-found errors become missing.""" + ... + @abstractmethod def get_table_freshness_metadata( self, @@ -153,7 +159,7 @@ def get_tables_freshness_metadata( connection: Any, requests: tuple[TableFreshnessRequest, ...], ) -> dict[TableFreshnessRequest, TableFreshnessMetadata]: - """Return comparable freshness metadata for physical tables in batch.""" + """Return one freshness outcome per requested table without failing the whole batch.""" ... @abstractmethod diff --git a/src/sqlbuild/adapter/contract/constants.py b/src/sqlbuild/adapter/contract/constants.py index f4f7f1e53a..db92dd0498 100644 --- a/src/sqlbuild/adapter/contract/constants.py +++ b/src/sqlbuild/adapter/contract/constants.py @@ -29,3 +29,7 @@ PYTHON_IDENTIFIER_REPLACEMENT: str = "_" SNAPSHOT_VERSION_START_COLUMN: str = "__version_start" +DUCKDB_NOT_FOUND_ERROR_CLASS_NAMES: frozenset[str] = frozenset( + {"CatalogException", "BinderException"} +) +DUCKDB_MISSING_OBJECT_MARKER: str = "does not exist" diff --git a/src/sqlbuild/adapter/contract/main/complete_table_freshness_results.py b/src/sqlbuild/adapter/contract/main/complete_table_freshness_results.py new file mode 100644 index 0000000000..d5a294d0ce --- /dev/null +++ b/src/sqlbuild/adapter/contract/main/complete_table_freshness_results.py @@ -0,0 +1,24 @@ +"""Complete batch table freshness results with an outcome for every request.""" + +from __future__ import annotations + +from sqlbuild.adapter.contract.models import TableFreshnessMetadata, TableFreshnessRequest + + +def complete_table_freshness_results( + *, + requests: tuple[TableFreshnessRequest, ...], + results: dict[TableFreshnessRequest, TableFreshnessMetadata], + adapter_label: str, +) -> dict[TableFreshnessRequest, TableFreshnessMetadata]: + """Return one outcome per request; a request without a metadata row is unavailable.""" + + completed: dict[TableFreshnessRequest, TableFreshnessMetadata] = dict(results) + request: TableFreshnessRequest + for request in requests: + if request in completed: + continue + completed[request] = TableFreshnessMetadata.unavailable( + message=f"{adapter_label} table freshness metadata not found for {request.name}" + ) + return completed diff --git a/src/sqlbuild/adapter/contract/main/run_relation_read_probe.py b/src/sqlbuild/adapter/contract/main/run_relation_read_probe.py new file mode 100644 index 0000000000..1d5fd1ed53 --- /dev/null +++ b/src/sqlbuild/adapter/contract/main/run_relation_read_probe.py @@ -0,0 +1,48 @@ +"""Direct relation read probe shared by adapter contract implementations.""" + +from __future__ import annotations + +from collections.abc import Callable +from typing import Any + +from sqlbuild.adapter.contract.models import RelationReadProbe +from sqlbuild.adapter.contract.types import AdapterExecute, RelationReadStatus + + +def run_relation_read_probe( + *, + execute: AdapterExecute[Any, Any], + connection: Any, + relation: str, + classify_not_found: Callable[[BaseException], RelationReadStatus | None], +) -> RelationReadProbe: + """Read nothing from a relation; only adapter-classified not-found errors become outcomes.""" + + try: + cursor: object = execute(connection=connection, sql=f"SELECT 1 FROM {relation} WHERE 1=0") + except Exception as error: + link: BaseException + for link in _error_chain(error): + status: RelationReadStatus | None = classify_not_found(link) + if status is not None: + return RelationReadProbe(status=status) + raise + del cursor + return RelationReadProbe(status=RelationReadStatus.READABLE) + + +def _error_chain(error: BaseException) -> tuple[BaseException, ...]: + chain: list[BaseException] = [] + pending: list[BaseException] = [error] + while pending: + current: BaseException = pending.pop() + if any(current is seen for seen in chain): + continue + chain.append(current) + original: object = getattr(current, "__dict__", {}).get("original_error") + pending.extend( + link + for link in (current.__cause__, current.__context__, original) + if isinstance(link, BaseException) + ) + return tuple(chain) diff --git a/src/sqlbuild/adapter/contract/models.py b/src/sqlbuild/adapter/contract/models.py index 15dbe08a7b..36ae6d97ec 100644 --- a/src/sqlbuild/adapter/contract/models.py +++ b/src/sqlbuild/adapter/contract/models.py @@ -16,10 +16,12 @@ HistoricalSnapshotInsertStyle, LifeCycleEventKind, MigrationTransfer, + RelationReadStatus, RetentionChangePhase, RetentionScope, SnapshotLatestVersionStyle, SnapshotUpdateStyle, + TableFreshnessStatus, TypeFamily, ) from sqlbuild.compiler.compile.types import FunctionLanguage @@ -135,13 +137,34 @@ def is_transient(self, *, database: str | None = None, schema: str | None, name: return bool(relation.is_transient) if relation is not None else False +@dataclass(frozen=True) +class RelationReadProbe: + """Whether a rendered relation is readable, and the role used when the adapter knows it.""" + + status: RelationReadStatus + role: str | None = None + + @dataclass(frozen=True) class TableFreshnessMetadata: - """Adapter-observed freshness metadata for one physical table source.""" + """Adapter freshness outcome for one physical table: observed or unavailable.""" data_version: object value_kind: str observed_at: datetime | None = None + status: TableFreshnessStatus = TableFreshnessStatus.OBSERVED + message: str | None = None + + @classmethod + def unavailable(cls, *, message: str) -> TableFreshnessMetadata: + """Return the outcome for a table whose freshness metadata cannot be compared.""" + + return cls( + data_version=None, + value_kind="timestamp", + status=TableFreshnessStatus.UNAVAILABLE, + message=message, + ) @dataclass(frozen=True) diff --git a/src/sqlbuild/adapter/contract/types.py b/src/sqlbuild/adapter/contract/types.py index 6ce905ee68..65a2965d67 100644 --- a/src/sqlbuild/adapter/contract/types.py +++ b/src/sqlbuild/adapter/contract/types.py @@ -26,6 +26,21 @@ class BuiltinAdapter(StrEnum): SQLSERVER = "sqlserver" +class RelationReadStatus(StrEnum): + """Outcome of probing whether a rendered relation can be read.""" + + READABLE = "readable" + MISSING = "missing" + MISSING_OR_UNREADABLE = "missing_or_unreadable" + + +class TableFreshnessStatus(StrEnum): + """Per-table outcome of an adapter freshness metadata lookup.""" + + OBSERVED = "observed" + UNAVAILABLE = "unavailable" + + class CursorKind(StrEnum): TIMESTAMP = "timestamp" INTEGER = "integer" diff --git a/src/sqlbuild/adapters/bigquery/classes/bigquery_adapter.py b/src/sqlbuild/adapters/bigquery/classes/bigquery_adapter.py index b870624511..dd2a4034e9 100644 --- a/src/sqlbuild/adapters/bigquery/classes/bigquery_adapter.py +++ b/src/sqlbuild/adapters/bigquery/classes/bigquery_adapter.py @@ -40,6 +40,10 @@ AdapterUserError, UnsupportedTypedSqlRenderingError, ) +from sqlbuild.adapter.contract.main.complete_table_freshness_results import ( + complete_table_freshness_results, +) +from sqlbuild.adapter.contract.main.run_relation_read_probe import run_relation_read_probe from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, @@ -51,6 +55,7 @@ QueryResult, RelationGrant, RelationInfo, + RelationReadProbe, RenderedRetentionChange, RetentionRequest, RetentionState, @@ -78,11 +83,13 @@ LoaderLogicalType, MigrationTransfer, PromotionStrategy, + RelationReadStatus, RetentionChangePhase, RetentionScope, SnapshotLatestVersionStyle, SnapshotUpdateStyle, StatementSizeLimit, + TableFreshnessStatus, TablePromotionMode, ) from sqlbuild.adapter.relations.main.get_columns_for_relations import ( @@ -455,6 +462,22 @@ def supports_relation_age_metadata(self) -> bool: def supports_table_freshness_metadata(self) -> bool: return True + def probe_relation_read(self, *, connection: Any, relation: str) -> RelationReadProbe: + return run_relation_read_probe( + execute=self.execute, + connection=connection, + relation=relation, + classify_not_found=self._classify_relation_not_found, + ) + + @classmethod + def _classify_relation_not_found(cls, error: BaseException) -> RelationReadStatus | None: + return ( + RelationReadStatus.MISSING + if isinstance(error, Exception) and cls._is_google_not_found(error) + else None + ) + def get_table_freshness_metadata( self, *, @@ -468,10 +491,13 @@ def get_table_freshness_metadata( schema=schema, name=name, ) - return self.get_tables_freshness_metadata( + metadata: TableFreshnessMetadata = self.get_tables_freshness_metadata( connection=connection, requests=(request,), )[request] + if metadata.status != TableFreshnessStatus.OBSERVED: + raise AdapterUserError(message=metadata.message or f"freshness unavailable for {name}") + return metadata def get_tables_freshness_metadata( self, @@ -481,26 +507,29 @@ def get_tables_freshness_metadata( ) -> dict[TableFreshnessRequest, TableFreshnessMetadata]: if not requests: return {} + results: dict[TableFreshnessRequest, TableFreshnessMetadata] = {} + lookup_requests: list[TableFreshnessRequest] = [] request: TableFreshnessRequest for request in requests: if request.schema is None: - raise AdapterUserError( + results[request] = TableFreshnessMetadata.unavailable( message="BigQuery table freshness metadata requires a dataset" ) - if TABLE_NAME_WILDCARD in request.name: - raise AdapterUserError( + elif TABLE_NAME_WILDCARD in request.name: + results[request] = TableFreshnessMetadata.unavailable( message="BigQuery metadata freshness does not support wildcard tables; " "configure a freshness column or query instead" ) + else: + lookup_requests.append(request) requests_by_location_project: dict[tuple[str, str | None], list[TableFreshnessRequest]] = {} - for request in requests: + for request in lookup_requests: location: str = self._metadata_location(connection=connection, request=request) requests_by_location_project.setdefault((location, request.database), []).append( request ) - results: dict[TableFreshnessRequest, TableFreshnessMetadata] = {} grouped_requests: list[TableFreshnessRequest] for (location, database), grouped_requests in requests_by_location_project.items(): clauses: str = " OR ".join( @@ -540,24 +569,19 @@ def get_tables_freshness_metadata( if matched_request is None: continue if row[2] is None: - raise AdapterUserError( + results[matched_request] = TableFreshnessMetadata.unavailable( message="BigQuery table freshness metadata is missing " f"storage_last_modified_time for {matched_request.name}" ) + continue results[matched_request] = TableFreshnessMetadata( data_version=row[2], value_kind="timestamp", observed_at=row[2] if isinstance(row[2], datetime) else None, ) - missing_requests: list[TableFreshnessRequest] = [ - request for request in requests if request not in results - ] - if missing_requests: - missing_names: str = ", ".join(request.name for request in missing_requests) - raise AdapterUserError( - message=f"BigQuery table freshness metadata not found for {missing_names}" - ) - return results + return complete_table_freshness_results( + requests=requests, results=results, adapter_label="BigQuery" + ) def _legacy_tables_freshness_rows( self, diff --git a/src/sqlbuild/adapters/databricks/classes/databricks_adapter.py b/src/sqlbuild/adapters/databricks/classes/databricks_adapter.py index 55325d3429..abbffa3210 100644 --- a/src/sqlbuild/adapters/databricks/classes/databricks_adapter.py +++ b/src/sqlbuild/adapters/databricks/classes/databricks_adapter.py @@ -37,7 +37,11 @@ DIFF_RIGHT_SIDE, ) from sqlbuild.adapter.contract.exceptions import AdapterUserError +from sqlbuild.adapter.contract.main.complete_table_freshness_results import ( + complete_table_freshness_results, +) from sqlbuild.adapter.contract.main.normalize_seed_csv_value import normalize_seed_csv_value +from sqlbuild.adapter.contract.main.run_relation_read_probe import run_relation_read_probe from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, @@ -49,6 +53,7 @@ QueryResult, RelationGrant, RelationInfo, + RelationReadProbe, RenderedRetentionChange, RetentionRequest, RetentionState, @@ -76,12 +81,14 @@ LoaderLogicalType, MigrationTransfer, PromotionStrategy, + RelationReadStatus, RelationType, RetentionChangePhase, RetentionScope, SnapshotLatestVersionStyle, SnapshotUpdateStyle, StatementSizeLimit, + TableFreshnessStatus, TablePromotionMode, ) from sqlbuild.adapter.relations.main.get_columns_for_relations import ( @@ -111,7 +118,9 @@ DELTA_DEFAULT_LOG_RETENTION_DAYS, DELTA_RELATION_FORMAT, NON_ROW_RESULT_COLUMN_NAMES, + RELATION_NOT_FOUND_ERROR_CLASSES, TABLE_RELATION_METADATA_TYPES, + UNDEFINED_TABLE_SQLSTATE, ) from sqlbuild.compiler.compile.types import FunctionLanguage from sqlbuild.compiler.source_freshness.models import SourceFreshnessRecord @@ -545,6 +554,23 @@ def render_prune_source_freshness_history_sql( def supports_table_freshness_metadata(self) -> bool: return True + def probe_relation_read(self, *, connection: Any, relation: str) -> RelationReadProbe: + return run_relation_read_probe( + execute=self.execute, + connection=connection, + relation=relation, + classify_not_found=self._classify_relation_not_found, + ) + + @staticmethod + def _classify_relation_not_found(error: BaseException) -> RelationReadStatus | None: + message: str = str(error) + not_found: bool = ( + any(error_class in message for error_class in RELATION_NOT_FOUND_ERROR_CLASSES) + or getattr(error, "sqlstate", None) == UNDEFINED_TABLE_SQLSTATE + ) + return RelationReadStatus.MISSING if not_found else None + def get_table_freshness_metadata( self, *, @@ -562,9 +588,12 @@ def get_table_freshness_metadata( schema=schema, name=name, ) - return self.get_tables_freshness_metadata(connection=connection, requests=(request,))[ - request - ] + metadata: TableFreshnessMetadata = self.get_tables_freshness_metadata( + connection=connection, requests=(request,) + )[request] + if metadata.status != TableFreshnessStatus.OBSERVED: + raise AdapterUserError(message=metadata.message or f"freshness unavailable for {name}") + return metadata def get_tables_freshness_metadata( self, @@ -574,22 +603,33 @@ def get_tables_freshness_metadata( ) -> dict[TableFreshnessRequest, TableFreshnessMetadata]: if not requests: return {} - request: TableFreshnessRequest - for request in requests: - if request.database is None or request.schema is None: - raise AdapterUserError( - message="Databricks table freshness metadata requires catalog and schema" - ) + results: dict[TableFreshnessRequest, TableFreshnessMetadata] = { + request: TableFreshnessMetadata.unavailable( + message="Databricks table freshness metadata requires catalog and schema" + ) + for request in requests + if request.database is None or request.schema is None + } + lookup_requests: tuple[TableFreshnessRequest, ...] = tuple( + request for request in requests if request not in results + ) + if not lookup_requests: + return results try: - return self._get_delta_history_freshness_metadata( - connection=connection, - requests=requests, + results.update( + self._get_delta_history_freshness_metadata( + connection=connection, + requests=lookup_requests, + ) ) except Exception: - return self._get_unity_catalog_freshness_metadata( - connection=connection, - requests=requests, + results.update( + self._get_unity_catalog_freshness_metadata( + connection=connection, + requests=lookup_requests, + ) ) + return results def _get_unity_catalog_freshness_metadata( self, @@ -637,29 +677,25 @@ def _get_unity_catalog_freshness_metadata( continue table_type: str = str(row[3]) if self._normalize_relation_type(table_type) != RelationType.TABLE: - raise AdapterUserError( + results[matched_request] = TableFreshnessMetadata.unavailable( message="Databricks table freshness metadata only supports Delta tables; " - f"found {table_type}" + f"found {table_type} for {matched_request.name}" ) + continue if row[4] is None: - raise AdapterUserError( + results[matched_request] = TableFreshnessMetadata.unavailable( message="Databricks table freshness metadata is missing LAST_ALTERED " f"for {matched_request.name}" ) + continue results[matched_request] = TableFreshnessMetadata( data_version=row[4], value_kind="timestamp", observed_at=row[4] if isinstance(row[4], datetime) else None, ) - missing_requests: list[TableFreshnessRequest] = [ - request for request in requests if request not in results - ] - if missing_requests: - missing_names: str = ", ".join(request.name for request in missing_requests) - raise AdapterUserError( - message=f"Databricks table freshness metadata not found for {missing_names}" - ) - return results + return complete_table_freshness_results( + requests=requests, results=results, adapter_label="Databricks" + ) def _get_delta_history_freshness_metadata( self, @@ -722,15 +758,9 @@ def _get_delta_history_freshness_metadata( value_kind="timestamp", observed_at=row[3] if isinstance(row[3], datetime) else None, ) - missing_requests: list[TableFreshnessRequest] = [ - request for request in requests if request not in results - ] - if missing_requests: - missing_names: str = ", ".join(request.name for request in missing_requests) - raise AdapterUserError( - message=f"Databricks Delta history not found for {missing_names}" - ) - return results + return complete_table_freshness_results( + requests=requests, results=results, adapter_label="Databricks Delta history" + ) def maximum_identifier_length(self) -> int: """Return the maximum unqualified identifier length supported by the adapter.""" diff --git a/src/sqlbuild/adapters/databricks/constants.py b/src/sqlbuild/adapters/databricks/constants.py index e957c550a8..c0e2a8635b 100644 --- a/src/sqlbuild/adapters/databricks/constants.py +++ b/src/sqlbuild/adapters/databricks/constants.py @@ -7,3 +7,10 @@ DELTA_DEFAULT_DELETED_FILE_RETENTION_DAYS: int = 7 DELTA_COLUMN_MAPPING_PROPERTY: str = "delta.columnMapping.mode" DELTA_COLUMN_MAPPING_NAME_MODE: str = "name" +RELATION_NOT_FOUND_ERROR_CLASSES: tuple[str, ...] = ( + "TABLE_OR_VIEW_NOT_FOUND", + "SCHEMA_NOT_FOUND", + "CATALOG_NOT_FOUND", + "NO_SUCH_CATALOG_EXCEPTION", +) +UNDEFINED_TABLE_SQLSTATE: str = "42P01" diff --git a/src/sqlbuild/adapters/postgres/classes/postgres_adapter.py b/src/sqlbuild/adapters/postgres/classes/postgres_adapter.py index 5bd8eeb213..0d3c7ca00c 100644 --- a/src/sqlbuild/adapters/postgres/classes/postgres_adapter.py +++ b/src/sqlbuild/adapters/postgres/classes/postgres_adapter.py @@ -40,6 +40,7 @@ ) from sqlbuild.adapter.contract.exceptions import AdapterUserError from sqlbuild.adapter.contract.main.normalize_seed_csv_value import normalize_seed_csv_value +from sqlbuild.adapter.contract.main.run_relation_read_probe import run_relation_read_probe from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, @@ -51,6 +52,7 @@ QueryResult, RelationGrant, RelationInfo, + RelationReadProbe, RowDiffColumnResult, RowDiffCoverage, RowDiffPreparedRelations, @@ -75,6 +77,7 @@ LoaderLogicalType, MigrationTransfer, PromotionStrategy, + RelationReadStatus, SnapshotLatestVersionStyle, SnapshotUpdateStyle, StatementSizeLimit, @@ -147,6 +150,23 @@ def supports_relation_age_metadata(self) -> bool: def supports_table_freshness_metadata(self) -> bool: return False + def probe_relation_read(self, *, connection: Any, relation: str) -> RelationReadProbe: + return run_relation_read_probe( + execute=self.execute, + connection=connection, + relation=relation, + classify_not_found=self._classify_relation_not_found, + ) + + @staticmethod + def _classify_relation_not_found(error: BaseException) -> RelationReadStatus | None: + undefined_table: str = "42P01" + codes: tuple[object, ...] = ( + getattr(error, "sqlstate", None), + getattr(error, "pgcode", None), + ) + return RelationReadStatus.MISSING if undefined_table in codes else None + def get_table_freshness_metadata( self, *, diff --git a/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py b/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py index c7a601b7ab..9a24d754e3 100644 --- a/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py +++ b/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py @@ -36,7 +36,11 @@ DIFF_RIGHT_SIDE, ) from sqlbuild.adapter.contract.exceptions import AdapterUserError +from sqlbuild.adapter.contract.main.complete_table_freshness_results import ( + complete_table_freshness_results, +) from sqlbuild.adapter.contract.main.normalize_seed_csv_value import normalize_seed_csv_value +from sqlbuild.adapter.contract.main.run_relation_read_probe import run_relation_read_probe from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, @@ -47,6 +51,7 @@ QueryResult, RelationGrant, RelationInfo, + RelationReadProbe, RenderedRetentionChange, RetentionRequest, RetentionState, @@ -74,6 +79,7 @@ LoaderLogicalType, MigrationTransfer, PromotionStrategy, + RelationReadStatus, RetentionChangePhase, RetentionScope, SnapshotLatestVersionStyle, @@ -727,6 +733,29 @@ def supports_relation_age_metadata(self) -> bool: def supports_table_freshness_metadata(self) -> bool: return True + def probe_relation_read(self, *, connection: Any, relation: str) -> RelationReadProbe: + probe: RelationReadProbe = run_relation_read_probe( + execute=self.execute, + connection=connection, + relation=relation, + classify_not_found=self._classify_relation_not_found, + ) + if probe.status != RelationReadStatus.MISSING_OR_UNREADABLE: + return probe + return RelationReadProbe( + status=probe.status, role=self._current_role_name(connection=connection) + ) + + @staticmethod + def _classify_relation_not_found(error: BaseException) -> RelationReadStatus | None: + ambiguous: bool = isinstance(error, Exception) and is_missing_object_error(error) + return RelationReadStatus.MISSING_OR_UNREADABLE if ambiguous else None + + @staticmethod + def _current_role_name(*, connection: Any) -> str | None: + role: object = getattr(getattr(connection, "raw_connection", None), "role", None) + return str(role) if isinstance(role, str) and role else None + def get_table_freshness_metadata( self, *, @@ -825,29 +854,25 @@ def get_tables_freshness_metadata( continue table_type: str = str(row[3]).upper() if table_type != BASE_TABLE_METADATA_TYPE: - raise AdapterUserError( + results[matched_request] = TableFreshnessMetadata.unavailable( message="Snowflake table freshness metadata only supports physical tables; " - f"found {table_type}" + f"found {table_type} for {matched_request.name}" ) + continue if row[4] is None: - raise AdapterUserError( + results[matched_request] = TableFreshnessMetadata.unavailable( message="Snowflake table freshness metadata is missing LAST_ALTERED " f"for {matched_request.name}" ) + continue results[matched_request] = TableFreshnessMetadata( data_version=row[4], value_kind="timestamp", observed_at=row[4] if isinstance(row[4], datetime) else None, ) - missing_requests: list[TableFreshnessRequest] = [ - request for request in requests if request not in results - ] - if missing_requests: - missing_names: str = ", ".join(request.name for request in missing_requests) - raise AdapterUserError( - message=f"Snowflake table freshness metadata not found for {missing_names}" - ) - return results + return complete_table_freshness_results( + requests=requests, results=results, adapter_label="Snowflake" + ) def persists_python_functions(self) -> bool: return True diff --git a/src/sqlbuild/adapters/sqlserver/classes/sqlserver_adapter.py b/src/sqlbuild/adapters/sqlserver/classes/sqlserver_adapter.py index a7271349bb..abd8a9d163 100644 --- a/src/sqlbuild/adapters/sqlserver/classes/sqlserver_adapter.py +++ b/src/sqlbuild/adapters/sqlserver/classes/sqlserver_adapter.py @@ -42,6 +42,7 @@ UnsupportedTypedSqlRenderingError, ) from sqlbuild.adapter.contract.main.normalize_seed_csv_value import normalize_seed_csv_value +from sqlbuild.adapter.contract.main.run_relation_read_probe import run_relation_read_probe from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, @@ -53,6 +54,7 @@ QueryResult, RelationGrant, RelationInfo, + RelationReadProbe, RowDiffColumnResult, RowDiffCoverage, RowDiffPreparedRelations, @@ -77,6 +79,7 @@ LoaderLogicalType, MigrationTransfer, PromotionStrategy, + RelationReadStatus, SnapshotLatestVersionStyle, SnapshotUpdateStyle, StatementSizeLimit, @@ -150,6 +153,20 @@ def get_columns_for_relations( def supports_table_freshness_metadata(self) -> bool: return False + def probe_relation_read(self, *, connection: Any, relation: str) -> RelationReadProbe: + return run_relation_read_probe( + execute=self.execute, + connection=connection, + relation=relation, + classify_not_found=self._classify_relation_not_found, + ) + + @staticmethod + def _classify_relation_not_found(error: BaseException) -> RelationReadStatus | None: + invalid_object_name: int = 208 + not_found: bool = bool(error.args) and error.args[0] == invalid_object_name + return RelationReadStatus.MISSING if not_found else None + def get_table_freshness_metadata( self, *, diff --git a/src/sqlbuild/cli/commands/_helpers/freshness/observe.py b/src/sqlbuild/cli/commands/_helpers/freshness/observe.py index e885d79e99..03281a4c2b 100644 --- a/src/sqlbuild/cli/commands/_helpers/freshness/observe.py +++ b/src/sqlbuild/cli/commands/_helpers/freshness/observe.py @@ -31,11 +31,16 @@ source_freshness_records_equivalent, ) from sqlbuild.compiler.source_freshness.models import ( + AdapterSourceFreshnessBatch, SourceFreshnessIdentity, SourceFreshnessObservation, SourceFreshnessRecord, + SourceFreshnessUnknown, +) +from sqlbuild.compiler.source_freshness.types import ( + SourceFreshnessAgeStatus, + SourceFreshnessUnknownReason, ) -from sqlbuild.compiler.source_freshness.types import SourceFreshnessAgeStatus from sqlbuild.runtime.observability.main.run_scope import run_scope from sqlbuild.spec.contracts.models import SourceEntry, SourceFreshnessConfig from sqlbuild.spec.contracts.types import SourceFreshnessStrategy @@ -124,11 +129,11 @@ def _observe_source_freshness_for_command( ): adapter_observation_sources.append(observation_source) - adapter_observations: dict[str, SourceFreshnessObservation] = {} + adapter_batch: AdapterSourceFreshnessBatch = AdapterSourceFreshnessBatch() adapter_observation_error: Exception | None = None if adapter_observation_sources: try: - adapter_observations = observe_adapter_sources_freshness( + adapter_batch = observe_adapter_sources_freshness( adapter=adapter, connection=connection, sources=tuple(adapter_observation_sources), @@ -153,7 +158,26 @@ def _observe_source_freshness_for_command( ) ) continue - observation: SourceFreshnessObservation | None = adapter_observations.get(source_name) + source_unknown: SourceFreshnessUnknown | None = adapter_batch.unknown.get(source_name) + if source_unknown is not None: + results.append( + FreshnessSourceResult( + name=source_name, + status=( + FreshnessSourceStatus.UNKNOWN + if source_unknown.reason == SourceFreshnessUnknownReason.UNAVAILABLE + else FreshnessSourceStatus.ERROR + ), + target_database=observation_source.database, + target_schema=observation_source.schema, + target_name=observation_source.table, + message=source_unknown.message, + ) + ) + continue + observation: SourceFreshnessObservation | None = adapter_batch.observations.get( + source_name + ) if observation is None: results.append( FreshnessSourceResult( diff --git a/src/sqlbuild/cli/output/_helpers/plan_text.py b/src/sqlbuild/cli/output/_helpers/plan_text.py index 842342a806..f26cda310d 100644 --- a/src/sqlbuild/cli/output/_helpers/plan_text.py +++ b/src/sqlbuild/cli/output/_helpers/plan_text.py @@ -16,6 +16,7 @@ from sqlbuild.cli.output.models import CursorPlanDetails from sqlbuild.cli.output.types import CursorBoundsOwner, CursorResolutionStatus, PlanRowKind from sqlbuild.compiler.pipeline.models import PythonPlanEntry +from sqlbuild.compiler.planner.constants import SOURCE_FRESHNESS_UNKNOWN_WARNING_TITLE from sqlbuild.compiler.planner.main.changes.query_diff import format_query_diff from sqlbuild.compiler.planner.main.execution.cursor_bound_display import cursor_bound_display from sqlbuild.compiler.planner.main.execution.inclusive_cursor_end import inclusive_cursor_end @@ -1301,6 +1302,10 @@ def _format_warnings( include_stale_input_warnings or not warning.message.startswith(_STALE_INPUT_WARNING_TITLE) ) + and not ( + include_direct_freshness_diagnostics + and warning.message.startswith(SOURCE_FRESHNESS_UNKNOWN_WARNING_TITLE) + ) ] style: CliStyle = CliStyle(use_color=True) lines = _format_diagnostic_section( diff --git a/src/sqlbuild/cli/progress/classes/native_progress_projector.py b/src/sqlbuild/cli/progress/classes/native_progress_projector.py index 24978fe052..d9306be657 100644 --- a/src/sqlbuild/cli/progress/classes/native_progress_projector.py +++ b/src/sqlbuild/cli/progress/classes/native_progress_projector.py @@ -76,6 +76,8 @@ _TTY_RESOURCE_OWNER_OPERATIONS: frozenset[str] = frozenset( {"audit_evaluation", "sql_test_assertion"} ) +_FRESHNESS_METADATA_OPERATION: str = "source_freshness_metadata_observation" +_FRESHNESS_QUERY_OPERATION: str = "source_freshness_query_observation" _RESOURCE_START: str = "resource_attempt_started" _OPERATION_START: str = "operation_started" _OPERATION_TERMINALS: frozenset[str] = frozenset({"operation_completed", "operation_failed"}) @@ -386,8 +388,10 @@ def _consume_operation_terminal(self, event: LifecycleEvent) -> None: ) prefix: str = " " if event.resource_attempt_id is not None else "" attempt: str = _operation_attempt_label(start) + detail: str = _operation_terminal_detail(operation_name=operation_name, event=event) self._write( - f"{prefix}{_VISIBLE_OPERATION_LABELS[operation_name]}{attempt} {status}{elapsed}" + f"{prefix}{_VISIBLE_OPERATION_LABELS[operation_name]}{attempt} " + f"{status}{detail}{elapsed}" ) def _consume_retry_scheduled(self, event: LifecycleEvent) -> None: @@ -515,3 +519,29 @@ def _operation_attempt_label(event: LifecycleEvent) -> str: if isinstance(attempt_number, int) and not isinstance(attempt_number, bool): return f" {attempt_number}" return "" + + +def _operation_terminal_detail(*, operation_name: str, event: LifecycleEvent) -> str: + if operation_name == _FRESHNESS_QUERY_OPERATION: + return " (source freshness unknown)" if event.event_type.endswith("failed") else "" + if operation_name != _FRESHNESS_METADATA_OPERATION: + return "" + metadata: object = event.payload.get("metadata") + typed_metadata: Mapping[str, object] = ( + cast(Mapping[str, object], metadata) if isinstance(metadata, Mapping) else {} + ) + item_count: object = typed_metadata.get("item_count") + if isinstance(item_count, bool) or not isinstance(item_count, int): + return "" + noun: str = "source" if item_count == 1 else "sources" + if event.event_type.endswith("failed"): + error_type: object = event.payload.get("error_type") + reason: str = f": {error_type}" if isinstance(error_type, str) else "" + return f" ({item_count} {noun} unknown, metadata lookup failed{reason})" + unknown_count: object = typed_metadata.get("unknown_count") + unknown: int = ( + unknown_count + if isinstance(unknown_count, int) and not isinstance(unknown_count, bool) + else 0 + ) + return f" ({item_count} {noun}, {unknown} unknown)" diff --git a/src/sqlbuild/compiler/planner/_helpers/output/plan_entry.py b/src/sqlbuild/compiler/planner/_helpers/output/plan_entry.py index a6ec403dcc..5042be73cd 100644 --- a/src/sqlbuild/compiler/planner/_helpers/output/plan_entry.py +++ b/src/sqlbuild/compiler/planner/_helpers/output/plan_entry.py @@ -8,11 +8,7 @@ from typing import Any from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter -from sqlbuild.adapter.contract.models import ColumnInfo, RelationInfo, RelationLookup -from sqlbuild.adapter.relations.main.get_columns_for_inspection import get_columns_for_inspection -from sqlbuild.adapter.relations.main.list_relations_for_inspection import ( - list_relations_for_inspection, -) +from sqlbuild.adapter.contract.models import ColumnInfo from sqlbuild.compiler.compile.constants import CURSOR_INPUTS_CONFIG_KEY from sqlbuild.compiler.compile.exceptions import CompileInputError from sqlbuild.compiler.compile.main._cursor_roles import resolve_cursor_input_roles @@ -78,13 +74,11 @@ from sqlbuild.compiler.planner._helpers.warehouse.semantic_sources import ( get_semantic_source_columns, ) +from sqlbuild.compiler.planner._helpers.warehouse.source_columns import gather_source_inspection from sqlbuild.compiler.planner._helpers.warehouse.source_deferral import ( build_source_read_map, with_declared_source_reads, ) -from sqlbuild.compiler.planner.constants import ( - METADATA_NAME_FILTER_LIMIT, -) from sqlbuild.compiler.planner.exceptions import PlannerInputError from sqlbuild.compiler.planner.main.changes._model_changes import detect_model_changes from sqlbuild.compiler.planner.main.changes.relation_replacement import ( @@ -276,6 +270,15 @@ def build_planner_relations_context( if effective_deferral.source_deferral_enabled else python_source_map ) + source_columns: dict[str, tuple[ColumnInfo, ...]] + listed_source_names: frozenset[str] + source_columns, listed_source_names = _resolve_source_warehouse_columns( + project=project, + adapter=adapter, + connection=connection, + source_read_map=source_read_map, + known_source_columns=known_source_columns, + ) return PlannerRelationsContext( model_locations=model_locations, seed_locations=seed_locations, @@ -289,15 +292,10 @@ def build_planner_relations_context( connection=connection, source_read_map=source_read_map, selected_keys=scope.selected_keys, - columns=_resolve_source_warehouse_columns( - project=project, - adapter=adapter, - connection=connection, - source_read_map=source_read_map, - known_source_columns=known_source_columns, - ), + columns=source_columns, ), star_exclude_keyword=adapter.star_exclude_keyword(), + listed_source_names=listed_source_names, ) @@ -308,16 +306,17 @@ def _resolve_source_warehouse_columns( connection: Any, source_read_map: dict[str, SourceEntry], known_source_columns: dict[str, tuple[ColumnInfo, ...]] | None, -) -> dict[str, tuple[ColumnInfo, ...]]: +) -> tuple[dict[str, tuple[ColumnInfo, ...]], frozenset[str]]: """Reuse already-gathered source columns for the read map or gather them once.""" if known_source_columns is not None: - return { + known: dict[str, tuple[ColumnInfo, ...]] = { name: known_source_columns[name] for name in source_read_map if name in known_source_columns } - return gather_source_columns( + return known, frozenset(known) + return gather_source_inspection( project=project, adapter=adapter, connection=connection, @@ -1024,102 +1023,6 @@ def resolve_cursor_overrides( return resolved_start, resolved_end -def gather_source_columns( - *, - project: CompiledProject, - adapter: BaseAdapter, - connection: Any, - source_entries: tuple[SourceEntry, ...] | None = None, -) -> dict[str, tuple[ColumnInfo, ...]]: - """Gather warehouse columns for all declared sources.""" - - result: dict[str, tuple[ColumnInfo, ...]] = {} - source_schemas: dict[str, set[str]] = {} - entries: tuple[SourceEntry, ...] = ( - source_entries - if source_entries is not None - else tuple(source.source_entry for source in project.sources) - ) - entry: SourceEntry - for entry in entries: - if entry.expression is not None: - if entry.type_enforcement: - column_names: tuple[str, ...] = adapter.query_column_names( - connection=connection, sql=entry.expression - ) - result[entry.name] = tuple(ColumnInfo(name=name, type="") for name in column_names) - continue - schema: str | None = entry.schema - if schema is None: - continue - db: str | None = entry.database - db_key: str = db or "" - source_schemas.setdefault(db_key, set()).add(schema) - - db_key_iter: str - schemas: set[str] - for db_key_iter, schemas in source_schemas.items(): - database: str | None = db_key_iter or None - scoped: dict[str, tuple[str | None, str | None, str]] = { - entry_iter.name: RelationLookup.key( - database=database, - schema=entry_iter.schema, - name=entry_iter.table if entry_iter.table is not None else entry_iter.name, - ) - for entry_iter in entries - if entry_iter.expression is None - and (entry_iter.database or None) == database - and entry_iter.schema in schemas - } - relations: tuple[RelationInfo, ...] = list_relations_for_inspection( - adapter=adapter, - connection=connection, - database=database, - schemas=tuple(sorted(schemas)), - names=_build_source_table_name_filter( - project=project, database=database, schemas=schemas, source_entries=entries - ), - ) - wanted: frozenset[tuple[str | None, str | None, str]] = frozenset(scoped.values()) - all_columns: dict[tuple[str | None, str | None, str], tuple[ColumnInfo, ...]] = ( - get_columns_for_inspection( - adapter=adapter, - connection=connection, - relations=tuple(relation for relation in relations if relation.identity in wanted), - ) - ) - result.update( - {name: all_columns[key] for name, key in scoped.items() if key in all_columns} - ) - - return result - - -def _build_source_table_name_filter( - *, - project: CompiledProject, - database: str | None, - schemas: set[str], - source_entries: tuple[SourceEntry, ...] | None = None, -) -> tuple[str, ...] | None: - names: set[str] = set() - entries: tuple[SourceEntry, ...] = ( - source_entries - if source_entries is not None - else tuple(source.source_entry for source in project.sources) - ) - entry: SourceEntry - for entry in entries: - if entry.expression is not None or entry.schema not in schemas: - continue - if (entry.database or None) != database: - continue - names.add(entry.table if entry.table is not None else entry.name) - if not names or len(names) > METADATA_NAME_FILTER_LIMIT: - return None - return tuple(sorted(names)) - - def _get_on_schema_change(model: CompiledModel) -> OnSchemaChange | None: """Extract on_schema_change from model config.""" diff --git a/src/sqlbuild/compiler/planner/_helpers/planning/buildability.py b/src/sqlbuild/compiler/planner/_helpers/planning/buildability.py index 8b965efd1a..7c20e68fa7 100644 --- a/src/sqlbuild/compiler/planner/_helpers/planning/buildability.py +++ b/src/sqlbuild/compiler/planner/_helpers/planning/buildability.py @@ -8,24 +8,31 @@ missing_upstream_message, ) from sqlbuild.compiler.planner._helpers.graph.core import build_execution_edge_origins +from sqlbuild.compiler.planner._helpers.warehouse.source_tables import ( + check_selected_source_tables_exist, +) from sqlbuild.compiler.planner.exceptions import PlannerInputError from sqlbuild.compiler.planner.models import ( DeferralInputs, MissingUpstream, + PlannerRuntime, PlannerScopeResolution, + PlannerWarehouseState, WarehouseSnapshot, ) def check_selected_scope_buildability( *, - project: CompiledProject, + runtime: PlannerRuntime, scopes: PlannerScopeResolution, - snapshot: WarehouseSnapshot, + warehouse: PlannerWarehouseState, deferral: DeferralInputs, ) -> None: - """Raise a planner input error when selected upstream dependencies are missing.""" + """Raise a planner input error when selected upstream inputs are missing.""" + project: CompiledProject = runtime.project + snapshot: WarehouseSnapshot = warehouse.snapshot external_seed_keys: frozenset[CompiledObjectKey] = frozenset( seed.key for seed in project.seeds if seed.external ) @@ -43,3 +50,10 @@ def check_selected_scope_buildability( ), code="S301", ) + check_selected_source_tables_exist( + project=project, + adapter=runtime.adapter, + connection=runtime.connection, + scope=scopes.selected_scope, + relations=warehouse.inspection_relations, + ) diff --git a/src/sqlbuild/compiler/planner/_helpers/planning/output_assembly.py b/src/sqlbuild/compiler/planner/_helpers/planning/output_assembly.py index 0a1ad90d5e..aa45292c50 100644 --- a/src/sqlbuild/compiler/planner/_helpers/planning/output_assembly.py +++ b/src/sqlbuild/compiler/planner/_helpers/planning/output_assembly.py @@ -13,6 +13,9 @@ from sqlbuild.compiler.planner._helpers.pruning.selection_staleness import ( build_stale_out_of_selection_warnings, ) +from sqlbuild.compiler.planner._helpers.warehouse.source_freshness_warnings import ( + build_source_freshness_unknown_warnings, +) from sqlbuild.compiler.planner.models import ( ColumnRenameHint, ModelPlanEntry, @@ -142,8 +145,16 @@ def with_plan_warnings( plan_output: PlanOutput, policies: PlannerPolicies, ) -> PlanOutput: - """Append stale-out-of-selection warnings to the plan.""" + """Append source freshness and stale-out-of-selection warnings to the plan.""" + freshness_warnings: tuple[PlanWarning, ...] = build_source_freshness_unknown_warnings( + source_freshness=source_freshness + ) + if freshness_warnings: + plan_output = replace( + plan_output, + warnings=(*plan_output.warnings, *freshness_warnings), + ) if not policies.selection_diagnostics: return plan_output stale_out_of_selection_warnings: tuple[PlanWarning, ...] = ( @@ -242,6 +253,14 @@ def _serialize_direct_source_freshness_metadata( "changed_source_names": changed_source_names, "unchanged_source_names": unchanged_source_names, "unknown_source_names": tuple(sorted(source_freshness.unknown_source_names)), + "unknown_source_details": tuple( + { + "source_name": unknown.source_name, + "reason": unknown.reason.value, + "message": unknown.message, + } + for unknown in source_freshness.unknown_sources.values() + ), "age_warning_source_names": age_warning_source_names, "age_error_source_names": age_error_source_names, "stale_model_names": stale_model_names, diff --git a/src/sqlbuild/compiler/planner/_helpers/resolve/sources.py b/src/sqlbuild/compiler/planner/_helpers/resolve/sources.py index c2f6fc1ea4..8f3761cab3 100644 --- a/src/sqlbuild/compiler/planner/_helpers/resolve/sources.py +++ b/src/sqlbuild/compiler/planner/_helpers/resolve/sources.py @@ -122,7 +122,7 @@ def _resolve_source_reference( source_entry: SourceEntry | None = source_map.get(source_name) if source_entry is None: return unknown_source_sql - resolved_source: str = _render_source_relation(adapter=adapter, source_entry=source_entry) + resolved_source: str = render_source_read_relation(adapter=adapter, source_entry=source_entry) warehouse_cols: tuple[ColumnInfo, ...] | None = source_warehouse_columns.get(source_name) if source_entry.expression is None and warehouse_cols: _validate_declared_columns( @@ -239,7 +239,9 @@ def _internal_source_alias(source_name: str) -> str: return f"__sqb_source_{alias_suffix}" -def _render_source_relation(*, adapter: BaseAdapter, source_entry: SourceEntry) -> str: +def render_source_read_relation(*, adapter: BaseAdapter, source_entry: SourceEntry) -> str: + """Render the relation name that planned model SQL reads for one source.""" + if source_entry.expression is not None: return render_source_relation(entry=source_entry, adapter=adapter) table_name: str = source_entry.table if source_entry.table is not None else source_entry.name diff --git a/src/sqlbuild/compiler/planner/_helpers/reuse/clone.py b/src/sqlbuild/compiler/planner/_helpers/reuse/clone.py index aedd035d43..da3a724b58 100644 --- a/src/sqlbuild/compiler/planner/_helpers/reuse/clone.py +++ b/src/sqlbuild/compiler/planner/_helpers/reuse/clone.py @@ -34,9 +34,6 @@ from sqlbuild.compiler.planner._helpers.identity.functions import ( build_compiled_function_fingerprint_sql, ) -from sqlbuild.compiler.planner._helpers.output.plan_entry import ( - gather_source_columns, -) from sqlbuild.compiler.planner._helpers.output.strategy import get_materialization_type from sqlbuild.compiler.planner._helpers.resolve.refs import ( build_function_locations, @@ -47,6 +44,7 @@ resolve_function_sql, resolve_model_sql, ) +from sqlbuild.compiler.planner._helpers.warehouse.source_columns import gather_source_columns from sqlbuild.compiler.planner.models import ( BackfillResult, CloneSourcePlanEntry, diff --git a/src/sqlbuild/compiler/planner/_helpers/warehouse/source_columns.py b/src/sqlbuild/compiler/planner/_helpers/warehouse/source_columns.py new file mode 100644 index 0000000000..67683fc637 --- /dev/null +++ b/src/sqlbuild/compiler/planner/_helpers/warehouse/source_columns.py @@ -0,0 +1,133 @@ +"""Warehouse source relation listing and column gathering for planning.""" + +from __future__ import annotations + +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapter.contract.models import ColumnInfo, RelationInfo, RelationLookup +from sqlbuild.adapter.relations.main.get_columns_for_inspection import get_columns_for_inspection +from sqlbuild.adapter.relations.main.list_relations_for_inspection import ( + list_relations_for_inspection, +) +from sqlbuild.compiler.compile.models import CompiledProject +from sqlbuild.compiler.planner.constants import METADATA_NAME_FILTER_LIMIT +from sqlbuild.spec.contracts.models import SourceEntry + + +def gather_source_columns( + *, + project: CompiledProject, + adapter: BaseAdapter, + connection: Any, + source_entries: tuple[SourceEntry, ...] | None = None, +) -> dict[str, tuple[ColumnInfo, ...]]: + """Gather warehouse columns for all declared sources.""" + + return gather_source_inspection( + project=project, + adapter=adapter, + connection=connection, + source_entries=source_entries, + )[0] + + +def gather_source_inspection( + *, + project: CompiledProject, + adapter: BaseAdapter, + connection: Any, + source_entries: tuple[SourceEntry, ...] | None, +) -> tuple[dict[str, tuple[ColumnInfo, ...]], frozenset[str]]: + """Gather source columns and the names of sources the relation listing found.""" + + result: dict[str, tuple[ColumnInfo, ...]] = {} + listed_names: set[str] = set() + source_schemas: dict[str, set[str]] = {} + entries: tuple[SourceEntry, ...] = ( + source_entries + if source_entries is not None + else tuple(source.source_entry for source in project.sources) + ) + entry: SourceEntry + for entry in entries: + if entry.expression is not None: + if entry.type_enforcement: + column_names: tuple[str, ...] = adapter.query_column_names( + connection=connection, sql=entry.expression + ) + result[entry.name] = tuple(ColumnInfo(name=name, type="") for name in column_names) + continue + schema: str | None = entry.schema + if schema is None: + continue + db: str | None = entry.database + db_key: str = db or "" + source_schemas.setdefault(db_key, set()).add(schema) + + db_key_iter: str + schemas: set[str] + for db_key_iter, schemas in source_schemas.items(): + database: str | None = db_key_iter or None + scoped: dict[str, tuple[str | None, str | None, str]] = { + entry_iter.name: RelationLookup.key( + database=database, + schema=entry_iter.schema, + name=entry_iter.table if entry_iter.table is not None else entry_iter.name, + ) + for entry_iter in entries + if entry_iter.expression is None + and (entry_iter.database or None) == database + and entry_iter.schema in schemas + } + relations: tuple[RelationInfo, ...] = list_relations_for_inspection( + adapter=adapter, + connection=connection, + database=database, + schemas=tuple(sorted(schemas)), + names=_build_source_table_name_filter( + project=project, database=database, schemas=schemas, source_entries=entries + ), + ) + listed_identities: frozenset[tuple[str | None, str | None, str]] = frozenset( + relation.identity for relation in relations + ) + listed_names.update(name for name, key in scoped.items() if key in listed_identities) + wanted: frozenset[tuple[str | None, str | None, str]] = frozenset(scoped.values()) + all_columns: dict[tuple[str | None, str | None, str], tuple[ColumnInfo, ...]] = ( + get_columns_for_inspection( + adapter=adapter, + connection=connection, + relations=tuple(relation for relation in relations if relation.identity in wanted), + ) + ) + result.update( + {name: all_columns[key] for name, key in scoped.items() if key in all_columns} + ) + + return result, frozenset(listed_names) + + +def _build_source_table_name_filter( + *, + project: CompiledProject, + database: str | None, + schemas: set[str], + source_entries: tuple[SourceEntry, ...] | None = None, +) -> tuple[str, ...] | None: + names: set[str] = set() + entries: tuple[SourceEntry, ...] = ( + source_entries + if source_entries is not None + else tuple(source.source_entry for source in project.sources) + ) + entry: SourceEntry + for entry in entries: + if entry.expression is not None or entry.schema not in schemas: + continue + if (entry.database or None) != database: + continue + names.add(entry.table if entry.table is not None else entry.name) + if not names or len(names) > METADATA_NAME_FILTER_LIMIT: + return None + return tuple(sorted(names)) diff --git a/src/sqlbuild/compiler/planner/_helpers/warehouse/source_freshness_warnings.py b/src/sqlbuild/compiler/planner/_helpers/warehouse/source_freshness_warnings.py new file mode 100644 index 0000000000..0d3636203d --- /dev/null +++ b/src/sqlbuild/compiler/planner/_helpers/warehouse/source_freshness_warnings.py @@ -0,0 +1,53 @@ +"""Plan warnings that explain why source freshness could not be observed.""" + +from __future__ import annotations + +from sqlbuild.compiler.planner.constants import SOURCE_FRESHNESS_UNKNOWN_WARNING_TITLE +from sqlbuild.compiler.planner.models import PlanWarning +from sqlbuild.compiler.planner.types import WarningSeverity +from sqlbuild.compiler.source_freshness.models import ( + DirectSourceFreshnessPlanningResult, + SourceFreshnessUnknown, +) +from sqlbuild.compiler.source_freshness.types import SourceFreshnessUnknownReason + +_NAME_LIMIT: int = 10 +_REASON_EXPLANATIONS: dict[SourceFreshnessUnknownReason, str] = { + SourceFreshnessUnknownReason.UNAVAILABLE: ( + "adapter freshness metadata is unavailable, for example for a view, an external table, " + "or a table without a last-modified time" + ), + SourceFreshnessUnknownReason.ERROR: "the freshness observation failed", +} + + +def build_source_freshness_unknown_warnings( + *, source_freshness: DirectSourceFreshnessPlanningResult +) -> tuple[PlanWarning, ...]: + """Return one warning per reason that left configured source freshness unknown.""" + + names_by_reason: dict[SourceFreshnessUnknownReason, list[str]] = {} + unknown: SourceFreshnessUnknown + for unknown in source_freshness.unknown_sources.values(): + if unknown.reason in _REASON_EXPLANATIONS: + names_by_reason.setdefault(unknown.reason, []).append(unknown.source_name) + return tuple( + PlanWarning( + model_name=None, + severity=WarningSeverity.WARNING, + message=( + f"{SOURCE_FRESHNESS_UNKNOWN_WARNING_TITLE} ({reason.value}) for " + f"{_format_names(names)}: {_REASON_EXPLANATIONS[reason]}; " + "run sqb freshness for each source's details" + ), + ) + for reason, names in names_by_reason.items() + ) + + +def _format_names(names: list[str]) -> str: + ordered: list[str] = sorted(names) + shown: str = ", ".join(ordered[:_NAME_LIMIT]) + if len(ordered) > _NAME_LIMIT: + return f"{shown} and {len(ordered) - _NAME_LIMIT} more" + return shown diff --git a/src/sqlbuild/compiler/planner/_helpers/warehouse/source_tables.py b/src/sqlbuild/compiler/planner/_helpers/warehouse/source_tables.py new file mode 100644 index 0000000000..f9030a58ec --- /dev/null +++ b/src/sqlbuild/compiler/planner/_helpers/warehouse/source_tables.py @@ -0,0 +1,193 @@ +"""Selected-scope validation that external source tables exist in the warehouse.""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from functools import partial +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapter.contract.models import RelationReadProbe +from sqlbuild.adapter.contract.types import RelationReadStatus +from sqlbuild.adapter.relations.main.run_bounded_inspections import run_bounded_inspections +from sqlbuild.compiler.compile.models import CompiledObjectKey, CompiledProject, CompiledSource +from sqlbuild.compiler.compile.types import CompiledResourceType +from sqlbuild.compiler.planner._helpers.resolve.sources import render_source_read_relation +from sqlbuild.compiler.planner.exceptions import PlannerInputError +from sqlbuild.compiler.planner.models import PlannerRelationsContext, PlannerScope +from sqlbuild.spec.contracts.models import SourceEntry + +_MESSAGE_ITEM_LIMIT: int = 10 +_WILDCARD_TABLE_CHARACTER: str = "*" + + +@dataclass(frozen=True) +class _MissingSourceTable: + source: CompiledSource + entry: SourceEntry + read_by: tuple[str, ...] + probe: RelationReadProbe + + +def check_selected_source_tables_exist( + *, + project: CompiledProject, + adapter: BaseAdapter, + connection: Any, + scope: PlannerScope, + relations: PlannerRelationsContext, +) -> None: + """Raise when a selected resource reads an external source table that does not exist.""" + + readers_by_source: dict[str, list[str]] = _selected_source_readers(scope=scope) + candidates: list[tuple[CompiledSource, SourceEntry, tuple[str, ...]]] = [] + source: CompiledSource + for source in project.sources: + readers: list[str] | None = readers_by_source.get(source.name) + if not readers: + continue + entry: SourceEntry = relations.source_read_map.get(source.name, source.source_entry) + if not _is_external_table_source(source.source_entry) or not _is_external_table_source( + entry + ): + continue + candidates.append((source, entry, tuple(sorted(readers)))) + if not candidates: + return + unlisted: tuple[tuple[CompiledSource, SourceEntry, tuple[str, ...]], ...] = tuple( + candidate + for candidate in candidates + if candidate[0].name not in relations.listed_source_names + ) + probes: list[RelationReadProbe] = run_bounded_inspections( + tasks=tuple( + partial( + adapter.probe_relation_read, + connection=connection, + relation=render_source_read_relation(adapter=adapter, source_entry=entry), + ) + for _, entry, _ in unlisted + ), + concurrency=max(1, int(adapter.metadata_inspection_concurrency)), + ) + missing: tuple[_MissingSourceTable, ...] = tuple( + _MissingSourceTable(source=source, entry=entry, read_by=read_by, probe=probe) + for (source, entry, read_by), probe in zip(unlisted, probes, strict=True) + if probe.status != RelationReadStatus.READABLE + ) + if missing: + raise PlannerInputError( + _missing_source_tables_message(missing=missing, adapter=adapter), + code="S405", + help=_missing_source_tables_help(missing=missing), + ) + + +def _selected_source_readers(*, scope: PlannerScope) -> dict[str, list[str]]: + readers_by_source: dict[str, list[str]] = {} + selected_key: CompiledObjectKey + for selected_key in scope.selected_keys: + if selected_key.resource_type == CompiledResourceType.SOURCE: + continue + dep_key: CompiledObjectKey + for dep_key in scope.upstream_deps.get(selected_key, ()): + if dep_key.resource_type != CompiledResourceType.SOURCE: + continue + readers_by_source.setdefault(dep_key.name, []).append(selected_key.name) + return readers_by_source + + +def _is_external_table_source(entry: SourceEntry) -> bool: + return ( + not entry.managed + and entry.loader is None + and entry.integration_loader is None + and entry.expression is None + and entry.schema is not None + and _WILDCARD_TABLE_CHARACTER not in _table_name(entry) + ) + + +def _table_name(entry: SourceEntry) -> str: + return entry.table if entry.table is not None else entry.name + + +def _missing_source_tables_message( + *, missing: tuple[_MissingSourceTable, ...], adapter: BaseAdapter +) -> str: + noun: str = "source table" if len(missing) == 1 else "source tables" + verb: str = "does" if len(missing) == 1 else "do" + readability: str = ( + " or is not readable by the current role" + if len(missing) == 1 + else " or are not readable by the current role" + ) + lines: list[str] = [ + f"cannot build selected scope: {len(missing)} {noun} read by selected resources " + f"{verb} not exist in the warehouse" + f"{readability if _any_unreadable(missing) else ''}:" + ] + item: _MissingSourceTable + for item in missing[:_MESSAGE_ITEM_LIMIT]: + relation: str = render_source_read_relation(adapter=adapter, source_entry=item.entry) + readers: str = ", ".join(item.read_by) + lines.append( + f" - source '{item.source.name}' ({relation}), read by {readers}" + f"{_unreadable_suffix(item.probe)}" + ) + if len(missing) > _MESSAGE_ITEM_LIMIT: + lines.append(f" - ... and {len(missing) - _MESSAGE_ITEM_LIMIT} more") + return "\n".join(lines) + + +def _unreadable_suffix(probe: RelationReadProbe) -> str: + if probe.status != RelationReadStatus.MISSING_OR_UNREADABLE: + return "" + role: str = f"role {probe.role}" if probe.role is not None else "the current role" + return f"; the warehouse reports it does not exist or is not readable by {role}" + + +def _any_unreadable(missing: tuple[_MissingSourceTable, ...]) -> bool: + return any(item.probe.status == RelationReadStatus.MISSING_OR_UNREADABLE for item in missing) + + +def _missing_source_tables_help(*, missing: tuple[_MissingSourceTable, ...]) -> str: + locations: str = ", ".join( + f"'{item.source.name}' at {_declaration_location(item.source)}" + for item in missing[:_MESSAGE_ITEM_LIMIT] + ) + roles: tuple[str, ...] = tuple( + sorted({item.probe.role for item in missing if item.probe.role is not None}) + ) + role_text: str = f" ({', '.join(roles)})" if roles else "" + grant: str = ( + f"grant SELECT on it to the role in use{role_text}, " if _any_unreadable(missing) else "" + ) + return ( + f"create or load the table, {grant}correct its database, schema, or table in the " + f"source declaration, or stop reading it from the selected models; declared {locations}" + ) + + +def _declaration_location(source: CompiledSource) -> str: + path: str = source.source_file.relative_path.as_posix() + line: int | None = _declaration_line(contents=source.source_file.contents, name=source.name) + return path if line is None else f"{path}:{line}" + + +def _declaration_line(*, contents: str, name: str) -> int | None: + pattern: re.Pattern[str] = re.compile( + rf"^(?P\s*)-\s*name:\s*(?P['\"]?){re.escape(name)}(?P=quote)\s*(#.*)?$" + ) + best: tuple[int, int] | None = None + index: int + line: str + for index, line in enumerate(contents.splitlines(), start=1): + match: re.Match[str] | None = pattern.match(line) + if match is None: + continue + indent: int = len(match.group("indent")) + if best is None or indent < best[0]: + best = (indent, index) + return None if best is None else best[1] diff --git a/src/sqlbuild/compiler/planner/constants.py b/src/sqlbuild/compiler/planner/constants.py index 3434990806..08f9371d91 100644 --- a/src/sqlbuild/compiler/planner/constants.py +++ b/src/sqlbuild/compiler/planner/constants.py @@ -111,3 +111,4 @@ "elsewhere or is no longer needed, remove migrate_from" ), } +SOURCE_FRESHNESS_UNKNOWN_WARNING_TITLE: str = "source freshness unknown" diff --git a/src/sqlbuild/compiler/planner/main/execution/execution.py b/src/sqlbuild/compiler/planner/main/execution/execution.py index 9c16a98509..1606725a53 100644 --- a/src/sqlbuild/compiler/planner/main/execution/execution.py +++ b/src/sqlbuild/compiler/planner/main/execution/execution.py @@ -107,9 +107,9 @@ def build_execution_plan( include_stale_warning_identities=policies.selection_diagnostics, ) check_selected_scope_buildability( - project=project, + runtime=runtime, scopes=scopes, - snapshot=warehouse.snapshot, + warehouse=warehouse, deferral=deferral, ) changes: PlannerChangeResults diff --git a/src/sqlbuild/compiler/planner/models.py b/src/sqlbuild/compiler/planner/models.py index 9acbb34cc1..6151b42ce1 100644 --- a/src/sqlbuild/compiler/planner/models.py +++ b/src/sqlbuild/compiler/planner/models.py @@ -749,6 +749,7 @@ class PlannerRelationsContext: python_source_read_map: dict[str, SourceEntry] source_warehouse_columns: dict[str, tuple[ColumnInfo, ...]] star_exclude_keyword: str + listed_source_names: frozenset[str] = frozenset() @dataclass(frozen=True) diff --git a/src/sqlbuild/compiler/source_freshness/_helpers/observation.py b/src/sqlbuild/compiler/source_freshness/_helpers/observation.py index 64674e5b64..a1dba0b765 100644 --- a/src/sqlbuild/compiler/source_freshness/_helpers/observation.py +++ b/src/sqlbuild/compiler/source_freshness/_helpers/observation.py @@ -11,10 +11,16 @@ TableFreshnessMetadata, TableFreshnessRequest, ) +from sqlbuild.adapter.contract.types import TableFreshnessStatus from sqlbuild.compiler.references.main.render_source_relation import render_source_relation from sqlbuild.compiler.source_freshness._helpers.datetime import normalize_presumed_utc_datetime from sqlbuild.compiler.source_freshness.exceptions import SourceFreshnessObservationError -from sqlbuild.compiler.source_freshness.models import SourceFreshnessObservation +from sqlbuild.compiler.source_freshness.models import ( + AdapterSourceFreshnessBatch, + SourceFreshnessObservation, + SourceFreshnessUnknown, +) +from sqlbuild.compiler.source_freshness.types import SourceFreshnessUnknownReason from sqlbuild.runtime.observability.classes.operation_lifecycle import OperationLifecycle from sqlbuild.runtime.observability.models import OperationAttributes from sqlbuild.spec.contracts.models import SourceEntry, SourceFreshnessConfig @@ -141,10 +147,11 @@ def _observe_adapter_freshness( schema=source.schema, name=source.table, ) - if metadata.data_version is None: - raise SourceFreshnessObservationError( - f"source '{source.name}' freshness data_version cannot be null" - ) + source_unknown: SourceFreshnessUnknown | None = _adapter_metadata_unknown( + source_name=source.name, metadata=metadata + ) + if source_unknown is not None: + raise SourceFreshnessObservationError(source_unknown.message) return SourceFreshnessObservation( source_name=source.name, strategy=SourceFreshnessStrategy.ADAPTER, @@ -164,23 +171,27 @@ def observe_adapter_sources_freshness( connection: Any, sources: tuple[SourceEntry, ...], observed_at: datetime, -) -> dict[str, SourceFreshnessObservation]: - """Observe adapter metadata freshness for physical table sources in one batch.""" +) -> AdapterSourceFreshnessBatch: + """Observe adapter metadata freshness per source, batching the warehouse lookups.""" if not sources: - return {} + return AdapterSourceFreshnessBatch() with OperationLifecycle( operation_kind="freshness", operation_name="source_freshness_metadata_observation", metadata={"item_count": len(sources)}, attributes=OperationAttributes(phase="observe", strategy="adapter", target_kind="source"), - ): - return _observe_adapter_sources_freshness( + ) as lifecycle: + batch: AdapterSourceFreshnessBatch = _observe_adapter_sources_freshness( adapter=adapter, connection=connection, sources=sources, observed_at=observed_at, ) + lifecycle.completed( + metadata={"item_count": len(sources), "unknown_count": len(batch.unknown)} + ) + return batch def _observe_adapter_sources_freshness( @@ -189,44 +200,53 @@ def _observe_adapter_sources_freshness( connection: Any, sources: tuple[SourceEntry, ...], observed_at: datetime, -) -> dict[str, SourceFreshnessObservation]: - if not adapter.supports_table_freshness_metadata(): - raise SourceFreshnessObservationError( - f"adapter '{adapter.adapter_name}' does not support table freshness metadata" - ) +) -> AdapterSourceFreshnessBatch: + unknown: dict[str, SourceFreshnessUnknown] = {} requests_by_source_name: dict[str, TableFreshnessRequest] = {} source: SourceEntry for source in sources: - if source.expression is not None or source.table is None: - raise SourceFreshnessObservationError( - f"source '{source.name}' adapter freshness requires a physical table source" + unsupported_message: str | None = _adapter_freshness_unsupported_message( + adapter=adapter, source=source + ) + if unsupported_message is not None: + unknown[source.name] = SourceFreshnessUnknown( + source_name=source.name, + reason=SourceFreshnessUnknownReason.UNAVAILABLE, + message=unsupported_message, ) + continue requests_by_source_name[source.name] = TableFreshnessRequest( database=source.database, schema=source.schema, - name=source.table, + name=str(source.table), ) + if not requests_by_source_name: + return AdapterSourceFreshnessBatch(unknown=unknown) metadata_by_request: dict[TableFreshnessRequest, TableFreshnessMetadata] = ( adapter.get_tables_freshness_metadata( connection=connection, - requests=tuple(requests_by_source_name.values()), + requests=tuple(dict.fromkeys(requests_by_source_name.values())), ) ) observations: dict[str, SourceFreshnessObservation] = {} - for source in sources: - request: TableFreshnessRequest = requests_by_source_name[source.name] - metadata: TableFreshnessMetadata | None = metadata_by_request.get(request) - if metadata is None: - raise SourceFreshnessObservationError( - f"source '{source.name}' freshness metadata was not returned" - ) - if metadata.data_version is None: - raise SourceFreshnessObservationError( - f"source '{source.name}' freshness data_version cannot be null" - ) - observations[source.name] = SourceFreshnessObservation( - source_name=source.name, + source_name: str + request: TableFreshnessRequest + for source_name, request in requests_by_source_name.items(): + metadata: TableFreshnessMetadata = metadata_by_request.get( + request, + TableFreshnessMetadata.unavailable( + message=f"source '{source_name}' freshness metadata was not returned" + ), + ) + source_unknown: SourceFreshnessUnknown | None = _adapter_metadata_unknown( + source_name=source_name, metadata=metadata + ) + if source_unknown is not None: + unknown[source_name] = source_unknown + continue + observations[source_name] = SourceFreshnessObservation( + source_name=source_name, strategy=SourceFreshnessStrategy.ADAPTER, data_version=_normalize_adapter_datetime(metadata.data_version), value_kind=SourceFreshnessValueKind(metadata.value_kind), @@ -236,7 +256,35 @@ def _observe_adapter_sources_freshness( else observed_at ), ) - return observations + return AdapterSourceFreshnessBatch(observations=observations, unknown=unknown) + + +def _adapter_freshness_unsupported_message( + *, adapter: StrictAdapter, source: SourceEntry +) -> str | None: + if not adapter.supports_table_freshness_metadata(): + return f"adapter '{adapter.adapter_name}' does not support table freshness metadata" + if source.expression is not None or source.table is None: + return f"source '{source.name}' adapter freshness requires a physical table source" + return None + + +def _adapter_metadata_unknown( + *, source_name: str, metadata: TableFreshnessMetadata +) -> SourceFreshnessUnknown | None: + if metadata.status == TableFreshnessStatus.UNAVAILABLE: + return SourceFreshnessUnknown( + source_name=source_name, + reason=SourceFreshnessUnknownReason.UNAVAILABLE, + message=metadata.message or f"source '{source_name}' freshness metadata unavailable", + ) + if metadata.data_version is None: + return SourceFreshnessUnknown( + source_name=source_name, + reason=SourceFreshnessUnknownReason.UNAVAILABLE, + message=f"source '{source_name}' freshness data_version is null", + ) + return None def _query_single_data_version( diff --git a/src/sqlbuild/compiler/source_freshness/_helpers/planning.py b/src/sqlbuild/compiler/source_freshness/_helpers/planning.py index 8d57baa2c6..eb530d0af4 100644 --- a/src/sqlbuild/compiler/source_freshness/_helpers/planning.py +++ b/src/sqlbuild/compiler/source_freshness/_helpers/planning.py @@ -33,13 +33,18 @@ source_freshness_records_equivalent, ) from sqlbuild.compiler.source_freshness.models import ( + AdapterSourceFreshnessBatch, DirectSourceFreshnessPlanningResult, SourceFreshnessIdentity, SourceFreshnessObservation, SourceFreshnessRecord, SourceFreshnessSet, + SourceFreshnessUnknown, +) +from sqlbuild.compiler.source_freshness.types import ( + SourceFreshnessAgeStatus, + SourceFreshnessUnknownReason, ) -from sqlbuild.compiler.source_freshness.types import SourceFreshnessAgeStatus from sqlbuild.spec.contracts.models import SourceEntry, SourceFreshnessConfig from sqlbuild.spec.contracts.types import SourceFreshnessStrategy @@ -73,7 +78,7 @@ def build_direct_source_freshness_planning_result( ) ) observed_records: list[SourceFreshnessRecord] = [] - unknown_source_names: list[str] = [] + unconfigured_sources: dict[str, SourceFreshnessUnknown] = {} changed_identities: set[SourceFreshnessIdentity] = set() unchanged_identities: set[SourceFreshnessIdentity] = set() age_statuses: dict[SourceFreshnessIdentity, SourceFreshnessAgeStatus] = {} @@ -89,7 +94,11 @@ def build_direct_source_freshness_planning_result( source=source, ) if observation_source is None: - unknown_source_names.append(source.name) + unconfigured_sources[source.name] = SourceFreshnessUnknown( + source_name=source.name, + reason=SourceFreshnessUnknownReason.NO_FRESHNESS_CONFIG, + message="no freshness config and adapter metadata unavailable", + ) continue observation_sources_by_name[source.name] = observation_source if observation_source.freshness is not None and ( @@ -97,23 +106,24 @@ def build_direct_source_freshness_planning_result( ): adapter_observation_sources.append(observation_source) - adapter_observations: dict[str, SourceFreshnessObservation] = {} - if adapter_observation_sources: - try: - adapter_observations = observe_adapter_sources_freshness( - adapter=adapter, - connection=connection, - sources=tuple(adapter_observation_sources), - observed_at=observed_at, - ) - except (AdapterUserError, SourceFreshnessObservationError): - unknown_source_names.extend(source.name for source in adapter_observation_sources) + adapter_batch: AdapterSourceFreshnessBatch = _observe_adapter_batch( + adapter=adapter, + connection=connection, + sources=tuple(adapter_observation_sources), + observed_at=observed_at, + ) + unknown_sources: dict[str, SourceFreshnessUnknown] = { + **unconfigured_sources, + **adapter_batch.unknown, + } for source_name, observation_source in observation_sources_by_name.items(): if observation_source.freshness is not None and ( observation_source.freshness.strategy == SourceFreshnessStrategy.ADAPTER ): - observation: SourceFreshnessObservation | None = adapter_observations.get(source_name) + observation: SourceFreshnessObservation | None = adapter_batch.observations.get( + source_name + ) if observation is None: continue else: @@ -124,13 +134,17 @@ def build_direct_source_freshness_planning_result( source=observation_source, observed_at=observed_at, ) - except AdapterUserError: - unknown_source_names.append(source_name) + except AdapterUserError as exc: + unknown_sources[source_name] = _error_unknown( + source_name=source_name, message=exc.message + ) continue except SourceFreshnessObservationError as exc: if _source_freshness_error_is_configuration_error(error=exc): raise - unknown_source_names.append(source_name) + unknown_sources[source_name] = _error_unknown( + source_name=source_name, message=str(exc) + ) continue observed_record: SourceFreshnessRecord = source_freshness_record_from_observation( observation=observation, @@ -171,11 +185,44 @@ def build_direct_source_freshness_planning_result( ), changed_identities=frozenset(changed_identities), unchanged_identities=frozenset(unchanged_identities), - unknown_source_names=tuple(sorted(unknown_source_names)), + unknown_source_names=tuple(sorted(unknown_sources)), + unknown_sources=dict(sorted(unknown_sources.items())), age_statuses=age_statuses, ) +def _observe_adapter_batch( + *, + adapter: StrictAdapter, + connection: Any, + sources: tuple[SourceEntry, ...], + observed_at: datetime, +) -> AdapterSourceFreshnessBatch: + try: + return observe_adapter_sources_freshness( + adapter=adapter, + connection=connection, + sources=sources, + observed_at=observed_at, + ) + except (AdapterUserError, SourceFreshnessObservationError) as exc: + message: str = exc.message if isinstance(exc, AdapterUserError) else str(exc) + return AdapterSourceFreshnessBatch( + unknown={ + source.name: _error_unknown(source_name=source.name, message=message) + for source in sources + } + ) + + +def _error_unknown(*, source_name: str, message: str) -> SourceFreshnessUnknown: + return SourceFreshnessUnknown( + source_name=source_name, + reason=SourceFreshnessUnknownReason.ERROR, + message=message, + ) + + def _source_freshness_error_is_configuration_error( *, error: SourceFreshnessObservationError ) -> bool: diff --git a/src/sqlbuild/compiler/source_freshness/main/adapter_observation.py b/src/sqlbuild/compiler/source_freshness/main/adapter_observation.py index ea4304932a..91263cf282 100644 --- a/src/sqlbuild/compiler/source_freshness/main/adapter_observation.py +++ b/src/sqlbuild/compiler/source_freshness/main/adapter_observation.py @@ -9,7 +9,7 @@ from sqlbuild.compiler.source_freshness._helpers.observation import ( observe_adapter_sources_freshness as _observe_adapter_sources_freshness, ) -from sqlbuild.compiler.source_freshness.models import SourceFreshnessObservation +from sqlbuild.compiler.source_freshness.models import AdapterSourceFreshnessBatch from sqlbuild.spec.contracts.models import SourceEntry @@ -19,8 +19,8 @@ def observe_adapter_sources_freshness( connection: Any, sources: tuple[SourceEntry, ...], observed_at: datetime, -) -> dict[str, SourceFreshnessObservation]: - """Observe adapter metadata freshness for physical table sources in one batch.""" +) -> AdapterSourceFreshnessBatch: + """Observe adapter metadata freshness per source, batching the warehouse lookups.""" return _observe_adapter_sources_freshness( adapter=adapter, diff --git a/src/sqlbuild/compiler/source_freshness/models.py b/src/sqlbuild/compiler/source_freshness/models.py index deece9f680..507b047ad1 100644 --- a/src/sqlbuild/compiler/source_freshness/models.py +++ b/src/sqlbuild/compiler/source_freshness/models.py @@ -8,7 +8,10 @@ from sqlbuild.adapter.contract.types import FrameworkType from sqlbuild.compiler.source_freshness._helpers.datetime import require_aware_utc_datetime -from sqlbuild.compiler.source_freshness.types import SourceFreshnessAgeStatus +from sqlbuild.compiler.source_freshness.types import ( + SourceFreshnessAgeStatus, + SourceFreshnessUnknownReason, +) from sqlbuild.spec.contracts.types import SourceFreshnessStrategy, SourceFreshnessValueKind @@ -47,6 +50,23 @@ def __post_init__(self) -> None: ) +@dataclass(frozen=True) +class SourceFreshnessUnknown: + """One source whose freshness could not be observed, with the reason.""" + + source_name: str + reason: SourceFreshnessUnknownReason + message: str + + +@dataclass(frozen=True) +class AdapterSourceFreshnessBatch: + """Per-source outcomes of one batched adapter metadata observation.""" + + observations: dict[str, SourceFreshnessObservation] = field(default_factory=dict) + unknown: dict[str, SourceFreshnessUnknown] = field(default_factory=dict) + + @dataclass(frozen=True) class SourceFreshnessIdentity: """Stable identity for one source freshness stream in direct state.""" @@ -121,6 +141,7 @@ class DirectSourceFreshnessPlanningResult: changed_identities: frozenset[SourceFreshnessIdentity] = frozenset() unchanged_identities: frozenset[SourceFreshnessIdentity] = frozenset() unknown_source_names: tuple[str, ...] = () + unknown_sources: dict[str, SourceFreshnessUnknown] = field(default_factory=dict) age_statuses: dict[SourceFreshnessIdentity, SourceFreshnessAgeStatus] = field( default_factory=dict ) diff --git a/src/sqlbuild/compiler/source_freshness/types.py b/src/sqlbuild/compiler/source_freshness/types.py index d0d3d88416..dbb1ae0448 100644 --- a/src/sqlbuild/compiler/source_freshness/types.py +++ b/src/sqlbuild/compiler/source_freshness/types.py @@ -22,3 +22,11 @@ class SourceFreshnessAgeStatus(StrEnum): WARN = "warn" ERROR = "error" UNKNOWN = "unknown" + + +class SourceFreshnessUnknownReason(StrEnum): + """Why one source's freshness could not be observed.""" + + UNAVAILABLE = "unavailable" + ERROR = "error" + NO_FRESHNESS_CONFIG = "no_freshness_config" diff --git a/src/sqlbuild/runtime/observability/constants.py b/src/sqlbuild/runtime/observability/constants.py index 5d38a218d0..d09e09cbca 100644 --- a/src/sqlbuild/runtime/observability/constants.py +++ b/src/sqlbuild/runtime/observability/constants.py @@ -156,6 +156,7 @@ "added_count", "removed_count", "altered_count", + "unknown_count", } ) OPERATION_PHASES: frozenset[str] = frozenset( diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/plan/_test_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/plan/_test_types.py index 25d2b69e7a..810b51661a 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/plan/_test_types.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/plan/_test_types.py @@ -1,4 +1,4 @@ -from dataclasses import dataclass +from dataclasses import dataclass, field @dataclass(frozen=True) @@ -55,3 +55,11 @@ class VerboseInspectionPlanE2ETestCase: command: tuple[str, ...] expected_inspection_output: bool expected_json_stdout: bool + + +@dataclass(frozen=True) +class MissingSourceTableE2ETestCase: + description: str + expected_output_fragments: tuple[str, ...] = () + expected_recorded_source_names: tuple[str, ...] = () + expected_statuses: dict[str, str] = field(default_factory=dict) diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/plan/helpers.py b/tests/e2e/src/sqlbuild/cli/commands/main/plan/helpers.py index 53f419c3d2..4a7dc4a4a7 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/plan/helpers.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/plan/helpers.py @@ -103,3 +103,66 @@ def prepare_changed_incremental_diamond_project(*, tmp_path: Path) -> Path: assert build.returncode == 0, build.stdout + build.stderr change_incremental_diamond_root(project_dir=project_dir) return project_dir + + +_MISSING_SOURCE_TABLES_SOURCES_YML: str = """sources: + - name: raw_orders + schema: raw + table: orders + freshness: + strategy: column + column: updated_at + type: timestamp + - name: raw_customers + schema: raw + table: customers + freshness: + strategy: column + column: updated_at + type: timestamp + - name: raw_payments + schema: raw + table: payments + freshness: + strategy: column + column: updated_at + type: timestamp +""" + + +def prepare_missing_source_tables_project(*, tmp_path: Path) -> Path: + """Write three source-reading models where only the payments source table is missing.""" + + project_dir: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="missing_source_tables", + repo_files={ + "sqlbuild_project.toml": ( + 'name = "missing_source_tables"\n' + 'adapter = "duckdb"\n\n' + "[connection]\n" + 'database = "warehouse.duckdb"\n' + ), + "sources/raw.yml": _MISSING_SOURCE_TABLES_SOURCES_YML, + "models/orders.sql": ( + 'MODEL (materialized table);\n\nSELECT * FROM __source("raw_orders")\n' + ), + "models/customers.sql": ( + 'MODEL (materialized table);\n\nSELECT * FROM __source("raw_customers")\n' + ), + "models/payments.sql": ( + 'MODEL (materialized table);\n\nSELECT * FROM __source("raw_payments")\n' + ), + }, + ) + db_path: Path = project_dir / "warehouse.duckdb" + statement: str + for statement in ( + "CREATE SCHEMA raw", + "CREATE TABLE raw.orders AS SELECT 1 AS order_id, " + "TIMESTAMP '2026-01-01 00:00:00' AS updated_at", + "CREATE TABLE raw.customers AS SELECT 1 AS customer_id, " + "TIMESTAMP '2026-01-02 00:00:00' AS updated_at", + ): + execute_duckdb(db_path=db_path, sql=statement) + return project_dir diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/plan/test_missing_source_tables.py b/tests/e2e/src/sqlbuild/cli/commands/main/plan/test_missing_source_tables.py new file mode 100644 index 0000000000..21e1406eb4 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/plan/test_missing_source_tables.py @@ -0,0 +1,244 @@ +"""E2E coverage for planning against source tables missing from the warehouse.""" + +from __future__ import annotations + +import json +import subprocess +from pathlib import Path +from typing import Any + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.plan._test_types import ( + MissingSourceTableE2ETestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.main.plan.helpers import ( + prepare_missing_source_tables_project, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import ( + execute_duckdb, + query_duckdb, + run_sqb, +) + + +@pytest.mark.parametrize( + "test_case", + [ + MissingSourceTableE2ETestCase( + description="one missing source among several fails plan and build", + expected_output_fragments=( + "error[S405]", + "1 source table read by selected resources does not exist in the warehouse", + "source 'raw_payments' (raw.payments), read by payments", + "'raw_payments' at sources/raw.yml:16", + ), + ) + ], + ids=lambda case: case.description, +) +def test_given_missing_source_table_when_planning_then_fails_naming_it( + test_case: MissingSourceTableE2ETestCase, + tmp_path: Path, +) -> None: + project_dir: Path = prepare_missing_source_tables_project(tmp_path=tmp_path) + + result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "plan"), project_dir=project_dir + ) + build_result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "build"), project_dir=project_dir + ) + unaffected_result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "plan", "--select", "orders"), project_dir=project_dir + ) + + assert result.returncode == 1, result.stdout + result.stderr + output: str = result.stdout + result.stderr + fragment: str + for fragment in test_case.expected_output_fragments: + assert fragment in output, output + assert build_result.returncode == 1, build_result.stdout + build_result.stderr + assert test_case.expected_output_fragments[0] in build_result.stdout + build_result.stderr + assert unaffected_result.returncode == 0, unaffected_result.stdout + unaffected_result.stderr + assert not query_duckdb( + db_path=project_dir / "warehouse.duckdb", + sql="SELECT table_name FROM information_schema.tables WHERE table_schema = 'main'", + ) + + +@pytest.mark.parametrize( + "test_case", + [ + MissingSourceTableE2ETestCase( + description="unread missing source leaves other freshness recorded and used", + expected_output_fragments=( + "Freshness query FAIL (source freshness unknown)", + "source freshness unknown (error) for raw_payments", + ), + expected_recorded_source_names=("raw_customers", "raw_orders"), + ) + ], + ids=lambda case: case.description, +) +def test_given_missing_source_reference_removed_when_building_then_other_sources_keep_freshness( + test_case: MissingSourceTableE2ETestCase, + tmp_path: Path, +) -> None: + project_dir: Path = prepare_missing_source_tables_project(tmp_path=tmp_path) + (project_dir / "models" / "payments.sql").unlink() + + build_result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "build"), project_dir=project_dir + ) + plan_result: subprocess.CompletedProcess[str] = run_sqb( + command=("plan", "--json"), project_dir=project_dir + ) + text_plan_result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "plan"), project_dir=project_dir + ) + + assert build_result.returncode == 0, build_result.stdout + build_result.stderr + recorded: list[tuple[Any, ...]] = query_duckdb( + db_path=project_dir / "warehouse.duckdb", + sql="SELECT DISTINCT source_name FROM main._sqlbuild_source_freshness ORDER BY 1", + ) + assert tuple(row[0] for row in recorded) == test_case.expected_recorded_source_names + assert plan_result.returncode == 0, plan_result.stdout + plan_result.stderr + payload: dict[str, Any] = json.loads(plan_result.stdout) + freshness: dict[str, Any] = payload["metadata"]["direct_source_freshness"] + assert freshness["unchanged_source_names"] == list(test_case.expected_recorded_source_names) + assert freshness["unknown_source_names"] == ["raw_payments"] + assert [ + (detail["source_name"], detail["reason"]) for detail in freshness["unknown_source_details"] + ] == [("raw_payments", "error")] + assert [warning["message"] for warning in payload["warnings"]] == [ + "source freshness unknown (error) for raw_payments: the freshness observation failed; " + "run sqb freshness for each source's details" + ] + assert text_plan_result.returncode == 0, text_plan_result.stdout + text_plan_result.stderr + text_output: str = text_plan_result.stdout + text_plan_result.stderr + fragment: str + for fragment in test_case.expected_output_fragments: + assert fragment in text_output, text_output + + +@pytest.mark.parametrize( + "test_case", + [ + MissingSourceTableE2ETestCase( + description="creating the missing table lets the plan succeed", + expected_output_fragments=("Plan ready 3 selected",), + ) + ], + ids=lambda case: case.description, +) +def test_given_missing_source_table_when_created_then_plan_succeeds( + test_case: MissingSourceTableE2ETestCase, + tmp_path: Path, +) -> None: + project_dir: Path = prepare_missing_source_tables_project(tmp_path=tmp_path) + execute_duckdb( + db_path=project_dir / "warehouse.duckdb", + sql="CREATE TABLE raw.payments AS SELECT 1 AS payment_id, " + "TIMESTAMP '2026-01-03 00:00:00' AS updated_at", + ) + + result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "plan"), project_dir=project_dir + ) + + assert result.returncode == 0, result.stdout + result.stderr + output: str = result.stdout + result.stderr + assert "error[S405]" not in output + fragment: str + for fragment in test_case.expected_output_fragments: + assert fragment in output, output + + +@pytest.mark.parametrize( + "test_case", + [ + MissingSourceTableE2ETestCase( + description="source declared with different identifier case plans", + expected_output_fragments=("Plan ready 3 selected",), + ) + ], + ids=lambda case: case.description, +) +def test_given_case_variant_source_declaration_when_planning_then_table_is_found( + test_case: MissingSourceTableE2ETestCase, + tmp_path: Path, +) -> None: + project_dir: Path = prepare_missing_source_tables_project(tmp_path=tmp_path) + execute_duckdb( + db_path=project_dir / "warehouse.duckdb", + sql="CREATE TABLE raw.payments AS SELECT 1 AS payment_id, " + "TIMESTAMP '2026-01-03 00:00:00' AS updated_at", + ) + sources_path: Path = project_dir / "sources" / "raw.yml" + sources_path.write_text( + sources_path.read_text(encoding="utf-8").replace( + "schema: raw\n table: orders", "schema: Raw\n table: Orders" + ), + encoding="utf-8", + ) + + result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "plan"), project_dir=project_dir + ) + + assert result.returncode == 0, result.stdout + result.stderr + output: str = result.stdout + result.stderr + assert "error[S405]" not in output + fragment: str + for fragment in test_case.expected_output_fragments: + assert fragment in output, output + + +@pytest.mark.parametrize( + "test_case", + [ + MissingSourceTableE2ETestCase( + description="freshness command reports each source on its own", + expected_output_fragments=( + "raw_customers value 2026-01-02T00:00:00+00:00", + "raw_orders value 2026-01-01T00:00:00+00:00", + "raw_payments", + "OBSERVED=2", + "ERROR=1", + ), + expected_statuses={ + "raw_customers": "observed", + "raw_orders": "observed", + "raw_payments": "error", + }, + ) + ], + ids=lambda case: case.description, +) +def test_given_missing_source_table_when_observing_freshness_then_reports_each_source( + test_case: MissingSourceTableE2ETestCase, + tmp_path: Path, +) -> None: + project_dir: Path = prepare_missing_source_tables_project(tmp_path=tmp_path) + + result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "freshness"), project_dir=project_dir + ) + json_result: subprocess.CompletedProcess[str] = run_sqb( + command=("freshness", "--json"), project_dir=project_dir + ) + + assert result.returncode == 0, result.stdout + result.stderr + fragment: str + for fragment in test_case.expected_output_fragments: + assert fragment in result.stdout, result.stdout + payload: dict[str, Any] = json.loads(json_result.stdout) + assert { + source["name"]: source["status"] for source in payload["sources"] + } == test_case.expected_statuses + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/integration/src/sqlbuild/compiler/pipeline/_test_types.py b/tests/integration/src/sqlbuild/compiler/pipeline/_test_types.py index 264e05aba9..77dce05d9b 100644 --- a/tests/integration/src/sqlbuild/compiler/pipeline/_test_types.py +++ b/tests/integration/src/sqlbuild/compiler/pipeline/_test_types.py @@ -179,6 +179,7 @@ class RunCompilePipelineIntegrationTestCase: expected_seed_count: int = 0 expected_manifest_node_count: int = 0 expected_declaration_usages: tuple[str, ...] = () + warehouse_setup_sql: tuple[str, ...] = () @dataclass(frozen=True) @@ -306,6 +307,7 @@ class SqlAnalysisChainCompileTargetIntegrationTestCase: compiled_test_path: str expected_fragments: tuple[str, ...] unexpected_fragments: tuple[str, ...] + warehouse_setup_sql: tuple[str, ...] = () @dataclass(frozen=True) diff --git a/tests/integration/src/sqlbuild/compiler/pipeline/helpers.py b/tests/integration/src/sqlbuild/compiler/pipeline/helpers.py index 55fc3feca7..8edeea1fde 100644 --- a/tests/integration/src/sqlbuild/compiler/pipeline/helpers.py +++ b/tests/integration/src/sqlbuild/compiler/pipeline/helpers.py @@ -202,3 +202,17 @@ def traced(*, preparation: CompactBatchPreparation) -> object: monkeypatch.setattr(compact, "_run_compact_analysis_batch", traced) return preparations + + +def apply_warehouse_setup_sql(*, project_dir: Path, statements: tuple[str, ...]) -> None: + """Create pre-existing warehouse objects in the project's file-backed DuckDB database.""" + + import duckdb + + connection: duckdb.DuckDBPyConnection = duckdb.connect(str(project_dir / "warehouse.duckdb")) + try: + statement: str + for statement in statements: + connection.execute(statement) + finally: + connection.close() diff --git a/tests/integration/src/sqlbuild/compiler/pipeline/test_main.py b/tests/integration/src/sqlbuild/compiler/pipeline/test_main.py index c24bab7445..2f6bdd4759 100644 --- a/tests/integration/src/sqlbuild/compiler/pipeline/test_main.py +++ b/tests/integration/src/sqlbuild/compiler/pipeline/test_main.py @@ -32,12 +32,16 @@ SqlAnalysisChainCompileTargetIntegrationTestCase, ) from tests.integration.src.sqlbuild.compiler.pipeline.helpers import ( + apply_warehouse_setup_sql, build_manifest_for_pipeline_result, run_compile_pipeline_for_project, validate_manifest_against_dbt_schema, ) _PROJECT_TOML: str = 'name = "demo"\nadapter = "duckdb"\n\n[connection]\ndatabase = ":memory:"\n' +_FILE_PROJECT_TOML: str = ( + 'name = "demo"\nadapter = "duckdb"\n\n[connection]\ndatabase = "warehouse.duckdb"\n' +) @pytest.mark.parametrize( @@ -234,8 +238,9 @@ def test_given_rule_finding_when_running_shared_compile_pipeline_then_planning_i ), RunCompilePipelineIntegrationTestCase( description="model with source reference resolves source to qualified name", + warehouse_setup_sql=("CREATE TABLE main.payments AS SELECT 1 AS payment_id",), project_files={ - "sqlbuild_project.toml": _PROJECT_TOML, + "sqlbuild_project.toml": _FILE_PROJECT_TOML, "sources/raw.yml": ( "sources:\n - name: raw_payments\n schema: main\n table: payments\n" ), @@ -368,8 +373,11 @@ def test_given_project_files_when_running_compile_pipeline_then_produces_valid_o test_case: RunCompilePipelineIntegrationTestCase, tmp_path: Path, write_repo_files: Callable[[Path, dict[str, str]], None], + monkeypatch: pytest.MonkeyPatch, ) -> None: write_repo_files(tmp_path, test_case.project_files) + monkeypatch.chdir(tmp_path) + apply_warehouse_setup_sql(project_dir=tmp_path, statements=test_case.warehouse_setup_sql) result: CompilePipelineResult = run_compile_pipeline_for_project( project_dir=tmp_path, @@ -683,7 +691,7 @@ def test_given_project_with_defer_to_when_compiling_then_resolves_refs_to_deferr 'name = "demo"\n' 'adapter = "duckdb"\n\n' "[connection]\n" - 'database = ":memory:"\n\n' + 'database = "warehouse.duckdb"\n\n' "[settings]\n" "sql_analysis = true\n" ), @@ -706,6 +714,7 @@ def test_given_project_with_defer_to_when_compiling_then_resolves_refs_to_deferr "SELECT 1\n" ), }, + warehouse_setup_sql=("CREATE TABLE main.raw AS SELECT 1 AS id, 100 AS amount",), compiled_test_path=( "target/compiled/tests/_chain_/fact_orders__stg_orders/test_chain.sql" ), @@ -737,8 +746,11 @@ def test_given_sql_analysis_enabled_chain_test_when_writing_compile_target_then_ test_case: SqlAnalysisChainCompileTargetIntegrationTestCase, tmp_path: Path, write_repo_files: Callable[[Path, dict[str, str]], None], + monkeypatch: pytest.MonkeyPatch, ) -> None: write_repo_files(tmp_path, test_case.project_files) + monkeypatch.chdir(tmp_path) + apply_warehouse_setup_sql(project_dir=tmp_path, statements=test_case.warehouse_setup_sql) result: CompilePipelineResult = run_compile_pipeline_for_project( project_dir=tmp_path, diff --git a/tests/integration/src/sqlbuild/compiler/planner/_helpers/test_warehouse_snapshot.py b/tests/integration/src/sqlbuild/compiler/planner/_helpers/test_warehouse_snapshot.py index c701f5e8df..0984716805 100644 --- a/tests/integration/src/sqlbuild/compiler/planner/_helpers/test_warehouse_snapshot.py +++ b/tests/integration/src/sqlbuild/compiler/planner/_helpers/test_warehouse_snapshot.py @@ -17,8 +17,8 @@ from sqlbuild.compiler.compile.types import CompiledResourceType from sqlbuild.compiler.fingerprints.main.write import write_fingerprint from sqlbuild.compiler.fingerprints.models import Fingerprint -from sqlbuild.compiler.planner._helpers.output.plan_entry import gather_source_columns from sqlbuild.compiler.planner._helpers.warehouse.snapshot import gather_warehouse_snapshot +from sqlbuild.compiler.planner._helpers.warehouse.source_columns import gather_source_columns from sqlbuild.compiler.planner.constants import METADATA_NAME_FILTER_LIMIT from sqlbuild.compiler.planner.exceptions import PlannerInputError from sqlbuild.compiler.planner.models import ( diff --git a/tests/integration/src/sqlbuild/compiler/planner/main/_test_types.py b/tests/integration/src/sqlbuild/compiler/planner/main/_test_types.py index 81238979af..295ba0a2c1 100644 --- a/tests/integration/src/sqlbuild/compiler/planner/main/_test_types.py +++ b/tests/integration/src/sqlbuild/compiler/planner/main/_test_types.py @@ -157,3 +157,12 @@ class SnowflakeMissingDatabaseTestCase: description: str warehouse_database: str expected_error_fragment: str + + +@dataclass(frozen=True) +class SourceTableProbeBudgetTestCase: + description: str + source_schema: str + source_table: str + expected_source_listings: int + expected_probes: tuple[str, ...] diff --git a/tests/integration/src/sqlbuild/compiler/planner/main/helpers.py b/tests/integration/src/sqlbuild/compiler/planner/main/helpers.py index 0ff2aae644..5d4b27c89f 100644 --- a/tests/integration/src/sqlbuild/compiler/planner/main/helpers.py +++ b/tests/integration/src/sqlbuild/compiler/planner/main/helpers.py @@ -12,6 +12,7 @@ from types import MappingProxyType from typing import Any, cast +from sqlbuild.adapter.contract.models import RelationInfo, RelationReadProbe from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter from sqlbuild.compiler.compile.models import ( CompiledFunction, @@ -550,3 +551,64 @@ def relations_in_database( """Move every synthetic relation into another database.""" return tuple(replace(relation, database=database) for relation in project.relations) + + +class CountingSourceDuckDbAdapter(DuckDbAdapter): + """Plan against one shared DuckDB connection, counting source listings and probes.""" + + def __init__(self, *, connection: Any) -> None: + super().__init__() + self._shared_connection: Any = connection + self.listed_schema_sets: list[tuple[str, ...]] = [] + self.probed_relations: list[str] = [] + + def connect(self, config: dict[str, Any]) -> Any: + del config + return self._shared_connection + + def close(self, connection: Any) -> None: + del connection + + def list_relations( + self, + *, + connection: Any, + database: str | None, + schemas: tuple[str, ...] | None, + names: tuple[str, ...] | None = None, + ) -> tuple[RelationInfo, ...]: + self.listed_schema_sets.append(tuple(schema.lower() for schema in schemas or ())) + return super().list_relations( + connection=connection, database=database, schemas=schemas, names=names + ) + + def probe_relation_read(self, *, connection: Any, relation: str) -> RelationReadProbe: + self.probed_relations.append(relation) + return super().probe_relation_read(connection=connection, relation=relation) + + +def plan_source_reading_project( + *, project_dir: Path, adapter: DuckDbAdapter, source_schema: str, source_table: str +) -> CompilePipelineResult: + """Write and plan one model reading one declared table source.""" + + project_files: dict[str, str] = { + "sqlbuild_project.toml": 'name = "orders"\nadapter = "duckdb"\n', + "sources/raw.yml": ( + f"sources:\n - name: raw_orders\n schema: {source_schema}\n" + f" table: {source_table}\n" + ), + "models/orders.sql": ( + 'MODEL (materialized table);\n\nSELECT * FROM __source("raw_orders")\n' + ), + } + relative_path: str + contents: str + for relative_path, contents in project_files.items(): + (project_dir / relative_path).parent.mkdir(parents=True, exist_ok=True) + (project_dir / relative_path).write_text(contents, encoding="utf-8") + return run_compile_pipeline( + discovered_inputs=discover_project_inputs(project_dir=project_dir), + adapter=adapter, + options=CompilePipelineOptions(no_sql_validation=True), + ) diff --git a/tests/integration/src/sqlbuild/compiler/planner/main/test_source_table_probes.py b/tests/integration/src/sqlbuild/compiler/planner/main/test_source_table_probes.py new file mode 100644 index 0000000000..3ac70e8357 --- /dev/null +++ b/tests/integration/src/sqlbuild/compiler/planner/main/test_source_table_probes.py @@ -0,0 +1,65 @@ +"""Planning reuses the source listing for S405 and probes only sources the listing missed.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import pytest + +from sqlbuild.compiler.pipeline.models import CompilePipelineResult +from tests.integration.src.sqlbuild.compiler.planner.main._test_types import ( + SourceTableProbeBudgetTestCase, +) +from tests.integration.src.sqlbuild.compiler.planner.main.helpers import ( + CountingSourceDuckDbAdapter, + plan_source_reading_project, +) + + +@pytest.mark.parametrize( + "test_case", + [ + SourceTableProbeBudgetTestCase( + description="listed source table plans without extra listing or probe", + source_schema="raw", + source_table="orders", + expected_source_listings=1, + expected_probes=(), + ), + SourceTableProbeBudgetTestCase( + description="case-variant source the listing misses is confirmed by one probe", + source_schema="Raw", + source_table="Orders", + expected_source_listings=1, + expected_probes=("Raw.Orders",), + ), + ], + ids=lambda case: case.description, +) +def test_given_existing_source_table_when_planning_then_listing_is_reused( + test_case: SourceTableProbeBudgetTestCase, + tmp_path: Path, + connection: Any, +) -> None: + connection.execute("CREATE SCHEMA raw") + connection.execute("CREATE TABLE raw.orders AS SELECT 1 AS order_id") + counting: CountingSourceDuckDbAdapter = CountingSourceDuckDbAdapter(connection=connection) + + result: CompilePipelineResult = plan_source_reading_project( + project_dir=tmp_path, + adapter=counting, + source_schema=test_case.source_schema, + source_table=test_case.source_table, + ) + + assert [entry.name for entry in result.plan_output.model_entries] == ["orders"] + assert ( + sum("raw" in schemas for schemas in counting.listed_schema_sets) + == test_case.expected_source_listings + ) + assert tuple(counting.probed_relations) == test_case.expected_probes + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/integration/src/sqlbuild/executor/build/helpers.py b/tests/integration/src/sqlbuild/executor/build/helpers.py index 8d2fa49301..d1e949528a 100644 --- a/tests/integration/src/sqlbuild/executor/build/helpers.py +++ b/tests/integration/src/sqlbuild/executor/build/helpers.py @@ -207,7 +207,8 @@ def compile_capped_microbatch_intermediary_project( project_files={ "sqlbuild_project.toml": 'name = "capped_dependency"\nadapter = "duckdb"\n', "sources/raw.yml": ( - "sources:\n - name: raw_events\n schema: main\n table: raw_events\n" + "sources:\n - name: raw_events\n expression: " + "SELECT 1 AS id, TIMESTAMP '2026-01-01 00:00:00' AS event_time\n" ), "models/capped_events.sql": capped_dependency_producer_sql( action=MicrobatchLimitAction.CAP_FROM_END @@ -240,7 +241,8 @@ def compile_capped_dependency_project( project_files: dict[str, str] = { "sqlbuild_project.toml": 'name = "capped_dependency"\nadapter = "duckdb"\n', "sources/raw.yml": ( - "sources:\n - name: raw_events\n schema: main\n table: raw_events\n" + "sources:\n - name: raw_events\n expression: " + "SELECT 1 AS id, TIMESTAMP '2026-01-01 00:00:00' AS event_time\n" ), "models/capped_events.sql": capped_dependency_producer_sql(action=action), "models/downstream_events.sql": consumer_sql, @@ -262,6 +264,21 @@ def compile_capped_dependency_project( ) +class SharedConnectionDuckDbAdapter(DuckDbAdapter): + """Plan against the same in-memory warehouse that the build executes against.""" + + def __init__(self, *, connection: Any) -> None: + super().__init__() + self._shared_connection: Any = connection + + def connect(self, config: dict[str, Any]) -> Any: + del config + return self._shared_connection + + def close(self, connection: Any) -> None: + del connection + + def run_build_for_project( *, test_case: BuildExecutionTestCase, @@ -278,7 +295,7 @@ def run_build_for_project( discovered: DiscoveredProjectInputs = discover_project_inputs(project_dir=project_dir) pipeline_result: CompilePipelineResult = run_compile_pipeline( discovered_inputs=discovered, - adapter=adapter, + adapter=SharedConnectionDuckDbAdapter(connection=connection), options=CompilePipelineOptions(no_sql_validation=True), ) diff --git a/tests/integration/src/sqlbuild/executor/build/test_snapshot_execution.py b/tests/integration/src/sqlbuild/executor/build/test_snapshot_execution.py index 48db88a64b..d853d2d81c 100644 --- a/tests/integration/src/sqlbuild/executor/build/test_snapshot_execution.py +++ b/tests/integration/src/sqlbuild/executor/build/test_snapshot_execution.py @@ -29,6 +29,7 @@ SnapshotTimestampFailureTestCase, ) from tests.integration.src.sqlbuild.executor.build.helpers import ( + SharedConnectionDuckDbAdapter, run_build_for_project, verify_model_statuses, ) @@ -3086,6 +3087,10 @@ def test_given_check_snapshot_with_execution_initial_validity_when_building_then 'SELECT customer_id, updated_at FROM __source("raw_customers")' ), }, + setup_sql=( + "CREATE TABLE main.raw_customers AS " + "SELECT 1 AS customer_id, TIMESTAMP '2024-01-01 00:00:00' AS updated_at", + ), expected_status=BuildStatus.SUCCESS, expected_model_statuses=(("customer_snapshot", ExecutionStatus.SUCCESS),), ) @@ -3096,14 +3101,17 @@ def test_given_snapshot_initial_validity_config_when_planning_then_plan_entry_pr test_case: BuildExecutionTestCase, tmp_path: Path, write_repo_files: Callable[[Path, dict[str, str]], None], - adapter: DuckDbAdapter, + connection: Any, ) -> None: write_repo_files(tmp_path, test_case.project_files) + sql: str + for sql in test_case.setup_sql: + connection.execute(sql) discovered: DiscoveredProjectInputs = discover_project_inputs(project_dir=tmp_path) pipeline_result: CompilePipelineResult = run_compile_pipeline( discovered_inputs=discovered, - adapter=adapter, + adapter=SharedConnectionDuckDbAdapter(connection=connection), options=CompilePipelineOptions(no_sql_validation=True), ) plan: PlanOutput = pipeline_result.plan_output diff --git a/tests/integration/src/sqlbuild/executor/testing/helpers.py b/tests/integration/src/sqlbuild/executor/testing/helpers.py index d907c19a6a..b13418d2be 100644 --- a/tests/integration/src/sqlbuild/executor/testing/helpers.py +++ b/tests/integration/src/sqlbuild/executor/testing/helpers.py @@ -18,6 +18,7 @@ from tests.integration.src.sqlbuild.compiler.pipeline.helpers import ( run_compile_pipeline_for_project, ) +from tests.integration.src.sqlbuild.executor.build.helpers import SharedConnectionDuckDbAdapter from tests.integration.src.sqlbuild.executor.testing._test_types import ( SqlTestExecutionTestCase, ) @@ -113,6 +114,35 @@ def render_project_test_step( result: CompilePipelineResult = run_compile_pipeline_for_project( project_dir=project_dir, adapter=adapter ) + return _project_test_step( + result=result, adapter=adapter, test_name=test_name, model_name=model_name + ) + + +def render_project_test_step_over_tables( + *, project_dir: Path, test_name: str, model_name: str, setup_sql: tuple[str, ...] +) -> tuple[str, str, str]: + """Plan against an in-memory warehouse holding the source tables, then render one step.""" + + adapter: DuckDbAdapter = DuckDbAdapter() + connection: Any = adapter.connect({"database": ":memory:"}) + try: + statement: str + for statement in setup_sql: + connection.execute(statement) + result: CompilePipelineResult = run_compile_pipeline_for_project( + project_dir=project_dir, adapter=SharedConnectionDuckDbAdapter(connection=connection) + ) + finally: + adapter.close(connection) + return _project_test_step( + result=result, adapter=adapter, test_name=test_name, model_name=model_name + ) + + +def _project_test_step( + *, result: CompilePipelineResult, adapter: DuckDbAdapter, test_name: str, model_name: str +) -> tuple[str, str, str]: entry: SqlTestPlanEntry = {item.name: item for item in result.plan_output.test_entries}[ test_name ] diff --git a/tests/integration/src/sqlbuild/executor/testing/test_sql_test_build_parity.py b/tests/integration/src/sqlbuild/executor/testing/test_sql_test_build_parity.py index 837bace792..be1fb73399 100644 --- a/tests/integration/src/sqlbuild/executor/testing/test_sql_test_build_parity.py +++ b/tests/integration/src/sqlbuild/executor/testing/test_sql_test_build_parity.py @@ -22,6 +22,7 @@ build_sql_matches_test_body, comparison_rows, render_project_test_step, + render_project_test_step_over_tables, ) _WAFFLE_SHOP: Path = Path(__file__).resolve().parents[6] / "tests/e2e/fixtures/waffle_shop" @@ -98,8 +99,11 @@ def test_given_model_ctes_when_rendering_then_build_sql_slices_appear_verbatim( ) -> None: write_repo_files(tmp_path, build_authored_cte_project_files()) - build_sql, test_body, rendered_sql = render_project_test_step( - project_dir=tmp_path, test_name=test_case.test_name, model_name=test_case.model_name + build_sql, test_body, rendered_sql = render_project_test_step_over_tables( + project_dir=tmp_path, + test_name=test_case.test_name, + model_name=test_case.model_name, + setup_sql=("CREATE TABLE main.raw_orders (id INTEGER, amount INTEGER)",), ) assert build_sql_matches_test_body(build_sql=build_sql, test_body=test_body), test_body diff --git a/tests/unit/src/sqlbuild/adapter/contract/main/_test_types.py b/tests/unit/src/sqlbuild/adapter/contract/main/_test_types.py index 6fe68e38fa..23f19534a1 100644 --- a/tests/unit/src/sqlbuild/adapter/contract/main/_test_types.py +++ b/tests/unit/src/sqlbuild/adapter/contract/main/_test_types.py @@ -4,6 +4,8 @@ from dataclasses import dataclass +from sqlbuild.adapter.contract.types import RelationReadStatus + @dataclass(frozen=True) class SameViewDefinitionTestCase: @@ -11,3 +13,13 @@ class SameViewDefinitionTestCase: definition: str sql: str expected_match: bool + + +@dataclass(frozen=True) +class RelationProbeClassificationTestCase: + description: str + adapter_name: str + errors: tuple[Exception, ...] + expected_status: RelationReadStatus | None + expected_role: str | None = None + role: str | None = None diff --git a/tests/unit/src/sqlbuild/adapter/contract/main/helpers.py b/tests/unit/src/sqlbuild/adapter/contract/main/helpers.py new file mode 100644 index 0000000000..a976f91ddd --- /dev/null +++ b/tests/unit/src/sqlbuild/adapter/contract/main/helpers.py @@ -0,0 +1,94 @@ +"""Driver-shaped error doubles and a recording connection for relation probe tests.""" + +from __future__ import annotations + +from types import SimpleNamespace +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapters.bigquery.classes.bigquery_adapter import BigQueryAdapter +from sqlbuild.adapters.databricks.classes.databricks_adapter import DatabricksAdapter +from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter +from sqlbuild.adapters.postgres.classes.postgres_adapter import PostgresAdapter +from sqlbuild.adapters.snowflake.classes.snowflake_adapter import SnowflakeAdapter +from sqlbuild.adapters.sqlserver.classes.sqlserver_adapter import SqlServerAdapter + +ADAPTERS_BY_NAME: dict[str, type[BaseAdapter]] = { + "bigquery": BigQueryAdapter, + "databricks": DatabricksAdapter, + "duckdb": DuckDbAdapter, + "postgres": PostgresAdapter, + "snowflake": SnowflakeAdapter, + "sqlserver": SqlServerAdapter, +} + + +PROBE_SQL_PREFIX: str = "SELECT 1 FROM " + + +class BinderException(Exception): + """DuckDB binder error shape.""" + + +class ProgrammingError(Exception): + """Snowflake connector error shape carrying an errno.""" + + def __init__(self, message: str, *, errno: int) -> None: + super().__init__(message) + self.errno: int = errno + + +class UndefinedTable(Exception): + """Psycopg error shape carrying a SQLSTATE.""" + + def __init__(self, message: str, *, sqlstate: str) -> None: + super().__init__(message) + self.sqlstate: str = sqlstate + + +class NotFound(Exception): + """Google API not-found error shape.""" + + code: int = 404 + + +class Forbidden(Exception): + """Google API permission error shape.""" + + code: int = 403 + + +class CatalogException(Exception): + """DuckDB catalog error shape.""" + + +class ProbeRows: + """Cursor double returning fixed rows.""" + + def __init__(self, rows: list[tuple[object, ...]]) -> None: + self.rows: list[tuple[object, ...]] = rows + + def fetchall(self) -> list[tuple[object, ...]]: + return self.rows + + +class RecordingProbeConnection: + """Connection double: probes raise configured driver errors; other reads return rows.""" + + def __init__( + self, + *, + errors: tuple[Exception, ...] = (), + role: str | None = None, + ) -> None: + self.errors_by_probe: dict[bool, tuple[Exception, ...]] = {True: errors, False: ()} + self.raw_connection: SimpleNamespace = SimpleNamespace(role=role) + self.statements: list[str] = [] + + def execute(self, sql: str, *args: Any, **kwargs: Any) -> ProbeRows: + del args, kwargs + self.statements.append(sql) + error: Exception + for error in self.errors_by_probe[sql.startswith(PROBE_SQL_PREFIX)]: + raise error + return ProbeRows([]) diff --git a/tests/unit/src/sqlbuild/adapter/contract/main/test_run_relation_read_probe.py b/tests/unit/src/sqlbuild/adapter/contract/main/test_run_relation_read_probe.py new file mode 100644 index 0000000000..30c744c754 --- /dev/null +++ b/tests/unit/src/sqlbuild/adapter/contract/main/test_run_relation_read_probe.py @@ -0,0 +1,173 @@ +from __future__ import annotations + +import pytest + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapter.contract.models import RelationReadProbe +from sqlbuild.adapter.contract.types import RelationReadStatus +from tests.unit.src.sqlbuild.adapter.contract.main._test_types import ( + RelationProbeClassificationTestCase, +) +from tests.unit.src.sqlbuild.adapter.contract.main.helpers import ( + ADAPTERS_BY_NAME, + BinderException, + CatalogException, + Forbidden, + NotFound, + ProgrammingError, + RecordingProbeConnection, + UndefinedTable, +) + + +@pytest.mark.parametrize( + "test_case", + [ + RelationProbeClassificationTestCase( + description="snowflake readable relation exists", + adapter_name="snowflake", + errors=(), + expected_status=RelationReadStatus.READABLE, + ), + RelationProbeClassificationTestCase( + description="snowflake 002003 does not exist or not authorized is missing", + adapter_name="snowflake", + errors=( + ProgrammingError( + "002003 (42S02): Object 'RAW.ORDERS' does not exist or not authorized.", + errno=2003, + ), + ), + expected_status=RelationReadStatus.MISSING_OR_UNREADABLE, + ), + RelationProbeClassificationTestCase( + description="snowflake 2043 object does not exist is missing", + adapter_name="snowflake", + errors=(ProgrammingError("Object does not exist", errno=2043),), + expected_status=RelationReadStatus.MISSING_OR_UNREADABLE, + ), + RelationProbeClassificationTestCase( + description="postgres 42P01 undefined table is missing", + adapter_name="postgres", + errors=(UndefinedTable('relation "raw.orders" does not exist', sqlstate="42P01"),), + expected_status=RelationReadStatus.MISSING, + ), + RelationProbeClassificationTestCase( + description="sql server 208 invalid object name is missing", + adapter_name="sqlserver", + errors=(Exception(208, b"Invalid object name 'raw.orders'."),), + expected_status=RelationReadStatus.MISSING, + ), + RelationProbeClassificationTestCase( + description="databricks table or view not found is missing", + adapter_name="databricks", + errors=(Exception("[TABLE_OR_VIEW_NOT_FOUND] The table or view `raw`.`orders`"),), + expected_status=RelationReadStatus.MISSING, + ), + RelationProbeClassificationTestCase( + description="bigquery not found is missing through the adapter error wrapper", + adapter_name="bigquery", + errors=(NotFound("Not found: Table example-project:raw.orders"),), + expected_status=RelationReadStatus.MISSING, + ), + RelationProbeClassificationTestCase( + description="databricks schema not found is missing", + adapter_name="databricks", + errors=(Exception("[SCHEMA_NOT_FOUND] The schema `raw` cannot be found."),), + expected_status=RelationReadStatus.MISSING, + ), + RelationProbeClassificationTestCase( + description="databricks catalog not found is missing", + adapter_name="databricks", + errors=(Exception("[CATALOG_NOT_FOUND] The catalog `lake` cannot be found."),), + expected_status=RelationReadStatus.MISSING, + ), + RelationProbeClassificationTestCase( + description="duckdb missing attached catalog is missing", + adapter_name="duckdb", + errors=(BinderException('Binder Error: Catalog "lake" does not exist!'),), + expected_status=RelationReadStatus.MISSING, + ), + RelationProbeClassificationTestCase( + description="duckdb catalog does not exist is missing", + adapter_name="duckdb", + errors=(CatalogException("Catalog Error: Table with name orders does not exist!"),), + expected_status=RelationReadStatus.MISSING, + ), + ], + ids=lambda case: case.description, +) +def test_given_probe_outcome_when_checking_relation_then_classifies_not_found( + test_case: RelationProbeClassificationTestCase, +) -> None: + adapter: BaseAdapter = ADAPTERS_BY_NAME[test_case.adapter_name]() + connection: RecordingProbeConnection = RecordingProbeConnection( + errors=test_case.errors, role=test_case.role + ) + + probe: RelationReadProbe = adapter.probe_relation_read( + connection=connection, relation="raw.orders" + ) + + assert probe.status == test_case.expected_status + assert probe.role == test_case.expected_role + assert connection.statements[0] == "SELECT 1 FROM raw.orders WHERE 1=0" + + +@pytest.mark.parametrize( + "test_case", + [ + RelationProbeClassificationTestCase( + description="snowflake session error is re-raised", + adapter_name="snowflake", + errors=(ProgrammingError("Authentication token has expired", errno=390114),), + expected_status=None, + ), + RelationProbeClassificationTestCase( + description="postgres permission error is re-raised", + adapter_name="postgres", + errors=(UndefinedTable("permission denied for table orders", sqlstate="42501"),), + expected_status=None, + ), + RelationProbeClassificationTestCase( + description="sql server permission error is re-raised", + adapter_name="sqlserver", + errors=(Exception(229, b"The SELECT permission was denied"),), + expected_status=None, + ), + RelationProbeClassificationTestCase( + description="databricks permission error is re-raised", + adapter_name="databricks", + errors=(Exception("[INSUFFICIENT_PERMISSIONS] User does not have SELECT"),), + expected_status=None, + ), + RelationProbeClassificationTestCase( + description="bigquery forbidden is re-raised", + adapter_name="bigquery", + errors=(Forbidden("Access Denied: Table example-project:raw.orders"),), + expected_status=None, + ), + RelationProbeClassificationTestCase( + description="duckdb io error is re-raised", + adapter_name="duckdb", + errors=(OSError("IO Error: could not read file"),), + expected_status=None, + ), + ], + ids=lambda case: case.description, +) +def test_given_other_probe_error_when_checking_relation_then_reraises( + test_case: RelationProbeClassificationTestCase, +) -> None: + adapter: BaseAdapter = ADAPTERS_BY_NAME[test_case.adapter_name]() + connection: RecordingProbeConnection = RecordingProbeConnection(errors=test_case.errors) + + with pytest.raises(Exception) as raised: + adapter.probe_relation_read(connection=connection, relation="raw.orders") + + assert test_case.expected_status is None + assert str(test_case.errors[0]) in str(raised.value) + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/unit/src/sqlbuild/adapters/bigquery/test_client.py b/tests/unit/src/sqlbuild/adapters/bigquery/test_client.py index 285762bc7d..4b6e1fac53 100644 --- a/tests/unit/src/sqlbuild/adapters/bigquery/test_client.py +++ b/tests/unit/src/sqlbuild/adapters/bigquery/test_client.py @@ -18,7 +18,11 @@ TableFreshnessMetadata, TableFreshnessRequest, ) -from sqlbuild.adapter.contract.types import CursorKind, FunctionNullabilityRule +from sqlbuild.adapter.contract.types import ( + CursorKind, + FunctionNullabilityRule, + TableFreshnessStatus, +) from sqlbuild.adapters.bigquery.classes.bigquery_adapter import BigQueryAdapter from sqlbuild.adapters.bigquery.classes.bigquery_connection import _BigQueryConnection from sqlbuild.adapters.bigquery.classes.bigquery_cursor import _BigQueryCursor @@ -246,14 +250,14 @@ def test_given_physical_tables_when_getting_freshness_metadata_then_bigquery_use "test_case", [ BigQueryTableFreshnessWildcardTestCase( - description="rejects wildcard table metadata freshness", + description="reports wildcard table metadata freshness unavailable", table_name="events_*", expected_error_fragment="does not support wildcard tables", ) ], ids=lambda case: case.description, ) -def test_given_wildcard_table_when_getting_freshness_metadata_then_bigquery_raises_clear_error( +def test_given_wildcard_table_when_getting_freshness_metadata_then_bigquery_reports_unavailable( test_case: BigQueryTableFreshnessWildcardTestCase, ) -> None: adapter: BigQueryAdapter = BigQueryAdapter() @@ -267,8 +271,12 @@ def test_given_wildcard_table_when_getting_freshness_metadata_then_bigquery_rais location="US", ) - with pytest.raises(AdapterUserError, match=test_case.expected_error_fragment): - adapter.get_tables_freshness_metadata(connection=connection, requests=(request,)) + metadata: TableFreshnessMetadata = adapter.get_tables_freshness_metadata( + connection=connection, requests=(request,) + )[request] + + assert metadata.status == TableFreshnessStatus.UNAVAILABLE + assert test_case.expected_error_fragment in str(metadata.message) @pytest.mark.parametrize( diff --git a/tests/unit/src/sqlbuild/adapters/snowflake/_test_types.py b/tests/unit/src/sqlbuild/adapters/snowflake/_test_types.py index 791904b063..ce18d26ebb 100644 --- a/tests/unit/src/sqlbuild/adapters/snowflake/_test_types.py +++ b/tests/unit/src/sqlbuild/adapters/snowflake/_test_types.py @@ -2,6 +2,7 @@ from datetime import datetime from sqlbuild.adapter.contract.models import FunctionInfo, SchemaDiffResult +from sqlbuild.adapter.contract.types import TableFreshnessStatus from sqlbuild.compiler.lineage.types import InferredNullability from sqlbuild.cost.types import CostStatus @@ -132,6 +133,14 @@ class SnowflakeTableFreshnessBatchTestCase: expected_query_fragments: tuple[str, ...] +@dataclass(frozen=True) +class SnowflakeTableFreshnessOutcomeTestCase: + description: str + rows: tuple[tuple[object, ...], ...] + expected_statuses: dict[str, TableFreshnessStatus] + expected_message_fragments: dict[str, str] + + @dataclass(frozen=True) class SnowflakeTableFreshnessMetadataErrorTestCase: description: str diff --git a/tests/unit/src/sqlbuild/adapters/snowflake/inspection/test_inspection_catalog.py b/tests/unit/src/sqlbuild/adapters/snowflake/inspection/test_inspection_catalog.py index 4ecaa7a34f..10fb349f1c 100644 --- a/tests/unit/src/sqlbuild/adapters/snowflake/inspection/test_inspection_catalog.py +++ b/tests/unit/src/sqlbuild/adapters/snowflake/inspection/test_inspection_catalog.py @@ -14,6 +14,7 @@ TableFreshnessMetadata, TableFreshnessRequest, ) +from sqlbuild.adapter.contract.types import TableFreshnessStatus from sqlbuild.adapter.relations.classes.inspection_catalog import InspectionCatalog from sqlbuild.adapter.relations.constants import INSPECTION_IN_LIST_LIMIT from sqlbuild.adapter.relations.main.open_inspection_catalog import open_inspection_catalog @@ -229,29 +230,33 @@ def test_given_open_catalog_when_reading_source_freshness_then_reads_requested_t "test_case", [ FreshnessErrorTestCase( - description="view freshness fails like the direct read", + description="view freshness is unavailable like the direct read", request=("analytics", "staging", "customers_v"), expected_error_fragment="only supports physical tables", ) ], ids=lambda case: case.description, ) -def test_given_open_catalog_when_freshness_targets_a_view_then_raises_like_direct_read( +def test_given_open_catalog_when_freshness_targets_a_view_then_reports_unavailable_like_direct_read( test_case: FreshnessErrorTestCase, ) -> None: adapter, connection, _ = build_offline_snowflake(relations=build_inspection_catalog_relations()) database, schema, name = test_case.request - requests: tuple[TableFreshnessRequest, ...] = ( - TableFreshnessRequest(database=database, schema=schema, name=name), + request: TableFreshnessRequest = TableFreshnessRequest( + database=database, schema=schema, name=name ) - with pytest.raises(AdapterUserError, match=test_case.expected_error_fragment): - _ = adapter.get_tables_freshness_metadata(connection=connection, requests=requests) + direct: TableFreshnessMetadata = adapter.get_tables_freshness_metadata( + connection=connection, requests=(request,) + )[request] - with ( - open_inspection_catalog(adapter=adapter, connection=connection), - pytest.raises(AdapterUserError, match=test_case.expected_error_fragment), - ): - _ = adapter.get_tables_freshness_metadata(connection=connection, requests=requests) + with open_inspection_catalog(adapter=adapter, connection=connection): + served: TableFreshnessMetadata = adapter.get_tables_freshness_metadata( + connection=connection, requests=(request,) + )[request] + + assert served == direct + assert direct.status == TableFreshnessStatus.UNAVAILABLE + assert test_case.expected_error_fragment in str(direct.message) @pytest.mark.parametrize( diff --git a/tests/unit/src/sqlbuild/adapters/snowflake/test_client.py b/tests/unit/src/sqlbuild/adapters/snowflake/test_client.py index 43e7e24ec8..25fb504975 100644 --- a/tests/unit/src/sqlbuild/adapters/snowflake/test_client.py +++ b/tests/unit/src/sqlbuild/adapters/snowflake/test_client.py @@ -17,7 +17,11 @@ TableFreshnessMetadata, TableFreshnessRequest, ) -from sqlbuild.adapter.contract.types import CursorKind, FunctionNullabilityRule +from sqlbuild.adapter.contract.types import ( + CursorKind, + FunctionNullabilityRule, + TableFreshnessStatus, +) from sqlbuild.adapters.snowflake.classes.snowflake_adapter import SnowflakeAdapter from sqlbuild.compiler.compile.models import ( FunctionArgument, @@ -46,6 +50,7 @@ SnowflakeTableFreshnessBatchTestCase, SnowflakeTableFreshnessMetadataErrorTestCase, SnowflakeTableFreshnessMetadataTestCase, + SnowflakeTableFreshnessOutcomeTestCase, SnowflakeTableTypeDdlTestCase, ) from tests.unit.src.sqlbuild.adapters.snowflake.helpers import ( @@ -959,6 +964,60 @@ def test_given_physical_tables_when_getting_freshness_metadata_then_batches_last assert cursor.closed is True +@pytest.mark.parametrize( + "test_case", + [ + SnowflakeTableFreshnessOutcomeTestCase( + description="isolates absent, view, and null metadata per table", + rows=( + ("ANALYTICS", "RAW", "ORDERS", "BASE TABLE", datetime(2026, 1, 2, 3, 4, 5)), + ("ANALYTICS", "RAW", "CUSTOMERS_V", "VIEW", datetime(2026, 1, 2, 3, 4, 5)), + ("ANALYTICS", "RAW", "PAYMENTS", "BASE TABLE", None), + ), + expected_statuses={ + "ORDERS": TableFreshnessStatus.OBSERVED, + "CUSTOMERS_V": TableFreshnessStatus.UNAVAILABLE, + "PAYMENTS": TableFreshnessStatus.UNAVAILABLE, + "SHIPMENTS": TableFreshnessStatus.UNAVAILABLE, + }, + expected_message_fragments={ + "CUSTOMERS_V": "only supports physical tables", + "PAYMENTS": "missing LAST_ALTERED", + "SHIPMENTS": "not found for SHIPMENTS", + }, + ) + ], + ids=lambda case: case.description, +) +def test_given_batch_with_missing_and_unusable_tables_when_getting_freshness_then_reports_each_table( + test_case: SnowflakeTableFreshnessOutcomeTestCase, +) -> None: + adapter: SnowflakeAdapter = SnowflakeAdapter() + requests: tuple[TableFreshnessRequest, ...] = tuple( + TableFreshnessRequest(database="ANALYTICS", schema="RAW", name=name) + for name in test_case.expected_statuses + ) + cursor: FakeSnowflakeMetadataCursor = FakeSnowflakeMetadataCursor(rows=list(test_case.rows)) + connection: FakeSnowflakeMetadataConnection = FakeSnowflakeMetadataConnection(cursor) + + metadata_by_request: dict[TableFreshnessRequest, TableFreshnessMetadata] = ( + adapter.get_tables_freshness_metadata(connection=connection, requests=requests) + ) + + assert { + request.name: metadata_by_request[request].status for request in requests + } == test_case.expected_statuses + assert metadata_by_request[requests[0]].data_version == test_case.rows[0][4] + messages_by_name: dict[str, str] = { + request.name: str(metadata.message) for request, metadata in metadata_by_request.items() + } + assert { + name: fragment in messages_by_name[name] + for name, fragment in test_case.expected_message_fragments.items() + } == dict.fromkeys(test_case.expected_message_fragments, True) + assert cursor.executed_params == ("ANALYTICS", "RAW", *test_case.expected_statuses) + + @pytest.mark.parametrize( "test_case", [ diff --git a/tests/unit/src/sqlbuild/cli/progress/classes/_test_types.py b/tests/unit/src/sqlbuild/cli/progress/classes/_test_types.py index 9d7acc928a..04d62b57e6 100644 --- a/tests/unit/src/sqlbuild/cli/progress/classes/_test_types.py +++ b/tests/unit/src/sqlbuild/cli/progress/classes/_test_types.py @@ -63,3 +63,11 @@ class LockOrderCase: description: str timeout_seconds: float expected_alive_threads: int + + +@dataclass(frozen=True) +class FreshnessMetadataProjectionCase: + description: str + terminal_event_type: str + terminal_payload: dict[str, object] + expected_terminal_line: str diff --git a/tests/unit/src/sqlbuild/cli/progress/classes/test_native_progress_projector.py b/tests/unit/src/sqlbuild/cli/progress/classes/test_native_progress_projector.py index 009ef2b432..56188ecd99 100644 --- a/tests/unit/src/sqlbuild/cli/progress/classes/test_native_progress_projector.py +++ b/tests/unit/src/sqlbuild/cli/progress/classes/test_native_progress_projector.py @@ -22,6 +22,7 @@ from sqlbuild.runtime.observability.classes.operation_lifecycle import publish_retry_scheduled from tests.unit.src.sqlbuild.cli.progress.classes._test_types import ( CursorCleanupCase, + FreshnessMetadataProjectionCase, LockOrderCase, NativeProjectionCase, RetryProjectionCase, @@ -660,6 +661,56 @@ def test_given_clone_finalization_lifecycle_when_projected_then_phase_is_visible assert stream.getvalue() == test_case.expected_output +@pytest.mark.parametrize( + "test_case", + ( + FreshnessMetadataProjectionCase( + description="completed batch reports source and unknown counts", + terminal_event_type="operation_completed", + terminal_payload={"metadata": {"item_count": 257, "unknown_count": 3}}, + expected_terminal_line="Freshness metadata OK (257 sources, 3 unknown) (0.01s)", + ), + FreshnessMetadataProjectionCase( + description="failed batch explains that every source is unknown", + terminal_event_type="operation_failed", + terminal_payload={"metadata": {"item_count": 2}, "error_type": "AdapterUserError"}, + expected_terminal_line=( + "Freshness metadata FAIL (2 sources unknown, metadata lookup failed: " + "AdapterUserError) (0.01s)" + ), + ), + ), + ids=lambda case: case.description, +) +def test_given_freshness_metadata_terminal_when_projected_then_line_explains_outcome( + test_case: FreshnessMetadataProjectionCase, +) -> None: + stream: _FlushRecordingStream = _FlushRecordingStream() + projector: NativeProgressProjector = NativeProgressProjector(stream=stream, use_color=False) + start: LifecycleEvent = lifecycle_event( + "operation_started", + operation_id="freshness-metadata", + payload={ + "operation_kind": "freshness", + "operation_name": "source_freshness_metadata_observation", + }, + ) + terminal: LifecycleEvent = replace( + start, + event_id="freshness-metadata-terminal", + event_type=test_case.terminal_event_type, + payload={**start.payload, **test_case.terminal_payload, "duration_ms": 10.0}, + ) + + projector.consume(start) + projector.consume(terminal) + + assert stream.getvalue().splitlines() == [ + "Freshness metadata START", + test_case.expected_terminal_line, + ] + + @pytest.mark.parametrize( "test_case", ( diff --git a/tests/unit/src/sqlbuild/compiler/planner/_helpers/test_source_columns.py b/tests/unit/src/sqlbuild/compiler/planner/_helpers/test_source_columns.py index 4971503a3b..8311a2054c 100644 --- a/tests/unit/src/sqlbuild/compiler/planner/_helpers/test_source_columns.py +++ b/tests/unit/src/sqlbuild/compiler/planner/_helpers/test_source_columns.py @@ -20,9 +20,9 @@ _build_cursor_input_relations, _runtime_cursor_producer_names, build_planner_relations_context, - gather_source_columns, validate_source_cursor_input_columns, ) +from sqlbuild.compiler.planner._helpers.warehouse.source_columns import gather_source_columns from sqlbuild.compiler.planner.exceptions import PlannerInputError from sqlbuild.compiler.planner.models import ( CursorInputRelation, diff --git a/tests/unit/src/sqlbuild/compiler/planner/_helpers/warehouse/_test_types.py b/tests/unit/src/sqlbuild/compiler/planner/_helpers/warehouse/_test_types.py index 31dc7138a7..93a280e580 100644 --- a/tests/unit/src/sqlbuild/compiler/planner/_helpers/warehouse/_test_types.py +++ b/tests/unit/src/sqlbuild/compiler/planner/_helpers/warehouse/_test_types.py @@ -2,6 +2,8 @@ from dataclasses import dataclass +from sqlbuild.spec.contracts.models import SourceEntry + @dataclass(frozen=True) class SemanticSourceBatchCase: @@ -41,3 +43,17 @@ class SnowflakeSourceCandidateCase: declared_schemas: tuple[str, ...] expected_candidates: tuple[tuple[str | None, str | None, str], ...] expected_query_kinds: tuple[str, ...] + + +@dataclass(frozen=True) +class SourceTableProbeTestCase: + description: str + adapter_name: str + source: SourceEntry + listed_source_names: frozenset[str] + probe_errors: tuple[Exception, ...] + expected_probe_statements: tuple[str, ...] + expected_error_fragments: tuple[str, ...] = () + expected_help_fragment: str = "" + unexpected_output_fragment: str = "__not_present__" + role: str | None = None diff --git a/tests/unit/src/sqlbuild/compiler/planner/_helpers/warehouse/helpers.py b/tests/unit/src/sqlbuild/compiler/planner/_helpers/warehouse/helpers.py index a5c2f7b300..427f15cb71 100644 --- a/tests/unit/src/sqlbuild/compiler/planner/_helpers/warehouse/helpers.py +++ b/tests/unit/src/sqlbuild/compiler/planner/_helpers/warehouse/helpers.py @@ -5,15 +5,25 @@ import threading import time from collections.abc import Callable -from typing import Any +from pathlib import Path +from types import SimpleNamespace +from typing import Any, cast +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter from sqlbuild.adapter.contract.models import ColumnInfo +from sqlbuild.compiler.compile.models import CompiledObjectKey, CompiledProject, CompiledSource +from sqlbuild.compiler.compile.types import CompiledResourceType +from sqlbuild.compiler.discovery.models import DiscoveredSourceFile from sqlbuild.compiler.planner._helpers.warehouse.snapshot import ( _CursorModelInfo, _PhysicalCursorQuery, ) +from sqlbuild.compiler.planner._helpers.warehouse.source_tables import ( + check_selected_source_tables_exist, +) +from sqlbuild.compiler.planner.models import PlannerRelationsContext, PlannerScope from sqlbuild.compiler.planner.types import CursorType -from sqlbuild.spec.contracts.models import StartCursorsConfig +from sqlbuild.spec.contracts.models import SourceEntry, StartCursorsConfig class CursorBoundRows: @@ -150,3 +160,56 @@ def column_names(columns: tuple[ColumnInfo, ...]) -> tuple[str, ...]: """Return inspected column names in order.""" return tuple(column.name for column in columns) + + +def check_one_source_table( + *, + adapter: BaseAdapter, + connection: object, + source: SourceEntry, + listed_source_names: frozenset[str], +) -> None: + """Run the S405 check for one model that reads one source.""" + + model_key: CompiledObjectKey = CompiledObjectKey( + resource_type=CompiledResourceType.MODEL, name="orders" + ) + source_key: CompiledObjectKey = CompiledObjectKey( + resource_type=CompiledResourceType.SOURCE, name=source.name + ) + compiled_source: CompiledSource = CompiledSource( + key=source_key, + deps=(), + name=source.name, + source_entry=source, + source_file=DiscoveredSourceFile( + file_path=Path("/project/sources/raw.yml"), + relative_path=Path("sources/raw.yml"), + contents=f"sources:\n - name: {source.name}\n", + source_entries=(source,), + ), + ) + check_selected_source_tables_exist( + project=cast(CompiledProject, SimpleNamespace(sources=(compiled_source,))), + adapter=adapter, + connection=connection, + scope=PlannerScope( + upstream_deps={model_key: (source_key,)}, + downstream_deps={source_key: (model_key,)}, + all_keys={"orders": model_key, source.name: source_key}, + models_by_name={}, + selected_keys=frozenset({model_key, source_key}), + execution_order=(source_key, model_key), + ), + relations=PlannerRelationsContext( + model_locations={}, + seed_locations={}, + function_locations={}, + source_map={source.name: source}, + source_read_map={source.name: source}, + python_source_read_map={}, + source_warehouse_columns={}, + star_exclude_keyword="EXCLUDE", + listed_source_names=listed_source_names, + ), + ) diff --git a/tests/unit/src/sqlbuild/compiler/planner/_helpers/warehouse/test_source_tables.py b/tests/unit/src/sqlbuild/compiler/planner/_helpers/warehouse/test_source_tables.py new file mode 100644 index 0000000000..574fbf0e21 --- /dev/null +++ b/tests/unit/src/sqlbuild/compiler/planner/_helpers/warehouse/test_source_tables.py @@ -0,0 +1,178 @@ +from __future__ import annotations + +import pytest + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.compiler.planner.exceptions import PlannerInputError +from sqlbuild.spec.contracts.models import SourceEntry +from tests.unit.src.sqlbuild.adapter.contract.main.helpers import ( + ADAPTERS_BY_NAME, + BinderException, + CatalogException, + ProgrammingError, + RecordingProbeConnection, +) +from tests.unit.src.sqlbuild.compiler.planner._helpers.warehouse._test_types import ( + SourceTableProbeTestCase, +) +from tests.unit.src.sqlbuild.compiler.planner._helpers.warehouse.helpers import ( + check_one_source_table, +) + + +@pytest.mark.parametrize( + "test_case", + [ + SourceTableProbeTestCase( + description="listed source needs no probe", + adapter_name="duckdb", + source=SourceEntry(name="raw_orders", schema="raw", table="orders"), + listed_source_names=frozenset({"raw_orders"}), + probe_errors=(), + expected_probe_statements=(), + ), + SourceTableProbeTestCase( + description="databricks source without a catalog is probed by its rendered name", + adapter_name="databricks", + source=SourceEntry(name="raw_events", schema="raw", table="events"), + listed_source_names=frozenset(), + probe_errors=(), + expected_probe_statements=("SELECT 1 FROM `raw`.`events` WHERE 1=0",), + ), + SourceTableProbeTestCase( + description="sql server cross-database source is probed fully qualified", + adapter_name="sqlserver", + source=SourceEntry(name="erp_orders", database="erp", schema="dbo", table="orders"), + listed_source_names=frozenset(), + probe_errors=(), + expected_probe_statements=("SELECT 1 FROM erp.dbo.orders WHERE 1=0",), + ), + SourceTableProbeTestCase( + description="postgres materialized view source exists by probe", + adapter_name="postgres", + source=SourceEntry(name="order_totals", schema="reporting", table="order_totals"), + listed_source_names=frozenset(), + probe_errors=(), + expected_probe_statements=("SELECT 1 FROM reporting.order_totals WHERE 1=0",), + ), + SourceTableProbeTestCase( + description="snowflake stream source exists by probe", + adapter_name="snowflake", + source=SourceEntry( + name="order_changes", database="ANALYTICS", schema="RAW", table="ORDER_STREAM" + ), + listed_source_names=frozenset(), + probe_errors=(), + expected_probe_statements=("SELECT 1 FROM ANALYTICS.RAW.ORDER_STREAM WHERE 1=0",), + ), + ], + ids=lambda case: case.description, +) +def test_given_source_the_listing_missed_when_probe_succeeds_then_plan_is_not_blocked( + test_case: SourceTableProbeTestCase, +) -> None: + adapter: BaseAdapter = ADAPTERS_BY_NAME[test_case.adapter_name]() + connection: RecordingProbeConnection = RecordingProbeConnection(errors=test_case.probe_errors) + + check_one_source_table( + adapter=adapter, + connection=connection, + source=test_case.source, + listed_source_names=test_case.listed_source_names, + ) + + assert tuple(connection.statements) == test_case.expected_probe_statements + + +@pytest.mark.parametrize( + "test_case", + [ + SourceTableProbeTestCase( + description="duckdb missing table fails with S405", + adapter_name="duckdb", + source=SourceEntry(name="raw_payments", schema="raw", table="payments"), + listed_source_names=frozenset(), + probe_errors=( + CatalogException("Catalog Error: Table with name payments does not exist!"), + ), + expected_probe_statements=("SELECT 1 FROM raw.payments WHERE 1=0",), + expected_error_fragments=( + "1 source table read by selected resources does not exist in the warehouse:", + "source 'raw_payments' (raw.payments), read by orders", + ), + expected_help_fragment="create or load the table, correct its database", + unexpected_output_fragment="not readable", + ), + SourceTableProbeTestCase( + description="duckdb missing attached catalog fails with S405", + adapter_name="duckdb", + source=SourceEntry(name="lake_orders", database="lake", schema="raw", table="orders"), + listed_source_names=frozenset(), + probe_errors=(BinderException('Binder Error: Catalog "lake" does not exist!'),), + expected_probe_statements=("SELECT 1 FROM lake.raw.orders WHERE 1=0",), + expected_error_fragments=("source 'lake_orders' (lake.raw.orders), read by orders",), + expected_help_fragment="create or load the table", + ), + SourceTableProbeTestCase( + description="databricks missing schema fails with S405", + adapter_name="databricks", + source=SourceEntry(name="lake_orders", database="lake", schema="raw", table="orders"), + listed_source_names=frozenset(), + probe_errors=( + Exception("[SCHEMA_NOT_FOUND] The schema `lake`.`raw` cannot be found."), + ), + expected_probe_statements=("SELECT 1 FROM `lake`.`raw`.`orders` WHERE 1=0",), + expected_error_fragments=("source 'lake_orders' (`lake`.`raw`.`orders`)",), + expected_help_fragment="create or load the table", + ), + SourceTableProbeTestCase( + description="snowflake missing or unreadable names the role", + adapter_name="snowflake", + source=SourceEntry( + name="raw_payments", database="ANALYTICS", schema="RAW", table="PAYMENTS" + ), + listed_source_names=frozenset(), + probe_errors=( + ProgrammingError( + "Object 'ANALYTICS.RAW.PAYMENTS' does not exist or not authorized.", + errno=2003, + ), + ), + role="TRANSFORMER", + expected_probe_statements=("SELECT 1 FROM ANALYTICS.RAW.PAYMENTS WHERE 1=0",), + expected_error_fragments=( + "does not exist in the warehouse or is not readable by the current role:", + "(ANALYTICS.RAW.PAYMENTS), read by orders; the warehouse reports it does not " + "exist or is not readable by role TRANSFORMER", + ), + expected_help_fragment="grant SELECT on it to the role in use (TRANSFORMER)", + ), + ], + ids=lambda case: case.description, +) +def test_given_source_probe_not_found_when_checking_then_raises_s405( + test_case: SourceTableProbeTestCase, +) -> None: + adapter: BaseAdapter = ADAPTERS_BY_NAME[test_case.adapter_name]() + connection: RecordingProbeConnection = RecordingProbeConnection( + errors=test_case.probe_errors, role=test_case.role + ) + + with pytest.raises(PlannerInputError) as raised: + check_one_source_table( + adapter=adapter, + connection=connection, + source=test_case.source, + listed_source_names=test_case.listed_source_names, + ) + + assert raised.value.code == "S405" + assert all(fragment in raised.value.message for fragment in test_case.expected_error_fragments) + assert test_case.expected_help_fragment in str(raised.value.help) + assert "sources/raw.yml:2" in str(raised.value.help) + assert test_case.unexpected_output_fragment not in raised.value.message + str(raised.value.help) + assert tuple(connection.statements) == test_case.expected_probe_statements + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/unit/src/sqlbuild/compiler/source_freshness/main/_test_types.py b/tests/unit/src/sqlbuild/compiler/source_freshness/main/_test_types.py index ba57c5375b..8f08e90378 100644 --- a/tests/unit/src/sqlbuild/compiler/source_freshness/main/_test_types.py +++ b/tests/unit/src/sqlbuild/compiler/source_freshness/main/_test_types.py @@ -2,6 +2,8 @@ from datetime import UTC, datetime from typing import Any +from sqlbuild.adapter.contract.models import TableFreshnessMetadata +from sqlbuild.compiler.source_freshness.types import SourceFreshnessUnknownReason from sqlbuild.spec.contracts.types import SourceFreshnessValueKind @@ -190,3 +192,11 @@ class DirectSourceFreshnessExpressionTestCase: expression: str column: str expected_data_version: str + + +@dataclass(frozen=True) +class AdapterBatchIsolationTestCase: + description: str + outcomes: dict[str, TableFreshnessMetadata] + expected_observed_names: tuple[str, ...] + expected_unknown_reasons: dict[str, SourceFreshnessUnknownReason] diff --git a/tests/unit/src/sqlbuild/compiler/source_freshness/main/helpers.py b/tests/unit/src/sqlbuild/compiler/source_freshness/main/helpers.py index a11d68cedb..b17c55f509 100644 --- a/tests/unit/src/sqlbuild/compiler/source_freshness/main/helpers.py +++ b/tests/unit/src/sqlbuild/compiler/source_freshness/main/helpers.py @@ -5,7 +5,11 @@ from types import MappingProxyType from typing import Any -from sqlbuild.adapter.contract.models import RelationLookup +from sqlbuild.adapter.contract.models import ( + RelationLookup, + TableFreshnessMetadata, + TableFreshnessRequest, +) from sqlbuild.adapter.contract.types import FrameworkType from sqlbuild.adapter.relations.main.relation_lookup import build_relation_lookup from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter @@ -22,6 +26,7 @@ SourceFreshnessRecord, SourceFreshnessRenderers, ) +from sqlbuild.spec.contracts.models import SourceEntry, SourceFreshnessConfig from sqlbuild.spec.contracts.types import SourceFreshnessStrategy, SourceFreshnessValueKind @@ -284,3 +289,43 @@ def compiled_key(raw: str) -> CompiledObjectKey: name: str raw_type, name = raw.split(":", 1) return CompiledObjectKey(resource_type=CompiledResourceType(raw_type), name=name) + + +class PerTableFreshnessDuckDbAdapter(DuckDbAdapter): + """DuckDB adapter double returning configured per-table freshness outcomes.""" + + def __init__(self, *, outcomes: dict[str, TableFreshnessMetadata]) -> None: + super().__init__() + self.outcomes: dict[str, TableFreshnessMetadata] = outcomes + self.batch_requests: list[tuple[TableFreshnessRequest, ...]] = [] + + def supports_table_freshness_metadata(self) -> bool: + return True + + def get_tables_freshness_metadata( + self, + *, + connection: Any, + requests: tuple[TableFreshnessRequest, ...], + ) -> dict[TableFreshnessRequest, TableFreshnessMetadata]: + del connection + self.batch_requests.append(requests) + requests_by_name: dict[str, TableFreshnessRequest] = { + request.name: request for request in requests + } + return {requests_by_name[name]: metadata for name, metadata in self.outcomes.items()} + + +def observed_table_freshness(version: int) -> TableFreshnessMetadata: + """Return an observed integer table freshness outcome.""" + + return TableFreshnessMetadata(data_version=version, value_kind="integer") + + +def physical_table_sources(*, freshness: SourceFreshnessConfig | None) -> tuple[SourceEntry, ...]: + """Return the orders, customers, and payments physical table sources.""" + + return tuple( + SourceEntry(name=name, schema="raw", table=name, freshness=freshness) + for name in ("orders", "customers", "payments") + ) diff --git a/tests/unit/src/sqlbuild/compiler/source_freshness/main/test_adapter_observation.py b/tests/unit/src/sqlbuild/compiler/source_freshness/main/test_adapter_observation.py new file mode 100644 index 0000000000..7d087e4ad2 --- /dev/null +++ b/tests/unit/src/sqlbuild/compiler/source_freshness/main/test_adapter_observation.py @@ -0,0 +1,164 @@ +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Any + +import pytest + +from sqlbuild.adapter.contract.models import TableFreshnessMetadata +from sqlbuild.compiler.source_freshness.main._planning import ( + build_direct_source_freshness_planning_result, +) +from sqlbuild.compiler.source_freshness.main.adapter_observation import ( + observe_adapter_sources_freshness, +) +from sqlbuild.compiler.source_freshness.models import ( + AdapterSourceFreshnessBatch, + DirectSourceFreshnessPlanningResult, +) +from sqlbuild.compiler.source_freshness.types import SourceFreshnessUnknownReason +from sqlbuild.observability import ( + EventDispatcher, + LifecycleEvent, + dispatcher_scope, + invocation_scope, +) +from sqlbuild.spec.contracts.models import SourceFreshnessConfig +from sqlbuild.spec.contracts.types import SourceFreshnessStrategy +from tests.unit.src.sqlbuild.compiler.source_freshness.main._test_types import ( + AdapterBatchIsolationTestCase, +) +from tests.unit.src.sqlbuild.compiler.source_freshness.main.helpers import ( + PerTableFreshnessDuckDbAdapter, + observed_table_freshness, + physical_table_sources, +) + +_OBSERVED_AT: datetime = datetime(2026, 1, 15, 12, 0, 0, tzinfo=UTC) + + +@pytest.mark.parametrize( + "test_case", + [ + AdapterBatchIsolationTestCase( + description="absent table metadata leaves other sources observed", + outcomes={ + "orders": observed_table_freshness(1), + "customers": observed_table_freshness(2), + "payments": TableFreshnessMetadata.unavailable( + message="table freshness metadata not found for payments" + ), + }, + expected_observed_names=("customers", "orders"), + expected_unknown_reasons={"payments": SourceFreshnessUnknownReason.UNAVAILABLE}, + ), + AdapterBatchIsolationTestCase( + description="null data version is unknown for that source only", + outcomes={ + "orders": observed_table_freshness(1), + "customers": TableFreshnessMetadata(data_version=None, value_kind="integer"), + "payments": observed_table_freshness(3), + }, + expected_observed_names=("orders", "payments"), + expected_unknown_reasons={"customers": SourceFreshnessUnknownReason.UNAVAILABLE}, + ), + AdapterBatchIsolationTestCase( + description="unavailable metadata and unreturned table are isolated", + outcomes={ + "orders": TableFreshnessMetadata.unavailable(message="found VIEW for orders"), + "customers": observed_table_freshness(2), + }, + expected_observed_names=("customers",), + expected_unknown_reasons={ + "orders": SourceFreshnessUnknownReason.UNAVAILABLE, + "payments": SourceFreshnessUnknownReason.UNAVAILABLE, + }, + ), + ], + ids=lambda case: case.description, +) +def test_given_batch_with_unobservable_table_when_observing_then_isolates_each_source( + test_case: AdapterBatchIsolationTestCase, +) -> None: + adapter: PerTableFreshnessDuckDbAdapter = PerTableFreshnessDuckDbAdapter( + outcomes=test_case.outcomes + ) + events: list[LifecycleEvent] = [] + dispatcher: EventDispatcher = EventDispatcher() + dispatcher.subscribe_lifecycle(subscriber=events.append, accepts_opaque=False) + + with invocation_scope("freshness-invocation"), dispatcher_scope(dispatcher): + batch: AdapterSourceFreshnessBatch = observe_adapter_sources_freshness( + adapter=adapter, + connection=object(), + sources=physical_table_sources( + freshness=SourceFreshnessConfig(strategy=SourceFreshnessStrategy.ADAPTER) + ), + observed_at=_OBSERVED_AT, + ) + + assert tuple(sorted(batch.observations)) == test_case.expected_observed_names + assert { + name: unknown.reason for name, unknown in batch.unknown.items() + } == test_case.expected_unknown_reasons + assert len(adapter.batch_requests) == 1 + assert events[-1].event_type == "operation_completed" + assert events[-1].payload["metadata"] == { + "item_count": 3, + "unknown_count": len(test_case.expected_unknown_reasons), + } + + +@pytest.mark.parametrize( + "test_case", + [ + AdapterBatchIsolationTestCase( + description="auto-observed sources keep freshness beside absent table metadata", + outcomes={ + "orders": observed_table_freshness(1), + "customers": observed_table_freshness(2), + "payments": TableFreshnessMetadata.unavailable( + message="table freshness metadata not found for payments" + ), + }, + expected_observed_names=("customers", "orders"), + expected_unknown_reasons={"payments": SourceFreshnessUnknownReason.UNAVAILABLE}, + ), + ], + ids=lambda case: case.description, +) +def test_given_batch_with_absent_table_metadata_when_planning_then_records_others_and_reasons( + test_case: AdapterBatchIsolationTestCase, +) -> None: + adapter: PerTableFreshnessDuckDbAdapter = PerTableFreshnessDuckDbAdapter( + outcomes=test_case.outcomes + ) + connection: Any = adapter.connect({"database": ":memory:"}) + try: + result: DirectSourceFreshnessPlanningResult = build_direct_source_freshness_planning_result( + adapter=adapter, + connection=connection, + sources=physical_table_sources(freshness=None), + state_database=None, + state_schemas=("state_schema",), + observed_at=_OBSERVED_AT, + run_id="planning", + render_qualified_name=adapter.render_qualified_name, + state_table_exists_by_schema={"state_schema": False}, + ) + finally: + adapter.close(connection) + + assert ( + tuple(sorted(record.source_name for record in result.observed_records)) + == test_case.expected_observed_names + ) + assert result.unknown_source_names == tuple(sorted(test_case.expected_unknown_reasons)) + assert { + name: unknown.reason for name, unknown in result.unknown_sources.items() + } == test_case.expected_unknown_reasons + assert len(result.changed_identities) == len(test_case.expected_observed_names) + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/website/src/content/docs/docs/cli/freshness.mdx b/website/src/content/docs/docs/cli/freshness.mdx index b36a34b56a..9938ad5f72 100644 --- a/website/src/content/docs/docs/cli/freshness.mdx +++ b/website/src/content/docs/docs/cli/freshness.mdx @@ -62,6 +62,12 @@ Summary: observed=3 changed=0 unchanged=0 tolerated=0 unknown=0 errors=0 Sources without explicit `freshness:` config are auto-observed using the `adapter` strategy if the adapter supports table metadata. Sources that can't be observed (expression sources, managed sources without freshness config on unsupported adapters) show as `unknown`. +Each source is reported on its own. Adapter metadata is fetched in one batched lookup, but a +source the metadata can't describe affects only that source: it shows as `unknown` with the +reason (for example a view, an external table, a table with no last-modified time, or a table the +metadata lookup did not find). A column or SQL freshness query that fails shows as `error`. The +other sources are still observed. + ## Comparing against state Use `--state` to compare current observations against the freshness state stored from the last successful build: @@ -93,7 +99,7 @@ Summary: observed=0 changed=1 unchanged=1 tolerated=1 unknown=0 errors=0 | `changed` | Data version differs from the stored state | | `unchanged` | Data version matches the stored state exactly | | `tolerated` | Data version differs but is within the `lag_tolerance` threshold | -| `unknown` | No freshness config and adapter metadata unavailable, or no previous state to compare against | +| `unknown` | No freshness config and adapter metadata unavailable, metadata unavailable for an existing table, or no previous state to compare against | | `error` | Observation failed (e.g. source table does not exist, query error) | ## CI integration diff --git a/website/src/content/docs/docs/cli/plan.mdx b/website/src/content/docs/docs/cli/plan.mdx index b11e5ec5f0..e7f024217d 100644 --- a/website/src/content/docs/docs/cli/plan.mdx +++ b/website/src/content/docs/docs/cli/plan.mdx @@ -114,3 +114,10 @@ error[S301]: cannot build selected scope: audit 'relationships' on 'stg_orders' Load or build the resource first (for a seed, `sqb seed`), or add it to the selection. When both kinds are missing, the error lists the missing upstream dependencies and then the audit reads. + +## Missing source tables + +When a selected model reads an external source whose table does not exist in the warehouse, +planning fails with `S405` before anything runs. The error names each missing source, its table, +the selected models that read it, and the source declaration file and line. See +[Source freshness: Missing source tables](/docs/concepts/planning/source-freshness#missing-source-tables). diff --git a/website/src/content/docs/docs/concepts/planning/source-freshness.mdx b/website/src/content/docs/docs/concepts/planning/source-freshness.mdx index da72f9a6e6..19fa8ab0af 100644 --- a/website/src/content/docs/docs/concepts/planning/source-freshness.mdx +++ b/website/src/content/docs/docs/concepts/planning/source-freshness.mdx @@ -21,6 +21,52 @@ During planning, SQLBuild observes the current data version of each source that Sources without explicit `freshness:` config are auto-observed using the `adapter` strategy if the adapter supports table metadata and the source has a physical table (not an expression source, not a managed source). +Adapter metadata for all sources is fetched in one batched lookup, but each source gets its own +result. When one source can't be observed, only that source is unknown; the others keep their +freshness. The `Freshness metadata` progress line reports the outcome, for example +`Freshness metadata OK (257 sources, 3 unknown)`. The plan adds one warning per reason, listing up +to 10 sources, such as `source freshness unknown (unavailable) for raw_events, raw_sessions: ...`. +Sources without freshness configuration that the adapter can't observe are not warned about. +Every unknown source is listed with its reason and message under `unknown_source_details` in the +`direct_source_freshness` metadata of `sqb plan --json`; run `sqb freshness` for the same detail +in text form. + +## Missing source tables + +A source table that a selected model reads must exist before anything runs. If it doesn't, +planning fails with `S405`, naming each missing source, its table, and the selected models that +read it, and pointing at the source declaration: + +``` +error[S405]: cannot build selected scope: 1 source table read by selected resources does not exist in the warehouse: + - source 'raw_payments' (raw.payments), read by payments + = help: create or load the table, correct its database, schema, or table in the source declaration, or stop reading it from the selected models; declared 'raw_payments' at sources/raw.yml:16 +``` + +Create the table, fix the declaration, or leave the reading models out of the selection. Managed +sources and sources with a loader are exempt, because SQLBuild creates their tables when it loads +them. Expression sources, sources without a schema, and wildcard tables are also exempt. A missing +table that only unselected models read does not fail the plan; its freshness is reported as unknown. + +The check first looks for each source in the warehouse listing that planning already reads for +source columns, so it normally issues no extra queries. A source the listing does not find (for +example because of identifier case, a stream, a materialized view, or a table in another database) +is then probed directly with `SELECT 1 FROM WHERE 1=0`, using the same relation name the +model SQL reads. Probes for several such sources run concurrently, within the same limit as other +warehouse inspection reads. Only a not-found error from that probe counts as missing, including a +missing schema, catalog, or attached database; any other error (for example a connection error) +fails planning with that error. On Snowflake the probe may need a running warehouse, but it only +runs for sources the listing did not find. + +Snowflake reports a missing table and a table the current role can't read with the same error, and +its listing hides tables the role can't access. For those sources `S405` says the table "does not +exist or is not readable by" the role, names the role in use when the connection reports it, and the help suggests both fixes: +create or load the table, or grant `SELECT` on it to that role. + +A table that a model's own `pre_hook` creates does not exist yet when the plan is made, so planning +fails with `S405`. Declare such a table as a managed source or a source with a loader, or create it +before planning. + ## Lag tolerance For timestamp-based freshness, `lag_tolerance` controls how much the observed value can drift before being considered a real change. If the current timestamp is within the tolerance of the previous observation, the source is treated as unchanged. This is useful for sources where the freshness timestamp moves by seconds or minutes on every query but the underlying data hasn't meaningfully changed.