diff --git a/fensu.toml b/fensu.toml index 1fda03ea27..b8872f79dc 100644 --- a/fensu.toml +++ b/fensu.toml @@ -278,3 +278,20 @@ contract = "src/sqlbuild/adapter/contract/classes/strict_adapter.py:StrictAdapte forbidden_owners = ["src/sqlbuild/adapter/contract/classes/base_adapter.py:BaseAdapter"] paths = ["src/sqlbuild/adapter/contract/classes/*", "src/sqlbuild/adapters/*/classes/*"] reason = "tests/unit/src/sqlbuild/adapter/contract/classes/strict_adapter/test_strict_adapter.py requires every first-class adapter to define each StrictAdapter method itself instead of inheriting it from BaseAdapter." + +[[targets.python.rule_exceptions]] +rule = "XSB051" +path = "src/sqlbuild/executor/migrations/_helpers/old_name_views.py" +symbols = [ + "present_old_name_views", + "recreate_old_name_views", + "refresh_old_name_views", + "run_planned_old_name_steps", +] +reason = "Old-name steps re-read facts and physical state per move so a retry never repeats a completed step, and compatibility views are re-created in dependency order, so a chained view must read the columns of the view it selects from after that view is re-created. Each view's existence is checked right before it is redefined, and a model has only its own short chain of old names." + +[[targets.python.rule_exceptions]] +rule = "XSB051" +path = "src/sqlbuild/executor/old_name_views/_helpers/janitor.py" +symbols = ["plan_old_name_views", "apply_old_name_view_drops"] +reason = "Janitor reads each compatibility view's stored definition right before deciding to drop it, so a relation that replaced the view is never dropped; a project has few live old names." diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md index 40d21e2602..0e700532a3 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md @@ -38,7 +38,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in - [Schemas](concepts/models/schemas.md) (`concepts/models/schemas`) - Declare model columns inline or reuse canonical inherited schemas. - [Type Enforcement](concepts/models/type-enforcement.md) (`concepts/models/type-enforcement`) - Understand declared model types, static checks, and runtime casting. - [Contracts](concepts/models/contracts.md) (`concepts/models/contracts`) - Validate required or exact model output schemas. -- [Model migrations](concepts/models/migrations.md) (`concepts/models/migrations`) - Keep the history of an incremental or snapshot model when you rename it. +- [Model migrations](concepts/models/migrations.md) (`concepts/models/migrations`) - Keep a renamed model's history, and keep its old name working for a while. - [Column migrations](concepts/models/column-migrations.md) (`concepts/models/column-migrations`) - Keep a column's history when you rename it in an incremental or snapshot model. - [Hooks](concepts/models/hooks.md) (`concepts/models/hooks`) - Run SQL or Python lifecycle hooks around model materialization. - [SQL Hooks](concepts/models/hooks/sql.md) (`concepts/models/hooks/sql`) - Define, parameterize, compile, and invoke reusable or inline SQL lifecycle hooks. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md index 4146f7ce83..253b81aa58 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md @@ -11,6 +11,7 @@ Online: https://sqlbuild.com/docs/cli/janitor/ - Usage - Flags - Lifecycle +- Compatibility views - Relation age - Audit record - Restoring an archive @@ -34,6 +35,7 @@ sqb --project-dir janitor [flags] | `--auto-approve` | Skip the confirmation prompt and run the planned actions immediately | | `--retention-days` | Override the configured retention period (days) before a stale relation is archived | | `--direct-state-history-versions` | Override how many state-history versions are kept per identity | +| `--drop-old-name-view ` | Drop the [compatibility view](../concepts/models/migrations.md#old-names) at a renamed model's old name before it expires. Repeat for several views. Fails with `C503` if no compatibility view is recorded at the name | ## Lifecycle @@ -58,6 +60,20 @@ Long names are shortened with a short hash so they fit the adapter's identifier Setting both `retention_days` and `archive_retention_days` to `0` archives and deletes stale relations in the same run. +## Compatibility views + +When a model is renamed, SQLBuild keeps a [compatibility view](../concepts/models/migrations.md#old-names) at its old name. The janitor lists each one with the model it reads and its expiry: + +``` +Old name views (1) +└── prod.revenue -> model:daily_revenue + └── expires 2026-10-28 (live; drop early with --drop-old-name-view prod.revenue) +``` + +A compatibility view is never archived as a stale relation. Once it expires, or when it is named with `--drop-old-name-view`, the janitor drops it and records the drop in `_sqlbuild_old_name_views`. Before dropping, the janitor checks that the relation at the old name is still the view SQLBuild created there, by comparing its stored definition with the view's SQL. Any other relation at that name, such as a view a project model built after an interrupted early drop, is left alone: the janitor records the compatibility view as dropped and lists it as `record dropped (name now used by another relation, model:revenue)`, naming the model when the name is a project destination. A compatibility view that no longer exists is recorded as dropped too. + +Run the same SQLBuild version for `sqb janitor` as for builds. An older janitor does not know compatibility views and archives them as stale relations. + ## Relation age `retention_days` is measured from the last time a relation was written, as reported by the warehouse. For a relation the project no longer builds, that is roughly when it stopped being built. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/column-migrations.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/column-migrations.md index 6ad05a0950..85eeb0161a 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/column-migrations.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/column-migrations.md @@ -46,7 +46,7 @@ After the rename is recorded, `migrate_from` has no further effect, and SQLBuild A declaration is rejected when compiling if: -- the model is not incremental or snapshot; +- the model is not incremental or snapshot, unless it is a table or view model that declares its own `migrate_from`; there the column's `migrate_from` only names the old column in the [compatibility view](migrations.md#old-names) at the model's old name; - `migrate_from` names the column itself, or a qualified name instead of one column; - two columns declare the same `migrate_from`; - a declared column migrates from another column that itself declares `migrate_from`. Chains and swaps can't be renamed in place safely; rename through a name no other column uses, or rebuild with `--full-refresh`. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md index 5c57343513..661019e3aa 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md @@ -23,7 +23,7 @@ Online: https://sqlbuild.com/docs/concepts/models/configuration/ | `materialized` | `view`, `table`, `incremental`, `snapshot`, or a custom materialization name | | `tags` | Tags used by selectors | | `description` | Human-readable model description | -| `columns` | Model-local column declarations, inherited-column audit augmentation, or column renames with `migrate_from` (incremental and snapshot models; see [Column migrations](column-migrations.md)) | +| `columns` | Model-local column declarations, inherited-column audit augmentation, or column renames with `migrate_from` (incremental and snapshot models; see [Column migrations](column-migrations.md)). On a table or view model that declares its own `migrate_from`, a column `migrate_from` only names the old column in the [compatibility view](migrations.md#old-names) | | `model_schema` | Reusable column schema name | | `audits` | Model-level audit instances | | `enums` | Model-local enum declarations; names must begin with `_` | @@ -81,8 +81,9 @@ Table promotion mode is a project setting rather than a `MODEL()` field. Staged | `full_refresh` | Optional model execution override: `false` always runs incrementally, `true` always full-refreshes, and omission follows the command | | `on_schema_change` | `append_new_columns`, `sync_all_columns`, `ignore`, or `fail` | | `replay_on_change` | `forward`, `full`, or `bounded-` | -| `migrate_from` | Old model name (or `schema.name`) whose data this model takes over; see [Model migrations](migrations.md). Also valid on snapshot models | -| `migrate_force` | `true` replaces a destination that already has its own build history during a migration | +| `migrate_from` | Old model name (or `schema.name`) whose data this model takes over; see [Model migrations](migrations.md). Also valid on snapshot, table, and view models | +| `migrate_force` | `true` replaces a destination that already has its own build history during a migration. Incremental and snapshot models only | +| `old_name_view` | How long a [compatibility view](migrations.md#old-names) keeps the old name working after a migration, such as `7d`, or `false` for none. Overrides `[migrations] old_name_views`; valid on every model | See [Incremental](../incremental.md) for full semantics. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md index 65baada20e..ed7efd0806 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md @@ -2,13 +2,24 @@ # Model migrations -> Keep the history of an incremental or snapshot model when you rename it. +> Keep a renamed model's history, and keep its old name working for a while. Online: https://sqlbuild.com/docs/concepts/models/migrations/ +## Contents + +- Declaring a migration +- Decisions +- Automatic discovery +- How a move runs +- Old names +- State +- Adapter support +- Previewing another target + Renaming an incremental or snapshot model normally means rebuilding it from scratch under the new name, which loses any history the source data can no longer reproduce. A model migration moves the existing relation's data to the new name instead, then continues building incrementally. -Migrations apply to incremental and snapshot models. To keep a column's history when you rename the column, see [Column migrations](column-migrations.md). +Table and view models can declare a migration too. They hold no history, so they are rebuilt under the new name, but they keep their identity for change detection. For every materialization, the old name keeps working for a while through a [compatibility view](#old-names). To keep a column's history when you rename the column, see [Column migrations](column-migrations.md). ## Declaring a migration @@ -25,7 +36,7 @@ MODEL ( Use `schema.name` when the old relation is in a different schema of the same database. -On the next build, SQLBuild copies the old relation's data to the new name through a staging table (see [How a move runs](#how-a-move-runs)), records the move, and builds the model incrementally from the migrated state. There is no first-run rebuild, and no `replay_on_change` replay caused by the rename. The old relation is never modified or dropped. Once it is no longer part of the project, [janitor](../../cli/janitor.md) archives it like any other stale relation. +On the next build, SQLBuild copies the old relation's data to the new name through a staging table (see [How a move runs](#how-a-move-runs)), records the move, and builds the model incrementally from the migrated state. There is no first-run rebuild, and no `replay_on_change` replay caused by the rename. After the model builds, the old relation is archived and a compatibility view takes its name (see [Old names](#old-names)). After the move is recorded, `migrate_from` has no further effect, and SQLBuild tells you it can be removed. Keep it until every target that needs the move has built. @@ -47,11 +58,13 @@ The origin and destination must be compatible under the model's normal `on_schem `migrate_force true` only affects the `conflict` case and is safe to leave in the header. +Table and view models only ever see `migrate`, `done`, and `origin missing`: nothing is replaced, so there is no conflict. `migrate_force` on a table or view model is a compile error. In `sqb plan --json`, their decision is `renamed`. + ## Automatic discovery If a selected model has never been built, SQLBuild compares it with models that were removed from the project but whose relations still exist. When exactly one removed model has equivalent logic, it is migrated automatically. Equivalent means the same query and configuration, ignoring the model's own name, CTE and table alias names, comments, formatting, and storage-only settings. Renamed upstream models are matched first, so a renamed chain of models is migrated together. Removed models whose data was already moved on by a recorded migration are not candidates, so a model renamed several times matches its latest table. -Renamed tables and views are matched the same way. They are still rebuilt under the new name, since they hold no history, but they keep their identity for change detection, so incremental models downstream of a renamed view are not replayed or rebuilt. These renames are recorded as `renamed` events. +Renamed tables and views are matched the same way. They are rebuilt under the new name, since they hold no history, but they keep their identity for change detection, so incremental models downstream of a renamed view are not replayed or rebuilt. These renames are recorded as `renamed` events. Automatic discovery never guesses. If a match is ambiguous, SQLBuild warns (`M107`) and builds from scratch; declare `migrate_from` to choose. An explicit `migrate_from` always wins. Automatic discovery covers renames within the project's schemas; use `migrate_from` to move a model to another schema. Relations last built before this feature carry no stored fingerprint and aren't matched automatically. @@ -63,15 +76,104 @@ A move never overwrites the destination directly: 2. The staging table is checked: it must exist, have the expected columns and table type, and, for a physical copy, the same row count as the old relation. 3. The staging table is swapped in as the destination. If a destination already existed, for example after `superseded replace` or `forced replace`, it is kept as `_sqb_archive____migration_previous__` rather than dropped. 4. The move is recorded, then the model builds as usual. +5. After the model builds successfully, the old relation is archived and a compatibility view is created at its name (see [Old names](#old-names)). -The old relation is never modified. Staging and replaced tables use the [janitor](../../cli/janitor.md) archive format, so an abandoned staging table or a replaced destination is deleted by janitor after `archive_retention_days`. A staging table left by an interrupted run is never reused; the next run stages again. Don't run janitor at the same time as a build in the same schema. +Until step 5, the old relation is not modified. Staging and replaced tables use the [janitor](../../cli/janitor.md) archive format, so an abandoned staging table or a replaced destination is deleted by janitor after `archive_retention_days`. A staging table left by an interrupted run is never reused; the next run stages again. Don't run janitor at the same time as a build in the same schema. `sqb plan` shows how each move will run, for example `physical copy, transient -> permanent, promote by swap`. +## Old names + +A rename breaks every query outside the project that still reads the old name: dashboards, notebooks, other teams' jobs. After the renamed model builds successfully, SQLBuild keeps the old name working for a while: + +1. The old relation is renamed to `_sqb_archive____migration_origin__`. +2. A view is created at the old name that selects from the new relation. +3. The view gets the privileges that were granted on the old relation, read from the archive. Only existing grants are copied; nothing new is granted. + +On DuckDB, MotherDuck, PostgreSQL, and SQL Server these steps and their records commit in one transaction. On Snowflake, BigQuery, and Databricks each step is recorded after it runs, and a re-run resumes from the first unrecorded step without repeating one. A failure is reported as `M117`; re-run the build. + +| Adapter | Privileges copied to the view | +|---------|-------------------------------| +| Snowflake | `SHOW GRANTS` on the archive, replayed to roles and database roles, keeping `WITH GRANT OPTION`. `OWNERSHIP`, grants to shares, and privileges that don't apply to views are skipped | +| PostgreSQL | Table and column privileges from the catalog, including grants to `PUBLIC` and `WITH GRANT OPTION`. The owner's own privileges are not copied | +| BigQuery | Table IAM role bindings from `INFORMATION_SCHEMA.OBJECT_PRIVILEGES`, replayed with `GRANT`. See the note below | +| Databricks | Unity Catalog grants on the table itself that apply to views, such as `SELECT`. `OWN`, `MODIFY`, and inherited catalog or schema grants are skipped | +| SQL Server | Object and column permissions, including `DENY` and `WITH GRANT OPTION`, that apply to views | +| DuckDB, MotherDuck | No object privileges exist | + +Column privileges are copied to the view's column of the same name, which is the old name when the view aliases a renamed column. A column privilege on a column the view no longer exposes is not copied; the view shows nothing of that column. + +The copied grants are recorded on the view's creation. After that, the view's own privileges are what counts: when SQLBuild redefines the view after a rebuild, the view ends up with exactly the privileges it had just before, so a privilege you revoke from the view stays revoked and one you grant on the view is kept. SQLBuild uses `CREATE OR REPLACE VIEW` on PostgreSQL, `COPY GRANTS` on Snowflake, and `ALTER VIEW` on SQL Server and Databricks. + +Creating or replacing a view can also apply privileges you did not give it: PostgreSQL applies `ALTER DEFAULT PRIVILEGES` when it re-creates a view, and Snowflake applies the schema's future grants for views on `CREATE OR REPLACE VIEW … COPY GRANTS`. SQLBuild reads the view's privileges before every create or redefinition and reconciles afterwards: it grants what went missing and revokes what was added. A new compatibility view likewise ends up with only the privileges copied from the old relation. Ownership and the owner's own privileges are never changed. + +A recorded view that no longer exists at the old name is not re-created by a build; [janitor](../../cli/janitor.md) records it as dropped. + +On Snowflake, BigQuery, Databricks, DuckDB, and MotherDuck, views read their tables by name when queried, so a build redefines a compatibility view only when its SQL would change, for example when the view lists columns and the new relation's columns changed. PostgreSQL views are bound to the table itself, so they are redefined whenever the table is replaced. + +BigQuery cannot rename views, so an old view model is re-created under the archive name with its IAM bindings, and then dropped. + +On BigQuery, a view reads its tables with the permissions of whoever queries it. A principal that could read the old table keeps access to the old name through the copied bindings, but also needs read access on the new table. SQLBuild does not grant that and does not create authorized views; the build warns with `M118` and names the principals. + +If columns were renamed, the view presents them under their old names, for example `SELECT order_id, revenue AS amount FROM analytics.daily_revenue`. Incremental and snapshot models alias every recorded [column migration](column-migrations.md). A table or view model that declares `migrate_from` can declare old column names the same way, as `revenue (migrate_from amount)`; this only aliases the column in the compatibility view, since the table is rebuilt with the new name anyway. The view follows the new relation's current columns after every build of the model. + +`sqb plan` shows what happens at the old name: + +``` +Migrations (1) +└── daily_revenue migrate analytics.revenue -> analytics.daily_revenue + ├── transfer rebuild (table) + └── old name analytics.revenue + ├── view until 2026-10-28 + ├── columns amount <- revenue + ├── grants copied from the old table + └── old table archived +``` + +A move resumed after the archive shows `old table already archived`. Once the view exists it shows `view live until 2026-10-28` and how many grants were copied. When no view is kept, the block shows `left for janitor` and the reason, such as `old_name_view false` or `name reused by model:revenue`. + +In `sqb plan --json`, each migration has an `old_name` object with `action` (`archive_and_view`, `view_only`, `live`, or `none`), `view`, `reads`, `expires_at`, `column_aliases`, `grants_copied` (the number of grants copied once the view exists, otherwise `null`), and `reason`. The top-level `old_names` list also includes steps that resume an earlier move. + +The view expires after 30 days by default. After that, [janitor](../../cli/janitor.md) drops it; `sqb janitor --drop-old-name-view analytics.revenue` drops it earlier. Change the default for the project, or turn the views off with `false`: + +```toml +[migrations] +old_name_views = "7d" +``` + +A model header overrides the project setting with `old_name_view 7d` or `old_name_view false`. The retention of a move is fixed when the move is recorded. + +With views off, or when a view can't be kept, the old relation is left in place and janitor archives it once it is no longer part of the project, as before. No view is created when: + +- the old relation was not built by SQLBuild; +- another model in the project builds at the old name; +- the old name is itself a compatibility view of an earlier rename. + +Moves recorded before a SQLBuild version that supports old-name views never gain a view later. + +### Code inside the project + +The compatibility view is for consumers outside the project. Project code must use the new model: + +- Python nodes, hooks, and loaders whose literal SQL names the old name fail with `P008`, at compile time for a declared `migrate_from` and at plan time for an automatically discovered rename. SQL models reading it through `__ref` are unaffected, since `__ref` resolves the new name. +- A new model whose name is a live compatibility view stops the build with `M114`, which names the view and its expiry. Drop the view early with janitor, or choose another name. + +### PostgreSQL views + +PostgreSQL views are bound to the table they read, not its name. When a build replaces the renamed model's table, or drops or retypes its columns, SQLBuild points its compatibility views at the new table in the same transaction, so they are never missing, even if the build is interrupted. A view is replaced in place when its columns allow it, so views you build on top of the compatibility view keep working; otherwise it is dropped and re-created with the privileges it had, which fails if another view depends on it. + +Views outside SQLBuild that read the old relation follow it into the archive and keep reading its old data; the build warns with `M115` and names them. Re-create them against the old name or the new relation. On the other adapters, views read relations by name, so no rebinding is needed. + +### Upgrading + +Upgrade the SQLBuild version that runs `sqb janitor` together with the one that runs builds. An older janitor does not know compatibility views and archives them as stale relations. + ## State Each move is recorded in an append-only `_sqlbuild_migrations` table in the destination schema. A missing record means completion is unrecorded; staging or promotion may already have happened. Re-running after an interruption recovers through the migration decisions above. +What happens at the old name is recorded in an append-only `_sqlbuild_old_name_views` table next to it, one row per step: `required` (with the move), `origin_archived`, `view_created`, and `view_dropped`. Rows are never updated. + ## Adapter support | Adapter | Staging | Promotion | diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md index 5f8ca1c9cf..0d23f38517 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md @@ -19,6 +19,7 @@ Online: https://sqlbuild.com/docs/concepts/project-configuration/ - References - Project variables - Janitor +- Migrations - Scenario - dbt - Skills @@ -502,6 +503,19 @@ exclude_patterns = ["audit_*", "tmp_*"] See [janitor](../cli/janitor.md) for the archive and delete lifecycle. +## Migrations + +Defaults for [model migrations](models/migrations.md): + +```toml +[migrations] +old_name_views = "30d" +``` + +| Field | Default | Description | +|-------|---------|-------------| +| `old_name_views` | `"30d"` | How long a renamed model's old name keeps working through a [compatibility view](models/migrations.md#old-names) before janitor drops it. `false` keeps the old relation in place instead, as before. A model's `old_name_view` header overrides it. | + ## Scenario Configuration for scenario run isolation, snapshot capture safety limits, and local type overrides: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/sql-references.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/sql-references.md index 2d73271471..3ed1ebdd4c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/sql-references.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/sql-references.md @@ -87,6 +87,12 @@ Loaders run before models and seeds, so a loader naming a model or seed fails wi cannot read them. A loader naming a source is told to use `ctx.source("name")`, or `ctx.loader(upstream_loader)` when the source is populated by a loader it depends on. +The old name of a [renamed model](../models/migrations.md#old-names) counts as a project +relation while a compatibility view is kept there, so literal SQL still reading it fails with +`P008` and names the new model. A declared `migrate_from` is checked at compile time; an +automatically discovered rename is checked when planning, and stops the build. A hook's own model +does not exempt its old name. + **At run time**, SQL built dynamically cannot be checked earlier. Before sending it, `ctx.query()` and `ctx.execute_sql()` parse it in the adapter's dialect, resolve unqualified names against the current schema, and compare them with the project relations resolved for the run. A diff --git a/src/sqlbuild/adapter/contract/_helpers/view_definitions.py b/src/sqlbuild/adapter/contract/_helpers/view_definitions.py new file mode 100644 index 0000000000..c15ebc9d39 --- /dev/null +++ b/src/sqlbuild/adapter/contract/_helpers/view_definitions.py @@ -0,0 +1,25 @@ +"""Normalize stored view definitions for comparison with compatibility view SQL.""" + +from __future__ import annotations + +import re + +_CREATE_PREFIX: re.Pattern[str] = re.compile( + r"^\s*(?:create|alter)\b.*?\bview\s+\S+.*?\bas\s+", re.IGNORECASE | re.DOTALL +) +_QUOTES: re.Pattern[str] = re.compile(r"[\"`\[\]]") +_WHITESPACE: re.Pattern[str] = re.compile(r"\s+") +_PUNCTUATION_SPACE: re.Pattern[str] = re.compile(r"\s*([,()])\s*") + + +def view_definitions_match(*, definition: str, sql: str) -> bool: + """Return whether two view bodies match, ignoring quoting, case, whitespace, and CREATE.""" + + return _normalized(definition) == _normalized(sql) + + +def _normalized(text: str) -> str: + body: str = _CREATE_PREFIX.sub("", text, count=1) + body = _QUOTES.sub("", body).lower() + body = _PUNCTUATION_SPACE.sub(r"\1", body) + return _WHITESPACE.sub(" ", body).strip().rstrip(";").strip() diff --git a/src/sqlbuild/adapter/contract/classes/base_adapter.py b/src/sqlbuild/adapter/contract/classes/base_adapter.py index 87f5350bd5..e1de3f9f62 100644 --- a/src/sqlbuild/adapter/contract/classes/base_adapter.py +++ b/src/sqlbuild/adapter/contract/classes/base_adapter.py @@ -30,6 +30,7 @@ AdapterUserError, UnsupportedTypedSqlRenderingError, ) +from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, CursorValue, @@ -38,6 +39,7 @@ FunctionInfo, MigrationStagePlan, QueryResult, + RelationGrant, RelationInfo, RowDiffCoverage, RowDiffPreparedRelations, @@ -101,6 +103,8 @@ class BaseAdapter(RetentionAdapterMixin, StrictAdapter): sql_analysis_dialect_name: ClassVar[str | None] = None max_identifier_length: ClassVar[int] = 63 state_tables_transient: ClassVar[bool] = False + relation_grants_supported: ClassVar[bool] = True + views_read_with_reader_access: ClassVar[bool] = False allows_implicit_managed_write_schema: ClassVar[bool] = False execution_duration_limit_seconds: ClassVar[int | None] = None _snapshot_sql_dialect: ClassVar[SnapshotSqlDialect] = SnapshotSqlDialect( @@ -707,6 +711,93 @@ def capture_dependent_view_rebinds( del connection, database, schema, name return () + def views_bind_to_relation_identity(self) -> bool: + """Return whether views follow a renamed relation instead of re-resolving its name.""" + + return False + + def list_dependent_view_names( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> tuple[str, ...]: + """Return qualified names of views bound to this relation.""" + + del connection, database, schema, name + return () + + def read_relation_grants( + self, + *, + connection: Any, + database: str | None, + schema: str, + name: str, + relation_type: str, + ) -> tuple[RelationGrant, ...]: + """Return the privileges granted or denied on a relation and its columns.""" + + del connection, database, schema, name, relation_type + return () + + def render_relation_grants( + self, *, grants: tuple[RelationGrant, ...], destination: str, columns: tuple[str, ...] + ) -> tuple[str, ...]: + """Render grants onto ``destination``, keeping column grants only for ``columns``.""" + + del grants, destination, columns + return () + + def render_replace_view_keeping_grants( + self, *, destination: str, sql: str + ) -> tuple[str, ...] | None: + """Return statements that redefine an existing view keeping its privileges, if any.""" + + del destination, sql + return None + + def read_view_definition( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> str | None: + """Return the stored definition of a view, or None when unknown.""" + + del connection, database, schema, name + return None + + def view_definition_matches( + self, *, connection: Any, database: str | None, schema: str, name: str, sql: str + ) -> bool: + """Return whether the view's stored definition is ``sql`` as this warehouse stores it.""" + + definition: str | None = self.read_view_definition( + connection=connection, database=database, schema=schema, name=name + ) + return definition is not None and same_view_definition(definition=definition, sql=sql) + + def render_relation_revokes( + self, *, grants: tuple[RelationGrant, ...], destination: str + ) -> tuple[str, ...]: + """Render statements that remove grants from ``destination``.""" + + del grants, destination + return () + + def rename_view( + self, + *, + connection: Any, + origin: str, + destination: str, + statement_recorder: StatementRecorder, + ) -> None: + """Rename a view in place, keeping its privileges.""" + + statements: tuple[str, ...] = self.render_rename_view( + origin=origin, destination=destination + ) + statement_recorder.record_many(statements) + statement: str + for statement in statements: + self.execute(connection=connection, sql=statement) + def supports_transactional_ddl(self) -> bool: return False @@ -2160,6 +2251,23 @@ def render_create_migration_state_table_sql(self, *, database: str | None, schem transient=self.state_tables_transient, ) + def render_create_old_name_view_state_table_sql( + self, *, database: str | None, schema: str + ) -> str: + """Render DDL that creates the old-name view fact table when it is missing.""" + + from sqlbuild.compiler.migrations.main.old_name_view_create_table_sql import ( + build_old_name_view_state_create_table_sql, + ) + + return build_old_name_view_state_create_table_sql( + database=database, + schema=schema, + render_qualified_name=self.render_qualified_name, + render_framework_type=self.render_framework_type, + transient=self.state_tables_transient, + ) + def render_create_column_migration_state_table_sql( self, *, database: str | None, schema: str ) -> str: diff --git a/src/sqlbuild/adapter/contract/classes/connection.py b/src/sqlbuild/adapter/contract/classes/connection.py index 5e4a3f539b..59a8821cd2 100644 --- a/src/sqlbuild/adapter/contract/classes/connection.py +++ b/src/sqlbuild/adapter/contract/classes/connection.py @@ -82,15 +82,19 @@ def rollback(self, connection: Any) -> None: @contextlib.contextmanager def transaction(self, connection: Any) -> Generator[None]: - """Context manager for a begin/commit/rollback boundary.""" - if not self.supports_transactions(): + """Context manager for a begin/commit/rollback boundary; nested use joins the outer one.""" + open_transactions: set[int] = self.__dict__.setdefault("_open_transaction_ids", set()) + if not self.supports_transactions() or id(connection) in open_transactions: yield return self.begin(connection) + open_transactions.add(id(connection)) try: yield + open_transactions.discard(id(connection)) self.commit(connection) except BaseException: + open_transactions.discard(id(connection)) self.rollback(connection) raise diff --git a/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py b/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py index 48c8a1d5f1..7766951cba 100644 --- a/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py +++ b/src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py @@ -35,6 +35,7 @@ QUALIFIED_NAME_SEPARATOR, ) from sqlbuild.adapter.contract.exceptions import AdapterUserError +from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, CursorValue, @@ -43,6 +44,7 @@ FunctionInfo, MigrationStagePlan, QueryResult, + RelationGrant, RelationInfo, RowDiffColumnResult, RowDiffCoverage, @@ -98,6 +100,7 @@ def get_columns_for_relations( ) sql_analysis_dialect_name: ClassVar[str | None] = "duckdb" + relation_grants_supported: ClassVar[bool] = False def supports_zero_copy_clone(self) -> bool: return False @@ -866,6 +869,21 @@ def render_create_migration_state_table_sql(self, *, database: str | None, schem render_framework_type=self.render_framework_type, ) + def render_create_old_name_view_state_table_sql( + self, *, database: str | None, schema: str + ) -> str: + from sqlbuild.compiler.migrations.main.old_name_view_create_table_sql import ( + build_old_name_view_state_create_table_sql, + ) + + return build_old_name_view_state_create_table_sql( + database=database, + schema=schema, + render_qualified_name=self.render_qualified_name, + render_framework_type=self.render_framework_type, + transient=self.state_tables_transient, + ) + def render_create_column_migration_state_table_sql( self, *, database: str | None, schema: str ) -> str: @@ -1540,6 +1558,84 @@ def capture_dependent_view_rebinds( del connection, database, schema, name return () + def views_bind_to_relation_identity(self) -> bool: + return False + + def list_dependent_view_names( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> tuple[str, ...]: + del connection, database, schema, name + return () + + def read_relation_grants( + self, + *, + connection: Any, + database: str | None, + schema: str, + name: str, + relation_type: str, + ) -> tuple[RelationGrant, ...]: + del connection, database, schema, name, relation_type + return () + + def render_relation_grants( + self, *, grants: tuple[RelationGrant, ...], destination: str, columns: tuple[str, ...] + ) -> tuple[str, ...]: + del grants, destination, columns + return () + + def render_replace_view_keeping_grants( + self, *, destination: str, sql: str + ) -> tuple[str, ...] | None: + return self.render_create_view_as(destination=destination, sql=sql) + + def read_view_definition( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> str | None: + database_filter: str = ( + "" if database is None else f" AND database_name = {_duckdb_string_literal(database)}" + ) + rows: list[tuple[Any, ...]] = self.execute( + connection=connection, + sql=( + "SELECT sql FROM duckdb_views() WHERE schema_name = " + f"{_duckdb_string_literal(schema)} AND view_name = {_duckdb_string_literal(name)}" + + database_filter + ), + ).fetchall() + return None if not rows else str(rows[0][0]) + + def view_definition_matches( + self, *, connection: Any, database: str | None, schema: str, name: str, sql: str + ) -> bool: + definition: str | None = self.read_view_definition( + connection=connection, database=database, schema=schema, name=name + ) + return definition is not None and same_view_definition(definition=definition, sql=sql) + + def render_relation_revokes( + self, *, grants: tuple[RelationGrant, ...], destination: str + ) -> tuple[str, ...]: + del grants, destination + return () + + def rename_view( + self, + *, + connection: Any, + origin: str, + destination: str, + statement_recorder: StatementRecorder, + ) -> None: + statements: tuple[str, ...] = self.render_rename_view( + origin=origin, destination=destination + ) + statement_recorder.record_many(statements) + statement: str + for statement in statements: + self.execute(connection=connection, sql=statement) + def supports_transactional_ddl(self) -> bool: return True diff --git a/src/sqlbuild/adapter/contract/classes/strict_adapter.py b/src/sqlbuild/adapter/contract/classes/strict_adapter.py index 1420b3bb21..e57a17b2af 100644 --- a/src/sqlbuild/adapter/contract/classes/strict_adapter.py +++ b/src/sqlbuild/adapter/contract/classes/strict_adapter.py @@ -16,6 +16,7 @@ CursorValue, ExpressionInferenceProfile, MigrationStagePlan, + RelationGrant, RowDiffTolerance, RowDiffTolerances, SnapshotChangeTarget, @@ -351,6 +352,85 @@ def capture_dependent_view_rebinds( """Capture statements that re-point identity-bound dependent views at this table name.""" ... + @abstractmethod + def views_bind_to_relation_identity(self) -> bool: + """Return whether views follow a renamed relation instead of re-resolving its name.""" + ... + + @abstractmethod + def list_dependent_view_names( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> tuple[str, ...]: + """Return qualified names of views bound to this relation.""" + ... + + @abstractmethod + def render_create_old_name_view_state_table_sql( + self, *, database: str | None, schema: str + ) -> str: + """Render DDL that creates the old-name view fact table when it is missing.""" + ... + + @abstractmethod + def read_relation_grants( + self, + *, + connection: Any, + database: str | None, + schema: str, + name: str, + relation_type: str, + ) -> tuple[RelationGrant, ...]: + """Return the privileges granted or denied on a relation and its columns.""" + ... + + @abstractmethod + def render_relation_grants( + self, *, grants: tuple[RelationGrant, ...], destination: str, columns: tuple[str, ...] + ) -> tuple[str, ...]: + """Render grants onto ``destination``, keeping column grants only for ``columns``.""" + ... + + @abstractmethod + def render_replace_view_keeping_grants( + self, *, destination: str, sql: str + ) -> tuple[str, ...] | None: + """Return statements that redefine an existing view keeping its privileges, if any.""" + ... + + @abstractmethod + def read_view_definition( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> str | None: + """Return the stored definition of a view, or None when unknown.""" + ... + + @abstractmethod + def view_definition_matches( + self, *, connection: Any, database: str | None, schema: str, name: str, sql: str + ) -> bool: + """Return whether the view's stored definition is ``sql`` as this warehouse stores it.""" + ... + + @abstractmethod + def render_relation_revokes( + self, *, grants: tuple[RelationGrant, ...], destination: str + ) -> tuple[str, ...]: + """Render statements that remove grants from ``destination``.""" + ... + + @abstractmethod + def rename_view( + self, + *, + connection: Any, + origin: str, + destination: str, + statement_recorder: StatementRecorder, + ) -> None: + """Rename a view in place, keeping its privileges.""" + ... + @abstractmethod def supports_transactional_ddl(self) -> bool: """Return whether renames and state inserts can commit or roll back together.""" diff --git a/src/sqlbuild/adapter/contract/main/grants_for_columns.py b/src/sqlbuild/adapter/contract/main/grants_for_columns.py new file mode 100644 index 0000000000..06b45950a7 --- /dev/null +++ b/src/sqlbuild/adapter/contract/main/grants_for_columns.py @@ -0,0 +1,21 @@ +"""Order relation grants for replay and drop column grants a view does not expose.""" + +from __future__ import annotations + +from sqlbuild.adapter.contract.models import RelationGrant + + +def grants_for_columns( + *, grants: tuple[RelationGrant, ...], columns: tuple[str, ...] +) -> tuple[RelationGrant, ...]: + """Return relation-level grants, then column grants on exposed columns only.""" + + exposed: frozenset[str] = frozenset(column.lower() for column in columns) + return ( + *(grant for grant in grants if grant.column is None), + *( + grant + for grant in grants + if grant.column is not None and grant.column.lower() in exposed + ), + ) diff --git a/src/sqlbuild/adapter/contract/main/same_view_definition.py b/src/sqlbuild/adapter/contract/main/same_view_definition.py new file mode 100644 index 0000000000..fcf3844e4b --- /dev/null +++ b/src/sqlbuild/adapter/contract/main/same_view_definition.py @@ -0,0 +1,11 @@ +"""Compare a stored view definition with the SELECT SQLBuild would create it from.""" + +from __future__ import annotations + +from sqlbuild.adapter.contract._helpers.view_definitions import view_definitions_match + + +def same_view_definition(*, definition: str, sql: str) -> bool: + """Return whether two view bodies match, ignoring quoting, case, whitespace, and CREATE.""" + + return view_definitions_match(definition=definition, sql=sql) diff --git a/src/sqlbuild/adapter/contract/models.py b/src/sqlbuild/adapter/contract/models.py index 72cc84fa63..15dbe08a7b 100644 --- a/src/sqlbuild/adapter/contract/models.py +++ b/src/sqlbuild/adapter/contract/models.py @@ -144,6 +144,18 @@ class TableFreshnessMetadata: observed_at: datetime | None = None +@dataclass(frozen=True) +class RelationGrant: + """One privilege granted or denied on a relation, or on one of its columns.""" + + privilege: str + grantee: str | None + column: str | None = None + grantable: bool = False + denied: bool = False + grantee_kind: str = "" + + @dataclass(frozen=True) class TableFreshnessRequest: """Physical table identity for adapter metadata freshness lookup.""" diff --git a/src/sqlbuild/adapters/bigquery/_helpers/grants.py b/src/sqlbuild/adapters/bigquery/_helpers/grants.py new file mode 100644 index 0000000000..3693c46c74 --- /dev/null +++ b/src/sqlbuild/adapters/bigquery/_helpers/grants.py @@ -0,0 +1,51 @@ +"""Read and replay BigQuery table IAM bindings onto a compatibility view.""" + +from __future__ import annotations + +from typing import Any + +from sqlbuild.adapter.contract.models import RelationGrant + +_ROLE_COLUMN: int = 0 +_GRANTEE_COLUMN: int = 1 + + +def bigquery_relation_grants(rows: list[tuple[Any, ...]]) -> tuple[RelationGrant, ...]: + """Decode OBJECT_PRIVILEGES role bindings.""" + + return tuple( + RelationGrant(privilege=str(row[_ROLE_COLUMN]), grantee=str(row[_GRANTEE_COLUMN])) + for row in rows + ) + + +def render_bigquery_view_grants( + *, grants: tuple[RelationGrant, ...], destination: str +) -> tuple[str, ...]: + """Render role bindings as DCL grants on ``destination``.""" + + return tuple( + f"GRANT `{grant.privilege.replace('`', '')}` ON VIEW {destination} " + f'TO "{(grant.grantee or "").replace(chr(34), "")}"' + for grant in grants + ) + + +def render_bigquery_view_revokes( + *, grants: tuple[RelationGrant, ...], destination: str +) -> tuple[str, ...]: + """Render role bindings to remove from ``destination``.""" + + return tuple( + f"REVOKE `{grant.privilege.replace('`', '')}` ON VIEW {destination} " + f'FROM "{(grant.grantee or "").replace(chr(34), "")}"' + for grant in grants + ) + + +def render_bigquery_view_move( + *, origin: str, destination: str, definition: str, grants: tuple[str, ...] +) -> tuple[str, ...]: + """Re-create a view under a new name with its grants, since BigQuery cannot rename views.""" + + return (f"CREATE VIEW {destination} AS {definition}", *grants, f"DROP VIEW {origin}") diff --git a/src/sqlbuild/adapters/bigquery/classes/bigquery_adapter.py b/src/sqlbuild/adapters/bigquery/classes/bigquery_adapter.py index 10e0284ffc..846a84416e 100644 --- a/src/sqlbuild/adapters/bigquery/classes/bigquery_adapter.py +++ b/src/sqlbuild/adapters/bigquery/classes/bigquery_adapter.py @@ -40,6 +40,7 @@ AdapterUserError, UnsupportedTypedSqlRenderingError, ) +from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, CursorValue, @@ -48,6 +49,7 @@ FunctionInfo, MigrationStagePlan, QueryResult, + RelationGrant, RelationInfo, RenderedRetentionChange, RetentionRequest, @@ -97,6 +99,12 @@ from sqlbuild.adapter.type_system.main.normalize_numeric_family import normalize_numeric_family from sqlbuild.adapter.type_system.main.types_equal import types_equal from sqlbuild.adapters.bigquery._helpers.clone_refusal import is_bigquery_clone_refusal +from sqlbuild.adapters.bigquery._helpers.grants import ( + bigquery_relation_grants, + render_bigquery_view_grants, + render_bigquery_view_move, + render_bigquery_view_revokes, +) from sqlbuild.adapters.bigquery._helpers.statement_telemetry import affected_rows from sqlbuild.adapters.bigquery.classes.bigquery_connection import _BigQueryConnection from sqlbuild.adapters.bigquery.classes.bigquery_cursor import _BigQueryCursor @@ -141,6 +149,7 @@ def get_columns_for_relations( adapter_name: ClassVar[str] = BuiltinAdapter.BIGQUERY.value sql_analysis_dialect_name: ClassVar[str | None] = "bigquery" + views_read_with_reader_access: ClassVar[bool] = True max_identifier_length: ClassVar[int] = 1024 _snapshot_sql_dialect: ClassVar[SnapshotSqlDialect] = SnapshotSqlDialect( timestamp_type="TIMESTAMP", @@ -337,6 +346,21 @@ def render_create_migration_state_table_sql(self, *, database: str | None, schem render_framework_type=self.render_framework_type, ) + def render_create_old_name_view_state_table_sql( + self, *, database: str | None, schema: str + ) -> str: + from sqlbuild.compiler.migrations.main.old_name_view_create_table_sql import ( + build_old_name_view_state_create_table_sql, + ) + + return build_old_name_view_state_create_table_sql( + database=database, + schema=schema, + render_qualified_name=self.render_qualified_name, + render_framework_type=self.render_framework_type, + transient=self.state_tables_transient, + ) + def render_create_column_migration_state_table_sql( self, *, database: str | None, schema: str ) -> str: @@ -1802,6 +1826,126 @@ def capture_dependent_view_rebinds( del connection, database, schema, name return () + def views_bind_to_relation_identity(self) -> bool: + return False + + def list_dependent_view_names( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> tuple[str, ...]: + del connection, database, schema, name + return () + + def read_relation_grants( + self, + *, + connection: Any, + database: str | None, + schema: str, + name: str, + relation_type: str, + ) -> tuple[RelationGrant, ...]: + del relation_type + location: str = self._metadata_location( + connection=connection, + request=TableFreshnessRequest(database=database, schema=schema, name=name), + ) + region: str = location.strip().lower() + region = region if region.startswith("region-") else f"region-{region}" + project: str = self._strip_identifier_quotes(database or str(connection.client.project)) + rows: list[tuple[Any, ...]] = self.execute( + connection=connection, + sql=( + f"SELECT privilege_type, grantee FROM `{project}.{region}`" + ".INFORMATION_SCHEMA.OBJECT_PRIVILEGES " + f"WHERE object_schema = '{self._escape_sql_string(schema)}' " + f"AND object_name = '{self._escape_sql_string(name)}' " + "ORDER BY grantee, privilege_type" + ), + ).fetchall() + return bigquery_relation_grants(rows) + + def render_relation_grants( + self, *, grants: tuple[RelationGrant, ...], destination: str, columns: tuple[str, ...] + ) -> tuple[str, ...]: + del columns + return render_bigquery_view_grants( + grants=grants, destination=self._quote_identifier_path(destination) + ) + + def render_replace_view_keeping_grants( + self, *, destination: str, sql: str + ) -> tuple[str, ...] | None: + del destination, sql + return None + + def read_view_definition( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> str | None: + dataset: str = self._build_dataset_id(database=database, schema=schema) + view_name: str = self._escape_sql_string(self._strip_identifier_quotes(name)) + rows: list[tuple[Any, ...]] = self.execute( + connection=connection, + sql=( + f"SELECT view_definition FROM `{dataset}`.INFORMATION_SCHEMA.VIEWS " + f"WHERE table_name = '{view_name}'" + ), + ).fetchall() + return None if not rows else str(rows[0][0]) + + def view_definition_matches( + self, *, connection: Any, database: str | None, schema: str, name: str, sql: str + ) -> bool: + definition: str | None = self.read_view_definition( + connection=connection, database=database, schema=schema, name=name + ) + return definition is not None and same_view_definition(definition=definition, sql=sql) + + def render_relation_revokes( + self, *, grants: tuple[RelationGrant, ...], destination: str + ) -> tuple[str, ...]: + return render_bigquery_view_revokes( + grants=grants, destination=self._quote_identifier_path(destination) + ) + + def rename_view( + self, + *, + connection: Any, + origin: str, + destination: str, + statement_recorder: StatementRecorder, + ) -> None: + parts: list[str] = self._strip_identifier_quotes(origin).split(".") + dataset: str = ".".join(parts[:-1]) + rows: list[tuple[Any, ...]] = self.execute( + connection=connection, + sql=( + f"SELECT view_definition FROM `{dataset}`.INFORMATION_SCHEMA.VIEWS " + f"WHERE table_name = '{self._escape_sql_string(parts[-1])}'" + ), + ).fetchall() + if not rows: + raise AdapterUserError(message=f"BigQuery view {origin} does not exist") + view_grants: tuple[RelationGrant, ...] = self.read_relation_grants( + connection=connection, + database=".".join(parts[:-2]) or None, + schema=parts[-2], + name=parts[-1], + relation_type="view", + ) + statements: tuple[str, ...] = render_bigquery_view_move( + origin=self._quote_identifier_path(origin), + destination=self._quote_identifier_path(destination), + definition=str(rows[0][0]), + grants=self.render_relation_grants( + grants=view_grants, destination=destination, columns=() + ), + ) + statement_recorder.record_many(statements) + statement: str + for statement in statements: + self.execute(connection=connection, sql=statement) + def supports_transactional_ddl(self) -> bool: return False diff --git a/src/sqlbuild/adapters/databricks/_helpers/__init__.py b/src/sqlbuild/adapters/databricks/_helpers/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/src/sqlbuild/adapters/databricks/_helpers/grants.py b/src/sqlbuild/adapters/databricks/_helpers/grants.py new file mode 100644 index 0000000000..8debcd1cfa --- /dev/null +++ b/src/sqlbuild/adapters/databricks/_helpers/grants.py @@ -0,0 +1,63 @@ +"""Read and replay Unity Catalog grants onto a compatibility view.""" + +from __future__ import annotations + +from typing import Any + +from sqlbuild.adapter.contract.models import RelationGrant +from sqlbuild.adapter.contract.types import RelationType +from sqlbuild.adapter.type_system.main.normalize_relation_type import normalize_relation_type + +_VIEW_PRIVILEGES: frozenset[str] = frozenset({"SELECT", "APPLY TAG", "ALL PRIVILEGES", "MANAGE"}) +_OBJECT_TYPES: frozenset[str] = frozenset({"TABLE", "VIEW"}) +_PRINCIPAL_COLUMN: int = 0 +_ACTION_COLUMN: int = 1 +_OBJECT_TYPE_COLUMN: int = 2 +_OBJECT_KEYWORDS: dict[RelationType, str] = {RelationType.VIEW: "VIEW"} + + +def databricks_relation_grants(rows: list[tuple[Any, ...]]) -> tuple[RelationGrant, ...]: + """Keep privileges granted on the relation itself that also apply to a view.""" + + return tuple( + RelationGrant( + privilege=str(row[_ACTION_COLUMN]).upper(), grantee=str(row[_PRINCIPAL_COLUMN]) + ) + for row in rows + if str(row[_ACTION_COLUMN]).upper() in _VIEW_PRIVILEGES + and str(row[_OBJECT_TYPE_COLUMN]).upper() in _OBJECT_TYPES + ) + + +def render_databricks_view_grants( + *, grants: tuple[RelationGrant, ...], destination: str +) -> tuple[str, ...]: + """Render Unity Catalog grants on ``destination``.""" + + return tuple( + f"GRANT {grant.privilege} ON VIEW {destination} TO " + + "`" + + (grant.grantee or "").replace("`", "``") + + "`" + for grant in grants + ) + + +def render_databricks_view_revokes( + *, grants: tuple[RelationGrant, ...], destination: str +) -> tuple[str, ...]: + """Render Unity Catalog grants to remove from ``destination``.""" + + return tuple( + f"REVOKE {grant.privilege} ON VIEW {destination} FROM " + + "`" + + (grant.grantee or "").replace("`", "``") + + "`" + for grant in grants + ) + + +def show_grants_object_kind(relation_type: str) -> str: + """Return the SHOW GRANTS object keyword for a listed relation type.""" + + return _OBJECT_KEYWORDS.get(normalize_relation_type(relation_type), "TABLE") diff --git a/src/sqlbuild/adapters/databricks/classes/databricks_adapter.py b/src/sqlbuild/adapters/databricks/classes/databricks_adapter.py index 0ff80f36d6..d3ef4c12eb 100644 --- a/src/sqlbuild/adapters/databricks/classes/databricks_adapter.py +++ b/src/sqlbuild/adapters/databricks/classes/databricks_adapter.py @@ -38,6 +38,7 @@ ) from sqlbuild.adapter.contract.exceptions import AdapterUserError from sqlbuild.adapter.contract.main.normalize_seed_csv_value import normalize_seed_csv_value +from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, CursorValue, @@ -46,6 +47,7 @@ FunctionInfo, MigrationStagePlan, QueryResult, + RelationGrant, RelationInfo, RenderedRetentionChange, RetentionRequest, @@ -95,6 +97,12 @@ from sqlbuild.adapter.type_system.main.first_arg_nullability import first_arg_nullability from sqlbuild.adapter.type_system.main.normalize_numeric_family import normalize_numeric_family from sqlbuild.adapter.type_system.main.types_equal import types_equal +from sqlbuild.adapters.databricks._helpers.grants import ( + databricks_relation_grants, + render_databricks_view_grants, + render_databricks_view_revokes, + show_grants_object_kind, +) from sqlbuild.adapters.databricks.classes.databricks_connection import _DatabricksConnection from sqlbuild.adapters.databricks.constants import ( DELTA_COLUMN_MAPPING_NAME_MODE, @@ -432,6 +440,21 @@ def render_create_migration_state_table_sql(self, *, database: str | None, schem render_framework_type=self.render_framework_type, ) + def render_create_old_name_view_state_table_sql( + self, *, database: str | None, schema: str + ) -> str: + from sqlbuild.compiler.migrations.main.old_name_view_create_table_sql import ( + build_old_name_view_state_create_table_sql, + ) + + return build_old_name_view_state_create_table_sql( + database=database, + schema=schema, + render_qualified_name=self.render_qualified_name, + render_framework_type=self.render_framework_type, + transient=self.state_tables_transient, + ) + def render_create_column_migration_state_table_sql( self, *, database: str | None, schema: str ) -> str: @@ -1777,6 +1800,97 @@ def capture_dependent_view_rebinds( del connection, database, schema, name return () + def views_bind_to_relation_identity(self) -> bool: + return False + + def list_dependent_view_names( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> tuple[str, ...]: + del connection, database, schema, name + return () + + def read_relation_grants( + self, + *, + connection: Any, + database: str | None, + schema: str, + name: str, + relation_type: str, + ) -> tuple[RelationGrant, ...]: + relation: str | None = self.render_qualified_name( + database=database, schema=schema, name=name + ) + kind: str = show_grants_object_kind(relation_type) + rows: list[tuple[Any, ...]] = self.execute( + connection=connection, sql=f"SHOW GRANTS ON {kind} {relation}" + ).fetchall() + return databricks_relation_grants(rows) + + def render_relation_grants( + self, *, grants: tuple[RelationGrant, ...], destination: str, columns: tuple[str, ...] + ) -> tuple[str, ...]: + del columns + return render_databricks_view_grants(grants=grants, destination=destination) + + def render_replace_view_keeping_grants( + self, *, destination: str, sql: str + ) -> tuple[str, ...] | None: + return (f"ALTER VIEW {destination} AS {sql}",) + + def read_view_definition( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> str | None: + views: str = ( + "information_schema.views" + if database is None + else (f"{database}.information_schema.views") + ) + rows: list[tuple[Any, ...]] = self.execute( + connection=connection, + sql=( + f"SELECT view_definition FROM {views} WHERE lower(table_schema) = lower(" + + "'" + + schema.replace("'", "''") + + "'" + + ") AND lower(table_name) = lower(" + + "'" + + name.replace("'", "''") + + "'" + + ")" + ), + ).fetchall() + return None if not rows else str(rows[0][0]) + + def view_definition_matches( + self, *, connection: Any, database: str | None, schema: str, name: str, sql: str + ) -> bool: + definition: str | None = self.read_view_definition( + connection=connection, database=database, schema=schema, name=name + ) + return definition is not None and same_view_definition(definition=definition, sql=sql) + + def render_relation_revokes( + self, *, grants: tuple[RelationGrant, ...], destination: str + ) -> tuple[str, ...]: + return render_databricks_view_revokes(grants=grants, destination=destination) + + def rename_view( + self, + *, + connection: Any, + origin: str, + destination: str, + statement_recorder: StatementRecorder, + ) -> None: + statements: tuple[str, ...] = self.render_rename_view( + origin=origin, destination=destination + ) + statement_recorder.record_many(statements) + statement: str + for statement in statements: + self.execute(connection=connection, sql=statement) + def supports_transactional_ddl(self) -> bool: return False diff --git a/src/sqlbuild/adapters/postgres/_helpers/grants.py b/src/sqlbuild/adapters/postgres/_helpers/grants.py new file mode 100644 index 0000000000..68db304147 --- /dev/null +++ b/src/sqlbuild/adapters/postgres/_helpers/grants.py @@ -0,0 +1,71 @@ +"""Read and replay PostgreSQL privileges onto a compatibility view.""" + +from __future__ import annotations + +from collections.abc import Callable +from typing import Any + +from sqlbuild.adapter.contract.main.grants_for_columns import grants_for_columns +from sqlbuild.adapter.contract.models import RelationGrant + + +def postgres_relation_grants( + *, relation_rows: list[tuple[Any, ...]], column_rows: list[tuple[Any, ...]] +) -> tuple[RelationGrant, ...]: + """Decode aclexplode rows of a relation and of its columns.""" + + return ( + *( + RelationGrant( + privilege=str(row[0]), + grantee=None if row[1] is None else str(row[1]), + grantable=bool(row[2]), + ) + for row in relation_rows + ), + *( + RelationGrant( + privilege=str(row[1]), + grantee=None if row[2] is None else str(row[2]), + column=str(row[0]), + grantable=bool(row[3]), + ) + for row in column_rows + ), + ) + + +def render_postgres_grants( + *, + grants: tuple[RelationGrant, ...], + destination: str, + columns: tuple[str, ...], + render_identifier: Callable[[str], str], +) -> tuple[str, ...]: + """Render table and exposed column privileges as grants on ``destination``.""" + + return tuple( + f"GRANT {grant.privilege}" + + ("" if grant.column is None else f" ({render_identifier(grant.column)})") + + f" ON {destination} TO " + + ("PUBLIC" if grant.grantee is None else render_identifier(grant.grantee)) + + (" WITH GRANT OPTION" if grant.grantable else "") + for grant in grants_for_columns(grants=grants, columns=columns) + ) + + +def render_postgres_revokes( + *, + grants: tuple[RelationGrant, ...], + destination: str, + render_identifier: Callable[[str], str], +) -> tuple[str, ...]: + """Render grants to remove from ``destination``, grant option included.""" + + return tuple( + f"REVOKE {grant.privilege}" + + ("" if grant.column is None else f" ({render_identifier(grant.column)})") + + f" ON {destination} FROM " + + ("PUBLIC" if grant.grantee is None else render_identifier(grant.grantee)) + for grant in grants + ) diff --git a/src/sqlbuild/adapters/postgres/classes/postgres_adapter.py b/src/sqlbuild/adapters/postgres/classes/postgres_adapter.py index c0b5621393..bcc08cc0a0 100644 --- a/src/sqlbuild/adapters/postgres/classes/postgres_adapter.py +++ b/src/sqlbuild/adapters/postgres/classes/postgres_adapter.py @@ -40,6 +40,7 @@ ) from sqlbuild.adapter.contract.exceptions import AdapterUserError from sqlbuild.adapter.contract.main.normalize_seed_csv_value import normalize_seed_csv_value +from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, CursorValue, @@ -48,6 +49,7 @@ FunctionInfo, MigrationStagePlan, QueryResult, + RelationGrant, RelationInfo, RowDiffColumnResult, RowDiffCoverage, @@ -86,6 +88,11 @@ ) from sqlbuild.adapter.type_system.main.normalize_numeric_family import normalize_numeric_family from sqlbuild.adapter.type_system.main.types_equal import types_equal +from sqlbuild.adapters.postgres._helpers.grants import ( + postgres_relation_grants, + render_postgres_grants, + render_postgres_revokes, +) from sqlbuild.adapters.postgres._helpers.view_rebind import render_postgres_view_rebind from sqlbuild.adapters.postgres.classes.postgres_connection import _PostgresConnection from sqlbuild.adapters.postgres.constants import TABLE_FUNCTION_RETURN_TYPE @@ -611,6 +618,142 @@ def capture_dependent_view_rebinds( for row in cursor.fetchall() ) + def views_bind_to_relation_identity(self) -> bool: + return True + + def list_dependent_view_names( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> tuple[str, ...]: + del database + query: str = ( + "SELECT DISTINCT dependent_namespace.nspname, dependent.relname " + "FROM pg_depend AS dependency " + "JOIN pg_rewrite AS rewrite ON rewrite.oid = dependency.objid " + "JOIN pg_class AS dependent ON dependent.oid = rewrite.ev_class " + "JOIN pg_namespace AS dependent_namespace " + "ON dependent_namespace.oid = dependent.relnamespace " + "JOIN pg_class AS referenced ON referenced.oid = dependency.refobjid " + "JOIN pg_namespace AS referenced_namespace " + "ON referenced_namespace.oid = referenced.relnamespace " + "WHERE dependency.classid = 'pg_rewrite'::regclass " + "AND dependency.refclassid = 'pg_class'::regclass " + "AND dependent.relkind IN ('v', 'm') " + "AND dependent.oid <> referenced.oid " + f"AND referenced_namespace.nspname = {_quote_sql_string(schema)} " + f"AND referenced.relname = {_quote_sql_string(name)} " + "ORDER BY 1, 2" + ) + cursor: Any = connection.execute(query) + return tuple(f"{row[0]}.{row[1]}" for row in cursor.fetchall()) + + def read_relation_grants( + self, + *, + connection: Any, + database: str | None, + schema: str, + name: str, + relation_type: str, + ) -> tuple[RelationGrant, ...]: + del database, relation_type + relation_filter: str = ( + "FROM pg_class AS relation " + "JOIN pg_namespace AS namespace ON namespace.oid = relation.relnamespace " + ) + where: str = ( + f"WHERE namespace.nspname = {_quote_sql_string(schema)} " + f"AND relation.relname = {_quote_sql_string(name)} " + "AND acl.grantee <> relation.relowner " + ) + grantee: str = "CASE WHEN acl.grantee = 0 THEN NULL ELSE pg_get_userbyid(acl.grantee) END" + relation_rows: list[tuple[Any, ...]] = connection.execute( + f"SELECT acl.privilege_type, {grantee}, acl.is_grantable " + + relation_filter + + "CROSS JOIN LATERAL aclexplode(relation.relacl) AS acl " + + where + + "ORDER BY 2 NULLS FIRST, 1" + ).fetchall() + column_rows: list[tuple[Any, ...]] = connection.execute( + f"SELECT attribute.attname, acl.privilege_type, {grantee}, acl.is_grantable " + + relation_filter + + "JOIN pg_attribute AS attribute ON attribute.attrelid = relation.oid " + "AND attribute.attnum > 0 AND NOT attribute.attisdropped " + "CROSS JOIN LATERAL aclexplode(attribute.attacl) AS acl " + + where + + "ORDER BY attribute.attnum, 3 NULLS FIRST, 2" + ).fetchall() + return postgres_relation_grants(relation_rows=relation_rows, column_rows=column_rows) + + def render_relation_grants( + self, *, grants: tuple[RelationGrant, ...], destination: str, columns: tuple[str, ...] + ) -> tuple[str, ...]: + return render_postgres_grants( + grants=grants, + destination=destination, + columns=columns, + render_identifier=self.render_identifier, + ) + + def render_replace_view_keeping_grants( + self, *, destination: str, sql: str + ) -> tuple[str, ...] | None: + return self.render_create_view_as(destination=destination, sql=sql) + + def read_view_definition( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> str | None: + del database + rows: list[tuple[Any, ...]] = connection.execute( + "SELECT pg_get_viewdef(relation.oid) FROM pg_class AS relation " + "JOIN pg_namespace AS namespace ON namespace.oid = relation.relnamespace " + f"WHERE relation.relkind = 'v' AND namespace.nspname = {_quote_sql_string(schema)} " + f"AND relation.relname = {_quote_sql_string(name)}" + ).fetchall() + return None if not rows else str(rows[0][0]) + + def view_definition_matches( + self, *, connection: Any, database: str | None, schema: str, name: str, sql: str + ) -> bool: + definition: str | None = self.read_view_definition( + connection=connection, database=database, schema=schema, name=name + ) + if definition is None: + return False + connection.execute("BEGIN") + try: + connection.execute(f"CREATE TEMPORARY VIEW _sqb_view_probe AS {sql}") + probe: str = str( + connection.execute("SELECT pg_get_viewdef('_sqb_view_probe'::regclass)").fetchone()[ + 0 + ] + ) + finally: + connection.execute("ROLLBACK") + return same_view_definition(definition=definition, sql=probe) + + def render_relation_revokes( + self, *, grants: tuple[RelationGrant, ...], destination: str + ) -> tuple[str, ...]: + return render_postgres_revokes( + grants=grants, destination=destination, render_identifier=self.render_identifier + ) + + def rename_view( + self, + *, + connection: Any, + origin: str, + destination: str, + statement_recorder: StatementRecorder, + ) -> None: + statements: tuple[str, ...] = self.render_rename_view( + origin=origin, destination=destination + ) + statement_recorder.record_many(statements) + statement: str + for statement in statements: + self.execute(connection=connection, sql=statement) + def supports_transactional_ddl(self) -> bool: return True @@ -1544,6 +1687,21 @@ def render_create_migration_state_table_sql(self, *, database: str | None, schem render_framework_type=self.render_framework_type, ) + def render_create_old_name_view_state_table_sql( + self, *, database: str | None, schema: str + ) -> str: + from sqlbuild.compiler.migrations.main.old_name_view_create_table_sql import ( + build_old_name_view_state_create_table_sql, + ) + + return build_old_name_view_state_create_table_sql( + database=database, + schema=schema, + render_qualified_name=self.render_qualified_name, + render_framework_type=self.render_framework_type, + transient=self.state_tables_transient, + ) + def render_create_column_migration_state_table_sql( self, *, database: str | None, schema: str ) -> str: diff --git a/src/sqlbuild/adapters/snowflake/_helpers/__init__.py b/src/sqlbuild/adapters/snowflake/_helpers/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/src/sqlbuild/adapters/snowflake/_helpers/grants.py b/src/sqlbuild/adapters/snowflake/_helpers/grants.py new file mode 100644 index 0000000000..02eb4a58c0 --- /dev/null +++ b/src/sqlbuild/adapters/snowflake/_helpers/grants.py @@ -0,0 +1,74 @@ +"""Read and replay Snowflake grants onto a compatibility view.""" + +from __future__ import annotations + +from typing import Any + +from sqlbuild.adapter.contract.models import RelationGrant +from sqlbuild.adapter.contract.types import RelationType +from sqlbuild.adapter.type_system.main.normalize_relation_type import normalize_relation_type + +_VIEW_PRIVILEGES: frozenset[str] = frozenset({"SELECT", "REFERENCES"}) +_GRANTEE_KEYWORDS: dict[str, str] = {"ROLE": "ROLE", "DATABASE_ROLE": "DATABASE ROLE"} +_PRIVILEGE_COLUMN: int = 1 +_GRANTED_TO_COLUMN: int = 4 +_GRANTEE_COLUMN: int = 5 +_GRANT_OPTION_COLUMN: int = 6 +_GRANT_OPTIONS: dict[str, bool] = {"true": True} +_OBJECT_KEYWORDS: dict[RelationType, str] = {RelationType.VIEW: "VIEW"} + + +def snowflake_relation_grants(rows: list[tuple[Any, ...]]) -> tuple[RelationGrant, ...]: + """Keep SHOW GRANTS rows that apply to a view, skipping OWNERSHIP and shares.""" + + grants: list[RelationGrant] = [] + row: tuple[Any, ...] + for row in rows: + privilege: str = str(row[_PRIVILEGE_COLUMN]).upper() + keyword: str | None = _GRANTEE_KEYWORDS.get(str(row[_GRANTED_TO_COLUMN]).upper()) + if privilege not in _VIEW_PRIVILEGES or keyword is None: + continue + grants.append( + RelationGrant( + privilege=privilege, + grantee=str(row[_GRANTEE_COLUMN]), + grantable=_GRANT_OPTIONS.get(str(row[_GRANT_OPTION_COLUMN]).lower(), False), + grantee_kind=keyword, + ) + ) + return tuple(grants) + + +def render_snowflake_view_grants( + *, grants: tuple[RelationGrant, ...], destination: str +) -> tuple[str, ...]: + """Render role and database-role grants on ``destination``.""" + + return tuple( + f"GRANT {grant.privilege} ON VIEW {destination} TO {grant.grantee_kind} " + + _quoted_grantee(grant.grantee or "") + + (" WITH GRANT OPTION" if grant.grantable else "") + for grant in grants + ) + + +def render_snowflake_view_revokes( + *, grants: tuple[RelationGrant, ...], destination: str +) -> tuple[str, ...]: + """Render role and database-role grants to remove from ``destination``.""" + + return tuple( + f"REVOKE {grant.privilege} ON VIEW {destination} FROM {grant.grantee_kind} " + + _quoted_grantee(grant.grantee or "") + for grant in grants + ) + + +def _quoted_grantee(grantee: str) -> str: + return ".".join('"' + part.replace('"', '""') + '"' for part in grantee.split(".")) + + +def show_grants_object_kind(relation_type: str) -> str: + """Return the SHOW GRANTS object keyword for a listed relation type.""" + + return _OBJECT_KEYWORDS.get(normalize_relation_type(relation_type), "TABLE") diff --git a/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py b/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py index f89fdb88ca..7d1f9964d0 100644 --- a/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py +++ b/src/sqlbuild/adapters/snowflake/classes/snowflake_adapter.py @@ -36,6 +36,7 @@ ) from sqlbuild.adapter.contract.exceptions import AdapterUserError from sqlbuild.adapter.contract.main.normalize_seed_csv_value import normalize_seed_csv_value +from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, ExpressionInferenceProfile, @@ -43,6 +44,7 @@ FunctionInfo, MigrationStagePlan, QueryResult, + RelationGrant, RelationInfo, RenderedRetentionChange, RetentionRequest, @@ -88,6 +90,12 @@ from sqlbuild.adapter.type_system.main.first_arg_nullability import first_arg_nullability from sqlbuild.adapter.type_system.main.normalize_numeric_family import normalize_numeric_family from sqlbuild.adapter.type_system.main.types_equal import types_equal +from sqlbuild.adapters.snowflake._helpers.grants import ( + render_snowflake_view_grants, + render_snowflake_view_revokes, + show_grants_object_kind, + snowflake_relation_grants, +) from sqlbuild.adapters.snowflake.classes.snowflake_connection import _SnowflakeConnection from sqlbuild.adapters.snowflake.constants import ( BASE_TABLE_METADATA_TYPE, @@ -513,6 +521,21 @@ def render_create_migration_state_table_sql(self, *, database: str | None, schem transient=self.state_tables_transient, ) + def render_create_old_name_view_state_table_sql( + self, *, database: str | None, schema: str + ) -> str: + from sqlbuild.compiler.migrations.main.old_name_view_create_table_sql import ( + build_old_name_view_state_create_table_sql, + ) + + return build_old_name_view_state_create_table_sql( + database=database, + schema=schema, + render_qualified_name=self.render_qualified_name, + render_framework_type=self.render_framework_type, + transient=self.state_tables_transient, + ) + def render_create_column_migration_state_table_sql( self, *, database: str | None, schema: str ) -> str: @@ -2307,6 +2330,97 @@ def capture_dependent_view_rebinds( del connection, database, schema, name return () + def views_bind_to_relation_identity(self) -> bool: + return False + + def list_dependent_view_names( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> tuple[str, ...]: + del connection, database, schema, name + return () + + def read_relation_grants( + self, + *, + connection: Any, + database: str | None, + schema: str, + name: str, + relation_type: str, + ) -> tuple[RelationGrant, ...]: + relation: str | None = self.render_qualified_name( + database=database, schema=schema, name=name + ) + kind: str = show_grants_object_kind(relation_type) + rows: list[tuple[Any, ...]] = self.execute( + connection=connection, sql=f"SHOW GRANTS ON {kind} {relation}" + ).fetchall() + return snowflake_relation_grants(rows) + + def render_relation_grants( + self, *, grants: tuple[RelationGrant, ...], destination: str, columns: tuple[str, ...] + ) -> tuple[str, ...]: + del columns + return render_snowflake_view_grants(grants=grants, destination=destination) + + def render_replace_view_keeping_grants( + self, *, destination: str, sql: str + ) -> tuple[str, ...] | None: + return (f"CREATE OR REPLACE VIEW {destination} COPY GRANTS AS {sql}",) + + def read_view_definition( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> str | None: + views: str = ( + "INFORMATION_SCHEMA.VIEWS" + if database is None + else (f"{database}.INFORMATION_SCHEMA.VIEWS") + ) + rows: list[tuple[Any, ...]] = self.execute( + connection=connection, + sql=( + f"SELECT VIEW_DEFINITION FROM {views} WHERE UPPER(TABLE_SCHEMA) = UPPER(" + + "'" + + schema.replace("'", "''") + + "'" + + ") AND UPPER(TABLE_NAME) = UPPER(" + + "'" + + name.replace("'", "''") + + "'" + + ")" + ), + ).fetchall() + return None if not rows else str(rows[0][0]) + + def view_definition_matches( + self, *, connection: Any, database: str | None, schema: str, name: str, sql: str + ) -> bool: + definition: str | None = self.read_view_definition( + connection=connection, database=database, schema=schema, name=name + ) + return definition is not None and same_view_definition(definition=definition, sql=sql) + + def render_relation_revokes( + self, *, grants: tuple[RelationGrant, ...], destination: str + ) -> tuple[str, ...]: + return render_snowflake_view_revokes(grants=grants, destination=destination) + + def rename_view( + self, + *, + connection: Any, + origin: str, + destination: str, + statement_recorder: StatementRecorder, + ) -> None: + statements: tuple[str, ...] = self.render_rename_view( + origin=origin, destination=destination + ) + statement_recorder.record_many(statements) + statement: str + for statement in statements: + self.execute(connection=connection, sql=statement) + def supports_transactional_ddl(self) -> bool: return False diff --git a/src/sqlbuild/adapters/sqlserver/_helpers/__init__.py b/src/sqlbuild/adapters/sqlserver/_helpers/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/src/sqlbuild/adapters/sqlserver/_helpers/grants.py b/src/sqlbuild/adapters/sqlserver/_helpers/grants.py new file mode 100644 index 0000000000..fb7e439e1a --- /dev/null +++ b/src/sqlbuild/adapters/sqlserver/_helpers/grants.py @@ -0,0 +1,92 @@ +"""Read and replay SQL Server object and column permissions onto a compatibility view.""" + +from __future__ import annotations + +from typing import Any + +from sqlbuild.adapter.contract.main.grants_for_columns import grants_for_columns +from sqlbuild.adapter.contract.models import RelationGrant + +_VIEW_PERMISSIONS: frozenset[str] = frozenset( + { + "ALTER", + "CONTROL", + "DELETE", + "INSERT", + "REFERENCES", + "SELECT", + "TAKE OWNERSHIP", + "UPDATE", + "VIEW CHANGE TRACKING", + "VIEW DEFINITION", + } +) +_COLUMN_PERMISSIONS: frozenset[str] = frozenset({"REFERENCES", "SELECT", "UPDATE"}) +_STATES: dict[str, tuple[bool, bool]] = { + "GRANT": (False, False), + "GRANT_WITH_GRANT_OPTION": (False, True), + "DENY": (True, False), +} +_STATE_COLUMN: int = 0 +_PERMISSION_COLUMN: int = 1 +_GRANTEE_COLUMN: int = 2 +_COLUMN_NAME_COLUMN: int = 3 + + +def sqlserver_relation_grants(rows: list[tuple[Any, ...]]) -> tuple[RelationGrant, ...]: + """Decode object and column grants and denies that also apply to a view.""" + + grants: list[RelationGrant] = [] + row: tuple[Any, ...] + for row in rows: + permission: str = str(row[_PERMISSION_COLUMN]).upper() + state: tuple[bool, bool] | None = _STATES.get(str(row[_STATE_COLUMN]).upper()) + column: str | None = ( + None if row[_COLUMN_NAME_COLUMN] is None else str(row[_COLUMN_NAME_COLUMN]) + ) + allowed: frozenset[str] = _VIEW_PERMISSIONS if column is None else _COLUMN_PERMISSIONS + if state is None or permission not in allowed: + continue + grants.append( + RelationGrant( + privilege=permission, + grantee=str(row[_GRANTEE_COLUMN]), + column=column, + grantable=state[1], + denied=state[0], + ) + ) + return tuple(grants) + + +def render_sqlserver_view_grants( + *, grants: tuple[RelationGrant, ...], destination: str, columns: tuple[str, ...] +) -> tuple[str, ...]: + """Render object permissions, then permissions on columns the view exposes.""" + + return tuple( + ("DENY" if grant.denied else "GRANT") + + f" {grant.privilege} ON OBJECT::{destination}" + + ("" if grant.column is None else " ([" + grant.column.replace("]", "]]") + "])") + + " TO [" + + (grant.grantee or "").replace("]", "]]") + + "]" + + (" WITH GRANT OPTION" if grant.grantable else "") + for grant in grants_for_columns(grants=grants, columns=columns) + ) + + +def render_sqlserver_view_revokes( + *, grants: tuple[RelationGrant, ...], destination: str +) -> tuple[str, ...]: + """Render grants or denies to remove from ``destination``, cascading grant options.""" + + return tuple( + f"REVOKE {grant.privilege} ON OBJECT::{destination}" + + ("" if grant.column is None else " ([" + grant.column.replace("]", "]]") + "])") + + " FROM [" + + (grant.grantee or "").replace("]", "]]") + + "]" + + (" CASCADE" if grant.grantable else "") + for grant in grants + ) diff --git a/src/sqlbuild/adapters/sqlserver/classes/sqlserver_adapter.py b/src/sqlbuild/adapters/sqlserver/classes/sqlserver_adapter.py index a3cff18841..dbdef8b84a 100644 --- a/src/sqlbuild/adapters/sqlserver/classes/sqlserver_adapter.py +++ b/src/sqlbuild/adapters/sqlserver/classes/sqlserver_adapter.py @@ -42,6 +42,7 @@ UnsupportedTypedSqlRenderingError, ) from sqlbuild.adapter.contract.main.normalize_seed_csv_value import normalize_seed_csv_value +from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition from sqlbuild.adapter.contract.models import ( ColumnInfo, CursorValue, @@ -50,6 +51,7 @@ FunctionInfo, MigrationStagePlan, QueryResult, + RelationGrant, RelationInfo, RowDiffColumnResult, RowDiffCoverage, @@ -86,6 +88,11 @@ from sqlbuild.adapter.state_sql.main.render_insert_source_freshness_records_sql import ( render_insert_source_freshness_records_sql, ) +from sqlbuild.adapters.sqlserver._helpers.grants import ( + render_sqlserver_view_grants, + render_sqlserver_view_revokes, + sqlserver_relation_grants, +) from sqlbuild.adapters.sqlserver.classes.sqlserver_connection import _SqlServerConnection from sqlbuild.adapters.sqlserver.constants import ( BOOLEAN_RETURN_TYPE, @@ -635,6 +642,27 @@ def render_create_migration_state_table_sql(self, *, database: str | None, schem table_name=MIGRATION_TABLE_NAME, ) + def render_create_old_name_view_state_table_sql( + self, *, database: str | None, schema: str + ) -> str: + from sqlbuild.compiler.migrations.constants import OLD_NAME_VIEW_TABLE_NAME + from sqlbuild.compiler.migrations.main.old_name_view_create_table_sql import ( + build_old_name_view_state_create_table_sql, + ) + + create_sql: str = build_old_name_view_state_create_table_sql( + database=database, + schema=schema, + render_qualified_name=self.render_qualified_name, + render_framework_type=self.render_framework_type, + ).replace("CREATE TABLE IF NOT EXISTS", "CREATE TABLE", 1) + return self._create_table_if_missing_sql( + create_sql=create_sql, + database=database, + schema=schema, + table_name=OLD_NAME_VIEW_TABLE_NAME, + ) + def render_create_column_migration_state_table_sql( self, *, database: str | None, schema: str ) -> str: @@ -1819,6 +1847,92 @@ def capture_dependent_view_rebinds( del connection, database, schema, name return () + def views_bind_to_relation_identity(self) -> bool: + return False + + def list_dependent_view_names( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> tuple[str, ...]: + del connection, database, schema, name + return () + + def read_relation_grants( + self, + *, + connection: Any, + database: str | None, + schema: str, + name: str, + relation_type: str, + ) -> tuple[RelationGrant, ...]: + del relation_type + relation: str = ".".join( + self.render_identifier(part) for part in (database, schema, name) if part + ).replace("'", "''") + query: str = ( + "SELECT permission.state_desc, permission.permission_name, grantee.name, " + "COL_NAME(permission.major_id, NULLIF(permission.minor_id, 0)) " + "FROM sys.database_permissions AS permission " + "JOIN sys.database_principals AS grantee " + "ON grantee.principal_id = permission.grantee_principal_id " + "WHERE permission.class = 1 " + f"AND permission.major_id = OBJECT_ID(N'{relation}') " + "ORDER BY permission.minor_id, grantee.name, permission.permission_name" + ) + rows: list[tuple[Any, ...]] = self.execute(connection=connection, sql=query).fetchall() + return sqlserver_relation_grants(rows) + + def render_relation_grants( + self, *, grants: tuple[RelationGrant, ...], destination: str, columns: tuple[str, ...] + ) -> tuple[str, ...]: + return render_sqlserver_view_grants(grants=grants, destination=destination, columns=columns) + + def render_replace_view_keeping_grants( + self, *, destination: str, sql: str + ) -> tuple[str, ...] | None: + return (f"ALTER VIEW {destination} AS {sql}",) + + def read_view_definition( + self, *, connection: Any, database: str | None, schema: str, name: str + ) -> str | None: + relation: str = ".".join( + self.render_identifier(part) for part in (database, schema, name) if part + ).replace("'", "''") + rows: list[tuple[Any, ...]] = self.execute( + connection=connection, + sql=f"SELECT OBJECT_DEFINITION(OBJECT_ID(N'{relation}', N'V'))", + ).fetchall() + return None if not rows or rows[0][0] is None else str(rows[0][0]) + + def view_definition_matches( + self, *, connection: Any, database: str | None, schema: str, name: str, sql: str + ) -> bool: + definition: str | None = self.read_view_definition( + connection=connection, database=database, schema=schema, name=name + ) + return definition is not None and same_view_definition(definition=definition, sql=sql) + + def render_relation_revokes( + self, *, grants: tuple[RelationGrant, ...], destination: str + ) -> tuple[str, ...]: + return render_sqlserver_view_revokes(grants=grants, destination=destination) + + def rename_view( + self, + *, + connection: Any, + origin: str, + destination: str, + statement_recorder: StatementRecorder, + ) -> None: + statements: tuple[str, ...] = self.render_rename_view( + origin=origin, destination=destination + ) + statement_recorder.record_many(statements) + statement: str + for statement in statements: + self.execute(connection=connection, sql=statement) + def supports_transactional_ddl(self) -> bool: return True diff --git a/src/sqlbuild/cli/commands/_helpers/build_planning/plan_policies.py b/src/sqlbuild/cli/commands/_helpers/build_planning/plan_policies.py index 594b49cbf5..5c8db4d7ee 100644 --- a/src/sqlbuild/cli/commands/_helpers/build_planning/plan_policies.py +++ b/src/sqlbuild/cli/commands/_helpers/build_planning/plan_policies.py @@ -24,8 +24,11 @@ ColumnMigrationPlanEntry, ModelMigrationPlanEntry, PlanOutput, + PlanWarning, ) +from sqlbuild.compiler.planner.types import WarningSeverity +_OLD_NAME_CODES: frozenset[str] = frozenset({"M114", "P008"}) _COLUMN_BLOCK_CODES: dict[ColumnMigrationDecision, str] = { ColumnMigrationDecision.SOURCE_MISSING: "M109", ColumnMigrationDecision.CONFLICT: "M110", @@ -41,6 +44,7 @@ def enforce_build_plan_policies( _enforce_model_migration_policy(plan=plan) _enforce_column_migration_policy(plan=plan) + _enforce_old_name_policy(plan=plan) enforce_model_execution_limit( model_count=executable_model_count(plan=plan), target_name=invocation.effective_target_name, @@ -130,3 +134,20 @@ def _enforce_column_migration_policy(*, plan: PlanOutput) -> None: code=_COLUMN_BLOCK_CODES.get(blocked[0].decision, "M109"), help="Run sqb plan to see why each column rename cannot be applied.", ) + + +def _enforce_old_name_policy(*, plan: PlanOutput) -> None: + """Refuse to build over a compatibility view or while code reads an old name.""" + + errors: tuple[PlanWarning, ...] = tuple( + warning + for warning in plan.warnings + if warning.severity == WarningSeverity.ERROR and warning.code in _OLD_NAME_CODES + ) + if not errors: + return + raise CliUserError( + "; ".join(warning.message for warning in errors), + code=errors[0].code or "M114", + help="Run sqb plan to see every compatibility view and old-name reference.", + ) diff --git a/src/sqlbuild/cli/commands/_helpers/entry/dispatch.py b/src/sqlbuild/cli/commands/_helpers/entry/dispatch.py index 02eb54d4d8..200ed070e9 100644 --- a/src/sqlbuild/cli/commands/_helpers/entry/dispatch.py +++ b/src/sqlbuild/cli/commands/_helpers/entry/dispatch.py @@ -404,6 +404,7 @@ def dispatch_cli_command(*, args: CliNamespace, handlers: CliEntrypointHandlers) auto_approve=args.auto_approve, retention_days=args.retention_days, direct_state_history_versions=args.direct_state_history_versions, + drop_old_name_views=tuple(args.drop_old_name_views), ) ) return _dispatch_local_command( diff --git a/src/sqlbuild/cli/commands/_helpers/entry/parser.py b/src/sqlbuild/cli/commands/_helpers/entry/parser.py index a69e57074d..23a39ed79d 100644 --- a/src/sqlbuild/cli/commands/_helpers/entry/parser.py +++ b/src/sqlbuild/cli/commands/_helpers/entry/parser.py @@ -581,6 +581,13 @@ def _add_maintenance_parsers( janitor_parser.add_argument("--auto-approve", action="store_true", default=False) janitor_parser.add_argument("--retention-days", type=int, default=None) janitor_parser.add_argument("--direct-state-history-versions", type=int, default=None) + janitor_parser.add_argument( + "--drop-old-name-view", + action="append", + default=[], + metavar="NAME", + dest="drop_old_name_views", + ) def _add_workspace_parsers( diff --git a/src/sqlbuild/cli/commands/_helpers/janitor_output/output.py b/src/sqlbuild/cli/commands/_helpers/janitor_output/output.py index de4d7cce9e..d7126bb924 100644 --- a/src/sqlbuild/cli/commands/_helpers/janitor_output/output.py +++ b/src/sqlbuild/cli/commands/_helpers/janitor_output/output.py @@ -5,16 +5,20 @@ from datetime import datetime from typing import TextIO +from sqlbuild.compiler.migrations.types import OldNameViewDropReason from sqlbuild.executor.janitor.models import ( JanitorArchiveCandidate, JanitorArchivedRelation, JanitorDeleteCandidate, + JanitorOldNameView, + JanitorOldNameViewPlanning, JanitorPlan, JanitorQueryDiffArtifactCandidate, JanitorSkippedRelation, JanitorSkippedSchema, ) from sqlbuild.presentation.classes.cli_style import CliStyle +from sqlbuild.presentation.main.tree_connector import tree_connector def write_disabled(*, stream: TextIO, use_color: bool = False) -> None: @@ -65,6 +69,15 @@ def _write_plan_summary(*, plan: JanitorPlan, stream: TextIO, style: CliStyle) - stream.write( f" {'archives retained':<22} {style.accent(str(len(plan.retained_archives)))}\n" ) + stream.write( + f" {'old name views':<22} {style.accent(str(len(plan.old_name_views.live)))}\n" + ) + _write_count_row( + stream=stream, + style=style, + label="old name views to drop", + items=plan.old_name_views.drops, + ) else: _write_count_row( stream=stream, @@ -145,6 +158,43 @@ def _write_archive_sections(*, plan: JanitorPlan, stream: TextIO, style: CliStyl ) +def _write_old_name_views(*, plan: JanitorPlan, stream: TextIO, style: CliStyle) -> None: + views: JanitorOldNameViewPlanning = plan.old_name_views + listed: tuple[JanitorOldNameView, ...] = (*views.drops, *views.live, *views.missing) + if not listed: + return + stream.write(f"\n{style.success('Old name views')} {style.muted(f'({len(listed)})')}\n") + index: int + view: JanitorOldNameView + for index, view in enumerate(listed): + last: bool = index == len(listed) - 1 + stream.write( + f"{tree_connector(style=style, last=last)} " + f"{style.object_name(view.key.display_name())} " + f"{style.muted(f'-> model:{view.destination_model}')}\n" + ) + label, detail = _old_name_view_detail(view) + stream.write( + f"{' ' if last else style.muted('│ ')}{tree_connector(style=style, last=True)} " + f"{style.muted(label)} {detail}\n" + ) + + +def _old_name_view_detail(view: JanitorOldNameView) -> tuple[str, str]: + expiry: str = "" if view.expires_at is None else f"{view.expires_at:%Y-%m-%d}" + name: str = view.key.display_name() + if view.drop_reason == OldNameViewDropReason.EARLY: + return "drop", f"now (requested; would expire {expiry})" + if view.drop_reason == OldNameViewDropReason.EXPIRED: + return "drop", f"now (expired {expiry})" + if view.occupied: + owner: str = "" if view.claimed_by is None else f", model:{view.claimed_by}" + return "record", f"dropped (name now used by another relation{owner})" + if view.drop_reason == OldNameViewDropReason.MISSING: + return "record", "dropped (the view no longer exists)" + return "expires", f"{expiry} (live; drop early with --drop-old-name-view {name})" + + def _utc(value: datetime) -> str: return f"{value:%Y-%m-%d %H:%M:%S} UTC" @@ -205,6 +255,8 @@ def write_plan(*, plan: JanitorPlan, stream: TextIO, use_color: bool = False) -> _write_archive_sections(plan=plan, stream=stream, style=style) + _write_old_name_views(plan=plan, stream=stream, style=style) + _write_query_artifact_candidates(plan=plan, stream=stream, style=style) if plan.direct_state_prune_candidates: @@ -249,7 +301,9 @@ def physical_janitor_deletion_count(plan: JanitorPlan) -> int: relation_count: int = ( len(plan.archive_deletion_candidates) if plan.direct_mode else len(plan.candidates) ) - return relation_count + len(plan.query_diff_artifact_candidates) + return ( + relation_count + len(plan.query_diff_artifact_candidates) + len(plan.old_name_views.drops) + ) def environment_label(plan: JanitorPlan) -> str: diff --git a/src/sqlbuild/cli/commands/_helpers/janitor_output/outputs.py b/src/sqlbuild/cli/commands/_helpers/janitor_output/outputs.py index f7d2a47eee..395b79fb6c 100644 --- a/src/sqlbuild/cli/commands/_helpers/janitor_output/outputs.py +++ b/src/sqlbuild/cli/commands/_helpers/janitor_output/outputs.py @@ -44,6 +44,8 @@ def janitor_plan_has_work(planning_result: JanitorPlanningResult) -> bool: or plan.archive_deletion_candidates or plan.query_diff_artifact_candidates or plan.direct_state_prune_candidates + or plan.old_name_views.drops + or plan.old_name_views.missing ) @@ -119,6 +121,7 @@ def _deletion_summary(*, result: JanitorExecutionResult) -> str: len(result.deleted) + len(result.deleted_archives) + len(result.deleted_query_diff_artifacts) + + len(result.dropped_old_name_views) ) objects: str = format_count_noun(count=deleted_object_count, singular="object") pruned_state_count: int = len(result.pruned_direct_state) diff --git a/src/sqlbuild/cli/commands/_helpers/janitor_runtime/invocation.py b/src/sqlbuild/cli/commands/_helpers/janitor_runtime/invocation.py index 25e3cdfcaa..996f79dc70 100644 --- a/src/sqlbuild/cli/commands/_helpers/janitor_runtime/invocation.py +++ b/src/sqlbuild/cli/commands/_helpers/janitor_runtime/invocation.py @@ -56,4 +56,5 @@ def resolve_janitor_settings( archive_retention_days=( invocation.discovered_inputs.project_config.janitor.archive_retention_days ), + drop_old_name_views=request.drop_old_name_views, ) diff --git a/src/sqlbuild/cli/commands/_helpers/janitor_runtime/planning.py b/src/sqlbuild/cli/commands/_helpers/janitor_runtime/planning.py index 329c57f9a5..531dfe13fe 100644 --- a/src/sqlbuild/cli/commands/_helpers/janitor_runtime/planning.py +++ b/src/sqlbuild/cli/commands/_helpers/janitor_runtime/planning.py @@ -5,6 +5,7 @@ import sys import time +from sqlbuild.cli.commands.exceptions import CliUserError from sqlbuild.cli.commands.models import ( JanitorCompileContext, JanitorConnectionContext, @@ -48,12 +49,20 @@ def build_janitor_execution_plan( state_history_versions=settings.direct_state_history_versions, archive_retention_days=settings.archive_retention_days, ), + early_old_name_view_drops=settings.drop_old_name_views, ) lifecycle.completed(metadata={"item_count": _janitor_candidate_count(plan)}) status.complete( message=f"Inspected warehouse state. ({time.perf_counter() - inspect_start:.2f}s)", blank_line_after=True, ) + if plan.old_name_views.unknown_requests: + raise CliUserError( + "no compatibility view is recorded at " + + ", ".join(plan.old_name_views.unknown_requests), + code="C503", + help="Run sqb janitor without --drop-old-name-view to list compatibility views.", + ) return JanitorPlanningResult(plan=plan) @@ -65,5 +74,6 @@ def _janitor_candidate_count(plan: JanitorPlan) -> int: plan.archive_deletion_candidates, plan.query_diff_artifact_candidates, plan.direct_state_prune_candidates, + plan.old_name_views.drops, ) ) diff --git a/src/sqlbuild/cli/commands/classes/cli_namespace.py b/src/sqlbuild/cli/commands/classes/cli_namespace.py index 44c1af0838..e4cac9dbda 100644 --- a/src/sqlbuild/cli/commands/classes/cli_namespace.py +++ b/src/sqlbuild/cli/commands/classes/cli_namespace.py @@ -63,6 +63,7 @@ "auto_approve": False, "retention_days": None, "direct_state_history_versions": None, + "drop_old_name_views": [], "bounded": None, "max_column_examples": None, "max_row_only_examples": None, @@ -222,6 +223,7 @@ class CliNamespace: auto_approve: bool retention_days: int | None direct_state_history_versions: int | None + drop_old_name_views: list[str] bounded: str | None max_column_examples: int | None max_row_only_examples: int | None diff --git a/src/sqlbuild/cli/commands/models.py b/src/sqlbuild/cli/commands/models.py index 2f80ebb3de..74abf7baf5 100644 --- a/src/sqlbuild/cli/commands/models.py +++ b/src/sqlbuild/cli/commands/models.py @@ -648,6 +648,7 @@ class JanitorCommandRequest: auto_approve: bool = False retention_days: int | None = None direct_state_history_versions: int | None = None + drop_old_name_views: tuple[str, ...] = () @dataclass(frozen=True) @@ -666,6 +667,7 @@ class JanitorSettings: retention_days: int direct_state_history_versions: int archive_retention_days: int = 14 + drop_old_name_views: tuple[str, ...] = () @dataclass(frozen=True) diff --git a/src/sqlbuild/cli/output/_helpers/plan_json.py b/src/sqlbuild/cli/output/_helpers/plan_json.py index 112c1c5f94..d35920e7dd 100644 --- a/src/sqlbuild/cli/output/_helpers/plan_json.py +++ b/src/sqlbuild/cli/output/_helpers/plan_json.py @@ -10,20 +10,23 @@ from sqlbuild.cli.output._helpers.future_cursor_safety import serialize_future_cursor_safety from sqlbuild.cli.output._helpers.maximum_start_safety import serialize_maximum_start_safety from sqlbuild.cli.output.models import CursorPlanDetails +from sqlbuild.compiler.migrations.types import MigrationDecision from sqlbuild.compiler.pipeline.models import PythonPlanEntry from sqlbuild.compiler.planner.models import ( CascadeResult, FunctionPlanEntry, FutureCursorSafetyEvidence, MaximumStartSafetyEvidence, + ModelMigrationPlanEntry, ModelPlanEntry, + OldNameViewPlanEntry, PlanOutput, PlanProviderUsage, PlanWarning, SeedPlanEntry, SourceLoadPlanEntry, ) -from sqlbuild.compiler.planner.types import IncrementalMode, PlanReason +from sqlbuild.compiler.planner.types import IncrementalMode, MaterializationType, PlanReason from sqlbuild.compiler.python_nodes.types import PythonIdentityStatus from sqlbuild.cursor_algebra.main.sentinel_to_token import sentinel_to_token @@ -99,6 +102,7 @@ def format_plan_json( "table_type_conversions": table_type_conversions, "migrations": _serialize_model_migrations(plan), "column_migrations": _serialize_column_migrations(plan), + "old_names": [_serialize_old_name(entry) for entry in plan.old_name_view_entries], } if plan.metadata: result["metadata"] = plan.metadata @@ -496,7 +500,26 @@ def _serialize_column_migrations(plan: PlanOutput) -> list[dict[str, object]]: ] +def _serialize_old_name(entry: OldNameViewPlanEntry) -> dict[str, object]: + return { + "model": entry.model_name, + "action": entry.action.value, + "view": entry.origin.qualified_name or entry.origin.name, + "reads": entry.destination.qualified_name or entry.destination.name, + "expires_at": entry.expires_at.isoformat() if entry.expires_at is not None else None, + "column_aliases": dict(entry.column_aliases), + "grants_copied": entry.grants_copied, + "reason": entry.reason, + } + + def _serialize_model_migrations(plan: PlanOutput) -> list[dict[str, object]]: + old_names: dict[str, OldNameViewPlanEntry] = { + entry.model_name: entry for entry in plan.old_name_view_entries + } + materializations: dict[str, str] = { + entry.name: str(entry.materialization_type) for entry in plan.model_entries + } return [ { "kind": "model_migration", @@ -510,7 +533,9 @@ def _serialize_model_migrations(plan: PlanOutput) -> list[dict[str, object]]: "destination": entry.destination.qualified_name or entry.destination.name, "target": entry.target_name, "origin_version_hash": entry.origin_version_hash, - "transfer": entry.transfer.value if entry.transfer is not None else None, + "transfer": _migration_transfer( + entry=entry, materialization=materializations.get(entry.model_name) + ), "transfer_fallback": ( entry.transfer_fallback.value if entry.transfer_fallback is not None else None ), @@ -519,6 +544,21 @@ def _serialize_model_migrations(plan: PlanOutput) -> list[dict[str, object]]: "completed_at": ( entry.completed_at.isoformat() if entry.completed_at is not None else None ), + "old_name": ( + _serialize_old_name(old_names[entry.model_name]) + if entry.model_name in old_names + else None + ), } for entry in plan.migration_entries ] + + +def _migration_transfer( + *, entry: ModelMigrationPlanEntry, materialization: str | None +) -> str | None: + if entry.transfer is not None: + return entry.transfer.value + if entry.decision != MigrationDecision.RENAMED or entry.completed_at is not None: + return None + return "recreate" if materialization == MaterializationType.VIEW else "rebuild" diff --git a/src/sqlbuild/compiler/compile/_helpers/config/model_validation.py b/src/sqlbuild/compiler/compile/_helpers/config/model_validation.py index bafc661f53..53fa6c6612 100644 --- a/src/sqlbuild/compiler/compile/_helpers/config/model_validation.py +++ b/src/sqlbuild/compiler/compile/_helpers/config/model_validation.py @@ -17,6 +17,7 @@ ) from sqlbuild.compiler.compile.exceptions import CompileInputError from sqlbuild.compiler.compile.models import CompileModelConfig +from sqlbuild.compiler.migrations.constants import OLD_NAME_VIEW_CONFIG_KEY from sqlbuild.compiler.planner.types import ( ContractPolicy, CursorGrain, @@ -119,6 +120,18 @@ "lookback", ) +_HISTORY_MATERIALIZATIONS: frozenset[str] = frozenset( + {MaterializationType.INCREMENTAL, MaterializationType.SNAPSHOT} +) +_MIGRATABLE_MATERIALIZATIONS: frozenset[str] = frozenset( + { + MaterializationType.TABLE, + MaterializationType.VIEW, + MaterializationType.INCREMENTAL, + MaterializationType.SNAPSHOT, + } +) + @dataclass(frozen=True) class _IncrementalConfigValues: @@ -953,17 +966,25 @@ def validate_non_incremental_config( def validate_model_migration_config(*, config: CompileModelConfig, model_name: str) -> None: - """Validate migrate_from and migrate_force on history-bearing materializations.""" + """Validate migrate_from, migrate_force, and old_name_view on model headers.""" + _validate_old_name_view_config(config=config, model_name=model_name) migrate_from: object | None = config.values.get(MIGRATE_FROM_CONFIG_KEY) migrate_force: object | None = config.values.get(MIGRATE_FORCE_CONFIG_KEY) if migrate_from is None and migrate_force is None: return materialized: str | None = get_config_str(values=config.values, key="materialized") - if materialized not in {MaterializationType.INCREMENTAL, MaterializationType.SNAPSHOT}: + if materialized not in _MIGRATABLE_MATERIALIZATIONS: + raise CompileInputError( + f"model '{model_name}': migrate_from is only valid for table, view, incremental " + f"and snapshot models" + ) + if migrate_force is not None and materialized not in _HISTORY_MATERIALIZATIONS: raise CompileInputError( - f"model '{model_name}': migrate_from is only valid for incremental and snapshot " - f"models; '{materialized}' models are rebuilt under their new name" + f"model '{model_name}': migrate_force is only valid for incremental and snapshot " + f"models; nothing is replaced when a '{materialized}' model migrates, because " + "tables and views are rebuilt under their new name", + help="remove migrate_force from the model header", ) if migrate_from is None: raise CompileInputError(f"model '{model_name}': migrate_force requires migrate_from") @@ -977,6 +998,21 @@ def validate_model_migration_config(*, config: CompileModelConfig, model_name: s raise CompileInputError(f"model '{model_name}': migrate_force must be true or false") +def _validate_old_name_view_config(*, config: CompileModelConfig, model_name: str) -> None: + if OLD_NAME_VIEW_CONFIG_KEY not in config.values: + return + value: object = config.values[OLD_NAME_VIEW_CONFIG_KEY] + if value is False: + return + if isinstance(value, str) and Duration.parse(value.strip()) is not None: + return + raise CompileInputError( + f"model '{model_name}': old_name_view must be a positive duration such as 7d, or " + f"false; got {value!r}", + help="write for example old_name_view 7d, or old_name_view false", + ) + + def validate_column_migration_config( *, config_values: dict[str, object], model_name: str, columns: tuple[SchemaColumn, ...] ) -> None: @@ -988,12 +1024,17 @@ def validate_column_migration_config( if not declared: return materialized: str | None = get_config_str(values=config_values, key="materialized") - if materialized not in {MaterializationType.INCREMENTAL, MaterializationType.SNAPSHOT}: + if materialized not in _HISTORY_MATERIALIZATIONS and ( + materialized not in _MIGRATABLE_MATERIALIZATIONS + or config_values.get(MIGRATE_FROM_CONFIG_KEY) is None + ): raise CompileInputError( f"model '{model_name}': column '{declared[0].name}' declares migrate_from, which is " - f"only valid for incremental and snapshot models; '{materialized}' models are " - "rebuilt with their new columns", - help="remove migrate_from from the column", + "only valid for incremental and snapshot models, or for table and view models " + "that declare migrate_from themselves (the old name's compatibility view then " + f"presents the column under its old name); '{materialized}' models are rebuilt " + "with their new columns", + help="remove migrate_from from the column, or declare the model's own migrate_from", ) sources: dict[str, str] = {} column: SchemaColumn diff --git a/src/sqlbuild/compiler/compile/_helpers/explicit_references/python_sql.py b/src/sqlbuild/compiler/compile/_helpers/explicit_references/python_sql.py index 3ad9224695..1c32f74bbe 100644 --- a/src/sqlbuild/compiler/compile/_helpers/explicit_references/python_sql.py +++ b/src/sqlbuild/compiler/compile/_helpers/explicit_references/python_sql.py @@ -10,19 +10,27 @@ from pathlib import Path from sqlbuild.compiler.compile.constants import HARD_CODED_PROJECT_RELATION_CODE -from sqlbuild.compiler.compile.models import CompiledProject, CompilerDiagnostic +from sqlbuild.compiler.compile.models import ( + CompiledProject, + CompilerDiagnostic, + PythonSqlReferenceReport, +) from sqlbuild.compiler.compile.types import DiagnosticPhase, DiagnosticSeverity from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.compiler.graph.main._model_python_hook_names import model_python_hook_names from sqlbuild.compiler.references.main._compiled_project_relations import ( compiled_project_relations, ) +from sqlbuild.compiler.references.main._old_name_reference_message import ( + old_name_reference_message, +) from sqlbuild.compiler.references.main.extract_relation_names import extract_relation_names from sqlbuild.compiler.references.main.hard_coded_relation_remedy import ( hard_coded_relation_remedy, ) from sqlbuild.compiler.references.main.match_project_relation import match_project_relation from sqlbuild.compiler.references.models import ( + LiteralSqlRelation, ProjectRelation, ProjectRelationIndex, RelationName, @@ -58,37 +66,44 @@ class _LiteralSql: def python_sql_reference_diagnostics( *, project: CompiledProject, discovered_inputs: DiscoveredProjectInputs -) -> tuple[CompilerDiagnostic, ...]: - """Return a P008 error for each project relation literal SQL in Python code names.""" +) -> PythonSqlReferenceReport: + """Return P008 errors for project relations literal SQL in Python code names.""" owners: tuple[_PythonSqlOwner, ...] = _owners( project=project, discovered_inputs=discovered_inputs ) if not owners: - return () - index: ProjectRelationIndex = compiled_project_relations(project=project) - if not index.relations: - return () + return PythonSqlReferenceReport() + index: ProjectRelationIndex = compiled_project_relations( + project=project, + old_name_retention=discovered_inputs.project_config.migrations.old_name_views, + ) diagnostics: dict[tuple[str, int, str, str], CompilerDiagnostic] = {} + unmatched: list[LiteralSqlRelation] = [] for owner in owners: temporary: frozenset[str] = frozenset() for literal in sorted(_literal_sql_calls(owner.function), key=lambda item: item.line): found: dict[tuple[str, int, str, str], CompilerDiagnostic] - temporary, found = _hard_coded_relations( + unnamed: tuple[LiteralSqlRelation, ...] + temporary, found, unnamed = _hard_coded_relations( owner=owner, literal=literal, index=index, dialect=project.sql_analysis_dialect, temporary=temporary, ) + unmatched.extend(unnamed) for key, diagnostic in found.items(): diagnostics.setdefault(key, diagnostic) - return tuple( - diagnostics[key] - for key in sorted( - diagnostics, - key=lambda key: (diagnostics[key].path or Path(), key[1], key[0], key[2], key[3]), - ) + return PythonSqlReferenceReport( + diagnostics=tuple( + diagnostics[key] + for key in sorted( + diagnostics, + key=lambda key: (diagnostics[key].path or Path(), key[1], key[0], key[2], key[3]), + ) + ), + unmatched=tuple(dict.fromkeys(unmatched)), ) @@ -240,13 +255,18 @@ def _hard_coded_relations( index: ProjectRelationIndex, dialect: str | None, temporary: frozenset[str], -) -> tuple[frozenset[str], dict[tuple[str, int, str, str], CompilerDiagnostic]]: +) -> tuple[ + frozenset[str], + dict[tuple[str, int, str, str], CompilerDiagnostic], + tuple[LiteralSqlRelation, ...], +]: extracted: tuple[tuple[RelationName, ...], frozenset[str]] | None = extract_relation_names( sql=literal.sql, dialect=dialect ) found: dict[tuple[str, int, str, str], CompilerDiagnostic] = {} + unmatched: list[LiteralSqlRelation] = [] if extracted is None: - return temporary, found + return temporary, found, () relations, created = extracted temporary = temporary | created for relation in relations: @@ -257,7 +277,19 @@ def _hard_coded_relations( if relation.schema is None and relation.name.casefold() in temporary: continue match: ProjectRelation | None = match_project_relation(index=index, relation=relation) - if match is None or match.ref in owner.own_refs: + if match is None: + unmatched.append( + LiteralSqlRelation( + owner_label=owner.label, + owner_kind=owner.kind, + relative_path=owner.relative_path, + line=literal.line, + method=literal.method, + relation=relation, + ) + ) + continue + if match.ref in owner.own_refs and match.compatibility_for is None: continue key: tuple[str, int, str, str] = ( owner.label, @@ -271,7 +303,7 @@ def _hard_coded_relations( owner=owner, literal=literal, relation=relation, match=match ), ) - return temporary, found + return temporary, found, tuple(unmatched) def _hard_coded_relation_diagnostic( @@ -300,8 +332,15 @@ def _hard_coded_relation_diagnostic( severity=DiagnosticSeverity.ERROR, code=HARD_CODED_PROJECT_RELATION_CODE, message=( - f"{owner.label} names {match.ref.kind.value}:{match.ref.name} as '{written}' in SQL " - f"passed to ctx.{literal.method}()" + old_name_reference_message( + owner_label=owner.label, + written=written, + method=literal.method, + model_name=match.compatibility_for, + ) + if match.compatibility_for is not None + else f"{owner.label} names {match.ref.kind.value}:{match.ref.name} as '{written}' " + f"in SQL passed to ctx.{literal.method}()" ), location=SourceLocation(path=owner.relative_path, line=literal.line, column=1), help=( diff --git a/src/sqlbuild/compiler/compile/main/_assemble_project.py b/src/sqlbuild/compiler/compile/main/_assemble_project.py index b7a12c0f8b..9cf506e6e5 100644 --- a/src/sqlbuild/compiler/compile/main/_assemble_project.py +++ b/src/sqlbuild/compiler/compile/main/_assemble_project.py @@ -21,6 +21,7 @@ CompiledProject, CompileProjectInputs, CompilerDiagnostic, + PythonSqlReferenceReport, ) from sqlbuild.compiler.lineage.types import ColumnLineageMode @@ -45,17 +46,22 @@ def assemble_project( analysis_cache_dir=analysis_cache_dir, analysis_model_names=analysis_model_names, ) + python_sql: PythonSqlReferenceReport = ( + python_sql_reference_diagnostics( + project=project, discovered_inputs=inputs.discovered_inputs + ) + if inputs.project_config.references.enforce_explicit + else PythonSqlReferenceReport() + ) reference_diagnostics: tuple[CompilerDiagnostic, ...] = ( *attached_audit_gate_diagnostics(project=project), *hook_read_diagnostics(project=project), - *( - python_sql_reference_diagnostics( - project=project, discovered_inputs=inputs.discovered_inputs - ) - if inputs.project_config.references.enforce_explicit - else () - ), + *python_sql.diagnostics, ) - if not reference_diagnostics: + if not reference_diagnostics and not python_sql.unmatched: return project - return replace(project, diagnostics=(*project.diagnostics, *reference_diagnostics)) + return replace( + project, + diagnostics=(*project.diagnostics, *reference_diagnostics), + unmatched_literal_sql_relations=python_sql.unmatched, + ) diff --git a/src/sqlbuild/compiler/compile/models.py b/src/sqlbuild/compiler/compile/models.py index 38e3e3236d..cbe21e788e 100644 --- a/src/sqlbuild/compiler/compile/models.py +++ b/src/sqlbuild/compiler/compile/models.py @@ -49,6 +49,7 @@ ColumnTransformKind, InferredNullability, ) +from sqlbuild.compiler.references.models import LiteralSqlRelation from sqlbuild.compiler.references.types import ExternalSqlReferenceResolver, SqlReferenceKind from sqlbuild.compiler.scopes.models import ( DeclarationIdentity, @@ -1026,6 +1027,14 @@ def __post_init__(self) -> None: ) +@dataclass(frozen=True) +class PythonSqlReferenceReport: + """P008 diagnostics plus literal SQL relations that named no project relation.""" + + diagnostics: tuple[CompilerDiagnostic, ...] = () + unmatched: tuple[LiteralSqlRelation, ...] = () + + @dataclass(frozen=True) class CompiledProject: """Planner-ready whole-project compile output.""" @@ -1057,6 +1066,7 @@ class CompiledProject: public_constants: dict[str, ConstantDeclaration] = field(default_factory=dict) loaded_macros: dict[str, LoadedMacro] = field(default_factory=dict) diagnostics: tuple[CompilerDiagnostic, ...] = field(default_factory=tuple) + unmatched_literal_sql_relations: tuple[LiteralSqlRelation, ...] = field(default_factory=tuple) external_sql_reference_resolver: ExternalSqlReferenceResolver | None = None scope_index: ScopeIndex = field(default_factory=ScopeIndex) sql_expansions: dict[Path, CompiledSqlExpansion] = field( diff --git a/src/sqlbuild/compiler/discovery/_helpers/yml/project.py b/src/sqlbuild/compiler/discovery/_helpers/yml/project.py index bee4de9067..b36a06510f 100644 --- a/src/sqlbuild/compiler/discovery/_helpers/yml/project.py +++ b/src/sqlbuild/compiler/discovery/_helpers/yml/project.py @@ -65,6 +65,7 @@ MaterializationRetentionDefaults, MicrobatchesConfig, MicrobatchLimitsConfig, + MigrationsConfig, ProjectConfig, ReferencesConfig, ScenarioConfig, @@ -109,6 +110,7 @@ {"named", _LIFECYCLE_SHUTDOWN_TIMEOUT_CONFIG_KEY} ) _LEGACY_EVENT_EXPORTERS_CONFIG_KEY: str = "event_exporters" +_OLD_NAME_VIEWS_CONFIG_KEY: str = "old_name_views" def load_project_config(*, project_dir: Path) -> ProjectConfig: @@ -156,6 +158,9 @@ def load_project_config(*, project_dir: Path) -> ProjectConfig: file_path=file_path, ) janitor: JanitorConfig = _load_janitor(payload=payload.get("janitor"), file_path=file_path) + migrations: MigrationsConfig = _load_migrations( + payload=payload.get("migrations"), file_path=file_path + ) snapshots: SnapshotsConfig = _load_snapshots( payload=payload.get("snapshots"), file_path=file_path ) @@ -188,6 +193,7 @@ def load_project_config(*, project_dir: Path) -> ProjectConfig: vars=vars_map, targets=targets, janitor=janitor, + migrations=migrations, snapshots=snapshots, scenario=scenario, dbt=dbt, @@ -677,6 +683,29 @@ def _load_microbatches(*, payload: object, file_path: Path) -> MicrobatchesConfi return MicrobatchesConfig(limits=MicrobatchLimitsConfig(max_batches=max_batches, action=action)) +def _load_migrations(*, payload: object, file_path: Path) -> MigrationsConfig: + mapping: dict[str, object] = _coerce_mapping( + payload=payload, label="migrations", file_path=file_path + ) + _validate_allowed_keys( + mapping=mapping, + allowed_keys=frozenset({_OLD_NAME_VIEWS_CONFIG_KEY}), + label="migrations", + file_path=file_path, + ) + if _OLD_NAME_VIEWS_CONFIG_KEY not in mapping: + return MigrationsConfig() + value: object = mapping[_OLD_NAME_VIEWS_CONFIG_KEY] + if value is False: + return MigrationsConfig(old_name_views=None) + if isinstance(value, str) and Duration.parse(value.strip()) is not None: + return MigrationsConfig(old_name_views=value.strip()) + raise ProjectConfigError( + f"{file_path} migrations.old_name_views must be a positive duration such as '30d', " + "or false to disable compatibility views at old model names" + ) + + def _load_scopes(*, payload: object, file_path: Path) -> ScopesConfig: mapping: dict[str, object] = _coerce_mapping( payload=payload, label="scopes", file_path=file_path diff --git a/src/sqlbuild/compiler/migrations/_helpers/old_name_sql.py b/src/sqlbuild/compiler/migrations/_helpers/old_name_sql.py new file mode 100644 index 0000000000..ea5a3a3996 --- /dev/null +++ b/src/sqlbuild/compiler/migrations/_helpers/old_name_sql.py @@ -0,0 +1,207 @@ +"""Portable SQL for append-only old-name view facts.""" + +from __future__ import annotations + +import json +from collections.abc import Callable, Iterable +from typing import Any + +from sqlbuild.adapter.contract.types import FrameworkType +from sqlbuild.adapter.state_sql.main.render_state_table_create_sql import ( + render_state_table_create_sql, +) +from sqlbuild.compiler.migrations._helpers.sql import ( + decode_event_timestamp, + optional_text, + render_event_exists_sql, + render_event_insert_sql, +) +from sqlbuild.compiler.migrations.constants import ( + OLD_NAME_VIEW_COLUMN_TYPES, + OLD_NAME_VIEW_COLUMNS, + OLD_NAME_VIEW_TABLE_NAME, +) +from sqlbuild.compiler.migrations.exceptions import MigrationStateError +from sqlbuild.compiler.migrations.models import MigrationRelation, OldNameViewEvent +from sqlbuild.compiler.migrations.types import OldNameViewDropReason, OldNameViewEventType + + +def qualified_old_name_view_table( + *, database: str | None, schema: str, render_qualified_name: Callable[..., str | None] +) -> str: + table: str | None = render_qualified_name( + database=database, schema=schema, name=OLD_NAME_VIEW_TABLE_NAME + ) + if table is None: + raise MigrationStateError("old-name view state requires a target schema") + return table + + +def build_old_name_create_table_sql( + *, + database: str | None, + schema: str, + render_qualified_name: Callable[..., str | None], + render_framework_type: Callable[[FrameworkType], str], + transient: bool, +) -> str: + return render_state_table_create_sql( + qualified_name=qualified_old_name_view_table( + database=database, schema=schema, render_qualified_name=render_qualified_name + ), + columns=OLD_NAME_VIEW_COLUMNS, + column_types=OLD_NAME_VIEW_COLUMN_TYPES, + required_columns=frozenset(), + render_framework_type=render_framework_type, + transient=transient, + ) + + +def build_old_name_existing_event_sql( + *, event: OldNameViewEvent, render_qualified_name: Callable[..., str | None] +) -> str: + return render_event_exists_sql( + table=_event_table(event=event, render_qualified_name=render_qualified_name), + event_id=event.event_id, + ) + + +def build_old_name_insert_sql( + *, event: OldNameViewEvent, render_qualified_name: Callable[..., str | None] +) -> str: + return render_event_insert_sql( + table=_event_table(event=event, render_qualified_name=render_qualified_name), + columns=OLD_NAME_VIEW_COLUMNS, + column_types=OLD_NAME_VIEW_COLUMN_TYPES, + values=( + event.event_id, + event.target_name, + event.event_type.value, + event.migration_event_id, + event.destination_model, + event.old.database, + event.old.schema, + event.old.name, + event.new.database, + event.new.schema, + event.new.name, + event.view_retention, + event.archive_name, + _encode_aliases(event.column_aliases) if event.column_aliases else None, + None if event.grants_copied is None else json.dumps(list(event.grants_copied)), + event.view_sql, + event.expires_at, + None if event.drop_reason is None else event.drop_reason.value, + event.run_id, + event.created_at, + ), + ) + + +def readable_old_name_columns(stored_columns: Iterable[str]) -> tuple[str, ...]: + """Return the known columns a stored table has; columns added later may be absent.""" + + stored: frozenset[str] = frozenset(column.lower() for column in stored_columns) + return tuple(column for column in OLD_NAME_VIEW_COLUMNS if column in stored) + + +def build_old_name_read_sql( + *, + database: str | None, + schema: str, + columns: tuple[str, ...], + render_qualified_name: Callable[..., str | None], +) -> str: + table: str = qualified_old_name_view_table( + database=database, schema=schema, render_qualified_name=render_qualified_name + ) + return f"SELECT {', '.join(columns)} FROM {table} ORDER BY created_at, event_id" + + +def decode_old_name_event_row( + *, row: tuple[Any, ...], columns: tuple[str, ...] = OLD_NAME_VIEW_COLUMNS +) -> OldNameViewEvent | None: + """Decode one row of ``columns``, reading absent ones as NULL; skip unknown event types.""" + + values: dict[str, Any] = dict(zip(columns, row, strict=False)) + try: + event_type: OldNameViewEventType = OldNameViewEventType(str(values.get("event_type"))) + except ValueError: + return None + raw_reason: str | None = optional_text(values.get("drop_reason")) + raw_expires: object | None = values.get("expires_at") + return OldNameViewEvent( + event_id=str(values["event_id"]), + target_name=optional_text(values.get("target_name")), + event_type=event_type, + migration_event_id=str(values["migration_event_id"]), + destination_model=str(values["destination_model"]), + old=MigrationRelation( + database=optional_text(values.get("old_database")), + schema=optional_text(values.get("old_schema")), + name=str(values["old_name"]), + ), + new=MigrationRelation( + database=optional_text(values.get("new_database")), + schema=optional_text(values.get("new_schema")), + name=str(values["new_name"]), + ), + run_id=str(values["run_id"]), + created_at=decode_event_timestamp(values["created_at"]), + view_retention=optional_text(values.get("view_retention")), + archive_name=optional_text(values.get("archive_name")), + column_aliases=_decode_aliases(optional_text(values.get("column_aliases"))), + expires_at=None if raw_expires is None else decode_event_timestamp(raw_expires), + drop_reason=_decode_reason(raw_reason), + grants_copied=_decode_grants(optional_text(values.get("grants_copied"))), + view_sql=optional_text(values.get("view_sql")), + ) + + +def _decode_reason(raw: str | None) -> OldNameViewDropReason | None: + if raw is None: + return None + try: + return OldNameViewDropReason(raw) + except ValueError: + return None + + +def _decode_grants(raw: str | None) -> tuple[str, ...] | None: + if raw is None: + return None + try: + payload: object = json.loads(raw) + except json.JSONDecodeError: + return None + if not isinstance(payload, list): + return None + return tuple(str(statement) for statement in payload) + + +def _encode_aliases(aliases: tuple[tuple[str, str], ...]) -> str: + return json.dumps(dict(aliases), sort_keys=True, separators=(",", ":")) + + +def _decode_aliases(raw: str | None) -> tuple[tuple[str, str], ...]: + if not raw: + return () + try: + payload: object = json.loads(raw) + except json.JSONDecodeError: + return () + if not isinstance(payload, dict): + return () + return tuple(sorted((str(old), str(new)) for old, new in payload.items())) + + +def _event_table( + *, event: OldNameViewEvent, render_qualified_name: Callable[..., str | None] +) -> str: + if event.new.schema is None: + raise MigrationStateError("old-name view facts require a destination schema") + return qualified_old_name_view_table( + database=event.new.database, + schema=event.new.schema, + render_qualified_name=render_qualified_name, + ) diff --git a/src/sqlbuild/compiler/migrations/constants.py b/src/sqlbuild/compiler/migrations/constants.py index 8a1713d3bd..07c1e30dcb 100644 --- a/src/sqlbuild/compiler/migrations/constants.py +++ b/src/sqlbuild/compiler/migrations/constants.py @@ -52,3 +52,35 @@ **{column: StateSqlValueType.STRING for column in COLUMN_MIGRATION_COLUMNS}, "created_at": StateSqlValueType.TEXT_TIMESTAMP, } + +OLD_NAME_VIEW_TABLE_NAME: str = "_sqlbuild_old_name_views" +OLD_NAME_VIEW_CONFIG_KEY: str = "old_name_view" + +OLD_NAME_VIEW_COLUMNS: tuple[str, ...] = ( + "event_id", + "target_name", + "event_type", + "migration_event_id", + "destination_model", + "old_database", + "old_schema", + "old_name", + "new_database", + "new_schema", + "new_name", + "view_retention", + "archive_name", + "column_aliases", + "grants_copied", + "view_sql", + "expires_at", + "drop_reason", + "run_id", + "created_at", +) + +OLD_NAME_VIEW_COLUMN_TYPES: dict[str, StateSqlValueType] = { + **{column: StateSqlValueType.STRING for column in OLD_NAME_VIEW_COLUMNS}, + "expires_at": StateSqlValueType.TEXT_TIMESTAMP, + "created_at": StateSqlValueType.TEXT_TIMESTAMP, +} diff --git a/src/sqlbuild/compiler/migrations/main/_project_old_name_views.py b/src/sqlbuild/compiler/migrations/main/_project_old_name_views.py new file mode 100644 index 0000000000..c77a46db97 --- /dev/null +++ b/src/sqlbuild/compiler/migrations/main/_project_old_name_views.py @@ -0,0 +1,40 @@ +"""Project old-name view histories from recorded moves and old-name facts.""" + +from __future__ import annotations + +from collections.abc import Iterable + +from sqlbuild.compiler.migrations.models import ( + MigrationEvent, + OldNameViewEvent, + OldNameViewHistory, +) +from sqlbuild.compiler.migrations.types import OldNameViewEventType + + +def project_old_name_views( + *, moves: Iterable[MigrationEvent], facts: Iterable[OldNameViewEvent] +) -> tuple[OldNameViewHistory, ...]: + """Return one history per recorded move whose old name needs a compatibility view.""" + + by_move: dict[str, dict[OldNameViewEventType, OldNameViewEvent]] = {} + fact: OldNameViewEvent + for fact in sorted(facts, key=lambda item: (item.created_at, item.event_id)): + _ = by_move.setdefault(fact.migration_event_id, {}).setdefault(fact.event_type, fact) + histories: list[OldNameViewHistory] = [] + move: MigrationEvent + for move in sorted(moves, key=lambda item: (item.created_at, item.event_id)): + recorded: dict[OldNameViewEventType, OldNameViewEvent] = by_move.get(move.event_id, {}) + required: OldNameViewEvent | None = recorded.get(OldNameViewEventType.REQUIRED) + if required is None: + continue + histories.append( + OldNameViewHistory( + move=move, + required=required, + archived=recorded.get(OldNameViewEventType.ORIGIN_ARCHIVED), + created=recorded.get(OldNameViewEventType.VIEW_CREATED), + dropped=recorded.get(OldNameViewEventType.VIEW_DROPPED), + ) + ) + return tuple(histories) diff --git a/src/sqlbuild/compiler/migrations/main/_resolve_old_name_view_retention.py b/src/sqlbuild/compiler/migrations/main/_resolve_old_name_view_retention.py new file mode 100644 index 0000000000..f65d17b304 --- /dev/null +++ b/src/sqlbuild/compiler/migrations/main/_resolve_old_name_view_retention.py @@ -0,0 +1,20 @@ +"""Resolve how long a migrated model keeps a compatibility view at its old name.""" + +from __future__ import annotations + +from collections.abc import Mapping + +from sqlbuild.compiler.migrations.constants import OLD_NAME_VIEW_CONFIG_KEY + + +def resolve_old_name_view_retention( + *, config_values: Mapping[str, object], project_retention: str | None +) -> str | None: + """Return the model's retention text, the project's when unset, or None when disabled.""" + + if OLD_NAME_VIEW_CONFIG_KEY not in config_values: + return project_retention + value: object = config_values[OLD_NAME_VIEW_CONFIG_KEY] + if value is False: + return None + return str(value).strip() diff --git a/src/sqlbuild/compiler/migrations/main/deterministic_old_name_view_event_id.py b/src/sqlbuild/compiler/migrations/main/deterministic_old_name_view_event_id.py new file mode 100644 index 0000000000..c1e36b912d --- /dev/null +++ b/src/sqlbuild/compiler/migrations/main/deterministic_old_name_view_event_id.py @@ -0,0 +1,18 @@ +"""Build stable identifiers for immutable old-name view facts.""" + +from __future__ import annotations + +import hashlib +import json + +from sqlbuild.compiler.migrations.types import OldNameViewEventType + + +def deterministic_old_name_view_event_id( + *, event_type: OldNameViewEventType, migration_event_id: str +) -> str: + """Return the content-addressed ID of one old-name step of one recorded move.""" + + identity: tuple[object, ...] = ("old_name", event_type.value, migration_event_id) + encoded: str = json.dumps(identity, separators=(",", ":"), ensure_ascii=True) + return hashlib.sha256(encoded.encode()).hexdigest() diff --git a/src/sqlbuild/compiler/migrations/main/old_name_view_create_table_sql.py b/src/sqlbuild/compiler/migrations/main/old_name_view_create_table_sql.py new file mode 100644 index 0000000000..4539d6bf19 --- /dev/null +++ b/src/sqlbuild/compiler/migrations/main/old_name_view_create_table_sql.py @@ -0,0 +1,27 @@ +"""Old-name view state table SQL entrypoint.""" + +from __future__ import annotations + +from collections.abc import Callable + +from sqlbuild.adapter.contract.types import FrameworkType +from sqlbuild.compiler.migrations._helpers.old_name_sql import build_old_name_create_table_sql + + +def build_old_name_view_state_create_table_sql( + *, + database: str | None, + schema: str, + render_qualified_name: Callable[..., str | None], + render_framework_type: Callable[[FrameworkType], str], + transient: bool = False, +) -> str: + """Build DDL that creates the old-name view fact table when it is missing.""" + + return build_old_name_create_table_sql( + database=database, + schema=schema, + render_qualified_name=render_qualified_name, + render_framework_type=render_framework_type, + transient=transient, + ) diff --git a/src/sqlbuild/compiler/migrations/main/read_old_name_view_events.py b/src/sqlbuild/compiler/migrations/main/read_old_name_view_events.py new file mode 100644 index 0000000000..a08d199876 --- /dev/null +++ b/src/sqlbuild/compiler/migrations/main/read_old_name_view_events.py @@ -0,0 +1,48 @@ +"""Read append-only old-name view facts from one schema.""" + +from __future__ import annotations + +from collections.abc import Callable, Iterable +from typing import Any + +from sqlbuild.adapter.contract.types import AdapterExecute +from sqlbuild.compiler.migrations._helpers.old_name_sql import ( + build_old_name_read_sql, + decode_old_name_event_row, + readable_old_name_columns, +) +from sqlbuild.compiler.migrations.exceptions import MigrationStateError +from sqlbuild.compiler.migrations.models import OldNameViewEvent + + +def read_old_name_view_events( + *, + connection: Any, + execute: AdapterExecute[Any, Any], + database: str | None, + schema: str, + stored_columns: Iterable[str], + render_qualified_name: Callable[..., str | None], +) -> tuple[OldNameViewEvent, ...]: + """Read every old-name fact this version understands from the columns the table has.""" + + columns: tuple[str, ...] = readable_old_name_columns(stored_columns) + sql: str = build_old_name_read_sql( + database=database, + schema=schema, + columns=columns, + render_qualified_name=render_qualified_name, + ) + try: + rows: list[tuple[Any, ...]] = execute(connection=connection, sql=sql).fetchall() + except Exception as error: + raise MigrationStateError( + f"Unable to read old-name view facts from schema '{schema}': {error}" + ) from error + events: list[OldNameViewEvent] = [] + row: tuple[Any, ...] + for row in rows: + event: OldNameViewEvent | None = decode_old_name_event_row(row=tuple(row), columns=columns) + if event is not None: + events.append(event) + return tuple(events) diff --git a/src/sqlbuild/compiler/migrations/main/read_old_name_view_histories.py b/src/sqlbuild/compiler/migrations/main/read_old_name_view_histories.py new file mode 100644 index 0000000000..da9607a15c --- /dev/null +++ b/src/sqlbuild/compiler/migrations/main/read_old_name_view_histories.py @@ -0,0 +1,44 @@ +"""Read the old-name view history of every recorded move in one schema.""" + +from __future__ import annotations + +from collections.abc import Callable, Iterable +from typing import Any + +from sqlbuild.adapter.contract.types import AdapterExecute +from sqlbuild.compiler.migrations.main._project_old_name_views import project_old_name_views +from sqlbuild.compiler.migrations.main._read_events import read_migration_events +from sqlbuild.compiler.migrations.main.read_old_name_view_events import ( + read_old_name_view_events, +) +from sqlbuild.compiler.migrations.models import OldNameViewHistory + + +def read_old_name_view_histories( + *, + connection: Any, + execute: AdapterExecute[Any, Any], + database: str | None, + schema: str, + stored_columns: Iterable[str], + render_qualified_name: Callable[..., str | None], +) -> tuple[OldNameViewHistory, ...]: + """Read moves and old-name facts from a schema that holds both state tables.""" + + return project_old_name_views( + moves=read_migration_events( + connection=connection, + execute=execute, + database=database, + schema=schema, + render_qualified_name=render_qualified_name, + ), + facts=read_old_name_view_events( + connection=connection, + execute=execute, + database=database, + schema=schema, + stored_columns=stored_columns, + render_qualified_name=render_qualified_name, + ), + ) diff --git a/src/sqlbuild/compiler/migrations/main/read_recorded_column_renames.py b/src/sqlbuild/compiler/migrations/main/read_recorded_column_renames.py new file mode 100644 index 0000000000..74f4de2165 --- /dev/null +++ b/src/sqlbuild/compiler/migrations/main/read_recorded_column_renames.py @@ -0,0 +1,41 @@ +"""Read the column renames recorded on one relation since a point in time.""" + +from __future__ import annotations + +from collections.abc import Callable +from datetime import datetime +from typing import Any + +from sqlbuild.adapter.contract.types import AdapterExecute +from sqlbuild.compiler.migrations.main._read_column_events import read_column_migration_events +from sqlbuild.compiler.migrations.models import ColumnMigrationEvent, MigrationRelation + + +def read_recorded_column_renames( + *, + connection: Any, + execute: AdapterExecute[Any, Any], + relation: MigrationRelation, + since: datetime, + render_qualified_name: Callable[..., str | None], +) -> tuple[ColumnMigrationEvent, ...]: + """Return renames recorded on the relation at or after ``since``, oldest first.""" + + return tuple( + sorted( + ( + event + for event in read_column_migration_events( + connection=connection, + execute=execute, + database=relation.database, + schema=relation.schema or "", + render_qualified_name=render_qualified_name, + ) + if event.relation.matches(relation) + and event.decision.records_event + and event.created_at >= since + ), + key=lambda event: (event.created_at, event.event_id), + ) + ) diff --git a/src/sqlbuild/compiler/migrations/main/stored_old_name_view_columns.py b/src/sqlbuild/compiler/migrations/main/stored_old_name_view_columns.py new file mode 100644 index 0000000000..3fd7053a88 --- /dev/null +++ b/src/sqlbuild/compiler/migrations/main/stored_old_name_view_columns.py @@ -0,0 +1,21 @@ +"""Inspect which columns a stored old-name view state table has.""" + +from __future__ import annotations + +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.compiler.migrations.constants import OLD_NAME_VIEW_TABLE_NAME + + +def stored_old_name_view_columns( + *, adapter: BaseAdapter, connection: Any, database: str | None, schema: str +) -> tuple[str, ...]: + """Return the column names of one schema's old-name view state table.""" + + return tuple( + column.name + for column in adapter.get_columns( + connection=connection, database=database, schema=schema, name=OLD_NAME_VIEW_TABLE_NAME + ) + ) diff --git a/src/sqlbuild/compiler/migrations/main/write_old_name_view_event.py b/src/sqlbuild/compiler/migrations/main/write_old_name_view_event.py new file mode 100644 index 0000000000..8c143f2e1f --- /dev/null +++ b/src/sqlbuild/compiler/migrations/main/write_old_name_view_event.py @@ -0,0 +1,41 @@ +"""Append one old-name view fact idempotently.""" + +from __future__ import annotations + +from collections.abc import Callable +from typing import Any + +from sqlbuild.adapter.contract.types import AdapterExecute +from sqlbuild.compiler.migrations._helpers.old_name_sql import ( + build_old_name_existing_event_sql, + build_old_name_insert_sql, +) +from sqlbuild.compiler.migrations._helpers.writes import append_event_row +from sqlbuild.compiler.migrations.constants import MIGRATION_WRITE_ATTEMPTS +from sqlbuild.compiler.migrations.models import OldNameViewEvent + + +def write_old_name_view_event( + *, + connection: Any, + execute: AdapterExecute[Any, Any], + event: OldNameViewEvent, + render_qualified_name: Callable[..., str | None], + create_table_sql: str | None, + attempts: int = MIGRATION_WRITE_ATTEMPTS, +) -> None: + """Create the state table from adapter DDL when given, then insert the fact if absent.""" + + _ = append_event_row( + connection=connection, + execute=execute, + existing_sql=build_old_name_existing_event_sql( + event=event, render_qualified_name=render_qualified_name + ), + insert_sql=build_old_name_insert_sql( + event=event, render_qualified_name=render_qualified_name + ), + create_table_sql=create_table_sql, + attempts=attempts, + subject=f"{event.destination_model} old name {event.old.name}", + ) diff --git a/src/sqlbuild/compiler/migrations/models.py b/src/sqlbuild/compiler/migrations/models.py index 124fe17454..3ad791bf8e 100644 --- a/src/sqlbuild/compiler/migrations/models.py +++ b/src/sqlbuild/compiler/migrations/models.py @@ -9,6 +9,9 @@ ColumnMigrationDecision, MigrationDecision, MigrationDiscovery, + OldNameViewDropReason, + OldNameViewEventType, + OldNameViewStatus, ) @@ -91,3 +94,62 @@ def renamed(self, *, origin_column: str, destination_column: str) -> bool: self.origin_column.lower() == origin_column.lower() and self.destination_column.lower() == destination_column.lower() ) + + +@dataclass(frozen=True) +class OldNameViewEvent: + """One immutable fact about the old name of one recorded move.""" + + event_id: str + target_name: str | None + event_type: OldNameViewEventType + migration_event_id: str + destination_model: str + old: MigrationRelation + new: MigrationRelation + run_id: str + created_at: datetime + view_retention: str | None = None + archive_name: str | None = None + column_aliases: tuple[tuple[str, str], ...] = () + expires_at: datetime | None = None + drop_reason: OldNameViewDropReason | None = None + grants_copied: tuple[str, ...] | None = None + view_sql: str | None = None + + +@dataclass(frozen=True) +class OldNameViewHistory: + """The recorded move and every old-name fact that references it.""" + + move: MigrationEvent + required: OldNameViewEvent + archived: OldNameViewEvent | None = None + created: OldNameViewEvent | None = None + dropped: OldNameViewEvent | None = None + + def status(self, *, now: datetime) -> OldNameViewStatus: + """Project the current state of this move's old-name steps.""" + + if self.dropped is not None: + return OldNameViewStatus.DROPPED + if self.created is not None: + expires_at: datetime | None = self.created.expires_at + if expires_at is not None and expires_at <= now: + return OldNameViewStatus.EXPIRED + return OldNameViewStatus.LIVE + if self.archived is not None: + return OldNameViewStatus.PENDING_VIEW + return OldNameViewStatus.PENDING_ARCHIVE + + @property + def old(self) -> MigrationRelation: + """Return the relation name the compatibility view occupies.""" + + return self.required.old + + @property + def new(self) -> MigrationRelation: + """Return the relation the compatibility view reads.""" + + return self.required.new diff --git a/src/sqlbuild/compiler/migrations/types.py b/src/sqlbuild/compiler/migrations/types.py index 7e8fa7fbbb..56eba6ffd6 100644 --- a/src/sqlbuild/compiler/migrations/types.py +++ b/src/sqlbuild/compiler/migrations/types.py @@ -120,6 +120,42 @@ def label(self) -> str: return _COLUMN_DECISION_LABELS[self] +class OldNameViewEventType(StrEnum): + """Immutable facts about what happened at a migrated relation's old name.""" + + REQUIRED = "required" + ORIGIN_ARCHIVED = "origin_archived" + VIEW_CREATED = "view_created" + VIEW_DROPPED = "view_dropped" + + +class OldNameViewDropReason(StrEnum): + """Why a compatibility view at an old name was dropped.""" + + EXPIRED = "expired" + EARLY = "early" + MISSING = "missing" + + +class OldNameViewStatus(StrEnum): + """Projected state of the old-name steps of one recorded move.""" + + PENDING_ARCHIVE = "pending_archive" + PENDING_VIEW = "pending_view" + LIVE = "live" + EXPIRED = "expired" + DROPPED = "dropped" + + +class OldNameViewAction(StrEnum): + """What a build does at a migrated model's old name.""" + + ARCHIVE_AND_VIEW = "archive_and_view" + VIEW_ONLY = "view_only" + LIVE = "live" + NONE = "none" + + _RECORDING_COLUMN_DECISIONS: frozenset[ColumnMigrationDecision] = frozenset( {ColumnMigrationDecision.RENAME, ColumnMigrationDecision.RECORD} ) diff --git a/src/sqlbuild/compiler/planner/_helpers/migrations/display.py b/src/sqlbuild/compiler/planner/_helpers/migrations/display.py index bc06a199be..8982d3f1b9 100644 --- a/src/sqlbuild/compiler/planner/_helpers/migrations/display.py +++ b/src/sqlbuild/compiler/planner/_helpers/migrations/display.py @@ -9,15 +9,19 @@ MigrationCompatibility, MigrationDecision, MigrationDiscovery, + OldNameViewAction, ) from sqlbuild.compiler.planner.models import ( ColumnMigrationPlanEntry, ModelMigrationPlanEntry, + OldNameViewPlanEntry, PlanOutput, ) +from sqlbuild.compiler.planner.types import MaterializationType from sqlbuild.presentation.classes.cli_style import CliStyle from sqlbuild.presentation.main.append_overflow_line import append_overflow_line from sqlbuild.presentation.main.count_header import count_header_style +from sqlbuild.presentation.main.tree_connector import tree_connector from sqlbuild.presentation.main.visible_entries import visible_entries from sqlbuild.presentation.models import DisplayOptions @@ -32,23 +36,28 @@ def format_model_migrations( header_style: Callable[[str], str] = count_header_style( style=style, title_style=style.plan_section ) - title: str - renamed: bool - for title, renamed in (("Migrations", False), ("Renamed", True)): - entries: tuple[ModelMigrationPlanEntry, ...] = tuple( - entry - for entry in plan.migration_entries - if (entry.decision == MigrationDecision.RENAMED) is renamed - ) - if not entries: - continue - result.append(header_style(f"{title} ({len(entries)})")) + entries: tuple[ModelMigrationPlanEntry, ...] = plan.migration_entries + old_names: dict[str, OldNameViewPlanEntry] = { + entry.model_name: entry for entry in plan.old_name_view_entries + } + materializations: dict[str, str] = { + entry.name: str(entry.materialization_type) for entry in plan.model_entries + } + if entries: + result.append(header_style(f"Migrations ({len(entries)})")) visible: Sequence[ModelMigrationPlanEntry] = visible_entries( entries=entries, options=display_options ) entry: ModelMigrationPlanEntry for entry in visible: - result.extend(_migration_lines(entry=entry, style=style)) + result.extend( + _migration_lines( + entry=entry, + style=style, + old_name=old_names.get(entry.model_name), + materialization=materializations.get(entry.model_name), + ) + ) result = append_overflow_line( lines=result, total_count=len(entries), @@ -56,11 +65,85 @@ def format_model_migrations( indent=" ", options=display_options, ) + result = _format_resumed_old_names( + lines=result, + plan=plan, + style=style, + header_style=header_style, + ) return _format_column_migrations( lines=result, plan=plan, style=style, header_style=header_style ) +def _format_resumed_old_names( + *, + lines: list[str], + plan: PlanOutput, + style: CliStyle, + header_style: Callable[[str], str], +) -> list[str]: + """Format old-name steps that resume a move no longer listed as a migration.""" + + migrated: frozenset[str] = frozenset(entry.model_name for entry in plan.migration_entries) + resumed: tuple[OldNameViewPlanEntry, ...] = tuple( + entry for entry in plan.old_name_view_entries if entry.model_name not in migrated + ) + if not resumed: + return lines + result: list[str] = list(lines) + result.append(header_style(f"Old names ({len(resumed)})")) + entry: OldNameViewPlanEntry + for entry in resumed: + origin: str = entry.origin.qualified_name or entry.origin.name + destination: str = entry.destination.qualified_name or entry.destination.name + result.append( + f" {style.object_name(entry.model_name)} {style.muted(f'{origin} ->')} {destination}" + ) + result.extend(old_name_rows(entry=entry, style=style)) + return result + + +def old_name_rows(*, entry: OldNameViewPlanEntry, style: CliStyle) -> list[str]: + """Render the old name of a migration as a leaf with one fact per nested row.""" + + origin: str = entry.origin.qualified_name or entry.origin.name + facts: list[tuple[str, str]] = old_name_facts(entry) + rows: list[str] = [_property_row(label="old name", value=origin, style=style)] + index: int + label: str + value: str + for index, (label, value) in enumerate(facts): + connector: str = tree_connector(style=style, last=index == len(facts) - 1) + rows.append(f" {connector} {style.muted(label)} {value}") + return rows + + +def old_name_facts(entry: OldNameViewPlanEntry) -> list[tuple[str, str]]: + """Return what happens at a migrated model's old name, one labelled fact per row.""" + + if entry.action == OldNameViewAction.NONE: + return [("left for janitor", entry.reason or "no compatibility view")] + until: str = "" if entry.expires_at is None else f"until {entry.expires_at:%Y-%m-%d}" + view: str = f"live {until}".strip() if entry.action == OldNameViewAction.LIVE else until + facts: list[tuple[str, str]] = [("view", view)] + if entry.column_aliases: + facts.append(("columns", ", ".join(f"{old} <- {new}" for old, new in entry.column_aliases))) + if entry.grants_supported: + facts.append(("grants", _grants_text(entry))) + archived: str = ( + "already archived" if entry.action == OldNameViewAction.VIEW_ONLY else "archived" + ) + facts.append(("old table", archived)) + return facts + + +def _grants_text(entry: OldNameViewPlanEntry) -> str: + if entry.grants_copied is None: + return "copied from the old table" + return f"{entry.grants_copied} copied from the old table" + + def _format_column_migrations( *, lines: list[str], @@ -104,15 +187,31 @@ def _column_migration_text(*, entry: ColumnMigrationPlanEntry, style: CliStyle) return f"{rename} {decision}{suffix}" -def _migration_lines(*, entry: ModelMigrationPlanEntry, style: CliStyle) -> list[str]: +def _migration_lines( + *, + entry: ModelMigrationPlanEntry, + style: CliStyle, + old_name: OldNameViewPlanEntry | None, + materialization: str | None, +) -> list[str]: origin: str = entry.origin.qualified_name or entry.origin.name destination: str = entry.destination.qualified_name or entry.destination.name relation_move: str = f"{style.muted(f'{origin} ->')} {destination}" name: str = style.object_name(entry.model_name) - if entry.decision == MigrationDecision.RENAMED: - return [f" {name} {relation_move} {style.muted('(identity handed over)')}"] decision: str = _decision_text(decision=entry.decision, style=style) + if entry.decision == MigrationDecision.RENAMED: + decision = ( + style.muted(MigrationDecision.DONE.label) + if entry.completed_at is not None + else style.accent_strong(MigrationDecision.MIGRATE.label) + ) rows: list[str] = [f" {name} {decision} {relation_move}"] + if entry.decision == MigrationDecision.RENAMED and entry.completed_at is None: + rows.append( + _property_row( + label="transfer", value=identity_transfer_label(materialization), style=style + ) + ) if entry.decision.checks_compatibility: rows.append( _property_row( @@ -149,6 +248,8 @@ def _migration_lines(*, entry: ModelMigrationPlanEntry, style: CliStyle) -> list style=style, ) ) + if old_name is not None: + rows.extend(old_name_rows(entry=old_name, style=style)) blocking: bool = ( entry.decision.blocks_build or entry.compatibility == MigrationCompatibility.INCOMPATIBLE ) @@ -157,6 +258,14 @@ def _migration_lines(*, entry: ModelMigrationPlanEntry, style: CliStyle) -> list return rows +def identity_transfer_label(materialization: str | None) -> str: + """Describe how a renamed table or view reaches its new name.""" + + if materialization == MaterializationType.VIEW: + return "recreate (view)" + return "rebuild (table)" + + def _property_row(*, label: str, value: str, style: CliStyle) -> str: return f" {style.muted(label)} {value}" diff --git a/src/sqlbuild/compiler/planner/_helpers/migrations/old_names.py b/src/sqlbuild/compiler/planner/_helpers/migrations/old_names.py new file mode 100644 index 0000000000..791b703601 --- /dev/null +++ b/src/sqlbuild/compiler/planner/_helpers/migrations/old_names.py @@ -0,0 +1,595 @@ +"""Plan what a build does at migrated models' old names.""" + +from __future__ import annotations + +from collections.abc import Iterable +from datetime import datetime + +from sqlbuild.adapter.contract.models import RelationInfo +from sqlbuild.adapter.contract.types import RelationType +from sqlbuild.adapter.type_system.main.normalize_relation_type import normalize_relation_type +from sqlbuild.compiler.compile.models import CompiledModel, CompiledRelationLocation +from sqlbuild.compiler.compile.types import CompiledResourceType +from sqlbuild.compiler.migrations.main._resolve_old_name_view_retention import ( + resolve_old_name_view_retention, +) +from sqlbuild.compiler.migrations.main.relation_for_location import migration_relation_for_location +from sqlbuild.compiler.migrations.models import MigrationRelation, OldNameViewHistory +from sqlbuild.compiler.migrations.types import ( + ColumnMigrationDecision, + MigrationDecision, + OldNameViewAction, + OldNameViewStatus, +) +from sqlbuild.compiler.planner._helpers.migrations.planning import ( + planning_database, + project_schemas, +) +from sqlbuild.compiler.planner.classes.migration_state_inspection import ( + MigrationStateInspection, +) +from sqlbuild.compiler.planner.models import ( + ColumnMigrationPlanEntry, + ModelMigrationPlanEntry, + OldNameView, + OldNameViewPlanEntry, + OldNameViewPlanning, + PlannerRuntime, + PlannerScope, + PlanWarning, + WarehouseSnapshot, +) +from sqlbuild.compiler.planner.types import MaterializationType, WarningSeverity +from sqlbuild.compiler.references.main._old_name_reference_message import ( + old_name_reference_message, +) +from sqlbuild.compiler.references.main.hard_coded_relation_remedy import ( + hard_coded_relation_remedy, +) +from sqlbuild.compiler.references.main.match_project_relation import match_project_relation +from sqlbuild.compiler.references.models import ( + LiteralSqlRelation, + ProjectRelation, + ProjectRelationIndex, + RelationName, +) +from sqlbuild.cursor_algebra.models import Duration +from sqlbuild.python_nodes.models import SqlResourceRef +from sqlbuild.python_nodes.types import SqlResourceRefKind +from sqlbuild.spec.contracts.main.get_config_str import get_config_str +from sqlbuild.spec.contracts.models import MigrationsConfig, SchemaColumn + +_HISTORY_MATERIALIZATIONS: frozenset[str] = frozenset( + {MaterializationType.INCREMENTAL, MaterializationType.SNAPSHOT} +) +_ALIAS_DECISIONS: frozenset[ColumnMigrationDecision] = frozenset( + {ColumnMigrationDecision.RENAME, ColumnMigrationDecision.RECORD} +) +_RESUMED_STATUSES: frozenset[OldNameViewStatus] = frozenset( + {OldNameViewStatus.PENDING_ARCHIVE, OldNameViewStatus.PENDING_VIEW} +) +_PENDING_ACTIONS: dict[OldNameViewStatus, OldNameViewAction] = { + OldNameViewStatus.PENDING_ARCHIVE: OldNameViewAction.ARCHIVE_AND_VIEW, + OldNameViewStatus.PENDING_VIEW: OldNameViewAction.VIEW_ONLY, + OldNameViewStatus.LIVE: OldNameViewAction.LIVE, + OldNameViewStatus.EXPIRED: OldNameViewAction.LIVE, +} + + +def plan_old_name_views( + *, + runtime: PlannerRuntime, + scope: PlannerScope, + snapshot: WarehouseSnapshot, + migration_entries: tuple[ModelMigrationPlanEntry, ...], + column_entries: tuple[ColumnMigrationPlanEntry, ...], +) -> OldNameViewPlanning: + """Decide old-name steps for selected models and collect recorded compatibility views.""" + + models_by_name: dict[str, CompiledModel] = { + model.name: model for model in runtime.project.models + } + selected: tuple[CompiledModel, ...] = tuple( + models_by_name[key.name] + for key in scope.execution_order + if key in scope.selected_keys + and key.resource_type == CompiledResourceType.MODEL + and key.name in models_by_name + ) + state: MigrationStateInspection = MigrationStateInspection( + adapter=runtime.adapter, + connection=runtime.connection, + database=planning_database(runtime=runtime), + ) + schemas: set[str] = _schemas_to_read( + snapshot=snapshot, + needed=_needed_schemas( + runtime=runtime, + selected=selected, + snapshot=snapshot, + migration_entries=migration_entries, + ), + ) + if schemas: + state.inspect_old_name_views(schemas=schemas) + target_name: str | None = runtime.project.effective_target_name + histories: tuple[OldNameViewHistory, ...] = tuple( + history + for history in state.old_name_histories + if history.move.target_name in (None, target_name) + ) + views: tuple[OldNameView, ...] = tuple( + _recorded_view(runtime=runtime, history=history, models=runtime.project.models) + for history in histories + if history.created is not None and history.dropped is None + ) + planner: _OldNamePlanner = _OldNamePlanner( + runtime=runtime, + models_by_name=models_by_name, + histories=histories, + views=views, + column_entries=column_entries, + ) + entries: list[OldNameViewPlanEntry] = [] + covered: set[str] = set() + entry: ModelMigrationPlanEntry + for entry in migration_entries: + planned: OldNameViewPlanEntry | None = planner.for_migration(entry) + if planned is not None: + entries.append(planned) + covered.add(entry.model_name) + model: CompiledModel + for model in selected: + if model.name in covered: + continue + pending: OldNameViewPlanEntry | None = planner.pending_for(model) + if pending is not None: + entries.append(pending) + return OldNameViewPlanning( + entries=tuple(entries), + views=views, + warnings=( + *_claim_errors( + selected=selected, snapshot=snapshot, views=views, now=runtime.invocation_time + ), + *_old_name_reference_errors(runtime=runtime, views=views, entries=tuple(entries)), + ), + ) + + +class _OldNamePlanner: + """Per-plan inputs shared by every old-name decision.""" + + def __init__( + self, + *, + runtime: PlannerRuntime, + models_by_name: dict[str, CompiledModel], + histories: tuple[OldNameViewHistory, ...], + views: tuple[OldNameView, ...], + column_entries: tuple[ColumnMigrationPlanEntry, ...], + ) -> None: + self._runtime: PlannerRuntime = runtime + self._models_by_name: dict[str, CompiledModel] = models_by_name + self._histories: tuple[OldNameViewHistory, ...] = histories + self._views: tuple[OldNameView, ...] = views + self._column_entries: tuple[ColumnMigrationPlanEntry, ...] = column_entries + config: MigrationsConfig = ( + runtime.project_config.migrations + if runtime.project_config is not None + else MigrationsConfig() + ) + self._project_retention: str | None = config.old_name_views + self._now: datetime = runtime.invocation_time + + def for_migration(self, entry: ModelMigrationPlanEntry) -> OldNameViewPlanEntry | None: + model: CompiledModel | None = self._models_by_name.get(entry.model_name) + if model is None or entry.blocks_build: + return None + if _starts_move(entry): + return self._new_move(entry=entry, model=model) + if entry.decision == MigrationDecision.DONE or ( + entry.decision == MigrationDecision.RENAMED and entry.completed_at is not None + ): + history: OldNameViewHistory | None = self._newest_history( + destination=migration_relation_for_location(entry.destination), + origin=migration_relation_for_location(entry.origin), + ) + return None if history is None else self._from_history(history=history, model=model) + return None + + def pending_for(self, model: CompiledModel) -> OldNameViewPlanEntry | None: + history: OldNameViewHistory | None = self._newest_history( + destination=migration_relation_for_location(model.destination), origin=None + ) + if history is None or history.move.destination_model != model.name: + return None + planned: OldNameViewPlanEntry | None = self._from_history(history=history, model=model) + if planned is None or not (planned.runs_steps or _skips_pending(planned)): + return None + return planned + + def _new_move( + self, *, entry: ModelMigrationPlanEntry, model: CompiledModel + ) -> OldNameViewPlanEntry: + retention: str | None = resolve_old_name_view_retention( + config_values=model.config.values, project_retention=self._project_retention + ) + reason: str | None = self._skip_reason( + origin=entry.origin, origin_tracked=entry.origin_tracked, retention=retention + ) + if reason is not None or retention is None: + return _skipped_entry(entry=entry, reason=reason or "old_name_view false") + duration: Duration | None = Duration.parse(retention) + return OldNameViewPlanEntry( + model_name=entry.model_name, + origin=entry.origin, + destination=entry.destination, + action=OldNameViewAction.ARCHIVE_AND_VIEW, + target_name=entry.target_name, + retention=retention, + expires_at=None if duration is None else duration.add_to(self._now), + column_aliases=self._planned_aliases(model), + stores_history=_stores_history(model), + records_requirement=True, + grants_supported=self._runtime.adapter.relation_grants_supported, + ) + + def _skip_reason( + self, + *, + origin: CompiledRelationLocation, + origin_tracked: bool | None, + retention: str | None, + ) -> str | None: + """Return why the old name gets no steps; ``origin_tracked`` None means unknown.""" + + if retention is None: + return "old_name_view false" + old: MigrationRelation = migration_relation_for_location(origin) + reuser: CompiledModel | None = next( + ( + model + for model in self._models_by_name.values() + if migration_relation_for_location(model.destination).matches(old) + ), + None, + ) + if reuser is not None: + return f"name reused by model:{reuser.name}" + if any(migration_relation_for_location(view.old).matches(old) for view in self._views): + return "old name is a compatibility view" + if origin_tracked is False: + return "not built by SQLBuild" + return None + + def _from_history( + self, *, history: OldNameViewHistory, model: CompiledModel + ) -> OldNameViewPlanEntry | None: + """Resume a recorded move; its record proves SQLBuild owned the origin when it moved.""" + + status: OldNameViewStatus = history.status(now=self._now) + action: OldNameViewAction | None = _PENDING_ACTIONS.get(status) + if action is None: + return None + if status in _RESUMED_STATUSES: + reason: str | None = self._skip_reason( + origin=_location(runtime=self._runtime, relation=history.old), + origin_tracked=None, + retention=resolve_old_name_view_retention( + config_values=model.config.values, project_retention=self._project_retention + ), + ) + if reason is not None: + return _skipped_resume(runtime=self._runtime, history=history, reason=reason) + duration: Duration | None = Duration.parse(history.required.view_retention or "") + expires_at: datetime | None = ( + history.created.expires_at + if history.created is not None + else (None if duration is None else duration.add_to(self._now)) + ) + return OldNameViewPlanEntry( + model_name=model.name, + origin=_location(runtime=self._runtime, relation=history.old), + destination=_location(runtime=self._runtime, relation=history.new), + action=action, + target_name=history.move.target_name, + retention=history.required.view_retention, + expires_at=expires_at, + column_aliases=( + history.created.column_aliases + if history.created is not None + else self._planned_aliases(model) + ), + stores_history=_stores_history(model), + migration_event_id=history.move.event_id, + grants_copied=( + None + if history.created is None or history.created.grants_copied is None + else len(history.created.grants_copied) + ), + grants_supported=self._runtime.adapter.relation_grants_supported, + archived=history.archived is not None, + ) + + def _newest_history( + self, *, destination: MigrationRelation, origin: MigrationRelation | None + ) -> OldNameViewHistory | None: + matching: tuple[OldNameViewHistory, ...] = tuple( + history + for history in self._histories + if history.move.destination.matches(destination) + and (origin is None or history.move.origin.matches(origin)) + ) + if not matching: + return None + return max(matching, key=lambda item: (item.move.created_at, item.move.event_id)) + + def _planned_aliases(self, model: CompiledModel) -> tuple[tuple[str, str], ...]: + if _stores_history(model): + return tuple( + sorted( + (entry.origin_column, entry.destination_column) + for entry in self._column_entries + if entry.model_name == model.name and entry.decision in _ALIAS_DECISIONS + ) + ) + return declared_column_aliases(model) + + +def declared_column_aliases(model: CompiledModel) -> tuple[tuple[str, str], ...]: + """Return old-to-new column names a table or view model declares with migrate_from.""" + + columns: tuple[SchemaColumn, ...] = ( + model.schema_entry.columns if model.schema_entry is not None else () + ) + return tuple( + sorted( + (column.migrate_from.strip(), column.name) + for column in columns + if column.migrate_from is not None + ) + ) + + +def _starts_move(entry: ModelMigrationPlanEntry) -> bool: + if entry.blocks_build: + return False + return entry.decision.moves_data or ( + entry.decision == MigrationDecision.RENAMED and entry.completed_at is None + ) + + +def _skips_pending(entry: OldNameViewPlanEntry) -> bool: + return entry.action == OldNameViewAction.NONE and entry.migration_event_id is not None + + +def _skipped_resume( + *, runtime: PlannerRuntime, history: OldNameViewHistory, reason: str +) -> OldNameViewPlanEntry: + return OldNameViewPlanEntry( + model_name=history.move.destination_model, + origin=_location(runtime=runtime, relation=history.old), + destination=_location(runtime=runtime, relation=history.new), + action=OldNameViewAction.NONE, + target_name=history.move.target_name, + migration_event_id=history.move.event_id, + reason=reason, + ) + + +def _stores_history(model: CompiledModel) -> bool: + return ( + get_config_str(values=model.config.values, key="materialized") in _HISTORY_MATERIALIZATIONS + ) + + +def _skipped_entry(*, entry: ModelMigrationPlanEntry, reason: str) -> OldNameViewPlanEntry: + return OldNameViewPlanEntry( + model_name=entry.model_name, + origin=entry.origin, + destination=entry.destination, + action=OldNameViewAction.NONE, + target_name=entry.target_name, + reason=reason, + ) + + +def _needed_schemas( + *, + runtime: PlannerRuntime, + selected: tuple[CompiledModel, ...], + snapshot: WarehouseSnapshot, + migration_entries: tuple[ModelMigrationPlanEntry, ...], +) -> set[str]: + """Return schemas whose old-name facts this plan needs.""" + + schemas: set[str] = { + model.destination.schema for model in selected if model.destination.schema is not None + } + schemas.update( + entry.origin.schema for entry in migration_entries if entry.origin.schema is not None + ) + if runtime.project.unmatched_literal_sql_relations or any( + _may_claim_view(model=model, snapshot=snapshot) for model in selected + ): + schemas.update(project_schemas(runtime=runtime)) + return schemas + + +def _schemas_to_read(*, snapshot: WarehouseSnapshot, needed: set[str]) -> set[str]: + """Skip schemas the warehouse snapshot already listed without finding old-name state.""" + + listed: frozenset[str] | None = snapshot.listed_state_schemas + return { + schema + for schema in needed + if schema.lower() in snapshot.old_name_view_state_schemas + or (listed is not None and schema.lower() not in listed) + } + + +def _may_claim_view(*, model: CompiledModel, snapshot: WarehouseSnapshot) -> bool: + relation: RelationInfo | None = snapshot.existing_relations.get(model.name) + return relation is not None and _is_view(relation) + + +def _is_view(relation: RelationInfo) -> bool: + return normalize_relation_type(relation.relation_type) == RelationType.VIEW + + +def _claim_errors( + *, + selected: Iterable[CompiledModel], + snapshot: WarehouseSnapshot, + views: tuple[OldNameView, ...], + now: datetime, +) -> tuple[PlanWarning, ...]: + """Refuse a model whose destination is a compatibility view still recorded at that name.""" + + errors: list[PlanWarning] = [] + model: CompiledModel + for model in selected: + if not _may_claim_view(model=model, snapshot=snapshot): + continue + destination: MigrationRelation = migration_relation_for_location(model.destination) + view: OldNameView | None = next( + ( + view + for view in views + if view.destination_model != model.name + and migration_relation_for_location(view.old).matches(destination) + ), + None, + ) + if view is not None: + errors.append( + PlanWarning( + model_name=model.name, + severity=WarningSeverity.ERROR, + message=claim_message(model_name=model.name, view=view, now=now), + code="M114", + ) + ) + return tuple(errors) + + +def _old_name_reference_errors( + *, + runtime: PlannerRuntime, + views: tuple[OldNameView, ...], + entries: tuple[OldNameViewPlanEntry, ...], +) -> tuple[PlanWarning, ...]: + """Refuse literal Python SQL that reads a name kept only as a compatibility view.""" + + literals: tuple[LiteralSqlRelation, ...] = runtime.project.unmatched_literal_sql_relations + if not literals: + return () + index: ProjectRelationIndex = ProjectRelationIndex( + relations=tuple( + dict.fromkeys( + ( + *( + _old_name_relation(old=view.old, model_name=view.destination_model) + for view in views + ), + *( + _old_name_relation(old=entry.origin, model_name=entry.model_name) + for entry in entries + if entry.runs_steps + ), + ) + ) + ) + ) + errors: list[PlanWarning] = [] + literal: LiteralSqlRelation + for literal in literals: + match: ProjectRelation | None = match_project_relation( + index=index, relation=literal.relation + ) + if match is None or match.compatibility_for is None: + continue + written: str = ".".join( + part + for part in (literal.relation.database, literal.relation.schema, literal.relation.name) + if part + ) + errors.append( + PlanWarning( + model_name=None, + severity=WarningSeverity.ERROR, + message=( + f"{literal.relative_path}:{literal.line}: " + + old_name_reference_message( + owner_label=literal.owner_label, + written=written, + method=literal.method, + model_name=match.compatibility_for, + ) + + "; " + + hard_coded_relation_remedy(owner_kind=literal.owner_kind, ref=match.ref) + ), + code="P008", + ) + ) + return tuple(errors) + + +def _old_name_relation(*, old: CompiledRelationLocation, model_name: str) -> ProjectRelation: + return ProjectRelation( + ref=SqlResourceRef(kind=SqlResourceRefKind.MODEL, name=model_name), + relation=RelationName(name=old.name, schema=old.schema, database=old.database), + compatibility_for=model_name, + ) + + +def claim_message(*, model_name: str, view: OldNameView, now: datetime) -> str: + """Describe why a model cannot take a name held by a compatibility view.""" + + name: str = view.old.qualified_name or view.old.name + if view.expires_at is not None and view.expires_at <= now: + return ( + f"model '{model_name}' would replace {name}, a compatibility view for " + f"model:{view.destination_model} that expired {view.expires_at:%Y-%m-%d}; run " + "`sqb janitor` to drop it first" + ) + until: str = "" if view.expires_at is None else f" until {view.expires_at:%Y-%m-%d}" + return ( + f"model '{model_name}' would replace {name}, a compatibility view for " + f"model:{view.destination_model}{until}; drop it early with " + f"`sqb janitor --drop-old-name-view {name}`" + ) + + +def _recorded_view( + *, runtime: PlannerRuntime, history: OldNameViewHistory, models: Iterable[CompiledModel] +) -> OldNameView: + return OldNameView( + destination_model=history.move.destination_model, + old=_location(runtime=runtime, relation=history.old), + new=_location(runtime=runtime, relation=history.new), + expires_at=None if history.created is None else history.created.expires_at, + column_aliases=() if history.created is None else history.created.column_aliases, + migration_event_id=history.move.event_id, + target_name=history.move.target_name, + name_reused_by=next( + ( + model.name + for model in models + if migration_relation_for_location(model.destination).matches(history.old) + ), + None, + ), + ) + + +def _location(*, runtime: PlannerRuntime, relation: MigrationRelation) -> CompiledRelationLocation: + return CompiledRelationLocation( + database=relation.database, + schema=relation.schema, + name=relation.name, + qualified_name=runtime.adapter.render_qualified_name( + database=relation.database, schema=relation.schema, name=relation.name + ), + ) diff --git a/src/sqlbuild/compiler/planner/_helpers/migrations/planning.py b/src/sqlbuild/compiler/planner/_helpers/migrations/planning.py index 50f3af0a86..7508bebf58 100644 --- a/src/sqlbuild/compiler/planner/_helpers/migrations/planning.py +++ b/src/sqlbuild/compiler/planner/_helpers/migrations/planning.py @@ -65,6 +65,10 @@ from sqlbuild.spec.contracts.models import SnapshotsConfig from sqlbuild.spec.contracts.types import TableType +_IDENTITY_MATERIALIZATIONS: frozenset[str] = frozenset( + {MaterializationType.TABLE, MaterializationType.VIEW} +) + def manual_migration_requests(*, scope: PlannerScope) -> tuple[ModelMigrationRequest, ...]: """Collect selected models that declare migrate_from.""" @@ -92,6 +96,8 @@ def _declared_request(model: CompiledModel) -> ModelMigrationRequest | None: discovery=MigrationDiscovery.MANUAL, raw_origin=raw_origin.strip(), force=model.config.values.get(MIGRATE_FORCE_CONFIG_KEY) is True, + identity_only=get_config_str(values=model.config.values, key="materialized") + in _IDENTITY_MATERIALIZATIONS, ) @@ -416,6 +422,7 @@ def _decide( origin_fingerprint.version_hash if origin_fingerprint is not None else "" ), origin_is_transient=bool(origin_relation is not None and origin_relation.is_transient), + origin_tracked=origin_fingerprint is not None, ) if request.identity_only: return _decide_rename( @@ -424,6 +431,8 @@ def _decide( newest=newest, origin=origin, origin_fingerprint=origin_fingerprint, + origin_exists=origin_relation is not None, + manual=request.discovery == MigrationDiscovery.MANUAL, ) if ( newest is not None @@ -493,8 +502,10 @@ def _decide_rename( newest: MigrationEvent | None, origin: CompiledRelationLocation, origin_fingerprint: Fingerprint | None, + origin_exists: bool, + manual: bool, ) -> tuple[ModelMigrationPlanEntry, Fingerprint | None, bool]: - """Hand a renamed table or view's identity to its unbuilt successor; no data moves.""" + """Hand a renamed table or view's identity to its successor; no data moves.""" recorded: bool = ( newest is not None @@ -502,6 +513,8 @@ def _decide_rename( and newest.destination.matches(migration_relation_for_location(model.destination)) and newest.origin.matches(migration_relation_for_location(origin)) ) + if manual and not recorded and not origin_exists: + return replace(base, decision=MigrationDecision.ORIGIN_MISSING), None, False return ( replace( base, @@ -722,7 +735,9 @@ def _has_cursor(*, runtime: PlannerRuntime, model_name: str) -> bool: def _entry_warning(entry: ModelMigrationPlanEntry) -> PlanWarning | None: origin: str = entry.origin.qualified_name or entry.origin.name destination: str = entry.destination.qualified_name or entry.destination.name - if entry.decision == MigrationDecision.DONE: + if entry.decision == MigrationDecision.DONE or ( + entry.decision == MigrationDecision.RENAMED and entry.completed_at is not None + ): if entry.discovery != MigrationDiscovery.MANUAL: return None completed: str = entry.completed_at.isoformat() if entry.completed_at else "unknown time" diff --git a/src/sqlbuild/compiler/planner/_helpers/planning/output_assembly.py b/src/sqlbuild/compiler/planner/_helpers/planning/output_assembly.py index c8e7b4de8c..8189e2536d 100644 --- a/src/sqlbuild/compiler/planner/_helpers/planning/output_assembly.py +++ b/src/sqlbuild/compiler/planner/_helpers/planning/output_assembly.py @@ -95,6 +95,8 @@ def with_storage_policies( ), migration_entries=warehouse.migration_entries, column_migration_entries=warehouse.column_migration_entries, + old_name_view_entries=warehouse.old_name_view_entries, + old_name_views=warehouse.old_name_views, warnings=(*warehouse.migration_warnings, *plan_output.warnings), table_type_entries=plan_table_types( runtime=runtime, warehouse=warehouse, scope=scopes.selected_scope diff --git a/src/sqlbuild/compiler/planner/_helpers/planning/warehouse_state.py b/src/sqlbuild/compiler/planner/_helpers/planning/warehouse_state.py index 7ea4b50db9..4ac3230fa9 100644 --- a/src/sqlbuild/compiler/planner/_helpers/planning/warehouse_state.py +++ b/src/sqlbuild/compiler/planner/_helpers/planning/warehouse_state.py @@ -5,6 +5,7 @@ import time from sqlbuild.compiler.planner._helpers.migrations.columns import plan_column_migrations +from sqlbuild.compiler.planner._helpers.migrations.old_names import plan_old_name_views from sqlbuild.compiler.planner._helpers.migrations.planning import plan_model_migrations from sqlbuild.compiler.planner._helpers.output.plan_entry import build_planner_relations_context from sqlbuild.compiler.planner._helpers.planning.full_refresh import ( @@ -16,6 +17,7 @@ CursorSnapshotScope, DeferralInputs, ModelMigrationPlanning, + OldNameViewPlanning, PlannerOverrides, PlannerRelationsContext, PlannerRuntime, @@ -92,6 +94,13 @@ def gather_planner_warehouse_state( deferral=deferral, source_columns=inspection_relations.source_warehouse_columns, ) + old_names: OldNameViewPlanning = plan_old_name_views( + runtime=runtime, + scope=scopes.selected_scope, + snapshot=snapshot, + migration_entries=migrations.entries, + column_entries=columns.entries, + ) if runtime.on_progress is not None: runtime.on_progress( f"Inspected warehouse state. ({time.monotonic() - warehouse_start:.2f}s)" @@ -101,7 +110,9 @@ def gather_planner_warehouse_state( snapshot=columns.snapshot, inspection_relations=inspection_relations, migration_entries=migrations.entries, - migration_warnings=(*migrations.warnings, *columns.warnings), + migration_warnings=(*migrations.warnings, *columns.warnings, *old_names.warnings), column_migration_entries=columns.entries, column_rename_hints=columns.hints, + old_name_view_entries=old_names.entries, + old_name_views=old_names.views, ) diff --git a/src/sqlbuild/compiler/planner/_helpers/warehouse/snapshot.py b/src/sqlbuild/compiler/planner/_helpers/warehouse/snapshot.py index 311b16e4de..c1935082a9 100644 --- a/src/sqlbuild/compiler/planner/_helpers/warehouse/snapshot.py +++ b/src/sqlbuild/compiler/planner/_helpers/warehouse/snapshot.py @@ -33,6 +33,7 @@ from sqlbuild.compiler.fingerprints.main.read import read_latest_fingerprints from sqlbuild.compiler.fingerprints.models import Fingerprint, FingerprintSet from sqlbuild.compiler.graph.main.transitive_closure_many import transitive_closure_many +from sqlbuild.compiler.migrations.constants import OLD_NAME_VIEW_TABLE_NAME from sqlbuild.compiler.planner._helpers.graph.buildability import ( check_buildability, missing_upstream_message, @@ -159,6 +160,15 @@ class _CursorGatherInputs: existing_columns: dict[str, tuple[ColumnInfo, ...]] | None = None +@dataclass(frozen=True) +class _StateTableSchemas: + """Lower-cased schemas where each planner-read state table exists.""" + + fingerprints: frozenset[str] + source_freshness: frozenset[str] + old_name_views: frozenset[str] + + def build_warehouse_snapshot( *, project: CompiledProject, @@ -232,9 +242,8 @@ def gather_warehouse_snapshot( query_schemas: tuple[str, ...] | None = schemas or None relations: dict[str, RelationInfo] - fingerprint_state_schemas: frozenset[str] - freshness_state_schemas: frozenset[str] - relations, fingerprint_state_schemas, freshness_state_schemas = _gather_relations( + state_schemas: _StateTableSchemas + relations, state_schemas = _gather_relations( project=project, adapter=adapter, connection=connection, @@ -242,6 +251,8 @@ def gather_warehouse_snapshot( schemas=query_schemas, names=metadata_names, ) + fingerprint_state_schemas: frozenset[str] = state_schemas.fingerprints + freshness_state_schemas: frozenset[str] = state_schemas.source_freshness columns: dict[str, tuple[ColumnInfo, ...]] = _gather_columns( adapter=adapter, connection=connection, @@ -295,6 +306,12 @@ def gather_warehouse_snapshot( cursor_snapshots=cursor_snapshots, source_freshness_state_schemas=freshness_state_schemas, column_dialect=adapter.sql_analysis_dialect(), + old_name_view_state_schemas=state_schemas.old_name_views, + listed_state_schemas=( + frozenset(schema.lower() for schema in query_schemas) + if query_schemas is not None + else None + ), ) @@ -471,6 +488,7 @@ def _build_metadata_name_filter( return None names.add(FINGERPRINT_TABLE_NAME) names.add(SOURCE_FRESHNESS_TABLE_NAME) + names.add(OLD_NAME_VIEW_TABLE_NAME) return tuple(sorted(names)) @@ -514,8 +532,8 @@ def _gather_relations( database: str | None, schemas: tuple[str, ...] | None, names: tuple[str, ...] | None, -) -> tuple[dict[str, RelationInfo], frozenset[str], frozenset[str]]: - """Fetch relations and the schemas where the fingerprint/freshness state tables exist.""" +) -> tuple[dict[str, RelationInfo], _StateTableSchemas]: + """Fetch relations and the schemas where fingerprint, freshness and old-name state live.""" relations: tuple[RelationInfo, ...] = adapter.list_relations( connection=connection, database=database, schemas=schemas, names=names @@ -533,8 +551,13 @@ def _gather_relations( logical_names_by_identity[_location_identity(function.destination)] = function.name fingerprint_schemas: set[str] = set() freshness_schemas: set[str] = set() + old_name_schemas: set[str] = set() relation: RelationInfo for relation in relations: + if relation.name == OLD_NAME_VIEW_TABLE_NAME: + if relation.schema is not None: + old_name_schemas.add(relation.schema.lower()) + continue if relation.name == FINGERPRINT_TABLE_NAME: if relation.schema is not None: fingerprint_schemas.add(relation.schema.lower()) @@ -544,7 +567,11 @@ def _gather_relations( freshness_schemas.add(relation.schema.lower()) continue result[logical_names_by_identity.get(relation.identity, relation.name)] = relation - return result, frozenset(fingerprint_schemas), frozenset(freshness_schemas) + return result, _StateTableSchemas( + fingerprints=frozenset(fingerprint_schemas), + source_freshness=frozenset(freshness_schemas), + old_name_views=frozenset(old_name_schemas), + ) def _location_identity( diff --git a/src/sqlbuild/compiler/planner/classes/migration_state_inspection.py b/src/sqlbuild/compiler/planner/classes/migration_state_inspection.py index bd4161c5d7..3824981075 100644 --- a/src/sqlbuild/compiler/planner/classes/migration_state_inspection.py +++ b/src/sqlbuild/compiler/planner/classes/migration_state_inspection.py @@ -13,13 +13,23 @@ from sqlbuild.compiler.migrations.constants import ( COLUMN_MIGRATION_TABLE_NAME, MIGRATION_TABLE_NAME, + OLD_NAME_VIEW_TABLE_NAME, ) +from sqlbuild.compiler.migrations.main._project_old_name_views import project_old_name_views from sqlbuild.compiler.migrations.main._read_column_events import read_column_migration_events from sqlbuild.compiler.migrations.main._read_events import read_migration_events +from sqlbuild.compiler.migrations.main.read_old_name_view_events import ( + read_old_name_view_events, +) +from sqlbuild.compiler.migrations.main.stored_old_name_view_columns import ( + stored_old_name_view_columns, +) from sqlbuild.compiler.migrations.models import ( ColumnMigrationEvent, MigrationEvent, MigrationRelation, + OldNameViewEvent, + OldNameViewHistory, ) @@ -37,6 +47,9 @@ def __init__(self, *, adapter: BaseAdapter, connection: Any, database: str | Non self._columns: dict[tuple[str, str], tuple[ColumnInfo, ...]] = {} self._column_event_schemas: set[str] = set() self._column_events: list[ColumnMigrationEvent] = [] + self._old_name_schemas: set[str] = set() + self._old_name_moves: list[MigrationEvent] = [] + self._old_name_facts: list[OldNameViewEvent] = [] @property def database(self) -> str | None: @@ -89,6 +102,62 @@ def inspect_column_events(self, *, schemas: set[str]) -> None: ) ) + @property + def old_name_histories(self) -> tuple[OldNameViewHistory, ...]: + """Return the old-name history of every move read by ``inspect_old_name_views``.""" + + return project_old_name_views(moves=self._old_name_moves, facts=self._old_name_facts) + + def inspect_old_name_views(self, *, schemas: set[str]) -> None: + """Read moves and old-name facts once for every schema not inspected yet.""" + + pending: tuple[str, ...] = tuple( + sorted(schema for schema in schemas if schema.lower() not in self._old_name_schemas) + ) + if not pending: + return + self._old_name_schemas.update(schema.lower() for schema in pending) + listed: tuple[RelationInfo, ...] = self._adapter.list_relations( + connection=self._connection, + database=self._database, + schemas=pending, + names=(MIGRATION_TABLE_NAME, OLD_NAME_VIEW_TABLE_NAME), + ) + state_tables: set[tuple[str, str]] = { + ((relation.schema or "").lower(), relation.name.lower()) for relation in listed + } + schema: str + for schema in pending: + if (schema.lower(), OLD_NAME_VIEW_TABLE_NAME) not in state_tables or ( + schema.lower(), + MIGRATION_TABLE_NAME, + ) not in state_tables: + continue + self._old_name_facts.extend( + read_old_name_view_events( + connection=self._connection, + execute=self._adapter.execute, + database=self._database, + schema=schema, + stored_columns=stored_old_name_view_columns( + adapter=self._adapter, + connection=self._connection, + database=self._database, + schema=schema, + ), + render_qualified_name=self._adapter.render_qualified_name, + ) + ) + self._old_name_moves.extend( + read_migration_events( + connection=self._connection, + execute=self._adapter.execute, + database=self._database, + schema=schema, + render_qualified_name=self._adapter.render_qualified_name, + ) + ) + def inspect_schemas(self, *, schemas: set[str]) -> None: """Read state tables once for every schema not inspected yet.""" diff --git a/src/sqlbuild/compiler/planner/models.py b/src/sqlbuild/compiler/planner/models.py index 98ac8acc53..b7d9aa9678 100644 --- a/src/sqlbuild/compiler/planner/models.py +++ b/src/sqlbuild/compiler/planner/models.py @@ -37,6 +37,7 @@ MigrationDecision, MigrationDiscovery, MigrationPromotion, + OldNameViewAction, ) from sqlbuild.compiler.planner.exceptions import PlannerInputError from sqlbuild.compiler.planner.types import ( @@ -626,6 +627,8 @@ class WarehouseSnapshot: source_freshness_state_schemas: frozenset[str] = field(default_factory=frozenset) column_dialect: str | None = None renamed_models: frozenset[str] = field(default_factory=frozenset) + old_name_view_state_schemas: frozenset[str] = field(default_factory=frozenset) + listed_state_schemas: frozenset[str] | None = field(default_factory=frozenset) @dataclass(frozen=True) @@ -920,6 +923,7 @@ class ModelMigrationPlanEntry: completed_at: datetime | None = None compatibility_findings: tuple[str, ...] = () message: str | None = None + origin_tracked: bool = False @property def storage_transition(self) -> str | None: @@ -945,6 +949,47 @@ def blocks_build(self) -> bool: ) +@dataclass(frozen=True) +class OldNameViewPlanEntry: + """What one build does at the old name of one migrated model.""" + + model_name: str + origin: CompiledRelationLocation + destination: CompiledRelationLocation + action: OldNameViewAction + target_name: str | None + retention: str | None = None + expires_at: datetime | None = None + column_aliases: tuple[tuple[str, str], ...] = () + stores_history: bool = False + migration_event_id: str | None = None + records_requirement: bool = False + reason: str | None = None + grants_copied: int | None = None + grants_supported: bool = False + archived: bool = False + + @property + def runs_steps(self) -> bool: + """Return whether this build archives the old relation or creates the view.""" + + return self.action in (OldNameViewAction.ARCHIVE_AND_VIEW, OldNameViewAction.VIEW_ONLY) + + +@dataclass(frozen=True) +class OldNameView: + """One compatibility view recorded at an old name and not recorded as dropped.""" + + destination_model: str + old: CompiledRelationLocation + new: CompiledRelationLocation + expires_at: datetime | None + column_aliases: tuple[tuple[str, str], ...] + migration_event_id: str + target_name: str | None + name_reused_by: str | None = None + + @dataclass(frozen=True) class ColumnMigrationPlanEntry: """One declared or detected in-place column rename and its per-run decision.""" @@ -1426,6 +1471,8 @@ class PlanOutput: table_type_entries: tuple[TableTypePlanEntry, ...] = field(default_factory=tuple) migration_entries: tuple[ModelMigrationPlanEntry, ...] = field(default_factory=tuple) column_migration_entries: tuple[ColumnMigrationPlanEntry, ...] = field(default_factory=tuple) + old_name_view_entries: tuple[OldNameViewPlanEntry, ...] = field(default_factory=tuple) + old_name_views: tuple[OldNameView, ...] = field(default_factory=tuple) upstream_deps: dict[CompiledObjectKey, tuple[CompiledObjectKey, ...]] = field( default_factory=dict ) @@ -1525,6 +1572,8 @@ class PlannerWarehouseState: migration_warnings: tuple[PlanWarning, ...] = () column_migration_entries: tuple[ColumnMigrationPlanEntry, ...] = () column_rename_hints: tuple[ColumnRenameHint, ...] = () + old_name_view_entries: tuple[OldNameViewPlanEntry, ...] = () + old_name_views: tuple[OldNameView, ...] = () @dataclass(frozen=True) @@ -1575,6 +1624,15 @@ class ModelMigrationPlanning: warnings: tuple[PlanWarning, ...] = () +@dataclass(frozen=True) +class OldNameViewPlanning: + """Old-name steps for this build plus every compatibility view still recorded.""" + + entries: tuple[OldNameViewPlanEntry, ...] = () + views: tuple[OldNameView, ...] = () + warnings: tuple[PlanWarning, ...] = () + + @dataclass(frozen=True) class ColumnMigrationPlanning: """Column rename decisions plus the warehouse snapshot as it will look after renames.""" diff --git a/src/sqlbuild/compiler/references/_helpers/project_relations.py b/src/sqlbuild/compiler/references/_helpers/project_relations.py index ef51c8de91..d634ef504a 100644 --- a/src/sqlbuild/compiler/references/_helpers/project_relations.py +++ b/src/sqlbuild/compiler/references/_helpers/project_relations.py @@ -4,13 +4,26 @@ from collections.abc import Iterable, Mapping -from sqlbuild.compiler.compile.models import CompiledProject, CompiledRelationLocation -from sqlbuild.compiler.references._helpers.relation_names import extract_relation_names_impl +from sqlbuild.compiler.compile.constants import MIGRATE_FROM_CONFIG_KEY +from sqlbuild.compiler.compile.models import ( + CompiledModel, + CompiledProject, + CompiledRelationLocation, +) +from sqlbuild.compiler.migrations.main._resolve_old_name_view_retention import ( + resolve_old_name_view_retention, +) +from sqlbuild.compiler.references._helpers.relation_names import ( + extract_relation_names_impl, + match_project_relation_impl, +) from sqlbuild.compiler.references.models import ProjectRelation, ProjectRelationIndex, RelationName from sqlbuild.python_nodes.models import SqlResourceRef from sqlbuild.python_nodes.types import SqlResourceRefKind from sqlbuild.spec.contracts.models import SourceEntry +_SCHEMA_QUALIFIED_PARTS: int = 2 + def build_project_relation_index_impl( relations: Iterable[ProjectRelation], @@ -20,8 +33,10 @@ def build_project_relation_index_impl( return ProjectRelationIndex(relations=tuple(dict.fromkeys(relations))) -def compiled_project_relations_impl(project: CompiledProject) -> ProjectRelationIndex: - """Return the project relations a compiled project declares.""" +def compiled_project_relations_impl( + *, project: CompiledProject, old_name_retention: str | None +) -> ProjectRelationIndex: + """Return the project relations a compiled project declares, with declared old names.""" relations: list[ProjectRelation] = [ _location_relation( @@ -49,7 +64,59 @@ def compiled_project_relations_impl(project: CompiledProject) -> ProjectRelation ), ) ) - return build_project_relation_index_impl(relations) + declared: ProjectRelationIndex = build_project_relation_index_impl(relations) + return build_project_relation_index_impl( + ( + *declared.relations, + *_declared_old_names( + project=project, declared=declared, old_name_retention=old_name_retention + ), + ) + ) + + +def _declared_old_names( + *, project: CompiledProject, declared: ProjectRelationIndex, old_name_retention: str | None +) -> tuple[ProjectRelation, ...]: + """Return old names that migrate_from declarations keep only as compatibility views.""" + + model_names: frozenset[str] = frozenset(model.name for model in project.models) + old_names: list[ProjectRelation] = [] + model: CompiledModel + for model in project.models: + raw: object = model.config.values.get(MIGRATE_FROM_CONFIG_KEY) + if not isinstance(raw, str) or ( + resolve_old_name_view_retention( + config_values=model.config.values, project_retention=old_name_retention + ) + is None + ): + continue + parts: list[str] = [part.strip().strip('"`') for part in raw.split(".")] + if len(parts) == 1 and parts[0] in model_names: + continue + relation: RelationName = RelationName( + name=parts[-1], + schema=parts[-2] if len(parts) > 1 else model.destination.schema, + database=parts[-3] + if len(parts) > _SCHEMA_QUALIFIED_PARTS + else model.destination.database, + ) + if ( + match_project_relation_impl( + index=declared, relation=relation, default_database=None, default_schema=None + ) + is not None + ): + continue + old_names.append( + ProjectRelation( + ref=SqlResourceRef(kind=SqlResourceRefKind.MODEL, name=model.name), + relation=relation, + compatibility_for=model.name, + ) + ) + return tuple(old_names) def runtime_project_relations_impl( diff --git a/src/sqlbuild/compiler/references/main/_compiled_project_relations.py b/src/sqlbuild/compiler/references/main/_compiled_project_relations.py index c4596c1288..317abacd55 100644 --- a/src/sqlbuild/compiler/references/main/_compiled_project_relations.py +++ b/src/sqlbuild/compiler/references/main/_compiled_project_relations.py @@ -7,7 +7,9 @@ from sqlbuild.compiler.references.models import ProjectRelationIndex -def compiled_project_relations(*, project: CompiledProject) -> ProjectRelationIndex: - """Return the project relations a compiled project declares.""" +def compiled_project_relations( + *, project: CompiledProject, old_name_retention: str | None +) -> ProjectRelationIndex: + """Return the project relations a compiled project declares, with declared old names.""" - return compiled_project_relations_impl(project) + return compiled_project_relations_impl(project=project, old_name_retention=old_name_retention) diff --git a/src/sqlbuild/compiler/references/main/_old_name_reference_message.py b/src/sqlbuild/compiler/references/main/_old_name_reference_message.py new file mode 100644 index 0000000000..ea7d3df620 --- /dev/null +++ b/src/sqlbuild/compiler/references/main/_old_name_reference_message.py @@ -0,0 +1,15 @@ +"""Describe literal SQL that still reads a renamed model through its old name.""" + +from __future__ import annotations + + +def old_name_reference_message( + *, owner_label: str, written: str, method: str, model_name: str +) -> str: + """Return the P008 message for SQL naming a relation kept only as a compatibility view.""" + + return ( + f"{owner_label} names '{written}' in SQL passed to ctx.{method}(); it is the old name " + f"of model:{model_name}, kept only as a compatibility view for consumers outside the " + "project" + ) diff --git a/src/sqlbuild/compiler/references/models.py b/src/sqlbuild/compiler/references/models.py index 3945155419..957eaa2491 100644 --- a/src/sqlbuild/compiler/references/models.py +++ b/src/sqlbuild/compiler/references/models.py @@ -3,7 +3,9 @@ from __future__ import annotations from dataclasses import dataclass, field +from pathlib import Path +from sqlbuild.compiler.references.types import HardCodedRelationOwnerKind from sqlbuild.python_nodes.models import SqlResourceRef @@ -22,6 +24,7 @@ class ProjectRelation: ref: SqlResourceRef relation: RelationName + compatibility_for: str | None = None @dataclass(frozen=True) @@ -29,3 +32,15 @@ class ProjectRelationIndex: """Every relation name that counts as a project relation for hard-coded name checks.""" relations: tuple[ProjectRelation, ...] = field(default_factory=tuple) + + +@dataclass(frozen=True) +class LiteralSqlRelation: + """A relation named in literal SQL in Python code that matched no project relation.""" + + owner_label: str + owner_kind: HardCodedRelationOwnerKind + relative_path: Path + line: int + method: str + relation: RelationName diff --git a/src/sqlbuild/executor/build/classes/build_scheduler.py b/src/sqlbuild/executor/build/classes/build_scheduler.py index f4ed3cf3c4..49e8dfc4cc 100644 --- a/src/sqlbuild/executor/build/classes/build_scheduler.py +++ b/src/sqlbuild/executor/build/classes/build_scheduler.py @@ -88,10 +88,15 @@ from sqlbuild.executor.load.main._resource_kind import load_resource_kind from sqlbuild.executor.load.main._skipped_result import skipped_load_result from sqlbuild.executor.load.models import LoadExecutionResult +from sqlbuild.executor.migrations.main._bound_old_name_view_guard import ( + bound_old_name_view_guard, +) +from sqlbuild.executor.migrations.main._complete_old_name_views import complete_old_name_views from sqlbuild.executor.python_nodes.types import PythonIdentityRecorder from sqlbuild.executor.run.classes.runtime_watermark_resolver import RuntimeWatermarkResolver from sqlbuild.executor.run.models import ( BatchWindow, + BoundViewGuard, MicrobatchPhaseOutcome, ModelExecutionResult, ModelMaterializationContext, @@ -1040,54 +1045,59 @@ def _execute_model_node( ), entry=model_entry, ) - result: ModelExecutionResult = _dispatch_model( - context=ModelMaterializationContext( - entry=model_entry, - adapter=self._adapter, - connection=connection, - model_locations=self._plan.model_locations, - seed_locations=self._plan.seed_locations, - source_map=self._plan.source_map, - python_source_read_map=self._plan.python_source_entries, - model_audits=model_audits, - run_id=self._run_id, - query_change_tracking=self._query_change_tracking, - schema_prepared=self._schema_prepared, - hook_functions=self._plan.hook_functions, - enforce_explicit_references=self._plan.enforce_explicit_references, - effective_target_name=self._target, - effective_vars=self._effective_vars, - providers=self._providers, - python_identity_recorder=self._python_identity_recorder, - microbatch_event_store=microbatch_event_store, - microbatch_event_store_resolver=microbatch_event_store_resolver, - microbatch_scope=microbatch_scope, - microbatch_model_version_hash=model_entry.fingerprint_version_hash, - microbatch_unaccounted_partition_policy=( - model_entry.unaccounted_partition_policy - or self._runtime.microbatch_unaccounted_partition_policy + bound_views: BoundViewGuard = bound_old_name_view_guard( + plan=self._plan, + adapter=self._adapter, + connection=connection, + destination=model_entry.destination, + ) + result: ModelExecutionResult = self._dispatch_keeping_old_names( + model_entry=model_entry, + connection=connection, + dispatch=lambda: _dispatch_model( + context=ModelMaterializationContext( + entry=model_entry, + adapter=self._adapter, + connection=connection, + model_locations=self._plan.model_locations, + seed_locations=self._plan.seed_locations, + source_map=self._plan.source_map, + python_source_read_map=self._plan.python_source_entries, + model_audits=model_audits, + run_id=self._run_id, + query_change_tracking=self._query_change_tracking, + schema_prepared=self._schema_prepared, + hook_functions=self._plan.hook_functions, + enforce_explicit_references=self._plan.enforce_explicit_references, + effective_target_name=self._target, + effective_vars=self._effective_vars, + providers=self._providers, + python_identity_recorder=self._python_identity_recorder, + microbatch_event_store=microbatch_event_store, + microbatch_event_store_resolver=microbatch_event_store_resolver, + microbatch_scope=microbatch_scope, + microbatch_model_version_hash=model_entry.fingerprint_version_hash, + microbatch_unaccounted_partition_policy=( + model_entry.unaccounted_partition_policy + or self._runtime.microbatch_unaccounted_partition_policy + ), + microbatch_lease_check=self._runtime.microbatch_lease_check, + microbatch_global_concurrency=self._max_concurrency, + microbatch_batch_runner=microbatch_batch_runner, + watermark_resolver=self._watermark_resolver, + bound_views=bound_views, ), - microbatch_lease_check=self._runtime.microbatch_lease_check, - microbatch_global_concurrency=self._max_concurrency, - microbatch_batch_runner=microbatch_batch_runner, - watermark_resolver=self._watermark_resolver, + promotion_mode=self._promotion_mode, + snapshots=self._snapshots, + allow_snapshot_schema_change=self._allow_snapshot_schema_change, + custom_materializations=self._custom_materializations, + target=self._target, + effective_vars=self._effective_vars, + warehouse_relations=self._warehouse_relations, + on_progress=self._on_sub_progress, ), - promotion_mode=self._promotion_mode, - snapshots=self._snapshots, - allow_snapshot_schema_change=self._allow_snapshot_schema_change, - custom_materializations=self._custom_materializations, - target=self._target, - effective_vars=self._effective_vars, - warehouse_relations=self._warehouse_relations, - on_progress=self._on_sub_progress, ) if result.status == ExecutionStatus.SUCCESS: - reconcile_model_retention( - plan=self._plan, - adapter=self._adapter, - connection=connection, - model_name=model_entry.name, - ) if microbatch_event_store is not None and microbatch_scope is not None: result = dataclasses.replace( result, @@ -1115,6 +1125,45 @@ def _execute_model_node( ) return completed_result + def _dispatch_keeping_old_names( + self, + *, + model_entry: ModelPlanEntry, + connection: Any, + dispatch: Callable[[], ModelExecutionResult], + ) -> ModelExecutionResult: + """Materialize, reconcile retention, then keep old names working after the build.""" + + result: ModelExecutionResult = dispatch() + if result.status != ExecutionStatus.SUCCESS: + return result + reconcile_model_retention( + plan=self._plan, + adapter=self._adapter, + connection=connection, + model_name=model_entry.name, + ) + try: + warnings: tuple[str, ...] = complete_old_name_views( + plan=self._plan, + adapter=self._adapter, + connection=connection, + model_name=model_entry.name, + destination=model_entry.destination, + run_id=self._run_id, + ) + except ExecutorInputError as error: + return dataclasses.replace( + result, + status=ExecutionStatus.FAILED, + error_code=error.code, + error_help=error.help, + error_message=error.message, + ) + if not warnings: + return result + return dataclasses.replace(result, warning_messages=(*result.warning_messages, *warnings)) + def _resolve_microbatch_state( self, *, model_entry: ModelPlanEntry, connection: Any ) -> tuple[MicrobatchEventStore, MicrobatchScope]: diff --git a/src/sqlbuild/executor/janitor/constants.py b/src/sqlbuild/executor/janitor/constants.py index d56339d28f..2d9db78951 100644 --- a/src/sqlbuild/executor/janitor/constants.py +++ b/src/sqlbuild/executor/janitor/constants.py @@ -6,6 +6,7 @@ from sqlbuild.compiler.migrations.constants import ( COLUMN_MIGRATION_TABLE_NAME, MIGRATION_TABLE_NAME, + OLD_NAME_VIEW_TABLE_NAME, ) from sqlbuild.compiler.source_freshness.constants import SOURCE_FRESHNESS_TABLE_NAME from sqlbuild.executor.audit_results.constants import AUDIT_RESULTS_TABLE_NAME @@ -22,6 +23,7 @@ JANITOR_EVENTS_TABLE_NAME, MIGRATION_TABLE_NAME, COLUMN_MIGRATION_TABLE_NAME, + OLD_NAME_VIEW_TABLE_NAME, ) DIRECT_DEFAULT_RETENTION_DAYS: int = 14 diff --git a/src/sqlbuild/executor/janitor/main/execute.py b/src/sqlbuild/executor/janitor/main/execute.py index f12bb5b6a2..e94b82ee62 100644 --- a/src/sqlbuild/executor/janitor/main/execute.py +++ b/src/sqlbuild/executor/janitor/main/execute.py @@ -17,9 +17,11 @@ JanitorDeleteCandidate, JanitorDirectStatePruneCandidate, JanitorExecutionResult, + JanitorOldNameView, JanitorPlan, JanitorQueryDiffArtifactCandidate, ) +from sqlbuild.executor.old_name_views.main._drop_old_name_views import drop_old_name_views from sqlbuild.runtime.observability.classes.operation_lifecycle import OperationLifecycle @@ -73,6 +75,13 @@ def _execute_janitor_plan( if plan.direct_mode else ((), ()) ) + dropped_old_name_views: tuple[JanitorOldNameView, ...] = drop_old_name_views( + plan=plan.old_name_views, + adapter=adapter, + connection=connection, + recorder=recorder, + run_id=janitor_run_id(), + ) query_artifact_candidate: JanitorQueryDiffArtifactCandidate for query_artifact_candidate in plan.query_diff_artifact_candidates: with OperationLifecycle(operation_kind="janitor", operation_name="janitor_cleanup_action"): @@ -103,6 +112,7 @@ def _execute_janitor_plan( deleted_archives=deleted_archives, deleted_query_diff_artifacts=plan.query_diff_artifact_candidates, pruned_direct_state=pruned_direct_state, + dropped_old_name_views=dropped_old_name_views, ) @@ -111,4 +121,9 @@ def _janitor_action_count(plan: JanitorPlan) -> int: relation_count += len(plan.query_diff_artifact_candidates) if plan.direct_mode: relation_count += len(plan.archive_candidates) + len(plan.archive_deletion_candidates) - return relation_count + len(plan.direct_state_prune_candidates) + return ( + relation_count + + len(plan.direct_state_prune_candidates) + + len(plan.old_name_views.drops) + + len(plan.old_name_views.missing) + ) diff --git a/src/sqlbuild/executor/janitor/main/plan.py b/src/sqlbuild/executor/janitor/main/plan.py index 5d744c433e..0550f41374 100644 --- a/src/sqlbuild/executor/janitor/main/plan.py +++ b/src/sqlbuild/executor/janitor/main/plan.py @@ -21,11 +21,15 @@ JanitorArchivePlanning, JanitorDirectModeSettings, JanitorDirectStatePruneCandidate, + JanitorOldNameViewPlanning, JanitorPlan, JanitorRelationScope, JanitorSchemaClassification, JanitorWarehouseFacts, ) +from sqlbuild.executor.old_name_views.main._plan_old_name_view_cleanup import ( + plan_old_name_view_cleanup, +) from sqlbuild.runtime.observability.classes.operation_lifecycle import OperationLifecycle @@ -39,6 +43,7 @@ def build_janitor_plan( exclude_patterns: tuple[str, ...] = (), relation_scope: JanitorRelationScope | None = None, direct_settings: JanitorDirectModeSettings | None = None, + early_old_name_view_drops: tuple[str, ...] = (), ) -> JanitorPlan: """Build a desired-vs-warehouse cleanup plan for target schemas.""" @@ -101,6 +106,18 @@ def build_janitor_plan( ) ) inspection.completed(metadata={"item_count": len(target_schemas)}) + old_names: JanitorOldNameViewPlanning + old_names, scope = plan_old_name_view_cleanup( + adapter=adapter, + connection=connection, + managed_target_schemas=managed_target_schemas, + relations_by_schema=facts.relations_by_schema, + target_name=project.effective_target_name, + early_drops=early_old_name_view_drops, + project=project, + scope=scope, + now=now, + ) age_reader: JanitorRelationAgeReader = JanitorRelationAgeReader( adapter=adapter, connection=connection ) @@ -144,12 +161,14 @@ def build_janitor_plan( skipped for skipped in schemas.skipped_relations if not QueryDiffArtifactLifecycle.is_artifact_name(skipped.key.name) + and skipped.key not in old_names.keys ), *archives.skipped_relations, *query_diff_artifact_skipped, ), skipped_schemas=schemas.skipped_schemas, blocked_schemas=archives.blocked_schemas, + old_name_views=old_names, scanned_schema_count=len(target_schemas | set(query_artifact_schemas)), age_metadata_supported=age_supported, planned_at=now, diff --git a/src/sqlbuild/executor/janitor/models.py b/src/sqlbuild/executor/janitor/models.py index 034fdf686f..4655857c9c 100644 --- a/src/sqlbuild/executor/janitor/models.py +++ b/src/sqlbuild/executor/janitor/models.py @@ -6,6 +6,8 @@ from datetime import datetime from sqlbuild.adapter.contract.models import RelationInfo, RelationLookup +from sqlbuild.compiler.migrations.models import OldNameViewHistory +from sqlbuild.compiler.migrations.types import OldNameViewDropReason @dataclass(frozen=True, eq=False) @@ -81,6 +83,41 @@ class JanitorArchivedRelation: original_key: JanitorRelationKey | None = None +@dataclass(frozen=True) +class JanitorOldNameView: + """One compatibility view recorded at a migrated model's old name.""" + + key: JanitorRelationKey + history: OldNameViewHistory + expires_at: datetime | None + drop_reason: OldNameViewDropReason | None = None + occupied: bool = False + claimed_by: str | None = None + + @property + def destination_model(self) -> str: + """Return the model whose relation the view reads.""" + + return self.history.move.destination_model + + +@dataclass(frozen=True) +class JanitorOldNameViewPlanning: + """Compatibility views to keep, drop, or record as already gone.""" + + live: tuple[JanitorOldNameView, ...] = () + drops: tuple[JanitorOldNameView, ...] = () + missing: tuple[JanitorOldNameView, ...] = () + unknown_requests: tuple[str, ...] = () + project_destinations: dict[JanitorRelationKey, str] = field(default_factory=dict) + + @property + def keys(self) -> frozenset[JanitorRelationKey]: + """Return every old name this planning owns, so general cleanup leaves it alone.""" + + return frozenset(view.key for view in (*self.live, *self.drops, *self.missing)) + + @dataclass(frozen=True) class JanitorQueryDiffArtifactCandidate: """One expired, fingerprint-owned query-diff artifact eligible for deletion.""" @@ -237,6 +274,7 @@ class JanitorPlan: skipped_relations: tuple[JanitorSkippedRelation, ...] = field(default_factory=tuple) skipped_schemas: tuple[JanitorSkippedSchema, ...] = field(default_factory=tuple) blocked_schemas: tuple[JanitorBlockedSchema, ...] = field(default_factory=tuple) + old_name_views: JanitorOldNameViewPlanning = field(default_factory=JanitorOldNameViewPlanning) scanned_schema_count: int = 0 age_metadata_supported: bool = False planned_at: datetime | None = None @@ -253,3 +291,4 @@ class JanitorExecutionResult: default_factory=tuple ) pruned_direct_state: tuple[JanitorDirectStatePruneCandidate, ...] = field(default_factory=tuple) + dropped_old_name_views: tuple[JanitorOldNameView, ...] = field(default_factory=tuple) diff --git a/src/sqlbuild/executor/migrations/_helpers/old_name_facts.py b/src/sqlbuild/executor/migrations/_helpers/old_name_facts.py new file mode 100644 index 0000000000..685e755c6c --- /dev/null +++ b/src/sqlbuild/executor/migrations/_helpers/old_name_facts.py @@ -0,0 +1,118 @@ +"""Build, read, and append old-name view facts during a build.""" + +from __future__ import annotations + +from datetime import datetime +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.compiler.migrations.constants import MIGRATION_WRITE_ATTEMPTS +from sqlbuild.compiler.migrations.main.deterministic_old_name_view_event_id import ( + deterministic_old_name_view_event_id, +) +from sqlbuild.compiler.migrations.main.read_old_name_view_events import ( + read_old_name_view_events, +) +from sqlbuild.compiler.migrations.main.relation_for_location import migration_relation_for_location +from sqlbuild.compiler.migrations.main.stored_old_name_view_columns import ( + stored_old_name_view_columns, +) +from sqlbuild.compiler.migrations.main.write_old_name_view_event import write_old_name_view_event +from sqlbuild.compiler.migrations.models import MigrationRelation, OldNameViewEvent +from sqlbuild.compiler.migrations.types import OldNameViewEventType +from sqlbuild.compiler.planner.models import OldNameViewPlanEntry + + +def old_name_fact( + *, + entry: OldNameViewPlanEntry, + event_type: OldNameViewEventType, + migration_event_id: str, + run_id: str, + created_at: datetime, + archive_name: str | None = None, + column_aliases: tuple[tuple[str, str], ...] = (), + expires_at: datetime | None = None, + grants_copied: tuple[str, ...] | None = None, + view_sql: str | None = None, +) -> OldNameViewEvent: + """Build one old-name fact for a planned old-name step.""" + + return OldNameViewEvent( + event_id=deterministic_old_name_view_event_id( + event_type=event_type, migration_event_id=migration_event_id + ), + target_name=entry.target_name, + event_type=event_type, + migration_event_id=migration_event_id, + destination_model=entry.model_name, + old=migration_relation_for_location(entry.origin), + new=migration_relation_for_location(entry.destination), + run_id=run_id, + created_at=created_at, + view_retention=entry.retention if event_type == OldNameViewEventType.REQUIRED else None, + archive_name=archive_name, + column_aliases=column_aliases, + expires_at=expires_at, + grants_copied=grants_copied, + view_sql=view_sql, + ) + + +def record_old_name_fact( + *, + adapter: BaseAdapter, + connection: Any, + event: OldNameViewEvent, + create_table: bool = True, + attempts: int = MIGRATION_WRITE_ATTEMPTS, +) -> None: + """Append one old-name fact through the adapter's state-table rendering.""" + + write_old_name_view_event( + connection=connection, + execute=adapter.execute, + event=event, + render_qualified_name=adapter.render_qualified_name, + create_table_sql=( + old_name_state_table_sql(adapter=adapter, relation=event.new) if create_table else None + ), + attempts=attempts, + ) + + +def old_name_state_table_sql(*, adapter: BaseAdapter, relation: MigrationRelation) -> str: + """Render the create-if-missing DDL of the fact table next to one destination.""" + + return adapter.render_create_old_name_view_state_table_sql( + database=relation.database, schema=relation.schema or "" + ) + + +def recorded_old_name_facts( + *, adapter: BaseAdapter, connection: Any, entry: OldNameViewPlanEntry, migration_event_id: str +) -> dict[OldNameViewEventType, OldNameViewEvent]: + """Re-read the facts of one move so every step decides from current state.""" + + relation: MigrationRelation = migration_relation_for_location(entry.destination) + _ = adapter.execute( + connection=connection, sql=old_name_state_table_sql(adapter=adapter, relation=relation) + ) + facts: dict[OldNameViewEventType, OldNameViewEvent] = {} + event: OldNameViewEvent + for event in read_old_name_view_events( + connection=connection, + execute=adapter.execute, + database=relation.database, + schema=relation.schema or "", + stored_columns=stored_old_name_view_columns( + adapter=adapter, + connection=connection, + database=relation.database, + schema=relation.schema or "", + ), + render_qualified_name=adapter.render_qualified_name, + ): + if event.migration_event_id == migration_event_id: + _ = facts.setdefault(event.event_type, event) + return facts diff --git a/src/sqlbuild/executor/migrations/_helpers/old_name_views.py b/src/sqlbuild/executor/migrations/_helpers/old_name_views.py new file mode 100644 index 0000000000..9603e3bb2e --- /dev/null +++ b/src/sqlbuild/executor/migrations/_helpers/old_name_views.py @@ -0,0 +1,772 @@ +"""Archive old relations and keep compatibility views at old model names.""" + +from __future__ import annotations + +import contextlib +import dataclasses +from collections.abc import Iterator +from datetime import UTC, datetime, timedelta +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapter.contract.classes.statement_recorder import StatementRecorder +from sqlbuild.adapter.contract.main.grants_for_columns import grants_for_columns +from sqlbuild.adapter.contract.models import ColumnInfo, RelationGrant, RelationInfo +from sqlbuild.adapter.contract.types import RelationType +from sqlbuild.adapter.relations.main.resolve_qualified_name_parts import ( + resolve_qualified_name_parts, +) +from sqlbuild.adapter.type_system.main.normalize_relation_type import normalize_relation_type +from sqlbuild.compiler.compile.models import CompiledRelationLocation +from sqlbuild.compiler.migrations.constants import COLUMN_MIGRATION_TABLE_NAME +from sqlbuild.compiler.migrations.main.deterministic_event_id import ( + deterministic_migration_event_id, +) +from sqlbuild.compiler.migrations.main.read_recorded_column_renames import ( + read_recorded_column_renames, +) +from sqlbuild.compiler.migrations.main.relation_for_location import migration_relation_for_location +from sqlbuild.compiler.migrations.models import ( + ColumnMigrationEvent, + MigrationRelation, + OldNameViewEvent, +) +from sqlbuild.compiler.migrations.types import OldNameViewEventType +from sqlbuild.compiler.planner.models import OldNameView, OldNameViewPlanEntry, PlanOutput +from sqlbuild.cursor_algebra.models import Duration +from sqlbuild.errors.contracts.exceptions import ExecutorInputError +from sqlbuild.executor.janitor.main._build_archive_name import build_janitor_archive_name +from sqlbuild.executor.migrations._helpers.old_name_facts import ( + old_name_fact, + record_old_name_fact, + recorded_old_name_facts, +) +from sqlbuild.executor.migrations.constants import ( + MIGRATION_NAME_ATTEMPTS, + MIGRATION_ORIGIN_ARCHIVE_KIND, +) +from sqlbuild.executor.migrations.models import OldNameViewSource + + +def run_old_name_steps( + *, + adapter: BaseAdapter, + connection: Any, + entry: OldNameViewPlanEntry, + migration_event_id: str, + run_id: str, + recorded_views: tuple[OldNameView, ...], +) -> tuple[str, ...]: + """Archive the old relation and create its compatibility view, resuming from facts.""" + + facts: dict[OldNameViewEventType, OldNameViewEvent] = recorded_old_name_facts( + adapter=adapter, connection=connection, entry=entry, migration_event_id=migration_event_id + ) + required: OldNameViewEvent | None = facts.get(OldNameViewEventType.REQUIRED) + if required is None or OldNameViewEventType.VIEW_CREATED in facts: + return () + aliases: tuple[tuple[str, str], ...] = ( + _recorded_column_aliases( + adapter=adapter, connection=connection, entry=entry, since=required + ) + if entry.stores_history + else entry.column_aliases + ) + source: OldNameViewSource = OldNameViewSource( + old=entry.origin, new=entry.destination, column_aliases=aliases + ) + select_sql: str = render_old_name_view_select( + adapter=adapter, connection=connection, source=source + ) + old_relation: RelationInfo | None = _listed( + adapter=adapter, connection=connection, location=entry.origin + ) + archived: bool = OldNameViewEventType.ORIGIN_ARCHIVED in facts + if archived and old_relation is not None and not _is_view(old_relation): + return ( + f"M116: {_display(entry.origin)} is held by another relation after it was archived; " + "no compatibility view was created", + ) + warnings: tuple[str, ...] = () + if not archived and old_relation is not None and adapter.views_bind_to_relation_identity(): + warnings = _external_dependents_warning( + adapter=adapter, connection=connection, entry=entry, recorded_views=recorded_views + ) + chained: tuple[OldNameViewSource, ...] = _chained_sources( + location=entry.origin, recorded_views=recorded_views + ) + now: datetime = datetime.now(tz=UTC) + duration: Duration | None = Duration.parse(required.view_retention or "") + with _ddl_transaction(adapter=adapter, connection=connection) as transactional: + attempts: int = 1 if transactional else 3 + archive_name: str | None = ( + facts[OldNameViewEventType.ORIGIN_ARCHIVED].archive_name if archived else None + ) + if not archived: + archive_name = _archive_old_relation( + adapter=adapter, connection=connection, entry=entry, old_relation=old_relation + ) + record_old_name_fact( + adapter=adapter, + connection=connection, + event=old_name_fact( + entry=entry, + event_type=OldNameViewEventType.ORIGIN_ARCHIVED, + migration_event_id=migration_event_id, + run_id=run_id, + created_at=now, + archive_name=archive_name, + ), + create_table=False, + attempts=attempts, + ) + copied: tuple[tuple[str, ...], tuple[RelationGrant, ...]] = _create_compatibility_view( + adapter=adapter, + connection=connection, + entry=entry, + source=source, + sql=select_sql, + archive_name=archive_name, + ) + grants: tuple[str, ...] = copied[0] + record_old_name_fact( + adapter=adapter, + connection=connection, + event=old_name_fact( + entry=entry, + event_type=OldNameViewEventType.VIEW_CREATED, + migration_event_id=migration_event_id, + run_id=run_id, + created_at=now, + column_aliases=aliases, + expires_at=None if duration is None else duration.add_to(now), + grants_copied=grants, + view_sql=select_sql, + ), + create_table=False, + attempts=attempts, + ) + _ = rebind_old_name_views(adapter=adapter, connection=connection, sources=chained) + return (*warnings, *reader_access_warning(adapter=adapter, entry=entry, grants=copied[1])) + + +def run_planned_old_name_steps( + *, plan: PlanOutput, adapter: BaseAdapter, connection: Any, model_name: str, run_id: str +) -> tuple[str, ...]: + """Run every pending old-name step planned for one successfully built model.""" + + warnings: list[str] = [] + entry: OldNameViewPlanEntry + for entry in plan.old_name_view_entries: + if entry.model_name != model_name or not entry.runs_steps: + continue + try: + warnings.extend( + run_old_name_steps( + adapter=adapter, + connection=connection, + entry=entry, + migration_event_id=_migration_event_id(entry=entry, run_id=run_id), + run_id=run_id, + recorded_views=plan.old_name_views, + ) + ) + except Exception as error: + raise ExecutorInputError( + f"model '{model_name}': keeping its old name {_display(entry.origin)} " + f"failed: {error}", + code="M117", + help=( + "Re-run the build; old-name steps resume from recorded facts and never " + "repeat a completed step." + ), + ) from error + return tuple(warnings) + + +def present_old_name_views( + *, adapter: BaseAdapter, connection: Any, sources: tuple[OldNameViewSource, ...] +) -> tuple[OldNameViewSource, ...]: + """Keep views that exist at their old names; a recorded, absent view is left for janitor.""" + + present: list[OldNameViewSource] = [] + source: OldNameViewSource + for source in sources: + relation: RelationInfo | None = _listed( + adapter=adapter, connection=connection, location=source.old + ) + if relation is not None and _is_view(relation): + present.append(source) + return tuple(present) + + +def refresh_old_name_views( + *, adapter: BaseAdapter, connection: Any, sources: tuple[OldNameViewSource, ...] +) -> int: + """Redefine changed compatibility views in order, keeping their privileges as they were.""" + + refreshed: int = 0 + source: OldNameViewSource + for source in sources: + destination: str = _qualified(adapter=adapter, location=source.old) + sql: str = render_old_name_view_select( + adapter=adapter, connection=connection, source=source + ) + if not adapter.views_bind_to_relation_identity() and adapter.view_definition_matches( + connection=connection, + database=source.old.database, + schema=source.old.schema or "", + name=source.old.name, + sql=sql, + ): + continue + before: tuple[RelationGrant, ...] = _view_grants( + adapter=adapter, connection=connection, location=source.old + ) + native: tuple[str, ...] | None = adapter.render_replace_view_keeping_grants( + destination=destination, sql=sql + ) + if native is None: + _create_view(adapter=adapter, connection=connection, old=source.old, sql=sql) + else: + _execute_all(adapter=adapter, connection=connection, statements=native) + _ = reconcile_view_grants( + adapter=adapter, connection=connection, source=source, target=before + ) + refreshed += 1 + return refreshed + + +def release_old_name_views( + *, adapter: BaseAdapter, connection: Any, sources: tuple[OldNameViewSource, ...] +) -> tuple[OldNameViewSource, ...]: + """Capture each view's current privileges, then drop the views, dependents first.""" + + released: tuple[OldNameViewSource, ...] = tuple( + dataclasses.replace( + source, + grants=_view_grants(adapter=adapter, connection=connection, location=source.old), + ) + for source in sources + ) + _ = drop_old_name_views(adapter=adapter, connection=connection, sources=released) + return released + + +def recreate_old_name_views( + *, adapter: BaseAdapter, connection: Any, sources: tuple[OldNameViewSource, ...] +) -> int: + """Re-create released views in order with the privileges captured before their drop.""" + + source: OldNameViewSource + for source in sources: + _create_view( + adapter=adapter, + connection=connection, + old=source.old, + sql=render_old_name_view_select(adapter=adapter, connection=connection, source=source), + ) + _ = reconcile_view_grants( + adapter=adapter, connection=connection, source=source, target=source.grants + ) + return len(sources) + + +def rebind_old_name_views( + *, adapter: BaseAdapter, connection: Any, sources: tuple[OldNameViewSource, ...] +) -> int: + """Point existing compatibility views at their relations' current columns, in order.""" + + present: tuple[OldNameViewSource, ...] = present_old_name_views( + adapter=adapter, connection=connection, sources=sources + ) + if not present: + return 0 + if adapter.views_bind_to_relation_identity() and not all( + _replaceable(adapter=adapter, connection=connection, source=source) for source in present + ): + return recreate_old_name_views( + adapter=adapter, + connection=connection, + sources=release_old_name_views(adapter=adapter, connection=connection, sources=present), + ) + return refresh_old_name_views(adapter=adapter, connection=connection, sources=present) + + +def rebind_old_name_views_atomically( + *, adapter: BaseAdapter, connection: Any, sources: tuple[OldNameViewSource, ...] +) -> int: + """Rebind compatibility views in one transaction where views bind to relation identity.""" + + with _ddl_transaction(adapter=adapter, connection=connection): + return rebind_old_name_views(adapter=adapter, connection=connection, sources=sources) + + +def exposed_columns( + *, adapter: BaseAdapter, connection: Any, source: OldNameViewSource +) -> tuple[str, ...]: + """Return the column names the view presents: old names for aliased columns.""" + + old_by_new: dict[str, str] = {new.lower(): old for old, new in source.column_aliases} + return tuple( + old_by_new.get(column.name.lower(), column.name) + for column in adapter.get_columns( + connection=connection, + database=source.new.database, + schema=source.new.schema, + name=source.new.name, + ) + ) + + +def _view_grants( + *, adapter: BaseAdapter, connection: Any, location: CompiledRelationLocation +) -> tuple[RelationGrant, ...]: + return adapter.read_relation_grants( + connection=connection, + database=location.database, + schema=location.schema or "", + name=location.name, + relation_type=RelationType.VIEW.value, + ) + + +def reconcile_view_grants( + *, + adapter: BaseAdapter, + connection: Any, + source: OldNameViewSource, + target: tuple[RelationGrant, ...], +) -> tuple[str, ...]: + """Revoke view privileges not in ``target`` and grant missing ones; return ``target`` SQL.""" + + destination: str = _qualified(adapter=adapter, location=source.old) + columns: tuple[str, ...] = exposed_columns( + adapter=adapter, connection=connection, source=source + ) + wanted: tuple[RelationGrant, ...] = grants_for_columns(grants=target, columns=columns) + current: tuple[RelationGrant, ...] = _view_grants( + adapter=adapter, connection=connection, location=source.old + ) + _execute_all( + adapter=adapter, + connection=connection, + statements=( + *adapter.render_relation_revokes( + grants=tuple(grant for grant in current if grant not in wanted), + destination=destination, + ), + *adapter.render_relation_grants( + grants=tuple(grant for grant in wanted if grant not in current), + destination=destination, + columns=columns, + ), + ), + ) + return adapter.render_relation_grants(grants=wanted, destination=destination, columns=columns) + + +def _replaceable(*, adapter: BaseAdapter, connection: Any, source: OldNameViewSource) -> bool: + """Return whether CREATE OR REPLACE VIEW can keep the view, which only allows appending.""" + + current: tuple[ColumnInfo, ...] = adapter.get_columns( + connection=connection, + database=source.old.database, + schema=source.old.schema, + name=source.old.name, + ) + old_by_new: dict[str, str] = {new.lower(): old for old, new in source.column_aliases} + projected: list[tuple[str, str]] = [ + (old_by_new.get(column.name.lower(), column.name).lower(), column.type.lower()) + for column in adapter.get_columns( + connection=connection, + database=source.new.database, + schema=source.new.schema, + name=source.new.name, + ) + ] + existing: list[tuple[str, str]] = [ + (column.name.lower(), column.type.lower()) for column in current + ] + return projected[: len(existing)] == existing + + +def drop_old_name_views( + *, adapter: BaseAdapter, connection: Any, sources: tuple[OldNameViewSource, ...] +) -> int: + """Drop compatibility views, dependents first, when they exist.""" + + source: OldNameViewSource + for source in reversed(sources): + adapter.drop_view( + connection=connection, + destination=_qualified(adapter=adapter, location=source.old), + if_exists=True, + statement_recorder=StatementRecorder(), + ) + return len(sources) + + +def _migration_event_id(*, entry: OldNameViewPlanEntry, run_id: str) -> str: + if entry.migration_event_id is not None: + return entry.migration_event_id + return deterministic_migration_event_id( + run_id=run_id, + target_name=entry.target_name, + origin=migration_relation_for_location(entry.origin), + destination=migration_relation_for_location(entry.destination), + ) + + +def views_reading( + *, location: CompiledRelationLocation, recorded_views: tuple[OldNameView, ...] +) -> tuple[OldNameViewSource, ...]: + """Return recorded views reading one relation, then chained views; reused names are skipped.""" + + ordered: list[OldNameViewSource] = [] + seen: set[tuple[str | None, str, str]] = set() + frontier: list[CompiledRelationLocation] = [location] + while frontier: + current: CompiledRelationLocation = frontier.pop(0) + view: OldNameView + for view in _direct_readers(location=current, recorded_views=recorded_views): + identity: tuple[str | None, str, str] = migration_relation_for_location( + view.old + ).identity + if identity in seen: + continue + seen.add(identity) + ordered.append( + OldNameViewSource(old=view.old, new=view.new, column_aliases=view.column_aliases) + ) + frontier.append(view.old) + return tuple(ordered) + + +def render_old_name_view_select( + *, adapter: BaseAdapter, connection: Any, source: OldNameViewSource +) -> str: + """Render the SELECT that presents the new relation under the old interface.""" + + relation: str = _qualified(adapter=adapter, location=source.new) + if not source.column_aliases: + return f"SELECT * FROM {relation}" + columns: tuple[ColumnInfo, ...] = adapter.get_columns( + connection=connection, + database=source.new.database, + schema=source.new.schema, + name=source.new.name, + ) + old_by_new: dict[str, str] = {new.lower(): old for old, new in source.column_aliases} + projections: list[str] = [] + column: ColumnInfo + for column in columns: + rendered: str = adapter.render_exact_identifier(column.name) + old_name: str | None = old_by_new.get(column.name.lower()) + projections.append( + rendered if old_name is None else f"{rendered} AS {adapter.render_identifier(old_name)}" + ) + return f"SELECT {', '.join(projections)} FROM {relation}" + + +def compose_column_aliases( + events: tuple[ColumnMigrationEvent, ...], +) -> tuple[tuple[str, str], ...]: + """Fold ordered column renames into one old-name-to-current-name map.""" + + current_by_old: dict[str, str] = {} + event: ColumnMigrationEvent + for event in sorted(events, key=lambda item: (item.created_at, item.event_id)): + origin: str = event.origin_column + replaced: bool = False + old: str + for old, current in tuple(current_by_old.items()): + if current.lower() == origin.lower(): + current_by_old[old] = event.destination_column + replaced = True + if not replaced: + current_by_old[origin] = event.destination_column + return tuple( + sorted((old, new) for old, new in current_by_old.items() if old.lower() != new.lower()) + ) + + +def _recorded_column_aliases( + *, adapter: BaseAdapter, connection: Any, entry: OldNameViewPlanEntry, since: OldNameViewEvent +) -> tuple[tuple[str, str], ...]: + relation: MigrationRelation = migration_relation_for_location(entry.destination) + if not adapter.list_relations( + connection=connection, + database=relation.database, + schemas=(relation.schema or "",), + names=(COLUMN_MIGRATION_TABLE_NAME,), + ): + return () + return compose_column_aliases( + read_recorded_column_renames( + connection=connection, + execute=adapter.execute, + relation=relation, + since=since.created_at, + render_qualified_name=adapter.render_qualified_name, + ) + ) + + +def _archive_old_relation( + *, + adapter: BaseAdapter, + connection: Any, + entry: OldNameViewPlanEntry, + old_relation: RelationInfo | None, +) -> str | None: + if old_relation is None: + return _existing_archive_name(adapter=adapter, connection=connection, entry=entry) + archive_name: str = _unused_archive_name(adapter=adapter, connection=connection, entry=entry) + origin: str = _qualified(adapter=adapter, location=entry.origin) + destination: str = resolve_qualified_name_parts( + adapter=adapter, + database=entry.origin.database, + schema=entry.origin.schema, + name=archive_name, + ) + if _is_view(old_relation): + adapter.rename_view( + connection=connection, + origin=origin, + destination=destination, + statement_recorder=StatementRecorder(), + ) + else: + adapter.rename( + connection=connection, + origin=origin, + destination=destination, + statement_recorder=StatementRecorder(), + ) + return archive_name + + +def _unused_archive_name( + *, adapter: BaseAdapter, connection: Any, entry: OldNameViewPlanEntry +) -> str: + now: datetime = datetime.now(tz=UTC) + candidates: tuple[str, ...] = tuple( + build_janitor_archive_name( + original_name=entry.origin.name, + archived_at=now - timedelta(seconds=offset), + identifier_limit=adapter.maximum_identifier_length(), + kind=MIGRATION_ORIGIN_ARCHIVE_KIND, + ) + for offset in range(MIGRATION_NAME_ATTEMPTS) + ) + existing: frozenset[str] = frozenset( + relation.name.lower() + for relation in adapter.list_relations( + connection=connection, + database=entry.origin.database, + schemas=(entry.origin.schema or "",), + names=candidates, + ) + ) + name: str + for name in candidates: + if name not in existing: + return name + return candidates[0] + + +def _existing_archive_name( + *, adapter: BaseAdapter, connection: Any, entry: OldNameViewPlanEntry +) -> str | None: + """Find the archive an interrupted run left, as physical evidence for the archive fact.""" + + suffix: str = f"__{MIGRATION_ORIGIN_ARCHIVE_KIND}__{entry.origin.name}".lower() + names: list[str] = sorted( + relation.name.lower() + for relation in adapter.list_relations( + connection=connection, + database=entry.origin.database, + schemas=(entry.origin.schema or "",), + names=None, + ) + if relation.name.lower().startswith("_sqb_archive__") + and relation.name.lower().endswith(suffix) + ) + return names[-1] if names else None + + +def _external_dependents_warning( + *, + adapter: BaseAdapter, + connection: Any, + entry: OldNameViewPlanEntry, + recorded_views: tuple[OldNameView, ...], +) -> tuple[str, ...]: + owned: frozenset[str] = frozenset( + f"{view.old.schema}.{view.old.name}".lower() for view in recorded_views + ) + dependents: tuple[str, ...] = tuple( + name + for name in adapter.list_dependent_view_names( + connection=connection, + database=entry.origin.database, + schema=entry.origin.schema or "", + name=entry.origin.name, + ) + if name.lower() not in owned + ) + if not dependents: + return () + return ( + f"M115: views not managed by SQLBuild depend on {_display(entry.origin)}: " + f"{', '.join(dependents)}. They stay bound to the archived relation and keep reading " + "its old data; re-create them against the old name, now a compatibility view, or " + f"against {_display(entry.destination)}", + ) + + +def _chained_sources( + *, location: CompiledRelationLocation, recorded_views: tuple[OldNameView, ...] +) -> tuple[OldNameViewSource, ...]: + return views_reading(location=location, recorded_views=recorded_views) + + +def _direct_readers( + *, location: CompiledRelationLocation, recorded_views: tuple[OldNameView, ...] +) -> tuple[OldNameView, ...]: + relation: MigrationRelation = migration_relation_for_location(location) + return tuple( + view + for view in recorded_views + if view.name_reused_by is None + and migration_relation_for_location(view.new).matches(relation) + ) + + +def _create_compatibility_view( + *, + adapter: BaseAdapter, + connection: Any, + entry: OldNameViewPlanEntry, + source: OldNameViewSource, + sql: str, + archive_name: str | None, +) -> tuple[tuple[str, ...], tuple[RelationGrant, ...]]: + """Publish the view at the old name with the archived relation's grants; return them.""" + + _create_view(adapter=adapter, connection=connection, old=entry.origin, sql=sql) + grants: tuple[RelationGrant, ...] = _archive_grants( + adapter=adapter, connection=connection, entry=entry, archive_name=archive_name + ) + statements: tuple[str, ...] = reconcile_view_grants( + adapter=adapter, connection=connection, source=source, target=grants + ) + return statements, grants + + +def _archive_grants( + *, + adapter: BaseAdapter, + connection: Any, + entry: OldNameViewPlanEntry, + archive_name: str | None, +) -> tuple[RelationGrant, ...]: + """Return the grants of the archived old relation; none when no archive is found.""" + + if archive_name is None: + return () + archive: RelationInfo | None = _listed( + adapter=adapter, + connection=connection, + location=CompiledRelationLocation( + database=entry.origin.database, + schema=entry.origin.schema, + name=archive_name, + qualified_name=None, + ), + ) + if archive is None: + return () + return adapter.read_relation_grants( + connection=connection, + database=archive.database, + schema=archive.schema or entry.origin.schema or "", + name=archive.name, + relation_type=normalize_relation_type(archive.relation_type).value, + ) + + +def reader_access_warning( + *, adapter: BaseAdapter, entry: OldNameViewPlanEntry, grants: tuple[RelationGrant, ...] +) -> tuple[str, ...]: + """Warn that principals given the view by copied grants also need the destination.""" + + principals: tuple[str, ...] = tuple( + sorted({grant.grantee for grant in grants if grant.grantee and not grant.denied}) + ) + if not adapter.views_read_with_reader_access or not principals: + return () + return ( + f"M118: {_display(entry.origin)} now reads {_display(entry.destination)} with each " + f"reader's own access. {', '.join(principals)} kept their access to the old name but " + f"also need read access on {_display(entry.destination)}; SQLBuild does not grant it", + ) + + +def _execute_all(*, adapter: BaseAdapter, connection: Any, statements: tuple[str, ...]) -> None: + statement: str + for statement in statements: + _ = adapter.execute(connection=connection, sql=statement) + + +def _create_view( + *, adapter: BaseAdapter, connection: Any, old: CompiledRelationLocation, sql: str +) -> None: + statement: str + for statement in adapter.render_create_view_as( + destination=_qualified(adapter=adapter, location=old), sql=sql + ): + _ = adapter.execute(connection=connection, sql=statement) + + +def _listed( + *, adapter: BaseAdapter, connection: Any, location: CompiledRelationLocation +) -> RelationInfo | None: + wanted: tuple[str, str] = ((location.schema or "").lower(), location.name.lower()) + return next( + ( + relation + for relation in adapter.list_relations( + connection=connection, + database=location.database, + schemas=(location.schema or "",), + names=(location.name,), + ) + if ((relation.schema or "").lower(), relation.name.lower()) == wanted + ), + None, + ) + + +def _is_view(relation: RelationInfo) -> bool: + return normalize_relation_type(relation.relation_type) == RelationType.VIEW + + +@contextlib.contextmanager +def _ddl_transaction(*, adapter: BaseAdapter, connection: Any) -> Iterator[bool]: + if not adapter.supports_transactional_ddl(): + yield False + return + with adapter.transaction(connection): + yield True + + +def _qualified(*, adapter: BaseAdapter, location: CompiledRelationLocation) -> str: + return location.qualified_name or resolve_qualified_name_parts( + adapter=adapter, database=location.database, schema=location.schema, name=location.name + ) + + +def _display(location: CompiledRelationLocation) -> str: + return location.qualified_name or location.name diff --git a/src/sqlbuild/executor/migrations/_helpers/promotion.py b/src/sqlbuild/executor/migrations/_helpers/promotion.py index ad6e6a8eef..e028be67bd 100644 --- a/src/sqlbuild/executor/migrations/_helpers/promotion.py +++ b/src/sqlbuild/executor/migrations/_helpers/promotion.py @@ -13,9 +13,14 @@ ) from sqlbuild.compiler.migrations.main.relation_for_location import migration_relation_for_location from sqlbuild.compiler.migrations.main.write_event import write_migration_event -from sqlbuild.compiler.migrations.models import MigrationEvent, MigrationRelation -from sqlbuild.compiler.migrations.types import MigrationDecision -from sqlbuild.compiler.planner.models import ModelMigrationPlanEntry +from sqlbuild.compiler.migrations.models import MigrationEvent, MigrationRelation, OldNameViewEvent +from sqlbuild.compiler.migrations.types import MigrationDecision, OldNameViewEventType +from sqlbuild.compiler.planner.models import ModelMigrationPlanEntry, OldNameViewPlanEntry +from sqlbuild.executor.migrations._helpers.old_name_facts import ( + old_name_fact, + old_name_state_table_sql, + record_old_name_fact, +) from sqlbuild.executor.migrations.models import MigrationArtifactNames from sqlbuild.executor.run.main.promote_staged_relation import promote_staged_relation @@ -27,10 +32,13 @@ def promote_and_record( entry: ModelMigrationPlanEntry, names: MigrationArtifactNames, event: MigrationEvent, + requirement: OldNameViewEvent | None = None, ) -> None: """Promote the stage, keeping a displaced destination, then append the migration event.""" if not adapter.supports_transactional_ddl(): + if requirement is not None: + record_old_name_fact(adapter=adapter, connection=connection, event=requirement) _promote(adapter=adapter, connection=connection, names=names) record_event(adapter=adapter, connection=connection, event=event) return @@ -40,6 +48,11 @@ def promote_and_record( database=event.destination.database, schema=event.destination.schema or "" ), ) + if requirement is not None: + _ = adapter.execute( + connection=connection, + sql=old_name_state_table_sql(adapter=adapter, relation=requirement.new), + ) with adapter.transaction(connection): rebinds: tuple[str, ...] = ( adapter.capture_dependent_view_rebinds( @@ -55,6 +68,14 @@ def promote_and_record( statement: str for statement in rebinds: _ = adapter.execute(connection=connection, sql=statement) + if requirement is not None: + record_old_name_fact( + adapter=adapter, + connection=connection, + event=requirement, + create_table=False, + attempts=1, + ) record_event( adapter=adapter, connection=connection, event=event, create_table=False, attempts=1 ) @@ -129,6 +150,7 @@ def record_renames( connection: Any, entries: tuple[ModelMigrationPlanEntry, ...], run_id: str, + old_name_entries: tuple[OldNameViewPlanEntry, ...] = (), ) -> None: """Record each newly handed-over table or view rename so retries keep the identity.""" @@ -142,8 +164,30 @@ def record_renames( schema=entry.destination.schema, statement_recorder=StatementRecorder(), ) - record_event( - adapter=adapter, - connection=connection, - event=migration_event(entry=entry, run_id=run_id), + event: MigrationEvent = migration_event(entry=entry, run_id=run_id) + requirement: OldNameViewEvent | None = old_name_requirement( + entries=old_name_entries, model_name=entry.model_name, event=event ) + if requirement is not None: + record_old_name_fact(adapter=adapter, connection=connection, event=requirement) + record_event(adapter=adapter, connection=connection, event=event) + + +def old_name_requirement( + *, entries: tuple[OldNameViewPlanEntry, ...], model_name: str, event: MigrationEvent +) -> OldNameViewEvent | None: + """Build the ``required`` fact a move records when its model keeps an old-name view.""" + + entry: OldNameViewPlanEntry | None = next( + (item for item in entries if item.model_name == model_name and item.records_requirement), + None, + ) + if entry is None: + return None + return old_name_fact( + entry=entry, + event_type=OldNameViewEventType.REQUIRED, + migration_event_id=event.event_id, + run_id=event.run_id, + created_at=event.created_at, + ) diff --git a/src/sqlbuild/executor/migrations/classes/__init__.py b/src/sqlbuild/executor/migrations/classes/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/src/sqlbuild/executor/migrations/classes/bound_old_name_views.py b/src/sqlbuild/executor/migrations/classes/bound_old_name_views.py new file mode 100644 index 0000000000..615205dcd8 --- /dev/null +++ b/src/sqlbuild/executor/migrations/classes/bound_old_name_views.py @@ -0,0 +1,51 @@ +"""Compatibility views bound to one model's relation during a build that replaces or alters it.""" + +from __future__ import annotations + +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.executor.migrations._helpers.old_name_views import ( + present_old_name_views, + rebind_old_name_views, + recreate_old_name_views, + release_old_name_views, +) +from sqlbuild.executor.migrations.models import OldNameViewSource + + +class BoundOldNameViews: + """Views released before column changes are re-created with the privileges they had.""" + + def __init__( + self, *, adapter: BaseAdapter, connection: Any, sources: tuple[OldNameViewSource, ...] + ) -> None: + self._adapter: BaseAdapter = adapter + self._connection: Any = connection + self._sources: tuple[OldNameViewSource, ...] = sources + self._released: tuple[OldNameViewSource, ...] = () + + def release(self) -> int: + """Capture privileges of the views that exist, then drop them.""" + + self._released = release_old_name_views( + adapter=self._adapter, + connection=self._connection, + sources=present_old_name_views( + adapter=self._adapter, connection=self._connection, sources=self._sources + ), + ) + return len(self._released) + + def rebind(self) -> int: + """Re-create released views, or point existing ones at the relation's new columns.""" + + released: tuple[OldNameViewSource, ...] = self._released + self._released = () + if released: + return recreate_old_name_views( + adapter=self._adapter, connection=self._connection, sources=released + ) + return rebind_old_name_views( + adapter=self._adapter, connection=self._connection, sources=self._sources + ) diff --git a/src/sqlbuild/executor/migrations/constants.py b/src/sqlbuild/executor/migrations/constants.py index 2798088f67..62d72ca44f 100644 --- a/src/sqlbuild/executor/migrations/constants.py +++ b/src/sqlbuild/executor/migrations/constants.py @@ -4,4 +4,5 @@ MIGRATION_STAGE_ARCHIVE_KIND: str = "migration_stage" MIGRATION_PREVIOUS_ARCHIVE_KIND: str = "migration_previous" +MIGRATION_ORIGIN_ARCHIVE_KIND: str = "migration_origin" MIGRATION_NAME_ATTEMPTS: int = 30 diff --git a/src/sqlbuild/executor/migrations/main/_apply.py b/src/sqlbuild/executor/migrations/main/_apply.py index d341b483a5..7c4fe28546 100644 --- a/src/sqlbuild/executor/migrations/main/_apply.py +++ b/src/sqlbuild/executor/migrations/main/_apply.py @@ -10,11 +10,17 @@ from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter from sqlbuild.adapter.contract.classes.statement_recorder import StatementRecorder from sqlbuild.adapter.contract.types import MigrationTransfer -from sqlbuild.compiler.planner.models import ModelMigrationPlanEntry, PlanOutput +from sqlbuild.compiler.migrations.models import MigrationEvent +from sqlbuild.compiler.planner.models import ( + ModelMigrationPlanEntry, + OldNameViewPlanEntry, + PlanOutput, +) from sqlbuild.errors.contracts.exceptions import ExecutorInputError from sqlbuild.executor.migrations._helpers.naming import resolve_artifact_names from sqlbuild.executor.migrations._helpers.promotion import ( migration_event, + old_name_requirement, promote_and_record, record_renames, ) @@ -52,9 +58,14 @@ def apply_model_migrations( connection=connection, run_id=run_id, on_progress=on_progress, + old_name_entries=plan.old_name_view_entries, ) _ = record_renames( - adapter=adapter, connection=connection, entries=plan.migration_entries, run_id=run_id + adapter=adapter, + connection=connection, + entries=plan.migration_entries, + run_id=run_id, + old_name_entries=plan.old_name_view_entries, ) @@ -65,6 +76,7 @@ def _apply_one( connection: Any, run_id: str, on_progress: Callable[[str], None] | None, + old_name_entries: tuple[OldNameViewPlanEntry, ...], ) -> None: origin_label: str = entry.origin.qualified_name or entry.origin.name destination_label: str = entry.destination.qualified_name or entry.destination.name @@ -73,7 +85,11 @@ def _apply_one( started: float = time.monotonic() try: transfer: MigrationTransfer = _stage_promote_and_record( - entry=entry, adapter=adapter, connection=connection, run_id=run_id + entry=entry, + adapter=adapter, + connection=connection, + run_id=run_id, + old_name_entries=old_name_entries, ) except Exception as error: raise ExecutorInputError( @@ -93,7 +109,12 @@ def _apply_one( def _stage_promote_and_record( - *, entry: ModelMigrationPlanEntry, adapter: BaseAdapter, connection: Any, run_id: str + *, + entry: ModelMigrationPlanEntry, + adapter: BaseAdapter, + connection: Any, + run_id: str, + old_name_entries: tuple[OldNameViewPlanEntry, ...], ) -> MigrationTransfer: adapter.ensure_schema( connection=connection, @@ -110,11 +131,15 @@ def _stage_promote_and_record( _ = verify_stage( adapter=adapter, connection=connection, entry=entry, names=names, transfer=transfer ) + event: MigrationEvent = migration_event(entry=entry, run_id=run_id) _ = promote_and_record( adapter=adapter, connection=connection, entry=entry, names=names, - event=migration_event(entry=entry, run_id=run_id), + event=event, + requirement=old_name_requirement( + entries=old_name_entries, model_name=entry.model_name, event=event + ), ) return transfer diff --git a/src/sqlbuild/executor/migrations/main/_bound_old_name_view_guard.py b/src/sqlbuild/executor/migrations/main/_bound_old_name_view_guard.py new file mode 100644 index 0000000000..820a3897f6 --- /dev/null +++ b/src/sqlbuild/executor/migrations/main/_bound_old_name_view_guard.py @@ -0,0 +1,42 @@ +"""Guard compatibility views bound to a model's relation while the build replaces or alters it.""" + +from __future__ import annotations + +from contextlib import AbstractContextManager, nullcontext +from functools import partial +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.compiler.compile.models import CompiledRelationLocation +from sqlbuild.compiler.planner.models import PlanOutput +from sqlbuild.executor.migrations._helpers.old_name_views import views_reading +from sqlbuild.executor.migrations.classes.bound_old_name_views import BoundOldNameViews +from sqlbuild.executor.migrations.models import OldNameViewSource +from sqlbuild.executor.run.models import BoundViewGuard + + +def bound_old_name_view_guard( + *, + plan: PlanOutput, + adapter: BaseAdapter, + connection: Any, + destination: CompiledRelationLocation, +) -> BoundViewGuard: + """Return the guard a materialization runs its promotion or column changes inside.""" + + if not adapter.views_bind_to_relation_identity(): + return BoundViewGuard() + sources: tuple[OldNameViewSource, ...] = views_reading( + location=destination, recorded_views=plan.old_name_views + ) + if not sources: + return BoundViewGuard() + views: BoundOldNameViews = BoundOldNameViews( + adapter=adapter, connection=connection, sources=sources + ) + transaction: partial[AbstractContextManager[object]] = partial(adapter.transaction, connection) + return BoundViewGuard( + transaction=transaction if adapter.supports_transactional_ddl() else nullcontext, + release=views.release, + rebind=views.rebind, + ) diff --git a/src/sqlbuild/executor/migrations/main/_compatibility_view_sql.py b/src/sqlbuild/executor/migrations/main/_compatibility_view_sql.py new file mode 100644 index 0000000000..c16178ce37 --- /dev/null +++ b/src/sqlbuild/executor/migrations/main/_compatibility_view_sql.py @@ -0,0 +1,57 @@ +"""Render the SQL a recorded compatibility view is defined by.""" + +from __future__ import annotations + +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.compiler.compile.models import CompiledRelationLocation +from sqlbuild.compiler.migrations.models import MigrationRelation, OldNameViewHistory +from sqlbuild.executor.migrations._helpers.old_name_views import render_old_name_view_select +from sqlbuild.executor.migrations.models import OldNameViewSource + + +def compatibility_view_sql( + *, adapter: BaseAdapter, connection: Any, history: OldNameViewHistory +) -> tuple[str, ...]: + """Return the SQL the view was created from and the SQL a build would redefine it with.""" + + recorded: str | None = None if history.created is None else history.created.view_sql + aliases: tuple[tuple[str, str], ...] = ( + () if history.created is None else history.created.column_aliases + ) + new: CompiledRelationLocation = _location(adapter=adapter, relation=history.new) + if adapter.views_bind_to_relation_identity() and not adapter.list_relations( + connection=connection, + database=new.database, + schemas=(new.schema or "",), + names=(new.name,), + ): + return () + rendered: tuple[str, ...] = () + if not aliases or adapter.get_columns( + connection=connection, database=new.database, schema=new.schema, name=new.name + ): + rendered = ( + render_old_name_view_select( + adapter=adapter, + connection=connection, + source=OldNameViewSource( + old=_location(adapter=adapter, relation=history.old), + new=new, + column_aliases=aliases, + ), + ), + ) + return tuple(dict.fromkeys((*rendered, *(() if recorded is None else (recorded,))))) + + +def _location(*, adapter: BaseAdapter, relation: MigrationRelation) -> CompiledRelationLocation: + return CompiledRelationLocation( + database=relation.database, + schema=relation.schema, + name=relation.name, + qualified_name=adapter.render_qualified_name( + database=relation.database, schema=relation.schema, name=relation.name + ), + ) diff --git a/src/sqlbuild/executor/migrations/main/_complete_old_name_views.py b/src/sqlbuild/executor/migrations/main/_complete_old_name_views.py new file mode 100644 index 0000000000..3457c71a0b --- /dev/null +++ b/src/sqlbuild/executor/migrations/main/_complete_old_name_views.py @@ -0,0 +1,36 @@ +"""Keep a migrated model's old name working after its successful build.""" + +from __future__ import annotations + +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.compiler.compile.models import CompiledRelationLocation +from sqlbuild.compiler.planner.models import PlanOutput +from sqlbuild.executor.migrations._helpers.old_name_views import ( + rebind_old_name_views_atomically, + run_planned_old_name_steps, + views_reading, +) + + +def complete_old_name_views( + *, + plan: PlanOutput, + adapter: BaseAdapter, + connection: Any, + model_name: str, + destination: CompiledRelationLocation, + run_id: str, +) -> tuple[str, ...]: + """Run pending old-name steps, then rebind every compatibility view reading the model.""" + + warnings: tuple[str, ...] = run_planned_old_name_steps( + plan=plan, adapter=adapter, connection=connection, model_name=model_name, run_id=run_id + ) + _ = rebind_old_name_views_atomically( + adapter=adapter, + connection=connection, + sources=views_reading(location=destination, recorded_views=plan.old_name_views), + ) + return warnings diff --git a/src/sqlbuild/executor/migrations/models.py b/src/sqlbuild/executor/migrations/models.py index 7a6994abee..23e6b87f43 100644 --- a/src/sqlbuild/executor/migrations/models.py +++ b/src/sqlbuild/executor/migrations/models.py @@ -4,6 +4,9 @@ from dataclasses import dataclass +from sqlbuild.adapter.contract.models import RelationGrant +from sqlbuild.compiler.compile.models import CompiledRelationLocation + @dataclass(frozen=True) class MigrationArtifactNames: @@ -15,3 +18,13 @@ class MigrationArtifactNames: displaced_qualified: str destination_qualified: str destination_exists: bool + + +@dataclass(frozen=True) +class OldNameViewSource: + """One compatibility view to (re)create: its name, what it reads, and its column aliases.""" + + old: CompiledRelationLocation + new: CompiledRelationLocation + column_aliases: tuple[tuple[str, str], ...] + grants: tuple[RelationGrant, ...] = () diff --git a/src/sqlbuild/executor/old_name_views/__init__.py b/src/sqlbuild/executor/old_name_views/__init__.py new file mode 100644 index 0000000000..46806d13b5 --- /dev/null +++ b/src/sqlbuild/executor/old_name_views/__init__.py @@ -0,0 +1 @@ +"""Janitor cleanup of compatibility views at old model names.""" diff --git a/src/sqlbuild/executor/old_name_views/_helpers/__init__.py b/src/sqlbuild/executor/old_name_views/_helpers/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/src/sqlbuild/executor/old_name_views/_helpers/janitor.py b/src/sqlbuild/executor/old_name_views/_helpers/janitor.py new file mode 100644 index 0000000000..1f29f25ef8 --- /dev/null +++ b/src/sqlbuild/executor/old_name_views/_helpers/janitor.py @@ -0,0 +1,414 @@ +"""Janitor planning and execution for compatibility views at old model names.""" + +from __future__ import annotations + +from dataclasses import replace +from datetime import UTC, datetime +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapter.contract.classes.statement_recorder import StatementRecorder +from sqlbuild.adapter.contract.models import RelationInfo +from sqlbuild.adapter.contract.types import RelationType +from sqlbuild.adapter.relations.main.resolve_qualified_name_parts import ( + resolve_qualified_name_parts, +) +from sqlbuild.adapter.type_system.main.normalize_relation_type import normalize_relation_type +from sqlbuild.compiler.compile.models import CompiledProject +from sqlbuild.compiler.migrations.constants import ( + MIGRATION_TABLE_NAME, + OLD_NAME_VIEW_TABLE_NAME, +) +from sqlbuild.compiler.migrations.main.deterministic_old_name_view_event_id import ( + deterministic_old_name_view_event_id, +) +from sqlbuild.compiler.migrations.main.read_old_name_view_histories import ( + read_old_name_view_histories, +) +from sqlbuild.compiler.migrations.main.stored_old_name_view_columns import ( + stored_old_name_view_columns, +) +from sqlbuild.compiler.migrations.main.write_old_name_view_event import write_old_name_view_event +from sqlbuild.compiler.migrations.models import OldNameViewEvent, OldNameViewHistory +from sqlbuild.compiler.migrations.types import ( + OldNameViewDropReason, + OldNameViewEventType, + OldNameViewStatus, +) +from sqlbuild.executor.janitor.models import ( + JanitorOldNameView, + JanitorOldNameViewPlanning, + JanitorRelationKey, + JanitorRelationScope, +) +from sqlbuild.executor.migrations.main._compatibility_view_sql import compatibility_view_sql + + +def plan_old_name_views( + *, + adapter: BaseAdapter, + connection: Any, + managed_target_schemas: set[tuple[str | None, str | None]], + relations_by_schema: dict[tuple[str | None, str | None], tuple[RelationInfo, ...]], + target_name: str | None, + early_drops: tuple[str, ...], + project_destinations: dict[JanitorRelationKey, str], + now: datetime, +) -> JanitorOldNameViewPlanning: + """Split recorded views into live, droppable, and gone; another relation's name is gone.""" + + histories: tuple[OldNameViewHistory, ...] = tuple( + history + for history in _read_histories( + adapter=adapter, + connection=connection, + managed_target_schemas=managed_target_schemas, + relations_by_schema=relations_by_schema, + ) + if history.move.target_name in (None, target_name) + and history.status(now=now) in (OldNameViewStatus.LIVE, OldNameViewStatus.EXPIRED) + ) + if not histories and not early_drops: + return JanitorOldNameViewPlanning() + physical: dict[JanitorRelationKey, RelationInfo] = _physical_old_names( + adapter=adapter, + connection=connection, + histories=histories, + relations_by_schema=relations_by_schema, + ) + requested: dict[str, JanitorOldNameView | None] = dict.fromkeys(early_drops) + live: list[JanitorOldNameView] = [] + drops: list[JanitorOldNameView] = [] + missing: list[JanitorOldNameView] = [] + history: OldNameViewHistory + for history in histories: + key: JanitorRelationKey = _old_key(history) + view: JanitorOldNameView = _view(history) + matched: tuple[str, ...] = tuple(name for name in requested if _names(name=name, key=key)) + requested.update(dict.fromkeys(matched, view)) + relation: RelationInfo | None = physical.get(key) + if relation is None: + missing.append(_with_reason(view=view, reason=OldNameViewDropReason.MISSING)) + elif not is_compatibility_view( + adapter=adapter, connection=connection, view=view, relation=relation + ): + missing.append(_occupied(view=view, project_destinations=project_destinations)) + elif matched: + drops.append(_with_reason(view=view, reason=OldNameViewDropReason.EARLY)) + elif history.status(now=now) == OldNameViewStatus.EXPIRED: + drops.append(_with_reason(view=view, reason=OldNameViewDropReason.EXPIRED)) + else: + live.append(view) + return JanitorOldNameViewPlanning( + live=tuple(live), + drops=tuple(drops), + missing=tuple(missing), + unknown_requests=tuple(name for name, view in requested.items() if view is None), + project_destinations=project_destinations, + ) + + +def protect_old_names( + *, scope: JanitorRelationScope, old_names: JanitorOldNameViewPlanning +) -> JanitorRelationScope: + """Keep general cleanup away from every old name this planning handles itself.""" + + if not old_names.keys: + return scope + reasons: dict[JanitorRelationKey, str] = dict(scope.protected_relation_reasons or {}) + reasons.update({key: "compatibility view at an old model name" for key in old_names.keys}) + return replace( + scope, + protected_relation_keys=scope.protected_relation_keys | old_names.keys, + protected_relation_reasons=reasons, + ) + + +def apply_old_name_view_drops( + *, + plan: JanitorOldNameViewPlanning, + adapter: BaseAdapter, + connection: Any, + recorder: StatementRecorder, + run_id: str, +) -> tuple[JanitorOldNameView, ...]: + """Drop expired or requested views, then record every drop and every vanished view.""" + + dropped: list[JanitorOldNameView] = [] + view: JanitorOldNameView + for view in plan.drops: + if not is_compatibility_view(adapter=adapter, connection=connection, view=view): + _record_drop( + adapter=adapter, + connection=connection, + view=_with_reason(view=view, reason=OldNameViewDropReason.MISSING), + run_id=run_id, + ) + continue + adapter.drop_view( + connection=connection, + destination=resolve_qualified_name_parts( + adapter=adapter, + database=view.key.database, + schema=view.key.schema, + name=view.key.name, + ), + if_exists=True, + statement_recorder=recorder, + ) + _record_drop(adapter=adapter, connection=connection, view=view, run_id=run_id) + dropped.append(view) + for view in plan.missing: + _record_drop(adapter=adapter, connection=connection, view=view, run_id=run_id) + return tuple(dropped) + + +def _record_drop( + *, adapter: BaseAdapter, connection: Any, view: JanitorOldNameView, run_id: str +) -> None: + required: OldNameViewEvent = view.history.required + write_old_name_view_event( + connection=connection, + execute=adapter.execute, + event=OldNameViewEvent( + event_id=deterministic_old_name_view_event_id( + event_type=OldNameViewEventType.VIEW_DROPPED, + migration_event_id=required.migration_event_id, + ), + target_name=required.target_name, + event_type=OldNameViewEventType.VIEW_DROPPED, + migration_event_id=required.migration_event_id, + destination_model=required.destination_model, + old=required.old, + new=required.new, + run_id=run_id, + created_at=datetime.now(tz=UTC), + drop_reason=view.drop_reason, + ), + render_qualified_name=adapter.render_qualified_name, + create_table_sql=adapter.render_create_old_name_view_state_table_sql( + database=required.new.database, schema=required.new.schema or "" + ), + ) + + +def _read_histories( + *, + adapter: BaseAdapter, + connection: Any, + managed_target_schemas: set[tuple[str | None, str | None]], + relations_by_schema: dict[tuple[str | None, str | None], tuple[RelationInfo, ...]], +) -> tuple[OldNameViewHistory, ...]: + histories: list[OldNameViewHistory] = [] + database: str | None + schema: str | None + for database, schema in sorted( + managed_target_schemas, key=lambda key: (key[0] or "", key[1] or "") + ): + names: frozenset[str] = frozenset( + relation.name.lower() for relation in relations_by_schema.get((database, schema), ()) + ) + if schema is None or not {OLD_NAME_VIEW_TABLE_NAME, MIGRATION_TABLE_NAME} <= names: + continue + histories.extend( + read_old_name_view_histories( + connection=connection, + execute=adapter.execute, + database=database, + schema=schema, + stored_columns=stored_old_name_view_columns( + adapter=adapter, connection=connection, database=database, schema=schema + ), + render_qualified_name=adapter.render_qualified_name, + ) + ) + return tuple(histories) + + +def _physical_old_names( + *, + adapter: BaseAdapter, + connection: Any, + histories: tuple[OldNameViewHistory, ...], + relations_by_schema: dict[tuple[str | None, str | None], tuple[RelationInfo, ...]], +) -> dict[JanitorRelationKey, RelationInfo]: + """Look up each old name, listing only schemas the janitor did not already scan.""" + + known: dict[str, tuple[RelationInfo, ...]] = { + (schema or "").lower(): relations for (_, schema), relations in relations_by_schema.items() + } + unscanned: tuple[OldNameViewHistory, ...] = tuple( + history for history in histories if (history.old.schema or "").lower() not in known + ) + listed: tuple[RelationInfo, ...] = ( + adapter.list_relations( + connection=connection, + database=unscanned[0].old.database, + schemas=tuple(sorted({history.old.schema or "" for history in unscanned})), + names=tuple(sorted({history.old.name for history in unscanned})), + ) + if unscanned + else () + ) + found: dict[JanitorRelationKey, RelationInfo] = {} + scanned: list[RelationInfo] = list(listed) + schema_relations: tuple[RelationInfo, ...] + for schema_relations in known.values(): + scanned.extend(schema_relations) + relation: RelationInfo + for relation in scanned: + _ = found.setdefault( + JanitorRelationKey( + database=relation.database, schema=relation.schema, name=relation.name + ), + relation, + ) + physical: dict[JanitorRelationKey, RelationInfo] = {} + history: OldNameViewHistory + for history in histories: + key: JanitorRelationKey = _old_key(history) + match: RelationInfo | None = _lookup(found=found, key=key) + if match is not None: + physical[key] = match + return physical + + +def _view(history: OldNameViewHistory) -> JanitorOldNameView: + return JanitorOldNameView( + key=_old_key(history), + history=history, + expires_at=None if history.created is None else history.created.expires_at, + ) + + +def project_destinations(project: CompiledProject) -> dict[JanitorRelationKey, str]: + """Map each project model's destination to the model's name.""" + + return { + JanitorRelationKey( + database=model.destination.database, + schema=model.destination.schema, + name=model.destination.name, + ): model.name + for model in project.models + } + + +def _old_key(history: OldNameViewHistory) -> JanitorRelationKey: + return JanitorRelationKey( + database=history.old.database, schema=history.old.schema, name=history.old.name + ) + + +def _lookup( + *, found: dict[JanitorRelationKey, RelationInfo], key: JanitorRelationKey +) -> RelationInfo | None: + exact: RelationInfo | None = found.get(key) + if exact is not None or key.database is not None: + return exact + return next( + ( + relation + for candidate, relation in found.items() + if (candidate.schema or "").lower() == (key.schema or "").lower() + and candidate.name.lower() == key.name.lower() + ), + None, + ) + + +def _names(*, name: str, key: JanitorRelationKey) -> bool: + parts: list[str] = [part.strip().strip('"').lower() for part in name.split(".")] + spelled: list[str] = [ + part.lower() for part in (key.database, key.schema, key.name) if part is not None + ] + return spelled[-len(parts) :] == parts + + +def _with_reason(*, view: JanitorOldNameView, reason: OldNameViewDropReason) -> JanitorOldNameView: + return JanitorOldNameView( + key=view.key, history=view.history, expires_at=view.expires_at, drop_reason=reason + ) + + +def is_compatibility_view( + *, + adapter: BaseAdapter, + connection: Any, + view: JanitorOldNameView, + relation: RelationInfo | None = None, +) -> bool: + """Return whether the relation at the old name is still the view SQLBuild defined there.""" + + listed: RelationInfo | None = relation or _listed_relation( + adapter=adapter, connection=connection, key=view.key + ) + if listed is None or not _is_view(listed) or view.key.schema is None: + return False + schema: str = view.key.schema + return any( + adapter.view_definition_matches( + connection=connection, + database=view.key.database, + schema=schema, + name=view.key.name, + sql=sql, + ) + for sql in compatibility_view_sql( + adapter=adapter, connection=connection, history=view.history + ) + ) + + +def _listed_relation( + *, adapter: BaseAdapter, connection: Any, key: JanitorRelationKey +) -> RelationInfo | None: + return next( + ( + relation + for relation in adapter.list_relations( + connection=connection, + database=key.database, + schemas=(key.schema or "",), + names=(key.name,), + ) + if relation.name.lower() == key.name.lower() + ), + None, + ) + + +def _occupied( + *, view: JanitorOldNameView, project_destinations: dict[JanitorRelationKey, str] +) -> JanitorOldNameView: + return JanitorOldNameView( + key=view.key, + history=view.history, + expires_at=view.expires_at, + drop_reason=OldNameViewDropReason.MISSING, + occupied=True, + claimed_by=_claiming_model(key=view.key, project_destinations=project_destinations), + ) + + +def _claiming_model( + *, key: JanitorRelationKey, project_destinations: dict[JanitorRelationKey, str] +) -> str | None: + return next( + ( + model_name + for destination, model_name in project_destinations.items() + if (destination.schema or "").lower() == (key.schema or "").lower() + and destination.name.lower() == key.name.lower() + and ( + destination.database is None + or key.database is None + or destination.database.lower() == key.database.lower() + ) + ), + None, + ) + + +def _is_view(relation: RelationInfo) -> bool: + return normalize_relation_type(relation.relation_type) == RelationType.VIEW diff --git a/src/sqlbuild/executor/old_name_views/main/__init__.py b/src/sqlbuild/executor/old_name_views/main/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/src/sqlbuild/executor/old_name_views/main/_drop_old_name_views.py b/src/sqlbuild/executor/old_name_views/main/_drop_old_name_views.py new file mode 100644 index 0000000000..b66fa14d88 --- /dev/null +++ b/src/sqlbuild/executor/old_name_views/main/_drop_old_name_views.py @@ -0,0 +1,25 @@ +"""Drop expired or requested compatibility views and record the drops.""" + +from __future__ import annotations + +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapter.contract.classes.statement_recorder import StatementRecorder +from sqlbuild.executor.janitor.models import JanitorOldNameView, JanitorOldNameViewPlanning +from sqlbuild.executor.old_name_views._helpers.janitor import apply_old_name_view_drops + + +def drop_old_name_views( + *, + plan: JanitorOldNameViewPlanning, + adapter: BaseAdapter, + connection: Any, + recorder: StatementRecorder, + run_id: str, +) -> tuple[JanitorOldNameView, ...]: + """Drop planned views, then append a drop fact for each dropped or vanished view.""" + + return apply_old_name_view_drops( + plan=plan, adapter=adapter, connection=connection, recorder=recorder, run_id=run_id + ) diff --git a/src/sqlbuild/executor/old_name_views/main/_plan_old_name_view_cleanup.py b/src/sqlbuild/executor/old_name_views/main/_plan_old_name_view_cleanup.py new file mode 100644 index 0000000000..ba7c45882e --- /dev/null +++ b/src/sqlbuild/executor/old_name_views/main/_plan_old_name_view_cleanup.py @@ -0,0 +1,43 @@ +"""Plan janitor cleanup of compatibility views at old model names.""" + +from __future__ import annotations + +from datetime import datetime +from typing import Any + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapter.contract.models import RelationInfo +from sqlbuild.compiler.compile.models import CompiledProject +from sqlbuild.executor.janitor.models import JanitorOldNameViewPlanning, JanitorRelationScope +from sqlbuild.executor.old_name_views._helpers.janitor import ( + plan_old_name_views, + project_destinations, + protect_old_names, +) + + +def plan_old_name_view_cleanup( + *, + adapter: BaseAdapter, + connection: Any, + managed_target_schemas: set[tuple[str | None, str | None]], + relations_by_schema: dict[tuple[str | None, str | None], tuple[RelationInfo, ...]], + target_name: str | None, + early_drops: tuple[str, ...], + project: CompiledProject, + scope: JanitorRelationScope, + now: datetime, +) -> tuple[JanitorOldNameViewPlanning, JanitorRelationScope]: + """Return old-name view actions and a scope that keeps general cleanup away from them.""" + + old_names: JanitorOldNameViewPlanning = plan_old_name_views( + adapter=adapter, + connection=connection, + managed_target_schemas=managed_target_schemas, + relations_by_schema=relations_by_schema, + target_name=target_name, + early_drops=early_drops, + project_destinations=project_destinations(project), + now=now, + ) + return old_names, protect_old_names(scope=scope, old_names=old_names) diff --git a/src/sqlbuild/executor/run/_helpers/execution/promotion.py b/src/sqlbuild/executor/run/_helpers/execution/promotion.py index 3bc21acf4d..99f041e51a 100644 --- a/src/sqlbuild/executor/run/_helpers/execution/promotion.py +++ b/src/sqlbuild/executor/run/_helpers/execution/promotion.py @@ -8,6 +8,7 @@ from sqlbuild.adapter.contract.classes.statement_recorder import StatementRecorder from sqlbuild.adapter.contract.types import PromotionStrategy from sqlbuild.errors.contracts.exceptions import ExecutorInputError +from sqlbuild.executor.run.models import BoundViewGuard from sqlbuild.runtime.observability.classes.operation_lifecycle import ( OperationAttributes, OperationLifecycle, @@ -31,9 +32,12 @@ def promote_relation_to_destination( destination_schema: str | None, destination_name: str, statement_recorder: StatementRecorder, + bound_views: BoundViewGuard | None = None, ) -> None: """Promote an already-created relation into its final destination.""" + guard: BoundViewGuard = bound_views or BoundViewGuard() + with OperationLifecycle( operation_kind="warehouse", operation_name="relation_promotion", @@ -57,31 +61,35 @@ def promote_relation_to_destination( ) try: if strategy == PromotionStrategy.ATOMIC_SWAP: - adapter.swap( - connection=connection, - left=destination_relation, - right=origin_relation, - statement_recorder=statement_recorder, - ) - adapter.drop( - connection=connection, - destination=origin_relation, - if_exists=True, - statement_recorder=statement_recorder, - ) + with guard.transaction(): + adapter.swap( + connection=connection, + left=destination_relation, + right=origin_relation, + statement_recorder=statement_recorder, + ) + _ = guard.rebind() + adapter.drop( + connection=connection, + destination=origin_relation, + if_exists=True, + statement_recorder=statement_recorder, + ) elif strategy == PromotionStrategy.ATOMIC_REPLACE: - adapter.replace_table_from_relation( - connection=connection, - destination=destination_relation, - origin=origin_relation, - statement_recorder=statement_recorder, - ) - adapter.drop( - connection=connection, - destination=origin_relation, - if_exists=True, - statement_recorder=statement_recorder, - ) + with guard.transaction(): + adapter.replace_table_from_relation( + connection=connection, + destination=destination_relation, + origin=origin_relation, + statement_recorder=statement_recorder, + ) + _ = guard.rebind() + adapter.drop( + connection=connection, + destination=origin_relation, + if_exists=True, + statement_recorder=statement_recorder, + ) elif strategy == RENAME_OPERATION_STRATEGY: adapter.rename( connection=connection, diff --git a/src/sqlbuild/executor/run/_helpers/materializations/full_refresh.py b/src/sqlbuild/executor/run/_helpers/materializations/full_refresh.py index 97abb0c853..aeb3029caa 100644 --- a/src/sqlbuild/executor/run/_helpers/materializations/full_refresh.py +++ b/src/sqlbuild/executor/run/_helpers/materializations/full_refresh.py @@ -11,7 +11,7 @@ from sqlbuild.adapter.relations.main.resolve_qualified_name_parts import ( resolve_qualified_name_parts, ) -from sqlbuild.executor.run.models import FullRefreshRelations +from sqlbuild.executor.run.models import BoundViewGuard, FullRefreshRelations from sqlbuild.runtime.observability.classes.operation_lifecycle import ( OperationAttributes, OperationLifecycle, @@ -82,9 +82,12 @@ def promote_full_refresh_rebuild( relations: FullRefreshRelations, target_exists: bool, statement_recorder: StatementRecorder, + bound_views: BoundViewGuard | None = None, ) -> None: """Promote a rebuild; dropping prev at the next swap is retention pruning only.""" + guard: BoundViewGuard = bound_views or BoundViewGuard() + with OperationLifecycle( operation_kind="warehouse", operation_name="relation_promotion", @@ -95,22 +98,24 @@ def promote_full_refresh_rebuild( target_kind="relation", ), ) as lifecycle: - if target_exists: - adapter.drop( + with guard.transaction(): + if target_exists: + adapter.drop( + connection=connection, + destination=relations.previous_qualified, + if_exists=True, + statement_recorder=statement_recorder, + ) + promote_staged_relation( + adapter=adapter, connection=connection, - destination=relations.previous_qualified, - if_exists=True, + target_qualified=relations.target_qualified, + staged_qualified=relations.rebuild_qualified, + displaced_qualified=relations.previous_qualified, + target_exists=target_exists, statement_recorder=statement_recorder, ) - promote_staged_relation( - adapter=adapter, - connection=connection, - target_qualified=relations.target_qualified, - staged_qualified=relations.rebuild_qualified, - displaced_qualified=relations.previous_qualified, - target_exists=target_exists, - statement_recorder=statement_recorder, - ) + _ = guard.rebind() lifecycle.completed(metadata={"changed_count": 1}) diff --git a/src/sqlbuild/executor/run/_helpers/materializations/incremental.py b/src/sqlbuild/executor/run/_helpers/materializations/incremental.py index 64eb3c599a..53f7f01d5a 100644 --- a/src/sqlbuild/executor/run/_helpers/materializations/incremental.py +++ b/src/sqlbuild/executor/run/_helpers/materializations/incremental.py @@ -47,6 +47,7 @@ ) from sqlbuild.executor.run._helpers.validation.type_enforcement import enforce_types_staged from sqlbuild.executor.run.models import ( + BoundViewGuard, FinalAuditRun, HookExecutionResult, ModelExecutionResult, @@ -228,6 +229,7 @@ def execute_incremental_entry( delta_columns=delta_columns, on_schema_change=entry.on_schema_change or _DEFAULT_ON_SCHEMA_CHANGE, statement_recorder=statement_recorder, + bound_views=context.bound_views, ) ) target_columns = inspect_runtime_relation_schema( @@ -589,9 +591,12 @@ def _apply_schema_change( delta_columns: tuple[ColumnInfo, ...], on_schema_change: OnSchemaChange, statement_recorder: StatementRecorder, + bound_views: BoundViewGuard | None = None, ) -> tuple[str, ...]: """Inspect runtime schema diff, apply on_schema_change policy, return new warnings.""" + guard: BoundViewGuard = bound_views or BoundViewGuard() + target_map: dict[str, str] = {col.name.lower(): col.type for col in target_columns} delta_map: dict[str, str] = {col.name.lower(): col.type for col in delta_columns} @@ -677,16 +682,21 @@ def _apply_schema_change( return () if on_schema_change == OnSchemaChange.SYNC_ALL_COLUMNS: - with OperationLifecycle( - operation_kind="warehouse", - operation_name="schema_synchronization", - attributes=OperationAttributes( - phase="apply", - strategy="sync_all_columns", - adapter=canonicalize_operation_adapter(adapter.adapter_name), - target_kind="relation", - ), - ) as lifecycle: + with ( + OperationLifecycle( + operation_kind="warehouse", + operation_name="schema_synchronization", + attributes=OperationAttributes( + phase="apply", + strategy="sync_all_columns", + adapter=canonicalize_operation_adapter(adapter.adapter_name), + target_kind="relation", + ), + ) as lifecycle, + guard.transaction(), + ): + if removed or type_changed: + _ = guard.release() if added: adapter.add_columns( connection=connection, @@ -708,6 +718,7 @@ def _apply_schema_change( columns=tuple(type_changed), statement_recorder=statement_recorder, ) + _ = guard.rebind() lifecycle.completed( metadata={ "changed_count": len(added) + len(removed) + len(type_changed), diff --git a/src/sqlbuild/executor/run/_helpers/materializations/microbatch.py b/src/sqlbuild/executor/run/_helpers/materializations/microbatch.py index 570af4625b..b57dd613fe 100644 --- a/src/sqlbuild/executor/run/_helpers/materializations/microbatch.py +++ b/src/sqlbuild/executor/run/_helpers/materializations/microbatch.py @@ -2845,6 +2845,7 @@ def _apply_microbatch_schema_change( delta_columns=delta_columns, on_schema_change=context.entry.on_schema_change or _DEFAULT_ON_SCHEMA_CHANGE, statement_recorder=state.statement_recorder, + bound_views=context.bound_views, ) except Exception as exc: return MicrobatchSchemaPhaseOutcome( @@ -3129,6 +3130,7 @@ def _promote_microbatch_full_refresh( relations=relations, target_exists=live_exists, statement_recorder=state.statement_recorder, + bound_views=context.bound_views, ) except Exception as exc: return build_failed_result( diff --git a/src/sqlbuild/executor/run/_helpers/materializations/snapshot.py b/src/sqlbuild/executor/run/_helpers/materializations/snapshot.py index fdf6a941b4..1572b3bfaa 100644 --- a/src/sqlbuild/executor/run/_helpers/materializations/snapshot.py +++ b/src/sqlbuild/executor/run/_helpers/materializations/snapshot.py @@ -556,6 +556,7 @@ def _snapshot_promotion_failure( relations=relations, target_exists=target_exists, statement_recorder=statement_recorder, + bound_views=context.bound_views, ) except Exception as exc: return build_failed_result( diff --git a/src/sqlbuild/executor/run/main/_execute.py b/src/sqlbuild/executor/run/main/_execute.py index ffc39cb663..ac5759dc65 100644 --- a/src/sqlbuild/executor/run/main/_execute.py +++ b/src/sqlbuild/executor/run/main/_execute.py @@ -304,6 +304,7 @@ def _staged_lifecycle( destination_schema=targets.target_schema, destination_name=targets.target_table, statement_recorder=statement_recorder, + bound_views=context.bound_views, ) except Exception as exc: return build_failed_result( diff --git a/src/sqlbuild/executor/run/models.py b/src/sqlbuild/executor/run/models.py index 29e7fef1c6..61cdfde382 100644 --- a/src/sqlbuild/executor/run/models.py +++ b/src/sqlbuild/executor/run/models.py @@ -3,6 +3,7 @@ from __future__ import annotations from collections.abc import Callable, Mapping +from contextlib import AbstractContextManager, nullcontext from dataclasses import dataclass, field from datetime import datetime from typing import Any, cast @@ -266,6 +267,15 @@ class AuditGateMetadataParseFailureDetail: detail: str +@dataclass(frozen=True) +class BoundViewGuard: + """Keeps views bound to a model's relation valid inside the DDL that replaces or alters it.""" + + transaction: Callable[[], AbstractContextManager[object]] = nullcontext + release: Callable[[], int] = int + rebind: Callable[[], int] = int + + @dataclass(frozen=True) class ModelMaterializationContext: """Shared runtime inputs for one model's materialization lifecycle.""" @@ -296,6 +306,7 @@ class ModelMaterializationContext: microbatch_batch_runner: MicrobatchBatchRunner | None = None watermark_resolver: WatermarkResolver | None = None enforce_explicit_references: bool = False + bound_views: BoundViewGuard = field(default_factory=BoundViewGuard) @dataclass(frozen=True, kw_only=True, init=False) diff --git a/src/sqlbuild/spec/contracts/models.py b/src/sqlbuild/spec/contracts/models.py index 044f123326..9559712e85 100644 --- a/src/sqlbuild/spec/contracts/models.py +++ b/src/sqlbuild/spec/contracts/models.py @@ -298,6 +298,13 @@ class JanitorConfig: exclude_patterns: tuple[str, ...] = field(default_factory=tuple) +@dataclass(frozen=True) +class MigrationsConfig: + """Model migration defaults.""" + + old_name_views: str | None = "30d" + + @dataclass(frozen=True) class SnapshotsConfig: """Snapshot materialization safety defaults.""" @@ -398,6 +405,7 @@ class ProjectConfig: vars: dict[str, str] = field(default_factory=dict) targets: dict[str, TargetConfig] = field(default_factory=dict) janitor: JanitorConfig = field(default_factory=JanitorConfig) + migrations: MigrationsConfig = field(default_factory=MigrationsConfig) snapshots: SnapshotsConfig = field(default_factory=SnapshotsConfig) scenario: ScenarioConfig = field(default_factory=ScenarioConfig) dbt: DbtConfig = field(default_factory=DbtConfig) diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/model_migrations/_test_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/model_migrations/_test_types.py index 5d2b72d9f3..228ea3e670 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/model_migrations/_test_types.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/model_migrations/_test_types.py @@ -24,3 +24,73 @@ class ColumnMigrationLifecycleE2ETestCase: expected_near_match_hint: str expected_final_columns: tuple[str, ...] expected_events: tuple[tuple[str, str, str, str], ...] + + +@dataclass(frozen=True) +class OldNameViewMaterializationE2ETestCase: + description: str + materialized: str + expected_old_name_type: str + expected_facts: tuple[str, ...] + expected_rows: tuple[tuple[int, int], ...] + + +@dataclass(frozen=True) +class OldNameViewAliasE2ETestCase: + description: str + materialized: str + origin_columns: str + destination_columns: str + destination_schema: str + expected_old_columns: tuple[str, ...] + expected_aliases: dict[str, str] + + +@dataclass(frozen=True) +class OldNameViewConfigE2ETestCase: + description: str + project_toml_extra: str + extra_config: str + expected_old_name_type: str + expected_facts: tuple[str, ...] + expected_retention: tuple[str | None, ...] + expected_plan_fragment: str + + +@dataclass(frozen=True) +class OldNameViewJanitorE2ETestCase: + description: str + extra_config: str + janitor_args: tuple[str, ...] + wait_seconds: float + expected_janitor_fragment: str + expected_old_name_type: str | None + expected_drop_reasons: tuple[str, ...] + + +@dataclass(frozen=True) +class OldNameClaimE2ETestCase: + description: str + expected_error_fragments: tuple[str, ...] + + +@dataclass(frozen=True) +class OldNameReferenceE2ETestCase: + description: str + header: str + command: str + expected_fragment: str + + +@dataclass(frozen=True) +class OldNameGatingE2ETestCase: + description: str + expected_old_name_type: str + expected_facts: tuple[str, ...] + + +@dataclass(frozen=True) +class OldNameUnknownDropE2ETestCase: + description: str + requested_name: str + expected_code: str diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/model_migrations/helpers.py b/tests/e2e/src/sqlbuild/cli/commands/main/model_migrations/helpers.py index 7005eb1b3c..26b32b58a4 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/model_migrations/helpers.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/model_migrations/helpers.py @@ -4,6 +4,7 @@ import json import subprocess +from collections.abc import Callable from pathlib import Path from typing import Any @@ -17,6 +18,7 @@ DATABASE_FILE: str = "orders.duckdb" _PROJECT_TOML: str = ( f'name = "orders_project"\nadapter = "duckdb"\n\n[connection]\ndatabase = "{DATABASE_FILE}"\n' + "\n[migrations]\nold_name_views = false\n" ) _SOURCES_YML: str = "sources:\n - name: raw_orders\n schema: main\n table: raw_orders\n" @@ -247,3 +249,218 @@ def column_migration_events(*, project_dir: Path) -> tuple[tuple[str, str, str, ), ) ) + + +OLD_NAME_ORIGIN: str = "revenue" +OLD_NAME_DESTINATION: str = "daily_revenue" +OLD_NAME_SCHEMA: str = "analytics" +_OLD_NAME_PROJECT_TOML: str = ( + 'name = "orders_project"\nadapter = "duckdb"\ndefault_target = "dev"\n\n' + f'[connection]\ndatabase = "{DATABASE_FILE}"\n\n' + f'[targets.dev]\nschema = "{OLD_NAME_SCHEMA}"\n\n' + "[janitor]\nenabled = true\n" +) +_OLD_NAME_HEADERS: dict[str, str] = { + "table": " materialized table,\n", + "view": " materialized view,\n", + "incremental": ( + " materialized incremental,\n" + " incremental_strategy delete_insert,\n" + " unique_key order_id,\n" + " cursor order_date,\n" + " cursor_type timestamp,\n" + " cursor_grain day,\n" + ' cursor_start "2026-01-01",\n' + ), + "snapshot": ( + " materialized snapshot,\n" + " unique_key [order_id],\n" + " snapshot_strategy timestamp,\n" + " updated_at order_date,\n" + ), +} +_OLD_NAME_FACT_ORDER: tuple[str, ...] = ( + "required", + "origin_archived", + "view_created", + "view_dropped", +) + + +def old_name_model_sql( + *, materialized: str, columns: str = "amount_cents", extra_config: str = "" +) -> str: + """Return a model of the given materialization over raw orders.""" + + return ( + "MODEL (\n" + f"{_OLD_NAME_HEADERS[materialized]}" + f"{extra_config}" + ");\n\n" + f'SELECT order_id, order_date, {columns} FROM __source("raw_orders")\n' + ) + + +def write_old_name_project( + *, + tmp_path: Path, + models: dict[str, str], + project_toml_extra: str = "", + python_files: dict[str, str] | None = None, +) -> Path: + """Write a project with exactly the given models and optional Python files.""" + + project_dir: Path = tmp_path / "orders_project" + stale: Path + for stale in (project_dir / "models").glob("*.sql"): + stale.unlink() + for stale in (project_dir / "python").glob("*.py"): + stale.unlink() + files: dict[str, str] = { + "sqlbuild_project.toml": _OLD_NAME_PROJECT_TOML + project_toml_extra, + "sources/raw.yml": _SOURCES_YML, + **{f"models/{name}.sql": sql for name, sql in models.items()}, + **{f"python/{name}": source for name, source in (python_files or {}).items()}, + } + return prepare_inline_project( + tmp_path=tmp_path, project_name="orders_project", repo_files=files + ) + + +def old_name_facts(*, project_dir: Path) -> tuple[str, ...]: + """Return old-name fact types in lifecycle order, or () when the table is absent.""" + + readers: dict[bool, Callable[[Path], tuple[str, ...]]] = { + True: _read_old_name_facts, + False: _no_old_name_facts, + } + exists: bool = ( + relation_type(project_dir=project_dir, name="_sqlbuild_old_name_views") is not None + ) + return readers[exists](project_dir) + + +def _read_old_name_facts(project_dir: Path) -> tuple[str, ...]: + return tuple( + str(row[0]) + for row in query_duckdb( + db_path=project_dir / DATABASE_FILE, + sql=( + f"SELECT event_type FROM {OLD_NAME_SCHEMA}._sqlbuild_old_name_views ORDER BY " + f"list_position({list(_OLD_NAME_FACT_ORDER)}, event_type), created_at" + ), + ) + ) + + +def _no_old_name_facts(project_dir: Path) -> tuple[str, ...]: + del project_dir + return () + + +def old_name_fact_values(*, project_dir: Path, event_type: str, column: str) -> tuple[Any, ...]: + """Return one column of every old-name fact of one type, or () when the table is absent.""" + + sql: str = ( + f"SELECT {column} FROM {OLD_NAME_SCHEMA}._sqlbuild_old_name_views " + f"WHERE event_type = '{event_type}' ORDER BY created_at" + ) + readers: dict[bool, Callable[[], tuple[Any, ...]]] = { + True: lambda: tuple( + row[0] for row in query_duckdb(db_path=project_dir / DATABASE_FILE, sql=sql) + ), + False: tuple, + } + table_type: str | None = relation_type(project_dir=project_dir, name="_sqlbuild_old_name_views") + return readers[table_type is not None]() + + +def relation_type(*, project_dir: Path, name: str) -> str | None: + """Return the information_schema table type of one relation in main, or None.""" + + rows: list[tuple[Any, ...]] = query_duckdb( + db_path=project_dir / DATABASE_FILE, + sql=( + "SELECT table_type FROM information_schema.tables " + f"WHERE table_schema = '{OLD_NAME_SCHEMA}' AND table_name = '{name}'" + ), + ) + return next((str(row[0]) for row in rows), None) + + +def relation_rows(*, project_dir: Path, relation: str, columns: str) -> tuple[tuple[Any, ...], ...]: + """Return the given columns of one relation ordered by order_id.""" + + return tuple( + tuple(row) + for row in query_duckdb( + db_path=project_dir / DATABASE_FILE, + sql=f"SELECT {columns} FROM {relation} ORDER BY order_id", + ) + ) + + +def origin_archive_count(*, project_dir: Path) -> int: + """Return how many migration_origin archives exist in main.""" + + return int( + query_duckdb( + db_path=project_dir / DATABASE_FILE, + sql=( + "SELECT count(*) FROM information_schema.tables " + f"WHERE table_schema = '{OLD_NAME_SCHEMA}' " + "AND contains(table_name, '__migration_origin__')" + ), + )[0][0] + ) + + +def sqb(*, project_dir: Path, args: tuple[str, ...]) -> subprocess.CompletedProcess[str]: + """Run one sqb command without colour in a subprocess.""" + + return run_sqb(command=("--no-color", *args), project_dir=project_dir) + + +OLD_NAME_MIGRATE_FROM: str = f" migrate_from {OLD_NAME_ORIGIN},\n" + + +def build_old_name_project(project_dir: Path) -> subprocess.CompletedProcess[str]: + """Run sqb build and require success.""" + + result: subprocess.CompletedProcess[str] = sqb(project_dir=project_dir, args=("build",)) + assert result.returncode == 0, result.stdout + result.stderr + return result + + +def prepare_old_name_rename( + *, + tmp_path: Path, + materialized: str, + origin_columns: str = "amount_cents", + destination_columns: str = "amount_cents", + destination_schema: str = "", + extra_config: str = "", + project_toml_extra: str = "", +) -> Path: + """Build the origin model, then declare its rename to the destination.""" + + project_dir: Path = write_old_name_project( + tmp_path=tmp_path, + models={ + OLD_NAME_ORIGIN: old_name_model_sql(materialized=materialized, columns=origin_columns) + }, + project_toml_extra=project_toml_extra, + ) + load_raw_orders(project_dir=project_dir, last_day=3) + _ = build_old_name_project(project_dir) + return write_old_name_project( + tmp_path=tmp_path, + models={ + OLD_NAME_DESTINATION: old_name_model_sql( + materialized=materialized, + columns=destination_columns, + extra_config=OLD_NAME_MIGRATE_FROM + destination_schema + extra_config, + ) + }, + project_toml_extra=project_toml_extra, + ) diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/model_migrations/test_old_name_views.py b/tests/e2e/src/sqlbuild/cli/commands/main/model_migrations/test_old_name_views.py new file mode 100644 index 0000000000..4622bb60a0 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/model_migrations/test_old_name_views.py @@ -0,0 +1,472 @@ +"""CLI e2e coverage for compatibility views that keep migrated models' old names working.""" + +from __future__ import annotations + +import subprocess +import time +from pathlib import Path +from typing import Any + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.model_migrations._test_types import ( + OldNameClaimE2ETestCase, + OldNameGatingE2ETestCase, + OldNameReferenceE2ETestCase, + OldNameUnknownDropE2ETestCase, + OldNameViewAliasE2ETestCase, + OldNameViewConfigE2ETestCase, + OldNameViewJanitorE2ETestCase, + OldNameViewMaterializationE2ETestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.main.model_migrations.helpers import ( + OLD_NAME_DESTINATION, + OLD_NAME_MIGRATE_FROM, + OLD_NAME_ORIGIN, + OLD_NAME_SCHEMA, + build_old_name_project, + load_raw_orders, + old_name_fact_values, + old_name_facts, + old_name_model_sql, + origin_archive_count, + plan_payload, + prepare_old_name_rename, + relation_rows, + relation_type, + sqb, + write_old_name_project, +) + +_ALL_FACTS: tuple[str, ...] = ("required", "origin_archived", "view_created") +_THREE_ORDERS: tuple[tuple[int, int], ...] = ((1, 101), (2, 102), (3, 103)) +_REPORT_TASK: str = ( + "from sqlbuild.tasks import task\n\n\n" + "@task\n" + "def report(ctx):\n" + f' ctx.query("SELECT count(*) FROM {OLD_NAME_SCHEMA}.{OLD_NAME_ORIGIN}").fetchall()\n' +) + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameViewMaterializationE2ETestCase( + description=materialized, + materialized=materialized, + expected_old_name_type="VIEW", + expected_facts=_ALL_FACTS, + expected_rows=_THREE_ORDERS, + ) + for materialized in ("table", "view", "incremental", "snapshot") + ], + ids=lambda case: case.description, +) +def test_given_renamed_model_when_building_then_old_name_reads_the_new_relation( + tmp_path: Path, test_case: OldNameViewMaterializationE2ETestCase +) -> None: + """Every materialization archives its old relation once and serves a view at the old name.""" + + project_dir: Path = prepare_old_name_rename( + tmp_path=tmp_path, materialized=test_case.materialized + ) + planned: dict[str, Any] = plan_payload(project_dir=project_dir) + + _ = build_old_name_project(project_dir) + load_raw_orders(project_dir=project_dir, last_day=4) + _ = build_old_name_project(project_dir) + rebuilt: dict[str, Any] = plan_payload(project_dir=project_dir) + + assert planned["old_names"][0]["action"] == "archive_and_view" + assert relation_type(project_dir=project_dir, name=OLD_NAME_ORIGIN) == ( + test_case.expected_old_name_type + ) + assert old_name_facts(project_dir=project_dir) == test_case.expected_facts + assert origin_archive_count(project_dir=project_dir) == 1 + assert relation_rows( + project_dir=project_dir, relation=f"{OLD_NAME_SCHEMA}.{OLD_NAME_ORIGIN}", columns="*" + ) == relation_rows( + project_dir=project_dir, relation=f"{OLD_NAME_SCHEMA}.{OLD_NAME_DESTINATION}", columns="*" + ) + assert ( + relation_rows( + project_dir=project_dir, + relation=f"{OLD_NAME_SCHEMA}.{OLD_NAME_ORIGIN}", + columns="order_id, amount_cents", + )[:3] + == test_case.expected_rows + ) + assert rebuilt["migrations"][0]["old_name"]["action"] == "live" + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameViewAliasE2ETestCase( + description="incremental column renamed in place", + materialized="incremental", + origin_columns="amount_cents", + destination_columns="amount_cents AS revenue_cents", + destination_schema=" columns (revenue_cents (migrate_from amount_cents)),\n", + expected_old_columns=("order_id", "order_date", "amount_cents"), + expected_aliases={"amount_cents": "revenue_cents"}, + ), + OldNameViewAliasE2ETestCase( + description="table column declared as an alias", + materialized="table", + origin_columns="amount_cents", + destination_columns="amount_cents AS revenue_cents", + destination_schema=" columns (revenue_cents (migrate_from amount_cents)),\n", + expected_old_columns=("order_id", "order_date", "amount_cents"), + expected_aliases={"amount_cents": "revenue_cents"}, + ), + ], + ids=lambda case: case.description, +) +def test_given_renamed_column_when_building_then_old_name_view_uses_old_column_names( + tmp_path: Path, test_case: OldNameViewAliasE2ETestCase +) -> None: + """The compatibility view presents renamed columns under the names consumers used.""" + + project_dir: Path = prepare_old_name_rename( + tmp_path=tmp_path, + materialized=test_case.materialized, + origin_columns=test_case.origin_columns, + destination_columns=test_case.destination_columns, + destination_schema=test_case.destination_schema, + ) + + _ = build_old_name_project(project_dir) + planned: dict[str, Any] = plan_payload(project_dir=project_dir) + + old_columns: tuple[str, ...] = tuple( + str(row[0]) + for row in relation_rows( + project_dir=project_dir, + relation=( + "(SELECT column_name, ordinal_position AS order_id FROM " + f"information_schema.columns WHERE table_schema = '{OLD_NAME_SCHEMA}' " + f"AND table_name = '{OLD_NAME_ORIGIN}')" + ), + columns="column_name", + ) + ) + assert old_columns == test_case.expected_old_columns + assert ( + relation_rows( + project_dir=project_dir, + relation=f"{OLD_NAME_SCHEMA}.{OLD_NAME_ORIGIN}", + columns="order_id, amount_cents", + ) + == _THREE_ORDERS + ) + assert planned["old_names"][0]["column_aliases"] == test_case.expected_aliases + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameViewConfigE2ETestCase( + description="project default keeps the view for thirty days", + project_toml_extra="", + extra_config="", + expected_old_name_type="VIEW", + expected_facts=_ALL_FACTS, + expected_retention=("30d",), + expected_plan_fragment=("└── old name analytics.revenue\n ├── view until "), + ), + OldNameViewConfigE2ETestCase( + description="model header overrides the project retention", + project_toml_extra='\n[migrations]\nold_name_views = "90d"\n', + extra_config=" old_name_view 7d,\n", + expected_old_name_type="VIEW", + expected_facts=_ALL_FACTS, + expected_retention=("7d",), + expected_plan_fragment=("└── old name analytics.revenue\n ├── view until "), + ), + OldNameViewConfigE2ETestCase( + description="project false keeps the old relation untouched", + project_toml_extra="\n[migrations]\nold_name_views = false\n", + extra_config="", + expected_old_name_type="BASE TABLE", + expected_facts=(), + expected_retention=(), + expected_plan_fragment=( + "└── old name analytics.revenue\n └── left for janitor old_name_view false" + ), + ), + OldNameViewConfigE2ETestCase( + description="model header false overrides the project default", + project_toml_extra="", + extra_config=" old_name_view false,\n", + expected_old_name_type="BASE TABLE", + expected_facts=(), + expected_retention=(), + expected_plan_fragment=( + "└── old name analytics.revenue\n └── left for janitor old_name_view false" + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_old_name_view_config_when_building_then_retention_follows_config( + tmp_path: Path, test_case: OldNameViewConfigE2ETestCase +) -> None: + """Project and model retention decide whether and how long the old name is kept.""" + + project_dir: Path = prepare_old_name_rename( + tmp_path=tmp_path, + materialized="table", + extra_config=test_case.extra_config, + project_toml_extra=test_case.project_toml_extra, + ) + plan_text: str = sqb(project_dir=project_dir, args=("plan",)).stdout + + _ = build_old_name_project(project_dir) + + assert test_case.expected_plan_fragment in plan_text, plan_text + assert relation_type(project_dir=project_dir, name=OLD_NAME_ORIGIN) == ( + test_case.expected_old_name_type + ) + assert old_name_facts(project_dir=project_dir) == test_case.expected_facts + assert ( + old_name_fact_values( + project_dir=project_dir, event_type="required", column="view_retention" + ) + == test_case.expected_retention + ) + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameGatingE2ETestCase( + description="move recorded while views were off", + expected_old_name_type="BASE TABLE", + expected_facts=(), + ) + ], + ids=lambda case: case.description, +) +def test_given_move_recorded_without_old_name_view_when_enabling_then_old_name_is_untouched( + tmp_path: Path, test_case: OldNameGatingE2ETestCase +) -> None: + """Moves recorded while views were off, as before the upgrade, never gain a view later.""" + + project_dir: Path = prepare_old_name_rename( + tmp_path=tmp_path, + materialized="table", + project_toml_extra="\n[migrations]\nold_name_views = false\n", + ) + _ = build_old_name_project(project_dir) + project_dir = write_old_name_project( + tmp_path=tmp_path, + models={ + OLD_NAME_DESTINATION: old_name_model_sql( + materialized="table", extra_config=OLD_NAME_MIGRATE_FROM + ) + }, + ) + + _ = build_old_name_project(project_dir) + + assert relation_type(project_dir=project_dir, name=OLD_NAME_ORIGIN) == ( + test_case.expected_old_name_type + ) + assert old_name_facts(project_dir=project_dir) == test_case.expected_facts + assert plan_payload(project_dir=project_dir)["old_names"] == [] + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameViewJanitorE2ETestCase( + description="unexpired view is kept and listed", + extra_config="", + janitor_args=(), + wait_seconds=0, + expected_janitor_fragment=( + "└── analytics.revenue -> model:daily_revenue\n └── expires " + ), + expected_old_name_type="VIEW", + expected_drop_reasons=(), + ), + OldNameViewJanitorE2ETestCase( + description="expired view is dropped", + extra_config=" old_name_view 1s,\n", + janitor_args=(), + wait_seconds=1.5, + expected_janitor_fragment=( + "└── analytics.revenue -> model:daily_revenue\n └── drop now (expired " + ), + expected_old_name_type=None, + expected_drop_reasons=("expired",), + ), + OldNameViewJanitorE2ETestCase( + description="requested view is dropped early", + extra_config="", + janitor_args=("--drop-old-name-view", "analytics.revenue"), + wait_seconds=0, + expected_janitor_fragment=" └── drop now (requested; would expire ", + expected_old_name_type=None, + expected_drop_reasons=("early",), + ), + ], + ids=lambda case: case.description, +) +def test_given_old_name_view_when_running_janitor_then_it_drops_only_expired_or_requested( + tmp_path: Path, test_case: OldNameViewJanitorE2ETestCase +) -> None: + """The janitor drops expired or requested views, records the drop, and spares live ones.""" + + project_dir: Path = prepare_old_name_rename( + tmp_path=tmp_path, materialized="table", extra_config=test_case.extra_config + ) + _ = build_old_name_project(project_dir) + time.sleep(test_case.wait_seconds) + + janitor: subprocess.CompletedProcess[str] = sqb( + project_dir=project_dir, args=("janitor", "--auto-approve", *test_case.janitor_args) + ) + + assert janitor.returncode == 0, janitor.stdout + janitor.stderr + assert test_case.expected_janitor_fragment in janitor.stdout, janitor.stdout + assert relation_type(project_dir=project_dir, name=OLD_NAME_ORIGIN) == ( + test_case.expected_old_name_type + ) + assert ( + old_name_fact_values( + project_dir=project_dir, event_type="view_dropped", column="drop_reason" + ) + == test_case.expected_drop_reasons + ) + assert origin_archive_count(project_dir=project_dir) == 1 + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameUnknownDropE2ETestCase( + description="name with no recorded compatibility view", + requested_name="analytics.orders", + expected_code="C503", + ) + ], + ids=lambda case: case.description, +) +def test_given_unknown_old_name_when_dropping_early_then_janitor_refuses( + tmp_path: Path, test_case: OldNameUnknownDropE2ETestCase +) -> None: + """An early drop must name a recorded compatibility view.""" + + project_dir: Path = prepare_old_name_rename(tmp_path=tmp_path, materialized="table") + _ = build_old_name_project(project_dir) + + janitor: subprocess.CompletedProcess[str] = sqb( + project_dir=project_dir, + args=("janitor", "--auto-approve", "--drop-old-name-view", test_case.requested_name), + ) + + assert janitor.returncode != 0 + assert test_case.expected_code in janitor.stdout + janitor.stderr + assert relation_type(project_dir=project_dir, name=OLD_NAME_ORIGIN) == "VIEW" + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameClaimE2ETestCase( + description="new model named like a live compatibility view", + expected_error_fragments=( + "M114", + "would replace analytics.revenue, a compatibility view for model:daily_revenue until", + "sqb janitor --drop-old-name-view analytics.revenue", + ), + ) + ], + ids=lambda case: case.description, +) +def test_given_live_compatibility_view_when_new_model_claims_its_name_then_build_is_refused( + tmp_path: Path, test_case: OldNameClaimE2ETestCase +) -> None: + """A model cannot silently replace a compatibility view; an early drop releases the name.""" + + project_dir: Path = prepare_old_name_rename(tmp_path=tmp_path, materialized="table") + _ = build_old_name_project(project_dir) + project_dir = write_old_name_project( + tmp_path=tmp_path, + models={ + OLD_NAME_DESTINATION: old_name_model_sql(materialized="table"), + OLD_NAME_ORIGIN: old_name_model_sql(materialized="table", columns="0 AS other"), + }, + ) + + refused: subprocess.CompletedProcess[str] = sqb(project_dir=project_dir, args=("build",)) + dropped: subprocess.CompletedProcess[str] = sqb( + project_dir=project_dir, + args=("janitor", "--auto-approve", "--drop-old-name-view", "analytics.revenue"), + ) + _ = build_old_name_project(project_dir) + + assert refused.returncode != 0 + assert all( + fragment in refused.stdout + refused.stderr + for fragment in test_case.expected_error_fragments + ), refused.stdout + refused.stderr + assert dropped.returncode == 0, dropped.stdout + dropped.stderr + assert relation_type(project_dir=project_dir, name=OLD_NAME_ORIGIN) == "BASE TABLE" + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameReferenceE2ETestCase( + description="declared rename is rejected at compile time", + header=OLD_NAME_MIGRATE_FROM, + command="compile", + expected_fragment=( + "error[P008]: task:report names 'analytics.revenue' in SQL passed to ctx.query(); " + "it is the old name of model:daily_revenue" + ), + ), + OldNameReferenceE2ETestCase( + description="discovered rename is rejected at plan time", + header="", + command="build", + expected_fragment=( + "task:report names 'analytics.revenue' in SQL passed to ctx.query(); " + "it is the old name of model:daily_revenue" + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_python_sql_reading_old_name_when_checking_then_p008_rejects_it( + tmp_path: Path, test_case: OldNameReferenceE2ETestCase +) -> None: + """Project code must move to the new model; the old name is only for outside consumers.""" + + project_dir: Path = write_old_name_project( + tmp_path=tmp_path, models={OLD_NAME_ORIGIN: old_name_model_sql(materialized="table")} + ) + load_raw_orders(project_dir=project_dir, last_day=3) + _ = build_old_name_project(project_dir) + project_dir = write_old_name_project( + tmp_path=tmp_path, + models={ + OLD_NAME_DESTINATION: old_name_model_sql( + materialized="table", extra_config=test_case.header + ) + }, + python_files={"tasks.py": _REPORT_TASK}, + ) + + result: subprocess.CompletedProcess[str] = sqb( + project_dir=project_dir, args=(test_case.command,) + ) + + assert result.returncode != 0, result.stdout + result.stderr + assert test_case.expected_fragment in result.stdout + result.stderr, ( + result.stdout + result.stderr + ) + assert relation_type(project_dir=project_dir, name=OLD_NAME_ORIGIN) == "BASE TABLE" diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/postgres/_test_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/postgres/_test_types.py index 1de6d26f50..eca7d7aec1 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/postgres/_test_types.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/postgres/_test_types.py @@ -1,5 +1,8 @@ +from collections.abc import Callable from dataclasses import dataclass, field +import pytest + @dataclass(frozen=True) class PostgresBuildE2ETestCase: @@ -171,3 +174,63 @@ class PostgresColumnMigrationRollbackE2ETestCase: expected_columns_after_failure: tuple[tuple[object, ...], ...] expected_final_columns: tuple[tuple[object, ...], ...] expected_events: tuple[tuple[object, ...], ...] + + +@dataclass(frozen=True) +class PostgresOldNameViewE2ETestCase: + description: str + materialized: str + expected_old_name_kind: str + expected_ids_after_rebuild: tuple[tuple[object, ...], ...] + expected_warning_fragment: str + expected_external_ids: tuple[tuple[object, ...], ...] + + +@dataclass(frozen=True) +class PostgresBoundViewCrashE2ETestCase: + description: str + materialized: str + changed_columns: str + install_failure: Callable[[pytest.MonkeyPatch], None] + expected_crash_exit_code: int + expected_view_ids_after_crash: tuple[tuple[object, ...], ...] + expected_view_ids_after_retry: tuple[tuple[object, ...], ...] + + +@dataclass(frozen=True) +class PostgresOldNameGrantsE2ETestCase: + description: str + reader_role: str + expected_reader_ids: tuple[tuple[object, ...], ...] + expected_grants: tuple[str, ...] + expected_new_table_error: str + expected_plan_fragment: str + + +@dataclass(frozen=True) +class PostgresOldNameRevokeE2ETestCase: + description: str + materialized: str + rebuilt_columns: str + revoked_role: str + granted_role: str + expected_revoked_error: str + expected_granted_ids: tuple[tuple[object, ...], ...] + + +@dataclass(frozen=True) +class PostgresOldNameJanitorE2ETestCase: + description: str + replacement_sql: str + expected_janitor_fragment: str + expected_old_name_kinds: tuple[tuple[object, ...], ...] + + +@dataclass(frozen=True) +class PostgresDefaultPrivilegesE2ETestCase: + description: str + materialized: str + rebuilt_columns: str + role: str + revoke_before_rebuild: bool + expected_error: str diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/postgres/helpers.py b/tests/e2e/src/sqlbuild/cli/commands/main/postgres/helpers.py index f12dac554c..56c6e93818 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/postgres/helpers.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/postgres/helpers.py @@ -16,6 +16,7 @@ prepare_inline_project, prepare_source_loader_strategies, prepare_waffle_shop, + run_sqb, stringify_warehouse_rows, ) @@ -680,3 +681,196 @@ def create_unwritable_column_migration_table( ), config=config, ) + + +_OLD_NAME_HEADERS: dict[str, str] = { + "table": " materialized table,\n", + "incremental": ( + " materialized incremental,\n" + " incremental_strategy delete_insert,\n" + " unique_key order_id,\n" + " cursor order_date,\n" + " cursor_type timestamp,\n" + " cursor_grain day,\n" + ' cursor_start "2026-01-01",\n' + ), + "incremental_sync": ( + " materialized incremental,\n" + " incremental_strategy delete_insert,\n" + " unique_key order_id,\n" + " cursor order_date,\n" + " cursor_type timestamp,\n" + " cursor_grain day,\n" + ' cursor_start "2026-01-01",\n' + " on_schema_change sync_all_columns,\n" + ), +} + + +def old_name_model_sql( + *, + materialized: str, + migrate_from: str = "", + columns: str = "order_id, order_date, amount_cents", +) -> str: + """Return a table or incremental orders model with an optional migrate_from header.""" + + migration: str = {"": ""}.get(migrate_from, f" migrate_from {migrate_from},\n") + return ( + f"MODEL (\n{_OLD_NAME_HEADERS[materialized]}{migration});\n\n" + f'SELECT {columns} FROM __source("raw_orders")\n' + ) + + +def build_ok(project_dir: Path) -> subprocess.CompletedProcess[str]: + """Run sqb build and require success.""" + + result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "build"), project_dir=project_dir + ) + assert result.returncode == 0, result.stdout + result.stderr + return result + + +def _crash(connection: Any) -> None: + """Die like a killed process: the session ends, so nothing after this point runs or commits.""" + + connection.close() + raise RuntimeError("simulated crash") + + +def _crashing_execute( + monkeypatch: pytest.MonkeyPatch, *, crashes_at: Callable[[str, list[str]], bool] +) -> None: + original: Callable[..., Any] = PostgresAdapter.execute + executed: list[str] = [] + + def crash(self: PostgresAdapter, *, connection: Any, sql: str) -> Any: + del self, sql + _crash(connection) + + outcomes: dict[bool, Callable[..., Any]] = {True: crash, False: original} + + def execute(self: PostgresAdapter, *, connection: Any, sql: str) -> Any: + executed.append(sql) + return outcomes[crashes_at(sql, executed)](self, connection=connection, sql=sql) + + monkeypatch.setattr(PostgresAdapter, "execute", execute) + + +def _is_second_swap_rename(sql: str, executed: list[str]) -> bool: + swap_renames: list[str] = [*filter(lambda statement: "__swap_staging" in statement, executed)] + return swap_renames[1:2] == [sql] + + +def _is_compatibility_view_rebind(sql: str, executed: list[str]) -> bool: + del executed + target: str = sql.split(" AS ")[0].replace('"', "") + return target.startswith("CREATE OR REPLACE VIEW") and target.endswith(".revenue") + + +def _is_displaced_drop(sql: str, executed: list[str]) -> bool: + del executed + return sql.startswith("DROP TABLE") and "__staging" in sql + + +def _is_column_drop(sql: str, executed: list[str]) -> bool: + del executed + return "DROP COLUMN" in sql + + +def fail_mid_swap(monkeypatch: pytest.MonkeyPatch) -> None: + """Crash on the second rename of a swap, after the new relation already holds the name.""" + + _crashing_execute(monkeypatch, crashes_at=_is_second_swap_rename) + + +def fail_bound_view_rebind(monkeypatch: pytest.MonkeyPatch) -> None: + """Crash while the compatibility view is being pointed at the new relation.""" + + _crashing_execute(monkeypatch, crashes_at=_is_compatibility_view_rebind) + + +def fail_displaced_drop(monkeypatch: pytest.MonkeyPatch) -> None: + """Crash after compatibility views follow the new relation, before the old one is dropped.""" + + _crashing_execute(monkeypatch, crashes_at=_is_displaced_drop) + + +def fail_column_drop(monkeypatch: pytest.MonkeyPatch) -> None: + """Crash on the column drop of an incremental schema change.""" + + _crashing_execute(monkeypatch, crashes_at=_is_column_drop) + + +def no_postgres_failure(monkeypatch: pytest.MonkeyPatch) -> None: + """Install nothing.""" + + del monkeypatch + + +def build_in_process(*, project_dir: Path, capsys: pytest.CaptureFixture[str]) -> int: + """Run sqb build in-process so faults can be injected, returning its exit code.""" + + from sqlbuild.cli.commands.main.entrypoint.entry import main + + _ = capsys.readouterr() + exit_code: int = main(["--project-dir", str(project_dir), "--no-color", "build"]) + output: str = "".join(capsys.readouterr()) + assert exit_code in (0, 1), output + return exit_code + + +def reader_rows( + *, config: dict[str, object], role: str, sql: str +) -> tuple[tuple[object, ...], ...]: + """Run one query as a login role that holds only the privileges granted to it.""" + + reader: dict[str, object] = {**config, "user": role, "password": role} + return fetch_postgres_rows(sql=sql, config=reader) + + +def reader_error(*, config: dict[str, object], role: str, sql: str) -> str: + """Return the error a login role gets for one query, or an empty string.""" + + try: + _ = reader_rows(config=config, role=role, sql=sql) + except Exception as error: + return str(error).splitlines()[0] + return "" + + +def create_login_role(*, role: str, config: dict[str, object]) -> None: + """Create a login role whose password is its name, unless it exists.""" + + execute_postgres_sql( + sql=( + f"DO $$ BEGIN CREATE ROLE {role} LOGIN PASSWORD '{role}'; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$" + ), + config=config, + ) + + +def replace_view(*, view: str, sql: str, config: dict[str, object]) -> None: + """Replace a view with one defined by ``sql``; an empty ``sql`` leaves it alone.""" + + statements: dict[bool, tuple[str, ...]] = { + True: (), + False: (f"DROP VIEW {view}", f"CREATE VIEW {view} AS {sql}"), + } + statement: str + for statement in statements[not sql]: + execute_postgres_sql(sql=statement, config=config) + + +def revoke_view_select(*, view: str, role: str, revoke: bool, config: dict[str, object]) -> None: + """Revoke a role's SELECT on a view when ``revoke`` is set.""" + + statements: dict[bool, tuple[str, ...]] = { + True: (f"REVOKE SELECT ON {view} FROM {role}",), + False: (), + } + statement: str + for statement in statements[revoke]: + execute_postgres_sql(sql=statement, config=config) diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/postgres/test_old_name_view_crashes.py b/tests/e2e/src/sqlbuild/cli/commands/main/postgres/test_old_name_view_crashes.py new file mode 100644 index 0000000000..25d0a2e02b --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/postgres/test_old_name_view_crashes.py @@ -0,0 +1,156 @@ +"""PostgreSQL fault injection proving compatibility views survive a crash at every promotion point.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.postgres._test_types import ( + PostgresBoundViewCrashE2ETestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.main.postgres.helpers import ( + build_in_process, + build_unique_schema_name, + cleanup_postgres_schema, + ensure_postgres_schema_ready, + execute_postgres_sql, + fail_bound_view_rebind, + fail_column_drop, + fail_displaced_drop, + fail_mid_swap, + load_raw_orders, + no_postgres_failure, + old_name_model_sql, + ordered_ids, + write_migration_project, +) + +_FIVE: tuple[tuple[object, ...], ...] = ((1,), (2,), (3,), (4,), (5,)) +_SIX: tuple[tuple[object, ...], ...] = (*_FIVE, (6,)) +_ALL_COLUMNS: str = "order_id, order_date, amount_cents" +_WITHOUT_AMOUNT: str = "order_id, order_date" + + +@pytest.mark.parametrize( + "test_case", + [ + PostgresBoundViewCrashE2ETestCase( + description="table rebuild crashing between swap renames", + materialized="table", + changed_columns=_ALL_COLUMNS, + install_failure=fail_mid_swap, + expected_crash_exit_code=1, + expected_view_ids_after_crash=_FIVE, + expected_view_ids_after_retry=_SIX, + ), + PostgresBoundViewCrashE2ETestCase( + description="table rebuild crashing before views follow the new table", + materialized="table", + changed_columns=_ALL_COLUMNS, + install_failure=fail_bound_view_rebind, + expected_crash_exit_code=1, + expected_view_ids_after_crash=_FIVE, + expected_view_ids_after_retry=_SIX, + ), + PostgresBoundViewCrashE2ETestCase( + description="table rebuild crashing before the displaced table is dropped", + materialized="table", + changed_columns=_ALL_COLUMNS, + install_failure=fail_displaced_drop, + expected_crash_exit_code=1, + expected_view_ids_after_crash=_FIVE, + expected_view_ids_after_retry=_SIX, + ), + PostgresBoundViewCrashE2ETestCase( + description="table rebuild without a crash", + materialized="table", + changed_columns=_ALL_COLUMNS, + install_failure=no_postgres_failure, + expected_crash_exit_code=0, + expected_view_ids_after_crash=_SIX, + expected_view_ids_after_retry=_SIX, + ), + PostgresBoundViewCrashE2ETestCase( + description="incremental column drop crashing after views are released", + materialized="incremental_sync", + changed_columns=_WITHOUT_AMOUNT, + install_failure=fail_column_drop, + expected_crash_exit_code=1, + expected_view_ids_after_crash=_FIVE, + expected_view_ids_after_retry=_SIX, + ), + ], + ids=lambda case: case.description, +) +def test_given_compatibility_view_when_rebuild_crashes_then_view_is_never_missing( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, + test_case: PostgresBoundViewCrashE2ETestCase, + postgres_e2e_config: dict[str, object], +) -> None: + """Views bound to the replaced relation change only inside the promoting transaction.""" + + schema_name: str = build_unique_schema_name(prefix="sqlbuild_old_name_crash") + raw_schema_name: str = f"{schema_name}_raw" + ensure_postgres_schema_ready(schema_name=schema_name, config=postgres_e2e_config) + load_raw_orders(raw_schema_name=raw_schema_name, config=postgres_e2e_config) + view: str = f"{schema_name}.revenue" + try: + project_dir: Path = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={"revenue": old_name_model_sql(materialized=test_case.materialized)}, + ) + assert build_in_process(project_dir=project_dir, capsys=capsys) == 0 + project_dir = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={ + "daily_revenue": old_name_model_sql( + materialized=test_case.materialized, migrate_from="revenue" + ) + }, + ) + assert build_in_process(project_dir=project_dir, capsys=capsys) == 0 + execute_postgres_sql( + sql=( + f"INSERT INTO {raw_schema_name}.raw_orders VALUES (6, TIMESTAMP '2026-01-06', 106)" + ), + config=postgres_e2e_config, + ) + project_dir = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={ + "daily_revenue": old_name_model_sql( + materialized=test_case.materialized, + migrate_from="revenue", + columns=test_case.changed_columns, + ) + }, + ) + with monkeypatch.context() as patch: + test_case.install_failure(patch) + crash_exit_code: int = build_in_process(project_dir=project_dir, capsys=capsys) + after_crash: tuple[tuple[object, ...], ...] = ordered_ids( + relation=view, config=postgres_e2e_config + ) + assert build_in_process(project_dir=project_dir, capsys=capsys) == 0 + after_retry: tuple[tuple[object, ...], ...] = ordered_ids( + relation=view, config=postgres_e2e_config + ) + finally: + cleanup_postgres_schema(schema_name=schema_name, config=postgres_e2e_config) + cleanup_postgres_schema(schema_name=raw_schema_name, config=postgres_e2e_config) + + assert crash_exit_code == test_case.expected_crash_exit_code + assert after_crash == test_case.expected_view_ids_after_crash + assert after_retry == test_case.expected_view_ids_after_retry diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/postgres/test_old_name_view_grants.py b/tests/e2e/src/sqlbuild/cli/commands/main/postgres/test_old_name_view_grants.py new file mode 100644 index 0000000000..2ab26147ed --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/postgres/test_old_name_view_grants.py @@ -0,0 +1,341 @@ +"""PostgreSQL e2e coverage for compatibility views keeping the old table's privileges.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.postgres._test_types import ( + PostgresDefaultPrivilegesE2ETestCase, + PostgresOldNameGrantsE2ETestCase, + PostgresOldNameRevokeE2ETestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.main.postgres.helpers import ( + build_ok, + build_unique_schema_name, + cleanup_postgres_schema, + create_login_role, + ensure_postgres_schema_ready, + execute_postgres_sql, + fetch_postgres_rows, + load_raw_orders, + old_name_model_sql, + reader_error, + reader_rows, + revoke_view_select, + write_migration_project, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import run_sqb + +_FIVE: tuple[tuple[object, ...], ...] = ((1,), (2,), (3,), (4,), (5,)) +_SIX: tuple[tuple[object, ...], ...] = (*_FIVE, (6,)) + + +@pytest.mark.parametrize( + "test_case", + [ + PostgresOldNameGrantsE2ETestCase( + description="select-only reader of the old table", + reader_role="orders_reader", + expected_reader_ids=_SIX, + expected_grants=('GRANT SELECT ON {view} TO "orders_reader"',), + expected_new_table_error="permission denied for table daily_revenue", + expected_plan_fragment=" ├── grants 1 copied from the old table\n", + ) + ], + ids=lambda case: case.description, +) +def test_given_reader_of_old_table_when_model_is_renamed_then_reader_keeps_access_through_view( + tmp_path: Path, + test_case: PostgresOldNameGrantsE2ETestCase, + postgres_e2e_config: dict[str, object], +) -> None: + """The view copies the archived table's privileges and grants nothing new.""" + + schema_name: str = build_unique_schema_name(prefix="sqlbuild_old_name_grants") + raw_schema_name: str = f"{schema_name}_raw" + role: str = test_case.reader_role + view: str = f"{schema_name}.revenue" + ensure_postgres_schema_ready(schema_name=schema_name, config=postgres_e2e_config) + load_raw_orders(raw_schema_name=raw_schema_name, config=postgres_e2e_config) + execute_postgres_sql( + sql=( + f"DO $$ BEGIN CREATE ROLE {role} LOGIN PASSWORD '{role}'; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$" + ), + config=postgres_e2e_config, + ) + try: + project_dir: Path = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={"revenue": old_name_model_sql(materialized="table")}, + ) + _ = build_ok(project_dir) + execute_postgres_sql( + sql=f"GRANT USAGE ON SCHEMA {schema_name} TO {role}", config=postgres_e2e_config + ) + execute_postgres_sql(sql=f"GRANT SELECT ON {view} TO {role}", config=postgres_e2e_config) + project_dir = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={ + "daily_revenue": old_name_model_sql(materialized="table", migrate_from="revenue") + }, + ) + _ = build_ok(project_dir) + execute_postgres_sql( + sql=( + f"INSERT INTO {raw_schema_name}.raw_orders VALUES (6, TIMESTAMP '2026-01-06', 106)" + ), + config=postgres_e2e_config, + ) + _ = build_ok(project_dir) + reader_ids: tuple[tuple[object, ...], ...] = reader_rows( + config=postgres_e2e_config, role=role, sql=f"SELECT order_id FROM {view} ORDER BY 1" + ) + new_table_error: str = reader_error( + config=postgres_e2e_config, + role=role, + sql=f"SELECT order_id FROM {schema_name}.daily_revenue", + ) + plan_text: str = run_sqb(command=("--no-color", "plan"), project_dir=project_dir).stdout + plan_json: dict[str, Any] = json.loads( + run_sqb(command=("--no-color", "plan", "--json"), project_dir=project_dir).stdout + ) + recorded: tuple[tuple[object, ...], ...] = fetch_postgres_rows( + sql=( + f"SELECT grants_copied FROM {schema_name}._sqlbuild_old_name_views " + "WHERE event_type = 'view_created'" + ), + config=postgres_e2e_config, + ) + finally: + cleanup_postgres_schema(schema_name=schema_name, config=postgres_e2e_config) + cleanup_postgres_schema(schema_name=raw_schema_name, config=postgres_e2e_config) + + assert test_case.expected_plan_fragment in plan_text, plan_text + assert [entry["grants_copied"] for entry in plan_json["old_names"]] == [ + len(test_case.expected_grants) + ] + assert reader_ids == test_case.expected_reader_ids + assert new_table_error == test_case.expected_new_table_error + assert tuple(json.loads(str(recorded[0][0]))) == tuple( + grant.format(view=view) for grant in test_case.expected_grants + ) + + +@pytest.mark.parametrize( + "test_case", + [ + PostgresOldNameRevokeE2ETestCase( + description="table rebuild replaces the view in place", + materialized="table", + rebuilt_columns="order_id, order_date, amount_cents", + revoked_role="orders_revoked_table", + granted_role="orders_granted_table", + expected_revoked_error="permission denied for view revenue", + expected_granted_ids=_SIX, + ), + PostgresOldNameRevokeE2ETestCase( + description="dropped column re-creates the view with its current privileges", + materialized="incremental_sync", + rebuilt_columns="order_id, order_date", + revoked_role="orders_revoked_sync", + granted_role="orders_granted_sync", + expected_revoked_error="permission denied for view revenue", + expected_granted_ids=_SIX, + ), + ], + ids=lambda case: case.description, +) +def test_given_view_privileges_changed_when_destination_rebuilds_then_current_privileges_stay( + tmp_path: Path, + test_case: PostgresOldNameRevokeE2ETestCase, + postgres_e2e_config: dict[str, object], +) -> None: + """A rebuild never restores a revoked privilege and never loses one granted on the view.""" + + schema_name: str = build_unique_schema_name(prefix="sqlbuild_old_name_revoke") + raw_schema_name: str = f"{schema_name}_raw" + view: str = f"{schema_name}.revenue" + revoked: str = test_case.revoked_role + granted: str = test_case.granted_role + ensure_postgres_schema_ready(schema_name=schema_name, config=postgres_e2e_config) + load_raw_orders(raw_schema_name=raw_schema_name, config=postgres_e2e_config) + create_login_role(role=revoked, config=postgres_e2e_config) + create_login_role(role=granted, config=postgres_e2e_config) + try: + project_dir: Path = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={"revenue": old_name_model_sql(materialized=test_case.materialized)}, + ) + _ = build_ok(project_dir) + execute_postgres_sql( + sql=f"GRANT USAGE ON SCHEMA {schema_name} TO {revoked}, {granted}", + config=postgres_e2e_config, + ) + execute_postgres_sql(sql=f"GRANT SELECT ON {view} TO {revoked}", config=postgres_e2e_config) + project_dir = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={ + "daily_revenue": old_name_model_sql( + materialized=test_case.materialized, migrate_from="revenue" + ) + }, + ) + _ = build_ok(project_dir) + execute_postgres_sql( + sql=f"REVOKE SELECT ON {view} FROM {revoked}", config=postgres_e2e_config + ) + execute_postgres_sql(sql=f"GRANT SELECT ON {view} TO {granted}", config=postgres_e2e_config) + execute_postgres_sql( + sql=( + f"INSERT INTO {raw_schema_name}.raw_orders VALUES (6, TIMESTAMP '2026-01-06', 106)" + ), + config=postgres_e2e_config, + ) + project_dir = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={ + "daily_revenue": old_name_model_sql( + materialized=test_case.materialized, + migrate_from="revenue", + columns=test_case.rebuilt_columns, + ) + }, + ) + _ = build_ok(project_dir) + revoked_error: str = reader_error( + config=postgres_e2e_config, role=revoked, sql=f"SELECT order_id FROM {view}" + ) + granted_ids: tuple[tuple[object, ...], ...] = reader_rows( + config=postgres_e2e_config, + role=granted, + sql=f"SELECT order_id FROM {view} ORDER BY 1", + ) + finally: + cleanup_postgres_schema(schema_name=schema_name, config=postgres_e2e_config) + cleanup_postgres_schema(schema_name=raw_schema_name, config=postgres_e2e_config) + + assert revoked_error == test_case.expected_revoked_error + assert granted_ids == test_case.expected_granted_ids + + +@pytest.mark.parametrize( + "test_case", + [ + PostgresDefaultPrivilegesE2ETestCase( + description="default privileges do not reach a new compatibility view", + materialized="table", + rebuilt_columns="order_id, order_date, amount_cents", + role="orders_default_create", + revoke_before_rebuild=False, + expected_error="permission denied for view revenue", + ), + PostgresDefaultPrivilegesE2ETestCase( + description="drop and re-create does not restore a revoked default privilege", + materialized="incremental_sync", + rebuilt_columns="order_id, order_date", + role="orders_default_recreate", + revoke_before_rebuild=True, + expected_error="permission denied for view revenue", + ), + ], + ids=lambda case: case.description, +) +def test_given_default_privileges_when_view_is_recreated_then_grants_match_before( + tmp_path: Path, + test_case: PostgresDefaultPrivilegesE2ETestCase, + postgres_e2e_config: dict[str, object], +) -> None: + """Schema default privileges applied by a re-create are revoked again.""" + + schema_name: str = build_unique_schema_name(prefix="sqlbuild_old_name_defaults") + raw_schema_name: str = f"{schema_name}_raw" + view: str = f"{schema_name}.revenue" + role: str = test_case.role + ensure_postgres_schema_ready(schema_name=schema_name, config=postgres_e2e_config) + load_raw_orders(raw_schema_name=raw_schema_name, config=postgres_e2e_config) + create_login_role(role=role, config=postgres_e2e_config) + try: + project_dir: Path = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={"revenue": old_name_model_sql(materialized=test_case.materialized)}, + ) + _ = build_ok(project_dir) + execute_postgres_sql( + sql=f"GRANT USAGE ON SCHEMA {schema_name} TO {role}", config=postgres_e2e_config + ) + execute_postgres_sql( + sql=( + f"ALTER DEFAULT PRIVILEGES IN SCHEMA {schema_name} GRANT SELECT ON TABLES TO {role}" + ), + config=postgres_e2e_config, + ) + project_dir = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={ + "daily_revenue": old_name_model_sql( + materialized=test_case.materialized, migrate_from="revenue" + ) + }, + ) + _ = build_ok(project_dir) + revoke_view_select( + view=view, + role=role, + revoke=test_case.revoke_before_rebuild, + config=postgres_e2e_config, + ) + project_dir = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={ + "daily_revenue": old_name_model_sql( + materialized=test_case.materialized, + migrate_from="revenue", + columns=test_case.rebuilt_columns, + ) + }, + ) + _ = build_ok(project_dir) + error: str = reader_error( + config=postgres_e2e_config, role=role, sql=f"SELECT order_id FROM {view}" + ) + finally: + execute_postgres_sql( + sql=( + f"ALTER DEFAULT PRIVILEGES IN SCHEMA {schema_name} " + f"REVOKE SELECT ON TABLES FROM {role}" + ), + config=postgres_e2e_config, + ) + cleanup_postgres_schema(schema_name=schema_name, config=postgres_e2e_config) + cleanup_postgres_schema(schema_name=raw_schema_name, config=postgres_e2e_config) + + assert error == test_case.expected_error diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/postgres/test_old_name_views.py b/tests/e2e/src/sqlbuild/cli/commands/main/postgres/test_old_name_views.py new file mode 100644 index 0000000000..dba1e4b5d9 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/postgres/test_old_name_views.py @@ -0,0 +1,186 @@ +"""PostgreSQL e2e coverage for compatibility views at migrated models' old names.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.postgres._test_types import ( + PostgresOldNameJanitorE2ETestCase, + PostgresOldNameViewE2ETestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.main.postgres.helpers import ( + build_ok, + build_unique_schema_name, + cleanup_postgres_schema, + ensure_postgres_schema_ready, + execute_postgres_sql, + fetch_postgres_rows, + load_raw_orders, + old_name_model_sql, + ordered_ids, + replace_view, + write_migration_project, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import run_sqb + +_ORIGIN: str = "revenue" +_DESTINATION: str = "daily_revenue" +_FIVE: tuple[tuple[object, ...], ...] = ((1,), (2,), (3,), (4,), (5,)) +_SIX: tuple[tuple[object, ...], ...] = (*_FIVE, (6,)) + + +@pytest.mark.parametrize( + "test_case", + [ + PostgresOldNameViewE2ETestCase( + description=materialized, + materialized=materialized, + expected_old_name_kind="v", + expected_ids_after_rebuild=_SIX, + expected_warning_fragment="M115: views not managed by SQLBuild depend on", + expected_external_ids=_FIVE, + ) + for materialized in ("table", "incremental") + ], + ids=lambda case: case.description, +) +def test_given_renamed_model_when_rebuilding_on_postgres_then_old_name_view_follows_it( + tmp_path: Path, + test_case: PostgresOldNameViewE2ETestCase, + postgres_e2e_config: dict[str, object], +) -> None: + """Identity-bound compatibility views are released and re-created around every rebuild.""" + + schema_name: str = build_unique_schema_name(prefix="sqlbuild_old_names") + raw_schema_name: str = f"{schema_name}_raw" + ensure_postgres_schema_ready(schema_name=schema_name, config=postgres_e2e_config) + load_raw_orders(raw_schema_name=raw_schema_name, config=postgres_e2e_config) + external: str = f"{raw_schema_name}.revenue_report" + try: + project_dir: Path = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={_ORIGIN: old_name_model_sql(materialized=test_case.materialized)}, + ) + _ = build_ok(project_dir) + execute_postgres_sql( + sql=f"CREATE VIEW {external} AS SELECT order_id FROM {schema_name}.{_ORIGIN}", + config=postgres_e2e_config, + ) + project_dir = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={ + _DESTINATION: old_name_model_sql( + materialized=test_case.materialized, migrate_from=_ORIGIN + ) + }, + ) + migrated: subprocess.CompletedProcess[str] = build_ok(project_dir) + execute_postgres_sql( + sql=( + f"INSERT INTO {raw_schema_name}.raw_orders VALUES (6, TIMESTAMP '2026-01-06', 106)" + ), + config=postgres_e2e_config, + ) + _ = build_ok(project_dir) + _ = build_ok(project_dir) + old_name_kind: tuple[tuple[object, ...], ...] = fetch_postgres_rows( + sql=( + "SELECT relkind FROM pg_class JOIN pg_namespace ON pg_namespace.oid = " + f"relnamespace WHERE nspname = '{schema_name}' AND relname = '{_ORIGIN}'" + ), + config=postgres_e2e_config, + ) + old_name_ids: tuple[tuple[object, ...], ...] = ordered_ids( + relation=f"{schema_name}.{_ORIGIN}", config=postgres_e2e_config + ) + external_ids: tuple[tuple[object, ...], ...] = ordered_ids( + relation=external, config=postgres_e2e_config + ) + finally: + execute_postgres_sql(sql=f"DROP VIEW IF EXISTS {external}", config=postgres_e2e_config) + cleanup_postgres_schema(schema_name=schema_name, config=postgres_e2e_config) + cleanup_postgres_schema(schema_name=raw_schema_name, config=postgres_e2e_config) + + assert old_name_kind == ((test_case.expected_old_name_kind,),) + assert old_name_ids == test_case.expected_ids_after_rebuild + assert test_case.expected_warning_fragment in migrated.stdout, migrated.stdout + assert external in migrated.stdout, migrated.stdout + assert external_ids == test_case.expected_external_ids + + +@pytest.mark.parametrize( + "test_case", + [ + PostgresOldNameJanitorE2ETestCase( + description="sqlbuild's own view is dropped early", + replacement_sql="", + expected_janitor_fragment="└── drop now (requested; would expire ", + expected_old_name_kinds=(), + ), + PostgresOldNameJanitorE2ETestCase( + description="another view at the old name is left alone", + replacement_sql="SELECT 1 AS order_id", + expected_janitor_fragment="└── record dropped (name now used by another relation)", + expected_old_name_kinds=(("v",),), + ), + ], + ids=lambda case: case.description, +) +def test_given_postgres_old_name_when_dropping_early_then_only_sqlbuilds_view_is_dropped( + tmp_path: Path, + test_case: PostgresOldNameJanitorE2ETestCase, + postgres_e2e_config: dict[str, object], +) -> None: + """Janitor compares the view's definition with SQLBuild's before dropping it.""" + + schema_name: str = build_unique_schema_name(prefix="sqlbuild_old_name_janitor") + raw_schema_name: str = f"{schema_name}_raw" + view: str = f"{schema_name}.{_ORIGIN}" + ensure_postgres_schema_ready(schema_name=schema_name, config=postgres_e2e_config) + load_raw_orders(raw_schema_name=raw_schema_name, config=postgres_e2e_config) + try: + project_dir: Path = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={_ORIGIN: old_name_model_sql(materialized="table")}, + ) + _ = build_ok(project_dir) + project_dir = write_migration_project( + tmp_path=tmp_path, + schema_name=schema_name, + raw_schema_name=raw_schema_name, + config=postgres_e2e_config, + models={_DESTINATION: old_name_model_sql(materialized="table", migrate_from=_ORIGIN)}, + ) + _ = build_ok(project_dir) + replace_view(view=view, sql=test_case.replacement_sql, config=postgres_e2e_config) + janitor: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "janitor", "--auto-approve", "--drop-old-name-view", view), + project_dir=project_dir, + ) + kinds: tuple[tuple[object, ...], ...] = fetch_postgres_rows( + sql=( + "SELECT relation.relkind FROM pg_class AS relation JOIN pg_namespace AS " + "namespace ON namespace.oid = relation.relnamespace WHERE namespace.nspname = " + f"'{schema_name}' AND relation.relname = '{_ORIGIN}'" + ), + config=postgres_e2e_config, + ) + finally: + cleanup_postgres_schema(schema_name=schema_name, config=postgres_e2e_config) + cleanup_postgres_schema(schema_name=raw_schema_name, config=postgres_e2e_config) + + assert janitor.returncode == 0, janitor.stdout + janitor.stderr + assert kinds == test_case.expected_old_name_kinds + assert test_case.expected_janitor_fragment in janitor.stdout, janitor.stdout diff --git a/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/helpers.py b/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/helpers.py index f024e3bd51..e916d75518 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/helpers.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/helpers.py @@ -27,6 +27,9 @@ [connection] database = "{DATABASE_FILE}" + + [migrations] + old_name_views = false """ ).lstrip() TARGETS_PROJECT_TOML: str = dedent( @@ -43,6 +46,9 @@ [targets.prod] schema = "prod" + + [migrations] + old_name_views = false """ ).lstrip() JANITOR_PROJECT_TOML: str = dedent( @@ -62,6 +68,9 @@ enabled = true archive_retention_days = 0 delete_tracked_only = false + + [migrations] + old_name_views = false """ ).lstrip() RAW_SOURCES_YML: str = dedent( diff --git a/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_identity_handover.py b/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_identity_handover.py index 2d0a1d4eef..e9aaf3069b 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_identity_handover.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_identity_handover.py @@ -148,19 +148,22 @@ def test_given_renamed_chain_with_full_replay_when_building_then_migrated_histor "test_case", [ RenamedPlanTextTestCase( - description="plan lists renamed views separately from data migrations", + description="plan lists a renamed view as a migration that recreates it", expected_fragment=( - "Renamed (1)\n└── customer_orders_enriched main.orders_enriched -> " - "main.customer_orders_enriched (identity handed over)" + "├── customer_orders_enriched migrate main.orders_enriched -> " + "main.customer_orders_enriched\n ├── transfer recreate (view)\n" + " ├── discovery automatic\n" + " └── old name main.orders_enriched\n" + " └── left for janitor old_name_view false" ), ) ], ids=lambda case: case.description, ) -def test_given_renamed_view_when_planning_then_text_lists_it_under_renamed( +def test_given_renamed_view_when_planning_then_text_lists_it_as_a_migration( test_case: RenamedPlanTextTestCase, tmp_path: Path, capsys: pytest.CaptureFixture[str] ) -> None: - """Renamed tables and views move no data and appear in their own plan section.""" + """Renamed tables and views move no data and are listed with the other migrations.""" write_project(project_dir=tmp_path, models=original_replay_order_models(middle="view")) load_raw_orders(project_dir=tmp_path, first_day=1, last_day=5) diff --git a/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_manual_migrations.py b/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_manual_migrations.py index c8ed592d5f..e58feff3fe 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_manual_migrations.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_manual_migrations.py @@ -455,14 +455,33 @@ def test_given_replay_on_change_full_when_migrating_then_destination_is_not_repl "test_case", [ MigrationCompileErrorTestCase( - description="table materialization", - model_sql='MODEL (materialized table, migrate_from "stg_orders");\nSELECT 1 AS id\n', - expected_fragment="migrate_from is only valid for incremental and snapshot models", + description="force on a table", + model_sql=( + 'MODEL (materialized table, migrate_from "stg_orders", migrate_force true);\n' + "SELECT 1 AS id\n" + ), + expected_fragment="migrate_force is only valid for incremental and snapshot models", + ), + MigrationCompileErrorTestCase( + description="force on a view", + model_sql=( + 'MODEL (materialized view, migrate_from "stg_orders", migrate_force true);\n' + "SELECT 1 AS id\n" + ), + expected_fragment="migrate_force is only valid for incremental and snapshot models", + ), + MigrationCompileErrorTestCase( + description="column alias on a table that is not migrating", + model_sql=( + "MODEL (materialized table, columns (revenue (migrate_from amount)));\n" + "SELECT 1 AS revenue\n" + ), + expected_fragment="or for table and view models that declare migrate_from themselves", ), MigrationCompileErrorTestCase( - description="view materialization", - model_sql='MODEL (materialized view, migrate_from "stg_orders");\nSELECT 1 AS id\n', - expected_fragment="migrate_from is only valid for incremental and snapshot models", + description="invalid old name view retention", + model_sql=incremental_orders_sql(extra_config=" old_name_view soon,\n"), + expected_fragment="old_name_view must be a positive duration", ), MigrationCompileErrorTestCase( description="force without origin", diff --git a/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_staged_execution.py b/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_staged_execution.py index 8c18d03f8e..3710e7a525 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_staged_execution.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/model_migrations/test_staged_execution.py @@ -128,7 +128,7 @@ def test_given_forced_replace_when_interrupted_then_retry_converges_without_losi expected_transfer="copy", expected_promotion="transactional_rename", expected_storage_transition=None, - expected_text_fragment="└── transfer physical copy, promote by transactional rename", + expected_text_fragment="├── transfer physical copy, promote by transactional rename", ), StagedPlanOutputTestCase( description="refused clone plans its copy fallback and builds by copy", @@ -139,7 +139,7 @@ def test_given_forced_replace_when_interrupted_then_retry_converges_without_losi expected_promotion="transactional_rename", expected_storage_transition=None, expected_text_fragment=( - "└── transfer zero-copy clone (physical copy if refused), " + "├── transfer zero-copy clone (physical copy if refused), " "promote by transactional rename" ), ), diff --git a/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/__init__.py b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/_test_types.py b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/_test_types.py new file mode 100644 index 0000000000..9951d192c2 --- /dev/null +++ b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/_test_types.py @@ -0,0 +1,67 @@ +"""Test case types for old-name compatibility view integration tests.""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass + +import pytest + + +@dataclass(frozen=True) +class OldNameResumeTestCase: + description: str + install_failure: Callable[[pytest.MonkeyPatch], None] + expected_first_exit_code: int + expected_facts_after_failure: tuple[str, ...] + expected_old_name_type_after_failure: str | None + expected_final_facts: tuple[str, ...] + expected_archive_count: int + + +@dataclass(frozen=True) +class OldNameJanitorResumeTestCase: + description: str + early_drop: tuple[str, ...] + builds_after_crash: int + expected_facts_after_crash: tuple[str, ...] + expected_retry_fragment: str + expected_final_facts: tuple[str, ...] + + +@dataclass(frozen=True) +class OldNameResumeSkipTestCase: + description: str + install_failure: Callable[[pytest.MonkeyPatch], None] + retry_models: dict[str, str] + expected_plan_fragment: str + expected_facts: tuple[str, ...] + expected_old_name_type: str | None + expected_old_name_ids: tuple[int, ...] | None + + +@dataclass(frozen=True) +class OldNameJanitorClaimTestCase: + description: str + janitor_args: tuple[str, ...] + install_build_fault: Callable[[pytest.MonkeyPatch], None] + claiming_builds: tuple[tuple[str, ...], ...] + expected_janitor_fragment: str + expected_old_name_type: str + expected_final_facts: tuple[str, ...] + + +@dataclass(frozen=True) +class OldNameAliasResumeTestCase: + description: str + install_failure: Callable[[pytest.MonkeyPatch], None] + expected_facts_after_failure: tuple[str, ...] + expected_plan_fragment: str + + +@dataclass(frozen=True) +class OldNameRefreshSkipTestCase: + description: str + renamed_sql: str + rebuilt_sql: str + expected_view_statements: int diff --git a/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/helpers.py b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/helpers.py new file mode 100644 index 0000000000..63c7799123 --- /dev/null +++ b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/helpers.py @@ -0,0 +1,291 @@ +"""Project builders and fault injection for old-name compatibility view integration tests.""" + +from __future__ import annotations + +from collections.abc import Callable +from pathlib import Path +from textwrap import dedent +from typing import Any + +import pytest + +from sqlbuild.compiler.migrations.models import OldNameViewEvent +from sqlbuild.compiler.migrations.types import OldNameViewEventType +from sqlbuild.executor.migrations._helpers import old_name_views as old_name_steps +from tests.integration.src.sqlbuild.cli.commands.main.model_migrations.helpers import ( + CliRun, + build_ok, + disable_transactional_ddl, + load_raw_orders, + query, + write_project, +) + +ORIGIN_MODEL: str = "revenue" +DESTINATION_MODEL: str = "daily_revenue" +PROJECT_TOML: str = dedent( + """ + name = "orders_project" + adapter = "duckdb" + + [connection] + database = "orders.duckdb" + """ +).lstrip() +_FACT_ORDER: tuple[str, ...] = ("required", "origin_archived", "view_created", "view_dropped") +_TABLE_SQL: str = ( + "MODEL (materialized table{extra});\n" + 'SELECT order_id, amount_cents FROM __source("raw_orders")\n' +) + + +def table_sql(*, migrate_from: str | None = None) -> str: + """Return a table model over raw orders with an optional migrate_from header.""" + + extra: str = {None: ""}.get(migrate_from, f", migrate_from {migrate_from}") + return _TABLE_SQL.format(extra=extra) + + +def prepare_table_rename( + *, project_dir: Path, capsys: pytest.CaptureFixture[str], project_toml: str = PROJECT_TOML +) -> None: + """Build the origin table, then declare its rename to the destination.""" + + write_project( + project_dir=project_dir, models={ORIGIN_MODEL: table_sql()}, project_toml=project_toml + ) + load_raw_orders(project_dir=project_dir, first_day=1, last_day=3) + _ = build_ok(project_dir=project_dir, capsys=capsys) + write_project( + project_dir=project_dir, + models={DESTINATION_MODEL: table_sql(migrate_from=ORIGIN_MODEL)}, + project_toml=project_toml, + ) + + +def old_name_facts(*, project_dir: Path, schema: str = "main") -> tuple[str, ...]: + """Return recorded old-name fact types in lifecycle order.""" + + return tuple( + str(row[0]) + for row in query( + project_dir=project_dir, + sql=( + f"SELECT event_type FROM {schema}._sqlbuild_old_name_views ORDER BY " + f"list_position({list(_FACT_ORDER)}, event_type), created_at" + ), + ) + ) + + +def relation_type(*, project_dir: Path, name: str) -> str | None: + """Return the information_schema type of one relation in main, or None.""" + + return relation_type_in(project_dir=project_dir, schema="main", name=name) + + +def relation_type_in(*, project_dir: Path, schema: str, name: str) -> str | None: + """Return the information_schema type of one relation, or None.""" + + rows: list[tuple[Any, ...]] = query( + project_dir=project_dir, + sql=( + "SELECT table_type FROM information_schema.tables " + f"WHERE table_schema = '{schema}' AND table_name = '{name}'" + ), + ) + return next((str(row[0]) for row in rows), None) + + +def origin_archive_count(*, project_dir: Path) -> int: + """Return how many migration_origin archives exist in main.""" + + return int( + query( + project_dir=project_dir, + sql=( + "SELECT count(*) FROM information_schema.tables WHERE table_schema = 'main' " + "AND contains(table_name, '__migration_origin__')" + ), + )[0][0] + ) + + +def _raise_interruption(*_: object, **__: object) -> None: + raise RuntimeError("simulated interruption") + + +def _fail_fact(*, monkeypatch: pytest.MonkeyPatch, event_type: OldNameViewEventType) -> None: + original: Callable[..., None] = old_name_steps.record_old_name_fact + + outcomes: dict[bool, Callable[..., None]] = {True: _raise_interruption, False: original} + + def failing(*, event: OldNameViewEvent, **kwargs: Any) -> None: + outcomes[event.event_type == event_type](event=event, **kwargs) + + monkeypatch.setattr(old_name_steps, "record_old_name_fact", failing) + + +def fail_destination_build(monkeypatch: pytest.MonkeyPatch) -> None: + """Fail the destination table build after its move is recorded.""" + + from sqlbuild.executor.build._helpers import scheduler as scheduler_module + + original: Callable[..., Any] = scheduler_module.execute_table_entry + failing: dict[str, Callable[..., Any]] = {DESTINATION_MODEL: _raise_interruption} + monkeypatch.setattr( + scheduler_module, + "execute_table_entry", + lambda **kwargs: failing.get(kwargs["context"].entry.name, original)(**kwargs), + ) + + +def fail_transactional_view(monkeypatch: pytest.MonkeyPatch) -> None: + """Fail creating the view inside the archive-and-view transaction.""" + + monkeypatch.setattr(old_name_steps, "_create_compatibility_view", _raise_interruption) + + +def fail_non_transactional_archive_fact(monkeypatch: pytest.MonkeyPatch) -> None: + """Crash after the old relation is renamed but before its archive is recorded.""" + + disable_transactional_ddl(monkeypatch) + _fail_fact(monkeypatch=monkeypatch, event_type=OldNameViewEventType.ORIGIN_ARCHIVED) + + +def fail_non_transactional_view(monkeypatch: pytest.MonkeyPatch) -> None: + """Crash after the archive is recorded but before the view exists.""" + + disable_transactional_ddl(monkeypatch) + monkeypatch.setattr(old_name_steps, "_create_compatibility_view", _raise_interruption) + + +def fail_non_transactional_view_fact(monkeypatch: pytest.MonkeyPatch) -> None: + """Crash after the view is created but before it is recorded.""" + + disable_transactional_ddl(monkeypatch) + _fail_fact(monkeypatch=monkeypatch, event_type=OldNameViewEventType.VIEW_CREATED) + + +def build_result(*, project_dir: Path, capsys: pytest.CaptureFixture[str]) -> CliRun: + """Run sqb build, reporting an escaped exception as a failed run.""" + + from tests.integration.src.sqlbuild.cli.commands.main.model_migrations.helpers import build + + return build(project_dir=project_dir, capsys=capsys) + + +JANITOR_PROJECT_TOML: str = dedent( + """ + name = "orders_project" + adapter = "duckdb" + default_target = "dev" + + [connection] + database = "orders.duckdb" + + [targets.dev] + schema = "dev" + + [janitor] + enabled = true + """ +).lstrip() + + +def fail_janitor_drop_fact(monkeypatch: pytest.MonkeyPatch) -> None: + """Crash after the janitor drops a view but before the drop is recorded.""" + + monkeypatch.setattr( + "sqlbuild.executor.old_name_views._helpers.janitor.write_old_name_view_event", + _raise_interruption, + ) + + +def disabled_table_sql(*, migrate_from: str) -> str: + """Return a table model that declares its rename with old-name views turned off.""" + + return ( + f"MODEL (materialized table, migrate_from {migrate_from}, old_name_view false);\n" + 'SELECT order_id, amount_cents FROM __source("raw_orders")\n' + ) + + +def plan_text(*, project_dir: Path, capsys: pytest.CaptureFixture[str]) -> str: + """Run sqb plan and return its text output.""" + + from tests.integration.src.sqlbuild.cli.commands.main.model_migrations.helpers import run_sqb + + return run_sqb(project_dir=project_dir, args=("plan",), capsys=capsys).output + + +def claiming_view_models() -> dict[str, str]: + """Return the renamed model plus a new view model built at the old name.""" + + return { + DESTINATION_MODEL: table_sql(migrate_from=ORIGIN_MODEL), + ORIGIN_MODEL: ( + "MODEL (materialized view);\n" + 'SELECT order_id, amount_cents * 2 AS doubled FROM __source("raw_orders")\n' + ), + } + + +def aliased_orders_sql(*, alias: str, migrate_from: str | None = None) -> str: + """Return one table model whose relation name comes from ``alias``.""" + + migration: str = {None: ""}.get(migrate_from, f", migrate_from {migrate_from}") + return ( + f"MODEL (materialized table, alias {alias}{migration});\n" + 'SELECT order_id, amount_cents FROM __source("raw_orders")\n' + ) + + +def _failed_fingerprint_write(**_: object) -> tuple[str, ...]: + return ("fingerprint write failed",) + + +def fail_view_fingerprint_write(monkeypatch: pytest.MonkeyPatch) -> None: + """Let view builds succeed while their best-effort fingerprint write fails.""" + + monkeypatch.setattr( + "sqlbuild.executor.run._helpers.materializations.view.try_write_fingerprint", + _failed_fingerprint_write, + ) + + +def no_build_fault(monkeypatch: pytest.MonkeyPatch) -> None: + """Install nothing.""" + + del monkeypatch + + +def aliased_table_sql(*, extra_column: str = "") -> str: + """Return the renamed table model with ``amount`` renamed to ``revenue`` in its old view.""" + + return ( + f"MODEL (materialized table, migrate_from {ORIGIN_MODEL}, " + "columns (revenue (migrate_from amount_cents)));\n" + f'SELECT order_id, amount_cents AS revenue{extra_column} FROM __source("raw_orders")\n' + ) + + +def record_old_name_view_ddl(monkeypatch: pytest.MonkeyPatch, statements: list[str]) -> None: + """Record every DDL statement that creates or replaces the view at the old name.""" + + from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter + + original: Callable[..., Any] = DuckDbAdapter.execute + + def execute(self: DuckDbAdapter, *, connection: Any, sql: str) -> Any: + statements.extend( + sql + for _ in range( + sql.lstrip().upper().startswith("CREATE") + and "VIEW" in sql.upper() + and f"{ORIGIN_MODEL} AS" in sql + ) + ) + return original(self, connection=connection, sql=sql) + + monkeypatch.setattr(DuckDbAdapter, "execute", execute) diff --git a/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_alias_resume.py b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_alias_resume.py new file mode 100644 index 0000000000..37092b73fa --- /dev/null +++ b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_alias_resume.py @@ -0,0 +1,82 @@ +"""Integration coverage for resuming the old-name steps of a same-model alias move.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from tests.integration.src.sqlbuild.cli.commands.main.model_migrations.helpers import ( + build_ok, + load_raw_orders, + order_ids, + write_project, +) +from tests.integration.src.sqlbuild.cli.commands.main.old_name_views._test_types import ( + OldNameAliasResumeTestCase, +) +from tests.integration.src.sqlbuild.cli.commands.main.old_name_views.helpers import ( + PROJECT_TOML, + aliased_orders_sql, + build_result, + fail_non_transactional_view, + fail_transactional_view, + old_name_facts, + plan_text, + relation_type, +) + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameAliasResumeTestCase( + description="view creation interrupted and rolled back with the archive", + install_failure=fail_transactional_view, + expected_facts_after_failure=("required",), + expected_plan_fragment=" └── old table archived", + ), + OldNameAliasResumeTestCase( + description="view creation interrupted after the archive", + install_failure=fail_non_transactional_view, + expected_facts_after_failure=("required", "origin_archived"), + expected_plan_fragment=" └── old table already archived", + ), + ], + ids=lambda case: case.description, +) +def test_given_interrupted_alias_move_when_retrying_then_view_is_created_at_the_old_alias( + test_case: OldNameAliasResumeTestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The recorded move proves ownership even after the model's own fingerprint moved on.""" + + write_project( + project_dir=tmp_path, + models={"orders": aliased_orders_sql(alias="legacy_orders")}, + project_toml=PROJECT_TOML, + ) + load_raw_orders(project_dir=tmp_path, first_day=1, last_day=3) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + write_project( + project_dir=tmp_path, + models={ + "orders": aliased_orders_sql(alias="current_orders", migrate_from="main.legacy_orders") + }, + project_toml=PROJECT_TOML, + ) + with monkeypatch.context() as patch: + test_case.install_failure(patch) + _ = build_result(project_dir=tmp_path, capsys=capsys) + facts_after_failure: tuple[str, ...] = old_name_facts(project_dir=tmp_path) + + plan: str = plan_text(project_dir=tmp_path, capsys=capsys) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + + assert facts_after_failure == test_case.expected_facts_after_failure + assert relation_type(project_dir=tmp_path, name="legacy_orders") == "VIEW" + assert order_ids(project_dir=tmp_path, relation="main.legacy_orders") == (1, 2, 3) + assert old_name_facts(project_dir=tmp_path) == ("required", "origin_archived", "view_created") + assert test_case.expected_plan_fragment in plan, plan diff --git a/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_janitor_resume.py b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_janitor_resume.py new file mode 100644 index 0000000000..cdf8d12bcd --- /dev/null +++ b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_janitor_resume.py @@ -0,0 +1,175 @@ +"""Integration coverage for a janitor crash between dropping a view and recording the drop.""" + +from __future__ import annotations + +import contextlib +from pathlib import Path + +import pytest + +from tests.integration.src.sqlbuild.cli.commands.main.model_migrations.helpers import ( + CliRun, + build, + build_ok, + relation_names, + run_sqb, + write_project, +) +from tests.integration.src.sqlbuild.cli.commands.main.old_name_views._test_types import ( + OldNameJanitorClaimTestCase, + OldNameJanitorResumeTestCase, +) +from tests.integration.src.sqlbuild.cli.commands.main.old_name_views.helpers import ( + DESTINATION_MODEL, + JANITOR_PROJECT_TOML, + ORIGIN_MODEL, + claiming_view_models, + fail_janitor_drop_fact, + fail_view_fingerprint_write, + no_build_fault, + old_name_facts, + prepare_table_rename, + relation_type_in, +) + +_ONE_BUILD: tuple[tuple[str, ...], ...] = (("build",),) +_CLAIMED: str = "└── record dropped (name now used by another relation, model:revenue)" + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameJanitorResumeTestCase( + description="early drop interrupted before its record", + early_drop=("janitor", "--auto-approve", "--drop-old-name-view", "dev.revenue"), + builds_after_crash=0, + expected_facts_after_crash=("required", "origin_archived", "view_created"), + expected_retry_fragment="└── record dropped (the view no longer exists)", + expected_final_facts=("required", "origin_archived", "view_created", "view_dropped"), + ), + OldNameJanitorResumeTestCase( + description="destination build after the interrupted drop leaves the view absent", + early_drop=("janitor", "--auto-approve", "--drop-old-name-view", "dev.revenue"), + builds_after_crash=1, + expected_facts_after_crash=("required", "origin_archived", "view_created"), + expected_retry_fragment="└── record dropped (the view no longer exists)", + expected_final_facts=("required", "origin_archived", "view_created", "view_dropped"), + ), + ], + ids=lambda case: case.description, +) +def test_given_janitor_crash_after_drop_when_rerunning_then_drop_is_recorded_as_missing( + test_case: OldNameJanitorResumeTestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A vanished compatibility view is recorded as dropped instead of lingering in state.""" + + prepare_table_rename(project_dir=tmp_path, capsys=capsys, project_toml=JANITOR_PROJECT_TOML) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + + with monkeypatch.context() as patch: + fail_janitor_drop_fact(patch) + try: + first: CliRun = run_sqb(project_dir=tmp_path, args=test_case.early_drop, capsys=capsys) + except RuntimeError as error: + first = CliRun(exit_code=1, output=str(error)) + facts_after_crash: tuple[str, ...] = old_name_facts(project_dir=tmp_path, schema="dev") + for _ in range(test_case.builds_after_crash): + _ = build_ok(project_dir=tmp_path, capsys=capsys) + names_before_retry: list[str] = relation_names(project_dir=tmp_path, schema="dev") + retry: CliRun = run_sqb(project_dir=tmp_path, args=("janitor", "--auto-approve"), capsys=capsys) + + assert first.exit_code != 0, first.output + assert facts_after_crash == test_case.expected_facts_after_crash + assert ORIGIN_MODEL not in names_before_retry + assert retry.exit_code == 0, retry.output + assert test_case.expected_retry_fragment in retry.output + assert old_name_facts(project_dir=tmp_path, schema="dev") == test_case.expected_final_facts + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameJanitorClaimTestCase( + description="plain janitor run records the claimed name without touching it", + janitor_args=("janitor", "--auto-approve"), + install_build_fault=no_build_fault, + claiming_builds=_ONE_BUILD, + expected_janitor_fragment=_CLAIMED, + expected_old_name_type="VIEW", + expected_final_facts=("required", "origin_archived", "view_created", "view_dropped"), + ), + OldNameJanitorClaimTestCase( + description="repeated early drop leaves the project's view in place", + janitor_args=("janitor", "--auto-approve", "--drop-old-name-view", "dev.revenue"), + install_build_fault=no_build_fault, + claiming_builds=_ONE_BUILD, + expected_janitor_fragment=_CLAIMED, + expected_old_name_type="VIEW", + expected_final_facts=("required", "origin_archived", "view_created", "view_dropped"), + ), + OldNameJanitorClaimTestCase( + description="repeated early drop spares a project view built without a fingerprint", + janitor_args=("janitor", "--auto-approve", "--drop-old-name-view", "dev.revenue"), + install_build_fault=fail_view_fingerprint_write, + claiming_builds=_ONE_BUILD, + expected_janitor_fragment=_CLAIMED, + expected_old_name_type="VIEW", + expected_final_facts=("required", "origin_archived", "view_created", "view_dropped"), + ), + OldNameJanitorClaimTestCase( + description="destination built after the project's view leaves that view alone", + janitor_args=("janitor", "--auto-approve", "--drop-old-name-view", "dev.revenue"), + install_build_fault=no_build_fault, + claiming_builds=( + ("build", "--select", ORIGIN_MODEL), + ("build", "--select", DESTINATION_MODEL), + ), + expected_janitor_fragment=_CLAIMED, + expected_old_name_type="VIEW", + expected_final_facts=("required", "origin_archived", "view_created", "view_dropped"), + ), + ], + ids=lambda case: case.description, +) +def test_given_interrupted_early_drop_when_project_builds_the_name_then_janitor_leaves_it( + test_case: OldNameJanitorClaimTestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Janitor never drops a relation the project built at a recorded view's name.""" + + prepare_table_rename(project_dir=tmp_path, capsys=capsys, project_toml=JANITOR_PROJECT_TOML) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + with monkeypatch.context() as patch: + fail_janitor_drop_fact(patch) + with contextlib.suppress(RuntimeError): + _ = run_sqb( + project_dir=tmp_path, + args=("janitor", "--auto-approve", "--drop-old-name-view", "dev.revenue"), + capsys=capsys, + ) + write_project( + project_dir=tmp_path, models=claiming_view_models(), project_toml=JANITOR_PROJECT_TOML + ) + with monkeypatch.context() as patch: + test_case.install_build_fault(patch) + for args in test_case.claiming_builds: + claiming: CliRun = run_sqb(project_dir=tmp_path, args=args, capsys=capsys) + assert claiming.exit_code == 0, claiming.output + + janitor: CliRun = run_sqb(project_dir=tmp_path, args=test_case.janitor_args, capsys=capsys) + old_name_type: str | None = relation_type_in( + project_dir=tmp_path, schema="dev", name=ORIGIN_MODEL + ) + facts: tuple[str, ...] = old_name_facts(project_dir=tmp_path, schema="dev") + rebuild: CliRun = build(project_dir=tmp_path, capsys=capsys) + + assert old_name_type == test_case.expected_old_name_type + assert facts == test_case.expected_final_facts + assert janitor.exit_code == 0, janitor.output + assert test_case.expected_janitor_fragment in janitor.output, janitor.output + assert rebuild.exit_code == 0, rebuild.output diff --git a/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_refresh_skip.py b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_refresh_skip.py new file mode 100644 index 0000000000..9e1d72d1ce --- /dev/null +++ b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_refresh_skip.py @@ -0,0 +1,84 @@ +"""Integration coverage for leaving unchanged late-binding compatibility views alone.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from tests.integration.src.sqlbuild.cli.commands.main.model_migrations.helpers import ( + build_ok, + load_raw_orders, + order_ids, + write_project, +) +from tests.integration.src.sqlbuild.cli.commands.main.old_name_views._test_types import ( + OldNameRefreshSkipTestCase, +) +from tests.integration.src.sqlbuild.cli.commands.main.old_name_views.helpers import ( + DESTINATION_MODEL, + ORIGIN_MODEL, + PROJECT_TOML, + aliased_table_sql, + record_old_name_view_ddl, + table_sql, +) + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameRefreshSkipTestCase( + description="unchanged select-star view is not redefined", + renamed_sql=table_sql(migrate_from=ORIGIN_MODEL), + rebuilt_sql=table_sql(migrate_from=ORIGIN_MODEL), + expected_view_statements=0, + ), + OldNameRefreshSkipTestCase( + description="unchanged aliased view is not redefined", + renamed_sql=aliased_table_sql(), + rebuilt_sql=aliased_table_sql(), + expected_view_statements=0, + ), + OldNameRefreshSkipTestCase( + description="aliased view whose column list changed is redefined once", + renamed_sql=aliased_table_sql(), + rebuilt_sql=aliased_table_sql(extra_column=", order_id * 10 AS order_rank"), + expected_view_statements=1, + ), + ], + ids=lambda case: case.description, +) +def test_given_live_view_when_destination_rebuilds_then_only_changed_sql_is_redefined( + test_case: OldNameRefreshSkipTestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Views that bind by name at query time need no DDL while their SQL is unchanged.""" + + write_project( + project_dir=tmp_path, models={ORIGIN_MODEL: table_sql()}, project_toml=PROJECT_TOML + ) + load_raw_orders(project_dir=tmp_path, first_day=1, last_day=3) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + write_project( + project_dir=tmp_path, + models={DESTINATION_MODEL: test_case.renamed_sql}, + project_toml=PROJECT_TOML, + ) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + write_project( + project_dir=tmp_path, + models={DESTINATION_MODEL: test_case.rebuilt_sql}, + project_toml=PROJECT_TOML, + ) + statements: list[str] = [] + + with monkeypatch.context() as patch: + record_old_name_view_ddl(patch, statements) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + + assert len(statements) == test_case.expected_view_statements, statements + assert order_ids(project_dir=tmp_path, relation=f"main.{ORIGIN_MODEL}") == (1, 2, 3) diff --git a/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_resume.py b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_resume.py new file mode 100644 index 0000000000..f973a2ad27 --- /dev/null +++ b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_resume.py @@ -0,0 +1,109 @@ +"""Integration coverage for resuming old-name steps after a crash in every failure window.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from tests.integration.src.sqlbuild.cli.commands.main.model_migrations.helpers import ( + CliRun, + build_ok, + order_ids, +) +from tests.integration.src.sqlbuild.cli.commands.main.old_name_views._test_types import ( + OldNameResumeTestCase, +) +from tests.integration.src.sqlbuild.cli.commands.main.old_name_views.helpers import ( + ORIGIN_MODEL, + build_result, + fail_destination_build, + fail_non_transactional_archive_fact, + fail_non_transactional_view, + fail_non_transactional_view_fact, + fail_transactional_view, + old_name_facts, + origin_archive_count, + prepare_table_rename, + relation_type, +) + +_ALL_FACTS: tuple[str, ...] = ("required", "origin_archived", "view_created") + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameResumeTestCase( + description="destination build fails after the move is recorded", + install_failure=fail_destination_build, + expected_first_exit_code=1, + expected_facts_after_failure=("required",), + expected_old_name_type_after_failure="BASE TABLE", + expected_final_facts=_ALL_FACTS, + expected_archive_count=1, + ), + OldNameResumeTestCase( + description="transactional crash creating the view rolls the archive back", + install_failure=fail_transactional_view, + expected_first_exit_code=1, + expected_facts_after_failure=("required",), + expected_old_name_type_after_failure="BASE TABLE", + expected_final_facts=_ALL_FACTS, + expected_archive_count=1, + ), + OldNameResumeTestCase( + description="non-transactional crash after archiving adopts the existing archive", + install_failure=fail_non_transactional_archive_fact, + expected_first_exit_code=1, + expected_facts_after_failure=("required",), + expected_old_name_type_after_failure=None, + expected_final_facts=_ALL_FACTS, + expected_archive_count=1, + ), + OldNameResumeTestCase( + description="non-transactional crash before the view creates only the view", + install_failure=fail_non_transactional_view, + expected_first_exit_code=1, + expected_facts_after_failure=("required", "origin_archived"), + expected_old_name_type_after_failure=None, + expected_final_facts=_ALL_FACTS, + expected_archive_count=1, + ), + OldNameResumeTestCase( + description="non-transactional crash after the view records it on retry", + install_failure=fail_non_transactional_view_fact, + expected_first_exit_code=1, + expected_facts_after_failure=("required", "origin_archived"), + expected_old_name_type_after_failure="VIEW", + expected_final_facts=_ALL_FACTS, + expected_archive_count=1, + ), + ], + ids=lambda case: case.description, +) +def test_given_interrupted_old_name_steps_when_rebuilding_then_steps_resume_once( + test_case: OldNameResumeTestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A retry completes the archive and view exactly once from facts and physical state.""" + + prepare_table_rename(project_dir=tmp_path, capsys=capsys) + + with monkeypatch.context() as patch: + test_case.install_failure(patch) + first: CliRun = build_result(project_dir=tmp_path, capsys=capsys) + facts_after_failure: tuple[str, ...] = old_name_facts(project_dir=tmp_path) + type_after_failure: str | None = relation_type(project_dir=tmp_path, name=ORIGIN_MODEL) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + + assert first.exit_code == test_case.expected_first_exit_code, first.output + assert facts_after_failure == test_case.expected_facts_after_failure + assert type_after_failure == test_case.expected_old_name_type_after_failure + assert old_name_facts(project_dir=tmp_path) == test_case.expected_final_facts + assert relation_type(project_dir=tmp_path, name=ORIGIN_MODEL) == "VIEW" + assert origin_archive_count(project_dir=tmp_path) == test_case.expected_archive_count + assert order_ids(project_dir=tmp_path, relation=f"main.{ORIGIN_MODEL}") == (1, 2, 3) diff --git a/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_resume_skips.py b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_resume_skips.py new file mode 100644 index 0000000000..8ca33c3cde --- /dev/null +++ b/tests/integration/src/sqlbuild/cli/commands/main/old_name_views/test_old_name_resume_skips.py @@ -0,0 +1,98 @@ +"""Integration coverage for resumed old-name steps that must leave the old name alone.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from tests.integration.src.sqlbuild.cli.commands.main.model_migrations.helpers import ( + build_ok, + order_ids, + write_project, +) +from tests.integration.src.sqlbuild.cli.commands.main.old_name_views._test_types import ( + OldNameResumeSkipTestCase, +) +from tests.integration.src.sqlbuild.cli.commands.main.old_name_views.helpers import ( + DESTINATION_MODEL, + ORIGIN_MODEL, + PROJECT_TOML, + build_result, + disabled_table_sql, + fail_destination_build, + fail_non_transactional_view, + old_name_facts, + origin_archive_count, + plan_text, + prepare_table_rename, + relation_type, + table_sql, +) + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameResumeSkipTestCase( + description="old model restored before the archive keeps its table", + install_failure=fail_destination_build, + retry_models={ORIGIN_MODEL: table_sql(), DESTINATION_MODEL: table_sql()}, + expected_plan_fragment=( + f" └── old name main.{ORIGIN_MODEL}\n" + f" └── left for janitor name reused by model:{ORIGIN_MODEL}\n" + ), + expected_facts=("required",), + expected_old_name_type="BASE TABLE", + expected_old_name_ids=(1, 2, 3), + ), + OldNameResumeSkipTestCase( + description="old model restored after the archive is rebuilt without a view", + install_failure=fail_non_transactional_view, + retry_models={ORIGIN_MODEL: table_sql(), DESTINATION_MODEL: table_sql()}, + expected_plan_fragment=( + f" └── left for janitor name reused by model:{ORIGIN_MODEL}\n" + ), + expected_facts=("required", "origin_archived"), + expected_old_name_type="BASE TABLE", + expected_old_name_ids=(1, 2, 3), + ), + OldNameResumeSkipTestCase( + description="old-name views turned off before the archive leave the old table", + install_failure=fail_destination_build, + retry_models={DESTINATION_MODEL: disabled_table_sql(migrate_from=ORIGIN_MODEL)}, + expected_plan_fragment=" └── left for janitor old_name_view false\n", + expected_facts=("required",), + expected_old_name_type="BASE TABLE", + expected_old_name_ids=(1, 2, 3), + ), + ], + ids=lambda case: case.description, +) +def test_given_pending_old_name_steps_when_old_name_is_no_longer_ours_then_resume_skips_them( + test_case: OldNameResumeSkipTestCase, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Resume applies the same checks as a new move: no archive and no view.""" + + prepare_table_rename(project_dir=tmp_path, capsys=capsys) + with monkeypatch.context() as patch: + test_case.install_failure(patch) + _ = build_result(project_dir=tmp_path, capsys=capsys) + write_project(project_dir=tmp_path, models=test_case.retry_models, project_toml=PROJECT_TOML) + + plan: str = plan_text(project_dir=tmp_path, capsys=capsys) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + _ = build_ok(project_dir=tmp_path, capsys=capsys) + + assert old_name_facts(project_dir=tmp_path) == test_case.expected_facts + assert relation_type(project_dir=tmp_path, name=ORIGIN_MODEL) == ( + test_case.expected_old_name_type + ) + assert order_ids(project_dir=tmp_path, relation=f"main.{ORIGIN_MODEL}") == ( + test_case.expected_old_name_ids + ) + assert origin_archive_count(project_dir=tmp_path) == len(test_case.expected_facts) - 1 + assert test_case.expected_plan_fragment in plan, plan diff --git a/tests/unit/src/sqlbuild/adapter/contract/classes/connection/_test_types.py b/tests/unit/src/sqlbuild/adapter/contract/classes/connection/_test_types.py index eda34fe334..1be92adf66 100644 --- a/tests/unit/src/sqlbuild/adapter/contract/classes/connection/_test_types.py +++ b/tests/unit/src/sqlbuild/adapter/contract/classes/connection/_test_types.py @@ -12,3 +12,10 @@ class SqlExecutionFailureCase: description: str sql: str expected_error: str + + +@dataclass(frozen=True) +class NestedTransactionCase: + description: str + fail_after_inner: bool + expected_tables: tuple[str, ...] diff --git a/tests/unit/src/sqlbuild/adapter/contract/classes/connection/helpers.py b/tests/unit/src/sqlbuild/adapter/contract/classes/connection/helpers.py new file mode 100644 index 0000000000..25aa493c60 --- /dev/null +++ b/tests/unit/src/sqlbuild/adapter/contract/classes/connection/helpers.py @@ -0,0 +1,13 @@ +"""Callbacks for connection contract tests.""" + +from __future__ import annotations + + +def interrupt() -> None: + """Raise as a crash inside a transaction would.""" + + raise RuntimeError("simulated interruption") + + +def carry_on() -> None: + """Do nothing.""" diff --git a/tests/unit/src/sqlbuild/adapter/contract/classes/connection/test_connection.py b/tests/unit/src/sqlbuild/adapter/contract/classes/connection/test_connection.py index 3b1a2df3f1..ff16725dbf 100644 --- a/tests/unit/src/sqlbuild/adapter/contract/classes/connection/test_connection.py +++ b/tests/unit/src/sqlbuild/adapter/contract/classes/connection/test_connection.py @@ -1,14 +1,23 @@ +import contextlib import inspect +from collections.abc import Callable from typing import Any, ClassVar +import duckdb import pytest from sqlbuild.adapter.contract.classes.connection import ConnectionMixin from sqlbuild.adapter.contract.models import QueryResult +from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter from tests.unit.src.sqlbuild.adapter.contract.classes.connection._test_types import ( ConnectionContractCase, + NestedTransactionCase, SqlExecutionFailureCase, ) +from tests.unit.src.sqlbuild.adapter.contract.classes.connection.helpers import ( + carry_on, + interrupt, +) class _LegacyExecuteOnlyAdapter(ConnectionMixin): @@ -111,3 +120,41 @@ def test_given_warehouse_failure_when_executing_then_exact_sql_is_attributed( assert vars(exc_info.value)["failed_sql"] == test_case.sql assert str(exc_info.value) == test_case.expected_error + + +@pytest.mark.parametrize( + "test_case", + [ + NestedTransactionCase( + description="nested transaction commits with the outer one", + fail_after_inner=False, + expected_tables=("inner_orders", "outer_orders"), + ), + NestedTransactionCase( + description="failure after a nested transaction rolls both back", + fail_after_inner=True, + expected_tables=(), + ), + ], + ids=lambda case: case.description, +) +def test_given_nested_transactions_when_outer_ends_then_one_boundary_decides( + test_case: NestedTransactionCase, +) -> None: + adapter: DuckDbAdapter = DuckDbAdapter() + connection: duckdb.DuckDBPyConnection = duckdb.connect(":memory:") + failures: dict[bool, Callable[[], None]] = {True: interrupt, False: carry_on} + + with contextlib.suppress(RuntimeError), adapter.transaction(connection): + _ = adapter.execute(connection=connection, sql="CREATE TABLE outer_orders (id INTEGER)") + with adapter.transaction(connection): + _ = adapter.execute(connection=connection, sql="CREATE TABLE inner_orders (id INTEGER)") + failures[test_case.fail_after_inner]() + + tables: tuple[str, ...] = tuple( + str(row[0]) + for row in connection.execute( + "SELECT table_name FROM information_schema.tables ORDER BY 1" + ).fetchall() + ) + assert tables == test_case.expected_tables diff --git a/tests/unit/src/sqlbuild/adapter/contract/main/__init__.py b/tests/unit/src/sqlbuild/adapter/contract/main/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/unit/src/sqlbuild/adapter/contract/main/_test_types.py b/tests/unit/src/sqlbuild/adapter/contract/main/_test_types.py new file mode 100644 index 0000000000..6fe68e38fa --- /dev/null +++ b/tests/unit/src/sqlbuild/adapter/contract/main/_test_types.py @@ -0,0 +1,13 @@ +"""Test case types for adapter contract entrypoints.""" + +from __future__ import annotations + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class SameViewDefinitionTestCase: + description: str + definition: str + sql: str + expected_match: bool diff --git a/tests/unit/src/sqlbuild/adapter/contract/main/test_same_view_definition.py b/tests/unit/src/sqlbuild/adapter/contract/main/test_same_view_definition.py new file mode 100644 index 0000000000..2aa30b8728 --- /dev/null +++ b/tests/unit/src/sqlbuild/adapter/contract/main/test_same_view_definition.py @@ -0,0 +1,67 @@ +"""Unit coverage for comparing stored view definitions with compatibility view SQL.""" + +from __future__ import annotations + +import pytest + +from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition +from tests.unit.src.sqlbuild.adapter.contract.main._test_types import SameViewDefinitionTestCase + +_SQL: str = 'SELECT "order_id", "revenue" AS "amount" FROM analytics.daily_revenue' + + +@pytest.mark.parametrize( + "test_case", + [ + SameViewDefinitionTestCase( + description="duckdb re-serialized create statement", + definition=( + "CREATE VIEW analytics.revenue AS SELECT order_id, revenue AS amount " + "FROM analytics.daily_revenue;" + ), + sql=_SQL, + expected_match=True, + ), + SameViewDefinitionTestCase( + description="snowflake create statement with copy grants", + definition=( + "CREATE OR REPLACE VIEW ANALYTICS.REVENUE COPY GRANTS AS " + 'SELECT "order_id", "revenue" AS "amount" FROM analytics.daily_revenue' + ), + sql=_SQL, + expected_match=True, + ), + SameViewDefinitionTestCase( + description="sql server bracketed alter statement", + definition=( + "ALTER VIEW [analytics].[revenue] AS SELECT [order_id], [revenue] AS [amount] " + "FROM analytics.daily_revenue" + ), + sql=_SQL, + expected_match=True, + ), + SameViewDefinitionTestCase( + description="postgres rewritten body on several lines", + definition=" SELECT order_id,\n revenue AS amount\n FROM analytics.daily_revenue;", + sql=" SELECT order_id,\n revenue AS amount\n FROM analytics.daily_revenue;", + expected_match=True, + ), + SameViewDefinitionTestCase( + description="another view at the old name", + definition=( + "CREATE VIEW analytics.revenue AS SELECT order_id, amount_cents * 2 AS doubled " + "FROM raw.raw_orders" + ), + sql=_SQL, + expected_match=False, + ), + ], + ids=lambda case: case.description, +) +def test_given_stored_definition_when_comparing_then_only_formatting_is_ignored( + test_case: SameViewDefinitionTestCase, +) -> None: + assert ( + same_view_definition(definition=test_case.definition, sql=test_case.sql) + == test_case.expected_match + ) diff --git a/tests/unit/src/sqlbuild/adapters/_test_types.py b/tests/unit/src/sqlbuild/adapters/_test_types.py index 8c847ee7b7..a436ca076a 100644 --- a/tests/unit/src/sqlbuild/adapters/_test_types.py +++ b/tests/unit/src/sqlbuild/adapters/_test_types.py @@ -2,6 +2,7 @@ from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter from sqlbuild.adapter.contract.classes.strict_adapter import StrictAdapter +from sqlbuild.adapter.contract.models import RelationGrant from sqlbuild.adapter.contract.types import MigrationTransfer @@ -96,3 +97,31 @@ class DatabricksColumnMappingTestCase: description: str property_rows: list[tuple[str, str]] expected_available: bool + + +@dataclass(frozen=True) +class RelationGrantCaptureTestCase: + description: str + adapter: BaseAdapter + relation_type: str + answers: tuple[list[tuple[object, ...]], ...] + destination: str + columns: tuple[str, ...] + expected_query_fragment: str + expected_statements: tuple[str, ...] + + +@dataclass(frozen=True) +class ViewReplaceTestCase: + description: str + adapter: BaseAdapter + expected_statements: tuple[str, ...] | None + + +@dataclass(frozen=True) +class RelationRevokeTestCase: + description: str + adapter: BaseAdapter + grants: tuple[RelationGrant, ...] + destination: str + expected_statements: tuple[str, ...] diff --git a/tests/unit/src/sqlbuild/adapters/helpers.py b/tests/unit/src/sqlbuild/adapters/helpers.py index 7c5a3613fa..b5d256ca35 100644 --- a/tests/unit/src/sqlbuild/adapters/helpers.py +++ b/tests/unit/src/sqlbuild/adapters/helpers.py @@ -2,7 +2,7 @@ from __future__ import annotations -from collections.abc import Callable +from collections.abc import Callable, Iterator from typing import Any @@ -29,3 +29,47 @@ def execute(*, connection: Any, sql: str) -> FetchedRows: return FetchedRows(rows) return execute + + +class GrantRows: + """A cursor result that returns fixed grant rows of any width.""" + + def __init__(self, rows: list[tuple[object, ...]]) -> None: + self._rows: list[tuple[object, ...]] = rows + + def fetchall(self) -> list[tuple[object, ...]]: + """Return the fixed rows.""" + + return self._rows + + +class GrantConnection: + """A connection that records SQL, answers queries in order, and names a project.""" + + def __init__(self, answers: tuple[list[tuple[object, ...]], ...]) -> None: + self.answers: Iterator[list[tuple[object, ...]]] = iter(answers) + self.executed: list[str] = [] + self.location: str = "US" + self.client: GrantClient = GrantClient() + + def execute(self, sql: str) -> GrantRows: + """Record the SQL and return the next answer, or no rows once they run out.""" + + self.executed.append(sql) + return GrantRows(next(self.answers, [])) + + +class GrantClient: + """The BigQuery client attribute of a fake connection.""" + + project: str = "orders-project" + + +def grant_execute(connection: GrantConnection) -> Callable[..., GrantRows]: + """Return an adapter execute stub that routes SQL to a recording connection.""" + + def execute(*, connection: Any, sql: str) -> GrantRows: + return connection.execute(sql) + + del connection + return execute diff --git a/tests/unit/src/sqlbuild/adapters/test_old_name_view_grants.py b/tests/unit/src/sqlbuild/adapters/test_old_name_view_grants.py new file mode 100644 index 0000000000..14f70e23ed --- /dev/null +++ b/tests/unit/src/sqlbuild/adapters/test_old_name_view_grants.py @@ -0,0 +1,375 @@ +"""Unit coverage for reading and replaying relation privileges onto a compatibility view.""" + +from __future__ import annotations + +import pytest + +from sqlbuild.adapter.contract.classes.statement_recorder import StatementRecorder +from sqlbuild.adapter.contract.models import RelationGrant +from sqlbuild.adapters.bigquery.classes.bigquery_adapter import BigQueryAdapter +from sqlbuild.adapters.databricks.classes.databricks_adapter import DatabricksAdapter +from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter +from sqlbuild.adapters.postgres.classes.postgres_adapter import PostgresAdapter +from sqlbuild.adapters.snowflake.classes.snowflake_adapter import SnowflakeAdapter +from sqlbuild.adapters.sqlserver.classes.sqlserver_adapter import SqlServerAdapter +from tests.unit.src.sqlbuild.adapters._test_types import ( + RelationGrantCaptureTestCase, + RelationRevokeTestCase, + ViewReplaceTestCase, +) +from tests.unit.src.sqlbuild.adapters.helpers import GrantConnection, grant_execute + +_ARCHIVE: str = "_sqb_archive__20260928t101500z__migration_origin__revenue" +_EXPOSED: tuple[str, ...] = ("order_id", "amount") + + +@pytest.mark.parametrize( + "test_case", + [ + RelationGrantCaptureTestCase( + description="duckdb has no object privileges", + adapter=DuckDbAdapter(), + relation_type="table", + answers=(), + destination="analytics.revenue", + columns=_EXPOSED, + expected_query_fragment="", + expected_statements=(), + ), + RelationGrantCaptureTestCase( + description="postgres replays table privileges and public grants", + adapter=PostgresAdapter(), + relation_type="table", + answers=([("SELECT", None, False), ("SELECT", "orders_reader", True)], []), + destination="analytics.revenue", + columns=_EXPOSED, + expected_query_fragment="aclexplode(relation.relacl)", + expected_statements=( + "GRANT SELECT ON analytics.revenue TO PUBLIC", + 'GRANT SELECT ON analytics.revenue TO "orders_reader" WITH GRANT OPTION', + ), + ), + RelationGrantCaptureTestCase( + description="postgres replays column privileges only on columns the view exposes", + adapter=PostgresAdapter(), + relation_type="table", + answers=( + [], + [ + ("amount", "SELECT", "orders_reader", False), + ("discount", "SELECT", "orders_reader", False), + ("order_id", "UPDATE", None, True), + ], + ), + destination="analytics.revenue", + columns=_EXPOSED, + expected_query_fragment="aclexplode(attribute.attacl)", + expected_statements=( + 'GRANT SELECT ("amount") ON analytics.revenue TO "orders_reader"', + 'GRANT UPDATE ("order_id") ON analytics.revenue TO PUBLIC WITH GRANT OPTION', + ), + ), + RelationGrantCaptureTestCase( + description="snowflake keeps view privileges and grant option, skips ownership", + adapter=SnowflakeAdapter(), + relation_type="table", + answers=( + [ + ("t", "OWNERSHIP", "TABLE", _ARCHIVE, "ROLE", "TRANSFORMER", "true", "X"), + ("t", "SELECT", "TABLE", _ARCHIVE, "ROLE", "REPORTING", "true", "X"), + ("t", "INSERT", "TABLE", _ARCHIVE, "ROLE", "LOADER", "false", "X"), + ( + "t", + "SELECT", + "TABLE", + _ARCHIVE, + "DATABASE_ROLE", + "ORDERS.READ", + "false", + "X", + ), + ("t", "SELECT", "TABLE", _ARCHIVE, "SHARE", "PARTNER_SHARE", "false", "X"), + ], + ), + destination="analytics.revenue", + columns=_EXPOSED, + expected_query_fragment=f"SHOW GRANTS ON TABLE analytics.{_ARCHIVE}", + expected_statements=( + 'GRANT SELECT ON VIEW analytics.revenue TO ROLE "REPORTING" WITH GRANT OPTION', + 'GRANT SELECT ON VIEW analytics.revenue TO DATABASE ROLE "ORDERS"."READ"', + ), + ), + RelationGrantCaptureTestCase( + description="snowflake reads grants of an archived view model as a view", + adapter=SnowflakeAdapter(), + relation_type="VIEW", + answers=([("t", "SELECT", "VIEW", _ARCHIVE, "ROLE", "REPORTING", "false", "X")],), + destination="analytics.revenue", + columns=_EXPOSED, + expected_query_fragment=f"SHOW GRANTS ON VIEW analytics.{_ARCHIVE}", + expected_statements=('GRANT SELECT ON VIEW analytics.revenue TO ROLE "REPORTING"',), + ), + RelationGrantCaptureTestCase( + description="databricks replays direct unity catalog grants that apply to views", + adapter=DatabricksAdapter(), + relation_type="table", + answers=( + [ + ("analysts", "SELECT", "TABLE", "main.analytics.x"), + ("loaders", "MODIFY", "TABLE", "main.analytics.x"), + ("owners", "OWN", "TABLE", "main.analytics.x"), + ("everyone", "USE SCHEMA", "SCHEMA", "main.analytics"), + ], + ), + destination="`main`.`analytics`.`revenue`", + columns=_EXPOSED, + expected_query_fragment="SHOW GRANTS ON TABLE", + expected_statements=( + "GRANT SELECT ON VIEW `main`.`analytics`.`revenue` TO `analysts`", + ), + ), + RelationGrantCaptureTestCase( + description="sql server replays object grants, grant options, and denies", + adapter=SqlServerAdapter(), + relation_type="table", + answers=( + [ + ("GRANT", "SELECT", "reporting", None), + ("GRANT_WITH_GRANT_OPTION", "SELECT", "leads", None), + ("DENY", "UPDATE", "reporting", None), + ("GRANT", "EXECUTE", "reporting", None), + ], + ), + destination="[analytics].[revenue]", + columns=_EXPOSED, + expected_query_fragment="sys.database_permissions", + expected_statements=( + "GRANT SELECT ON OBJECT::[analytics].[revenue] TO [reporting]", + "GRANT SELECT ON OBJECT::[analytics].[revenue] TO [leads] WITH GRANT OPTION", + "DENY UPDATE ON OBJECT::[analytics].[revenue] TO [reporting]", + ), + ), + RelationGrantCaptureTestCase( + description="sql server keeps column denies on exposed columns after object grants", + adapter=SqlServerAdapter(), + relation_type="table", + answers=( + [ + ("DENY", "SELECT", "reporting", "amount"), + ("GRANT", "SELECT", "reporting", None), + ("GRANT", "SELECT", "leads", "order_id"), + ("DENY", "SELECT", "reporting", "discount"), + ], + ), + destination="[analytics].[revenue]", + columns=_EXPOSED, + expected_query_fragment="COL_NAME(permission.major_id, NULLIF(permission.minor_id, 0))", + expected_statements=( + "GRANT SELECT ON OBJECT::[analytics].[revenue] TO [reporting]", + "DENY SELECT ON OBJECT::[analytics].[revenue] ([amount]) TO [reporting]", + "GRANT SELECT ON OBJECT::[analytics].[revenue] ([order_id]) TO [leads]", + ), + ), + RelationGrantCaptureTestCase( + description="bigquery replays table iam role bindings", + adapter=BigQueryAdapter(), + relation_type="table", + answers=([("roles/bigquery.dataViewer", "group:analysts@example.com")],), + destination="orders-project.analytics.revenue", + columns=_EXPOSED, + expected_query_fragment=( + "`orders-project.region-us`.INFORMATION_SCHEMA.OBJECT_PRIVILEGES" + ), + expected_statements=( + "GRANT `roles/bigquery.dataViewer` ON VIEW `orders-project.analytics.revenue` " + 'TO "group:analysts@example.com"', + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_archived_relation_grants_when_replaying_then_view_statements_copy_them( + test_case: RelationGrantCaptureTestCase, monkeypatch: pytest.MonkeyPatch +) -> None: + connection: GrantConnection = GrantConnection(test_case.answers) + monkeypatch.setattr(test_case.adapter, "execute", grant_execute(connection)) + + grants: tuple[RelationGrant, ...] = test_case.adapter.read_relation_grants( + connection=connection, + database=None, + schema="analytics", + name=_ARCHIVE, + relation_type=test_case.relation_type, + ) + statements: tuple[str, ...] = test_case.adapter.render_relation_grants( + grants=grants, destination=test_case.destination, columns=test_case.columns + ) + + assert statements == test_case.expected_statements + assert test_case.expected_query_fragment in "".join(connection.executed) + + +@pytest.mark.parametrize( + "test_case", + [ + RelationGrantCaptureTestCase( + description="bigquery copies the view's iam bindings to the archive before dropping", + adapter=BigQueryAdapter(), + relation_type="view", + answers=( + [("SELECT order_id FROM `orders-project.analytics.daily_revenue`",)], + [("roles/bigquery.dataViewer", "group:analysts@example.com")], + ), + destination=f"orders-project.analytics.{_ARCHIVE}", + columns=(), + expected_query_fragment="INFORMATION_SCHEMA.OBJECT_PRIVILEGES", + expected_statements=( + f"CREATE VIEW `orders-project.analytics.{_ARCHIVE}` AS " + "SELECT order_id FROM `orders-project.analytics.daily_revenue`", + "GRANT `roles/bigquery.dataViewer` ON VIEW " + f'`orders-project.analytics.{_ARCHIVE}` TO "group:analysts@example.com"', + "DROP VIEW `orders-project.analytics.revenue`", + ), + ) + ], + ids=lambda case: case.description, +) +def test_given_bigquery_view_when_archiving_then_definition_and_grants_move_before_drop( + test_case: RelationGrantCaptureTestCase, monkeypatch: pytest.MonkeyPatch +) -> None: + connection: GrantConnection = GrantConnection(test_case.answers) + monkeypatch.setattr(test_case.adapter, "execute", grant_execute(connection)) + recorder: StatementRecorder = StatementRecorder() + + test_case.adapter.rename_view( + connection=connection, + origin="orders-project.analytics.revenue", + destination=test_case.destination, + statement_recorder=recorder, + ) + + assert "INFORMATION_SCHEMA.VIEWS" in connection.executed[0] + assert test_case.expected_query_fragment in connection.executed[1] + assert tuple(connection.executed[2:]) == test_case.expected_statements + + +@pytest.mark.parametrize( + "test_case", + [ + ViewReplaceTestCase( + description="postgres create or replace keeps privileges", + adapter=PostgresAdapter(), + expected_statements=("CREATE OR REPLACE VIEW analytics.revenue AS SELECT 1",), + ), + ViewReplaceTestCase( + description="snowflake copies grants onto the replacement", + adapter=SnowflakeAdapter(), + expected_statements=( + "CREATE OR REPLACE VIEW analytics.revenue COPY GRANTS AS SELECT 1", + ), + ), + ViewReplaceTestCase( + description="sql server alters the view in place", + adapter=SqlServerAdapter(), + expected_statements=("ALTER VIEW analytics.revenue AS SELECT 1",), + ), + ViewReplaceTestCase( + description="databricks alters the view in place", + adapter=DatabricksAdapter(), + expected_statements=("ALTER VIEW analytics.revenue AS SELECT 1",), + ), + ViewReplaceTestCase( + description="bigquery has no privilege-preserving replace", + adapter=BigQueryAdapter(), + expected_statements=None, + ), + ], + ids=lambda case: case.description, +) +def test_given_existing_view_when_redefining_then_adapter_keeps_its_privileges( + test_case: ViewReplaceTestCase, +) -> None: + statements: tuple[str, ...] | None = test_case.adapter.render_replace_view_keeping_grants( + destination="analytics.revenue", sql="SELECT 1" + ) + + assert statements == test_case.expected_statements + + +@pytest.mark.parametrize( + "test_case", + [ + RelationRevokeTestCase( + description="duckdb has nothing to revoke", + adapter=DuckDbAdapter(), + grants=(RelationGrant(privilege="SELECT", grantee="reporting"),), + destination="analytics.revenue", + expected_statements=(), + ), + RelationRevokeTestCase( + description="postgres revokes table, column, and public privileges", + adapter=PostgresAdapter(), + grants=( + RelationGrant(privilege="SELECT", grantee="reporting", grantable=True), + RelationGrant(privilege="UPDATE", grantee=None, column="amount"), + ), + destination="analytics.revenue", + expected_statements=( + 'REVOKE SELECT ON analytics.revenue FROM "reporting"', + 'REVOKE UPDATE ("amount") ON analytics.revenue FROM PUBLIC', + ), + ), + RelationRevokeTestCase( + description="snowflake revokes a future grant from a role", + adapter=SnowflakeAdapter(), + grants=(RelationGrant(privilege="SELECT", grantee="REPORTING", grantee_kind="ROLE"),), + destination="analytics.revenue", + expected_statements=('REVOKE SELECT ON VIEW analytics.revenue FROM ROLE "REPORTING"',), + ), + RelationRevokeTestCase( + description="databricks revokes from a principal", + adapter=DatabricksAdapter(), + grants=(RelationGrant(privilege="SELECT", grantee="analysts"),), + destination="`main`.`analytics`.`revenue`", + expected_statements=( + "REVOKE SELECT ON VIEW `main`.`analytics`.`revenue` FROM `analysts`", + ), + ), + RelationRevokeTestCase( + description="sql server revokes grants and denies, cascading grant options", + adapter=SqlServerAdapter(), + grants=( + RelationGrant(privilege="SELECT", grantee="leads", grantable=True), + RelationGrant( + privilege="SELECT", grantee="reporting", column="amount", denied=True + ), + ), + destination="[analytics].[revenue]", + expected_statements=( + "REVOKE SELECT ON OBJECT::[analytics].[revenue] FROM [leads] CASCADE", + "REVOKE SELECT ON OBJECT::[analytics].[revenue] ([amount]) FROM [reporting]", + ), + ), + RelationRevokeTestCase( + description="bigquery removes a role binding", + adapter=BigQueryAdapter(), + grants=( + RelationGrant(privilege="roles/bigquery.dataViewer", grantee="group:a@example.com"), + ), + destination="orders-project.analytics.revenue", + expected_statements=( + "REVOKE `roles/bigquery.dataViewer` ON VIEW `orders-project.analytics.revenue` " + 'FROM "group:a@example.com"', + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_added_grants_when_reconciling_then_adapter_renders_revokes( + test_case: RelationRevokeTestCase, +) -> None: + statements: tuple[str, ...] = test_case.adapter.render_relation_revokes( + grants=test_case.grants, destination=test_case.destination + ) + + assert statements == test_case.expected_statements diff --git a/tests/unit/src/sqlbuild/compiler/discovery/_helpers/test_yml_project.py b/tests/unit/src/sqlbuild/compiler/discovery/_helpers/test_yml_project.py index 998232e4db..9ab93dc182 100644 --- a/tests/unit/src/sqlbuild/compiler/discovery/_helpers/test_yml_project.py +++ b/tests/unit/src/sqlbuild/compiler/discovery/_helpers/test_yml_project.py @@ -1634,6 +1634,28 @@ def test_given_local_target_execution_limits_when_loading_then_limits_are_typed( """.strip(), expected_error_fragment="janitor.archive_retention_days must be >= 0", ), + LoadProjectConfigErrorTestCase( + description="raises when old name view retention is not a duration", + project_file_contents=""" +name = "demo" +adapter = "duckdb" + +[migrations] +old_name_views = 30 +""".strip(), + expected_error_fragment="migrations.old_name_views must be a positive duration", + ), + LoadProjectConfigErrorTestCase( + description="raises when migrations contain unknown key", + project_file_contents=""" +name = "demo" +adapter = "duckdb" + +[migrations] +old_name_view = "7d" +""".strip(), + expected_error_fragment=r"migrations contains unknown key\(s\): old_name_view", + ), LoadProjectConfigErrorTestCase( description="raises when project settings contain unknown key", project_file_contents=""" diff --git a/tests/unit/src/sqlbuild/compiler/migrations/main/_test_types.py b/tests/unit/src/sqlbuild/compiler/migrations/main/_test_types.py index c320687fcc..1813408fa0 100644 --- a/tests/unit/src/sqlbuild/compiler/migrations/main/_test_types.py +++ b/tests/unit/src/sqlbuild/compiler/migrations/main/_test_types.py @@ -35,3 +35,28 @@ class NewestColumnEventTestCase: origin_column: str destination_column: str expected_newest_index: int | None + + +@dataclass(frozen=True) +class OldNameViewStorageTestCase: + description: str + write_count: int + foreign_event_types: tuple[str, ...] + expected_event_types: tuple[str, ...] + + +@dataclass(frozen=True) +class OldNameViewProjectionTestCase: + description: str + recorded_types: tuple[str, ...] + recorded_moves: int + expires_in_days: int + expected_statuses: tuple[str, ...] + + +@dataclass(frozen=True) +class OldNameViewAbsentColumnsTestCase: + description: str + absent_columns: tuple[str, ...] + expected_statuses: tuple[str, ...] + expected_archive_name: str | None diff --git a/tests/unit/src/sqlbuild/compiler/migrations/main/helpers.py b/tests/unit/src/sqlbuild/compiler/migrations/main/helpers.py index 68708910a6..8d6e7e1191 100644 --- a/tests/unit/src/sqlbuild/compiler/migrations/main/helpers.py +++ b/tests/unit/src/sqlbuild/compiler/migrations/main/helpers.py @@ -10,15 +10,20 @@ from sqlbuild.compiler.migrations.main.deterministic_event_id import ( deterministic_migration_event_id, ) +from sqlbuild.compiler.migrations.main.deterministic_old_name_view_event_id import ( + deterministic_old_name_view_event_id, +) from sqlbuild.compiler.migrations.models import ( ColumnMigrationEvent, MigrationEvent, MigrationRelation, + OldNameViewEvent, ) from sqlbuild.compiler.migrations.types import ( ColumnMigrationDecision, MigrationDecision, MigrationDiscovery, + OldNameViewEventType, ) _STARTED_AT: datetime = datetime(2026, 1, 1, tzinfo=UTC) @@ -75,3 +80,34 @@ def column_migration_event(*, origin: str, destination: str, day: int) -> Column run_id=f"run-{day}", created_at=_STARTED_AT + timedelta(days=day), ) + + +def old_name_fact( + *, + move: MigrationEvent, + event_type: OldNameViewEventType, + expires_at: datetime | None = None, + column_aliases: tuple[tuple[str, str], ...] = (), + archive_name: str | None = None, + grants_copied: tuple[str, ...] | None = None, +) -> OldNameViewEvent: + """Return one old-name fact of a move from main.revenue to main.daily_revenue.""" + + return OldNameViewEvent( + event_id=deterministic_old_name_view_event_id( + event_type=event_type, migration_event_id=move.event_id + ), + target_name="dev", + event_type=event_type, + migration_event_id=move.event_id, + destination_model=move.destination_model, + old=main_relation("revenue"), + new=main_relation("daily_revenue"), + run_id=move.run_id, + created_at=move.created_at, + view_retention="30d", + column_aliases=column_aliases, + expires_at=expires_at, + archive_name=archive_name, + grants_copied=grants_copied, + ) diff --git a/tests/unit/src/sqlbuild/compiler/migrations/main/test_old_name_view_events.py b/tests/unit/src/sqlbuild/compiler/migrations/main/test_old_name_view_events.py new file mode 100644 index 0000000000..4e7638cbf7 --- /dev/null +++ b/tests/unit/src/sqlbuild/compiler/migrations/main/test_old_name_view_events.py @@ -0,0 +1,250 @@ +"""Unit coverage for append-only old-name view facts and their projection.""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta + +import duckdb +import pytest + +from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter +from sqlbuild.compiler.migrations.constants import OLD_NAME_VIEW_COLUMNS +from sqlbuild.compiler.migrations.main._project_old_name_views import project_old_name_views +from sqlbuild.compiler.migrations.main.read_old_name_view_events import ( + read_old_name_view_events, +) +from sqlbuild.compiler.migrations.main.write_event import write_migration_event +from sqlbuild.compiler.migrations.main.write_old_name_view_event import write_old_name_view_event +from sqlbuild.compiler.migrations.models import ( + MigrationEvent, + OldNameViewEvent, + OldNameViewHistory, +) +from sqlbuild.compiler.migrations.types import OldNameViewEventType +from sqlbuild.compiler.planner.classes.migration_state_inspection import ( + MigrationStateInspection, +) +from tests.unit.src.sqlbuild.compiler.migrations.main._test_types import ( + OldNameViewAbsentColumnsTestCase, + OldNameViewProjectionTestCase, + OldNameViewStorageTestCase, +) +from tests.unit.src.sqlbuild.compiler.migrations.main.helpers import ( + migration_event, + old_name_fact, +) + +_NOW: datetime = datetime(2026, 2, 1, tzinfo=UTC) + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameViewStorageTestCase( + description="writing the same fact twice stores it once", + write_count=2, + foreign_event_types=(), + expected_event_types=("view_created",), + ), + OldNameViewStorageTestCase( + description="facts of a type from a newer version are skipped", + write_count=1, + foreign_event_types=("view_renamed_by_a_future_version",), + expected_event_types=("view_created",), + ), + ], + ids=lambda case: case.description, +) +def test_given_old_name_facts_when_writing_and_reading_then_rows_are_idempotent( + test_case: OldNameViewStorageTestCase, +) -> None: + adapter: DuckDbAdapter = DuckDbAdapter() + connection: duckdb.DuckDBPyConnection = duckdb.connect(":memory:") + move: MigrationEvent = migration_event(origin="revenue", destination="daily_revenue", day=1) + fact: OldNameViewEvent = old_name_fact( + move=move, + event_type=OldNameViewEventType.VIEW_CREATED, + expires_at=_NOW, + column_aliases=(("amount", "revenue"),), + ) + + for _ in range(test_case.write_count): + write_old_name_view_event( + connection=connection, + execute=adapter.execute, + event=fact, + render_qualified_name=adapter.render_qualified_name, + create_table_sql=adapter.render_create_old_name_view_state_table_sql( + database=None, schema="main" + ), + ) + for foreign in test_case.foreign_event_types: + _ = connection.execute( + "INSERT INTO main._sqlbuild_old_name_views (event_id, event_type, " + "migration_event_id, destination_model, old_name, new_name, run_id, created_at) " + f"VALUES ('future', '{foreign}', 'm', 'daily_revenue', 'revenue', 'daily_revenue', " + "'run-9', TIMESTAMP '2026-01-09')" + ) + stored: tuple[OldNameViewEvent, ...] = read_old_name_view_events( + connection=connection, + execute=adapter.execute, + database=None, + schema="main", + stored_columns=OLD_NAME_VIEW_COLUMNS, + render_qualified_name=adapter.render_qualified_name, + ) + + assert tuple(event.event_type.value for event in stored) == test_case.expected_event_types + assert stored == (fact,) + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameViewProjectionTestCase( + description="move recorded before old-name views existed has no history", + recorded_types=(), + recorded_moves=1, + expires_in_days=1, + expected_statuses=(), + ), + OldNameViewProjectionTestCase( + description="requirement without its move is ignored", + recorded_types=("required",), + recorded_moves=0, + expires_in_days=1, + expected_statuses=(), + ), + OldNameViewProjectionTestCase( + description="required move waits for its archive", + recorded_types=("required",), + recorded_moves=1, + expires_in_days=1, + expected_statuses=("pending_archive",), + ), + OldNameViewProjectionTestCase( + description="archived move waits for its view", + recorded_types=("required", "origin_archived"), + recorded_moves=1, + expires_in_days=1, + expected_statuses=("pending_view",), + ), + OldNameViewProjectionTestCase( + description="created view is live until it expires", + recorded_types=("required", "origin_archived", "view_created"), + recorded_moves=1, + expires_in_days=1, + expected_statuses=("live",), + ), + OldNameViewProjectionTestCase( + description="created view past its expiry is expired", + recorded_types=("required", "origin_archived", "view_created"), + recorded_moves=1, + expires_in_days=-1, + expected_statuses=("expired",), + ), + OldNameViewProjectionTestCase( + description="dropped view is dropped", + recorded_types=("required", "origin_archived", "view_created", "view_dropped"), + recorded_moves=1, + expires_in_days=-1, + expected_statuses=("dropped",), + ), + ], + ids=lambda case: case.description, +) +def test_given_recorded_facts_when_projecting_then_status_follows_the_newest_step( + test_case: OldNameViewProjectionTestCase, +) -> None: + move: MigrationEvent = migration_event(origin="revenue", destination="daily_revenue", day=1) + facts: tuple[OldNameViewEvent, ...] = tuple( + old_name_fact( + move=move, + event_type=OldNameViewEventType(event_type), + expires_at=_NOW + timedelta(days=test_case.expires_in_days), + ) + for event_type in test_case.recorded_types + ) + + histories: tuple[OldNameViewHistory, ...] = project_old_name_views( + moves=(move,)[: test_case.recorded_moves], facts=facts + ) + + assert tuple(history.status(now=_NOW).value for history in histories) == ( + test_case.expected_statuses + ) + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameViewAbsentColumnsTestCase( + description="table without grants_copied reads grants as unknown", + absent_columns=("grants_copied",), + expected_statuses=("live",), + expected_archive_name="_sqb_archive__revenue", + ), + OldNameViewAbsentColumnsTestCase( + description="table without grants_copied and archive_name reads both as null", + absent_columns=("grants_copied", "archive_name"), + expected_statuses=("live",), + expected_archive_name=None, + ), + ], + ids=lambda case: case.description, +) +def test_given_state_table_without_optional_columns_when_planning_then_absent_columns_read_null( + test_case: OldNameViewAbsentColumnsTestCase, +) -> None: + adapter: DuckDbAdapter = DuckDbAdapter() + connection: duckdb.DuckDBPyConnection = duckdb.connect(":memory:") + move: MigrationEvent = migration_event(origin="revenue", destination="daily_revenue", day=1) + write_migration_event( + connection=connection, + execute=adapter.execute, + event=move, + render_qualified_name=adapter.render_qualified_name, + create_table_sql=adapter.render_create_migration_state_table_sql( + database=None, schema="main" + ), + ) + event_type: OldNameViewEventType + for event_type in ( + OldNameViewEventType.REQUIRED, + OldNameViewEventType.ORIGIN_ARCHIVED, + OldNameViewEventType.VIEW_CREATED, + ): + write_old_name_view_event( + connection=connection, + execute=adapter.execute, + event=old_name_fact( + move=move, + event_type=event_type, + expires_at=_NOW + timedelta(days=30), + archive_name="_sqb_archive__revenue", + grants_copied=("GRANT SELECT ON main.revenue TO reporting",), + ), + render_qualified_name=adapter.render_qualified_name, + create_table_sql=adapter.render_create_old_name_view_state_table_sql( + database=None, schema="main" + ), + ) + column: str + for column in test_case.absent_columns: + _ = connection.execute(f"ALTER TABLE main._sqlbuild_old_name_views DROP COLUMN {column}") + inspection: MigrationStateInspection = MigrationStateInspection( + adapter=adapter, connection=connection, database=None + ) + + inspection.inspect_old_name_views(schemas={"main"}) + + histories: tuple[OldNameViewHistory, ...] = inspection.old_name_histories + assert tuple(history.status(now=_NOW).value for history in histories) == ( + test_case.expected_statuses + ) + created: OldNameViewEvent | None = histories[0].created + archived: OldNameViewEvent | None = histories[0].archived + assert created is not None + assert created.grants_copied is None + assert archived is not None + assert archived.archive_name == test_case.expected_archive_name diff --git a/tests/unit/src/sqlbuild/compiler/planner/_helpers/migrations/_test_types.py b/tests/unit/src/sqlbuild/compiler/planner/_helpers/migrations/_test_types.py index 345a72273b..d5ab78845a 100644 --- a/tests/unit/src/sqlbuild/compiler/planner/_helpers/migrations/_test_types.py +++ b/tests/unit/src/sqlbuild/compiler/planner/_helpers/migrations/_test_types.py @@ -67,3 +67,14 @@ class UnreadableQueryShapeTestCase: description: str query_sql: str expected_shape: None = None + + +@dataclass(frozen=True) +class OldNameRowsTestCase: + description: str + action: str + column_aliases: tuple[tuple[str, str], ...] + grants_supported: bool + grants_copied: int | None + reason: str | None + expected_rows: tuple[str, ...] diff --git a/tests/unit/src/sqlbuild/compiler/planner/_helpers/migrations/helpers.py b/tests/unit/src/sqlbuild/compiler/planner/_helpers/migrations/helpers.py index d75a770cd9..2a04d45b9f 100644 --- a/tests/unit/src/sqlbuild/compiler/planner/_helpers/migrations/helpers.py +++ b/tests/unit/src/sqlbuild/compiler/planner/_helpers/migrations/helpers.py @@ -3,10 +3,13 @@ from __future__ import annotations import json +from datetime import UTC, datetime +from sqlbuild.compiler.compile.models import CompiledRelationLocation +from sqlbuild.compiler.migrations.types import OldNameViewAction from sqlbuild.compiler.planner._helpers.migrations.fingerprint import build_migration_fingerprint from sqlbuild.compiler.planner._helpers.migrations.projections import parse_query_shape -from sqlbuild.compiler.planner.models import QueryShape +from sqlbuild.compiler.planner.models import OldNameViewPlanEntry, QueryShape from sqlbuild.compiler.planner.types import InputColumns BASE_CONFIG: dict[str, object] = { @@ -46,3 +49,32 @@ def input_lookup(relations: dict[tuple[str, str], frozenset[str]]) -> InputColum """Return an input-column lookup over (reference kind, relation name) pairs.""" return lambda kind, name: relations.get((kind, name)) + + +def old_name_entry( + *, + action: str, + column_aliases: tuple[tuple[str, str], ...], + grants_supported: bool, + grants_copied: int | None, + reason: str | None, +) -> OldNameViewPlanEntry: + """Return an old-name plan entry for prod.revenue expiring on 2026-10-28.""" + + return OldNameViewPlanEntry( + model_name="daily_revenue", + origin=CompiledRelationLocation( + database=None, schema="prod", name="revenue", qualified_name="prod.revenue" + ), + destination=CompiledRelationLocation( + database=None, schema="prod", name="daily_revenue", qualified_name="prod.daily_revenue" + ), + action=OldNameViewAction(action), + target_name="prod", + retention="30d", + expires_at=datetime(2026, 10, 28, tzinfo=UTC), + column_aliases=column_aliases, + reason=reason, + grants_copied=grants_copied, + grants_supported=grants_supported, + ) diff --git a/tests/unit/src/sqlbuild/compiler/planner/_helpers/migrations/test_old_name_rows.py b/tests/unit/src/sqlbuild/compiler/planner/_helpers/migrations/test_old_name_rows.py new file mode 100644 index 0000000000..929f6a6529 --- /dev/null +++ b/tests/unit/src/sqlbuild/compiler/planner/_helpers/migrations/test_old_name_rows.py @@ -0,0 +1,89 @@ +"""Unit coverage for the nested old-name block in migration plan text.""" + +from __future__ import annotations + +import pytest + +from sqlbuild.compiler.planner._helpers.migrations.display import old_name_rows +from sqlbuild.presentation.classes.cli_style import CliStyle +from tests.unit.src.sqlbuild.compiler.planner._helpers.migrations._test_types import ( + OldNameRowsTestCase, +) +from tests.unit.src.sqlbuild.compiler.planner._helpers.migrations.helpers import old_name_entry + + +@pytest.mark.parametrize( + "test_case", + [ + OldNameRowsTestCase( + description="planned move with renamed columns and grants", + action="archive_and_view", + column_aliases=(("amount", "revenue"),), + grants_supported=True, + grants_copied=None, + reason=None, + expected_rows=( + " old name prod.revenue", + " ├── view until 2026-10-28", + " ├── columns amount <- revenue", + " ├── grants copied from the old table", + " └── old table archived", + ), + ), + OldNameRowsTestCase( + description="resumed after the archive without grants support", + action="view_only", + column_aliases=(), + grants_supported=False, + grants_copied=None, + reason=None, + expected_rows=( + " old name prod.revenue", + " ├── view until 2026-10-28", + " └── old table already archived", + ), + ), + OldNameRowsTestCase( + description="live view reports the grants it copied", + action="live", + column_aliases=(), + grants_supported=True, + grants_copied=2, + reason=None, + expected_rows=( + " old name prod.revenue", + " ├── view live until 2026-10-28", + " ├── grants 2 copied from the old table", + " └── old table archived", + ), + ), + OldNameRowsTestCase( + description="disabled view is left for janitor with its reason", + action="none", + column_aliases=(), + grants_supported=True, + grants_copied=None, + reason="name reused by model:revenue", + expected_rows=( + " old name prod.revenue", + " └── left for janitor name reused by model:revenue", + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_old_name_step_when_rendering_then_each_fact_has_its_own_row( + test_case: OldNameRowsTestCase, +) -> None: + rows: list[str] = old_name_rows( + entry=old_name_entry( + action=test_case.action, + column_aliases=test_case.column_aliases, + grants_supported=test_case.grants_supported, + grants_copied=test_case.grants_copied, + reason=test_case.reason, + ), + style=CliStyle(use_color=False), + ) + + assert tuple(rows) == test_case.expected_rows diff --git a/tests/unit/src/sqlbuild/compiler/planner/main/pre_build/test_pre_build_work_display.py b/tests/unit/src/sqlbuild/compiler/planner/main/pre_build/test_pre_build_work_display.py index 1e94d810b9..583a7ca25a 100644 --- a/tests/unit/src/sqlbuild/compiler/planner/main/pre_build/test_pre_build_work_display.py +++ b/tests/unit/src/sqlbuild/compiler/planner/main/pre_build/test_pre_build_work_display.py @@ -99,14 +99,15 @@ expected_fragments=(f"{_BOLD_RED}origin missing{_RESET}",), ), MigrationDecisionStyleTestCase( - description="renamed hands over identity with dim origin and note", + description="renamed table is listed as a bold blue migration that rebuilds it", decision=MigrationDecision.RENAMED, compatibility=MigrationCompatibility.NOT_CHECKED, findings=(), expected_fragments=( - f"\033[1mRenamed{_RESET} {_DIM}(1){_RESET}", - f" daily_revenue {_DIM}prod.revenue ->{_RESET} prod.daily_revenue " - f"{_DIM}(identity handed over){_RESET}", + f"\033[1mMigrations{_RESET} {_DIM}(1){_RESET}", + f" daily_revenue {_BOLD_BLUE}migrate{_RESET} {_DIM}prod.revenue ->{_RESET} " + "prod.daily_revenue", + f" {_DIM}transfer{_RESET} rebuild (table)", ), ), ), diff --git a/tests/unit/src/sqlbuild/executor/migrations/_helpers/_test_types.py b/tests/unit/src/sqlbuild/executor/migrations/_helpers/_test_types.py index 43a93c0c51..16e5152903 100644 --- a/tests/unit/src/sqlbuild/executor/migrations/_helpers/_test_types.py +++ b/tests/unit/src/sqlbuild/executor/migrations/_helpers/_test_types.py @@ -4,6 +4,9 @@ from dataclasses import dataclass +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapter.contract.models import RelationGrant + @dataclass(frozen=True) class OverlappingColumnRenameTestCase: @@ -11,3 +14,38 @@ class OverlappingColumnRenameTestCase: transactional: bool expected_columns: tuple[str, ...] expected_decisions: tuple[str, ...] + + +@dataclass(frozen=True) +class ComposeColumnAliasesTestCase: + description: str + renames: tuple[tuple[str, str], ...] + expected_aliases: tuple[tuple[str, str], ...] + + +@dataclass(frozen=True) +class AdapterOldNameSqlTestCase: + description: str + adapter: BaseAdapter + database: str | None + column_aliases: tuple[tuple[str, str], ...] + expected_statements: tuple[str, ...] + expected_state_table_prefix: str + expected_transactional: bool + + +@dataclass(frozen=True) +class ReaderAccessWarningTestCase: + description: str + adapter: BaseAdapter + grants: tuple[RelationGrant, ...] + expected_warnings: tuple[str, ...] + + +@dataclass(frozen=True) +class GrantReconcileTestCase: + description: str + adapter: BaseAdapter + current: tuple[RelationGrant, ...] + target: tuple[RelationGrant, ...] + expected_statements: tuple[str, ...] diff --git a/tests/unit/src/sqlbuild/executor/migrations/_helpers/helpers.py b/tests/unit/src/sqlbuild/executor/migrations/_helpers/helpers.py index a9c27990b7..b4b5e1776a 100644 --- a/tests/unit/src/sqlbuild/executor/migrations/_helpers/helpers.py +++ b/tests/unit/src/sqlbuild/executor/migrations/_helpers/helpers.py @@ -3,14 +3,24 @@ from __future__ import annotations from collections.abc import Callable +from datetime import UTC, datetime, timedelta from typing import Any import duckdb +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter from sqlbuild.adapters.duckdb.classes.duckdb_adapter import DuckDbAdapter from sqlbuild.compiler.compile.models import CompiledRelationLocation -from sqlbuild.compiler.migrations.types import ColumnMigrationDecision, MigrationDiscovery -from sqlbuild.compiler.planner.models import ColumnMigrationPlanEntry +from sqlbuild.compiler.migrations.main.deterministic_column_event_id import ( + deterministic_column_migration_event_id, +) +from sqlbuild.compiler.migrations.models import ColumnMigrationEvent, MigrationRelation +from sqlbuild.compiler.migrations.types import ( + ColumnMigrationDecision, + MigrationDiscovery, + OldNameViewAction, +) +from sqlbuild.compiler.planner.models import ColumnMigrationPlanEntry, OldNameViewPlanEntry RELATION: str = "main.fct_orders" @@ -65,3 +75,59 @@ def live_column_names(connection: duckdb.DuckDBPyConnection) -> tuple[str, ...]: "WHERE table_name = 'fct_orders' ORDER BY ordinal_position" ).fetchall() ) + + +_RENAMES_STARTED_AT: datetime = datetime(2026, 1, 1, tzinfo=UTC) +_RENAMED_RELATION: MigrationRelation = MigrationRelation( + database=None, schema="analytics", name="daily_revenue" +) + + +def column_rename(*, origin: str, destination: str, day: int) -> ColumnMigrationEvent: + """Return one manual column rename on analytics.daily_revenue.""" + + return ColumnMigrationEvent( + event_id=deterministic_column_migration_event_id( + run_id=f"run-{day}", + target_name=None, + relation=_RENAMED_RELATION, + origin_column=origin, + destination_column=destination, + ), + target_name=None, + model_name="daily_revenue", + relation=_RENAMED_RELATION, + origin_column=origin, + destination_column=destination, + discovery=MigrationDiscovery.MANUAL, + decision=ColumnMigrationDecision.RENAME, + run_id=f"run-{day}", + created_at=_RENAMES_STARTED_AT + timedelta(days=day), + ) + + +def adapter_location( + *, adapter: BaseAdapter, database: str | None, name: str +) -> CompiledRelationLocation: + """Return an analytics relation rendered the way the adapter qualifies it.""" + + return CompiledRelationLocation( + database=database, + schema="analytics", + name=name, + qualified_name=adapter.render_qualified_name( + database=database, schema="analytics", name=name + ), + ) + + +def old_name_plan_entry(*, adapter: BaseAdapter) -> OldNameViewPlanEntry: + """Return a planned move of analytics.revenue to analytics.daily_revenue.""" + + return OldNameViewPlanEntry( + model_name="daily_revenue", + origin=adapter_location(adapter=adapter, database=None, name="revenue"), + destination=adapter_location(adapter=adapter, database=None, name="daily_revenue"), + action=OldNameViewAction.ARCHIVE_AND_VIEW, + target_name="prod", + ) diff --git a/tests/unit/src/sqlbuild/executor/migrations/_helpers/test_old_name_views.py b/tests/unit/src/sqlbuild/executor/migrations/_helpers/test_old_name_views.py new file mode 100644 index 0000000000..0be0493ac7 --- /dev/null +++ b/tests/unit/src/sqlbuild/executor/migrations/_helpers/test_old_name_views.py @@ -0,0 +1,300 @@ +"""Unit coverage for compatibility view SQL and column alias composition.""" + +from __future__ import annotations + +from typing import Any + +import pytest + +from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter +from sqlbuild.adapter.contract.models import ColumnInfo, RelationGrant +from sqlbuild.adapters.bigquery.classes.bigquery_adapter import BigQueryAdapter +from sqlbuild.adapters.databricks.classes.databricks_adapter import DatabricksAdapter +from sqlbuild.adapters.postgres.classes.postgres_adapter import PostgresAdapter +from sqlbuild.adapters.snowflake.classes.snowflake_adapter import SnowflakeAdapter +from sqlbuild.adapters.sqlserver.classes.sqlserver_adapter import SqlServerAdapter +from sqlbuild.compiler.migrations.models import ColumnMigrationEvent +from sqlbuild.executor.migrations._helpers.old_name_views import ( + compose_column_aliases, + reader_access_warning, + reconcile_view_grants, + render_old_name_view_select, +) +from sqlbuild.executor.migrations.models import OldNameViewSource +from tests.unit.src.sqlbuild.executor.migrations._helpers._test_types import ( + AdapterOldNameSqlTestCase, + ComposeColumnAliasesTestCase, + GrantReconcileTestCase, + ReaderAccessWarningTestCase, +) +from tests.unit.src.sqlbuild.executor.migrations._helpers.helpers import ( + adapter_location, + column_rename, + old_name_plan_entry, +) + +_ANALYSTS: RelationGrant = RelationGrant(privilege="SELECT", grantee="analysts") +_FUTURE: RelationGrant = RelationGrant(privilege="SELECT", grantee="reporting") + + +@pytest.mark.parametrize( + "test_case", + [ + ComposeColumnAliasesTestCase( + description="one rename aliases the new column back", + renames=(("amount", "revenue"),), + expected_aliases=(("amount", "revenue"),), + ), + ComposeColumnAliasesTestCase( + description="chained renames alias the first name to the latest", + renames=(("amount", "revenue"), ("revenue", "net_revenue")), + expected_aliases=(("amount", "net_revenue"),), + ), + ComposeColumnAliasesTestCase( + description="a rename back to the original name needs no alias", + renames=(("amount", "revenue"), ("revenue", "amount")), + expected_aliases=(), + ), + ], + ids=lambda case: case.description, +) +def test_given_recorded_column_renames_when_composing_then_old_names_map_to_current( + test_case: ComposeColumnAliasesTestCase, +) -> None: + events: tuple[ColumnMigrationEvent, ...] = tuple( + column_rename(origin=origin, destination=destination, day=day) + for day, (origin, destination) in enumerate(test_case.renames, start=1) + ) + + assert compose_column_aliases(events) == test_case.expected_aliases + + +@pytest.mark.parametrize( + "test_case", + [ + AdapterOldNameSqlTestCase( + description="snowflake view without aliases selects every column", + adapter=SnowflakeAdapter(), + database=None, + column_aliases=(), + expected_statements=( + "CREATE OR REPLACE VIEW analytics.revenue AS SELECT * FROM analytics.daily_revenue", + ), + expected_state_table_prefix="CREATE TRANSIENT TABLE IF NOT EXISTS", + expected_transactional=False, + ), + AdapterOldNameSqlTestCase( + description="snowflake aliased column keeps its exact name and exposes the old one", + adapter=SnowflakeAdapter(), + database=None, + column_aliases=(("amount", "revenue"),), + expected_statements=( + "CREATE OR REPLACE VIEW analytics.revenue AS " + 'SELECT "ORDER_ID", "REVENUE" AS "AMOUNT" FROM analytics.daily_revenue', + ), + expected_state_table_prefix="CREATE TRANSIENT TABLE IF NOT EXISTS", + expected_transactional=False, + ), + AdapterOldNameSqlTestCase( + description="bigquery view reads the new table by name", + adapter=BigQueryAdapter(), + database="orders-project", + column_aliases=(("amount", "revenue"),), + expected_statements=( + "CREATE OR REPLACE VIEW `orders-project.analytics.revenue` AS SELECT `ORDER_ID`, " + "`REVENUE` AS `amount` FROM `orders-project.analytics.daily_revenue`", + ), + expected_state_table_prefix=( + "CREATE TABLE IF NOT EXISTS `orders-project.analytics._sqlbuild_old_name_views`" + ), + expected_transactional=False, + ), + AdapterOldNameSqlTestCase( + description="databricks view reads the new table by name", + adapter=DatabricksAdapter(), + database="main", + column_aliases=(("amount", "revenue"),), + expected_statements=( + "CREATE OR REPLACE VIEW `main`.`analytics`.`revenue` AS SELECT `ORDER_ID`, " + "`REVENUE` AS `amount` FROM `main`.`analytics`.`daily_revenue`", + ), + expected_state_table_prefix=( + "CREATE TABLE IF NOT EXISTS `main`.`analytics`.`_sqlbuild_old_name_views`" + ), + expected_transactional=False, + ), + AdapterOldNameSqlTestCase( + description="sql server re-creates the view inside a transaction", + adapter=SqlServerAdapter(), + database=None, + column_aliases=(("amount", "revenue"),), + expected_statements=( + "DROP VIEW IF EXISTS analytics.revenue", + "CREATE VIEW analytics.revenue AS SELECT [ORDER_ID], [REVENUE] AS [amount] " + "FROM analytics.daily_revenue", + ), + expected_state_table_prefix="IF NOT EXISTS (SELECT 1 FROM information_schema.tables", + expected_transactional=True, + ), + ], + ids=lambda case: case.description, +) +def test_given_adapter_compatibility_view_when_rendering_then_sql_reads_new_relation_by_name( + test_case: AdapterOldNameSqlTestCase, monkeypatch: pytest.MonkeyPatch +) -> None: + adapter: BaseAdapter = test_case.adapter + + def columns(**_: Any) -> tuple[ColumnInfo, ...]: + return ( + ColumnInfo(name="ORDER_ID", type="NUMBER"), + ColumnInfo(name="REVENUE", type="NUMBER"), + ) + + monkeypatch.setattr(adapter, "get_columns", columns) + source: OldNameViewSource = OldNameViewSource( + old=adapter_location(adapter=adapter, database=test_case.database, name="revenue"), + new=adapter_location(adapter=adapter, database=test_case.database, name="daily_revenue"), + column_aliases=test_case.column_aliases, + ) + + select_sql: str = render_old_name_view_select(adapter=adapter, connection=None, source=source) + statements: tuple[str, ...] = adapter.render_create_view_as( + destination=source.old.qualified_name or "", sql=select_sql + ) + + assert statements == test_case.expected_statements + assert not adapter.views_bind_to_relation_identity() + assert adapter.supports_transactional_ddl() == test_case.expected_transactional + assert adapter.render_create_old_name_view_state_table_sql( + database=test_case.database, schema="analytics" + ).startswith(test_case.expected_state_table_prefix) + + +@pytest.mark.parametrize( + "test_case", + [ + ReaderAccessWarningTestCase( + description="bigquery names principals that also need the destination table", + adapter=BigQueryAdapter(), + grants=( + RelationGrant(privilege="roles/bigquery.dataViewer", grantee="group:b@example.com"), + RelationGrant(privilege="roles/bigquery.dataViewer", grantee="group:a@example.com"), + ), + expected_warnings=( + "M118: `analytics.revenue` now reads `analytics.daily_revenue` with each " + "reader's own access. group:a@example.com, group:b@example.com kept their " + "access to the old name but also need read access on " + "`analytics.daily_revenue`; SQLBuild does not grant it", + ), + ), + ReaderAccessWarningTestCase( + description="bigquery without copied grants has nothing to warn about", + adapter=BigQueryAdapter(), + grants=(), + expected_warnings=(), + ), + ReaderAccessWarningTestCase( + description="postgres views read with the owner's access", + adapter=PostgresAdapter(), + grants=(RelationGrant(privilege="SELECT", grantee="orders_reader"),), + expected_warnings=(), + ), + ], + ids=lambda case: case.description, +) +def test_given_copied_grants_when_views_read_with_reader_access_then_m118_names_principals( + test_case: ReaderAccessWarningTestCase, +) -> None: + warnings: tuple[str, ...] = reader_access_warning( + adapter=test_case.adapter, + entry=old_name_plan_entry(adapter=test_case.adapter), + grants=test_case.grants, + ) + + assert warnings == test_case.expected_warnings + + +@pytest.mark.parametrize( + "test_case", + [ + GrantReconcileTestCase( + description="postgres revokes a default privilege the re-create added", + adapter=PostgresAdapter(), + current=(_ANALYSTS, _FUTURE), + target=(_ANALYSTS,), + expected_statements=('REVOKE SELECT ON analytics.revenue FROM "reporting"',), + ), + GrantReconcileTestCase( + description="postgres grants what the drop removed, on exposed columns only", + adapter=PostgresAdapter(), + current=(), + target=( + _ANALYSTS, + RelationGrant(privilege="SELECT", grantee="leads", column="amount"), + RelationGrant(privilege="SELECT", grantee="leads", column="discount"), + ), + expected_statements=( + 'GRANT SELECT ON analytics.revenue TO "analysts"', + 'GRANT SELECT ("amount") ON analytics.revenue TO "leads"', + ), + ), + GrantReconcileTestCase( + description="postgres replaces a grant whose grant option changed", + adapter=PostgresAdapter(), + current=(RelationGrant(privilege="SELECT", grantee="analysts", grantable=True),), + target=(_ANALYSTS,), + expected_statements=( + 'REVOKE SELECT ON analytics.revenue FROM "analysts"', + 'GRANT SELECT ON analytics.revenue TO "analysts"', + ), + ), + GrantReconcileTestCase( + description="snowflake revokes a future grant copy grants reapplied", + adapter=SnowflakeAdapter(), + current=(RelationGrant(privilege="SELECT", grantee="REPORTING", grantee_kind="ROLE"),), + target=(), + expected_statements=('REVOKE SELECT ON VIEW analytics.revenue FROM ROLE "REPORTING"',), + ), + GrantReconcileTestCase( + description="unchanged privileges issue nothing", + adapter=PostgresAdapter(), + current=(_ANALYSTS,), + target=(_ANALYSTS,), + expected_statements=(), + ), + ], + ids=lambda case: case.description, +) +def test_given_view_grants_after_refresh_when_reconciling_then_they_equal_the_grants_before( + test_case: GrantReconcileTestCase, monkeypatch: pytest.MonkeyPatch +) -> None: + adapter: BaseAdapter = test_case.adapter + executed: list[str] = [] + + def columns(**_: Any) -> tuple[ColumnInfo, ...]: + return ( + ColumnInfo(name="order_id", type="INTEGER"), + ColumnInfo(name="amount", type="INTEGER"), + ) + + def current(**_: Any) -> tuple[RelationGrant, ...]: + return test_case.current + + def execute(*, connection: Any, sql: str) -> None: + del connection + executed.append(sql) + + monkeypatch.setattr(adapter, "get_columns", columns) + monkeypatch.setattr(adapter, "read_relation_grants", current) + monkeypatch.setattr(adapter, "execute", execute) + source: OldNameViewSource = OldNameViewSource( + old=adapter_location(adapter=adapter, database=None, name="revenue"), + new=adapter_location(adapter=adapter, database=None, name="daily_revenue"), + column_aliases=(), + ) + + _ = reconcile_view_grants( + adapter=adapter, connection=None, source=source, target=test_case.target + ) + + assert tuple(executed) == test_case.expected_statements diff --git a/website/src/content/docs/docs/cli/janitor.mdx b/website/src/content/docs/docs/cli/janitor.mdx index ab9a63931a..55bdb6d92f 100644 --- a/website/src/content/docs/docs/cli/janitor.mdx +++ b/website/src/content/docs/docs/cli/janitor.mdx @@ -20,6 +20,7 @@ sqb --project-dir janitor [flags] | `--auto-approve` | Skip the confirmation prompt and run the planned actions immediately | | `--retention-days` | Override the configured retention period (days) before a stale relation is archived | | `--direct-state-history-versions` | Override how many state-history versions are kept per identity | +| `--drop-old-name-view ` | Drop the [compatibility view](/docs/concepts/models/migrations#old-names) at a renamed model's old name before it expires. Repeat for several views. Fails with `C503` if no compatibility view is recorded at the name | ## Lifecycle @@ -44,6 +45,20 @@ Long names are shortened with a short hash so they fit the adapter's identifier Setting both `retention_days` and `archive_retention_days` to `0` archives and deletes stale relations in the same run. +## Compatibility views + +When a model is renamed, SQLBuild keeps a [compatibility view](/docs/concepts/models/migrations#old-names) at its old name. The janitor lists each one with the model it reads and its expiry: + +``` +Old name views (1) +└── prod.revenue -> model:daily_revenue + └── expires 2026-10-28 (live; drop early with --drop-old-name-view prod.revenue) +``` + +A compatibility view is never archived as a stale relation. Once it expires, or when it is named with `--drop-old-name-view`, the janitor drops it and records the drop in `_sqlbuild_old_name_views`. Before dropping, the janitor checks that the relation at the old name is still the view SQLBuild created there, by comparing its stored definition with the view's SQL. Any other relation at that name, such as a view a project model built after an interrupted early drop, is left alone: the janitor records the compatibility view as dropped and lists it as `record dropped (name now used by another relation, model:revenue)`, naming the model when the name is a project destination. A compatibility view that no longer exists is recorded as dropped too. + +Run the same SQLBuild version for `sqb janitor` as for builds. An older janitor does not know compatibility views and archives them as stale relations. + ## Relation age `retention_days` is measured from the last time a relation was written, as reported by the warehouse. For a relation the project no longer builds, that is roughly when it stopped being built. diff --git a/website/src/content/docs/docs/concepts/models/column-migrations.mdx b/website/src/content/docs/docs/concepts/models/column-migrations.mdx index 921e8f8d4d..7cc03397f1 100644 --- a/website/src/content/docs/docs/concepts/models/column-migrations.mdx +++ b/website/src/content/docs/docs/concepts/models/column-migrations.mdx @@ -34,7 +34,7 @@ After the rename is recorded, `migrate_from` has no further effect, and SQLBuild A declaration is rejected when compiling if: -- the model is not incremental or snapshot; +- the model is not incremental or snapshot, unless it is a table or view model that declares its own `migrate_from`; there the column's `migrate_from` only names the old column in the [compatibility view](/docs/concepts/models/migrations#old-names) at the model's old name; - `migrate_from` names the column itself, or a qualified name instead of one column; - two columns declare the same `migrate_from`; - a declared column migrates from another column that itself declares `migrate_from`. Chains and swaps can't be renamed in place safely; rename through a name no other column uses, or rebuild with `--full-refresh`. diff --git a/website/src/content/docs/docs/concepts/models/configuration.mdx b/website/src/content/docs/docs/concepts/models/configuration.mdx index 7ad3f1895f..d87b8ae539 100644 --- a/website/src/content/docs/docs/concepts/models/configuration.mdx +++ b/website/src/content/docs/docs/concepts/models/configuration.mdx @@ -10,7 +10,7 @@ description: "MODEL() header fields and SQL-analysis controls." | `materialized` | `view`, `table`, `incremental`, `snapshot`, or a custom materialization name | | `tags` | Tags used by selectors | | `description` | Human-readable model description | -| `columns` | Model-local column declarations, inherited-column audit augmentation, or column renames with `migrate_from` (incremental and snapshot models; see [Column migrations](/docs/concepts/models/column-migrations)) | +| `columns` | Model-local column declarations, inherited-column audit augmentation, or column renames with `migrate_from` (incremental and snapshot models; see [Column migrations](/docs/concepts/models/column-migrations)). On a table or view model that declares its own `migrate_from`, a column `migrate_from` only names the old column in the [compatibility view](/docs/concepts/models/migrations#old-names) | | `model_schema` | Reusable column schema name | | `audits` | Model-level audit instances | | `enums` | Model-local enum declarations; names must begin with `_` | @@ -68,8 +68,9 @@ Table promotion mode is a project setting rather than a `MODEL()` field. Staged | `full_refresh` | Optional model execution override: `false` always runs incrementally, `true` always full-refreshes, and omission follows the command | | `on_schema_change` | `append_new_columns`, `sync_all_columns`, `ignore`, or `fail` | | `replay_on_change` | `forward`, `full`, or `bounded-` | -| `migrate_from` | Old model name (or `schema.name`) whose data this model takes over; see [Model migrations](/docs/concepts/models/migrations). Also valid on snapshot models | -| `migrate_force` | `true` replaces a destination that already has its own build history during a migration | +| `migrate_from` | Old model name (or `schema.name`) whose data this model takes over; see [Model migrations](/docs/concepts/models/migrations). Also valid on snapshot, table, and view models | +| `migrate_force` | `true` replaces a destination that already has its own build history during a migration. Incremental and snapshot models only | +| `old_name_view` | How long a [compatibility view](/docs/concepts/models/migrations#old-names) keeps the old name working after a migration, such as `7d`, or `false` for none. Overrides `[migrations] old_name_views`; valid on every model | See [Incremental](/docs/concepts/incremental) for full semantics. diff --git a/website/src/content/docs/docs/concepts/models/migrations.mdx b/website/src/content/docs/docs/concepts/models/migrations.mdx index 5d955663a5..31388aec32 100644 --- a/website/src/content/docs/docs/concepts/models/migrations.mdx +++ b/website/src/content/docs/docs/concepts/models/migrations.mdx @@ -1,11 +1,11 @@ --- title: "Model migrations" -description: "Keep the history of an incremental or snapshot model when you rename it." +description: "Keep a renamed model's history, and keep its old name working for a while." --- Renaming an incremental or snapshot model normally means rebuilding it from scratch under the new name, which loses any history the source data can no longer reproduce. A model migration moves the existing relation's data to the new name instead, then continues building incrementally. -Migrations apply to incremental and snapshot models. To keep a column's history when you rename the column, see [Column migrations](/docs/concepts/models/column-migrations). +Table and view models can declare a migration too. They hold no history, so they are rebuilt under the new name, but they keep their identity for change detection. For every materialization, the old name keeps working for a while through a [compatibility view](#old-names). To keep a column's history when you rename the column, see [Column migrations](/docs/concepts/models/column-migrations). ## Declaring a migration @@ -22,7 +22,7 @@ MODEL ( Use `schema.name` when the old relation is in a different schema of the same database. -On the next build, SQLBuild copies the old relation's data to the new name through a staging table (see [How a move runs](#how-a-move-runs)), records the move, and builds the model incrementally from the migrated state. There is no first-run rebuild, and no `replay_on_change` replay caused by the rename. The old relation is never modified or dropped. Once it is no longer part of the project, [janitor](/docs/cli/janitor) archives it like any other stale relation. +On the next build, SQLBuild copies the old relation's data to the new name through a staging table (see [How a move runs](#how-a-move-runs)), records the move, and builds the model incrementally from the migrated state. There is no first-run rebuild, and no `replay_on_change` replay caused by the rename. After the model builds, the old relation is archived and a compatibility view takes its name (see [Old names](#old-names)). After the move is recorded, `migrate_from` has no further effect, and SQLBuild tells you it can be removed. Keep it until every target that needs the move has built. @@ -44,11 +44,13 @@ The origin and destination must be compatible under the model's normal `on_schem `migrate_force true` only affects the `conflict` case and is safe to leave in the header. +Table and view models only ever see `migrate`, `done`, and `origin missing`: nothing is replaced, so there is no conflict. `migrate_force` on a table or view model is a compile error. In `sqb plan --json`, their decision is `renamed`. + ## Automatic discovery If a selected model has never been built, SQLBuild compares it with models that were removed from the project but whose relations still exist. When exactly one removed model has equivalent logic, it is migrated automatically. Equivalent means the same query and configuration, ignoring the model's own name, CTE and table alias names, comments, formatting, and storage-only settings. Renamed upstream models are matched first, so a renamed chain of models is migrated together. Removed models whose data was already moved on by a recorded migration are not candidates, so a model renamed several times matches its latest table. -Renamed tables and views are matched the same way. They are still rebuilt under the new name, since they hold no history, but they keep their identity for change detection, so incremental models downstream of a renamed view are not replayed or rebuilt. These renames are recorded as `renamed` events. +Renamed tables and views are matched the same way. They are rebuilt under the new name, since they hold no history, but they keep their identity for change detection, so incremental models downstream of a renamed view are not replayed or rebuilt. These renames are recorded as `renamed` events. Automatic discovery never guesses. If a match is ambiguous, SQLBuild warns (`M107`) and builds from scratch; declare `migrate_from` to choose. An explicit `migrate_from` always wins. Automatic discovery covers renames within the project's schemas; use `migrate_from` to move a model to another schema. Relations last built before this feature carry no stored fingerprint and aren't matched automatically. @@ -60,15 +62,104 @@ A move never overwrites the destination directly: 2. The staging table is checked: it must exist, have the expected columns and table type, and, for a physical copy, the same row count as the old relation. 3. The staging table is swapped in as the destination. If a destination already existed, for example after `superseded replace` or `forced replace`, it is kept as `_sqb_archive____migration_previous__` rather than dropped. 4. The move is recorded, then the model builds as usual. +5. After the model builds successfully, the old relation is archived and a compatibility view is created at its name (see [Old names](#old-names)). -The old relation is never modified. Staging and replaced tables use the [janitor](/docs/cli/janitor) archive format, so an abandoned staging table or a replaced destination is deleted by janitor after `archive_retention_days`. A staging table left by an interrupted run is never reused; the next run stages again. Don't run janitor at the same time as a build in the same schema. +Until step 5, the old relation is not modified. Staging and replaced tables use the [janitor](/docs/cli/janitor) archive format, so an abandoned staging table or a replaced destination is deleted by janitor after `archive_retention_days`. A staging table left by an interrupted run is never reused; the next run stages again. Don't run janitor at the same time as a build in the same schema. `sqb plan` shows how each move will run, for example `physical copy, transient -> permanent, promote by swap`. +## Old names + +A rename breaks every query outside the project that still reads the old name: dashboards, notebooks, other teams' jobs. After the renamed model builds successfully, SQLBuild keeps the old name working for a while: + +1. The old relation is renamed to `_sqb_archive____migration_origin__`. +2. A view is created at the old name that selects from the new relation. +3. The view gets the privileges that were granted on the old relation, read from the archive. Only existing grants are copied; nothing new is granted. + +On DuckDB, MotherDuck, PostgreSQL, and SQL Server these steps and their records commit in one transaction. On Snowflake, BigQuery, and Databricks each step is recorded after it runs, and a re-run resumes from the first unrecorded step without repeating one. A failure is reported as `M117`; re-run the build. + +| Adapter | Privileges copied to the view | +|---------|-------------------------------| +| Snowflake | `SHOW GRANTS` on the archive, replayed to roles and database roles, keeping `WITH GRANT OPTION`. `OWNERSHIP`, grants to shares, and privileges that don't apply to views are skipped | +| PostgreSQL | Table and column privileges from the catalog, including grants to `PUBLIC` and `WITH GRANT OPTION`. The owner's own privileges are not copied | +| BigQuery | Table IAM role bindings from `INFORMATION_SCHEMA.OBJECT_PRIVILEGES`, replayed with `GRANT`. See the note below | +| Databricks | Unity Catalog grants on the table itself that apply to views, such as `SELECT`. `OWN`, `MODIFY`, and inherited catalog or schema grants are skipped | +| SQL Server | Object and column permissions, including `DENY` and `WITH GRANT OPTION`, that apply to views | +| DuckDB, MotherDuck | No object privileges exist | + +Column privileges are copied to the view's column of the same name, which is the old name when the view aliases a renamed column. A column privilege on a column the view no longer exposes is not copied; the view shows nothing of that column. + +The copied grants are recorded on the view's creation. After that, the view's own privileges are what counts: when SQLBuild redefines the view after a rebuild, the view ends up with exactly the privileges it had just before, so a privilege you revoke from the view stays revoked and one you grant on the view is kept. SQLBuild uses `CREATE OR REPLACE VIEW` on PostgreSQL, `COPY GRANTS` on Snowflake, and `ALTER VIEW` on SQL Server and Databricks. + +Creating or replacing a view can also apply privileges you did not give it: PostgreSQL applies `ALTER DEFAULT PRIVILEGES` when it re-creates a view, and Snowflake applies the schema's future grants for views on `CREATE OR REPLACE VIEW … COPY GRANTS`. SQLBuild reads the view's privileges before every create or redefinition and reconciles afterwards: it grants what went missing and revokes what was added. A new compatibility view likewise ends up with only the privileges copied from the old relation. Ownership and the owner's own privileges are never changed. + +A recorded view that no longer exists at the old name is not re-created by a build; [janitor](/docs/cli/janitor) records it as dropped. + +On Snowflake, BigQuery, Databricks, DuckDB, and MotherDuck, views read their tables by name when queried, so a build redefines a compatibility view only when its SQL would change, for example when the view lists columns and the new relation's columns changed. PostgreSQL views are bound to the table itself, so they are redefined whenever the table is replaced. + +BigQuery cannot rename views, so an old view model is re-created under the archive name with its IAM bindings, and then dropped. + +On BigQuery, a view reads its tables with the permissions of whoever queries it. A principal that could read the old table keeps access to the old name through the copied bindings, but also needs read access on the new table. SQLBuild does not grant that and does not create authorized views; the build warns with `M118` and names the principals. + +If columns were renamed, the view presents them under their old names, for example `SELECT order_id, revenue AS amount FROM analytics.daily_revenue`. Incremental and snapshot models alias every recorded [column migration](/docs/concepts/models/column-migrations). A table or view model that declares `migrate_from` can declare old column names the same way, as `revenue (migrate_from amount)`; this only aliases the column in the compatibility view, since the table is rebuilt with the new name anyway. The view follows the new relation's current columns after every build of the model. + +`sqb plan` shows what happens at the old name: + +``` +Migrations (1) +└── daily_revenue migrate analytics.revenue -> analytics.daily_revenue + ├── transfer rebuild (table) + └── old name analytics.revenue + ├── view until 2026-10-28 + ├── columns amount <- revenue + ├── grants copied from the old table + └── old table archived +``` + +A move resumed after the archive shows `old table already archived`. Once the view exists it shows `view live until 2026-10-28` and how many grants were copied. When no view is kept, the block shows `left for janitor` and the reason, such as `old_name_view false` or `name reused by model:revenue`. + +In `sqb plan --json`, each migration has an `old_name` object with `action` (`archive_and_view`, `view_only`, `live`, or `none`), `view`, `reads`, `expires_at`, `column_aliases`, `grants_copied` (the number of grants copied once the view exists, otherwise `null`), and `reason`. The top-level `old_names` list also includes steps that resume an earlier move. + +The view expires after 30 days by default. After that, [janitor](/docs/cli/janitor) drops it; `sqb janitor --drop-old-name-view analytics.revenue` drops it earlier. Change the default for the project, or turn the views off with `false`: + +```toml +[migrations] +old_name_views = "7d" +``` + +A model header overrides the project setting with `old_name_view 7d` or `old_name_view false`. The retention of a move is fixed when the move is recorded. + +With views off, or when a view can't be kept, the old relation is left in place and janitor archives it once it is no longer part of the project, as before. No view is created when: + +- the old relation was not built by SQLBuild; +- another model in the project builds at the old name; +- the old name is itself a compatibility view of an earlier rename. + +Moves recorded before a SQLBuild version that supports old-name views never gain a view later. + +### Code inside the project + +The compatibility view is for consumers outside the project. Project code must use the new model: + +- Python nodes, hooks, and loaders whose literal SQL names the old name fail with `P008`, at compile time for a declared `migrate_from` and at plan time for an automatically discovered rename. SQL models reading it through `__ref` are unaffected, since `__ref` resolves the new name. +- A new model whose name is a live compatibility view stops the build with `M114`, which names the view and its expiry. Drop the view early with janitor, or choose another name. + +### PostgreSQL views + +PostgreSQL views are bound to the table they read, not its name. When a build replaces the renamed model's table, or drops or retypes its columns, SQLBuild points its compatibility views at the new table in the same transaction, so they are never missing, even if the build is interrupted. A view is replaced in place when its columns allow it, so views you build on top of the compatibility view keep working; otherwise it is dropped and re-created with the privileges it had, which fails if another view depends on it. + +Views outside SQLBuild that read the old relation follow it into the archive and keep reading its old data; the build warns with `M115` and names them. Re-create them against the old name or the new relation. On the other adapters, views read relations by name, so no rebinding is needed. + +### Upgrading + +Upgrade the SQLBuild version that runs `sqb janitor` together with the one that runs builds. An older janitor does not know compatibility views and archives them as stale relations. + ## State Each move is recorded in an append-only `_sqlbuild_migrations` table in the destination schema. A missing record means completion is unrecorded; staging or promotion may already have happened. Re-running after an interruption recovers through the migration decisions above. +What happens at the old name is recorded in an append-only `_sqlbuild_old_name_views` table next to it, one row per step: `required` (with the move), `origin_archived`, `view_created`, and `view_dropped`. Rows are never updated. + ## Adapter support | Adapter | Staging | Promotion | diff --git a/website/src/content/docs/docs/concepts/project-configuration.mdx b/website/src/content/docs/docs/concepts/project-configuration.mdx index 5e33c5bbc1..4af4b22801 100644 --- a/website/src/content/docs/docs/concepts/project-configuration.mdx +++ b/website/src/content/docs/docs/concepts/project-configuration.mdx @@ -480,6 +480,19 @@ exclude_patterns = ["audit_*", "tmp_*"] See [janitor](/docs/cli/janitor) for the archive and delete lifecycle. +## Migrations + +Defaults for [model migrations](/docs/concepts/models/migrations): + +```toml +[migrations] +old_name_views = "30d" +``` + +| Field | Default | Description | +|-------|---------|-------------| +| `old_name_views` | `"30d"` | How long a renamed model's old name keeps working through a [compatibility view](/docs/concepts/models/migrations#old-names) before janitor drops it. `false` keeps the old relation in place instead, as before. A model's `old_name_view` header overrides it. | + ## Scenario Configuration for scenario run isolation, snapshot capture safety limits, and local type overrides: diff --git a/website/src/content/docs/docs/concepts/python-nodes/sql-references.mdx b/website/src/content/docs/docs/concepts/python-nodes/sql-references.mdx index 16d94f7902..7c31978f8e 100644 --- a/website/src/content/docs/docs/concepts/python-nodes/sql-references.mdx +++ b/website/src/content/docs/docs/concepts/python-nodes/sql-references.mdx @@ -83,6 +83,12 @@ Loaders run before models and seeds, so a loader naming a model or seed fails wi cannot read them. A loader naming a source is told to use `ctx.source("name")`, or `ctx.loader(upstream_loader)` when the source is populated by a loader it depends on. +The old name of a [renamed model](/docs/concepts/models/migrations#old-names) counts as a project +relation while a compatibility view is kept there, so literal SQL still reading it fails with +`P008` and names the new model. A declared `migrate_from` is checked at compile time; an +automatically discovered rename is checked when planning, and stops the build. A hook's own model +does not exempt its old name. + **At run time**, SQL built dynamically cannot be checked earlier. Before sending it, `ctx.query()` and `ctx.execute_sql()` parse it in the adapter's dialect, resolve unqualified names against the current schema, and compare them with the project relations resolved for the run. A