Skip to content

Commit 6701700

Browse files
authored
fix: use the current declaration layout in the playground (#820)
1 parent 7e067f4 commit 6701700

11 files changed

Lines changed: 205 additions & 23 deletions

File tree

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in
3636
- [Schemas](concepts/models/schemas.md) (`concepts/models/schemas`) - Declare model columns inline or reuse canonical inherited schemas.
3737
- [Type Enforcement](concepts/models/type-enforcement.md) (`concepts/models/type-enforcement`) - Understand declared model types, static checks, and runtime casting.
3838
- [Contracts](concepts/models/contracts.md) (`concepts/models/contracts`) - Validate required or exact model output schemas.
39+
- [Model migrations](concepts/models/migrations.md) (`concepts/models/migrations`) - Keep the history of an incremental or snapshot model when you rename it.
3940
- [Hooks](concepts/models/hooks.md) (`concepts/models/hooks`) - Run SQL or Python lifecycle hooks around model materialization.
4041
- [SQL Hooks](concepts/models/hooks/sql.md) (`concepts/models/hooks/sql`) - Define, parameterize, compile, and invoke reusable or inline SQL lifecycle hooks.
4142
- [Python Hooks](concepts/models/hooks/python.md) (`concepts/models/hooks/python`) - Define Python lifecycle hooks with runtime context, providers, SQL access, and skips.
@@ -137,7 +138,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in
137138
- [dag](cli/dag.md) (`cli/dag`) - Generate the static DAG artifact for Dagster and other integrations.
138139
- [query](cli/query.md) (`cli/query`) - Run ad hoc SQL queries against the project database.
139140
- [debug](cli/debug.md) (`cli/debug`) - Validate project configuration and test the warehouse connection.
140-
- [janitor](cli/janitor.md) (`cli/janitor`) - Clean up stale warehouse relations.
141+
- [janitor](cli/janitor.md) (`cli/janitor`) - Archive and then delete stale warehouse relations.
141142
- [clean](cli/clean.md) (`cli/clean`) - Remove compiled artifacts from the target directory.
142143
- [dbt](cli/dbt.md) (`cli/dbt`) - Coordinate dbt and SQLBuild projects.
143144
- [state](cli/state.md) (`cli/state`) - Manage the virtual mode state store.

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md‎

Lines changed: 82 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,27 @@
22

33
# janitor
44

5-
> Clean up stale warehouse relations.
5+
> Archive and then delete stale warehouse relations.
66
77
Online: https://docs.sqlbuild.com/cli/janitor
88

9+
## Contents
10+
11+
- Usage
12+
- Flags
13+
- Direct-mode lifecycle
14+
- Relation age
15+
- Audit record
16+
- Restoring an archive
17+
- Configuration
18+
- Examples
19+
- State history pruning
20+
- Virtual environments
21+
- Safety
22+
923
# sqb janitor
1024

11-
Identifies and removes stale warehouse relations that are no longer part of the project. Requires `janitor.enabled = true` in `sqlbuild_project.toml`.
25+
Finds stale warehouse relations that are no longer part of the project, archives them, and deletes archives once they expire. Requires `janitor.enabled = true` in `sqlbuild_project.toml`.
1226

1327
## Usage
1428

@@ -20,8 +34,61 @@ sqb --project-dir <path> janitor [flags]
2034

2135
| Flag | Description |
2236
|------|-------------|
23-
| `--auto-approve` | Skip the confirmation prompt and delete immediately |
24-
| `--retention-days` | Override the configured retention period (days) |
37+
| `--auto-approve` | Skip the confirmation prompt and run the planned actions immediately |
38+
| `--retention-days` | Override the configured retention period (days) before a stale relation is archived |
39+
| `--direct-state-history-versions` | Override how many state-history versions are kept per identity |
40+
41+
## Direct-mode lifecycle
42+
43+
In direct mode, the janitor never drops a live relation directly. Each stale relation goes through two stages:
44+
45+
1. **Archive.** Once a relation has been stale for `retention_days`, the janitor renames it in place, in the same schema, to:
46+
47+
```
48+
_SQB_ARCHIVE__<UTC timestamp>__<original name>
49+
```
50+
51+
For example, `stg_orders` becomes `_SQB_ARCHIVE__20260924T101500Z__stg_orders`. A rename copies no data, and keeps the relation's table type, grants and time travel.
52+
2. **Delete.** Once an archive is older than `archive_retention_days`, the janitor drops it.
53+
54+
Stale relations are identified exactly as before. A relation is a candidate only if it is in a schema the project writes to, it is not produced by the current complete project, it was tracked by SQLBuild (when `delete_tracked_only = true`), it matches no exclude pattern, and it is older than the retention period.
55+
56+
The age of an archive comes from the timestamp in its name, so deletion needs no other state. The janitor only deletes relations whose names **strictly** match the archive format and that are in a schema the project writes to. Names that only look similar are reported and never touched. The janitor also only acts on relations whose names are plain lowercase identifiers (letters, digits and underscores), and that don't collide case-insensitively with another relation in the same schema. Anything else is reported and left alone.
57+
58+
Archive names are written as unquoted lowercase identifiers, so warehouses that fold unquoted names show them in their own case. On Snowflake, for example, the archive appears as `_SQB_ARCHIVE__20260924T101500Z__STG_ORDERS`.
59+
60+
Long names are shortened with a short hash so they fit the adapter's identifier limit. The prefix and timestamp are never shortened.
61+
62+
Setting both `retention_days` and `archive_retention_days` to `0` archives and deletes stale relations in the same run.
63+
64+
## Relation age
65+
66+
`retention_days` is measured from the last time a relation was written, as reported by the warehouse. For a relation the project no longer builds, that is roughly when it stopped being built.
67+
68+
| Adapter | Age source |
69+
|---------|------------|
70+
| Snowflake | `INFORMATION_SCHEMA.TABLES` `CREATED` and `LAST_ALTERED`, which change on every write |
71+
| Databricks | `DESCRIBE DETAIL` `lastModified` for tables, and `INFORMATION_SCHEMA.TABLES` for views |
72+
| BigQuery | `__TABLES__` `creation_time` and `last_modified_time` |
73+
| PostgreSQL, SQL Server, DuckDB, MotherDuck | Not available |
74+
75+
When the warehouse can't report a relation's age, the janitor skips that relation whenever `retention_days` is greater than `0`. SQL Server is in this group because its modification date ignores inserts and updates, so a recently written table could look old.
76+
77+
Ages are only looked up for relations that are otherwise candidates, and not at all when `retention_days` is `0`.
78+
79+
## Audit record
80+
81+
Every archive and every delete is recorded in an append-only `_sqlbuild_janitor_events` table in the affected schema. It records the original name, the archive name, the time and the invocation. The janitor never reads this table to decide what to do. It exists for auditing, and for restoring archives whose names were shortened.
82+
83+
## Restoring an archive
84+
85+
Rename the archive back to its original name before it expires, using your warehouse's rename statement. For example, on Snowflake:
86+
87+
```sql
88+
ALTER TABLE analytics._SQB_ARCHIVE__20260924T101500Z__stg_orders RENAME TO analytics.stg_orders;
89+
```
90+
91+
If the archive's name was shortened, look up the original name in `_sqlbuild_janitor_events`. After an archive has been deleted, your warehouse's own time travel or undrop feature may still be able to recover it within its retention window.
2592

2693
## Configuration
2794

@@ -30,20 +97,21 @@ Configure janitor behavior in `sqlbuild_project.toml`:
3097
```toml
3198
[janitor]
3299
enabled = true
33-
retention_days = 30
100+
retention_days = 14
101+
archive_retention_days = 14
34102
delete_tracked_only = true
35103
exclude_patterns = ["audit_*", "tmp_*"]
36104
```
37105

38-
See [Project Configuration](../concepts/project-configuration.md) for details on janitor settings.
106+
When `retention_days` is not set, direct mode uses `14` and virtual-environment mode uses `30`. See [Project Configuration](../concepts/project-configuration.md#janitor) for every janitor setting.
39107

40108
## Examples
41109

42110
```bash
43-
# Interactive mode (prompts for confirmation)
111+
# Interactive mode (shows the plan and prompts for confirmation)
44112
sqb janitor
45113

46-
# Auto-approve deletion
114+
# Run the planned archives and deletes without prompting
47115
sqb janitor --auto-approve
48116

49117
# Override retention to 7 days
@@ -52,8 +120,12 @@ sqb janitor --retention-days 7
52120

53121
## State history pruning
54122

55-
In addition to cleaning up stale relations, the janitor prunes old rows from `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness` tables, retaining only the latest record per identity. This keeps state tables compact without affecting change detection.
123+
In addition to archiving stale relations, the janitor prunes old rows from `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness`, keeping the latest `direct_state_history_versions` rows per identity (default `20`). This is configured separately from relation retention.
124+
125+
## Virtual environments
126+
127+
In virtual-environment mode, the janitor also cleans up expired environments, old checkpoints and unreferenced physical versions. See [Virtual environments: janitor](../concepts/virtual-environments/janitor.md). Direct-mode archiving doesn't apply there. In both modes, relations whose names differ only by case from another relation in the same schema are reported and never dropped, and exclude patterns match regardless of case.
56128

57129
## Safety
58130

59-
Janitor prompts for confirmation before deleting. The confirmation requires typing an exact string to prevent accidental deletion. Use `--auto-approve` only in CI or when you're certain.
131+
The janitor shows every planned archive and delete, with exact names, ages and expiry, then asks you to type an exact confirmation phrase. Use `--auto-approve` only in automation, or when you're certain.

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ sqb --project-dir <path> plan [flags]
2525
| `--changes-only` | Virtual mode only: show work not already bound to its expected VDE version |
2626
| `--no-python` | Exclude read-side Python tasks and assets from the plan |
2727
| `--defer-to` | Resolve unselected model references against another target |
28+
| `--as` | Compile for another target and show its plan, including [model migration](../concepts/models/migrations.md) decisions, using the current connection. Inspection only. |
2829
| `--json` | Output the plan as JSON |
2930
| `--full-refresh` | Plan a full rebuild for selected models unless a model sets `full_refresh false`; `full_refresh true` forces a model without this flag |
3031
| `--start-cursor-ts` | Override start cursor for timestamp incremental models (ISO format) |

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,8 @@ Table promotion mode is a project setting rather than a `MODEL()` field. Staged
8181
| `full_refresh` | Optional model execution override: `false` always runs incrementally, `true` always full-refreshes, and omission follows the command |
8282
| `on_schema_change` | `append_new_columns`, `sync_all_columns`, `ignore`, or `fail` |
8383
| `replay_on_change` | `forward`, `full`, or `bounded-<duration>` |
84+
| `migrate_from` | Old model name (or `schema.name`) whose data this model takes over; see [Model migrations](migrations.md). Also valid on snapshot models |
85+
| `migrate_force` | `true` replaces a destination that already has its own build history during a migration |
8486

8587
See [Incremental](../incremental.md) for full semantics.
8688

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
<!-- generated-by: sqlbuild skills -->
2+
3+
# Model migrations
4+
5+
> Keep the history of an incremental or snapshot model when you rename it.
6+
7+
Online: https://docs.sqlbuild.com/concepts/models/migrations
8+
9+
# Model migrations
10+
11+
Renaming an incremental or snapshot model normally means rebuilding it from scratch under the new name, which loses any history the source data can no longer reproduce. A model migration moves the existing relation's data to the new name instead, then continues building incrementally.
12+
13+
Migrations apply to incremental and snapshot models in direct mode. Virtual-environment builds reject `migrate_from`.
14+
15+
## Declaring a migration
16+
17+
Add `migrate_from` to the renamed model's header, naming the old model:
18+
19+
```sql
20+
MODEL (
21+
materialized incremental,
22+
migrate_from stg_orders,
23+
cursor order_date,
24+
...
25+
);
26+
```
27+
28+
Use `schema.name` when the old relation is in a different schema of the same database.
29+
30+
On the next build, SQLBuild copies the old relation's data to the new name through a staging table (see [How a move runs](#how-a-move-runs)), records the move, and builds the model incrementally from the migrated state. There is no first-run rebuild, and no `replay_on_change` replay caused by the rename. The old relation is never modified or dropped. Once it is no longer part of the project, [janitor](../../cli/janitor.md) archives it like any other stale relation.
31+
32+
After the move is recorded, `migrate_from` has no further effect, and SQLBuild tells you it can be removed. Keep it until every target that needs the move has built.
33+
34+
## Decisions
35+
36+
`sqb plan` shows the decision for every migration:
37+
38+
| Decision | Meaning |
39+
|----------|---------|
40+
| `migrate` | The new relation doesn't exist. Move the old one's data to it. |
41+
| `done` | The move is already recorded. Nothing to do. |
42+
| `redo` | The new relation exists but has no build history and no recorded move, which is what an interrupted migration leaves behind. Move it again. |
43+
| `superseded replace` | The new relation was itself the origin of the latest recorded move, for example after renaming a model back. Replace it with the newer data. |
44+
| `forced replace` | The new relation has its own build history and `migrate_force true` is set. Replace it. |
45+
| `conflict` | The new relation has its own build history. The build stops (`M103`) until you set `migrate_force true` or remove `migrate_from`. |
46+
| `origin missing` | The old relation doesn't exist and no move was recorded. The build stops (`M102`). |
47+
48+
The origin and destination must be compatible under the model's normal `on_schema_change` rules; otherwise the build stops (`M104`).
49+
50+
`migrate_force true` only affects the `conflict` case and is safe to leave in the header.
51+
52+
## Automatic discovery
53+
54+
If a selected model has never been built, SQLBuild compares it with models that were removed from the project but whose relations still exist. When exactly one removed model has equivalent logic, it is migrated automatically. Equivalent means the same query and configuration, ignoring the model's own name, CTE and table alias names, comments, formatting, and storage-only settings. Renamed upstream models are matched first, so a renamed chain of models is migrated together. Removed models whose data was already moved on by a recorded migration are not candidates, so a model renamed several times matches its latest table.
55+
56+
Renamed tables and views are matched the same way. They are still rebuilt under the new name, since they hold no history, but they keep their identity for change detection, so incremental models downstream of a renamed view are not replayed or rebuilt. These renames are recorded as `renamed` events.
57+
58+
Automatic discovery never guesses. If a match is ambiguous, SQLBuild warns (`M107`) and builds from scratch; declare `migrate_from` to choose. An explicit `migrate_from` always wins. Automatic discovery covers renames within the project's schemas; use `migrate_from` to move a model to another schema. Relations last built before this feature carry no stored fingerprint and aren't matched automatically.
59+
60+
## How a move runs
61+
62+
A move never overwrites the destination directly:
63+
64+
1. The old relation is cloned or copied into a new staging table named `_sqb_archive__<UTC timestamp>__migration_stage__<name>`, with the destination's configured table type.
65+
2. The staging table is checked: it must exist, have the expected columns and table type, and, for a physical copy, the same row count as the old relation.
66+
3. The staging table is swapped in as the destination. If a destination already existed, for example after `superseded replace` or `forced replace`, it is kept as `_sqb_archive__<UTC timestamp>__migration_previous__<name>` rather than dropped.
67+
4. The move is recorded, then the model builds as usual.
68+
69+
The old relation is never modified. Staging and replaced tables use the [janitor](../../cli/janitor.md) archive format, so an abandoned staging table or a replaced destination is deleted by janitor after `archive_retention_days`. A staging table left by an interrupted run is never reused; the next run stages again. Don't run janitor at the same time as a build in the same schema.
70+
71+
`sqb plan` shows how each move will run, for example `physical copy, transient -> permanent, promote by swap`.
72+
73+
## State
74+
75+
Each move is recorded in an append-only `_sqlbuild_migrations` table in the destination schema. A missing record means completion is unrecorded; staging or promotion may already have happened. Re-running after an interruption recovers through the migration decisions above.
76+
77+
## Adapter support
78+
79+
| Adapter | Staging | Promotion |
80+
|---------|---------|-----------|
81+
| Snowflake | Zero-copy clone with `COPY GRANTS`. Transient → permanent is a physical copy, because Snowflake can't clone a transient table into a permanent one. | `ALTER TABLE … SWAP WITH` for an existing destination; rename for a missing destination |
82+
| BigQuery | `CREATE TABLE … CLONE`, or `CREATE TABLE … COPY` when BigQuery refuses the clone | Two renames; the destination is briefly missing between them, and an interrupted run recovers on retry |
83+
| Databricks | `DEEP CLONE` | Two renames, as for BigQuery |
84+
| PostgreSQL, SQL Server, DuckDB, MotherDuck | Physical copy | Renames and the migration record in one transaction |
85+
86+
On PostgreSQL, views that depend on the destination are re-pointed to the new table in the same transaction, keeping their options, owner and grants. Materialized views are not re-pointed.
87+
88+
## Previewing another target
89+
90+
`sqb plan --as <target>` compiles the project for another target, for example production, and shows its migration decisions using your current connection. It only inspects; nothing is written.

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -372,22 +372,28 @@ See [Macros](macros.md) for details on variable substitution and how variables i
372372

373373
## Janitor
374374

375-
Configuration for the `sqb janitor` command, which cleans up stale warehouse relations:
375+
Configuration for the `sqb janitor` command, which archives and then deletes stale warehouse relations:
376376

377377
```toml
378378
[janitor]
379379
enabled = false
380-
retention_days = 30
380+
retention_days = 14
381+
archive_retention_days = 14
381382
delete_tracked_only = true
382383
exclude_patterns = ["audit_*", "tmp_*"]
383384
```
384385

385386
| Field | Default | Description |
386387
|-------|---------|-------------|
387388
| `enabled` | `false` | Whether janitor is active |
388-
| `retention_days` | `30` | Only clean relations older than this many days |
389+
| `retention_days` | `14` in direct mode, `30` in virtual-environment mode | How many days a relation must be stale before it is archived (direct mode) or cleaned up (virtual-environment mode) |
390+
| `archive_retention_days` | `14` | Direct mode only: how many days an archive is kept before it is deleted. `0` deletes it in the same run. |
389391
| `delete_tracked_only` | `true` | Only clean relations that appear in fingerprint tracking |
390392
| `exclude_patterns` | `[]` | Glob patterns for relations to skip |
393+
| `direct_state_history_versions` | `20` | How many state-history rows to keep per identity in `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness` |
394+
| `max_checkpoints` | `20` | Virtual-environment mode: how many checkpoints to keep |
395+
396+
See [janitor](../cli/janitor.md) for the archive and delete lifecycle.
391397

392398
## Scenario
393399

0 commit comments

Comments
 (0)