Repository navigation
feat(janitor): archive stale relations before deletion in direct mode - #809
Merged
Merged
Conversation
Direct-mode janitor now renames eligible stale relations in place to _SQB_ARCHIVE__<UTC timestamp>__<fitted name> after retention_days, and drops relations in managed schemas whose names strictly match the archive grammar once their embedded timestamp is older than archive_retention_days. Malformed look-alike names are reported and never deleted. Deletion decisions use only the name grammar, the embedded timestamp, and managed schema scope. Each archive and delete appends a deterministic, idempotent row to the new _sqlbuild_janitor_events audit table, which is never read for decisions. retention_days now defaults to 14 and the new archive_retention_days defaults to 14; 0/0 archives and deletes in one run. Identifier fitting moves to a shared adapter relation helper, reused by scenario artifacts and archive names. Virtual-environment janitor behaviour is unchanged.
… retention default Direct-mode janitor now renames or drops a relation only when its catalog name is a plain lowercase identifier that does not collide case-insensitively with another relation in the same schema listing. Other names are reported as skipped and never touched. Archive names are generated in case-folded form so the archives the janitor creates always pass this guard. An unset [janitor] retention_days now resolves to 14 days in direct mode and keeps the 30-day default in virtual-environment mode; explicit values and the --retention-days override apply to both modes as before.
…catalog identifiers Listed relations are matched to target schemas case-insensitively and respelled with the project's schema spelling, so warehouses that fold unquoted identifiers (for example lowercase listings of uppercase schemas) produce janitor candidates. Relation keys, source-schema overlap detection, and scan-schema collection compare case-insensitively while rendering keeps an addressable spelling.
Snowflake relation listings already read created and last_altered for tables and views. The adapter now reports age metadata support and normalizes both timestamps to UTC so retention-based janitor cleanup can evaluate relation age.
Databricks relation listings read created and last_altered from information_schema.tables. BigQuery keeps its list_tables listing and adds an on-demand with_relation_age_metadata hook that reads creation and last-modified times from one __TABLES__ query per dataset; janitor planning calls it, so other commands pay no extra query jobs. Catalog timestamps share one UTC normalizer. SQL Server, Postgres, DuckDB, and MotherDuck document why they do not report relation age.
…ite times on Databricks Janitor classification now asks the adapter for age metadata only for relations that survive the desired, tracked, exclude, protection, and addressing filters, in both direct and virtual mode. Databricks information_schema last_altered tracks DDL only, so non-view relations no longer take ages from it; the Databricks hook reads createdAt and lastModified from DESCRIBE DETAIL per candidate and leaves the age unknown when that fails. Views keep their information_schema timestamps.
…eanup metadata-free Folded relation identity made tracked-only membership case-insensitive, so virtual mode could drop an excluded lowercase table through an untracked quoted twin. Virtual mode now applies the case-collision guard, and exclude patterns match case-insensitively in both modes. BigQuery age metadata is keyed by the exact case-sensitive table ID. Janitor no longer reads relation ages when retention_days is 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
In direct mode the janitor only reported stale relations, so cleanup was manual and a dropped relation was hard to recover. It also found nothing on case-folding warehouses: schema keys from the catalog and the project differed in case, and no adapter reported relation age, so any
retention_daysabove 0 skipped every relation.Changes
_sqb_archive__<UTC timestamp>__<name>, then drops archives older thanarchive_retention_days. Deletion is decided by strict name grammar, embedded timestamp and managed schema only. Look-alike names are reported and never touched._sqlbuild_janitor_eventsrow per archive and delete, with deterministic IDs. It is never read for decisions.retention_days14 in direct mode (30 in virtual mode when unset),archive_retention_days14, and0/0allowed.DESCRIBE DETAILfor tables) and BigQuery, fetched only for candidates and never at retention 0.render_rename_viewadapter hook; BigQuery rejects view renames.fit_artifact_logical_name. The disabled-janitor guidance now shows TOML.Verification
make testandmake check-cipass locally;fensu checkreports 0 faults.UNDROPand manual restore worked.