Skip to content

feat: show which janitor candidates are in use, and measure and cut custom rule overhead - #1232

Merged
kvlonge merged 16 commits into
mainfrom
feat/rules-and-janitor-usage
Oct 23, 2026
Merged

kvlonge merged 16 commits into
mainfrom
feat/rules-and-janitor-usage

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Why

sqb janitor judged a relation stale by project membership and age, not by whether anything still reads it, so archiving could silently break readers. Separately, custom Rules dominated cold compiles with no per-Rule cost breakdown, --no-cache still reused Rule results, and SQBRPROJECT107 needed one entry per qualifier.

Changes

  • Janitor usage check: adapter relation_usage returns latest read, readers, query count and the views a relation was read through, or "not checked" with a reason.
    • Snowflake ACCESS_HISTORY (direct and base objects) plus OBJECT_DEPENDENCIES view chains; BigQuery INFORMATION_SCHEMA.JOBS; Databricks system.access.table_lineage; ClickHouse system.query_log.
    • Missing access means "not checked" naming the grant; other errors fail.
  • Preview In use section with via and a frozen flag; in-use count in the prompt; keep_in_use / --keep-in-use keep them as Kept (in use); in_use_window defaults to "7d"; additive --json fields.
  • compile_timings adds per-Rule invocations, CPU and cache counts, plus monitor and guard CPU; existing keys unchanged.
  • compile --no-cache turns the Rules cache off.
  • The module-state monitor checks once per model (project Rules still after each invocation), so split-host findings match a single process.
  • SQBRPROJECT107 entries accept a qualifier glob and a paths form.
  • Docs.

Verification

  • Custom Rules: synthetic 1,000-model project, 20 Rules, true cold compile: 100.8 G -> 72.6 G instructions (-28%).
  • Janitor: per-adapter unit tests (SQL, parameters, chunking, names, errors, clone false positives); DuckDB E2E via a test usage adapter; ClickHouse E2E (testcontainers) with a view chain read by another user. Not run against live Snowflake, BigQuery or Databricks.
  • Rules: E2E for timings, --no-cache, persistent vs undone module state across 1 and 2 hosts, globs and paths.
  • Focused E2E and integration files on the combined branch, make check-ci.

@kvlonge
kvlonge enabled auto-merge (squash) October 23, 2026 07:06
@kvlonge
kvlonge merged commit 90970f2 into main Oct 23, 2026
46 checks passed
@kvlonge
kvlonge deleted the feat/rules-and-janitor-usage branch October 23, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant