Skip to content

feat: show janitor progress, inspect and archive faster - #1226

Merged
kvlonge merged 10 commits into
mainfrom
feat/janitor-progress
Oct 23, 2026
Merged

kvlonge merged 10 commits into
mainfrom
feat/janitor-progress

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Why

sqb janitor was silent while working, listed every object in its preview, and was slow on large warehouses: repeated schema listings, per-view definition queries, and an event lookup plus insert per archive, on one connection.

Changes

  • Progress: per-phase start and closing lines, one n/N line per action (OK/FAIL, time, error); the run ends with ✓ Janitor completed archived A, deleted D, pruned P (time) or ✗ Janitor failed ….
  • Preview shows 10 entries per section, skipped objects grouped by reason; --verbose lists everything; --json puts plan and result (or an error document) on stdout and progress on stderr; options have help.
  • Inspection uses the shared inspection catalog: each schema is listed once, old-name view definitions read in one query per schema.
  • Actions within a phase run on up to the target's concurrency connections where the adapter allows concurrent DDL (Snowflake, BigQuery, Databricks), else 1. Phase order is unchanged.
  • Audit events: one existing-ID read per schema per run, then multi-row inserts (≤100 rows) per schema after each chunk of actions, and for every completed action even when an action, the progress output or an interrupt fails. Shortened archive names get their event right after the rename. IDs, schema and append-only semantics are unchanged. This window is safe: archive age and expiry come from archive names, never events; events serve auditing and shortened-name restores (written immediately); Recall records each rename.
  • Docs updated.

Verification

  • New DuckDB CLI E2E tests: ordered lines, capping, --verbose, --json (incl. error documents), mid-phase failure, per-schema statement counts, concurrent end state equal to sequential (incl. a failing rename), shortened-name event.
  • Unit/integration tests for renderers, the event writer, and events surviving a broken progress stream or interrupt.
  • Focused janitor, old-name-view, refactor, clone and diff tests pass; make check-ci passes.

@kvlonge
kvlonge enabled auto-merge (squash) October 23, 2026 01:19
@kvlonge
kvlonge merged commit bc1245c into main Oct 23, 2026
46 checks passed
@kvlonge
kvlonge deleted the feat/janitor-progress branch October 23, 2026 01:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant