Skip to content

fix: evaluate project-wide rules over the whole project on selected runs - #1198

Merged
kvlonge merged 3 commits into
mainfrom
fix/project-rules-whole-project
Oct 21, 2026
Merged

kvlonge merged 3 commits into
mainfrom
fix/project-rules-whole-project

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Why

sqb plan -s, sqb build -s, sqb compile -s and sqb rules run --select evaluated every configured Rule on the selected slice of models. Rules that compare models with each other, such as SQBRPROJECT107 (one published name shape per domain, decided by majority), then gave different findings from a whole-project sqb compile. A selection could report a model that compile accepts.

Changes

  • With a model selection, model-scoped Rules still evaluate only the slice. Project-wide Rules (built-in and custom) evaluate the whole project with the full, unfiltered exception policy, exactly as compile does. Their findings are then limited to selected models and to files that are not models. Their results are always a subset of compile's.
  • The whole-project pass runs after the slice pass and keeps its native, response-memo, fixture and custom-rule caches in bulk/whole-project/, so the two passes never overwrite each other's cache.
  • Selected custom Rules stay selected in the whole-project pass without running, so SQBRTEST301 coverage matches compile.
  • The slice keeps every selected Rule in its selection, so its per-model cache key and entries match compile's; its project-wide findings are discarded in favour of the whole-project pass, and it does not run project-wide custom Rules.
  • Evaluated-model counts stay those of the slice.
  • Documented the behaviour in Execution and caching.

Verification

  • New DuckDB E2E test through the real CLI (fails before, passes after): the reported majority-flip repro, plus a case where model-scoped and project-wide findings outside the slice are hidden and those inside are shown, on both cold and warm runs; and compile, selected plan, edit, compile reuses the built-in Rules cache.
  • Updated one integration test that encoded the old slice-only result for a project-wide custom Rule.
  • Focused rules, compile-selection and integration Rules tests pass. make check-ci passes.

A selection evaluated every configured rule on its slice of models, so rules
that compare models with each other, such as the published name-shape rule,
could report findings that a whole-project compile does not.

Selected runs now evaluate model-scoped rules on the slice and project-wide
rules on the whole project with the complete exception policy, keeping only
their findings on selected models and on files that are not models. The
whole-project pass caches in its own bulk cache directory.
Dropping project-wide codes from a selected run's rule selection changed the
per-model built-in rules cache key, so compile and a selected plan replaced
each other's cache bucket. The slice selects every rule again, as before; its
project-wide findings are still discarded in favour of the whole-project pass,
and it no longer runs project-wide custom rules that pass replaces.
@kvlonge
kvlonge enabled auto-merge (squash) October 21, 2026 11:40
@kvlonge
kvlonge merged commit 5556991 into main Oct 21, 2026
46 checks passed
@kvlonge
kvlonge deleted the fix/project-rules-whole-project branch October 21, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant