Repository navigation
perf(snowflake): cut per-model statement overhead, and write a Rules-failed compile to target/compiled-failed - #1265
Merged
Merged
Conversation
Staging and delta tables are created with CREATE OR REPLACE where the adapter replaces, dropped only as tables, kept between microbatch batches that replace them, and counted only when DML reports no row count. A table without an enforced contract skips the staging column read.
…a listings A run lists a schema with SHOW TABLES and SHOW VIEWS once it looks up a second relation there, and reads retention with SHOW instead of INFORMATION_SCHEMA. Promotion reads the destination again when another session changed it.
A build sets up each fingerprint table once and skips rows identical to the latest one, except in projects with model migration history, where rename matching compares rows with rename events.
Scenario cleanup lists the namespace with one SHOW TERSE OBJECTS on Snowflake, drops each relation by its listed kind, and answers the scenario's later existence checks for relations it confirmed absent.
…failed/ A compile whose Rules fail writes every model's SQL to target/compiled-failed/, laid out like target/compiled/, with the not-buildable header on failing models, and never touches target/compiled/. Each failing compile replaces the folder as a whole and the next passing compile deletes it, so target/compiled/ always holds exactly the last passing compile. The report names each failing model's file. The record of written model files is gone. Compile reuse, delta compiles and delta verification cover the new folder: a failing delta writes the models it did not rerun from the stored compile, and a passing delta after a failed compile brings target/compiled/ up to date.
A compile whose Rules fail writes no SQL tests, functions or audits, so a delta on top of it left those files as an older passing compile wrote them. The next compile after a Rules failure now compiles in full, and the delta paths that restored models from the failed compile's SQL are removed. Delta tests whose stored project failed its Rules by accident now give their custom Rules a harness case, or turn off the built-in Rules that cannot parse PIVOT models, so they keep exercising deltas.
A compile whose Rules fail removes target/manifest.json, but with --manifest its report still said it wrote one. The text and JSON reports now leave the manifest out after a Rules failure.
A per-model query tag hook such as ALTER SESSION SET QUERY_TAG, and USE WAREHOUSE, change no relation and no name resolution, but they invalidated every cached listing, so the next lookup listed the whole schema again. Setting or resetting one parameter known not to affect name resolution, and changing the warehouse, now leave the cache alone. Other session parameters, USE of a database, schema or role, and ambiguous statements such as CALL or EXECUTE IMMEDIATE still invalidate it.
…several parameters An unquoted session parameter value ended only at whitespace, so a statement setting a neutral parameter followed by a comma and one that changes name resolution kept the cache. An unquoted value now ends before a comma or an equals sign, and such a statement clears the cache.
kvlonge
enabled auto-merge (squash)
October 28, 2026 16:09
The view whose post hook drops the fingerprint table now reads every other model, so they all record before the drop and the expected rows no longer depend on which independent model the scheduler runs first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Each Snowflake statement is a round trip, and builds spent many on housekeeping: drops of object kinds never created,
INFORMATION_SCHEMAreads, identical fingerprint rows, per-target scenario cleanup. A Rules-failed compile also wrote intotarget/compiled/and needed a side record.Changes
SHOWschema listings and retention; session-only statements such asQUERY_TAGkeep them.LAST_ALTEREDread, thesqb testpath.target/compiled-failed/(with the not-buildable header) and never touchestarget/compiled/. The next passing compile deletes it, and the report names each failing file.target/compiled-models.txtis removed.failure/rule-custom-finding.json: model SQL moves fromcompiledtocompiled_failed.failure/rule-unused-cte.json: same, with the header.failure/rule-unused-cte-after-variable-and-macro.json: same, with the header.Verification
DuckDB housekeeping statements, synthetic orders project:
Snowflake counts confirmed on a scratch schema.
make test,make check-ciand the build, scenario, adapters and compile E2E directories pass; new CLI statement-count, compiled-failed sequence and offline Snowflake tests.