Repository navigation
perf(snowflake): read build-time relation metadata with SHOW and cache it per build - #900
Merged
Merged
Conversation
…ON_SCHEMA Build-time relation_exists and get_columns ran one INFORMATION_SCHEMA query per lookup, which needs a running warehouse and queues behind model work. They now use SHOW TERSE TABLES/VIEWS LIKE ... IN SCHEMA (exact stored-name match on the result) and SHOW COLUMNS IN TABLE, then IN VIEW, with the SHOW COLUMNS type translation shared with planning. A missing relation or schema still reports absent/no columns; a missing database still fails, disambiguated with SHOW SCHEMAS as in planning. Unqualified-schema lookups keep the INFORMATION_SCHEMA query. Refs CHI-455
Materializations looked the same relation up repeatedly: an incremental read its delta's columns three times and its target's twice, and a microbatch read delta and target columns in every batch phase. Each build now opens one relation metadata cache. Runtime column and existence lookups return the previous identical answer until a statement may have changed that relation. Every finished outermost statement is reported to the cache through an in-process StatementLifecycle listener: CREATE/DROP/ALTER/COPY and SELECT ... INTO evict entries with the same unqualified name (RENAME TO and SWAP WITH evict both sides); reads, DML, and transaction control keep them; USE, ALTER SESSION, and anything unrecognized evict everything. A read that overlaps an eviction of its relation is not stored, and on adapters with transactional DDL a COMMIT or ROLLBACK replays that thread's evictions. Nothing is persisted across invocations; migrations keep reading live columns directly. Refs CHI-455
Compile, plan, and build a table, a view, a merge incremental, and a three-day microbatch model twice through the real build pipeline against a simulated Snowflake whose catalog follows the DDL the build issues. Each build must send no INFORMATION_SCHEMA query, must never repeat a metadata read without DDL on that relation in between, and must read columns exactly once per relation version, with one and four workers. A delta that gains a column must reach the merge through a fresh read of the altered target. Refs CHI-455
…guous The statement classifier now lexes literals, quoted identifiers, comments, and dollar-quoted bodies before classifying, so a comment marker or separator inside a literal no longer hides a second statement. It names a relation only when the name is followed by a clause it expects (DROP of several relations, ALTER TABLE ONLY, unquoted non-ASCII names, and unexpected clauses fall back to evicting everything), classifies under standard and backslash-escaped literal rules and evicts everything when they disagree, and keys quoted dotted names the same way lookups do. Refs CHI-455
replace_table_from_relation and load_seed change relations through client jobs that no statement listener sees; they now evict the destination (and the copy origin) from the build's relation metadata cache. Refs CHI-455
The current database used to qualify unqualified SHOW reads was remembered per connection for the life of the session. A USE, CALL, or EXECUTE statement on that connection now clears it, so build and planning lookups re-resolve it. Refs CHI-455
The previous leading-whitespace/comment regex backtracked exponentially on leading whitespace and ran on every Snowflake execute. A small scanner now skips whitespace and comments in linear time, then clears the remembered database for USE, CALL, EXECUTE, and scripting blocks (BEGIN, DECLARE), or when a leading block comment never closes. Refs CHI-455
The literal-aware lexer is now an explicit scanner that stops at the first unterminated comment, literal, or dollar tag instead of rescanning, and the name and tail patterns use possessive quantifiers. CREATE ... SHALLOW/DEEP CLONE evicts only its target, digit-leading unquoted names keep their digits, and adapter-supplied names that cannot be parsed evict everything. A runtime guard covers adversarial inputs. Refs CHI-455
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
In a real Snowflake build (582 resources, 47 min), single-table
information_schemalookups took about 40% of all query time: 1,325 column lookups over 525 tables plus 405 table lookups, against 89 query-minutes of real model work.INFORMATION_SCHEMAneeds a running warehouse and queues behind other work, and the same table was looked up 2-4 times per model or batch (CHI-455 item 7).Changes
SHOW TERSE TABLES/VIEWS LIKE ... IN SCHEMA(exact name match) andSHOW COLUMNS IN TABLE(metadata service, no warehouse), reusing perf(snowflake): plan from SHOW metadata and run inspection reads concurrently #896's type mapping and missing schema/database handling.USE/CALL/scripting blocks reset the remembered session database.Verification
INFORMATION_SCHEMA(including order), existence and missing schema/database cases, fresh reads afterALTER/DROP; scratch objects dropped.INFORMATION_SCHEMAqueries, no repeated read without intervening DDL, exact per-relation counts; invalidation test.make test,make check-ci; one independent review plus a follow-up.