Repository navigation
perf(snowflake): plan from SHOW metadata and run inspection reads concurrently - #896
Merged
Merged
Conversation
Add a shared InspectionCatalog that serves list_relations and get_columns_for_relations requests for one planning invocation. Adapters implementing SchemaScopedMetadataReader (Snowflake) read each schema's table list and column list at most once with unfiltered INFORMATION_SCHEMA queries and filter client-side using the stored identifier case, so no metadata query carries a large IN list. Sparse column requests keep exact SHOW COLUMNS reads. Independent schema reads run with bounded concurrency on per-thread cursors. Other adapters keep their request shapes; identical requests are memoized. Snowflake source freshness reuses the catalog's schema listing when a catalog is open, and every inspection read can be recorded for verbose output. Refs CHI-455
Open one inspection catalog per execution plan and route snapshot, migration-state, source-column, and semantic-source metadata lookups through it, so each schema is listed once and state-table and source lookups reuse the listing. Target and source schemas are prefetched together (best effort; a failed speculative read is re-read and reported by the lookup that needs it). Snapshot columns are read only for listed relations that planning looks up by node name. Cursor-bound MIN/MAX reads stay one statement per relation and run with the adapter's bounded inspection concurrency (8 on Snowflake, sequential elsewhere); results merge in query order and a failed read still leaves only that relation unavailable. Bounded eligible-max reads fail the plan naming the model and relation, as before. Adds a recording-warehouse budget guard over a synthetic 4-schema project, an equivalence check against the direct adapter path, and an architecture guard keeping catalog SQL in adapter modules. Refs CHI-455
With -v, sqb plan and sqb build write each planning inspection read (truncated SQL, elapsed time, row count or error) and a final total to stderr, keeping JSON stdout machine readable. Refs CHI-455
…MATION_SCHEMA INFORMATION_SCHEMA queries need a running warehouse and queue behind other work on a shared warehouse; SHOW commands are served by the metadata service. Planning schema listings now use SHOW TABLES and SHOW VIEWS (paged past the 10,000-row cap) and SHOW COLUMNS IN SCHEMA, mapped onto the same RelationInfo and type strings the INFORMATION_SCHEMA path produced. A missing or unauthorized schema lists nothing, as before. SHOW COLUMNS silently truncates at 10,000 rows, so a schema that reaches the cap is reread from INFORMATION_SCHEMA.COLUMNS. SHOW has no LAST_ALTERED, so source freshness reads it only for observed sources, in IN lists of at most 50 names run with bounded concurrency. SHOW COLUMNS type translation lives in one helper shared with the per-relation path and is checked against INFORMATION_SCHEMA output for every Snowflake type family. The budget guard now requires SHOW reads per schema and no INFORMATION_SCHEMA reads beyond freshness. Refs CHI-455
SHOW COLUMNS IN SCHEMA returned every row on a real account well past 10,000, so a result is now reread from INFORMATION_SCHEMA only when it is exactly 10,000 rows. SHOW TABLES/VIEWS paging already continues only on a full 10,000-row page. Views and materialized views now report is_transient as None, matching INFORMATION_SCHEMA.TABLES. Refs CHI-455
- Semantic source fallback lists candidates by name in every schema and matches schemas case-insensitively, as before the catalog; schema-scoped catalogs answer declared schemas from listings first. - Bounded inspections cancel queued reads and re-raise promptly on any exception in the coordinating thread (including Ctrl-C); after a read fails, running reads finish and the lowest-index failure is raised. - Snowflake SHOW is always qualified with a database (the session's when none is given); only a missing schema lists empty, a missing database fails as before. - Column reads choose per-relation SHOW COLUMNS by the request's total size, as before, not per schema; a relation dropped after listing has no columns instead of failing. - A cap-sized SHOW COLUMNS result rereads only the needed relations from INFORMATION_SCHEMA in concurrent IN chunks of at most 50 names. Refs CHI-455
… SCHEMAS - SHOW TABLES/VIEWS report the same generic 2043 error for a missing schema and a missing database, so any missing-object error from a SHOW ... IN SCHEMA read is disambiguated once per plan with SHOW SCHEMAS LIKE ... IN DATABASE: a missing schema lists empty (a fresh target plans as a first run), a missing or unusable database fails naming it, and an existing schema re-raises the original error. - Bounded inspection reads started inside a bounded worker run serially, so capped-schema fallback chunks share the outer concurrency bound. - Semantic source candidates are deduplicated by relation identity, so a relation seen by both SHOW and the name lookup is matched once. Refs CHI-455
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On Snowflake,
sqb planspent most of its time in warehouse inspection: serialINFORMATION_SCHEMAqueries with IN lists of every relation (one took 35 s), repeated table-list reads, and about 200 cursor-bound queries one at a time.INFORMATION_SCHEMAalso needs a running warehouse and queues behind other work: on a busy shared warehouse a 1 s query waited 6 s (CHI-455 items 5-6).Changes
SHOW TABLES,SHOW VIEWSandSHOW COLUMNS IN SCHEMA(metadata service, no warehouse), once per schema, schemas in parallel (8). JSON column types map to the existing type strings; exact-cap and missing schema/database cases handled.LAST_ALTEREDreads only the needed sources, at most 50 names per query, in parallel.UNION ALL, no caching across runs).sqb plan -v/sqb build -vlog every inspection query (SQL, time, rows) to stderr.Verification
INFORMATION_SCHEMAfor 3,036 relations and ~83,000 columns; a not-yet-created target schema plans as first run.sqb plan --as prod --no-cache: planning 25-30 s over three runs (was ~135 s); compile excluded.make test,make check-ci; one independent review plus a follow-up.