Repository navigation
Commit f7c8faf
authored
perf(snowflake): plan from SHOW metadata and run inspection reads concurrently (#896)
* perf(adapter): add an invocation-scoped warehouse inspection catalog
Add a shared InspectionCatalog that serves list_relations and
get_columns_for_relations requests for one planning invocation.
Adapters implementing SchemaScopedMetadataReader (Snowflake) read each
schema's table list and column list at most once with unfiltered
INFORMATION_SCHEMA queries and filter client-side using the stored
identifier case, so no metadata query carries a large IN list. Sparse
column requests keep exact SHOW COLUMNS reads. Independent schema reads
run with bounded concurrency on per-thread cursors. Other adapters keep
their request shapes; identical requests are memoized.
Snowflake source freshness reuses the catalog's schema listing when a
catalog is open, and every inspection read can be recorded for verbose
output.
Refs CHI-455
* perf(planner): reuse schema metadata and read cursor bounds concurrently
Open one inspection catalog per execution plan and route snapshot,
migration-state, source-column, and semantic-source metadata lookups
through it, so each schema is listed once and state-table and source
lookups reuse the listing. Target and source schemas are prefetched
together (best effort; a failed speculative read is re-read and reported
by the lookup that needs it). Snapshot columns are read only for listed
relations that planning looks up by node name.
Cursor-bound MIN/MAX reads stay one statement per relation and run with
the adapter's bounded inspection concurrency (8 on Snowflake, sequential
elsewhere); results merge in query order and a failed read still leaves
only that relation unavailable. Bounded eligible-max reads fail the plan
naming the model and relation, as before.
Adds a recording-warehouse budget guard over a synthetic 4-schema
project, an equivalence check against the direct adapter path, and an
architecture guard keeping catalog SQL in adapter modules.
Refs CHI-455
* feat(cli): log warehouse inspection queries under plan and build -v
With -v, sqb plan and sqb build write each planning inspection read
(truncated SQL, elapsed time, row count or error) and a final total to
stderr, keeping JSON stdout machine readable.
Refs CHI-455
* perf(snowflake): inspect planning metadata with SHOW instead of INFORMATION_SCHEMA
INFORMATION_SCHEMA queries need a running warehouse and queue behind
other work on a shared warehouse; SHOW commands are served by the
metadata service. Planning schema listings now use SHOW TABLES and
SHOW VIEWS (paged past the 10,000-row cap) and SHOW COLUMNS IN SCHEMA,
mapped onto the same RelationInfo and type strings the
INFORMATION_SCHEMA path produced. A missing or unauthorized schema lists
nothing, as before.
SHOW COLUMNS silently truncates at 10,000 rows, so a schema that
reaches the cap is reread from INFORMATION_SCHEMA.COLUMNS. SHOW has no
LAST_ALTERED, so source freshness reads it only for observed sources,
in IN lists of at most 50 names run with bounded concurrency.
SHOW COLUMNS type translation lives in one helper shared with the
per-relation path and is checked against INFORMATION_SCHEMA output for
every Snowflake type family. The budget guard now requires SHOW reads
per schema and no INFORMATION_SCHEMA reads beyond freshness.
Refs CHI-455
* fix(snowflake): treat only a cap-sized SHOW COLUMNS result as truncated
SHOW COLUMNS IN SCHEMA returned every row on a real account well past
10,000, so a result is now reread from INFORMATION_SCHEMA only when it is
exactly 10,000 rows. SHOW TABLES/VIEWS paging already continues only on
a full 10,000-row page.
Views and materialized views now report is_transient as None, matching
INFORMATION_SCHEMA.TABLES.
Refs CHI-455
* fix(planner): harden shared inspection reads from review findings
- Semantic source fallback lists candidates by name in every schema and
matches schemas case-insensitively, as before the catalog; schema-scoped
catalogs answer declared schemas from listings first.
- Bounded inspections cancel queued reads and re-raise promptly on any
exception in the coordinating thread (including Ctrl-C); after a read
fails, running reads finish and the lowest-index failure is raised.
- Snowflake SHOW is always qualified with a database (the session's when
none is given); only a missing schema lists empty, a missing database
fails as before.
- Column reads choose per-relation SHOW COLUMNS by the request's total
size, as before, not per schema; a relation dropped after listing has
no columns instead of failing.
- A cap-sized SHOW COLUMNS result rereads only the needed relations from
INFORMATION_SCHEMA in concurrent IN chunks of at most 50 names.
Refs CHI-455
* fix(snowflake): tell missing schemas from missing databases with SHOW SCHEMAS
- SHOW TABLES/VIEWS report the same generic 2043 error for a missing
schema and a missing database, so any missing-object error from a
SHOW ... IN SCHEMA read is disambiguated once per plan with
SHOW SCHEMAS LIKE ... IN DATABASE: a missing schema lists empty (a
fresh target plans as a first run), a missing or unusable database
fails naming it, and an existing schema re-raises the original error.
- Bounded inspection reads started inside a bounded worker run serially,
so capped-schema fallback chunks share the outer concurrency bound.
- Semantic source candidates are deduplicated by relation identity, so a
relation seen by both SHOW and the name lookup is matched once.
Refs CHI-4551 parent dd06e90 commit f7c8faf
53 files changed
Lines changed: 5776 additions & 214 deletions
File tree
- src/sqlbuild
- .agents/skills/sqlbuild/references/docs/concepts/adapters
- adapters/snowflake
- _helpers
- classes
- adapter
- contract/classes
- relations
- _helpers
- classes
- main
- cli
- commands
- _helpers/build_planning
- main/project
- progress
- classes
- main
- compiler/planner
- _helpers
- output
- warehouse
- classes
- main/execution
- tests
- e2e/src/sqlbuild/cli/commands/main/plan
- integration/src/sqlbuild/compiler/planner
- _helpers
- main
- unit/src/sqlbuild
- adapters/snowflake/inspection
- adapter/relations
- main/run_bounded_inspections
- cli/progress/main/verbose_inspection_output
- compiler/planner/_helpers/warehouse
- website/src/content/docs/docs/concepts/adapters
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
44 | 57 | | |
45 | 58 | | |
46 | 59 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
107 | 107 | | |
108 | 108 | | |
109 | 109 | | |
| 110 | + | |
110 | 111 | | |
111 | 112 | | |
112 | 113 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
0 commit comments