Skip to content

Commit f7c8faf

Browse files
authored
perf(snowflake): plan from SHOW metadata and run inspection reads concurrently (#896)
* perf(adapter): add an invocation-scoped warehouse inspection catalog Add a shared InspectionCatalog that serves list_relations and get_columns_for_relations requests for one planning invocation. Adapters implementing SchemaScopedMetadataReader (Snowflake) read each schema's table list and column list at most once with unfiltered INFORMATION_SCHEMA queries and filter client-side using the stored identifier case, so no metadata query carries a large IN list. Sparse column requests keep exact SHOW COLUMNS reads. Independent schema reads run with bounded concurrency on per-thread cursors. Other adapters keep their request shapes; identical requests are memoized. Snowflake source freshness reuses the catalog's schema listing when a catalog is open, and every inspection read can be recorded for verbose output. Refs CHI-455 * perf(planner): reuse schema metadata and read cursor bounds concurrently Open one inspection catalog per execution plan and route snapshot, migration-state, source-column, and semantic-source metadata lookups through it, so each schema is listed once and state-table and source lookups reuse the listing. Target and source schemas are prefetched together (best effort; a failed speculative read is re-read and reported by the lookup that needs it). Snapshot columns are read only for listed relations that planning looks up by node name. Cursor-bound MIN/MAX reads stay one statement per relation and run with the adapter's bounded inspection concurrency (8 on Snowflake, sequential elsewhere); results merge in query order and a failed read still leaves only that relation unavailable. Bounded eligible-max reads fail the plan naming the model and relation, as before. Adds a recording-warehouse budget guard over a synthetic 4-schema project, an equivalence check against the direct adapter path, and an architecture guard keeping catalog SQL in adapter modules. Refs CHI-455 * feat(cli): log warehouse inspection queries under plan and build -v With -v, sqb plan and sqb build write each planning inspection read (truncated SQL, elapsed time, row count or error) and a final total to stderr, keeping JSON stdout machine readable. Refs CHI-455 * perf(snowflake): inspect planning metadata with SHOW instead of INFORMATION_SCHEMA INFORMATION_SCHEMA queries need a running warehouse and queue behind other work on a shared warehouse; SHOW commands are served by the metadata service. Planning schema listings now use SHOW TABLES and SHOW VIEWS (paged past the 10,000-row cap) and SHOW COLUMNS IN SCHEMA, mapped onto the same RelationInfo and type strings the INFORMATION_SCHEMA path produced. A missing or unauthorized schema lists nothing, as before. SHOW COLUMNS silently truncates at 10,000 rows, so a schema that reaches the cap is reread from INFORMATION_SCHEMA.COLUMNS. SHOW has no LAST_ALTERED, so source freshness reads it only for observed sources, in IN lists of at most 50 names run with bounded concurrency. SHOW COLUMNS type translation lives in one helper shared with the per-relation path and is checked against INFORMATION_SCHEMA output for every Snowflake type family. The budget guard now requires SHOW reads per schema and no INFORMATION_SCHEMA reads beyond freshness. Refs CHI-455 * fix(snowflake): treat only a cap-sized SHOW COLUMNS result as truncated SHOW COLUMNS IN SCHEMA returned every row on a real account well past 10,000, so a result is now reread from INFORMATION_SCHEMA only when it is exactly 10,000 rows. SHOW TABLES/VIEWS paging already continues only on a full 10,000-row page. Views and materialized views now report is_transient as None, matching INFORMATION_SCHEMA.TABLES. Refs CHI-455 * fix(planner): harden shared inspection reads from review findings - Semantic source fallback lists candidates by name in every schema and matches schemas case-insensitively, as before the catalog; schema-scoped catalogs answer declared schemas from listings first. - Bounded inspections cancel queued reads and re-raise promptly on any exception in the coordinating thread (including Ctrl-C); after a read fails, running reads finish and the lowest-index failure is raised. - Snowflake SHOW is always qualified with a database (the session's when none is given); only a missing schema lists empty, a missing database fails as before. - Column reads choose per-relation SHOW COLUMNS by the request's total size, as before, not per schema; a relation dropped after listing has no columns instead of failing. - A cap-sized SHOW COLUMNS result rereads only the needed relations from INFORMATION_SCHEMA in concurrent IN chunks of at most 50 names. Refs CHI-455 * fix(snowflake): tell missing schemas from missing databases with SHOW SCHEMAS - SHOW TABLES/VIEWS report the same generic 2043 error for a missing schema and a missing database, so any missing-object error from a SHOW ... IN SCHEMA read is disambiguated once per plan with SHOW SCHEMAS LIKE ... IN DATABASE: a missing schema lists empty (a fresh target plans as a first run), a missing or unusable database fails naming it, and an existing schema re-raises the original error. - Bounded inspection reads started inside a bounded worker run serially, so capped-schema fallback chunks share the outer concurrency bound. - Semantic source candidates are deduplicated by relation identity, so a relation seen by both SHOW and the name lookup is matched once. Refs CHI-455
1 parent dd06e90 commit f7c8faf

53 files changed

Lines changed: 5776 additions & 214 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,19 @@ On connect, SQLBuild applies the connection's role and warehouse, then applies t
4141
target's authoritative database and schema. These ensure the session context is set correctly
4242
regardless of the user's default settings.
4343

44+
## Planning metadata
45+
46+
Planning reads relation and column metadata with `SHOW TABLES`, `SHOW VIEWS`, and `SHOW COLUMNS`
47+
once per schema, several schemas at a time. `SHOW` commands are served by Snowflake's metadata
48+
service, so they don't wait for, or resume, a warehouse. Two reads still use `INFORMATION_SCHEMA`
49+
and need a running warehouse:
50+
51+
- `LAST_ALTERED` for source freshness, read only for the sources the plan observes.
52+
- Columns for a schema whose `SHOW COLUMNS` output is exactly 10,000 rows, the documented
53+
limit at which the output may be cut off.
54+
55+
Run `sqb plan -v` to see each metadata query with its elapsed time and row count.
56+
4457
## Shared connections across targets
4558

4659
Use multiple targets to reuse one Snowflake connection while selecting different authoritative

‎src/sqlbuild/adapter/contract/classes/base_adapter.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,7 @@ class BaseAdapter(RetentionAdapterMixin, StrictAdapter):
107107
views_read_with_reader_access: ClassVar[bool] = False
108108
allows_implicit_managed_write_schema: ClassVar[bool] = False
109109
execution_duration_limit_seconds: ClassVar[int | None] = None
110+
metadata_inspection_concurrency: ClassVar[int] = 1
110111
_snapshot_sql_dialect: ClassVar[SnapshotSqlDialect] = SnapshotSqlDialect(
111112
timestamp_type="TIMESTAMP",
112113
distinct_condition=render_is_distinct_from,
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
"""Invocation-local inspection catalog and verbose query sink state."""
2+
3+
from __future__ import annotations
4+
5+
import logging
6+
from collections.abc import Callable
7+
from contextvars import ContextVar, Token
8+
from typing import TYPE_CHECKING, Any
9+
10+
from sqlbuild.adapter.relations.models import InspectionQueryRecord
11+
from sqlbuild.diagnostics.main.log_debug_event import log_debug_event
12+
from sqlbuild.diagnostics.main.log_sql import log_sql
13+
14+
if TYPE_CHECKING:
15+
from sqlbuild.adapter.relations.classes.inspection_catalog import InspectionCatalog
16+
17+
_LOGGER: logging.Logger = logging.getLogger("sqlbuild.inspection")
18+
_ACTIVE_CATALOG: ContextVar[InspectionCatalog | None] = ContextVar(
19+
"sqlbuild_inspection_catalog", default=None
20+
)
21+
_INSIDE_WORKER: ContextVar[bool] = ContextVar("sqlbuild_inside_inspection_worker", default=False)
22+
_QUERY_SINK: ContextVar[Callable[[InspectionQueryRecord], None] | None] = ContextVar(
23+
"sqlbuild_inspection_query_sink", default=None
24+
)
25+
26+
27+
def current_catalog(*, adapter: Any, connection: Any) -> InspectionCatalog | None:
28+
"""Return the open catalog for exactly this adapter and connection, if any."""
29+
30+
catalog: InspectionCatalog | None = _ACTIVE_CATALOG.get()
31+
if catalog is None or catalog.adapter is not adapter or catalog.connection is not connection:
32+
return None
33+
return catalog
34+
35+
36+
def emit_inspection_record(*, record: InspectionQueryRecord) -> None:
37+
"""Log one inspection read and forward it to the active verbose sink."""
38+
39+
log_sql(logger=_LOGGER, sql=record.sql, action="inspection")
40+
log_debug_event(
41+
logger=_LOGGER,
42+
message="warehouse inspection query",
43+
sqlbuild_elapsed_seconds=f"{record.elapsed_seconds:.3f}",
44+
sqlbuild_row_count=record.row_count,
45+
sqlbuild_error=record.error,
46+
)
47+
sink: Callable[[InspectionQueryRecord], None] | None = _QUERY_SINK.get()
48+
if sink is not None:
49+
sink(record)
50+
51+
52+
def activate_catalog(catalog: InspectionCatalog) -> Token[InspectionCatalog | None]:
53+
"""Make ``catalog`` the open catalog for this context."""
54+
55+
return _ACTIVE_CATALOG.set(catalog)
56+
57+
58+
def deactivate_catalog(token: Token[InspectionCatalog | None]) -> None:
59+
"""Restore the catalog that was open before ``activate_catalog``."""
60+
61+
_ACTIVE_CATALOG.reset(token)
62+
63+
64+
def activate_query_sink(
65+
sink: Callable[[InspectionQueryRecord], None],
66+
) -> Token[Callable[[InspectionQueryRecord], None] | None]:
67+
"""Deliver recorded inspection reads in this context to ``sink``."""
68+
69+
return _QUERY_SINK.set(sink)
70+
71+
72+
def deactivate_query_sink(token: Token[Callable[[InspectionQueryRecord], None] | None]) -> None:
73+
"""Restore the sink that was active before ``activate_query_sink``."""
74+
75+
_QUERY_SINK.reset(token)
76+
77+
78+
def inside_inspection_worker() -> bool:
79+
"""Return whether this thread is running one bounded inspection read."""
80+
81+
return _INSIDE_WORKER.get()
82+
83+
84+
def mark_inspection_worker() -> None:
85+
"""Mark this worker context so nested bounded reads run serially."""
86+
87+
_ = _INSIDE_WORKER.set(True)

0 commit comments

Comments
 (0)