Repository navigation
feat(location): sync every state table without waking a suspended warehouse - #1419
Merged
Merged
Conversation
…cs read A statistics read that timed out halved the shared page limit and rewound to catalog. Finishing catalog reset that limit to the maximum, so statistics retried at full size, timed out again and looped, keeping the warehouse busy. Statistics now has its own page limit: it survives the rewind and returns to the maximum once a statistics scan completes.
…warehouse A code location's catalog sync listed relations and columns through information_schema, and every Snowflake connection checked its session schema through information_schema.schemata. Both need a running warehouse, so each sync cycle resumed it and kept it from suspending. The catalog page now uses the schema-scoped SHOW listings the build already uses when the adapter provides them, and schema_exists asks SHOW SCHEMAS, falling back to the information schema for names SHOW LIKE cannot express.
…n state The location sync now reads every state table SQLBuild writes. Column migrations, node results and consumption join the other kinds, ahead of catalog and statistics. Node result and consumption rows have no event ID, so the server derives one from their identity columns, and retried identical rows collapse to one. Each kind names its source state table, and a test fails when a state table has no kind. The janitor's built-in exclusions now use the same list. The warehouse_metadata main modules are grouped into ingest, location-read and person-read buckets.
…Server A table without an event_id column pages with a NULL placeholder key. The history SQL wrapped that key in the adapter's binary conversion, and Snowflake rejects TO_BINARY(NULL, 'UTF-8'). The placeholder now stays a bare NULL and is left out of ORDER BY, where SQL Server rejects constants.
…and person-read modules
kvlonge
enabled auto-merge (squash)
November 11, 2026 15:21
…to consolidate/sync-state
…o consolidate/sync-state
This was referenced Nov 11, 2026
Contributor
Author
|
Correction to the description above, where "without waking a suspended warehouse" overstates what this PR does.
A follow-up will skip each state-table SELECT when the table's last-altered time hasn't moved since the last sync. It will verify both cases, no new state and new state, with warehouse event and metering history. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A code location's metadata sync kept a suspended Snowflake warehouse running and could loop forever on slow statistics reads. It also mirrored only 7 of the 10 state tables SQLBuild writes. This combines three fixes so they share one CI run. Each was reviewed and passed checks on its own.
Changes
statistics_limit, which halves after a failed read and survives the rewind through catalog. A page too large to deliver halves the current kind's limit.SHOW TABLES,SHOW VIEWS,SHOW COLUMNS IN SCHEMA).schema_existsusesSHOW SCHEMAS LIKE … IN DATABASEwith an exact, case-insensitive match. Other adapters are unchanged.column_migrations,node_resultsandconsumption. A test fails if a name inSTATE_TABLE_NAMEShas no kind. Rows without an event ID get one derived from their identity columns.event_idcan now be paged on Snowflake and SQL Server.warehouse_metadata/mainis split into ingest, location-read and person-read modules.DuckShop: new state-table fixtures. The paging and adapter-listing changes are not project shapes.
Verification
make check-cipasses.location_metadatatests, and the Snowflake unit tests.