Repository navigation
feat: skip unchanged state-table reads during warehouse sync - #1428
Merged
Merged
Conversation
A code location's sync now asks the adapter for a change token before reading each state table. When the token matches the last complete read of that table, the sync skips it without reading any rows. On Snowflake the token is SYSTEM$LAST_CHANGE_COMMIT_TIME, which cloud services answer without resuming the warehouse, so a sync with nothing new leaves the warehouse suspended. A reconciling scan skips only on a token recorded by a reconciling read, and a scan never skips when the mirror has lost rows that read had. Adapters without the capability, missing tables and token errors fall back to reading as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A code location re-reads every warehouse state table on each sync, even when nothing has changed. On Snowflake each read resumes the warehouse, so an idle project pays for compute on every sync.
Changes
RelationChangeTokenReaderprotocol. Snowflake usesSYSTEM$LAST_CHANGE_COMMIT_TIME, which cloud services answer without resuming the warehouse.No DuckShop case applies: this changes how the location reads state tables during sync, not a project shape, data value or build lifecycle.
Verification
make check-cipasses.