Skip to content

feat: skip unchanged state-table reads during warehouse sync - #1428

Merged
chio-labs-delivery[bot] merged 1 commit into
mainfrom
feat/skip-unchanged-state-tables
Nov 12, 2026
Merged

chio-labs-delivery[bot] merged 1 commit into
mainfrom
feat/skip-unchanged-state-tables

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Why

A code location re-reads every warehouse state table on each sync, even when nothing has changed. On Snowflake each read resumes the warehouse, so an idle project pays for compute on every sync.

Changes

  • Adapters can report a per-table change token through an optional RelationChangeTokenReader protocol. Snowflake uses SYSTEM$LAST_CHANGE_COMMIT_TIME, which cloud services answer without resuming the warehouse.
  • Before a scan's first page, the native host reads the token. When it matches the token from the last complete read of that table, it returns an unchanged page and reads no rows.
  • The CLI keeps tokens in memory, recorded only when a scan completes. A reconciling scan skips only on a token recorded by a reconciling read, and a scan never skips when the mirror lost rows that the recorded read had.
  • Adapters without the capability, missing tables and token errors read as before.
  • The server docs say a Snowflake sync with nothing new doesn't resume the warehouse.

No DuckShop case applies: this changes how the location reads state tables during sync, not a project shape, data value or build lifecycle.

Verification

  • Rust cases for when a recorded token may skip, including a regression case that fails without the incremental-read guard.
  • E2E: a served DuckDB location with a token hook reads nothing when tokens are unchanged, only the changed table after a write, and re-reads a table whose mirror rows were deleted.
  • Existing automatic-history and incremental-overlap E2E tests pass.
  • Live Snowflake check: an unchanged sync left the warehouse suspended; after one inserted row the sync resumed it.
  • make check-ci passes.

A code location's sync now asks the adapter for a change token before
reading each state table. When the token matches the last complete read
of that table, the sync skips it without reading any rows. On Snowflake
the token is SYSTEM$LAST_CHANGE_COMMIT_TIME, which cloud services answer
without resuming the warehouse, so a sync with nothing new leaves the
warehouse suspended.

A reconciling scan skips only on a token recorded by a reconciling read,
and a scan never skips when the mirror has lost rows that read had.
Adapters without the capability, missing tables and token errors fall
back to reading as before.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 12, 2026
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit e1b095e Nov 12, 2026
59 of 80 checks passed
@chio-labs-delivery
chio-labs-delivery Bot deleted the feat/skip-unchanged-state-tables branch November 12, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant