Repository navigation
feat(location): sync a location only after runs end or on request - #1421
Merged
Merged
Conversation
…cs read A statistics read that timed out halved the shared page limit and rewound to catalog. Finishing catalog reset that limit to the maximum, so statistics retried at full size, timed out again and looped, keeping the warehouse busy. Statistics now has its own page limit: it survives the rewind and returns to the maximum once a statistics scan completes.
…warehouse A code location's catalog sync listed relations and columns through information_schema, and every Snowflake connection checked its session schema through information_schema.schemata. Both need a running warehouse, so each sync cycle resumed it and kept it from suspending. The catalog page now uses the schema-scoped SHOW listings the build already uses when the adapter provides them, and schema_exists asks SHOW SCHEMAS, falling back to the information schema for names SHOW LIKE cannot express.
…n state The location sync now reads every state table SQLBuild writes. Column migrations, node results and consumption join the other kinds, ahead of catalog and statistics. Node result and consumption rows have no event ID, so the server derives one from their identity columns, and retried identical rows collapse to one. Each kind names its source state table, and a test fails when a state table has no kind. The janitor's built-in exclusions now use the same list. The warehouse_metadata main modules are grouped into ingest, location-read and person-read buckets.
…Server A table without an event_id column pages with a NULL placeholder key. The history SQL wrapped that key in the adapter's binary conversion, and Snowflake rejects TO_BINARY(NULL, 'UTF-8'). The placeholder now stays a bare NULL and is left out of ORDER BY, where SQL Server rejects constants.
…and person-read modules
…to consolidate/sync-state
…o consolidate/sync-state
An idle location no longer polls the warehouse every 30 seconds. It asks the control server every 5 seconds whether a run for its target ended since its last synchronization began, and starts one only then. The first synchronization still runs at startup, and full reconciliation still runs on the first synchronization after its interval.
People with connection-test access can ask a location to read SQLBuild's state and catalog from the warehouse now, with Sync now on the location page or sqb control sync. The request is audited location work that the location acknowledges and follows with a sync, or one more sync after the current one.
Test suites, development scripts and benchmarks are marked as documentation for Linguist.
…nd role The sync-due read refuses another location, another target's session and a stale generation, and answers only two times. A sync request is refused outside the Developer's location and target and for Viewer and Operator.
Locations now sync only after runs, so location health reads every retained day instead of the last two. Sync now waits for a changed sync report instead of comparing browser and server clocks, and stops waiting when the request expires. sqb control sql without a node and sync with a node are usage errors.
A requested sync that fails the same way as before now ends the wait too.
# Conflicts: # crates/sqlbuild-orchestration/src/api/_helpers/resources/warehouse_metadata.rs # crates/sqlbuild-orchestration/src/warehouse_metadata/main/location_reads/mod.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A location read the warehouse every 30 s even when nothing had changed; its state changes when a run ends. People also could not ask for a fresh read.
Changes
POST .../metadata/sync-due. It syncs only on start-up, after a run ends, or on request.sqb control syncqueue audited location work that makes the location sync now. Both need connection-test authority (Developer or Admin).sqb control sqlwithout a node andsyncwith a node are usage errors..gitattributesmarks tests, scripts and benchmarks aslinguist-documentation. Languages before (GitHub API): Rust 48.2%, Python 46.5%, MDX 1.9%, TypeScript 1.7%, Svelte 1.4%. Expected after (local bytes): Rust 73.1%, Python 17.0%, MDX 3.8%, TypeScript 2.7%, Svelte 2.7%.Verification
make check-ci, fensu check and dupes, and 46 integration tests (person SQL and sync, location work, incremental overlap).{runs_ended_at, now};location.work_requested.