Skip to content

Commit 49817f3

Browse files
authored
feat(location): sync a location only after runs end or on request (#1421)
* fix(location): keep a smaller statistics page after a failed statistics read A statistics read that timed out halved the shared page limit and rewound to catalog. Finishing catalog reset that limit to the maximum, so statistics retried at full size, timed out again and looped, keeping the warehouse busy. Statistics now has its own page limit: it survives the rewind and returns to the maximum once a statistics scan completes. * fix(snowflake): sync the catalog and check schemas without a running warehouse A code location's catalog sync listed relations and columns through information_schema, and every Snowflake connection checked its session schema through information_schema.schemata. Both need a running warehouse, so each sync cycle resumed it and kept it from suspending. The catalog page now uses the schema-scoped SHOW listings the build already uses when the adapter provides them, and schema_exists asks SHOW SCHEMAS, falling back to the information schema for names SHOW LIKE cannot express. * feat(location): mirror column migrations, node results and consumption state The location sync now reads every state table SQLBuild writes. Column migrations, node results and consumption join the other kinds, ahead of catalog and statistics. Node result and consumption rows have no event ID, so the server derives one from their identity columns, and retried identical rows collapse to one. Each kind names its source state table, and a test fails when a state table has no kind. The janitor's built-in exclusions now use the same list. The warehouse_metadata main modules are grouped into ingest, location-read and person-read buckets. * fix(state): page state tables without event IDs on Snowflake and SQL Server A table without an event_id column pages with a NULL placeholder key. The history SQL wrapped that key in the adapter's binary conversion, and Snowflake rejects TO_BINARY(NULL, 'UTF-8'). The placeholder now stays a bare NULL and is left out of ORDER BY, where SQL Server rejects constants. * chore(location): point duplicate-code exemptions at the moved ingest and person-read modules * refactor(location): keep each sync kind's names in one table * test(location): accept one relation prefix in migration history fixtures * feat(location): read the warehouse only after a run ends An idle location no longer polls the warehouse every 30 seconds. It asks the control server every 5 seconds whether a run for its target ended since its last synchronization began, and starts one only then. The first synchronization still runs at startup, and full reconciliation still runs on the first synchronization after its interval. * feat(location): sync a location from the warehouse on request People with connection-test access can ask a location to read SQLBuild's state and catalog from the warehouse now, with Sync now on the location page or sqb control sync. The request is audited location work that the location acknowledges and follows with a sync, or one more sync after the current one. * chore: count only shipped code in GitHub's language statistics Test suites, development scripts and benchmarks are marked as documentation for Linguist. * test(location): refuse sync requests outside their location, target and role The sync-due read refuses another location, another target's session and a stale generation, and answers only two times. A sync request is refused outside the Developer's location and target and for Viewer and Operator. * fix(location): keep a quiet location's last sync visible Locations now sync only after runs, so location health reads every retained day instead of the last two. Sync now waits for a changed sync report instead of comparing browser and server clocks, and stops waiting when the request expires. sqb control sql without a node and sync with a node are usage errors. * fix(ui): end the Sync now wait on any new sync report A requested sync that fails the same way as before now ends the wait too.
1 parent 16747e6 commit 49817f3

51 files changed

Lines changed: 1176 additions & 139 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitattributes‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# GitHub's language bar counts shipped code only: test suites, development scripts and
2+
# benchmarks are documentation to Linguist, in every language.
3+
tests/** linguist-documentation
4+
scripts/** linguist-documentation
5+
benchmarks/** linguist-documentation
6+
crates/**/tests/** linguist-documentation
7+
crates/**/tests.rs linguist-documentation
8+
ui/src/**/*.test.ts linguist-documentation
9+
ui/scripts/** linguist-documentation
10+
clients/typescript/tests/** linguist-documentation
11+
clients/typescript/scripts/** linguist-documentation

‎clients/typescript/src/schema.d.ts‎

Lines changed: 165 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -818,6 +818,22 @@ export interface paths {
818818
patch?: never;
819819
trace?: never;
820820
};
821+
"/api/control/locations/{location_id}/targets/{target}/warehouse-syncs": {
822+
parameters: {
823+
query?: never;
824+
header?: never;
825+
path?: never;
826+
cookie?: never;
827+
};
828+
get?: never;
829+
put?: never;
830+
post: operations["requestWarehouseSync"];
831+
delete?: never;
832+
options?: never;
833+
head?: never;
834+
patch?: never;
835+
trace?: never;
836+
};
821837
"/api/control/platform/events": {
822838
parameters: {
823839
query?: never;
@@ -1010,6 +1026,22 @@ export interface paths {
10101026
patch?: never;
10111027
trace?: never;
10121028
};
1029+
"/api/locations/{location_id}/targets/{target}/metadata/sync-due": {
1030+
parameters: {
1031+
query?: never;
1032+
header?: never;
1033+
path?: never;
1034+
cookie?: never;
1035+
};
1036+
get?: never;
1037+
put?: never;
1038+
post: operations["sync_due"];
1039+
delete?: never;
1040+
options?: never;
1041+
head?: never;
1042+
patch?: never;
1043+
trace?: never;
1044+
};
10131045
"/api/locations/{location_id}/targets/{target}/metadata/watermark": {
10141046
parameters: {
10151047
query?: never;
@@ -1524,11 +1556,6 @@ export interface components {
15241556
rejected_sha256: string;
15251557
table: components["schemas"]["Table"];
15261558
};
1527-
ConnectionTestRequest: {
1528-
reason: string;
1529-
/** Format: uuid */
1530-
request_id: string;
1531-
};
15321559
/** @description What the UI needs first: how people sign in, and what this person may see and do. */
15331560
ControlBootstrap: {
15341561
access: components["schemas"]["PersonAccess"];
@@ -1776,6 +1803,9 @@ export interface components {
17761803
LocationWork: {
17771804
/** @enum {string} */
17781805
kind: "check_connection";
1806+
} | {
1807+
/** @enum {string} */
1808+
kind: "sync_warehouse";
17791809
} | {
17801810
evaluation: Record<string, never>;
17811811
/** @enum {string} */
@@ -1810,6 +1840,9 @@ export interface components {
18101840
document: Record<string, never>;
18111841
/** @enum {string} */
18121842
kind: "plan";
1843+
} | {
1844+
/** @enum {string} */
1845+
kind: "sync_started";
18131846
} | {
18141847
/** @enum {string} */
18151848
kind: "superseded";
@@ -2193,6 +2226,12 @@ export interface components {
21932226
/** @description A running schedule's next due times from `next_due_at` on, at most five; empty otherwise. */
21942227
upcoming: string[];
21952228
};
2229+
/** @description A person asks a location for one named read-only operation, such as a connection test. */
2230+
PersonWorkRequest: {
2231+
reason: string;
2232+
/** Format: uuid */
2233+
request_id: string;
2234+
};
21962235
/** @description Observed physical evidence is not a live warehouse-existence guarantee. */
21972236
PhysicalEvidence: {
21982237
observation: components["schemas"]["CatalogObservation"];
@@ -3107,6 +3146,20 @@ export interface components {
31073146
state_database?: string | null;
31083147
state_schema: string;
31093148
};
3149+
/** @description Both times come from the server clock, so the reader compares them without clock skew. */
3150+
WarehouseSyncDue: {
3151+
/** Format: date-time */
3152+
now: string;
3153+
/** Format: date-time */
3154+
runs_ended_at?: string | null;
3155+
};
3156+
/** @description A registered reader asks whether a run ended since its last synchronization began. */
3157+
WarehouseSyncDueRead: {
3158+
/** Format: int64 */
3159+
generation: number;
3160+
/** Format: uuid */
3161+
session_id: string;
3162+
};
31103163
/**
31113164
* @description Closed failure stages keep warehouse errors and credentials out of health observations.
31123165
* @enum {string}
@@ -4104,7 +4157,7 @@ export interface operations {
41044157
};
41054158
requestBody: {
41064159
content: {
4107-
"application/json": components["schemas"]["ConnectionTestRequest"];
4160+
"application/json": components["schemas"]["PersonWorkRequest"];
41084161
};
41094162
};
41104163
responses: {
@@ -5308,6 +5361,57 @@ export interface operations {
53085361
};
53095362
};
53105363
};
5364+
requestWarehouseSync: {
5365+
parameters: {
5366+
query?: never;
5367+
header: {
5368+
"x-sqb-csrf": string;
5369+
};
5370+
path: {
5371+
location_id: string;
5372+
target: string;
5373+
};
5374+
cookie?: never;
5375+
};
5376+
requestBody: {
5377+
content: {
5378+
"application/json": components["schemas"]["PersonWorkRequest"];
5379+
};
5380+
};
5381+
responses: {
5382+
200: {
5383+
headers: {
5384+
[name: string]: unknown;
5385+
};
5386+
content: {
5387+
"application/json": components["schemas"]["MutationResponse"];
5388+
};
5389+
};
5390+
400: {
5391+
headers: {
5392+
[name: string]: unknown;
5393+
};
5394+
content: {
5395+
"application/json": components["schemas"]["ApiErrorBody"];
5396+
};
5397+
};
5398+
/** @description Session rejected */
5399+
401: {
5400+
headers: {
5401+
[name: string]: unknown;
5402+
};
5403+
content?: never;
5404+
};
5405+
403: {
5406+
headers: {
5407+
[name: string]: unknown;
5408+
};
5409+
content: {
5410+
"application/json": components["schemas"]["ApiErrorBody"];
5411+
};
5412+
};
5413+
};
5414+
};
53115415
listPersonPlatformEvents: {
53125416
parameters: {
53135417
query?: {
@@ -5836,6 +5940,61 @@ export interface operations {
58365940
};
58375941
};
58385942
};
5943+
sync_due: {
5944+
parameters: {
5945+
query?: never;
5946+
header?: never;
5947+
path: {
5948+
location_id: string;
5949+
target: string;
5950+
};
5951+
cookie?: never;
5952+
};
5953+
requestBody: {
5954+
content: {
5955+
"application/json": components["schemas"]["WarehouseSyncDueRead"];
5956+
};
5957+
};
5958+
responses: {
5959+
200: {
5960+
headers: {
5961+
[name: string]: unknown;
5962+
};
5963+
content: {
5964+
"application/json": components["schemas"]["WarehouseSyncDue"];
5965+
};
5966+
};
5967+
400: {
5968+
headers: {
5969+
[name: string]: unknown;
5970+
};
5971+
content: {
5972+
"application/json": components["schemas"]["ApiErrorBody"];
5973+
};
5974+
};
5975+
/** @description Invalid credential */
5976+
401: {
5977+
headers: {
5978+
[name: string]: unknown;
5979+
};
5980+
content?: never;
5981+
};
5982+
/** @description Location not authorized */
5983+
403: {
5984+
headers: {
5985+
[name: string]: unknown;
5986+
};
5987+
content?: never;
5988+
};
5989+
/** @description Reader generation fenced */
5990+
409: {
5991+
headers: {
5992+
[name: string]: unknown;
5993+
};
5994+
content?: never;
5995+
};
5996+
};
5997+
};
58395998
watermark: {
58405999
parameters: {
58416000
query?: never;

‎crates/sqlbuild-cli/src/argv/command_table.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -82,12 +82,12 @@
8282
"name": "control",
8383
"parser": {
8484
"prog": "sqb control",
85-
"description": "Read one exact node's current published SQL using your saved sqb login session. Results are JSON; progress goes to stderr. No project or warehouse connection is opened.",
85+
"description": "Use your saved sqb login session to read one exact node's current published SQL, or to ask a location to sync from the warehouse now. Results are JSON; progress goes to stderr. No project or warehouse connection is opened.",
8686
"groups": [{"title": "positional arguments"}, {"title": "options"}],
8787
"actions": [
8888
{"kind": "help", "options": ["-h", "--help"], "dest": "help", "nargs": 0, "default": {"suppress": true}, "help": "show this help message and exit", "group": 1},
89-
{"kind": "store", "dest": "person_action", "choices": ["sql"], "help": "inspect published raw and compiled SQL", "group": 0},
90-
{"kind": "store", "dest": "person_node", "help": "exact published node key, for example model:orders (not a selector)", "group": 0},
89+
{"kind": "store", "dest": "person_action", "choices": ["sql", "sync"], "help": "sql: inspect published raw and compiled SQL; sync: ask the location to read SQLBuild's state and catalog from the warehouse now", "group": 0},
90+
{"kind": "store", "dest": "person_node", "nargs": "?", "help": "for sql: exact published node key, for example model:orders (not a selector)", "group": 0},
9191
{"kind": "store", "options": ["--server"], "dest": "login_server", "required": true, "help": "HTTPS SQLBuild server origin used by sqb login", "group": 1},
9292
{"kind": "store", "options": ["--ca-cert"], "dest": "login_ca", "help": "additional trusted PEM CA certificate", "group": 1},
9393
{"kind": "store", "options": ["--location-id"], "dest": "person_location", "required": true, "help": "published location UUID", "group": 1},

‎crates/sqlbuild-cli/src/native_commands/_helpers/execution.rs‎

Lines changed: 35 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ use sqlbuild_orchestration::store::models::ControlStore;
3737
use sqlbuild_orchestration::warehouse_metadata::constants::MAX_HISTORY_RECORDS;
3838
use sqlbuild_orchestration::warehouse_metadata::models::{
3939
MetadataBatch, MetadataScan, StatisticsDueRead, StatisticsRead, StatisticsRetention,
40-
WarehouseHistoryKind, WarehouseHistoryPage, WarehouseStateLocation,
40+
WarehouseHistoryKind, WarehouseHistoryPage, WarehouseStateLocation, WarehouseSyncDueRead,
4141
};
4242
use sqlbuild_orchestration::webhooks::constants::SECURITY_AUDIT_WEBHOOK_VARIABLE;
4343
use sqlbuild_orchestration::webhooks::main::{
@@ -70,7 +70,8 @@ use crate::native_commands::_helpers::terminal::{
7070
};
7171
use crate::native_commands::constants::{
7272
ADMIN_COMMAND, COLUMNS_VARIABLE, FALLBACK_COLUMNS, METADATA_POLL_SECONDS,
73-
METADATA_RECONCILE_SECONDS, SCHEDULER_COMMAND, UNCAUGHT_EXCEPTION_STATUS, UNKNOWN_COMMAND,
73+
METADATA_RECONCILE_SECONDS, METADATA_SYNC_CHECK_SECONDS, SCHEDULER_COMMAND,
74+
UNCAUGHT_EXCEPTION_STATUS, UNKNOWN_COMMAND,
7475
};
7576
use crate::native_commands::errors::PythonNeeded;
7677
use crate::native_commands::main::_run_admin::run_admin;
@@ -349,6 +350,8 @@ pub(crate) fn metadata_sync(
349350
statistics_limit: MAX_HISTORY_RECORDS,
350351
pending: None,
351352
next_at: Instant::now(),
353+
idle: true,
354+
cycle_started_at: None,
352355
}
353356
}
354357

@@ -447,6 +450,34 @@ pub(crate) fn freeze_history_page(
447450
Ok(state)
448451
}
449452

453+
/// An idle location asks the server, never the warehouse, whether a run ended since it last began.
454+
pub(crate) async fn begin_after_run(
455+
mut state: LocationMetadataSync,
456+
(client, poll): (&LocationWorkClient, &LocationWorkPoll),
457+
target: &str,
458+
) -> (LocationMetadataSync, Result<bool, ClientError>) {
459+
state.next_at = Instant::now() + Duration::from_secs(METADATA_SYNC_CHECK_SECONDS);
460+
let request = WarehouseSyncDueRead {
461+
session_id: poll.session_id,
462+
generation: state.generation,
463+
};
464+
let due = match client.metadata_sync_due(target, &request).await {
465+
Ok(due) => due,
466+
Err(error) => return (state, Err(error)),
467+
};
468+
let ran = match (state.cycle_started_at, due.runs_ended_at) {
469+
(None, _) => true,
470+
(Some(started_at), Some(ended_at)) => ended_at >= started_at,
471+
(Some(_), None) => false,
472+
};
473+
if ran {
474+
state.idle = false;
475+
state.cycle_started_at = Some(due.now);
476+
state.next_at = Instant::now();
477+
}
478+
(state, Ok(ran))
479+
}
480+
450481
pub(crate) fn acknowledge_metadata_page(mut state: LocationMetadataSync) -> LocationMetadataSync {
451482
let Some(pending) = state.pending.take() else {
452483
return state;
@@ -470,7 +501,8 @@ pub(crate) fn acknowledge_metadata_page(mut state: LocationMetadataSync) -> Loca
470501
state.reconcile_at =
471502
Instant::now() + Duration::from_secs(u64::from(state.reconcile_seconds));
472503
}
473-
state.next_at = Instant::now() + Duration::from_secs(METADATA_POLL_SECONDS);
504+
state.idle = true;
505+
state.next_at = Instant::now();
474506
drop(write_output(&[(
475507
Stream::Stderr,
476508
"Warehouse history synchronization completed.\n",

‎crates/sqlbuild-cli/src/native_commands/constants.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,10 @@ pub(crate) const UNCAUGHT_EXCEPTION_STATUS: i32 = 1;
9393
/// Reserve half the location work claim for acknowledgement retries after killing a callback.
9494
pub(crate) const SENSOR_EVALUATION_SECONDS: u64 = 30;
9595
pub(crate) const SENSOR_RESULT_BYTES: u64 = 1_048_576;
96-
/// Independent read timeout and delay between automatic history polls.
96+
/// Read timeout, and the delay before retrying a failed page of an unfinished synchronization.
9797
pub(crate) const METADATA_POLL_SECONDS: u64 = 30;
98+
/// How often an idle location asks the server, never the warehouse, whether a run ended.
99+
pub(crate) const METADATA_SYNC_CHECK_SECONDS: u64 = 5;
98100
/// Full key-set reconciliation is independent of the incremental poll and overlap window.
99101
pub(crate) const METADATA_RECONCILE_SECONDS: u32 = 300;
100102
pub(crate) const METADATA_OVERLAP_SECONDS: i64 = 300;

‎crates/sqlbuild-cli/src/native_commands/main/_location_metadata.rs‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ use tokio::time::Instant;
1616

1717
use crate::argv::models::Stream;
1818
use crate::native_commands::_helpers::execution::{
19-
acknowledge_metadata_page, freeze_history_page, history_request, location_result,
20-
metadata_health, statistics_page,
19+
acknowledge_metadata_page, begin_after_run, freeze_history_page, history_request,
20+
location_result, metadata_health, statistics_page,
2121
};
2222
use crate::native_commands::_helpers::paths::location_inputs;
2323
use crate::native_commands::_helpers::terminal::write_output;
@@ -41,6 +41,18 @@ pub(super) async fn sync_location_metadata(
4141
}
4242
let started = Instant::now();
4343
tokio::pin!(stopped);
44+
if state.idle {
45+
let (next, due) = tokio::select! {
46+
result = &mut stopped => { result?; return Ok(None); },
47+
result = begin_after_run(state, (client, poll), &options.target) => result,
48+
};
49+
state = next;
50+
match due {
51+
Ok(true) => {}
52+
Ok(false) => return Ok(Some(state)),
53+
Err(error) => return retry_metadata(state, error),
54+
}
55+
}
4456
if state.pending.is_none() {
4557
if state.kind() == WarehouseHistoryKind::Statistics && state.statistics.is_none() {
4658
state.statistics = Some(tokio::select! {

0 commit comments

Comments
 (0)