Repository navigation
feat: combined scheduler, web UI, parser and Verify delivery - #1382
Merged
Merged
Conversation
…use Recall relations portably SQL Server's render_create_function stripped a table function's SELECT and wrapped it in BEGIN RETURN (...) END, which the engine rejects. A TABLE return now renders RETURNS TABLE AS RETURN (<query>). sqb recall schema --backend warehouse matched state relations with LIKE '\_sqlbuild\_%' ESCAPE '\', which ClickHouse cannot parse. A prefix comparison with SUBSTRING needs no escape in any dialect.
A native lint response can report that the engine does not parse the project's dialect yet. Each such file now gets one L007 warning carrying the engine's reason, instead of looking lint-clean. A non-string reason is rejected as an invalid response.
…n and refuse WITH source expressions render_create_schema doubled quotes only in the sys.schemas lookup, so a schema name with a quote broke the EXEC string. render_merge now ends with the semicolon SQL Server requires. A source expression starting with WITH cannot sit in a derived table on SQL Server, so it now fails with A715 and a rewrite hint instead of a warehouse syntax error.
… of SQL library probes The unknown-function check (B101) asked the SQL library's parser which built-in node each unaccepted call spelling becomes, and took DuckDB's accepted names from the library's function catalogue. The check now reads a table of the 41 rejected spellings, each with its argument count and the spelling the dialect accepts. Every DuckDB row was confirmed unknown to DuckDB 2.0.0. The call scan takes the analysis dialect name, so analysis off the library can run it unchanged. Findings are identical on every DuckDB and Snowflake query of the compiler differential corpus and on every candidate spelling at 0 to 9 arguments. The workspace no longer depends on the library's function-catalogue crate directly; resolved features are unchanged.
…le and rules run compile and rules run kept only selected Rule codes, so an unavailable dialect looked clean there. L007 now passes through as a warning-severity Rule diagnostic in compile and is left out of rules run's failing-finding count. counts_toward_exit owns the exemption.
The plan and build rules gate failed on any finding, so an L007-only project could not plan. Rule exceptions also validate every finding's code as a rule code and rejected L007 with an invalid-lifecycle error. The gate now counts only findings that count toward exit, and L007 passes through rule exceptions unsuppressed. An end-to-end test replays a cached finding as L007 and as a rule code through compile, rules run, plan and build.
…g issues A same-run time ratio cannot see one store round trip per node: that cost is linear. Tests now count the statements each task issues, and the wide listing must read builds 100 nodes per statement: 17 statements for 1,000 nodes and 37 for 3,000 on SQLite and PostgreSQL, 1,008 with one node per statement. The ratio check stays and now guards super-linear growth only.
The dynamic column contract proof parses pivot SQL with SQLQuack's shared parser (DuckDB, MotherDuck, Snowflake) instead of Polyglot. DuckDB trees match Polyglot's exactly; a Snowflake aggregate keeps its argument in a call's argument list, which the proof now reads. Other dialects keep the existing unsupported-dialect message.
…nto feat/rule-facts-sqlquack
Local scenario capture no longer asks Polyglot to read warehouse column types. Postgres types go through sqlquack-syntax-postgres's type parser, and Snowflake and BigQuery types through their SQLQuack cast-target parse, behind one native function (parse_warehouse_type). DuckDB and MotherDuck spellings pass through unchanged. Databricks nested types map recursively to DuckDB's LIST, MAP and STRUCT, and a nested spelling that cannot be mapped is refused with a named message instead of falling back to JSON. Other types keep their integer width or fall back to VARCHAR.
SQL Server has no SQLQuack type parser, so its information_schema names reached the generic fallback, which held datetime2, smalldatetime, datetimeoffset and uniqueidentifier as VARCHAR. A SQL Server pre-mapper now maps them to TIMESTAMP, TIMESTAMPTZ and UUID, bit to BOOLEAN, money and smallmoney to exact decimals, and SQL Server's 8-byte float to DOUBLE. The fallback keeps 4-byte floats (FLOAT, FLOAT4, REAL) as FLOAT instead of widening them to DOUBLE. A panic in the warehouse type parser now surfaces as the ValueError capture already falls back on.
A run's priority decides which queued run is claimed first, and any person allowed to run a job or rerun could set any priority. A person's job run or rerun above priority 0 now also needs run.prioritise on the run's nodes. Only Admin grants it among built-in roles, project roles may grant it, and it has no default role. Priority 0 and below need nothing extra, and trusted local launches are unchanged.
…ge to resend plain A snapshot whose plain JSON exceeds the wire bound can never be accepted uncompressed, so a 400 for the gzip body is returned as is instead of being masked by a second upload that ends in 413.
A parsed snapshot holds about 7.6 times its JSON size on the server, so a body just under 512 MiB peaked at 3.8 GiB RSS. 320 MiB keeps four times the headroom of a 10,000-model project with column lineage.
Only queueing a job run or a rerun above priority 0 needs run.prioritise. Cancelling an urgent run keeps needing only run.cancel.
…uler fault recovery The scheduler ticks only the latest missed occurrence, so a trigger first evaluated after the next minute begins ticks that minute and skips the earlier one. When the final recovery drain crossed a minute boundary on a slow runner, the fault test counted those triggers as missing their latest tick, and its duplicate probe at that minute was accepted for the same reason. Recovery now cycles until every trigger has ticked at or after the latest due time at settling, bounded by a five-minute deadline. Each trigger's newest tick must reach that time, and the duplicate probe re-inserts a time the trigger actually ticked. The duplicate, off-schedule, run, event and fencing checks are unchanged.
Requiring each trigger's newest tick to be at or after the latest due time has no upper bound, so a scheduler that ticked the next occurrence early would pass. Each tick records the clock it was planned with, and no tick may be for a later time.
…rify Every pull request push ran the whole matrix: wheels on five platforms, Windows and macOS jobs, DuckDB end-to-end shards, warehouse, compiler golden and timing jobs, for about two hours of the organisation's shared job slots. Pushes now run a fast Linux core: static checks, unit and integration tests, native workspace checks and the UI checks. The rest runs when the pull request is labelled full-verify and is not a draft, and always on merge groups and manual runs. A run that leaves out suites its change needs reports its aggregate as a fast-core check, so the required Verify stays missing and blocks the merge; a full run fails Verify when any selected suite was skipped. Changes confined to the web UI skip the Python and native checks they cannot affect. The aggregate now runs unless the run is cancelled, so a cancelled run no longer queues it for a runner. A label other than full-verify starts a run in its own concurrency group that verifies nothing and reports under another name, so it cannot cancel or satisfy the real run.
# Conflicts: # benchmarks/duckshop/edge_cases/orchestration/administrator_lifecycle.json
# Conflicts: # clients/typescript/src/schema.d.ts # crates/sqlbuild-orchestration/src/api/_helpers/resources/person_reads.rs
People check and launch `sqb build` from the run panel. The server resolves the selection against the current publication, authorises run.adhoc plus run.warehouse, run.prioritise and history-rewriting flags, and refuses any flag outside the ad-hoc allowlist by name. The dry run reads only and records nothing. Workers refuse an ad-hoc run whose warehouse the location does not configure for the target, before any statement runs; published jobs keep their authored warehouse.
The release gate requires the aggregate to run with always(): a job skipped when a run is cancelled would report Verify as skipped, which satisfies a required check. Stale runs are force-cancelled instead.
# Conflicts: # ui/src/lib/api/http-client.ts
…rites Reading audit pages authorizes nothing; it reads the facts the credentials audit module appends and prunes. Together the administrator role reads and the ad-hoc scope decisions put 21 modules in authorization/main, one over the container limit.
…he merge gate The release gate now names the aggregate with the label and fast-core facts it reads, and requires that a pull request whose fast core passed without the full matrix, or whose only run followed an unrelated label, stays blocked. The aggregate tests cover full runs with a selected suite skipped, fast-core runs and label-only runs.
# Conflicts: # ui/src/lib/api/http-client.test.ts # ui/src/lib/api/http-client.ts
# Conflicts: # crates/sqlbuild-orchestration/tests/api/test_reads.rs # fensu.toml # ui/src/lib/api/http-catalog.test.ts # ui/src/lib/api/http-catalog.ts # ui/src/lib/api/http-client.test.ts # ui/src/lib/api/http-client.ts # ui/src/lib/api/http-runs.ts
The ad-hoc launch looked up its new run through the shared runs listing without clearing it, so a listing started before the launch could answer without the run and report it missing while it was already queued. The launch now clears shared listings after its write, as a job launch does.
kvlonge
added
the
full-verify
Amp for GitHub
kvlonge
enabled auto-merge (squash)
November 8, 2026 22:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Each pull request costs a full Verify run, so ready work lands in one, each part as its own merge. Verify now runs a fast core per push and the full matrix only with the
full-verifylabel.Changes
Parts: #1378, #1380, #1355, #1351, #1374 with #1356 and #1360, #1379, #1376, #1367, ad-hoc commands, run reads, and the Verify change (aggregate stays on
always(); release-gate tests prove a partial run cannot pass). Security audit reads move beside the audit writes for the module limit. Conflicts were resolved as unions and the TypeScript client regenerated. Column lineage follows separately.Breaking: job runs and reruns above priority 0 now need
run.prioritise(#1379).Verification
make check-ci; fensu 0 faults, dupes 0 clusters.DuckShop: no new case; each part carried its own note.
BEGIN_COMMIT_OVERRIDE
ci: run the full Verify matrix only on pull requests labelled full-verify
feat(orchestration): gzip location snapshot publishes
test: wait for every trigger to reach the latest due time after scheduler fault recovery
test(orchestration): count the store statements a wide catalog listing issues
fix: render SQL Server functions, schemas, merges and sources the engine accepts, and list warehouse Recall relations portably
feat: parse rule facts, types and syntax trees with SQLQuack
refactor(analysis): read B101 spellings from a recorded table
feat(contracts): prove dynamic pivots from SQLQuack parse trees
feat!: require run.prioritise for job runs and reruns above priority 0
feat(ui): wire the platform and admin pages to the control API
fix(ui): stop a burst of control API requests on every live refresh
feat: run ad-hoc sqb commands from the web UI
feat: show run events, statements, SQL and output to people
END_COMMIT_OVERRIDE