Skip to content

Commit 7f41309

Browse files
authored
feat: combined scheduler, web UI, parser and Verify delivery (#1382)
* fix: render SQL Server table functions as inline TVFs and list warehouse Recall relations portably SQL Server's render_create_function stripped a table function's SELECT and wrapped it in BEGIN RETURN (...) END, which the engine rejects. A TABLE return now renders RETURNS TABLE AS RETURN (<query>). sqb recall schema --backend warehouse matched state relations with LIKE '\_sqlbuild\_%' ESCAPE '\', which ClickHouse cannot parse. A prefix comparison with SUBSTRING needs no escape in any dialect. * feat(lint): warn when SQL lint is unavailable for a dialect A native lint response can report that the engine does not parse the project's dialect yet. Each such file now gets one L007 warning carrying the engine's reason, instead of looking lint-clean. A non-string reason is rejected as an invalid response. * fix(sqlserver): quote schema names in EXEC, end MERGE with a semicolon and refuse WITH source expressions render_create_schema doubled quotes only in the sys.schemas lookup, so a schema name with a quote broke the EXEC string. render_merge now ends with the semicolon SQL Server requires. A source expression starting with WITH cannot sit in a derived table on SQL Server, so it now fails with A715 and a rewrite hint instead of a warehouse syntax error. * refactor(analysis): read B101 spellings from a recorded table instead of SQL library probes The unknown-function check (B101) asked the SQL library's parser which built-in node each unaccepted call spelling becomes, and took DuckDB's accepted names from the library's function catalogue. The check now reads a table of the 41 rejected spellings, each with its argument count and the spelling the dialect accepts. Every DuckDB row was confirmed unknown to DuckDB 2.0.0. The call scan takes the analysis dialect name, so analysis off the library can run it unchanged. Findings are identical on every DuckDB and Snowflake query of the compiler differential corpus and on every candidate spelling at 0 to 9 arguments. The workspace no longer depends on the library's function-catalogue crate directly; resolved features are unchanged. * feat(lint): keep L007 out of every exit decision and show it in compile and rules run compile and rules run kept only selected Rule codes, so an unavailable dialect looked clean there. L007 now passes through as a warning-severity Rule diagnostic in compile and is left out of rules run's failing-finding count. counts_toward_exit owns the exemption. * fix(lint): keep L007 out of rule exceptions and the plan rules gate The plan and build rules gate failed on any finding, so an L007-only project could not plan. Rule exceptions also validate every finding's code as a rule code and rejected L007 with an invalid-lifecycle error. The gate now counts only findings that count toward exit, and L007 passes through rule exceptions unsuppressed. An end-to-end test replays a cached finding as L007 and as a rule code through compile, rules run, plan and build. * docs(rules): name plan and build, not format, in the L007 exit note * test(orchestration): count the store statements a wide catalog listing issues A same-run time ratio cannot see one store round trip per node: that cost is linear. Tests now count the statements each task issues, and the wide listing must read builds 100 nodes per statement: 17 statements for 1,000 nodes and 37 for 3,000 on SQLite and PostgreSQL, 1,008 with one node per statement. The ratio check stays and now guards super-linear growth only. * refactor(analysis): share diagnostic location and the catalog's options view * feat(contracts): prove dynamic pivots from SQLQuack parse trees The dynamic column contract proof parses pivot SQL with SQLQuack's shared parser (DuckDB, MotherDuck, Snowflake) instead of Polyglot. DuckDB trees match Polyglot's exactly; a Snowflake aggregate keeps its argument in a call's argument list, which the proof now reads. Other dialects keep the existing unsupported-dialect message. * fix(contracts): refuse pivot SQL nested deeper than 512 levels before serialising it * wip: read rule facts with SQLQuack * wip: locate cast types and Snowflake date parts without a generator * wip: report built-in rules unavailable for unparsed dialects * wip: e2e for built-in rules on unparsed dialects * chore(deps): bump sqlquack to a9dde6a0 * wip: docs and tests for built-in rules on unparsed dialects * wip: format tests * fix(rules): refuse rule facts for statements deeper than 512 levels * fix(rules): anchor nested comparisons and Snowflake LAST_DAY date parts * feat(scenario): map scenario column types with SQLQuack's type parsers Local scenario capture no longer asks Polyglot to read warehouse column types. Postgres types go through sqlquack-syntax-postgres's type parser, and Snowflake and BigQuery types through their SQLQuack cast-target parse, behind one native function (parse_warehouse_type). DuckDB and MotherDuck spellings pass through unchanged. Databricks nested types map recursively to DuckDB's LIST, MAP and STRUCT, and a nested spelling that cannot be mapped is refused with a named message instead of falling back to JSON. Other types keep their integer width or fall back to VARCHAR. * fix(scenario): map SQL Server date, time, uuid and money types locally SQL Server has no SQLQuack type parser, so its information_schema names reached the generic fallback, which held datetime2, smalldatetime, datetimeoffset and uniqueidentifier as VARCHAR. A SQL Server pre-mapper now maps them to TIMESTAMP, TIMESTAMPTZ and UUID, bit to BOOLEAN, money and smallmoney to exact decimals, and SQL Server's 8-byte float to DOUBLE. The fallback keeps 4-byte floats (FLOAT, FLOAT4, REAL) as FLOAT instead of widening them to DOUBLE. A panic in the warehouse type parser now surfaces as the ValueError capture already falls back on. * chore(fensu): allow the cast-target walk next to the other JSON tree walks * refactor(rules): name the date-part and unavailable-rule loops * docs(analysis): keep the warehouse type binding's doc comment to one line * feat(types): normalize adapter types without Polyglot Read each type with SQLQuack's parser for the dialect, or SQLBuild's own type reader where a dialect has none, and spell it the way the Polyglot-based normalization did so the same aliases stay equal. normalize_type now raises only ValueError, for an unknown dialect. * fix(ui): stop a burst of control API requests on every live refresh The person trigger listing now carries each trigger's newest tick, so the jobs and triggers pages no longer read ticks once per trigger. Pages coalesce change notices into one reload, and the HTTP client shares an in-flight listing between pages that refresh together. * fix(types): read only shipped dialects and ClickHouse nested parameters MotherDuck now reads and spells types as DuckDB. MySQL and the other SQL analysis dialects no adapter ships raise Unknown dialect instead of reading as generic SQL. Under ClickHouse the type reader keeps nested type arguments, enum members with values, and named tuple fields. * refactor(rules): read the shared-parser dialect from one lookup * chore(deps): bump sqlquack to 19c51fa2 sqlquack main 19c51fa2 carries the Postgres parser work and the train that followed a9dde6a0. sqlquack-syntax-postgres now depends on stacker, which re-resolves a few Windows-only windows-sys edges in the lock. * build(deps): pin SQLQuack before its Snowflake grammar checks SQLQuack's next commit refuses a bare call in a Snowflake FROM clause, which rejects the __ref and __source placeholders that compiled model SQL keeps, so custom-rule syntax trees fail on Snowflake models. This rev keeps the BigQuery parser the warehouse type parsers need. * fix(rules): read relation markers as tables in custom-rule syntax trees Compiled query_sql keeps relation markers such as __ref("orders"). Snowflake does not accept a bare function call in FROM, so newer SQLQuack parsers refuse every Snowflake model that references another. A shared helper in sqlbuild-analysis (relation_markers::markers_as_names) now writes each marker call in code as a short indexed name, padded so all later text keeps its line and column, and returns the calls by index. sql_tree reads SQL through it for every dialect. The rules crate's dialect lexical syntax moves to sqlbuild-sqltext so both use one mapping. * build(deps): keep Windows windows-sys edges at main's resolution Adding stacker through the SQLQuack crates re-resolved errno, rustix, tempfile, junction, quinn-udp and winapi-util to older windows-sys releases. On Windows the extension build then compiled windows-sys 0.52 with features the sqb build lacks. The lockfile keeps main's edges, so both builds share one windows-sys again. * docs(types): keep the ClickHouse type parameter doc comment to one line * feat(ui): wire the lineage and changes pages to the live control API Add a person read of a target's published snapshots with the subjects each one changed, behind the same listing authorisation as jobs. The Changes page reads it, and the Lineage page builds its graph and inspector from the catalog listing, so neither page needs mock data. * feat(server): page the security audit newest first with actor names Person reads of the security audit accept an exclusive upper cursor that returns the newest entries first, and name each principal actor. Exports keep their payload unchanged. * feat(ui): wire the audit log to the security audit read The audit log reads the newest security facts first, lists a denial under the action it refused with its decision ID and reason, and names principal actors. Its actor and action filters come from the entries, and links to the person page and access checker stay hidden until the people read exists. * feat(server): list people and role holdings for administrators Add GET /api/admin/people, which needs user.manage everywhere, and GET /api/admin/roles, which needs role.assign everywhere. People carry sign-in facts only: password presence, SSO groups, live session count and last sign-in. Role holdings list person assignments and SSO group mappings. * feat(ui): wire the people page to the people and role reads The people page lists everyone with their highest role, its source and last sign-in. A person's sign-in source is shown only when the server knows it. Links to the person page and access checker, and the risky actions filter, stay hidden until their reads exist. * fix(server): count no active sessions for a disabled person Disabling a person blocks their sessions without revoking them, so the people read no longer counts those sessions as active. * fix(contracts): read relation markers as relations in dynamic pivot proofs Newer SQLQuack Snowflake parsers refuse a bare __ref(...) or __source(...) in FROM, so the proof parses the model SQL through the shared markers_as_names helper and renames each dependency marker's table or column node back to the relation its call names. * refactor(analysis): share the statement depth guard between pivot proofs and rule facts Both crates carried the same iterative deeper_than; sql_tree now owns it. The deep pivot test asserts its expected outcome directly. * fix(contracts): keep non-dependency markers out of pivot relations A __dbt_ref passthrough named its internal placeholder in the failure reason; it now fails as before. The marker scan reads comments and strings with the dialect the proof parses, so MotherDuck scans as DuckDB. * fix(types): spell ClickHouse Nullable and BINARY as the engine does Under ClickHouse, Nullable(T) is read as a wrapper, so an alias inside it is respelled: Nullable(INT) is Nullable(Int32), as bare INT is. BINARY(n) spells as FixedString(n). Bare BINARY, and a Nullable that wraps another Nullable or a nested type, are refused as the engine refuses them. * feat(server): let people read a target's location health Add GET /api/control/locations/{location_id}/targets/{target}/health behind the same listing gate as jobs. It returns the target's newest health observations and instance check-ins, shared with the machine read through one helper. * feat(ui): show live location health on the Locations page Health comes from each location's check-ins and observations: no check-in for 90 seconds is broken, a failed compile at or after the connected generation is falling back, and an unhealthy connection is degraded. People without listing view see the health as unknown. Server status, detail links and the deploy policy column appear only when supported. * fix(server): limit a target's health read to its own observations The person health read filtered by target only after the location's newest 100 observations were taken across all targets, so a busier target could push out another target's compile or connection reports. The target filter now applies before each limit. * feat(server): let people read live platform problems Add GET /api/control/platform/problems for the UI's alert banner. It derives problems at read time with the watchdog's thresholds: whether the scheduler has a leader, location targets whose instances all missed two check-ins, and running schedules more than 30 seconds late. Broken locations and late ticks appear only for targets the person lists, and access is checked per target before that target's ticks are read. * feat(ui): show live platform problems in the alert banner The banner reads the new problems endpoint every 30 seconds, because missed check-ins and late ticks come from time passing, not change notices. A stopped scheduler and each broken location are critical, and late schedules are grouped per location. The read now names each location and dates a stopped scheduler from its last lease. Links go to the Locations list until location detail pages are supported. * docs(ui): keep the locations doc comment on listLocations * feat(server): let role administrators read a target's role definitions Add GET /api/admin/locations/{location_id}/targets/{target}/roles behind server-wide role.assign, like the role holdings read. It returns the built-in roles with their actions, then the target's published custom roles with their description, grants and the nodes each selector matched, and how many active people hold each role there. * feat(ui): show role definitions on the Roles pages The Roles list and role pages read the new role definitions endpoint. Built-in roles show their actions on every node, and custom roles show their published grants, the nodes each selector matched, the published definition and who holds them on this location. Holder counts include SSO group members, counted once per person. The SSO mappings tab and the base-role comparison stay hidden until the server provides them. * feat(ui): show warehouse sync status in the top bar The top bar reads the target's warehouse sync reports from the health read: when a table last synced cleanly, and which tables' newest sync failed and at what stage. It rereads every 30 seconds with the alert banner, because sync reports send no change notice. * fix(ui): stop rereading warehouse sync after a refused read Every refused health read writes an audit denial, so a person who sees a location but may not list all of its nodes would add one each poll. The shell now asks once per target per page load and keeps the indicator at its unknown state after a refusal. * fix(rules): wrap Snowflake table functions in TABLE( ) before parsing The rules engine stubs __table_fn calls before parsing a model. SQLQuack's Snowflake grammar, like Snowflake, accepts a table function in FROM only inside TABLE( ), which is how SQLBuild renders the call, so a bare stub made every built-in Rule report the model unevaluated. Wrap the stub the same way unless the model already wrote TABLE( ), and assert in the Snowflake evaluation tests that the model was actually evaluated. * fix(rules): read Snowflake table function markers inside TABLE( ) in rule trees Custom Rules parse a model through the rule tree, which renames each relation marker to a placeholder name first. SQLQuack's Snowflake grammar refuses the bare table function call that leaves, so a custom Rule on a Snowflake model with a table function failed to parse it. Wrap those calls in TABLE( ) over the marker's own padding and a following space, so node positions keep their line and column, and share the TABLE( ) detection with the built-in Rules' stubbing. * feat(ui): show catalog node detail from the control API and finish lineage labels The catalog node page, the lineage inspector and node links across Catalog, Checks, Changes and Lineage now load from the catalog listing, node status, node SQL and check reads. SQL is read only where the location grants view.sql, so viewers cause no refused, audited reads. Graph cards show the node name without its kind prefix. Sections the server does not report (run history, jobs, history, drift without built SQL, ad-hoc runs) are hidden instead of claiming there are none. * fix(ui): ask for refused node SQL once until reload view.sql in a location's bootstrap actions may hold on only some nodes, so the node SQL read can still be refused. Each refusal is audited and node pages refetch on change signals, so remember the refused node and skip its SQL read until the page reloads. * fix(rules): keep columns after a wrapped table function in rule trees When no space or tab follows a Snowflake table function call, the rule tree inserts its closing parenthesis. Write the TABLE( ) wrapper over one more padding space in that case, so text after the call keeps its column. * fix(ui): wrap SQL previews and drop the kind prefix from SQL file labels Preview lines scrolled sideways with no visible scrollbar, so long lines looked cut off; previews now wrap and the full viewer still scrolls. SQL file labels show the node name, like the graph cards. * feat: publish column lineage and show it on catalog node pages Location publication now adds a lineage snapshot document: the compiler's FAST column lineage edges for every model, as source and target node keys, columns and transform kind. A person read returns the edges reaching or leaving one node, with each node's inferred columns, behind the same view-on-every-node gate as the catalog listing. It carries no SQL or expressions, which need view.sql. The catalog node page's Column lineage tab maps the read onto the existing column lineage view. * fix(ui): show full names and transform kinds in column lineage Traced identifiers and node names wrap instead of truncating, column types stay on one line, and transformed edges name the compiler's transform kind, such as Cast or Aggregation. * refactor: move column lineage to its own change Column lineage publication would push large projects past the publish body limit, so it lands after the publish request is compressed. This change keeps node detail, labels and the SQL preview fixes. * feat: show per-node run history and selecting jobs on catalog node pages A person read lists the scheduler runs that built one node in a target, newest first, behind the same view-on-every-node gate as each run's node list, and carries the same metadata. The catalog node page fills its Runs tab, recent build strip and last duration from it, and lists the jobs and groups whose selection includes the node from the existing jobs and triggers reads. The catalog list drops the kind prefix from node names, and the runs list gives its Started column room for queued times. * feat(orchestration): gzip location snapshot publishes A location now sends its snapshot with Content-Encoding: gzip. The server keeps the 64 MiB wire bound and, after authenticating the location token, streams the body through a 512 MiB decompressed-size bound into the JSON parser. Encodings other than gzip get 415, and a body over either bound gets 413. A server that refuses the encoded body with 400 or 415 gets the snapshot again as plain JSON, so new locations still publish to older servers. * feat: show run events, statements, SQL and output to people Person reads serve a run's Recall rows, scoped to its scheduler and claimed worker invocations. Event and statement metadata need view on every node of the target, as the run's node list does, and leave out event payloads, which can carry SQL and error text. Statement SQL also needs view.sql, and captured output with invocation and diagnostic error text also needs view.evidence, on every node the run selected; both are audited like node SQL and check evidence. The run page's Statements, Events and Output tabs read them. The UI asks for SQL and output only where the bootstrap grants them and stops asking for a run after a refusal until reload, so viewers do not repeat audited denials. SQL blocks wrap instead of clipping. * feat: require run.prioritise for runs above priority 0 A run's priority decides which queued run is claimed first, and any person allowed to run a job or rerun could set any priority. A person's job run or rerun above priority 0 now also needs run.prioritise on the run's nodes. Only Admin grants it among built-in roles, project roles may grant it, and it has no default role. Priority 0 and below need nothing extra, and trusted local launches are unchanged. * fix(orchestration): report the server's refusal of a snapshot too large to resend plain A snapshot whose plain JSON exceeds the wire bound can never be accepted uncompressed, so a 400 for the gzip body is returned as is instead of being masked by a second upload that ends in 413. * fix(orchestration): bound decompressed publish bodies at 320 MiB A parsed snapshot holds about 7.6 times its JSON size on the server, so a body just under 512 MiB peaked at 3.8 GiB RSS. 320 MiB keeps four times the headroom of a 10,000-model project with column lineage. * fix: keep run cancellation independent of run.prioritise Only queueing a job run or a rerun above priority 0 needs run.prioritise. Cancelling an urgent run keeps needing only run.cancel. * test: wait for every trigger to reach the latest due time after scheduler fault recovery The scheduler ticks only the latest missed occurrence, so a trigger first evaluated after the next minute begins ticks that minute and skips the earlier one. When the final recovery drain crossed a minute boundary on a slow runner, the fault test counted those triggers as missing their latest tick, and its duplicate probe at that minute was accepted for the same reason. Recovery now cycles until every trigger has ticked at or after the latest due time at settling, bounded by a five-minute deadline. Each trigger's newest tick must reach that time, and the duplicate probe re-inserts a time the trigger actually ticked. The duplicate, off-schedule, run, event and fencing checks are unchanged. * test: count ticks scheduled after their own evaluation as off schedule Requiring each trigger's newest tick to be at or after the latest due time has no upper bound, so a scheduler that ticked the next occurrence early would pass. Each tick records the clock it was planned with, and no tick may be for a later time. * ci: run the full Verify matrix only on pull requests labelled full-verify Every pull request push ran the whole matrix: wheels on five platforms, Windows and macOS jobs, DuckDB end-to-end shards, warehouse, compiler golden and timing jobs, for about two hours of the organisation's shared job slots. Pushes now run a fast Linux core: static checks, unit and integration tests, native workspace checks and the UI checks. The rest runs when the pull request is labelled full-verify and is not a draft, and always on merge groups and manual runs. A run that leaves out suites its change needs reports its aggregate as a fast-core check, so the required Verify stays missing and blocks the merge; a full run fails Verify when any selected suite was skipped. Changes confined to the web UI skip the Python and native checks they cannot affect. The aggregate now runs unless the run is cancelled, so a cancelled run no longer queues it for a runner. A label other than full-verify starts a run in its own concurrency group that verifies nothing and reports under another name, so it cannot cancel or satisfy the real run. * feat: run ad-hoc sqb commands from the web UI People check and launch `sqb build` from the run panel. The server resolves the selection against the current publication, authorises run.adhoc plus run.warehouse, run.prioritise and history-rewriting flags, and refuses any flag outside the ad-hoc allowlist by name. The dry run reads only and records nothing. Workers refuse an ad-hoc run whose warehouse the location does not configure for the target, before any statement runs; published jobs keep their authored warehouse. * ci: keep the Verify aggregate on always() The release gate requires the aggregate to run with always(): a job skipped when a run is cancelled would report Verify as skipped, which satisfies a required check. Stale runs are force-cancelled instead. * refactor(orchestration): page security audit facts beside the audit writes Reading audit pages authorizes nothing; it reads the facts the credentials audit module appends and prunes. Together the administrator role reads and the ad-hoc scope decisions put 21 modules in authorization/main, one over the container limit. * test(ci): prove a fast-core or label-only Verify run cannot satisfy the merge gate The release gate now names the aggregate with the label and fast-core facts it reads, and requires that a pull request whose fast core passed without the full matrix, or whose only run followed an unrelated label, stays blocked. The aggregate tests cover full runs with a selected suite skipped, fast-core runs and label-only runs. * fix(ui): find an ad-hoc run launched while a runs listing is in flight The ad-hoc launch looked up its new run through the shared runs listing without clearing it, so a listing started before the launch could answer without the run and report it missing while it was already queued. The launch now clears shared listings after its write, as a job launch does.
1 parent b8149cc commit 7f41309

316 files changed

Lines changed: 17245 additions & 8316 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/verify.yml‎

Lines changed: 112 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -2,23 +2,34 @@ name: Verify
22

33
on:
44
pull_request:
5+
# Labelling a pull request `full-verify` (or removing the label) changes how much it runs.
6+
types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]
57
merge_group:
68
workflow_dispatch:
79

810
permissions:
911
contents: read
1012

1113
# Each merge group has its own gh-readonly-queue ref, so queued groups never cancel each other; the
12-
# queue itself discards a group's run when it rebuilds the group.
14+
# queue itself discards a group's run when it rebuilds the group. A push to a pull request cancels
15+
# its older run. Adding or removing any label other than `full-verify` starts a run in a group of its
16+
# own that verifies nothing, so it never cancels the pull request's real run.
1317
concurrency:
14-
group: verify-${{ github.workflow }}-${{ github.event.merge_group.head_ref || github.ref }}
18+
group: >-
19+
verify-${{ github.workflow }}-${{ github.event.merge_group.head_ref || github.ref }}${{
20+
(github.event.action == 'labeled' || github.event.action == 'unlabeled')
21+
&& github.event.label.name != 'full-verify' && format('-label-{0}', github.run_id) || '' }}
1522
cancel-in-progress: ${{ github.event_name != 'merge_group' }}
1623

1724
jobs:
1825
changes:
1926
name: Classify verification scope
27+
# A label other than `full-verify` changes nothing this workflow verifies.
28+
if: ${{ !((github.event.action == 'labeled' || github.event.action == 'unlabeled') && github.event.label.name != 'full-verify') }}
2029
runs-on: ubuntu-24.04
2130
outputs:
31+
full: ${{ steps.depth.outputs.full }}
32+
partial: ${{ steps.depth.outputs.partial }}
2233
checks: ${{ steps.scope.outputs.checks }}
2334
compiler_differential: ${{ steps.scope.outputs.compiler_differential }}
2435
duckdb: ${{ steps.scope.outputs.duckdb }}
@@ -138,6 +149,9 @@ jobs:
138149
# skill pages generated from the website docs skips the Python checks and test suites.
139150
# The website builds in its own workflow.
140151
docs_only=true
152+
# A change that only touches the web UI (and documentation) runs the UI checks and the
153+
# wheels that embed its build, but not the Python and native checks it cannot affect.
154+
ui_only=true
141155
while IFS= read -r -d '' path; do
142156
changed=true
143157
case "${path}" in
@@ -147,6 +161,13 @@ jobs:
147161
*.md) ;;
148162
*) docs_only=false ;;
149163
esac
164+
case "${path}" in
165+
ui/*|website/*|.github/workflows/website.yml|.github/demos/*) ;;
166+
src/sqlbuild/.agents/skills/sqlbuild/references/docs/*.md) ;;
167+
*/*) ui_only=false ;;
168+
*.md) ;;
169+
*) ui_only=false ;;
170+
esac
150171
case "${path}" in
151172
crates/*|ui/*|Cargo.toml|Cargo.lock|pyproject.toml|.github/workflows/verify.yml|.github/workflows/build-distributions.yml|deploy/Dockerfile|deploy/project/*|deploy/compose*|compose.yml|.dockerignore|Makefile|tests/integration/src/sqlbuild/compiler/compiled_project/publication/test_compose.py|tests/integration/src/sqlbuild/compiler/compiled_project/publication/docker/compose/*|benchmarks/duckshop/edge_cases/orchestration/container_deployment.json)
152173
wheels=true
@@ -232,7 +253,7 @@ jobs:
232253
fi
233254
234255
checks=true
235-
if [[ "${docs_only}" == "true" ]]; then
256+
if [[ "${docs_only}" == "true" || "${ui_only}" == "true" ]]; then
236257
checks=false
237258
fi
238259
@@ -256,8 +277,36 @@ jobs:
256277
"${recovery}" \
257278
"${ui}"
258279
280+
# Pull requests run the fast Linux core: static checks, unit and integration tests, native
281+
# workspace checks and the UI checks. The rest of the matrix (wheels, Compose, Windows, macOS,
282+
# DuckDB E2E, warehouse, compiler-golden and timing jobs) runs only once a pull request is
283+
# labelled `full-verify` and is not a draft, and always on merge groups and manual runs. A
284+
# run that leaves out suites its change needs reports under another name, so the required
285+
# Verify stays missing and blocks the merge until the full matrix has passed.
286+
- name: Choose the fast core or the full matrix
287+
id: depth
288+
env:
289+
FULL: >-
290+
${{ github.event_name != 'pull_request'
291+
|| (contains(github.event.pull_request.labels.*.name, 'full-verify')
292+
&& !github.event.pull_request.draft) }}
293+
HEAVY_SELECTED: >-
294+
${{ steps.scope.outputs.duckdb == 'true' || steps.scope.outputs.postgres == 'true' || steps.scope.outputs.clickhouse == 'true' || steps.scope.outputs.performance == 'true' || steps.scope.outputs.format_performance == 'true' || steps.scope.outputs.rules_performance == 'true' || steps.scope.outputs.wheels == 'true' || steps.scope.outputs.compiler_differential == 'true' || steps.scope.outputs.recovery == 'true' }}
295+
run: |
296+
partial=false
297+
if [[ "${FULL}" != "true" && "${HEAVY_SELECTED}" == "true" ]]; then
298+
partial=true
299+
echo "Fast core only: label the pull request full-verify to run the full matrix Verify needs" \
300+
| tee -a "${GITHUB_STEP_SUMMARY}"
301+
fi
302+
{
303+
echo "full=${FULL}"
304+
echo "partial=${partial}"
305+
} >> "${GITHUB_OUTPUT}"
306+
259307
lockfile:
260308
name: Lockfile consistency
309+
if: ${{ !((github.event.action == 'labeled' || github.event.action == 'unlabeled') && github.event.label.name != 'full-verify') }}
261310
runs-on: ubuntu-24.04
262311
steps:
263312
- uses: actions/checkout@v4
@@ -365,7 +414,7 @@ jobs:
365414
e2e_shards:
366415
name: DuckDB E2E (${{ matrix.name }})
367416
needs: [changes, lockfile]
368-
if: needs.changes.outputs.duckdb == 'true'
417+
if: needs.changes.outputs.duckdb == 'true' && needs.changes.outputs.full == 'true'
369418
runs-on: ubuntu-24.04
370419
timeout-minutes: 45
371420
strategy:
@@ -442,7 +491,7 @@ jobs:
442491
compiler_differential:
443492
name: Compiler golden outputs
444493
needs: [changes, lockfile]
445-
if: needs.changes.outputs.compiler_differential == 'true'
494+
if: needs.changes.outputs.compiler_differential == 'true' && needs.changes.outputs.full == 'true'
446495
runs-on: ubuntu-24.04
447496
timeout-minutes: 45
448497
steps:
@@ -483,7 +532,7 @@ jobs:
483532
compiler_differential_python_versions:
484533
name: Native compiler on Python ${{ matrix.python }}
485534
needs: [changes, lockfile]
486-
if: needs.changes.outputs.compiler_differential == 'true'
535+
if: needs.changes.outputs.compiler_differential == 'true' && needs.changes.outputs.full == 'true'
487536
runs-on: ubuntu-24.04
488537
timeout-minutes: 45
489538
strategy:
@@ -523,7 +572,7 @@ jobs:
523572
native_discovery_windows:
524573
name: Native compiler on Windows
525574
needs: [changes, lockfile]
526-
if: needs.changes.outputs.compiler_differential == 'true'
575+
if: needs.changes.outputs.compiler_differential == 'true' && needs.changes.outputs.full == 'true'
527576
runs-on: windows-2025
528577
timeout-minutes: 45
529578
defaults:
@@ -580,7 +629,7 @@ jobs:
580629
scheduler_recovery:
581630
name: Scheduler recovery (${{ matrix.name }})
582631
needs: [changes, lockfile]
583-
if: needs.changes.outputs.recovery == 'true'
632+
if: needs.changes.outputs.recovery == 'true' && needs.changes.outputs.full == 'true'
584633
runs-on: ${{ matrix.runner }}
585634
timeout-minutes: 75
586635
strategy:
@@ -632,7 +681,7 @@ jobs:
632681
format_performance:
633682
name: Formatter performance guards
634683
needs: [changes, lockfile]
635-
if: needs.changes.outputs.format_performance == 'true'
684+
if: needs.changes.outputs.format_performance == 'true' && needs.changes.outputs.full == 'true'
636685
runs-on: ubuntu-24.04
637686
timeout-minutes: 45
638687
steps:
@@ -696,7 +745,7 @@ jobs:
696745
cold_compile_performance:
697746
name: Fresh-process compile (${{ matrix.models }} models)
698747
needs: [changes, lockfile]
699-
if: needs.changes.outputs.performance == 'true'
748+
if: needs.changes.outputs.performance == 'true' && needs.changes.outputs.full == 'true'
700749
runs-on: ubuntu-24.04
701750
timeout-minutes: 45
702751
strategy:
@@ -747,7 +796,7 @@ jobs:
747796
dense_compile_performance:
748797
name: Dense all-rules compile (${{ matrix.models }} models)
749798
needs: [changes, lockfile]
750-
if: needs.changes.outputs.performance == 'true'
799+
if: needs.changes.outputs.performance == 'true' && needs.changes.outputs.full == 'true'
751800
runs-on: ubuntu-24.04
752801
# The 3k and 10k cells also run the delta edit-loop guard, about five minutes at 10k.
753802
timeout-minutes: 45
@@ -791,7 +840,7 @@ jobs:
791840
compile_performance_ratio:
792841
name: Same-runner compile ratio (${{ matrix.label || matrix.kind }} 3000 models)
793842
needs: [changes, lockfile]
794-
if: needs.changes.outputs.performance == 'true'
843+
if: needs.changes.outputs.performance == 'true' && needs.changes.outputs.full == 'true'
795844
runs-on: ubuntu-24.04
796845
timeout-minutes: 45
797846
strategy:
@@ -894,7 +943,7 @@ jobs:
894943
cache_compile_performance:
895944
name: Fresh-process compile cache (${{ matrix.models }} models)
896945
needs: [changes, lockfile]
897-
if: needs.changes.outputs.performance == 'true'
946+
if: needs.changes.outputs.performance == 'true' && needs.changes.outputs.full == 'true'
898947
runs-on: ubuntu-24.04
899948
timeout-minutes: 45
900949
strategy:
@@ -1019,7 +1068,7 @@ jobs:
10191068
postgres:
10201069
name: PostgreSQL integration and E2E tests
10211070
needs: [changes, lockfile]
1022-
if: needs.changes.outputs.postgres == 'true'
1071+
if: needs.changes.outputs.postgres == 'true' && needs.changes.outputs.full == 'true'
10231072
runs-on: ubuntu-24.04
10241073
timeout-minutes: 45
10251074
steps:
@@ -1056,7 +1105,7 @@ jobs:
10561105
clickhouse:
10571106
name: ClickHouse integration and E2E tests
10581107
needs: [changes, lockfile]
1059-
if: needs.changes.outputs.clickhouse == 'true'
1108+
if: needs.changes.outputs.clickhouse == 'true' && needs.changes.outputs.full == 'true'
10601109
runs-on: ubuntu-24.04
10611110
timeout-minutes: 45
10621111
steps:
@@ -1093,7 +1142,7 @@ jobs:
10931142
wheels:
10941143
name: Wheels
10951144
needs: [changes]
1096-
if: needs.changes.outputs.wheels == 'true'
1145+
if: needs.changes.outputs.wheels == 'true' && needs.changes.outputs.full == 'true'
10971146
uses: ./.github/workflows/build-distributions.yml
10981147
with:
10991148
sdist: false
@@ -1105,7 +1154,7 @@ jobs:
11051154
compose:
11061155
name: Packaged Compose lifecycle
11071156
needs: [changes, lockfile, wheels]
1108-
if: needs.changes.outputs.wheels == 'true'
1157+
if: needs.changes.outputs.wheels == 'true' && needs.changes.outputs.full == 'true'
11091158
runs-on: ubuntu-24.04
11101159
timeout-minutes: 20
11111160
steps:
@@ -1158,11 +1207,16 @@ jobs:
11581207
# metadata-only pass can never satisfy Verify first; a skipped job would count as passing, so
11591208
# the job always runs and only its name changes. Until those checks post Verify on the head,
11601209
# the pull request waits for an expected check, and every new push to the branch waits again.
1210+
# The same holds for a fast-core run that leaves out suites its change needs, and for a run
1211+
# started by an unrelated label: neither reports as Verify.
11611212
name: >-
11621213
${{ (github.event_name == 'pull_request'
11631214
&& github.event.pull_request.head.repo.full_name == github.repository
11641215
&& startsWith(github.event.pull_request.head.ref, 'release-please--branches--'))
1165-
&& 'Release pull request suites' || 'Verify' }}
1216+
&& 'Release pull request suites'
1217+
|| ((github.event.action == 'labeled' || github.event.action == 'unlabeled') && github.event.label.name != 'full-verify') && 'Label change (nothing verified)'
1218+
|| needs.changes.outputs.partial == 'true' && 'Fast core (label full-verify for Verify)'
1219+
|| 'Verify' }}
11661220
if: always()
11671221
needs:
11681222
[
@@ -1215,7 +1269,21 @@ jobs:
12151269
COMPOSE_RESULT: ${{ needs.compose.result }}
12161270
UI_SELECTED: ${{ needs.changes.outputs.ui }}
12171271
UI_RESULT: ${{ needs.ui.result }}
1272+
LABEL_ONLY: ${{ (github.event.action == 'labeled' || github.event.action == 'unlabeled') && github.event.label.name != 'full-verify' }}
1273+
PARTIAL: ${{ needs.changes.outputs.partial }}
1274+
DUCKDB_SELECTED: ${{ needs.changes.outputs.duckdb }}
1275+
COMPILER_DIFFERENTIAL_SELECTED: ${{ needs.changes.outputs.compiler_differential }}
1276+
RECOVERY_SELECTED: ${{ needs.changes.outputs.recovery }}
1277+
FORMAT_PERFORMANCE_SELECTED: ${{ needs.changes.outputs.format_performance }}
1278+
PERFORMANCE_SELECTED: ${{ needs.changes.outputs.performance }}
1279+
POSTGRES_SELECTED: ${{ needs.changes.outputs.postgres }}
1280+
CLICKHOUSE_SELECTED: ${{ needs.changes.outputs.clickhouse }}
1281+
WHEELS_SELECTED: ${{ needs.changes.outputs.wheels }}
12181282
run: |
1283+
if [[ "${LABEL_ONLY}" == "true" ]]; then
1284+
echo "A label other than full-verify changed; this run verifies nothing"
1285+
exit 0
1286+
fi
12191287
if [[ "${CHANGES_RESULT}" != "success" || "${LOCKFILE_RESULT}" != "success" ]]; then
12201288
echo "Required classification or lockfile checks did not pass" >&2
12211289
exit 1
@@ -1258,4 +1326,30 @@ jobs:
12581326
echo "Required UI checks did not pass: ${UI_RESULT}" >&2
12591327
exit 1
12601328
fi
1329+
# A full run must have run every suite its change selected; a skip here is never a pass.
1330+
if [[ "${PARTIAL}" != "true" ]]; then
1331+
for pair in \
1332+
"${DUCKDB_SELECTED}:${E2E_SHARDS_RESULT}" \
1333+
"${COMPILER_DIFFERENTIAL_SELECTED}:${COMPILER_DIFFERENTIAL_RESULT}" \
1334+
"${COMPILER_DIFFERENTIAL_SELECTED}:${COMPILER_DIFFERENTIAL_PYTHON_VERSIONS_RESULT}" \
1335+
"${COMPILER_DIFFERENTIAL_SELECTED}:${NATIVE_DISCOVERY_WINDOWS_RESULT}" \
1336+
"${RECOVERY_SELECTED}:${SCHEDULER_RECOVERY_RESULT}" \
1337+
"${FORMAT_PERFORMANCE_SELECTED}:${FORMAT_PERFORMANCE_RESULT}" \
1338+
"${PERFORMANCE_SELECTED}:${COLD_COMPILE_PERFORMANCE_RESULT}" \
1339+
"${PERFORMANCE_SELECTED}:${DENSE_COMPILE_PERFORMANCE_RESULT}" \
1340+
"${PERFORMANCE_SELECTED}:${COMPILE_PERFORMANCE_RATIO_RESULT}" \
1341+
"${PERFORMANCE_SELECTED}:${CACHE_COMPILE_PERFORMANCE_RESULT}" \
1342+
"${POSTGRES_SELECTED}:${POSTGRES_RESULT}" \
1343+
"${CLICKHOUSE_SELECTED}:${CLICKHOUSE_RESULT}" \
1344+
"${WHEELS_SELECTED}:${WHEELS_RESULT}" \
1345+
"${WHEELS_SELECTED}:${COMPOSE_RESULT}"; do
1346+
if [[ "${pair%%:*}" == "true" && "${pair#*:}" != "success" ]]; then
1347+
echo "A suite this change needs did not run to success: ${pair#*:}" >&2
1348+
exit 1
1349+
fi
1350+
done
1351+
else
1352+
echo "Fast core passed; the full matrix runs once the pull request is labelled full-verify"
1353+
exit 0
1354+
fi
12611355
echo "All path-relevant verification jobs passed"

‎CONTRIBUTING.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,14 @@ For broader validation, run:
9494
make verify
9595
```
9696

97+
In CI, every push to a pull request runs a fast Linux core: static checks, unit and integration
98+
tests and native workspace checks, plus the UI checks for UI changes. A change confined to the web
99+
UI and documentation runs only the UI checks. The full matrix (wheels, Compose, Windows, macOS,
100+
DuckDB end-to-end, warehouse, compiler-golden and timing jobs) runs once a maintainer labels a
101+
pull request that is not a draft `full-verify`, normally when it is next to merge. The required
102+
`Verify` check only reports after every suite the change needs has passed, so a pull request that
103+
needs the full matrix cannot merge until it has run.
104+
97105
For focused iteration, run the narrowest relevant pytest target, for example:
98106

99107
```bash

0 commit comments

Comments
 (0)