Skip to content

Commit 4070f12

Browse files
authored
feat: column lineage, worker ad-hoc commands and UI pages (#1384)
* fix: render SQL Server table functions as inline TVFs and list warehouse Recall relations portably SQL Server's render_create_function stripped a table function's SELECT and wrapped it in BEGIN RETURN (...) END, which the engine rejects. A TABLE return now renders RETURNS TABLE AS RETURN (<query>). sqb recall schema --backend warehouse matched state relations with LIKE '\_sqlbuild\_%' ESCAPE '\', which ClickHouse cannot parse. A prefix comparison with SUBSTRING needs no escape in any dialect. * feat(lint): warn when SQL lint is unavailable for a dialect A native lint response can report that the engine does not parse the project's dialect yet. Each such file now gets one L007 warning carrying the engine's reason, instead of looking lint-clean. A non-string reason is rejected as an invalid response. * fix(sqlserver): quote schema names in EXEC, end MERGE with a semicolon and refuse WITH source expressions render_create_schema doubled quotes only in the sys.schemas lookup, so a schema name with a quote broke the EXEC string. render_merge now ends with the semicolon SQL Server requires. A source expression starting with WITH cannot sit in a derived table on SQL Server, so it now fails with A715 and a rewrite hint instead of a warehouse syntax error. * refactor(analysis): read B101 spellings from a recorded table instead of SQL library probes The unknown-function check (B101) asked the SQL library's parser which built-in node each unaccepted call spelling becomes, and took DuckDB's accepted names from the library's function catalogue. The check now reads a table of the 41 rejected spellings, each with its argument count and the spelling the dialect accepts. Every DuckDB row was confirmed unknown to DuckDB 2.0.0. The call scan takes the analysis dialect name, so analysis off the library can run it unchanged. Findings are identical on every DuckDB and Snowflake query of the compiler differential corpus and on every candidate spelling at 0 to 9 arguments. The workspace no longer depends on the library's function-catalogue crate directly; resolved features are unchanged. * feat(lint): keep L007 out of every exit decision and show it in compile and rules run compile and rules run kept only selected Rule codes, so an unavailable dialect looked clean there. L007 now passes through as a warning-severity Rule diagnostic in compile and is left out of rules run's failing-finding count. counts_toward_exit owns the exemption. * fix(lint): keep L007 out of rule exceptions and the plan rules gate The plan and build rules gate failed on any finding, so an L007-only project could not plan. Rule exceptions also validate every finding's code as a rule code and rejected L007 with an invalid-lifecycle error. The gate now counts only findings that count toward exit, and L007 passes through rule exceptions unsuppressed. An end-to-end test replays a cached finding as L007 and as a rule code through compile, rules run, plan and build. * docs(rules): name plan and build, not format, in the L007 exit note * test(orchestration): count the store statements a wide catalog listing issues A same-run time ratio cannot see one store round trip per node: that cost is linear. Tests now count the statements each task issues, and the wide listing must read builds 100 nodes per statement: 17 statements for 1,000 nodes and 37 for 3,000 on SQLite and PostgreSQL, 1,008 with one node per statement. The ratio check stays and now guards super-linear growth only. * refactor(analysis): share diagnostic location and the catalog's options view * feat(contracts): prove dynamic pivots from SQLQuack parse trees The dynamic column contract proof parses pivot SQL with SQLQuack's shared parser (DuckDB, MotherDuck, Snowflake) instead of Polyglot. DuckDB trees match Polyglot's exactly; a Snowflake aggregate keeps its argument in a call's argument list, which the proof now reads. Other dialects keep the existing unsupported-dialect message. * fix(contracts): refuse pivot SQL nested deeper than 512 levels before serialising it * wip: read rule facts with SQLQuack * wip: locate cast types and Snowflake date parts without a generator * wip: report built-in rules unavailable for unparsed dialects * wip: e2e for built-in rules on unparsed dialects * chore(deps): bump sqlquack to a9dde6a0 * wip: docs and tests for built-in rules on unparsed dialects * wip: format tests * fix(rules): refuse rule facts for statements deeper than 512 levels * fix(rules): anchor nested comparisons and Snowflake LAST_DAY date parts * feat(scenario): map scenario column types with SQLQuack's type parsers Local scenario capture no longer asks Polyglot to read warehouse column types. Postgres types go through sqlquack-syntax-postgres's type parser, and Snowflake and BigQuery types through their SQLQuack cast-target parse, behind one native function (parse_warehouse_type). DuckDB and MotherDuck spellings pass through unchanged. Databricks nested types map recursively to DuckDB's LIST, MAP and STRUCT, and a nested spelling that cannot be mapped is refused with a named message instead of falling back to JSON. Other types keep their integer width or fall back to VARCHAR. * fix(scenario): map SQL Server date, time, uuid and money types locally SQL Server has no SQLQuack type parser, so its information_schema names reached the generic fallback, which held datetime2, smalldatetime, datetimeoffset and uniqueidentifier as VARCHAR. A SQL Server pre-mapper now maps them to TIMESTAMP, TIMESTAMPTZ and UUID, bit to BOOLEAN, money and smallmoney to exact decimals, and SQL Server's 8-byte float to DOUBLE. The fallback keeps 4-byte floats (FLOAT, FLOAT4, REAL) as FLOAT instead of widening them to DOUBLE. A panic in the warehouse type parser now surfaces as the ValueError capture already falls back on. * chore(fensu): allow the cast-target walk next to the other JSON tree walks * refactor(rules): name the date-part and unavailable-rule loops * docs(analysis): keep the warehouse type binding's doc comment to one line * feat(types): normalize adapter types without Polyglot Read each type with SQLQuack's parser for the dialect, or SQLBuild's own type reader where a dialect has none, and spell it the way the Polyglot-based normalization did so the same aliases stay equal. normalize_type now raises only ValueError, for an unknown dialect. * fix(ui): stop a burst of control API requests on every live refresh The person trigger listing now carries each trigger's newest tick, so the jobs and triggers pages no longer read ticks once per trigger. Pages coalesce change notices into one reload, and the HTTP client shares an in-flight listing between pages that refresh together. * fix(types): read only shipped dialects and ClickHouse nested parameters MotherDuck now reads and spells types as DuckDB. MySQL and the other SQL analysis dialects no adapter ships raise Unknown dialect instead of reading as generic SQL. Under ClickHouse the type reader keeps nested type arguments, enum members with values, and named tuple fields. * refactor(rules): read the shared-parser dialect from one lookup * chore(deps): bump sqlquack to 19c51fa2 sqlquack main 19c51fa2 carries the Postgres parser work and the train that followed a9dde6a0. sqlquack-syntax-postgres now depends on stacker, which re-resolves a few Windows-only windows-sys edges in the lock. * build(deps): pin SQLQuack before its Snowflake grammar checks SQLQuack's next commit refuses a bare call in a Snowflake FROM clause, which rejects the __ref and __source placeholders that compiled model SQL keeps, so custom-rule syntax trees fail on Snowflake models. This rev keeps the BigQuery parser the warehouse type parsers need. * fix(rules): read relation markers as tables in custom-rule syntax trees Compiled query_sql keeps relation markers such as __ref("orders"). Snowflake does not accept a bare function call in FROM, so newer SQLQuack parsers refuse every Snowflake model that references another. A shared helper in sqlbuild-analysis (relation_markers::markers_as_names) now writes each marker call in code as a short indexed name, padded so all later text keeps its line and column, and returns the calls by index. sql_tree reads SQL through it for every dialect. The rules crate's dialect lexical syntax moves to sqlbuild-sqltext so both use one mapping. * build(deps): keep Windows windows-sys edges at main's resolution Adding stacker through the SQLQuack crates re-resolved errno, rustix, tempfile, junction, quinn-udp and winapi-util to older windows-sys releases. On Windows the extension build then compiled windows-sys 0.52 with features the sqb build lacks. The lockfile keeps main's edges, so both builds share one windows-sys again. * docs(types): keep the ClickHouse type parameter doc comment to one line * feat(ui): wire the lineage and changes pages to the live control API Add a person read of a target's published snapshots with the subjects each one changed, behind the same listing authorisation as jobs. The Changes page reads it, and the Lineage page builds its graph and inspector from the catalog listing, so neither page needs mock data. * feat(server): page the security audit newest first with actor names Person reads of the security audit accept an exclusive upper cursor that returns the newest entries first, and name each principal actor. Exports keep their payload unchanged. * feat(ui): wire the audit log to the security audit read The audit log reads the newest security facts first, lists a denial under the action it refused with its decision ID and reason, and names principal actors. Its actor and action filters come from the entries, and links to the person page and access checker stay hidden until the people read exists. * feat(server): list people and role holdings for administrators Add GET /api/admin/people, which needs user.manage everywhere, and GET /api/admin/roles, which needs role.assign everywhere. People carry sign-in facts only: password presence, SSO groups, live session count and last sign-in. Role holdings list person assignments and SSO group mappings. * feat(ui): wire the people page to the people and role reads The people page lists everyone with their highest role, its source and last sign-in. A person's sign-in source is shown only when the server knows it. Links to the person page and access checker, and the risky actions filter, stay hidden until their reads exist. * fix(server): count no active sessions for a disabled person Disabling a person blocks their sessions without revoking them, so the people read no longer counts those sessions as active. * fix(contracts): read relation markers as relations in dynamic pivot proofs Newer SQLQuack Snowflake parsers refuse a bare __ref(...) or __source(...) in FROM, so the proof parses the model SQL through the shared markers_as_names helper and renames each dependency marker's table or column node back to the relation its call names. * refactor(analysis): share the statement depth guard between pivot proofs and rule facts Both crates carried the same iterative deeper_than; sql_tree now owns it. The deep pivot test asserts its expected outcome directly. * fix(contracts): keep non-dependency markers out of pivot relations A __dbt_ref passthrough named its internal placeholder in the failure reason; it now fails as before. The marker scan reads comments and strings with the dialect the proof parses, so MotherDuck scans as DuckDB. * fix(types): spell ClickHouse Nullable and BINARY as the engine does Under ClickHouse, Nullable(T) is read as a wrapper, so an alias inside it is respelled: Nullable(INT) is Nullable(Int32), as bare INT is. BINARY(n) spells as FixedString(n). Bare BINARY, and a Nullable that wraps another Nullable or a nested type, are refused as the engine refuses them. * feat(server): let people read a target's location health Add GET /api/control/locations/{location_id}/targets/{target}/health behind the same listing gate as jobs. It returns the target's newest health observations and instance check-ins, shared with the machine read through one helper. * feat(ui): show live location health on the Locations page Health comes from each location's check-ins and observations: no check-in for 90 seconds is broken, a failed compile at or after the connected generation is falling back, and an unhealthy connection is degraded. People without listing view see the health as unknown. Server status, detail links and the deploy policy column appear only when supported. * fix(server): limit a target's health read to its own observations The person health read filtered by target only after the location's newest 100 observations were taken across all targets, so a busier target could push out another target's compile or connection reports. The target filter now applies before each limit. * feat(server): let people read live platform problems Add GET /api/control/platform/problems for the UI's alert banner. It derives problems at read time with the watchdog's thresholds: whether the scheduler has a leader, location targets whose instances all missed two check-ins, and running schedules more than 30 seconds late. Broken locations and late ticks appear only for targets the person lists, and access is checked per target before that target's ticks are read. * feat(ui): show live platform problems in the alert banner The banner reads the new problems endpoint every 30 seconds, because missed check-ins and late ticks come from time passing, not change notices. A stopped scheduler and each broken location are critical, and late schedules are grouped per location. The read now names each location and dates a stopped scheduler from its last lease. Links go to the Locations list until location detail pages are supported. * docs(ui): keep the locations doc comment on listLocations * feat(server): let role administrators read a target's role definitions Add GET /api/admin/locations/{location_id}/targets/{target}/roles behind server-wide role.assign, like the role holdings read. It returns the built-in roles with their actions, then the target's published custom roles with their description, grants and the nodes each selector matched, and how many active people hold each role there. * feat(ui): show role definitions on the Roles pages The Roles list and role pages read the new role definitions endpoint. Built-in roles show their actions on every node, and custom roles show their published grants, the nodes each selector matched, the published definition and who holds them on this location. Holder counts include SSO group members, counted once per person. The SSO mappings tab and the base-role comparison stay hidden until the server provides them. * feat(ui): show warehouse sync status in the top bar The top bar reads the target's warehouse sync reports from the health read: when a table last synced cleanly, and which tables' newest sync failed and at what stage. It rereads every 30 seconds with the alert banner, because sync reports send no change notice. * fix(ui): stop rereading warehouse sync after a refused read Every refused health read writes an audit denial, so a person who sees a location but may not list all of its nodes would add one each poll. The shell now asks once per target per page load and keeps the indicator at its unknown state after a refusal. * fix(rules): wrap Snowflake table functions in TABLE( ) before parsing The rules engine stubs __table_fn calls before parsing a model. SQLQuack's Snowflake grammar, like Snowflake, accepts a table function in FROM only inside TABLE( ), which is how SQLBuild renders the call, so a bare stub made every built-in Rule report the model unevaluated. Wrap the stub the same way unless the model already wrote TABLE( ), and assert in the Snowflake evaluation tests that the model was actually evaluated. * fix(rules): read Snowflake table function markers inside TABLE( ) in rule trees Custom Rules parse a model through the rule tree, which renames each relation marker to a placeholder name first. SQLQuack's Snowflake grammar refuses the bare table function call that leaves, so a custom Rule on a Snowflake model with a table function failed to parse it. Wrap those calls in TABLE( ) over the marker's own padding and a following space, so node positions keep their line and column, and share the TABLE( ) detection with the built-in Rules' stubbing. * feat(ui): show catalog node detail from the control API and finish lineage labels The catalog node page, the lineage inspector and node links across Catalog, Checks, Changes and Lineage now load from the catalog listing, node status, node SQL and check reads. SQL is read only where the location grants view.sql, so viewers cause no refused, audited reads. Graph cards show the node name without its kind prefix. Sections the server does not report (run history, jobs, history, drift without built SQL, ad-hoc runs) are hidden instead of claiming there are none. * fix(ui): ask for refused node SQL once until reload view.sql in a location's bootstrap actions may hold on only some nodes, so the node SQL read can still be refused. Each refusal is audited and node pages refetch on change signals, so remember the refused node and skip its SQL read until the page reloads. * fix(rules): keep columns after a wrapped table function in rule trees When no space or tab follows a Snowflake table function call, the rule tree inserts its closing parenthesis. Write the TABLE( ) wrapper over one more padding space in that case, so text after the call keeps its column. * fix(ui): wrap SQL previews and drop the kind prefix from SQL file labels Preview lines scrolled sideways with no visible scrollbar, so long lines looked cut off; previews now wrap and the full viewer still scrolls. SQL file labels show the node name, like the graph cards. * feat: publish column lineage and show it on catalog node pages Location publication now adds a lineage snapshot document: the compiler's FAST column lineage edges for every model, as source and target node keys, columns and transform kind. A person read returns the edges reaching or leaving one node, with each node's inferred columns, behind the same view-on-every-node gate as the catalog listing. It carries no SQL or expressions, which need view.sql. The catalog node page's Column lineage tab maps the read onto the existing column lineage view. * fix(ui): show full names and transform kinds in column lineage Traced identifiers and node names wrap instead of truncating, column types stay on one line, and transformed edges name the compiler's transform kind, such as Cast or Aggregation. * refactor: move column lineage to its own change Column lineage publication would push large projects past the publish body limit, so it lands after the publish request is compressed. This change keeps node detail, labels and the SQL preview fixes. * feat: show per-node run history and selecting jobs on catalog node pages A person read lists the scheduler runs that built one node in a target, newest first, behind the same view-on-every-node gate as each run's node list, and carries the same metadata. The catalog node page fills its Runs tab, recent build strip and last duration from it, and lists the jobs and groups whose selection includes the node from the existing jobs and triggers reads. The catalog list drops the kind prefix from node names, and the runs list gives its Started column room for queued times. * feat(orchestration): gzip location snapshot publishes A location now sends its snapshot with Content-Encoding: gzip. The server keeps the 64 MiB wire bound and, after authenticating the location token, streams the body through a 512 MiB decompressed-size bound into the JSON parser. Encodings other than gzip get 415, and a body over either bound gets 413. A server that refuses the encoded body with 400 or 415 gets the snapshot again as plain JSON, so new locations still publish to older servers. * feat: show run events, statements, SQL and output to people Person reads serve a run's Recall rows, scoped to its scheduler and claimed worker invocations. Event and statement metadata need view on every node of the target, as the run's node list does, and leave out event payloads, which can carry SQL and error text. Statement SQL also needs view.sql, and captured output with invocation and diagnostic error text also needs view.evidence, on every node the run selected; both are audited like node SQL and check evidence. The run page's Statements, Events and Output tabs read them. The UI asks for SQL and output only where the bootstrap grants them and stops asking for a run after a refusal until reload, so viewers do not repeat audited denials. SQL blocks wrap instead of clipping. * feat: require run.prioritise for runs above priority 0 A run's priority decides which queued run is claimed first, and any person allowed to run a job or rerun could set any priority. A person's job run or rerun above priority 0 now also needs run.prioritise on the run's nodes. Only Admin grants it among built-in roles, project roles may grant it, and it has no default role. Priority 0 and below need nothing extra, and trusted local launches are unchanged. * fix(orchestration): report the server's refusal of a snapshot too large to resend plain A snapshot whose plain JSON exceeds the wire bound can never be accepted uncompressed, so a 400 for the gzip body is returned as is instead of being masked by a second upload that ends in 413. * fix(orchestration): bound decompressed publish bodies at 320 MiB A parsed snapshot holds about 7.6 times its JSON size on the server, so a body just under 512 MiB peaked at 3.8 GiB RSS. 320 MiB keeps four times the headroom of a 10,000-model project with column lineage. * fix: keep run cancellation independent of run.prioritise Only queueing a job run or a rerun above priority 0 needs run.prioritise. Cancelling an urgent run keeps needing only run.cancel. * test: wait for every trigger to reach the latest due time after scheduler fault recovery The scheduler ticks only the latest missed occurrence, so a trigger first evaluated after the next minute begins ticks that minute and skips the earlier one. When the final recovery drain crossed a minute boundary on a slow runner, the fault test counted those triggers as missing their latest tick, and its duplicate probe at that minute was accepted for the same reason. Recovery now cycles until every trigger has ticked at or after the latest due time at settling, bounded by a five-minute deadline. Each trigger's newest tick must reach that time, and the duplicate probe re-inserts a time the trigger actually ticked. The duplicate, off-schedule, run, event and fencing checks are unchanged. * test: count ticks scheduled after their own evaluation as off schedule Requiring each trigger's newest tick to be at or after the latest due time has no upper bound, so a scheduler that ticked the next occurrence early would pass. Each tick records the clock it was planned with, and no tick may be for a later time. * ci: run the full Verify matrix only on pull requests labelled full-verify Every pull request push ran the whole matrix: wheels on five platforms, Windows and macOS jobs, DuckDB end-to-end shards, warehouse, compiler golden and timing jobs, for about two hours of the organisation's shared job slots. Pushes now run a fast Linux core: static checks, unit and integration tests, native workspace checks and the UI checks. The rest runs when the pull request is labelled full-verify and is not a draft, and always on merge groups and manual runs. A run that leaves out suites its change needs reports its aggregate as a fast-core check, so the required Verify stays missing and blocks the merge; a full run fails Verify when any selected suite was skipped. Changes confined to the web UI skip the Python and native checks they cannot affect. The aggregate now runs unless the run is cancelled, so a cancelled run no longer queues it for a runner. A label other than full-verify starts a run in its own concurrency group that verifies nothing and reports under another name, so it cannot cancel or satisfy the real run. * feat: run ad-hoc sqb commands from the web UI People check and launch `sqb build` from the run panel. The server resolves the selection against the current publication, authorises run.adhoc plus run.warehouse, run.prioritise and history-rewriting flags, and refuses any flag outside the ad-hoc allowlist by name. The dry run reads only and records nothing. Workers refuse an ad-hoc run whose warehouse the location does not configure for the target, before any statement runs; published jobs keep their authored warehouse. * ci: keep the Verify aggregate on always() The release gate requires the aggregate to run with always(): a job skipped when a run is cancelled would report Verify as skipped, which satisfies a required check. Stale runs are force-cancelled instead. * refactor(orchestration): page security audit facts beside the audit writes Reading audit pages authorizes nothing; it reads the facts the credentials audit module appends and prunes. Together the administrator role reads and the ad-hoc scope decisions put 21 modules in authorization/main, one over the container limit. * test(ci): prove a fast-core or label-only Verify run cannot satisfy the merge gate The release gate now names the aggregate with the label and fast-core facts it reads, and requires that a pull request whose fast core passed without the full matrix, or whose only run followed an unrelated label, stays blocked. The aggregate tests cover full runs with a selected suite skipped, fast-core runs and label-only runs. * feat: publish column lineage and show it on catalog node pages Locations publish the compiler's column lineage with each snapshot, and a catalog node page shows the columns that feed and read the node. The read needs view on every current node, like the catalog listing, returns only node keys, kinds, column names and types and the transform kind, and loads only the definitions its edges touch, in bounded batches. Snapshot node keys now come from one shared read. * fix(lineage): publish snapshot column lineage from native edge records Reading the edges getter materialised every edge as a Python object and tripped the strict native-field guard. The snapshot now reads the compact edge records directly, with the same rows in the same order. * fix(ui): find an ad-hoc run launched while a runs listing is in flight The ad-hoc launch looked up its new run through the shared runs listing without clearing it, so a listing started before the launch could answer without the run and report it missing while it was already queued. The launch now clears shared listings after its write, as a job launch does. * feat: run test, seed, audit and freshness ad-hoc commands on workers Workers now run every verb the ad-hoc planner accepts, not only build. - The planner checks each verb's settings against its own parser. Seed runs keep only the selected seeds. Freshness runs take the selected sources and the sources upstream of the selection, less those the exclusions reach. - Runs freeze their command. The worker rebuilds that command's argv from the frozen settings and selection, and refuses a parse that dispatches anything else. - Test and audit runs report each SQL test or audit the snapshot publishes as a check, and each selected node takes the worst result of its checks. Seed runs refuse seeds outside the frozen write set before connecting. Freshness runs report each source. - Only build and seed have a write set, so test, audit and freshness runs take no write claims. * fix: run every ad-hoc verb from the worker's verified project Test, audit, seed and freshness runs re-read the location's checkout after the worker verified it against the frozen snapshot. A checkout that changed in that window could run SQL, audits or seed contents that were never verified. Every verb now runs from the verified inputs, as build already did. A test or audit run that was refused, or that raised, no longer records an unchecked node as successful. Such nodes are recorded as skipped. * fix(ui): show the approved logos and favicon from the website The web UI shipped older copies of the SQLBuild logo and favicon. It now imports the website's approved files through a $website alias, so a branding change reaches both, and a unit test rejects image copies or literal logo paths under ui/. * ci: run the UI checks when the website logos or favicon change The UI now imports the website's logo and favicon files, so renaming or moving them breaks the UI build. Route those three paths to the fast UI job. * style: move the checks-only outcome note into its doc comment * feat(ui): show the Overview from live runs, checks, models and deploys The Overview, where every sign-in lands, ran only on mock data, so the server showed "Not available" there. It now reads the runs, checks, catalog nodes, publications and platform problems the person can already list: runs and outcomes in the last 24 hours, failed and unresponsive runs, check outcomes, deployed models and this week's model changes, and recent deploys. The server has no project-wide build state, cost or group read yet, so stale counts, cost and group status stay hidden rather than showing zeros, as does the activity timeline. Recent deploys show the short commit. * feat(ui): tell a partial viewer why the Overview is empty A person with view on only part of a target signed in to a page error, because every listing the Overview reads needs view on the whole target. The client now answers null for that refusal and the page explains it, naming the target and linking the pages that do not list it. Other errors still show as errors, and no partial counts are shown. * fix(ui): show Overview errors and drop a stale partial-view message The partial-view message used whatever data the last load left, so after switching location it named the new target while that target loaded, and a failed load never showed its error. The message now needs a settled, error-free load, and errors show in every state. * feat(ui): show renames, old names and clean-ups on the Maintenance page The Maintenance page ran only on mock data. It now reads the synced maintenance history the person can list for the target: renames and moves, the old-name views kept for them with each recorded step, and clean-up archives and deletes. Each old-name view takes every fact from the newest step that records it; a relation recorded without its schema shows by its bare name. * fix: regenerate the TypeScript control client for ad-hoc run commands * ci: check the generated control client in make check-ci The client check ran only as its own Verify step, so a stale client passed make check-ci locally. check-ci now runs it, and Verify runs it through check-ci after installing the packages. * feat(ui): give and remove roles and disable people from the admin pages The person page's role and account controls had no server calls behind them. Giving a role, removing a manual one, and disabling or enabling a person now use the server's existing admin endpoints, then read the result back. The add control offers only what the server accepts: built-in roles for every location, a location's own roles there, and admin only for every location. The page also hides its effective-access and access-checker panels and the audit "Check this" links where the server has no read for them, and long audit subjects wrap. * feat(ui): test a location's warehouse connection from the Locations page Each location row now has Test connection for people who may test that location. The UI asks the location through the server's connection-test request, then reads the result until the location answers or the request expires, and shows whether it connected. The location reports no reason, so a failure points to sqb debug in the location.
1 parent e0136b6 commit 4070f12

93 files changed

Lines changed: 2678 additions & 391 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/verify.yml‎

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -174,9 +174,10 @@ jobs:
174174
;;
175175
esac
176176
case "${path}" in
177-
ui/*|clients/typescript/*|.github/workflows/verify.yml)
177+
ui/*|clients/typescript/*|.github/workflows/verify.yml|website/src/assets/logo-light.png|website/src/assets/logo-dark.png|website/public/favicon.png)
178178
# The web UI's own checks and unit tests; the wheels embed its build. It compiles
179-
# against the generated control client, so client changes run them too.
179+
# against the generated control client, so client changes run them too, and it
180+
# imports the website's logos and favicon, so moving them fails its build.
180181
ui=true
181182
;;
182183
esac
@@ -357,11 +358,6 @@ jobs:
357358
node-version: "22"
358359
cache: npm
359360
cache-dependency-path: clients/typescript/package-lock.json
360-
- name: Check generated control client against the built native API
361-
run: |
362-
npm --prefix clients/typescript ci
363-
uv run --no-sync npm --prefix clients/typescript run check
364-
uv run --no-sync npm --prefix clients/typescript test
365361
- name: Fetch the base for the new-duplicate check
366362
run: git fetch --no-tags --depth=1 origin "$BASE_REF"
367363
env:

‎Makefile‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
SHELL := /bin/bash
22

3-
.PHONY: verify verify-quick verify-pg coverage cli-preview rust-check check-native-layering check-dupes-new check-native-stub check-build-features check-python-boundary check-release-gate check-docs-skill \
3+
.PHONY: verify verify-quick verify-pg coverage cli-preview rust-check check-native-layering check-dupes-new check-native-stub check-build-features check-python-boundary check-release-gate check-docs-skill check-control-client \
44
check-e2e-shards test-e2e-duckdb test-e2e-duckdb-build-core \
55
test-e2e-duckdb-build-incremental \
66
test-e2e-duckdb-cli-data test-e2e-duckdb-cli test-e2e-duckdb-inspection \
@@ -706,14 +706,21 @@ verify-quick:
706706
exit $$status
707707

708708

709-
check-ci: check-dupes-new check-e2e-shards check-native-layering check-native-stub check-build-features check-python-boundary check-release-gate check-docs-skill rust-lint
709+
check-ci: check-dupes-new check-e2e-shards check-native-layering check-native-stub check-build-features check-python-boundary check-release-gate check-docs-skill check-control-client rust-lint
710710
uv run ruff format --check .
711711
uv run ruff check .
712712
uv run ty check src tests
713713
uv run pytest tests/unit/src/sqlbuild/adapter/contract/classes/strict_adapter/test_strict_adapter.py -q
714714
uv run fensu check
715715

716716

717+
# The generated TypeScript control client must match the OpenAPI document of the installed sqb.
718+
check-control-client:
719+
npm --prefix clients/typescript ci
720+
uv run --no-sync npm --prefix clients/typescript run check
721+
uv run --no-sync npm --prefix clients/typescript test
722+
723+
717724
check-native-layering:
718725
uv run python -m scripts.check_native_layering
719726

‎clients/typescript/src/schema.d.ts‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -338,6 +338,22 @@ export interface paths {
338338
patch?: never;
339339
trace?: never;
340340
};
341+
"/api/control/locations/{location_id}/targets/{target}/column-lineage": {
342+
parameters: {
343+
query?: never;
344+
header?: never;
345+
path?: never;
346+
cookie?: never;
347+
};
348+
get: operations["readPersonColumnLineage"];
349+
put?: never;
350+
post?: never;
351+
delete?: never;
352+
options?: never;
353+
head?: never;
354+
patch?: never;
355+
trace?: never;
356+
};
341357
"/api/control/locations/{location_id}/targets/{target}/connection-tests": {
342358
parameters: {
343359
query?: never;
@@ -1334,6 +1350,21 @@ export interface components {
13341350
CliLoginStartRequest: {
13351351
client_hash: string;
13361352
};
1353+
/** @description One published column lineage edge; `transform` is the compiler's transform kind, such as `direct` or `cast`. */
1354+
ColumnLineageEdge: {
1355+
source_column: string;
1356+
source_key: string;
1357+
target_column: string;
1358+
target_key: string;
1359+
transform: string;
1360+
};
1361+
/** @description The column lineage reaching or leaving one node, from the current snapshot; empty before one publishes lineage. */
1362+
ColumnLineageRead: {
1363+
edges: components["schemas"]["ColumnLineageEdge"][];
1364+
nodes: components["schemas"]["LineageNodeColumns"][];
1365+
/** Format: uuid */
1366+
snapshot_id?: string | null;
1367+
};
13371368
/** @description A row whose key was already stored with different content; the first row is kept. */
13381369
ConflictEvidence: {
13391370
existing_sha256: string;
@@ -1472,6 +1503,16 @@ export interface components {
14721503
target: string;
14731504
window?: null | components["schemas"]["RunWindow"];
14741505
};
1506+
LineageColumn: {
1507+
data_type?: string | null;
1508+
name: string;
1509+
};
1510+
/** @description A node's columns as its compiled definition infers them; `data_type` is absent when inference could not tell. */
1511+
LineageNodeColumns: {
1512+
columns: components["schemas"]["LineageColumn"][];
1513+
kind: string;
1514+
node_key: string;
1515+
};
14751516
/** @description One location target the person can view, with the actions allowed on at least one of its nodes. */
14761517
LocationAccess: {
14771518
actions: components["schemas"]["Action"][];
@@ -1989,6 +2030,11 @@ export interface components {
19892030
claim_id: string;
19902031
request: components["schemas"]["RunRequest"];
19912032
};
2033+
/**
2034+
* @description The `sqb` command a run executes; job and group runs always build.
2035+
* @enum {string}
2036+
*/
2037+
RunCommand: "build" | "test" | "seed" | "audit" | "freshness";
19922038
RunErrorText: {
19932039
error_type?: string | null;
19942040
invocation_id: string;
@@ -2121,6 +2167,7 @@ export interface components {
21212167
};
21222168
/** @description The authorized selection and pinned execution inputs are immutable once requested. */
21232169
RunRequest: {
2170+
command?: components["schemas"]["RunCommand"];
21242171
concurrency_groups: string[];
21252172
image_id?: string | null;
21262173
job_key?: string | null;
@@ -3379,6 +3426,46 @@ export interface operations {
33793426
};
33803427
};
33813428
};
3429+
readPersonColumnLineage: {
3430+
parameters: {
3431+
query: {
3432+
/** @description The node whose upstream and downstream column lineage to return */
3433+
node_key: string;
3434+
};
3435+
header?: never;
3436+
path: {
3437+
location_id: string;
3438+
target: string;
3439+
};
3440+
cookie?: never;
3441+
};
3442+
requestBody?: never;
3443+
responses: {
3444+
200: {
3445+
headers: {
3446+
[name: string]: unknown;
3447+
};
3448+
content: {
3449+
"application/json": components["schemas"]["ColumnLineageRead"];
3450+
};
3451+
};
3452+
/** @description Session rejected */
3453+
401: {
3454+
headers: {
3455+
[name: string]: unknown;
3456+
};
3457+
content?: never;
3458+
};
3459+
403: {
3460+
headers: {
3461+
[name: string]: unknown;
3462+
};
3463+
content: {
3464+
"application/json": components["schemas"]["ApiErrorBody"];
3465+
};
3466+
};
3467+
};
3468+
};
33823469
requestConnectionTest: {
33833470
parameters: {
33843471
query?: never;

‎crates/sqlbuild-cli/src/adhoc/_helpers/planner.rs‎

Lines changed: 75 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,10 @@ use sqlbuild_orchestration::runs::models::{CommandPlan, PlannerResource};
1111
use sqlbuild_orchestration::runs::types::CommandPlanner;
1212

1313
use crate::adhoc::constants::{
14-
ADHOC_VERBS, ALLOWED_DESTS, BUILD_VERB, COMMAND_DEST, INTEGER_CONVERTERS, LOAD_SOURCES_DEST,
15-
POSITIVE_INTEGER_CONVERTER, REFUSED_DESTS, SELECTOR_DESTS,
14+
ADHOC_VERBS, ALLOWED_DESTS, BUILD_VERB, COMMAND_DEST, FRESHNESS_VERB, INTEGER_CONVERTERS,
15+
LOAD_SOURCES_DEST, POSITIVE_INTEGER_CONVERTER, REFUSED_DESTS, SEED_VERB, SELECTOR_DESTS,
1616
};
17+
use crate::argv::main::command_settings_arguments::command_settings_arguments;
1718
use crate::argv::main::job_settings_arguments::job_settings_arguments;
1819
use crate::argv::main::parse_cli_arguments::parse_cli_arguments;
1920
use crate::argv::models::{ArgvSettings, Namespace, TypeFailure, Value};
@@ -52,13 +53,29 @@ impl CommandPlanner for CliPlanner {
5253
}
5354
if verb == BUILD_VERB {
5455
job_settings_arguments(&settings)?;
56+
} else {
57+
command_settings_arguments(verb, &settings)?;
5558
}
5659
let mut select: Vec<String> = selectors.remove("select").unwrap_or_default();
5760
select.extend(selectors.remove("positional_select").unwrap_or_default());
5861
let exclude: Vec<String> = selectors.remove("exclude").unwrap_or_default();
62+
let selection: BTreeSet<String> = match verb.as_str() {
63+
SEED_VERB => {
64+
let seeds: BTreeSet<String> = resolve(resources, &select, &exclude)?
65+
.into_iter()
66+
.filter(|key| key.starts_with("seed:"))
67+
.collect();
68+
if seeds.is_empty() {
69+
return Err("the selectors match no seeds in the current publication".into());
70+
}
71+
seeds
72+
}
73+
FRESHNESS_VERB => freshness_sources(resources, &select, &exclude)?,
74+
_ => resolve(resources, &select, &exclude)?,
75+
};
5976
Ok(CommandPlan {
6077
verb: verb.clone(),
61-
selection: resolve(resources, &select, &exclude)?,
78+
selection,
6279
settings,
6380
})
6481
}
@@ -163,18 +180,69 @@ fn resolve(
163180
resources: &[PlannerResource],
164181
select: &[String],
165182
exclude: &[String],
183+
) -> Result<BTreeSet<String>, String> {
184+
let selected: BTreeSet<String> = resolved(resources, select, exclude)?;
185+
if selected.is_empty() {
186+
return Err("the selectors match no nodes in the current publication".into());
187+
}
188+
Ok(selected)
189+
}
190+
191+
fn resolved(
192+
resources: &[PlannerResource],
193+
select: &[String],
194+
exclude: &[String],
166195
) -> Result<BTreeSet<String>, String> {
167196
let graph_resources: Vec<GraphResource> = resources.iter().filter_map(graph_resource).collect();
168197
let graph = build_project_graph(&graph_resources);
169-
let selected: BTreeSet<String> = resolve_selectors(&graph, select, exclude)
198+
Ok(resolve_selectors(&graph, select, exclude)
170199
.map_err(|error| format!("{}: {}", error.code, error.message))?
171200
.into_iter()
172201
.map(|(kind, name)| format!("{kind}:{name}"))
202+
.collect())
203+
}
204+
205+
/// Sources `sqb freshness` observes: selected or upstream sources, less those the exclusions reach.
206+
fn freshness_sources(
207+
resources: &[PlannerResource],
208+
select: &[String],
209+
exclude: &[String],
210+
) -> Result<BTreeSet<String>, String> {
211+
let selected: BTreeSet<String> =
212+
upstream_sources(resources, &resolved(resources, select, &[])?);
213+
let excluded: BTreeSet<String> = if exclude.is_empty() {
214+
BTreeSet::new()
215+
} else {
216+
upstream_sources(resources, &resolved(resources, exclude, &[])?)
217+
};
218+
let sources: BTreeSet<String> = selected.difference(&excluded).cloned().collect();
219+
if sources.is_empty() {
220+
return Err("no sources are selected or upstream of the selection".into());
221+
}
222+
Ok(sources)
223+
}
224+
225+
fn upstream_sources(resources: &[PlannerResource], keys: &BTreeSet<String>) -> BTreeSet<String> {
226+
let deps: BTreeMap<&str, &[String]> = resources
227+
.iter()
228+
.map(|resource| (resource.key.as_str(), resource.deps.as_slice()))
173229
.collect();
174-
if selected.is_empty() {
175-
return Err("the selectors match no nodes in the current publication".into());
230+
let mut reached: BTreeSet<&str> = BTreeSet::new();
231+
let mut pending: Vec<&str> = keys.iter().map(String::as_str).collect();
232+
while let Some(key) = pending.pop() {
233+
if reached.insert(key) {
234+
pending.extend(
235+
deps.get(key)
236+
.into_iter()
237+
.flat_map(|deps| deps.iter().map(String::as_str)),
238+
);
239+
}
176240
}
177-
Ok(selected)
241+
reached
242+
.into_iter()
243+
.filter(|key| key.starts_with("source:"))
244+
.map(str::to_owned)
245+
.collect()
178246
}
179247

180248
/// A published node as the selector graph sees it; keys without a kind are not selectable.

‎crates/sqlbuild-cli/src/adhoc/constants.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,12 @@ pub(crate) const ADHOC_VERBS: [&str; 5] = ["build", "test", "seed", "audit", "fr
66
/// The verb whose settings must also form valid job settings.
77
pub(crate) const BUILD_VERB: &str = "build";
88

9+
/// The verb that runs only the seeds of its selection.
10+
pub(crate) const SEED_VERB: &str = "seed";
11+
12+
/// The verb that observes sources, including those upstream of selected models.
13+
pub(crate) const FRESHNESS_VERB: &str = "freshness";
14+
915
/// The parser destination holding the verb itself, which is not a setting.
1016
pub(crate) const COMMAND_DEST: &str = "command";
1117

‎crates/sqlbuild-cli/src/adhoc/tests/test_adhoc_planner.rs‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,6 +180,34 @@ fn given_ad_hoc_commands_when_planned_then_flags_follow_the_allowlist_and_select
180180
command: &["test", "-s", "orders", "--case", "happy_path"],
181181
expected_plan: Ok((&["model:orders"], json!({"case": "happy_path"}))),
182182
},
183+
AdhocPlanTestCase {
184+
description: "audit settings come from the audit parser",
185+
command: &["audit", "-s", "orders", "--concurrency", "2"],
186+
expected_plan: Ok((&["model:orders"], json!({"concurrency": 2}))),
187+
},
188+
AdhocPlanTestCase {
189+
description: "seed runs only the seeds its selectors reach",
190+
command: &["seed", "+orders"],
191+
expected_plan: Ok((&["seed:currencies"], json!({}))),
192+
},
193+
AdhocPlanTestCase {
194+
description: "seed refuses a selection without seeds, which would run nothing",
195+
command: &["seed", "orders"],
196+
expected_plan: Err("the selectors match no seeds"),
197+
},
198+
AdhocPlanTestCase {
199+
description: "freshness observes every source upstream of a selected model",
200+
command: &["freshness", "order_totals", "--state", "--fail-on-stale"],
201+
expected_plan: Ok((
202+
&["source:shop.raw_orders"],
203+
json!({"fail_on_stale": true, "state": true}),
204+
)),
205+
},
206+
AdhocPlanTestCase {
207+
description: "freshness excludes the sources an excluded model reads",
208+
command: &["freshness", "--exclude", "orders"],
209+
expected_plan: Err("no sources are selected or upstream of the selection"),
210+
},
183211
];
184212
let planner = adhoc_planner();
185213
for test_case in test_cases {
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
//! Turn frozen settings back into one command's flags, against the same table as the command-line parser.
2+
3+
use std::collections::BTreeMap;
4+
5+
use serde_json::Value as Json;
6+
7+
use crate::argv::_helpers::job_settings::{setting_argument, validate_mutex};
8+
use crate::argv::_helpers::table::root_parser;
9+
10+
/// Produce argument tokens for `command`, never shell text; a setting the command lacks is refused.
11+
pub fn command_settings_arguments(
12+
command: &str,
13+
settings: &BTreeMap<String, Json>,
14+
) -> Result<Vec<String>, String> {
15+
let parser = root_parser()
16+
.actions
17+
.iter()
18+
.flat_map(|action| &action.parsers)
19+
.find(|(name, _)| name == command)
20+
.map(|(_, parser)| parser)
21+
.ok_or_else(|| format!("{command} command is unavailable"))?;
22+
let mut arguments: Vec<String> = Vec::new();
23+
for (name, value) in settings {
24+
if let Some(argument) = setting_argument(parser, name, value)? {
25+
arguments.push(argument);
26+
}
27+
}
28+
validate_mutex(parser, &arguments)?;
29+
Ok(arguments)
30+
}

0 commit comments

Comments
 (0)