Repository navigation
feat: failed-node errors, job detail, redaction gaps and run history flags - #1388
Merged
Merged
Conversation
…use Recall relations portably SQL Server's render_create_function stripped a table function's SELECT and wrapped it in BEGIN RETURN (...) END, which the engine rejects. A TABLE return now renders RETURNS TABLE AS RETURN (<query>). sqb recall schema --backend warehouse matched state relations with LIKE '\_sqlbuild\_%' ESCAPE '\', which ClickHouse cannot parse. A prefix comparison with SUBSTRING needs no escape in any dialect.
A native lint response can report that the engine does not parse the project's dialect yet. Each such file now gets one L007 warning carrying the engine's reason, instead of looking lint-clean. A non-string reason is rejected as an invalid response.
…n and refuse WITH source expressions render_create_schema doubled quotes only in the sys.schemas lookup, so a schema name with a quote broke the EXEC string. render_merge now ends with the semicolon SQL Server requires. A source expression starting with WITH cannot sit in a derived table on SQL Server, so it now fails with A715 and a rewrite hint instead of a warehouse syntax error.
… of SQL library probes The unknown-function check (B101) asked the SQL library's parser which built-in node each unaccepted call spelling becomes, and took DuckDB's accepted names from the library's function catalogue. The check now reads a table of the 41 rejected spellings, each with its argument count and the spelling the dialect accepts. Every DuckDB row was confirmed unknown to DuckDB 2.0.0. The call scan takes the analysis dialect name, so analysis off the library can run it unchanged. Findings are identical on every DuckDB and Snowflake query of the compiler differential corpus and on every candidate spelling at 0 to 9 arguments. The workspace no longer depends on the library's function-catalogue crate directly; resolved features are unchanged.
…le and rules run compile and rules run kept only selected Rule codes, so an unavailable dialect looked clean there. L007 now passes through as a warning-severity Rule diagnostic in compile and is left out of rules run's failing-finding count. counts_toward_exit owns the exemption.
The plan and build rules gate failed on any finding, so an L007-only project could not plan. Rule exceptions also validate every finding's code as a rule code and rejected L007 with an invalid-lifecycle error. The gate now counts only findings that count toward exit, and L007 passes through rule exceptions unsuppressed. An end-to-end test replays a cached finding as L007 and as a rule code through compile, rules run, plan and build.
…g issues A same-run time ratio cannot see one store round trip per node: that cost is linear. Tests now count the statements each task issues, and the wide listing must read builds 100 nodes per statement: 17 statements for 1,000 nodes and 37 for 3,000 on SQLite and PostgreSQL, 1,008 with one node per statement. The ratio check stays and now guards super-linear growth only.
The dynamic column contract proof parses pivot SQL with SQLQuack's shared parser (DuckDB, MotherDuck, Snowflake) instead of Polyglot. DuckDB trees match Polyglot's exactly; a Snowflake aggregate keeps its argument in a call's argument list, which the proof now reads. Other dialects keep the existing unsupported-dialect message.
…nto feat/rule-facts-sqlquack
Local scenario capture no longer asks Polyglot to read warehouse column types. Postgres types go through sqlquack-syntax-postgres's type parser, and Snowflake and BigQuery types through their SQLQuack cast-target parse, behind one native function (parse_warehouse_type). DuckDB and MotherDuck spellings pass through unchanged. Databricks nested types map recursively to DuckDB's LIST, MAP and STRUCT, and a nested spelling that cannot be mapped is refused with a named message instead of falling back to JSON. Other types keep their integer width or fall back to VARCHAR.
SQL Server has no SQLQuack type parser, so its information_schema names reached the generic fallback, which held datetime2, smalldatetime, datetimeoffset and uniqueidentifier as VARCHAR. A SQL Server pre-mapper now maps them to TIMESTAMP, TIMESTAMPTZ and UUID, bit to BOOLEAN, money and smallmoney to exact decimals, and SQL Server's 8-byte float to DOUBLE. The fallback keeps 4-byte floats (FLOAT, FLOAT4, REAL) as FLOAT instead of widening them to DOUBLE. A panic in the warehouse type parser now surfaces as the ValueError capture already falls back on.
The client check ran only as its own Verify step, so a stale client passed make check-ci locally. check-ci now runs it, and Verify runs it through check-ci after installing the packages.
The person page's role and account controls had no server calls behind them. Giving a role, removing a manual one, and disabling or enabling a person now use the server's existing admin endpoints, then read the result back. The add control offers only what the server accepts: built-in roles for every location, a location's own roles there, and admin only for every location. The page also hides its effective-access and access-checker panels and the audit "Check this" links where the server has no read for them, and long audit subjects wrap.
…ombined-next-local
Each location row now has Test connection for people who may test that location. The UI asks the location through the server's connection-test request, then reads the result until the location answers or the request expires, and shows whether it connected. The location reports no reason, so a failure points to sqb debug in the location.
…th reads The location page now opens from the server's target health, newest snapshot and live problems: one process per reporting session, the publish-time connection check, a failed reload shown as its health, and open problems. The scheduler strip reports whether a server holds the lease. Facts the server doesn't report (images, commits, deploy policy, heartbeat, forwarding, problem history, tokens) stay hidden, and the list counts open problems per location and no longer offers the page to people who can't read the target's health.
…eturning them The maintenance read returned each old-name view's copied GRANT statements, which name warehouse roles, to anyone who may view the target. It now returns only how many grants were copied, read the way local sqb reads the stored list; the stored rows are unchanged.
People who may rewrite history can give a start and an end when they run a job or group. Whole numbers on both ends send an integer window; anything else goes as typed for the server to validate. Server error bodies that aren't JSON, such as a request the route couldn't read, are now shown as the server wrote them instead of as the bare status.
Axum's own 500 rejections, such as a missing request extension, are plain text that names Rust types. The server now replaces any plain-text 5xx body with the same JSON error its handlers return. 4xx rejections keep their text, which describes the caller's own request.
Workers now report a failed node's error message with its outcome. The server redacts named secrets and connection-string passwords, including ones a warehouse cut short, then bounds the text to 2,000 characters when it reads the report, so the stored fact never holds the raw text. It stays in the existing node-finished fact, so the store schema is unchanged. A new person read returns that text, the error code and the node's project file. It needs view.evidence on that node, because error text can quote warehouse data, and each read is audited. The run page fetches it when a failed node's details are opened.
…r worker's report A server refuses a run report whose shape it does not know. That only happens when the worker is newer than the server, for example a worker that sends a failed node's error text to a server without that field. The worker now stops with a message that says to upgrade the server before its workers, and the server page documents the upgrade order.
… rules Error text redaction and its 2,000-character cut move into core redaction as redact_error_text, shared by the worker and the server. The generic 5xx layer lives with the other API resources, the run node error read sits with the other warehouse-history reads, and its record is RunNodeFailure. No behaviour changes; only the OpenAPI schema name changes.
…d launch a full refresh
A new person read, GET .../targets/{target}/jobs/{job_key}, returns one
published job with its settings, a group's members in run order with what each
waits for, the groups it belongs to and its critical nodes. It needs the same
view as the job list. vars shows only that it is set, and text settings pass
through secret redaction.
A job run request can ask for a full refresh. It freezes full_refresh into the
run's settings and into each group child, needs run.rewrite_history on the
write set as a backfill does, and cannot be combined with a backfill window.
The job page shows the new facts and offers the full refresh to people who
may rewrite history.
… and passwords containing @ Secret keys glued to a prefix, such as db_password or PGPASSWORD, are now redacted; pwd still has to stand alone so working-directory variables keep their paths. Quoted PASSWORD values and IDENTIFIED BY values in SQL are redacted, while prose such as "password authentication failed" is kept. A URI password now runs to the last @ before the host. In error text, a value after scheme://name: counts as a port only when the authority ends after it; a value cut off before that is redacted even when it is all digits.
…D WITH plugin BY A value after scheme://name: stays only when it is all digits and the authority ends after it, so a cut password containing / is redacted again. IDENTIFIED WITH <plugin> BY '...' is redacted like IDENTIFIED BY.
…-node-errors-job-detail-redaction
… from the job read Queued runs now store rewrite_history from their own frozen request: a backfill window, a full refresh, a cursor setting or an ad-hoc destructive flag sets it, for group children and reruns too. A requeue keeps the value instead of resetting it to false. The job read withholds a defer_* setting whose value is not a target name, such as a state path, as it withholds vars.
…e-errors-job-detail-redaction
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
One pull request for audited server, UI and redaction work: one full Verify run, each part its own changelog entry.
Changes
rewrite_history.Verification
make check-cipasses, including the control client check.DuckShop: no project shape, data value or lifecycle is involved.
BEGIN_COMMIT_OVERRIDE
fix(server): the server no longer sends framework error text that names internal types; a plain-text 5xx becomes the generic JSON error
feat(runs): the run page shows a failed node's error text and project file, redacted and limited to 2,000 characters, for people with view.evidence on that node. Upgrade the server before its workers: an older server refuses a newer worker's failed-node report
fix(worker): workers stop with a clear message when an older server refuses their report. Upgrade the server before its locations and workers
feat(jobs): the job page shows a job's settings, its group's members in run order, the groups it belongs to and its critical nodes, and can start a full refresh for people with run.rewrite_history
fix(redaction): diagnostic and error-text redaction now also covers keys glued to a prefix (such as PGPASSWORD), SQL PASSWORD and IDENTIFIED [WITH plugin] BY values, passwords containing @, and digit passwords in cut connection strings
fix(runs): runs record whether they rewrite history (backfills and full refreshes), and the job page no longer shows deferral paths
END_COMMIT_OVERRIDE