Repository navigation
feat: add a ClickHouse adapter and fix CLI rename, errors and test functions - #1152
Merged
Merged
Conversation
kvlonge
enabled auto-merge (squash)
October 19, 2026 06:25
…h the column A MODEL columns block without commas between entries hid every entry after the first from sqb rename, so the model's own declaration was left for a hand edit that the next run then refused because the project no longer compiled. Column entries now split where the compiler splits them. String arguments of attached audits are renamed only when the generic audit's SQL definition splices them in as SQL (`@where_condition`), and only where they reference the audited model: a subquery reading another model keeps that model's column, and one reading the audited model is renamed too. Arguments the audit quotes (`@'prefix'`), arguments of audits with no SQL definition, unparseable ones and ones reading relations of unknown columns are listed for a hand edit. Refs: CHI-640
Reading a missing or unreadable selector file raised a raw FileNotFoundError with a traceback from compile, plan and build. The read now fails with error[C114] naming the file, in text and --json output. Refs: CHI-501
With only declaration rules selected, model rule evaluation took the first model unconditionally, so a source-only selection sliced an empty model list and panicked. The model limit is now capped by the number of models. Refs: CHI-530
Without --auto-approve, janitor prompted on a non-interactive stdin and ended with an EOFError traceback. Like build confirmations, it now refuses before prompting with error[C503] and tells the user to pass --auto-approve or run it in a terminal. Refs: CHI-534
A dict key such as [1] or (1, [2]) parsed, then failed with a raw TypeError when the Python dict was built. The argument parser now rejects list and dict keys, also inside tuple keys, with a located P001 error and help. Refs: CHI-571
When the warehouse connection could not be opened, for example a DuckDB file in a missing directory, build, seed, audit, test, plan, load, query and diff printed the failed connection line and then the adapter's raw exception with a traceback. Connection failures now raise error[C115] naming the adapter and the reason, with help pointing at sqb debug. SQLBuild's own coded errors pass through unchanged; a client library error that carries its own `code`, such as a Google API error, is still reported as C115. Refs: CHI-596
…errors sqb diff with a --key column that does not exist, or with the relation missing on either side, failed with a raw warehouse error and a traceback. Key columns are now checked against both relations (X311), and a failed schema read whose relation does not exist reports which side lacks it (X310). Refs: CHI-600
Detail rows under every plan entry used a blank prefix, so the vertical bar of a non-last entry broke under its cursor, bounds, reason and diff rows. Detail and nested rows now carry the bar when their entry has a later sibling. Refs: CHI-533
Seed, source and end audits printed as 'audit not_null OK', without their target, and a failing warn-severity audit still showed OK. Their rows now name the target and column, as in 'not_null (customers.customer_id) PASS', and show PASS, WARN, FAIL or INSUFFICIENT from the audit outcome. The partial semantic checks notice also says '1 model' instead of '1 models'. Refs: CHI-519
sqb plan always exited 0 and printed 'Plan complete with errors' when the plan had error-severity findings that sqb build refuses, such as M114 name claims, so CI could not gate on it. Text and --json plans now exit 1 and the final line reads 'Plan failed with errors'; plans with only warnings still exit 0. The plan docs record the change as a release note. Refs: CHI-429
`sqb test` called the built warehouse function, so an edited UDF or table function was tested at its last-built definition, and a never-built one failed with "Run `sqb build` first". Before the tests run, SQLBuild now creates a session-scoped copy of every function the selected tests call, and of every function those call (the test plan now carries the transitive closure), then points calls in the tests' SQL code at the copies; calls in string literals and comments keep their text. No permanent DDL is issued. The copy set is decided on the first test connection and recreated on every other connection and on recovery reconnects; a connection that cannot create it stops the run with error[T006] naming the function and connection. Adapters opt in through the optional StrictAdapter method `render_create_temporary_function`: DuckDB and MotherDuck use TEMP MACRO, PostgreSQL uses pg_temp, Snowflake and Databricks use prefixed TEMPORARY FUNCTIONs. BigQuery and SQL Server keep calling the built function. When no copy can be created, tests fall back to the built function (a caller whose callee has no copy too), and a missing one reports the reason with the build hint. Refs: CHI-555
The bundled skill reference pages under src/sqlbuild/.agents/skills/sqlbuild/references/docs drifted from the website docs and nothing caught it. `make check-docs-skill`, now part of `make check-ci`, regenerates the pages in memory with the same generator as `make skills` and fails, listing every changed, missing or retired page, with `make skills` as the fix. Refs: CHI-629
The seeds page attached `expression_is_true` without showing its
definition, and the definition people copy from the playground reads
`__ref("@model")`. An attached audit receives its owner kind's name
argument (`model`, `source` or `seed`) plus `relation`, so on a seed it
failed with a bare "missing argument 'model'". Passing the seed as
`model` would not help: `__ref` resolves only models.
The seeds page now shows a seed-compatible `@relation` definition and
explains the owner arguments, the audits page documents them, and the
missing-argument error names the argument the owner does receive and
points at @relation. A DuckDB E2E builds the documented example.
Refs: CHI-632
CHI-429 makes sqb plan exit 1 when the plan reports an error finding, so the schema backfill case that plans an on_schema_change fail model now expects that exit code.
Adds a ClickHouse adapter over clickhouse-connect (HTTP, sessionless, per-statement query ids, interrupt via KILL QUERY). A SQLBuild schema is a ClickHouse database. Tables build aside and swap with EXCHANGE TABLES; merge, delete_insert, cursor windows and microbatches rewrite whole partitions with REPLACE PARTITION, so a rerun after a failure converges. Snapshots stage on a working copy and exchange in. Seeds stream as CSV. SQL functions deploy as <schema>__<name>; Python and table functions are refused with coded errors. Historical snapshot SQL now aliases its key projections so engines that name qualified projections after the qualifier resolve them.
Sources accept 'final: true'. Every generated read of such a source (model references, source audits and freshness) is rendered natively as (SELECT * FROM <relation> FINAL) so replacing engines return one row per key. Adapters whose SQL dialect is not ClickHouse refuse the option with A010, and expression sources cannot set it.
… adapter Adds end-to-end CLI coverage on a real ClickHouse server (views, tables, seeds, functions, merge, concurrent microbatches, final sources, query, diff, clone and the snapshot matrix), a ClickHouse CI job and test-real-clickhouse target, and the adapter documentation page.
Microbatch events order by created_at then event ID when choosing the latest completion and requirement. ClickHouse's TIMESTAMP is DateTime, so casting state literals to it cut them to whole seconds and let events from the same second order by ID instead of time. Adapters may now name the type state timestamps cast to; ClickHouse uses DateTime64(6, 'UTC').
… model Table and incremental models accept engine, order_by, partition_by, primary_key and table_settings in the MODEL header. Compile validates their shapes (MergeTree-family single-server engines, primary key a prefix of the sorting key, scalar settings) and rejects them on views and snapshots; planning rejects them on other adapters with A011. The keys join the model's version identity, the plan entry carries them to every target-creating CREATE TABLE, and ClickHouse renders them as table clauses. An E2E shows an incremental window rewriting only its partition.
…s a full refresh Changing engine, order_by, partition_by, primary_key or table_settings on an incremental model changes its version, but runs keep writing into the existing table with its old layout. The plan now carries a PHYSICAL_LAYOUT_PENDING warning naming the changed keys and the full refresh that applies them, in text and JSON output. Plan decisions are unchanged.
…ment A sorting key written as one tuple string such as 'customer_id, ordered_at' was compared whole against the primary key, so a valid prefix was rejected. Both keys are now split at their top-level commas, outside quotes and brackets and after removing enclosing parentheses, before the prefix check.
User connection settings were applied over the settings SQLBuild's generated SQL depends on, so for example join_use_nulls = 0 silently broke snapshot inserts. Required settings are now applied last, and a user value that differs from one fails to connect with A710 naming the setting.
The ClickHouse connection runs client queries inside its statement lifecycle, like the other adapters' connection wrappers, but was missing from the raw statement audit's approved boundaries.
ClickHouse tables built from a query take each column's nullability from
it, and inserting NULL into a non-nullable column stored the type's
default ('', 0, the epoch) under on_schema_change ignore. Incremental and
microbatch runs now widen such target columns to Nullable before the
schema comparison through a new widen_target_nullability adapter method
(a no-op elsewhere), insert_null_as_default is pinned to 0 so any other
mismatch fails, and diff_schema reports nullability differences.
…ange A rewrite that touched several partitions swapped them in one at a time. An interruption between swaps could leave a newer partition published and an older one not, and the next run's cursor, read from the table's maximum, then skipped the unpublished rows for good. Several partitions are now rebuilt in a copy of the table that hard-links every kept part and are published with one EXCHANGE TABLES, under an in-process exclusive lock on the table; single-partition swaps keep REPLACE PARTITION under a shared table lock and their partition locks. The copy carries the table's generation in its comment, so microbatch state keeps its scope across the exchange. A fault test interrupts every statement of the rewrite and checks the next cursor run against the DuckDB oracle.
…chain
On ClickHouse a UDF marker call such as __udf("name")(args) parses as a
parametric function, so SQL-test planning took the parsed path, which
only replaced relation markers and left the UDF marker in the real
model's SQL. The parsed path now resolves UDF and table-function calls
the same way the textual path does.
The bundled waffle shop runs through the real CLI on ClickHouse: a full build of seeds, sources, models and audits, sqb test, and a second build over the incremental and microbatch paths, checking exit codes and row counts. The helper drops the SQL table function and the Python UDF (neither deploys to ClickHouse), aliases qualified columns in multi-join models, and writes CURRENT_TIMESTAMP() with parentheses; the adapter page documents both SQL differences.
In a query joining three or more tables, ClickHouse names a selected alias.column after the alias when another joined table has the same column, so a table could silently get a column such as 'o.order_id' where every other warehouse and SQL analysis say 'order_id'. Plan entries now carry the output columns SQL analysis inferred, and after creating a model's table or delta the ClickHouse adapter drops it and fails with A711 when a column is named alias.column for an expected column, naming the alias to add. Models without an inferred column list are not checked.
…allows it Nullability widening ran before the on_schema_change check, so fail and ignore models still had their columns altered. Widening now happens only under append_new_columns and sync_all_columns; fail reports the type change through its usual path, and ignore leaves the table alone so a NULL insert fails with A713 explaining the fix. A key column that would need widening fails with A712 explaining allow_nullable_key and coalesce() instead of a raw server error.
Copies made with CREATE TABLE ... AS inherited the source's generation comment, so a clone, hard copy, migration stage or snapshot working copy shared the original table's microbatch generation. Every such copy now clears the comment and takes its own UUID as its generation; only the partition-rewrite exchange carries the generation over.
Assembled sources now carry the ClickHouse final read option, which defaults to false, so the two assembly golden digests that include sources change.
kvlonge
changed the title
kvlonge
force-pushed
the
feat/recall-clickhouse
branch
from
October 19, 2026 07:12
d8b839f to
9fa3cc4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A native ClickHouse adapter that writes safely without transactions (CHI-650, CHI-651; Recall epic CHI-577), combined with thirteen CLI fixes (formerly #1149) because both touch the same adapter contract files.
Changes
ClickHouse (single server):
EXCHANGE TABLES; incrementals by partition rewrite, multi-partition runs published atomically from a hard-linked copy; microbatch, snapshots, diff, functions, clone.MODEL()layout keysengine,order_by,partition_by,primary_key,table_settings;final truesources; nullability followson_schema_change(A712/A713); required settings fixed (A710); alias-named columns fail (A711). CI job on a ClickHouse container; docs.CLI:
sqb renamerewrites MODEL() column declarations and audit SQL arguments that resolve to the column (CHI-640).sqb planexits 1 when the plan has errors (CHI-429, release note).sqb testuses session-temporary copies of called functions and callees (CHI-555).make check-docs-skillincheck-ci(CHI-629); seed audit docs (CHI-632).Verification
clickhouse/finalin two compiler goldens, assembly goldens recordfinal, one intended CLI golden.