Repository navigation
Commit 920a933
authored
fix: align SQL scanners and remove obsolete implementations (#815)
* fix(sql-lint): honour backtick identifiers for the project dialect
SQL lint's interpolation-site lexer treated backticks as ordinary code, so
in BigQuery and Databricks projects a close paren inside a backtick
identifier ended a macro call early, and an apostrophe inside a backtick
identifier hid every later macro site. `sqb format` then rewrote the hidden
macro (for example `@label(...)` became `@LABEL(...)`), breaking compile.
Lint now derives backtick quoting from the configured lint dialect using the
same policy rules use, in both the native preparation pass and the Python
fallback used for formatting and non-ASCII bodies. Postgres and SQL Server
keep treating backticks as code; backslash escapes are unchanged.
* fix(rules): skip comments in the generic parser fallback normalisation
When a model's dialect parser rejected it, the generic fallback rewrote `:`
paths and `->` lambdas while skipping only quoted text. An apostrophe in a
`--` or `/* */` comment (for example `-- don't`) opened a phantom quote that
never closed, so normalisation stopped and the fallback parse failed. The
fallback now skips comments and quotes through the shared scanner.
* fix(sql-tests): share the expected-CTE UNION splitter with the Polyglot fallback
When Polyglot could not parse an expected CTE, the fallback split set
operations with its own scanner: it ignored comments (an apostrophe in
`-- don't` hid every later branch, and `UNION` inside a comment split),
left `DISTINCT` at the start of a `UNION DISTINCT` branch so the branch
parse raised a raw parser error, and split on INTERSECT/EXCEPT even though
the parsed path rejects them.
Both the fallback and the textual expected-projection path now use one
shared `split_union_branches` over the comment- and quote-aware
`iter_code_positions`, handling `UNION [ALL|DISTINCT]`. The fallback rejects
top-level INTERSECT/EXCEPT with the parsed path's error, and defers to the
textual path when a split branch still cannot be parsed.
* fix(sql-tests): merge textual assertion CTEs into a leading WITH
The textual assertion fallback prefixed lifted CTEs as `WITH ... {body}`
even when the assertion body began with its own `WITH`, producing
`WITH a AS (...) WITH b AS (...) SELECT ...`. That resolved SQL is what the
renderer emits when an assertion's lifted CTEs conflict with earlier steps.
The fallback now uses the shared `with_leading_ctes`, merging lifted CTEs
into the body's leading WITH clause like every other planning path.
* refactor(native): share one top-level code scan in the SQL-test extractor
`split_unions`, `split_top_level` and `find_top_level_keyword` each repeated
the same loop around the shared scanner: skip comments and quotes, track
parenthesis depth, then act on top-level code. They now run on a single
`scan_code` walker whose visitor advances, resumes after a consumed
keyword, or stops with a result. Characterisation tests pin union splits,
comma splits, keyword offsets, multibyte text, unbalanced parentheses and
unclosed-input errors; results are unchanged.
* fix(identity): include declared loader columns in version identity
Loader version identity hashed the destination, write strategy, cursor,
unique key and contract but not the declared `columns`, so editing a
loader's column declarations (for example through a module-level column
list) left its recorded version unchanged. Loaders now include `columns`
in their identity decorator config, matching assets. Retry and connection
mode remain excluded for every kind.
Upgrading gives one round of new versions for loaders with declared
columns: their first run after the upgrade records a new version hash.
* fix(sql-tests): support INTERSECT and EXCEPT in expected CTEs and check every branch
Expected CTEs now accept UNION, INTERSECT and EXCEPT, each with an optional
ALL or DISTINCT, in model, macro, UDF and table-function tests, and every
branch's projection names and count must match the first branch.
- Native extractor: `split_unions` becomes `split_set_operations`, so an
INTERSECT or EXCEPT branch is validated instead of being folded into the
first branch; `* EXCEPT (...)` star modifiers are not treated as operators.
- Polyglot path: Intersect and Except nodes are flattened like Union instead
of being rejected as non-SELECT branches.
- Shared Python fallback: `split_union_branches` becomes
`split_set_operation_branches` and no longer rejects INTERSECT/EXCEPT.
- Model-test expected-CTE errors now name the actual CTE (for example
`__expected__orders`) instead of `__expected__<model>`, matching
direct-logic tests; the message wording is otherwise unchanged.
* refactor(compiler): remove unused code
* refactor(cli): remove unused code
* refactor(executor): remove unused code
* refactor(virtual): remove unused code
* refactor(runtime): remove unused code
* refactor(integrations): remove unused code
* refactor(adapters): remove unused code
* refactor(core): remove unused code
* refactor(tests): remove orphaned test case types
* refactor(compiler): remove no-op cursor alias deprecation shim
* refactor(compiler): remove unused legacy hook key copies
* refactor(adapters): remove unused adapter contract methods
Drop render_query_with_cursor_bounds, render_seed_select_after_cursor, get_relation_max_cursor and count_rows from BaseAdapter, StrictAdapter, DiffMixin, DuckDbBackedAdapter and every first-class adapter, plus PromotionStrategy.CREATE_NEW. No engine code called them. Tests that used count_rows for verification use a small COUNT(*) helper; the unsupported promotion strategy test now uses the catalogued but unhandled build_aside strategy.
* refactor(executor): remove unwired same-target audit gate reuse decision
same_target_audit_gate_reuse_decision was never called by the executor. Remove it, the BINDING_SET_CHANGED reason only it produced, and its tests. The invalid-metadata diagnostic test now exercises the live reuse_from decision, which shares the parsing path.
* refactor(compiler): remove test-only header parser and state backend methods
Remove the Python _ModelHeaderParser, which only served as a cross-check oracle for the native MODEL header parser, together with its corpus parity test. Remove the StateBackend methods replace_virtual_environment_function_refs, replace_virtual_environment_seed_refs and get_physical_relation_ancestry, which only tests called; tests now write refs through the node-ref APIs production uses. No state-table layout, stored format or reader used by the engine changes. append_microbatch_event is kept: retargeting its Postgres tests to append_microbatch_events exposed a type-resolution failure in the bulk path that needs a separate fix.
* refactor(execution-history): remove deprecated EventLogStorage alias
This removes a deprecated public alias: sqlbuild.execution_history.EventLogStorage is gone. Use LifecycleEventLogStorage, which it pointed to. docs/execution-history.md is updated accordingly.
* refactor(lint): remove unused code
Remove the unused fix-run machinery (FixRunResult, LintFixRecord, fix planning helpers and their CLI renderers), other unreferenced lint helpers and constants, and the test-only map_neutralized_offset. LINT_ENGINE_SQLBUILD is kept and now used by native.py instead of the duplicated "sqlbuild" literal.
* refactor(rule-engine): remove unused code
Remove unreferenced SQL helper modules (decision signatures, model-name grammar, passthrough detection), table_parts, ModelNameParts and unused constants. The public custom-rule API is unchanged: SqlDocument.subqueries and RuleOption.integer_list stay.
* refactor(compiler): remove unused SQL analysis, scenario and SQL-test code
Remove unused Polyglot analysis constants, the scenario _dedupe_names helper and the never-emitted S505 scenario code, the unused _LEGACY_MODEL_HOOK_KEYS copy in SQL-test attachment, and the SQL-test CTE SQLite cache module, which has had no production caller since #758.
* refactor(python-nodes): remove unused code
Remove the unused loader-to-node conversion module, the placeholder DiscoveredPython{Task,Asset,Check}Node classes, PythonSqlRunLifecyclePlan.python_phase, and fields that were set but never read: PythonNodeIdentity.object_module/object_qualname, PythonNodeDependencyEdge.upstream_function/downstream_function and PythonSqlRunLifecyclePlan.read_side_sql_keys. The identity definition payload and hashes are unchanged.
* refactor(executor): remove unread microbatch applied intervals
ModelExecutionResult.microbatch_applied_intervals and MicrobatchPhaseOutcome.applied_intervals were accumulated for every batch but never read. Remove the fields and their bookkeeping; execution behaviour is unchanged.
* refactor(virtual): share ref model-version lookup
virtual diff and promote each carried an identical private _read_model_versions. Both now use one read_ref_model_versions entry in virtual/state/main/environments.
* refactor(planner): rename stale SchemaColumnSource.SQLGLOT member
sqlglot was removed as a runtime dependency; the member is now SQL_ANALYSIS. Only the Python name changes: the enum value stays "sql_analysis", so nothing serialised or displayed changes.
* fix(state): type bulk microbatch event inserts on Postgres
* refactor(state): remove single microbatch event append
* test(ci): run Python node identity regression in build-core shard1 parent afa88d4 commit 920a933
249 files changed
Lines changed: 1889 additions & 8593 deletions
File tree
- crates/sqlbuild-rules-native/src
- bindings/_helpers
- compiler
- _helpers/sql_tests
- tests
- engine/tests
- rules
- _helpers
- main
- sql_lint
- _helpers
- main
- sql_scan
- docs
- src/sqlbuild
- adapters
- bigquery/classes
- databricks/classes
- postgres/classes
- snowflake/classes
- sqlserver/classes
- adapter/contract
- classes
- cli
- commands
- _helpers
- entry
- lint
- skills
- classes
- output
- _helpers
- progress/classes
- compiler
- compile
- _helpers
- analysis
- attachment
- config
- render
- sql_tests
- main
- contracts
- discovery
- _helpers
- filesystem
- sql
- fingerprints
- lineage
- _helpers
- planner
- _helpers
- changes
- command_planning
- identity
- output
- scenario
- python_nodes
- _helpers
- scopes
- _helpers
- main
- sql_analysis
- _helpers
- main
- cost
- cursor_algebra
- diagnostics/_helpers
- executor
- build
- _helpers
- classes
- clone
- _helpers
- load/_helpers
- python_nodes
- _helpers
- classes
- main
- run
- _helpers
- materializations
- reuse
- main
- integrations
- dagster
- dbt
- _helpers/manifest
- main/manifest
- dlt
- ingestr/_helpers
- rivers
- lint
- _helpers
- main
- presentation
- rule_engine
- _helpers/sql
- runtime/event_exporting/classes
- spec/contracts
- sql_values
- virtual
- diff/_helpers
- executor
- _helpers
- planner/_helpers
- state
- _helpers/state_storage
- classes
- main
- checkpoints
- environments
- tests
- e2e/src/sqlbuild/cli/commands/main
- build
- plan
- postgres
- test
- integration/src/sqlbuild
- adapters
- duckdb
- postgres
- snowflake
- cli/commands/main
- executor/run/incremental
- virtual/state/classes
- postgres
- unit/src/sqlbuild
- adapters
- bigquery
- duckdb
- adapter/contract/classes/base_adapter
- cli/commands
- _helpers
- check
- main
- compile
- entry
- compiler
- compile/_helpers
- discovery/_helpers
- lineage
- planner
- _helpers
- changes
- main
- python_nodes/_helpers
- scopes
- executor
- build/_helpers
- janitor/main
- load
- python_nodes/_helpers
- run/_helpers
- integrations
- dbt
- dlt
- lint
- _helpers
- runtime
- event_exporting
- execution_history
- virtual
- executor/_helpers
- state
- _helpers
- main
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
| 76 | + | |
76 | 77 | | |
77 | 78 | | |
78 | 79 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
41 | 51 | | |
42 | 52 | | |
43 | 53 | | |
44 | | - | |
45 | | - | |
46 | | - | |
| 54 | + | |
47 | 55 | | |
48 | 56 | | |
49 | | - | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
50 | 63 | | |
51 | 64 | | |
52 | 65 | | |
53 | 66 | | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
54 | 72 | | |
55 | 73 | | |
56 | 74 | | |
| |||
325 | 343 | | |
326 | 344 | | |
327 | 345 | | |
| 346 | + | |
328 | 347 | | |
329 | 348 | | |
330 | 349 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
202 | 202 | | |
203 | 203 | | |
204 | 204 | | |
205 | | - | |
| 205 | + | |
| 206 | + | |
206 | 207 | | |
207 | 208 | | |
208 | 209 | | |
| |||
431 | 432 | | |
432 | 433 | | |
433 | 434 | | |
434 | | - | |
| 435 | + | |
435 | 436 | | |
436 | 437 | | |
437 | 438 | | |
| |||
844 | 845 | | |
845 | 846 | | |
846 | 847 | | |
847 | | - | |
848 | | - | |
849 | | - | |
850 | | - | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
851 | 862 | | |
852 | 863 | | |
853 | 864 | | |
| |||
860 | 871 | | |
861 | 872 | | |
862 | 873 | | |
863 | | - | |
864 | | - | |
865 | | - | |
866 | | - | |
867 | | - | |
868 | | - | |
869 | | - | |
870 | | - | |
871 | | - | |
872 | | - | |
873 | | - | |
874 | | - | |
875 | | - | |
876 | | - | |
877 | | - | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
878 | 878 | | |
879 | | - | |
880 | 879 | | |
881 | | - | |
882 | | - | |
883 | | - | |
884 | | - | |
885 | | - | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
886 | 885 | | |
887 | 886 | | |
888 | | - | |
| 887 | + | |
| 888 | + | |
889 | 889 | | |
890 | 890 | | |
891 | | - | |
892 | | - | |
893 | | - | |
894 | | - | |
895 | | - | |
896 | | - | |
897 | | - | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
898 | 894 | | |
899 | | - | |
900 | | - | |
901 | | - | |
902 | | - | |
903 | | - | |
904 | | - | |
905 | | - | |
906 | | - | |
907 | | - | |
908 | | - | |
909 | | - | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
910 | 904 | | |
911 | | - | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
912 | 918 | | |
913 | | - | |
914 | | - | |
915 | | - | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
916 | 922 | | |
917 | | - | |
918 | 923 | | |
919 | 924 | | |
920 | | - | |
921 | | - | |
922 | | - | |
923 | | - | |
924 | | - | |
925 | | - | |
926 | | - | |
927 | | - | |
928 | | - | |
929 | | - | |
930 | | - | |
931 | | - | |
932 | | - | |
933 | | - | |
934 | | - | |
935 | | - | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
936 | 932 | | |
937 | | - | |
938 | | - | |
939 | | - | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
940 | 953 | | |
941 | 954 | | |
942 | 955 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1056 | 1056 | | |
1057 | 1057 | | |
1058 | 1058 | | |
1059 | | - | |
1060 | | - | |
1061 | | - | |
1062 | | - | |
1063 | | - | |
1064 | | - | |
1065 | | - | |
1066 | | - | |
1067 | | - | |
| 1059 | + | |
1068 | 1060 | | |
1069 | 1061 | | |
1070 | 1062 | | |
| |||
0 commit comments