Repository navigation
Commit fac17f3
authored
refactor: port clone, load, check, test, seed and diff commands, rule caches and state helpers to the native extension (#1102)
* refactor(cli): run sqb clone natively
* chore: update the Python boundary ratchet
* refactor(state): build migration event IDs, event SQL and freshness versions natively
The new plain-Rust sqlbuild-state crate computes the content-addressed
migration, column-rename and old-name view event IDs, renders the
idempotent event lookup and typed INSERT statements, and normalizes,
hashes, compares and age-checks source freshness data versions. Stored
IDs and hashes are pinned to values the Python implementation wrote.
Reads and writes still go through the adapter, and warehouse
orchestration stays in Python.
* fix: read native freshness helpers from the extension in the freshness command
* chore: update the Python boundary ratchet
* refactor: normalize dbt profiles and write dbt init config natively
* refactor: build and render the sqb dbt plan natively
* refactor: index dbt manifests and resolve dbt refs natively
* chore: update the Python boundary ratchet
* refactor(cli): run sqb load natively
* chore: update the Python boundary ratchet
* refactor: register command handler bindings with the CLI bindings
* fix(cli): route progress to stderr for sqb dbt --json
* chore: update the Python boundary ratchet
* refactor(diagnostics): redact diagnostic secrets natively
Secret redaction of diagnostic text and structured logging fields runs in
sqlbuild-core with Rust regex. The Python logging handlers, formatters,
filters and the LogRecord field collection stay in Python and are listed
as stays in python-boundary.toml.
* refactor(rules): key and look up the SQL rule result cache natively
SQL rule cache keys (lint settings, rule codes, model SQL and dynamic
column contract, non-model file text), stored-entry lookup and the cache
file now live in sqlbuild-rules as pure functions with a new key format
(sql-rules-v6), so the first run after upgrading misses once. A new
integration test pins cold misses, warm hits and invalidation on model,
threshold, exception-policy and rule-selection edits.
* chore: update the Python boundary ratchet
* fix: drop command binding imports from the main registration
* refactor(cli): run sqb check natively
* chore: update the Python boundary ratchet
* refactor(cli): render sqb diff output and resolve its exit codes natively
* chore: update the Python boundary ratchet
* refactor: import janitor, load, clone and freshness Python by complete module paths
* test: check native Python module paths resolve and are never built at runtime
* feat: run sqb test natively
* chore: update the Python boundary ratchet
* refactor(rules): key, look up and store the custom-rule result cache natively
Custom-rule identities (code identity, host runtime, rule metadata, source
fingerprint, rules settings, dialect, project root), subject identities,
read-set lookup, result merging, finding validation and ordering, and the
cache file now live in sqlbuild-rules as pure functions behind a
CustomRuleCache binding, with a new key format (custom-rules-native-v1),
so the first run after upgrading misses once. Fact digests stay with the
Python fact views the rule hosts read. A new integration test pins cold
misses, warm hits, a hit through an unread model edit, and invalidation
on a rule source edit.
* refactor(rules): load rules configuration natively
The native loader now returns the RulesConfig models directly and raises
RulesError or RulesConfigError itself; the allow_exceptions policy check
moves to sqlbuild-rules. The JSON configuration hand-off and its Python
normalisation are removed. Selector resolution against the compiled
project stays with the planner scope in Python.
* refactor(rules): convert rule findings and SQL test rows natively
The rules request binding now reads compiled SQL tests, scenarios and
initial findings straight from the Python objects, and evaluation,
finalisation, the SQL rule cache lookup and the custom-rule cache return
Finding models built natively. Finalisation keeps each original finding's
fixable mark natively, and undecodable cached SQL findings are dropped as
misses inside the lookup, so its handled-error entry is removed. The
Python row builders and finding decoders are deleted; a new
NativeRulesRequest.sql_facts_json accessor lets tests inspect the facts
sent to the engine.
* refactor(rules): separate rule-run thread and custom-rule dispatch modules
Thread pools, futures and stop events around a rules run move to
run/rule_threads.py, and custom-rule submission beside native evaluation
moves to engine/custom_dispatch.py. Run planning, SQL rule evaluation
and result finalisation stay in run/rules.py as plain functions. The two
new modules and engine/custom_rules.py, which dispatches to the rule-host
worker pool, are listed as user-code stays.
* refactor(rules): build the built-in rule catalogue natively
Built-in Rule models, their declared options and guidance are built by a
native builtin_rules binding from the sqlbuild-rules catalogue, and the
early-lint SQL code selection runs natively. The JSON catalogue hand-off
and the Python option table are removed. Loading custom rule modules and
validating their declared options stay in Python with the user code.
* chore: update the Python boundary ratchet
* refactor(cli): run sqb diff validation, orchestration and query lifecycle natively
* chore: update the Python boundary ratchet
* feat: run sqb seed natively
* chore: update the Python boundary ratchet
* fix(cli): render sqb check rows before encoding check metadata1 parent 74fcac8 commit fac17f3
530 files changed
Lines changed: 22337 additions & 11498 deletions
File tree
- crates
- sqlbuild-cli/src
- diff_report
- _helpers
- main
- tests
- dispatch
- sqlbuild-config/src
- dbt_manifest
- _helpers
- main
- tests
- dbt_profile
- _helpers
- main
- tests
- sqlbuild-core
- src
- check_command
- _helpers
- main
- tests
- cli_text
- _helpers
- main
- tests
- clone_text
- _helpers
- main
- tests
- load_command
- _helpers
- main
- tests
- progress
- main
- tests
- redaction
- _helpers
- main
- tests
- seed_command
- main
- tests
- test_command
- main
- tests
- sqlbuild-python
- src/bindings
- _helpers
- check
- cli
- clone
- command_runtime
- dbt_interop
- diagnostics
- diff_report
- freshness
- janitor
- load
- rules
- seed_command
- sql_test_command
- state_rows
- main
- sqlbuild-rules/src
- configuration
- _helpers
- main
- custom_rule_cache
- _helpers
- main
- tests
- rule_cache
- _helpers
- main
- rules
- main
- sqlbuild-state
- src
- events
- _helpers
- main
- tests
- freshness
- _helpers
- main
- tests
- scripts/native_module_paths
- _helpers
- src/sqlbuild
- cli/commands
- _helpers
- build_execution
- check
- clone
- diff
- entry
- load
- seed
- test
- classes
- main
- dbt
- execution
- inspection
- compiler
- compile/main
- discovery/main
- migrations
- _helpers
- main
- pipeline/main
- planner/main/commands
- source_freshness
- _helpers
- main
- diagnostics
- _helpers
- classes
- executor
- diff
- main
- migrations/_helpers
- old_name_views/_helpers
- integrations/dbt
- _helpers
- graph
- manifest
- pipeline
- planning
- profile
- classes
- main
- manifest
- pipeline
- planning
- presentation
- _helpers
- main
- rule_engine
- _helpers
- engine
- run
- main
- tests
- integration/src/sqlbuild
- adapters
- bigquery
- databricks
- snowflake
- cli/commands
- _helpers
- diff
- runtime
- main
- executor/diff
- integrations/dbt
- unit
- scripts/native_module_paths
- _helpers
- src/sqlbuild
- cli
- commands
- _helpers
- check
- clone
- diff
- freshness
- load_selection
- main
- dbt
- entry
- output/_helpers
- compiler
- migrations/main
- planner
- _helpers
- resolve
- sql_test_assembly
- main
- source_freshness
- _helpers
- main
- configured_observation
- executor
- load
- migrations/_helpers
- python_nodes/_helpers
- integrations/dbt
- profile
- rule_engine/_helpers
- engine
- run
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| 30 | + | |
29 | 31 | | |
30 | 32 | | |
31 | 33 | | |
| |||
104 | 106 | | |
105 | 107 | | |
106 | 108 | | |
| 109 | + | |
107 | 110 | | |
108 | 111 | | |
109 | 112 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
34 | | - | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| |||
56 | 57 | | |
57 | 58 | | |
58 | 59 | | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
59 | 75 | | |
60 | 76 | | |
61 | 77 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
0 commit comments