Repository navigation
Commit bbd4e53
authored
refactor: host remaining Rust-only shapes, event export and janitor planning natively (#1115)
* test: drop stale imports of removed rule and diagnostics modules
* refactor: host lint, lineage, scope, discovery and analysis-session records and errors natively
* refactor: host compile, discovery, rule, planner and dbt value records natively
* chore: record staying executor, adapter and API shapes and move migration relation naming to the executor
* refactor: declare compile input protocols in the extension stub and record the staying compile view shapes
* chore: reconcile boundary after data-model leftovers
* chore: drop stale stub imports after data-model integration
* refactor: add native lifecycle-event export policy and priority queue
Add the destination-neutral export policy derived from the lifecycle catalogue,
severity comparison and narrowing, exception-name sanitisation, the bounded
finite-priority queue with displacement, per-sink routing and accounting
snapshots to sqlbuild-core, with unit tests.
* refactor: host lifecycle-event export natively
Move the event exporter dispatcher (priority queue, per-sink accounting, drain
deadlines, invocation timeouts, failure notifications and periodic and final
health summaries), the export policy, severity narrowing, error-name
sanitisation and the command scope with its context variable into Rust. The
dispatcher hosts its work on named Python daemon threads under a Python
condition, joins its health thread at shutdown and stops its threads at
interpreter exit. Each lifecycle sink call runs on a thread whose target is
the Python invocation shim, so user code never runs above native frames that
finalization would abort.
Python keeps the public sink decorators and readers, the shapes re-exported by
sqlbuild.sinks and the two shims that call user sink functions.
* fix: stop command-output delivery before interpreter finalization
A command-output sink still running when the interpreter finalized ran above
native frames on the capture worker thread, and CPython ending that daemon
thread aborted the process. Each command-output sink call now runs on its own
daemon thread whose target is the Python invocation shim, which reports
completion back to the native export. Writes from those threads stay
uncaptured. At exit, hooks set a shared exit signal that ends native waits,
then stop and join the capture worker and the event exporter threads with a
bounded wait. Delivered records and accounting are unchanged.
* chore: drop duplicated stub imports after event export integration
* fix: join background SQL test planning before interpreter exit
When planning failed before the SQL test plan was awaited, the background
planning thread could still be running native code as the interpreter
finalized, and CPython ending that daemon thread aborted the process instead
of reporting the planning error. Execution planning now joins the planning
thread on every exit path, and an exit hook joins any still-running planning
thread with a bounded wait. Planned output is unchanged.
* test: import native SqlTestSelection in planning exit test
* refactor: plan janitor cleanup natively around adapter callbacks
* refactor: build imported project Python discovery records natively around a module shim
* refactor: parse Dagster invocation arguments, logs and payloads natively
* refactor: build integration loader records natively around runtime loader factories
* fix: keep provider settings unloaded during plain compile discovery
* fix: run command-output sinks on one thread per event exporter dispatcher
Each command-output sink call started its own daemon thread, so a provider
used only by output sinks that creates a thread-bound resource on first use
(such as a SQLite connection) failed on the next record. All command-output
sink calls for a dispatcher now run in order on a single long-lived thread
whose target is a Python shim waiting on a Python queue, so no native frame
is on that thread when interpreter finalization can end it. Records,
ordering, failure counting, filtering and provider teardown are unchanged.1 parent 845980a commit bbd4e53
707 files changed
Lines changed: 20371 additions & 10332 deletions
File tree
- crates
- sqlbuild-analysis/src
- janitor_planning
- _helpers
- main
- tests
- sqlbuild-cache/src/stage_capture
- sqlbuild-cli/src/dispatch
- sqlbuild-core/src
- dagster_invocation
- _helpers
- main
- tests
- event_export
- main
- tests
- sqlbuild-discovery/src/yaml_entries
- _helpers
- main
- sqlbuild-python/src/bindings
- _helpers
- analysis_session
- analysis
- audit_command
- auditing
- build_command
- check
- cli
- clone
- command_runtime
- compile_attachment
- compile_command
- compile_driver
- compile_records
- compiled_project
- compiler_records
- contract_command
- contracts
- dag_artifact
- dagster_invocation
- dbt_command
- dbt_events
- dbt_interop
- discovery
- event_exporting
- fingerprints
- format_command
- freshness
- frontier
- graph
- hand_built
- janitor
- lineage
- manifest
- model_config
- native_threads
- output_capture
- plan_command
- plan_output
- planner_entries
- planner_migrations
- planner_models
- planner_scenario
- planner_scope
- planner_selection
- planner_strategy
- records
- refactor
- relation_naming
- render
- rules
- scenario_command
- scopes
- seed_command
- semantic_checks
- semantic_shapes
- sql_test_command
- sql_test_glue
- sqltext
- state_rows
- main
- scripts
- cold_compile_performance/_helpers
- compiler_differential/_helpers
- corpus
- coverage
- fensu_policy
- src/sqlbuild
- cli/commands/_helpers/freshness
- compiler
- analysis_session
- auditing
- compile
- contract_adoption
- contracts
- discovery
- _helpers/filesystem
- main
- fingerprints
- frontier
- graph
- lineage
- migrations
- main
- pipeline
- planner
- python_nodes
- _helpers
- refactoring
- references
- resource_names
- scopes
- source_freshness
- sql_analysis
- sql_test_glue
- executor
- janitor
- _helpers
- classes
- main
- migrations/_helpers
- old_name_views
- _helpers
- main
- integrations
- dagster
- _helpers
- classes
- dbt
- dlt/main
- ingestr/main
- lint
- presentation
- provider/_helpers
- rule_engine
- runtime/event_exporting
- _helpers
- classes
- main
- spec/contracts
- sql_values
- tests
- e2e/src/sqlbuild/cli/commands/main
- compile
- providers
- integration/src/sqlbuild
- cli/commands/main
- compiler
- analysis_session
- attachments
- compiled_project
- compile
- native_model_config
- contracts
- discovery
- explicit_references
- lineage
- model_config
- pipeline
- planner
- _helpers
- resolve
- sql_test_assembly
- main
- project_assembly
- scopes
- sql_test_glue
- executor
- build
- concurrent
- custom
- snowflake_metadata
- diff
- migrations/_helpers
- integrations/dbt
- lint/main
- rule_engine/native_rows
- unit
- scripts
- compiler_differential/_helpers/coverage
- native_module_paths/_helpers
- src/sqlbuild
- adapters
- bigquery
- databricks
- duckdb
- postgres
- snowflake
- sqlserver
- adapter/contract/classes/base_adapter
- cli
- commands
- _helpers
- entry
- runtime
- classes
- main
- build
- compile
- dbt
- entrypoint
- entry
- compile_reuse/main
- output/main
- plan_json
- plan
- compiler
- compile
- _helpers
- analysis
- config
- render
- contract_adoption/_helpers
- contracts
- dag/main
- discovery
- _helpers
- fingerprints/main
- frontier
- lineage
- macro_bridge/main
- manifest
- migrations/main
- pipeline
- _helpers
- target_validation
- main
- planner
- _helpers
- changes
- migrations
- planning
- resolve
- selection
- sql_test_assembly
- warehouse
- classes
- main
- execution
- identity
- pre_build
- python_nodes/_helpers
- scopes
- source_freshness/main
- sql_analysis/main
- executor
- clone/_helpers
- diff
- janitor
- _helpers
- main
- migrations/_helpers
- python_nodes/_helpers
- testing/main
- integrations
- dbt
- profile
- dlt
- ingestr
- lint
- _helpers
- rule_engine
- _helpers
- engine
- host
- run
- classes
- audit_facts
- main
- evaluate_rule
- evaluate
- run_rules
- skills
- runtime
- event_exporting
- classes
- output_capture/classes
- spec/contracts
- _helpers
- main
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
0 commit comments