Repository navigation
refactor(native): move discovery, scopes and the model config build into plain-Rust crates - #1182
Merged
Merged
Conversation
…o sqlbuild-discovery The header key sets each statement accepts and the options built from them now live in `sqlbuild-discovery` (`constants`, `model_file_options`, `declaration_file_options`, `sql_test_file_options`), and the bindings call them. The delta compile's header comparison moves too: `header_changes` digests each header key, decides how an edit changed a model header and whether a custom rule reads a changed key. The bindings keep only the recording of digests into the compile image. The comparison gains unit tests.
…ild-scopes The conversions from a cached full index to the report and projection views (`cached_report_records`, `cached_projection_index` and their parts), the scope path and qualified identity messages, the `as_posix` path text helpers, and the merge of new runtime usages (`merged_usages`, with `UsageKey`) now live in `sqlbuild-scopes`. The bindings read Python objects into those types and convert the results back, nothing more. `distinct_positions` and `merged_dependencies` become helpers of the usage merge, their only remaining caller.
…ls with it `sqlbuild-model-config` gains `config_values`: `ConfigNode<H>` holds the builtin Python values config is made of (None, bool, int, float, str, list, tuple, dict) as Rust data, and every other value as an opaque host value carrying its handle and the facts config logic reads of it, taken once with the same rules as before. It implements `AuthoredNode`, so the existing config scans read it unchanged, and `layered_config` merges one layer over another as `_merged_with_tag_union` does. The bindings convert Python values into nodes and project nodes back; opaque values project to the very same object. Externally described models now layer defaults, path defaults and headers through the typed merge, the first consumer.
…el-config The model config build (layering with tag, row-diff and tolerance unions, hook checks, the three model template passes, the target namespace and its templates, storage key removal, the no-macros check and header keys) now runs in `sqlbuild-model-config::model_config_build` over `ConfigNode`s. It asks its caller only for environment reads, the project-var text of non-string values and the exact error of a value that is not UTF-8 text, through `ConfigBuildHost`. `NativeModelConfigBuilder` converts the project inputs once per compile and each header once per model, and projects the built values back. Opaque values are carried as their own Python objects. Which header entries are still the header's own is tracked by provenance instead of object identity, so native header facts apply exactly where they did. Output is unchanged: the compiler differential passes, and a new integration test pins values, types, order, header keys and errors for builtin, subclassed, str-enum, date and opaque header values against the previous build. The build runs about 2% fewer instructions on a 600-model templated project.
`sqlbuild-model-config` gains the typed config reads (`checked_text`, `checked_flag`, `checked_integer`, `checked_texts`, `checked_cursor_bound`) with the type rules and error contents of the `get_config_*` readers; the Python-facing readers now check through them and still return the caller's own objects. A config record keeps its values as typed nodes, converted from `values` on first read and cached on the record. The planner's phase objects, change detection, display plan, column migrations, lineage and SQL test cursor window read through it instead of the Python dict.
Since configs are built as typed nodes, every built container was projected as a fresh Python object. The model loop reuses natively parsed header columns only when `values["columns"]` is the header's own object, so any model with `columns` beside another config key fell back to parsing its columns again in Python. The build now reports which header keys still hold the header's own values (the provenance it already tracks, with set-aside hooks included), and the projection uses the header's objects for them, so identity holds exactly where it did before. Stage counters record whether a model's header columns were reused or parsed again, and an integration test pins the native path for models with `columns` beside `materialized`, `tags` or a description.
…rints it exactly Opaque config values were read eagerly at conversion: `str()` twice, `repr()`, a date check and three hook-entry checks, with a raising `__repr__` read as empty. Only cheap facts (kind, text, numbers) are now read when a value is converted. Date checks and display text are asked of the value's handle when a scan needs them, and the build asks its host for `str()` and hook-entry checks, so their errors raise as before. Text is produced in Rust only where it is exact: `None`, booleans, integers, floats and a string's own text. A string's repr, containers and opaque values use Python's `str()`/`repr()` of the projected object for template operands and tag/row-diff unions. That replaces an approximate Rust repr that printed some format, private-use and unassigned characters unescaped. Hook-template context values read only `database`, `schema` and `alias` instead of converting the whole values dict. Integration tests pin container text, item errors in template operands and unions, and that the build never reads an opaque value's text.
…to whole Every header value was converted into config nodes, then cloned through layering and expansion, even large `columns` trees the build only passes through. A header container whose native parse found no template or macro, under a key the build never reads item by item (anything but tags, row-diff, hook and storage keys), is now carried as the header's own object. Dense compiles return to the instruction count they had before the typed build. Because such containers stay the header's own objects, models whose other config values hold templates also reuse their natively parsed columns now; an integration case pins it.
The first typed read of a config record converted every value in `config.values` into typed nodes, just to read one scalar key such as `incremental_mode`. A config record now keeps a per-key cache: a value is converted the first time its key is read, through the same `values.get(key)`, and kept for later reads. Checked reads keep their type rules and errors. Lineage's microbatch check compares the mode text as Python's `==` does again, so a string that cannot be encoded reads as not-microbatch instead of raising. A stage counter records each typed conversion, and an integration test pins that keys no planner reader reads are never converted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Second step of splitting
sqlbuild-pythoninto plain-Rust crates: move discovery, scope and model config logic out of the PyO3 bindings, so it can be tested and reused without Python, with unchanged output.Changes
sqlbuild-discovery; plain scope report, index and usage logic intosqlbuild-scopes.sqlbuild-model-configgainsConfigNode, a typed config value: builtins as Rust data, any other value an opaque handle to the original object.sqlbuild-model-config::model_config_build; the bindings convert inputs once and project results back.str()/repr()is used otherwise, with the same errors.Verification
make check-ci,cargo test, clippy-D warnings, fensu: clean; XSB072/073 budgets only lowered.make compiler-differential: 373/373; planner differential: 11/11 against a baseline recorded on the base commit.sqb planwarm and cold on dense 2,000 and a 10,000-model project within run-to-run noise (0.3%), plan output byte-identical.