Repository navigation
refactor(native): move the compiled project records into sqlbuild-project - #1197
Merged
Merged
Conversation
…o sqlbuild-discovery The header key sets each statement accepts and the options built from them now live in `sqlbuild-discovery` (`constants`, `model_file_options`, `declaration_file_options`, `sql_test_file_options`), and the bindings call them. The delta compile's header comparison moves too: `header_changes` digests each header key, decides how an edit changed a model header and whether a custom rule reads a changed key. The bindings keep only the recording of digests into the compile image. The comparison gains unit tests.
…ild-scopes The conversions from a cached full index to the report and projection views (`cached_report_records`, `cached_projection_index` and their parts), the scope path and qualified identity messages, the `as_posix` path text helpers, and the merge of new runtime usages (`merged_usages`, with `UsageKey`) now live in `sqlbuild-scopes`. The bindings read Python objects into those types and convert the results back, nothing more. `distinct_positions` and `merged_dependencies` become helpers of the usage merge, their only remaining caller.
…ls with it `sqlbuild-model-config` gains `config_values`: `ConfigNode<H>` holds the builtin Python values config is made of (None, bool, int, float, str, list, tuple, dict) as Rust data, and every other value as an opaque host value carrying its handle and the facts config logic reads of it, taken once with the same rules as before. It implements `AuthoredNode`, so the existing config scans read it unchanged, and `layered_config` merges one layer over another as `_merged_with_tag_union` does. The bindings convert Python values into nodes and project nodes back; opaque values project to the very same object. Externally described models now layer defaults, path defaults and headers through the typed merge, the first consumer.
…el-config The model config build (layering with tag, row-diff and tolerance unions, hook checks, the three model template passes, the target namespace and its templates, storage key removal, the no-macros check and header keys) now runs in `sqlbuild-model-config::model_config_build` over `ConfigNode`s. It asks its caller only for environment reads, the project-var text of non-string values and the exact error of a value that is not UTF-8 text, through `ConfigBuildHost`. `NativeModelConfigBuilder` converts the project inputs once per compile and each header once per model, and projects the built values back. Opaque values are carried as their own Python objects. Which header entries are still the header's own is tracked by provenance instead of object identity, so native header facts apply exactly where they did. Output is unchanged: the compiler differential passes, and a new integration test pins values, types, order, header keys and errors for builtin, subclassed, str-enum, date and opaque header values against the previous build. The build runs about 2% fewer instructions on a 600-model templated project.
`sqlbuild-model-config` gains the typed config reads (`checked_text`, `checked_flag`, `checked_integer`, `checked_texts`, `checked_cursor_bound`) with the type rules and error contents of the `get_config_*` readers; the Python-facing readers now check through them and still return the caller's own objects. A config record keeps its values as typed nodes, converted from `values` on first read and cached on the record. The planner's phase objects, change detection, display plan, column migrations, lineage and SQL test cursor window read through it instead of the Python dict.
Since configs are built as typed nodes, every built container was projected as a fresh Python object. The model loop reuses natively parsed header columns only when `values["columns"]` is the header's own object, so any model with `columns` beside another config key fell back to parsing its columns again in Python. The build now reports which header keys still hold the header's own values (the provenance it already tracks, with set-aside hooks included), and the projection uses the header's objects for them, so identity holds exactly where it did before. Stage counters record whether a model's header columns were reused or parsed again, and an integration test pins the native path for models with `columns` beside `materialized`, `tags` or a description.
…rints it exactly Opaque config values were read eagerly at conversion: `str()` twice, `repr()`, a date check and three hook-entry checks, with a raising `__repr__` read as empty. Only cheap facts (kind, text, numbers) are now read when a value is converted. Date checks and display text are asked of the value's handle when a scan needs them, and the build asks its host for `str()` and hook-entry checks, so their errors raise as before. Text is produced in Rust only where it is exact: `None`, booleans, integers, floats and a string's own text. A string's repr, containers and opaque values use Python's `str()`/`repr()` of the projected object for template operands and tag/row-diff unions. That replaces an approximate Rust repr that printed some format, private-use and unassigned characters unescaped. Hook-template context values read only `database`, `schema` and `alias` instead of converting the whole values dict. Integration tests pin container text, item errors in template operands and unions, and that the build never reads an opaque value's text.
…to whole Every header value was converted into config nodes, then cloned through layering and expansion, even large `columns` trees the build only passes through. A header container whose native parse found no template or macro, under a key the build never reads item by item (anything but tags, row-diff, hook and storage keys), is now carried as the header's own object. Dense compiles return to the instruction count they had before the typed build. Because such containers stay the header's own objects, models whose other config values hold templates also reuse their natively parsed columns now; an integration case pins it.
The first typed read of a config record converted every value in `config.values` into typed nodes, just to read one scalar key such as `incremental_mode`. A config record now keeps a per-key cache: a value is converted the first time its key is read, through the same `values.get(key)`, and kept for later reads. Checked reads keep their type rules and errors. Lineage's microbatch check compares the mode text as Python's `==` does again, so a string that cannot be encoded reads as not-microbatch instead of raising. A stage counter records each typed conversion, and an integration test pins that keys no planner reader reads are never converted.
Cursor model bounds, model SQL resolution, append inclusivity, hook entries, cursor overrides, cursor ranges and cursor policies read the model's config through the typed per-key values instead of the Python dict. String comparisons use `config_text_is`, which compares as Python's `==` does, so they raise only where the old checked reads did.
Model plan entries (cursor and snapshot fields, batch concurrency, hard deletes, merge exclusions, storage settings, execution limits, lookback), cursor input relations, fixture authority and materialization types read the model's config through the typed per-key values. Helpers shared with Python callers keep reading the values dict.
…ld-python # Conflicts: # crates/sqlbuild-python/src/bindings/_helpers/model_config/typed_config.rs # fensu.toml
Cursor bound reads of upstream models, the semantic source analysis check, source deferral hook reads, rename detection in the snapshot and watermark contract checks read config through the typed per-key values.
Clone plan enablement, contract identity metadata, migration compatibility and declared migrations, full refresh, retention materialization and tag selectors read the model's config through the typed per-key values instead of the Python dict.
An empty plain-Rust crate for the compiled project's records, placed above sqlbuild-refactor and below sqlbuild-python in the native layer order. The records move into it in the following commits.
…project Reference rows, destination facts, metadata texts and query expansion records are plain Rust types in sqlbuild-project. Expansion spans use sqlbuild-sqltext's `CharSpan`, which `SpanRow` already copied field for field. The tuples Python reads are built by small positional rows in the bindings, so Python sees the same values.
`ModelInputRecord` and `SchemaRecord` are generic records in sqlbuild-project over a `ProjectHost`, whose associated types are the host's slot beside each record. The extension's slot keeps the model file, config, declarations, usages, header columns and the lazily built Python projections, so every Python value stays the same object.
…qlbuild-project `CompiledRecord` and `AnalysisRecord` are generic records in sqlbuild-project; their lazily built Python projections are the extension's host slots. The lineage row and edge reads are plain methods with a unit test, and the Python projections of lineage, binding diagnostics and contract proof are an extension trait in the bindings, built once as before.
…lbuild-project Attached sources, functions and audits and their compiled records are generic records in sqlbuild-project. Each keeps its Python values in an attached slot and its lazily built projections in a record slot. Namespace, dependency and name reads are plain methods; key facts and entries are an extension trait in the bindings. The generic audit's fallible reads still run in their original order, so the same error is raised first.
`SeedRecord` is a generic record in sqlbuild-project with its name and resolved destination; the extension's slot keeps the discovered CSV file, declaration and schema file and the key its views build. The reads through those objects are an extension trait in the bindings.
…roject `CompiledProject` holds the project facts and every resource's records in sqlbuild-project, with input retention, schema fact lookup and the record lists as plain methods and a unit test. `NativeCompiledProject` wraps it and keeps only the Python view classes, the shared inferred column projections and the attachment loop.
The crate table names the compiled project's records, the plain reads they answer, and how the host's values and lazy views sit beside them.
kvlonge
enabled auto-merge (squash)
October 21, 2026 07:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The planner and the compile driver read the compiled project through Python records. Before either can become a plain-Rust crate, the compiled project's records must be plain Rust. This adds
sqlbuild-projectand moves the records into it. Behaviour and output do not change.Changes
sqlbuild-project, placed betweensqlbuild-refactorandsqlbuild-pythonin the native layer order.CompiledProjectand the per-resource records (model inputs and schemas, compiled models and analysis, sources, functions, audits, seeds) are generic over aProjectHosttrait. Each record has one host slot holding its Python values and lazily built views.sqlbuild-sqltext'sCharSpan. The tuples Python reads are built by small positional rows.Verification
make check-ci, clippy-D warnings, fensu with 0 faults; budgets unchanged (XSB072 3900, XSB073 1530, XSB075 0).cargo test -p sqlbuild-project; compiler differential 373/373; planner differential 11/11; focused compiler, attachment, seed, audit and compile pytest.sqb plan(warm and cold, two large projects) are at instruction-count parity withmain, and plan output is byte-identical.sqlbuild-pythonclean release and incremental builds are at instruction-count parity withmain. The new crate builds alongsidesqlbuild-rules, so the critical path is unchanged.