Skip to content

Commit 6f3bee6

Browse files
authored
refactor: digest rule facts and read declared columns natively (#1122)
* refactor(rules): digest custom-rule facts natively Recompute and digest recorded custom-rule facts from the native compiled project in Rust instead of rebuilding Python fact views and encoding them with orjson. Rule hosts digest live filesystem reads through the same native encoder, project-tree file reads are native, and the custom-rule cache runtime version is bumped so entries keyed on the old digests miss. * refactor(rules): read per-model rule inputs natively Read model names, project paths and materializations for custom-rule cache rows, rule selection, run summaries and the native rules request through the compiled model views directly instead of per-model attribute lookups. * refactor(compile): read declared model columns and SQL from native facts Answer declared column rows, custom-rule SQL facts and semantic-check audit metadata of compiled model views from the native facts and schema records the compile already holds, instead of rebuilding schema entries and reading each column back through Python attributes. * refactor(compile): build declared column facts natively and read them in schema readers Build each model's declared column facts (name, type, nullability, description, migration source, audit definitions and location) from the parsed header and the named schema while the column specs are built, instead of reading every merged and enum-resolved column back through Python attributes. Audit attachment, declared relation types and nullability, relation keys and semantic-check audit metadata read these facts and touch a column's Python spec only for its audits. Model input records keep the model name, so readers no longer derive it from the file path. * refactor(compile): run model analysis directly from project assembly Project assembly now starts, runs and finishes the native model analysis session through Rust calls on typed request and result structs, instead of building a NativeModelAnalysisRequest record and calling analyze_native_model_sql and start_model_analysis_session through the extension module. The request and result record classes and both module functions are removed. The analysis session tests now assemble real projects with a cold, a warm and no analysis cache and compare those assemblies; their goldens are unchanged. Tests that could only reach unreachable internal invariants through the removed module seam are dropped; the duplicate-name invariant stays covered by the native session tests. * perf(rules): lint shared compile expansions without copying them up front Early SQL lint and SQL rules keep references to the compile's shared authored and expanded SQL texts instead of copying every model's expansion into owned strings before linting. A file's expansion is converted only when that file is linted and dropped afterwards. * perf(compile): free the analysis request early and keep schema facts lean Model analysis converts its session request before running the session, so the per-model request rows are freed instead of living through the run. Declared column facts keep only what readers use: whether a column is described, and the location of columns declared outside the model file. Attached schemas no longer copy every column fact; the declared-relation reader asks for them on its own. * fix: tag numbers in custom-rule fact digests and keep Windows read errors and model path lookups * fix: load project Python from current source so same-size edits are never hidden by cached bytecode * refactor: name user-code spec arguments and split root registration * fix: keep installed packages on cached bytecode and order the user-code finder after built-in importers * test: pause the slow lint macro by expanded file, not by thread
1 parent ecc31b7 commit 6f3bee6

110 files changed

Lines changed: 3534 additions & 2059 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎crates/sqlbuild-analysis/src/compiled_project/models.rs‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -73,9 +73,13 @@ pub struct SchemaColumnFact {
7373
pub name: String,
7474
pub data_type: Option<String>,
7575
pub nullable: Option<bool>,
76-
pub audit_count: u32,
77-
/// Whether the column is declared outside the model file, in a named schema.
78-
pub declared_outside_model: bool,
76+
/// Whether the column declares a description.
77+
pub described: bool,
78+
pub migrate_from: Option<String>,
79+
/// Each column audit's definition name, in declaration order.
80+
pub audit_definitions: Vec<String>,
81+
/// The file of a column declared outside the model file, in a named schema.
82+
pub location_path: Option<String>,
7983
}
8084

8185
/// One declared dynamic-column family; `key` is its name casefolded.

‎crates/sqlbuild-cache/src/hosted_shapes/constants.rs‎

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
//! The Python shape names of the natively hosted compiler shapes.
22
33
/// The Python model type a natively hosted compiler shape is captured as.
4-
pub const HOSTED_SHAPE_TYPES: [(&str, &str); 492] = [
4+
pub const HOSTED_SHAPE_TYPES: [(&str, &str); 490] = [
55
(
66
"sqlbuild._native:AlignedInterval",
77
"sqlbuild.cursor_algebra.models:AlignedInterval",
@@ -1104,14 +1104,6 @@ pub const HOSTED_SHAPE_TYPES: [(&str, &str); 492] = [
11041104
"sqlbuild._native:NativeLintError",
11051105
"sqlbuild.lint.exceptions:NativeLintError",
11061106
),
1107-
(
1108-
"sqlbuild._native:NativeModelAnalyses",
1109-
"sqlbuild.compiler.analysis_session.models:NativeModelAnalyses",
1110-
),
1111-
(
1112-
"sqlbuild._native:NativeModelAnalysisRequest",
1113-
"sqlbuild.compiler.analysis_session.models:NativeModelAnalysisRequest",
1114-
),
11151107
(
11161108
"sqlbuild._native:NativeModelConfigInputs",
11171109
"sqlbuild.compiler.compile.models:NativeModelConfigInputs",

‎crates/sqlbuild-python/src/bindings/_helpers/analysis_session/model_analyses.rs‎

Lines changed: 141 additions & 174 deletions
Large diffs are not rendered by default.

‎crates/sqlbuild-python/src/bindings/_helpers/analysis_session/registration.rs‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,14 @@
33
use pyo3::prelude::{Bound, PyModule, PyModuleMethods, PyResult};
44

55
use crate::bindings::_helpers::analysis_session::{
6-
fixture_columns, fixture_inferences, model_analyses, oracle, outcome_objects, query_columns,
7-
session, stage_functions,
6+
fixture_columns, fixture_inferences, oracle, outcome_objects, query_columns, session,
7+
stage_functions,
88
};
99

1010
pub(crate) fn register(module: &Bound<'_, PyModule>) -> PyResult<()> {
1111
session::register(module)?;
1212
fixture_columns::register(module)?;
1313
module.add_class::<fixture_inferences::FixtureColumnInferences>()?;
14-
model_analyses::register(module)?;
1514
outcome_objects::register(module)?;
1615
query_columns::register(module)?;
1716
stage_functions::register(module)?;

‎crates/sqlbuild-python/src/bindings/_helpers/analysis_session/session.rs‎

Lines changed: 22 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
33
use pyo3::prelude::{Bound, Py, PyAny, PyErr, PyModule, PyModuleMethods, PyRef, PyResult, Python};
44
use pyo3::types::PyDict;
5-
use pyo3::{pyclass, pyfunction, pymethods, wrap_pyfunction};
5+
use pyo3::{pyclass, pyfunction, pymethods};
66
use sqlbuild_analysis::assembly::analysis_session::main::attach_analysis_cache::attach_analysis_cache;
77
use sqlbuild_analysis::assembly::analysis_session::main::expression_shapes::expression_shapes;
88
use sqlbuild_analysis::assembly::analysis_session::main::finish_analysis_session::finish_analysis_session;
@@ -58,7 +58,7 @@ type ModelRow = (
5858
Vec<FamilyRow>,
5959
);
6060
/// The profile settings, relation shapes and models one session analyses.
61-
type RequestRow = (
61+
pub(crate) type RequestRow = (
6262
String,
6363
bool,
6464
Pairs,
@@ -72,7 +72,7 @@ type RequestRow = (
7272
Vec<ModelRow>,
7373
);
7474
/// `(publications, failures)`.
75-
type StepRow = (Shapes, Vec<String>);
75+
pub(crate) type StepRow = (Shapes, Vec<String>);
7676
/// One model's analysis, lineage source and cleaned SQL.
7777
type OutcomeRow = (
7878
bool,
@@ -108,9 +108,9 @@ type PivotRequestRow = (
108108
/// `("absent" | "deferred" | "proof", proof)`.
109109
pub(crate) type ContractRow = (&'static str, Option<ProofRow>);
110110
/// `(hits, misses, stored, why the store was not read or saved)`.
111-
type CacheStatsRow = (usize, usize, usize, Option<String>);
111+
pub(crate) type CacheStatsRow = (usize, usize, usize, Option<String>);
112112
/// `(catalog schema additions, analysis-shape names, whether each model's analysis succeeded)`.
113-
type FinishRow = (Shapes, Vec<String>, Vec<bool>);
113+
pub(crate) type FinishRow = (Shapes, Vec<String>, Vec<bool>);
114114

115115
/// One compile's native model analysis; an internal failure raises `NativeCompilerError`.
116116
#[pyclass(module = "sqlbuild._native")]
@@ -160,19 +160,9 @@ fn internal_error(reason: &str) -> PyErr {
160160
compiler_error(native_failure(SESSION_CONTEXT, reason))
161161
}
162162

163-
#[pymethods]
164163
impl NativeModelAnalysisSession {
165-
/// Models whose output names and lineage the finished session keeps for later stages.
166-
#[getter]
167-
pub(crate) fn fact_models(&self) -> Vec<String> {
168-
self.finished
169-
.as_ref()
170-
.map(finished_fact_models)
171-
.unwrap_or_default()
172-
}
173-
174164
/// Analyse every model.
175-
fn run(&mut self, py: Python<'_>) -> PyResult<StepRow> {
165+
pub(crate) fn run(&mut self, py: Python<'_>) -> PyResult<StepRow> {
176166
let mut session: AnalysisSession = self.running()?;
177167
let result: Result<(AnalysisSession, SessionStep), String> =
178168
py.compiler_detach(move || {
@@ -188,7 +178,7 @@ impl NativeModelAnalysisSession {
188178
}
189179

190180
/// Every model's outcome; saves the cache best-effort.
191-
fn finish(&mut self, py: Python<'_>) -> PyResult<FinishRow> {
181+
pub(crate) fn finish(&mut self, py: Python<'_>) -> PyResult<FinishRow> {
192182
let mut session: AnalysisSession = self.running()?;
193183
let cache: Option<(NativeStore, AnalysisCacheStats)> = take_analysis_cache(&mut session);
194184
let (outcome, finished) = catch_compiler_panic(|| finish_analysis_session(session))
@@ -213,9 +203,21 @@ impl NativeModelAnalysisSession {
213203
self.outcomes = models.into_iter().zip(dynamic_contracts).collect();
214204
Ok((schema_additions, analysis_names, succeeded))
215205
}
206+
}
207+
208+
#[pymethods]
209+
impl NativeModelAnalysisSession {
210+
/// Models whose output names and lineage the finished session keeps for later stages.
211+
#[getter]
212+
pub(crate) fn fact_models(&self) -> Vec<String> {
213+
self.finished
214+
.as_ref()
215+
.map(finished_fact_models)
216+
.unwrap_or_default()
217+
}
216218

217219
/// Each requested model's raw binding diagnostic rows, in request order.
218-
fn binding_rows(&self) -> Vec<Vec<DiagnosticRow>> {
220+
pub(crate) fn binding_rows(&self) -> Vec<Vec<DiagnosticRow>> {
219221
self.outcomes
220222
.iter()
221223
.map(|(outcome, _)| outcome.analysis.binding_diagnostics.clone())
@@ -236,7 +238,7 @@ impl NativeModelAnalysisSession {
236238

237239
/// The analysis cache's `(hits, misses, stored, failure)`; None without a cache.
238240
#[getter]
239-
fn cache_stats(&self) -> Option<CacheStatsRow> {
241+
pub(crate) fn cache_stats(&self) -> Option<CacheStatsRow> {
240242
self.cache_stats.clone()
241243
}
242244

@@ -265,9 +267,7 @@ impl NativeModelAnalysisSession {
265267
}
266268

267269
/// Start a session on `catalog`.
268-
#[pyfunction]
269-
#[pyo3(signature = (catalog, request, cache=None, adapter_rules=None))]
270-
fn start_model_analysis_session(
270+
pub(crate) fn start_model_analysis_session(
271271
catalog: PyRef<'_, ProjectCatalog>,
272272
request: RequestRow,
273273
cache: Option<(PathBuf, String)>,
@@ -514,6 +514,5 @@ fn pivot_model(row: PivotModelRow) -> PivotModel {
514514

515515
pub(crate) fn register(module: &Bound<'_, PyModule>) -> PyResult<()> {
516516
module.add_class::<NativeModelAnalysisSession>()?;
517-
module.add_function(wrap_pyfunction!(start_model_analysis_session, module)?)?;
518517
Ok(())
519518
}

‎crates/sqlbuild-python/src/bindings/_helpers/compile_driver/assembly_analysis.rs‎

Lines changed: 19 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,9 @@ use pyo3::types::{PyBool, PyDict, PyDictMethods, PyFrozenSet, PyString, PyTuple}
66
use sqlbuild_analysis::assembly::analysis_session::main::native_dialect::native_dialect;
77

88
use crate::bindings::_helpers::analysis::binding_catalog::{BindingCatalog, ignores_quoted_case};
9+
use crate::bindings::_helpers::analysis_session::model_analyses::{
10+
ModelAnalyses, ModelAnalysisRequest, analyze_models,
11+
};
912
use crate::bindings::_helpers::compile_driver::analysis_cache::analysis_cache_context;
1013
use crate::bindings::_helpers::compile_driver::assembly::AssemblyRequest;
1114
use crate::bindings::_helpers::compile_driver::compile_timings::{
@@ -30,8 +33,8 @@ pub(crate) struct AnalysisContext<'py> {
3033
pub(crate) nullability: Bound<'py, PyAny>,
3134
pub(crate) complete_schemas: Bound<'py, PyAny>,
3235
pub(crate) dynamic_families: Bound<'py, PyAny>,
33-
/// The `NativeModelAnalyses`, or `None` when no model was analysed.
34-
pub(crate) analyses: Bound<'py, PyAny>,
36+
/// The model analyses, or `None` when SQL analysis does not run.
37+
pub(crate) analyses: Option<ModelAnalyses<'py>>,
3538
}
3639

3740
/// Whether `model_input` is selected for analysis by `names` (`None` selects every model).
@@ -90,7 +93,7 @@ pub(crate) fn analysis_context<'py>(
9093
nullability,
9194
complete_schemas,
9295
dynamic_families,
93-
analyses: py.None().into_bound(py),
96+
analyses: None,
9497
};
9598
if context.sql_analysis_enabled.is_truthy()? {
9699
record_expression_shapes(inputs, &context)?;
@@ -324,10 +327,10 @@ fn record_expression_shapes<'py>(
324327
fn model_analyses<'py>(
325328
request: &AssemblyRequest,
326329
context: &AnalysisContext<'py>,
327-
) -> PyResult<Bound<'py, PyAny>> {
330+
) -> PyResult<Option<ModelAnalyses<'py>>> {
328331
let py: Python<'py> = context.profile.py();
329332
if !context.sql_analysis_enabled.is_truthy()? {
330-
return Ok(py.None().into_bound(py));
333+
return Ok(None);
331334
}
332335
let inputs: &Bound<'py, PyAny> = request.inputs.bind(py);
333336
let names: &Bound<'py, PyAny> = request.analysis_model_names.bind(py);
@@ -346,8 +349,6 @@ fn model_analyses<'py>(
346349
};
347350
record_compile_timing("model_analysis_ms", || {
348351
record_compile_cpu_timing(py, "model_analysis_cpu_ms", || {
349-
let (known_functions, known_types): (Bound<'py, PyAny>, Bound<'py, PyAny>) =
350-
declared_names(inputs, &context.column_types)?;
351352
let mut model_inputs: Vec<Bound<'py, PyAny>> = Vec::new();
352353
for model_input in inputs.getattr("model_inputs")?.try_iter()? {
353354
let model_input: Bound<'py, PyAny> = model_input?;
@@ -357,29 +358,17 @@ fn model_analyses<'py>(
357358
model_inputs.push(model_input);
358359
}
359360
}
360-
let analysis_request: Bound<'py, PyAny> = call_with(
361-
&registry::class(py, RecordId::NativeModelAnalysisRequest)?,
362-
&[
363-
("known_functions", known_functions),
364-
("known_types", known_types),
365-
("model_inputs", PyTuple::new(py, model_inputs)?.into_any()),
366-
("column_nullability_by_table", context.nullability.clone()),
367-
("column_types_by_table", context.column_types.clone()),
368-
("inference_profile", context.profile.clone()),
369-
("allow_compact_analysis", flags.0.clone()),
370-
("rich_type_inference", flags.1.clone()),
371-
("analysis_cache", cache.clone()),
372-
("complete_binding_schemas", context.complete_schemas.clone()),
373-
(
374-
"dynamic_families_by_table",
375-
context.dynamic_families.clone(),
376-
),
377-
],
378-
)?;
379-
call_with(
380-
&registry::native(py, NativeAttr::AnalyzeNativeModelSql)?,
381-
&[("request", analysis_request)],
382-
)
361+
analyze_models(&ModelAnalysisRequest {
362+
model_inputs,
363+
profile: context.profile.clone(),
364+
column_types: context.column_types.clone(),
365+
nullability: context.nullability.clone(),
366+
complete_schemas: context.complete_schemas.clone(),
367+
dynamic_families: context.dynamic_families.clone(),
368+
rich_type_inference: rich,
369+
cache,
370+
})
371+
.map(Some)
383372
})
384373
})
385374
}

‎crates/sqlbuild-python/src/bindings/_helpers/compile_driver/assembly_facts.rs‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -24,16 +24,15 @@ pub(crate) fn native_project_facts<'py>(
2424
let py: Python<'py> = inputs.py();
2525
let model_inputs: Bound<'py, PyAny> = inputs.getattr("model_inputs")?;
2626
let dialect: Bound<'py, PyAny> = context.profile.getattr("sql_analysis_dialect")?;
27-
let analysed: bool = !context.analyses.is_none();
28-
let succeeded: Bound<'py, PyAny> = if analysed {
29-
context.analyses.getattr("succeeded")?
30-
} else {
31-
PyFrozenSet::empty(py)?.into_any()
32-
};
33-
let positions: Bound<'py, PyAny> = if analysed {
34-
context.analyses.getattr("positions")?
35-
} else {
36-
PyDict::new(py).into_any()
27+
let (succeeded, positions): (Bound<'py, PyAny>, Bound<'py, PyAny>) = match &context.analyses {
28+
Some(analyses) => (
29+
analyses.succeeded.clone().into_any(),
30+
analyses.positions.clone().into_any(),
31+
),
32+
None => (
33+
PyFrozenSet::empty(py)?.into_any(),
34+
PyDict::new(py).into_any(),
35+
),
3736
};
3837
let syntax_checks: Bound<'py, PyAny> =
3938
model_syntax_checks(&model_inputs.clone(), &names.clone(), &succeeded)

0 commit comments

Comments
 (0)