perf(plan): cut plan and build planning time on large projects (#875)

* perf(plan): compute migration fingerprints once and only where needed

Rename discovery fingerprinted every unbuilt model under each rename
map even when no earlier model could match, and the plan fingerprinted
every entry again for storage. Skip matching without candidates, reuse
fingerprints across rename maps and phases, attach stored fingerprints
only to entries that will be built, and skip them for sqb plan, which
never stores them.

* perf(plan): remove quadratic union, ordering, and logging work

Resolving column reads through a UNION chain materialised every
branch's full argument tree at each level, so fixture validation grew
cubically with fixture rows. Read only the needed arguments. Order keys
with a heap instead of re-sorting the ready list, and look up the
package version once instead of per logged statement.

* perf(plan): skip rebinding sources whose inspection adds no evidence

Expression sources report column names without types, so every source
looked changed and every downstream model was analysed again during
planning. Treat inspected columns as changed only when names differ or a
reported type differs from the compile-time shape.

* perf(build): reuse migration fingerprints from the local compile cache

Every build fingerprinted every model it built, about 7 ms on average
and linear in query size, so a warm build of an unchanged 3,000-model
project spent 25 s recomputing identical values. Persist fingerprints in
the project-local compile cache, keyed by query, metadata, dialect,
visible renames, algorithm, and sqlbuild and polyglot versions, with
digest-checked entries that fall back to recomputing when damaged.
Also stop walking each query's local names twice.

* perf(plan): discover projects for plan and build like compile

Plan and build located every model's output columns during discovery,
which took about 2.4 s on the 3,000-model benchmark, while compile skips
it. The only consumer, contract diagnostics, already computes those
locations lazily when they are missing. Skip the eager scan for plan and
build as compile does.

* perf(plan): infer each distinct SQL test fixture once per plan

Fixture validation re-inferred the columns of every fixture body for
every test, although identical bodies under the same inference profile
always infer the same columns. Memoise inference per planning context,
keyed by fixture SQL and an equal profile, and bypass the memo for
profiles that carry a binding catalog.

* test(perf): guard plan and build time, memory, and scaling

Bound sqb plan --json on the 3,000-model inspection benchmark relative
to a warm compile measured in the same test, cap its peak RSS, and fail
when planning CPU grows super-linearly from a 300-model copy. Build a
1,000-model DuckDB benchmark from an empty warehouse with wall and RSS
limits, then plan against its stored state after editing models and
after renaming one. Raise the lineage ceilings to 3.0 s for CI headroom.

Refs: CHI-432

* fix(plan): treat non-ASCII fingerprint cache digests as cache misses

* test: count the migration fingerprint cache among compiler caches

Author: kvlonge

.github/workflows/verify.yml

@@ -84,8 +84,8 @@ jobs:
                 broad=true
                 performance=true
                 ;;
-              src/sqlbuild/cli/commands/_helpers/lineage/*|src/sqlbuild/cli/commands/_helpers/plan/*|src/sqlbuild/cli/commands/_helpers/scope/*|src/sqlbuild/cli/commands/main/inspection/*|src/sqlbuild/cli/commands/main/project/*|src/sqlbuild/cli/output/*|src/sqlbuild/presentation/*)
-                # Inspection command guards run inside the fresh-process compile jobs.
+              src/sqlbuild/cli/commands/_helpers/lineage/*|src/sqlbuild/cli/commands/_helpers/plan/*|src/sqlbuild/cli/commands/_helpers/scope/*|src/sqlbuild/cli/commands/_helpers/build_planning/*|src/sqlbuild/cli/commands/main/inspection/*|src/sqlbuild/cli/commands/main/project/*|src/sqlbuild/cli/output/*|src/sqlbuild/executor/build/*|src/sqlbuild/executor/run/*|src/sqlbuild/presentation/*)
+                # Inspection, plan, and build guards run inside the fresh-process compile jobs.
                 broad=true
                 performance=true
                 ;;

Makefile

@@ -237,6 +237,7 @@ test-e2e-cold-compile-performance:
 	env PYTHONUNBUFFERED=1 SQLBUILD_CONCURRENCY=4 uv run pytest \
 		tests/e2e/src/sqlbuild/cli/commands/main/compile/test_fresh_process_compile_performance.py \
 		tests/e2e/src/sqlbuild/cli/commands/main/compile/test_inspection_command_performance.py \
+		tests/e2e/src/sqlbuild/cli/commands/main/compile/test_build_performance.py \
 		-m cold_compile_performance -k "models_$(SQLBUILD_BENCHMARK_MODELS)" \
 		-vv --log-cli-level=INFO --color=yes
 

src/sqlbuild/cli/commands/_helpers/build_planning/invocation.py

@@ -35,7 +35,7 @@ def resolve_build_invocation(*, request: BuildCommandRequest) -> BuildInvocation
         request.project_dir if request.project_dir is not None else Path.cwd()
     )
     discovered_inputs: DiscoveredProjectInputs = discover_project_inputs(
-        project_dir=effective_project_dir
+        project_dir=effective_project_dir, extract_output_column_locations=False
     )
     effective_defer_clone_from: str | None = _resolve_defer_clone_from(
         discovered_inputs=discovered_inputs,

src/sqlbuild/cli/commands/_helpers/plan/invocation.py

@@ -32,7 +32,7 @@ def resolve_plan_invocation(*, request: PlanCommandRequest) -> PlanInvocation:
         request.project_dir if request.project_dir is not None else Path.cwd()
     )
     discovered_inputs: DiscoveredProjectInputs = discover_project_inputs(
-        project_dir=effective_project_dir
+        project_dir=effective_project_dir, extract_output_column_locations=False
     )
     _validate_preview_target(discovered_inputs=discovered_inputs, as_target=request.as_target)
     adapter_context: AdapterConnectionContext = resolve_adapter_connection_context(

src/sqlbuild/cli/commands/_helpers/plan/planning.py

@@ -53,6 +53,7 @@ def compile_plan_pipeline(
             resolve_python_run_selectors=request.include_python,
             max_microbatches=request.max_microbatches,
             selection_diagnostics=request.selection_diagnostics,
+            record_migration_fingerprints=False,
         ),
         hooks=ConnectionHooks(
             on_progress=invocation.planning_progress.on_progress,

src/sqlbuild/compiler/compile/_helpers/analysis/required_columns.py

@@ -217,7 +217,7 @@ def _resolve_set_operation_reads(
     branches: list[_RelationReads] = []
     required: list[CompiledLineageSourceFact] = []
     for side in ("left", "right"):
-        branch: Any | None = getattr(operation, "args", {}).get(side)
+        branch: Any | None = operation.arg(side)
         if branch is None:
             return _RelationReads(), _deduplicated_sources(required)
         branch_reads, branch_required = _resolve_query_reads(
@@ -228,7 +228,7 @@ def _resolve_set_operation_reads(
         branches.append(branch_reads)
         required.extend(branch_required)
     left, right = branches
-    if bool(getattr(operation, "args", {}).get("by_name")):
+    if bool(operation.arg("by_name")):
         right_by_name: dict[str, tuple[CompiledLineageSourceFact, ...]] = {
             name.casefold(): leaves for name, leaves in right.slots
         }

src/sqlbuild/compiler/compile/_helpers/assembly/source_bindings.py

@@ -82,7 +82,7 @@ def get_source_binding_diagnostics(
     for name, value in columns.items():
         if value:
             raw_shape: dict[str, str] = {column.name: column.type for column in value}
-            if raw_shape != shapes.get(name) or (
+            if _adds_physical_evidence(physical=raw_shape, known=shapes.get(name)) or (
                 profile.sql_analysis_dialect in CASE_SENSITIVE_BINDING_DIALECTS
                 and not profile.quoted_identifiers_ignore_case
             ):
@@ -202,6 +202,16 @@ def get_source_binding_diagnostics(
     return tuple(diagnostics)
 
 
+def _adds_physical_evidence(*, physical: dict[str, str], known: dict[str, str] | None) -> bool:
+    """Return whether inspected columns differ from the shape; untyped names match any type."""
+
+    if known is None or physical.keys() != known.keys():
+        return True
+    return any(
+        physical_type and physical_type != known[name] for name, physical_type in physical.items()
+    )
+
+
 def _placeholders(model: CompiledModel) -> dict[str, str] | None:
     values: object = model.config.values.get("placeholders")
     if isinstance(values, dict):

src/sqlbuild/compiler/pipeline/main/compile.py

@@ -247,6 +247,7 @@ def _build_result(
             auto_load_sources=options.auto_load_sources,
             selection_diagnostics=options.selection_diagnostics,
             plan_sql_tests=options.plan_sql_tests,
+            record_migration_fingerprints=options.record_migration_fingerprints,
         ),
         on_progress=on_progress,
         project_config=discovered_inputs.project_config,

src/sqlbuild/compiler/pipeline/models.py

@@ -61,6 +61,7 @@ class CompilePipelineOptions:
     max_microbatches: int | None = None
     selection_diagnostics: bool = False
     plan_sql_tests: bool = True
+    record_migration_fingerprints: bool = True
 
 
 @dataclass(frozen=True)

src/sqlbuild/compiler/planner/_helpers/fixtures/completion.py

@@ -11,7 +11,6 @@
 from sqlbuild.compiler.compile.main._analyze_columns_and_lineage import (
     analyze_resolved_column_reads,
 )
-from sqlbuild.compiler.compile.main._infer_fixture_columns import infer_fixture_column_facts
 from sqlbuild.compiler.compile.models import (
     CompiledLineageColumnFact,
     CompiledLineageSourceFact,
@@ -28,6 +27,7 @@
     ColumnTransformKind,
 )
 from sqlbuild.compiler.planner._helpers.fixtures.empty_fixture import is_empty_fixture_query
+from sqlbuild.compiler.planner.classes.fixture_column_inferences import FixtureColumnInferences
 from sqlbuild.compiler.planner.models import (
     FixtureColumnMetadata,
     FixtureRelationMetadata,
@@ -74,6 +74,7 @@ def build_relation_fixture_completion(
         fixture_groups=fixture_groups,
         relations=relations,
         adapter=adapter,
+        inferences=context.fixture_inferences,
     )
 
     fixture_keys_requiring_analysis: frozenset[FixtureKey] = _fixture_keys_requiring_analysis(
@@ -217,6 +218,7 @@ def _prepare_fixture_inputs(
     fixture_groups: FixtureGroups,
     relations: dict[FixtureKey, FixtureRelationMetadata],
     adapter: BaseAdapter,
+    inferences: FixtureColumnInferences,
 ) -> tuple[
     dict[FixtureKey, str],
     dict[FixtureKey, tuple[InferredColumn, ...]],
@@ -250,7 +252,7 @@ def _prepare_fixture_inputs(
                     continue
                 fixture_sql = empty_sql
             fixture_sql_by_key[key] = fixture_sql
-            inference: FixtureColumnInference | None = infer_fixture_column_facts(
+            inference: FixtureColumnInference | None = inferences.infer(
                 query_sql=fixture_sql,
                 inference_profile=adapter.expression_inference_profile(),
             )