Skip to content

perf(compile): cut cache-write overhead and fresh compile preparation cost - #882

Merged
kvlonge merged 9 commits into
mainfrom
perf/chi-411-386-fresh-and-cache
Sep 29, 2026
Merged

kvlonge merged 9 commits into
mainfrom
perf/chi-411-386-fresh-and-cache

Conversation

@kvlonge

@kvlonge kvlonge commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Why

Writing the compile cache made cold compiles 14-18% slower than --no-cache, and fresh compile was ~1.22x of 0.119.1. Profiling on 0.126.2 showed a whole-SQL identifier regex in binding-query preparation, dataclass construction just to serialise cache rows, full model SQL sent to native SQL-test chain resolution, and per-model scans in metadata validation.

Changes

  • Qualified-reference scan searches each reference name directly (old regex kept as fallback for quoted, commented or non-ASCII SQL).
  • Cache writes serialise compact lineage rows directly and compute each output signature once; stored payloads byte-identical, so no cache version change.
  • SQL-test chain resolution sends names and dependencies only.
  • Metadata validation serialises each select once and indexes argument column types.
  • New guard: cold cache-writing compile vs --no-cache (CPU <= 1.10, wall <= 1.15, same fingerprint). Cache job warm/edit-to-cold ratios raised to 0.70 because cold got faster while edit and warm were unchanged (to be tightened with the warm-path work).

Verification

  • Cache overhead at 3k: +18.3% -> +6.5% wall; dense 5k cold 42.5 s -> 37.2 s; fresh 3000 9.70 s -> 9.27 s (1.175x of 0.119.1, so fresh budgets stay as they are).
  • Fingerprints, compiled artifacts, diagnostics and cache payloads identical to main.
  • make test, make check-ci, cache and cold-compile performance targets under 4 GiB; one independent review.

@kvlonge
kvlonge enabled auto-merge (squash) September 29, 2026 08:50
@kvlonge
kvlonge merged commit aea4484 into main Sep 29, 2026
29 checks passed
@kvlonge
kvlonge deleted the perf/chi-411-386-fresh-and-cache branch September 29, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant