Skip to content

perf: accelerate warm incremental compiles - #766

Merged
kvlonge merged 1 commit into
mainfrom
perf/warm-incremental-cache
Sep 21, 2026
Merged

kvlonge merged 1 commit into
mainfrom
perf/warm-incremental-cache

Conversation

@kvlonge

@kvlonge kvlonge commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Why

Fresh-process warm compiles still repeated whole-project work despite valid fact caches. On the representative 3k semantic project, the unchanged path took 21.0s; SQL-test artifact handling alone rebuilt the full model dependency index once per test.

Changes

  • Build SQL-test model closures once per compile and share them across artifact identities.
  • Add v10 dependency-aware compact analysis generations with content/version identities, Merkle-style upstream keys, partial reuse after edits, per-entry fallback, integrity validation, two-generation retention, and corruption/contention-as-miss behavior.
  • Report analysis batch hits, entry hits, misses, and bypasses in compile JSON.
  • Document the target cache architecture and delivered vertical slice.
  • Add 3k/5k/10k fresh-process CI guards covering cold, unchanged, leaf edit, macro edit, broad config invalidation, post-edit reuse, semantic fingerprints, cache size, and peak RSS under a 4 GiB cgroup.

Verification

  • Focused compiler, target-writer, JSON, and seed-target tests: 131 passed.
  • Sanctioned unit/integration run: 6,768 passed; 12 optional Dagster collection errors were resolved by installing all extras, then all 86 affected tests passed.
  • uv sync --all-extras
  • make check-ci
  • Public-repository hygiene scan and independent bounded review: clean.
  • Calibrated 3k: 5.99s unchanged, 6.17s leaf edit (about 400 MiB warm RSS).
  • Calibrated 10k: 20.18s unchanged, 21.23s leaf edit (about 1.0 GiB warm RSS); cold/config peaks remained below 2 GiB.

@kvlonge
kvlonge merged commit 82e353d into main Sep 21, 2026
24 checks passed
@kvlonge
kvlonge deleted the perf/warm-incremental-cache branch September 21, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant