|
| 1 | +# Compiler cache architecture |
| 2 | + |
| 3 | +The compiler cache exists to avoid proving unchanged facts again. It must never turn an uncertain |
| 4 | +input into a cache hit, and a missing, stale, corrupt, oversized, or contended cache must behave like |
| 5 | +an ordinary uncached compile. |
| 6 | + |
| 7 | +This document defines the target architecture. The current vertical slice implements versioned |
| 8 | +model-analysis entries, dependency-aware compact analysis generations, shared SQL-test closure |
| 9 | +indexes, bounded compact-generation retention, corruption fallback, and analysis hit/miss telemetry. |
| 10 | +Project inventory, discovery/attachment reuse, and command snapshots remain later layers; their |
| 11 | +sections below are design constraints rather than claims of current implementation. |
| 12 | + |
| 13 | +## Goals |
| 14 | + |
| 15 | +- Make unchanged fresh-process compiles proportional to project inventory and requested output, not |
| 16 | + total SQL complexity. |
| 17 | +- Make a small edit proportional to the changed resource and its semantic dependents. |
| 18 | +- Reuse downstream analysis when an upstream implementation changes without changing its exported |
| 19 | + column interface. |
| 20 | +- Preserve diagnostics, bindings, types and nullability, lineage, tests, audits, hooks, macros, |
| 21 | + ordering, artifacts, and exit codes exactly. |
| 22 | +- Keep cache data project-local, bounded, inspectable, non-executable, and safe under concurrent |
| 23 | + compiler processes. |
| 24 | + |
| 25 | +## Non-goals |
| 26 | + |
| 27 | +- The cache is not a second source of project truth. |
| 28 | +- The cache does not make arbitrary Python execution deterministic. A stage with dynamic inputs that |
| 29 | + are not represented in its identity must bypass persistent reuse. |
| 30 | +- The cache does not store pickles, code objects, callables, imported modules, adapter instances, or |
| 31 | + open connections. |
| 32 | +- The cache does not replace warehouse execution state or change build selection semantics. |
| 33 | + |
| 34 | +## Layers |
| 35 | + |
| 36 | +### 1. Project inventory |
| 37 | + |
| 38 | +The inventory records every compiler-owned authored input as a normalized project-relative path, |
| 39 | +resource role, content digest, size, and filesystem identity metadata. Configuration, ignore files, |
| 40 | +adapter identity, compiler and SQL-analysis engine versions, selected target, CLI variables, and |
| 41 | +compile modes are explicit identity inputs. |
| 42 | + |
| 43 | +Filesystem metadata may identify an unchanged candidate, but reusable semantic facts remain |
| 44 | +content-addressed. Ambiguous or concurrently changing observations are misses. Added, removed, |
| 45 | +renamed, and role-changing paths are first-class changes. |
| 46 | + |
| 47 | +### 2. Static discovery facts |
| 48 | + |
| 49 | +Parsing results for SQL, YAML, CSV metadata, and statically inspected Python declarations are keyed |
| 50 | +by content digest, parser version, resource role, and parse options. These records contain only |
| 51 | +validated JSON-compatible values. Runtime Python discovery still imports current project code and |
| 52 | +is never reconstructed from executable cache data. |
| 53 | + |
| 54 | +### 3. Attached resource facts |
| 55 | + |
| 56 | +Attachment resolves configuration, declarations, macros, hooks, schemas, tests, and references. |
| 57 | +Each record carries the exact identities it observed: |
| 58 | + |
| 59 | +- authored resource content; |
| 60 | +- effective target, variables, and relevant configuration; |
| 61 | +- visible enum and constant declarations; |
| 62 | +- macro implementation and dependency closure; |
| 63 | +- referenced schema or hook declarations; |
| 64 | +- adapter rendering contract. |
| 65 | + |
| 66 | +If any observed identity is absent or unstable, that resource is attached normally. A global |
| 67 | +configuration change is broad only when the affected value was observed broadly. |
| 68 | + |
| 69 | +### 4. Semantic analysis facts |
| 70 | + |
| 71 | +Model SQL analysis remains keyed by expanded SQL, analysis profile, referenced relation interfaces, |
| 72 | +binding schema, and engine versions. The persistent representation should retain compact interned |
| 73 | +facts so warm reads do not reconstruct duplicate lineage objects entry by entry. |
| 74 | + |
| 75 | +Each model publishes a deterministic interface signature containing the output columns, types, |
| 76 | +nullability, wildcard status, and any other fact consumed by downstream analysis. A changed model |
| 77 | +invalidates downstream semantic analysis only when this exported interface changes. Its own compiled |
| 78 | +artifact still changes whenever its implementation changes. |
| 79 | + |
| 80 | +### 5. Shared derived indexes |
| 81 | + |
| 82 | +Project-wide indexes are built once per compile and shared by all consumers. They include model |
| 83 | +dependencies, test model closures, declaration visibility, selector indexes, and artifact identities. |
| 84 | +No per-test, per-rule, or per-artifact path may rebuild an unchanged whole-project index. |
| 85 | + |
| 86 | +### 6. Command snapshots |
| 87 | + |
| 88 | +A command snapshot records validated, non-executable output facts after all lower layers complete. |
| 89 | +It may provide a no-op fast path only when every input observed by that command is represented in |
| 90 | +the snapshot identity. Commands involving untracked runtime values bypass this layer while still |
| 91 | +using safe lower-layer facts. |
| 92 | + |
| 93 | +## Invalidation |
| 94 | + |
| 95 | +Invalidation follows recorded dependency edges rather than directory-wide guesses: |
| 96 | + |
| 97 | +1. Compare the current inventory with the last committed inventory. |
| 98 | +2. Invalidate directly changed static facts. |
| 99 | +3. Follow declaration, macro, configuration, schema, hook, and test dependency edges. |
| 100 | +4. Reattach directly affected resources. |
| 101 | +5. Reanalyze changed models and compare exported interface signatures. |
| 102 | +6. Reanalyze downstream models only when a consumed interface changed. |
| 103 | +7. Rebuild tests, audits, rules, and artifacts only when their recorded inputs changed. |
| 104 | + |
| 105 | +Removing an edge is itself an invalidating change. Cache readers must not infer validity from the |
| 106 | +absence of a newly introduced dependency record. |
| 107 | + |
| 108 | +## Storage and publication |
| 109 | + |
| 110 | +- Records are schema-versioned and include algorithm and package fingerprints. |
| 111 | +- Payloads have explicit size and collection-count limits before semantic acceptance. |
| 112 | +- Digests cover keys and canonical serialized payloads. |
| 113 | +- Writers publish in one transaction or by atomic replacement; readers observe one committed |
| 114 | + generation. |
| 115 | +- Concurrent lock or I/O failure is a cache miss, never a compile failure. |
| 116 | +- Cache growth is bounded by generation and byte limits. Pruning removes old immutable records and |
| 117 | + does not mutate current semantic state. |
| 118 | +- `--no-cache`, target configuration, and the compile-cache environment override bypass every |
| 119 | + persistent layer and do not publish new records. |
| 120 | + |
| 121 | +## Observability and guards |
| 122 | + |
| 123 | +The target machine-readable contract reports per-layer hits, misses, invalidations, bytes read and |
| 124 | +written, and phase timings. The current vertical slice reports model-analysis batch hits, entry |
| 125 | +hits, misses, bypasses, and phase timings. Counts describe semantic records, not low-level database |
| 126 | +operations. |
| 127 | + |
| 128 | +Fresh-process performance guards use representative semantic projects and cover: |
| 129 | + |
| 130 | +- cold compile with a clean target; |
| 131 | +- unchanged warm compile; |
| 132 | +- leaf-model implementation edit; |
| 133 | +- upstream implementation edit with an unchanged interface; |
| 134 | +- interface-changing upstream edit; |
| 135 | +- test-only and macro edits; |
| 136 | +- broad project-configuration edit; |
| 137 | +- cache corruption and `--no-cache` fallback. |
| 138 | + |
| 139 | +Every path verifies exact resource counts, zero unexpected diagnostics, deterministic semantic |
| 140 | +fingerprints, bounded peak RSS, and expected hit and invalidation counts. Wall-time ceilings are |
| 141 | +calibrated only after the behavior and identity assertions are stable. |
0 commit comments