feat(dbt): dbt mart optimizer commands, skills and a review schema-lag lane

.claude-plugin/marketplace.json

@@ -0,0 +1,20 @@
+{
+  "name": "altimate-code",
+  "owner": {
+    "name": "Altimate AI",
+    "email": "support@altimate.ai"
+  },
+  "metadata": {
+    "description": "Altimate Code as a Claude Code plugin — deterministic dbt/SQL analysis driven by the altimate engine",
+    "version": "0.1.0"
+  },
+  "plugins": [
+    {
+      "name": "altimate-data",
+      "source": "./plugins/claude-code",
+      "description": "Deterministic dbt and SQL analysis: lint, grade, PII, column-level lineage, blast radius, and schema verification, powered by the altimate CLI",
+      "category": "data-engineering",
+      "keywords": ["dbt", "sql", "lineage", "trino", "data-engineering"]
+    }
+  ]
+}

.opencode/skills/dbt-factor/SKILL.md

@@ -0,0 +1,27 @@
+---
+name: dbt-factor
+description: Use when the same or near-duplicate SQL (CTEs, filters, aggregations) is repeated across several dbt models, or when asked to compute it once and reuse it, to deduplicate model logic, or to cut repeated scans. Proposes a shared intermediate model, with an equivalence proof for each consumer.
+---
+
+# Compute repeated logic once
+
+## Requirements
+**Agent:** `builder` (edits model files)
+**Tools used:** `bash` (`altimate dbt-factor`, `altimate dbt-impact`, `altimate dbt-parity`), `read`, `edit`
+
+```bash
+altimate dbt-factor --project [--min-occurrences 2] [--json]
+altimate dbt-factor --model <model>          # only proposals that touch this model
+altimate dbt-factor --apply [--proposal N]   # write the patch
+```
+
+Each proposal is a multi-file patch:
+- a new `models/intermediate/int_<name>.sql` with a yml stub;
+- an edit to each consumer so it reads the new model;
+- a proof for each consumer, plus the savings estimate and the storage trade-off (the
+  new model is materialised).
+
+`--apply` writes the files only when no consumer proof is REFUTED and every consumer's
+Jinja maps back safely. It then runs `dbt-impact` on each consumer it touched and
+prints a `dbt-parity` command for each one. Run those commands after building
+(skill `dbt-parity`). Set a `{{ config() }}` on the new model before you build it.

.opencode/skills/dbt-impact/SKILL.md

@@ -0,0 +1,60 @@
+---
+name: dbt-impact
+description: Use before changing, dropping, renaming or retyping a dbt model or column, adding a column that another job or model reads, changing its grain or materialization, or when asked "what depends on this?" or "what breaks downstream?". Lists every dbt model, loader and served table hop by hop, with a BREAKING / RISKY / SAFE verdict for each.
+---
+
+# What breaks downstream
+
+## Requirements
+**Agent:** any (read-only analysis)
+**Tools used:** `bash` (`altimate dbt-impact`), `read`
+
+```bash
+altimate dbt-impact --model <model> [--column <col>] \
+  --kind drop|rename|retype|grain|materialization|add \
+  [--new-name x | --to-type t | --to incremental [--strategy delete+insert] [--unique-key k]] \
+  [--sinks sink_map.json | --no-exposure-sinks] [--json]
+# a branch's changes: both sides compiled with dbt, compiled SQL compared
+altimate dbt-impact --base origin/main [--head HEAD] [--sinks sink_map.json] \
+  [--jobs jobs.json | --jobs-from selectors] [--live] [--profiles-dir d] [--dbt-bin path]
+# or from two already compiled manifests
+altimate dbt-impact --base-manifest base/manifest.json --head-manifest head/manifest.json [--jobs ...]
+```
+
+- Each hop is a dbt child, a dbt descendant, a loader, or a served table, with a
+  verdict and its evidence. `Overall` is the worst verdict, followed by its reasons.
+- Without `--sinks`, the sink map comes from exposures whose `meta.sink` declares
+  `{loader, target, columns, read_mode}`. That map is partial: a mart with no exposure has
+  UNKNOWN loaders, not none, unless the dbt var `sink_map_complete: true` is set. An
+  explicit `--sinks` map is complete. With no map at all the verdict is UNKNOWN
+  (`sinks_unknown`), never SAFE. A clean dbt graph alone does not show that a mart column
+  is unused. A loader that reads a column tolerantly (fills NULL) stays BREAKING.
+- `--kind materialization` defaults to the `append` strategy. Pass `--strategy` (and
+  `--unique-key` when the new config sets one); the report then carries `delete_scope`
+  and the `reader_requirements` every reader must meet.
+- Diff mode compiles base and head (`dbt compile`, offline first: no warehouse needed
+  unless a model introspects) and compares `compiled_code`, so `{% set %}`, vars and
+  macros classify. It detects added, renamed and retyped columns (`resolved_kind`).
+  `--no-compile` falls back to a naive render; models it cannot render are
+  `UNCLASSIFIED`. Run those again with `--model`.
+- **Schema lag** (`schema_lag`): CI builds everything at once, production does not.
+  For every column a change adds, renames or retypes, readers are checked for:
+  - `schema_lag:cross_job` (BREAKING): the reader's job does not build the producer.
+    Needs a job map: `--jobs jobs.json` (`{"jobs": {name: {"select", "exclude",
+    "schedule"}}}`, dbt selection syntax, resolved with `dbt ls`) or
+    `--jobs-from selectors` (one job per `selectors.yml` selector; cron from a
+    `schedules.json` next to it). Without one, cross-job is skipped and the report says so.
+  - `schema_lag:incremental_needs_full_refresh` (BREAKING): the producer is incremental
+    with `on_schema_change` ignore (the default) or fail.
+  - `schema_lag:source_not_landed`: the reader uses a source column newly declared in
+    YAML. RISKY without a live catalog; `--live` reads the deployed columns over the
+    configured connection and makes it BREAKING when the column is missing.
+  - Ephemeral producers are inlined and never lag.
+  Each finding names the consumer, column, producer, jobs (with cron and worst-case
+  window) and one fix (e.g. `+` parent selection, `on_schema_change:
+  append_new_columns`, a two-phase release).
+- Use `--fail-on breaking|risky` to gate CI.
+
+Report BREAKING hops and schema-lag findings first and name the consumers, jobs and the
+suggested fix. Do not call a change safe when the sink map was missing or when cross-job
+was skipped for lack of a job map on a project that runs several jobs.

.opencode/skills/dbt-optimize/SKILL.md

@@ -0,0 +1,57 @@
+---
+name: dbt-optimize
+description: Use when asked to optimise, speed up, or cut the cost / memory / bytes scanned of a dbt model, or to find and fix its performance or cost problem — especially when the output rows must stay exactly the same. Finds the cost problem, proposes rewrites with an equivalence proof, applies one safely, and hands off to a data parity check.
+---
+
+# Optimise a dbt model without changing its rows
+
+## Requirements
+**Agent:** `builder` (edits model files)
+**Tools used:** `bash` (`altimate dbt-perf`, `altimate dbt-optimize`, `altimate dbt-parity`), `read`, `edit`
+
+Run from the dbt project directory. All steps are deterministic engine calls.
+
+## 0. Compile first
+`altimate` reads `target/manifest.json` and needs the COMPILED SQL in it:
+`dbt compile` (not `dbt parse`). Check `had_compiled_sql: true` in step 1's summary.
+
+## 1. Find the cost problem
+```bash
+altimate dbt-perf --format json [--run-results target/run_results.json --sort cost]
+```
+Keep the findings whose `model` is the target. `--run-results` adds measured runtime
+and ranks by it. A finding's `guardrail` of kind `materialization_change` comes with
+`delete_scope` and `reader_requirements` — quote both if you act on it.
+
+## 2. Get proven rewrites
+```bash
+altimate dbt-optimize --model <model> [--json]
+```
+Each proposal has a proof:
+- **PROVEN** — the engine showed both versions return the same rows. Apply it.
+- **REFUTED** — it changes the rows; the differences are listed. Never apply it.
+- **UNDECIDABLE** — unknown; apply only if step 4 then passes.
+
+## 3. Apply
+```bash
+altimate dbt-optimize --model <model> --apply [--proposal N] [--allow-unproven]
+```
+It keeps the `{{ config() }}` block and turns relations back into `ref()`s. When it
+prints `NOT APPLIED` (Jinja such as `{% if %}`, macros or `var()`), it prints the
+compiled-SQL patch: port that change into the model by hand, keeping the Jinja.
+If no proposal fits the finding, write the fix yourself. The step 4 bar still applies.
+
+## 4. Prove the rows did not change
+Build the edited model into a separate relation (e.g. a dev target), then run the
+command step 3 printed:
+```bash
+altimate dbt-parity --model <model> --before <current relation> --after <new relation> [--columns a,b,c]
+```
+`PARITY: MATCH` (exit 0) is the bar. Details are in the `dbt-parity` skill.
+Without a warehouse connection, say so and report the proof from step 2 as the only
+evidence.
+
+## Report
+State the finding (with cost, if measured), the change, its proof status, and the
+parity result. Before you change a column or the materialization, run
+`altimate dbt-impact` (skill `dbt-impact`).

.opencode/skills/dbt-parity/SKILL.md

@@ -0,0 +1,37 @@
+---
+name: dbt-parity
+description: Use to verify that a changed or rewritten dbt model returns exactly the same rows as before. It proves the data is unchanged: row counts, duplicate grain keys, EXCEPT both ways, and per-column match rates, run over a warehouse connection.
+---
+
+# Prove a dbt model change kept the same data
+
+## Requirements
+**Agent:** any (read-only analysis)
+**Tools used:** `bash` (`altimate dbt-parity`), `read`
+
+```bash
+altimate dbt-parity --model <model> --before <relation before> --after <relation after> \
+  [--connection <name>] [--columns a,b,c] [--sample-rows N] [--json]
+```
+
+- `--before`: the relation built from the old SQL (usually prod). `--after`: the same
+  model built from the new SQL into another schema or target.
+- The grain comes from the model's uniqueness test, its contract primary key, or its
+  `unique_key`. The output says which one it used.
+- `--columns` turns on per-column match rates, which name the column that differs.
+- Exit 0 and `PARITY: MATCH` mean the rows are the same. `MISMATCH` and `INCONCLUSIVE`
+  exit 1. `INCONCLUSIVE` means a check errored. It is never a pass.
+- `--connection`: a configured warehouse connection. Without one, the first configured
+  connection is used. A connection can also come from an environment variable:
+  `ALTIMATE_CODE_CONN_<NAME>='{"type":"trino","host":…,"port":…,"user":…,"catalog":…}'`.
+
+## PII: samples are opt-in
+Without `--sample-rows`, only counts leave the warehouse. With `--sample-rows N`,
+up to N differing rows are printed each way, and they end up in this conversation.
+Ask the user before you use it on tables that may hold personal or regulated data.
+
+## Reading a mismatch
+- Row delta with EXCEPT = 0: rows were duplicated or dropped. Check
+  `Duplicated grain keys`.
+- EXCEPT > 0 both ways with equal counts: values changed. The column match rates name
+  the columns.

.opencode/skills/dbt-pr-review/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: dbt-pr-review
-description: Cloudflare-style AI code review for dbt/SQL pull requests. Produces a signed APPROVE/COMMENT/REQUEST_CHANGES verdict where every blocking finding is backed by a deterministic engine call — column-lineage blast radius, query equivalence, PII classification, and A–F grade. Use to review a dbt PR or the working-tree changes before merge.
+description: Cloudflare-style AI code review for dbt/SQL pull requests. Produces a signed APPROVE/COMMENT/REQUEST_CHANGES verdict where every blocking finding is backed by a deterministic engine call — column-lineage blast radius, query equivalence, PII classification, and A–F grade. Use to review a dbt PR or the working-tree changes before merge. Reviews a DIFF; for the whole project use `dbt-project-review`.
 ---
 
 # dbt PR Review
@@ -9,6 +9,14 @@ description: Cloudflare-style AI code review for dbt/SQL pull requests. Produces
 **Agent:** `reviewer` (read-only) — also works from `analyst`/`builder`.
 **Tools used:** `dbt_pr_review` (primary), `impact_analysis`, `altimate_core_equivalence`, `altimate_core_check`, `lineage_check`, read-only `git`.
 
+## Scope — diff, not project
+
+This reviews **a diff**: base..head, defaulting to the merge-base with `origin/main`.
+A model that has been wrong since the day it was written will not appear unless it is in
+the diff. If the question is "is this project healthy?" rather than "is this change
+safe?", use `dbt-project-review` (`altimate review --all`), which reviews every model in
+the manifest and reports the base-vs-head lanes as skipped.
+
 ## When to Use This Skill
 
 Use when the user wants to:

.opencode/skills/dbt-project-review/SKILL.md

@@ -0,0 +1,144 @@
+---
+name: dbt-project-review
+applyPaths:
+  - "dbt_project.yml"
+  - "**/dbt_project.yml"
+description: Review an ENTIRE dbt project rather than a diff — every model in the manifest, plus a deterministic performance pass. Use when the question is "is this project healthy?", "what technical debt is in here?", "audit the whole warehouse", or when onboarding to an unfamiliar dbt repo. For "is this change safe?" use `dbt-pr-review` instead, which reviews a diff.
+---
+
+# dbt project review
+
+## Requirements
+**Agent:** `reviewer` (read-only) — also works from `analyst`.
+**Tools used:** `bash` (`altimate review --all`, `altimate dbt-perf`), read-only `git`, `read`.
+
+## Diff vs project — pick the right one
+
+| Question | Command | Skill |
+| --- | --- | --- |
+| Is **this change** safe to merge? | `altimate review` | `dbt-pr-review` |
+| Is **this project** healthy? | `altimate review --all` | this one |
+
+They are not interchangeable. `altimate review` defaults to the merge-base with
+`origin/main`; a model that has been wrong since the day it was written never appears in
+its output. In a mature repo that is most of the findings — a project running
+`altimate review` in CI for months can still be sitting on dozens of untouched
+project-level issues, because none of them was ever in a recent diff.
+
+## Preflight — compile, do not just parse
+
+The engine analyses SQL, not Jinja. **`dbt parse` is not enough for a project review.**
+It writes a manifest (so the DAG, contracts and tests resolve) but no compiled SQL, and
+every model that falls back to raw Jinja fails to parse at `{{ config(...) }}` on line 1
+— the AST lanes then return **zero findings for that model**, which is indistinguishable
+from a clean one.
+
+```bash
+cd <dbt project dir>
+dbt deps
+dbt compile --profiles-dir .     # writes target/manifest.json AND target/compiled/**
+```
+
+`dbt compile` needs credentials to resolve `source()` metadata but does not run queries.
+If compile is impossible, run anyway — but read the compiled-coverage line in the report
+(below) and say plainly how much of the project was actually analysed.
+
+## 1. Project review
+
+```bash
+altimate review --all \
+  --cwd <dbt project dir> \
+  --manifest target/manifest.json \
+  --severity warning \
+  --no-ai \
+  --json
+```
+
+- `--all` is mutually exclusive with `--base`/`--head` — it has no base ref.
+- `--severity suggestion` for everything; `warning` is the useful default on a first pass.
+- `--max-models N` caps the run (`0` lifts the cap; the default is 500). Any truncation
+  is announced on stderr — never assume it didn't happen.
+- `--mode gate` exits non-zero on `REQUEST_CHANGES`, for a scheduled CI health check.
+- `--no-ai` keeps it deterministic and free.
+
+### Read `lanes` before you read `findings`
+
+The envelope carries a `lanes` object that the diff mode does not have:
+
+```jsonc
+"lanes": {
+  "ran": ["core_lint", "dbt_patterns", "lexical_scan", "dbt_config_lint",
+          "grain_mismatch", "missing_grain_test", "sibling_filter_consistency",
+          "pii_classify", "pii_detect"],
+  "skipped": [{ "lane": "semantic_change", "reason": "proves base-vs-head row equivalence; …" }],
+  "compiledModels": 7,
+  "totalModels": 214
+}
+```
+
+Two rules:
+
+- **Every base-vs-head lane is skipped and says so.** Equivalence, structural diff,
+  column breakage, data-diff, config diff, lineage breakage, topology equivalence and
+  the schema.yml test-removal detector all compare two versions of a model; project
+  scope has one. Do not report their silence as "no issues found there" — report them
+  as not run, and point the user at `dbt-pr-review` for those.
+- **`compiledModels` < `totalModels` means the report is partial.** Say the number. A
+  review where 207 of 214 models had no compiled SQL analysed 3% of the project with
+  the AST lanes, and the honest headline is that, not the finding count.
+
+## 2. Performance pass
+
+`altimate review --all` covers correctness, PII, contracts and grain. It does not do
+the cost/performance rule set — that is a separate deterministic command:
+
+```bash
+altimate dbt-perf --cwd <dbt project dir> --manifest target/manifest.json \
+  --format json --severity warning
+```
+
+- It prints `skipped_rules` for every rule that could not run, and a summary line
+  carrying `had_compiled_sql` / `had_sink_map` / `had_run_results` / `parse_failures`. Surface both: a rule that could not run
+  is not a clean result.
+- Where a finding carries a `guardrail` of kind `materialization_change`, report its
+  `delete_scope` and `reader_requirements` verbatim. Those fields exist because a past
+  performance fix switched a materialization without dropping the old relation and
+  caused a data incident. A recommendation to change a materialization that omits them
+  is not a complete recommendation.
+- `--fail-on warning|error` for CI.
+- `--run-results target/run_results.json --sort cost` ranks findings by measured runtime;
+  `--sinks <map>` lets the column rules see loaders outside dbt.
+- If it reports that it needs a newer `@altimateai/altimate-core`, say so and stop —
+  do not substitute a hand-rolled performance opinion for the engine's.
+
+## Acting on the findings
+
+This skill reports. To fix what it finds, use these skills, which work on one model
+or one change at a time:
+
+- Performance or cost findings → `dbt-optimize` (`altimate dbt-optimize --model X`):
+  rewrites with an equivalence proof, applied safely.
+- Logic repeated across models (P104 and similar) → `dbt-factor`
+  (`altimate dbt-factor --project`).
+- Before you drop, rename or retype a column, or change a materialization →
+  `dbt-impact` (`altimate dbt-impact --model X --column c`). It includes loaders and
+  served tables when you pass `--sinks`.
+- After any rewrite → `dbt-parity` to prove the rows are unchanged.
+
+## 3. Reporting
+
+Lead with **scope and coverage**, then findings:
+
+1. How many models were analysed, how many had compiled SQL, which lanes ran and which
+   did not.
+2. Findings grouped by category (not by file) and ordered by severity — in a project
+   review the pattern matters more than the instance. "31 models cross join without an
+   `ON` clause" is the finding; the 31 paths are the evidence.
+3. The three or four themes worth fixing first, with the concrete first step.
+
+Two rules carried over from `dbt-pr-review`:
+
+- **Do not restate a finding you cannot see in the source.** Open the file and confirm
+  the line before repeating what the engine said.
+- **Never present a partial run as a complete one.** If compile did not happen, or the
+  model cap bit, or a lane was skipped, that belongs in the first paragraph.

.opencode/skills/query-optimize/SKILL.md

@@ -11,6 +11,14 @@ description: Analyze and optimize SQL queries for better performance
 
 Analyze SQL queries for performance issues and suggest concrete optimizations including rewritten SQL.
 
+## dbt models: use `dbt-optimize` instead
+
+If the SQL is a **dbt model** (a file under `models/` or a model name), use the
+`dbt-optimize` skill. It reads the compiled project, ties rewrites to the project's
+perf findings, writes the proven rewrite back into the model while keeping its Jinja,
+and hands off to `dbt-parity` to prove the rows did not change. Use `dbt-factor` when
+the cost is logic repeated across models. This skill is for standalone SQL.
+
 ## Workflow
 
 1. **Get the SQL query** -- Either: