Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,14 @@ The playground runs on local DuckDB, with no warehouse credentials.
- **Tidy up safely.** The [janitor](https://sqlbuild.com/docs/cli/janitor/) archives stale tables
before anything is deleted.

### Find out what happened

- **Every run is recorded.** [Recall](https://sqlbuild.com/docs/concepts/observability/recall/)
keeps each command's statements, SQL, audit results and exact output. Agents and people query it
with SQL through [`sqb recall`](https://sqlbuild.com/docs/cli/recall/), then check live state in
the warehouse with the same command. History stays in a SQLite file in the project, or in
ClickHouse for a team.

Ingestion with Python loaders, and Python tasks, assets and checks, run in the same graph as your SQL
models. See the [docs](https://sqlbuild.com/docs/) for everything else.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Online: https://sqlbuild.com/docs/concepts/adapters/clickhouse/
- Settings SQLBuild pins
- SQL differences to watch
- Limits
- Recall store

The ClickHouse adapter is in beta and supports a single server. Clusters (`ON CLUSTER`, replicated and distributed tables) are not supported yet. Please [report issues](https://github.com/chio-labs/sqlbuild/issues).

Expand Down Expand Up @@ -187,3 +188,11 @@ ClickHouse returns a type's default value (`0`, the epoch, an empty string) from
- Snapshots always use an unsorted `MergeTree` table.
- Seeds load through the HTTP insert interface as CSV, so large seeds never hit the `max_query_size` limit.
- Retention and time travel are not available.

## Recall store

A ClickHouse connection can also hold [Recall](../observability/recall.md)'s command
history for a whole team, whichever adapter the project builds with. Set
`[recall] connection` to it; Recall creates its tables on first use, keeps every output byte by
default, and needs only `SELECT` and `INSERT` once the tables exist. See
[Sharing history in ClickHouse](../observability/recall.md#sharing-history-in-clickhouse).
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Online: https://sqlbuild.com/docs/concepts/adapters/sqlite/
- Capabilities
- Scenarios
- Limits
- Recall store

SQLite uses the `sqlite3` module that ships with Python. No extra installation is needed.
SQLBuild requires SQLite 3.44 or newer; Python builds installed by `uv` bundle a recent version.
Expand Down Expand Up @@ -130,3 +131,9 @@ as `datetime(ordered_at, 'start of day')`, have no DuckDB translation and fail l
| SQLite older than 3.44 | `A701` |
| Another process holds the write lock beyond `busy_timeout` | `A702` |
| Concurrent microbatch writes: one file has one writer | refused at plan time |

## Recall store

[Recall](../observability/recall.md) keeps its local history in a SQLite file,
`logs/recall.sqlite3`, for every project, whichever adapter it builds with. Point
`[recall] connection` at a `sqlite` connection to put the file somewhere else.
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,28 @@ whether work started, completed, failed, skipped, or retried.
|--------|---------|-----------|
| `LifecycleEvent` | Immutable invocation, run, resource-attempt, operation, statement, retry, and audit facts | Canonical lifecycle evidence once durably stored |
| `DiagnosticLog` | Structured framework diagnostics | Explanatory only |
| `stdout.log` / `stderr.log` | Exact host-local process output | Troubleshooting transcript |
| [Recall](observability/recall.md) | Local queryable history of invocations, statements, SQL, and output | Troubleshooting history; best effort |
| [Recall](observability/recall.md) | Queryable history of invocations, statements, SQL, audit results, and output, locally or in ClickHouse | Troubleshooting history; best effort |
| `stdout.log` / `stderr.log` | Exact host-local process output (compute log files) | Troubleshooting transcript; also stored by Recall |
| `CommandOutputRecord` | Bounded, ANSI-free remote stdout/stderr chunks | Potentially sensitive and lossy troubleshooting data |
| Final JSON output | End-of-command aggregate | Result projection, not event history |

Human-readable CLI output remains the normal operator interface. Structured records are a separate
contract for integrations and durable consumers.

## Which to use

| You want to | Use |
|-------------|-----|
| Find out what a past run did: what ran, its SQL, audit results, errors and output | [Recall](observability/recall.md) and [`sqb recall`](../cli/recall.md) |
| Send events or output to your own systems, such as a message queue or a monitoring tool | [Typed sinks](observability/sinks.md) |
| Store lifecycle events and query run state from your own Python application | [Execution history](observability/execution-history.md) |
| Read the result of the command you just ran | Its final `--json` output |

Recall is on by default and covers investigation for people and agents. Sinks and execution history
are for building on SQLBuild's events elsewhere. The `stdout.log`, `stderr.log` and
`diagnostics.jsonl` compute log files under `logs/<date>/` are still written alongside Recall and
hold the same bytes; prefer `sqb recall output`, which works the same with a shared store.

## Lifecycle envelope

New events use lifecycle schema version 2:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ published, rebuildable run projection. Diagnostic and compute logs are never acc
contract. The public types, filters, pages, records, errors, and helper functions are exported from
`sqlbuild.execution_history`.

This is a Python API for applications that store and serve SQLBuild events themselves. To look into
what past runs did, use [Recall](recall.md), which records every command
automatically and is queried with [`sqb recall`](../../cli/recall.md).

## Public contracts

`LifecycleEventLogStorage` provides `append_event`, `append_events`, `get_events`, `get_schema_version`,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Online: https://sqlbuild.com/docs/concepts/observability/recall/

## Contents

- An investigation, start to finish
- What is recorded
- Settings
- Output limit
Expand All @@ -22,6 +23,64 @@ Recall records each `sqb` invocation in a local SQLite store, or in a shared
[ClickHouse store](#sharing-history-in-clickhouse), so you can ask what happened after the fact: which command ran, how it ended, which statements it executed and how long they took, and
exactly what it printed. Recall is on by default and needs no setup.

## An investigation, start to finish

The waffle shop example's `daily_revenue` model has an audit, `revenue_is_non_negative`, at warn
severity. One night the payments feed starts recording a refund as a negative payment. The unit tests
still pass, because they run on mocked data. The build finishes with a warning that nobody reads.

The next morning, ask when the audit started failing:

```bash
sqb recall query --format table \
"SELECT invocation_started_at AS run, outcome, violation_count FROM audit_results
WHERE audit_name = 'revenue_is_non_negative' ORDER BY run DESC LIMIT 3"
```

```
run | outcome | violation_count
--------------------------- | ------- | ---------------
2026-10-20T00:01:44.784683Z | warn | 1
2026-10-20T00:00:34.652468Z | pass | 0
2026-10-19T23:59:23.827831Z | pass | 0

3 rows
```

See what that run printed, exactly as the scheduler saw it:

```bash
sqb recall output last --tail 4
```

```
daily_revenue
audit revenue_is_non_negative returned 1 row

✓ Completed with warnings PASS=89 WARN=1 FAIL=0 INSUFFICIENT=0 SKIP=0 TOTAL=90 (3.08s)
```

Then check what is true in the warehouse now, with the same command:

```bash
sqb recall query --backend warehouse --format table \
"SELECT r.revenue_date, r.total_revenue_cents, p.payment_id, p.amount_cents
FROM prod.daily_revenue r JOIN prod.stg_payments p
ON CAST(p.paid_at AS DATE) = r.revenue_date
WHERE p.amount_cents < 0"
```

```
revenue_date | total_revenue_cents | payment_id | amount_cents
------------ | ------------------- | ---------- | ------------
2026-04-03 | -1250 | 9 | -2200

1 row
```

The bad row is still in prod, and payment 9 explains it. A coding agent can follow the same steps:
every answer is a SQL query over the [views](#querying-history), with the same output in `--json`.

## What is recorded

For every recorded invocation, Recall keeps:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -118,11 +118,11 @@ SQLBuild, dbt, and SQLMesh are all SQL pipeline frameworks. They share common gr
| Feature | SQLBuild | dbt | SQLMesh |
|---------|----------|-----|---------|
| Reference syntax | `__ref()` - parses as valid SQL | `{{ ref() }}` - Jinja template | `model_name` with dependency tracking |
| Adapters | DuckDB, MotherDuck, Snowflake, BigQuery, Databricks, PostgreSQL, SQL Server | 30+ (community adapters) | DuckDB, Snowflake, BigQuery, Databricks, Spark, Redshift, Postgres, Trino, MySQL |
| Adapters | DuckDB, MotherDuck, Snowflake, BigQuery, Databricks, PostgreSQL, SQL Server, ClickHouse, SQLite | 30+ (community adapters) | DuckDB, Snowflake, BigQuery, Databricks, Spark, Redshift, Postgres, Trino, MySQL |
| State requirements | Stateless by default | manifest.json + target/ | Requires state store (local database or PostgreSQL for production) |
| Playground | `sqb playground` | Clone example repo | Example project |
| AI agent skills | General guidance with `sqb skills`; Rules guidance with `sqb rules skills` | No | No |
| Execution observability | Immutable schema-versioned lifecycle facts, invocation-local ordering, orchestration context, and typed project sinks | Events/artifacts plus external observability | Plans, state, and external observability |
| Execution observability | Recall: every command's statements, SQL, audit results and output, queryable with SQL locally or in ClickHouse; immutable schema-versioned lifecycle facts and typed project sinks | Events/artifacts plus external observability | Plans, state, and external observability |

## Where each tool fits

Expand All @@ -134,4 +134,4 @@ SQLBuild, dbt, and SQLMesh are all SQL pipeline frameworks. They share common gr

## Not yet in SQLBuild

- **Broader adapter support** - ClickHouse, Redshift, Trino, Spark, Athena
- **Broader adapter support** - Redshift, Trino, Spark, Athena
10 changes: 9 additions & 1 deletion src/sqlbuild/.agents/skills/sqlbuild/references/docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,12 @@ updated, a [migration](concepts/models/migrations.md) moves its table instead of
history, and the old name keeps working as a view. Columns can be renamed the same way. The
[janitor](cli/janitor.md) archives tables the project no longer builds before it deletes them.

## Investigate

[Recall](concepts/observability/recall.md) records every command: what ran, the SQL it executed,
audit results and everything it printed. Agents and people query that history with SQL through
[`sqb recall`](cli/recall.md), then check live state in the warehouse with the same command.

## How it fits together

| Step | Command | What it does |
Expand All @@ -51,8 +57,10 @@ history, and the old name keeps working as a view. Columns can be renamed the sa
| Plan | [`sqb plan`](cli/plan.md) | Shows what will run, why, and how much it will rebuild |
| Build | [`sqb build`](cli/build.md) | Runs tests, builds models in dependency order, and blocks bad data with audits |
| Compare | [`sqb diff`](cli/diff.md) | Compares schemas and rows between two targets |
| Investigate | [`sqb recall`](cli/recall.md) | Queries the history of past runs, then live warehouse state |

SQLBuild works with DuckDB, MotherDuck, Snowflake, BigQuery, Databricks, PostgreSQL and SQL Server.
SQLBuild works with DuckDB, MotherDuck, Snowflake, BigQuery, Databricks, PostgreSQL, SQL Server,
ClickHouse and SQLite.
DuckDB runs locally, so you can try everything without warehouse credentials. See
[Adapters](concepts/adapters.md) for connection setup.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ A self-hosted web app for your SQLBuild project:

## Later

- Moving BigQuery, Databricks and SQL Server from beta to supported.
- More warehouses, such as ClickHouse, Redshift and Trino.
- Moving BigQuery, ClickHouse, Databricks, SQLite and SQL Server from beta to supported.
- More warehouses, such as Redshift and Trino.

## Not planned

Expand Down
35 changes: 34 additions & 1 deletion website/public/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -362,12 +362,45 @@ <h3>Archive old tables before deleting them</h3>
<span class="ok">Relations to archive</span>
prod.weekly_revenue -&gt; prod._sqb_archive__20260925t155417z__weekly_revenue <span class="dim">…</span></pre>
</div>
<p class="step-group">After production changes</p>
<div class="step">
<div class="step-text">
<span class="num">09 · Investigate</span>
<h3>Find out what happened, then check what's true now</h3>
<p>
Every run is recorded: what ran, the SQL it executed, audit results and everything it printed. Agents and people ask in SQL, then check live state with the same command.
</p>
</div>
<pre class="terminal"><span class="cmd">sqb recall query &quot;SELECT substr(invocation_started_at, 1, 16) AS run, outcome,
violation_count FROM audit_results WHERE audit_name = 'revenue_is_non_negative'
ORDER BY run DESC LIMIT 3&quot;</span>
run <span class="dim">|</span> outcome <span class="dim">|</span> violation_count
2026-10-20T00:01 <span class="dim">|</span> <span class="warn">warn</span> <span class="dim">|</span> <span class="warn">1</span>
2026-10-20T00:00 <span class="dim">|</span> <span class="ok">pass</span> <span class="dim">|</span> 0
2026-10-19T23:59 <span class="dim">|</span> <span class="ok">pass</span> <span class="dim">|</span> 0

<span class="cmd">sqb recall output last --tail 4</span>
daily_revenue
audit revenue_is_non_negative returned 1 row
<span class="warn strong">✓ Completed with warnings</span> PASS=89 WARN=1 FAIL=0 <span class="dim">…</span>

<span class="cmd">sqb recall query --backend warehouse &quot;SELECT r.revenue_date, r.total_revenue_cents,
p.payment_id, p.amount_cents FROM prod.daily_revenue r JOIN prod.stg_payments p
ON CAST(p.paid_at AS DATE) = r.revenue_date WHERE p.amount_cents &lt; 0&quot;</span>
revenue_date <span class="dim">|</span> total_revenue_cents <span class="dim">|</span> payment_id <span class="dim">|</span> amount_cents
2026-04-03 <span class="dim">|</span> <span class="warn">-1250</span> <span class="dim">|</span> 9 <span class="dim">|</span> <span class="warn">-2200</span></pre>
</div>
</div>
<p class="caption">
Real output from SQLBuild's example project, trimmed to the relevant lines. During a migration the old
table is never modified, archives can be renamed back until they expire, and every move is recorded.
In step 09 the audit started warning on the latest run, the build still finished, and the bad
row is still in prod: a refund recorded as a negative payment. Three queries, no log digging.
History stays in a SQLite file in your project, or in ClickHouse for a team. No service, no
account, no paid tier.
<a href="/docs/concepts/models/migrations/">How migrations work</a> ·
<a href="/docs/cli/janitor/">How the janitor works</a>
<a href="/docs/cli/janitor/">How the janitor works</a> ·
<a href="/docs/concepts/observability/recall/">How Recall works</a>
</p>
</div>
</section>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -174,3 +174,11 @@ ClickHouse returns a type's default value (`0`, the epoch, an empty string) from
- Snapshots always use an unsorted `MergeTree` table.
- Seeds load through the HTTP insert interface as CSV, so large seeds never hit the `max_query_size` limit.
- Retention and time travel are not available.

## Recall store

A ClickHouse connection can also hold [Recall](/docs/concepts/observability/recall)'s command
history for a whole team, whichever adapter the project builds with. Set
`[recall] connection` to it; Recall creates its tables on first use, keeps every output byte by
default, and needs only `SELECT` and `INSERT` once the tables exist. See
[Sharing history in ClickHouse](/docs/concepts/observability/recall#sharing-history-in-clickhouse).
6 changes: 6 additions & 0 deletions website/src/content/docs/docs/concepts/adapters/sqlite.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -117,3 +117,9 @@ as `datetime(ordered_at, 'start of day')`, have no DuckDB translation and fail l
| SQLite older than 3.44 | `A701` |
| Another process holds the write lock beyond `busy_timeout` | `A702` |
| Concurrent microbatch writes: one file has one writer | refused at plan time |

## Recall store

[Recall](/docs/concepts/observability/recall) keeps its local history in a SQLite file,
`logs/recall.sqlite3`, for every project, whichever adapter it builds with. Point
`[recall] connection` at a `sqlite` connection to put the file somewhere else.
18 changes: 16 additions & 2 deletions website/src/content/docs/docs/concepts/observability.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,28 @@ whether work started, completed, failed, skipped, or retried.
|--------|---------|-----------|
| `LifecycleEvent` | Immutable invocation, run, resource-attempt, operation, statement, retry, and audit facts | Canonical lifecycle evidence once durably stored |
| `DiagnosticLog` | Structured framework diagnostics | Explanatory only |
| `stdout.log` / `stderr.log` | Exact host-local process output | Troubleshooting transcript |
| [Recall](/docs/concepts/observability/recall) | Local queryable history of invocations, statements, SQL, and output | Troubleshooting history; best effort |
| [Recall](/docs/concepts/observability/recall) | Queryable history of invocations, statements, SQL, audit results, and output, locally or in ClickHouse | Troubleshooting history; best effort |
| `stdout.log` / `stderr.log` | Exact host-local process output (compute log files) | Troubleshooting transcript; also stored by Recall |
| `CommandOutputRecord` | Bounded, ANSI-free remote stdout/stderr chunks | Potentially sensitive and lossy troubleshooting data |
| Final JSON output | End-of-command aggregate | Result projection, not event history |

Human-readable CLI output remains the normal operator interface. Structured records are a separate
contract for integrations and durable consumers.

## Which to use

| You want to | Use |
|-------------|-----|
| Find out what a past run did: what ran, its SQL, audit results, errors and output | [Recall](/docs/concepts/observability/recall) and [`sqb recall`](/docs/cli/recall) |
| Send events or output to your own systems, such as a message queue or a monitoring tool | [Typed sinks](/docs/concepts/observability/sinks) |
| Store lifecycle events and query run state from your own Python application | [Execution history](/docs/concepts/observability/execution-history) |
| Read the result of the command you just ran | Its final `--json` output |

Recall is on by default and covers investigation for people and agents. Sinks and execution history
are for building on SQLBuild's events elsewhere. The `stdout.log`, `stderr.log` and
`diagnostics.jsonl` compute log files under `logs/<date>/` are still written alongside Recall and
hold the same bytes; prefer `sqb recall output`, which works the same with a shared store.

## Lifecycle envelope

New events use lifecycle schema version 2:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ published, rebuildable run projection. Diagnostic and compute logs are never acc
contract. The public types, filters, pages, records, errors, and helper functions are exported from
`sqlbuild.execution_history`.

This is a Python API for applications that store and serve SQLBuild events themselves. To look into
what past runs did, use [Recall](/docs/concepts/observability/recall), which records every command
automatically and is queried with [`sqb recall`](/docs/cli/recall).

## Public contracts

`LifecycleEventLogStorage` provides `append_event`, `append_events`, `get_events`, `get_schema_version`,
Expand Down
Loading
Loading