diff --git a/README.md b/README.md index 17c0e9dd83..a6cb9ebdab 100644 --- a/README.md +++ b/README.md @@ -3,40 +3,25 @@

- Verify early. Test properly. Deploy reversibly. SQL pipelines with the rigor of real software. + The refactorable warehouse. Verify early, test properly, and refactor safely.

-**Valid isn't the same as correct.** Your SQL compiles, runs, and returns rows; none of that means the number is right, and a silently-wrong number a stakeholder already trusted is the bug that actually hurts. - -SQLBuild brings software-engineering rigor to SQL pipelines: catch errors before the warehouse runs them, test your logic locally, and opt into change-aware execution when you need it. It is a standalone, open-source framework for building SQL and Python data pipelines. - -All state is persisted as append-only tables in the warehouse alongside your data: no external state database, no manifest files, no paid add-on. Start with straightforward SQL models, then add ingestion, Python nodes, and opt-in virtual environments as your project grows. - -## Key features - -- **Test your logic, not just your columns.** Multi-model SQL tests resolve every intermediate model from its real SQL, plus end-to-end scenarios with local DuckDB replay for fast CI with no warehouse. Catch wrong logic before it ships, not just nulls. -- **Verify early.** Define models as SQL files with `MODEL()` headers. SQLBuild resolves references, validates SQL, infers columns, checks contracts, and computes column lineage before anything runs, all offline. It fails at compile, not halfway through a warehouse run. -- **Fast and open static analysis.** SQL parsing, validation, column inference, lineage, and transpilation run on [Polyglot](https://github.com/tobilg/polyglot), a Rust SQL engine (MIT, 32+ dialects), so compile stays fast on large projects. The analysis is part of the Apache-2.0 core: no proprietary engine, no login, no paid tier. -- **Audits that block bad data.** Audits run before data reaches the target table. Full table builds materialize into a staging table and only promote if audits pass; incremental models validate each batch before DML. -- **Deploy reversibly (opt-in).** Virtual environments add instant low-copy branching, partial promotion, rollback, checkpoints, and reconciliation. Opt-in, not a tax you pay upfront. -- **Opt-in change-aware execution.** Models, seeds, UDFs, and Python nodes are fingerprinted, and source freshness is tracked. In virtual environments, pass `--changes-only` or set `changes_only = true` to skip work that is already current; commands otherwise run the full selected scope. -- **Warehouse-native state.** All change-tracking state lives in append-only tables (`_sqlbuild_fingerprints`, `_sqlbuild_source_freshness`, `_sqlbuild_node_results`) in your warehouse schemas. No external state machine, no corruption risk. -- **Cursor-based incremental processing.** Automatic gap detection and resume, with microbatch mode for large ranges. No external checkpoint to maintain. -- **Ingestion and Python nodes.** Load external data with Python `@loader` functions, and run `@task`, `@asset`, and `@check` nodes as first-class members of the same DAG as your SQL models. +

+ Website · + Docs · + Quickstart · + Roadmap +

-See the [documentation](https://docs.sqlbuild.com) for the full feature set, including providers, lifecycle hooks, Python macros, UDFs, custom materializations, data diffs, zero-copy cloning, and virtual environments. To coordinate dbt and SQLBuild projects, see the [dbt compatibility guide](https://docs.sqlbuild.com/concepts/dbt-compatibility/overview). +Change your warehouse as often as your code. SQLBuild brings compile-time checks, tests and diffs to +your SQL, so change is safe. It is a free, open-source framework for SQL and Python data pipelines, +and it keeps its state in append-only tables in your own warehouse: no external state database, no +manifest files and no paid tier. ## Quick start ```bash pip install sqlbuild -# or -uv pip install sqlbuild -``` - -Create and run the included playground project: - -```bash sqb playground waffle-shop cd waffle-shop sqb plan @@ -44,9 +29,50 @@ sqb build sqb test ``` +The playground runs on local DuckDB, with no warehouse credentials. + +## What it does + +### Catch mistakes before anything runs + +- **Compile-time checks.** SQLBuild resolves references, validates SQL, infers column types, checks + [contracts](https://sqlbuild.com/docs/concepts/models/contracts/) and computes column lineage, + all offline. A typo'd column fails in seconds, not halfway through a warehouse run. +- **Your conventions as rules.** Built-in and custom Python + [rules](https://sqlbuild.com/docs/concepts/rules/) turn review comments into compile errors: for + example, marts can't read sources directly, or every final model declares its key. + +### Prove it works + +- **Tests across models.** SQL [tests](https://sqlbuild.com/docs/concepts/testing/) mock the + sources and check the result through every model in between, with macros as test helpers. Macro, + UDF and table-function tests are built in. +- **End-to-end scenarios.** Build the real graph against fixture data, capture fixtures from the + warehouse, and replay them locally on DuckDB in CI. + See [scenarios](https://sqlbuild.com/docs/concepts/scenarios/). +- **Audits and diffs.** Audits run before data reaches the target table, and + [data diffs](https://sqlbuild.com/docs/concepts/diff/) compare dev against prod or any query. + +### Change it without rebuilding everything + +- **Renames keep their history.** Rename or move an incremental or snapshot model and SQLBuild + [migrates](https://sqlbuild.com/docs/concepts/models/migrations/) the existing table instead of + rebuilding it. +- **Replay on change.** When a model's SQL changes, choose how far back to reprocess, from only the + new data to the last 14 days to a full rebuild, with + [`replay_on_change`](https://sqlbuild.com/docs/concepts/incremental/#replay-on-change). +- **Macros don't have to be global.** Keep macros, enums and constants next to the models that use + them, and preview what a move would break with `sqb scope`. See + [declaration scopes](https://sqlbuild.com/docs/concepts/declaration-scopes/). +- **Tidy up safely.** The [janitor](https://sqlbuild.com/docs/cli/janitor/) archives stale tables + before anything is deleted. + +Ingestion with Python loaders, and Python tasks, assets and checks, run in the same graph as your SQL +models. See the [docs](https://sqlbuild.com/docs/) for everything else. + ## Example -A model is a SQL file with a `MODEL()` header and a `SELECT`. References use `__ref()` and `__source()`, and configuration, schema, and audits are declared inline: +A model is a SQL file with a `MODEL()` header and a `SELECT`: ```sql MODEL ( @@ -65,7 +91,8 @@ FROM __ref("stg_orders") o JOIN __ref("stg_payments") p USING (order_id) ``` -A unit test mocks sources and asserts on the model, resolving every intermediate model automatically: +A test mocks the sources and asserts on the model, resolving every model in between from its real +SQL: ```sql TEST(); @@ -87,230 +114,30 @@ __expected__fact_orders AS ( SELECT 1 ``` -Relation fixtures can omit a column when the compiled test or scenario closure requires it and -SQLBuild knows its adapter type, unless the column is explicitly non-nullable. SQLBuild completes -that test-only fixture column with a typed null such as `CAST(NULL AS VARCHAR)`; it never changes -model SQL or warehouse defaults. Required columns declared with `nullable false` and columns with -unknown types must be supplied explicitly. When the relation's complete column set is authoritative, -misspelled or unknown supplied fixture columns are rejected. - -A direct `NULL AS column_name` projection receives the authoritative relation type in compiled test -SQL, including expected-output CTEs for contracted models. To represent a contracted upstream with -no rows, use `SELECT * FROM __empty_fixture()` as the complete body of a `__ref__`, `__source__`, or -`__seed__` fixture CTE, or a contracted `__expected__` CTE. SQLBuild expands it to the relation's -full typed schema with a false filter. - -See the [documentation](https://docs.sqlbuild.com) for incremental models, scenarios, loaders, and more. - -### Python project layout - -Project-owned Python must live in a supported extension location such as `factories/`, `libs/`, -`macros/`, `providers/`, or another documented Python resource root. Factory locations contain -normal Python: constants, classes, undecorated helper functions, and modules such as `_helpers.py` -are allowed, while decorators determine which functions become SQLBuild resources. Compilation -rejects Python under invented project roots so indirectly importable modules cannot create an -unofficial project structure. Keep repository pytest tests outside the SQLBuild project's `tests/` -directory, which is reserved for SQLBuild SQL tests and scenarios. Documented integration paths -such as `dagster/`, `rivers_pipeline/`, and their `definitions.py` modules are also supported. - -### Python macro declaration context - -Python SQL macros receive the constants and enums visible to the SQL resource that calls them. -Use the typed mappings for Python control flow, and use the rendering methods when inserting a -declaration into generated SQL so quoting and collection syntax follow the active adapter: - -```python -def minimum_order_filter(ctx) -> str: - minimum = ctx.constants["minimum_order_value"] - if minimum is None: # The visible declaration explicitly has a NULL value. - return "TRUE" - return f"order_value >= {ctx.render_constant('minimum_order_value')}" - - -def active_status_filter(ctx) -> str: - status = ctx.render_enum_member(enum_name="order_status", member_name="active") - return f"status = {status}" -``` - -Callers can still pass explicit `@const(...)` or `@enum(...)` values as macro arguments. Context -lookups are intended for policy owned by the macro; both forms use the caller's declaration scope. - -## Grouped declarations - -Keep folder-scoped macros, enums, and constants together without mixing declaration directories into -resource listings: - -```text -models/orders/ -├── _sqlbuild/ -│ ├── macros/ # visible in orders/ and its descendants -│ ├── enums/ -│ ├── constants/ -│ ├── _macros/ # visible only to resources directly in orders/ -│ ├── _enums/ -│ └── _constants/ -├── intermediate/ -└── mart/ -``` - -The containing `orders/` directory remains the declaration owner. Existing declaration directories -directly below an owner remain supported. Placement diagnostics recommend the grouped layout. -`_sqlbuild/` is reserved for the six declaration-role directories shown above; other direct entries -are rejected rather than silently treated as resources. A grouped directory must sit below a -concrete owner: `models/_sqlbuild/` is invalid because declarations at that boundary belong in the -project-wide `macros/`, `enums/`, or `constants/` roots. - -SQL tests retain their own lexical declaration scope and also receive the deterministic union of -file-based declarations visible to their inferred tested resources. This lets model and macro tests -exercise scoped production macros without promoting those macros globally. Mock fixture resources -do not broaden test visibility. - -## Compiler-integrated Rules - -Rules turn repeatable SQL and project review decisions into compile-time diagnostics. Mandatory -compiler correctness still runs first. SQLBuild then evaluates selected native built-ins, followed by -selected custom Python rules, before completing compile artifacts. `sqb compile` is authoritative; -build and execution commands enforce the same configuration. Rules report findings and never rewrite -SQL. `sqb format` remains a separate source-rewriting command. - -Long model and scenario descriptions are reflowed deterministically. Ordinary authored line breaks -are normalized as spaces, while blank lines preserve paragraph boundaries. Configure the maximum -physical line width in `sqlbuild_project.toml` (the default is `100`): - -```toml -[format] -line_width = 100 -``` - -Prefer family selection in `sqlbuild_project.toml` over hand-maintained lists of exact codes: - -```toml -[rules] -select = ["SQBRSQL", "SQBRMODEL", "SQBRGRAPH", "XSQBRARCH"] -ignore = ["SQBRSQL004"] -``` - -Built-in codes use `SQBR`, such as `SQBRSQL001` and -`SQBRGRAPH101`. Custom codes use `XSQBR`, such as -`XSQBRARCH001`. A family is always the code with its final three digits removed. - -Family selectors automatically include new built-in rules on upgrade. Exact-code lists retain -their current membership and must be updated manually. In particular, selecting `SQBRSQL` enables -`SQBRSQL040` (plain JOIN predicates) and `SQBRSQL041` (terminal CTE naming); existing projects may -need to extract computed join keys into input CTEs and rename their last CTE to `final`. -See the [SQL rule conventions](https://sqlbuild.com/docs/concepts/rules/configuration-and-selection/#join-keys-and-final-cte-names) -for their exact scope and relationship to `SQBRSQL035`. - -`sqb format` reports a file-specific `format-unsafe` fault whenever a SQL body cannot be safely -formatted, including parser, comment-attachment, interpolation-restoration, and idempotence -failures. The entire original file is retained (including its headers and fixtures), the reason -appears in human and JSON output, and both -formatting and `sqb format --check` exit nonzero. A declined body is never counted as canonical. - -Formatting preserves authored cast types, postfix casts, quoted literals, variant paths, typed -lambda parameters, and supported SQL function spellings while applying canonical layout. It uses -the compiler's trusted-SQL function-depth budget and does not impose the separate browser-oriented -UNION-chain limit from Polyglot's convenience formatting API. SQLBuild calls retain their authored -spelling, including zero-argument cursor and empty-fixture intrinsics. CTE-producing macros remain -authored calls rather than expanded project SQL, with each call on its own CTE-list line and -leading comments attached to the node they describe. - -Custom rules are ordinary Python beneath `rules/**/*.py`. Only `@rule` functions register; helper -functions, constants, dataclasses, classes, and nested packages remain ordinary Python. Typed, -keyword-only parameters determine whether a rule runs once per model or once per project: - -```python -from sqlbuild.rules import Finding, Model, RuleContext, rule - - -@rule( - code="XSQBRARCH001", - message="Final models must declare an order identifier", - remediation="Declare order_id in the model contract.", -) -def final_order_identifier(*, model: Model, ctx: RuleContext) -> list[Finding]: - declared = {column.name for column in ctx.columns.declared(model)} - return [] if "order_id" in declared else [ctx.finding(subject=model)] -``` - -Use `Project` instead of `Model` for an invariant with no natural model subject. A model rule can -still inspect project-wide facts. `RuleContext` exposes compiler-owned SQL, graph, columns, -contracts, tests, audits, declarations, project metadata, and a deterministic project tree. Common -SQL facts are typed and lazy; the full Polyglot AST is an explicit escape hatch at -`ctx.sql.for_model(model).expanded.polyglot_ast()`. - -Custom rules are deterministic and cacheable. Environment, network, subprocess, time, randomness, -and untracked filesystem access are rejected. Tracked project text must be read through -`ctx.project.tree`, and implementation, options, subject facts, helper code, project observations, -and backend compatibility participate in cache identity. +## Warehouses -Inspect and run focused selections with: - -```bash -sqb rules list -sqb rules show SQBRSQL001 -sqb rules run SQBRSQL -sqb rules run XSQBRARCH --select customer_orders -sqb rules skills --check -``` - -Test custom rules through the real discovery and compiler path with `RuleCase` and `evaluate_rule` -from `sqlbuild.rules.testing`. - -The neutral large-project benchmark supports 1,000, 3,000, 5,000, and 10,000-model profiles and -reports repeated median/p95 timings with cache accounting and phase breakdowns: - -```bash -uv run python -m scripts.benchmark_rules --models 3000 --iterations 5 -uv run python -m scripts.benchmark_rules --models 5000 --iterations 5 -``` - -## Supported adapters - -| Adapter | Status | -|---------|--------| +| Warehouse | Status | +|-----------|--------| +| Snowflake | Supported | | DuckDB | Supported | | MotherDuck | Supported | -| Snowflake | Supported | -| BigQuery | Supported | -| Databricks | Supported | | PostgreSQL | Supported | -| SQL Server | Supported | - -ClickHouse, Redshift, Trino, Spark, and Athena are on the way. - -## Snowflake cost estimates - -Native Snowflake builds automatically show a compact per-run busy-compute estimate. SQLBuild -attributes visible overlapping query intervals fairly across active queries, converts attributed -seconds using the warehouse-size credit rate, and estimates USD from the configured rate: - -```toml -[cost] -usd_per_credit = 3.00 -``` - -The default is `3.00` USD per credit and is visibly marked as a default. Configure the value with -your Snowflake contract rate. Use `sqb cost`, `sqb cost latest`, `sqb cost `, or -`sqb cost history --since 7d` to inspect persisted records. `--json` and `--json-output PATH` -provide a versioned, decimal-safe output contract. Pending detail records are refreshed from -Snowflake when inspected again. +| BigQuery | Beta | +| Databricks | Beta | +| SQL Server | Beta | -These values are attributed compute credits and estimated cost, not Snowflake-billed credits or -invoice reconciliation. The estimate uses only query history visible to the executing role and -does not reconstruct invisible concurrent work, warehouse resume or idle tail, the 60-second -minimum, cloud-services credits, contract adjustments, or multi-cluster billing. Run metadata and -query IDs are stored under `target/executions//`; that statement ledger stores only an SQL -digest, not SQL text. Executed SQL artifacts are stored separately under the sensitive -`target/run/` tree. +Snowflake is the main target. Beta adapters build, test and plan, but have had less production use +so far. See [adapters](https://sqlbuild.com/docs/concepts/adapters/). -## Documentation +## Free and independent -Full documentation is available at [docs.sqlbuild.com](https://docs.sqlbuild.com). +SQLBuild is Apache 2.0 and will stay free: no paid tier, no commercial edition, and no feature held +back for one. Its state lives in your warehouse, next to your data. See the +[roadmap](https://sqlbuild.com/docs/roadmap/) for what's next. ## Contributing -We welcome contributions. Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. +Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md). ## License diff --git a/scripts/docs_skill/_helpers/clone.py b/scripts/docs_skill/_helpers/clone.py deleted file mode 100644 index 1d79a20ae5..0000000000 --- a/scripts/docs_skill/_helpers/clone.py +++ /dev/null @@ -1,19 +0,0 @@ -"""Docs repository cloning for skill generation.""" - -import shutil -import subprocess -from pathlib import Path - - -def clone_docs_repo(*, repo_url: str, clone_dir: Path) -> Path: - """Clone the configured docs repository into a clean directory.""" - - if clone_dir.exists(): - shutil.rmtree(clone_dir) - - clone_dir.parent.mkdir(parents=True, exist_ok=True) - subprocess.run( - ["git", "clone", "--depth", "1", repo_url, str(clone_dir)], - check=True, - ) - return clone_dir diff --git a/scripts/docs_skill/_helpers/mdx.py b/scripts/docs_skill/_helpers/mdx.py index bc855fe1cd..94c0424f03 100644 --- a/scripts/docs_skill/_helpers/mdx.py +++ b/scripts/docs_skill/_helpers/mdx.py @@ -2,7 +2,7 @@ import re -from scripts.docs_skill.constants import FRONTMATTER_DELIMITER +from scripts.docs_skill.constants import COMPONENT_TAG_END, FRONTMATTER_DELIMITER from scripts.docs_skill.models import MdxPage @@ -41,6 +41,7 @@ def _split_frontmatter(contents: str) -> tuple[dict[str, str], str]: def _clean_mdx_body(body: str) -> str: cleaned_lines: list[str] = [] in_code_block: bool = False + in_component_tag: bool = False for raw_line in body.splitlines(): line: str = raw_line.rstrip() @@ -53,6 +54,12 @@ def _clean_mdx_body(body: str) -> str: continue stripped: str = line.strip() + if in_component_tag: + in_component_tag = COMPONENT_TAG_END not in stripped + continue + if re.match(r"<[A-Z][\w.]*\b", stripped) and COMPONENT_TAG_END not in stripped: + in_component_tag = True + continue if _is_mdx_only_line(stripped): continue cleaned_lines.append(line) diff --git a/scripts/docs_skill/_helpers/navigation.py b/scripts/docs_skill/_helpers/navigation.py index 72890b2327..e72c6cba20 100644 --- a/scripts/docs_skill/_helpers/navigation.py +++ b/scripts/docs_skill/_helpers/navigation.py @@ -4,83 +4,61 @@ from pathlib import Path from typing import Any -from scripts.docs_skill.constants import UNGROUPED_LABEL +from scripts.docs_skill.constants import DOCS_SLUG, INDEX_PAGE, UNGROUPED_LABEL from scripts.docs_skill.models import NavigationGroup -def list_ordered_page_paths(*, docs_root: Path) -> list[Path]: - """Return existing MDX pages in navigation order followed by unlisted pages.""" +def list_ordered_page_paths(*, docs_root: Path, sidebar_path: Path | None = None) -> list[Path]: + """Return existing pages in sidebar order followed by unlisted pages.""" - docs_json_path: Path = docs_root / "docs.json" - if docs_json_path.exists(): - docs_json: dict[str, Any] = json.loads(docs_json_path.read_text(encoding="utf-8")) - ordered_pages: list[str] = _collect_navigation_pages(docs_json.get("navigation", {})) - else: - ordered_pages = sorted(path.with_suffix("").as_posix() for path in docs_root.rglob("*.mdx")) - - page_paths: list[Path] = [] - seen_pages: set[str] = set() - for page in ordered_pages: - page_path: Path = Path(f"{page}.mdx") - if page in seen_pages or not (docs_root / page_path).exists(): - continue - page_paths.append(page_path) - seen_pages.add(page) - - for mdx_path in sorted(docs_root.rglob("*.mdx")): - page: str = mdx_path.relative_to(docs_root).with_suffix("").as_posix() - if page not in seen_pages: - page_paths.append(mdx_path.relative_to(docs_root)) - seen_pages.add(page) - - return page_paths + paths: list[Path] = [] + for group in list_navigation_groups(docs_root=docs_root, sidebar_path=sidebar_path): + paths.extend(group.page_paths) + return paths def _collect_navigation_pages(node: Any) -> list[str]: - pages: list[str] = [] - if isinstance(node, str): - return [node] + return [INDEX_PAGE if node == DOCS_SLUG else node.removeprefix(f"{DOCS_SLUG}/")] if isinstance(node, list): + pages: list[str] = [] for item in node: pages.extend(_collect_navigation_pages(item)) return pages if isinstance(node, dict): - for key in ("root", "groups", "pages"): - pages.extend(_collect_navigation_pages(node.get(key, []))) + return _collect_navigation_pages(node.get("slug", node.get("items", []))) + return [] - return pages +def list_navigation_groups( + *, docs_root: Path, sidebar_path: Path | None = None +) -> list[NavigationGroup]: + """Flatten nested sidebar groups within each top-level group, then add unlisted pages.""" -def list_navigation_groups(*, docs_root: Path) -> list[NavigationGroup]: - """Return pages grouped by top-level navigation group, then unlisted pages.""" - - ordered_paths: list[Path] = list_ordered_page_paths(docs_root=docs_root) - docs_json_path: Path = docs_root / "docs.json" - top_level_nodes: list[Any] = [] - if docs_json_path.exists(): - navigation: Any = json.loads(docs_json_path.read_text(encoding="utf-8")).get( - "navigation", {} - ) - top_level_nodes = navigation.get("groups", []) if isinstance(navigation, dict) else [] - + sidebar_path = sidebar_path or docs_root.resolve().parents[2] / "sidebar.json" + nodes: list[Any] = ( + json.loads(sidebar_path.read_text(encoding="utf-8")) if sidebar_path.exists() else [] + ) + available: dict[str, Path] = {} + for suffix in ("*.md", "*.mdx"): + for path in sorted(docs_root.rglob(suffix)): + relative_path: Path = path.relative_to(docs_root) + available[relative_path.with_suffix("").as_posix()] = relative_path groups: list[NavigationGroup] = [] - grouped_pages: set[str] = set() - for node in top_level_nodes: - label: str = str(node.get("group", UNGROUPED_LABEL)) if isinstance(node, dict) else "" - member_pages: set[str] = set(_collect_navigation_pages(node)) - page_paths: tuple[Path, ...] = tuple( - path - for path in ordered_paths - if path.with_suffix("").as_posix() in member_pages - and path.with_suffix("").as_posix() not in grouped_pages - ) - grouped_pages.update(path.with_suffix("").as_posix() for path in page_paths) - if page_paths: - groups.append(NavigationGroup(label=label, page_paths=page_paths)) - + seen: set[str] = set() + for node in nodes: + members: list[Path] = [] + for page in _collect_navigation_pages(node): + if page in available and page not in seen: + members.append(available[page]) + seen.add(page) + if members: + label: str = ( + node.get("label", UNGROUPED_LABEL) if isinstance(node, dict) else UNGROUPED_LABEL + ) + groups.append(NavigationGroup(label=label, page_paths=tuple(members))) remaining: tuple[Path, ...] = tuple( - path for path in ordered_paths if path.with_suffix("").as_posix() not in grouped_pages + available[page] for page in sorted(available) if page not in seen ) if remaining: groups.append(NavigationGroup(label=UNGROUPED_LABEL, page_paths=remaining)) diff --git a/scripts/docs_skill/_helpers/render.py b/scripts/docs_skill/_helpers/render.py index 0de0ddc788..95081283ca 100644 --- a/scripts/docs_skill/_helpers/render.py +++ b/scripts/docs_skill/_helpers/render.py @@ -8,8 +8,11 @@ from scripts.docs_skill._helpers.navigation import list_navigation_groups from scripts.docs_skill.constants import ( DOCS_BASE_URL, + DOCS_URL_PATH, GENERATED_MARKER, INDEX_FILENAME, + INDEX_PAGE, + SITE_BASE_URL, TABLE_OF_CONTENTS_MIN_LINES, ) from scripts.docs_skill.models import MdxPage, NavigationGroup @@ -18,10 +21,12 @@ _SECTION_HEADING: re.Pattern[str] = re.compile(r"^## (.+)$", re.MULTILINE) -def build_reference_pages(*, docs_root: Path) -> dict[str, str]: +def build_reference_pages(*, docs_root: Path, sidebar_path: Path | None = None) -> dict[str, str]: """Render every documentation page plus a grouped index, keyed by relative output path.""" - groups: list[NavigationGroup] = list_navigation_groups(docs_root=docs_root) + groups: list[NavigationGroup] = list_navigation_groups( + docs_root=docs_root, sidebar_path=sidebar_path + ) page_names: set[str] = set() for group in groups: page_names.update(path.with_suffix("").as_posix() for path in group.page_paths) @@ -49,7 +54,8 @@ def _render_page(*, page: MdxPage, page_name: str, known_pages: frozenset[str]) return ( normalize_blank_lines( f"{GENERATED_MARKER}\n\n# {page.title}\n\n{description_block}" - f"Online: {DOCS_BASE_URL}/{page_name}\n\n{contents_block}{body}" + f"Online: {DOCS_BASE_URL}/{'' if page_name == INDEX_PAGE else page_name + '/'}" + f"\n\n{contents_block}{body}" ).strip() + "\n" ) @@ -72,10 +78,13 @@ def _rewrite_internal_links(*, body: str, page_name: str, known_pages: frozenset source_dir: PurePosixPath = PurePosixPath(page_name).parent def replace(match: re.Match[str]) -> str: - target: str = match.group(1).strip("/") or "index" + path: str = match.group(1) anchor: str = match.group(2) or "" + if path != DOCS_URL_PATH and not path.startswith(f"{DOCS_URL_PATH}/"): + return f"]({SITE_BASE_URL}{path}{anchor})" + target: str = path.removeprefix(DOCS_URL_PATH).strip("/") or INDEX_PAGE if target not in known_pages: - return f"]({DOCS_BASE_URL}/{target}{anchor})" + return f"]({DOCS_BASE_URL}/{target}/{anchor})" relative: str = os.path.relpath(f"{target}.md", start=source_dir.as_posix() or ".") return f"]({PurePosixPath(relative).as_posix()}{anchor})" diff --git a/scripts/docs_skill/constants.py b/scripts/docs_skill/constants.py index a487a207c7..9e230cf9b3 100644 --- a/scripts/docs_skill/constants.py +++ b/scripts/docs_skill/constants.py @@ -1,12 +1,15 @@ """Stable constants for SQLBuild docs reference generation.""" -import tempfile from pathlib import Path -DEFAULT_REPO_URL: str = "https://github.com/chio-labs/sqlbuild-docs" -DEFAULT_CLONE_DIR: Path = Path(tempfile.gettempdir()) / "sqlbuild-docs-skill-source" +DEFAULT_DOCS_ROOT: Path = Path(__file__).resolve().parents[2] / "website/src/content/docs/docs" DEFAULT_OUTPUT_DIR: Path = Path("src/sqlbuild/.agents/skills/sqlbuild/references/docs") -DOCS_BASE_URL: str = "https://docs.sqlbuild.com" +SITE_BASE_URL: str = "https://sqlbuild.com" +DOCS_BASE_URL: str = f"{SITE_BASE_URL}/docs" +DOCS_SLUG: str = "docs" +DOCS_URL_PATH: str = f"/{DOCS_SLUG}" +INDEX_PAGE: str = "index" +COMPONENT_TAG_END: str = ">" GENERATED_MARKER: str = "" INDEX_FILENAME: str = "CONTENTS.md" FRONTMATTER_DELIMITER: str = "---" diff --git a/scripts/docs_skill/main/generate.py b/scripts/docs_skill/main/generate.py index aeeee74279..94d9deaac5 100644 --- a/scripts/docs_skill/main/generate.py +++ b/scripts/docs_skill/main/generate.py @@ -5,23 +5,18 @@ import argparse from pathlib import Path -from scripts.docs_skill._helpers.clone import clone_docs_repo from scripts.docs_skill._helpers.output import write_reference_pages from scripts.docs_skill._helpers.render import build_reference_pages -from scripts.docs_skill.constants import DEFAULT_CLONE_DIR, DEFAULT_OUTPUT_DIR, DEFAULT_REPO_URL +from scripts.docs_skill.constants import DEFAULT_DOCS_ROOT, DEFAULT_OUTPUT_DIR def generate_docs_skill(argv: list[str] | None = None) -> int: """Generate the bundled documentation reference pages for the SQLBuild skill.""" args: argparse.Namespace = _parse_args(argv) - docs_root: Path = args.docs_root or clone_docs_repo( - repo_url=args.repo_url, - clone_dir=args.clone_dir, - ) write_reference_pages( output_dir=args.output_dir, - pages=build_reference_pages(docs_root=docs_root), + pages=build_reference_pages(docs_root=args.docs_root), ) return 0 @@ -29,11 +24,9 @@ def generate_docs_skill(argv: list[str] | None = None) -> int: def _parse_args(argv: list[str] | None) -> argparse.Namespace: parser: argparse.ArgumentParser = argparse.ArgumentParser( description=( - "Clone sqlbuild-docs and convert each MDX page into a bundled skill reference page." + "Convert local SQLBuild website documentation into bundled skill reference pages." ) ) - parser.add_argument("--repo-url", default=DEFAULT_REPO_URL) - parser.add_argument("--clone-dir", type=Path, default=DEFAULT_CLONE_DIR) - parser.add_argument("--docs-root", type=Path) + parser.add_argument("--docs-root", type=Path, default=DEFAULT_DOCS_ROOT) parser.add_argument("--output-dir", type=Path, default=DEFAULT_OUTPUT_DIR) return parser.parse_args(argv) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/SKILL.md b/src/sqlbuild/.agents/skills/sqlbuild/SKILL.md index 55033028ee..2e102a1ab1 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/SKILL.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/SKILL.md @@ -163,4 +163,4 @@ diff. - Full documentation for this SQLBuild version: [references/docs/CONTENTS.md](references/docs/CONTENTS.md) lists every page; open only the page you need, for example `references/docs/cli/build.md` or `references/docs/concepts/snapshots.md`. The same pages are online at - https://docs.sqlbuild.com if you need a newer version. + https://sqlbuild.com/docs/ if you need a newer version. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md index 25e70c7cb0..34057c0f06 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md @@ -6,9 +6,10 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in ## Getting Started -- [Introduction](index.md) (`index`) - Verify early, test properly, and deploy reversibly. SQL pipelines with the rigor of real software - free and open source. +- [Introduction](index.md) (`index`) - The refactorable warehouse. Verify early, test properly, and refactor safely. - [Quickstart](quickstart.md) (`quickstart`) - Get a SQLBuild project running locally with DuckDB in under a minute. - [Feature Comparison](feature-comparison.md) (`feature-comparison`) - Feature comparison between SQLBuild, dbt, and SQLMesh. +- [Roadmap](roadmap.md) (`roadmap`) - What's coming next in SQLBuild. - [Benchmarks](benchmarks.md) (`benchmarks`) - Complete compiler performance on projects with thousands of models, SQL tests, and audit attachments. ## dbt Compatibility @@ -22,12 +23,12 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in - [Project Configuration](concepts/project-configuration.md) (`concepts/project-configuration`) - Configure your SQLBuild project with sqlbuild_project.toml and sqlbuild_local.toml. - [Resource Identities](concepts/resource-identities.md) (`concepts/resource-identities`) - Canonical names for SQLBuild resources, selectors, state, and integrations. - [Overview](concepts/adapters.md) (`concepts/adapters`) - Supported database engines and their connection configuration. +- [Snowflake](concepts/adapters/snowflake.md) (`concepts/adapters/snowflake`) - Snowflake adapter configuration for SQLBuild. - [DuckDB](concepts/adapters/duckdb.md) (`concepts/adapters/duckdb`) - DuckDB adapter configuration for SQLBuild. - [MotherDuck](concepts/adapters/motherduck.md) (`concepts/adapters/motherduck`) - MotherDuck adapter configuration for SQLBuild. -- [Snowflake](concepts/adapters/snowflake.md) (`concepts/adapters/snowflake`) - Snowflake adapter configuration for SQLBuild. +- [PostgreSQL](concepts/adapters/postgres.md) (`concepts/adapters/postgres`) - PostgreSQL adapter configuration for SQLBuild. - [BigQuery](concepts/adapters/bigquery.md) (`concepts/adapters/bigquery`) - BigQuery adapter configuration for SQLBuild. - [Databricks](concepts/adapters/databricks.md) (`concepts/adapters/databricks`) - Databricks adapter configuration for SQLBuild. -- [PostgreSQL](concepts/adapters/postgres.md) (`concepts/adapters/postgres`) - PostgreSQL adapter configuration for SQLBuild. - [SQL Server](concepts/adapters/sqlserver.md) (`concepts/adapters/sqlserver`) - Microsoft SQL Server adapter configuration for SQLBuild. - [Sources](concepts/sources.md) (`concepts/sources`) - Declare external data inputs for your pipeline. - [Seeds](concepts/seeds.md) (`concepts/seeds`) - Load static CSV data into your pipeline as tables. @@ -65,12 +66,13 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in - [Scenarios](concepts/scenarios.md) (`concepts/scenarios`) - End-to-end tests that build real project graphs against coherent fixture data. - [Selectors](concepts/selectors.md) (`concepts/selectors`) - Target specific models, paths, tags, or DAG subsets with select and exclude flags. - [Column Lineage](concepts/column-lineage.md) (`concepts/column-lineage`) - Trace individual columns through your SQL pipeline - understand where data comes from and where it goes. -- [Data Diffs](concepts/diff.md) (`concepts/diff`) - Compare schemas and data between targets or virtual environments to validate changes before promotion. +- [Data Diffs](concepts/diff.md) (`concepts/diff`) - Compare schemas and data between targets to validate changes before they reach production. ## Advanced Concepts - [Execution Observability](concepts/observability.md) (`concepts/observability`) - Choose authoritative lifecycle facts, readable logs, or command-output records. - [Typed Sinks](concepts/observability/sinks.md) (`concepts/observability/sinks`) - Export lifecycle facts and command output through project-owned providers. +- [Execution history](concepts/observability/execution-history.md) (`concepts/observability/execution-history`) - Store and query lifecycle events and run history from Python, with SQLite locally or PostgreSQL when deployed. - [Overview](concepts/declaration-scopes.md) (`concepts/declaration-scopes`) - Limit enums, constants, and macros to the parts of a project that use them. - [How Visibility Works](concepts/declaration-scopes/visibility.md) (`concepts/declaration-scopes/visibility`) - See which enums, constants, and macros are available to each SQL file. - [Where to Put Declarations](concepts/declaration-scopes/placement.md) (`concepts/declaration-scopes/placement`) - Choose the narrowest folder that contains every real use. @@ -90,21 +92,6 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in - [Providers](concepts/python-nodes/providers.md) (`concepts/python-nodes/providers`) - Shared runtime services for Python nodes and hooks. - [SQL References](concepts/python-nodes/sql-references.md) (`concepts/python-nodes/sql-references`) - Read SQL models and sources from Python nodes without creating SQL dependencies. -## Virtual Environments (Alpha) - -- [Overview](concepts/virtual-environments.md) (`concepts/virtual-environments`) - Version-controlled SQL pipeline environments with instant promotion and rollback. -- [Setup](concepts/virtual-environments/setup.md) (`concepts/virtual-environments/setup`) - Configure virtual environments and initialize the state store. -- [Building](concepts/virtual-environments/building.md) (`concepts/virtual-environments/building`) - Virtual builds, VDE creation, partial builds, and seeded incrementals. -- [Promotion](concepts/virtual-environments/promotion.md) (`concepts/virtual-environments/promotion`) - Promote VDE refs and diff virtual environments. -- [Rollback](concepts/virtual-environments/rollback.md) (`concepts/virtual-environments/rollback`) - Checkpoints and rollback for virtual environments. -- [Adopt and Detach](concepts/virtual-environments/adopt-detach.md) (`concepts/virtual-environments/adopt-detach`) - Migrate existing projects to and from virtual mode. -- [Clone](concepts/virtual-environments/clone.md) (`concepts/virtual-environments/clone`) - Hydrate physical versions from a source warehouse. -- [Diff](concepts/virtual-environments/diff.md) (`concepts/virtual-environments/diff`) - Compare virtual data environments. -- [Reconcile](concepts/virtual-environments/reconcile.md) (`concepts/virtual-environments/reconcile`) - Diagnose and repair drift between state and warehouse. -- [Locks](concepts/virtual-environments/locks.md) (`concepts/virtual-environments/locks`) - Advisory locks for concurrent access control. -- [Janitor](concepts/virtual-environments/janitor.md) (`concepts/virtual-environments/janitor`) - Cleanup of virtual environment artifacts and retention policies. -- [Recovery](concepts/virtual-environments/recovery.md) (`concepts/virtual-environments/recovery`) - Diagnosing and recovering from failures in virtual mode. - ## Integrations - [Overview](integrations/dagster.md) (`integrations/dagster`) - Orchestrate SQLBuild pipelines with Dagster scheduling, retries, and asset UI. @@ -115,36 +102,32 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in ## CLI Reference -- [init](cli/init.md) (`cli/init`) - Scaffold a new SQLBuild project. -- [playground](cli/playground.md) (`cli/playground`) - Create a self-contained SQLBuild project to explore locally. -- [skills](cli/skills.md) (`cli/skills`) - Install SQLBuild skill files for AI coding agents. -- [compile](cli/compile.md) (`cli/compile`) - Compile models into resolved SQL, validate contracts, and write target artifacts - fully offline. -- [format](cli/format.md) (`cli/format`) - Apply canonical SQLBuild SQL formatting. -- [contract](cli/contract.md) (`cli/contract`) - Compare or generate repository contracts from physical warehouse schemas. +- [sqb init](cli/init.md) (`cli/init`) - Scaffold a new SQLBuild project. +- [sqb playground](cli/playground.md) (`cli/playground`) - Create a self-contained SQLBuild project to explore locally. +- [sqb skills](cli/skills.md) (`cli/skills`) - Install SQLBuild skill files for AI coding agents. +- [sqb compile](cli/compile.md) (`cli/compile`) - Compile models into resolved SQL, validate contracts, and write target artifacts - fully offline. +- [sqb format](cli/format.md) (`cli/format`) - Apply canonical SQLBuild SQL formatting. +- [sqb contract](cli/contract.md) (`cli/contract`) - Compare or generate repository contracts from physical warehouse schemas. - [scope](cli/scope.md) (`cli/scope`) - Inspect declaration visibility, usage, placement, and move impact offline. -- [rules](cli/rules.md) (`cli/rules`) - List, inspect, or run compiler-integrated Rules and generate project guidance. -- [plan](cli/plan.md) (`cli/plan`) - Preview what SQLBuild will do before executing. -- [build](cli/build.md) (`cli/build`) - Compile, plan, and execute the selected build lifecycle. -- [load](cli/load.md) (`cli/load`) - Load managed sources into the warehouse. -- [seed](cli/seed.md) (`cli/seed`) - Load seed CSV files into the warehouse. -- [test](cli/test.md) (`cli/test`) - Run SQL unit tests and multi-model tests in isolation. -- [scenario](cli/scenario.md) (`cli/scenario`) - Run end-to-end scenario tests against the warehouse or locally with DuckDB. -- [audit](cli/audit.md) (`cli/audit`) - Run data quality audits in isolation. -- [freshness](cli/freshness.md) (`cli/freshness`) - Observe source freshness without writing state. -- [check](cli/check.md) (`cli/check`) - Run Python checks against tasks, assets, and loaders. -- [clone](cli/clone.md) (`cli/clone`) - Copy model relations between configured targets. -- [diff](cli/diff.md) (`cli/diff`) - Compare schemas and data between targets or virtual environments. +- [sqb rules](cli/rules.md) (`cli/rules`) - List, inspect, or run compiler-integrated Rules and generate project guidance. +- [sqb plan](cli/plan.md) (`cli/plan`) - Preview what SQLBuild will do before executing. +- [sqb build](cli/build.md) (`cli/build`) - Compile, plan, and execute the selected build lifecycle. +- [sqb load](cli/load.md) (`cli/load`) - Load managed sources into the warehouse. +- [sqb seed](cli/seed.md) (`cli/seed`) - Load seed CSV files into the warehouse. +- [sqb test](cli/test.md) (`cli/test`) - Run SQL unit tests and multi-model tests in isolation. +- [sqb scenario](cli/scenario.md) (`cli/scenario`) - Run end-to-end scenario tests against the warehouse or locally with DuckDB. +- [sqb audit](cli/audit.md) (`cli/audit`) - Run data quality audits in isolation. +- [sqb freshness](cli/freshness.md) (`cli/freshness`) - Observe source freshness without writing state. +- [sqb check](cli/check.md) (`cli/check`) - Run Python checks against tasks, assets, and loaders. +- [sqb clone](cli/clone.md) (`cli/clone`) - Copy model relations between configured targets. +- [diff](cli/diff.md) (`cli/diff`) - Compare schemas and data between targets. - [lineage](cli/lineage.md) (`cli/lineage`) - Explore model and column-level dependency graphs from the command line. -- [dag](cli/dag.md) (`cli/dag`) - Generate the static DAG artifact for Dagster and other integrations. -- [query](cli/query.md) (`cli/query`) - Run ad hoc SQL queries against the project database. +- [sqb dag](cli/dag.md) (`cli/dag`) - Generate the static DAG artifact for Dagster and other integrations. +- [sqb query](cli/query.md) (`cli/query`) - Run ad hoc SQL queries against the project database. - [debug](cli/debug.md) (`cli/debug`) - Validate project configuration and test the warehouse connection. -- [janitor](cli/janitor.md) (`cli/janitor`) - Archive and then delete stale warehouse relations. -- [clean](cli/clean.md) (`cli/clean`) - Remove compiled artifacts from the target directory. -- [dbt](cli/dbt.md) (`cli/dbt`) - Coordinate dbt and SQLBuild projects. -- [state](cli/state.md) (`cli/state`) - Manage the virtual mode state store. -- [promote](cli/promote.md) (`cli/promote`) - Promote VDE refs from one virtual environment to another. -- [rollback](cli/rollback.md) (`cli/rollback`) - Roll back a VDE to a prior finalized checkpoint. -- [reconcile](cli/reconcile.md) (`cli/reconcile`) - Diagnose and repair drift between virtual state and warehouse. +- [sqb janitor](cli/janitor.md) (`cli/janitor`) - Archive and then delete stale warehouse relations. +- [sqb clean](cli/clean.md) (`cli/clean`) - Remove compiled artifacts from the target directory. +- [sqb dbt](cli/dbt.md) (`cli/dbt`) - Coordinate dbt and SQLBuild projects. ## Other pages diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md index 44fa20d9ee..2f78137285 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md @@ -4,7 +4,7 @@ > Complete compiler performance on projects with thousands of models, SQL tests, and audit attachments. -Online: https://docs.sqlbuild.com/benchmarks +Online: https://sqlbuild.com/docs/benchmarks/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md index dcc7418bfa..16adc5a1ee 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/audit.md @@ -1,12 +1,10 @@ -# audit +# sqb audit > Run data quality audits in isolation. -Online: https://docs.sqlbuild.com/cli/audit - -# sqb audit +Online: https://sqlbuild.com/docs/cli/audit/ Runs all attached audits without rebuilding models. Useful for verifying data quality on existing warehouse state. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md index 68a929e6e7..06916f9533 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md @@ -1,10 +1,10 @@ -# build +# sqb build > Compile, plan, and execute the selected build lifecycle. -Online: https://docs.sqlbuild.com/cli/build +Online: https://sqlbuild.com/docs/cli/build/ ## Contents @@ -18,11 +18,7 @@ Online: https://docs.sqlbuild.com/cli/build - Fingerprints - Runtime artifacts -# sqb build - -Compiles, plans, and executes the selected build lifecycle. Direct mode runs the full selected -scope. [Virtual environments](../concepts/virtual-environments.md) can additionally use -`--changes-only` to build work not already represented by the target VDE's bound versions. Use +Compiles, plans, and executes the selected build lifecycle, running the full selected scope. Use `--no-tests` and `--no-audits` to skip validation for fast iteration. ## Usage @@ -36,7 +32,6 @@ sqb --project-dir build [flags] | Flag | Description | |------|-------------| | `--target` | Build against a configured target instead of the active/default target | -| `--changes-only` | Virtual mode only: build work not already bound to its expected VDE version | | `--no-tests` | Skip SQL unit tests | | `--no-audits` | Skip audits | | `--no-python` | Skip read-side Python tasks and assets (loader-side Python still runs for selected sources) | @@ -76,7 +71,7 @@ This replaces the former `sqb run` command. The full lifecycle (tests + audits) 2. Seeds are loaded (if changed) 3. Source audits run before their dependent models (unless `--no-audits`) 4. SQL unit tests run before their target model (unless `--no-tests`) -5. Models are materialized in DAG topological order (virtual `--changes-only` limits this to stale work) +5. Models are materialized in DAG topological order 6. Error-severity audits run against the staging table before promotion to the target (unless `--no-audits`) ## Output diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md index e80ed16ea7..986f891925 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/check.md @@ -1,12 +1,10 @@ -# check +# sqb check > Run Python checks against tasks, assets, and loaders. -Online: https://docs.sqlbuild.com/cli/check - -# sqb check +Online: https://sqlbuild.com/docs/cli/check/ Runs Python [checks](../concepts/python-nodes/checks.md) in isolation. Checks validate the output of tasks, assets, and loaders. For SQL relation validation, use [`sqb audit`](audit.md) instead. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clean.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clean.md index de8fdc9c2b..a2f363c462 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clean.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clean.md @@ -1,12 +1,10 @@ -# clean +# sqb clean > Remove compiled artifacts from the target directory. -Online: https://docs.sqlbuild.com/cli/clean - -# sqb clean +Online: https://sqlbuild.com/docs/cli/clean/ Removes the `target/` directory containing compiled artifacts, runtime SQL recordings, and other build outputs. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md index bb830b801d..d51a437337 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/clone.md @@ -1,12 +1,10 @@ -# clone +# sqb clone > Copy model relations between configured targets. -Online: https://docs.sqlbuild.com/cli/clone - -# sqb clone +Online: https://sqlbuild.com/docs/cli/clone/ Copies selected relations from one target to another. It uses adapter-native cloning where supported and physical copies where required; `--hard-copy` forces a physical copy on adapters that support both. @@ -18,8 +16,8 @@ When `--to` is omitted, the destination is the active target selected by `--targ `--from` selects only the origin namespace and its clone-origin policy. `--to`, or the active target when it is omitted, selects the sole physical connection used by the operation. The origin namespace must therefore be visible and readable through the destination connection. -SQLBuild never resolves or opens the origin target's connection, including for direct, -deferred, and virtual clone. Clone cannot perform a cross-account, cross-server, or cross-file +SQLBuild never resolves or opens the origin target's connection, including for direct and +deferred clone. Clone cannot perform a cross-account, cross-server, or cross-file transfer by opening a second connection. ## Usage diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md index 1653e621bb..43b86abd84 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md @@ -1,10 +1,10 @@ -# compile +# sqb compile > Compile models into resolved SQL, validate contracts, and write target artifacts - fully offline. -Online: https://docs.sqlbuild.com/cli/compile +Online: https://sqlbuild.com/docs/cli/compile/ ## Contents @@ -17,8 +17,6 @@ Online: https://docs.sqlbuild.com/cli/compile - Column lineage modes - Examples -# sqb compile - Discovers the complete project, resolves references, expands macros, validates SQL, checks column contracts, computes column lineage, and writes compiled artifacts to `target/`. The compile command is fully offline—it does not connect to the warehouse. ## Usage diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md index 161074b015..b8d9463b8c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/contract.md @@ -1,12 +1,10 @@ -# contract +# sqb contract > Compare or generate repository contracts from physical warehouse schemas. -Online: https://docs.sqlbuild.com/cli/contract - -# sqb contract +Online: https://sqlbuild.com/docs/cli/contract/ Existing-schema comparison is explicitly online and read-only with respect to the warehouse. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dag.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dag.md index 51ded90d7c..7aa68e393a 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dag.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dag.md @@ -1,10 +1,10 @@ -# dag +# sqb dag > Generate the static DAG artifact for Dagster and other integrations. -Online: https://docs.sqlbuild.com/cli/dag +Online: https://sqlbuild.com/docs/cli/dag/ ## Contents @@ -14,8 +14,6 @@ Online: https://docs.sqlbuild.com/cli/dag - Output format - Examples -# sqb dag - Compiles the project and outputs the static DAG artifact. The artifact contains every node (source, seed, model, function), dependency edge, and check (test, audit, scenario) in your project as structured JSON. It is the bridge between SQLBuild and external orchestrators like Dagster. This artifact describes resource dependencies, not column-level lineage. Use [`sqb lineage`](lineage.md) to inspect or export column traces. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md index 26be5dcf32..676c9733f5 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/dbt.md @@ -1,12 +1,10 @@ -# dbt +# sqb dbt > Coordinate dbt and SQLBuild projects. -Online: https://docs.sqlbuild.com/cli/dbt - -# sqb dbt +Online: https://sqlbuild.com/docs/cli/dbt/ Orchestrate dbt and SQLBuild together. Each subcommand runs dbt first, then SQLBuild, with selection logic across both project graphs. dbt remains responsible for dbt-owned models; SQLBuild validates and executes SQLBuild-owned models downstream. See [Using SQLBuild with dbt](../concepts/dbt-compatibility/overview.md) for scope and selection behavior. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md index 4b70bcd03f..f53dbe7dde 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/debug.md @@ -4,7 +4,7 @@ > Validate project configuration and test the warehouse connection. -Online: https://docs.sqlbuild.com/cli/debug +Online: https://sqlbuild.com/docs/cli/debug/ Checks that your project config is valid, the adapter is resolvable, and the warehouse connection works. Useful for diagnosing setup issues. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md index d849384294..9f03bdc4a1 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/diff.md @@ -2,25 +2,32 @@ # diff -> Compare schemas and data between targets or virtual environments. +> Compare schemas and data between targets. -Online: https://docs.sqlbuild.com/cli/diff +Online: https://sqlbuild.com/docs/cli/diff/ -Compares schemas and optionally row-level data between two build contexts: two targets (e.g. `prod:dev`) in direct mode, or two virtual environments when virtual mode is enabled. See [Data Diffs](../concepts/diff.md) for detailed usage. +Compares schemas and optionally row-level data between two targets (e.g. `prod:dev`), or between two read-only SQL queries. See [Data Diffs](../concepts/diff.md) for detailed usage. ## Usage ```bash -sqb diff : [flags] +# Model mode +sqb diff : --select [flags] + +# Query mode +sqb diff --left-query --right-query (--key | --unkeyed) [flags] +sqb diff --left-query --right-query --schema-only [flags] ``` -The first argument is a positional `FROM:TO` range. Exactly one mode is required: `--full`, `--schema-only`, or `--bounded `. +In model mode, the first argument is a positional `FROM:TO` range, `--select` is required, and exactly one mode is required: `--full`, `--schema-only`, or `--bounded `. -In direct mode, `FROM` and `TO` are configured target names. Their database/schema namespaces remain +`FROM` and `TO` are configured target names. Their database/schema namespaces remain authoritative, while the `TO` target's named connection executes the complete comparison and must be able to read both namespaces. -Full and bounded row comparisons require the model to define `unique_key`. Bounded mode uses the model's cursor and falls back to a full row comparison when no cursor is configured. +Full and bounded row comparisons match rows on the model's `unique_key`, or on `--key` columns when given; use `--unkeyed` for an exact full-row comparison. Bounded mode uses the model's cursor and falls back to a full row comparison when no cursor is configured. + +In query mode, omit `FROM:TO` and model selectors; each query identifies its own input and the active target's connection (or `--target`) runs both. A full comparison is the default and needs `--key` or `--unkeyed`; `--schema-only` takes neither, and `--bounded` is model-only. ## Flags @@ -39,7 +46,19 @@ Full and bounded row comparisons require the model to define `unique_key`. Bound | `--max-models` | Fail when the selected scope contains more models than this limit | | `--max-columns` | Fail when either side of a model has more columns than this limit | | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | -| `--select`, `-s` | Select specific models to diff (required in v1) | +| `--key` | Row key column; repeat for a composite key. Overrides the model's `unique_key` | +| `--unkeyed` | Compare exact full-row multisets when there is no stable row key | +| `--exclude-column` | Leave a column out of the comparison; repeatable | +| `--tolerance` | Numeric tolerance such as `amount:absolute=1` or `rate:relative=0.001`; repeatable | +| `--left-query`, `--right-query` | Compare two read-only SQL queries instead of models | +| `--left-query-file`, `--right-query-file` | Read each query from a UTF-8 file | +| `--left-label`, `--right-label` | Names for the two query sides in the output | +| `--target` | Target whose connection runs both queries | +| `--max-value-length` | Maximum characters shown per example value (default 160) | +| `--no-example-values` | Keep keys and counts but hide example values | +| `--full-example-values` | Show complete example values | +| `--json` | Print one JSON document to stdout | +| `--select`, `-s` | Select specific models to diff (required in model mode) | | `--exclude` | Exclude specific models from diffing | ## Examples @@ -59,8 +78,11 @@ sqb diff prod:dev --bounded 14d --sample-rows 50000 --sample-seed 7 --select ord # Force exhaustive comparison despite inherited sampling defaults sqb diff prod:dev --bounded 14d --exhaustive --select order_lines + +# Compare two queries on a key +sqb diff --left-query-file queries/orders_before.sql --right-query-file queries/orders_after.sql --key order_id ``` ## Exit codes -Returns `0` when all selected models have no differences, `1` when any model has schema or row differences. +Returns `0` when all selected models have no differences, `1` when any model has schema or row differences. Query mode also returns `2` when a safety, schema, or key check prevented a complete comparison, and `3` when setup, execution, or cleanup failed; see [Query output and exit codes](../concepts/diff.md#query-output-and-exit-codes). diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/format.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/format.md index 49650950e5..c7775839a3 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/format.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/format.md @@ -1,12 +1,10 @@ -# format +# sqb format > Apply canonical SQLBuild SQL formatting. -Online: https://docs.sqlbuild.com/cli/format - -# sqb format +Online: https://sqlbuild.com/docs/cli/format/ Formats SQL files using SQLBuild's canonical, dialect-aware representation. @@ -31,3 +29,30 @@ sqb format --diff --select tag:marts Formatting owns presentation and source rewriting. Compiler-integrated diagnostics belong to configured [Rules](../concepts/rules.md), which never mutate source. + +## Line width and descriptions + +Long model and scenario descriptions are reflowed deterministically. Ordinary authored line breaks +are normalized as spaces, while blank lines preserve paragraph boundaries. Configure the maximum +physical line width in `sqlbuild_project.toml` (the default is `100`): + +```toml +[format] +line_width = 100 +``` + +## What formatting preserves + +Formatting preserves authored cast types, postfix casts, quoted literals, variant paths, typed +lambda parameters, and supported SQL function spellings while applying canonical layout. SQLBuild +calls keep their authored spelling, including zero-argument cursor and empty-fixture intrinsics. +CTE-producing macros remain authored calls rather than expanded project SQL, with each call on its +own CTE-list line and leading comments attached to the node they describe. + +## Unsafe bodies + +`sqb format` reports a file-specific `format-unsafe` fault whenever a SQL body cannot be safely +formatted, including parser, comment-attachment, interpolation-restoration, and idempotence +failures. The entire original file is kept (including its headers and fixtures), the reason +appears in human and JSON output, and both formatting and `sqb format --check` exit nonzero. A +declined body is never counted as canonical. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md index ebe5c6e8cf..514e82f8a8 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/freshness.md @@ -1,10 +1,10 @@ -# freshness +# sqb freshness > Observe source freshness without writing state. -Online: https://docs.sqlbuild.com/cli/freshness +Online: https://sqlbuild.com/docs/cli/freshness/ ## Contents @@ -17,8 +17,6 @@ Online: https://docs.sqlbuild.com/cli/freshness - JSON output - See also -# sqb freshness - Observes the current data version of each source and reports whether the data has changed since the last build. Does not write any state or trigger builds. ## Usage @@ -34,7 +32,6 @@ sqb --project-dir freshness [flags] | `--state` | Compare observations against stored freshness state from the last build | | `--fail-on-error` | Exit with code 1 if any source observation fails or is unknown | | `--fail-on-stale` | Exit with code 1 if any source has changed, is unknown, or errored (requires `--state`) | -| `--virtual-env` | Read previous state from the specified virtual environment instead of direct state | | `--json` | Output as JSON instead of human-readable text | | `--json-output` | Write JSON output to a file path (also prints text to stdout unless `--json` is set) | | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | @@ -111,14 +108,6 @@ Summary: observed=0 changed=1 unchanged=1 tolerated=1 unknown=0 errors=0 | `unknown` | No freshness config and adapter metadata unavailable, or no previous state to compare against | | `error` | Observation failed (e.g. source table does not exist, query error) | -### Virtual environment state - -To compare against state stored in a virtual environment instead of direct mode state: - -```bash -sqb freshness --state --virtual-env pr_123 -``` - ## CI integration Use `--fail-on-error` to fail the pipeline if any source can't be observed: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/init.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/init.md index dca0fc030f..51fe4c115c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/init.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/init.md @@ -1,12 +1,10 @@ -# init +# sqb init > Scaffold a new SQLBuild project. -Online: https://docs.sqlbuild.com/cli/init - -# sqb init +Online: https://sqlbuild.com/docs/cli/init/ Creates a new SQLBuild project with a minimal directory structure and configuration files. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md index f9c4d58bf1..4146f7ce83 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md @@ -1,27 +1,24 @@ -# janitor +# sqb janitor > Archive and then delete stale warehouse relations. -Online: https://docs.sqlbuild.com/cli/janitor +Online: https://sqlbuild.com/docs/cli/janitor/ ## Contents - Usage - Flags -- Direct-mode lifecycle +- Lifecycle - Relation age - Audit record - Restoring an archive - Configuration - Examples - State history pruning -- Virtual environments - Safety -# sqb janitor - Finds stale warehouse relations that are no longer part of the project, archives them, and deletes archives once they expire. Requires `janitor.enabled = true` in `sqlbuild_project.toml`. ## Usage @@ -38,9 +35,9 @@ sqb --project-dir janitor [flags] | `--retention-days` | Override the configured retention period (days) before a stale relation is archived | | `--direct-state-history-versions` | Override how many state-history versions are kept per identity | -## Direct-mode lifecycle +## Lifecycle -In direct mode, the janitor never drops a live relation directly. Each stale relation goes through two stages: +The janitor never drops a live relation directly. Each stale relation goes through two stages: 1. **Archive.** Once a relation has been stale for `retention_days`, the janitor renames it in place, in the same schema, to: @@ -103,7 +100,7 @@ delete_tracked_only = true exclude_patterns = ["audit_*", "tmp_*"] ``` -When `retention_days` is not set, direct mode uses `14` and virtual-environment mode uses `30`. See [Project Configuration](../concepts/project-configuration.md#janitor) for every janitor setting. +When `retention_days` is not set, it defaults to `14`. See [Project Configuration](../concepts/project-configuration.md#janitor) for every janitor setting. ## Examples @@ -122,10 +119,6 @@ sqb janitor --retention-days 7 In addition to archiving stale relations, the janitor prunes old rows from `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness`, keeping the latest `direct_state_history_versions` rows per identity (default `20`). This is configured separately from relation retention. -## Virtual environments - -In virtual-environment mode, the janitor also cleans up expired environments, old checkpoints and unreferenced physical versions. See [Virtual environments: janitor](../concepts/virtual-environments/janitor.md). Direct-mode archiving doesn't apply there. In both modes, relations whose names differ only by case from another relation in the same schema are reported and never dropped, and exclude patterns match regardless of case. - ## Safety -The janitor shows every planned archive and delete, with exact names, ages and expiry, then asks you to type an exact confirmation phrase. Use `--auto-approve` only in automation, or when you're certain. +The janitor shows every planned archive and delete, with exact names, ages and expiry, then asks you to type an exact confirmation phrase. Relations whose names differ only by case from another relation in the same schema are reported and never dropped, and exclude patterns match regardless of case. Use `--auto-approve` only in automation, or when you're certain. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/lineage.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/lineage.md index 4ec2ed8766..86fd57c2b6 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/lineage.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/lineage.md @@ -4,7 +4,7 @@ > Explore model and column-level dependency graphs from the command line. -Online: https://docs.sqlbuild.com/cli/lineage +Online: https://sqlbuild.com/docs/cli/lineage/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md index 99cc03766e..28cd952723 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/load.md @@ -1,12 +1,10 @@ -# load +# sqb load > Load managed sources into the warehouse. -Online: https://docs.sqlbuild.com/cli/load - -# sqb load +Online: https://sqlbuild.com/docs/cli/load/ Runs source loader functions and writes data into their target tables using the configured write strategy. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md index d950316ac4..52baa3731d 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md @@ -1,12 +1,10 @@ -# plan +# sqb plan > Preview what SQLBuild will do before executing. -Online: https://docs.sqlbuild.com/cli/plan - -# sqb plan +Online: https://sqlbuild.com/docs/cli/plan/ Shows the execution plan without making any changes. Useful for inspecting change detection, backfill policies, and selector scope before building. @@ -22,7 +20,6 @@ sqb --project-dir plan [flags] |------|-------------| | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | | `--no-cache` | Bypass the reusable compile-analysis cache for this invocation | -| `--changes-only` | Virtual mode only: show work not already bound to its expected VDE version | | `--no-python` | Exclude read-side Python tasks and assets from the plan | | `--defer-to` | Resolve unselected model references against another target | | `--as` | Compile for another target and show its plan, including [model migration](../concepts/models/migrations.md) decisions, using the current connection. Inspection only. | diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/playground.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/playground.md index 5cce645ea1..326d8cc75c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/playground.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/playground.md @@ -1,12 +1,10 @@ -# playground +# sqb playground > Create a self-contained SQLBuild project to explore locally. -Online: https://docs.sqlbuild.com/cli/playground - -# sqb playground +Online: https://sqlbuild.com/docs/cli/playground/ Creates a self-contained waffle shop project with DuckDB. No warehouse credentials, no git clone, no external data - just a working project you can compile, build, test, and explore immediately. @@ -25,7 +23,6 @@ The positional argument is the directory to create (default `sqlbuild-playground | `waffle_shop` | Default. DuckDB-backed project with models, tests, scenarios, and macros. | | `dagster` | Waffle shop project plus a `dagster/` directory with a ready-to-run `definitions.py`. | | `rivers` | Waffle shop project plus a `rivers_pipeline/` directory with a Rivers repository definition. | -| `virtual` | Waffle shop with virtual environments enabled, a local DuckDB state store, loaders, and the full virtual lifecycle (build, promote, rollback). | | `python_nodes` | A small DuckDB project demonstrating [Python nodes](../concepts/python-nodes/overview.md): a task feeding a loader, a model read by a Python asset through `ctx.relation(model(...))`, a soft-skip fan-in, `materialized=False`, and a Python check. | ## What it creates @@ -72,14 +69,6 @@ cd sqlbuild-playground uv pip install 'sqlbuild[rivers]' uv run rivers dev rivers_pipeline.definitions -# With virtual environments -sqb playground --template virtual -cd sqlbuild-playground -sqb state init -sqb build -sqb build --virtual-env pr -sqb promote --from pr --to dev - # With Python nodes sqb playground --template python_nodes cd sqlbuild-playground diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/promote.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/promote.md deleted file mode 100644 index 67ee037a54..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/promote.md +++ /dev/null @@ -1,48 +0,0 @@ - - -# promote - -> Promote VDE refs from one virtual environment to another. - -Online: https://docs.sqlbuild.com/cli/promote - -# sqb promote - -Promotes model version refs from a source VDE to a target VDE. No models are rebuilt - promotion swaps pointers and refreshes logical views. Virtual mode only. - -## Usage - -```bash -sqb promote --from --to [flags] -``` - -## Flags - -| Flag | Description | -|------|-------------| -| `--from` | Source VDE name (required) | -| `--to` | Target VDE name (required) | -| `--select`, `-s` | Select specific models to promote | -| `--exclude` | Exclude specific models | -| `--allow-partial-promotion` | Accept a working target VDE when partial promotion leaves downstream stale | -| `--include-stale-upstreams` | Add required stale upstream refs to the promotion scope | -| `--verbose`, `-v` | Show uncapped model sets | -| `--var` | Set project variables | - -## Examples - -```bash -# Promote all models from pr_123 to dev -sqb promote --from pr_123 --to dev - -# Promote specific models -sqb promote --from pr_123 --to dev --select fact_orders - -# Partial promotion with working target accepted -sqb promote --from pr_123 --to dev --select fact_orders --allow-partial-promotion - -# Include stale upstreams for coherent partial promotion -sqb promote --from pr_123 --to dev --select fact_orders --include-stale-upstreams -``` - -See [Promotion](../concepts/virtual-environments/promotion.md) for details on promotion behavior, source requirements, and guards. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md index 4ecac29e9f..617f80f46e 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/query.md @@ -1,12 +1,10 @@ -# query +# sqb query > Run ad hoc SQL queries against the project database. -Online: https://docs.sqlbuild.com/cli/query - -# sqb query +Online: https://sqlbuild.com/docs/cli/query/ Execute ad hoc SQL against the active project connection. Useful for inspecting data, debugging models, or running one-off queries without leaving the SQLBuild CLI. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/reconcile.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/reconcile.md deleted file mode 100644 index e059f710e7..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/reconcile.md +++ /dev/null @@ -1,66 +0,0 @@ - - -# reconcile - -> Diagnose and repair drift between virtual state and warehouse. - -Online: https://docs.sqlbuild.com/cli/reconcile - -# sqb reconcile - -Detects and repairs inconsistencies between the virtual state store and the warehouse. Virtual mode only. - -## Usage - -```bash -sqb reconcile [--virtual-env ] [--model ] -sqb reconcile repair-view --virtual-env --model -sqb reconcile attach --virtual-env --model --physical-relation -``` - -## Subcommands - -### (default) - -Run a diagnostic report without changing anything: - -```bash -sqb reconcile --virtual-env dev -``` - -### repair-view - -Recreate a logical VDE view from trusted state refs: - -```bash -sqb reconcile repair-view --virtual-env dev --model fact_orders -``` - -No confirmation needed. Idempotent (`CREATE OR REPLACE VIEW`). - -### attach - -Rebind a VDE model ref to a different tracked physical relation: - -```bash -sqb reconcile attach --virtual-env dev --model fact_orders \ - --physical-relation dev__sqb_physical.fact_orders__v_8f3a9c12 -``` - -Requires confirmation by default. - -## Examples - -```bash -# Diagnose issues -sqb reconcile --virtual-env dev - -# Fix a missing view -sqb reconcile repair-view --virtual-env dev --model fact_orders - -# Rebind a model to a different physical version -sqb reconcile attach --virtual-env dev --model fact_orders \ - --physical-relation dev__sqb_physical.fact_orders__v_71d0e4ab -``` - -See [Reconcile](../concepts/virtual-environments/reconcile.md) for details on guards and when to use each subcommand. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rollback.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rollback.md deleted file mode 100644 index 35e21054c8..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rollback.md +++ /dev/null @@ -1,48 +0,0 @@ - - -# rollback - -> Roll back a VDE to a prior finalized checkpoint. - -Online: https://docs.sqlbuild.com/cli/rollback - -# sqb rollback - -Restores a virtual environment to a prior finalized checkpoint by rebinding its refs and refreshing logical views. Virtual mode only. - -## Usage - -```bash -sqb rollback [flags] -``` - -## Flags - -| Flag | Description | -|------|-------------| -| `--virtual-env` | Target VDE name (defaults to active target name) | -| `--checkpoint-id` | Restore a specific checkpoint instead of the previous one | -| `--select`, `-s` | Roll back specific models only | -| `--exclude` | Exclude specific models | -| `--allow-partial-rollback` | Accept a working VDE when partial rollback leaves downstream stale | -| `--include-stale-upstreams` | Add required upstream refs from the checkpoint | -| `--verbose`, `-v` | Show uncapped model sets | -| `--var` | Set project variables | - -## Examples - -```bash -# Roll back to the previous finalized checkpoint -sqb rollback - -# Roll back a specific VDE -sqb rollback --virtual-env pr_123 - -# Roll back to a specific checkpoint -sqb rollback --checkpoint-id - -# Partial rollback -sqb rollback --select fact_orders --allow-partial-rollback -``` - -See [Rollback](../concepts/virtual-environments/rollback.md) for details on checkpoints, rollback behavior, and guards. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rules.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rules.md index fafdb2cb7c..6678e5438f 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rules.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rules.md @@ -1,12 +1,10 @@ -# rules +# sqb rules > List, inspect, or run compiler-integrated Rules and generate project guidance. -Online: https://docs.sqlbuild.com/cli/rules - -# sqb rules +Online: https://sqlbuild.com/docs/cli/rules/ Configured Rules run automatically through `sqb compile` and build planning. The `rules` command provides focused catalogue, inspection, execution, and guidance operations. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md index e9fec8b062..1e55f722d0 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scenario.md @@ -1,10 +1,10 @@ -# scenario +# sqb scenario > Run end-to-end scenario tests against the warehouse or locally with DuckDB. -Online: https://docs.sqlbuild.com/cli/scenario +Online: https://sqlbuild.com/docs/cli/scenario/ ## Contents @@ -12,8 +12,6 @@ Online: https://docs.sqlbuild.com/cli/scenario - sqb scenario capture - Runtime artifacts -# sqb scenario - Run end-to-end scenario tests. Scenarios materialize fixture inputs as physical relations, build the real project graph against them, and evaluate expected outputs and assertions. See [Scenarios](../concepts/scenarios.md) for concepts and authoring details. ## sqb scenario test diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scope.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scope.md index c6fcebc506..929a65c0d9 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scope.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/scope.md @@ -4,7 +4,7 @@ > Inspect declaration visibility, usage, placement, and move impact offline. -Online: https://docs.sqlbuild.com/cli/scope +Online: https://sqlbuild.com/docs/cli/scope/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md index ae01a8a65c..43947809f0 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/seed.md @@ -1,12 +1,10 @@ -# seed +# sqb seed > Load seed CSV files into the warehouse. -Online: https://docs.sqlbuild.com/cli/seed - -# sqb seed +Online: https://sqlbuild.com/docs/cli/seed/ Loads seed CSV files into the warehouse as tables. Seeds are fully replaced on every run. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/skills.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/skills.md index 6ed7315808..893ec2d700 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/skills.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/skills.md @@ -1,12 +1,10 @@ -# skills +# sqb skills > Install SQLBuild skill files for AI coding agents. -Online: https://docs.sqlbuild.com/cli/skills - -# sqb skills +Online: https://sqlbuild.com/docs/cli/skills/ Install or update SQLBuild skill files so AI coding agents (Claude Code, OpenCode, Cursor, etc.) understand your project's framework, syntax, and conventions. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/state.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/state.md deleted file mode 100644 index f0cfeea979..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/state.md +++ /dev/null @@ -1,128 +0,0 @@ - - -# state - -> Manage the virtual mode state store. - -Online: https://docs.sqlbuild.com/cli/state - -## Contents - -- Usage -- Subcommands - -# sqb state - -Manages the virtual mode state store lifecycle, locks, and checkpoints. Requires `virtual_environments = true`. - -## Usage - -```bash -sqb state [flags] -``` - -## Subcommands - -### init - -Create state tables in the configured schema. - -```bash -sqb state init -``` - -### migrate - -Back up the current state schema and re-initialize state tables. Creates a backup schema (e.g. `sqlbuild_state__backup_`) before re-initializing. - -```bash -sqb state migrate -``` - -### rollback - -Restore state from a backup. - -```bash -# Restore from the latest backup -sqb state rollback - -# Restore from a specific backup -sqb state rollback --backup-id -``` - -### reset - -Drop all state tables. Requires `allow_reset = true` in the target state config and `--auto-approve` on the command line. - -```bash -sqb state reset --auto-approve -``` - -### adopt - -Convert existing stateless warehouse relations into versioned physical relations with VDE views. Interactive only - requires typed confirmation. - -```bash -sqb state adopt [--allow-copy] -``` - -| Flag | Description | -|------|-------------| -| `--allow-copy` | Allow CTAS copy fallback when native rename is not available | - -See [Adopt and Detach](../concepts/virtual-environments/adopt-detach.md) for details. - -### detach - -Collapse a VDE back into stateless warehouse relations. Interactive only - requires typed confirmation. The VDE must be finalized. - -```bash -sqb state detach [--allow-copy] -``` - -| Flag | Description | -|------|-------------| -| `--allow-copy` | Allow CTAS copy fallback when native rename is not available | - -### locks - -List active locks in the state store. - -```bash -sqb state locks -``` - -### locks clear - -Clear a stuck lock. - -```bash -sqb state locks clear --force -``` - -Only clear locks when you are certain the holding operation is no longer running. - -### checkpoints list - -List finalized checkpoints for a VDE. - -```bash -sqb state checkpoints list [--virtual-env ] -``` - -### checkpoints show - -Show the model refs stored in a checkpoint. - -```bash -sqb state checkpoints show [--virtual-env ] -``` - -### checkpoints diff - -Diff current VDE refs against a checkpoint. - -```bash -sqb state checkpoints diff [--virtual-env ] -``` diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md index e8cc148409..23b6ca4bca 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md @@ -1,12 +1,10 @@ -# test +# sqb test > Run SQL unit tests and multi-model tests in isolation. -Online: https://docs.sqlbuild.com/cli/test - -# sqb test +Online: https://sqlbuild.com/docs/cli/test/ Runs SQL unit tests, independently reported parameterized cases, and multi-model tests without building models. Useful for validating test logic independently. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md index 5a0c190078..9c7f1f0bd0 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters.md @@ -4,28 +4,31 @@ > Supported database engines and their connection configuration. -Online: https://docs.sqlbuild.com/concepts/adapters +Online: https://sqlbuild.com/docs/concepts/adapters/ ## Contents - Custom adapters -SQLBuild uses adapters to connect to different database engines. +SQLBuild uses adapters to connect to different database engines. Snowflake is the main target. + +- **Supported:** ready for production use. +- **Beta:** builds, tests and plans work, but with less production use so far. Expect rough edges, and please [report issues](https://github.com/chio-labs/sqlbuild/issues). | Adapter | Status | Install | |---------|--------|---------| +| [Snowflake](adapters/snowflake.md) | Supported | `sqlbuild[snowflake]` | | [DuckDB](adapters/duckdb.md) | Supported | included by default | | [MotherDuck](adapters/motherduck.md) | Supported | included by default (uses DuckDB) | -| [Snowflake](adapters/snowflake.md) | Supported | `sqlbuild[snowflake]` | -| [BigQuery](adapters/bigquery.md) | Supported | `sqlbuild[bigquery]` | -| [Databricks](adapters/databricks.md) | Supported | `sqlbuild[databricks]` | | [PostgreSQL](adapters/postgres.md) | Supported | `sqlbuild[postgres]` | -| [SQL Server](adapters/sqlserver.md) | Supported | `sqlbuild[sqlserver]` | -| ClickHouse | Coming soon | | -| Redshift | Coming soon | | -| Trino | Coming soon | | -| Spark | Coming soon | | -| Athena | Coming soon | | +| [BigQuery](adapters/bigquery.md) | Beta | `sqlbuild[bigquery]` | +| [Databricks](adapters/databricks.md) | Beta | `sqlbuild[databricks]` | +| [SQL Server](adapters/sqlserver.md) | Beta | `sqlbuild[sqlserver]` | +| ClickHouse | Planned | | +| Redshift | Planned | | +| Trino | Planned | | +| Spark | Planned | | +| Athena | Planned | | Set the adapter in `sqlbuild_project.toml`: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/bigquery.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/bigquery.md index 50b5cf7f9e..8d0141e523 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/bigquery.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/bigquery.md @@ -4,7 +4,9 @@ > BigQuery adapter configuration for SQLBuild. -Online: https://docs.sqlbuild.com/concepts/adapters/bigquery +Online: https://sqlbuild.com/docs/concepts/adapters/bigquery/ + +The BigQuery adapter is in beta: builds, tests and plans work, but it has had less production use so far. Please [report issues](https://github.com/chio-labs/sqlbuild/issues). BigQuery requires the optional `google-cloud-bigquery` dependency: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/databricks.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/databricks.md index e21871b989..61748c1a68 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/databricks.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/databricks.md @@ -4,7 +4,9 @@ > Databricks adapter configuration for SQLBuild. -Online: https://docs.sqlbuild.com/concepts/adapters/databricks +Online: https://sqlbuild.com/docs/concepts/adapters/databricks/ + +The Databricks adapter is in beta: builds, tests and plans work, but it has had less production use so far. Please [report issues](https://github.com/chio-labs/sqlbuild/issues). Databricks requires the optional `databricks-sql-connector` dependency: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/duckdb.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/duckdb.md index 988f8bddb6..51a57dccb7 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/duckdb.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/duckdb.md @@ -4,7 +4,7 @@ > DuckDB adapter configuration for SQLBuild. -Online: https://docs.sqlbuild.com/concepts/adapters/duckdb +Online: https://sqlbuild.com/docs/concepts/adapters/duckdb/ DuckDB is included as a core dependency. No extra installation needed. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/motherduck.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/motherduck.md index bc87e4f571..d54f46af87 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/motherduck.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/motherduck.md @@ -4,7 +4,7 @@ > MotherDuck adapter configuration for SQLBuild. -Online: https://docs.sqlbuild.com/concepts/adapters/motherduck +Online: https://sqlbuild.com/docs/concepts/adapters/motherduck/ MotherDuck uses DuckDB's built-in `md:` connection support. No extra installation needed beyond the core DuckDB dependency. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/postgres.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/postgres.md index 6f260044d3..41ff1723bf 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/postgres.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/postgres.md @@ -4,7 +4,7 @@ > PostgreSQL adapter configuration for SQLBuild. -Online: https://docs.sqlbuild.com/concepts/adapters/postgres +Online: https://sqlbuild.com/docs/concepts/adapters/postgres/ PostgreSQL requires the optional `psycopg` dependency: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md index 5d910b454b..9bd6b73918 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md @@ -4,7 +4,7 @@ > Snowflake adapter configuration for SQLBuild. -Online: https://docs.sqlbuild.com/concepts/adapters/snowflake +Online: https://sqlbuild.com/docs/concepts/adapters/snowflake/ Snowflake requires the optional `snowflake-connector-python` dependency: @@ -66,3 +66,27 @@ connection = "warehouse" database = "DEV_DB" schema = "dev" ``` + +## Cost estimates + +Snowflake builds show a compact per-run busy-compute estimate. SQLBuild attributes visible +overlapping query intervals fairly across active queries, converts attributed seconds using the +warehouse-size credit rate, and estimates USD from the configured rate: + +```toml +[cost] +usd_per_credit = 3.00 +``` + +The default is `3.00` USD per credit and is visibly marked as a default. Set it to your Snowflake +contract rate. Use `sqb cost`, `sqb cost latest`, `sqb cost `, or +`sqb cost history --since 7d` to inspect persisted records. `--json` and `--json-output PATH` +provide a versioned, decimal-safe output contract. Pending detail records are refreshed from +Snowflake when inspected again. + +These values are attributed compute credits and estimated cost, not Snowflake-billed credits or +invoice reconciliation. The estimate uses only query history visible to the executing role and does +not reconstruct invisible concurrent work, warehouse resume or idle tail, the 60-second minimum, +cloud-services credits, contract adjustments, or multi-cluster billing. Run metadata and query IDs +are stored under `target/executions//`; that statement ledger stores only an SQL digest, +not SQL text. Executed SQL artifacts are stored separately under the sensitive `target/run/` tree. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/sqlserver.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/sqlserver.md index f46232bb3a..3bba118653 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/sqlserver.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/sqlserver.md @@ -4,7 +4,9 @@ > Microsoft SQL Server adapter configuration for SQLBuild. -Online: https://docs.sqlbuild.com/concepts/adapters/sqlserver +Online: https://sqlbuild.com/docs/concepts/adapters/sqlserver/ + +The SQL Server adapter is in beta: builds, tests and plans work, but it has had less production use so far. Please [report issues](https://github.com/chio-labs/sqlbuild/issues). SQL Server requires the optional `pymssql` dependency: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md index 3c32bbc01c..39c69e4258 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/audits.md @@ -4,7 +4,7 @@ > Violation and measurement checks that gate data and record quality outcomes. -Online: https://docs.sqlbuild.com/concepts/audits +Online: https://sqlbuild.com/docs/concepts/audits/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/column-lineage.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/column-lineage.md index 11d7e2a6a6..592d0cb948 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/column-lineage.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/column-lineage.md @@ -4,7 +4,7 @@ > Trace individual columns through your SQL pipeline - understand where data comes from and where it goes. -Online: https://docs.sqlbuild.com/concepts/column-lineage +Online: https://sqlbuild.com/docs/concepts/column-lineage/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/constants.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/constants.md index 0221dddd05..7a89f9f1c9 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/constants.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/constants.md @@ -4,7 +4,7 @@ > Define reusable compiler-validated values and reference them safely from SQL. -Online: https://docs.sqlbuild.com/concepts/constants +Online: https://sqlbuild.com/docs/concepts/constants/ Constants give a name to a value used in SQL. SQLBuild validates the value during compilation and asks the active adapter to render it safely for its SQL dialect. Constant values are data, never raw diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/constants/collections-and-rendering.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/constants/collections-and-rendering.md index cde2d8aac4..2c76f2d1d9 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/constants/collections-and-rendering.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/constants/collections-and-rendering.md @@ -4,7 +4,7 @@ > Define list, set, and object constants and control their adapter-specific SQL rendering. -Online: https://docs.sqlbuild.com/concepts/constants/collections-and-rendering +Online: https://sqlbuild.com/docs/concepts/constants/collections-and-rendering/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/adding-sqlbuild-models.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/adding-sqlbuild-models.md index d9119a290a..e2e1624208 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/adding-sqlbuild-models.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/adding-sqlbuild-models.md @@ -4,7 +4,7 @@ > Grow into SQLBuild's own models, tests, audits, and scenarios downstream of your dbt project. -Online: https://docs.sqlbuild.com/concepts/dbt-compatibility/adding-sqlbuild-models +Online: https://sqlbuild.com/docs/concepts/dbt-compatibility/adding-sqlbuild-models/ The dbt compatibility bridge does not require SQLBuild models. You can optionally add SQLBuild models, tests, audits, and scenarios downstream of dbt outputs. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/overview.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/overview.md index ba4e816ded..faad954af2 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/overview.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/overview.md @@ -4,7 +4,7 @@ > Run dbt and SQLBuild side by side with coordinated selection and SQLBuild models downstream. -Online: https://docs.sqlbuild.com/concepts/dbt-compatibility/overview +Online: https://sqlbuild.com/docs/concepts/dbt-compatibility/overview/ ## Contents @@ -17,7 +17,7 @@ Online: https://docs.sqlbuild.com/concepts/dbt-compatibility/overview Use the dbt compatibility bridge to coordinate dbt selections with SQLBuild-owned models downstream of their outputs. -SQLBuild reads the dbt manifest and drives the `dbt` CLI as a subprocess. dbt remains responsible for compiling and executing dbt-owned models; SQLBuild statically analyzes and executes only SQLBuild-owned models downstream. `sqb dbt` runs in direct mode, so change-aware execution and virtual environments are not supported by the bridge. +SQLBuild reads the dbt manifest and drives the `dbt` CLI as a subprocess. dbt remains responsible for compiling and executing dbt-owned models; SQLBuild statically analyzes and executes only SQLBuild-owned models downstream. Change-aware execution is not supported by the bridge. ## Start with your existing dbt project diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/selection.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/selection.md index 5fb93f5448..98cb2dafc2 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/selection.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/dbt-compatibility/selection.md @@ -4,7 +4,7 @@ > How select and exclude flags route work across the dbt and SQLBuild graphs. -Online: https://docs.sqlbuild.com/concepts/dbt-compatibility/selection +Online: https://sqlbuild.com/docs/concepts/dbt-compatibility/selection/ The `sqb dbt` commands use `--select` and `--exclude` to scope what runs. Selectors work across both dbt and SQLBuild, with the system determining which side owns each selector and how to route work. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md index 5684b65381..76dcf16555 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes.md @@ -4,7 +4,7 @@ > Limit enums, constants, and macros to the parts of a project that use them. -Online: https://docs.sqlbuild.com/concepts/declaration-scopes +Online: https://sqlbuild.com/docs/concepts/declaration-scopes/ Most projects can keep enums, constants, and macros in their ordinary top-level directories. Those declarations are available throughout the project. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/explorer.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/explorer.md index bdcd7c657f..b30beb77ac 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/explorer.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/explorer.md @@ -4,7 +4,7 @@ > Inspect visibility, explain resolution, browse declarations, and preview moves offline. -Online: https://docs.sqlbuild.com/concepts/declaration-scopes/explorer +Online: https://sqlbuild.com/docs/concepts/declaration-scopes/explorer/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/placement.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/placement.md index 237daf3c38..da199664f1 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/placement.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/placement.md @@ -4,7 +4,7 @@ > Choose the narrowest folder that contains every real use. -Online: https://docs.sqlbuild.com/concepts/declaration-scopes/placement +Online: https://sqlbuild.com/docs/concepts/declaration-scopes/placement/ Start with the ordinary project-wide directories unless you have a reason to limit access: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/visibility.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/visibility.md index 3ce53b7c77..dfe3fec1ef 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/visibility.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/declaration-scopes/visibility.md @@ -4,7 +4,7 @@ > See which enums, constants, and macros are available to each SQL file. -Online: https://docs.sqlbuild.com/concepts/declaration-scopes/visibility +Online: https://sqlbuild.com/docs/concepts/declaration-scopes/visibility/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/diff.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/diff.md index 45448fde82..094d42f7ee 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/diff.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/diff.md @@ -2,9 +2,9 @@ # Data Diffs -> Compare schemas and data between targets or virtual environments to validate changes before promotion. +> Compare schemas and data between targets to validate changes before they reach production. -Online: https://docs.sqlbuild.com/concepts/diff +Online: https://sqlbuild.com/docs/concepts/diff/ ## Contents @@ -18,18 +18,14 @@ Online: https://docs.sqlbuild.com/concepts/diff - Structured output - Invocation safety limits - Selectors +- Compare raw queries - Exit codes -SQLBuild can compare schemas and row-level data between two build contexts. This lets you validate that changes produce the expected results before promoting them. +SQLBuild can compare schemas and row-level data between two targets. This lets you validate that changes produce the expected results before they reach production. -`sqb diff FROM:TO` compares: +`sqb diff FROM:TO` compares two targets, for example `prod:dev`. -- **two targets** (e.g. `prod:dev`) in direct mode, or -- **two virtual environments** (VDEs) when [virtual environments](virtual-environments.md) are enabled. - -The mechanics below are identical for both; only what `FROM` and `TO` refer to changes. - -In direct mode, `FROM` and `TO` resolve the authoritative database/schema namespaces to compare. +`FROM` and `TO` resolve the authoritative database/schema namespaces to compare. SQLBuild uses the `TO` target's named connection for the complete comparison and accesses both namespaces through fully qualified relations. It does not resolve or open the `FROM` target's connection. The `TO` connection must therefore be able to read both namespaces; missing credentials @@ -133,7 +129,21 @@ side. ## Row matching -Rows are matched between the two sides using the model's `unique_key`. Models without a `unique_key` can use schema-only diff but cannot run full or bounded row comparisons. +Rows are matched between the two sides using a row key. By default, row identity comes from each model's `unique_key`. Repeated `--key` arguments override +that configuration for one invocation: + +```bash +sqb diff prod:dev --full --select order_lines \ + --key order_id \ + --key line_id +``` + +For a model without a stable row key, opt in to exact full-row multiset comparison with +`--unkeyed`. Duplicate rows retain their multiplicity; SQLBuild never guesses a key. + +```bash +sqb diff prod:dev --full --select daily_order_totals --unkeyed +``` The diff output categorises rows as: - **Equal** - same key, same values on both sides @@ -237,6 +247,141 @@ sqb diff prod:dev --schema-only --select path:models/marts sqb diff prod:dev --full --select tag:acceptance ``` +## Compare raw queries + +Query mode omits `FROM:TO` because each query identifies its own input. Supply each side inline or +from a UTF-8 file, and choose keyed or unkeyed comparison explicitly: + +```bash +sqb diff \ + --left-query 'SELECT order_id, amount_cents FROM analytics.orders_before' \ + --right-query 'SELECT order_id, amount_cents FROM analytics.orders_after' \ + --left-label before \ + --right-label after \ + --key order_id +``` + +```bash +sqb diff \ + --left-query-file queries/orders_before.sql \ + --right-query-file queries/orders_after.sql \ + --key account_id \ + --key order_id +``` + +Raw inputs must contain exactly one read-only `SELECT`, `WITH`, or `VALUES` expression. SQLBuild +does not expand `ref()` or other SQLBuild templates in these inputs. The active target connection +executes both sides; use `--target NAME` to select another configured connection. + +Query mode performs a full comparison when no comparison mode flag is present. `--schema-only` is +also available. `--bounded` is model-specific; filter both raw queries directly when a bounded +query comparison is needed. + +Raw-query discovery loads only project configuration, target configuration, and the selected +adapter. It does not import unrelated project sinks, hooks, loaders, or providers. + +### Sampling, tolerances, and exclusions + +Keyed query comparisons support deterministic union-key sampling and the same bounded mismatch +examples as model comparisons: + +```bash +sqb diff \ + --left-query-file queries/orders_before.sql \ + --right-query-file queries/orders_after.sql \ + --key order_id \ + --sample-rows 50000 \ + --sample-seed 7 +``` + +Use repeated invocation overrides where needed: + +```bash +sqb diff \ + --left-query-file queries/orders_before.sql \ + --right-query-file queries/orders_after.sql \ + --key order_id \ + --exclude-column loaded_at \ + --tolerance amount_cents:absolute=1 \ + --tolerance conversion_rate:relative=0.001 +``` + +`--exhaustive` disables inherited or requested sampling. Unkeyed comparison is always exhaustive +and exact, so it does not accept sampling or numeric tolerances. + +### Bounded example values + +Comparison remains exact regardless of how examples are rendered. By default, long values are +limited to 160 characters and show context around the first differing character. Every shortened +value is marked as truncated and includes its original length. + +Control evidence independently from the number of collected examples: + +```bash +# Increase the displayed context without changing comparison scope. +sqb diff --left-query-file before.sql --right-query-file after.sql \ + --key order_id --max-value-length 500 + +# Retain keys and aggregate counts but suppress example values. +sqb diff --left-query-file before.sql --right-query-file after.sql \ + --key order_id --no-example-values + +# Explicitly opt in to complete values. --verbose alone remains bounded. +sqb diff --left-query-file before.sql --right-query-file after.sql \ + --key order_id --full-example-values +``` + +Ordinary keys are kept complete. Exceptionally long keys are shortened with an explicit length and +stable digest so an operator can still identify them without flooding terminal output. + +### Stable query results and cleanup + +SQLBuild materializes each raw query once in the selected target's default schema before comparing +it. Artifacts use a reserved run-owned name and are dropped immediately when the command finishes, +including after comparison errors. + +Each successfully created artifact publishes an immutable `query_diff_artifact` fingerprint. The +fingerprint is ownership evidence for crash recovery only: running and comparing the queries never +reads historical fingerprints, and fingerprints do not represent planned, running, complete, or +cleaned workflow states. + +If a process stops before immediate cleanup, the next query diff and `sqb janitor` can remove an +expired artifact only when its exact physical identity and reserved name match its ownership +fingerprint. A matching name without ownership evidence is reported for manual investigation and +is not deleted automatically. + +The recovery TTL defaults to 24 hours and accepts SQLBuild fixed-duration syntax: + +```toml +[diff] +query_artifact_ttl = "24h" +``` + +The TTL is a crash-recovery backstop, not the normal cleanup path. Historical ownership inspection +problems are reported but do not prevent the current queries from running. Failure to publish +ownership for a newly created artifact, or failure to remove a current-run artifact, is an execution +error. + +### Query output and exit codes + +Use `--json-output PATH`, or `--json` to emit one JSON document to stdout while all +lifecycle progress remains on stderr. Query results include `input_kind = "query"`, named left and +right inputs, `query_comparisons`, sparse nonzero `column_mismatches`, lifecycle timings, and +explicit example truncation metadata. Existing `models` output remains available for compatibility. + +Raw-query automation distinguishes complete findings from incomplete evidence and execution +failure: + +| Exit | Outcome | +|---:|---| +| `0` | Complete comparison with no differences | +| `1` | Complete comparison with findings | +| `2` | Safety, schema, or key preflight prevented complete value evidence | +| `3` | Setup, connection, materialization, comparison execution, ownership, or cleanup failed | + +Human output and JSON use the same outcome. Lifecycle messages include elapsed connection, +reconciliation, inspection, materialization, comparison, cleanup, and total durations. + ## Exit codes -`sqb diff` returns exit code `0` when all selected models have no differences, and `1` when any model has schema or row differences. This makes it usable in CI pipelines as a validation gate. +In model mode, `sqb diff` returns exit code `0` when all selected models have no differences, and `1` when any model has schema or row differences. Raw-query mode uses the exit codes in [Query output and exit codes](#query-output-and-exit-codes). This makes it usable in CI pipelines as a validation gate. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums-and-constants.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums-and-constants.md index 2a40b4da12..ba4b4207a6 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums-and-constants.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums-and-constants.md @@ -4,7 +4,7 @@ > Find the new focused guides for enums, constants, collections, contracts, and private values. -Online: https://docs.sqlbuild.com/concepts/enums-and-constants +Online: https://sqlbuild.com/docs/concepts/enums-and-constants/ Enums and constants now have separate guides so each feature is easier to find and learn. This page remains available for existing links and bookmarks but is not part of the main navigation. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums.md index 379f6f88fe..2aa4dd7789 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums.md @@ -4,7 +4,7 @@ > Define a fixed set of named string or integer values and use them safely in SQL. -Online: https://docs.sqlbuild.com/concepts/enums +Online: https://sqlbuild.com/docs/concepts/enums/ Enums give a name to a fixed set of allowed values. SQLBuild checks the enum and every member reference during compilation, then renders the selected value as a safe SQL literal. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums/model-contracts.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums/model-contracts.md index ee4d263134..066c988fdf 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums/model-contracts.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/enums/model-contracts.md @@ -4,7 +4,7 @@ > Use an enum as a portable model-column domain with generated accepted-value validation. -Online: https://docs.sqlbuild.com/concepts/enums/model-contracts +Online: https://sqlbuild.com/docs/concepts/enums/model-contracts/ An enum can describe the allowed domain of a model column as well as provide individual SQL literals. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/functions.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/functions.md index 4b8fb79242..6421a8ad5f 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/functions.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/functions.md @@ -4,7 +4,7 @@ > User-defined functions and table functions managed as part of your project. -Online: https://docs.sqlbuild.com/concepts/functions +Online: https://sqlbuild.com/docs/concepts/functions/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/incremental.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/incremental.md index 5e2211f92d..4af85a13c2 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/incremental.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/incremental.md @@ -4,7 +4,7 @@ > Cursor-based incremental strategies, microbatch execution, and backfill policies. -Online: https://docs.sqlbuild.com/concepts/incremental +Online: https://sqlbuild.com/docs/concepts/incremental/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/interpolation.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/interpolation.md index d34eca9790..f81dc680db 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/interpolation.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/interpolation.md @@ -4,7 +4,7 @@ > How SQLBuild processes variables, context, and dynamic content in SQL and config. -Online: https://docs.sqlbuild.com/concepts/interpolation +Online: https://sqlbuild.com/docs/concepts/interpolation/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros.md index b5c4227799..5135a22a7e 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros.md @@ -4,7 +4,7 @@ > Write Python functions that generate reusable SQL fragments at compile time. -Online: https://docs.sqlbuild.com/concepts/macros +Online: https://sqlbuild.com/docs/concepts/macros/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/composition-and-context.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/composition-and-context.md index 5b667d86f8..837c30fdcd 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/composition-and-context.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/macros/composition-and-context.md @@ -4,7 +4,7 @@ > Compose macros through Python, use compile context, and understand scoped imports. -Online: https://docs.sqlbuild.com/concepts/macros/composition-and-context +Online: https://sqlbuild.com/docs/concepts/macros/composition-and-context/ ## Contents @@ -137,3 +137,24 @@ def schema_qualified(ctx, table: str) -> str: Use context when generated SQL genuinely differs by adapter or target. Prefer ordinary parameters for values that the SQL caller should choose explicitly. + +### Constants and enums + +Python SQL macros receive the constants and enums visible to the SQL resource that calls them. Use +the typed mappings for Python control flow, and use the rendering methods when inserting a +declaration into generated SQL so quoting and collection syntax follow the active adapter: + +```python +def minimum_order_filter(ctx) -> str: + minimum = ctx.constants["minimum_order_value"] + if minimum is None: # The visible declaration explicitly has a NULL value. + return "TRUE" + return f"order_value >= {ctx.render_constant('minimum_order_value')}" + +def active_status_filter(ctx) -> str: + status = ctx.render_enum_member(enum_name="order_status", member_name="active") + return f"status = {status}" +``` + +Callers can still pass explicit `@const(...)` or `@enum(...)` values as macro arguments. Context +lookups are intended for policy owned by the macro; both forms use the caller's declaration scope. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/model-private-values.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/model-private-values.md index cea0bbd4a5..b654019d97 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/model-private-values.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/model-private-values.md @@ -4,7 +4,7 @@ > Keep enums and constants inside one model when no other resource should use them. -Online: https://docs.sqlbuild.com/concepts/model-private-values +Online: https://sqlbuild.com/docs/concepts/model-private-values/ Put an enum or constant directly in `MODEL()` when it belongs to one model and should not enter the project-wide namespace. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models.md index 6d136060f7..3154c504b8 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models.md @@ -4,7 +4,7 @@ > SQL model anatomy, references, dependencies, and the model documentation guide. -Online: https://docs.sqlbuild.com/concepts/models +Online: https://sqlbuild.com/docs/concepts/models/ A model is a SQL file that defines one transformation step and produces a table or view in the warehouse. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md index b6c4fd8161..5c524be8ff 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md @@ -4,7 +4,7 @@ > MODEL() header fields and SQL-analysis controls. -Online: https://docs.sqlbuild.com/concepts/models/configuration +Online: https://sqlbuild.com/docs/concepts/models/configuration/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/contracts.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/contracts.md index bd49cdf51c..755de65810 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/contracts.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/contracts.md @@ -4,7 +4,7 @@ > Validate required or exact model output schemas. -Online: https://docs.sqlbuild.com/concepts/models/contracts +Online: https://sqlbuild.com/docs/concepts/models/contracts/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks.md index e1055781f6..47788859f7 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks.md @@ -4,7 +4,7 @@ > Run SQL or Python lifecycle hooks around model materialization. -Online: https://docs.sqlbuild.com/concepts/models/hooks +Online: https://sqlbuild.com/docs/concepts/models/hooks/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/python.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/python.md index 92bf8b2198..2eff732396 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/python.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/python.md @@ -4,7 +4,7 @@ > Define Python lifecycle hooks with runtime context, providers, SQL access, and skips. -Online: https://docs.sqlbuild.com/concepts/models/hooks/python +Online: https://sqlbuild.com/docs/concepts/models/hooks/python/ Python hooks run model lifecycle logic that needs Python control flow, providers, warehouse queries, logging, or explicit skip decisions. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/sql.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/sql.md index a1c7ac417c..a1c878a35b 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/sql.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/hooks/sql.md @@ -4,7 +4,7 @@ > Define, parameterize, compile, and invoke reusable or inline SQL lifecycle hooks. -Online: https://docs.sqlbuild.com/concepts/models/hooks/sql +Online: https://sqlbuild.com/docs/concepts/models/hooks/sql/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md index 52aefa9e51..b4470341b9 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md @@ -4,7 +4,7 @@ > Choose how SQLBuild persists model output. -Online: https://docs.sqlbuild.com/concepts/models/materializations +Online: https://sqlbuild.com/docs/concepts/models/materializations/ The `materialized` field selects how a model becomes a warehouse relation. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md index 3ec30497a7..2192cff3cf 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md @@ -4,13 +4,11 @@ > Keep the history of an incremental or snapshot model when you rename it. -Online: https://docs.sqlbuild.com/concepts/models/migrations - -# Model migrations +Online: https://sqlbuild.com/docs/concepts/models/migrations/ Renaming an incremental or snapshot model normally means rebuilding it from scratch under the new name, which loses any history the source data can no longer reproduce. A model migration moves the existing relation's data to the new name instead, then continues building incrementally. -Migrations apply to incremental and snapshot models in direct mode. Virtual-environment builds reject `migrate_from`. +Migrations apply to incremental and snapshot models. ## Declaring a migration diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/schemas.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/schemas.md index 4d3b9b25e6..47f3758e15 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/schemas.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/schemas.md @@ -4,7 +4,7 @@ > Declare model columns inline or reuse canonical inherited schemas. -Online: https://docs.sqlbuild.com/concepts/models/schemas +Online: https://sqlbuild.com/docs/concepts/models/schemas/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/type-enforcement.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/type-enforcement.md index c86f72603f..348bbcfd67 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/type-enforcement.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/type-enforcement.md @@ -4,7 +4,7 @@ > Understand declared model types, static checks, and runtime casting. -Online: https://docs.sqlbuild.com/concepts/models/type-enforcement +Online: https://sqlbuild.com/docs/concepts/models/type-enforcement/ Type enforcement makes authored column types operational. It is a model feature as well as a related source feature, and it is separate from whether a model uses an exact contract. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability.md index 9771a28f5a..36bb9f197d 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability.md @@ -4,7 +4,7 @@ > Choose authoritative lifecycle facts, readable logs, or command-output records. -Online: https://docs.sqlbuild.com/concepts/observability +Online: https://sqlbuild.com/docs/concepts/observability/ SQLBuild keeps execution facts separate from human output. Do not parse terminal text to determine whether work started, completed, failed, skipped, or retried. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/execution-history.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/execution-history.md new file mode 100644 index 0000000000..1399e3390c --- /dev/null +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/execution-history.md @@ -0,0 +1,164 @@ + + +# Execution history + +> Store and query lifecycle events and run history from Python, with SQLite locally or PostgreSQL when deployed. + +Online: https://sqlbuild.com/docs/concepts/observability/execution-history/ + +## Contents + +- Public contracts +- Local SQLite +- Deployed PostgreSQL + +`LifecycleEventLogStorage` stores canonical immutable lifecycle facts. `RunStorage` serves an atomically +published, rebuildable run projection. Diagnostic and compute logs are never accepted by either +contract. The public types, filters, pages, records, errors, and helper functions are exported from +`sqlbuild.execution_history`. + +## Public contracts + +`LifecycleEventLogStorage` provides `append_event`, `append_events`, `get_events`, `get_schema_version`, +`upgrade_schema`, `close`, `dispose`, and context management. `RunStorage` provides `get_run`, +`get_runs`, `project`, `rebuild_from_events`, the schema/lifecycle methods, and context management. +The concrete SQLite and PostgreSQL classes additionally provide transactional +`append_and_project`, `reconcile`, and `check_health` operations. + +The deprecated `EventLogStorage` alias has been removed; use `LifecycleEventLogStorage`. Storage +tables use the lifecycle-specific names `lifecycle_event_log` in SQLite and +`sqlbuild_lifecycle_event_log` in PostgreSQL; the former `event_log` and `sqlbuild_event_log` names +are retired. + +Each newly stored event receives a positive, increasing `storage_order`, opaque stable `cursor`, +and UTC `received_at`. Pages ascend in storage order. `after_cursor` is exclusive, +`next_cursor` identifies the last returned record and is `None` for an empty page, and `has_more` +states whether another page exists. Limits default to 100 and cannot exceed 1000. Cursors are +backend- and storage-instance-owned; persist them verbatim and never parse them. They identify a +global position and remain valid if a later call changes filters. + +`EventFilter` supports invocation ID, run ID, event types, event family, producer, and inclusive +event-time bounds. Event types and family are mutually exclusive. `EventFamily` contains +invocation, run, resource-attempt, operation, and statement; select retry facts with +`EventFilter(event_types=("retry_scheduled",))`. `RunFilter` supports invocation, status, and +inclusive projection-creation bounds. A run cursor identifies the global +`(created_at, run_id)` order and is also filter-independent. + +Projection applies durable facts in `storage_order`, never `occurred_at` order. `created_at` is the +first run fact's `received_at`; start/end times retain event time. A run with no terminal is +`RunStatus.UNKNOWN` and `is_complete=False`. The last durable terminal wins if conflicting terminal +facts exist. Incremental projection and a full rebuild produce the same result, and one projection +call publishes all changes or none. + +## Local SQLite + +The CLI does not open local history implicitly. Construct and subscribe this backend only when an +application has a concrete durable-history consumer. + +```python +from pathlib import Path + +from sqlbuild.execution_history import EventFilter, RunFilter +from sqlbuild.sqlite_history import SQLiteExecutionHistory + +with SQLiteExecutionHistory(project_dir=Path(".")) as history: + assert history.check_health() + + cursor = None + while True: + page = history.get_events( + event_filter=EventFilter(run_id="RUN_ID"), + after_cursor=cursor, + ) + for stored in page.records: + print(stored.storage_order, stored.event) + if not page.has_more: + break + cursor = page.next_cursor + + for run in history.get_runs(run_filter=RunFilter()).records: + print(run.run_id, run.status, run.is_complete) +``` + +With no arguments, `SQLiteExecutionHistory()` uses the current directory. Passing `project_dir` +uses `/.sqlbuild/history.sqlite3`; passing `path` selects an exact file, and `path` may +be `":memory:"`. Do not pass both. Startup creates/migrates schema version 1 and reconciles the run +projection from all event facts. + +Schema v1 has `execution_history_metadata`, append-only `lifecycle_event_log`, and disposable +`run_projection` tables plus run, invocation, type, and created-time indexes. SQLite uses WAL mode, +foreign keys, a configurable 5000 ms default busy timeout, a process-local reentrant lock, and +transactions for append/projection publication. Multiple processes may use SQLite subject to normal +SQLite/WAL host-filesystem constraints and busy timeout; it is a host-local backend, not a deployed +coordination service. Startup refuses unknown future schema versions and never resets or downgrades +data. + +## Deployed PostgreSQL + +Install `sqlbuild[postgres]` only in deployed processes and construct the backend explicitly. The +CLI never selects PostgreSQL implicitly. Neither backend is selected by ordinary CLI commands. + +```python +import os +from datetime import UTC, datetime + +from sqlbuild.execution_history import EventFilter +from sqlbuild.observability import create_lifecycle_event, invocation_scope, run_scope +from sqlbuild.postgres_history import PostgresExecutionHistory + +occurred_at = datetime(2026, 9, 2, 10, 15, tzinfo=UTC) +with invocation_scope("deployment-20260902-1015"): + with run_scope("run-20260902-1015"): + run_started = create_lifecycle_event( + event_type="run_started", + event_id="run-20260902-1015-started", + occurred_at=occurred_at, + payload={"run_kind": "build", "selected_count": 12}, + ) + +with PostgresExecutionHistory(os.environ["SQLBUILD_HISTORY_DSN"]) as history: + first = history.append_and_project((run_started,))[0] + retry = history.append_event(run_started) + assert retry.storage_order == first.storage_order + + page = history.get_events( + event_filter=EventFilter(run_id="run-20260902-1015"), + after_cursor=None, + ) + checkpoint = page.next_cursor + +# Persist checkpoint securely. A later process resumes after that exact global position. +with PostgresExecutionHistory(os.environ["SQLBUILD_HISTORY_DSN"]) as history: + later = history.get_events( + event_filter=EventFilter(run_id="run-20260902-1015"), + after_cursor=checkpoint, + ) +``` + +The DSN must be resolved from a secret before construction. SQLBuild neither retains it for +diagnostics nor includes it in exceptions or `repr`. PostgreSQL schema v1 uses +`sqlbuild_storage_migrations`, authoritative `sqlbuild_lifecycle_event_log`, and disposable +`sqlbuild_run_projection`. Startup takes a transaction-scoped advisory migration lock, performs +forward-only migrations, and reconciles projection. All instances should run the same SQLBuild +version during rollout. + +Append and projection serialize under a history advisory lock. Only serialization failures and +deadlocks (`40001`, `40P01`) are retried internally, up to three retries by default. If connection +loss makes commit acknowledgement uncertain, discard the backend, reconnect, and retry the exact +same deterministic event ID and content. Equal content returns the existing stored fact; different +content raises `IntegrityConflictError`. This is storage idempotency, not exporter exactly-once +delivery. + +Grant deployed application roles only the required schema/table privileges and reserve migration, +backup, retention, and destructive privileges for operators. Back up `sqlbuild_lifecycle_event_log` and +`sqlbuild_storage_migrations`, preserving event IDs, canonical JSON, identity sequence values, and +storage namespace. `sqlbuild_run_projection` can be rebuilt with `reconcile()`. PostgreSQL performs +no automatic retention, partitioning, or archival. + +These are selected operational examples rather than an exhaustive API reference. The public +`sqlbuild.execution_history` facade also exports canonical content/ID helpers, page-limit +validation, projection helpers, schema and paging constants, storage protocols, models, filters, +statuses, and typed storage/cursor/schema errors. + +See [Observability](../observability.md) for the lifecycle event envelope, correlation, and +failure behavior. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/sinks.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/sinks.md index 7b59dfdf64..328cdc4752 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/sinks.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/observability/sinks.md @@ -4,7 +4,7 @@ > Export lifecycle facts and command output through project-owned providers. -Online: https://docs.sqlbuild.com/concepts/observability/sinks +Online: https://sqlbuild.com/docs/concepts/observability/sinks/ Project sinks consume one declared record type. Lifecycle and command-output streams remain separate even when they share a provider and destination. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning.md index de2aad4ccc..86291e7b00 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning.md @@ -4,7 +4,7 @@ > How SQLBuild explains build work, detects changes, and chooses safe model actions. -Online: https://docs.sqlbuild.com/concepts/planning +Online: https://sqlbuild.com/docs/concepts/planning/ ## Contents @@ -27,14 +27,9 @@ Planning and selection are separate concerns: resource needs work, chooses the correct action for incremental models, propagates upstream changes, and warns when a partial selection would be incoherent. -In the default **direct mode**, a build runs its selected scope. Change detection still matters: the -plan can show query and configuration differences, calculate replay or full-refresh actions, and -identify stale upstreams. It does not filter a direct build to stale models. - -[Virtual environments](virtual-environments.md) additionally bind each model to a versioned -physical relation. Virtual builds can use those bindings to reuse existing versions or limit an -invocation to stale work. That virtual execution behavior is documented under -[Building in a virtual environment](virtual-environments/building.md#stale-driven-execution). +A build runs its selected scope. Change detection still matters: the plan can show query and +configuration differences, calculate replay or full-refresh actions, and identify stale upstreams. +It does not filter a build to stale models. ## Reading a plan @@ -103,16 +98,15 @@ The plan assigns reasons to explain detected work: | Upstream changed | A dependency change propagated to this model | | Run despite unchanged | A table's explicit policy requires another run despite an unchanged identity | -These reasons explain the plan; they are not a promise that direct mode will skip entries labelled -current. Direct builds execute the selected scope. Virtual mode can compare expected identities with -the versions already bound to a VDE and avoid unnecessary version creation. +These reasons explain the plan; they are not a promise that a build will skip entries labelled +current. Builds execute the selected scope. ## Cascades and incremental actions Changes propagate through the DAG in dependency order. The resulting action depends on the materialization: -- views are recreated when selected for a direct build; +- views are recreated when selected; - tables rebuild when selected; - incremental models use their cursor state, backfill policy, and `replay_on_change` setting to determine the affected range; and @@ -132,19 +126,16 @@ stale-upstream handling. ## Recorded state -Direct mode records append-only planning evidence in the warehouse: +SQLBuild records append-only planning evidence in the warehouse: - `_sqlbuild_fingerprints` stores resource identities; - `_sqlbuild_source_freshness` stores source observations; and - `_sqlbuild_node_results` stores Python node outcomes. -The planner reads the latest applicable facts and appends new facts after successful work. Virtual -mode stores version bindings and environment-scoped planning state in its configured state backend; -see [Virtual environment setup](virtual-environments/setup.md). +The planner reads the latest applicable facts and appends new facts after successful work. ## Related topics - [Cascade propagation](planning/cascade-propagation.md) - [Source freshness](planning/source-freshness.md) - [Selection and staleness](planning/selection-and-staleness.md) -- [Building in a virtual environment](virtual-environments/building.md) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/cascade-propagation.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/cascade-propagation.md index a6fdd644fb..7ced95ea76 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/cascade-propagation.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/cascade-propagation.md @@ -4,7 +4,7 @@ > How a change signal propagates downstream through the DAG, and how each materialization type responds. -Online: https://docs.sqlbuild.com/concepts/planning/cascade-propagation +Online: https://sqlbuild.com/docs/concepts/planning/cascade-propagation/ When a model or function changes, the change signal propagates downstream through the DAG. Every model downstream of a changed node is marked `Upstream changed` in the plan, even if its own SQL and config are identical. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/selection-and-staleness.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/selection-and-staleness.md index c548b17ad5..38f2ad6d21 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/selection-and-staleness.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/selection-and-staleness.md @@ -4,7 +4,7 @@ > How selection interacts with change detection, and the stale warnings that stop silent partial rebuilds. -Online: https://docs.sqlbuild.com/concepts/planning/selection-and-staleness +Online: https://sqlbuild.com/docs/concepts/planning/selection-and-staleness/ Change detection is selection-aware. When you scope a run with `--select`, SQLBuild only runs (and only displays) the resources you selected, but it still reasons about the *whole* graph to keep the result honest. The key case it handles: a selected model whose upstream changed but is **not** in the selection. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md index af8cae2b16..58f3efcb22 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/planning/source-freshness.md @@ -4,12 +4,11 @@ > Observe external source changes and propagate them through planning. -Online: https://docs.sqlbuild.com/concepts/planning/source-freshness +Online: https://sqlbuild.com/docs/concepts/planning/source-freshness/ Source freshness lets SQLBuild observe whether external source data changed between runs. A changed observation propagates through the dependency graph, so plans can explain which downstream models -are affected and choose the correct incremental action. In a virtual environment, the same signal -also participates in [stale-driven execution](../virtual-environments/building.md#stale-driven-execution). +are affected and choose the correct incremental action. ## Configuration @@ -31,9 +30,8 @@ For timestamp-based freshness, `lag_tolerance` controls how much the observed va ## State storage -In direct mode, source freshness observations are appended to `_sqlbuild_source_freshness` in each -target schema. In virtual mode, observations are scoped to the VDE in the configured state backend. -In both modes, failed work does not replace the prior successful observation, so the next plan still +Source freshness observations are appended to `_sqlbuild_source_freshness` in each target schema. +Failed work does not replace the prior successful observation, so the next plan still sees the pending source change. Direct observations are resolved across all target schemas in the project, so a source referenced by diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md index 10bef8ed84..ef42122937 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md @@ -4,7 +4,7 @@ > Configure your SQLBuild project with sqlbuild_project.toml and sqlbuild_local.toml. -Online: https://docs.sqlbuild.com/concepts/project-configuration +Online: https://sqlbuild.com/docs/concepts/project-configuration/ ## Contents @@ -20,6 +20,7 @@ Online: https://docs.sqlbuild.com/concepts/project-configuration - Scenario - dbt - Skills +- Project Python - sqlbuild_local.toml SQLBuild projects are configured with two files in the project root: @@ -235,6 +236,75 @@ allow_as_clone_destination = true Both policies default to `false`. `sqb clone --from prod --to dev` requires `allow_as_clone_origin = true` on `prod` and `allow_as_clone_destination = true` on `dev`. +### Execution limits + +Named targets can set optional limits for `sqb build`. Limits are useful for development targets +where an unexpectedly broad selector or long-running command should stop instead of consuming an +unbounded amount of warehouse compute. + +```toml +[targets.dev] +connection = "warehouse" +database = "analytics" +schema = "dev_alice" + +[targets.dev.execution_limits] +max_models = 125 +max_duration = "45m" +remediation = """ +Review the expanded plan and confirm that the selected scope is intentional. +Automated tools must request approval before changing this policy. +""" +``` + +All fields are optional. A target without an `execution_limits` section retains unlimited existing +behavior, so production or orchestrated targets do not need placeholder values. + +#### Model limit + +`max_models` is a positive integer. SQLBuild expands selectors and counts model entries that would +execute. When the count exceeds the target limit, the build exits before clone, load, or model +execution begins. The error reports the target, selected count, configured maximum, configuration +key, and confirms that no warehouse changes were made. + +`sqb plan` remains unrestricted and can be used to inspect a broad selection safely. + +#### Duration limit + +`max_duration` is a positive fixed duration such as `30m`, `2h`, or `90s`. It covers planning and +warehouse-mutating build phases after target resolution. SQLBuild stops dispatching work at the +deadline and passes the remaining duration to each warehouse statement so in-flight work is +cancelled by the adapter. + +Safe in-flight cancellation is currently supported by the Snowflake adapter, whose maximum accepted +duration is seven days. SQLBuild rejects an unsupported duration or an adapter that cannot guarantee +active statement cancellation rather than presenting a limit that may leave warehouse work running. + +A duration failure may occur after earlier models have completed. Its error therefore distinguishes +the timeout from a pre-execution model-limit failure and states that warehouse changes may already +have occurred. + +#### Custom remediation + +`remediation` is optional project-authored text. SQLBuild appends it to model-count, duration, and +unsupported-adapter errors after the factual error details. This lets each project define its own +approval or recovery process without SQLBuild recommending that users weaken local policy. + +#### Local overrides + +The same section is accepted in `sqlbuild_local.toml` and follows normal field-by-field local target +precedence: + +```toml +[targets.dev.execution_limits] +max_models = 200 +max_duration = "90m" +``` + +This permits an intentional developer override while keeping the shared target policy visible in +`sqlbuild_project.toml`. Other target fields and any execution-limit fields omitted locally continue +to inherit their project values. + ## Defaults Project-wide model defaults. Any field you can set in a `MODEL()` header can be set here as a default: @@ -318,8 +388,6 @@ default_audit_run_scope = "final" | Field | Default | Description | |-------|---------|-------------| | `sql_analysis` | `true` | Enable SQL syntax analysis, column binding, type inference, output inference, and semantic validation at compile time | -| `changes_only` | `false` | In virtual mode, limit `plan` and `build` to [stale-driven execution](virtual-environments/building.md#stale-driven-execution) without passing `--changes-only`. Rejected in direct mode. Can also be set per target under `[targets.]`. The CLI flag takes precedence, then the selected target, local settings, and this project setting. | -| `virtual_environments` | `false` | Enable [virtual environments](virtual-environments.md) (versioned model outputs, promotion, rollback, state management). When `false`, the project runs in direct mode. | | `query_change_tracking` | `true` | Track query fingerprints for change detection | | `column_contract_mode` | `implicit` | Controls whether column declarations on models without a `contract` declaration activate static shape/nullability validation. `implicit` preserves that validation; `explicit` treats columns as metadata and audit attachment unless the model declares `contract enforced`. Model-level `contract enforced` and `contract none` override this setting. Explicit type enforcement remains independent. See [Contracts](models/contracts.md). | | `concurrency` | `1` | Maximum parallel model execution (currently serial only) | @@ -386,12 +454,11 @@ exclude_patterns = ["audit_*", "tmp_*"] | Field | Default | Description | |-------|---------|-------------| | `enabled` | `false` | Whether janitor is active | -| `retention_days` | `14` in direct mode, `30` in virtual-environment mode | How many days a relation must be stale before it is archived (direct mode) or cleaned up (virtual-environment mode) | -| `archive_retention_days` | `14` | Direct mode only: how many days an archive is kept before it is deleted. `0` deletes it in the same run. | +| `retention_days` | `14` | How many days a relation must be stale before it is archived | +| `archive_retention_days` | `14` | How many days an archive is kept before it is deleted. `0` deletes it in the same run. | | `delete_tracked_only` | `true` | Only clean relations that appear in fingerprint tracking | | `exclude_patterns` | `[]` | Glob patterns for relations to skip | | `direct_state_history_versions` | `20` | How many state-history rows to keep per identity in `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness` | -| `max_checkpoints` | `20` | Virtual-environment mode: how many checkpoints to keep | See [janitor](../cli/janitor.md) for the archive and delete lifecycle. @@ -462,6 +529,19 @@ auto_update = false See [skills CLI reference](../cli/skills.md) for usage details. +## Project Python + +Project-owned Python must live in a supported extension location such as `factories/`, `libs/`, +`macros/`, `providers/`, or another documented Python resource root. Factory locations contain +normal Python: constants, classes, undecorated helper functions, and modules such as `_helpers.py` +are allowed, while decorators determine which functions become SQLBuild resources. Compilation +rejects Python under unsupported project roots, so indirectly importable modules cannot create an +unofficial project structure. + +Keep repository pytest tests outside the SQLBuild project's `tests/` directory, which is reserved for +SQLBuild SQL tests and scenarios. Documented integration paths such as `dagster/`, +`rivers_pipeline/`, and their `definitions.py` modules are also supported. + ## sqlbuild_local.toml Local developer overrides. This optional file is loaded automatically and should be diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/assets.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/assets.md index 1837d0b325..d2fe11ed27 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/assets.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/assets.md @@ -4,7 +4,7 @@ > Produce or observe external artifacts as nodes in the SQLBuild graph. -Online: https://docs.sqlbuild.com/concepts/python-nodes/assets +Online: https://sqlbuild.com/docs/concepts/python-nodes/assets/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/checks.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/checks.md index b14f845951..fc6946658f 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/checks.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/checks.md @@ -4,7 +4,7 @@ > Validate tasks, assets, and loaders with Python checks. -Online: https://docs.sqlbuild.com/concepts/python-nodes/checks +Online: https://sqlbuild.com/docs/concepts/python-nodes/checks/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/factories.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/factories.md index 196aa0881d..76c1df8031 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/factories.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/factories.md @@ -4,7 +4,7 @@ > Generate Python nodes programmatically with @factory. -Online: https://docs.sqlbuild.com/concepts/python-nodes/factories +Online: https://sqlbuild.com/docs/concepts/python-nodes/factories/ A factory is a function that **generates** Python nodes instead of authoring them one at a time. Use `@factory` when you want to create many similar [tasks](tasks.md), [assets](assets.md), [loaders](loaders.md), or [checks](checks.md) from a list, a config, or a loop, rather than hand-writing each one. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md index ac9dd81c60..d97906b3ab 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/loaders.md @@ -4,7 +4,7 @@ > Load external data into source tables with Python functions. -Online: https://docs.sqlbuild.com/concepts/python-nodes/loaders +Online: https://sqlbuild.com/docs/concepts/python-nodes/loaders/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/overview.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/overview.md index d80f3cfe76..dcfc26c99f 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/overview.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/overview.md @@ -4,7 +4,7 @@ > Tasks, assets, loaders, and checks as first-class nodes in the SQLBuild graph. -Online: https://docs.sqlbuild.com/concepts/python-nodes/overview +Online: https://sqlbuild.com/docs/concepts/python-nodes/overview/ ## Contents @@ -125,7 +125,7 @@ Downstream nodes run only if at least one upstream succeeded. If all upstreams a ## Result persistence -Node results (payload, metadata, status, errors) are persisted after each execution. In direct mode, results are stored in `_sqlbuild_node_results` in the warehouse alongside your data. In virtual mode, results are stored in the VDE state backend scoped per environment. Results persist across runs, so they are available for observability, debugging, and downstream consumption. +Node results (payload, metadata, status, errors) are persisted after each execution. Results are stored in `_sqlbuild_node_results` in the warehouse alongside your data. Results persist across runs, so they are available for observability, debugging, and downstream consumption. ## Selection diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/providers.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/providers.md index 0c02d37600..b42ac14b84 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/providers.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/providers.md @@ -4,7 +4,7 @@ > Shared runtime services for Python nodes and hooks. -Online: https://docs.sqlbuild.com/concepts/python-nodes/providers +Online: https://sqlbuild.com/docs/concepts/python-nodes/providers/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/sql-references.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/sql-references.md index 6f074a7af8..ec849b5955 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/sql-references.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/sql-references.md @@ -4,9 +4,9 @@ > Read SQL models and sources from Python nodes without creating SQL dependencies. -Online: https://docs.sqlbuild.com/concepts/python-nodes/sql-references +Online: https://sqlbuild.com/docs/concepts/python-nodes/sql-references/ -Python nodes can **read** SQL models and sources at runtime, but they cannot depend on them as graph edges (see the [SQL boundary](overview.md#the-sql-boundary)). Typed references make this read-only access explicit and safe across direct and virtual modes. +Python nodes can **read** SQL models and sources at runtime, but they cannot depend on them as graph edges (see the [SQL boundary](overview.md#the-sql-boundary)). Typed references make this read-only access explicit and safe. ## Declaring a reference @@ -46,13 +46,13 @@ def export_orders(ctx): `ctx.relation(...)` returns the adapter-qualified relation name for the current run. Passing a reference that was not declared in `depends_on` raises an error. -Do not hardcode model or source names in raw SQL. In virtual mode a model resolves to a versioned VDE relation (for example `marts__feature_x.fact_orders`), not the bare name. A query like `ctx.query("SELECT * FROM fact_orders")` will fail or read the wrong relation. Always use `ctx.relation(model("fact_orders"))`. +Do not hardcode model or source names in raw SQL. A model's relation depends on the target, its schema and any deferral, so a query like `ctx.query("SELECT * FROM fact_orders")` can fail or read the wrong relation. Always use `ctx.relation(model("fact_orders"))`. ## model vs source | Reference | Resolves to | |-----------|-------------| -| `model("name")` | The built model relation (the VDE logical relation in virtual mode) | +| `model("name")` | The built model relation | | `source("name")` | The source read relation, following deferred-source semantics | `source(...)` respects `defer_sources_to`, so a node reading a source in `dev` can read the deferred target's data just like SQL models do. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/tasks.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/tasks.md index 71d3828b52..6929c54594 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/tasks.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/tasks.md @@ -4,7 +4,7 @@ > Run Python computation and side effects as nodes in the SQLBuild graph. -Online: https://docs.sqlbuild.com/concepts/python-nodes/tasks +Online: https://sqlbuild.com/docs/concepts/python-nodes/tasks/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/resource-identities.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/resource-identities.md index 6e6c9a3820..1cd032f86d 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/resource-identities.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/resource-identities.md @@ -4,7 +4,7 @@ > Canonical names for SQLBuild resources, selectors, state, and integrations. -Online: https://docs.sqlbuild.com/concepts/resource-identities +Online: https://sqlbuild.com/docs/concepts/resource-identities/ SQLBuild resource identities use lowercase ASCII snake_case. A name must start with a lowercase letter, end with a lowercase letter or digit, and contain only lowercase letters, digits, and diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules.md index ae0422f808..993ccef02d 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules.md @@ -4,7 +4,7 @@ > Turn repeated SQL and project review decisions into compiler-enforced requirements. -Online: https://docs.sqlbuild.com/concepts/rules +Online: https://sqlbuild.com/docs/concepts/rules/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/configuration-and-selection.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/configuration-and-selection.md index 2a12024177..19424f3424 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/configuration-and-selection.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/configuration-and-selection.md @@ -4,7 +4,7 @@ > Select built-in and custom Rules by exact code or family prefix. -Online: https://docs.sqlbuild.com/concepts/rules/configuration-and-selection +Online: https://sqlbuild.com/docs/concepts/rules/configuration-and-selection/ Configure built-in and custom Rules together: @@ -73,3 +73,35 @@ Focused execution constructs the compiler facts needed by the selected Rules. It the complete configured enforcement performed by `sqb compile`. See every `sqb rules` command and exit code. + +## Join keys and final CTE names + +`SQBRSQL040` checks every `JOIN ... ON` in the SQL covered by SQL Rules, including joins inside CTEs +and subqueries. `USING (...)` is accepted. Allowed predicates are: + +- Plain column comparisons, including equality, inequality, range and null-safe comparisons. +- A plain column compared with a literal, including a signed numeric literal. +- Literal-only predicates such as `ON TRUE`, `ON FALSE` and `ON 1 = 1`, including + `LEFT JOIN LATERAL FLATTEN(...) AS f ON TRUE`. Functions or casts applied to literals are still + computed expressions. +- `BETWEEN` / `NOT BETWEEN` on a plain column with plain-column or literal bounds. +- `IN` / `NOT IN` on a plain column with a literal list. +- `IS NULL` / `IS NOT NULL` on a plain column. +- Parenthesized predicates, and `AND` or `OR` combinations in which every branch is allowed. + +Functions, casts (including `::`), arithmetic, concatenation, `CASE`, subqueries and other computed +operands are findings. For example, replace `ON LOWER(o.category) = c.category_key` with a CTE that +projects `LOWER(category) AS category_key`, then join `ON o.category_key = c.category_key`. Snowflake +variant path access such as `ON a.data:key = b.id` is also computed: project the value as a named +column before joining. + +`SQBRSQL041` checks CTE naming: the last top-level CTE must be named `final` (compared +case-insensitively, including quoted identifiers), and no other CTE, including a nested one, may use +that name. Queries without CTEs are unaffected, and SQL-test fixture CTEs are exempt. `SQBRSQL035` +owns the terminal `SELECT`'s plain projection from that last CTE, so select both, normally through +the `SQBRSQL` family, for the complete convention. Parentheses around the entire query do not create +a nested CTE scope. + +Family selectors such as `select = ["SQBRSQL", "SQBRMODEL", "SQBRGRAPH"]` pick up new built-in Rules +on upgrade, so a release that adds `SQBRSQL040` and `SQBRSQL041` can expose computed join keys or +differently named final CTEs that were accepted before. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/overview.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/overview.md index b68d4119b5..269095813b 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/overview.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/overview.md @@ -4,7 +4,7 @@ > Define repository-owned checks with one typed Python API. -Online: https://docs.sqlbuild.com/concepts/rules/custom-rules/overview +Online: https://sqlbuild.com/docs/concepts/rules/custom-rules/overview/ Custom Rules are ordinary Python under `rules/**/*.py`. Only functions decorated with `@rule` register; helpers, constants, dataclasses, and classes remain ordinary Python. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/rule-context.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/rule-context.md index 300b392770..58b0857ffc 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/rule-context.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/rule-context.md @@ -4,7 +4,7 @@ > Inspect stable compiler-owned facts and opt into lazy SQL AST access. -Online: https://docs.sqlbuild.com/concepts/rules/custom-rules/rule-context +Online: https://sqlbuild.com/docs/concepts/rules/custom-rules/rule-context/ `RuleContext` exposes typed, read-only views over the compiled project: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/testing-and-determinism.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/testing-and-determinism.md index 6d786b2e8e..dca63b7350 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/testing-and-determinism.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/custom-rules/testing-and-determinism.md @@ -4,7 +4,7 @@ > Test custom findings and keep implementations safe for dependency-aware caching. -Online: https://docs.sqlbuild.com/concepts/rules/custom-rules/testing-and-determinism +Online: https://sqlbuild.com/docs/concepts/rules/custom-rules/testing-and-determinism/ ## Test Rules diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/execution-and-caching.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/execution-and-caching.md index 47f78c109c..27d0688515 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/execution-and-caching.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/execution-and-caching.md @@ -4,7 +4,7 @@ > Understand Rule enforcement order, focused runs, and dependency-aware reuse. -Online: https://docs.sqlbuild.com/concepts/rules/execution-and-caching +Online: https://sqlbuild.com/docs/concepts/rules/execution-and-caching/ SQLBuild evaluates a project in this order: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/findings-and-exceptions.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/findings-and-exceptions.md index ffe81e254b..887b4b818e 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/findings-and-exceptions.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules/findings-and-exceptions.md @@ -4,7 +4,7 @@ > Understand Rule diagnostics and record intentional exceptions safely. -Online: https://docs.sqlbuild.com/concepts/rules/findings-and-exceptions +Online: https://sqlbuild.com/docs/concepts/rules/findings-and-exceptions/ A Rule finding identifies an exact code, project-relative path, line, column, message, and remediation. Error findings make compilation fail and prevent artifact completion. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md index 04fdfe7214..2afbe938fc 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md @@ -4,7 +4,7 @@ > End-to-end tests that build real project graphs against coherent fixture data. -Online: https://docs.sqlbuild.com/concepts/scenarios +Online: https://sqlbuild.com/docs/concepts/scenarios/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/seeds.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/seeds.md index 8a902e3372..068e555c86 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/seeds.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/seeds.md @@ -4,7 +4,7 @@ > Load static CSV data into your pipeline as tables. -Online: https://docs.sqlbuild.com/concepts/seeds +Online: https://sqlbuild.com/docs/concepts/seeds/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/selectors.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/selectors.md index f0a12610a3..4664f9edc4 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/selectors.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/selectors.md @@ -4,7 +4,7 @@ > Target specific models, paths, tags, or DAG subsets with select and exclude flags. -Online: https://docs.sqlbuild.com/concepts/selectors +Online: https://sqlbuild.com/docs/concepts/selectors/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/snapshots.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/snapshots.md index aaff877967..0fbaf5415a 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/snapshots.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/snapshots.md @@ -4,7 +4,7 @@ > Preserve row history over time using SCD Type 2 semantics with timestamp or check-based change detection. -Online: https://docs.sqlbuild.com/concepts/snapshots +Online: https://sqlbuild.com/docs/concepts/snapshots/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md index 339e750117..b9e06ca23e 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/sources.md @@ -4,7 +4,7 @@ > Declare external data inputs for your pipeline. -Online: https://docs.sqlbuild.com/concepts/sources +Online: https://sqlbuild.com/docs/concepts/sources/ ## Contents @@ -125,8 +125,7 @@ For common ingestion you can declare the source entirely in YAML, with no `@load Source freshness lets SQLBuild observe whether a source's data changed between runs. This feeds into [planning and change detection](planning.md): changed observations propagate to downstream -models and can alter their planned action. Virtual environments can also use the signal for -[stale-driven execution](virtual-environments/building.md#stale-driven-execution). +models and can alter their planned action. Configure freshness per source with a `freshness:` block: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md index 7a72e04ae0..f49af3a666 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md @@ -4,7 +4,7 @@ > SQL unit tests and multi-model tests with macro support, assertions, and model chaining. -Online: https://docs.sqlbuild.com/concepts/testing +Online: https://sqlbuild.com/docs/concepts/testing/ ## Contents @@ -12,6 +12,7 @@ Online: https://docs.sqlbuild.com/concepts/testing - CTE conventions - Multi-model tests - Mocking refs and seeds +- Fixture columns - Mocking table functions in model tests - Multiple expected models - Macro-powered mocks @@ -224,6 +225,21 @@ SELECT 1 You can mix `__source__`, `__ref__`, and `__seed__` mocks in the same test. As long as every leaf dependency is satisfied (either by a source mock, a ref mock, a seed mock, or by being in the expected chain), the test resolves. +## Fixture columns + +A relation fixture can omit a column when the compiled test or scenario closure requires it and +SQLBuild knows its adapter type, unless the column is explicitly non-nullable. SQLBuild completes +that test-only fixture column with a typed null such as `CAST(NULL AS VARCHAR)`; it never changes +model SQL or warehouse defaults. Required columns declared with `nullable false` and columns with +unknown types must be supplied explicitly. When the relation's complete column set is +authoritative, misspelled or unknown supplied fixture columns are rejected. + +A direct `NULL AS column_name` projection receives the authoritative relation type in compiled test +SQL, including expected-output CTEs for contracted models. To represent a contracted upstream with +no rows, use `SELECT * FROM __empty_fixture()` as the complete body of a `__ref__`, `__source__`, or +`__seed__` fixture CTE, or a contracted `__expected__` CTE. SQLBuild expands it to the relation's +full typed schema with a false filter. + ## Mocking table functions in model tests Use `__table_fn__` when the model under test reads a managed table function but the test diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments.md deleted file mode 100644 index a5048588f5..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments.md +++ /dev/null @@ -1,117 +0,0 @@ - - -# Overview - -> Version-controlled SQL pipeline environments with instant promotion and rollback. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments - -## Contents - -- When to use virtual environments -- How it works -- Try it -- Example workflow -- What's next - -Virtual environments are in alpha. The core workflow (build, promote, rollback, reconcile) is functional and tested across supported adapters. The API and CLI surface may evolve based on feedback. Do not use virtual environments for production workloads yet. - -Virtual environments (VDEs) let you build, preview, and promote SQL pipeline changes without recomputing models. Each VDE is a set of pointers to versioned physical relations. Creating a new VDE is instant (pointer copy, no data duplication), and promoting one VDE into another is a view swap, not a rebuild. - -## When to use virtual environments - -- **PR preview environments** - build a VDE per pull request against a production warehouse, validate with audits and tests, then promote the built versions to production without rebuilding -- **Blue/green deployments** - build into a staging VDE, promote to production atomically -- **Multi-developer isolation** - each developer works in their own VDE without conflicting with others, sharing physical versions when code is identical -- **Instant rollback** - revert production to a prior finalized state by restoring a checkpoint's pointer set - -Virtual environments are opt-in via `virtual_environments = true` (under `[settings]`) and require a state store. Projects that don't need environment isolation or promotion workflows should use the default direct mode. - -## How it works - -### Targets and virtual environments - -In virtual mode there are two separate concepts: - -**Targets** are the existing SQLBuild build contexts from `sqlbuild_project.toml` (e.g. `dev`, `prod`). They choose the warehouse connection, schema, and state database. In virtual mode they are sometimes called *physical targets* to distinguish them from VDEs. - -**Virtual data environments (VDEs)** are versioned pointer sets stored in the state database. They choose which model versions the logical views point to. - -```bash -sqb --target prod build --virtual-env pr_123 -``` - -This means: use the `prod` physical target (warehouse, state DB), but build into the `pr_123` virtual environment. - -### Version identity - -Model versions are identified by content hashes, not sequence numbers. The hash includes the model's query SQL, version-identity config, and upstream version hashes. If two developers compile identical code with identical upstream versions, they get the same hash and reuse the same physical relation. No data is duplicated. - -### Physical and logical relations - -Virtual mode creates two types of warehouse objects: - -- **Physical version relations** store actual data: `__sqb_physical.__v_` -- **Logical VDE views** point to physical versions: `__.` as `SELECT * FROM ` - -Users query the logical views. SQLBuild manages the physical layer. - -### Zero-copy branching - -Creating a new VDE from a baseline copies only pointer rows in the state database, not data. Unchanged models share the same physical relations across VDEs. - -### Instant promotion - -Promoting VDE `pr_123` to `prod` updates the pointer rows and refreshes the logical views. No models are rebuilt. If every model in the source VDE has already been built and validated, promotion is a metadata operation. - -## Try it - -```bash -sqb playground --template virtual -cd sqlbuild-playground -sqb state init -sqb build -sqb build --virtual-env pr -sqb diff dev:pr --schema-only --allow-partial-diff -sqb promote --from pr --to dev -sqb state checkpoints list -sqb rollback -``` - -This creates a self-contained DuckDB project with loaders, models, tests, scenarios, and a local state store. No warehouse credentials required. - -## Example workflow - -```bash -# Initialize state store -sqb state init - -# First build creates physical versions and default VDE -sqb build - -# Create a PR preview VDE -sqb build --virtual-env pr_123 - -# Modify a model, rebuild in the PR VDE -sqb build --virtual-env pr_123 - -# Compare VDEs -sqb diff dev:pr_123 - -# Promote PR versions to the default VDE -sqb promote --from pr_123 --to dev -``` - -## What's next - -- [Setup](virtual-environments/setup.md) - configuration and state initialization -- [Building](virtual-environments/building.md) - virtual builds, partial builds, seeded incrementals -- [Promotion](virtual-environments/promotion.md) - promoting VDEs -- [Rollback](virtual-environments/rollback.md) - checkpoints and rollback -- [Clone](virtual-environments/clone.md) - hydrating physical versions from a source warehouse -- [Diff](virtual-environments/diff.md) - comparing VDE ref sets -- [Adopt and Detach](virtual-environments/adopt-detach.md) - migrating existing projects -- [Reconcile](virtual-environments/reconcile.md) - diagnosing and repairing drift -- [Locks](virtual-environments/locks.md) - concurrent access control -- [Janitor](virtual-environments/janitor.md) - cleanup and retention -- [Recovery](virtual-environments/recovery.md) - what to do when things break diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/adopt-detach.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/adopt-detach.md deleted file mode 100644 index 21f75fbbb5..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/adopt-detach.md +++ /dev/null @@ -1,123 +0,0 @@ - - -# Adopt and Detach - -> Migrate existing projects to and from virtual mode. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments/adopt-detach - -## Contents - -- Adopt -- Detach -- Interrupted operations - -Adopt converts an existing stateless project into virtual mode. Detach reverses the process. Both are interactive operations that require typed confirmation. - -## Adopt - -`sqb state adopt` takes existing warehouse relations (tables and views) and converts them into versioned physical relations with VDE views at the original names. - -### Prerequisites - -1. State store must be initialized (`sqb state init`) -2. `unsuffixed_virtual_env` must be configured so existing object names are preserved: - -```toml -[targets.dev.state] -backend = "duckdb" -schema = "sqlbuild_state" -unsuffixed_virtual_env = "dev" - -[targets.dev.state.connection] -database = "state.duckdb" -``` - -Without `unsuffixed_virtual_env`, adopt blocks with a config error. This is intentional - without it, existing relations would be renamed to suffixed schemas (e.g. `dev__dev.fact_orders`), breaking existing consumers. - -### What happens - -1. For each model, the existing table is moved/renamed into the physical schema (`dev__sqb_physical.fact_orders__v_`) -2. A logical VDE view is created at the original name (`dev.fact_orders`) pointing to the physical version -3. Model versions, physical relations, VDE record, and VDE refs are persisted in state - -### Usage - -```bash -sqb state adopt --allow-copy -``` - -The command prints an adoption plan and requires typed confirmation: - -``` -Type "adopt dev" to confirm: adopt dev -``` - -`--allow-copy` is required when the adapter does not support native same-schema rename (cross-schema moves, some adapters). Without it, adopt blocks with `requires --allow-copy` if a copy fallback would be needed. - -### View models - -View models are adopted the same way - a versioned physical view is created in the physical schema, and the original name becomes a logical VDE view. - -## Detach - -`sqb state detach` reverses adoption, collapsing a VDE back into normal stateless relations. - -### Prerequisites - -The VDE must be `finalized`. If the VDE is working (has stale models), detach blocks: - -``` -error: detach requires a finalized virtual environment -``` - -Build the VDE to finalize it first, or resolve any pending changes. - -### What happens - -1. For each table model, the physical version is moved/renamed back to the original target name -2. For each view model, the view is recreated from compiled SQL at the original target name (not copied from the physical ref) -3. The VDE is marked `detached` in state -4. VDE refs are preserved for audit/recovery and janitor protection - -### Usage - -```bash -sqb state detach --allow-copy -``` - -Requires typed confirmation: - -``` -Type "detach dev" to confirm: detach dev -``` - -### After detach - -A detached VDE is blocked from further virtual operations: - -- `sqb build` blocks with "virtual environment is detached" -- `sqb promote --from ` or `--to ` blocks -- `sqb rollback` on a detached VDE blocks - -The project can continue operating in direct mode, or you can re-adopt to return to virtual mode. - -### Detached VDE cleanup - -Detached VDE refs and state rows are cleaned up by the [janitor](janitor.md), not by detach itself. This preserves refs for recovery if detach fails partway through. - -## Interrupted operations - -Adopt and detach are multi-step warehouse operations that cannot be wrapped in a single transaction. If an operation fails partway through: - -- A `failed` operation record is persisted in state with the error message -- VDE refs and checkpoints remain intact -- The warehouse may have partial artifacts (e.g. table moved to physical schema but view not yet created) - -Recovery path: - -1. Run `sqb reconcile` to diagnose the current state -2. Use `sqb reconcile repair-view` or manual warehouse repair as needed -3. Retry the adopt or detach operation - -There is no automatic resume command. SQLBuild records the failure state and leaves recovery to the operator. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/building.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/building.md deleted file mode 100644 index d846e664e7..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/building.md +++ /dev/null @@ -1,232 +0,0 @@ - - -# Building - -> Virtual builds, VDE creation, partial builds, and seeded incrementals. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments/building - -## Contents - -- Default VDE -- Explicit VDE -- Physical naming -- VDE status -- Partial builds -- Stale-driven execution -- Stale detection -- Seeded incremental builds -- Custom materializations -- Functions -- Plan - -Virtual builds create versioned physical relations and update VDE pointer sets. The build lifecycle is the same as direct mode (seeds, tests, models, audits), but model outputs are written to versioned physical tables and exposed through logical VDE views. - -Virtual builds run ingress (loaders and Python nodes that feed sources) as a separate phase before SQL model execution. This means independent SQL models that do not depend on loaders will wait for all ingress to complete before starting. Direct mode does not have this limitation. This keeps VDE state persistence simpler and safer but may add wall time when ingress is slow and independent SQL work is available. A future optimization may allow overlapping ingress with independent SQL execution. - -## Default VDE - -When you run `sqb build` in virtual mode without `--virtual-env`, the VDE name defaults to the active physical target name: - -```bash -sqb build -``` - -If the active environment is `dev`, this builds into VDE `dev`. On first run, it creates physical version relations and logical VDE views for all models. - -## Explicit VDE - -Use `--virtual-env` to build into a named VDE: - -```bash -sqb build --virtual-env pr_123 -``` - -If VDE `pr_123` does not exist, SQLBuild creates it by inheriting refs from the baseline VDE (the default VDE for the active physical target). Unchanged models share the same physical relations as the baseline - no data is copied. - -Only models with changed code, config, or upstream versions get new physical version relations. - -## Physical naming - -Physical version relations are stored in a dedicated schema: - -``` -__sqb_physical.__v_ -``` - -For example: `dev__sqb_physical.fact_orders__v_8f3a9c12` - -Logical VDE views are created in a VDE-suffixed schema: - -``` -__. -``` - -For example: `dev__pr_123.fact_orders` - -When `unsuffixed_virtual_env` is configured, the named VDE uses the base schema directly: - -``` -dev.fact_orders -``` - -## VDE status - -| Status | Meaning | -|--------|---------| -| `finalized` | All models match their expected version hashes. The VDE is coherent and eligible for promotion. | -| `active` | Some models are stale (partial build or pending changes). The VDE is a work in progress. | -| `detached` | The VDE has been detached via `sqb state detach`. Build, promote, and rollback are blocked. | - -## Partial builds - -Build a subset of models with `--select`: - -```bash -sqb build --virtual-env pr_123 --select fact_orders -``` - -Partial builds leave the VDE in `active` (working) status if downstream models remain stale. A -follow-up `sqb build --virtual-env pr_123 --changes-only` without `--select` builds the remaining -stale models to finalize the VDE. A plain `sqb build --virtual-env pr_123` runs the whole selection. - -### Stale upstream coherence - -If a selected model has stale required upstream models, the build blocks by default: - -```bash -sqb build --virtual-env pr_123 --select fact_orders -# error: selected models have stale required upstreams: stg_orders -``` - -Pass `--include-stale-upstreams` to expand the selection to the minimal set of stale ancestor models needed to make the selected scope coherent: - -```bash -sqb build --virtual-env pr_123 --select fact_orders --include-stale-upstreams -``` - -## Stale-driven execution - -Virtual environment builds run the full selection by default. Add `--changes-only` to limit a plan -or build to work whose expected version is not already bound in the target VDE: - -```bash -sqb plan --virtual-env pr_123 --changes-only -sqb build --virtual-env pr_123 --changes-only -sqb build --virtual-env pr_123 --select path:models/marts --changes-only -``` - -With selectors, SQLBuild intersects the selected scope with the stale set. Models with changed SQL or -configuration, pending backfills, changed functions, or affected upstreams remain in the plan; -models already bound to their expected version do not need another physical version. - -`--changes-only` is a virtual-mode execution option. Direct mode always runs its selected build -scope, although its plan still reports change reasons and uses them to choose incremental actions. -See [Planning and change detection](../planning.md) for the shared identity and cascade model. - -You can make stale-driven execution the default for virtual plans and builds: - -```toml -[settings] -virtual_environments = true -changes_only = true - -[targets.dev] -changes_only = true -``` - -The CLI flag takes precedence, followed by the selected target, local settings, and project -settings. - -### Run despite unchanged - -`run_despite_unchanged` lets a table remain eligible during stale-driven execution even when its -version identity is unchanged. This is useful when external data changes cannot be represented by -source freshness. - -```sql -MODEL ( - materialized table, - run_despite_unchanged "always", -); -``` - -- `always` runs whenever the model is selected. -- A duration such as `24h`, `30d`, or `90m` keeps the model eligible while the newest timestamp data - version from an upstream source is no older than that duration; it is not a periodic scheduler. - -Only table materializations support this setting. When it triggers, downstream models are also -stale. - -## Stale detection - -SQLBuild determines which models and seeds need rebuilding by comparing expected version hashes against bound version hashes in the VDE refs: - -1. **Expected local hash** - derived from the node's query SQL (for models), content hash (for seeds), version-identity config, and source freshness hashes -2. **Expected version hash** - local hash propagated through upstream dependencies (upstream hash changes cascade downstream) -3. **Bound version hash** - the hash currently stored in the VDE refs from the last successful build - -A node is stale when `bound != expected`. Stale nodes whose own local hash changed are roots (`query changed`, `config changed`, `function changed`). Others are stale due to `upstream changed`. - -Seeds participate in version identity the same way as models. They are loaded into versioned physical tables with logical VDE views on top, and their version hashes and refs are tracked per virtual environment in the state backend. Unchanged seeds are not reloaded. - -Source freshness observations and Python node identities are also persisted per virtual environment and included in version hash computation. When a source's observed data version changes, its downstream models become stale. See [Sources: Source freshness](../sources.md#source-freshness) for configuration. - -## Seeded incremental builds - -When an incremental model's version hash changes, SQLBuild seeds the new physical version from the prior physical version before running the incremental delta. This avoids full rebuilds of large incremental tables. - -| Adapter | Seed strategy | -|---------|--------------| -| Snowflake | Zero-copy clone | -| BigQuery | Table clone | -| Databricks | Deep clone | -| DuckDB, Postgres, SQL Server | CTAS copy | - -For append models with bounded replay (`replay_on_change bounded-7d`), the seed copies only rows before the replay window cutoff. The incremental delta then appends the bounded range without duplicating rows. - -## Custom materializations - -Custom materializations are supported in virtual mode. By default, SQLBuild seeds new physical versions using the direct clone/copy strategy before calling the custom `materialize` function. - -For custom materializations that need different seeding behavior, define a `prepare_version` function alongside `materialize`: - -```python -from sqlbuild.virtual.executor.models import VersionPrepareContext -from sqlbuild.executor.custom.models import MaterializationContext, MaterializationResult - -def prepare_version(ctx: VersionPrepareContext) -> None: - """Prepare the new physical destination from the prior version.""" - ctx.execute_sql(f"CREATE TABLE {ctx.destination} AS SELECT * FROM {ctx.prior_relation}") - -def materialize(ctx: MaterializationContext) -> MaterializationResult: - """Run the custom materialization logic against the prepared destination.""" - ... -``` - -If `prepare_version` is not defined, the framework uses the default clone/copy. Most custom materializations do not need to define it. - -`VersionPrepareContext` provides `prior_relation` (the source physical version), `destination` (the new physical destination relation), `adapter`, `connection`, `execute_sql()`, `config`, and `vars`. - -Custom materializations in virtual mode must write only to `ctx.destination`. Side-effect writes to other relations are not tracked by virtual state, not cleaned up by janitor, and not restored by rollback. - -## Functions - -Functions are published into the logical VDE schema, not the physical layer. Each VDE has its own copy of function definitions. Function versions are tracked in state and participate in promotion and rollback. - -## Plan - -Use `sqb plan` to preview what a virtual build would do without executing: - -```bash -sqb plan -sqb plan --virtual-env pr_123 -sqb plan --select fact_orders -``` - -Virtual plan output shows: -- Virtual environment name and status (finalized/working) -- Stale root count and root set -- Stale model count -- Query diffs from prior bound versions -- Remaining stale models after partial selection diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/clone.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/clone.md deleted file mode 100644 index ff3748de20..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/clone.md +++ /dev/null @@ -1,77 +0,0 @@ - - -# Clone - -> Hydrate physical versions from a source warehouse. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments/clone - -In virtual mode, `sqb clone` hydrates physical version relations from a source warehouse into the target physical storage layer. It does not copy VDE pointer sets or create logical views - it copies the underlying physical data so that builds and promotions can reference those versions locally. - -This is useful for seeding a new target from an existing one, or restoring physical versions that were cleaned up by the janitor. - -As in direct mode, `--from` selects an origin namespace and clone-origin policy, not a -connection. `--to` (or the active target) supplies the only physical connection. The origin -physical namespace must be readable through that destination connection; SQLBuild never opens -the origin target connection or transfers data across accounts, servers, or files using a -second connection. - -## How it works - -Virtual clone looks up expected physical version relations in the source warehouse by their naming convention (`__sqb_physical.__v_`) and copies them into the target warehouse. It then registers the copied relations in the target state store. - -Target VDE refs and logical views are not changed. Clone only populates the physical layer. - -## Default mode - -Without `--virtual-env`, clone hydrates physical versions matching the current workspace's expected fingerprints: - -```bash -sqb clone --from prod --to dev -``` - -This computes expected version hashes from the current code, looks for matching physical relations in the source warehouse, and copies them into the target. - -## VDE ref mode - -With `--virtual-env`, clone hydrates physical versions referenced by a specific VDE in the target state store: - -```bash -sqb clone --from prod --to dev --virtual-env pr_123 -``` - -This reads VDE `pr_123`'s refs from the target state, looks for those physical relations in the source warehouse, and copies them into the target. Useful when a VDE's physical versions have been deleted (by janitor or manual cleanup) but the state refs still exist. - -## Selection - -Scope which models are hydrated: - -```bash -sqb clone --from prod --to dev --select fact_orders -``` - -## Missing source artifacts - -If a source physical relation does not exist, clone reports it as missing and continues with the remaining models. The exit code is non-zero if any models are missing. - -## Model version locks - -If a target model version is locked (another process is building that version), clone blocks by default: - -```bash -# Skip locked versions and hydrate the rest -sqb clone --from prod --to dev --skip-locked -``` - -## What clone does not do - -- Does not create or update VDE refs -- Does not create or refresh logical VDE views -- Does not read the source state database (it uses warehouse-level artifact discovery) -- Does not change the target VDE status - -Clone is a physical-layer operation. VDE pointer management is handled by [build](building.md) and [promote](promotion.md). - -## Comparison with direct-mode clone - -In direct mode, `sqb clone` copies model relations between targets using zero-copy cloning where supported. In virtual mode, clone hydrates versioned physical relations instead. The source and target are still physical targets, but the copied objects are physical version relations rather than normal model targets. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/diff.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/diff.md deleted file mode 100644 index b99d70b20b..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/diff.md +++ /dev/null @@ -1,61 +0,0 @@ - - -# Diff - -> Compare virtual data environments. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments/diff - -In virtual mode, `sqb diff` compares VDE ref sets within the same physical target. It shows which models have different version hashes and, for changed models, reports schema and row-level differences. - -## Basic usage - -```bash -sqb diff dev:pr_123 -``` - -The format is `:`. Both VDEs must exist in the active physical target's state store. - -## What diff shows - -1. **Ref comparison** - which models have different version hashes between the two VDEs -2. **Schema differences** - column additions, removals, and type changes for changed models -3. **Row differences** - row counts, matched/unmatched rows, and changed column values - -Identical refs are skipped by default so the output focuses on models that actually differ. A model with a different version hash is included in the comparison even if its resulting data happens to be identical. - -## Options - -```bash -# Schema differences only (no row comparison) -sqb diff dev:pr_123 --schema-only - -# Full row-level comparison -sqb diff dev:pr_123 --full - -# Compare specific models -sqb diff dev:pr_123 --select fact_orders - -# No color output -sqb --no-color diff dev:pr_123 -``` - -## Working VDE guard - -If either VDE is working (has stale models that haven't been built yet), diff is blocked by default because the comparison may be incomplete: - -```bash -# Blocked -sqb diff dev:pr_123 - -# Allowed with explicit override -sqb diff dev:pr_123 --allow-partial-diff -``` - -This guard prevents misleading diff output when one VDE has pending changes that haven't been materialized yet. - -## Comparison with direct-mode diff - -In direct mode, `sqb diff prod:dev` compares physical target schemas and data directly in the warehouse. In virtual mode, `sqb diff dev:pr_123` compares VDE pointer sets within a single physical target, then inspects the physical versions those pointers reference. - -The output format is the same - schema diffs, row counts, changed columns, and example rows. The difference is what is being compared: physical targets vs virtual pointer sets. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/janitor.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/janitor.md deleted file mode 100644 index fb5f3137ea..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/janitor.md +++ /dev/null @@ -1,76 +0,0 @@ - - -# Janitor - -> Cleanup of virtual environment artifacts and retention policies. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments/janitor - -The janitor manages cleanup of virtual mode artifacts: expired VDEs, old checkpoints, unreferenced physical versions, stale state backups, and expired locks. All cleanup runs through `sqb janitor` with preview and confirmation. - -## Physical version protection - -Physical version relations are never deleted while referenced by: - -- Any active (non-detached) VDE's current refs -- Any retained checkpoint's refs - -The janitor resolves the complete set of protected physical relations before considering any deletions. - -## Checkpoint retention - -Checkpoints are retained according to `[janitor] max_checkpoints` (default: 20): - -```toml -[janitor] -max_checkpoints = 20 -``` - -Values below 1 are rejected. Checkpoint creation never prunes history - pruning is janitor-only, behind preview and confirmation. - -When old checkpoints are pruned, physical versions that were protected only by those checkpoints become eligible for deletion in the same janitor run (if not protected by active VDE refs or remaining checkpoints). - -## Expired VDE cleanup - -Non-active, non-detached VDEs older than `[janitor] retention_days` are pruned: - -```toml -[janitor] -retention_days = 30 -``` - -Active/default VDEs are always protected. This catches abandoned PR preview VDEs that were never promoted or cleaned up. - -## Detached VDE cleanup - -Detached VDEs (created by `sqb state detach`) are eligible for cleanup after `retention_days`: - -- `retention_days = 0` makes them eligible immediately -- Refs, function refs, and VDE row are deleted -- Checkpoint rows remain governed by checkpoint retention -- Physical versions newly unprotected by removed refs can be deleted in the same run - -Active VDE refs continue to protect physical versions even when detached refs are pruned. - -## State cleanup - -The janitor also prunes state-only artifacts: - -- **Migration backups**: old backup schemas are deleted, but the latest backup is always preserved -- **Expired locks**: lock rows with `expires_at` in the past are deleted; active locks are never touched - -## Execution order - -The janitor drops warehouse physical versions before pruning state rows. If a warehouse drop fails, the corresponding state refs are preserved so the janitor can retry on the next run. - -## Usage - -```bash -# Preview what would be cleaned -sqb janitor - -# Execute cleanup with confirmation -sqb janitor --auto-approve -``` - -The janitor shows a preview of all candidates (checkpoints, VDEs, physical versions, state items) and requires confirmation before executing any destructive operations. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/locks.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/locks.md deleted file mode 100644 index 1d147cde52..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/locks.md +++ /dev/null @@ -1,60 +0,0 @@ - - -# Locks - -> Advisory locks for concurrent access control. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments/locks - -Virtual mode uses advisory locks with TTL to prevent concurrent operations from conflicting. Locks are stored in the state database and are scoped to specific resources. - -## Lock types - -| Lock key | Protects | Acquired by | -|----------|----------|-------------| -| `virtual_env:` | VDE pointer set and views | build, promote, rollback, reconcile, adopt, detach | -| `model_version::` | Physical version creation | build (per model version) | -| `state_migration` | State schema changes | state init, migrate, rollback, reset | - -Different VDEs can be locked concurrently. Two builds targeting different VDEs do not block each other. - -## Lock behavior - -- Locks have an expiry time (`expires_at`). Active locks are those where `expires_at > now`. -- When a lock is successfully released, the lock row is deleted. -- Acquiring a lock over an expired lock replaces it. -- Owner identity is checked on release - only the owner that acquired the lock can release it. - -## When locks block - -If a lock is active and an operation requires it, the operation fails immediately with a clear error: - -``` -error[S014]: virtual environment 'dev' is locked -``` - -The operation does not wait or retry. This is intentional - SQLBuild does not implement lock queuing. If a lock is active, either wait for the holding operation to complete or clear the lock manually. - -## Inspecting locks - -List active locks: - -```bash -sqb state locks -``` - -## Clearing stuck locks - -If a process crashed while holding a lock, the lock may remain active until it expires. To clear it immediately: - -```bash -sqb state locks clear virtual_env:dev --force -``` - -Only clear locks when you are certain the holding operation is no longer running. Clearing a lock while the operation is still active can cause state corruption. - -## Lock expiry - -Locks are acquired with a TTL (typically 10 minutes for VDE locks). If an operation takes longer than the TTL, the lock expires and another operation can acquire it. This is a safety net against abandoned locks, not a normal operating condition. - -If operations consistently exceed the lock TTL, the TTL may need to be increased in a future configuration option. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/promotion.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/promotion.md deleted file mode 100644 index 7a4dad0d6e..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/promotion.md +++ /dev/null @@ -1,72 +0,0 @@ - - -# Promotion - -> Promote VDE refs and diff virtual environments. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments/promotion - -Promotion copies model version refs from one VDE to another and refreshes the target's logical views. No models are rebuilt - it is a pointer swap. - -## Basic usage - -```bash -sqb promote --from pr_123 --to dev -``` - -This updates VDE `dev` to point at the same physical versions that VDE `pr_123` uses. The target's logical views (`dev__dev.*` or `dev.*` if unsuffixed) are refreshed to point at the promoted physical relations. - -## What happens during promotion - -1. Target VDE lock is acquired -2. Source VDE is validated (finalized, current with workspace) -3. Source model refs are copied to target VDE refs -4. Source function refs are copied and function definitions are republished in the target schema -5. Target logical VDE views are refreshed -6. A checkpoint is created if the target is finalized -7. Target VDE lock is released - -## Partial promotion - -Promote a subset of models with `--select`: - -```bash -sqb promote --from pr_123 --to dev --select fact_orders -``` - -Partial promotion can leave the target VDE with stale downstream models - models that still point at older versions than the promoted scope expects. When this happens, promotion is blocked by default so you don't accidentally leave the target in a working state. - -To accept a working target, pass `--allow-partial-promotion`: - -```bash -sqb promote --from pr_123 --to dev --select fact_orders --allow-partial-promotion -``` - -The target VDE is marked `working` after the promotion. You can finalize it later by promoting or building the remaining models. - -If the models you select depend on upstream models that are themselves stale in the source, the promoted scope would not be coherent on its own. Pass `--include-stale-upstreams` to expand the selection to include the minimal set of required upstream models: - -```bash -sqb promote --from pr_123 --to dev --select fact_orders --include-stale-upstreams -``` - -## Source VDE requirements - -**Whole promotion** requires the source VDE to be finalized and current with the workspace (no stale models vs current code). If the source VDE has been built but the code has since changed, you need to rebuild the source VDE first or use partial promotion. - -**Partial promotion** does not require a finalized source. A working source VDE is allowed when using `--select` for a coherent scope. - -## Guards - -| Condition | Behavior | -|-----------|----------| -| Source VDE not finalized (whole promotion) | Blocks. Use `--select` for partial promotion from a working source. | -| Source VDE has stale models vs workspace | Blocks. Rebuild the source VDE or use `--select`. | -| Target VDE is locked | Blocks. Wait or clear the lock with `sqb state locks clear`. | -| Target VDE is detached | Blocks. Detached VDEs cannot be promoted to. | -| Source VDE is detached | Blocks. Detached VDEs cannot be promoted from. | -| Partial promotion leaves target working | Blocks unless `--allow-partial-promotion` is set. | - -## Comparing VDEs before promotion - -Use `sqb diff` to compare VDE ref sets before promoting. See [Diff](diff.md) for details. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/reconcile.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/reconcile.md deleted file mode 100644 index de75534193..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/reconcile.md +++ /dev/null @@ -1,74 +0,0 @@ - - -# Reconcile - -> Diagnose and repair drift between state and warehouse. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments/reconcile - -Reconcile detects and repairs inconsistencies between the virtual state store and the actual warehouse objects. - -## Report - -Run reconcile without a subcommand to get a diagnostic report: - -```bash -sqb reconcile --virtual-env dev -``` - -This inspects the VDE's refs, checks that logical views exist and point to the expected physical relations, and reports any issues without changing anything. - -## Repair view - -Recreate a logical VDE view from trusted state: - -```bash -sqb reconcile repair-view --virtual-env dev --model fact_orders -``` - -This runs `CREATE OR REPLACE VIEW` for the logical VDE view, pointing it at the physical version relation recorded in the VDE's refs. It is idempotent - running it when the view is already correct is a no-op. - -### Guards - -| Condition | Behavior | -|-----------|----------| -| Logical target is a table (not a view) | Blocks. Drop the table manually first, then retry. | -| Physical relation is missing | Blocks. Rebuild the model with `sqb build --select `. | -| Target VDE is locked | Blocks. Wait or clear the lock. | - -No confirmation is needed. The command is explicit and idempotent. - -## Attach - -Rebind a VDE model ref to a different tracked physical relation: - -```bash -sqb reconcile attach --virtual-env dev --model fact_orders \ - --physical-relation dev__sqb_physical.fact_orders__v_8f3a9c12 -``` - -This updates the VDE ref for `fact_orders` to point at the specified physical relation and refreshes the logical view. - -### Guards - -| Condition | Behavior | -|-----------|----------| -| Physical relation not tracked in state | Blocks. Only physical relations registered in SQLBuild state can be attached. | -| Physical relation tracked for a different model | Blocks. Cannot attach a relation that belongs to another model. | -| Logical target is a table (not a view) | Blocks. | -| Target VDE is locked | Blocks. | -| Wrong confirmation | Blocks. Refs remain unchanged. | - -Attach requires confirmation by default. Type the confirmation text when prompted, or cancel to leave refs unchanged. - -## When to use reconcile - -- **Missing logical views** after a failed build, promotion, or detach - use `repair-view` -- **Wrong physical version** if a VDE ref was corrupted or you need to manually override which version a model points to - use `attach` -- **Diagnostic inspection** before or after recovery operations - use the default report - -Reconcile records events in the `reconcile_events` state table for audit trail. - -## Limitations - -Reconcile repairs pointer/view state. It does not rebuild physical versions. If a physical relation is missing from the warehouse, the only remedy is rebuilding the model with `sqb build --select `. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/recovery.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/recovery.md deleted file mode 100644 index 7ebd25f17f..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/recovery.md +++ /dev/null @@ -1,127 +0,0 @@ - - -# Recovery - -> Diagnosing and recovering from failures in virtual mode. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments/recovery - -## Contents - -- State corruption -- Warehouse drift -- Lock conflicts -- Interrupted operations -- Mode guards -- Detached VDE guards -- Adopt guards -- Detach guards -- General recovery strategy - -Virtual mode has explicit recovery paths for common failure scenarios. SQLBuild blocks cleanly rather than leaving ambiguous state, and error messages point to the specific recovery action needed. - -## State corruption - -| Scenario | Error | Recovery | -|----------|-------|----------| -| Missing state table | "Cannot backup invalid state schema" on `state migrate` | `sqb state reset --auto-approve` then `sqb state init` | -| Missing state column | Same | Same | -| Wrong state column type | Same | Same | -| Deleted backup schema (explicit id) | Rollback blocks with backup-id error | Use a different backup or reset | -| All backups deleted | "No state backup is available for rollback" | `sqb state reset --auto-approve` then reinitialize | - -State corruption is detected by schema validation during `state migrate`. If the current state schema is invalid, SQLBuild refuses to back it up (to avoid persisting a broken snapshot) and directs you to reset. - -## Warehouse drift - -| Scenario | Detection | Recovery | -|----------|-----------|----------| -| Missing logical VDE view | `sqb reconcile` reports missing view | `sqb reconcile repair-view` for the affected model | -| Missing physical relation | `sqb reconcile` reports it, `repair-view` blocks | Rebuild with `sqb build --select` | -| Logical target is a table, not a view | `repair-view` and `attach` block | Drop the table manually, then `repair-view` | -| Checkpoint physical relation missing | `sqb rollback` blocks | Use a more recent checkpoint, or rebuild | -| Promoted physical relation missing | `sqb promote` blocks | Rebuild the source VDE first | - -## Lock conflicts - -| Scenario | Error | Recovery | -|----------|-------|----------| -| VDE locked by another process | "virtual environment is locked" | Wait for the other process, or clear the lock | -| Lock held by crashed process | Same | `sqb state locks clear` with `--force` | - -Only clear locks when you are certain the holding operation is no longer running. - -## Interrupted operations - -| Scenario | Behavior | Recovery | -|----------|----------|----------| -| Interrupted adopt | Failed operation recorded. Physical schema may have partial artifacts. | `sqb reconcile` to diagnose, manual repair if needed, retry. | -| Interrupted detach | Failed operation recorded. VDE stays finalized. Physical version preserved. | `sqb reconcile` to diagnose, retry detach. | -| Interrupted promotion | Failed operation recorded. Target VDE lock released. | Retry promote. | - -There is no automatic resume command for interrupted operations. SQLBuild records the failure in `state_operations` and `state_operation_events`, preserves VDE refs and checkpoints, and leaves recovery to the operator. - -To inspect a failed operation: - -```sql -SELECT operation_id, operation_type, status, virtual_environment_name -FROM sqlbuild_state.state_operations -WHERE status = 'failed'; - -SELECT operation_id, action, status, message -FROM sqlbuild_state.state_operation_events -WHERE operation_id = '' -ORDER BY created_at; -``` - -## Mode guards - -SQLBuild blocks operations that don't apply to the current mode: - -| Operation | Direct mode | Virtual mode | -|-----------|-------------|--------------| -| `sqb state` subcommands | Blocked | Allowed | -| `sqb build --no-tests --no-audits` | Allowed | Blocked (use `sqb build`) | -| `sqb build --defer-to` | Allowed | Blocked (VDE refs handle upstream resolution) | -| `sqb promote` | Blocked | Allowed | -| `sqb rollback` | Blocked | Allowed | -| `sqb reconcile` | Blocked | Allowed | - -## Detached VDE guards - -After `sqb state detach`, the VDE is marked `detached` and blocked from further virtual operations: - -| Operation | Behavior | -|-----------|----------| -| `sqb build` | Blocks: "virtual environment is detached" | -| `sqb promote --from ` | Blocks | -| `sqb promote --to ` | Blocks | -| `sqb rollback` | Blocks | - -The project can continue in direct mode or re-adopt to return to virtual mode. - -## Adopt guards - -| Condition | Error | -|-----------|-------| -| No `unsuffixed_virtual_env` configured | Blocks with config guidance | -| Copy fallback needed without `--allow-copy` | `requires --allow-copy` | -| Wrong typed confirmation | Cancelled, no changes made | - -## Detach guards - -| Condition | Error | -|-----------|-------| -| VDE not finalized | "requires a finalized virtual environment" | -| Copy fallback needed without `--allow-copy` | `requires --allow-copy` | -| Wrong typed confirmation | Cancelled, no changes made | - -## General recovery strategy - -1. **Diagnose** with `sqb reconcile` or by querying state tables directly -2. **Repair views** with `sqb reconcile repair-view` for missing/broken logical views -3. **Rebuild** with `sqb build --select ` for missing physical versions -4. **Roll back** with `sqb rollback` to restore a prior finalized state -5. **Reset state** with `sqb state reset --auto-approve` as a last resort (drops all virtual state) - -When in doubt, `sqb reconcile` is the starting point. It reports what's wrong without changing anything. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/rollback.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/rollback.md deleted file mode 100644 index 6a68b17923..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/rollback.md +++ /dev/null @@ -1,119 +0,0 @@ - - -# Rollback - -> Checkpoints and rollback for virtual environments. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments/rollback - -## Contents - -- Checkpoints -- Basic rollback -- Explicit checkpoint -- What happens during rollback -- Partial rollback -- Guards -- Checkpoint inspection - -Rollback restores a VDE to a prior finalized state by rebinding its refs to a stored checkpoint. - -## Checkpoints - -Checkpoints are created automatically when a VDE reaches `finalized` status: - -- After a whole build where all models match expected versions -- After a whole promotion where the target VDE is finalized - -Each checkpoint stores the complete set of model refs and function refs for that VDE at that point in time. Checkpoints are retained according to `[janitor] max_checkpoints` (default: 20). - -## Basic rollback - -Restore the previous finalized checkpoint for the default VDE: - -```bash -sqb rollback -``` - -For an explicit VDE: - -```bash -sqb rollback --virtual-env pr_123 -``` - -## Explicit checkpoint - -Restore a specific checkpoint: - -```bash -sqb rollback --checkpoint-id -``` - -Use `sqb state checkpoints list` to see available checkpoints. - -## What happens during rollback - -1. Target VDE lock is acquired -2. The target checkpoint is located (previous finalized by default, or explicit id) -3. Checkpoint physical relations are validated to still exist in the warehouse -4. VDE refs are replaced with the checkpoint's ref set -5. VDE function refs are replaced with the checkpoint's function ref set -6. Logical VDE views are refreshed to point at the restored physical versions -7. Function definitions are republished from the checkpoint's function versions -8. Target VDE lock is released - -## Partial rollback - -Roll back a subset of models with `--select`: - -```bash -sqb rollback --select fact_orders -``` - -Rolling back some models but not others can leave the VDE with stale models - models whose restored versions no longer line up with the rest of the VDE. When this happens, rollback is blocked by default so you don't leave the VDE in a working state by accident. - -To accept a working VDE, pass `--allow-partial-rollback`: - -```bash -sqb rollback --select fact_orders --allow-partial-rollback -``` - -The VDE is marked `working` after the rollback. You can finalize it later by building or rolling back the remaining models. - -If the models you select depend on upstream models that also need to be restored for the scope to be coherent, pass `--include-stale-upstreams` to expand the selection to the minimal set of required upstream refs from the checkpoint: - -```bash -sqb rollback --select fact_orders --include-stale-upstreams -``` - -## Guards - -| Condition | Behavior | -|-----------|----------| -| No previous checkpoint exists | Blocks. Build the VDE first to create a finalized checkpoint. | -| Checkpoint physical relations deleted | Blocks. The physical version tables have been cleaned up by janitor. Use a more recent checkpoint or rebuild. | -| Unknown checkpoint id | Blocks with error. | -| Target VDE is locked | Blocks. Wait or clear the lock. | -| Target VDE is detached | Blocks. Detached VDEs cannot be rolled back. | -| Partial rollback leaves VDE working | Blocks unless `--allow-partial-rollback` is set. | - -## Checkpoint inspection - -List checkpoints for a VDE: - -```bash -sqb state checkpoints list -sqb state checkpoints list --virtual-env pr_123 -``` - -Show a checkpoint's model refs: - -```bash -sqb state checkpoints show -``` - -Diff current VDE refs against a checkpoint: - -```bash -sqb state checkpoints diff -``` diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/setup.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/setup.md deleted file mode 100644 index 414cdcedd8..0000000000 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/virtual-environments/setup.md +++ /dev/null @@ -1,189 +0,0 @@ - - -# Setup - -> Configure virtual environments and initialize the state store. - -Online: https://docs.sqlbuild.com/concepts/virtual-environments/setup - -## Contents - -- Project configuration -- State configuration -- State initialization -- State lifecycle -- Local overrides -- Per-target state -- Unsuffixed VDE naming -- Janitor configuration -- State tables - -Virtual environments require two things: `virtual_environments = true` in your project settings, and a state store configured for your active target. - -## Project configuration - -```toml -name = "my_project" -adapter = "duckdb" -default_target = "dev" - -[settings] -virtual_environments = true - -[connections.local] -database = "warehouse.duckdb" - -[targets.dev] -connection = "local" -schema = "dev" - -[targets.dev.state] -backend = "duckdb" -schema = "sqlbuild_state" - -[targets.dev.state.connection] -database = "state.duckdb" -``` - -The `virtual_environments` setting switches the project from direct mode (default) to virtual mode. All state, plan, build, promote, rollback, and reconcile commands route through the virtual path when this is enabled. - -## State configuration - -Each physical target that uses virtual mode needs a `[targets..state]` block: - -| Field | Required | Description | -|-------|----------|-------------| -| `backend` | Yes | State store engine: `duckdb` or `postgres` | -| `schema` | Yes | Schema name for state tables | -| `connection` | Yes | Backend-specific connection config | -| `allow_reset` | No | Whether `sqb state reset` is permitted (default: `false`) | -| `unsuffixed_virtual_env` | No | VDE name that uses the base schema without a suffix (for [adopt/detach](adopt-detach.md)) | - -### DuckDB state backend - -```toml -[targets.dev.state] -backend = "duckdb" -schema = "sqlbuild_state" - -[targets.dev.state.connection] -database = "state.duckdb" -``` - -DuckDB state stores are file-based. Relative paths are resolved against the project directory. Suitable for local development and single-user workflows. Not recommended when multiple processes or CI jobs need concurrent state access. - -### Postgres state backend - -```toml -[targets.prod.state] -backend = "postgres" -schema = "sqlbuild_state" - -[targets.prod.state.connection] -host = "state-db.internal" -port = 5432 -user = "sqlbuild_state" -password = "${ENV:STATE_DB_PASSWORD}" -dbname = "sqlbuild_state" -``` - -Postgres is recommended for production. It supports concurrent access from multiple developers or CI jobs. - -## State initialization - -Before using virtual commands, initialize the state store: - -```bash -sqb state init -``` - -This creates the state tables in the configured schema. Run it once per physical target. - -## State lifecycle - -| Command | Description | -|---------|-------------| -| `sqb state init` | Create state tables | -| `sqb state migrate` | Back up current state and re-initialize tables | -| `sqb state rollback` | Restore from the latest backup | -| `sqb state rollback --backup-id ` | Restore from a specific backup | -| `sqb state reset --auto-approve` | Drop all state tables (requires `allow_reset = true`) | - -`state migrate` creates a backup schema (e.g. `sqlbuild_state__backup_`) before re-initializing. This lets you roll back if a migration causes problems. - -`state reset` is destructive and requires both `allow_reset = true` in config and `--auto-approve` on the command line. - -## Local overrides - -Use `sqlbuild_local.toml` to override state connection config per developer: - -```toml -[targets.dev.state.connection] -database = "local-state.duckdb" -``` - -This is useful when each developer uses a local state file while the project config points to a shared state database. - -## Per-target state - -Different physical targets can use different state backends: - -```toml -[targets.dev.state] -backend = "duckdb" -schema = "sqlbuild_state" - -[targets.dev.state.connection] -database = "state.duckdb" - -[targets.prod.state] -backend = "postgres" -schema = "sqlbuild_state" - -[targets.prod.state.connection] -host = "prod-state.internal" -dbname = "sqlbuild_state" -``` - -Each physical target has its own state store. VDEs are scoped to their physical target and cannot be promoted or compared across different physical targets. - -## Unsuffixed VDE naming - -By default, all VDE logical views use a suffixed schema: `dev__kevin.fact_orders`. For the primary VDE that consumers query directly (e.g. the production VDE), you typically want clean unsuffixed names: `dev.fact_orders`. - -Configure this with `unsuffixed_virtual_env`: - -```toml -[targets.dev.state] -backend = "duckdb" -schema = "sqlbuild_state" -unsuffixed_virtual_env = "dev" - -[targets.dev.state.connection] -database = "state.duckdb" -``` - -With this config, VDE `dev` uses `dev.fact_orders` while other VDEs like `kevin` use `dev__kevin.fact_orders`. - -This setting is also required for [adopt and detach](adopt-detach.md) operations so that existing consumer queries continue to work after migrating to virtual mode. - -## Janitor configuration - -When using virtual environments, configure the janitor to run periodically to clean up expired VDEs, old checkpoints, and unreferenced physical versions: - -```toml -[janitor] -enabled = true -retention_days = 30 -max_checkpoints = 20 -``` - -See [Janitor](janitor.md) for details on what gets cleaned up and how retention works. - -## State tables - -The state store contains current-state tables and append-only history tables. You do not need to interact with these directly, but understanding what is stored helps when debugging: - -**Current state:** `state_versions`, `model_versions`, `function_versions`, `physical_relations`, `physical_relation_ancestry`, `virtual_environments`, `virtual_environment_refs`, `virtual_environment_function_refs`, `virtual_environment_checkpoints`, `virtual_environment_checkpoint_refs`, `virtual_environment_checkpoint_function_refs`, `locks`, `state_operations` - -**History:** `plan_runs`, `virtual_environment_ref_events`, `reconcile_events`, `state_migration_events`, `state_operation_events` diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/feature-comparison.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/feature-comparison.md index 6364977ef6..8b2bcb405f 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/feature-comparison.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/feature-comparison.md @@ -4,7 +4,7 @@ > Feature comparison between SQLBuild, dbt, and SQLMesh. -Online: https://docs.sqlbuild.com/feature-comparison +Online: https://sqlbuild.com/docs/feature-comparison/ ## Contents @@ -68,18 +68,15 @@ SQLBuild, dbt, and SQLMesh are all SQL pipeline frameworks. They share common gr | Feature | SQLBuild | dbt | SQLMesh | |---------|----------|-----|---------| -| Stale-driven virtual builds | Optional `--changes-only` execution over VDE version bindings | dbt State (paid) | Version hash comparison | | Warehouse-native state | Append-only tables in the warehouse; no external state database | manifest.json artifacts | Requires external state store (SQLite/PostgreSQL) | | Source freshness | `sqb freshness` with adapter/column/sql strategies, lag tolerance, and CI gating | `dbt source freshness` | No dedicated freshness command; `signals` gate model evaluation until external data is ready | -| Reuse across environments | Virtual environments reuse fingerprint-matched physical tables across environments (shared physical storage) | dbt State clone (paid) | Virtual environments reuse fingerprint-matched physical tables across environments (shared physical storage) | | Cascade propagation | Topological walk with `replay_on_change` policy inheritance and override | No cascade control | Cascades through version hashes | ### Environments | Feature | SQLBuild | dbt | SQLMesh | |---------|----------|-----|---------| -| Virtual environments | Pointer swaps with hash-based version reuse (opt-in) | No | Pointer swaps, no compute cost | -| Data diffs | Full row-level data comparison across targets or virtual environments | No | Table diff | +| Data diffs | Full row-level data comparison across targets | No | Table diff | | Zero-copy cloning | `sqb clone` | No | No | ### Models @@ -131,9 +128,9 @@ SQLBuild, dbt, and SQLMesh are all SQL pipeline frameworks. They share common gr | Tool | Best for | |------|----------| -| **SQLBuild** | Rigor-first SQL pipelines: compile-time verification, pre-promotion audit gating, multi-model tests, and local E2E replay by default. Opt into change-aware builds and warehouse-native state when full rebuilds get expensive, plus ingestion, Python nodes, and virtual environments as the project grows. | +| **SQLBuild** | Rigor-first SQL pipelines: compile-time verification, pre-promotion audit gating, multi-model tests, and local E2E replay by default. Opt into change-aware builds and warehouse-native state when full rebuilds get expensive, plus ingestion and Python nodes as the project grows. | | **dbt** | The most widely adopted SQL transformation framework with the largest adapter and community ecosystem. | -| **SQLMesh** | State-managed pipelines with virtual environments, interval tracking, and cross-dialect transpilation. | +| **SQLMesh** | State-managed pipelines with interval tracking and cross-dialect transpilation. | ## Not yet in SQLBuild diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/index.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/index.md index 250334cbd4..eef8d5fb4b 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/index.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/index.md @@ -2,253 +2,58 @@ # Introduction -> Verify early, test properly, and deploy reversibly. SQL pipelines with the rigor of real software - free and open source. +> The refactorable warehouse. Verify early, test properly, and refactor safely. -Online: https://docs.sqlbuild.com/index +Online: https://sqlbuild.com/docs/ -## Contents +SQLBuild is a free, open-source framework for SQL and Python data pipelines. It brings compile-time +checks, tests and diffs to your warehouse, so you can change it as often as you change your code. -- Test properly -- Verify early -- Plans that explain every build -- Deploy reversibly (opt-in) -- Supported adapters -- More in SQLBuild -- What's next -- Quick links - -**Valid isn't the same as correct.** Your SQL compiles, runs, and returns rows - none of that means the number is right, and a silently-wrong number a stakeholder already trusted is the bug that actually hurts. - -SQLBuild brings software-engineering rigor to SQL pipelines: **verify early, test properly, deploy reversibly.** Catch errors before the warehouse runs them and test your logic locally - with change-aware builds and reversible deploys there when you need them, not forced on you on day one. It is a standalone framework for building SQL and Python data pipelines - free and open source. - -## Test properly - -Most data testing checks for nulls and uniqueness. That tells you a column isn't empty, not that your logic is right. SQLBuild tests the logic. - -- **Multi-model tests.** Mock your sources, assert on the model you care about, and SQLBuild resolves every intermediate model from its real SQL. One test file can cover a whole stretch of your pipeline. -- **E2E scenario tests with local replay.** Build the real model graph against coherent fixtures in isolated relations, capture the result as JSONL, and replay it locally through DuckDB. CI runs full end-to-end tests with zero warehouse credentials or compute. -- **Macro-powered mocks.** Tests are SQL, so they support macro calls - write reusable mock generators instead of copy-pasting fixtures. - -```sql --- Mock two sources, assert on the final mart. --- stg_orders and stg_payments resolve automatically from their real SQL. -TEST(); - -WITH -__source__raw__orders AS ( - @mock_orders() -), -__source__raw__payments AS ( - SELECT - 1 AS payment_id, - 1 AS order_id, - 1500 AS amount_cents, - 'credit_card' AS method -), -__expected__fact_orders AS ( - SELECT - 1 AS order_id, - 100 AS customer_id, - 1500 AS total_cents, - 'credit_card' AS payment_method -) -SELECT 1 -``` - -See [Testing](concepts/testing.md) for SQL unit tests and [Scenarios](concepts/scenarios.md) for local -E2E replay. +Valid SQL isn't the same as correct SQL. A query can compile, run and return rows, and still produce +a wrong number that people already trust. SQLBuild is built to catch that before it reaches +production. ## Verify early -Before any model runs, SQLBuild does static analysis of your project - offline, no warehouse connection needed. - -- **Catch errors at compile.** SQL syntax, type inference, contract checks, and column lineage all run before execution. A bad reference or a type mismatch fails at compile, with an error that points at the line - not halfway through a warehouse run. -- **Turn review decisions into compiler checks.** Configure [Rules](concepts/rules.md) over compiler-owned SQL, models, dependencies, contracts, tests, and project paths. -- **Format separately.** `sqb format` owns deterministic source rewriting; Rules remain diagnostic-only. -- **Fast, because it's Rust where it counts.** Static analysis runs on [Polyglot](https://github.com/tobilg/polyglot), a Rust SQL engine (MIT, 32+ dialects), so compile stays quick even on large projects. -- **Open, not paywalled.** The static analysis is part of the Apache-2.0 core - no proprietary engine, no separate login, and no paid tier gating the smart checks. - -## Plans that explain every build - -**Start simple.** By default, `sqb build` runs your full selection - the same predictable mental model as dbt, with nothing to configure. For many projects, that is all they will ever need. - -Every model, seed, function, and Python node has a versioned identity. SQLBuild uses those identities -to explain query and configuration changes, propagate upstream changes, choose incremental replay -actions, and protect scoped builds from stale dependencies. Direct mode still runs the selected -scope; change detection makes that plan explicit rather than silently changing what was selected. - -[Virtual environments](concepts/virtual-environments.md) add version reuse and optional stale-driven -execution when a project needs isolated previews, promotion, and rollback. - -The planner carries that evidence across the whole graph: - -- **Models and functions:** query, configuration, and dependency identities explain local and cascaded changes. -- **Seeds:** content and load-affecting configuration identify when a seed changed. -- **Audits:** audit evidence is associated with the model version it validated. -- **Source freshness:** observable source data versions can affect downstream plans, with lag tolerance to avoid jitter. -- **Cascade propagation:** when a model does change, the signal propagates downstream, with configurable replay windows (`replay_on_change`). -- **Python nodes:** source and dependency identities appear in plans; skip/run remains user-controlled through `ctx.skip()`. - -In direct mode, this evidence is stored as append-only rows in your warehouse. Virtual environments -use their configured state backend for environment-scoped version bindings. - -## Deploy reversibly (opt-in) - -By default, SQLBuild runs in direct mode with state as append-only rows in your warehouse. When you want more, [virtual environments](concepts/virtual-environments.md) add a reversibility layer on top: - -- **Instant branching, promotion, and rollback** as low-copy pointer operations. -- **Partial promotion.** Promote the models that are ready without re-running everything downstream of them - you don't have to rebuild the whole closure to ship one fix. -- **Checkpoints and reconciliation** so a bad change is something you undo, not an incident. - -Virtual environments are opt-in, not a tax you pay upfront - direct mode stays the default, so the floor stays low and you reach for them only when a workflow needs them. - -## Supported adapters - -SQLBuild works with **DuckDB, MotherDuck, Snowflake, BigQuery, Databricks, PostgreSQL, and SQL Server** today. Support for ClickHouse, Redshift, Trino, Spark, and Athena is on the way. - -DuckDB runs entirely locally, so you can try SQLBuild and run full E2E tests without any warehouse credentials. Head to the [Quickstart](quickstart.md) to get a project running in minutes, or see [Adapters](concepts/adapters.md) for connection setup. - -## More in SQLBuild - -### Familiar SQL models - -Models are SQL files with a `MODEL()` header and a `SELECT`. References to other models and sources use `__ref()` and `__source()`, and configuration, schema, and audits are declared inline in the header. If you know dbt or SQLMesh, you already know the shape. - -```sql --- models/marts/fact_orders.sql -MODEL ( - materialized table, - columns ( - order_id (audits [not_null, unique]), - ), -); +Before anything runs, `sqb compile` analyses the whole project offline, with no warehouse +connection. It checks SQL, model references and declared [contracts](concepts/models/contracts.md), +and traces [column lineage](concepts/column-lineage.md). [Rules](concepts/rules.md) turn +decisions from code review into compile errors, and +[declaration scopes](concepts/declaration-scopes.md) keep enums, constants and macros visible only +where they are used. -SELECT - o.order_id, - o.customer_id, - p.amount_cents AS total_cents, - p.method AS payment_method -FROM __ref("stg_orders") o -JOIN __ref("stg_payments") p USING (order_id) -``` - -### Python macros, not Jinja - -- **Real Python functions:** Macros are plain Python, called with `@macro()` in SQL. Testable, debuggable, and composable with standard tooling - no Jinja, no string-templating language to wrangle. - -```python -# macros/grant_target.py -def grant_target(target): - return f"GRANT SELECT ON {target} TO analyst_role" -``` - -### Audits that block bad data - -- **Full table builds:** SQLBuild materializes into a staging table and runs `error`-severity audits before promotion. If any fail, the swap is blocked and the production table is untouched. -- **Incremental models:** Delta-phase audits validate each batch before DML is applied. Bad data is caught before it reaches the target. -- **Measured quality:** Measurement audits evaluate values against warning/error thresholds, distinguish insufficient samples, and can retain bounded evidence and immutable result history. - -### Incremental processing - -- **Cursor-based replay:** SQLBuild resumes by reading the highest timestamp or integer value already in the target table. If a model fails for several runs, the next successful build picks up from the last data it actually wrote, with no manual backfilling. -- **Microbatch mode:** Split large replay windows into configurable batches, each with its own audit cycle. Or process the full range in one pass, the choice is per-model. -- **Replay on change:** When a model's version identity changes, `replay_on_change` controls how much data to reprocess: `forward` (default, just run the next delta), `bounded-14d` (replay a window), or `full` (rebuild the table). - -### Python you can read and extend - -- **The framework is Python.** Adapters, macros, hooks, providers, custom materializations, and Python nodes are all plain Python you can read and extend. (The SQL analysis underneath runs on Rust via Polyglot - see [Verify early](#verify-early) - so the code you work with stays Python while compile stays fast.) - -### Testing in depth - -Beyond the multi-model tests shown above, SQLBuild runs end-to-end scenario tests against the real model graph, with local replay and property assertions. - -- **Real graph execution:** Define coherent fixture inputs, and SQLBuild builds the real model graph against them in isolated warehouse relations. Test end-to-end business logic across your entire pipeline, not just one model at a time. -- **Local replay without a warehouse:** Capture scenario fixtures as JSONL snapshots, then replay them locally through DuckDB. CI pipelines run full E2E tests with zero warehouse credentials or compute cost. -- **Zero-row assertions:** Write property checks that pass when no rows violate a condition. Useful for invariants like "no negative revenue" or "no duplicate customer IDs" alongside full expected-output comparisons. - -```sql -SCENARIO ( - description "Daily revenue includes only successful payments", - tags ["revenue"] -); - -WITH -__ref__stg_orders AS ( - SELECT 1 AS order_id, 10 AS customer_id, 'completed' AS status -), -__ref__stg_payments AS ( - SELECT - 1 AS payment_id, - 1 AS order_id, - 1700 AS amount_cents, - 'success' AS payment_status -), -__expected__daily_revenue AS ( - SELECT - CAST('2026-04-01' AS DATE) AS revenue_date, - 1700 AS total_revenue_cents -), -__assert__all_orders_have_payments AS ( - SELECT * FROM __ref("fact_orders") WHERE payment_amount_cents IS NULL -) -SELECT 1 -``` - -### Python nodes - -Grow beyond warehouse-only SQL without leaving the graph. Python nodes are ordinary functions, decorated to become first-class nodes in the same DAG as your SQL models, and they run as part of `sqb build`. There are four kinds: - -- **Loaders** (`@loader`) load external data into managed sources, with incremental write strategies (table, append, delete_insert, merge), cursor-based loading, and concurrent execution. -- **Tasks** (`@task`) run Python computation or side effects as graph nodes. -- **Assets** (`@asset`) produce or observe external artifacts, with optional columns and lineage. -- **Checks** (`@check`) validate tasks, assets, and loaders, and run during `sqb build` or on their own with `sqb check`. - -```python -from sqlbuild.loaders import loader -from sqlbuild.executor.load.models import LoaderContext - -@loader -def raw_orders(ctx: LoaderContext): - if ctx.current_cursor_value is None: - return fetch_all_orders() - return fetch_orders_since(ctx.current_cursor_value) -``` - -SQL models never depend on Python nodes - the only path from Python into SQL is a loader populating a source - so the SQL graph stays fully analyzable on its own. Nodes can also be generated programmatically with `@factory`. See [Python Nodes](concepts/python-nodes/overview.md). - -### Multi-target workflows - -- **Data diffs:** Compare schemas and row-level data between targets (or virtual environments) with `sqb diff prod:dev`. - -- **Zero-copy cloning:** Branch targets instantly with `sqb clone` without duplicating data. -- **Deferred references:** Resolve a production namespace with `--defer-to` through the active target's connection while building in dev. -- **No manifest required:** Clone, diff, and defer work directly against live targets. No `manifest.json` generation, no artifact management, no stale state. - -### Extensibility +## Test properly -- **User-defined functions:** SQL and Python UDFs managed as part of your project. Functions participate in the DAG - definition changes trigger rebuilds of dependent models. Table functions provide predicate-pushdown-friendly alternatives to final-layer views. -- **Custom materializations:** Write materialization logic in Python with full framework integration - including audit hooks, schema change signals, and query change detection. +[SQL unit tests](concepts/testing.md) mock your sources, resolve every model in between from its +real SQL, and compare the output with what you expect. [Scenarios](concepts/scenarios.md) build the +real model graph against test data, then replay it locally on DuckDB in CI. +[Audits](concepts/audits.md) check each build before it replaces production. -```python -def materialize(ctx: MaterializationContext) -> MaterializationResult: - stale = find_untracked_partitions(ctx) +## Refactor safely - for partition in stale: - ctx.adapter.create_table_as(ctx.connection, target=staging, sql=partition_sql) - ctx.run_audits(staging) # same audit guarantees as built-in types - ctx.execute_sql(f"INSERT INTO {ctx.destination} SELECT * FROM {staging}") +[`sqb plan`](concepts/planning.md) shows why each model will run and how much it will rebuild. +[`sqb diff`](concepts/diff.md) compares real data between targets before a change ships. When you +rename a model, a [migration](concepts/models/migrations.md) moves its table instead of rebuilding +its history, and the [janitor](cli/janitor.md) archives tables the project no longer builds before +it deletes them. - return MaterializationResult(relation=ctx.destination) -``` -- **Path-between selectors:** `--select fact_orders~daily_activity_rollup` selects every model on the shortest path between two nodes, with optional upstream/downstream expansion. +## How it fits together -## What's next +| Step | Command | What it does | +|------|---------|--------------| +| Compile | [`sqb compile`](cli/compile.md) | Checks the project offline | +| Test | [`sqb test`](cli/test.md) | Runs SQL unit tests | +| Plan | [`sqb plan`](cli/plan.md) | Shows what will run, why, and how much it will rebuild | +| Build | [`sqb build`](cli/build.md) | Runs tests, builds models in dependency order, and blocks bad data with audits | +| Compare | [`sqb diff`](cli/diff.md) | Compares schemas and rows between two targets | -- **Broader adapter support** - ClickHouse, Redshift, Trino, Spark, Athena +SQLBuild works with DuckDB, MotherDuck, Snowflake, BigQuery, Databricks, PostgreSQL and SQL Server. +DuckDB runs locally, so you can try everything without warehouse credentials. See +[Adapters](concepts/adapters.md) for connection setup. -## Quick links +## Where to start - Get a project running locally in minutes. - Reference for SQLBuild commands and flags. - Understand models, incremental strategies, audits, and selectors. - E2E tests with local DuckDB replay. + Build a complete example project locally in a minute. + Run SQLBuild alongside an existing dbt project. + Models, tests, audits, planning and the rest. + Every command and flag. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster-reference.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster-reference.md index 68c5e8c130..c7a4339a9d 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster-reference.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster-reference.md @@ -4,7 +4,7 @@ > Dagster integration classes, decorators, and translator hooks. -Online: https://docs.sqlbuild.com/integrations/dagster-reference +Online: https://sqlbuild.com/docs/integrations/dagster-reference/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster.md index 18fa403805..3d224501da 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dagster.md @@ -4,7 +4,7 @@ > Orchestrate SQLBuild pipelines with Dagster scheduling, retries, and asset UI. -Online: https://docs.sqlbuild.com/integrations/dagster +Online: https://sqlbuild.com/docs/integrations/dagster/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dlt.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dlt.md index a3409ba7d4..518cd79879 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dlt.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/dlt.md @@ -4,7 +4,7 @@ > Declarative dlt sources in YAML, or full dlt pipelines inside Python source loaders. -Online: https://docs.sqlbuild.com/integrations/dlt +Online: https://sqlbuild.com/docs/integrations/dlt/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/ingestr.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/ingestr.md index 68e355cbf3..9621233a2c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/ingestr.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/ingestr.md @@ -4,7 +4,7 @@ > Declarative data ingestion from 50+ sources using ingestr. -Online: https://docs.sqlbuild.com/integrations/ingestr +Online: https://sqlbuild.com/docs/integrations/ingestr/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/rivers.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/rivers.md index d028d112ec..6608230229 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/rivers.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/integrations/rivers.md @@ -4,7 +4,7 @@ > Orchestrate SQLBuild pipelines with Rivers scheduling, jobs, and asset tracking. -Online: https://docs.sqlbuild.com/integrations/rivers +Online: https://sqlbuild.com/docs/integrations/rivers/ ## Contents diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/quickstart.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/quickstart.md index 5d7117be89..761f0c7509 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/quickstart.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/quickstart.md @@ -4,7 +4,7 @@ > Get a SQLBuild project running locally with DuckDB in under a minute. -Online: https://docs.sqlbuild.com/quickstart +Online: https://sqlbuild.com/docs/quickstart/ ## Contents @@ -39,7 +39,7 @@ The playground creates the shared `sqlbuild_project.toml`; that is sufficient fo quickstart. In a team project, you can additionally create a gitignored `sqlbuild_local.toml` to select your usual target and override your personal development schema or credentials without changing the committed project config. See -[Project Configuration](concepts/project-configuration.md#sqlbuild_local-toml). +[Project Configuration](concepts/project-configuration.md#sqlbuild_localtoml). ## 2. Plan diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/roadmap.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/roadmap.md new file mode 100644 index 0000000000..25ad459e93 --- /dev/null +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/roadmap.md @@ -0,0 +1,30 @@ + + +# Roadmap + +> What's coming next in SQLBuild. + +Online: https://sqlbuild.com/docs/roadmap/ + +SQLBuild is free and open source, and will stay that way. There is no paid tier, no commercial edition +and no feature held back for one. Everything below ships in the open-source project and works on its own. + +## Next: the SQLBuild UI + +A self-hosted web app for your SQLBuild project: + +- Browse models, lineage and column types. +- Follow runs, tests and audits as they happen. +- Run builds and set them on a schedule. +- See what your builds cost. +- Sign in with SSO, with fine-grained permissions and audit logs. + +## Later + +- Moving BigQuery, Databricks and SQL Server from beta to supported. +- More warehouses, such as ClickHouse, Redshift and Trino. + +## Not planned + +- **Virtual environments.** They add a lot of complexity, and most of what they offer can be done more + simply in other ways, without the need for an external state store. diff --git a/src/sqlbuild/runtime/compute_logs/README.md b/src/sqlbuild/runtime/compute_logs/README.md index 8534f6c998..bc8c8b8e69 100644 --- a/src/sqlbuild/runtime/compute_logs/README.md +++ b/src/sqlbuild/runtime/compute_logs/README.md @@ -16,4 +16,4 @@ ownership and the process umask. It is host-local and may contain sensitive outp provide object storage, lifecycle authority, or a cross-host log service. Operator documentation is at -[docs.sqlbuild.com/concepts/observability](https://docs.sqlbuild.com/concepts/observability). +[sqlbuild.com/docs/concepts/observability](https://sqlbuild.com/docs/concepts/observability/). diff --git a/src/sqlbuild/runtime/event_exporting/README.md b/src/sqlbuild/runtime/event_exporting/README.md index 68ff9dfb11..e405764971 100644 --- a/src/sqlbuild/runtime/event_exporting/README.md +++ b/src/sqlbuild/runtime/event_exporting/README.md @@ -75,4 +75,4 @@ not import project providers or construct exporter queue, dispatcher, or notific Normal command-owned project discovery remains responsible for providers needed by nodes and hooks. The project declaration, configuration, delivery, security, and failure contract is documented at -[docs.sqlbuild.com/concepts/observability/sinks](https://docs.sqlbuild.com/concepts/observability/sinks). +[sqlbuild.com/docs/concepts/observability/sinks](https://sqlbuild.com/docs/concepts/observability/sinks/). diff --git a/src/sqlbuild/runtime/observability/README.md b/src/sqlbuild/runtime/observability/README.md index d2ed7e72e0..5534d64d6f 100644 --- a/src/sqlbuild/runtime/observability/README.md +++ b/src/sqlbuild/runtime/observability/README.md @@ -122,4 +122,4 @@ never copied into the integration stream. They remain available through final ag compute logs after process exit. Operator documentation is at -[docs.sqlbuild.com/concepts/observability](https://docs.sqlbuild.com/concepts/observability). +[sqlbuild.com/docs/concepts/observability](https://sqlbuild.com/docs/concepts/observability/). diff --git a/tests/unit/scripts/docs_skill/_helpers/_test_types.py b/tests/unit/scripts/docs_skill/_helpers/_test_types.py index 135fb2e584..3fd455a9ce 100644 --- a/tests/unit/scripts/docs_skill/_helpers/_test_types.py +++ b/tests/unit/scripts/docs_skill/_helpers/_test_types.py @@ -4,15 +4,16 @@ @dataclass(frozen=True) class OrderedNavigationPagesTestCase: description: str - docs_json: str + sidebar_json: str page_paths: tuple[str, ...] expected_paths: tuple[str, ...] + expected_labels: tuple[str, ...] @dataclass(frozen=True) class ReferencePagesTestCase: description: str - docs_json: str + sidebar_json: str pages: dict[str, str] stale_page: str expected_files: tuple[str, ...] diff --git a/tests/unit/scripts/docs_skill/_helpers/test_navigation.py b/tests/unit/scripts/docs_skill/_helpers/test_navigation.py index 01b6d2d650..13f8dc2398 100644 --- a/tests/unit/scripts/docs_skill/_helpers/test_navigation.py +++ b/tests/unit/scripts/docs_skill/_helpers/test_navigation.py @@ -2,7 +2,7 @@ import pytest -from scripts.docs_skill._helpers.navigation import list_ordered_page_paths +from scripts.docs_skill._helpers.navigation import list_navigation_groups, list_ordered_page_paths from tests.unit.scripts.docs_skill._helpers._test_types import ( OrderedNavigationPagesTestCase, ) @@ -12,21 +12,26 @@ "test_case", [ OrderedNavigationPagesTestCase( - description="group root precedes child pages and unlisted pages", - docs_json=( - '{"navigation":{"groups":[{"group":"Hooks",' - '"root":"concepts/hooks","pages":["concepts/hooks/sql"]}]}}' + description="nested sidebar groups preserve ordering with index and unlisted pages", + sidebar_json=( + '[{"label":"Start","items":["docs"]},{"label":"Hooks","items":[' + '{"label":"Overview","slug":"docs/concepts/hooks"},' + '{"label":"Nested","collapsed":true,"items":["docs/concepts/hooks/sql",' + '"docs/missing","docs/concepts/hooks"]}]}]' ), page_paths=( + "index.mdx", "concepts/hooks.mdx", "concepts/hooks/sql.mdx", - "concepts/unlisted.mdx", + "concepts/unlisted.md", ), expected_paths=( + "index.mdx", "concepts/hooks.mdx", "concepts/hooks/sql.mdx", - "concepts/unlisted.mdx", + "concepts/unlisted.md", ), + expected_labels=("Start", "Hooks", "Other pages"), ), ], ids=lambda case: case.description, @@ -35,16 +40,24 @@ def test_given_navigation_root_when_listing_pages_then_root_precedes_children( test_case: OrderedNavigationPagesTestCase, tmp_path: Path, ) -> None: - (tmp_path / "docs.json").write_text(test_case.docs_json, encoding="utf-8") + sidebar_path: Path = tmp_path / "sidebar.json" + sidebar_path.write_text(test_case.sidebar_json, encoding="utf-8") page_path: str for page_path in test_case.page_paths: target: Path = tmp_path / page_path target.parent.mkdir(parents=True, exist_ok=True) target.write_text("---\ntitle: Test\n---\n", encoding="utf-8") - result: list[Path] = list_ordered_page_paths(docs_root=tmp_path) + result: list[Path] = list_ordered_page_paths(docs_root=tmp_path, sidebar_path=sidebar_path) assert tuple(path.as_posix() for path in result) == test_case.expected_paths + assert ( + tuple( + group.label + for group in list_navigation_groups(docs_root=tmp_path, sidebar_path=sidebar_path) + ) + == test_case.expected_labels + ) if __name__ == "__main__": diff --git a/tests/unit/scripts/docs_skill/_helpers/test_render.py b/tests/unit/scripts/docs_skill/_helpers/test_render.py index 577a474981..c5b61bcd75 100644 --- a/tests/unit/scripts/docs_skill/_helpers/test_render.py +++ b/tests/unit/scripts/docs_skill/_helpers/test_render.py @@ -2,9 +2,8 @@ import pytest -from scripts.docs_skill._helpers.output import write_reference_pages -from scripts.docs_skill._helpers.render import build_reference_pages from scripts.docs_skill.constants import GENERATED_MARKER +from scripts.docs_skill.main.generate import generate_docs_skill from tests.unit.scripts.docs_skill._helpers._test_types import ReferencePagesTestCase @@ -13,16 +12,22 @@ [ ReferencePagesTestCase( description="pages become files with relative links and a grouped index", - docs_json=( - '{"navigation":{"groups":[' - '{"group":"Concepts","pages":["concepts/diff","concepts/models"]},' - '{"group":"CLI Reference","pages":["cli/diff"]}]}}' + sidebar_json=( + '[{"label":"Start","items":["docs"]},' + '{"label":"Concepts","items":["docs/concepts/diff","docs/concepts/models"]},' + '{"label":"CLI Reference","items":["docs/cli/diff"]}]' ), pages={ + "index.mdx": '---\ntitle: "Introduction"\n---\n\nWelcome.\n', "concepts/diff.mdx": ( '---\ntitle: "Data Diffs"\ndescription: "Compare data."\n---\n\n' - "See [models](/concepts/models#schemas), [the CLI](/cli/diff) and " - "[elsewhere](/missing/page).\n\n\nkept text\n\n" + "import Note from '../../../../components/mintlify/Note.astro';\n" + "See [models](/docs/concepts/models/#schemas), [the CLI](/docs/cli/diff) and " + "[elsewhere](/docs/missing/page#details).\n\n\nkept text\n\n" + "[docs](/docs/) [home](/) [LLMs](/llms.txt)\n" + '\n\n' + "Card body.\n\n\n\nSteps body.\n\n" + '```python\n\n```\n' ), "concepts/models.mdx": '---\ntitle: "Models"\n---\n\n## Schemas\n\nBody.\n', "cli/diff.mdx": '---\ntitle: "diff"\ndescription: "Diff command."\n---\n\nUsage.\n', @@ -33,15 +38,26 @@ "cli/diff.md", "concepts/diff.md", "concepts/models.md", + "index.md", ), expected_fragments=( ("concepts/diff.md", GENERATED_MARKER), ("concepts/diff.md", "# Data Diffs"), ("concepts/diff.md", "> Compare data."), - ("concepts/diff.md", "Online: https://docs.sqlbuild.com/concepts/diff"), + ("concepts/diff.md", "Online: https://sqlbuild.com/docs/concepts/diff/\n"), + ("index.md", "Online: https://sqlbuild.com/docs/\n"), ("concepts/diff.md", "[models](models.md#schemas)"), ("concepts/diff.md", "[the CLI](../cli/diff.md)"), - ("concepts/diff.md", "[elsewhere](https://docs.sqlbuild.com/missing/page)"), + ( + "concepts/diff.md", + "[elsewhere](https://sqlbuild.com/docs/missing/page/#details)", + ), + ("concepts/diff.md", "[docs](../index.md)"), + ("concepts/diff.md", "[home](https://sqlbuild.com/)"), + ("concepts/diff.md", "[LLMs](https://sqlbuild.com/llms.txt)"), + ("concepts/diff.md", "Card body."), + ("concepts/diff.md", "Steps body."), + ("concepts/diff.md", '```python\n\n```'), ("concepts/diff.md", "kept text"), ("CONTENTS.md", "## Concepts"), ( @@ -59,18 +75,21 @@ def test_given_docs_pages_when_writing_references_then_each_page_is_a_linked_fil test_case: ReferencePagesTestCase, tmp_path: Path, ) -> None: - docs_root: Path = tmp_path / "docs" + docs_root: Path = tmp_path / "src/content/docs/docs" output_dir: Path = tmp_path / "references" / "docs" - (docs_root).mkdir() - (docs_root / "docs.json").write_text(test_case.docs_json, encoding="utf-8") + docs_root.mkdir(parents=True) + (tmp_path / "src/sidebar.json").write_text(test_case.sidebar_json, encoding="utf-8") for relative_path, contents in test_case.pages.items(): (docs_root / relative_path).parent.mkdir(parents=True, exist_ok=True) (docs_root / relative_path).write_text(contents, encoding="utf-8") (output_dir / test_case.stale_page).parent.mkdir(parents=True, exist_ok=True) (output_dir / test_case.stale_page).write_text(f"{GENERATED_MARKER}\nold\n", encoding="utf-8") - write_reference_pages(output_dir=output_dir, pages=build_reference_pages(docs_root=docs_root)) + result: int = generate_docs_skill( + ["--docs-root", str(docs_root), "--output-dir", str(output_dir)] + ) + assert result == 0 written: dict[str, str] = { path.relative_to(output_dir).as_posix(): path.read_text(encoding="utf-8") for path in sorted(output_dir.rglob("*.md")) @@ -81,6 +100,10 @@ def test_given_docs_pages_when_writing_references_then_each_page_is_a_linked_fil for name, fragment in test_case.expected_fragments } assert found == dict.fromkeys(test_case.expected_fragments, True) + assert "import Note" not in written["concepts/diff.md"] + assert ": [flags] +# Model mode +sqb diff : --select [flags] + +# Query mode +sqb diff --left-query --right-query (--key | --unkeyed) [flags] +sqb diff --left-query --right-query --schema-only [flags] ``` -The first argument is a positional `FROM:TO` range. Exactly one mode is required: `--full`, `--schema-only`, or `--bounded `. +In model mode, the first argument is a positional `FROM:TO` range, `--select` is required, and exactly one mode is required: `--full`, `--schema-only`, or `--bounded `. `FROM` and `TO` are configured target names. Their database/schema namespaces remain authoritative, while the `TO` target's named connection executes the complete comparison and must be able to read both namespaces. -Full and bounded row comparisons require the model to define `unique_key`. Bounded mode uses the model's cursor and falls back to a full row comparison when no cursor is configured. +Full and bounded row comparisons match rows on the model's `unique_key`, or on `--key` columns when given; use `--unkeyed` for an exact full-row comparison. Bounded mode uses the model's cursor and falls back to a full row comparison when no cursor is configured. + +In query mode, omit `FROM:TO` and model selectors; each query identifies its own input and the active target's connection (or `--target`) runs both. A full comparison is the default and needs `--key` or `--unkeyed`; `--schema-only` takes neither, and `--bounded` is model-only. ## Flags @@ -36,7 +43,19 @@ Full and bounded row comparisons require the model to define `unique_key`. Bound | `--max-models` | Fail when the selected scope contains more models than this limit | | `--max-columns` | Fail when either side of a model has more columns than this limit | | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | -| `--select`, `-s` | Select specific models to diff (required in v1) | +| `--key` | Row key column; repeat for a composite key. Overrides the model's `unique_key` | +| `--unkeyed` | Compare exact full-row multisets when there is no stable row key | +| `--exclude-column` | Leave a column out of the comparison; repeatable | +| `--tolerance` | Numeric tolerance such as `amount:absolute=1` or `rate:relative=0.001`; repeatable | +| `--left-query`, `--right-query` | Compare two read-only SQL queries instead of models | +| `--left-query-file`, `--right-query-file` | Read each query from a UTF-8 file | +| `--left-label`, `--right-label` | Names for the two query sides in the output | +| `--target` | Target whose connection runs both queries | +| `--max-value-length` | Maximum characters shown per example value (default 160) | +| `--no-example-values` | Keep keys and counts but hide example values | +| `--full-example-values` | Show complete example values | +| `--json` | Print one JSON document to stdout | +| `--select`, `-s` | Select specific models to diff (required in model mode) | | `--exclude` | Exclude specific models from diffing | ## Examples @@ -56,8 +75,11 @@ sqb diff prod:dev --bounded 14d --sample-rows 50000 --sample-seed 7 --select ord # Force exhaustive comparison despite inherited sampling defaults sqb diff prod:dev --bounded 14d --exhaustive --select order_lines + +# Compare two queries on a key +sqb diff --left-query-file queries/orders_before.sql --right-query-file queries/orders_after.sql --key order_id ``` ## Exit codes -Returns `0` when all selected models have no differences, `1` when any model has schema or row differences. +Returns `0` when all selected models have no differences, `1` when any model has schema or row differences. Query mode also returns `2` when a safety, schema, or key check prevented a complete comparison, and `3` when setup, execution, or cleanup failed; see [Query output and exit codes](/docs/concepts/diff#query-output-and-exit-codes). diff --git a/website/src/content/docs/docs/cli/format.mdx b/website/src/content/docs/docs/cli/format.mdx index 6d182ca19e..f60577016f 100644 --- a/website/src/content/docs/docs/cli/format.mdx +++ b/website/src/content/docs/docs/cli/format.mdx @@ -28,3 +28,30 @@ sqb format --diff --select tag:marts Formatting owns presentation and source rewriting. Compiler-integrated diagnostics belong to configured [Rules](/docs/concepts/rules), which never mutate source. + +## Line width and descriptions + +Long model and scenario descriptions are reflowed deterministically. Ordinary authored line breaks +are normalized as spaces, while blank lines preserve paragraph boundaries. Configure the maximum +physical line width in `sqlbuild_project.toml` (the default is `100`): + +```toml +[format] +line_width = 100 +``` + +## What formatting preserves + +Formatting preserves authored cast types, postfix casts, quoted literals, variant paths, typed +lambda parameters, and supported SQL function spellings while applying canonical layout. SQLBuild +calls keep their authored spelling, including zero-argument cursor and empty-fixture intrinsics. +CTE-producing macros remain authored calls rather than expanded project SQL, with each call on its +own CTE-list line and leading comments attached to the node they describe. + +## Unsafe bodies + +`sqb format` reports a file-specific `format-unsafe` fault whenever a SQL body cannot be safely +formatted, including parser, comment-attachment, interpolation-restoration, and idempotence +failures. The entire original file is kept (including its headers and fixtures), the reason +appears in human and JSON output, and both formatting and `sqb format --check` exit nonzero. A +declined body is never counted as canonical. diff --git a/website/src/content/docs/docs/concepts/adapters/snowflake.mdx b/website/src/content/docs/docs/concepts/adapters/snowflake.mdx index 1e955f1e31..f3a12b355d 100644 --- a/website/src/content/docs/docs/concepts/adapters/snowflake.mdx +++ b/website/src/content/docs/docs/concepts/adapters/snowflake.mdx @@ -63,3 +63,27 @@ connection = "warehouse" database = "DEV_DB" schema = "dev" ``` + +## Cost estimates + +Snowflake builds show a compact per-run busy-compute estimate. SQLBuild attributes visible +overlapping query intervals fairly across active queries, converts attributed seconds using the +warehouse-size credit rate, and estimates USD from the configured rate: + +```toml +[cost] +usd_per_credit = 3.00 +``` + +The default is `3.00` USD per credit and is visibly marked as a default. Set it to your Snowflake +contract rate. Use `sqb cost`, `sqb cost latest`, `sqb cost `, or +`sqb cost history --since 7d` to inspect persisted records. `--json` and `--json-output PATH` +provide a versioned, decimal-safe output contract. Pending detail records are refreshed from +Snowflake when inspected again. + +These values are attributed compute credits and estimated cost, not Snowflake-billed credits or +invoice reconciliation. The estimate uses only query history visible to the executing role and does +not reconstruct invisible concurrent work, warehouse resume or idle tail, the 60-second minimum, +cloud-services credits, contract adjustments, or multi-cluster billing. Run metadata and query IDs +are stored under `target/executions//`; that statement ledger stores only an SQL digest, +not SQL text. Executed SQL artifacts are stored separately under the sensitive `target/run/` tree. diff --git a/website/src/content/docs/docs/concepts/diff.mdx b/website/src/content/docs/docs/concepts/diff.mdx index d50aec53e6..3e435a4246 100644 --- a/website/src/content/docs/docs/concepts/diff.mdx +++ b/website/src/content/docs/docs/concepts/diff.mdx @@ -117,7 +117,21 @@ side. ## Row matching -Rows are matched between the two sides using the model's `unique_key`. Models without a `unique_key` can use schema-only diff but cannot run full or bounded row comparisons. +Rows are matched between the two sides using a row key. By default, row identity comes from each model's `unique_key`. Repeated `--key` arguments override +that configuration for one invocation: + +```bash +sqb diff prod:dev --full --select order_lines \ + --key order_id \ + --key line_id +``` + +For a model without a stable row key, opt in to exact full-row multiset comparison with +`--unkeyed`. Duplicate rows retain their multiplicity; SQLBuild never guesses a key. + +```bash +sqb diff prod:dev --full --select daily_order_totals --unkeyed +``` The diff output categorises rows as: - **Equal** - same key, same values on both sides @@ -221,6 +235,141 @@ sqb diff prod:dev --schema-only --select path:models/marts sqb diff prod:dev --full --select tag:acceptance ``` +## Compare raw queries + +Query mode omits `FROM:TO` because each query identifies its own input. Supply each side inline or +from a UTF-8 file, and choose keyed or unkeyed comparison explicitly: + +```bash +sqb diff \ + --left-query 'SELECT order_id, amount_cents FROM analytics.orders_before' \ + --right-query 'SELECT order_id, amount_cents FROM analytics.orders_after' \ + --left-label before \ + --right-label after \ + --key order_id +``` + +```bash +sqb diff \ + --left-query-file queries/orders_before.sql \ + --right-query-file queries/orders_after.sql \ + --key account_id \ + --key order_id +``` + +Raw inputs must contain exactly one read-only `SELECT`, `WITH`, or `VALUES` expression. SQLBuild +does not expand `ref()` or other SQLBuild templates in these inputs. The active target connection +executes both sides; use `--target NAME` to select another configured connection. + +Query mode performs a full comparison when no comparison mode flag is present. `--schema-only` is +also available. `--bounded` is model-specific; filter both raw queries directly when a bounded +query comparison is needed. + +Raw-query discovery loads only project configuration, target configuration, and the selected +adapter. It does not import unrelated project sinks, hooks, loaders, or providers. + +### Sampling, tolerances, and exclusions + +Keyed query comparisons support deterministic union-key sampling and the same bounded mismatch +examples as model comparisons: + +```bash +sqb diff \ + --left-query-file queries/orders_before.sql \ + --right-query-file queries/orders_after.sql \ + --key order_id \ + --sample-rows 50000 \ + --sample-seed 7 +``` + +Use repeated invocation overrides where needed: + +```bash +sqb diff \ + --left-query-file queries/orders_before.sql \ + --right-query-file queries/orders_after.sql \ + --key order_id \ + --exclude-column loaded_at \ + --tolerance amount_cents:absolute=1 \ + --tolerance conversion_rate:relative=0.001 +``` + +`--exhaustive` disables inherited or requested sampling. Unkeyed comparison is always exhaustive +and exact, so it does not accept sampling or numeric tolerances. + +### Bounded example values + +Comparison remains exact regardless of how examples are rendered. By default, long values are +limited to 160 characters and show context around the first differing character. Every shortened +value is marked as truncated and includes its original length. + +Control evidence independently from the number of collected examples: + +```bash +# Increase the displayed context without changing comparison scope. +sqb diff --left-query-file before.sql --right-query-file after.sql \ + --key order_id --max-value-length 500 + +# Retain keys and aggregate counts but suppress example values. +sqb diff --left-query-file before.sql --right-query-file after.sql \ + --key order_id --no-example-values + +# Explicitly opt in to complete values. --verbose alone remains bounded. +sqb diff --left-query-file before.sql --right-query-file after.sql \ + --key order_id --full-example-values +``` + +Ordinary keys are kept complete. Exceptionally long keys are shortened with an explicit length and +stable digest so an operator can still identify them without flooding terminal output. + +### Stable query results and cleanup + +SQLBuild materializes each raw query once in the selected target's default schema before comparing +it. Artifacts use a reserved run-owned name and are dropped immediately when the command finishes, +including after comparison errors. + +Each successfully created artifact publishes an immutable `query_diff_artifact` fingerprint. The +fingerprint is ownership evidence for crash recovery only: running and comparing the queries never +reads historical fingerprints, and fingerprints do not represent planned, running, complete, or +cleaned workflow states. + +If a process stops before immediate cleanup, the next query diff and `sqb janitor` can remove an +expired artifact only when its exact physical identity and reserved name match its ownership +fingerprint. A matching name without ownership evidence is reported for manual investigation and +is not deleted automatically. + +The recovery TTL defaults to 24 hours and accepts SQLBuild fixed-duration syntax: + +```toml +[diff] +query_artifact_ttl = "24h" +``` + +The TTL is a crash-recovery backstop, not the normal cleanup path. Historical ownership inspection +problems are reported but do not prevent the current queries from running. Failure to publish +ownership for a newly created artifact, or failure to remove a current-run artifact, is an execution +error. + +### Query output and exit codes + +Use `--json-output PATH`, or `--json` to emit one JSON document to stdout while all +lifecycle progress remains on stderr. Query results include `input_kind = "query"`, named left and +right inputs, `query_comparisons`, sparse nonzero `column_mismatches`, lifecycle timings, and +explicit example truncation metadata. Existing `models` output remains available for compatibility. + +Raw-query automation distinguishes complete findings from incomplete evidence and execution +failure: + +| Exit | Outcome | +|---:|---| +| `0` | Complete comparison with no differences | +| `1` | Complete comparison with findings | +| `2` | Safety, schema, or key preflight prevented complete value evidence | +| `3` | Setup, connection, materialization, comparison execution, ownership, or cleanup failed | + +Human output and JSON use the same outcome. Lifecycle messages include elapsed connection, +reconciliation, inspection, materialization, comparison, cleanup, and total durations. + ## Exit codes -`sqb diff` returns exit code `0` when all selected models have no differences, and `1` when any model has schema or row differences. This makes it usable in CI pipelines as a validation gate. +In model mode, `sqb diff` returns exit code `0` when all selected models have no differences, and `1` when any model has schema or row differences. Raw-query mode uses the exit codes in [Query output and exit codes](#query-output-and-exit-codes). This makes it usable in CI pipelines as a validation gate. diff --git a/website/src/content/docs/docs/concepts/macros/composition-and-context.mdx b/website/src/content/docs/docs/concepts/macros/composition-and-context.mdx index 9d947485f3..b1d336e161 100644 --- a/website/src/content/docs/docs/concepts/macros/composition-and-context.mdx +++ b/website/src/content/docs/docs/concepts/macros/composition-and-context.mdx @@ -131,3 +131,25 @@ def schema_qualified(ctx, table: str) -> str: Use context when generated SQL genuinely differs by adapter or target. Prefer ordinary parameters for values that the SQL caller should choose explicitly. + +### Constants and enums + +Python SQL macros receive the constants and enums visible to the SQL resource that calls them. Use +the typed mappings for Python control flow, and use the rendering methods when inserting a +declaration into generated SQL so quoting and collection syntax follow the active adapter: + +```python +def minimum_order_filter(ctx) -> str: + minimum = ctx.constants["minimum_order_value"] + if minimum is None: # The visible declaration explicitly has a NULL value. + return "TRUE" + return f"order_value >= {ctx.render_constant('minimum_order_value')}" + + +def active_status_filter(ctx) -> str: + status = ctx.render_enum_member(enum_name="order_status", member_name="active") + return f"status = {status}" +``` + +Callers can still pass explicit `@const(...)` or `@enum(...)` values as macro arguments. Context +lookups are intended for policy owned by the macro; both forms use the caller's declaration scope. diff --git a/website/src/content/docs/docs/concepts/observability/execution-history.mdx b/website/src/content/docs/docs/concepts/observability/execution-history.mdx new file mode 100644 index 0000000000..6291fdd6c8 --- /dev/null +++ b/website/src/content/docs/docs/concepts/observability/execution-history.mdx @@ -0,0 +1,155 @@ +--- +title: "Execution history" +description: "Store and query lifecycle events and run history from Python, with SQLite locally or PostgreSQL when deployed." +--- + +`LifecycleEventLogStorage` stores canonical immutable lifecycle facts. `RunStorage` serves an atomically +published, rebuildable run projection. Diagnostic and compute logs are never accepted by either +contract. The public types, filters, pages, records, errors, and helper functions are exported from +`sqlbuild.execution_history`. + +## Public contracts + +`LifecycleEventLogStorage` provides `append_event`, `append_events`, `get_events`, `get_schema_version`, +`upgrade_schema`, `close`, `dispose`, and context management. `RunStorage` provides `get_run`, +`get_runs`, `project`, `rebuild_from_events`, the schema/lifecycle methods, and context management. +The concrete SQLite and PostgreSQL classes additionally provide transactional +`append_and_project`, `reconcile`, and `check_health` operations. + +The deprecated `EventLogStorage` alias has been removed; use `LifecycleEventLogStorage`. Storage +tables use the lifecycle-specific names `lifecycle_event_log` in SQLite and +`sqlbuild_lifecycle_event_log` in PostgreSQL; the former `event_log` and `sqlbuild_event_log` names +are retired. + +Each newly stored event receives a positive, increasing `storage_order`, opaque stable `cursor`, +and UTC `received_at`. Pages ascend in storage order. `after_cursor` is exclusive, +`next_cursor` identifies the last returned record and is `None` for an empty page, and `has_more` +states whether another page exists. Limits default to 100 and cannot exceed 1000. Cursors are +backend- and storage-instance-owned; persist them verbatim and never parse them. They identify a +global position and remain valid if a later call changes filters. + +`EventFilter` supports invocation ID, run ID, event types, event family, producer, and inclusive +event-time bounds. Event types and family are mutually exclusive. `EventFamily` contains +invocation, run, resource-attempt, operation, and statement; select retry facts with +`EventFilter(event_types=("retry_scheduled",))`. `RunFilter` supports invocation, status, and +inclusive projection-creation bounds. A run cursor identifies the global +`(created_at, run_id)` order and is also filter-independent. + +Projection applies durable facts in `storage_order`, never `occurred_at` order. `created_at` is the +first run fact's `received_at`; start/end times retain event time. A run with no terminal is +`RunStatus.UNKNOWN` and `is_complete=False`. The last durable terminal wins if conflicting terminal +facts exist. Incremental projection and a full rebuild produce the same result, and one projection +call publishes all changes or none. + +## Local SQLite + +The CLI does not open local history implicitly. Construct and subscribe this backend only when an +application has a concrete durable-history consumer. + +```python +from pathlib import Path + +from sqlbuild.execution_history import EventFilter, RunFilter +from sqlbuild.sqlite_history import SQLiteExecutionHistory + +with SQLiteExecutionHistory(project_dir=Path(".")) as history: + assert history.check_health() + + cursor = None + while True: + page = history.get_events( + event_filter=EventFilter(run_id="RUN_ID"), + after_cursor=cursor, + ) + for stored in page.records: + print(stored.storage_order, stored.event) + if not page.has_more: + break + cursor = page.next_cursor + + for run in history.get_runs(run_filter=RunFilter()).records: + print(run.run_id, run.status, run.is_complete) +``` + +With no arguments, `SQLiteExecutionHistory()` uses the current directory. Passing `project_dir` +uses `/.sqlbuild/history.sqlite3`; passing `path` selects an exact file, and `path` may +be `":memory:"`. Do not pass both. Startup creates/migrates schema version 1 and reconciles the run +projection from all event facts. + +Schema v1 has `execution_history_metadata`, append-only `lifecycle_event_log`, and disposable +`run_projection` tables plus run, invocation, type, and created-time indexes. SQLite uses WAL mode, +foreign keys, a configurable 5000 ms default busy timeout, a process-local reentrant lock, and +transactions for append/projection publication. Multiple processes may use SQLite subject to normal +SQLite/WAL host-filesystem constraints and busy timeout; it is a host-local backend, not a deployed +coordination service. Startup refuses unknown future schema versions and never resets or downgrades +data. + +## Deployed PostgreSQL + +Install `sqlbuild[postgres]` only in deployed processes and construct the backend explicitly. The +CLI never selects PostgreSQL implicitly. Neither backend is selected by ordinary CLI commands. + +```python +import os +from datetime import UTC, datetime + +from sqlbuild.execution_history import EventFilter +from sqlbuild.observability import create_lifecycle_event, invocation_scope, run_scope +from sqlbuild.postgres_history import PostgresExecutionHistory + +occurred_at = datetime(2026, 9, 2, 10, 15, tzinfo=UTC) +with invocation_scope("deployment-20260902-1015"): + with run_scope("run-20260902-1015"): + run_started = create_lifecycle_event( + event_type="run_started", + event_id="run-20260902-1015-started", + occurred_at=occurred_at, + payload={"run_kind": "build", "selected_count": 12}, + ) + +with PostgresExecutionHistory(os.environ["SQLBUILD_HISTORY_DSN"]) as history: + first = history.append_and_project((run_started,))[0] + retry = history.append_event(run_started) + assert retry.storage_order == first.storage_order + + page = history.get_events( + event_filter=EventFilter(run_id="run-20260902-1015"), + after_cursor=None, + ) + checkpoint = page.next_cursor + +# Persist checkpoint securely. A later process resumes after that exact global position. +with PostgresExecutionHistory(os.environ["SQLBUILD_HISTORY_DSN"]) as history: + later = history.get_events( + event_filter=EventFilter(run_id="run-20260902-1015"), + after_cursor=checkpoint, + ) +``` + +The DSN must be resolved from a secret before construction. SQLBuild neither retains it for +diagnostics nor includes it in exceptions or `repr`. PostgreSQL schema v1 uses +`sqlbuild_storage_migrations`, authoritative `sqlbuild_lifecycle_event_log`, and disposable +`sqlbuild_run_projection`. Startup takes a transaction-scoped advisory migration lock, performs +forward-only migrations, and reconciles projection. All instances should run the same SQLBuild +version during rollout. + +Append and projection serialize under a history advisory lock. Only serialization failures and +deadlocks (`40001`, `40P01`) are retried internally, up to three retries by default. If connection +loss makes commit acknowledgement uncertain, discard the backend, reconnect, and retry the exact +same deterministic event ID and content. Equal content returns the existing stored fact; different +content raises `IntegrityConflictError`. This is storage idempotency, not exporter exactly-once +delivery. + +Grant deployed application roles only the required schema/table privileges and reserve migration, +backup, retention, and destructive privileges for operators. Back up `sqlbuild_lifecycle_event_log` and +`sqlbuild_storage_migrations`, preserving event IDs, canonical JSON, identity sequence values, and +storage namespace. `sqlbuild_run_projection` can be rebuilt with `reconcile()`. PostgreSQL performs +no automatic retention, partitioning, or archival. + +These are selected operational examples rather than an exhaustive API reference. The public +`sqlbuild.execution_history` facade also exports canonical content/ID helpers, page-limit +validation, projection helpers, schema and paging constants, storage protocols, models, filters, +statuses, and typed storage/cursor/schema errors. + +See [Observability](/docs/concepts/observability) for the lifecycle event envelope, correlation, and +failure behavior. diff --git a/website/src/content/docs/docs/concepts/project-configuration.mdx b/website/src/content/docs/docs/concepts/project-configuration.mdx index fdfc8c2726..d60f9978b5 100644 --- a/website/src/content/docs/docs/concepts/project-configuration.mdx +++ b/website/src/content/docs/docs/concepts/project-configuration.mdx @@ -216,6 +216,75 @@ allow_as_clone_destination = true Both policies default to `false`. `sqb clone --from prod --to dev` requires `allow_as_clone_origin = true` on `prod` and `allow_as_clone_destination = true` on `dev`. +### Execution limits + +Named targets can set optional limits for `sqb build`. Limits are useful for development targets +where an unexpectedly broad selector or long-running command should stop instead of consuming an +unbounded amount of warehouse compute. + +```toml +[targets.dev] +connection = "warehouse" +database = "analytics" +schema = "dev_alice" + +[targets.dev.execution_limits] +max_models = 125 +max_duration = "45m" +remediation = """ +Review the expanded plan and confirm that the selected scope is intentional. +Automated tools must request approval before changing this policy. +""" +``` + +All fields are optional. A target without an `execution_limits` section retains unlimited existing +behavior, so production or orchestrated targets do not need placeholder values. + +#### Model limit + +`max_models` is a positive integer. SQLBuild expands selectors and counts model entries that would +execute. When the count exceeds the target limit, the build exits before clone, load, or model +execution begins. The error reports the target, selected count, configured maximum, configuration +key, and confirms that no warehouse changes were made. + +`sqb plan` remains unrestricted and can be used to inspect a broad selection safely. + +#### Duration limit + +`max_duration` is a positive fixed duration such as `30m`, `2h`, or `90s`. It covers planning and +warehouse-mutating build phases after target resolution. SQLBuild stops dispatching work at the +deadline and passes the remaining duration to each warehouse statement so in-flight work is +cancelled by the adapter. + +Safe in-flight cancellation is currently supported by the Snowflake adapter, whose maximum accepted +duration is seven days. SQLBuild rejects an unsupported duration or an adapter that cannot guarantee +active statement cancellation rather than presenting a limit that may leave warehouse work running. + +A duration failure may occur after earlier models have completed. Its error therefore distinguishes +the timeout from a pre-execution model-limit failure and states that warehouse changes may already +have occurred. + +#### Custom remediation + +`remediation` is optional project-authored text. SQLBuild appends it to model-count, duration, and +unsupported-adapter errors after the factual error details. This lets each project define its own +approval or recovery process without SQLBuild recommending that users weaken local policy. + +#### Local overrides + +The same section is accepted in `sqlbuild_local.toml` and follows normal field-by-field local target +precedence: + +```toml +[targets.dev.execution_limits] +max_models = 200 +max_duration = "90m" +``` + +This permits an intentional developer override while keeping the shared target policy visible in +`sqlbuild_project.toml`. Other target fields and any execution-limit fields omitted locally continue +to inherit their project values. + ## Defaults Project-wide model defaults. Any field you can set in a `MODEL()` header can be set here as a default: @@ -440,6 +509,19 @@ auto_update = false See [skills CLI reference](/docs/cli/skills) for usage details. +## Project Python + +Project-owned Python must live in a supported extension location such as `factories/`, `libs/`, +`macros/`, `providers/`, or another documented Python resource root. Factory locations contain +normal Python: constants, classes, undecorated helper functions, and modules such as `_helpers.py` +are allowed, while decorators determine which functions become SQLBuild resources. Compilation +rejects Python under unsupported project roots, so indirectly importable modules cannot create an +unofficial project structure. + +Keep repository pytest tests outside the SQLBuild project's `tests/` directory, which is reserved for +SQLBuild SQL tests and scenarios. Documented integration paths such as `dagster/`, +`rivers_pipeline/`, and their `definitions.py` modules are also supported. + ## sqlbuild_local.toml Local developer overrides. This optional file is loaded automatically and should be diff --git a/website/src/content/docs/docs/concepts/testing.mdx b/website/src/content/docs/docs/concepts/testing.mdx index 5a7dedb77d..c8d97f449e 100644 --- a/website/src/content/docs/docs/concepts/testing.mdx +++ b/website/src/content/docs/docs/concepts/testing.mdx @@ -202,6 +202,21 @@ SELECT 1 You can mix `__source__`, `__ref__`, and `__seed__` mocks in the same test. As long as every leaf dependency is satisfied (either by a source mock, a ref mock, a seed mock, or by being in the expected chain), the test resolves. +## Fixture columns + +A relation fixture can omit a column when the compiled test or scenario closure requires it and +SQLBuild knows its adapter type, unless the column is explicitly non-nullable. SQLBuild completes +that test-only fixture column with a typed null such as `CAST(NULL AS VARCHAR)`; it never changes +model SQL or warehouse defaults. Required columns declared with `nullable false` and columns with +unknown types must be supplied explicitly. When the relation's complete column set is +authoritative, misspelled or unknown supplied fixture columns are rejected. + +A direct `NULL AS column_name` projection receives the authoritative relation type in compiled test +SQL, including expected-output CTEs for contracted models. To represent a contracted upstream with +no rows, use `SELECT * FROM __empty_fixture()` as the complete body of a `__ref__`, `__source__`, or +`__seed__` fixture CTE, or a contracted `__expected__` CTE. SQLBuild expands it to the relation's +full typed schema with a false filter. + ## Mocking table functions in model tests Use `__table_fn__` when the model under test reads a managed table function but the test diff --git a/website/src/navigation.mjs b/website/src/navigation.mjs index fd8f105fb0..f1084d9dd8 100644 --- a/website/src/navigation.mjs +++ b/website/src/navigation.mjs @@ -1,213 +1,6 @@ -// Sidebar and redirects, first generated from the Mintlify docs.json. -export const sidebar = [ - { - "label": "Getting Started", - "items": [ - "docs", - "docs/quickstart", - "docs/feature-comparison", - "docs/roadmap", - "docs/benchmarks" - ] - }, - { - "label": "dbt Compatibility", - "items": [ - "docs/concepts/dbt-compatibility/overview", - "docs/concepts/dbt-compatibility/selection", - "docs/concepts/dbt-compatibility/adding-sqlbuild-models" - ] - }, - { - "label": "Concepts", - "items": [ - "docs/concepts/project-configuration", - "docs/concepts/resource-identities", - { - "label": "Adapters", - "collapsed": true, - "items": [ - "docs/concepts/adapters", - "docs/concepts/adapters/snowflake", - "docs/concepts/adapters/duckdb", - "docs/concepts/adapters/motherduck", - "docs/concepts/adapters/postgres", - "docs/concepts/adapters/bigquery", - "docs/concepts/adapters/databricks", - "docs/concepts/adapters/sqlserver" - ] - }, - "docs/concepts/sources", - "docs/concepts/seeds", - { - "label": "Models", - "collapsed": true, - "items": [ - "docs/concepts/models", - "docs/concepts/models/materializations", - "docs/concepts/models/schemas", - "docs/concepts/models/type-enforcement", - "docs/concepts/models/contracts", - "docs/concepts/models/migrations", - { - "label": "Hooks", - "collapsed": true, - "items": [ - { - "label": "Overview", - "slug": "docs/concepts/models/hooks" - }, - "docs/concepts/models/hooks/sql", - "docs/concepts/models/hooks/python" - ] - }, - "docs/concepts/models/configuration" - ] - }, - { - "label": "Enums and Constants", - "collapsed": true, - "items": [ - "docs/concepts/enums", - "docs/concepts/constants", - "docs/concepts/constants/collections-and-rendering", - "docs/concepts/enums/model-contracts", - "docs/concepts/model-private-values" - ] - }, - { - "label": "Python Macros", - "collapsed": true, - "items": [ - "docs/concepts/macros", - "docs/concepts/macros/composition-and-context" - ] - }, - "docs/concepts/interpolation", - "docs/concepts/functions", - "docs/concepts/incremental", - { - "label": "Planning and Change Detection", - "collapsed": true, - "items": [ - "docs/concepts/planning", - "docs/concepts/planning/cascade-propagation", - "docs/concepts/planning/source-freshness", - "docs/concepts/planning/selection-and-staleness" - ] - }, - "docs/concepts/snapshots", - "docs/concepts/audits", - { - "label": "Rules", - "collapsed": true, - "items": [ - "docs/concepts/rules", - "docs/concepts/rules/configuration-and-selection", - "docs/concepts/rules/findings-and-exceptions", - "docs/concepts/rules/execution-and-caching" - ] - }, - "docs/concepts/testing", - "docs/concepts/scenarios", - "docs/concepts/selectors", - "docs/concepts/column-lineage", - "docs/concepts/diff" - ] - }, - { - "label": "Advanced Concepts", - "items": [ - { - "label": "Observability", - "collapsed": true, - "items": [ - "docs/concepts/observability", - "docs/concepts/observability/sinks" - ] - }, - { - "label": "Declarations and Scopes", - "collapsed": true, - "items": [ - "docs/concepts/declaration-scopes", - "docs/concepts/declaration-scopes/visibility", - "docs/concepts/declaration-scopes/placement", - "docs/concepts/declaration-scopes/explorer" - ] - }, - { - "label": "Custom Rules", - "collapsed": true, - "items": [ - "docs/concepts/rules/custom-rules/overview", - "docs/concepts/rules/custom-rules/rule-context", - "docs/concepts/rules/custom-rules/testing-and-determinism" - ] - } - ] - }, - { - "label": "Python Nodes", - "items": [ - "docs/concepts/python-nodes/overview", - "docs/concepts/python-nodes/loaders", - "docs/concepts/python-nodes/tasks", - "docs/concepts/python-nodes/assets", - "docs/concepts/python-nodes/checks", - "docs/concepts/python-nodes/factories", - "docs/concepts/python-nodes/providers", - "docs/concepts/python-nodes/sql-references" - ] - }, - { - "label": "Integrations", - "items": [ - { - "label": "Dagster", - "collapsed": true, - "items": [ - "docs/integrations/dagster", - "docs/integrations/dagster-reference" - ] - }, - "docs/integrations/rivers", - "docs/integrations/dlt", - "docs/integrations/ingestr" - ] - }, - { - "label": "CLI Reference", - "items": [ - "docs/cli/init", - "docs/cli/playground", - "docs/cli/skills", - "docs/cli/compile", - "docs/cli/format", - "docs/cli/contract", - "docs/cli/scope", - "docs/cli/rules", - "docs/cli/plan", - "docs/cli/build", - "docs/cli/load", - "docs/cli/seed", - "docs/cli/test", - "docs/cli/scenario", - "docs/cli/audit", - "docs/cli/freshness", - "docs/cli/check", - "docs/cli/clone", - "docs/cli/diff", - "docs/cli/lineage", - "docs/cli/dag", - "docs/cli/query", - "docs/cli/debug", - "docs/cli/janitor", - "docs/cli/clean", - "docs/cli/dbt" - ] - } -]; +import sidebarData from './sidebar.json'; + +export const sidebar = sidebarData; export const redirects = { "/docs/concepts/rules/overview": "/docs/concepts/rules", diff --git a/website/src/sidebar.json b/website/src/sidebar.json new file mode 100644 index 0000000000..d3eb5201b1 --- /dev/null +++ b/website/src/sidebar.json @@ -0,0 +1,210 @@ +[ + { + "label": "Getting Started", + "items": [ + "docs", + "docs/quickstart", + "docs/feature-comparison", + "docs/roadmap", + "docs/benchmarks" + ] + }, + { + "label": "dbt Compatibility", + "items": [ + "docs/concepts/dbt-compatibility/overview", + "docs/concepts/dbt-compatibility/selection", + "docs/concepts/dbt-compatibility/adding-sqlbuild-models" + ] + }, + { + "label": "Concepts", + "items": [ + "docs/concepts/project-configuration", + "docs/concepts/resource-identities", + { + "label": "Adapters", + "collapsed": true, + "items": [ + "docs/concepts/adapters", + "docs/concepts/adapters/snowflake", + "docs/concepts/adapters/duckdb", + "docs/concepts/adapters/motherduck", + "docs/concepts/adapters/postgres", + "docs/concepts/adapters/bigquery", + "docs/concepts/adapters/databricks", + "docs/concepts/adapters/sqlserver" + ] + }, + "docs/concepts/sources", + "docs/concepts/seeds", + { + "label": "Models", + "collapsed": true, + "items": [ + "docs/concepts/models", + "docs/concepts/models/materializations", + "docs/concepts/models/schemas", + "docs/concepts/models/type-enforcement", + "docs/concepts/models/contracts", + "docs/concepts/models/migrations", + { + "label": "Hooks", + "collapsed": true, + "items": [ + { + "label": "Overview", + "slug": "docs/concepts/models/hooks" + }, + "docs/concepts/models/hooks/sql", + "docs/concepts/models/hooks/python" + ] + }, + "docs/concepts/models/configuration" + ] + }, + { + "label": "Enums and Constants", + "collapsed": true, + "items": [ + "docs/concepts/enums", + "docs/concepts/constants", + "docs/concepts/constants/collections-and-rendering", + "docs/concepts/enums/model-contracts", + "docs/concepts/model-private-values" + ] + }, + { + "label": "Python Macros", + "collapsed": true, + "items": [ + "docs/concepts/macros", + "docs/concepts/macros/composition-and-context" + ] + }, + "docs/concepts/interpolation", + "docs/concepts/functions", + "docs/concepts/incremental", + { + "label": "Planning and Change Detection", + "collapsed": true, + "items": [ + "docs/concepts/planning", + "docs/concepts/planning/cascade-propagation", + "docs/concepts/planning/source-freshness", + "docs/concepts/planning/selection-and-staleness" + ] + }, + "docs/concepts/snapshots", + "docs/concepts/audits", + { + "label": "Rules", + "collapsed": true, + "items": [ + "docs/concepts/rules", + "docs/concepts/rules/configuration-and-selection", + "docs/concepts/rules/findings-and-exceptions", + "docs/concepts/rules/execution-and-caching" + ] + }, + "docs/concepts/testing", + "docs/concepts/scenarios", + "docs/concepts/selectors", + "docs/concepts/column-lineage", + "docs/concepts/diff" + ] + }, + { + "label": "Advanced Concepts", + "items": [ + { + "label": "Observability", + "collapsed": true, + "items": [ + "docs/concepts/observability", + "docs/concepts/observability/sinks", + "docs/concepts/observability/execution-history" + ] + }, + { + "label": "Declarations and Scopes", + "collapsed": true, + "items": [ + "docs/concepts/declaration-scopes", + "docs/concepts/declaration-scopes/visibility", + "docs/concepts/declaration-scopes/placement", + "docs/concepts/declaration-scopes/explorer" + ] + }, + { + "label": "Custom Rules", + "collapsed": true, + "items": [ + "docs/concepts/rules/custom-rules/overview", + "docs/concepts/rules/custom-rules/rule-context", + "docs/concepts/rules/custom-rules/testing-and-determinism" + ] + } + ] + }, + { + "label": "Python Nodes", + "items": [ + "docs/concepts/python-nodes/overview", + "docs/concepts/python-nodes/loaders", + "docs/concepts/python-nodes/tasks", + "docs/concepts/python-nodes/assets", + "docs/concepts/python-nodes/checks", + "docs/concepts/python-nodes/factories", + "docs/concepts/python-nodes/providers", + "docs/concepts/python-nodes/sql-references" + ] + }, + { + "label": "Integrations", + "items": [ + { + "label": "Dagster", + "collapsed": true, + "items": [ + "docs/integrations/dagster", + "docs/integrations/dagster-reference" + ] + }, + "docs/integrations/rivers", + "docs/integrations/dlt", + "docs/integrations/ingestr" + ] + }, + { + "label": "CLI Reference", + "items": [ + "docs/cli/init", + "docs/cli/playground", + "docs/cli/skills", + "docs/cli/compile", + "docs/cli/format", + "docs/cli/contract", + "docs/cli/scope", + "docs/cli/rules", + "docs/cli/plan", + "docs/cli/build", + "docs/cli/load", + "docs/cli/seed", + "docs/cli/test", + "docs/cli/scenario", + "docs/cli/audit", + "docs/cli/freshness", + "docs/cli/check", + "docs/cli/clone", + "docs/cli/diff", + "docs/cli/lineage", + "docs/cli/dag", + "docs/cli/query", + "docs/cli/debug", + "docs/cli/janitor", + "docs/cli/clean", + "docs/cli/dbt" + ] + } +]