Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 71 additions & 4 deletions benchmarks/duckshop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ Everything in it is synthetic: order references use a `DS-O-` prefix, email addr
```text
benchmarks/duckshop/
├── project/ the SQLBuild project (Snowflake adapter), with its data loaders in python/loaders/
├── scripts/ tooling that is not part of the project: the construct census and cleanup SQL
├── scripts/ tooling that is not part of the project: the construct census, cleanup SQL, the
│ core-only copy and edge-case expectation capture
├── tests/ offline checks for the synthetic data loaders
└── RUNBOOK.md loading, building, testing and capturing on Snowflake, and cleaning up
```
Expand All @@ -39,6 +40,8 @@ benchmarks/duckshop/

## Project shape

The core project, without the [edge-case area](#edge-cases):

- 252 models: 99 staging, 81 intermediate, 66 marts and 6 snapshots. Marts are grouped by area:
core, orders, customers, catalogue, inventory, fulfilment, support, clickstream, growth,
finance, partners and executive.
Expand All @@ -50,15 +53,15 @@ benchmarks/duckshop/

## Tests and scenarios

`tests/unit/` mirrors `models/` and holds 266 unit tests:
`tests/unit/` mirrors `models/` and holds 266 core unit tests:

- 260 model tests, including cursor-window tests on incremental models, a table-function mock, a
macro mock, a fixture built by a test-only Python macro, an empty fixture for a contracted
source, and a file with two `TEST` blocks;
- 3 macro tests, 2 UDF tests and 1 table-function test, most driven by `VALUES` case tables;
- assertions in 98 tests, including a key check on every mart.

`tests/scenarios/` holds 8 scenarios: one each for catalogue, clickstream, orders, fulfilment,
`tests/scenarios/` holds 8 core scenarios: one each for catalogue, clickstream, orders, fulfilment,
support and inventory, and two that cross areas (a regional scorecard, and campaign return on ad
spend). Two use `expected_columns "all"`. Expected values are derived by hand from the fixtures,
and every test and scenario passes on Snowflake (see [Status](#status)); the scenarios' captured
Expand Down Expand Up @@ -102,6 +105,46 @@ uv run --isolated --no-project --with pytest --with pyyaml pytest benchmarks/duc

See [RUNBOOK.md](RUNBOOK.md) for loading, building and testing on Snowflake.

## Edge cases

The edge-case area holds small, hand-picked cases for SQL whose results are easy to get subtly
wrong when translating Snowflake SQL or replaying it on another engine. It lives beside the core
project and does not change it:

- models in `models/staging/edge_cases/`, `models/intermediate/edge_cases/` and
`models/marts/edge_cases/`, each tagged `edge_case`. The `XSQBRDUCK002` Rule fails a model in an
`edge_cases` folder without the tag, or a model with the tag outside one;
- unit tests in the matching `tests/unit/<layer>/edge_cases/` folders and scenarios in
`tests/scenarios/edge_cases/`;
- raw `edge__*` sources in `sources/edge_cases.yml`, loaded by `python/loaders/edge_cases.py` from
fixed rows that `data_scale` does not change, so core data and core expectations stay the same.

It has 38 models (7 staging, 22 intermediate, 9 marts), 61 unit tests and 4 scenarios, in four
groups:

| Group | Covers |
|-------|--------|
| Semi-structured | JSON null against SQL NULL and missing keys; document text with sorted, case-differing and escaped keys; JSON number fidelity and casts out of `VARIANT`; path-like keys (`''`, `*`, `$`-prefixed, dotted) with `GET` and `GET_IGNORE_CASE`; `OBJECT_CONSTRUCT_KEEP_NULL`, `OBJECT_INSERT`, `OBJECT_DELETE`, `OBJECT_PICK`; array builders, `ARRAY_EXCEPT` and number spellings in `ARRAY_POSITION`; `FLATTEN` outer; lenient and malformed `TRY_PARSE_JSON`; typed lambdas |
| Dates and times | `TRY_TO_TIMESTAMP_NTZ` and its siblings over free text; epoch milliseconds as numbers and text; daylight-saving gaps and overlaps; fractional offsets; nanosecond truncation and epoch parts after 2242; pre-1970 text with sub-microsecond digits; date-part normalisation; month ends and leap days; week numbering across a year; sentinel timestamps |
| Ordering, numbers and text | NULL ordering under `DESC`; ties in ordered aggregates, `MAX_BY`, `MODE` and `ANY_VALUE`; default window frames; `UNION ALL BY NAME`; `ROUND` half-way values, decimal division, `MOD` and 38-digit amounts; integers past 2^53 against `FLOAT`; regular expressions and backslash escapes; `LIKE ... ESCAPE` and `ILIKE ANY`; `SPLIT_PART` and `SUBSTR` before the start; `NOT IN` with a NULL; names in many scripts |
| Scenarios and lifecycle | typed values across model boundaries; `VARIANT` expected output; a star under a lower-case contract and a quoted camelCase export; a cursor-window incremental model over far-apart fixture loads; a change log merged across two loads, checked against a full rebuild |

Unit-test expectations in the edge-case area were captured on Snowflake under the `edge` target's
pinned session, rather than derived by hand; scenario expectations, and the assertions in tests
about ties, are written by hand from those captured values. `fct_edge_delivery_events` is disabled: it is the
acceptance case for cursor `delete_insert` models that emit rows outside the window they replace.
See [RUNBOOK.md](RUNBOOK.md), step 7.

Benchmarks and core measurements leave the area out:

- `python benchmarks/duckshop/scripts/core_project.py <destination>` copies the project without
it. The core-copy compile guard compares the copy's compiled files with a recorded checksum
(`core_project.py --checksum`), so a change to the core project's compiled SQL fails the guard
until the checksum is updated on purpose and noted for the benchmark;
- warehouse runs exclude it with `--exclude tag:edge_case "source:edge__*"`, and scenario runs
with `--exclude edge_cases`;
- the construct census counts the core project unless given `--area edge` or `--area all`.

## Status

Last verified on Snowflake with SQLBuild 0.175.0 at `data_scale = "1"`:
Expand All @@ -113,7 +156,7 @@ Last verified on Snowflake with SQLBuild 0.175.0 at `data_scale = "1"`:
| `sqb test` | 266 of 266 pass |
| `sqb scenario test` | 8 of 8 pass |
| `sqb scenario capture` | Captures all 8; snapshots are in `project/tests/_scenario_snapshots/` |
| `sqb scenario test --local --strict` | 0 of 8 pass: 5 fail in translation, 3 report a stale snapshot |
| `sqb scenario test --local --strict` | In the core copy, 0 of 8 pass: 5 fail in translation, 3 report a stale snapshot |

Local replay translates each scenario's Snowflake SQL to DuckDB on a best-effort basis. These five
failures are the baseline that better translation should remove:
Expand All @@ -132,6 +175,21 @@ The other three scenarios (`campaign_roas__attributes_orders_to_channel_spend`,
reported stale straight after capture. The snapshot fingerprint now hashes the authored fixture
SQL, so these results predate that fix; recapture the snapshots to replay them.

The project no longer sets `QUOTED_IDENTIFIERS_IGNORE_CASE = true` on its connection. With SQLBuild
0.184.0, a core build without it passes with the same results as with it: every core model holds
the same rows, apart from the snapshots, which record build times, and three models whose results
vary from one run to the next (`int_cart_funnel`, `int_delivery_slot_utilisation` and
`rpt_delivery_exceptions_monthly`).

The edge-case area, verified on Snowflake with SQLBuild 0.184.0 on the `edge` target:

| Step | Result |
|------|--------|
| `sqb build --select tag:edge_case --exclude stg_edge_partner_listings+` | Passes; the disabled model is skipped |
| `sqb test --select tag:edge_case` | 61 of 61 pass |
| `sqb scenario test --select edge_cases` | 4 of 4 pass |
| Two-load replay of the delivery change log | The merged status equals the full rebuild after each load |

## Expected divergences

Nine models use constructs whose results depend on the engine or the session. They are tagged
Expand All @@ -155,6 +213,15 @@ noted here:
- Compilation accepts SQL that Snowflake rejects: `CONNECT_BY_ISLEAF`, which Snowflake does not
have, and a CTE whose body is a bare `VALUES` list. `int_category_paths` finds leaf categories
with a join instead, and test case tables use `SELECT ... FROM VALUES ... AS v(...)`.
- A source column whose name is quoted mixed case, such as `"listingId"`, cannot be read once the
table exists: planning folds the quoted reference to lower case and fails with `B002`. Fixtures
cannot name such columns either (`B302` in unit tests, `S510` in scenarios), and a `MODEL`
header cannot declare a quoted column name. `stg_edge_partner_listings` has a reviewed
`SQBRTEST202` ignore, `rpt_edge_partner_listing_export` a reviewed `SQBRTEST201` ignore, and the
edge build excludes `stg_edge_partner_listings+`.
- Local scenario replay stops at compile (`P001`, unclosed quoted string) on a model with a
backslash-escaped quote in a string literal, `'O\'Brien'` in `stg_edge_ticket_notes`, so local
replay runs in the core copy.
- `SQBRSQL018` can only take proof from single-column keys. Append-only feeds redeliver rows, so
their id is unique only together with `_loaded_at` (the `unique_per_delivery` audit), and the
staging models that keep the latest delivery carry reviewed exceptions.
Expand Down
133 changes: 115 additions & 18 deletions benchmarks/duckshop/RUNBOOK.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# DuckShop runbook

How to load DuckShop's synthetic data into Snowflake, build and test the project, capture the
scenario snapshots, and clean up afterwards. Run every command from `benchmarks/duckshop/project`
unless it says otherwise.
scenario snapshots, run the edge-case area, and clean up afterwards. Run every command from
`benchmarks/duckshop/project` unless it says otherwise.

Steps 3 to 6 run the core project only: they exclude the edge-case area (models tagged
`edge_case`, their `edge__*` sources and the `edge_cases` scenario folder), which step 7 runs on its
own target.

## 1. Configure the connection

Expand All @@ -15,11 +19,13 @@ cp sqlbuild_local.toml.example sqlbuild_local.toml
```

Fill in the account, user, authenticator, role and warehouse, and the `database` and `schema` of
`[targets.dev]`. Keep the project's `session_parameters` setting on the connection (see Known
limitations in the README): `sqlbuild_local.toml` adds to the `[connections.warehouse]` table in
`sqlbuild_project.toml` rather than replacing it, so leave `session_parameters` out of the local
file. `sqlbuild_local.toml` is gitignored; never commit it, and never copy its values
into tracked files, commit messages or pull requests.
`[targets.dev]`. For the edge-case area (step 7), fill in `[connections.edge]` and `[targets.edge]`
the same way, with a different schema from `[targets.dev]`: two targets that write managed sources
to the same schema fail with `S102`. `sqlbuild_local.toml` adds to the connection tables in
`sqlbuild_project.toml` rather than replacing them, so the edge connection keeps the project's
pinned `session_parameters`; leave `session_parameters` out of the local file.
`sqlbuild_local.toml` is gitignored; never commit it, and never copy its values into tracked
files, commit messages or pull requests.

Check the connection, then confirm the project compiles:

Expand All @@ -43,13 +49,20 @@ sqb query --file target/check.sql --no-limit
```

The query lists every existing table, view or function with a DuckShop name, which is everything
step 7 drops, and any SQLBuild state tables (`_SQLBUILD_*`). Before the first build it should list
no DuckShop names; if it does, stop and pick another schema. Note which state tables already existed, for the cleanup in step 7.
step 8 drops, and any SQLBuild state tables (`_SQLBUILD_*`). Before the first build it should list
no DuckShop names; if it does, stop and pick another schema. Note which state tables already existed, for the cleanup in step 8.
The list includes the edge-case area's names; run the check with `--target edge` too before its
first build.

The project no longer sets `QUOTED_IDENTIFIERS_IGNORE_CASE = true` on its connections. A schema
built while it did may hold objects and SQLBuild state that no longer resolve under Snowflake's
default case rules, so rebuild into a clean schema: drop everything as in step 8, or pick a new
schema, before the first build.

## 3. Load the synthetic data

```bash
sqb load --concurrency 4
sqb load --concurrency 4 --exclude "source:edge__*"
```

All 76 managed sources load, about 60,000 rows in total. Each loader replaces its table, so loading
Expand All @@ -64,7 +77,7 @@ data_scale = "10"
## 4. Build

```bash
sqb build --concurrency 8
sqb build --concurrency 8 --exclude tag:edge_case "source:edge__*"
```

The build reloads the managed sources first. SQLBuild 0.175.0 does this even with `--no-load`
Expand Down Expand Up @@ -108,8 +121,8 @@ sqb build --select fct_web_sessions --full-refresh \
## 5. Run the unit tests and scenarios on Snowflake

```bash
sqb test --concurrency 8
sqb scenario test --concurrency 4
sqb test --concurrency 8 --exclude tag:edge_case
sqb scenario test --concurrency 4 --exclude edge_cases
```

Unit tests and scenarios read only their own fixtures, not the loaded data, so they can run before
Expand All @@ -119,14 +132,17 @@ on Snowflake. Report any failure rather than editing the expectations to match.
## 6. Capture the scenario snapshots and commit them

```bash
sqb scenario capture
sqb scenario test --local --strict
sqb scenario capture --exclude edge_cases
python ../scripts/core_project.py /tmp/duckshop-core
(cd /tmp/duckshop-core && sqb scenario test --local --strict)
```

Capture materialises each scenario's fixtures in Snowflake, downloads them as JSONL into
`tests/_scenario_snapshots/`, and drops what it created. The local run then replays all 8
scenarios on your machine from those files, with no warehouse connection; `--strict` fails on a
missing or stale snapshot instead of skipping it.
missing or stale snapshot instead of skipping it. It runs in a copy of the project without the
edge-case area, because local replay of the full project stops at compile on an edge-case model
(see Known limitations in the README). Delete the copy afterwards.

With SQLBuild 0.175.0 the local run is expected to end with `ERROR=8`: five scenarios fail while
translating Snowflake SQL to DuckDB, and three report `X602` (stale snapshot) because of a
Expand All @@ -144,7 +160,86 @@ git commit -m "test(benchmarks): capture DuckShop scenario snapshots"

The `grep` must print nothing. The repository's publication checks run on commit and push as usual.

## 7. Clean up
## 7. Edge cases

The edge-case area runs on the `edge` target, whose connection pins the session its expected
values were captured under: UTC, and Snowflake's defaults for every parameter that changes a
result or how text is parsed, including `QUOTED_IDENTIFIERS_IGNORE_CASE = false`. Run it on that
target only; under another session some expected values differ by design.

`DEFAULT_DDL_COLLATION` also changes results (comparisons, ordering and `LIKE` on text columns),
but it is an account, database or schema parameter, not a session one, so the connection cannot
pin it. The edge target's schema, and its database and account, must leave it at the default (no
collation). Check with `SHOW PARAMETERS LIKE 'DEFAULT_DDL_COLLATION' IN SCHEMA <database>.<schema>`.

### Load, build and test

```bash
sqb build --target edge --select tag:edge_case --exclude stg_edge_partner_listings+ --concurrency 8
sqb test --target edge --select tag:edge_case --concurrency 8
sqb scenario test --target edge --select edge_cases
```

The build loads the `edge__*` sources from fixed rows that `data_scale` does not change, then
builds the edge models and runs their tests and audits. `fct_edge_delivery_events` is disabled and
reports `skip: disabled` (see [Pending acceptance cases](#pending-acceptance-cases)).
`stg_edge_partner_listings` and its downstream models are excluded because planning fails once the
`edge__partner_listings` table exists (see Known limitations in the README); the
`edge_partner_listings__star_contract_and_quoted_export` scenario covers the models downstream of
it.

### Replay the delivery change log as two loads

`int_edge_delivery_status` merges a shipment delivery change log incrementally, and the
`edge_delivery_status_matches_rebuild` audit fails if it differs from a rebuild from the whole log.
To exercise the merge across loads, start without the table, load the first load only, build, then
load everything and build again:

```bash
sqb query --target edge "DROP TABLE IF EXISTS <database>.<schema>.int_edge_delivery_status"
sqb load --target edge --select edge__delivery_updates --vars '{"edge_delivery_batch": "1"}'
sqb build --target edge --no-load \
--select int_edge_delivery_update_versions int_edge_delivery_status int_edge_delivery_status_rebuilt
sqb load --target edge --select edge__delivery_updates
sqb build --target edge --no-load \
--select int_edge_delivery_update_versions int_edge_delivery_status int_edge_delivery_status_rebuilt
```

The second build plans `int_edge_delivery_status` as a `merge` over the second load, which carries
a late older version, a redelivered version, a delete and a new shipment, and the audit passes.
Load with `sqb load` and build with `--no-load`, so that each build reads exactly the load you
chose.

### Capture expected values

Unit-test expectations in the edge-case area are Snowflake's own results on the fixtures, not
values derived by hand. To capture them for a new or changed edge test, write its `__expected__<model>` CTE with the
columns to compare (any values, such as `NULL AS column_name`), then run:

```bash
python ../scripts/capture_edge_expectations.py --write <test name> ...
sqb format tests/unit/staging/edge_cases tests/unit/intermediate/edge_cases tests/unit/marts/edge_cases
```

The script runs the edge tests on the `edge` target, reads each selected test's executed SQL, and
replaces each expected CTE with the actual rows as typed literals. Without a test name it
recaptures every edge test, which must leave the files unchanged. Check every captured value
against the case the test describes before committing: a capture records what Snowflake returned,
which is the point, but a fixture mistake is captured as faithfully as a real edge. Scenario
expectations, and the assertions in tests about ties (where Snowflake may return any of the tied
rows), are written by hand from the captured values.

Capture scenario snapshots for the edge scenarios with
`sqb scenario capture --target edge --select edge_cases`.

### Pending acceptance cases

`fct_edge_delivery_events` filters its input by order date but is rebuilt by delivery date, so a
rerun of a window deletes only that window's deliveries and inserts deliveries outside it again.
It stays disabled (`enabled false`) until cursor `delete_insert` models guard rows outside the
window; its unit test records the rows outside the window that the model emits today.

## 8. Clean up

Drop everything the project created, by name:

Expand All @@ -156,7 +251,9 @@ sqb clean
```

`drop.sql` is one Snowflake Scripting block of `DROP ... IF EXISTS` statements for every model,
snapshot, seed, raw source table and function, so it is safe to run twice. Scenario runs and
snapshot, seed, raw source table and function, edge-case area included, so it is safe to run
twice. Run the same three commands with `--target edge` (on `sqb compile` and `sqb query`) to clean
up the edge target. Scenario runs and
captures already drop their own relations.

SQLBuild also keeps state tables (`_SQLBUILD_*`) in the schema. If step 2 showed none before the
Expand Down
Loading
Loading