Skip to content

Commit e68476e

Browse files
kvlongekvlonge
authored
feat(benchmarks): add DuckShop, a dense Snowflake example project (#1245)
1 parent fe1308e commit e68476e

660 files changed

Lines changed: 65402 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎benchmarks/duckshop/README.md‎

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
# DuckShop
2+
3+
DuckShop is the online shop of Mallard & Co., a fictional company that sells duck-themed home goods:
4+
rubber ducks (collectible series and limited editions), duck-print textiles, decoys, pond and garden
5+
supplies, and bath goods. It is written as a dense SQLBuild project in Snowflake SQL, and is a
6+
realistic corpus for exercising the SQLBuild compiler, Rules, unit tests and scenarios, and for
7+
testing translation from Snowflake SQL to other dialects.
8+
9+
The theme lives only in the data and descriptions: product lines named after duck species
10+
(Mallard, Pintail, Teal, Eider, Merganser), warehouses such as The Pond and The Nest, and loyalty
11+
tiers from Duckling to Golden Duck. Model, column and source names stay conventional, and the
12+
business mechanics are those of any multi-region online shop.
13+
14+
Everything in it is synthetic: order references use a `DS-O-` prefix, email addresses use
15+
`example.com`-style domains, and every brand, carrier and product name is invented.
16+
17+
## Layout
18+
19+
```text
20+
benchmarks/duckshop/
21+
├── project/ the SQLBuild project (Snowflake adapter), with its data loaders in python/loaders/
22+
├── scripts/ tooling that is not part of the project: the construct census and cleanup SQL
23+
├── tests/ offline checks for the synthetic data loaders
24+
└── RUNBOOK.md loading, building, testing and capturing on Snowflake, and cleaning up
25+
```
26+
27+
## Domain
28+
29+
- Storefront regions: EU (GBP and EUR), US (USD), APAC (JPY and AUD) and MENA (KWD), so currencies
30+
with 0, 2 and 3 minor-unit digits all occur. Warehouses: The Pond (`Europe/London`), The Nest
31+
(`America/New_York`) and The Marsh (`Australia/Sydney`).
32+
- Raw tables are append-only change logs with `_op`, `_seq` and `_loaded_at`. Staging keeps the
33+
latest surviving version of each key with `QUALIFY ROW_NUMBER()`, through the `cdc_latest`
34+
CTE-list macro.
35+
- Payloads arrive as `VARIANT`, `OBJECT` and `ARRAY` columns, plus raw JSON text that needs
36+
`TRY_PARSE_JSON`. Money is in integer minor units with an ISO currency. Timestamps arrive as
37+
`TIMESTAMP_TZ` with offsets, epoch milliseconds in JSON, warehouse-local `TIMESTAMP_NTZ` with an
38+
IANA zone, and one deliberately session-dependent `TIMESTAMP_LTZ`.
39+
40+
## Project shape
41+
42+
- 252 models: 99 staging, 81 intermediate, 66 marts and 6 snapshots. Marts are grouped by area:
43+
core, orders, customers, catalogue, inventory, fulfilment, support, clickstream, growth,
44+
finance, partners and executive.
45+
- 5 seeds, 2 SQL functions (a scalar UDF and a table function) and a Python UDF.
46+
- SQL and Python macros, folder-scoped constants and enums, generic and singular audits, model
47+
schemas, hooks, incremental models with cursor windows, and snapshots with both strategies.
48+
- Rules select `SQBRSQL`, `SQBRGRAPH`, `SQBRCONTRACT`, `SQBRTEST` and the project's own
49+
`XSQBRDUCK` rules, with reviewed exceptions in `sqlbuild_project.toml`. Every model must have at least one unit test (`SQBRTEST202`).
50+
51+
## Tests and scenarios
52+
53+
`tests/unit/` mirrors `models/` and holds 266 unit tests:
54+
55+
- 260 model tests, including cursor-window tests on incremental models, a table-function mock, a
56+
macro mock, a fixture built by a test-only Python macro, an empty fixture for a contracted
57+
source, and a file with two `TEST` blocks;
58+
- 3 macro tests, 2 UDF tests and 1 table-function test, most driven by `VALUES` case tables;
59+
- assertions in 98 tests, including a key check on every mart.
60+
61+
`tests/scenarios/` holds 8 scenarios: one each for catalogue, clickstream, orders, fulfilment,
62+
support and inventory, and two that cross areas (a regional scorecard, and campaign return on ad
63+
spend). Two use `expected_columns "all"`. Expected values are derived by hand from the fixtures,
64+
and every test and scenario passes on Snowflake (see [Status](#status)); the scenarios' captured
65+
snapshots are in `tests/_scenario_snapshots/`. Unit tests and scenarios read only their own fixtures, so they do not depend on the
66+
synthetic data, and capturing a scenario snapshots its fixtures rather than loaded tables.
67+
68+
```bash
69+
cd benchmarks/duckshop/project
70+
sqb test --inspect
71+
```
72+
73+
## Synthetic data
74+
75+
Every raw source is a managed source with a Python loader in `project/python/loaders/`, run by
76+
`sqb load` (or automatically by `sqb build`). The loaders generate one coherent world: about 300
77+
customers, 120 SKUs and 1,200 orders between January and March 2026, with the payments, refunds,
78+
shipments, returns, stock movements, clickstream sessions, support tickets, campaigns and partner
79+
messages that follow from them, about 60,000 rows in all.
80+
81+
- Generation is deterministic: every table draws from its own stream seeded from a fixed seed, so
82+
the same code version always produces the same rows. The `data_scale` project variable
83+
multiplies volume (`"1"` by default).
84+
- Change logs carry inserts, updates and deletes with `_op`, `_seq` and `_loaded_at`. Some versions
85+
arrive days late or out of order, and some are delivered twice.
86+
- JSON payloads have missing keys, JSON nulls next to SQL NULLs, nested arrays, numbers sent as
87+
strings and odd precision. Partner order messages come in two payload versions, some with a
88+
JSON-null order id and the id nested elsewhere.
89+
- Timestamps carry offsets from nine time zones, with orders, stock movements and delivery slots on
90+
the 2026 daylight-saving changes, including wall-clock times that do not exist. Web sessions run
91+
past midnight, and a few events arrive a day or two late.
92+
- Support tickets and comments contain order references, phone numbers, email addresses and
93+
contact reasons for the regular-expression models to find.
94+
95+
`tests/test_synthetic_data.py` checks the loaders offline: determinism, the declared column shape
96+
of every managed source, staging keys and audited columns, references between tables, and the edge
97+
cases above. It needs only `pytest` and `pyyaml`:
98+
99+
```bash
100+
uv run --isolated --no-project --with pytest --with pyyaml pytest benchmarks/duckshop/tests
101+
```
102+
103+
See [RUNBOOK.md](RUNBOOK.md) for loading, building and testing on Snowflake.
104+
105+
## Status
106+
107+
Last verified on Snowflake with SQLBuild 0.175.0 at `data_scale = "1"`:
108+
109+
| Step | Result |
110+
|------|--------|
111+
| `sqb load` | All 76 managed sources load |
112+
| `sqb build` | Passes; one warn-severity audit (`orders_balance_to_payments`) reports findings |
113+
| `sqb test` | 266 of 266 pass |
114+
| `sqb scenario test` | 8 of 8 pass |
115+
| `sqb scenario capture` | Captures all 8; snapshots are in `project/tests/_scenario_snapshots/` |
116+
| `sqb scenario test --local --strict` | 0 of 8 pass: 5 fail in translation, 3 report a stale snapshot |
117+
118+
Local replay translates each scenario's Snowflake SQL to DuckDB on a best-effort basis. These five
119+
failures are the baseline that better translation should remove:
120+
121+
| Scenario | Model | Construct that fails |
122+
|----------|-------|----------------------|
123+
| `category_tree__rolls_sales_up_built_paths` | `stg_categories` | `OBJECT_KEYS` is passed through unchanged |
124+
| `cart_funnel__assigns_furthest_stage_per_session` | `int_cart_funnel` | `::ARRAY` cast of a lambda result becomes an untyped `CAST(... AS ARRAY)` |
125+
| `region_scorecard__combines_sales_and_recognised_revenue` | `rpt_daily_sales_by_region` | `TO_VARCHAR` is passed through unchanged |
126+
| `support_sla__weekly_breaches_and_monthly_reasons` | `rpt_contact_reasons_monthly` | `TO_VARCHAR` is passed through unchanged |
127+
| `stock_outs__finds_longest_run_per_week` | `rpt_stock_outs_weekly` | `DATEADD('day', <window function>, ...)` becomes an invalid `(...) DAY` interval |
128+
129+
The other three scenarios (`campaign_roas__attributes_orders_to_channel_spend`,
130+
`carrier_performance__weekly_rates_from_shipment_facts` and
131+
`payments__fully_refunded_orders_collect_nothing`) never reach translation: their snapshots are
132+
reported stale straight after capture, a SQLBuild issue described under known limitations.
133+
134+
## Expected divergences
135+
136+
Nine models use constructs whose results depend on the engine or the session. They are tagged
137+
`known_divergence` and say so in their descriptions, so translation and replay tools can report
138+
them as expected differences or refusals instead of errors:
139+
140+
| Construct | Models |
141+
|-----------|--------|
142+
| `HASH` | `int_customer_identity`, `dim_customers`, `fct_web_sessions` |
143+
| `TIMESTAMP_LTZ` | `stg_tickets`, `int_ticket_first_response`, `rpt_customer_order_lookup` |
144+
| `CURRENT_TIMESTAMP()` | `int_promotion_performance`, `int_ticket_sla_breaches`, `rpt_customer_order_lookup` |
145+
146+
## Known limitations
147+
148+
These SQLBuild limitations shape parts of the project; each workaround is a reviewed exception or
149+
noted here:
150+
151+
- Fixture checking asks for `CONNECT BY` pseudo-columns and lambda parameters as mock columns, so
152+
`int_category_paths` and `int_cart_funnel` have no unit test. Scenarios cover both, and the
153+
category scenario builds the hierarchy from a source fixture for the same reason.
154+
- Parameterised tests (`parameters` and `cases`) fail with `L005` while `SQBRSQL` is selected, so
155+
repeated cases use multi-row fixtures or `VALUES` case tables instead.
156+
- Macro tests can only reach project-wide macros, so folder-scoped staging macros such as
157+
`cdc_latest` are covered through the models that call them.
158+
- Scenarios cannot build incremental models that use cursor intrinsics, so scenarios that read
159+
them provide the incremental model as a fixture.
160+
- `sqb format` rewrites a typed `WHERE FALSE` fixture into `__empty_fixture()` even where the
161+
relation's schema is not authoritative, so the empty fixtures here use `WHERE 1 = 0`.
162+
- A macro mock must be an unaliased `SELECT` of one string literal, which `SQBRSQL022` rejects.
163+
- A file with several `TEST` blocks cannot satisfy `SQBRTEST102`, which names the file after each
164+
block.
165+
- Loaders that return rows write JSON values as `PARSE_JSON(...)` inside a `VALUES` list, which
166+
Snowflake rejects because `VALUES` accepts only constants. The DuckShop loaders therefore write
167+
their own tables, parsing JSON in the select list of each `INSERT` (`python/loaders/_writer.py`).
168+
- Once a source table exists, Snowflake planning rebinds models to the table's inspected columns.
169+
SQLBuild reads those names in lower case and then quotes them, so they only match quoted
170+
lower-case references, and every unquoted column reference fails with `B002`. The connection
171+
therefore sets `QUOTED_IDENTIFIERS_IGNORE_CASE = true` in `session_parameters`, which the binder
172+
honours. DuckShop quotes no identifier whose case matters, so the setting changes no results.
173+
- Snowflake does not support concurrent microbatch `delete_insert` batches (`P001`), so
174+
`int_web_sessions` runs its batches one at a time.
175+
- `sqb build --no-load` and `auto_load_sources = false` still load managed sources when the build
176+
has no `--select`, because the default selection includes the sources.
177+
- Self-managed loaders always report `rows=0`, since SQLBuild does not see the rows they insert.
178+
- Compilation accepts SQL that Snowflake rejects: `CONNECT_BY_ISLEAF`, which Snowflake does not
179+
have, and a CTE whose body is a bare `VALUES` list. `int_category_paths` finds leaf categories
180+
with a join instead, and test case tables use `SELECT ... FROM VALUES ... AS v(...)`.
181+
- `SQBRSQL018` can only take proof from single-column keys. Append-only feeds redeliver rows, so
182+
their id is unique only together with `_loaded_at` (the `unique_per_delivery` audit), and the
183+
staging models that keep the latest delivery carry reviewed exceptions.
184+
- Source fixtures are not cast to the source's declared types, so a typed function over an
185+
untyped `NULL` fixture column (such as `YEAR(NULL)`) fails on Snowflake; such fixtures cast it.
186+
- The scenario fixture checker treats lambda parameters (`i -> ...`, `(acc, q) -> ...`) as columns
187+
the model reads, so the `cart_funnel` scenario mocks carry placeholder `acc`, `i` and `q` columns.
188+
- A `relationships` audit with a quoted relation name, `to "fct_orders"`, compiles cleanly but
189+
queries an unqualified table and does not wait for it; the relation must be `to __ref(...)`.
190+
- A local scenario snapshot is reported stale (`X602`) straight after capture when a fixture has a
191+
bare `NULL` column, which SQLBuild wraps in a typed `__sqlbuild_partial_fixture` subquery. The
192+
snapshot fingerprint hashes that adapter-rendered SQL: capture renders the alias in Snowflake's
193+
upper case and local replay in DuckDB's lower case, and capture can also cast columns whose types
194+
it only knows from the warehouse. The fingerprint can never match, so the snapshot cannot be
195+
replayed. Three of the eight scenarios are affected.
196+
- A unit test of a table function, and an end-of-build audit, still run when the function or the
197+
relation they read was skipped, and then fail with a missing-object error.
198+
199+
## Compile
200+
201+
Compiling needs no warehouse connection:
202+
203+
```bash
204+
cd benchmarks/duckshop/project
205+
sqb compile
206+
sqb format --check
207+
```
208+
209+
CI compiles a copy of the project with the SQLBuild built from the same pull request and fails on
210+
any error or warning (`tests/e2e/src/sqlbuild/cli/commands/main/compile/test_duckshop_benchmark_compile.py`).
211+
212+
`sqlbuild_project.toml` holds placeholder connection values only. To run against a warehouse, copy
213+
`sqlbuild_local.toml.example` to `sqlbuild_local.toml` (gitignored) and fill in your own details.
214+
215+
## Construct census
216+
217+
After compiling, measure how often each construct family occurs, per layer:
218+
219+
```bash
220+
python benchmarks/duckshop/scripts/census.py
221+
```

0 commit comments

Comments
 (0)