diff --git a/benchmarks/duckshop/README.md b/benchmarks/duckshop/README.md new file mode 100644 index 0000000000..a932874863 --- /dev/null +++ b/benchmarks/duckshop/README.md @@ -0,0 +1,221 @@ +# DuckShop + +DuckShop is the online shop of Mallard & Co., a fictional company that sells duck-themed home goods: +rubber ducks (collectible series and limited editions), duck-print textiles, decoys, pond and garden +supplies, and bath goods. It is written as a dense SQLBuild project in Snowflake SQL, and is a +realistic corpus for exercising the SQLBuild compiler, Rules, unit tests and scenarios, and for +testing translation from Snowflake SQL to other dialects. + +The theme lives only in the data and descriptions: product lines named after duck species +(Mallard, Pintail, Teal, Eider, Merganser), warehouses such as The Pond and The Nest, and loyalty +tiers from Duckling to Golden Duck. Model, column and source names stay conventional, and the +business mechanics are those of any multi-region online shop. + +Everything in it is synthetic: order references use a `DS-O-` prefix, email addresses use +`example.com`-style domains, and every brand, carrier and product name is invented. + +## Layout + +```text +benchmarks/duckshop/ +├── project/ the SQLBuild project (Snowflake adapter), with its data loaders in python/loaders/ +├── scripts/ tooling that is not part of the project: the construct census and cleanup SQL +├── tests/ offline checks for the synthetic data loaders +└── RUNBOOK.md loading, building, testing and capturing on Snowflake, and cleaning up +``` + +## Domain + +- Storefront regions: EU (GBP and EUR), US (USD), APAC (JPY and AUD) and MENA (KWD), so currencies + with 0, 2 and 3 minor-unit digits all occur. Warehouses: The Pond (`Europe/London`), The Nest + (`America/New_York`) and The Marsh (`Australia/Sydney`). +- Raw tables are append-only change logs with `_op`, `_seq` and `_loaded_at`. Staging keeps the + latest surviving version of each key with `QUALIFY ROW_NUMBER()`, through the `cdc_latest` + CTE-list macro. +- Payloads arrive as `VARIANT`, `OBJECT` and `ARRAY` columns, plus raw JSON text that needs + `TRY_PARSE_JSON`. Money is in integer minor units with an ISO currency. Timestamps arrive as + `TIMESTAMP_TZ` with offsets, epoch milliseconds in JSON, warehouse-local `TIMESTAMP_NTZ` with an + IANA zone, and one deliberately session-dependent `TIMESTAMP_LTZ`. + +## Project shape + +- 252 models: 99 staging, 81 intermediate, 66 marts and 6 snapshots. Marts are grouped by area: + core, orders, customers, catalogue, inventory, fulfilment, support, clickstream, growth, + finance, partners and executive. +- 5 seeds, 2 SQL functions (a scalar UDF and a table function) and a Python UDF. +- SQL and Python macros, folder-scoped constants and enums, generic and singular audits, model + schemas, hooks, incremental models with cursor windows, and snapshots with both strategies. +- Rules select `SQBRSQL`, `SQBRGRAPH`, `SQBRCONTRACT`, `SQBRTEST` and the project's own + `XSQBRDUCK` rules, with reviewed exceptions in `sqlbuild_project.toml`. Every model must have at least one unit test (`SQBRTEST202`). + +## Tests and scenarios + +`tests/unit/` mirrors `models/` and holds 266 unit tests: + +- 260 model tests, including cursor-window tests on incremental models, a table-function mock, a + macro mock, a fixture built by a test-only Python macro, an empty fixture for a contracted + source, and a file with two `TEST` blocks; +- 3 macro tests, 2 UDF tests and 1 table-function test, most driven by `VALUES` case tables; +- assertions in 98 tests, including a key check on every mart. + +`tests/scenarios/` holds 8 scenarios: one each for catalogue, clickstream, orders, fulfilment, +support and inventory, and two that cross areas (a regional scorecard, and campaign return on ad +spend). Two use `expected_columns "all"`. Expected values are derived by hand from the fixtures, +and every test and scenario passes on Snowflake (see [Status](#status)); the scenarios' captured +snapshots are in `tests/_scenario_snapshots/`. Unit tests and scenarios read only their own fixtures, so they do not depend on the +synthetic data, and capturing a scenario snapshots its fixtures rather than loaded tables. + +```bash +cd benchmarks/duckshop/project +sqb test --inspect +``` + +## Synthetic data + +Every raw source is a managed source with a Python loader in `project/python/loaders/`, run by +`sqb load` (or automatically by `sqb build`). The loaders generate one coherent world: about 300 +customers, 120 SKUs and 1,200 orders between January and March 2026, with the payments, refunds, +shipments, returns, stock movements, clickstream sessions, support tickets, campaigns and partner +messages that follow from them, about 60,000 rows in all. + +- Generation is deterministic: every table draws from its own stream seeded from a fixed seed, so + the same code version always produces the same rows. The `data_scale` project variable + multiplies volume (`"1"` by default). +- Change logs carry inserts, updates and deletes with `_op`, `_seq` and `_loaded_at`. Some versions + arrive days late or out of order, and some are delivered twice. +- JSON payloads have missing keys, JSON nulls next to SQL NULLs, nested arrays, numbers sent as + strings and odd precision. Partner order messages come in two payload versions, some with a + JSON-null order id and the id nested elsewhere. +- Timestamps carry offsets from nine time zones, with orders, stock movements and delivery slots on + the 2026 daylight-saving changes, including wall-clock times that do not exist. Web sessions run + past midnight, and a few events arrive a day or two late. +- Support tickets and comments contain order references, phone numbers, email addresses and + contact reasons for the regular-expression models to find. + +`tests/test_synthetic_data.py` checks the loaders offline: determinism, the declared column shape +of every managed source, staging keys and audited columns, references between tables, and the edge +cases above. It needs only `pytest` and `pyyaml`: + +```bash +uv run --isolated --no-project --with pytest --with pyyaml pytest benchmarks/duckshop/tests +``` + +See [RUNBOOK.md](RUNBOOK.md) for loading, building and testing on Snowflake. + +## Status + +Last verified on Snowflake with SQLBuild 0.175.0 at `data_scale = "1"`: + +| Step | Result | +|------|--------| +| `sqb load` | All 76 managed sources load | +| `sqb build` | Passes; one warn-severity audit (`orders_balance_to_payments`) reports findings | +| `sqb test` | 266 of 266 pass | +| `sqb scenario test` | 8 of 8 pass | +| `sqb scenario capture` | Captures all 8; snapshots are in `project/tests/_scenario_snapshots/` | +| `sqb scenario test --local --strict` | 0 of 8 pass: 5 fail in translation, 3 report a stale snapshot | + +Local replay translates each scenario's Snowflake SQL to DuckDB on a best-effort basis. These five +failures are the baseline that better translation should remove: + +| Scenario | Model | Construct that fails | +|----------|-------|----------------------| +| `category_tree__rolls_sales_up_built_paths` | `stg_categories` | `OBJECT_KEYS` is passed through unchanged | +| `cart_funnel__assigns_furthest_stage_per_session` | `int_cart_funnel` | `::ARRAY` cast of a lambda result becomes an untyped `CAST(... AS ARRAY)` | +| `region_scorecard__combines_sales_and_recognised_revenue` | `rpt_daily_sales_by_region` | `TO_VARCHAR` is passed through unchanged | +| `support_sla__weekly_breaches_and_monthly_reasons` | `rpt_contact_reasons_monthly` | `TO_VARCHAR` is passed through unchanged | +| `stock_outs__finds_longest_run_per_week` | `rpt_stock_outs_weekly` | `DATEADD('day', , ...)` becomes an invalid `(...) DAY` interval | + +The other three scenarios (`campaign_roas__attributes_orders_to_channel_spend`, +`carrier_performance__weekly_rates_from_shipment_facts` and +`payments__fully_refunded_orders_collect_nothing`) never reach translation: their snapshots are +reported stale straight after capture, a SQLBuild issue described under known limitations. + +## Expected divergences + +Nine models use constructs whose results depend on the engine or the session. They are tagged +`known_divergence` and say so in their descriptions, so translation and replay tools can report +them as expected differences or refusals instead of errors: + +| Construct | Models | +|-----------|--------| +| `HASH` | `int_customer_identity`, `dim_customers`, `fct_web_sessions` | +| `TIMESTAMP_LTZ` | `stg_tickets`, `int_ticket_first_response`, `rpt_customer_order_lookup` | +| `CURRENT_TIMESTAMP()` | `int_promotion_performance`, `int_ticket_sla_breaches`, `rpt_customer_order_lookup` | + +## Known limitations + +These SQLBuild limitations shape parts of the project; each workaround is a reviewed exception or +noted here: + +- Fixture checking asks for `CONNECT BY` pseudo-columns and lambda parameters as mock columns, so + `int_category_paths` and `int_cart_funnel` have no unit test. Scenarios cover both, and the + category scenario builds the hierarchy from a source fixture for the same reason. +- Parameterised tests (`parameters` and `cases`) fail with `L005` while `SQBRSQL` is selected, so + repeated cases use multi-row fixtures or `VALUES` case tables instead. +- Macro tests can only reach project-wide macros, so folder-scoped staging macros such as + `cdc_latest` are covered through the models that call them. +- Scenarios cannot build incremental models that use cursor intrinsics, so scenarios that read + them provide the incremental model as a fixture. +- `sqb format` rewrites a typed `WHERE FALSE` fixture into `__empty_fixture()` even where the + relation's schema is not authoritative, so the empty fixtures here use `WHERE 1 = 0`. +- A macro mock must be an unaliased `SELECT` of one string literal, which `SQBRSQL022` rejects. +- A file with several `TEST` blocks cannot satisfy `SQBRTEST102`, which names the file after each + block. +- Loaders that return rows write JSON values as `PARSE_JSON(...)` inside a `VALUES` list, which + Snowflake rejects because `VALUES` accepts only constants. The DuckShop loaders therefore write + their own tables, parsing JSON in the select list of each `INSERT` (`python/loaders/_writer.py`). +- Once a source table exists, Snowflake planning rebinds models to the table's inspected columns. + SQLBuild reads those names in lower case and then quotes them, so they only match quoted + lower-case references, and every unquoted column reference fails with `B002`. The connection + therefore sets `QUOTED_IDENTIFIERS_IGNORE_CASE = true` in `session_parameters`, which the binder + honours. DuckShop quotes no identifier whose case matters, so the setting changes no results. +- Snowflake does not support concurrent microbatch `delete_insert` batches (`P001`), so + `int_web_sessions` runs its batches one at a time. +- `sqb build --no-load` and `auto_load_sources = false` still load managed sources when the build + has no `--select`, because the default selection includes the sources. +- Self-managed loaders always report `rows=0`, since SQLBuild does not see the rows they insert. +- Compilation accepts SQL that Snowflake rejects: `CONNECT_BY_ISLEAF`, which Snowflake does not + have, and a CTE whose body is a bare `VALUES` list. `int_category_paths` finds leaf categories + with a join instead, and test case tables use `SELECT ... FROM VALUES ... AS v(...)`. +- `SQBRSQL018` can only take proof from single-column keys. Append-only feeds redeliver rows, so + their id is unique only together with `_loaded_at` (the `unique_per_delivery` audit), and the + staging models that keep the latest delivery carry reviewed exceptions. +- Source fixtures are not cast to the source's declared types, so a typed function over an + untyped `NULL` fixture column (such as `YEAR(NULL)`) fails on Snowflake; such fixtures cast it. +- The scenario fixture checker treats lambda parameters (`i -> ...`, `(acc, q) -> ...`) as columns + the model reads, so the `cart_funnel` scenario mocks carry placeholder `acc`, `i` and `q` columns. +- A `relationships` audit with a quoted relation name, `to "fct_orders"`, compiles cleanly but + queries an unqualified table and does not wait for it; the relation must be `to __ref(...)`. +- A local scenario snapshot is reported stale (`X602`) straight after capture when a fixture has a + bare `NULL` column, which SQLBuild wraps in a typed `__sqlbuild_partial_fixture` subquery. The + snapshot fingerprint hashes that adapter-rendered SQL: capture renders the alias in Snowflake's + upper case and local replay in DuckDB's lower case, and capture can also cast columns whose types + it only knows from the warehouse. The fingerprint can never match, so the snapshot cannot be + replayed. Three of the eight scenarios are affected. +- A unit test of a table function, and an end-of-build audit, still run when the function or the + relation they read was skipped, and then fail with a missing-object error. + +## Compile + +Compiling needs no warehouse connection: + +```bash +cd benchmarks/duckshop/project +sqb compile +sqb format --check +``` + +CI compiles a copy of the project with the SQLBuild built from the same pull request and fails on +any error or warning (`tests/e2e/src/sqlbuild/cli/commands/main/compile/test_duckshop_benchmark_compile.py`). + +`sqlbuild_project.toml` holds placeholder connection values only. To run against a warehouse, copy +`sqlbuild_local.toml.example` to `sqlbuild_local.toml` (gitignored) and fill in your own details. + +## Construct census + +After compiling, measure how often each construct family occurs, per layer: + +```bash +python benchmarks/duckshop/scripts/census.py +``` diff --git a/benchmarks/duckshop/RUNBOOK.md b/benchmarks/duckshop/RUNBOOK.md new file mode 100644 index 0000000000..857f1acce7 --- /dev/null +++ b/benchmarks/duckshop/RUNBOOK.md @@ -0,0 +1,173 @@ +# DuckShop runbook + +How to load DuckShop's synthetic data into Snowflake, build and test the project, capture the +scenario snapshots, and clean up afterwards. Run every command from `benchmarks/duckshop/project` +unless it says otherwise. + +## 1. Configure the connection + +Install SQLBuild with Snowflake support (`pip install "sqlbuild[snowflake]"`), then create the +untracked local configuration: + +```bash +cd benchmarks/duckshop/project +cp sqlbuild_local.toml.example sqlbuild_local.toml +``` + +Fill in the account, user, authenticator, role and warehouse, and the `database` and `schema` of +`[targets.dev]`. Keep the project's `session_parameters` setting on the connection (see Known +limitations in the README): `sqlbuild_local.toml` adds to the `[connections.warehouse]` table in +`sqlbuild_project.toml` rather than replacing it, so leave `session_parameters` out of the local +file. `sqlbuild_local.toml` is gitignored; never commit it, and never copy its values +into tracked files, commit messages or pull requests. + +Check the connection, then confirm the project compiles: + +```bash +sqb debug +sqb compile +``` + +`sqb compile` must report `0 errors, 0 warnings`. + +## 2. Check the schema before the first build + +A build creates or replaces about 340 objects in the target schema: 252 models, 5 seeds, 76 raw +source tables and 3 functions. If the schema holds other work, make sure none of it has a DuckShop +name: + +```bash +sqb compile --json > target/duckshop-compile.json +python ../scripts/relations.py check target/duckshop-compile.json > target/check.sql +sqb query --file target/check.sql --no-limit +``` + +The query lists every existing table, view or function with a DuckShop name, which is everything +step 7 drops, and any SQLBuild state tables (`_SQLBUILD_*`). Before the first build it should list +no DuckShop names; if it does, stop and pick another schema. Note which state tables already existed, for the cleanup in step 7. + +## 3. Load the synthetic data + +```bash +sqb load --concurrency 4 +``` + +All 76 managed sources load, about 60,000 rows in total. Each loader replaces its table, so loading +again gives identical data. The volume multiplier is the `data_scale` project variable; for a larger +run, set it in `sqlbuild_local.toml`: + +```toml +[vars] +data_scale = "10" +``` + +## 4. Build + +```bash +sqb build --concurrency 8 +``` + +The build reloads the managed sources first. SQLBuild 0.175.0 does this even with `--no-load` +when no `--select` is given, because the default selection includes the sources themselves; the +reload writes the same rows as step 3, so it only costs time. The first build replays the +clickstream microbatch models from the start of their history, about 90 six-hour batches, so it +takes longer than later builds. + +Audits run during the build. Error-severity failures stop the dependent models and fail the +command. A few warn-severity audits, such as `orders_balance_to_payments` (orders part-paid with +gift cards) and the buyer match rate on partner orders, may report findings without failing. + +### Rebuilding after a model change + +Incremental models are forward-only by default: when a model's SQL changes, rows already built keep +their old results and only new cursor ranges use the new query, and the plan prints a warning that +says so. Each model's `replay_on_change` setting decides how much history its own change +reprocesses. Upstream changes never replay a downstream model. + +DuckShop's history is small and fixed (January to March 2026 at every `data_scale`), so its +incremental models replay all of it when they change: + +| Model | Policy | +|-------|--------| +| `stg_web_events`, `int_web_sessions` (microbatch) | `replay_on_change full` | +| `fct_web_sessions`, `fct_support_tickets`, `fct_inventory_daily` | `replay_on_change bounded-120d`, which covers all synthetic history | +| `stg_app_events` (append) | forward-only; replaying an append model would duplicate rows | + +`full` is not used for the three non-microbatch models because a full rebuild of a model that +filters on `__cursor_start()` needs an explicit interval. To reprocess a model by hand, for example +after an upstream change or for `stg_app_events`: + +```bash +# Microbatch models +sqb build --select stg_web_events --full-refresh +# Models that filter on cursor intrinsics without microbatches: give the interval explicitly +sqb build --select fct_web_sessions --full-refresh \ + --start-cursor-ts 2026-03-01T00:00:00 --end-cursor-ts 2026-04-02T00:00:00 +``` + +## 5. Run the unit tests and scenarios on Snowflake + +```bash +sqb test --concurrency 8 +sqb scenario test --concurrency 4 +``` + +Unit tests and scenarios read only their own fixtures, not the loaded data, so they can run before +or after the build. Their expected values were derived by hand; all 266 tests and 8 scenarios pass +on Snowflake. Report any failure rather than editing the expectations to match. + +## 6. Capture the scenario snapshots and commit them + +```bash +sqb scenario capture +sqb scenario test --local --strict +``` + +Capture materialises each scenario's fixtures in Snowflake, downloads them as JSONL into +`tests/_scenario_snapshots/`, and drops what it created. The local run then replays all 8 +scenarios on your machine from those files, with no warehouse connection; `--strict` fails on a +missing or stale snapshot instead of skipping it. + +With SQLBuild 0.175.0 the local run is expected to end with `ERROR=8`: five scenarios fail while +translating Snowflake SQL to DuckDB, and three report `X602` (stale snapshot) because of a +SQLBuild fingerprint issue. Both are described in the README's status section. Capturing again does +not clear the `X602` errors. + +The snapshots hold only the synthetic fixture rows and column types, but check them before +committing. From the repository root: + +```bash +grep -ril "\|" benchmarks/duckshop/project/tests/_scenario_snapshots +git add benchmarks/duckshop/project/tests/_scenario_snapshots +git commit -m "test(benchmarks): capture DuckShop scenario snapshots" +``` + +The `grep` must print nothing. The repository's publication checks run on commit and push as usual. + +## 7. Clean up + +Drop everything the project created, by name: + +```bash +sqb compile --json > target/duckshop-compile.json +python ../scripts/relations.py drop target/duckshop-compile.json > target/drop.sql +sqb query --file target/drop.sql +sqb clean +``` + +`drop.sql` is one Snowflake Scripting block of `DROP ... IF EXISTS` statements for every model, +snapshot, seed, raw source table and function, so it is safe to run twice. Scenario runs and +captures already drop their own relations. + +SQLBuild also keeps state tables (`_SQLBUILD_*`) in the schema. If step 2 showed none before the +first build, they belong to DuckShop alone and you can drop them too. List them, then drop each +one by name: + +```bash +sqb query "SHOW TABLES LIKE '\\_SQLBUILD\\_%'" +sqb query "DROP TABLE IF EXISTS " +``` + +`sqb janitor` is not used here. It archives relations the current project no longer produces and +keeps the ones it does, so it would leave DuckShop's own tables in place. In a schema that other +SQLBuild projects also write to, it could archive their relations instead. diff --git a/benchmarks/duckshop/project/.gitignore b/benchmarks/duckshop/project/.gitignore new file mode 100644 index 0000000000..2a4e91ca5e --- /dev/null +++ b/benchmarks/duckshop/project/.gitignore @@ -0,0 +1,4 @@ +sqlbuild_local.toml +target/ +logs/ +.sqlbuild/ diff --git a/benchmarks/duckshop/project/audits/generic/unique_per_delivery.sql b/benchmarks/duckshop/project/audits/generic/unique_per_delivery.sql new file mode 100644 index 0000000000..896aad7cd4 --- /dev/null +++ b/benchmarks/duckshop/project/audits/generic/unique_per_delivery.sql @@ -0,0 +1,19 @@ +AUDIT (); + +-- Append-only feeds are delivered at least once, so an id may arrive again in a later load. Within +-- one load (one `_loaded_at`) each id must still appear once; staging keeps the latest delivery. +WITH final AS ( + SELECT + @column, + _loaded_at, + COUNT(*) AS delivery_count + FROM @relation + WHERE @column IS NOT NULL + GROUP BY + @column, + _loaded_at + HAVING COUNT(*) > 1 +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/constants/history_start_date.sql b/benchmarks/duckshop/project/constants/history_start_date.sql new file mode 100644 index 0000000000..f9ce3e40f4 --- /dev/null +++ b/benchmarks/duckshop/project/constants/history_start_date.sql @@ -0,0 +1 @@ +CONSTANT (name history_start_date, value "2025-01-01"); diff --git a/benchmarks/duckshop/project/enums/fulfilment_method.sql b/benchmarks/duckshop/project/enums/fulfilment_method.sql new file mode 100644 index 0000000000..a2a711d151 --- /dev/null +++ b/benchmarks/duckshop/project/enums/fulfilment_method.sql @@ -0,0 +1,8 @@ +ENUM ( + name fulfilment_method, + members ( + HOME_DELIVERY "home_delivery", + CLICK_AND_COLLECT "click_and_collect", + PARTNER "partner", + ), +); diff --git a/benchmarks/duckshop/project/enums/order_status.sql b/benchmarks/duckshop/project/enums/order_status.sql new file mode 100644 index 0000000000..9ed7759d16 --- /dev/null +++ b/benchmarks/duckshop/project/enums/order_status.sql @@ -0,0 +1,4 @@ +ENUM ( + name order_status, + members [PENDING, PAID, PACKED, SHIPPED, DELIVERED, CANCELLED, RETURNED], +); diff --git a/benchmarks/duckshop/project/enums/return_reason.sql b/benchmarks/duckshop/project/enums/return_reason.sql new file mode 100644 index 0000000000..a2a67872df --- /dev/null +++ b/benchmarks/duckshop/project/enums/return_reason.sql @@ -0,0 +1,12 @@ +ENUM ( + name return_reason, + members ( + DAMAGED "damaged", + WRONG_ITEM "wrong_item", + SIZE_FIT "size_fit", + NOT_AS_DESCRIBED "not_as_described", + CHANGED_MIND "changed_mind", + LATE_DELIVERY "late_delivery", + OTHER "other", + ), +); diff --git a/benchmarks/duckshop/project/enums/sales_channel.sql b/benchmarks/duckshop/project/enums/sales_channel.sql new file mode 100644 index 0000000000..7f0816b8e7 --- /dev/null +++ b/benchmarks/duckshop/project/enums/sales_channel.sql @@ -0,0 +1,10 @@ +ENUM ( + name sales_channel, + members ( + WEB "web", + APP "app", + STORE "store", + PARTNER "partner", + PHONE "phone", + ), +); diff --git a/benchmarks/duckshop/project/enums/ticket_priority.sql b/benchmarks/duckshop/project/enums/ticket_priority.sql new file mode 100644 index 0000000000..c49e356ce9 --- /dev/null +++ b/benchmarks/duckshop/project/enums/ticket_priority.sql @@ -0,0 +1,4 @@ +ENUM ( + name ticket_priority, + members (URGENT 1, HIGH 2, NORMAL 3, LOW 4), +); diff --git a/benchmarks/duckshop/project/functions/python/luhn_valid.py b/benchmarks/duckshop/project/functions/python/luhn_valid.py new file mode 100644 index 0000000000..07be03ffd1 --- /dev/null +++ b/benchmarks/duckshop/project/functions/python/luhn_valid.py @@ -0,0 +1,21 @@ +from sqlbuild.functions import udf + + +@udf( + arguments={"digits": "VARCHAR"}, + returns="BOOLEAN", + runtime_version="3.11", +) +def main(digits: str | None) -> bool | None: + """Whether a string of digits passes the Luhn checksum; NULL for empty or non-numeric input.""" + if not digits or not digits.isdigit(): + return None + total = 0 + for index, char in enumerate(reversed(digits)): + value = int(char) + if index % 2 == 1: + value *= 2 + if value > 9: + value -= 9 + total += value + return total % 10 == 0 diff --git a/benchmarks/duckshop/project/functions/sql/table_fn__customer_orders.sql b/benchmarks/duckshop/project/functions/sql/table_fn__customer_orders.sql new file mode 100644 index 0000000000..18b1302d48 --- /dev/null +++ b/benchmarks/duckshop/project/functions/sql/table_fn__customer_orders.sql @@ -0,0 +1,32 @@ +FUNCTION ( + description "Orders placed by one customer with their USD value, status and sequence number", + arguments (p_customer_id VARCHAR), + returns table ( + order_id VARCHAR, + placed_at TIMESTAMP_TZ, + order_status VARCHAR, + amount_usd "NUMBER(18, 2)", + order_number INT, + is_first_order BOOLEAN + ) +); + +WITH orders AS ( + SELECT * + FROM __ref("fct_orders") +), + +final AS ( + SELECT + o.order_id, + o.placed_at, + o.order_status, + o.amount_usd, + o.order_number, + o.is_first_order + FROM orders AS o + WHERE o.customer_id = p_customer_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/functions/sql/udf__sku_family.sql b/benchmarks/duckshop/project/functions/sql/udf__sku_family.sql new file mode 100644 index 0000000000..f71f6398fc --- /dev/null +++ b/benchmarks/duckshop/project/functions/sql/udf__sku_family.sql @@ -0,0 +1,7 @@ +FUNCTION ( + description "Product family prefix of a canonical SKU, such as DECOY for DECOY-TEAL-08", + arguments (sku VARCHAR), + returns VARCHAR, +); + +UPPER(REGEXP_SUBSTR(sku, '^[A-Za-z]+')) diff --git a/benchmarks/duckshop/project/macros/python/rounding.py b/benchmarks/duckshop/project/macros/python/rounding.py new file mode 100644 index 0000000000..da71d6f45f --- /dev/null +++ b/benchmarks/duckshop/project/macros/python/rounding.py @@ -0,0 +1,8 @@ +"""Adapter-aware rounding.""" + + +def round_half_even(ctx, expression: str, places: int = 2) -> str: + """Banker's rounding of a decimal expression, in the adapter's native form when it has one.""" + if ctx.adapter_name == "snowflake": + return f"ROUND({expression}, {places}, 'HALF_TO_EVEN')" + return f"ROUND_EVEN({expression}, {places})" diff --git a/benchmarks/duckshop/project/macros/sql/money_columns.sql b/benchmarks/duckshop/project/macros/sql/money_columns.sql new file mode 100644 index 0000000000..913bce67bf --- /dev/null +++ b/benchmarks/duckshop/project/macros/sql/money_columns.sql @@ -0,0 +1,19 @@ +MACRO ( + description """ + Major-unit amount and its USD value from a minor-unit amount, currency exponent and USD rate + """, + arguments ( + src RELATION ( + amount_minor NUMBER(38, 0), + minor_unit_exponent NUMBER(38, 0), + rate_to_usd NUMBER(18, 8), + ), + ), +); + +SELECT + @sql.money_from_minor(@src.amount_minor, @src.minor_unit_exponent, places := 6) AS amount, + ROUND( + @sql.money_from_minor(@src.amount_minor, @src.minor_unit_exponent, places := 6) * @src.rate_to_usd, + 2 + )::NUMBER(18, 2) AS amount_usd diff --git a/benchmarks/duckshop/project/macros/sql/money_from_minor.sql b/benchmarks/duckshop/project/macros/sql/money_from_minor.sql new file mode 100644 index 0000000000..e844a7c3e5 --- /dev/null +++ b/benchmarks/duckshop/project/macros/sql/money_from_minor.sql @@ -0,0 +1,7 @@ +MACRO ( + description "Converts an integer amount in minor units to major units for a currency exponent", + arguments (amount NUMBER(38, 0), exponent NUMBER(38, 0), places INT CONST DEFAULT 2), + returns NUMBER(38, 6), +); + +ROUND(@sql.safe_divide(@amount, POWER(10, @exponent)::NUMBER(38, 0)), @places)::NUMBER(38, 6) diff --git a/benchmarks/duckshop/project/macros/sql/safe_divide.sql b/benchmarks/duckshop/project/macros/sql/safe_divide.sql new file mode 100644 index 0000000000..5f8085cefc --- /dev/null +++ b/benchmarks/duckshop/project/macros/sql/safe_divide.sql @@ -0,0 +1,9 @@ +MACRO ( + description "Division that returns NULL instead of failing on a zero denominator", + arguments (numerator NUMBER(38, 6), denominator NUMBER(38, 6)), + returns NUMBER(38, 6), +); + +( + @numerator / NULLIF(@denominator, 0) +)::NUMBER(38, 6) diff --git a/benchmarks/duckshop/project/models/intermediate/_sqlbuild/audits/generic/match_rate.sql b/benchmarks/duckshop/project/models/intermediate/_sqlbuild/audits/generic/match_rate.sql new file mode 100644 index 0000000000..1b06415e6d --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/_sqlbuild/audits/generic/match_rate.sql @@ -0,0 +1,29 @@ +AUDIT ( + evaluation measurement, + value match_rate, + sample_count total_rows, + sample_unit rows, +); + +MEASURE ( + WITH final AS ( + SELECT + COUNT(*) AS total_rows, + 100.0 * AVG(IFF(@matched, 1, 0)) AS match_rate + FROM @relation +) + +SELECT * +FROM final +); + +EVIDENCE ( + WITH final AS ( + SELECT * + FROM @relation + WHERE NOT COALESCE(@matched, FALSE) +) + +SELECT * +FROM final +); diff --git a/benchmarks/duckshop/project/models/intermediate/_sqlbuild/constants/supported_regions.sql b/benchmarks/duckshop/project/models/intermediate/_sqlbuild/constants/supported_regions.sql new file mode 100644 index 0000000000..11a320e4ba --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/_sqlbuild/constants/supported_regions.sql @@ -0,0 +1 @@ +CONSTANT (name supported_regions, value ["EU", "US", "APAC", "MENA"]); diff --git a/benchmarks/duckshop/project/models/intermediate/_sqlbuild/macros/sql/date_spine.sql b/benchmarks/duckshop/project/models/intermediate/_sqlbuild/macros/sql/date_spine.sql new file mode 100644 index 0000000000..2982029732 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/_sqlbuild/macros/sql/date_spine.sql @@ -0,0 +1,18 @@ +MACRO ( + description "One row per calendar day from a start date for a fixed number of days", + arguments (start_date VARCHAR CONST, day_count INT CONST), +); + +WITH spine_offsets AS ( + SELECT ARRAY_GENERATE_RANGE(0, @day_count) AS day_offsets +), + +date_spine AS ( + SELECT + DATEADD( + 'day', + d.value::INT, + @start_date::DATE + )::DATE AS calendar_date + FROM spine_offsets AS o, LATERAL FLATTEN(input => o.day_offsets) AS d +) diff --git a/benchmarks/duckshop/project/models/intermediate/_sqlbuild/macros/sql/in_regions.sql b/benchmarks/duckshop/project/models/intermediate/_sqlbuild/macros/sql/in_regions.sql new file mode 100644 index 0000000000..149d336bdb --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/_sqlbuild/macros/sql/in_regions.sql @@ -0,0 +1,7 @@ +MACRO ( + description "Whether a region code is one of a compile-time list of regions", + arguments (region VARCHAR, regions LIST VARCHAR), + returns BOOLEAN, +); + +COALESCE(@region IN @regions, FALSE) diff --git a/benchmarks/duckshop/project/models/intermediate/_sqlbuild/macros/sql/local_business_date.sql b/benchmarks/duckshop/project/models/intermediate/_sqlbuild/macros/sql/local_business_date.sql new file mode 100644 index 0000000000..f3190048f2 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/_sqlbuild/macros/sql/local_business_date.sql @@ -0,0 +1,7 @@ +MACRO ( + description "Calendar date of a timestamp in a storefront's IANA time zone", + arguments (ts TIMESTAMP_TZ, tz VARCHAR), + returns DATE, +); + +CONVERT_TIMEZONE(@tz, @ts)::DATE diff --git a/benchmarks/duckshop/project/models/intermediate/catalogue/int_bundle_pricing.sql b/benchmarks/duckshop/project/models/intermediate/catalogue/int_bundle_pricing.sql new file mode 100644 index 0000000000..8d8afa7980 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/catalogue/int_bundle_pricing.sql @@ -0,0 +1,149 @@ +MODEL ( + description """ + One row per bundle and currency with the summed list price of its required components, the + bundle price under its pricing mode, and the saving offered. + """, + columns ( + bundle_price_key (type VARCHAR, nullable false, audits [not_null, unique]), + bundle_sku (type VARCHAR, audits [not_null]), + currency_code (type VARCHAR, audits [not_null]), + component_count (type INT), + components_list_minor (type INT), + bundle_price_minor (type INT), + saving_minor (type INT), + saving_pct (type NUMBER(9, 4)), + has_unpriced_component (type BOOLEAN), + ), +); + +WITH bundles AS ( + SELECT * + FROM __ref("stg_bundles") +), + +components AS ( + SELECT * + FROM __ref("stg_bundle_components") +), + +price_history AS ( + SELECT * + FROM __ref("int_price_history") +), + +current_prices AS ( + SELECT + ph.sku, + ph.currency_code, + ph.list_amount_minor + FROM price_history AS ph + WHERE ph.effective_valid_to IS NULL +), + +required_components AS ( + SELECT + c.bundle_sku, + c.component_sku, + c.quantity + FROM components AS c + WHERE NOT c.is_optional +), + +priced_components AS ( + SELECT + rc.bundle_sku, + cp.currency_code, + rc.quantity * cp.list_amount_minor AS extended_minor, + cp.list_amount_minor AS unit_list_minor + FROM required_components AS rc + INNER JOIN current_prices AS cp + ON rc.component_sku = cp.sku +), + +component_counts AS ( + SELECT + rc.bundle_sku, + COUNT(*) AS component_count + FROM required_components AS rc + GROUP BY + rc.bundle_sku +), + +totals AS ( + SELECT + pc.bundle_sku, + pc.currency_code, + COUNT(*) AS priced_count, + SUM(pc.extended_minor) AS components_list_minor, + MIN(pc.unit_list_minor) AS cheapest_unit_minor + FROM priced_components AS pc + GROUP BY + pc.bundle_sku, + pc.currency_code +), + +final AS ( + SELECT + ( + t.bundle_sku || ':' || t.currency_code + )::VARCHAR AS bundle_price_key, + t.bundle_sku, + t.currency_code, + cc.component_count::INT AS component_count, + t.components_list_minor::INT AS components_list_minor, + CASE b.pricing_mode + WHEN 'fixed' + THEN IFF(b.pricing_currency_code = t.currency_code, b.fixed_amount_minor, NULL) + WHEN 'percent_off' + THEN ROUND(t.components_list_minor + * ( + 1 - b.percent_off / 100 + )) + WHEN 'cheapest_free' + THEN t.components_list_minor - t.cheapest_unit_minor + END::INT AS bundle_price_minor, + ( + t.components_list_minor + - CASE b.pricing_mode + WHEN 'fixed' + THEN IFF(b.pricing_currency_code = t.currency_code, b.fixed_amount_minor, NULL) + WHEN 'percent_off' + THEN ROUND(t.components_list_minor + * ( + 1 - b.percent_off / 100 + )) + WHEN 'cheapest_free' + THEN t.components_list_minor - t.cheapest_unit_minor + END + )::INT AS saving_minor, + ROUND( + 100 + * ( + 1 + - CASE b.pricing_mode + WHEN 'fixed' + THEN IFF(b.pricing_currency_code = t.currency_code, b.fixed_amount_minor, NULL) + WHEN 'percent_off' + THEN ROUND(t.components_list_minor + * ( + 1 - b.percent_off / 100 + )) + WHEN 'cheapest_free' + THEN t.components_list_minor - t.cheapest_unit_minor + END + / NULLIF(t.components_list_minor, 0) + ), + 4 + )::NUMBER(9, 4) AS saving_pct, + ( + t.priced_count < cc.component_count + ) AS has_unpriced_component + FROM totals AS t + INNER JOIN bundles AS b + ON t.bundle_sku = b.bundle_sku + INNER JOIN component_counts AS cc + ON t.bundle_sku = cc.bundle_sku +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/catalogue/int_category_paths.sql b/benchmarks/duckshop/project/models/intermediate/catalogue/int_category_paths.sql new file mode 100644 index 0000000000..d702f87739 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/catalogue/int_category_paths.sql @@ -0,0 +1,69 @@ +MODEL ( + description """ + One row per category node with its depth, its full slug path from the root and its root + category, walked with CONNECT BY over the parent links. Snowflake has no CONNECT_BY_ISLEAF, so a + node is a leaf when no category names it as a parent. + """, + columns ( + category_id (type VARCHAR, nullable false, audits [not_null, unique]), + parent_category_id (type VARCHAR), + root_category_id (type VARCHAR), + slug (type VARCHAR), + display_name (type VARCHAR), + depth (type INT), + slug_path (type VARCHAR), + is_leaf (type BOOLEAN), + sibling_rank (type INT), + ), +); + +WITH categories AS ( + SELECT * + FROM __ref("stg_categories") +), + +tree AS ( + SELECT + c.category_id, + c.parent_category_id, + c.slug, + c.display_name, + c.sort_order, + LEVEL AS depth, + SYS_CONNECT_BY_PATH(c.slug, '/') AS slug_path, + CONNECT_BY_ROOT c.category_id AS root_category_id + FROM categories AS c +START WITH c.is_root +CONNECT BY PRIOR c.category_id = c.parent_category_id +), + +parents AS ( + SELECT DISTINCT + c.parent_category_id + FROM categories AS c + WHERE c.parent_category_id IS NOT NULL +), + +final AS ( + SELECT + t.category_id, + t.parent_category_id, + t.root_category_id::VARCHAR AS root_category_id, + t.slug, + t.display_name, + t.depth::INT AS depth, + LTRIM(t.slug_path, '/')::VARCHAR AS slug_path, + ( + p.parent_category_id IS NULL + ) AS is_leaf, + RANK() OVER ( + PARTITION BY t.parent_category_id + ORDER BY t.sort_order, t.slug + )::INT AS sibling_rank + FROM tree AS t + LEFT JOIN parents AS p + ON t.category_id = p.parent_category_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/catalogue/int_price_history.sql b/benchmarks/duckshop/project/models/intermediate/catalogue/int_price_history.sql new file mode 100644 index 0000000000..fba778bfb2 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/catalogue/int_price_history.sql @@ -0,0 +1,83 @@ +MODEL ( + description """ + One row per list price period of a SKU in a currency, with the previous price, the size of the + change, and whether it was a markdown, with periods closed by the next price where the source + left them open. + """, + columns ( + price_list_id (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + currency_code (type VARCHAR), + list_amount_minor (type INT), + valid_from (type TIMESTAMP_TZ), + effective_valid_to (type TIMESTAMP_TZ), + previous_amount_minor (type INT), + change_pct (type NUMBER(9, 4)), + is_markdown (type BOOLEAN), + price_version (type INT), + days_at_price (type INT), + lowest_price_30d_minor (type INT), + ), +); + +WITH price_lists AS ( + SELECT * + FROM __ref("stg_price_lists") +), + +ordered AS ( + SELECT + p.price_list_id, + p.sku, + p.currency_code, + p.list_amount_minor, + p.valid_from, + COALESCE( + p.valid_to, + LEAD(p.valid_from) OVER ( + PARTITION BY p.sku, p.currency_code + ORDER BY p.valid_from, p.price_list_id + ) + ) AS effective_valid_to, + LAG(p.list_amount_minor) OVER ( + PARTITION BY p.sku, p.currency_code + ORDER BY p.valid_from, p.price_list_id + ) AS previous_amount_minor, + ROW_NUMBER() OVER ( + PARTITION BY p.sku, p.currency_code + ORDER BY p.valid_from, p.price_list_id + ) AS price_version, + MIN(p.list_amount_minor) OVER ( + PARTITION BY p.sku, p.currency_code + ORDER BY p.valid_from + RANGE BETWEEN INTERVAL '30 days' PRECEDING AND CURRENT ROW + ) AS lowest_price_30d_minor + FROM price_lists AS p +), + +final AS ( + SELECT + o.price_list_id, + o.sku, + o.currency_code, + o.list_amount_minor, + o.valid_from, + o.effective_valid_to, + o.previous_amount_minor::INT AS previous_amount_minor, + ROUND( + 100 + * ( + o.list_amount_minor - o.previous_amount_minor + ) + / NULLIF(o.previous_amount_minor, 0), + 4 + )::NUMBER(9, 4) AS change_pct, + COALESCE(o.list_amount_minor < o.previous_amount_minor, FALSE) AS is_markdown, + o.price_version::INT AS price_version, + DATEDIFF('day', o.valid_from, o.effective_valid_to)::INT AS days_at_price, + o.lowest_price_30d_minor::INT AS lowest_price_30d_minor + FROM ordered AS o +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/catalogue/int_product_catalogue.sql b/benchmarks/duckshop/project/models/intermediate/catalogue/int_product_catalogue.sql new file mode 100644 index 0000000000..58a7366b9a --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/catalogue/int_product_catalogue.sql @@ -0,0 +1,127 @@ +MODEL ( + description """ + One row per sellable variant with its product, brand, category path, primary image and flattened + specs: the catalogue as customers see it. + """, + columns ( + sku (type VARCHAR, nullable false, audits [not_null, unique]), + product_id (type VARCHAR, audits [not_null]), + sku_family (type VARCHAR), + title (type VARCHAR), + variant_label (type VARCHAR), + brand_code (type VARCHAR), + brand_name (type VARCHAR), + is_private_label (type BOOLEAN), + category_id (type VARCHAR), + category_slug_path (type VARCHAR), + category_level_1 (type VARCHAR), + material (type VARCHAR), + weight_kg (type NUMBER(10, 3)), + primary_image_url (type VARCHAR), + image_count (type INT), + is_active (type BOOLEAN), + launched_on (type DATE), + is_bundle_component (type BOOLEAN), + ), +); + +WITH variants AS ( + SELECT * + FROM __ref("stg_product_variants") +), + +products AS ( + SELECT * + FROM __ref("stg_products") +), + +brands AS ( + SELECT * + FROM __ref("stg_brands") +), + +category_paths AS ( + SELECT * + FROM __ref("int_category_paths") +), + +media AS ( + SELECT * + FROM __ref("stg_product_media") +), + +specs AS ( + SELECT * + FROM __ref("int_product_specs_flat") +), + +bundle_components AS ( + SELECT * + FROM __ref("stg_bundle_components") +), + +leaf_categories AS ( + SELECT + cp.category_id, + cp.slug_path + FROM category_paths AS cp +), + +images AS ( + SELECT + m.product_id, + MIN_BY(m.media_url, m.position) AS primary_image_url, + COUNT(*) AS image_count + FROM media AS m + WHERE m.media_type = 'image' + GROUP BY + m.product_id +), + +component_skus AS ( + SELECT DISTINCT bc.component_sku + FROM bundle_components AS bc +), + +final AS ( + SELECT + v.sku, + v.product_id, + __udf("udf__sku_family")(v.sku)::VARCHAR AS sku_family, + p.title, + NULLIF( + CONCAT_WS(' / ', v.colour, v.size_label), + '' + )::VARCHAR AS variant_label, + p.brand_code, + b.brand_name, + COALESCE(b.is_private_label, FALSE) AS is_private_label, + lc.category_id, + COALESCE(lc.slug_path, p.category_path)::VARCHAR AS category_slug_path, + p.category_level_1, + s.material, + v.weight_kg, + i.primary_image_url::VARCHAR AS primary_image_url, + COALESCE(i.image_count, 0)::INT AS image_count, + p.is_active, + p.launched_on, + ( + cs.component_sku IS NOT NULL + ) AS is_bundle_component + FROM variants AS v + INNER JOIN products AS p + ON v.product_id = p.product_id + LEFT JOIN brands AS b + ON p.brand_code = b.brand_code + LEFT JOIN leaf_categories AS lc + ON p.category_path = lc.slug_path + LEFT JOIN images AS i + ON v.product_id = i.product_id + LEFT JOIN specs AS s + ON v.product_id = s.product_id + LEFT JOIN component_skus AS cs + ON v.sku = cs.component_sku +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/catalogue/int_product_margin.sql b/benchmarks/duckshop/project/models/intermediate/catalogue/int_product_margin.sql new file mode 100644 index 0000000000..ae8f7460c7 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/catalogue/int_product_margin.sql @@ -0,0 +1,119 @@ +MODEL ( + description """ + One row per SKU and currency with its current list price, landed cost in the same currency, unit + margin and margin percentage. + """, + columns ( + sku_price_key (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + currency_code (type VARCHAR), + list_price (type NUMBER(18, 4)), + unit_cost (type NUMBER(18, 4)), + unit_margin (type NUMBER(18, 4)), + margin_pct (type NUMBER(9, 2)), + is_below_cost (type BOOLEAN), + category_level_1 (type VARCHAR), + ), +); + +WITH catalogue AS ( + SELECT * + FROM __ref("int_product_catalogue") +), + +price_history AS ( + SELECT * + FROM __ref("int_price_history") +), + +costs AS ( + SELECT * + FROM __ref("int_supplier_cost_current") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +current_prices AS ( + SELECT + ph.sku, + ph.currency_code, + ph.list_amount_minor, + '@@valuation_date'::DATE AS valuation_date + FROM price_history AS ph + WHERE ph.effective_valid_to IS NULL +), + +with_exponent AS ( + SELECT + cp.sku, + cp.currency_code, + cp.list_amount_minor, + cp.valuation_date, + c.minor_unit_exponent + FROM current_prices AS cp + INNER JOIN currencies AS c + ON cp.currency_code = c.currency_code +), + +with_rate AS ( + SELECT + w.sku, + w.currency_code, + w.list_amount_minor, + w.minor_unit_exponent, + fx.usd_to_rate + FROM with_exponent AS w + ASOF JOIN fx_rates AS fx MATCH_CONDITION (w.valuation_date >= fx.rate_date) + ON w.currency_code = fx.currency_code +), + +final AS ( + SELECT + ( + r.sku || ':' || r.currency_code + )::VARCHAR AS sku_price_key, + r.sku, + r.currency_code, + @sql.money_from_minor(r.list_amount_minor, r.minor_unit_exponent, places := 4)::NUMBER(18, 4) AS list_price, + ROUND(c.unit_cost_usd * r.usd_to_rate, 4)::NUMBER(18, 4) AS unit_cost, + ( + @sql.money_from_minor(r.list_amount_minor, r.minor_unit_exponent, places := 4) + - ROUND(c.unit_cost_usd * r.usd_to_rate, 4) + )::NUMBER(18, 4) AS unit_margin, + ROUND( + 100 + * ( + 1 + - ROUND(c.unit_cost_usd * r.usd_to_rate, 4) + / NULLIF( + @sql.money_from_minor(r.list_amount_minor, r.minor_unit_exponent, places := 4), + 0 + ) + ), + 2 + )::NUMBER(9, 2) AS margin_pct, + COALESCE( + ROUND( + c.unit_cost_usd * r.usd_to_rate, + 4 + ) > @sql.money_from_minor(r.list_amount_minor, r.minor_unit_exponent, places := 4), + FALSE + ) AS is_below_cost, + cat.category_level_1 + FROM with_rate AS r + LEFT JOIN costs AS c + ON r.sku = c.sku + LEFT JOIN catalogue AS cat + ON r.sku = cat.sku +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/catalogue/int_product_specs_flat.sql b/benchmarks/duckshop/project/models/intermediate/catalogue/int_product_specs_flat.sql new file mode 100644 index 0000000000..a0af2f4a62 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/catalogue/int_product_specs_flat.sql @@ -0,0 +1,146 @@ +MODEL ( + description """ + One row per product with its specification object flattened: the most common spec keys pivoted + into typed columns, unit-normalised dimensions, and the remaining keys re-aggregated into an + object. + """, + columns ( + product_id (type VARCHAR, nullable false, audits [not_null, unique]), + material (type VARCHAR), + colour_family (type VARCHAR), + width_cm (type NUMBER(10, 2)), + height_cm (type NUMBER(10, 2)), + capacity_ml (type INT), + is_dishwasher_safe (type BOOLEAN), + spec_value_types (type ARRAY), + other_specs (type OBJECT), + numeric_spec_count (type INT), + ), +); + +WITH products AS ( + SELECT * + FROM __ref("stg_products") +), + +spec_entries AS ( + SELECT + p.product_id, + LOWER(REGEXP_REPLACE( + k.value::VARCHAR, + '[^A-Za-z0-9]+', + '_' + )) AS spec_key, + GET(p.specs, k.value::VARCHAR) AS spec_value + FROM products AS p, LATERAL FLATTEN(input => OBJECT_KEYS(p.specs)) AS k +), + +typed_entries AS ( + SELECT + e.product_id, + e.spec_key, + e.spec_value, + TYPEOF(e.spec_value) AS value_type, + CASE + WHEN IS_DECIMAL(e.spec_value) + OR IS_INTEGER(e.spec_value) + THEN e.spec_value::NUMBER(18, 4) + WHEN REGEXP_LIKE( + e.spec_value::VARCHAR, + '^\\s*[0-9]+(\\.[0-9]+)?\\s*(mm|cm|m)\\s*$', + 'i' + ) + THEN TO_NUMBER(REGEXP_SUBSTR(e.spec_value::VARCHAR, '[0-9]+(\\.[0-9]+)?'), 18, 4) + * DECODE( + LOWER(REGEXP_SUBSTR( + e.spec_value::VARCHAR, + '(mm|cm|m)', + 1, + 1, + 'i' + )), + 'mm', 0.1, + 'm', 100, + 1 + ) + END AS numeric_value + FROM spec_entries AS e +), + +pivot_source AS ( + SELECT + t.product_id, + t.spec_key, + t.spec_value + FROM typed_entries AS t + WHERE t.spec_key IN ('material', 'colour_family', 'dishwasher_safe') +), + +pivoted AS ( + SELECT + p.product_id, + p.material, + p.colour_family, + p.dishwasher_safe + FROM pivot_source PIVOT( + MAX(spec_value) FOR spec_key IN ('material', 'colour_family', 'dishwasher_safe') + ) AS p(product_id, material, colour_family, dishwasher_safe) +), + +dimensions AS ( + SELECT + t.product_id, + MAX(IFF(t.spec_key IN ('width', 'width_cm'), t.numeric_value, NULL)) AS width_cm, + MAX(IFF(t.spec_key IN ('height', 'height_cm'), t.numeric_value, NULL)) AS height_cm, + MAX(IFF(t.spec_key IN ('capacity', 'capacity_ml'), t.numeric_value, NULL)) AS capacity_ml, + COUNT(t.numeric_value) AS numeric_spec_count, + ARRAY_AGG(DISTINCT t.value_type) WITHIN GROUP (ORDER BY t.value_type) AS spec_value_types + FROM typed_entries AS t + GROUP BY + t.product_id +), + +leftovers AS ( + SELECT + t.product_id, + OBJECT_AGG(t.spec_key, t.spec_value) AS other_specs + FROM typed_entries AS t + WHERE + t.spec_key NOT IN ( + 'material', + 'colour_family', + 'dishwasher_safe', + 'width', + 'width_cm', + 'height', + 'height_cm', + 'capacity', + 'capacity_ml' + ) + GROUP BY + t.product_id +), + +final AS ( + SELECT + p.product_id, + INITCAP(pv.material::VARCHAR)::VARCHAR AS material, + LOWER(pv.colour_family::VARCHAR)::VARCHAR AS colour_family, + ROUND(d.width_cm, 2)::NUMBER(10, 2) AS width_cm, + ROUND(d.height_cm, 2)::NUMBER(10, 2) AS height_cm, + d.capacity_ml::INT AS capacity_ml, + TRY_TO_BOOLEAN(pv.dishwasher_safe::VARCHAR)::BOOLEAN AS is_dishwasher_safe, + COALESCE(d.spec_value_types, ARRAY_CONSTRUCT())::ARRAY AS spec_value_types, + COALESCE(l.other_specs, OBJECT_CONSTRUCT())::OBJECT AS other_specs, + COALESCE(d.numeric_spec_count, 0)::INT AS numeric_spec_count + FROM products AS p + LEFT JOIN pivoted AS pv + ON p.product_id = pv.product_id + LEFT JOIN dimensions AS d + ON p.product_id = d.product_id + LEFT JOIN leftovers AS l + ON p.product_id = l.product_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/catalogue/int_supplier_cost_current.sql b/benchmarks/duckshop/project/models/intermediate/catalogue/int_supplier_cost_current.sql new file mode 100644 index 0000000000..ff393b958f --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/catalogue/int_supplier_cost_current.sql @@ -0,0 +1,133 @@ +MODEL ( + description """ + One row per SKU with its cheapest current supplier cost converted to USD (ASOF the cost's + effective date), the supplier, and how many suppliers quote it. + """, + columns ( + sku (type VARCHAR, nullable false, audits [not_null, unique]), + supplier_code (type VARCHAR), + supplier_name (type VARCHAR), + currency_code (type VARCHAR), + unit_cost_minor (type INT), + unit_cost_usd (type NUMBER(18, 4)), + effective_from (type DATE), + supplier_count (type INT), + payment_terms_days (type INT), + cost_spread_pct (type NUMBER(9, 2)), + ), +); + +WITH supplier_costs AS ( + SELECT * + FROM __ref("stg_supplier_costs") +), + +suppliers AS ( + SELECT * + FROM __ref("stg_suppliers") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +current_costs AS ( + SELECT + c.cost_id, + c.supplier_code, + c.sku, + c.currency_code, + COALESCE(c.best_break_cost_minor, c.unit_cost_minor) AS unit_cost_minor, + c.effective_from + FROM supplier_costs AS c + WHERE + c.effective_to IS NULL + OR c.effective_to >= '@@valuation_date'::DATE +), + +with_exponent AS ( + SELECT + cc.cost_id, + cc.supplier_code, + cc.sku, + cc.currency_code, + cc.unit_cost_minor, + cc.effective_from, + cur.minor_unit_exponent + FROM current_costs AS cc + INNER JOIN currencies AS cur + ON cc.currency_code = cur.currency_code +), + +in_usd AS ( + SELECT + w.cost_id, + w.supplier_code, + w.sku, + w.currency_code, + w.unit_cost_minor, + w.effective_from, + ROUND( + @sql.money_from_minor(w.unit_cost_minor, w.minor_unit_exponent, places := 6) * fx.rate_to_usd, + 4 + ) AS unit_cost_usd + FROM with_exponent AS w + ASOF JOIN fx_rates AS fx MATCH_CONDITION (w.effective_from >= fx.rate_date) + ON w.currency_code = fx.currency_code +), + +per_sku AS ( + SELECT + u.sku, + MIN_BY(u.cost_id, u.unit_cost_usd) AS cheapest_cost_id, + COUNT(DISTINCT u.supplier_code) AS supplier_count, + MAX(u.unit_cost_usd) AS max_cost_usd + FROM in_usd AS u + GROUP BY + u.sku +), + +ranked AS ( + SELECT + p.sku, + u.supplier_code, + u.currency_code, + u.unit_cost_minor, + u.unit_cost_usd, + u.effective_from, + p.supplier_count, + p.max_cost_usd + FROM per_sku AS p + INNER JOIN in_usd AS u + ON p.cheapest_cost_id = u.cost_id +), + +final AS ( + SELECT + r.sku, + r.supplier_code, + s.supplier_name, + r.currency_code, + r.unit_cost_minor::INT AS unit_cost_minor, + r.unit_cost_usd::NUMBER(18, 4) AS unit_cost_usd, + r.effective_from, + r.supplier_count::INT AS supplier_count, + s.payment_terms_days, + ROUND(100 + * ( + r.max_cost_usd - r.unit_cost_usd + ) + / NULLIF(r.unit_cost_usd, 0), 2)::NUMBER(9, 2) AS cost_spread_pct + FROM ranked AS r + LEFT JOIN suppliers AS s + ON r.supplier_code = s.supplier_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/clickstream/_sqlbuild/_constants/session_gap_minutes.sql b/benchmarks/duckshop/project/models/intermediate/clickstream/_sqlbuild/_constants/session_gap_minutes.sql new file mode 100644 index 0000000000..3fb348547b --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/clickstream/_sqlbuild/_constants/session_gap_minutes.sql @@ -0,0 +1 @@ +CONSTANT (name session_gap_minutes, value 30); diff --git a/benchmarks/duckshop/project/models/intermediate/clickstream/int_app_sessions.sql b/benchmarks/duckshop/project/models/intermediate/clickstream/int_app_sessions.sql new file mode 100644 index 0000000000..d7381e4adb --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/clickstream/int_app_sessions.sql @@ -0,0 +1,99 @@ +MODEL ( + description """ + One row per app session, split on 30 minutes of inactivity per device, with screens viewed and + push-driven starts + """, + columns ( + app_session_id (type VARCHAR, nullable false, audits [not_null, unique]), + device_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + platform (type VARCHAR), + app_major_version (type INT), + session_start_utc (type TIMESTAMP_NTZ), + session_end_utc (type TIMESTAMP_NTZ), + duration_seconds (type INT), + event_count (type INT), + screen_count (type INT), + started_from_push (type BOOLEAN), + ), +); + +WITH app_events AS ( + SELECT * + FROM __ref("stg_app_events") +), + +device_events AS ( + SELECT + e.event_id, + e.device_id, + e.customer_id, + e.platform, + e.app_major_version, + e.event_name, + e.event_at_utc, + IFF(COALESCE(DATEDIFF( + 'minute', + LAG(e.event_at_utc) OVER (PARTITION BY e.device_id ORDER BY e.event_at_utc, e.event_id), + e.event_at_utc + ), 999) > @const("session_gap_minutes"), 1, 0) AS starts_session + FROM app_events AS e + WHERE + e.device_id IS NOT NULL + AND e.event_at_utc IS NOT NULL +), + +numbered AS ( + SELECT + d.device_id, + d.customer_id, + d.platform, + d.app_major_version, + d.event_name, + d.event_at_utc, + d.starts_session, + SUM(d.starts_session) OVER ( + PARTITION BY d.device_id + ORDER BY d.event_at_utc, d.event_id + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) AS session_number + FROM device_events AS d +), + +sessions AS ( + SELECT + n.device_id, + MAX(n.customer_id) AS customer_id, + ANY_VALUE(n.platform) AS platform, + MAX(n.app_major_version) AS app_major_version, + MIN(n.event_at_utc) AS session_start_utc, + MAX(n.event_at_utc) AS session_end_utc, + COUNT(*) AS event_count, + COUNT_IF(n.event_name = 'screen_view') AS screen_count, + BOOLOR_AGG(n.starts_session = 1 AND n.event_name = 'push_open') AS started_from_push + FROM numbered AS n + GROUP BY + n.device_id, + n.session_number +), + +final AS ( + SELECT + ( + s.device_id || ':' || TO_CHAR(s.session_start_utc, 'YYYYMMDDHH24MISS') + )::VARCHAR AS app_session_id, + s.device_id, + s.customer_id::VARCHAR AS customer_id, + s.platform::VARCHAR AS platform, + s.app_major_version::INT AS app_major_version, + s.session_start_utc, + s.session_end_utc, + DATEDIFF('second', s.session_start_utc, s.session_end_utc)::INT AS duration_seconds, + s.event_count::INT AS event_count, + s.screen_count::INT AS screen_count, + s.started_from_push::BOOLEAN AS started_from_push + FROM sessions AS s +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/clickstream/int_cart_funnel.sql b/benchmarks/duckshop/project/models/intermediate/clickstream/int_cart_funnel.sql new file mode 100644 index 0000000000..df1ff3cef5 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/clickstream/int_cart_funnel.sql @@ -0,0 +1,127 @@ +MODEL ( + description """ + One row per web session with its cart activity worked out from event item arrays using FILTER + and TRANSFORM lambdas: items added, removed, net units, and the checkout steps reached. + """, + columns ( + session_id (type VARCHAR, nullable false, audits [not_null, unique]), + anonymous_id (type VARCHAR), + session_start_utc (type TIMESTAMP_NTZ), + added_skus (type ARRAY), + removed_sku_count (type INT), + net_units (type INT), + max_added_unit_minor (type INT), + reached_checkout (type BOOLEAN), + reached_payment (type BOOLEAN), + converted (type BOOLEAN), + funnel_stage (type VARCHAR, audits [ + accepted_values (values ["browse", "cart", "checkout", "payment", "order"]), + ]), + ), +); + +WITH sessions AS ( + SELECT * + FROM __ref("int_web_sessions") +), + +web_events AS ( + SELECT * + FROM __ref("stg_web_events") +), + +checkout_steps AS ( + SELECT * + FROM __ref("stg_web_checkout_steps") +), + +cart_events AS ( + SELECT + e.anonymous_id, + e.event_at_utc, + e.event_type, + FILTER( + e.items, + i -> i:sku IS NOT NULL + AND COALESCE(i:quantity::INT, 1) > 0 + ) AS valid_items + FROM web_events AS e + WHERE e.event_type IN ('add_to_cart', 'remove_from_cart') +), + +session_cart AS ( + SELECT + s.session_id, + ARRAY_AGG(IFF(c.event_type = 'add_to_cart', c.valid_items, NULL)) AS added_item_lists, + SUM(IFF(c.event_type = 'remove_from_cart', ARRAY_SIZE(c.valid_items), 0)) AS removed_sku_count, + SUM(IFF(c.event_type = 'add_to_cart', 1, -1) + * REDUCE( + TRANSFORM( + c.valid_items, + i -> COALESCE(i:quantity::INT, 1) + ), + 0, + (acc, q) -> acc + q + )) AS net_units, + MAX(ARRAY_MAX( + TRANSFORM(c.valid_items, i -> i:price_minor::INT) + )) AS max_added_unit_minor + FROM sessions AS s + INNER JOIN cart_events AS c + ON s.anonymous_id = c.anonymous_id + AND c.event_at_utc BETWEEN s.session_start_utc AND s.session_end_utc + GROUP BY + s.session_id +), + +session_checkout AS ( + SELECT + s.session_id, + MAX(cs.step_number) AS max_step + FROM sessions AS s + INNER JOIN checkout_steps AS cs + ON s.anonymous_id = cs.anonymous_id + AND cs.first_reached_at_utc BETWEEN s.session_start_utc AND s.session_end_utc + GROUP BY + s.session_id +), + +final AS ( + SELECT + s.session_id, + s.anonymous_id, + s.session_start_utc, + ARRAY_DISTINCT(TRANSFORM( + ARRAY_FLATTEN(COALESCE(sc.added_item_lists, ARRAY_CONSTRUCT())), + i -> UPPER(i:sku::VARCHAR) + ))::ARRAY AS added_skus, + COALESCE(sc.removed_sku_count, 0)::INT AS removed_sku_count, + COALESCE(sc.net_units, 0)::INT AS net_units, + sc.max_added_unit_minor::INT AS max_added_unit_minor, + ( + COALESCE(ck.max_step, 0) >= 1 + ) AS reached_checkout, + ( + COALESCE(ck.max_step, 0) >= 3 + ) AS reached_payment, + s.has_order AS converted, + CASE + WHEN s.has_order + THEN 'order' + WHEN COALESCE(ck.max_step, 0) >= 3 + THEN 'payment' + WHEN COALESCE(ck.max_step, 0) >= 1 + THEN 'checkout' + WHEN s.has_add_to_cart + THEN 'cart' + ELSE 'browse' + END::VARCHAR AS funnel_stage + FROM sessions AS s + LEFT JOIN session_cart AS sc + ON s.session_id = sc.session_id + LEFT JOIN session_checkout AS ck + ON s.session_id = ck.session_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/clickstream/int_cross_device_identity.sql b/benchmarks/duckshop/project/models/intermediate/clickstream/int_cross_device_identity.sql new file mode 100644 index 0000000000..94497605ed --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/clickstream/int_cross_device_identity.sql @@ -0,0 +1,97 @@ +MODEL ( + description """ + One row per customer with the web visitors and app devices linked to them, the platforms they + use, and a stable household-independent identity graph key. + """, + columns ( + customer_id (type VARCHAR, nullable false, audits [not_null, unique]), + web_visitor_count (type INT), + app_device_count (type INT), + platforms (type ARRAY), + is_multi_device (type BOOLEAN), + first_seen_at_utc (type TIMESTAMP_NTZ), + identity_graph (type OBJECT), + ), +); + +WITH web_identities AS ( + SELECT * + FROM __ref("stg_web_identifies") +), + +app_events AS ( + SELECT * + FROM __ref("stg_app_events") +), + +web_links AS ( + SELECT + w.customer_id::VARCHAR AS customer_id, + w.anonymous_id::VARCHAR AS identifier, + 'web'::VARCHAR AS platform, + w.first_seen_at_utc::TIMESTAMP_NTZ AS first_seen_at_utc + FROM web_identities AS w +), + +app_links AS ( + SELECT + a.customer_id::VARCHAR AS customer_id, + a.device_id::VARCHAR AS identifier, + a.platform::VARCHAR AS platform, + MIN(a.event_at_utc)::TIMESTAMP_NTZ AS first_seen_at_utc + FROM app_events AS a + WHERE + a.customer_id IS NOT NULL + AND a.device_id IS NOT NULL + GROUP BY + a.customer_id, + a.device_id, + a.platform +), + +all_links AS ( + SELECT + wl.customer_id::VARCHAR AS customer_id, + wl.identifier::VARCHAR AS identifier, + wl.platform::VARCHAR AS platform, + wl.first_seen_at_utc::TIMESTAMP_NTZ AS first_seen_at_utc + FROM web_links AS wl + UNION ALL + SELECT + al.customer_id::VARCHAR AS customer_id, + al.identifier::VARCHAR AS identifier, + al.platform::VARCHAR AS platform, + al.first_seen_at_utc::TIMESTAMP_NTZ AS first_seen_at_utc + FROM app_links AS al +), + +per_platform AS ( + SELECT + l.customer_id, + l.platform, + ARRAY_AGG(DISTINCT l.identifier) WITHIN GROUP (ORDER BY l.identifier) AS identifiers, + MIN(l.first_seen_at_utc) AS first_seen_at_utc + FROM all_links AS l + GROUP BY + l.customer_id, + l.platform +), + +final AS ( + SELECT + p.customer_id, + SUM(IFF(p.platform = 'web', ARRAY_SIZE(p.identifiers), 0))::INT AS web_visitor_count, + SUM(IFF(p.platform <> 'web', ARRAY_SIZE(p.identifiers), 0))::INT AS app_device_count, + ARRAY_AGG(p.platform) WITHIN GROUP (ORDER BY p.platform)::ARRAY AS platforms, + ( + SUM(ARRAY_SIZE(p.identifiers)) > 1 + ) AS is_multi_device, + MIN(p.first_seen_at_utc) AS first_seen_at_utc, + OBJECT_AGG(p.platform, p.identifiers::VARIANT)::OBJECT AS identity_graph + FROM per_platform AS p + GROUP BY + p.customer_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/clickstream/int_email_engagement.sql b/benchmarks/duckshop/project/models/intermediate/clickstream/int_email_engagement.sql new file mode 100644 index 0000000000..3855a25400 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/clickstream/int_email_engagement.sql @@ -0,0 +1,119 @@ +MODEL ( + description """ + One row per email message with its delivery outcome and human engagement, machine opens + excluded, pivoted from the event stream, with time from send to first click. + """, + columns ( + message_id (type VARCHAR, nullable false, audits [not_null, unique]), + campaign_id (type VARCHAR), + recipient_email (type VARCHAR), + sent_at (type TIMESTAMP_TZ), + is_delivered (type BOOLEAN), + bounce_type (type VARCHAR), + human_open_count (type INT), + click_count (type INT), + clicked_domains (type ARRAY), + unsubscribed (type BOOLEAN), + complained (type BOOLEAN), + minutes_to_first_click (type INT), + customer_id (type VARCHAR), + campaign_channel (type VARCHAR), + is_campaign_in_flight (type BOOLEAN), + ), +); + +WITH email_events AS ( + SELECT * + FROM __ref("stg_email_events") +), + +customers AS ( + SELECT * + FROM __ref("stg_customers") +), + +campaigns AS ( + SELECT * + FROM __ref("stg_campaigns") +), + +email_accounts AS ( + SELECT + c.email_normalised, + MIN(c.customer_id) AS customer_id + FROM customers AS c + WHERE c.email_normalised IS NOT NULL + GROUP BY + c.email_normalised +), + +per_message AS ( + SELECT + e.message_id, + MAX(e.campaign_id) AS campaign_id, + MAX(e.recipient_email) AS recipient_email, + MIN(IFF(e.event_type = 'sent', e.occurred_at, NULL)) AS sent_at, + BOOLOR_AGG(e.event_type = 'delivered') AS is_delivered, + MAX(e.bounce_type) AS bounce_type, + COUNT_IF(e.event_type = 'open' AND NOT COALESCE(e.is_machine_open, FALSE)) AS human_open_count, + COUNT_IF(e.event_type = 'click') AS click_count, + ARRAY_AGG(DISTINCT LOWER(PARSE_URL(e.clicked_url, 1)['host']::VARCHAR)) WITHIN GROUP ( + ORDER BY LOWER(PARSE_URL(e.clicked_url, 1)['host']::VARCHAR) + ) AS clicked_domains, + BOOLOR_AGG(e.event_type = 'unsubscribe') AS unsubscribed, + BOOLOR_AGG(e.event_type = 'complaint') AS complained, + MIN(IFF(e.event_type = 'click', e.occurred_at, NULL)) AS first_click_at + FROM email_events AS e + GROUP BY + e.message_id +), + +with_context AS ( + SELECT + p.message_id, + p.campaign_id, + p.recipient_email, + p.sent_at, + p.is_delivered, + p.bounce_type, + p.human_open_count, + p.click_count, + p.clicked_domains, + p.unsubscribed, + p.complained, + p.first_click_at, + a.customer_id, + c.channel AS campaign_channel, + CONVERT_TIMEZONE('UTC', p.sent_at)::DATE BETWEEN c.starts_on AND COALESCE( + c.ends_on, + '@@valuation_date'::DATE + ) AS is_campaign_in_flight + FROM per_message AS p + LEFT JOIN email_accounts AS a + ON p.recipient_email = a.email_normalised + LEFT JOIN campaigns AS c + ON p.campaign_id = c.campaign_id +), + +final AS ( + SELECT + p.message_id, + p.campaign_id::VARCHAR AS campaign_id, + p.recipient_email::VARCHAR AS recipient_email, + p.sent_at, + COALESCE(p.is_delivered, FALSE)::BOOLEAN AS is_delivered, + p.bounce_type::VARCHAR AS bounce_type, + p.human_open_count::INT AS human_open_count, + p.click_count::INT AS click_count, + p.clicked_domains::ARRAY AS clicked_domains, + COALESCE(p.unsubscribed, FALSE)::BOOLEAN AS unsubscribed, + COALESCE(p.complained, FALSE)::BOOLEAN AS complained, + DATEDIFF('minute', p.sent_at, p.first_click_at)::INT AS minutes_to_first_click, + p.customer_id::VARCHAR AS customer_id, + p.campaign_channel::VARCHAR AS campaign_channel, + COALESCE(p.is_campaign_in_flight, FALSE)::BOOLEAN AS is_campaign_in_flight + FROM with_context AS p +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/clickstream/int_experiment_exposures.sql b/benchmarks/duckshop/project/models/intermediate/clickstream/int_experiment_exposures.sql new file mode 100644 index 0000000000..c48658ceb0 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/clickstream/int_experiment_exposures.sql @@ -0,0 +1,79 @@ +MODEL ( + description """ + One row per experiment assignment with the visitor's sessions and conversions after assignment, + to measure each variant against control. + """, + columns ( + assignment_id (type VARCHAR, nullable false, audits [not_null, unique]), + experiment_key (type VARCHAR, audits [not_null]), + variant_key (type VARCHAR, audits [not_null]), + is_control (type BOOLEAN), + anonymous_id (type VARCHAR), + assigned_at_utc (type TIMESTAMP_NTZ), + sessions_after (type INT), + converted (type BOOLEAN), + days_to_convert (type NUMBER(10, 2)), + assignment_surface (type VARCHAR), + ), +); + +WITH assignments AS ( + SELECT * + FROM __ref("stg_experiment_assignments") +), + +sessions AS ( + SELECT * + FROM __ref("int_web_sessions") +), + +assigned AS ( + SELECT + a.assignment_id, + a.experiment_key, + a.variant_key, + a.anonymous_id, + a.assignment_surface, + CONVERT_TIMEZONE('UTC', a.assigned_at)::TIMESTAMP_NTZ AS assigned_at_utc + FROM assignments AS a +), + +after_assignment AS ( + SELECT + x.assignment_id, + COUNT(s.session_id) AS sessions_after, + MIN(IFF(s.has_order, s.session_start_utc, NULL)) AS first_order_session_utc + FROM assigned AS x + LEFT JOIN sessions AS s + ON x.anonymous_id = s.anonymous_id + AND s.session_start_utc >= x.assigned_at_utc + GROUP BY + x.assignment_id +), + +final AS ( + SELECT + x.assignment_id, + x.experiment_key, + x.variant_key, + ( + x.variant_key IN ('control', 'a', 'baseline') + ) AS is_control, + x.anonymous_id, + x.assigned_at_utc, + COALESCE(f.sessions_after, 0)::INT AS sessions_after, + ( + f.first_order_session_utc IS NOT NULL + ) AS converted, + ROUND( + DATEDIFF('minute', x.assigned_at_utc, f.first_order_session_utc) / 1440, + 2 + )::NUMBER(10, 2) AS days_to_convert, + x.assignment_surface + FROM assigned AS x + LEFT JOIN after_assignment AS f + ON x.assignment_id = f.assignment_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/clickstream/int_product_view_affinity.sql b/benchmarks/duckshop/project/models/intermediate/clickstream/int_product_view_affinity.sql new file mode 100644 index 0000000000..1ff4cddd10 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/clickstream/int_product_view_affinity.sql @@ -0,0 +1,96 @@ +MODEL ( + description """ + One row per ordered pair of SKUs viewed by the same visitor within an hour, with the number of + visitors who viewed both and the lift over chance. + """, + columns ( + sku_pair_key (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + co_viewed_sku (type VARCHAR, audits [not_null]), + co_viewers (type INT), + sku_viewers (type INT), + co_view_rate (type NUMBER(9, 6)), + lift (type NUMBER(12, 4)), + pair_rank (type INT), + ), +); + +WITH product_views AS ( + SELECT * + FROM __ref("stg_web_product_views") +), + +visitor_views AS ( + SELECT + v.anonymous_id, + v.sku, + v.viewed_minute_utc, + DATEADD('hour', 1, v.viewed_minute_utc) AS window_end_utc + FROM product_views AS v +), + +pairs AS ( + SELECT DISTINCT + a.anonymous_id, + a.sku, + b.sku AS co_viewed_sku + FROM visitor_views AS a + INNER JOIN visitor_views AS b + ON a.anonymous_id = b.anonymous_id + AND a.sku <> b.sku + AND b.viewed_minute_utc BETWEEN a.viewed_minute_utc AND a.window_end_utc +), + +pair_counts AS ( + SELECT + p.sku, + p.co_viewed_sku, + COUNT(DISTINCT p.anonymous_id) AS co_viewers + FROM pairs AS p + GROUP BY + p.sku, + p.co_viewed_sku +), + +sku_counts AS ( + SELECT + v.sku, + COUNT(DISTINCT v.anonymous_id) AS sku_viewers + FROM visitor_views AS v + GROUP BY + v.sku +), + +visitor_total AS ( + SELECT COUNT(DISTINCT v.anonymous_id) AS total_viewers + FROM visitor_views AS v +), + +final AS ( + SELECT + ( + pc.sku || '>' || pc.co_viewed_sku + )::VARCHAR AS sku_pair_key, + pc.sku, + pc.co_viewed_sku, + pc.co_viewers::INT AS co_viewers, + a.sku_viewers::INT AS sku_viewers, + ROUND(pc.co_viewers / NULLIF(a.sku_viewers, 0), 6)::NUMBER(9, 6) AS co_view_rate, + ROUND( + ( + pc.co_viewers / NULLIF(a.sku_viewers, 0) + ) + / NULLIF(b.sku_viewers / NULLIF(t.total_viewers, 0), 0), + 4 + )::NUMBER(12, 4) AS lift, + RANK() OVER (PARTITION BY pc.sku ORDER BY pc.co_viewers DESC)::INT AS pair_rank + FROM pair_counts AS pc + INNER JOIN sku_counts AS a + ON pc.sku = a.sku + INNER JOIN sku_counts AS b + ON pc.co_viewed_sku = b.sku + CROSS JOIN visitor_total AS t +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/clickstream/int_search_conversion.sql b/benchmarks/duckshop/project/models/intermediate/clickstream/int_search_conversion.sql new file mode 100644 index 0000000000..93c3b5c9d3 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/clickstream/int_search_conversion.sql @@ -0,0 +1,108 @@ +MODEL ( + description """ + One row per normalised search term with search volume across the site search log and the web + tracker, zero-result rate, and how often a product view followed within ten minutes. + """, + columns ( + search_term (type VARCHAR, nullable false, audits [not_null, unique]), + search_count (type INT), + searching_visitors (type INT), + zero_result_rate (type NUMBER(9, 4)), + view_after_search_rate (type NUMBER(9, 4)), + avg_results (type NUMBER(12, 2)), + top_filter_category (type VARCHAR), + first_searched_at_utc (type TIMESTAMP_NTZ), + ), +); + +WITH search_log AS ( + SELECT * + FROM __ref("stg_search_queries") +), + +web_searches AS ( + SELECT * + FROM __ref("stg_web_searches") +), + +product_views AS ( + SELECT * + FROM __ref("stg_web_product_views") +), + +all_searches AS ( + SELECT + q.query_id::VARCHAR AS search_key, + q.anonymous_id::VARCHAR AS anonymous_id, + q.query_text::VARCHAR AS search_term, + q.results_count::INT AS result_count, + CONVERT_TIMEZONE('UTC', q.searched_at)::TIMESTAMP_NTZ AS searched_at_utc, + q.filter_category::VARCHAR AS filter_category + FROM search_log AS q + UNION ALL + SELECT + w.event_id::VARCHAR AS search_key, + w.anonymous_id::VARCHAR AS anonymous_id, + w.search_term::VARCHAR AS search_term, + w.result_count::INT AS result_count, + w.searched_at_utc::TIMESTAMP_NTZ AS searched_at_utc, + NULL::VARCHAR AS filter_category + FROM web_searches AS w +), + +search_windows AS ( + SELECT + s.search_key, + s.anonymous_id, + s.searched_at_utc, + DATEADD('minute', 10, s.searched_at_utc) AS window_end_utc + FROM all_searches AS s +), + +followed AS ( + SELECT + w.search_key, + COUNT(v.product_view_key) > 0 AS has_view + FROM search_windows AS w + LEFT JOIN product_views AS v + ON w.anonymous_id = v.anonymous_id + AND v.viewed_minute_utc BETWEEN w.searched_at_utc AND w.window_end_utc + GROUP BY + w.search_key +), + +by_term AS ( + SELECT + s.search_term, + COUNT(*) AS search_count, + COUNT(DISTINCT s.anonymous_id) AS searching_visitors, + COUNT_IF(COALESCE(s.result_count, 0) = 0) AS zero_result_count, + COUNT_IF(f.has_view) AS viewed_count, + AVG(s.result_count) AS avg_results, + MODE(s.filter_category) AS top_filter_category, + MIN(s.searched_at_utc) AS first_searched_at_utc + FROM all_searches AS s + INNER JOIN followed AS f + ON s.search_key = f.search_key + WHERE + s.search_term IS NOT NULL + AND s.search_term <> '' + GROUP BY + s.search_term +), + +final AS ( + SELECT + b.search_term, + b.search_count::INT AS search_count, + b.searching_visitors::INT AS searching_visitors, + ROUND(b.zero_result_count / b.search_count, 4)::NUMBER(9, 4) AS zero_result_rate, + ROUND(b.viewed_count / b.search_count, 4)::NUMBER(9, 4) AS view_after_search_rate, + ROUND(b.avg_results, 2)::NUMBER(12, 2) AS avg_results, + b.top_filter_category::VARCHAR AS top_filter_category, + b.first_searched_at_utc + FROM by_term AS b +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/clickstream/int_session_attribution.sql b/benchmarks/duckshop/project/models/intermediate/clickstream/int_session_attribution.sql new file mode 100644 index 0000000000..707808b768 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/clickstream/int_session_attribution.sql @@ -0,0 +1,128 @@ +MODEL ( + description """ + One row per web session with UTM parameters parsed from the landing URL, a reporting channel + from the channel mapping seed, and the visitor's first-touch and last-touch channels. + """, + columns ( + session_id (type VARCHAR, nullable false, audits [not_null, unique]), + anonymous_id (type VARCHAR), + customer_id (type VARCHAR), + session_start_utc (type TIMESTAMP_NTZ), + utm_source (type VARCHAR), + utm_medium (type VARCHAR), + utm_campaign (type VARCHAR), + channel_code (type VARCHAR, audits [not_null]), + is_paid (type BOOLEAN), + first_touch_channel (type VARCHAR), + last_touch_channel (type VARCHAR), + touch_number (type INT), + ), +); + +WITH sessions AS ( + SELECT * + FROM __ref("int_web_sessions") +), + +channel_mapping AS ( + SELECT * + FROM __seed("channel_mapping") +), + +parsed AS ( + SELECT + s.session_id, + s.anonymous_id, + s.customer_id, + s.session_start_utc, + s.referrer_host, + LOWER(PARSE_URL(s.landing_page_url, 1)['parameters']['utm_source']::VARCHAR) AS utm_source, + LOWER( + COALESCE( + PARSE_URL(s.landing_page_url, 1)['parameters']['utm_medium']::VARCHAR, + REGEXP_SUBSTR(s.landing_page_url, '[?&]utm_medium=([^&#]+)', 1, 1, 'e', 1) + ) + ) AS utm_medium, + LOWER( + REGEXP_SUBSTR(s.landing_page_url, '[?&]utm_campaign=([^&#]+)', 1, 1, 'e', 1) + ) AS utm_campaign + FROM sessions AS s +), + +exact_matches AS ( + SELECT + p.session_id, + MIN(m.channel_code) AS channel_code, + BOOLOR_AGG(m.is_paid) AS is_paid + FROM parsed AS p + INNER JOIN channel_mapping AS m + ON p.utm_source = m.utm_source + AND p.utm_medium = m.utm_medium + GROUP BY + p.session_id +), + +medium_matches AS ( + SELECT + p.session_id, + MIN(m.channel_code) AS channel_code, + BOOLOR_AGG(m.is_paid) AS is_paid + FROM parsed AS p + INNER JOIN channel_mapping AS m + ON p.utm_medium = m.utm_medium + WHERE m.utm_source = '-' + GROUP BY + p.session_id +), + +classified AS ( + SELECT + p.session_id, + p.anonymous_id, + p.customer_id, + p.session_start_utc, + p.utm_source, + p.utm_medium, + p.utm_campaign, + COALESCE( + e.channel_code, + mm.channel_code, + IFF(p.referrer_host IS NULL OR p.referrer_host LIKE '%duckshop%', 'DIRECT', 'REFERRAL') + ) AS channel_code, + COALESCE(e.is_paid, mm.is_paid, FALSE) AS is_paid + FROM parsed AS p + LEFT JOIN exact_matches AS e + ON p.session_id = e.session_id + LEFT JOIN medium_matches AS mm + ON p.session_id = mm.session_id +), + +final AS ( + SELECT + c.session_id, + c.anonymous_id, + c.customer_id, + c.session_start_utc, + c.utm_source::VARCHAR AS utm_source, + c.utm_medium::VARCHAR AS utm_medium, + c.utm_campaign::VARCHAR AS utm_campaign, + c.channel_code::VARCHAR AS channel_code, + c.is_paid::BOOLEAN AS is_paid, + FIRST_VALUE(c.channel_code) OVER ( + PARTITION BY c.anonymous_id + ORDER BY c.session_start_utc, c.session_id + )::VARCHAR AS first_touch_channel, + LAST_VALUE(c.channel_code) OVER ( + PARTITION BY c.anonymous_id + ORDER BY c.session_start_utc, c.session_id + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + )::VARCHAR AS last_touch_channel, + ROW_NUMBER() OVER ( + PARTITION BY c.anonymous_id + ORDER BY c.session_start_utc, c.session_id + )::INT AS touch_number + FROM classified AS c +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/clickstream/int_web_sessions.sql b/benchmarks/duckshop/project/models/intermediate/clickstream/int_web_sessions.sql new file mode 100644 index 0000000000..4cd4262ee2 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/clickstream/int_web_sessions.sql @@ -0,0 +1,166 @@ +MODEL ( + description """ + One row per web session: a visitor's events split wherever 30 minutes pass without activity, + built in daily microbatches from the hourly event stream, with landing page, duration, depth and + the visitor's local start time from the tracker's reported time zone. Each batch keeps only the + sessions that start inside it, reading a day of events either side so that sessions crossing + the batch edges are whole; sessions are assumed to last less than a day. The three-day + lookback rebuilds recent days when late events arrive. + """, + constants ( + _read_margin_hours 24, + ), + materialized incremental, + incremental_strategy delete_insert, + incremental_mode microbatch, + microbatch_strategy watermark, + cursor_watermark_mode all, + cursor session_day, + cursor_type timestamp, + cursor_grain day, + batch_size 1d, + lookback 3d, + cursor_inputs ( + stg_web_events (column received_at, roles [watermark]), + ), + replay_on_change full, + on_schema_change sync_all_columns, + columns ( + session_id (type VARCHAR, nullable false, audits [not_null (run_scope delta_and_final), unique]), + anonymous_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + session_day (type TIMESTAMP_NTZ, audits [not_null]), + session_start_utc (type TIMESTAMP_NTZ), + session_end_utc (type TIMESTAMP_NTZ), + session_start_visitor_local (type TIMESTAMP_NTZ), + duration_seconds (type INT), + event_count (type INT), + page_view_count (type INT), + landing_page_path (type VARCHAR), + exit_page_path (type VARCHAR), + referrer_host (type VARCHAR), + landing_page_url (type VARCHAR), + is_bounce (type BOOLEAN), + has_add_to_cart (type BOOLEAN), + has_order (type BOOLEAN), + ), +); + +WITH web_events AS ( + SELECT * + FROM __ref("stg_web_events") +), + +human_events AS ( + SELECT + e.event_id, + e.anonymous_id, + e.customer_id, + e.event_type, + e.event_at_utc, + e.page_url, + e.page_path, + e.referrer_host, + e.client_timezone + FROM web_events AS e + WHERE + e.event_at_utc >= DATEADD('hour', -@const("_read_margin_hours"), __cursor_start()) + AND e.event_at_utc < DATEADD('hour', @const("_read_margin_hours"), __cursor_end()) + AND NOT e.is_bot + AND e.anonymous_id IS NOT NULL +), + +gaps AS ( + SELECT + h.event_id, + h.anonymous_id, + h.customer_id, + h.event_type, + h.event_at_utc, + h.page_url, + h.page_path, + h.referrer_host, + h.client_timezone, + IFF(DATEDIFF( + 'minute', + LAG(h.event_at_utc) OVER (PARTITION BY h.anonymous_id ORDER BY h.event_at_utc, h.event_id), + h.event_at_utc + ) <= @const("session_gap_minutes"), 0, 1) AS is_session_start + FROM human_events AS h +), + +numbered AS ( + SELECT + g.anonymous_id, + g.customer_id, + g.event_type, + g.event_at_utc, + g.page_url, + g.page_path, + g.referrer_host, + g.client_timezone, + SUM(g.is_session_start) OVER ( + PARTITION BY g.anonymous_id + ORDER BY g.event_at_utc, g.event_id + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) AS session_number + FROM gaps AS g +), + +sessions AS ( + SELECT + n.anonymous_id, + MAX(n.customer_id) AS customer_id, + MIN(n.event_at_utc) AS session_start_utc, + MAX(n.event_at_utc) AS session_end_utc, + COUNT(*) AS event_count, + COUNT_IF(n.event_type = 'page_view') AS page_view_count, + MIN_BY(n.page_path, n.event_at_utc) AS landing_page_path, + MAX_BY(n.page_path, n.event_at_utc) AS exit_page_path, + MIN_BY(n.referrer_host, n.event_at_utc) AS referrer_host, + MIN_BY(n.page_url, n.event_at_utc) AS landing_page_url, + MIN_BY(n.client_timezone, n.event_at_utc) AS client_timezone, + BOOLOR_AGG(n.event_type = 'add_to_cart') AS has_add_to_cart, + BOOLOR_AGG(n.event_type = 'order_completed') AS has_order + FROM numbered AS n + GROUP BY + n.anonymous_id, + n.session_number +), + +final AS ( + SELECT + ( + s.anonymous_id || ':' || TO_CHAR(s.session_start_utc, 'YYYYMMDDHH24MISS') + )::VARCHAR AS session_id, + s.anonymous_id, + s.customer_id::VARCHAR AS customer_id, + DATE_TRUNC('day', s.session_start_utc)::TIMESTAMP_NTZ AS session_day, + s.session_start_utc, + s.session_end_utc, + CONVERT_TIMEZONE( + 'UTC', + COALESCE(s.client_timezone, 'UTC'), + s.session_start_utc + )::TIMESTAMP_NTZ AS session_start_visitor_local, + DATEDIFF('second', s.session_start_utc, s.session_end_utc)::INT AS duration_seconds, + s.event_count::INT AS event_count, + s.page_view_count::INT AS page_view_count, + s.landing_page_path::VARCHAR AS landing_page_path, + s.exit_page_path::VARCHAR AS exit_page_path, + s.referrer_host::VARCHAR AS referrer_host, + s.landing_page_url::VARCHAR AS landing_page_url, + ( + s.page_view_count <= 1 + AND s.event_count <= 1 + ) AS is_bounce, + s.has_add_to_cart::BOOLEAN AS has_add_to_cart, + s.has_order::BOOLEAN AS has_order + FROM sessions AS s + WHERE + s.session_start_utc >= __cursor_start() + AND s.session_start_utc < __cursor_end() +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/customers/_sqlbuild/_constants/vip_ltv_threshold.sql b/benchmarks/duckshop/project/models/intermediate/customers/_sqlbuild/_constants/vip_ltv_threshold.sql new file mode 100644 index 0000000000..38a73d7fba --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/_sqlbuild/_constants/vip_ltv_threshold.sql @@ -0,0 +1 @@ +CONSTANT (name vip_ltv_threshold, type decimal, value "2500.00"); diff --git a/benchmarks/duckshop/project/models/intermediate/customers/_sqlbuild/_macros/sql/is_status.sql b/benchmarks/duckshop/project/models/intermediate/customers/_sqlbuild/_macros/sql/is_status.sql new file mode 100644 index 0000000000..5681b53be6 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/_sqlbuild/_macros/sql/is_status.sql @@ -0,0 +1,7 @@ +MACRO ( + description "Whether the caller's order_status column equals one order status", + arguments (status ENUM order_status), + returns BOOLEAN, +); + +order_status = @status diff --git a/benchmarks/duckshop/project/models/intermediate/customers/int_customer_address_current.sql b/benchmarks/duckshop/project/models/intermediate/customers/int_customer_address_current.sql new file mode 100644 index 0000000000..59b20eb641 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/int_customer_address_current.sql @@ -0,0 +1,70 @@ +MODEL ( + description """ + One row per customer with their current default shipping address, falling back to the latest + address + """, + columns ( + customer_id (type VARCHAR, nullable false, audits [not_null, unique]), + address_id (type VARCHAR, audits [not_null]), + city (type VARCHAR), + postcode_canonical (type VARCHAR), + country_code (type VARCHAR), + region_code (type VARCHAR), + latitude (type NUMBER(9, 6)), + longitude (type NUMBER(9, 6)), + is_explicit_default (type BOOLEAN), + address_count (type INT), + distinct_countries (type INT), + ), +); + +WITH addresses AS ( + SELECT * + FROM __ref("stg_customer_addresses") +), + +countries AS ( + SELECT * + FROM __seed("countries") +), + +ranked AS ( + SELECT + a.customer_id, + a.address_id, + a.city, + a.postcode_canonical, + a.country_code, + a.latitude, + a.longitude, + a.is_default, + COUNT(*) OVER (PARTITION BY a.customer_id) AS address_count, + COUNT(DISTINCT a.country_code) OVER (PARTITION BY a.customer_id) AS distinct_countries + FROM addresses AS a + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY a.customer_id + ORDER BY a.is_default DESC, a.valid_from DESC, a.address_id DESC + ) = 1 +), + +final AS ( + SELECT + r.customer_id, + r.address_id, + r.city, + r.postcode_canonical, + r.country_code, + c.region_code::VARCHAR AS region_code, + r.latitude, + r.longitude, + r.is_default AS is_explicit_default, + r.address_count::INT AS address_count, + r.distinct_countries::INT AS distinct_countries + FROM ranked AS r + LEFT JOIN countries AS c + ON r.country_code = c.country_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/customers/int_customer_consent_state.sql b/benchmarks/duckshop/project/models/intermediate/customers/int_customer_consent_state.sql new file mode 100644 index 0000000000..f09095e0ad --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/int_customer_consent_state.sql @@ -0,0 +1,90 @@ +MODEL ( + description """ + One row per customer with the current state of each consent type, pivoted into columns, and the + full state as an object for downstream audit. + """, + columns ( + customer_id (type VARCHAR, nullable false, audits [not_null, unique]), + email_marketing (type BOOLEAN), + sms_marketing (type BOOLEAN), + personalisation (type BOOLEAN), + third_party_sharing (type BOOLEAN), + consent_state (type OBJECT), + last_consent_change_at (type TIMESTAMP_TZ), + has_legacy_consent (type BOOLEAN), + ), +); + +WITH consents AS ( + SELECT * + FROM __ref("stg_customer_consents") +), + +current_consents AS ( + SELECT + c.customer_id, + c.consent_type, + c.is_granted, + c.captured_at, + c.is_legacy + FROM consents AS c + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY c.customer_id, c.consent_type + ORDER BY c.captured_at DESC, c.consent_id DESC + ) = 1 +), + +consent_matrix AS ( + SELECT + cc.customer_id, + cc.consent_type, + cc.is_granted + FROM current_consents AS cc +), + +pivoted AS ( + SELECT + p.customer_id, + p.email_marketing, + p.sms_marketing, + p.personalisation, + p.third_party_sharing + FROM consent_matrix PIVOT( + MAX(is_granted) FOR consent_type IN ( + 'EMAIL_MARKETING', + 'SMS_MARKETING', + 'PERSONALISATION', + 'THIRD_PARTY_SHARING' + ) + ) AS p(customer_id, email_marketing, sms_marketing, personalisation, third_party_sharing) +), + +summary AS ( + SELECT + cc.customer_id, + OBJECT_AGG(cc.consent_type, cc.is_granted::VARIANT) AS consent_state, + MAX(cc.captured_at) AS last_consent_change_at, + BOOLOR_AGG(cc.is_legacy) AS has_legacy_consent + FROM current_consents AS cc + GROUP BY + cc.customer_id +), + +final AS ( + SELECT + s.customer_id, + COALESCE(p.email_marketing, FALSE)::BOOLEAN AS email_marketing, + COALESCE(p.sms_marketing, FALSE)::BOOLEAN AS sms_marketing, + COALESCE(p.personalisation, FALSE)::BOOLEAN AS personalisation, + COALESCE(p.third_party_sharing, FALSE)::BOOLEAN AS third_party_sharing, + s.consent_state::OBJECT AS consent_state, + s.last_consent_change_at, + s.has_legacy_consent::BOOLEAN AS has_legacy_consent + FROM summary AS s + LEFT JOIN pivoted AS p + ON s.customer_id = p.customer_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/customers/int_customer_identity.sql b/benchmarks/duckshop/project/models/intermediate/customers/int_customer_identity.sql new file mode 100644 index 0000000000..3250a757e6 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/int_customer_identity.sql @@ -0,0 +1,159 @@ +MODEL ( + description """ + One row per customer account with its identity keys across channels: a HASH-based dedupe key on + the normalised email (HASH output is engine-specific, an expected divergence across engines), + linked anonymous web visitors, newsletter and support footprints, and the most similar other + account sharing a phone number. + """, + tags [known_divergence], + audits [ + match_rate ( + name "email_resolved", + matched "email_normalised IS NOT NULL", + minimum_samples 50, + evidence_limit 20, + thresholds (warn (below 90), error (below 50)), + ), + ], + columns ( + customer_id (type VARCHAR, nullable false, audits [not_null, unique]), + email_normalised (type VARCHAR), + email_domain (type VARCHAR), + email_dedupe_key (type NUMBER(38, 0)), + phone_normalised (type VARCHAR), + full_name (type VARCHAR), + anonymous_ids (type ARRAY), + anonymous_id_count (type INT), + newsletter_lists (type ARRAY), + ticket_count (type INT), + duplicate_candidate_id (type VARCHAR), + duplicate_name_similarity (type INT), + accounts_sharing_email (type INT), + ), +); + +WITH customers AS ( + SELECT * + FROM __ref("stg_customers") +), + +web_identities AS ( + SELECT * + FROM __ref("stg_web_identifies") +), + +newsletter AS ( + SELECT * + FROM __ref("stg_newsletter_subscriptions") +), + +tickets AS ( + SELECT * + FROM __ref("stg_tickets") +), + +base AS ( + SELECT + c.customer_id, + c.email_normalised, + SPLIT_PART(c.email_normalised, '@', 2) AS email_domain, + HASH(c.email_normalised) AS email_dedupe_key, + c.phone_normalised, + TRIM(CONCAT_WS(' ', c.first_name, c.last_name)) AS full_name + FROM customers AS c +), + +anonymous_links AS ( + SELECT + w.customer_id, + ARRAY_AGG(DISTINCT w.anonymous_id) WITHIN GROUP (ORDER BY w.anonymous_id) AS anonymous_ids + FROM web_identities AS w + GROUP BY + w.customer_id +), + +newsletter_links AS ( + SELECT + b.customer_id, + ARRAY_AGG(DISTINCT n.list_code) WITHIN GROUP (ORDER BY n.list_code) AS newsletter_lists + FROM base AS b + INNER JOIN newsletter AS n + ON b.email_normalised = n.email + WHERE n.list_status = 'subscribed' + GROUP BY + b.customer_id +), + +ticket_links AS ( + SELECT + b.customer_id, + COUNT(DISTINCT t.ticket_id) AS ticket_count + FROM base AS b + INNER JOIN tickets AS t + ON b.email_normalised = t.customer_email + GROUP BY + b.customer_id +), + +shared_phone_pairs AS ( + SELECT + a.customer_id, + b.customer_id AS other_customer_id, + JAROWINKLER_SIMILARITY(LOWER(a.full_name), LOWER(b.full_name)) AS name_similarity + FROM base AS a + INNER JOIN base AS b + ON a.phone_normalised = b.phone_normalised + AND a.customer_id <> b.customer_id +), + +best_duplicate AS ( + SELECT + p.customer_id, + MAX_BY(p.other_customer_id, p.name_similarity) AS other_customer_id, + MAX(p.name_similarity) AS name_similarity + FROM shared_phone_pairs AS p + WHERE p.name_similarity >= 85 + GROUP BY + p.customer_id +), + +email_collisions AS ( + SELECT + b.email_dedupe_key, + COUNT(*) AS accounts_sharing_email + FROM base AS b + WHERE b.email_normalised IS NOT NULL + GROUP BY + b.email_dedupe_key +), + +final AS ( + SELECT + b.customer_id, + b.email_normalised, + b.email_domain::VARCHAR AS email_domain, + b.email_dedupe_key::NUMBER(38, 0) AS email_dedupe_key, + b.phone_normalised, + b.full_name::VARCHAR AS full_name, + COALESCE(a.anonymous_ids, ARRAY_CONSTRUCT())::ARRAY AS anonymous_ids, + ARRAY_SIZE(COALESCE(a.anonymous_ids, ARRAY_CONSTRUCT()))::INT AS anonymous_id_count, + COALESCE(n.newsletter_lists, ARRAY_CONSTRUCT())::ARRAY AS newsletter_lists, + COALESCE(t.ticket_count, 0)::INT AS ticket_count, + d.other_customer_id::VARCHAR AS duplicate_candidate_id, + d.name_similarity::INT AS duplicate_name_similarity, + COALESCE(e.accounts_sharing_email, 1)::INT AS accounts_sharing_email + FROM base AS b + LEFT JOIN anonymous_links AS a + ON b.customer_id = a.customer_id + LEFT JOIN newsletter_links AS n + ON b.customer_id = n.customer_id + LEFT JOIN ticket_links AS t + ON b.customer_id = t.customer_id + LEFT JOIN best_duplicate AS d + ON b.customer_id = d.customer_id + LEFT JOIN email_collisions AS e + ON b.email_dedupe_key = e.email_dedupe_key +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/customers/int_customer_order_sequence.sql b/benchmarks/duckshop/project/models/intermediate/customers/int_customer_order_sequence.sql new file mode 100644 index 0000000000..b6344f11bd --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/int_customer_order_sequence.sql @@ -0,0 +1,92 @@ +MODEL ( + description """ + One row per order with its position in the customer's order history, the gap since the previous + order, and running lifetime totals in USD. + """, + columns ( + order_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR, audits [not_null]), + placed_at (type TIMESTAMP_TZ), + placed_date_local (type DATE), + order_number (type INT), + is_first_order (type BOOLEAN), + days_since_previous_order (type INT), + amount_usd (type NUMBER(18, 2)), + lifetime_revenue_usd (type NUMBER(18, 2)), + rolling_90d_revenue_usd (type NUMBER(18, 2)), + previous_sales_channel (type VARCHAR), + is_channel_switch (type BOOLEAN), + cohort_month (type DATE), + ), +); + +WITH orders AS ( + SELECT * + FROM __ref("int_orders_usd") +), + +valid_orders AS ( + SELECT + o.order_id, + o.customer_id, + o.placed_at, + o.placed_date_local, + o.amount_usd, + o.sales_channel + FROM orders AS o + WHERE NOT @sql.is_status(@enum("order_status").CANCELLED) +), + +sequenced AS ( + SELECT + v.order_id, + v.customer_id, + v.placed_at, + v.placed_date_local, + v.amount_usd, + v.sales_channel, + ROW_NUMBER() OVER (PARTITION BY v.customer_id ORDER BY v.placed_at, v.order_id) AS order_number, + LAG(v.placed_date_local) OVER ( + PARTITION BY v.customer_id + ORDER BY v.placed_at, v.order_id + ) AS previous_date, + LAG(v.sales_channel) OVER ( + PARTITION BY v.customer_id + ORDER BY v.placed_at, v.order_id + ) AS previous_sales_channel, + SUM(v.amount_usd) OVER ( + PARTITION BY v.customer_id + ORDER BY v.placed_at, v.order_id + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) AS lifetime_revenue_usd, + SUM(v.amount_usd) OVER ( + PARTITION BY v.customer_id + ORDER BY v.placed_date_local + RANGE BETWEEN INTERVAL '89 days' PRECEDING AND CURRENT ROW + ) AS rolling_90d_revenue_usd, + MIN(v.placed_date_local) OVER (PARTITION BY v.customer_id) AS first_order_date + FROM valid_orders AS v +), + +final AS ( + SELECT + s.order_id, + s.customer_id, + s.placed_at, + s.placed_date_local, + s.order_number::INT AS order_number, + ( + s.order_number = 1 + ) AS is_first_order, + DATEDIFF('day', s.previous_date, s.placed_date_local)::INT AS days_since_previous_order, + s.amount_usd, + s.lifetime_revenue_usd::NUMBER(18, 2) AS lifetime_revenue_usd, + s.rolling_90d_revenue_usd::NUMBER(18, 2) AS rolling_90d_revenue_usd, + s.previous_sales_channel::VARCHAR AS previous_sales_channel, + COALESCE(s.previous_sales_channel <> s.sales_channel, FALSE) AS is_channel_switch, + DATE_TRUNC('month', s.first_order_date)::DATE AS cohort_month + FROM sequenced AS s +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/customers/int_customer_rfm.sql b/benchmarks/duckshop/project/models/intermediate/customers/int_customer_rfm.sql new file mode 100644 index 0000000000..20a3a6d0b7 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/int_customer_rfm.sql @@ -0,0 +1,100 @@ +MODEL ( + description """ + One row per ordering customer with recency, frequency and monetary scores in quintiles and a + named segment. + """, + enums ( + _segment [CHAMPION, LOYAL, PROMISING, AT_RISK, HIBERNATING, NEW], + ), + columns ( + customer_id (type VARCHAR, nullable false, audits [not_null, unique]), + last_order_date (type DATE), + days_since_last_order (type INT), + order_count (type INT), + lifetime_revenue_usd (type NUMBER(18, 2)), + average_order_usd (type NUMBER(18, 2)), + recency_score (type INT), + frequency_score (type INT), + monetary_score (type INT), + rfm_code (type VARCHAR), + segment (type VARCHAR), + is_vip (type BOOLEAN), + ), +); + +WITH order_sequence AS ( + SELECT * + FROM __ref("int_customer_order_sequence") +), + +as_of AS ( + SELECT MAX(o.placed_date_local) AS as_of_date + FROM order_sequence AS o +), + +per_customer AS ( + SELECT + o.customer_id, + MAX(o.placed_date_local) AS last_order_date, + COUNT(*) AS order_count, + MAX(o.lifetime_revenue_usd) AS lifetime_revenue_usd + FROM order_sequence AS o + GROUP BY + o.customer_id +), + +scored AS ( + SELECT + p.customer_id, + p.last_order_date, + DATEDIFF('day', p.last_order_date, a.as_of_date) AS days_since_last_order, + p.order_count, + p.lifetime_revenue_usd, + 6 + - NTILE(5) OVER ( + ORDER BY DATEDIFF('day', p.last_order_date, a.as_of_date), p.customer_id + ) AS recency_score, + NTILE(5) OVER (ORDER BY p.order_count, p.customer_id) AS frequency_score, + NTILE(5) OVER (ORDER BY p.lifetime_revenue_usd, p.customer_id) AS monetary_score + FROM per_customer AS p + CROSS JOIN as_of AS a +), + +final AS ( + SELECT + s.customer_id, + s.last_order_date, + s.days_since_last_order::INT AS days_since_last_order, + s.order_count::INT AS order_count, + s.lifetime_revenue_usd, + ROUND(s.lifetime_revenue_usd / NULLIF(s.order_count, 0), 2)::NUMBER(18, 2) AS average_order_usd, + s.recency_score::INT AS recency_score, + s.frequency_score::INT AS frequency_score, + s.monetary_score::INT AS monetary_score, + ( + s.recency_score::VARCHAR || s.frequency_score::VARCHAR || s.monetary_score::VARCHAR + )::VARCHAR AS rfm_code, + CASE + WHEN s.order_count = 1 + AND s.recency_score >= 4 + THEN @enum("_segment").NEW + WHEN s.recency_score >= 4 + AND s.frequency_score >= 4 + THEN @enum("_segment").CHAMPION + WHEN s.frequency_score >= 3 + AND s.recency_score >= 3 + THEN @enum("_segment").LOYAL + WHEN s.recency_score >= 3 + THEN @enum("_segment").PROMISING + WHEN s.frequency_score >= 3 + THEN @enum("_segment").AT_RISK + ELSE @enum("_segment").HIBERNATING + END::VARCHAR AS segment, + ( + s.lifetime_revenue_usd >= @const("vip_ltv_threshold") + ) AS is_vip + FROM scored AS s +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/customers/int_gift_card_balances.sql b/benchmarks/duckshop/project/models/intermediate/customers/int_gift_card_balances.sql new file mode 100644 index 0000000000..008934d91b --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/int_gift_card_balances.sql @@ -0,0 +1,122 @@ +MODEL ( + description """ + One row per gift card with its remaining balance in major units and USD and its liability status + """, + columns ( + gift_card_id (type VARCHAR, nullable false, audits [not_null, unique]), + currency_code (type VARCHAR), + issued_at (type TIMESTAMP_TZ), + expires_on (type DATE), + initial_amount (type NUMBER(38, 6)), + redeemed_amount (type NUMBER(38, 6)), + balance_amount (type NUMBER(38, 6)), + balance_usd (type NUMBER(18, 2)), + transaction_count (type INT), + last_used_at (type TIMESTAMP_TZ), + liability_status (type VARCHAR, audits [ + accepted_values (values ["active", "fully_redeemed", "expired_unused", "expired_partial"]), + ]), + ), +); + +WITH gift_cards AS ( + SELECT * + FROM __ref("stg_gift_cards") +), + +transactions AS ( + SELECT * + FROM __ref("stg_gift_card_transactions") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +movement AS ( + SELECT + t.gift_card_id, + SUM(IFF(t.signed_amount_minor < 0, -t.signed_amount_minor, 0)) AS redeemed_minor, + SUM(t.signed_amount_minor) AS net_movement_minor, + COUNT(*) AS transaction_count, + MAX(t.occurred_at) AS last_used_at + FROM transactions AS t + GROUP BY + t.gift_card_id +), + +balances AS ( + SELECT + g.gift_card_id, + g.currency_code, + g.issued_at, + g.expires_on, + g.initial_amount_minor, + COALESCE(m.redeemed_minor, 0) AS redeemed_minor, + GREATEST(g.initial_amount_minor + COALESCE(m.net_movement_minor, 0), 0) AS balance_minor, + COALESCE(m.transaction_count, 0) AS transaction_count, + m.last_used_at, + c.minor_unit_exponent, + '@@valuation_date'::DATE AS valuation_date + FROM gift_cards AS g + INNER JOIN currencies AS c + ON g.currency_code = c.currency_code + LEFT JOIN movement AS m + ON g.gift_card_id = m.gift_card_id +), + +valued AS ( + SELECT + b.gift_card_id, + b.currency_code, + b.issued_at, + b.expires_on, + b.initial_amount_minor, + b.redeemed_minor, + b.balance_minor, + b.transaction_count, + b.last_used_at, + b.minor_unit_exponent, + b.valuation_date, + fx.rate_to_usd + FROM balances AS b + ASOF JOIN fx_rates AS fx MATCH_CONDITION (b.valuation_date >= fx.rate_date) + ON b.currency_code = fx.currency_code +), + +final AS ( + SELECT + v.gift_card_id, + v.currency_code, + v.issued_at, + v.expires_on, + @sql.money_from_minor(v.initial_amount_minor, v.minor_unit_exponent, places := 3) AS initial_amount, + @sql.money_from_minor(v.redeemed_minor, v.minor_unit_exponent, places := 3) AS redeemed_amount, + @sql.money_from_minor(v.balance_minor, v.minor_unit_exponent, places := 3) AS balance_amount, + ROUND( + @sql.money_from_minor(v.balance_minor, v.minor_unit_exponent, places := 6) * v.rate_to_usd, + 2 + )::NUMBER(18, 2) AS balance_usd, + v.transaction_count::INT AS transaction_count, + v.last_used_at, + CASE + WHEN v.balance_minor = 0 + THEN 'fully_redeemed' + WHEN v.expires_on < v.valuation_date + AND v.redeemed_minor = 0 + THEN 'expired_unused' + WHEN v.expires_on < v.valuation_date + THEN 'expired_partial' + ELSE 'active' + END::VARCHAR AS liability_status + FROM valued AS v +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/customers/int_loyalty_balances.sql b/benchmarks/duckshop/project/models/intermediate/customers/int_loyalty_balances.sql new file mode 100644 index 0000000000..22246baa20 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/int_loyalty_balances.sql @@ -0,0 +1,84 @@ +MODEL ( + description """ + One row per loyalty ledger posting with the account's running points balance, the points that + expire unused, and the balance valued at the programme rate. + """, + constants ( + _point_value_usd constant(type decimal, value "0.0100"), + ), + columns ( + entry_id (type VARCHAR, nullable false, audits [not_null, unique]), + loyalty_account_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + posted_at (type TIMESTAMP_TZ), + points_delta (type INT), + running_points (type INT), + expiring_points (type INT), + is_expired (type BOOLEAN), + running_value_usd (type NUMBER(18, 4)), + reason_code (type VARCHAR), + tier_code (type VARCHAR), + ), +); + +WITH ledger AS ( + SELECT * + FROM __ref("stg_loyalty_ledger") +), + +accounts AS ( + SELECT * + FROM __ref("stg_loyalty_accounts") +), + +postings AS ( + SELECT + l.entry_id, + l.loyalty_account_id, + l.posted_at, + l.points_delta, + l.reason_code, + l.expires_on, + SUM(l.points_delta) OVER ( + PARTITION BY l.loyalty_account_id + ORDER BY l.posted_at, l.entry_id + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) AS running_points + FROM ledger AS l +), + +expiry AS ( + SELECT + p.entry_id, + IFF( + p.points_delta > 0 AND p.expires_on < '@@valuation_date'::DATE, + p.points_delta, + 0 + ) AS expiring_points + FROM postings AS p +), + +final AS ( + SELECT + p.entry_id, + p.loyalty_account_id, + a.customer_id, + p.posted_at, + p.points_delta, + p.running_points::INT AS running_points, + e.expiring_points::INT AS expiring_points, + ( + e.expiring_points > 0 + ) AS is_expired, + ROUND(p.running_points * @const("_point_value_usd"), 4)::NUMBER(18, 4) AS running_value_usd, + p.reason_code, + a.tier_code + FROM postings AS p + INNER JOIN expiry AS e + ON p.entry_id = e.entry_id + LEFT JOIN accounts AS a + ON p.loyalty_account_id = a.loyalty_account_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/customers/int_loyalty_tier_changes.sql b/benchmarks/duckshop/project/models/intermediate/customers/int_loyalty_tier_changes.sql new file mode 100644 index 0000000000..87474144f0 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/int_loyalty_tier_changes.sql @@ -0,0 +1,93 @@ +MODEL ( + description """ + One row per loyalty tier change, flattened from each account's tier history with direction and + tenure + """, + columns ( + tier_change_key (type VARCHAR, nullable false, audits [not_null, unique]), + loyalty_account_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + change_sequence (type INT), + tier_code (type VARCHAR), + previous_tier_code (type VARCHAR), + changed_at (type TIMESTAMP_TZ), + direction (type VARCHAR, audits [ + accepted_values (values ["enrol", "upgrade", "downgrade", "same"]), + ]), + days_in_previous_tier (type INT), + ), +); + +WITH accounts AS ( + SELECT * + FROM __ref("stg_loyalty_accounts") +), + +history AS ( + SELECT + a.loyalty_account_id, + a.customer_id, + h.index::INT AS change_sequence, + UPPER(h.value:tier::VARCHAR) AS tier_code, + TRY_TO_TIMESTAMP_TZ(h.value:at::VARCHAR) AS changed_at + FROM accounts AS a, LATERAL FLATTEN(input => a.tier_history) AS h +), + +with_previous AS ( + SELECT + h.loyalty_account_id, + h.customer_id, + h.change_sequence, + h.tier_code, + h.changed_at, + LAG(h.tier_code) OVER ( + PARTITION BY h.loyalty_account_id + ORDER BY h.change_sequence + ) AS previous_tier_code, + LAG(h.changed_at) OVER ( + PARTITION BY h.loyalty_account_id + ORDER BY h.change_sequence + ) AS previous_changed_at + FROM history AS h +), + +final AS ( + SELECT + ( + w.loyalty_account_id || ':' || w.change_sequence::VARCHAR + )::VARCHAR AS tier_change_key, + w.loyalty_account_id, + w.customer_id, + w.change_sequence, + w.tier_code::VARCHAR AS tier_code, + w.previous_tier_code::VARCHAR AS previous_tier_code, + w.changed_at, + CASE + WHEN w.previous_tier_code IS NULL + THEN 'enrol' + WHEN DECODE( + w.tier_code, + 'DUCKLING', 1, + 'DRAKE', 2, + 'GOLDEN_DUCK', 3, + 'PLATINUM_DUCK', 4, + 0 + ) > DECODE( + w.previous_tier_code, + 'DUCKLING', 1, + 'DRAKE', 2, + 'GOLDEN_DUCK', 3, + 'PLATINUM_DUCK', 4, + 0 + ) + THEN 'upgrade' + WHEN w.tier_code = w.previous_tier_code + THEN 'same' + ELSE 'downgrade' + END::VARCHAR AS direction, + DATEDIFF('day', w.previous_changed_at, w.changed_at)::INT AS days_in_previous_tier + FROM with_previous AS w +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/customers/int_store_credit_balances.sql b/benchmarks/duckshop/project/models/intermediate/customers/int_store_credit_balances.sql new file mode 100644 index 0000000000..1a474b5375 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/int_store_credit_balances.sql @@ -0,0 +1,120 @@ +MODEL ( + description """ + One row per customer and currency with the store credit balance, the amount expiring within 30 + days and its age + """, + columns ( + store_credit_key (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR, audits [not_null]), + currency_code (type VARCHAR), + balance_minor (type INT), + expiring_30d_minor (type INT), + first_posted_at (type TIMESTAMP_TZ), + last_posted_at (type TIMESTAMP_TZ), + posting_count (type INT), + goodwill_share (type NUMBER(9, 6)), + balance_usd (type NUMBER(18, 2)), + ), +); + +WITH ledger AS ( + SELECT * + FROM __ref("stg_store_credit_ledger") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +grouped AS ( + SELECT + l.customer_id, + l.currency_code, + SUM(l.amount_minor) AS balance_minor, + SUM( + IFF( + l.amount_minor > 0 + AND l.expires_on BETWEEN '@@valuation_date'::DATE AND DATEADD( + 'day', + 30, + '@@valuation_date'::DATE + ), + l.amount_minor, + 0 + ) + ) AS expiring_30d_minor, + MIN(l.posted_at) AS first_posted_at, + MAX(l.posted_at) AS last_posted_at, + COUNT(*) AS posting_count, + SUM(IFF(l.reason_code LIKE 'GOODWILL%', l.amount_minor, 0)) AS goodwill_minor, + SUM(IFF(l.amount_minor > 0, l.amount_minor, 0)) AS issued_minor + FROM ledger AS l + GROUP BY + l.customer_id, + l.currency_code +), + +valued AS ( + SELECT + g.customer_id, + g.currency_code, + g.balance_minor, + g.expiring_30d_minor, + g.first_posted_at, + g.last_posted_at, + g.posting_count, + g.goodwill_minor, + g.issued_minor, + '@@valuation_date'::DATE AS valuation_date, + c.minor_unit_exponent + FROM grouped AS g + LEFT JOIN currencies AS c + ON g.currency_code = c.currency_code +), + +with_rate AS ( + SELECT + v.customer_id, + v.currency_code, + v.balance_minor, + v.expiring_30d_minor, + v.first_posted_at, + v.last_posted_at, + v.posting_count, + v.goodwill_minor, + v.issued_minor, + v.minor_unit_exponent, + fx.rate_to_usd + FROM valued AS v + ASOF JOIN fx_rates AS fx MATCH_CONDITION (v.valuation_date >= fx.rate_date) + ON v.currency_code = fx.currency_code +), + +final AS ( + SELECT + ( + g.customer_id || ':' || g.currency_code + )::VARCHAR AS store_credit_key, + g.customer_id, + g.currency_code, + g.balance_minor::INT AS balance_minor, + LEAST(g.expiring_30d_minor, GREATEST(g.balance_minor, 0))::INT AS expiring_30d_minor, + g.first_posted_at, + g.last_posted_at, + g.posting_count::INT AS posting_count, + ROUND(g.goodwill_minor / NULLIF(g.issued_minor, 0), 6)::NUMBER(9, 6) AS goodwill_share, + ROUND( + @sql.money_from_minor(g.balance_minor, COALESCE(g.minor_unit_exponent, 2), places := 6) * g.rate_to_usd, + 2 + )::NUMBER(18, 2) AS balance_usd + FROM with_rate AS g +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/customers/int_wishlist_demand.sql b/benchmarks/duckshop/project/models/intermediate/customers/int_wishlist_demand.sql new file mode 100644 index 0000000000..04c5a525c8 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/customers/int_wishlist_demand.sql @@ -0,0 +1,100 @@ +MODEL ( + description """ + One row per SKU with how many customers have it on a wishlist and how many of them later bought + it + """, + columns ( + sku (type VARCHAR, nullable false, audits [not_null, unique]), + wishlisting_customers (type INT), + public_wishlist_count (type INT), + purchased_after_wishlist (type INT), + wishlist_conversion_rate (type NUMBER(9, 6)), + median_days_to_purchase (type NUMBER(10, 2)), + first_added_at (type TIMESTAMP_TZ), + ), +); + +WITH wishlist_items AS ( + SELECT * + FROM __ref("stg_wishlist_items") +), + +wishlists AS ( + SELECT * + FROM __ref("stg_wishlists") +), + +order_lines AS ( + SELECT * + FROM __ref("stg_order_lines") +), + +orders AS ( + SELECT * + FROM __ref("stg_orders") +), + +saved AS ( + SELECT + i.sku, + i.customer_id, + MIN(i.added_at) AS added_at, + MAX(IFF(w.is_public, 1, 0)) AS on_public_list + FROM wishlist_items AS i + INNER JOIN wishlists AS w + ON i.wishlist_id = w.wishlist_id + GROUP BY + i.sku, + i.customer_id +), + +purchases AS ( + SELECT + l.sku, + o.customer_id, + MIN(o.placed_at) AS first_purchased_at + FROM order_lines AS l + INNER JOIN orders AS o + ON l.order_id = o.order_id + GROUP BY + l.sku, + o.customer_id +), + +matched AS ( + SELECT + s.sku, + s.customer_id, + s.added_at, + s.on_public_list, + p.first_purchased_at, + IFF( + p.first_purchased_at >= s.added_at, + DATEDIFF('hour', s.added_at, p.first_purchased_at) / 24, + NULL + ) AS days_to_purchase + FROM saved AS s + LEFT JOIN purchases AS p + ON s.sku = p.sku + AND s.customer_id = p.customer_id +), + +final AS ( + SELECT + m.sku, + COUNT(DISTINCT m.customer_id)::INT AS wishlisting_customers, + SUM(m.on_public_list)::INT AS public_wishlist_count, + COUNT_IF(m.first_purchased_at >= m.added_at)::INT AS purchased_after_wishlist, + ROUND( + COUNT_IF(m.first_purchased_at >= m.added_at) / NULLIF(COUNT(DISTINCT m.customer_id), 0), + 6 + )::NUMBER(9, 6) AS wishlist_conversion_rate, + MEDIAN(m.days_to_purchase)::NUMBER(10, 2) AS median_days_to_purchase, + MIN(m.added_at) AS first_added_at + FROM matched AS m + GROUP BY + m.sku +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/finance/_sqlbuild/_constants/reporting_timezone_by_region.sql b/benchmarks/duckshop/project/models/intermediate/finance/_sqlbuild/_constants/reporting_timezone_by_region.sql new file mode 100644 index 0000000000..7f124571a1 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/finance/_sqlbuild/_constants/reporting_timezone_by_region.sql @@ -0,0 +1,9 @@ +CONSTANT ( + name reporting_timezone_by_region, + value ( + EU "Europe/Berlin", + US "America/New_York", + APAC "Asia/Tokyo", + MENA "Asia/Kuwait", + ), +); diff --git a/benchmarks/duckshop/project/models/intermediate/finance/int_chargeback_exposure.sql b/benchmarks/duckshop/project/models/intermediate/finance/int_chargeback_exposure.sql new file mode 100644 index 0000000000..2ce5d4127d --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/finance/int_chargeback_exposure.sql @@ -0,0 +1,189 @@ +MODEL ( + description """ + One row per chargeback with its amount in USD, the evidence document kinds submitted, the + response deadline for its reason family, and the amount still at risk on the valuation date. + """, + constants ( + _response_days_by_family (fraud 20, consumer_dispute 30, processing 45), + ), + columns ( + chargeback_id (type VARCHAR, nullable false, audits [not_null, unique]), + payment_id (type VARCHAR, audits [not_null]), + order_id (type VARCHAR), + customer_id (type VARCHAR), + payment_method (type VARCHAR), + reason_code (type VARCHAR), + reason_family (type VARCHAR), + opened_at (type TIMESTAMP_TZ), + response_due_on (type DATE), + outcome (type VARCHAR), + amount_usd (type NUMBER(18, 2)), + share_of_payment (type NUMBER(9, 6)), + evidence_kinds (type ARRAY), + has_delivery_proof (type BOOLEAN), + days_open (type INT), + is_overdue (type BOOLEAN), + at_risk_usd (type NUMBER(18, 2)), + prior_chargebacks_for_customer (type INT), + ), +); + +WITH chargebacks AS ( + SELECT * + FROM __ref("stg_chargebacks") +), + +payments AS ( + SELECT * + FROM __ref("stg_payments") +), + +orders AS ( + SELECT * + FROM __ref("int_orders_usd") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +evidence AS ( + SELECT + c.chargeback_id, + ARRAY_AGG(DISTINCT LOWER(d.value:kind::VARCHAR)) WITHIN GROUP ( + ORDER BY LOWER(d.value:kind::VARCHAR) + ) AS evidence_kinds, + BOOLOR_AGG(REGEXP_LIKE( + LOWER(d.value:kind::VARCHAR), + '.*(tracking|delivery|signature|pod).*' + )) AS has_delivery_proof + FROM chargebacks AS c, LATERAL FLATTEN(input => c.evidence:documents) AS d + GROUP BY + c.chargeback_id +), + +with_payment AS ( + SELECT + c.chargeback_id, + c.payment_id, + p.order_id, + p.payment_method, + c.reason_code, + c.reason_family, + c.opened_at, + CONVERT_TIMEZONE('UTC', c.opened_at)::DATE AS opened_date, + c.closed_at, + c.outcome, + c.amount_minor, + c.currency_code, + p.amount_minor AS payment_amount_minor + FROM chargebacks AS c + LEFT JOIN payments AS p + ON c.payment_id = p.payment_id +), + +with_rate AS ( + SELECT + w.chargeback_id, + w.payment_id, + w.order_id, + w.payment_method, + w.reason_code, + w.reason_family, + w.opened_at, + w.opened_date, + w.closed_at, + w.outcome, + w.amount_minor, + w.payment_amount_minor, + cur.minor_unit_exponent, + fx.rate_to_usd + FROM with_payment AS w + LEFT JOIN currencies AS cur + ON w.currency_code = cur.currency_code + ASOF JOIN fx_rates AS fx MATCH_CONDITION (w.opened_date >= fx.rate_date) + ON w.currency_code = fx.currency_code +), + +with_order AS ( + SELECT + r.chargeback_id, + r.payment_id, + r.order_id, + o.customer_id, + r.payment_method, + r.reason_code, + r.reason_family, + r.opened_at, + r.opened_date, + DATEADD( + 'day', + COALESCE(@const("_response_days_by_family")[r.reason_family]::INT, 30), + r.opened_date + ) AS response_due_on, + r.closed_at, + r.outcome, + ROUND( + @sql.money_from_minor(r.amount_minor, COALESCE(r.minor_unit_exponent, 2), places := 6) * r.rate_to_usd, + 2 + ) AS amount_usd, + r.amount_minor / NULLIF(r.payment_amount_minor, 0) AS share_of_payment + FROM with_rate AS r + LEFT JOIN orders AS o + ON r.order_id = o.order_id +), + +customer_history AS ( + SELECT + w.chargeback_id, + COUNT(*) OVER ( + PARTITION BY w.customer_id + ORDER BY w.opened_at + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) + - 1 AS prior_chargebacks_for_customer + FROM with_order AS w +), + +final AS ( + SELECT + w.chargeback_id, + w.payment_id, + w.order_id, + w.customer_id, + w.payment_method, + w.reason_code, + w.reason_family, + w.opened_at, + w.response_due_on::DATE AS response_due_on, + w.outcome, + w.amount_usd::NUMBER(18, 2) AS amount_usd, + ROUND(w.share_of_payment, 6)::NUMBER(9, 6) AS share_of_payment, + COALESCE(e.evidence_kinds, ARRAY_CONSTRUCT())::ARRAY AS evidence_kinds, + COALESCE(e.has_delivery_proof, FALSE)::BOOLEAN AS has_delivery_proof, + DATEDIFF( + 'day', + w.opened_date, + COALESCE(CONVERT_TIMEZONE('UTC', w.closed_at)::DATE, '@@valuation_date'::DATE) + )::INT AS days_open, + ( + w.closed_at IS NULL + AND w.response_due_on < '@@valuation_date'::DATE + )::BOOLEAN AS is_overdue, + IFF(w.outcome IN ('open', 'lost', 'pending'), w.amount_usd, 0)::NUMBER(18, 2) AS at_risk_usd, + h.prior_chargebacks_for_customer::INT AS prior_chargebacks_for_customer + FROM with_order AS w + INNER JOIN customer_history AS h + ON w.chargeback_id = h.chargeback_id + LEFT JOIN evidence AS e + ON w.chargeback_id = e.chargeback_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/finance/int_invoices_reconciled.sql b/benchmarks/duckshop/project/models/intermediate/finance/int_invoices_reconciled.sql new file mode 100644 index 0000000000..cea09fce59 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/finance/int_invoices_reconciled.sql @@ -0,0 +1,187 @@ +MODEL ( + description """ + One row per invoice with its number parsed into series and sequence, gaps in each series' + numbering, and the invoice's tax reconciled three ways: the header total, the embedded tax-line + array and the separately loaded tax lines, against the order's own tax. + """, + constants ( + _tax_tolerance_minor 1, + ), + columns ( + invoice_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + invoice_series (type VARCHAR), + invoice_sequence (type INT), + sequence_gap (type INT), + issued_at (type TIMESTAMP_TZ), + issued_date_local (type DATE), + currency_code (type VARCHAR), + net (type NUMBER(38, 6)), + tax (type NUMBER(38, 6)), + gross (type NUMBER(38, 6)), + embedded_tax_minor (type INT), + loaded_tax_minor (type INT), + order_tax_minor (type INT), + tax_line_count (type INT), + tax_codes (type ARRAY), + header_balances (type BOOLEAN), + tax_lines_agree (type BOOLEAN), + matches_order_tax (type BOOLEAN), + invoice_currency_matches_order (type BOOLEAN), + ), +); + +WITH invoices AS ( + SELECT * + FROM __ref("stg_invoices") +), + +tax_lines AS ( + SELECT * + FROM __ref("stg_invoice_tax_lines") +), + +orders AS ( + SELECT * + FROM __ref("int_orders_enriched") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +numbered AS ( + SELECT + i.invoice_id, + i.order_id, + COALESCE( + i.invoice_series, + REGEXP_SUBSTR(i.invoice_number, '^([A-Z]+)-', 1, 1, 'e', 1), + 'NONE' + ) AS invoice_series, + TRY_TO_NUMBER(REGEXP_SUBSTR(i.invoice_number, '([0-9]+)$', 1, 1, 'e', 1)) AS invoice_sequence, + i.issued_at, + i.currency_code, + i.net_minor, + i.tax_minor, + i.gross_minor, + i.tax_lines + FROM invoices AS i +), + +sequenced AS ( + SELECT + n.invoice_id, + n.invoice_sequence + - LAG(n.invoice_sequence) OVER ( + PARTITION BY n.invoice_series + ORDER BY n.invoice_sequence, n.invoice_id + ) + - 1 AS sequence_gap + FROM numbered AS n +), + +embedded AS ( + SELECT + n.invoice_id, + SUM(t.value:tax_minor::INT) AS embedded_tax_minor, + COUNT(t.value) AS embedded_line_count + FROM numbered AS n, LATERAL FLATTEN(input => n.tax_lines) AS t + GROUP BY + n.invoice_id +), + +loaded AS ( + SELECT + l.invoice_id, + SUM(l.tax_minor) AS loaded_tax_minor, + COUNT(*) AS loaded_line_count, + ARRAY_AGG(DISTINCT l.tax_code) WITHIN GROUP (ORDER BY l.tax_code) AS tax_codes + FROM tax_lines AS l + GROUP BY + l.invoice_id +), + +order_tax AS ( + SELECT + o.order_id, + o.tax_amount_minor, + o.currency_code, + o.storefront_timezone + FROM orders AS o +), + +joined AS ( + SELECT + n.invoice_id, + n.order_id, + n.invoice_series, + n.invoice_sequence, + s.sequence_gap, + n.issued_at, + n.currency_code, + n.net_minor, + n.tax_minor, + n.gross_minor, + c.minor_unit_exponent, + e.embedded_tax_minor, + e.embedded_line_count, + l.loaded_tax_minor, + l.loaded_line_count, + l.tax_codes, + o.tax_amount_minor AS order_tax_minor, + o.currency_code AS order_currency_code, + o.storefront_timezone + FROM numbered AS n + INNER JOIN sequenced AS s + ON n.invoice_id = s.invoice_id + LEFT JOIN currencies AS c + ON n.currency_code = c.currency_code + LEFT JOIN embedded AS e + ON n.invoice_id = e.invoice_id + LEFT JOIN loaded AS l + ON n.invoice_id = l.invoice_id + LEFT JOIN order_tax AS o + ON n.order_id = o.order_id +), + +final AS ( + SELECT + j.invoice_id, + j.order_id, + j.invoice_series::VARCHAR AS invoice_series, + j.invoice_sequence::INT AS invoice_sequence, + NULLIF(j.sequence_gap, 0)::INT AS sequence_gap, + j.issued_at, + @sql.local_business_date(j.issued_at, COALESCE(j.storefront_timezone, 'UTC')) AS issued_date_local, + j.currency_code, + @sql.money_from_minor(j.net_minor, COALESCE(j.minor_unit_exponent, 2)) AS net, + @sql.money_from_minor(j.tax_minor, COALESCE(j.minor_unit_exponent, 2)) AS tax, + @sql.money_from_minor(j.gross_minor, COALESCE(j.minor_unit_exponent, 2)) AS gross, + j.embedded_tax_minor::INT AS embedded_tax_minor, + j.loaded_tax_minor::INT AS loaded_tax_minor, + j.order_tax_minor, + GREATEST( + COALESCE(j.embedded_line_count, 0), + COALESCE(j.loaded_line_count, 0) + )::INT AS tax_line_count, + COALESCE(j.tax_codes, ARRAY_CONSTRUCT())::ARRAY AS tax_codes, + ( + j.net_minor + j.tax_minor = j.gross_minor + )::BOOLEAN AS header_balances, + COALESCE( + ABS(j.embedded_tax_minor - j.loaded_tax_minor) <= @const("_tax_tolerance_minor") + AND ABS(j.loaded_tax_minor - j.tax_minor) <= @const("_tax_tolerance_minor"), + FALSE + )::BOOLEAN AS tax_lines_agree, + COALESCE( + ABS(j.tax_minor - j.order_tax_minor) <= @const("_tax_tolerance_minor"), + FALSE + )::BOOLEAN AS matches_order_tax, + EQUAL_NULL(j.currency_code, j.order_currency_code)::BOOLEAN AS invoice_currency_matches_order + FROM joined AS j +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/finance/int_ledger_balances_daily.sql b/benchmarks/duckshop/project/models/intermediate/finance/int_ledger_balances_daily.sql new file mode 100644 index 0000000000..4f3c92c619 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/finance/int_ledger_balances_daily.sql @@ -0,0 +1,141 @@ +MODEL ( + description """ + One row per ledger account, currency and calendar day from the account's first posting to the + valuation date, with the day's debits and credits, the closing balance in major units, and the + month-to-date movement. Days without postings carry the previous balance forward. + """, + columns ( + balance_key (type VARCHAR, nullable false, audits [not_null, unique]), + account_code (type VARCHAR, audits [not_null]), + account_class (type VARCHAR), + currency_code (type VARCHAR), + calendar_date (type DATE, audits [not_null]), + debit_minor (type INT), + credit_minor (type INT), + posting_count (type INT), + closing_balance_minor (type INT), + closing_balance (type NUMBER(38, 6)), + month_to_date_movement_minor (type INT), + is_month_end (type BOOLEAN), + is_normal_side (type BOOLEAN), + ), +); + +WITH entries AS ( + SELECT * + FROM __ref("stg_ledger_entries") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +@sql.date_spine(@const("history_start_date"), 730) + +posting_days AS ( + SELECT + e.account_code, + e.account_class, + e.currency_code, + CONVERT_TIMEZONE('UTC', e.posted_at)::DATE AS posted_date, + e.signed_amount_minor + FROM entries AS e +), daily_postings AS ( + SELECT + p.account_code, + p.currency_code, + p.posted_date, + SUM(IFF(p.signed_amount_minor > 0, p.signed_amount_minor, 0)) AS debit_minor, + SUM(IFF(p.signed_amount_minor < 0, -p.signed_amount_minor, 0)) AS credit_minor, + SUM(p.signed_amount_minor) AS net_minor, + COUNT(*) AS posting_count + FROM posting_days AS p + GROUP BY + p.account_code, + p.currency_code, + p.posted_date +), accounts AS ( + SELECT + p.account_code, + p.currency_code, + MAX(p.account_class) AS account_class, + MIN(p.posted_date) AS first_posted_date + FROM posting_days AS p + GROUP BY + p.account_code, + p.currency_code +), account_days AS ( + SELECT + a.account_code, + a.currency_code, + a.account_class, + d.calendar_date + FROM accounts AS a + INNER JOIN date_spine AS d + ON d.calendar_date >= a.first_posted_date + WHERE d.calendar_date <= '@@valuation_date'::DATE +), filled AS ( + SELECT + a.account_code, + a.currency_code, + a.account_class, + a.calendar_date, + COALESCE(p.debit_minor, 0) AS debit_minor, + COALESCE(p.credit_minor, 0) AS credit_minor, + COALESCE(p.net_minor, 0) AS net_minor, + COALESCE(p.posting_count, 0) AS posting_count + FROM account_days AS a + LEFT JOIN daily_postings AS p + ON a.account_code = p.account_code + AND a.currency_code = p.currency_code + AND a.calendar_date = p.posted_date +), balances AS ( + SELECT + f.account_code, + f.currency_code, + f.account_class, + f.calendar_date, + f.debit_minor, + f.credit_minor, + f.posting_count, + SUM(f.net_minor) OVER ( + PARTITION BY f.account_code, f.currency_code + ORDER BY f.calendar_date + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) AS closing_balance_minor, + SUM(f.net_minor) OVER ( + PARTITION BY f.account_code, f.currency_code, DATE_TRUNC('month', f.calendar_date) + ORDER BY f.calendar_date + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) AS month_to_date_movement_minor + FROM filled AS f +), final AS ( + SELECT + ( + b.account_code || ':' || b.currency_code || ':' || TO_VARCHAR(b.calendar_date, 'YYYY-MM-DD') + )::VARCHAR AS balance_key, + b.account_code, + b.account_class::VARCHAR AS account_class, + b.currency_code, + b.calendar_date, + b.debit_minor::INT AS debit_minor, + b.credit_minor::INT AS credit_minor, + b.posting_count::INT AS posting_count, + b.closing_balance_minor::INT AS closing_balance_minor, + @sql.money_from_minor(b.closing_balance_minor, COALESCE(c.minor_unit_exponent, 2)) AS closing_balance, + b.month_to_date_movement_minor::INT AS month_to_date_movement_minor, + ( + b.calendar_date = LAST_DAY(b.calendar_date) + )::BOOLEAN AS is_month_end, + CASE + WHEN b.account_class IN ('asset', 'expense') + THEN b.closing_balance_minor >= 0 + ELSE b.closing_balance_minor <= 0 + END::BOOLEAN AS is_normal_side + FROM balances AS b + LEFT JOIN currencies AS c + ON b.currency_code = c.currency_code +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/finance/int_payout_reconciliation.sql b/benchmarks/duckshop/project/models/intermediate/finance/int_payout_reconciliation.sql new file mode 100644 index 0000000000..40a387f486 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/finance/int_payout_reconciliation.sql @@ -0,0 +1,212 @@ +MODEL ( + description """ + One row per provider payout reconciled against its own line items, the payments those lines + reference, and the provider's settlement report for the payout date. Lines that reference no + known payment are counted as orphans. + """, + constants ( + _tolerance_minor 5, + ), + columns ( + payout_id (type VARCHAR, nullable false, audits [not_null, unique]), + provider_code (type VARCHAR, audits [not_null]), + currency_code (type VARCHAR), + paid_out_on (type DATE), + payout_net_minor (type INT), + lines_net_minor (type INT), + charge_line_count (type INT), + refund_line_count (type INT), + orphan_line_count (type INT), + embedded_line_count (type INT), + loaded_line_count (type INT), + matched_payment_minor (type INT), + report_id (type VARCHAR), + report_net_minor (type INT), + lines_match_payout (type BOOLEAN), + report_matches_payout (type BOOLEAN), + reconciliation_status (type VARCHAR), + days_since_previous_payout (type INT), + ), +); + +WITH payouts AS ( + SELECT * + FROM __ref("stg_payouts") +), + +payout_lines AS ( + SELECT * + FROM __ref("stg_payout_lines") +), + +payments AS ( + SELECT * + FROM __ref("stg_payments") +), + +reports AS ( + SELECT * + FROM __ref("stg_settlement_reports") +), + +payout_sequence AS ( + SELECT + p.payout_id, + DATEDIFF( + 'day', + LAG(p.paid_out_on) OVER ( + PARTITION BY p.provider_code, p.currency_code + ORDER BY p.paid_out_on, p.payout_id + ), + p.paid_out_on + ) AS days_since_previous_payout, + ARRAY_SIZE(COALESCE(p.lines, ARRAY_CONSTRUCT())) AS embedded_line_count + FROM payouts AS p +), + +line_totals AS ( + SELECT + l.payout_id, + SUM( + IFF(l.line_type = 'refund', -ABS(l.gross_minor), l.gross_minor) - COALESCE(l.fee_minor, 0) + ) AS lines_net_minor, + COUNT_IF(l.line_type = 'charge') AS charge_line_count, + COUNT_IF(l.line_type = 'refund') AS refund_line_count, + COUNT(*) AS loaded_line_count + FROM payout_lines AS l + GROUP BY + l.payout_id +), + +orphan_payment_ids AS ( + SELECT l.payment_id::VARCHAR AS payment_id + FROM payout_lines AS l + WHERE l.payment_id IS NOT NULL + MINUS + SELECT p.payment_id::VARCHAR AS payment_id + FROM payments AS p +), + +line_matches AS ( + SELECT + l.payout_id, + COUNT(o.payment_id) AS orphan_line_count, + SUM( + IFF(o.payment_id IS NULL AND l.line_type = 'charge', p.amount_minor, 0) + ) AS matched_payment_minor + FROM payout_lines AS l + LEFT JOIN orphan_payment_ids AS o + ON l.payment_id = o.payment_id + LEFT JOIN payments AS p + ON l.payment_id = p.payment_id + GROUP BY + l.payout_id +), + +daily_reports AS ( + SELECT + r.provider_code, + r.report_date, + r.currency_code, + r.report_id, + r.gross_minor - COALESCE(r.fees_minor, 0) AS report_net_minor + FROM reports AS r + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY r.provider_code, r.report_date, r.currency_code + ORDER BY r.line_count DESC, r.report_id DESC + ) = 1 +), + +joined AS ( + SELECT + p.payout_id, + p.provider_code, + p.currency_code, + p.paid_out_on, + p.net_minor AS payout_net_minor, + COALESCE(t.lines_net_minor, 0) AS lines_net_minor, + COALESCE(t.charge_line_count, 0) AS charge_line_count, + COALESCE(t.refund_line_count, 0) AS refund_line_count, + COALESCE(m.orphan_line_count, 0) AS orphan_line_count, + s.embedded_line_count, + COALESCE(t.loaded_line_count, 0) AS loaded_line_count, + COALESCE(m.matched_payment_minor, 0) AS matched_payment_minor, + r.report_id, + r.report_net_minor, + s.days_since_previous_payout + FROM payouts AS p + INNER JOIN payout_sequence AS s + ON p.payout_id = s.payout_id + LEFT JOIN line_totals AS t + ON p.payout_id = t.payout_id + LEFT JOIN line_matches AS m + ON p.payout_id = m.payout_id + LEFT JOIN daily_reports AS r + ON p.provider_code = r.provider_code + AND p.paid_out_on = r.report_date + AND p.currency_code = r.currency_code +), + +checked AS ( + SELECT + j.payout_id, + j.provider_code, + j.currency_code, + j.paid_out_on, + j.payout_net_minor, + j.lines_net_minor, + j.charge_line_count, + j.refund_line_count, + j.orphan_line_count, + j.embedded_line_count, + j.loaded_line_count, + j.matched_payment_minor, + j.report_id, + j.report_net_minor, + j.days_since_previous_payout, + ABS(j.payout_net_minor - j.lines_net_minor) <= @const("_tolerance_minor") AS lines_match_payout, + COALESCE( + ABS(j.payout_net_minor - j.report_net_minor) <= @const("_tolerance_minor"), + FALSE + ) AS report_matches_payout + FROM joined AS j +), + +final AS ( + SELECT + c.payout_id, + c.provider_code, + c.currency_code, + c.paid_out_on, + c.payout_net_minor::INT AS payout_net_minor, + c.lines_net_minor::INT AS lines_net_minor, + c.charge_line_count::INT AS charge_line_count, + c.refund_line_count::INT AS refund_line_count, + c.orphan_line_count::INT AS orphan_line_count, + c.embedded_line_count::INT AS embedded_line_count, + c.loaded_line_count::INT AS loaded_line_count, + c.matched_payment_minor::INT AS matched_payment_minor, + c.report_id, + c.report_net_minor::INT AS report_net_minor, + c.lines_match_payout::BOOLEAN AS lines_match_payout, + c.report_matches_payout::BOOLEAN AS report_matches_payout, + CASE + WHEN c.lines_match_payout + AND c.report_matches_payout + AND c.orphan_line_count = 0 + THEN 'reconciled' + WHEN c.report_id IS NULL + THEN 'awaiting_report' + WHEN c.orphan_line_count > 0 + THEN 'orphan_lines' + WHEN NOT c.lines_match_payout + THEN 'line_mismatch' + ELSE 'report_mismatch' + END::VARCHAR AS reconciliation_status, + c.days_since_previous_payout::INT AS days_since_previous_payout + FROM checked AS c +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/finance/int_revenue_recognition_daily.sql b/benchmarks/duckshop/project/models/intermediate/finance/int_revenue_recognition_daily.sql new file mode 100644 index 0000000000..ccd729f206 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/finance/int_revenue_recognition_daily.sql @@ -0,0 +1,172 @@ +MODEL ( + description """ + One row per recognition day, region and revenue stream. Product revenue is recognised on + delivery in the region's reporting time zone, refunds reverse it on the refund date, and + subscription revenue accrues daily. Includes month-to-date and trailing seven-day totals. + """, + columns ( + recognition_key (type VARCHAR, nullable false, audits [not_null, unique]), + recognition_date (type DATE, audits [not_null]), + region_code (type VARCHAR), + revenue_stream (type VARCHAR, audits [ + not_null, + accepted_values (values ["product", "refund", "subscription"]), + ]), + revenue_usd (type NUMBER(18, 2)), + document_count (type INT), + month_to_date_usd (type NUMBER(18, 2)), + trailing_7d_usd (type NUMBER(18, 2)), + share_of_day (type NUMBER(9, 6)), + ), +); + +WITH orders AS ( + SELECT * + FROM __ref("int_orders_usd") +), + +timelines AS ( + SELECT * + FROM __ref("int_order_status_timeline") +), + +refunds AS ( + SELECT * + FROM __ref("int_refunds_usd") +), + +subscription_days AS ( + SELECT * + FROM __ref("int_subscription_mrr_daily") +), + +customers AS ( + SELECT * + FROM __ref("stg_customers") +), + +delivered_orders AS ( + SELECT + o.order_id, + o.region_code, + o.amount_usd - COALESCE(o.tax_amount_usd, 0) AS recognisable_usd, + t.first_delivered_at + FROM orders AS o + INNER JOIN timelines AS t + ON o.order_id = t.order_id + WHERE t.first_delivered_at IS NOT NULL +), + +order_regions AS ( + SELECT + o.order_id, + o.region_code + FROM orders AS o +), + +subscription_regions AS ( + SELECT + d.subscription_day_key, + d.calendar_date, + d.mrr_usd, + c.region_code + FROM subscription_days AS d + LEFT JOIN customers AS c + ON d.customer_id = c.customer_id +), + +recognition_events AS ( + SELECT + CONVERT_TIMEZONE( + COALESCE(@const("reporting_timezone_by_region")[d.region_code]::VARCHAR, 'UTC'), + d.first_delivered_at + )::DATE AS recognition_date, + d.region_code::VARCHAR AS region_code, + 'product'::VARCHAR AS revenue_stream, + d.recognisable_usd::NUMBER(18, 4) AS amount_usd, + d.order_id::VARCHAR AS document_id + FROM delivered_orders AS d + UNION ALL + SELECT + r.refunded_date::DATE AS recognition_date, + o.region_code::VARCHAR AS region_code, + 'refund'::VARCHAR AS revenue_stream, + ( + -r.amount_usd + )::NUMBER(18, 4) AS amount_usd, + r.refund_id::VARCHAR AS document_id + FROM refunds AS r + LEFT JOIN order_regions AS o + ON r.order_id = o.order_id + UNION ALL + SELECT + s.calendar_date::DATE AS recognition_date, + s.region_code::VARCHAR AS region_code, + 'subscription'::VARCHAR AS revenue_stream, + ( + s.mrr_usd * 12 / 365 + )::NUMBER(18, 4) AS amount_usd, + s.subscription_day_key::VARCHAR AS document_id + FROM subscription_regions AS s +), + +daily AS ( + SELECT + e.recognition_date, + COALESCE(e.region_code, 'UNKNOWN') AS region_code, + e.revenue_stream, + SUM(e.amount_usd) AS revenue_usd, + COUNT(DISTINCT e.document_id) AS document_count + FROM recognition_events AS e + WHERE e.recognition_date <= '@@valuation_date'::DATE + GROUP BY + e.recognition_date, + COALESCE(e.region_code, 'UNKNOWN'), + e.revenue_stream +), + +rolling AS ( + SELECT + d.recognition_date, + d.region_code, + d.revenue_stream, + d.revenue_usd, + d.document_count, + SUM(d.revenue_usd) OVER ( + PARTITION BY d.region_code, d.revenue_stream, DATE_TRUNC('month', d.recognition_date) + ORDER BY d.recognition_date + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) AS month_to_date_usd, + SUM(d.revenue_usd) OVER ( + PARTITION BY d.region_code, d.revenue_stream + ORDER BY d.recognition_date + RANGE BETWEEN INTERVAL '6 days' PRECEDING AND CURRENT ROW + ) AS trailing_7d_usd, + RATIO_TO_REPORT(d.revenue_usd) OVER ( + PARTITION BY d.recognition_date, d.region_code + ) AS share_of_day + FROM daily AS d +), + +final AS ( + SELECT + ( + TO_VARCHAR(r.recognition_date, 'YYYY-MM-DD') + || ':' + || r.region_code + || ':' + || r.revenue_stream + )::VARCHAR AS recognition_key, + r.recognition_date, + r.region_code::VARCHAR AS region_code, + r.revenue_stream, + ROUND(r.revenue_usd, 2)::NUMBER(18, 2) AS revenue_usd, + r.document_count::INT AS document_count, + ROUND(r.month_to_date_usd, 2)::NUMBER(18, 2) AS month_to_date_usd, + ROUND(r.trailing_7d_usd, 2)::NUMBER(18, 2) AS trailing_7d_usd, + ROUND(r.share_of_day, 6)::NUMBER(9, 6) AS share_of_day + FROM rolling AS r +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/finance/int_settlement_lines.sql b/benchmarks/duckshop/project/models/intermediate/finance/int_settlement_lines.sql new file mode 100644 index 0000000000..b65814f202 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/finance/int_settlement_lines.sql @@ -0,0 +1,167 @@ +MODEL ( + description """ + One row per line of each provider settlement report, flattened from the report JSON, with the + payment reference parsed from free-text descriptions when the structured field is missing, and + the line matched to a known payment or refund. + """, + columns ( + settlement_line_key (type VARCHAR, nullable false, audits [not_null, unique]), + report_id (type VARCHAR, audits [not_null]), + provider_code (type VARCHAR), + report_date (type DATE), + line_number (type INT), + line_type (type VARCHAR), + payment_reference (type VARCHAR), + reference_source (type VARCHAR), + gross_minor (type INT), + fee_minor (type INT), + net_minor (type INT), + matched_payment_id (type VARCHAR), + matched_refund_id (type VARCHAR), + is_matched (type BOOLEAN), + amount_matches (type BOOLEAN), + fee_rate (type NUMBER(9, 6)), + ), +); + +WITH reports AS ( + SELECT * + FROM __ref("stg_settlement_reports") +), + +payments AS ( + SELECT * + FROM __ref("stg_payments") +), + +refunds AS ( + SELECT * + FROM __ref("stg_refunds") +), + +raw_lines AS ( + SELECT + r.report_id, + r.provider_code, + r.report_date, + l.index AS line_index, + l.value AS line + FROM reports AS r, LATERAL FLATTEN(input => r.report:lines) AS l +), + +typed_lines AS ( + SELECT + l.report_id, + l.provider_code, + l.report_date, + l.line_index, + LOWER(COALESCE(l.line:type::VARCHAR, l.line:record_type::VARCHAR, 'unknown')) AS line_type, + NULLIF( + TRIM(l.line:reference::VARCHAR), + '' + ) AS structured_reference, + REGEXP_SUBSTR( + l.line:description::VARCHAR, + '(pay|pi|ch)_[A-Za-z0-9]{6,}' + ) AS described_reference, + COALESCE( + l.line:gross_minor::INT, + TRY_TO_NUMBER(l.line:gross::VARCHAR, 18, 2) * 100 + ) AS gross_minor, + COALESCE(l.line:fee_minor::INT, 0) AS fee_minor + FROM raw_lines AS l +), + +referenced AS ( + SELECT + t.report_id, + t.provider_code, + t.report_date, + t.line_index, + t.line_type, + COALESCE(t.structured_reference, t.described_reference) AS payment_reference, + CASE + WHEN t.structured_reference IS NOT NULL + THEN 'structured' + WHEN t.described_reference IS NOT NULL + THEN 'description' + ELSE 'none' + END AS reference_source, + t.gross_minor, + t.fee_minor + FROM typed_lines AS t +), + +payment_refs AS ( + SELECT + p.gateway_reference::VARCHAR AS payment_reference, + MIN(p.payment_id) AS payment_id, + MIN(p.amount_minor) AS payment_amount_minor + FROM payments AS p + WHERE p.gateway_reference IS NOT NULL + GROUP BY + p.gateway_reference::VARCHAR +), + +refund_refs AS ( + SELECT + f.refund_id AS refund_reference, + MIN(f.amount_minor) AS refund_amount_minor + FROM refunds AS f + GROUP BY + f.refund_id +), + +matched AS ( + SELECT + r.report_id, + r.provider_code, + r.report_date, + r.line_index, + r.line_type, + r.payment_reference, + r.reference_source, + r.gross_minor, + r.fee_minor, + p.payment_id AS matched_payment_id, + f.refund_reference AS matched_refund_id, + COALESCE(p.payment_amount_minor, f.refund_amount_minor) AS matched_amount_minor + FROM referenced AS r + LEFT JOIN payment_refs AS p + ON r.payment_reference = p.payment_reference + LEFT JOIN refund_refs AS f + ON r.payment_reference = f.refund_reference +), + +final AS ( + SELECT + ( + m.report_id || ':' || LPAD(m.line_index::VARCHAR, 5, '0') + )::VARCHAR AS settlement_line_key, + m.report_id, + m.provider_code, + m.report_date, + ( + m.line_index + 1 + )::INT AS line_number, + m.line_type::VARCHAR AS line_type, + m.payment_reference::VARCHAR AS payment_reference, + m.reference_source::VARCHAR AS reference_source, + m.gross_minor::INT AS gross_minor, + m.fee_minor::INT AS fee_minor, + ( + m.gross_minor - m.fee_minor + )::INT AS net_minor, + m.matched_payment_id::VARCHAR AS matched_payment_id, + m.matched_refund_id::VARCHAR AS matched_refund_id, + ( + m.matched_payment_id IS NOT NULL + OR m.matched_refund_id IS NOT NULL + )::BOOLEAN AS is_matched, + COALESCE(ABS(m.gross_minor) = ABS(m.matched_amount_minor), FALSE)::BOOLEAN AS amount_matches, + ROUND(m.fee_minor / NULLIF(ABS(m.gross_minor), 0), 6)::NUMBER(9, 6) AS fee_rate + FROM matched AS m +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/finance/int_tax_applied.sql b/benchmarks/duckshop/project/models/intermediate/finance/int_tax_applied.sql new file mode 100644 index 0000000000..8dc5c53beb --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/finance/int_tax_applied.sql @@ -0,0 +1,154 @@ +MODEL ( + description """ + One row per invoice tax line with the published tax rate in force for the ship-to country and + tax category on the issue date, the tax that rate implies, and the variance from the tax + charged. + """, + columns ( + invoice_tax_line_key (type VARCHAR, nullable false, audits [not_null, unique]), + invoice_id (type VARCHAR, audits [not_null]), + order_id (type VARCHAR), + tax_code (type VARCHAR), + tax_category (type VARCHAR), + ship_to_country_code (type VARCHAR), + issued_date (type DATE), + charged_rate_pct (type NUMBER(9, 4)), + published_rate_pct (type NUMBER(9, 4)), + taxable_minor (type INT), + charged_tax_minor (type INT), + expected_tax_minor (type INT), + variance_minor (type INT), + has_published_rate (type BOOLEAN), + is_rate_mismatch (type BOOLEAN), + candidate_rate_count (type INT), + ), +); + +WITH tax_lines AS ( + SELECT * + FROM __ref("stg_invoice_tax_lines") +), + +invoices AS ( + SELECT * + FROM __ref("stg_invoices") +), + +orders AS ( + SELECT * + FROM __ref("int_orders_enriched") +), + +tax_rates AS ( + SELECT * + FROM __ref("stg_tax_rates") +), + +rate_windows AS ( + SELECT + r.tax_rate_id, + r.country_code, + LOWER(r.tax_category) AS tax_category, + r.rate_pct, + r.valid_from, + COALESCE(r.valid_to, '9999-12-31'::DATE) AS valid_to_effective + FROM tax_rates AS r +), + +line_context AS ( + SELECT + l.invoice_tax_line_key, + l.invoice_id, + i.order_id, + UPPER(l.tax_code) AS tax_code, + LOWER( + COALESCE(REGEXP_SUBSTR(l.tax_code, '^[A-Za-z]+[_-]([A-Za-z]+)', 1, 1, 'e', 1), 'standard') + ) AS tax_category, + CONVERT_TIMEZONE('UTC', i.issued_at)::DATE AS issued_date, + l.tax_rate_pct AS charged_rate_pct, + l.taxable_minor, + l.tax_minor AS charged_tax_minor + FROM tax_lines AS l + INNER JOIN invoices AS i + ON l.invoice_id = i.invoice_id +), + +with_country AS ( + SELECT + c.invoice_tax_line_key, + c.invoice_id, + c.order_id, + c.tax_code, + c.tax_category, + o.ship_to_country_code, + c.issued_date, + c.charged_rate_pct, + c.taxable_minor, + c.charged_tax_minor + FROM line_context AS c + LEFT JOIN orders AS o + ON c.order_id = o.order_id +), + +candidate_rates AS ( + SELECT + w.invoice_tax_line_key, + MAX_BY(r.rate_pct, r.valid_from) AS published_rate_pct, + COUNT(r.tax_rate_id) AS candidate_rate_count + FROM with_country AS w + INNER JOIN rate_windows AS r + ON w.ship_to_country_code = r.country_code + AND w.tax_category = r.tax_category + AND w.issued_date >= r.valid_from + AND w.issued_date < r.valid_to_effective + GROUP BY + w.invoice_tax_line_key +), + +expected AS ( + SELECT + w.invoice_tax_line_key, + w.invoice_id, + w.order_id, + w.tax_code, + w.tax_category, + w.ship_to_country_code, + w.issued_date, + w.charged_rate_pct, + c.published_rate_pct, + w.taxable_minor, + w.charged_tax_minor, + ROUND(w.taxable_minor * c.published_rate_pct / 100, 0) AS expected_tax_minor, + COALESCE(c.candidate_rate_count, 0) AS candidate_rate_count + FROM with_country AS w + LEFT JOIN candidate_rates AS c + ON w.invoice_tax_line_key = c.invoice_tax_line_key +), + +final AS ( + SELECT + e.invoice_tax_line_key, + e.invoice_id, + e.order_id, + e.tax_code::VARCHAR AS tax_code, + e.tax_category::VARCHAR AS tax_category, + e.ship_to_country_code, + e.issued_date, + e.charged_rate_pct, + e.published_rate_pct::NUMBER(9, 4) AS published_rate_pct, + e.taxable_minor, + e.charged_tax_minor, + e.expected_tax_minor::INT AS expected_tax_minor, + ( + e.charged_tax_minor - e.expected_tax_minor + )::INT AS variance_minor, + ( + e.published_rate_pct IS NOT NULL + )::BOOLEAN AS has_published_rate, + COALESCE(e.charged_rate_pct <> e.published_rate_pct, FALSE)::BOOLEAN AS is_rate_mismatch, + e.candidate_rate_count::INT AS candidate_rate_count + FROM expected AS e +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/fulfilment/_sqlbuild/_constants/unknown_region_code.sql b/benchmarks/duckshop/project/models/intermediate/fulfilment/_sqlbuild/_constants/unknown_region_code.sql new file mode 100644 index 0000000000..de21599cbe --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/fulfilment/_sqlbuild/_constants/unknown_region_code.sql @@ -0,0 +1 @@ +CONSTANT (name unknown_region_code, value null); diff --git a/benchmarks/duckshop/project/models/intermediate/fulfilment/_sqlbuild/_macros/python/currency_exponents.py b/benchmarks/duckshop/project/models/intermediate/fulfilment/_sqlbuild/_macros/python/currency_exponents.py new file mode 100644 index 0000000000..88e91439ef --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/fulfilment/_sqlbuild/_macros/python/currency_exponents.py @@ -0,0 +1,11 @@ +"""Currency exponent lookup that needs compile-time logic.""" + +# Minor-unit exponents per ISO currency, mirroring the currencies seed. Used where a CASE expression +# is cheaper than a join, such as inside window frames. +_EXPONENTS = {"USD": 2, "EUR": 2, "GBP": 2, "JPY": 0, "AUD": 2, "KWD": 3} + + +def minor_unit_exponent(currency_column: str) -> str: + """CASE expression giving the minor-unit exponent of a currency code column (default 2).""" + branches = " ".join(f"WHEN '{code}' THEN {exp}" for code, exp in sorted(_EXPONENTS.items())) + return f"(CASE {currency_column} {branches} ELSE 2 END)" diff --git a/benchmarks/duckshop/project/models/intermediate/fulfilment/int_carrier_webhook_events.sql b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_carrier_webhook_events.sql new file mode 100644 index 0000000000..7c431aeae5 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_carrier_webhook_events.sql @@ -0,0 +1,67 @@ +MODEL ( + description """ + One row per carrier status update matched to a shipment by tracking number, keeping the first + delivery of each carrier event id, with the receive lag. + """, + columns ( + carrier_update_key (type VARCHAR, nullable false, audits [not_null, unique]), + shipment_id (type VARCHAR, audits [not_null]), + carrier_code (type VARCHAR), + status_code (type VARCHAR), + status_at (type TIMESTAMP_TZ), + received_at (type TIMESTAMP_NTZ), + receive_lag_minutes (type INT), + is_batched (type BOOLEAN), + ), +); + +WITH webhooks AS ( + SELECT * + FROM __ref("stg_carrier_webhooks") +), + +shipments AS ( + SELECT * + FROM __ref("stg_shipments") +), + +first_delivery AS ( + SELECT + w.carrier_update_key, + w.carrier_code, + w.tracking_number, + w.status_code, + w.status_at, + w.received_at, + w.is_batched + FROM webhooks AS w + WHERE w.tracking_number IS NOT NULL + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY w.carrier_code, COALESCE(w.carrier_event_id::VARCHAR, w.carrier_update_key) + ORDER BY w.received_at, w.carrier_update_key + ) = 1 +), + +final AS ( + SELECT + f.carrier_update_key, + s.shipment_id, + f.carrier_code, + f.status_code, + f.status_at, + f.received_at, + DATEDIFF( + 'minute', + CONVERT_TIMEZONE('UTC', f.status_at)::TIMESTAMP_NTZ, + f.received_at + )::INT AS receive_lag_minutes, + f.is_batched + FROM first_delivery AS f + INNER JOIN shipments AS s + ON f.tracking_number = s.tracking_number + AND f.carrier_code = s.carrier_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/fulfilment/int_delivery_exceptions_classified.sql b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_delivery_exceptions_classified.sql new file mode 100644 index 0000000000..0301e1a970 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_delivery_exceptions_classified.sql @@ -0,0 +1,87 @@ +MODEL ( + description """ + One row per delivery exception with a responsibility class, its time to resolve and compensation + in USD + """, + columns ( + exception_id (type VARCHAR, nullable false, audits [not_null, unique]), + shipment_id (type VARCHAR, audits [not_null]), + carrier_code (type VARCHAR), + exception_code (type VARCHAR), + responsibility (type VARCHAR, audits [ + accepted_values (values ["carrier", "customer", "warehouse", "unknown"]), + ]), + occurred_at (type TIMESTAMP_TZ), + hours_to_resolve (type NUMBER(10, 2)), + resolution_type (type VARCHAR), + compensation_usd (type NUMBER(18, 2)), + exception_rank_in_shipment (type INT), + ), +); + +WITH exceptions AS ( + SELECT * + FROM __ref("stg_delivery_exceptions") +), + +shipments AS ( + SELECT * + FROM __ref("stg_shipments") +), + +orders AS ( + SELECT * + FROM __ref("int_orders_usd") +), + +classified AS ( + SELECT + e.exception_id, + e.shipment_id, + e.exception_code, + e.occurred_at, + e.resolved_at, + e.resolution_type, + e.compensation_minor, + CASE + WHEN REGEXP_LIKE(e.exception_code, '^(ADDR|NOT_HOME|REFUSED|ACCESS).*') + THEN 'customer' + WHEN REGEXP_LIKE(e.exception_code, '^(LOST|DAMAGED|DELAY|MISROUTE).*') + THEN 'carrier' + WHEN REGEXP_LIKE(e.exception_code, '^(LABEL|PACK|WRONG_ITEM).*') + THEN 'warehouse' + ELSE 'unknown' + END AS responsibility + FROM exceptions AS e +), + +final AS ( + SELECT + c.exception_id, + c.shipment_id, + s.carrier_code, + c.exception_code, + c.responsibility::VARCHAR AS responsibility, + c.occurred_at, + ROUND( + DATEDIFF('minute', c.occurred_at, c.resolved_at) / 60, + 2 + )::NUMBER(10, 2) AS hours_to_resolve, + c.resolution_type, + ROUND( + c.compensation_minor / POWER(10, @py.minor_unit_exponent("o.currency_code")) * o.rate_to_usd, + 2 + )::NUMBER(18, 2) AS compensation_usd, + RANK() OVER ( + PARTITION BY c.shipment_id + ORDER BY c.occurred_at + )::INT AS exception_rank_in_shipment + FROM classified AS c + INNER JOIN shipments AS s + ON c.shipment_id = s.shipment_id + LEFT JOIN orders AS o + ON s.order_id = o.order_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/fulfilment/int_delivery_slot_utilisation.sql b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_delivery_slot_utilisation.sql new file mode 100644 index 0000000000..a78bb34b5e --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_delivery_slot_utilisation.sql @@ -0,0 +1,122 @@ +MODEL ( + description """ + One row per warehouse and local day with delivery slot capacity, bookings, utilisation and the + busiest slot + """, + columns ( + slot_day_key (type VARCHAR, nullable false, audits [not_null, unique]), + warehouse_code (type VARCHAR, audits [not_null]), + slot_date_local (type DATE), + slot_count (type INT), + total_capacity (type INT), + total_booked (type INT), + utilisation (type NUMBER(9, 4)), + full_slot_count (type INT), + busiest_slot_start_local (type TIMESTAMP_NTZ), + is_weekend (type BOOLEAN), + region_code (type VARCHAR), + is_holiday (type BOOLEAN), + utilisation_vs_weekday_avg (type NUMBER(9, 4)), + ), +); + +WITH slots AS ( + SELECT * + FROM __ref("stg_delivery_slots") +), + +warehouses AS ( + SELECT * + FROM __ref("stg_warehouses") +), + +countries AS ( + SELECT * + FROM __seed("countries") +), + +holidays AS ( + SELECT * + FROM __seed("holiday_calendar") +), + +warehouse_regions AS ( + SELECT + w.warehouse_code, + c.region_code + FROM warehouses AS w + INNER JOIN countries AS c + ON w.country_code = c.country_code +), + +by_day AS ( + SELECT + s.warehouse_code, + s.slot_start_local::DATE AS slot_date_local, + COUNT(*) AS slot_count, + SUM(s.capacity) AS total_capacity, + SUM(s.booked) AS total_booked, + COUNT_IF(s.remaining_capacity = 0) AS full_slot_count, + MAX_BY(s.slot_start_local, s.booked / NULLIF(s.capacity, 0)) AS busiest_slot_start_local + FROM slots AS s + GROUP BY + s.warehouse_code, + s.slot_start_local::DATE +), + +with_calendar AS ( + SELECT + d.warehouse_code, + d.slot_date_local, + d.slot_count, + d.total_capacity, + d.total_booked, + d.full_slot_count, + d.busiest_slot_start_local, + r.region_code, + ( + h.holiday_date IS NOT NULL + ) AS is_holiday, + d.total_booked / NULLIF(d.total_capacity, 0) AS utilisation_raw + FROM by_day AS d + LEFT JOIN warehouse_regions AS r + ON d.warehouse_code = r.warehouse_code + LEFT JOIN holidays AS h + ON r.region_code = h.region_code + AND d.slot_date_local = h.holiday_date +), + +final AS ( + SELECT + ( + d.warehouse_code || '|' || TO_CHAR(d.slot_date_local, 'YYYY-MM-DD') + )::VARCHAR AS slot_day_key, + d.warehouse_code, + d.slot_date_local, + d.slot_count::INT AS slot_count, + d.total_capacity::INT AS total_capacity, + d.total_booked::INT AS total_booked, + ROUND(d.utilisation_raw, 4)::NUMBER(9, 4) AS utilisation, + d.full_slot_count::INT AS full_slot_count, + d.busiest_slot_start_local::TIMESTAMP_NTZ AS busiest_slot_start_local, + ( + DAYOFWEEKISO(d.slot_date_local) >= 6 + ) AS is_weekend, + IFF( + @sql.in_regions(d.region_code, @const("supported_regions")), + d.region_code, + @const("unknown_region_code") + )::VARCHAR AS region_code, + d.is_holiday::BOOLEAN AS is_holiday, + ROUND( + d.utilisation_raw + - AVG(d.utilisation_raw) OVER ( + PARTITION BY d.warehouse_code, DAYOFWEEKISO(d.slot_date_local) + ), + 4 + )::NUMBER(9, 4) AS utilisation_vs_weekday_avg + FROM with_calendar AS d +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/fulfilment/int_fulfilment_sla.sql b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_fulfilment_sla.sql new file mode 100644 index 0000000000..9e310205d0 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_fulfilment_sla.sql @@ -0,0 +1,149 @@ +MODEL ( + description """ + One row per shipped order with dispatch time measured in the shipping warehouse's business hours + (opening hours, weekends and regional holidays excluded) against a one-business-day dispatch + promise, and delivery time from the tracking feed. + """, + constants ( + _dispatch_promise_business_minutes 480, + ), + columns ( + shipment_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR), + carrier_code (type VARCHAR), + placed_at_local (type TIMESTAMP_NTZ), + shipped_at_local (type TIMESTAMP_NTZ), + calendar_days_to_ship (type INT), + business_days_to_ship (type INT), + holidays_in_window (type INT), + dispatch_business_minutes (type INT), + is_dispatch_on_time (type BOOLEAN), + delivered_at_utc (type TIMESTAMP_NTZ), + hours_to_deliver (type NUMBER(10, 2)), + ), +); + +WITH shipments AS ( + SELECT * + FROM __ref("stg_shipments") +), + +orders AS ( + SELECT * + FROM __ref("int_orders_enriched") +), + +warehouses AS ( + SELECT * + FROM __ref("stg_warehouses") +), + +tracking AS ( + SELECT * + FROM __ref("int_shipment_tracking_events") +), + +countries AS ( + SELECT * + FROM __seed("countries") +), + +holidays AS ( + SELECT * + FROM __seed("holiday_calendar") +), + +@sql.date_spine(@const("history_start_date"), 730) + +local_times AS ( + SELECT + s.shipment_id, + s.order_id, + s.warehouse_code, + s.carrier_code, + c.region_code, + w.opening_minutes, + CONVERT_TIMEZONE(w.iana_timezone, o.placed_at)::TIMESTAMP_NTZ AS placed_at_local, + CONVERT_TIMEZONE(w.iana_timezone, s.shipped_at)::TIMESTAMP_NTZ AS shipped_at_local, + CONVERT_TIMEZONE(w.iana_timezone, o.placed_at)::DATE AS placed_date_local, + CONVERT_TIMEZONE(w.iana_timezone, s.shipped_at)::DATE AS shipped_date_local + FROM shipments AS s + INNER JOIN orders AS o + ON s.order_id = o.order_id + INNER JOIN warehouses AS w + ON s.warehouse_code = w.warehouse_code + LEFT JOIN countries AS c + ON w.country_code = c.country_code + WHERE s.shipped_at IS NOT NULL +), window_days AS ( + SELECT + l.shipment_id, + DAYOFWEEKISO(d.calendar_date) AS iso_weekday, + h.holiday_date + FROM local_times AS l + INNER JOIN date_spine AS d + ON d.calendar_date > l.placed_date_local + AND d.calendar_date <= l.shipped_date_local + LEFT JOIN holidays AS h + ON l.region_code = h.region_code + AND d.calendar_date = h.holiday_date +), business_days AS ( + SELECT + wd.shipment_id, + COUNT_IF(wd.iso_weekday <= 5 AND wd.holiday_date IS NULL) AS business_days_to_ship, + COUNT_IF(wd.holiday_date IS NOT NULL) AS holidays_in_window + FROM window_days AS wd + GROUP BY + wd.shipment_id +), delivery AS ( + SELECT + t.shipment_id, + MIN(t.event_at_utc) AS delivered_at_utc + FROM tracking AS t + WHERE t.is_delivery_event + GROUP BY + t.shipment_id +), final AS ( + SELECT + l.shipment_id, + l.order_id, + l.warehouse_code, + l.carrier_code, + l.placed_at_local, + l.shipped_at_local, + DATEDIFF('day', l.placed_at_local, l.shipped_at_local)::INT AS calendar_days_to_ship, + COALESCE(b.business_days_to_ship, 0)::INT AS business_days_to_ship, + COALESCE(b.holidays_in_window, 0)::INT AS holidays_in_window, + IFF( + COALESCE(b.business_days_to_ship, 0) = 0, + DATEDIFF('minute', l.placed_at_local, l.shipped_at_local), + COALESCE(b.business_days_to_ship, 0) * l.opening_minutes + )::INT AS dispatch_business_minutes, + ( + IFF( + COALESCE(b.business_days_to_ship, 0) = 0, + DATEDIFF('minute', l.placed_at_local, l.shipped_at_local), + COALESCE(b.business_days_to_ship, 0) * l.opening_minutes + ) <= @const("_dispatch_promise_business_minutes") + ) AS is_dispatch_on_time, + dl.delivered_at_utc, + ROUND( + DATEDIFF( + 'minute', + CONVERT_TIMEZONE(w.iana_timezone, 'UTC', l.shipped_at_local), + dl.delivered_at_utc + ) + / 60, + 2 + )::NUMBER(10, 2) AS hours_to_deliver + FROM local_times AS l + INNER JOIN warehouses AS w + ON l.warehouse_code = w.warehouse_code + LEFT JOIN business_days AS b + ON l.shipment_id = b.shipment_id + LEFT JOIN delivery AS dl + ON l.shipment_id = dl.shipment_id +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/fulfilment/int_pick_performance.sql b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_pick_performance.sql new file mode 100644 index 0000000000..67aeb28b26 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_pick_performance.sql @@ -0,0 +1,130 @@ +MODEL ( + description """ + One row per picker, warehouse and local work date with pick throughput and its rank within the + warehouse that day + """, + columns ( + picker_day_key (type VARCHAR, nullable false, audits [not_null, unique]), + picker_id (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR), + work_date_local (type DATE), + task_count (type INT), + units_picked (type INT), + active_hours (type NUMBER(10, 2)), + units_per_hour (type NUMBER(10, 2)), + p90_task_seconds (type INT), + rank_in_warehouse (type INT), + first_task_hour_local (type INT), + shift_coverage (type NUMBER(9, 4)), + is_holiday (type BOOLEAN), + ), +); + +WITH pick_tasks AS ( + SELECT * + FROM __ref("stg_pick_tasks") +), + +warehouses AS ( + SELECT * + FROM __ref("stg_warehouses") +), + +countries AS ( + SELECT * + FROM __seed("countries") +), + +holidays AS ( + SELECT * + FROM __seed("holiday_calendar") +), + +warehouse_context AS ( + SELECT + w.warehouse_code, + w.opening_minutes, + c.region_code + FROM warehouses AS w + LEFT JOIN countries AS c + ON w.country_code = c.country_code +), + +completed AS ( + SELECT + t.picker_id, + t.warehouse_code, + t.started_at_local, + t.started_at_local::DATE AS work_date_local, + t.pick_seconds, + t.units_picked + FROM pick_tasks AS t + WHERE + t.completed_at_local IS NOT NULL + AND t.pick_seconds > 0 +), + +daily AS ( + SELECT + c.picker_id, + c.warehouse_code, + c.work_date_local, + COUNT(*) AS task_count, + SUM(c.units_picked) AS units_picked, + SUM(c.pick_seconds) / 3600 AS active_hours, + PERCENTILE_DISC(0.9) WITHIN GROUP (ORDER BY c.pick_seconds) AS p90_task_seconds, + MIN(HOUR(c.started_at_local)) AS first_task_hour_local + FROM completed AS c + GROUP BY + c.picker_id, + c.warehouse_code, + c.work_date_local +), + +with_context AS ( + SELECT + d.picker_id, + d.warehouse_code, + d.work_date_local, + d.task_count, + d.units_picked, + d.active_hours, + d.p90_task_seconds, + d.first_task_hour_local, + 60 * d.active_hours / NULLIF(w.opening_minutes, 0) AS shift_coverage, + ( + h.holiday_date IS NOT NULL + ) AS is_holiday + FROM daily AS d + LEFT JOIN warehouse_context AS w + ON d.warehouse_code = w.warehouse_code + LEFT JOIN holidays AS h + ON w.region_code = h.region_code + AND d.work_date_local = h.holiday_date +), + +final AS ( + SELECT + ( + d.picker_id || '|' || d.warehouse_code || '|' || TO_CHAR(d.work_date_local, 'YYYY-MM-DD') + )::VARCHAR AS picker_day_key, + d.picker_id, + d.warehouse_code, + d.work_date_local, + d.task_count::INT AS task_count, + d.units_picked::INT AS units_picked, + ROUND(d.active_hours, 2)::NUMBER(10, 2) AS active_hours, + ROUND(d.units_picked / NULLIF(d.active_hours, 0), 2)::NUMBER(10, 2) AS units_per_hour, + d.p90_task_seconds::INT AS p90_task_seconds, + DENSE_RANK() OVER ( + PARTITION BY d.warehouse_code, d.work_date_local + ORDER BY d.units_picked / NULLIF(d.active_hours, 0) DESC NULLS LAST + )::INT AS rank_in_warehouse, + d.first_task_hour_local::INT AS first_task_hour_local, + ROUND(d.shift_coverage, 4)::NUMBER(9, 4) AS shift_coverage, + d.is_holiday::BOOLEAN AS is_holiday + FROM with_context AS d +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/fulfilment/int_return_outcomes.sql b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_return_outcomes.sql new file mode 100644 index 0000000000..b035788f46 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_return_outcomes.sql @@ -0,0 +1,99 @@ +MODEL ( + description """ + One row per return with its inspection grade, the refund paid against the return's order, the + days from request to receipt and to refund, and the final outcome. + """, + columns ( + return_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR), + sku (type VARCHAR), + return_reason (type VARCHAR), + grade (type VARCHAR), + is_resellable (type BOOLEAN), + refund_usd (type NUMBER(18, 2)), + days_to_receive (type INT), + days_to_refund (type INT), + outcome (type VARCHAR, audits [ + accepted_values (values ["restocked", "written_off", "refunded_not_received", "pending"]), + ]), + defect_count (type INT), + ), +); + +WITH returns AS ( + SELECT * + FROM __ref("int_return_reasons") +), + +raw_returns AS ( + SELECT * + FROM __ref("stg_returns") +), + +inspections AS ( + SELECT * + FROM __ref("stg_return_inspections") +), + +refunds AS ( + SELECT * + FROM __ref("int_refunds_usd") +), + +latest_inspection AS ( + SELECT + i.return_id, + i.grade, + i.is_resellable, + ARRAY_SIZE(i.defect_codes) AS defect_count + FROM inspections AS i + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY i.return_id + ORDER BY i.inspected_at_local DESC, i.inspection_id DESC + ) = 1 +), + +order_refunds AS ( + SELECT + f.order_id, + SUM(f.amount_usd) AS refund_usd, + MIN(f.refunded_at) AS first_refunded_at + FROM refunds AS f + WHERE f.reason_category <> 'fraud' + GROUP BY + f.order_id +), + +final AS ( + SELECT + r.return_id, + r.order_id, + r.sku, + r.return_reason, + li.grade, + li.is_resellable, + orf.refund_usd::NUMBER(18, 2) AS refund_usd, + DATEDIFF('day', rr.requested_at, rr.received_at)::INT AS days_to_receive, + DATEDIFF('day', rr.requested_at, orf.first_refunded_at)::INT AS days_to_refund, + CASE + WHEN li.is_resellable + THEN 'restocked' + WHEN li.grade IS NOT NULL + THEN 'written_off' + WHEN orf.refund_usd IS NOT NULL + THEN 'refunded_not_received' + ELSE 'pending' + END::VARCHAR AS outcome, + COALESCE(li.defect_count, 0)::INT AS defect_count + FROM returns AS r + INNER JOIN raw_returns AS rr + ON r.return_id = rr.return_id + LEFT JOIN latest_inspection AS li + ON r.return_id = li.return_id + LEFT JOIN order_refunds AS orf + ON r.order_id = orf.order_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/fulfilment/int_return_reasons.sql b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_return_reasons.sql new file mode 100644 index 0000000000..0070067642 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_return_reasons.sql @@ -0,0 +1,125 @@ +MODEL ( + description """ + One row per return with its free-text reason classified into the return_reason enum by pattern + """, + columns ( + return_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR), + sku (type VARCHAR), + qty (type INT), + reason_text (type VARCHAR), + return_reason (type return_reason), + classified_by (type VARCHAR), + requested_at (type TIMESTAMP_TZ), + category_level_1 (type VARCHAR), + ordered_qty (type INT), + is_over_return (type BOOLEAN), + days_after_order (type INT), + ), +); + +WITH returns AS ( + SELECT * + FROM __ref("stg_returns") +), + +order_lines AS ( + SELECT * + FROM __ref("stg_order_lines") +), + +orders AS ( + SELECT * + FROM __ref("stg_orders") +), + +catalogue AS ( + SELECT * + FROM __ref("int_product_catalogue") +), + +ordered_quantities AS ( + SELECT + l.order_id, + l.sku, + SUM(l.quantity) AS ordered_qty + FROM order_lines AS l + GROUP BY + l.order_id, + l.sku +), + +normalised AS ( + SELECT + r.return_id, + r.order_id, + r.sku, + r.qty, + r.reason_text, + r.requested_at, + LOWER(REGEXP_REPLACE( + COALESCE(r.reason_text, ''), + '[^A-Za-z ]', + ' ' + )) AS reason_words + FROM returns AS r +), + +with_order AS ( + SELECT + n.return_id, + n.order_id, + n.sku, + n.qty, + n.reason_text, + n.requested_at, + n.reason_words, + c.category_level_1, + q.ordered_qty, + o.placed_at + FROM normalised AS n + LEFT JOIN catalogue AS c + ON n.sku = c.sku + LEFT JOIN ordered_quantities AS q + ON n.order_id = q.order_id + AND n.sku = q.sku + LEFT JOIN orders AS o + ON n.order_id = o.order_id +), + +final AS ( + SELECT + n.return_id, + n.order_id, + n.sku, + n.qty, + n.reason_text, + CASE + WHEN REGEXP_LIKE(n.reason_words, '.*(damag|broken|crack|dent|scratch).*') + THEN @enum("return_reason").DAMAGED + WHEN REGEXP_LIKE( + n.reason_words, + '.*(wrong item|wrong colou?r|not what i ordered|different).*' + ) + THEN @enum("return_reason").WRONG_ITEM + WHEN REGEXP_LIKE(n.reason_words, '.*(too (big|small|large)|size|fit).*') + THEN @enum("return_reason").SIZE_FIT + WHEN REGEXP_LIKE(n.reason_words, '.*(not as described|quality|looks different|photo).*') + THEN @enum("return_reason").NOT_AS_DESCRIBED + WHEN REGEXP_LIKE(n.reason_words, '.*(changed my mind|no longer|don t need|dont need).*') + THEN @enum("return_reason").CHANGED_MIND + WHEN REGEXP_LIKE(n.reason_words, '.*(late|delay|too long).*') + THEN @enum("return_reason").LATE_DELIVERY + ELSE @enum("return_reason").OTHER + END::VARCHAR AS return_reason, + IFF(TRIM(n.reason_words) = '', 'default', 'pattern')::VARCHAR AS classified_by, + n.requested_at, + n.category_level_1::VARCHAR AS category_level_1, + n.ordered_qty::INT AS ordered_qty, + COALESCE(n.qty > n.ordered_qty, TRUE)::BOOLEAN AS is_over_return, + DATEDIFF('day', n.placed_at, n.requested_at)::INT AS days_after_order + FROM with_order AS n +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/fulfilment/int_shipment_costs.sql b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_shipment_costs.sql new file mode 100644 index 0000000000..f2897746d9 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_shipment_costs.sql @@ -0,0 +1,121 @@ +MODEL ( + description """ + One row per shipment package with its chargeable weight (the greater of actual and volumetric) + and the carrier rate from the matching rate card weight band. + """, + columns ( + package_id (type VARCHAR, nullable false, audits [not_null, unique]), + shipment_id (type VARCHAR, audits [not_null]), + carrier_code (type VARCHAR), + service_level (type VARCHAR), + chargeable_weight_kg (type NUMBER(10, 3)), + weight_basis (type VARCHAR), + band_max_kg (type NUMBER(10, 3)), + rate_minor (type INT), + rate_currency_code (type VARCHAR), + has_rate (type BOOLEAN), + ), +); + +WITH packages AS ( + SELECT * + FROM __ref("stg_shipment_packages") +), + +shipments AS ( + SELECT * + FROM __ref("stg_shipments") +), + +warehouses AS ( + SELECT * + FROM __ref("stg_warehouses") +), + +orders AS ( + SELECT * + FROM __ref("int_orders_enriched") +), + +rate_cards AS ( + SELECT * + FROM __ref("stg_carrier_rate_cards") +), + +package_lanes AS ( + SELECT + p.package_id, + p.shipment_id, + s.carrier_code, + s.service_level, + w.country_code AS origin_country, + o.ship_to_country_code AS destination_country, + s.shipped_at::DATE AS shipped_date, + GREATEST(p.weight_grams / 1000, COALESCE(p.volumetric_weight_kg, 0)) AS chargeable_weight_kg, + IFF( + COALESCE(p.volumetric_weight_kg, 0) > p.weight_grams / 1000, + 'volumetric', + 'actual' + ) AS weight_basis + FROM packages AS p + INNER JOIN shipments AS s + ON p.shipment_id = s.shipment_id + INNER JOIN warehouses AS w + ON s.warehouse_code = w.warehouse_code + LEFT JOIN orders AS o + ON s.order_id = o.order_id +), + +bands AS ( + SELECT + r.carrier_code, + r.service_level, + r.origin_country, + r.destination_country, + r.currency_code, + r.valid_from, + r.valid_to, + b.value:max_kg::NUMBER(10, 3) AS band_max_kg, + b.value:amount_minor::INT AS rate_minor + FROM rate_cards AS r, LATERAL FLATTEN(input => r.bands) AS b +), + +matched AS ( + SELECT + l.package_id, + MIN(b.band_max_kg) AS band_max_kg, + MIN_BY(b.rate_minor, b.band_max_kg) AS rate_minor, + MIN_BY(b.currency_code, b.band_max_kg) AS rate_currency_code + FROM package_lanes AS l + INNER JOIN bands AS b + ON l.carrier_code = b.carrier_code + AND l.service_level = b.service_level + AND l.origin_country = b.origin_country + AND l.destination_country = b.destination_country + AND l.shipped_date BETWEEN b.valid_from AND b.valid_to + AND l.chargeable_weight_kg <= b.band_max_kg + GROUP BY + l.package_id +), + +final AS ( + SELECT + l.package_id, + l.shipment_id, + l.carrier_code, + l.service_level, + ROUND(l.chargeable_weight_kg, 3)::NUMBER(10, 3) AS chargeable_weight_kg, + l.weight_basis::VARCHAR AS weight_basis, + m.band_max_kg::NUMBER(10, 3) AS band_max_kg, + m.rate_minor::INT AS rate_minor, + m.rate_currency_code::VARCHAR AS rate_currency_code, + ( + m.rate_minor IS NOT NULL + ) AS has_rate + FROM package_lanes AS l + LEFT JOIN matched AS m + ON l.package_id = m.package_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/fulfilment/int_shipment_tracking_events.sql b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_shipment_tracking_events.sql new file mode 100644 index 0000000000..351faec78c --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/fulfilment/int_shipment_tracking_events.sql @@ -0,0 +1,110 @@ +MODEL ( + description """ + One row per tracking event of a shipment from carrier scans, recorded exceptions and webhooks, + deduplicated by code and minute, in UTC and in the shipping warehouse's local time. + """, + columns ( + tracking_event_key (type VARCHAR, nullable false, audits [not_null, unique]), + shipment_id (type VARCHAR, audits [not_null]), + event_code (type VARCHAR), + event_family (type VARCHAR), + event_at_utc (type TIMESTAMP_NTZ), + event_at_warehouse_local (type TIMESTAMP_NTZ), + event_origin (type VARCHAR), + is_delivery_event (type BOOLEAN), + ), +); + +WITH tracking_events AS ( + SELECT * + FROM __ref("stg_shipment_tracking_events") +), + +webhook_events AS ( + SELECT * + FROM __ref("int_carrier_webhook_events") +), + +shipments AS ( + SELECT * + FROM __ref("stg_shipments") +), + +warehouses AS ( + SELECT * + FROM __ref("stg_warehouses") +), + +combined AS ( + SELECT + t.tracking_event_key::VARCHAR AS tracking_event_key, + t.shipment_id::VARCHAR AS shipment_id, + t.event_code::VARCHAR AS event_code, + CONVERT_TIMEZONE('UTC', t.event_at)::TIMESTAMP_NTZ AS event_at_utc, + t.event_source::VARCHAR AS event_origin + FROM tracking_events AS t + WHERE t.event_at IS NOT NULL + UNION ALL + SELECT + ( + 'W:' || w.carrier_update_key + )::VARCHAR AS tracking_event_key, + w.shipment_id::VARCHAR AS shipment_id, + w.status_code::VARCHAR AS event_code, + CONVERT_TIMEZONE('UTC', w.status_at)::TIMESTAMP_NTZ AS event_at_utc, + 'webhook'::VARCHAR AS event_origin + FROM webhook_events AS w + WHERE w.status_at IS NOT NULL +), + +deduplicated AS ( + SELECT + c.tracking_event_key, + c.shipment_id, + c.event_code, + c.event_at_utc, + c.event_origin + FROM combined AS c + QUALIFY + COUNT(*) OVER ( + PARTITION BY c.shipment_id, c.event_code, DATE_TRUNC('minute', c.event_at_utc) + ORDER BY c.event_origin, c.tracking_event_key + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) = 1 +), + +final AS ( + SELECT + d.tracking_event_key, + d.shipment_id, + d.event_code, + CASE + WHEN d.event_code RLIKE '^(DELIVERED|DLV|POD).*' + THEN 'delivered' + WHEN d.event_code RLIKE '^(OUT_FOR_DELIVERY|OFD).*' + THEN 'out_for_delivery' + WHEN d.event_code RLIKE '^EXCEPTION_.*|^(EXC|DELAY|DAMAGE).*' + THEN 'exception' + WHEN d.event_code RLIKE '^(PICKED_UP|PU|ACCEPTED).*' + THEN 'collected' + ELSE 'in_transit' + END::VARCHAR AS event_family, + d.event_at_utc, + CONVERT_TIMEZONE( + 'UTC', + w.iana_timezone, + d.event_at_utc + )::TIMESTAMP_NTZ AS event_at_warehouse_local, + d.event_origin, + ( + d.event_code RLIKE '^(DELIVERED|DLV|POD).*' + ) AS is_delivery_event + FROM deduplicated AS d + INNER JOIN shipments AS s + ON d.shipment_id = s.shipment_id + INNER JOIN warehouses AS w + ON s.warehouse_code = w.warehouse_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/growth/_sqlbuild/_macros/sql/any_to_text.sql b/benchmarks/duckshop/project/models/intermediate/growth/_sqlbuild/_macros/sql/any_to_text.sql new file mode 100644 index 0000000000..eb8dce8a35 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/growth/_sqlbuild/_macros/sql/any_to_text.sql @@ -0,0 +1,7 @@ +MACRO ( + description "Trimmed text form of any scalar, or NULL when blank", + arguments (value ANY), + returns VARCHAR, +); + +NULLIF(TRIM(CAST(@value AS VARCHAR)), '') diff --git a/benchmarks/duckshop/project/models/intermediate/growth/int_affiliate_commissions_usd.sql b/benchmarks/duckshop/project/models/intermediate/growth/int_affiliate_commissions_usd.sql new file mode 100644 index 0000000000..3c629e238d --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/growth/int_affiliate_commissions_usd.sql @@ -0,0 +1,166 @@ +MODEL ( + description """ + One row per affiliate conversion with the commission in USD, the order it claims, whether it is + the last-click conversion that earns the commission, and whether the order was later cancelled + or returned so the commission should be reversed. + """, + columns ( + conversion_id (type VARCHAR, nullable false, audits [not_null, unique]), + affiliate_code (type VARCHAR, audits [not_null]), + affiliate_network (type VARCHAR), + sub_channel (type VARCHAR), + order_id (type VARCHAR), + converted_at (type TIMESTAMP_TZ), + click_to_conversion_hours (type INT), + commission (type NUMBER(38, 6)), + commission_usd (type NUMBER(18, 2)), + order_amount_usd (type NUMBER(18, 2)), + commission_rate (type NUMBER(9, 6)), + claims_on_order (type INT), + is_last_click (type BOOLEAN), + is_reversible (type BOOLEAN), + payable_commission_usd (type NUMBER(18, 2)), + ), +); + +WITH conversions AS ( + SELECT * + FROM __ref("stg_affiliate_conversions") +), + +orders AS ( + SELECT * + FROM __ref("int_orders_usd") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +ranked_claims AS ( + SELECT + c.conversion_id, + c.affiliate_code, + c.order_id, + c.commission_minor, + c.currency_code, + c.converted_at, + CONVERT_TIMEZONE('UTC', c.converted_at)::DATE AS converted_date, + c.click_to_conversion_hours, + c.sub_id, + ROW_NUMBER() OVER ( + PARTITION BY c.order_id + ORDER BY c.clicked_at DESC NULLS LAST, c.conversion_id + ) AS claim_rank, + COUNT(*) OVER (PARTITION BY c.order_id) AS claims_on_order + FROM conversions AS c +), + +parsed_codes AS ( + SELECT + r.conversion_id, + UPPER( + COALESCE(REGEXP_SUBSTR(r.affiliate_code, '^([A-Za-z]{2,5})[-_]', 1, 1, 'e', 1), 'DIRECT') + ) AS affiliate_network, + LOWER(REGEXP_REPLACE( + REGEXP_SUBSTR(r.sub_id, '^[a-zA-Z]+'), + '[^a-z]', + '' + )) AS sub_channel + FROM ranked_claims AS r +), + +with_exponent AS ( + SELECT + r.conversion_id, + r.affiliate_code, + r.order_id, + r.commission_minor, + r.currency_code, + r.converted_at, + r.converted_date, + r.click_to_conversion_hours, + r.claim_rank, + r.claims_on_order, + cur.minor_unit_exponent + FROM ranked_claims AS r + LEFT JOIN currencies AS cur + ON r.currency_code = cur.currency_code +), + +with_rate AS ( + SELECT + w.conversion_id, + w.affiliate_code, + w.order_id, + w.commission_minor, + w.converted_at, + w.click_to_conversion_hours, + w.claim_rank, + w.claims_on_order, + w.minor_unit_exponent, + fx.rate_to_usd + FROM with_exponent AS w + ASOF JOIN fx_rates AS fx MATCH_CONDITION (w.converted_date >= fx.rate_date) + ON w.currency_code = fx.currency_code +), + +with_order AS ( + SELECT + w.conversion_id, + w.affiliate_code, + p.affiliate_network, + p.sub_channel, + w.order_id, + w.converted_at, + w.click_to_conversion_hours, + @sql.money_from_minor(w.commission_minor, COALESCE(w.minor_unit_exponent, 2), places := 6) AS commission, + w.rate_to_usd, + w.claim_rank, + w.claims_on_order, + o.amount_usd AS order_amount_usd, + o.order_status + FROM with_rate AS w + INNER JOIN parsed_codes AS p + ON w.conversion_id = p.conversion_id + LEFT JOIN orders AS o + ON w.order_id = o.order_id +), + +final AS ( + SELECT + o.conversion_id, + o.affiliate_code, + o.affiliate_network::VARCHAR AS affiliate_network, + NULLIF(o.sub_channel, '')::VARCHAR AS sub_channel, + o.order_id, + o.converted_at, + o.click_to_conversion_hours, + o.commission::NUMBER(38, 6) AS commission, + ROUND(o.commission * o.rate_to_usd, 2)::NUMBER(18, 2) AS commission_usd, + o.order_amount_usd, + ROUND( + o.commission * o.rate_to_usd / NULLIF(o.order_amount_usd, 0), + 6 + )::NUMBER(9, 6) AS commission_rate, + o.claims_on_order::INT AS claims_on_order, + ( + o.claim_rank = 1 + )::BOOLEAN AS is_last_click, + COALESCE(o.order_status IN ('CANCELLED', 'RETURNED'), FALSE)::BOOLEAN AS is_reversible, + IFF( + o.claim_rank = 1 AND COALESCE(o.order_status NOT IN ('CANCELLED', 'RETURNED'), FALSE), + ROUND(o.commission * o.rate_to_usd, 2), + 0 + )::NUMBER(18, 2) AS payable_commission_usd + FROM with_order AS o +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/growth/int_campaign_performance.sql b/benchmarks/duckshop/project/models/intermediate/growth/int_campaign_performance.sql new file mode 100644 index 0000000000..dc4590fc7a --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/growth/int_campaign_performance.sql @@ -0,0 +1,227 @@ +MODEL ( + description """ + One row per campaign with USD spend, paid and email touches, attributed sessions and orders, and + return on ad spend. Orders and sessions are attributed when their utm_campaign matches the + campaign identifier. + """, + columns ( + campaign_id (type VARCHAR, nullable false, audits [not_null, unique]), + campaign_name (type VARCHAR), + channel (type VARCHAR), + objective (type VARCHAR), + starts_on (type DATE), + ends_on (type DATE), + budget_usd (type NUMBER(18, 2)), + spend_usd (type NUMBER(18, 2)), + budget_used_pct (type NUMBER(9, 2)), + paid_clicks (type INT), + email_clicks (type INT), + attributed_sessions (type INT), + attributed_orders (type INT), + attributed_revenue_usd (type NUMBER(18, 2)), + return_on_ad_spend (type NUMBER(12, 4)), + target_region_count (type INT), + unsupported_target_regions (type ARRAY), + spend_rank_in_channel (type INT), + ), +); + +WITH campaigns AS ( + SELECT * + FROM __ref("stg_campaigns") +), + +spend AS ( + SELECT * + FROM __ref("int_campaign_spend_usd") +), + +sessions AS ( + SELECT * + FROM __ref("int_session_attribution") +), + +orders AS ( + SELECT * + FROM __ref("int_orders_usd") +), + +emails AS ( + SELECT * + FROM __ref("int_email_engagement") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +campaign_keys AS ( + SELECT + c.campaign_id, + LOWER(c.campaign_id) AS utm_campaign_key, + c.campaign_name, + c.channel, + c.objective, + c.starts_on, + c.ends_on, + c.budget_minor, + c.currency_code, + c.target_regions + FROM campaigns AS c +), + +budget_rate AS ( + SELECT + k.campaign_id, + ROUND( + @sql.money_from_minor(k.budget_minor, cur.minor_unit_exponent, places := 6) * fx.rate_to_usd, + 2 + ) AS budget_usd + FROM campaign_keys AS k + INNER JOIN currencies AS cur + ON k.currency_code = cur.currency_code + ASOF JOIN fx_rates AS fx MATCH_CONDITION (k.starts_on >= fx.rate_date) + ON k.currency_code = fx.currency_code +), + +target_regions AS ( + SELECT + k.campaign_id, + COUNT(r.value) AS target_region_count, + ARRAY_AGG( + IFF(@sql.in_regions(r.value::VARCHAR, @const("supported_regions")), NULL, r.value::VARCHAR) + ) WITHIN GROUP (ORDER BY r.value::VARCHAR) AS unsupported_target_regions + FROM campaign_keys AS k, LATERAL FLATTEN(input => k.target_regions) AS r + GROUP BY + k.campaign_id +), + +touches AS ( + SELECT + s.campaign_id::VARCHAR AS campaign_id, + 'paid'::VARCHAR AS touch_type, + COALESCE(s.clicks, 0)::INT AS touch_count + FROM spend AS s + WHERE s.campaign_id IS NOT NULL + UNION ALL + SELECT + e.campaign_id::VARCHAR AS campaign_id, + 'email'::VARCHAR AS touch_type, + e.click_count::INT AS touch_count + FROM emails AS e + WHERE + e.campaign_id IS NOT NULL + AND e.click_count > 0 +), + +touch_totals AS ( + SELECT + t.campaign_id, + SUM(IFF(t.touch_type = 'paid', t.touch_count, 0)) AS paid_clicks, + SUM(IFF(t.touch_type = 'email', t.touch_count, 0)) AS email_clicks + FROM touches AS t + GROUP BY + t.campaign_id +), + +spend_totals AS ( + SELECT + s.campaign_id, + SUM(s.spend_usd) AS spend_usd + FROM spend AS s + GROUP BY + s.campaign_id +), + +session_totals AS ( + SELECT + s.utm_campaign, + COUNT(*) AS attributed_sessions + FROM sessions AS s + WHERE s.utm_campaign IS NOT NULL + GROUP BY + s.utm_campaign +), + +order_totals AS ( + SELECT + o.utm_campaign, + COUNT(*) AS attributed_orders, + SUM(o.amount_usd) AS attributed_revenue_usd + FROM orders AS o + WHERE + o.utm_campaign IS NOT NULL + AND o.order_status NOT IN ('CANCELLED', 'PENDING') + GROUP BY + o.utm_campaign +), + +combined AS ( + SELECT + k.campaign_id, + k.campaign_name, + k.channel, + k.objective, + k.starts_on, + k.ends_on, + b.budget_usd, + COALESCE(sp.spend_usd, 0) AS spend_usd, + COALESCE(t.paid_clicks, 0) AS paid_clicks, + COALESCE(t.email_clicks, 0) AS email_clicks, + COALESCE(st.attributed_sessions, 0) AS attributed_sessions, + COALESCE(ot.attributed_orders, 0) AS attributed_orders, + COALESCE(ot.attributed_revenue_usd, 0) AS attributed_revenue_usd, + COALESCE(tr.target_region_count, 0) AS target_region_count, + COALESCE(tr.unsupported_target_regions, ARRAY_CONSTRUCT()) AS unsupported_target_regions + FROM campaign_keys AS k + LEFT JOIN budget_rate AS b + ON k.campaign_id = b.campaign_id + LEFT JOIN spend_totals AS sp + ON k.campaign_id = sp.campaign_id + LEFT JOIN touch_totals AS t + ON k.campaign_id = t.campaign_id + LEFT JOIN session_totals AS st + ON k.utm_campaign_key = st.utm_campaign + LEFT JOIN order_totals AS ot + ON k.utm_campaign_key = ot.utm_campaign + LEFT JOIN target_regions AS tr + ON k.campaign_id = tr.campaign_id +), + +final AS ( + SELECT + c.campaign_id, + c.campaign_name, + c.channel, + c.objective, + c.starts_on, + c.ends_on, + c.budget_usd::NUMBER(18, 2) AS budget_usd, + c.spend_usd::NUMBER(18, 2) AS spend_usd, + ROUND(100 * c.spend_usd / NULLIF(c.budget_usd, 0), 2)::NUMBER(9, 2) AS budget_used_pct, + c.paid_clicks::INT AS paid_clicks, + c.email_clicks::INT AS email_clicks, + c.attributed_sessions::INT AS attributed_sessions, + c.attributed_orders::INT AS attributed_orders, + c.attributed_revenue_usd::NUMBER(18, 2) AS attributed_revenue_usd, + ROUND( + c.attributed_revenue_usd / NULLIF(c.spend_usd, 0), + 4 + )::NUMBER(12, 4) AS return_on_ad_spend, + c.target_region_count::INT AS target_region_count, + c.unsupported_target_regions::ARRAY AS unsupported_target_regions, + DENSE_RANK() OVER ( + PARTITION BY c.channel + ORDER BY c.spend_usd DESC + )::INT AS spend_rank_in_channel + FROM combined AS c +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/growth/int_campaign_spend_usd.sql b/benchmarks/duckshop/project/models/intermediate/growth/int_campaign_spend_usd.sql new file mode 100644 index 0000000000..8ad329ecfa --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/growth/int_campaign_spend_usd.sql @@ -0,0 +1,176 @@ +MODEL ( + description """ + One row per ad platform spend report with its long-format metrics pivoted to columns, the spend + converted to USD at the reference rate on the spend date, and whether the campaign was in + flight. + """, + columns ( + spend_id (type VARCHAR, nullable false, audits [not_null, unique]), + campaign_id (type VARCHAR), + campaign_name (type VARCHAR), + channel (type VARCHAR, audits [not_null]), + platform_account_id (type VARCHAR), + spend_date (type DATE, audits [not_null]), + currency_code (type VARCHAR), + spend_amount (type NUMBER(38, 6)), + spend_usd (type NUMBER(18, 2)), + impressions (type INT), + clicks (type INT), + conversions (type INT), + click_through_rate (type NUMBER(9, 6)), + cost_per_click_usd (type NUMBER(18, 4)), + metric_count (type INT), + unknown_metrics (type ARRAY), + is_in_flight (type BOOLEAN), + ), +); + +WITH spend AS ( + SELECT * + FROM __ref("stg_campaign_spend") +), + +campaigns AS ( + SELECT * + FROM __ref("stg_campaigns") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +report_headers AS ( + SELECT + s.spend_id, + s.spend_date, + s.channel, + s.campaign_id, + s.currency_code, + s.platform_account_id + FROM spend AS s + QUALIFY ROW_NUMBER() OVER (PARTITION BY s.spend_id ORDER BY s.spend_metric_key) = 1 +), + +metric_values AS ( + SELECT + s.spend_id, + MAX(IFF(s.metric_name = 'spend', s.metric_value, NULL)) AS spend_amount, + MAX(IFF(s.metric_name = 'impressions', s.metric_value, NULL)) AS impressions, + MAX(IFF(s.metric_name IN ('clicks', 'link_clicks'), s.metric_value, NULL)) AS clicks, + MAX( + IFF(REGEXP_LIKE(s.metric_name, '^(conversions|purchases)(_[0-9]+d)?$'), s.metric_value, NULL) + ) AS conversions, + COUNT(*) AS metric_count, + ARRAY_AGG( + DISTINCT IFF( + REGEXP_LIKE( + s.metric_name, + '^(spend|impressions|clicks|link_clicks|conversions|purchases)(_[0-9]+d)?$' + ), + NULL, + s.metric_name + ) + ) WITHIN GROUP ( + ORDER BY IFF( + REGEXP_LIKE( + s.metric_name, + '^(spend|impressions|clicks|link_clicks|conversions|purchases)(_[0-9]+d)?$' + ), + NULL, + s.metric_name + ) + ) AS unknown_metrics + FROM spend AS s + GROUP BY + s.spend_id +), + +reports AS ( + SELECT + h.spend_id, + h.spend_date, + h.channel, + h.campaign_id, + h.currency_code, + h.platform_account_id, + m.spend_amount, + m.impressions, + m.clicks, + m.conversions, + m.metric_count, + m.unknown_metrics + FROM report_headers AS h + INNER JOIN metric_values AS m + ON h.spend_id = m.spend_id +), + +with_rate AS ( + SELECT + r.spend_id, + r.spend_date, + r.channel, + r.campaign_id, + r.currency_code, + r.platform_account_id, + r.spend_amount, + r.impressions, + r.clicks, + r.conversions, + r.metric_count, + r.unknown_metrics, + fx.rate_to_usd + FROM reports AS r + ASOF JOIN fx_rates AS fx MATCH_CONDITION (r.spend_date >= fx.rate_date) + ON r.currency_code = fx.currency_code +), + +with_campaign AS ( + SELECT + w.spend_id, + w.campaign_id, + c.campaign_name, + COALESCE(w.channel, c.channel) AS channel, + w.platform_account_id, + w.spend_date, + w.currency_code, + w.spend_amount, + w.spend_amount * w.rate_to_usd AS spend_usd_unrounded, + w.impressions, + w.clicks, + w.conversions, + w.metric_count, + w.unknown_metrics, + COALESCE( + w.spend_date BETWEEN c.starts_on AND COALESCE(c.ends_on, '@@valuation_date'::DATE), + FALSE + ) AS is_in_flight + FROM with_rate AS w + LEFT JOIN campaigns AS c + ON w.campaign_id = c.campaign_id +), + +final AS ( + SELECT + c.spend_id, + c.campaign_id, + c.campaign_name::VARCHAR AS campaign_name, + c.channel::VARCHAR AS channel, + c.platform_account_id, + c.spend_date, + c.currency_code, + c.spend_amount::NUMBER(38, 6) AS spend_amount, + @py.round_half_even("c.spend_usd_unrounded", 2)::NUMBER(18, 2) AS spend_usd, + c.impressions::INT AS impressions, + c.clicks::INT AS clicks, + c.conversions::INT AS conversions, + ROUND(c.clicks / NULLIF(c.impressions, 0), 6)::NUMBER(9, 6) AS click_through_rate, + ROUND(c.spend_usd_unrounded / NULLIF(c.clicks, 0), 4)::NUMBER(18, 4) AS cost_per_click_usd, + c.metric_count::INT AS metric_count, + c.unknown_metrics::ARRAY AS unknown_metrics, + c.is_in_flight::BOOLEAN AS is_in_flight + FROM with_campaign AS c +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/growth/int_influencer_promo_orders.sql b/benchmarks/duckshop/project/models/intermediate/growth/int_influencer_promo_orders.sql new file mode 100644 index 0000000000..50e9799db2 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/growth/int_influencer_promo_orders.sql @@ -0,0 +1,140 @@ +MODEL ( + description """ + One row per creator post with the orders that used its promo code. When creators reuse a code, + each order is credited to the latest post published before it, and orders more than the + attribution window after that post are not credited. + """, + constants ( + _attribution_window_days 14, + ), + columns ( + post_id (type VARCHAR, nullable false, audits [not_null, unique]), + creator_handle (type VARCHAR), + platform (type VARCHAR), + posted_at (type TIMESTAMP_TZ), + promo_code (type VARCHAR), + impressions (type INT), + engagements (type INT), + saves (type INT), + has_tracking_link (type BOOLEAN), + credited_orders (type INT), + credited_customers (type INT), + credited_revenue_usd (type NUMBER(18, 2)), + discount_given_usd (type NUMBER(18, 2)), + first_order_hours_after_post (type NUMBER(10, 2)), + orders_per_thousand_impressions (type NUMBER(12, 4)), + ), +); + +WITH posts AS ( + SELECT * + FROM __ref("stg_influencer_posts") +), + +orders AS ( + SELECT * + FROM __ref("int_orders_usd") +), + +post_codes AS ( + SELECT + p.post_id, + LOWER(REGEXP_REPLACE(p.creator_handle, '^@+|\\s+', '')) AS creator_handle, + LOWER(p.platform) AS platform, + p.posted_at, + UPPER(p.promo_code) AS promo_code, + p.impressions, + COALESCE(p.likes, 0) + + COALESCE(p.metrics:comments::INT, 0) + + COALESCE(p.metrics:shares::INT, 0) AS engagements, + p.metrics:saves::INT AS saves, + REGEXP_LIKE( + COALESCE(p.metrics:link_url::VARCHAR, ''), + '^https?://[^ ]+[?&]utm_source=.*' + ) AS has_tracking_link + FROM posts AS p + WHERE p.promo_code IS NOT NULL +), + +promo_orders AS ( + SELECT + o.order_id, + o.customer_id, + UPPER(o.promo_code) AS promo_code, + o.placed_at, + o.amount_usd, + o.discount_amount_usd + FROM orders AS o + WHERE + o.promo_code IS NOT NULL + AND o.order_status <> 'CANCELLED' +), + +credited AS ( + SELECT + o.order_id, + o.customer_id, + o.placed_at, + o.amount_usd, + o.discount_amount_usd, + p.post_id, + p.posted_at + FROM promo_orders AS o + ASOF JOIN post_codes AS p MATCH_CONDITION (o.placed_at >= p.posted_at) + ON o.promo_code = p.promo_code +), + +in_window AS ( + SELECT + c.post_id, + c.order_id, + c.customer_id, + c.amount_usd, + c.discount_amount_usd, + DATEDIFF('minute', c.posted_at, c.placed_at) / 60 AS hours_after_post + FROM credited AS c + WHERE + c.post_id IS NOT NULL + AND DATEDIFF('day', c.posted_at, c.placed_at) <= @const("_attribution_window_days") +), + +per_post AS ( + SELECT + w.post_id, + COUNT(DISTINCT w.order_id) AS credited_orders, + COUNT(DISTINCT w.customer_id) AS credited_customers, + SUM(w.amount_usd) AS credited_revenue_usd, + SUM(w.discount_amount_usd) AS discount_given_usd, + MIN(w.hours_after_post) AS first_order_hours_after_post + FROM in_window AS w + GROUP BY + w.post_id +), + +final AS ( + SELECT + p.post_id, + p.creator_handle::VARCHAR AS creator_handle, + p.platform::VARCHAR AS platform, + p.posted_at, + p.promo_code::VARCHAR AS promo_code, + p.impressions, + p.engagements::INT AS engagements, + p.saves::INT AS saves, + p.has_tracking_link::BOOLEAN AS has_tracking_link, + COALESCE(s.credited_orders, 0)::INT AS credited_orders, + COALESCE(s.credited_customers, 0)::INT AS credited_customers, + COALESCE(s.credited_revenue_usd, 0)::NUMBER(18, 2) AS credited_revenue_usd, + COALESCE(s.discount_given_usd, 0)::NUMBER(18, 2) AS discount_given_usd, + ROUND(s.first_order_hours_after_post, 2)::NUMBER(10, 2) AS first_order_hours_after_post, + ROUND( + 1000 * COALESCE(s.credited_orders, 0) / NULLIF(p.impressions, 0), + 4 + )::NUMBER(12, 4) AS orders_per_thousand_impressions + FROM post_codes AS p + LEFT JOIN per_post AS s + ON p.post_id = s.post_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/growth/int_newsletter_status.sql b/benchmarks/duckshop/project/models/intermediate/growth/int_newsletter_status.sql new file mode 100644 index 0000000000..ef0f6428ff --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/growth/int_newsletter_status.sql @@ -0,0 +1,132 @@ +MODEL ( + description """ + One row per subscriber email with the latest status on each newsletter list, the lists the + address currently receives, and whether the address belongs to a customer account or is a + prospect who has never ordered. + """, + columns ( + email (type VARCHAR, nullable false, audits [not_null, unique]), + email_domain (type VARCHAR), + is_role_address (type BOOLEAN), + customer_id (type VARCHAR), + is_prospect (type BOOLEAN), + list_status_by_list (type OBJECT), + subscribed_lists (type ARRAY), + subscribed_list_count (type INT), + first_subscribed_at (type TIMESTAMP_TZ), + last_unsubscribed_at (type TIMESTAMP_TZ), + is_globally_unsubscribed (type BOOLEAN), + has_double_opt_in (type BOOLEAN), + first_signup_form (type VARCHAR), + ), +); + +WITH subscriptions AS ( + SELECT * + FROM __ref("stg_newsletter_subscriptions") +), + +customers AS ( + SELECT * + FROM __ref("stg_customers") +), + +latest_per_list AS ( + SELECT + s.email, + s.list_code, + s.list_status, + s.subscribed_at, + s.unsubscribed_at, + s.is_double_opt_in + FROM subscriptions AS s + WHERE s.email IS NOT NULL + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY s.email, s.list_code + ORDER BY COALESCE( + s.unsubscribed_at, + s.subscribed_at + ) DESC NULLS LAST, s.list_subscription_id DESC + ) = 1 +), + +first_signup AS ( + SELECT + s.email, + s.signup_form AS first_signup_form + FROM subscriptions AS s + WHERE s.email IS NOT NULL + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY s.email + ORDER BY s.subscribed_at NULLS LAST, s.list_subscription_id + ) = 1 +), + +by_email AS ( + SELECT + l.email, + OBJECT_AGG(l.list_code, l.list_status::VARIANT) AS list_status_by_list, + ARRAY_AGG( + IFF(l.list_status = 'subscribed', l.list_code, NULL) + ) WITHIN GROUP (ORDER BY l.list_code) AS subscribed_lists, + COUNT_IF(l.list_status = 'subscribed') AS subscribed_list_count, + MIN(l.subscribed_at) AS first_subscribed_at, + MAX(l.unsubscribed_at) AS last_unsubscribed_at, + BOOLAND_AGG(l.list_status IN ('unsubscribed', 'cleaned')) AS is_globally_unsubscribed, + BOOLOR_AGG(COALESCE(l.is_double_opt_in, FALSE)) AS has_double_opt_in + FROM latest_per_list AS l + GROUP BY + l.email +), + +customer_emails AS ( + SELECT + c.email_normalised AS email, + MIN(c.customer_id) AS customer_id + FROM customers AS c + WHERE c.email_normalised IS NOT NULL + GROUP BY + c.email_normalised +), + +prospect_emails AS ( + SELECT b.email::VARCHAR AS email + FROM by_email AS b + MINUS + SELECT c.email::VARCHAR AS email + FROM customer_emails AS c +), + +final AS ( + SELECT + b.email, + SPLIT_PART(b.email, '@', 2)::VARCHAR AS email_domain, + REGEXP_LIKE( + b.email, + '^(info|admin|sales|support|hello|contact|noreply|no-reply)@.*' + )::BOOLEAN AS is_role_address, + c.customer_id, + ( + p.email IS NOT NULL + )::BOOLEAN AS is_prospect, + b.list_status_by_list::OBJECT AS list_status_by_list, + b.subscribed_lists::ARRAY AS subscribed_lists, + b.subscribed_list_count::INT AS subscribed_list_count, + b.first_subscribed_at, + b.last_unsubscribed_at, + b.is_globally_unsubscribed::BOOLEAN AS is_globally_unsubscribed, + b.has_double_opt_in::BOOLEAN AS has_double_opt_in, + f.first_signup_form::VARCHAR AS first_signup_form + FROM by_email AS b + LEFT JOIN customer_emails AS c + ON b.email = c.email + LEFT JOIN prospect_emails AS p + ON b.email = p.email + LEFT JOIN first_signup AS f + ON b.email = f.email +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/growth/int_referral_conversions.sql b/benchmarks/duckshop/project/models/intermediate/growth/int_referral_conversions.sql new file mode 100644 index 0000000000..45d1140783 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/growth/int_referral_conversions.sql @@ -0,0 +1,166 @@ +MODEL ( + description """ + One row per referral invite with the invitee matched to a customer account by normalised email, + the invitee's first order, whether it landed inside the qualifying window, and the reward value + in USD. + """, + constants ( + _qualifying_window_days 30, + _minimum_qualifying_order_usd 20.00, + ), + columns ( + invite_id (type VARCHAR, nullable false, audits [not_null, unique]), + referrer_customer_id (type VARCHAR, audits [not_null]), + invitee_customer_id (type VARCHAR), + invitee_email_domain (type VARCHAR), + sent_at (type TIMESTAMP_TZ), + accepted_at (type TIMESTAMP_TZ), + hours_to_accept (type NUMBER(10, 2)), + first_order_id (type VARCHAR), + first_order_usd (type NUMBER(18, 2)), + days_to_first_order (type INT), + is_self_referral (type BOOLEAN), + is_qualified (type BOOLEAN), + reward_usd (type NUMBER(18, 2)), + invite_sequence_for_referrer (type INT), + referrer_lifetime_orders (type INT), + ), +); + +WITH invites AS ( + SELECT * + FROM __ref("stg_referral_invites") +), + +identities AS ( + SELECT * + FROM __ref("int_customer_identity") +), + +order_sequence AS ( + SELECT * + FROM __ref("int_customer_order_sequence") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +sequenced_invites AS ( + SELECT + i.invite_id, + i.referrer_customer_id, + i.invitee_email, + REGEXP_SUBSTR(i.invitee_email, '@(.+)$', 1, 1, 'e', 1) AS invitee_email_domain, + i.sent_at, + i.accepted_at, + CONVERT_TIMEZONE('UTC', i.sent_at)::DATE AS sent_date, + i.reward_amount_minor, + i.reward_currency_code, + ROW_NUMBER() OVER ( + PARTITION BY i.referrer_customer_id + ORDER BY i.sent_at, i.invite_id + ) AS invite_sequence_for_referrer + FROM invites AS i +), + +email_accounts AS ( + SELECT + d.email_normalised, + MIN_BY(d.customer_id, d.customer_id) AS customer_id + FROM identities AS d + WHERE d.email_normalised IS NOT NULL + GROUP BY + d.email_normalised +), + +first_orders AS ( + SELECT + s.customer_id, + MIN_BY(s.order_id, s.placed_at) AS first_order_id, + MIN(s.placed_at) AS first_order_at, + MIN_BY(s.amount_usd, s.placed_at) AS first_order_usd + FROM order_sequence AS s + GROUP BY + s.customer_id +), + +referrer_orders AS ( + SELECT + s.customer_id, + COUNT(*) AS lifetime_orders + FROM order_sequence AS s + GROUP BY + s.customer_id +), + +reward_rates AS ( + SELECT + v.invite_id, + @sql.money_from_minor(v.reward_amount_minor, cur.minor_unit_exponent, places := 6) * fx.rate_to_usd AS reward_usd + FROM sequenced_invites AS v + INNER JOIN currencies AS cur + ON v.reward_currency_code = cur.currency_code + ASOF JOIN fx_rates AS fx MATCH_CONDITION (v.sent_date >= fx.rate_date) + ON v.reward_currency_code = fx.currency_code +), + +matched AS ( + SELECT + v.invite_id, + v.referrer_customer_id, + a.customer_id AS invitee_customer_id, + v.invitee_email_domain, + v.sent_at, + v.accepted_at, + v.invite_sequence_for_referrer, + f.first_order_id, + f.first_order_at, + f.first_order_usd, + r.reward_usd, + COALESCE(ro.lifetime_orders, 0) AS referrer_lifetime_orders + FROM sequenced_invites AS v + LEFT JOIN email_accounts AS a + ON v.invitee_email = a.email_normalised + LEFT JOIN first_orders AS f + ON a.customer_id = f.customer_id + LEFT JOIN reward_rates AS r + ON v.invite_id = r.invite_id + LEFT JOIN referrer_orders AS ro + ON v.referrer_customer_id = ro.customer_id +), + +final AS ( + SELECT + m.invite_id, + m.referrer_customer_id, + m.invitee_customer_id::VARCHAR AS invitee_customer_id, + m.invitee_email_domain::VARCHAR AS invitee_email_domain, + m.sent_at, + m.accepted_at, + ROUND(DATEDIFF('minute', m.sent_at, m.accepted_at) / 60, 2)::NUMBER(10, 2) AS hours_to_accept, + m.first_order_id::VARCHAR AS first_order_id, + m.first_order_usd::NUMBER(18, 2) AS first_order_usd, + DATEDIFF('day', m.sent_at, m.first_order_at)::INT AS days_to_first_order, + EQUAL_NULL(m.invitee_customer_id, m.referrer_customer_id)::BOOLEAN AS is_self_referral, + COALESCE( + m.first_order_at >= m.sent_at + AND DATEDIFF('day', m.sent_at, m.first_order_at) <= @const("_qualifying_window_days") + AND m.first_order_usd >= @const("_minimum_qualifying_order_usd") + AND NOT EQUAL_NULL(m.invitee_customer_id, m.referrer_customer_id), + FALSE + )::BOOLEAN AS is_qualified, + ROUND(m.reward_usd, 2)::NUMBER(18, 2) AS reward_usd, + m.invite_sequence_for_referrer::INT AS invite_sequence_for_referrer, + m.referrer_lifetime_orders::INT AS referrer_lifetime_orders + FROM matched AS m +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/growth/int_survey_answers_flat.sql b/benchmarks/duckshop/project/models/intermediate/growth/int_survey_answers_flat.sql new file mode 100644 index 0000000000..2936ed2df0 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/growth/int_survey_answers_flat.sql @@ -0,0 +1,144 @@ +MODEL ( + description """ + One row per answered question in each customer's latest response to each survey, with the answer + typed by its JSON kind, free-text answers cleaned, and the response's NPS group. JSON null + answers are kept as unanswered rows; missing keys produce no row. + """, + columns ( + answer_key (type VARCHAR, nullable false, audits [not_null, unique]), + response_id (type VARCHAR, audits [not_null]), + survey_code (type VARCHAR), + customer_id (type VARCHAR), + submitted_at (type TIMESTAMP_TZ), + question_key (type VARCHAR), + answer_kind (type VARCHAR), + answer_number (type NUMBER(18, 4)), + answer_flag (type BOOLEAN), + answer_text (type VARCHAR), + answer_choices (type ARRAY), + is_answered (type BOOLEAN), + nps_group (type VARCHAR), + question_position (type INT), + ), +); + +WITH responses AS ( + SELECT * + FROM __ref("stg_survey_responses") +), + +latest_responses AS ( + SELECT + r.response_id, + r.survey_code, + r.customer_id, + r.submitted_at, + r.nps_score, + r.answers + FROM responses AS r + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY r.customer_id, r.survey_code + ORDER BY r.submitted_at DESC NULLS LAST, r.response_id DESC + ) = 1 +), + +answers AS ( + SELECT + r.response_id, + r.survey_code, + r.customer_id, + r.submitted_at, + r.nps_score, + a.key AS raw_question_key, + a.value AS answer_value, + TYPEOF(a.value) AS value_type + FROM latest_responses AS r, LATERAL FLATTEN(input => r.answers) AS a +), + +typed AS ( + SELECT + a.response_id, + a.survey_code, + a.customer_id, + a.submitted_at, + a.nps_score, + LOWER(REGEXP_REPLACE(TRIM(a.raw_question_key), '[^A-Za-z0-9]+', '_')) AS question_key, + CASE a.value_type + WHEN 'NULL_VALUE' + THEN 'null' + WHEN 'INTEGER' + THEN 'number' + WHEN 'DECIMAL' + THEN 'number' + WHEN 'DOUBLE' + THEN 'number' + WHEN 'BOOLEAN' + THEN 'flag' + WHEN 'ARRAY' + THEN 'choices' + WHEN 'VARCHAR' + THEN IFF(TRY_TO_NUMBER(a.answer_value::VARCHAR, 18, 4) IS NULL, 'text', 'number') + ELSE 'other' + END AS answer_kind, + a.answer_value + FROM answers AS a +), + +cleaned AS ( + SELECT + t.response_id, + t.survey_code, + t.customer_id, + t.submitted_at, + t.nps_score, + t.question_key, + t.answer_kind, + IFF( + t.answer_kind = 'number', + TRY_TO_NUMBER(t.answer_value::VARCHAR, 18, 4), + NULL + ) AS answer_number, + IFF(t.answer_kind = 'flag', t.answer_value::BOOLEAN, NULL) AS answer_flag, + IFF( + t.answer_kind = 'text', + REGEXP_REPLACE(@sql.any_to_text(t.answer_value), '\\s{2,}', ' '), + NULL + ) AS answer_text, + IFF(t.answer_kind = 'choices', t.answer_value::ARRAY, NULL) AS answer_choices + FROM typed AS t +), + +final AS ( + SELECT + ( + c.response_id || ':' || c.question_key + )::VARCHAR AS answer_key, + c.response_id, + c.survey_code, + c.customer_id, + c.submitted_at, + c.question_key::VARCHAR AS question_key, + c.answer_kind::VARCHAR AS answer_kind, + c.answer_number::NUMBER(18, 4) AS answer_number, + c.answer_flag::BOOLEAN AS answer_flag, + c.answer_text::VARCHAR AS answer_text, + c.answer_choices::ARRAY AS answer_choices, + ( + c.answer_kind <> 'null' + )::BOOLEAN AS is_answered, + CASE + WHEN c.nps_score IS NULL + THEN NULL + WHEN c.nps_score >= 9 + THEN 'promoter' + WHEN c.nps_score >= 7 + THEN 'passive' + ELSE 'detractor' + END::VARCHAR AS nps_group, + RANK() OVER (PARTITION BY c.response_id ORDER BY c.question_key)::INT AS question_position + FROM cleaned AS c +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/inventory/int_inventory_ledger.sql b/benchmarks/duckshop/project/models/intermediate/inventory/int_inventory_ledger.sql new file mode 100644 index 0000000000..5f69ea6297 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/inventory/int_inventory_ledger.sql @@ -0,0 +1,183 @@ +MODEL ( + description """ + One row per stock change of a SKU in a warehouse from every source (movements, purchase order + receipts and transfers in and out), in UTC, with the running on-hand quantity. + """, + columns ( + ledger_entry_key (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR, audits [not_null]), + entry_source (type VARCHAR, audits [ + accepted_values (values ["movement", "receipt", "transfer_out", "transfer_in"]), + ]), + qty_delta (type INT), + occurred_at_utc (type TIMESTAMP_NTZ), + occurred_date_local (type DATE), + running_on_hand (type INT), + entry_sequence (type INT), + is_negative_balance (type BOOLEAN), + ), +); + +WITH movements AS ( + SELECT * + FROM __ref("stg_stock_movements") +), + +receipts AS ( + SELECT * + FROM __ref("stg_po_receipts") +), + +transfers AS ( + SELECT * + FROM __ref("stg_stock_transfers") +), + +warehouses AS ( + SELECT * + FROM __ref("stg_warehouses") +), + +movement_entries AS ( + SELECT + ( + 'M:' || m.movement_id + )::VARCHAR AS ledger_entry_key, + m.sku::VARCHAR AS sku, + m.warehouse_code::VARCHAR AS warehouse_code, + 'movement'::VARCHAR AS entry_source, + m.qty_delta::INT AS qty_delta, + CONVERT_TIMEZONE(w.iana_timezone, 'UTC', m.moved_at_local)::TIMESTAMP_NTZ AS occurred_at_utc, + m.moved_at_local::DATE AS occurred_date_local + FROM movements AS m + INNER JOIN warehouses AS w + ON m.warehouse_code = w.warehouse_code + WHERE m.reference_type <> 'PO' +), + +receipt_entries AS ( + SELECT + ( + 'R:' || r.receipt_id + )::VARCHAR AS ledger_entry_key, + r.sku::VARCHAR AS sku, + r.warehouse_code::VARCHAR AS warehouse_code, + 'receipt'::VARCHAR AS entry_source, + r.sellable_qty::INT AS qty_delta, + CONVERT_TIMEZONE(w.iana_timezone, 'UTC', r.received_at_local)::TIMESTAMP_NTZ AS occurred_at_utc, + r.received_at_local::DATE AS occurred_date_local + FROM receipts AS r + INNER JOIN warehouses AS w + ON r.warehouse_code = w.warehouse_code +), + +transfer_out_entries AS ( + SELECT + ( + 'TO:' || t.transfer_id + )::VARCHAR AS ledger_entry_key, + t.sku::VARCHAR AS sku, + t.from_warehouse_code::VARCHAR AS warehouse_code, + 'transfer_out'::VARCHAR AS entry_source, + ( + -t.qty + )::INT AS qty_delta, + CONVERT_TIMEZONE('UTC', t.shipped_at)::TIMESTAMP_NTZ AS occurred_at_utc, + CONVERT_TIMEZONE(w.iana_timezone, t.shipped_at)::DATE AS occurred_date_local + FROM transfers AS t + INNER JOIN warehouses AS w + ON t.from_warehouse_code = w.warehouse_code + WHERE t.shipped_at IS NOT NULL +), + +transfer_in_entries AS ( + SELECT + ( + 'TI:' || t.transfer_id + )::VARCHAR AS ledger_entry_key, + t.sku::VARCHAR AS sku, + t.to_warehouse_code::VARCHAR AS warehouse_code, + 'transfer_in'::VARCHAR AS entry_source, + t.qty::INT AS qty_delta, + CONVERT_TIMEZONE('UTC', t.arrived_at)::TIMESTAMP_NTZ AS occurred_at_utc, + CONVERT_TIMEZONE(w.iana_timezone, t.arrived_at)::DATE AS occurred_date_local + FROM transfers AS t + INNER JOIN warehouses AS w + ON t.to_warehouse_code = w.warehouse_code + WHERE t.arrived_at IS NOT NULL +), + +all_entries AS ( + SELECT + me.ledger_entry_key::VARCHAR AS ledger_entry_key, + me.sku::VARCHAR AS sku, + me.warehouse_code::VARCHAR AS warehouse_code, + me.entry_source::VARCHAR AS entry_source, + me.qty_delta::INT AS qty_delta, + me.occurred_at_utc::TIMESTAMP_NTZ AS occurred_at_utc, + me.occurred_date_local::DATE AS occurred_date_local + FROM movement_entries AS me + UNION ALL + SELECT + re.ledger_entry_key::VARCHAR AS ledger_entry_key, + re.sku::VARCHAR AS sku, + re.warehouse_code::VARCHAR AS warehouse_code, + re.entry_source::VARCHAR AS entry_source, + re.qty_delta::INT AS qty_delta, + re.occurred_at_utc::TIMESTAMP_NTZ AS occurred_at_utc, + re.occurred_date_local::DATE AS occurred_date_local + FROM receipt_entries AS re + UNION ALL + SELECT + toe.ledger_entry_key::VARCHAR AS ledger_entry_key, + toe.sku::VARCHAR AS sku, + toe.warehouse_code::VARCHAR AS warehouse_code, + toe.entry_source::VARCHAR AS entry_source, + toe.qty_delta::INT AS qty_delta, + toe.occurred_at_utc::TIMESTAMP_NTZ AS occurred_at_utc, + toe.occurred_date_local::DATE AS occurred_date_local + FROM transfer_out_entries AS toe + UNION ALL + SELECT + tie.ledger_entry_key::VARCHAR AS ledger_entry_key, + tie.sku::VARCHAR AS sku, + tie.warehouse_code::VARCHAR AS warehouse_code, + tie.entry_source::VARCHAR AS entry_source, + tie.qty_delta::INT AS qty_delta, + tie.occurred_at_utc::TIMESTAMP_NTZ AS occurred_at_utc, + tie.occurred_date_local::DATE AS occurred_date_local + FROM transfer_in_entries AS tie +), + +final AS ( + SELECT + a.ledger_entry_key, + a.sku, + a.warehouse_code, + a.entry_source, + a.qty_delta, + a.occurred_at_utc, + a.occurred_date_local, + SUM(a.qty_delta) OVER ( + PARTITION BY a.sku, a.warehouse_code + ORDER BY a.occurred_at_utc, a.ledger_entry_key + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + )::INT AS running_on_hand, + COUNT(*) OVER ( + PARTITION BY a.sku, a.warehouse_code + ORDER BY a.occurred_at_utc, a.ledger_entry_key + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + )::INT AS entry_sequence, + ( + SUM(a.qty_delta) OVER ( + PARTITION BY a.sku, a.warehouse_code + ORDER BY a.occurred_at_utc, a.ledger_entry_key + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) < 0 + ) AS is_negative_balance + FROM all_entries AS a +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/inventory/int_inventory_valuation_daily.sql b/benchmarks/duckshop/project/models/intermediate/inventory/int_inventory_valuation_daily.sql new file mode 100644 index 0000000000..55ffb6493a --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/inventory/int_inventory_valuation_daily.sql @@ -0,0 +1,78 @@ +MODEL ( + description """ + One row per warehouse and day with stock units and their value in USD at current supplier cost + """, + columns ( + valuation_key (type VARCHAR, nullable false, audits [not_null, unique]), + warehouse_code (type VARCHAR, audits [not_null]), + calendar_date (type DATE, audits [not_null]), + sku_count (type INT), + units_on_hand (type INT), + stock_value_usd (type NUMBER(18, 2)), + unvalued_units (type INT), + stocked_out_skus (type INT), + value_change_usd (type NUMBER(18, 2)), + ), +); + +WITH stock_days AS ( + SELECT * + FROM __ref("int_stock_on_hand_daily") +), + +costs AS ( + SELECT * + FROM __ref("int_supplier_cost_current") +), + +valued AS ( + SELECT + s.warehouse_code, + s.calendar_date, + s.sku, + s.on_hand_qty, + s.is_stocked_out, + s.on_hand_qty * c.unit_cost_usd AS stock_value_usd, + IFF(c.unit_cost_usd IS NULL, s.on_hand_qty, 0) AS unvalued_units + FROM stock_days AS s + LEFT JOIN costs AS c + ON s.sku = c.sku +), + +daily AS ( + SELECT + v.warehouse_code, + v.calendar_date, + COUNT(DISTINCT v.sku) AS sku_count, + SUM(GREATEST(v.on_hand_qty, 0)) AS units_on_hand, + SUM(GREATEST(v.stock_value_usd, 0)) AS stock_value_usd, + SUM(v.unvalued_units) AS unvalued_units, + COUNT_IF(v.is_stocked_out) AS stocked_out_skus + FROM valued AS v + GROUP BY + v.warehouse_code, + v.calendar_date +), + +final AS ( + SELECT + ( + d.warehouse_code || '|' || TO_CHAR(d.calendar_date, 'YYYY-MM-DD') + )::VARCHAR AS valuation_key, + d.warehouse_code, + d.calendar_date, + d.sku_count::INT AS sku_count, + d.units_on_hand::INT AS units_on_hand, + ROUND(d.stock_value_usd, 2)::NUMBER(18, 2) AS stock_value_usd, + d.unvalued_units::INT AS unvalued_units, + d.stocked_out_skus::INT AS stocked_out_skus, + ROUND( + d.stock_value_usd + - LAG(d.stock_value_usd) OVER (PARTITION BY d.warehouse_code ORDER BY d.calendar_date), + 2 + )::NUMBER(18, 2) AS value_change_usd + FROM daily AS d +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/inventory/int_purchase_order_lines_received.sql b/benchmarks/duckshop/project/models/intermediate/inventory/int_purchase_order_lines_received.sql new file mode 100644 index 0000000000..1f4a1f9922 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/inventory/int_purchase_order_lines_received.sql @@ -0,0 +1,101 @@ +MODEL ( + description """ + One row per purchase order line with the quantity received, damaged and still outstanding, the + fill rate, and how late the first receipt was against the expected date. + """, + columns ( + po_line_key (type VARCHAR, nullable false, audits [not_null, unique]), + po_id (type VARCHAR, audits [not_null]), + supplier_code (type VARCHAR), + warehouse_code (type VARCHAR), + sku (type VARCHAR), + ordered_qty (type INT), + received_qty (type INT), + damaged_qty (type INT), + outstanding_qty (type INT), + fill_rate (type NUMBER(9, 4)), + first_received_at_local (type TIMESTAMP_NTZ), + days_late (type INT), + receipt_count (type INT), + discrepancy_codes (type ARRAY), + ), +); + +WITH po_lines AS ( + SELECT * + FROM __ref("stg_purchase_order_lines") +), + +purchase_orders AS ( + SELECT * + FROM __ref("stg_purchase_orders") +), + +receipts AS ( + SELECT * + FROM __ref("stg_po_receipts") +), + +warehouses AS ( + SELECT * + FROM __ref("stg_warehouses") +), + +receipt_totals AS ( + SELECT + r.po_id, + r.sku, + SUM(r.received_qty) AS received_qty, + SUM(r.damaged_qty) AS damaged_qty, + MIN(r.received_at_local) AS first_received_at_local, + COUNT(*) AS receipt_count, + ARRAY_DISTINCT(ARRAY_FLATTEN(ARRAY_AGG(r.discrepancy_codes))) AS discrepancy_codes + FROM receipts AS r + GROUP BY + r.po_id, + r.sku +), + +expected_local AS ( + SELECT + p.po_id, + p.supplier_code, + p.warehouse_code, + CONVERT_TIMEZONE(w.iana_timezone, p.expected_at)::TIMESTAMP_NTZ AS expected_at_local + FROM purchase_orders AS p + INNER JOIN warehouses AS w + ON p.warehouse_code = w.warehouse_code +), + +final AS ( + SELECT + l.po_line_key, + l.po_id, + e.supplier_code, + e.warehouse_code, + l.sku, + l.ordered_qty, + COALESCE(r.received_qty, 0)::INT AS received_qty, + COALESCE(r.damaged_qty, 0)::INT AS damaged_qty, + GREATEST(l.ordered_qty - COALESCE(r.received_qty, 0), 0)::INT AS outstanding_qty, + ROUND( + LEAST(COALESCE(r.received_qty, 0) / NULLIF(l.ordered_qty, 0), 1), + 4 + )::NUMBER(9, 4) AS fill_rate, + r.first_received_at_local, + GREATEST( + DATEDIFF('day', e.expected_at_local, r.first_received_at_local), + 0 + )::INT AS days_late, + COALESCE(r.receipt_count, 0)::INT AS receipt_count, + COALESCE(r.discrepancy_codes, ARRAY_CONSTRUCT())::ARRAY AS discrepancy_codes + FROM po_lines AS l + INNER JOIN expected_local AS e + ON l.po_id = e.po_id + LEFT JOIN receipt_totals AS r + ON l.po_id = r.po_id + AND l.sku = r.sku +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/inventory/int_reservations_open.sql b/benchmarks/duckshop/project/models/intermediate/inventory/int_reservations_open.sql new file mode 100644 index 0000000000..5b26dcb247 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/inventory/int_reservations_open.sql @@ -0,0 +1,79 @@ +MODEL ( + description """ + One row per SKU and warehouse with open reservations against the latest on-hand quantity, the + available-to-promise quantity and the oldest open reservation. + """, + columns ( + stock_key (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR), + open_reservations (type INT), + reserved_qty (type INT), + backordered_qty (type INT), + on_hand_qty (type INT), + available_to_promise (type INT), + oldest_reserved_at (type TIMESTAMP_TZ), + ), +); + +WITH reservations AS ( + SELECT * + FROM __ref("stg_stock_reservations") +), + +ledger AS ( + SELECT * + FROM __ref("int_inventory_ledger") +), + +latest_balance AS ( + SELECT + l.sku, + l.warehouse_code, + l.running_on_hand + FROM ledger AS l + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY l.sku, l.warehouse_code + ORDER BY l.entry_sequence DESC, l.ledger_entry_key DESC + ) = 1 +), + +open_holds AS ( + SELECT + r.sku, + r.warehouse_code, + COUNT(*) AS open_reservations, + SUM(IFF(r.is_backorder, 0, r.qty)) AS reserved_qty, + SUM(IFF(r.is_backorder, r.qty, 0)) AS backordered_qty, + MIN(r.reserved_at) AS oldest_reserved_at + FROM reservations AS r + WHERE r.is_open + GROUP BY + r.sku, + r.warehouse_code +), + +final AS ( + SELECT + ( + o.sku || '|' || o.warehouse_code + )::VARCHAR AS stock_key, + o.sku, + o.warehouse_code, + o.open_reservations::INT AS open_reservations, + o.reserved_qty::INT AS reserved_qty, + o.backordered_qty::INT AS backordered_qty, + COALESCE(b.running_on_hand, 0)::INT AS on_hand_qty, + ( + COALESCE(b.running_on_hand, 0) - o.reserved_qty + )::INT AS available_to_promise, + o.oldest_reserved_at + FROM open_holds AS o + LEFT JOIN latest_balance AS b + ON o.sku = b.sku + AND o.warehouse_code = b.warehouse_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/inventory/int_stock_count_variances.sql b/benchmarks/duckshop/project/models/intermediate/inventory/int_stock_count_variances.sql new file mode 100644 index 0000000000..d5106bada9 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/inventory/int_stock_count_variances.sql @@ -0,0 +1,100 @@ +MODEL ( + description """ + One row per cycle count with the quantity the ledger expected at that moment (ASOF the ledger), + the variance, and whether it exceeds the warehouse's tolerance. + """, + constants ( + _tolerance_units 2, + _tolerance_pct constant(type decimal, value "0.05"), + ), + columns ( + count_id (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR), + counted_at_local (type TIMESTAMP_NTZ), + counted_qty (type INT), + expected_qty (type INT), + variance_qty (type INT), + variance_pct (type NUMBER(9, 4)), + is_out_of_tolerance (type BOOLEAN), + is_blind_count (type BOOLEAN), + recount_sequence (type INT), + ), +); + +WITH counts AS ( + SELECT * + FROM __ref("stg_stock_counts") +), + +ledger AS ( + SELECT * + FROM __ref("int_inventory_ledger") +), + +warehouses AS ( + SELECT * + FROM __ref("stg_warehouses") +), + +counts_utc AS ( + SELECT + c.count_id, + c.sku, + c.warehouse_code, + c.counted_at_local, + c.counted_qty, + c.is_blind_count, + CONVERT_TIMEZONE(w.iana_timezone, 'UTC', c.counted_at_local)::TIMESTAMP_NTZ AS counted_at_utc + FROM counts AS c + INNER JOIN warehouses AS w + ON c.warehouse_code = w.warehouse_code +), + +with_expected AS ( + SELECT + c.count_id, + c.sku, + c.warehouse_code, + c.counted_at_local, + c.counted_qty, + c.is_blind_count, + COALESCE(l.running_on_hand, 0) AS expected_qty + FROM counts_utc AS c + ASOF JOIN ledger AS l MATCH_CONDITION (c.counted_at_utc >= l.occurred_at_utc) + ON c.sku = l.sku + AND c.warehouse_code = l.warehouse_code +), + +final AS ( + SELECT + e.count_id, + e.sku, + e.warehouse_code, + e.counted_at_local, + e.counted_qty, + e.expected_qty::INT AS expected_qty, + ( + e.counted_qty - e.expected_qty + )::INT AS variance_qty, + ROUND(( + e.counted_qty - e.expected_qty + ) + / NULLIF(e.expected_qty, 0), 4)::NUMBER(9, 4) AS variance_pct, + ( + ABS(e.counted_qty - e.expected_qty) > @const("_tolerance_units") + AND ABS( + e.counted_qty - e.expected_qty + ) > @const("_tolerance_pct") * GREATEST(e.expected_qty, 1) + ) AS is_out_of_tolerance, + e.is_blind_count, + COUNT(*) OVER ( + PARTITION BY e.sku, e.warehouse_code, e.counted_at_local::DATE + ORDER BY e.counted_at_local, e.count_id + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + )::INT AS recount_sequence + FROM with_expected AS e +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/inventory/int_stock_on_hand_daily.sql b/benchmarks/duckshop/project/models/intermediate/inventory/int_stock_on_hand_daily.sql new file mode 100644 index 0000000000..ceb2e74058 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/inventory/int_stock_on_hand_daily.sql @@ -0,0 +1,109 @@ +MODEL ( + description """ + One row per SKU, warehouse and calendar day with the on-hand quantity at the end of the + warehouse's local day (ASOF the stock ledger) and the most recent cycle count before it. + """, + columns ( + stock_day_key (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR, audits [not_null]), + calendar_date (type DATE, audits [not_null]), + month_start (type DATE), + on_hand_qty (type INT), + last_counted_qty (type INT), + last_counted_date (type DATE), + days_since_count (type INT), + is_stocked_out (type BOOLEAN), + ), +); + +WITH ledger AS ( + SELECT * + FROM __ref("int_inventory_ledger") +), + +counts AS ( + SELECT * + FROM __ref("stg_stock_counts") +), + +warehouses AS ( + SELECT * + FROM __ref("stg_warehouses") +), + +@sql.date_spine(@const("history_start_date"), 730) + +stock_keys AS ( + SELECT + l.sku, + l.warehouse_code, + MIN(l.occurred_date_local) AS first_date + FROM ledger AS l + GROUP BY + l.sku, + l.warehouse_code +), key_days AS ( + SELECT + k.sku, + k.warehouse_code, + d.calendar_date, + CONVERT_TIMEZONE( + w.iana_timezone, + 'UTC', + DATEADD('day', 1, d.calendar_date)::TIMESTAMP_NTZ + )::TIMESTAMP_NTZ AS day_end_utc, + DATEADD('day', 1, d.calendar_date)::TIMESTAMP_NTZ AS day_end_local + FROM stock_keys AS k + INNER JOIN date_spine AS d + ON d.calendar_date >= k.first_date + INNER JOIN warehouses AS w + ON k.warehouse_code = w.warehouse_code + WHERE d.calendar_date <= '@@valuation_date'::DATE +), with_balance AS ( + SELECT + kd.sku, + kd.warehouse_code, + kd.calendar_date, + kd.day_end_local, + l.running_on_hand + FROM key_days AS kd + ASOF JOIN ledger AS l MATCH_CONDITION (kd.day_end_utc > l.occurred_at_utc) + ON kd.sku = l.sku + AND kd.warehouse_code = l.warehouse_code +), with_count AS ( + SELECT + b.sku, + b.warehouse_code, + b.calendar_date, + b.running_on_hand, + c.counted_qty, + c.counted_at_local + FROM with_balance AS b + ASOF JOIN counts AS c MATCH_CONDITION (b.day_end_local > c.counted_at_local) + ON b.sku = c.sku + AND b.warehouse_code = c.warehouse_code +), final AS ( + SELECT + ( + c.sku || '|' || c.warehouse_code || '|' || TO_CHAR(c.calendar_date, 'YYYY-MM-DD') + )::VARCHAR AS stock_day_key, + c.sku, + c.warehouse_code, + c.calendar_date, + DATE_FROM_PARTS(YEAR(c.calendar_date), MONTH(c.calendar_date), 1) AS month_start, + COALESCE(c.running_on_hand, 0)::INT AS on_hand_qty, + c.counted_qty::INT AS last_counted_qty, + c.counted_at_local::DATE AS last_counted_date, + DATEDIFF( + 'day', + c.counted_at_local::DATE, + c.calendar_date + )::INT AS days_since_count, + ( + COALESCE(c.running_on_hand, 0) <= 0 + ) AS is_stocked_out + FROM with_count AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/inventory/int_stock_transfers_in_transit.sql b/benchmarks/duckshop/project/models/intermediate/inventory/int_stock_transfers_in_transit.sql new file mode 100644 index 0000000000..634c65d5f1 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/inventory/int_stock_transfers_in_transit.sql @@ -0,0 +1,75 @@ +MODEL ( + description """ + One row per stock transfer with its state and the expected arrival from the lane's historical + median transit time + """, + columns ( + transfer_id (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR), + from_warehouse_code (type VARCHAR), + to_warehouse_code (type VARCHAR), + qty (type INT), + transfer_state (type VARCHAR, audits [ + accepted_values (values ["planned", "in_transit", "arrived"]), + ]), + shipped_at (type TIMESTAMP_TZ), + expected_arrival_at (type TIMESTAMP_TZ), + lane_median_transit_hours (type NUMBER(10, 2)), + is_overdue (type BOOLEAN), + ), +); + +WITH transfers AS ( + SELECT * + FROM __ref("stg_stock_transfers") +), + +lane_history AS ( + SELECT + t.from_warehouse_code, + t.to_warehouse_code, + MEDIAN(t.transit_hours) AS lane_median_transit_hours + FROM transfers AS t + WHERE t.arrived_at IS NOT NULL + GROUP BY + t.from_warehouse_code, + t.to_warehouse_code +), + +final AS ( + SELECT + t.transfer_id, + t.sku, + t.from_warehouse_code, + t.to_warehouse_code, + t.qty, + CASE + WHEN t.arrived_at IS NOT NULL + THEN 'arrived' + WHEN t.shipped_at IS NOT NULL + THEN 'in_transit' + ELSE 'planned' + END::VARCHAR AS transfer_state, + t.shipped_at, + TIMESTAMPADD( + 'minute', + ROUND(COALESCE(h.lane_median_transit_hours, 48) * 60)::INT, + t.shipped_at + )::TIMESTAMP_TZ AS expected_arrival_at, + h.lane_median_transit_hours::NUMBER(10, 2) AS lane_median_transit_hours, + ( + t.arrived_at IS NULL + AND TIMESTAMPADD( + 'minute', + ROUND(COALESCE(h.lane_median_transit_hours, 48) * 60)::INT, + t.shipped_at + ) < '@@valuation_date'::TIMESTAMP_TZ + ) AS is_overdue + FROM transfers AS t + LEFT JOIN lane_history AS h + ON t.from_warehouse_code = h.from_warehouse_code + AND t.to_warehouse_code = h.to_warehouse_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/inventory/int_supplier_lead_times.sql b/benchmarks/duckshop/project/models/intermediate/inventory/int_supplier_lead_times.sql new file mode 100644 index 0000000000..2759f27a91 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/inventory/int_supplier_lead_times.sql @@ -0,0 +1,83 @@ +MODEL ( + description """ + One row per supplier with lead-time percentiles, on-time rate and fill rate over its received + purchase orders + """, + columns ( + supplier_code (type VARCHAR, nullable false, audits [not_null, unique]), + supplier_name (type VARCHAR), + received_po_count (type INT), + lead_time_p50_days (type NUMBER(10, 2)), + lead_time_p90_days (type NUMBER(10, 2)), + lead_time_stddev_days (type NUMBER(10, 2)), + on_time_rate (type NUMBER(9, 4)), + avg_fill_rate (type NUMBER(9, 4)), + latest_po_at (type TIMESTAMP_TZ), + ), +); + +WITH purchase_orders AS ( + SELECT * + FROM __ref("stg_purchase_orders") +), + +po_lines AS ( + SELECT * + FROM __ref("int_purchase_order_lines_received") +), + +suppliers AS ( + SELECT * + FROM __ref("stg_suppliers") +), + +po_fill AS ( + SELECT + l.po_id, + SUM(l.received_qty) / NULLIF(SUM(l.ordered_qty), 0) AS po_fill_rate, + MAX(l.days_late) AS max_days_late + FROM po_lines AS l + GROUP BY + l.po_id +), + +received AS ( + SELECT + p.supplier_code, + p.ordered_at, + DATEDIFF('hour', p.ordered_at, p.received_at) / 24 AS lead_time_days, + f.po_fill_rate, + f.max_days_late + FROM purchase_orders AS p + INNER JOIN po_fill AS f + ON p.po_id = f.po_id + WHERE p.received_at IS NOT NULL +), + +final AS ( + SELECT + r.supplier_code, + s.supplier_name, + COUNT(*)::INT AS received_po_count, + ROUND( + PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY r.lead_time_days), + 2 + )::NUMBER(10, 2) AS lead_time_p50_days, + ROUND( + PERCENTILE_CONT(0.9) WITHIN GROUP (ORDER BY r.lead_time_days), + 2 + )::NUMBER(10, 2) AS lead_time_p90_days, + ROUND(STDDEV_SAMP(r.lead_time_days), 2)::NUMBER(10, 2) AS lead_time_stddev_days, + ROUND(COUNT_IF(r.max_days_late = 0) / COUNT(*), 4)::NUMBER(9, 4) AS on_time_rate, + ROUND(AVG(r.po_fill_rate), 4)::NUMBER(9, 4) AS avg_fill_rate, + MAX(r.ordered_at) AS latest_po_at + FROM received AS r + LEFT JOIN suppliers AS s + ON r.supplier_code = s.supplier_code + GROUP BY + r.supplier_code, + s.supplier_name +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_cart_abandonment.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_cart_abandonment.sql new file mode 100644 index 0000000000..fa5b130344 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_cart_abandonment.sql @@ -0,0 +1,127 @@ +MODEL ( + description """ + One row per cart with its value, whether it converted to an order by the same customer within a + day of its last update, and the hours it sat idle. + """, + constants ( + _conversion_window_hours 24, + ), + columns ( + cart_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR), + storefront_code (type VARCHAR), + cart_status (type VARCHAR), + item_count (type INT), + units (type INT), + cart_value_minor (type INT), + last_activity_at (type TIMESTAMP_TZ), + converted_order_id (type VARCHAR), + is_converted (type BOOLEAN), + is_abandoned (type BOOLEAN), + hours_to_convert (type NUMBER(10, 2)), + distinct_sku_families (type INT), + ), +); + +WITH carts AS ( + SELECT * + FROM __ref("stg_carts") +), + +cart_items AS ( + SELECT * + FROM __ref("stg_cart_items") +), + +orders AS ( + SELECT * + FROM __ref("stg_orders") +), + +item_totals AS ( + SELECT + i.cart_id, + COUNT(*) AS item_count, + SUM(COALESCE(i.quantity, 1)) AS units, + SUM(COALESCE(i.quantity, 1) * COALESCE(i.unit_amount_minor, 0)) AS cart_value_minor, + COUNT(DISTINCT __udf("udf__sku_family")(i.sku)) AS distinct_sku_families + FROM cart_items AS i + GROUP BY + i.cart_id +), + +cart_activity AS ( + SELECT + c.cart_id, + c.customer_id, + c.storefront_code, + c.cart_status, + COALESCE(c.updated_at, c.created_at) AS last_activity_at + FROM carts AS c +), + +customer_orders AS ( + SELECT + o.order_id, + o.customer_id, + o.placed_at + FROM orders AS o + WHERE o.customer_id IS NOT NULL +), + +matches AS ( + SELECT + a.cart_id, + co.order_id, + co.placed_at + FROM cart_activity AS a + INNER JOIN customer_orders AS co + ON a.customer_id = co.customer_id + AND co.placed_at >= a.last_activity_at + WHERE DATEDIFF('hour', a.last_activity_at, co.placed_at) <= @const("_conversion_window_hours") +), + +first_match AS ( + SELECT + m.cart_id, + MIN_BY(m.order_id, m.placed_at) AS converted_order_id, + MIN(m.placed_at) AS converted_at + FROM matches AS m + GROUP BY + m.cart_id +), + +final AS ( + SELECT + a.cart_id, + a.customer_id, + a.storefront_code, + a.cart_status, + COALESCE(t.item_count, 0)::INT AS item_count, + COALESCE(t.units, 0)::INT AS units, + COALESCE(t.cart_value_minor, 0)::INT AS cart_value_minor, + a.last_activity_at, + f.converted_order_id::VARCHAR AS converted_order_id, + ( + f.converted_order_id IS NOT NULL + OR a.cart_status = 'ordered' + ) AS is_converted, + ( + f.converted_order_id IS NULL + AND a.cart_status NOT IN ('ordered', 'merged') + AND COALESCE(t.item_count, 0) > 0 + ) AS is_abandoned, + ROUND( + DATEDIFF('minute', a.last_activity_at, f.converted_at) / 60, + 2 + )::NUMBER(10, 2) AS hours_to_convert, + COALESCE(t.distinct_sku_families, 0)::INT AS distinct_sku_families + FROM cart_activity AS a + LEFT JOIN item_totals AS t + ON a.cart_id = t.cart_id + LEFT JOIN first_match AS f + ON a.cart_id = f.cart_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_order_lines_discounted.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_order_lines_discounted.sql new file mode 100644 index 0000000000..a2d1cfef3c --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_order_lines_discounted.sql @@ -0,0 +1,112 @@ +MODEL ( + description """ + One row per order line with its discounts rolled up by kind, the promotions behind them, and the + net amount after discounts in minor units. + """, + columns ( + order_line_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + sku (type VARCHAR, audits [not_null]), + quantity (type INT), + gross_amount_minor (type INT), + discount_amount_minor (type INT), + net_amount_minor (type INT), + tax_amount_minor (type INT), + discount_by_kind (type OBJECT), + promo_codes (type ARRAY), + has_stacked_promotions (type BOOLEAN), + discount_share (type NUMBER(9, 6)), + ), +); + +WITH order_lines AS ( + SELECT * + FROM __ref("stg_order_lines") +), + +line_discounts AS ( + SELECT * + FROM __ref("stg_order_line_discounts") +), + +promotions AS ( + SELECT * + FROM __ref("stg_promotions") +), + +discounts_with_promotion AS ( + SELECT + d.order_line_id, + d.discount_kind, + COALESCE(d.discount_amount_minor, 0) AS discount_amount_minor, + p.promo_code, + COALESCE(p.is_stackable, FALSE) AS is_stackable + FROM line_discounts AS d + LEFT JOIN promotions AS p + ON d.discount_code = p.promo_code +), + +kind_totals AS ( + SELECT + dp.order_line_id, + dp.discount_kind, + SUM(dp.discount_amount_minor) AS kind_amount_minor + FROM discounts_with_promotion AS dp + GROUP BY + dp.order_line_id, + dp.discount_kind +), + +kind_objects AS ( + SELECT + k.order_line_id, + OBJECT_AGG(k.discount_kind, k.kind_amount_minor::VARIANT) AS discount_by_kind, + SUM(k.kind_amount_minor) AS discount_amount_minor + FROM kind_totals AS k + GROUP BY + k.order_line_id +), + +promotion_sets AS ( + SELECT + dp.order_line_id, + ARRAY_AGG(DISTINCT dp.promo_code) WITHIN GROUP (ORDER BY dp.promo_code) AS promo_codes, + COUNT(DISTINCT dp.promo_code) AS promotion_count, + BOOLAND_AGG(dp.is_stackable) AS all_stackable + FROM discounts_with_promotion AS dp + WHERE dp.promo_code IS NOT NULL + GROUP BY + dp.order_line_id +), + +final AS ( + SELECT + l.order_line_id, + l.order_id, + l.sku, + l.quantity, + l.gross_amount_minor, + LEAST(COALESCE(k.discount_amount_minor, 0), l.gross_amount_minor)::INT AS discount_amount_minor, + ( + l.gross_amount_minor - LEAST(COALESCE(k.discount_amount_minor, 0), l.gross_amount_minor) + )::INT AS net_amount_minor, + l.tax_amount_minor, + COALESCE(k.discount_by_kind, OBJECT_CONSTRUCT())::OBJECT AS discount_by_kind, + COALESCE(ps.promo_codes, ARRAY_CONSTRUCT())::ARRAY AS promo_codes, + ( + COALESCE(ps.promotion_count, 0) > 1 + AND COALESCE(ps.all_stackable, FALSE) + ) AS has_stacked_promotions, + ROUND( + COALESCE(k.discount_amount_minor, 0) / NULLIF(l.gross_amount_minor, 0), + 6 + )::NUMBER(9, 6) AS discount_share + FROM order_lines AS l + LEFT JOIN kind_objects AS k + ON l.order_line_id = k.order_line_id + LEFT JOIN promotion_sets AS ps + ON l.order_line_id = ps.order_line_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_order_lines_priced.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_order_lines_priced.sql new file mode 100644 index 0000000000..49681c3c71 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_order_lines_priced.sql @@ -0,0 +1,134 @@ +MODEL ( + description """ + One row per order line with the list price that applied when the order was placed, the markdown + against list, and product attributes for downstream margin and mix analysis. + """, + columns ( + order_line_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + sku (type VARCHAR), + sku_family (type VARCHAR), + product_id (type VARCHAR), + quantity (type INT), + unit_amount_minor (type INT), + list_amount_minor (type INT), + markdown_pct (type NUMBER(9, 4)), + gross_amount_minor (type INT), + net_amount_minor (type INT), + discount_amount_minor (type INT), + tax_amount_minor (type INT), + line_weight_kg (type NUMBER(12, 3)), + has_list_price (type BOOLEAN), + ), +); + +WITH lines AS ( + SELECT * + FROM __ref("int_order_lines_discounted") +), + +raw_lines AS ( + SELECT * + FROM __ref("stg_order_lines") +), + +orders AS ( + SELECT * + FROM __ref("stg_orders") +), + +price_lists AS ( + SELECT * + FROM __ref("stg_price_lists") +), + +variants AS ( + SELECT * + FROM __ref("stg_product_variants") +), + +line_context AS ( + SELECT + l.order_line_id, + l.order_id, + l.sku, + l.quantity, + r.unit_amount_minor, + l.gross_amount_minor, + l.net_amount_minor, + l.discount_amount_minor, + l.tax_amount_minor, + o.currency_code, + o.placed_at, + o.sales_channel + FROM lines AS l + INNER JOIN raw_lines AS r + ON l.order_line_id = r.order_line_id + INNER JOIN orders AS o + ON l.order_id = o.order_id +), + +candidate_prices AS ( + SELECT + c.order_line_id, + p.list_amount_minor, + p.valid_from + FROM line_context AS c + INNER JOIN price_lists AS p + ON c.sku = p.sku + AND c.currency_code = p.currency_code + AND c.placed_at >= p.valid_from + AND ( + c.placed_at < p.valid_to + OR p.valid_to IS NULL + ) + WHERE + c.quantity >= p.min_quantity + AND ( + ARRAY_SIZE(p.channel_restrictions) = 0 + OR ARRAY_CONTAINS(c.sales_channel::VARIANT, p.channel_restrictions) + ) +), + +applied_prices AS ( + SELECT + cp.order_line_id, + MAX_BY(cp.list_amount_minor, cp.valid_from) AS list_amount_minor + FROM candidate_prices AS cp + GROUP BY + cp.order_line_id +), + +final AS ( + SELECT + c.order_line_id, + c.order_id, + c.sku, + __udf("udf__sku_family")(c.sku)::VARCHAR AS sku_family, + v.product_id, + c.quantity, + c.unit_amount_minor, + a.list_amount_minor::INT AS list_amount_minor, + ROUND(100 + * ( + 1 - c.unit_amount_minor / NULLIF(a.list_amount_minor, 0) + ), 4)::NUMBER(9, 4) AS markdown_pct, + c.gross_amount_minor, + c.net_amount_minor, + c.discount_amount_minor, + c.tax_amount_minor, + ( + c.quantity * COALESCE(v.weight_kg, 0) + )::NUMBER(12, 3) AS line_weight_kg, + ( + a.list_amount_minor IS NOT NULL + ) AS has_list_price + FROM line_context AS c + LEFT JOIN applied_prices AS a + ON c.order_line_id = a.order_line_id + LEFT JOIN variants AS v + ON c.sku = v.sku +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_order_payments_rollup.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_order_payments_rollup.sql new file mode 100644 index 0000000000..6b48725307 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_order_payments_rollup.sql @@ -0,0 +1,162 @@ +MODEL ( + description """ + One row per order with captured, refunded and charged-back amounts in minor units, pivoted by + payment method, plus gift card and store credit tender. + """, + columns ( + order_id (type VARCHAR, nullable false, audits [not_null, unique]), + card_captured_minor (type INT), + wallet_captured_minor (type INT), + bank_transfer_captured_minor (type INT), + other_captured_minor (type INT), + captured_minor (type INT), + refunded_minor (type INT), + chargeback_minor (type INT), + gift_card_tender_minor (type INT), + net_collected_minor (type INT), + failed_attempts (type INT), + tender_methods (type ARRAY), + ), +); + +WITH attempts AS ( + SELECT * + FROM __ref("int_payment_attempts") +), + +refunds AS ( + SELECT * + FROM __ref("stg_refunds") +), + +chargebacks AS ( + SELECT * + FROM __ref("stg_chargebacks") +), + +gift_card_transactions AS ( + SELECT * + FROM __ref("stg_gift_card_transactions") +), + +successful AS ( + SELECT + a.order_id, + CASE + WHEN a.payment_method IN ('card', 'credit_card', 'debit_card') + THEN 'CARD' + WHEN a.payment_method IN ('wallet', 'phone_wallet', 'browser_wallet', 'account_wallet') + THEN 'WALLET' + WHEN a.payment_method IN ('bank_transfer', 'sepa', 'bank_redirect') + THEN 'BANK' + ELSE 'OTHER' + END AS method_group, + a.amount_minor + FROM attempts AS a + WHERE a.is_successful +), + +captured_by_method AS ( + SELECT + p.order_id, + COALESCE(p.card, 0) AS card_captured_minor, + COALESCE(p.wallet, 0) AS wallet_captured_minor, + COALESCE(p.bank, 0) AS bank_transfer_captured_minor, + COALESCE(p.other, 0) AS other_captured_minor + FROM successful PIVOT( + SUM(amount_minor) FOR method_group IN ('CARD', 'WALLET', 'BANK', 'OTHER') + ) AS p(order_id, card, wallet, bank, other) +), + +attempt_stats AS ( + SELECT + a.order_id, + COUNT_IF(NOT a.is_successful) AS failed_attempts, + ARRAY_AGG(DISTINCT a.payment_method) WITHIN GROUP (ORDER BY a.payment_method) AS tender_methods + FROM attempts AS a + GROUP BY + a.order_id +), + +payment_orders AS ( + SELECT + a.payment_id, + a.order_id + FROM attempts AS a +), + +refunded AS ( + SELECT + po.order_id, + SUM(r.amount_minor) AS refunded_minor + FROM refunds AS r + INNER JOIN payment_orders AS po + ON r.payment_id = po.payment_id + GROUP BY + po.order_id +), + +charged_back AS ( + SELECT + po.order_id, + SUM(c.amount_minor) AS chargeback_minor + FROM chargebacks AS c + INNER JOIN payment_orders AS po + ON c.payment_id = po.payment_id + WHERE c.outcome <> 'won' + GROUP BY + po.order_id +), + +gift_card_tender AS ( + SELECT + g.order_id, + -SUM(g.signed_amount_minor) AS gift_card_tender_minor + FROM gift_card_transactions AS g + WHERE + g.transaction_type IN ('REDEEM', 'REVERSAL') + AND g.order_id IS NOT NULL + GROUP BY + g.order_id +), + +final AS ( + SELECT + s.order_id, + COALESCE(m.card_captured_minor, 0)::INT AS card_captured_minor, + COALESCE(m.wallet_captured_minor, 0)::INT AS wallet_captured_minor, + COALESCE(m.bank_transfer_captured_minor, 0)::INT AS bank_transfer_captured_minor, + COALESCE(m.other_captured_minor, 0)::INT AS other_captured_minor, + ( + COALESCE(m.card_captured_minor, 0) + + COALESCE(m.wallet_captured_minor, 0) + + COALESCE(m.bank_transfer_captured_minor, 0) + + COALESCE(m.other_captured_minor, 0) + )::INT AS captured_minor, + COALESCE(r.refunded_minor, 0)::INT AS refunded_minor, + COALESCE(cb.chargeback_minor, 0)::INT AS chargeback_minor, + COALESCE(g.gift_card_tender_minor, 0)::INT AS gift_card_tender_minor, + ( + COALESCE(m.card_captured_minor, 0) + + COALESCE(m.wallet_captured_minor, 0) + + COALESCE(m.bank_transfer_captured_minor, 0) + + COALESCE(m.other_captured_minor, 0) + + COALESCE(g.gift_card_tender_minor, 0) + - COALESCE(r.refunded_minor, 0) + - COALESCE(cb.chargeback_minor, 0) + )::INT AS net_collected_minor, + s.failed_attempts::INT AS failed_attempts, + s.tender_methods::ARRAY AS tender_methods + FROM attempt_stats AS s + LEFT JOIN captured_by_method AS m + ON s.order_id = m.order_id + LEFT JOIN refunded AS r + ON s.order_id = r.order_id + LEFT JOIN charged_back AS cb + ON s.order_id = cb.order_id + LEFT JOIN gift_card_tender AS g + ON s.order_id = g.order_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_order_status_timeline.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_order_status_timeline.sql new file mode 100644 index 0000000000..650502e7fa --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_order_status_timeline.sql @@ -0,0 +1,129 @@ +MODEL ( + description """ + One row per order summarising its status history: when each milestone was first reached, time + spent in each status, the path taken, and whether the status ever moved backwards. + """, + constants ( + _status_rank constant(value (pending 1, paid 2, packed 3, shipped 4, delivered 5, returned 6, cancelled 7)), + ), + columns ( + order_id (type VARCHAR, nullable false, audits [not_null, unique]), + transition_count (type INT), + first_paid_at (type TIMESTAMP_TZ), + first_shipped_at (type TIMESTAMP_TZ), + first_delivered_at (type TIMESTAMP_TZ), + cancelled_at (type TIMESTAMP_TZ), + minutes_to_pay (type INT), + hours_paid_to_shipped (type NUMBER(12, 2)), + status_path (type VARCHAR), + has_regression (type BOOLEAN), + last_status (type VARCHAR), + longest_status (type VARCHAR), + ), +); + +WITH status_events AS ( + SELECT * + FROM __ref("stg_order_status_events") +), + +orders AS ( + SELECT * + FROM __ref("stg_orders") +), + +ranked_events AS ( + SELECT + e.event_id, + e.order_id, + e.to_status, + e.changed_at, + @const("_status_rank")[e.to_status]::INT AS status_rank, + LEAD(e.changed_at) OVER ( + PARTITION BY e.order_id + ORDER BY e.changed_at, e.event_id + ) AS next_changed_at + FROM status_events AS e +), + +with_previous AS ( + SELECT + r.event_id, + r.order_id, + r.to_status, + r.changed_at, + r.status_rank, + r.next_changed_at, + LAG(r.status_rank) OVER ( + PARTITION BY r.order_id + ORDER BY r.changed_at, r.event_id + ) AS previous_rank + FROM ranked_events AS r +), + +durations AS ( + SELECT + w.order_id, + w.to_status, + SUM( + DATEDIFF('minute', w.changed_at, COALESCE(w.next_changed_at, w.changed_at)) + ) AS minutes_in_status + FROM with_previous AS w + GROUP BY + w.order_id, + w.to_status +), + +longest AS ( + SELECT + d.order_id, + MAX_BY(d.to_status, d.minutes_in_status) AS longest_status + FROM durations AS d + GROUP BY + d.order_id +), + +milestones AS ( + SELECT + w.order_id, + COUNT(*) AS transition_count, + MIN(IFF(w.to_status = 'paid', w.changed_at, NULL)) AS first_paid_at, + MIN(IFF(w.to_status = 'shipped', w.changed_at, NULL)) AS first_shipped_at, + MIN(IFF(w.to_status = 'delivered', w.changed_at, NULL)) AS first_delivered_at, + MAX(IFF(w.to_status = 'cancelled', w.changed_at, NULL)) AS cancelled_at, + LISTAGG(w.to_status, ' > ') WITHIN GROUP (ORDER BY w.changed_at, w.event_id) AS status_path, + BOOLOR_AGG( + w.status_rank < w.previous_rank AND w.to_status NOT IN ('cancelled', 'returned') + ) AS has_regression, + MAX_BY(w.to_status, w.changed_at) AS last_status + FROM with_previous AS w + GROUP BY + w.order_id +), + +final AS ( + SELECT + m.order_id, + m.transition_count::INT AS transition_count, + m.first_paid_at, + m.first_shipped_at, + m.first_delivered_at, + m.cancelled_at, + DATEDIFF('minute', o.placed_at, m.first_paid_at)::INT AS minutes_to_pay, + ROUND( + DATEDIFF('minute', m.first_paid_at, m.first_shipped_at) / 60, + 2 + )::NUMBER(12, 2) AS hours_paid_to_shipped, + m.status_path::VARCHAR AS status_path, + COALESCE(m.has_regression, FALSE) AS has_regression, + m.last_status::VARCHAR AS last_status, + l.longest_status::VARCHAR AS longest_status + FROM milestones AS m + INNER JOIN orders AS o + ON m.order_id = o.order_id + LEFT JOIN longest AS l + ON m.order_id = l.order_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_orders_enriched.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_orders_enriched.sql new file mode 100644 index 0000000000..2418c28505 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_orders_enriched.sql @@ -0,0 +1,190 @@ +MODEL ( + description """ + One row per order with its line totals, payment position, customer, shipping destination, + storefront region and local business date, ready for currency conversion. + """, + enums ( + _basket_size [SINGLE, SMALL, LARGE], + ), + constants ( + _large_basket_units 6, + ), + columns ( + order_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR, audits [not_null]), + storefront_code (type VARCHAR), + region_code (type VARCHAR), + storefront_timezone (type VARCHAR), + currency_code (type VARCHAR, audits [not_null]), + placed_at (type TIMESTAMP_TZ), + placed_date_local (type DATE), + order_status (type VARCHAR), + sales_channel (type VARCHAR), + line_count (type INT), + units (type INT), + gross_amount_minor (type INT), + discount_amount_minor (type INT), + net_amount_minor (type INT), + tax_amount_minor (type INT), + captured_amount_minor (type INT), + outstanding_amount_minor (type INT), + promo_code (type VARCHAR), + ship_to_country_code (type VARCHAR), + is_cross_border (type BOOLEAN), + is_first_party_customer (type BOOLEAN), + basket_size (type VARCHAR), + is_gift (type BOOLEAN), + utm_source (type VARCHAR), + utm_medium (type VARCHAR), + utm_campaign (type VARCHAR), + total_weight_kg (type NUMBER(12, 3)), + ), +); + +WITH orders AS ( + SELECT * + FROM __ref("stg_orders") +), + +lines AS ( + SELECT * + FROM __ref("int_order_lines_priced") +), + +customers AS ( + SELECT * + FROM __ref("stg_customers") +), + +payments AS ( + SELECT * + FROM __ref("stg_payments") +), + +addresses AS ( + SELECT * + FROM __ref("stg_customer_addresses") +), + +redemptions AS ( + SELECT * + FROM __ref("stg_promotion_redemptions") +), + +countries AS ( + SELECT * + FROM __seed("countries") +), + +line_totals AS ( + SELECT + l.order_id, + COUNT(*) AS line_count, + SUM(l.quantity) AS units, + SUM(l.gross_amount_minor) AS gross_amount_minor, + SUM(l.discount_amount_minor) AS discount_amount_minor, + SUM(l.net_amount_minor) AS net_amount_minor, + SUM(l.tax_amount_minor) AS tax_amount_minor, + SUM(l.line_weight_kg) AS total_weight_kg + FROM lines AS l + GROUP BY + l.order_id +), + +captured AS ( + SELECT + p.order_id, + SUM( + IFF(p.gateway_status IN ('captured', 'settled'), p.amount_minor, 0) + ) AS captured_amount_minor + FROM payments AS p + GROUP BY + p.order_id +), + +first_redemption AS ( + SELECT + r.order_id, + r.promo_code + FROM redemptions AS r + QUALIFY ROW_NUMBER() OVER (PARTITION BY r.order_id ORDER BY r.redeemed_at, r.redemption_id) = 1 +), + +shipping AS ( + SELECT + o.order_id, + a.country_code AS ship_to_country_code + FROM orders AS o + INNER JOIN addresses AS a + ON o.shipping_address_id = a.address_id +), + +storefronts AS ( + SELECT + c.country_code AS storefront_code, + c.region_code, + c.default_timezone + FROM countries AS c +), + +final AS ( + SELECT + o.order_id, + o.customer_id, + o.storefront_code, + COALESCE(s.region_code, cu.region_code)::VARCHAR AS region_code, + COALESCE(s.default_timezone, 'UTC')::VARCHAR AS storefront_timezone, + o.currency_code, + o.placed_at, + @sql.local_business_date(o.placed_at, COALESCE(s.default_timezone, 'UTC')) AS placed_date_local, + o.order_status, + COALESCE(o.sales_channel, @enum("sales_channel").WEB)::VARCHAR AS sales_channel, + COALESCE(t.line_count, 0)::INT AS line_count, + COALESCE(t.units, 0)::INT AS units, + COALESCE(t.gross_amount_minor, 0)::INT AS gross_amount_minor, + COALESCE(t.discount_amount_minor, 0)::INT AS discount_amount_minor, + COALESCE(t.net_amount_minor, 0)::INT AS net_amount_minor, + COALESCE(t.tax_amount_minor, 0)::INT AS tax_amount_minor, + COALESCE(c.captured_amount_minor, 0)::INT AS captured_amount_minor, + GREATEST( + COALESCE(t.net_amount_minor, 0) + + COALESCE(t.tax_amount_minor, 0) + - COALESCE(c.captured_amount_minor, 0), + 0 + )::INT AS outstanding_amount_minor, + r.promo_code, + sh.ship_to_country_code, + COALESCE(sh.ship_to_country_code <> o.storefront_code, FALSE) AS is_cross_border, + NOT RLIKE( + COALESCE(cu.email_normalised, ''), + '.*@example\\.(org|net)$' + ) AS is_first_party_customer, + CASE + WHEN COALESCE(t.units, 0) <= 1 + THEN @enum("_basket_size").SINGLE + WHEN t.units < @const("_large_basket_units") + THEN @enum("_basket_size").SMALL + ELSE @enum("_basket_size").LARGE + END::VARCHAR AS basket_size, + COALESCE(o.is_gift, FALSE) AS is_gift, + NULLIF(LOWER(o.utm_source), '')::VARCHAR AS utm_source, + NULLIF(LOWER(o.utm_medium), '')::VARCHAR AS utm_medium, + NULLIF(LOWER(o.utm_campaign), '')::VARCHAR AS utm_campaign, + COALESCE(t.total_weight_kg, 0)::NUMBER(12, 3) AS total_weight_kg + FROM orders AS o + LEFT JOIN line_totals AS t + ON o.order_id = t.order_id + LEFT JOIN captured AS c + ON o.order_id = c.order_id + LEFT JOIN first_redemption AS r + ON o.order_id = r.order_id + LEFT JOIN shipping AS sh + ON o.order_id = sh.order_id + LEFT JOIN storefronts AS s + ON o.storefront_code = s.storefront_code + LEFT JOIN customers AS cu + ON o.customer_id = cu.customer_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_orders_usd.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_orders_usd.sql new file mode 100644 index 0000000000..3e66badbda --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_orders_usd.sql @@ -0,0 +1,150 @@ +MODEL ( + description """ + One row per order with amounts converted to major units and to USD at the latest reference rate + on or before the local order date (an ASOF join against daily rates). + """, + audits [ + match_rate ( + name "fx_rate_found", + matched "rate_to_usd IS NOT NULL", + minimum_samples 50, + evidence_limit 20, + thresholds (warn (below 99.5), error (below 95)), + ), + ], + columns ( + order_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR), + region_code (type VARCHAR), + currency_code (type VARCHAR), + placed_at (type TIMESTAMP_TZ), + placed_date_local (type DATE), + order_status (type VARCHAR), + sales_channel (type VARCHAR), + units (type INT), + amount (type NUMBER(38, 6)), + amount_usd (type NUMBER(18, 2)), + gross_amount_usd (type NUMBER(18, 2)), + discount_amount_usd (type NUMBER(18, 2)), + tax_amount_usd (type NUMBER(18, 2)), + rate_to_usd (type NUMBER(18, 8)), + rate_date (type DATE), + rate_age_days (type INT), + is_rate_stale (type BOOLEAN), + promo_code (type VARCHAR), + utm_source (type VARCHAR), + utm_medium (type VARCHAR), + utm_campaign (type VARCHAR), + basket_size (type VARCHAR), + ), +); + +WITH orders AS ( + SELECT * + FROM __ref("int_orders_enriched") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +orders_with_exponent AS ( + SELECT + o.order_id, + o.customer_id, + o.region_code, + o.currency_code, + o.placed_at, + o.placed_date_local, + o.order_status, + o.sales_channel, + o.units, + o.net_amount_minor AS amount_minor, + o.gross_amount_minor, + o.discount_amount_minor, + o.tax_amount_minor, + o.promo_code, + o.utm_source, + o.utm_medium, + o.utm_campaign, + o.basket_size, + c.minor_unit_exponent + FROM orders AS o + INNER JOIN currencies AS c + ON o.currency_code = c.currency_code +), + +orders_with_rate AS ( + SELECT + o.order_id, + o.customer_id, + o.region_code, + o.currency_code, + o.placed_at, + o.placed_date_local, + o.order_status, + o.sales_channel, + o.units, + o.amount_minor, + o.gross_amount_minor, + o.discount_amount_minor, + o.tax_amount_minor, + o.promo_code, + o.utm_source, + o.utm_medium, + o.utm_campaign, + o.basket_size, + o.minor_unit_exponent, + fx.rate_to_usd, + fx.rate_date + FROM orders_with_exponent AS o + ASOF JOIN fx_rates AS fx MATCH_CONDITION (o.placed_date_local >= fx.rate_date) + ON o.currency_code = fx.currency_code +), + +final AS ( + SELECT + r.order_id, + r.customer_id, + r.region_code, + r.currency_code, + r.placed_at, + r.placed_date_local, + r.order_status, + r.sales_channel, + r.units, + @sql.money_columns(r), + ROUND( + @sql.money_from_minor(r.gross_amount_minor, r.minor_unit_exponent, places := 6) * r.rate_to_usd, + 2 + )::NUMBER(18, 2) AS gross_amount_usd, + ROUND( + @sql.money_from_minor(r.discount_amount_minor, r.minor_unit_exponent, places := 6) * r.rate_to_usd, + 2 + )::NUMBER(18, 2) AS discount_amount_usd, + ROUND( + @sql.money_from_minor(r.tax_amount_minor, r.minor_unit_exponent, places := 6) * r.rate_to_usd, + 2 + )::NUMBER(18, 2) AS tax_amount_usd, + r.rate_to_usd, + r.rate_date, + DATEDIFF('day', r.rate_date, r.placed_date_local)::INT AS rate_age_days, + ( + DATEDIFF('day', r.rate_date, r.placed_date_local) > 7 + ) AS is_rate_stale, + r.promo_code, + r.utm_source, + r.utm_medium, + r.utm_campaign, + r.basket_size + FROM orders_with_rate AS r +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_payment_attempts.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_payment_attempts.sql new file mode 100644 index 0000000000..43429bbead --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_payment_attempts.sql @@ -0,0 +1,161 @@ +MODEL ( + description """ + One row per payment attempt with its position among the order's attempts, the time since the + previous attempt, a normalised decline reason and card checks. + """, + enums ( + _decline_family [NONE, INSUFFICIENT_FUNDS, SUSPECTED_FRAUD, AUTHENTICATION, TECHNICAL, OTHER], + ), + columns ( + payment_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + attempt_number (type INT, audits [not_null]), + attempt_count (type INT), + is_final_attempt (type BOOLEAN), + payment_method (type VARCHAR), + amount_minor (type INT), + currency_code (type VARCHAR), + authorised_at (type TIMESTAMP_TZ), + seconds_since_previous_attempt (type INT), + gateway_status (type VARCHAR), + is_successful (type BOOLEAN), + decline_family (type VARCHAR), + gateway_error_code (type VARCHAR), + risk_score (type NUMBER(5, 2)), + risk_band (type VARCHAR), + three_ds_outcome (type VARCHAR), + card_bin_valid (type BOOLEAN), + card_scheme (type VARCHAR), + had_chargeback (type BOOLEAN), + ), +); + +WITH payments AS ( + SELECT * + FROM __ref("stg_payments") +), + +chargebacks AS ( + SELECT * + FROM __ref("stg_chargebacks") +), + +sequenced AS ( + SELECT + p.payment_id, + p.order_id, + p.payment_method, + p.amount_minor, + p.currency_code, + p.authorised_at, + p.gateway_status, + p.gateway_error_code, + p.gateway_response, + p.risk_score, + p.three_ds_outcome, + p.card_bin, + ROW_NUMBER() OVER ( + PARTITION BY p.order_id + ORDER BY p.authorised_at, p.payment_id + ) AS attempt_number, + COUNT(*) OVER (PARTITION BY p.order_id) AS attempt_count, + LAG(p.authorised_at) OVER ( + PARTITION BY p.order_id + ORDER BY p.authorised_at, p.payment_id + ) AS previous_authorised_at + FROM payments AS p +), + +decline_reasons AS ( + SELECT + s.payment_id, + LOWER(COALESCE(s.gateway_response:status.message::VARCHAR, '')) AS decline_message + FROM sequenced AS s + WHERE s.gateway_status NOT IN ('captured', 'settled', 'authorised') +), + +classified AS ( + SELECT + d.payment_id, + CASE + WHEN REGEXP_LIKE(d.decline_message, '.*(insufficient|nsf|limit exceeded).*') + THEN @enum("_decline_family").INSUFFICIENT_FUNDS + WHEN REGEXP_LIKE(d.decline_message, '.*(fraud|stolen|lost card|pickup).*') + THEN @enum("_decline_family").SUSPECTED_FRAUD + WHEN REGEXP_LIKE(d.decline_message, '.*(3ds|authentication|challenge).*') + THEN @enum("_decline_family").AUTHENTICATION + WHEN REGEXP_LIKE(d.decline_message, '.*(timeout|unavailable|error [0-9]{3}).*') + THEN @enum("_decline_family").TECHNICAL + ELSE @enum("_decline_family").OTHER + END AS decline_family + FROM decline_reasons AS d +), + +chargeback_flags AS ( + SELECT + c.payment_id, + COUNT(*) AS chargeback_count + FROM chargebacks AS c + GROUP BY + c.payment_id +), + +final AS ( + SELECT + s.payment_id, + s.order_id, + s.attempt_number::INT AS attempt_number, + s.attempt_count::INT AS attempt_count, + ( + s.attempt_number = s.attempt_count + ) AS is_final_attempt, + s.payment_method, + s.amount_minor::INT AS amount_minor, + s.currency_code, + s.authorised_at, + DATEDIFF( + 'second', + s.previous_authorised_at, + s.authorised_at + )::INT AS seconds_since_previous_attempt, + s.gateway_status, + ( + s.gateway_status IN ('captured', 'settled', 'authorised') + ) AS is_successful, + COALESCE(c.decline_family, @enum("_decline_family").NONE)::VARCHAR AS decline_family, + s.gateway_error_code, + s.risk_score, + CASE + WHEN s.risk_score IS NULL + THEN 'unscored' + WHEN s.risk_score >= 75 + THEN 'high' + WHEN s.risk_score >= 40 + THEN 'medium' + ELSE 'low' + END::VARCHAR AS risk_band, + s.three_ds_outcome, + __udf("luhn_valid")(s.card_bin)::BOOLEAN AS card_bin_valid, + CASE + WHEN s.card_bin RLIKE '^4[0-9]*' + THEN 'network_a' + WHEN s.card_bin RLIKE '^(5[1-5]|2[2-7])[0-9]*' + THEN 'network_b' + WHEN s.card_bin RLIKE '^3[47][0-9]*' + THEN 'network_c' + WHEN s.card_bin IS NULL + THEN NULL + ELSE 'other' + END::VARCHAR AS card_scheme, + ( + COALESCE(cb.chargeback_count, 0) > 0 + ) AS had_chargeback + FROM sequenced AS s + LEFT JOIN classified AS c + ON s.payment_id = c.payment_id + LEFT JOIN chargeback_flags AS cb + ON s.payment_id = cb.payment_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_promotion_performance.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_promotion_performance.sql new file mode 100644 index 0000000000..f7838fcdda --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_promotion_performance.sql @@ -0,0 +1,117 @@ +MODEL ( + description """ + One row per promotion with redemption counts, discounted revenue in USD, the share of orders + that met the best tier, and incremental basket size against non-promotion orders. Whether a + promotion is active now is read from CURRENT_TIMESTAMP(), which is session-dependent and an + expected divergence between runs and engines. + """, + tags [known_divergence], + columns ( + promo_code (type VARCHAR, nullable false, audits [not_null, unique]), + promo_name (type VARCHAR), + is_active_now (type BOOLEAN), + redemption_count (type INT), + redeeming_customers (type INT), + revenue_usd (type NUMBER(18, 2)), + discount_usd (type NUMBER(18, 2)), + discount_rate (type NUMBER(9, 6)), + avg_order_usd (type NUMBER(18, 2)), + baseline_avg_order_usd (type NUMBER(18, 2)), + basket_uplift_pct (type NUMBER(9, 2)), + top_tier_threshold_minor (type INT), + eligible_region_count (type INT), + ), +); + +WITH promotions AS ( + SELECT * + FROM __ref("stg_promotions") +), + +redemptions AS ( + SELECT * + FROM __ref("stg_promotion_redemptions") +), + +orders_usd AS ( + SELECT * + FROM __ref("int_orders_usd") +), + +tiers AS ( + SELECT + p.promo_code, + t.value:threshold_minor::INT AS threshold_minor + FROM promotions AS p, LATERAL FLATTEN(input => p.rules:tiers, outer => TRUE) AS t +), + +top_tiers AS ( + SELECT + t.promo_code, + MAX(t.threshold_minor) AS top_tier_threshold_minor + FROM tiers AS t + GROUP BY + t.promo_code +), + +redeemed_orders AS ( + SELECT + r.promo_code, + r.order_id, + o.customer_id, + o.amount_usd, + o.discount_amount_usd + FROM redemptions AS r + INNER JOIN orders_usd AS o + ON r.order_id = o.order_id +), + +promo_totals AS ( + SELECT + ro.promo_code, + COUNT(DISTINCT ro.order_id) AS redemption_count, + COUNT(DISTINCT ro.customer_id) AS redeeming_customers, + SUM(ro.amount_usd) AS revenue_usd, + SUM(ro.discount_amount_usd) AS discount_usd, + AVG(ro.amount_usd) AS avg_order_usd + FROM redeemed_orders AS ro + GROUP BY + ro.promo_code +), + +baseline AS ( + SELECT AVG(o.amount_usd) AS baseline_avg_order_usd + FROM orders_usd AS o + WHERE o.promo_code IS NULL +), + +final AS ( + SELECT + p.promo_code, + p.promo_name, + ( + CURRENT_TIMESTAMP() BETWEEN p.starts_at AND COALESCE(p.ends_at, '9999-12-31'::TIMESTAMP_TZ) + ) AS is_active_now, + COALESCE(pt.redemption_count, 0)::INT AS redemption_count, + COALESCE(pt.redeeming_customers, 0)::INT AS redeeming_customers, + COALESCE(pt.revenue_usd, 0)::NUMBER(18, 2) AS revenue_usd, + COALESCE(pt.discount_usd, 0)::NUMBER(18, 2) AS discount_usd, + @sql.safe_divide(pt.discount_usd, pt.revenue_usd + pt.discount_usd)::NUMBER(9, 6) AS discount_rate, + ROUND(pt.avg_order_usd, 2)::NUMBER(18, 2) AS avg_order_usd, + ROUND(b.baseline_avg_order_usd, 2)::NUMBER(18, 2) AS baseline_avg_order_usd, + ROUND(100 + * ( + pt.avg_order_usd / NULLIF(b.baseline_avg_order_usd, 0) - 1 + ), 2)::NUMBER(9, 2) AS basket_uplift_pct, + tt.top_tier_threshold_minor::INT AS top_tier_threshold_minor, + ARRAY_SIZE(p.eligible_regions)::INT AS eligible_region_count + FROM promotions AS p + LEFT JOIN promo_totals AS pt + ON p.promo_code = pt.promo_code + LEFT JOIN top_tiers AS tt + ON p.promo_code = tt.promo_code + CROSS JOIN baseline AS b +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_refunds_usd.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_refunds_usd.sql new file mode 100644 index 0000000000..a36e013db0 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_refunds_usd.sql @@ -0,0 +1,121 @@ +MODEL ( + description """ + One row per refund in major units and USD at the reference rate on or before the refund date + """, + audits [ + match_rate ( + name "fx_rate_found", + matched "amount_usd IS NOT NULL", + minimum_samples 50, + evidence_limit 20, + thresholds (warn (below 99.5), error (below 95)), + ), + ], + columns ( + refund_id (type VARCHAR, nullable false, audits [not_null, unique]), + payment_id (type VARCHAR, audits [not_null]), + order_id (type VARCHAR), + currency_code (type VARCHAR), + refunded_at (type TIMESTAMP_TZ), + refunded_date (type DATE), + reason_category (type VARCHAR), + amount (type NUMBER(38, 6)), + amount_usd (type NUMBER(18, 2)), + days_after_payment (type INT), + is_full_refund (type BOOLEAN), + ), +); + +WITH refunds AS ( + SELECT * + FROM __ref("stg_refunds") +), + +payments AS ( + SELECT * + FROM __ref("stg_payments") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +refund_context AS ( + SELECT + r.refund_id, + r.payment_id, + p.order_id, + p.currency_code, + r.refunded_at, + CONVERT_TIMEZONE('UTC', r.refunded_at)::DATE AS refunded_date, + r.reason_category, + r.amount_minor, + p.amount_minor AS payment_amount_minor, + p.captured_at + FROM refunds AS r + INNER JOIN payments AS p + ON r.payment_id = p.payment_id +), + +with_exponent AS ( + SELECT + rc.refund_id, + rc.payment_id, + rc.order_id, + rc.currency_code, + rc.refunded_at, + rc.refunded_date, + rc.reason_category, + rc.amount_minor, + rc.payment_amount_minor, + rc.captured_at, + c.minor_unit_exponent + FROM refund_context AS rc + INNER JOIN currencies AS c + ON rc.currency_code = c.currency_code +), + +with_rate AS ( + SELECT + w.refund_id, + w.payment_id, + w.order_id, + w.currency_code, + w.refunded_at, + w.refunded_date, + w.reason_category, + w.amount_minor, + w.payment_amount_minor, + w.captured_at, + w.minor_unit_exponent, + fx.rate_to_usd + FROM with_exponent AS w + ASOF JOIN fx_rates AS fx MATCH_CONDITION (w.refunded_date >= fx.rate_date) + ON w.currency_code = fx.currency_code +), + +final AS ( + SELECT + r.refund_id, + r.payment_id, + r.order_id, + r.currency_code, + r.refunded_at, + r.refunded_date, + r.reason_category, + @sql.money_columns(r), + DATEDIFF('day', r.captured_at, r.refunded_at)::INT AS days_after_payment, + ( + r.amount_minor >= r.payment_amount_minor + ) AS is_full_refund + FROM with_rate AS r +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_reviews_moderated.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_reviews_moderated.sql new file mode 100644 index 0000000000..ccd3f9c616 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_reviews_moderated.sql @@ -0,0 +1,104 @@ +MODEL ( + description """ + One row per approved or pending product review with a verified-purchase flag, a keyword + sentiment score and its weight in the product's rating. + """, + constants ( + _positive_terms "(love|great|excellent|perfect|sturdy|beautiful)", + _negative_terms "(broke|broken|cheap|flimsy|disappoint|return|rust)", + ), + columns ( + review_id (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + product_id (type VARCHAR), + customer_id (type VARCHAR), + rating (type INT), + submitted_at (type TIMESTAMP_TZ), + is_verified_purchase (type BOOLEAN), + sentiment_score (type INT), + review_weight (type NUMBER(6, 3)), + rating_rank_in_product (type INT), + ), +); + +WITH reviews AS ( + SELECT * + FROM __ref("stg_reviews") +), + +order_lines AS ( + SELECT * + FROM __ref("int_order_lines_priced") +), + +orders AS ( + SELECT * + FROM __ref("stg_orders") +), + +variants AS ( + SELECT * + FROM __ref("stg_product_variants") +), + +purchases AS ( + SELECT DISTINCT + o.customer_id, + l.sku + FROM order_lines AS l + INNER JOIN orders AS o + ON l.order_id = o.order_id +), + +visible AS ( + SELECT + r.review_id, + r.sku, + r.customer_id, + r.rating, + r.title, + r.submitted_at, + r.body_length + FROM reviews AS r + WHERE + r.moderation_status IN ('approved', 'pending') + AND ARRAY_SIZE(r.flagged_terms) = 0 +), + +scored AS ( + SELECT + v.review_id, + REGEXP_COUNT(LOWER(v.title), @const("_positive_terms")) + - REGEXP_COUNT(LOWER(v.title), @const("_negative_terms")) AS sentiment_score + FROM visible AS v +), + +final AS ( + SELECT + v.review_id, + v.sku, + pv.product_id, + v.customer_id, + v.rating, + v.submitted_at, + ( + p.customer_id IS NOT NULL + ) AS is_verified_purchase, + s.sentiment_score::INT AS sentiment_score, + ROUND( + IFF(p.customer_id IS NOT NULL, 1.5, 1.0) * LEAST(1 + LN(1 + v.body_length / 100), 3), + 3 + )::NUMBER(6, 3) AS review_weight, + RANK() OVER (PARTITION BY v.sku ORDER BY v.rating DESC)::INT AS rating_rank_in_product + FROM visible AS v + INNER JOIN scored AS s + ON v.review_id = s.review_id + LEFT JOIN purchases AS p + ON v.customer_id = p.customer_id + AND v.sku = p.sku + LEFT JOIN variants AS pv + ON v.sku = pv.sku +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_subscription_mrr_daily.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_subscription_mrr_daily.sql new file mode 100644 index 0000000000..b9de75b5df --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_subscription_mrr_daily.sql @@ -0,0 +1,99 @@ +MODEL ( + description """ + One row per subscription and day it was active, with its monthly recurring revenue in USD + normalised from its billing interval. + """, + columns ( + subscription_day_key (type VARCHAR, nullable false, audits [not_null, unique]), + subscription_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + calendar_date (type DATE, audits [not_null]), + mrr_usd (type NUMBER(18, 4)), + is_first_day (type BOOLEAN), + ), +); + +WITH periods AS ( + SELECT * + FROM __ref("int_subscription_periods") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +@sql.date_spine(@const("history_start_date"), 730) + +active_periods AS ( + SELECT + p.subscription_id, + p.customer_id, + p.period_start::DATE AS start_date, + COALESCE(p.period_end::DATE, '2026-12-31'::DATE) AS end_date, + p.billing_amount_minor, + p.billing_currency_code, + p.interval_days + FROM periods AS p + WHERE p.period_state = 'active' +), active_days AS ( + SELECT + a.subscription_id, + a.customer_id, + d.calendar_date, + a.billing_amount_minor, + a.billing_currency_code, + a.interval_days + FROM active_periods AS a + INNER JOIN date_spine AS d + ON d.calendar_date >= a.start_date + AND d.calendar_date < a.end_date +), with_exponent AS ( + SELECT + ad.subscription_id, + ad.customer_id, + ad.calendar_date, + ad.billing_amount_minor, + ad.billing_currency_code, + ad.interval_days, + c.minor_unit_exponent + FROM active_days AS ad + INNER JOIN currencies AS c + ON ad.billing_currency_code = c.currency_code +), with_rate AS ( + SELECT + w.subscription_id, + w.customer_id, + w.calendar_date, + w.billing_amount_minor, + w.interval_days, + w.minor_unit_exponent, + fx.rate_to_usd + FROM with_exponent AS w + ASOF JOIN fx_rates AS fx MATCH_CONDITION (w.calendar_date >= fx.rate_date) + ON w.billing_currency_code = fx.currency_code +), final AS ( + SELECT + ( + r.subscription_id || ':' || TO_CHAR(r.calendar_date, 'YYYYMMDD') + )::VARCHAR AS subscription_day_key, + r.subscription_id, + r.customer_id, + r.calendar_date, + ROUND( + @sql.money_from_minor(r.billing_amount_minor, r.minor_unit_exponent, places := 6) * r.rate_to_usd * 30.4375 + / NULLIF(r.interval_days, 0), + 4 + )::NUMBER(18, 4) AS mrr_usd, + ( + r.calendar_date = MIN(r.calendar_date) OVER (PARTITION BY r.subscription_id) + ) AS is_first_day + FROM with_rate AS r +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/orders/int_subscription_periods.sql b/benchmarks/duckshop/project/models/intermediate/orders/int_subscription_periods.sql new file mode 100644 index 0000000000..9044a171d5 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/orders/int_subscription_periods.sql @@ -0,0 +1,162 @@ +MODEL ( + description """ + One row per active, paused or cancelled period of a subscription, built as gaps-and-islands over + its lifecycle events: the subscription starts active, and every change of state starts a new + period. + """, + columns ( + subscription_period_key (type VARCHAR, nullable false, audits [not_null, unique]), + subscription_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + period_number (type INT), + period_state (type VARCHAR, audits [accepted_values (values ["active", "paused", "cancelled"])]), + period_start (type TIMESTAMP_TZ), + period_end (type TIMESTAMP_TZ), + period_days (type NUMBER(10, 2)), + billing_amount_minor (type INT), + billing_currency_code (type VARCHAR), + interval_days (type INT), + ), +); + +WITH subscriptions AS ( + SELECT * + FROM __ref("stg_subscriptions") +), + +events AS ( + SELECT * + FROM __ref("stg_subscription_events") +), + +state_events AS ( + SELECT + e.subscription_id, + e.event_id, + e.occurred_at, + CASE e.event_type + WHEN 'PAUSE' + THEN 'paused' + WHEN 'CANCEL' + THEN 'cancelled' + ELSE 'active' + END AS state + FROM events AS e + WHERE e.event_type IN ('PAUSE', 'RESUME', 'CANCEL') +), + +changes AS ( + SELECT + s.subscription_id, + s.event_id, + s.occurred_at, + s.state, + LAG(s.state, 1, 'active') OVER ( + PARTITION BY s.subscription_id + ORDER BY s.occurred_at, s.event_id + ) AS previous_state + FROM state_events AS s +), + +islands AS ( + SELECT + c.subscription_id, + c.occurred_at, + c.state, + SUM(IFF(c.previous_state <> c.state, 1, 0)) OVER ( + PARTITION BY c.subscription_id + ORDER BY c.occurred_at, c.event_id + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) AS island_number + FROM changes AS c +), + +event_periods AS ( + SELECT + i.subscription_id, + i.island_number, + ANY_VALUE(i.state) AS period_state, + MIN(i.occurred_at) AS period_start + FROM islands AS i + WHERE i.island_number >= 1 + GROUP BY + i.subscription_id, + i.island_number +), + +bounded AS ( + SELECT + p.subscription_id, + p.island_number + 1 AS period_number, + p.period_state, + p.period_start, + LEAD(p.period_start) OVER ( + PARTITION BY p.subscription_id + ORDER BY p.island_number + ) AS period_end + FROM event_periods AS p +), + +first_change AS ( + SELECT + p.subscription_id, + MIN(p.period_start) AS first_change_at + FROM event_periods AS p + GROUP BY + p.subscription_id +), + +final AS ( + SELECT + ( + s.subscription_id || '#1' + )::VARCHAR AS subscription_period_key, + s.subscription_id::VARCHAR AS subscription_id, + s.customer_id::VARCHAR AS customer_id, + 1::INT AS period_number, + 'active'::VARCHAR AS period_state, + s.started_at::TIMESTAMP_TZ AS period_start, + COALESCE(f.first_change_at, s.cancelled_at)::TIMESTAMP_TZ AS period_end, + ROUND( + DATEDIFF('hour', s.started_at, COALESCE(f.first_change_at, s.cancelled_at)) / 24, + 2 + )::NUMBER(10, 2) AS period_days, + s.billing_amount_minor::INT AS billing_amount_minor, + s.billing_currency_code::VARCHAR AS billing_currency_code, + s.interval_days::INT AS interval_days + FROM subscriptions AS s + LEFT JOIN first_change AS f + ON s.subscription_id = f.subscription_id + UNION ALL + SELECT + ( + b.subscription_id || '#' || b.period_number::VARCHAR + )::VARCHAR AS subscription_period_key, + b.subscription_id::VARCHAR AS subscription_id, + s.customer_id::VARCHAR AS customer_id, + b.period_number::INT AS period_number, + b.period_state::VARCHAR AS period_state, + b.period_start::TIMESTAMP_TZ AS period_start, + COALESCE( + b.period_end, + IFF(b.period_state = 'cancelled', b.period_start, s.cancelled_at) + )::TIMESTAMP_TZ AS period_end, + ROUND( + DATEDIFF( + 'hour', + b.period_start, + COALESCE(b.period_end, IFF(b.period_state = 'cancelled', b.period_start, s.cancelled_at)) + ) + / 24, + 2 + )::NUMBER(10, 2) AS period_days, + s.billing_amount_minor::INT AS billing_amount_minor, + s.billing_currency_code::VARCHAR AS billing_currency_code, + s.interval_days::INT AS interval_days + FROM bounded AS b + INNER JOIN subscriptions AS s + ON b.subscription_id = s.subscription_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/partners/_sqlbuild/_constants/default_currency.sql b/benchmarks/duckshop/project/models/intermediate/partners/_sqlbuild/_constants/default_currency.sql new file mode 100644 index 0000000000..3c39d76160 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/partners/_sqlbuild/_constants/default_currency.sql @@ -0,0 +1 @@ +CONSTANT (name default_currency, value "USD"); diff --git a/benchmarks/duckshop/project/models/intermediate/partners/_sqlbuild/_macros/python/relations.py b/benchmarks/duckshop/project/models/intermediate/partners/_sqlbuild/_macros/python/relations.py new file mode 100644 index 0000000000..06ab88e16c --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/partners/_sqlbuild/_macros/python/relations.py @@ -0,0 +1,13 @@ +"""Helpers that take typed relation references.""" + +from sqlbuild.refs import SqlResourceRef + + +def union_by_columns(relations: list[SqlResourceRef], columns: list[str]) -> str: + """UNION ALL of the same named columns from each relation, tagged with the relation name.""" + projection = ", ".join(columns) + return "\nUNION ALL\n".join( + f"SELECT CAST('{relation.name}' AS VARCHAR) AS source_relation, {projection} " + f"FROM {relation}" + for relation in relations + ) diff --git a/benchmarks/duckshop/project/models/intermediate/partners/int_partner_fees_usd.sql b/benchmarks/duckshop/project/models/intermediate/partners/int_partner_fees_usd.sql new file mode 100644 index 0000000000..5e789e1c67 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/partners/int_partner_fees_usd.sql @@ -0,0 +1,204 @@ +MODEL ( + description """ + One row per partner settlement statement with the partner's stated commission compared with the + commission implied by each order line in the period and the listing's agreed commission rate, + all converted to USD at the rate on the period end. + """, + constants ( + _variance_tolerance_pct 0.5, + ), + columns ( + settlement_id (type VARCHAR, nullable false, audits [not_null, unique]), + partner_code (type VARCHAR, audits [not_null]), + period_start (type DATE), + period_end (type DATE), + period_days (type INT), + currency_code (type VARCHAR), + gross_sales_usd (type NUMBER(18, 2)), + stated_commission_usd (type NUMBER(18, 2)), + expected_commission_usd (type NUMBER(18, 2)), + other_fees_usd (type NUMBER(18, 2)), + net_payable_usd (type NUMBER(18, 2)), + period_order_count (type INT), + lines_without_listing (type INT), + effective_commission_pct (type NUMBER(9, 4)), + commission_variance_pct (type NUMBER(9, 4)), + is_within_tolerance (type BOOLEAN), + statement_balances (type BOOLEAN), + ), +); + +WITH settlements AS ( + SELECT * + FROM __ref("stg_partner_settlements") +), + +orders AS ( + SELECT * + FROM __ref("int_partner_orders_unified") +), + +order_lines AS ( + SELECT * + FROM __ref("stg_partner_order_lines") +), + +listings AS ( + SELECT * + FROM __ref("stg_partner_listings") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +listing_terms AS ( + SELECT + l.partner_code, + l.sku, + l.commission_pct + FROM listings AS l + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY l.partner_code, l.sku + ORDER BY l.listed_at DESC NULLS LAST, l.listing_id DESC + ) = 1 +), + +line_context AS ( + SELECT + o.partner_code, + o.placed_date, + o.partner_order_key, + l.sku, + l.quantity * l.unit_amount_minor AS line_amount_minor + FROM order_lines AS l + INNER JOIN orders AS o + ON l.partner_order_key = o.partner_order_key +), + +line_commission AS ( + SELECT + c.partner_code, + c.placed_date, + c.partner_order_key, + c.line_amount_minor * t.commission_pct / 100 AS expected_commission_minor, + ( + t.commission_pct IS NULL + ) AS has_no_listing + FROM line_context AS c + LEFT JOIN listing_terms AS t + ON c.partner_code = t.partner_code + AND c.sku = t.sku +), + +period_lines AS ( + SELECT + s.settlement_id, + SUM(c.expected_commission_minor) AS expected_commission_minor, + COUNT(DISTINCT c.partner_order_key) AS period_order_count, + COUNT_IF(c.has_no_listing) AS lines_without_listing + FROM settlements AS s + INNER JOIN line_commission AS c + ON s.partner_code = c.partner_code + AND c.placed_date BETWEEN s.period_start AND s.period_end + GROUP BY + s.settlement_id +), + +with_rate AS ( + SELECT + s.settlement_id, + s.partner_code, + s.period_start, + s.period_end, + s.currency_code, + s.gross_sales_minor, + s.commission_minor, + s.other_fees_minor, + s.net_payable_minor, + cur.minor_unit_exponent, + fx.rate_to_usd + FROM settlements AS s + LEFT JOIN currencies AS cur + ON s.currency_code = cur.currency_code + ASOF JOIN fx_rates AS fx MATCH_CONDITION (s.period_end >= fx.rate_date) + ON s.currency_code = fx.currency_code +), + +converted AS ( + SELECT + w.settlement_id, + w.partner_code, + w.period_start, + w.period_end, + w.currency_code, + w.gross_sales_minor, + w.commission_minor, + w.other_fees_minor, + w.net_payable_minor, + @sql.money_from_minor(w.gross_sales_minor, COALESCE(w.minor_unit_exponent, 2), places := 6) * w.rate_to_usd AS gross_sales_usd, + @sql.money_from_minor(w.commission_minor, COALESCE(w.minor_unit_exponent, 2), places := 6) * w.rate_to_usd AS stated_commission_usd, + @sql.money_from_minor(p.expected_commission_minor, COALESCE(w.minor_unit_exponent, 2), places := 6) * w.rate_to_usd AS expected_commission_usd, + @sql.money_from_minor(w.other_fees_minor, COALESCE(w.minor_unit_exponent, 2), places := 6) * w.rate_to_usd AS other_fees_usd, + @sql.money_from_minor(w.net_payable_minor, COALESCE(w.minor_unit_exponent, 2), places := 6) * w.rate_to_usd AS net_payable_usd, + COALESCE(p.period_order_count, 0) AS period_order_count, + COALESCE(p.lines_without_listing, 0) AS lines_without_listing + FROM with_rate AS w + LEFT JOIN period_lines AS p + ON w.settlement_id = p.settlement_id +), + +final AS ( + SELECT + c.settlement_id, + c.partner_code, + c.period_start, + c.period_end, + ( + DATEDIFF('day', c.period_start, c.period_end) + 1 + )::INT AS period_days, + c.currency_code, + ROUND(c.gross_sales_usd, 2)::NUMBER(18, 2) AS gross_sales_usd, + ROUND(c.stated_commission_usd, 2)::NUMBER(18, 2) AS stated_commission_usd, + ROUND(c.expected_commission_usd, 2)::NUMBER(18, 2) AS expected_commission_usd, + ROUND(c.other_fees_usd, 2)::NUMBER(18, 2) AS other_fees_usd, + ROUND(c.net_payable_usd, 2)::NUMBER(18, 2) AS net_payable_usd, + c.period_order_count::INT AS period_order_count, + c.lines_without_listing::INT AS lines_without_listing, + ROUND( + 100 * c.commission_minor / NULLIF(c.gross_sales_minor, 0), + 4 + )::NUMBER(9, 4) AS effective_commission_pct, + ROUND( + 100 + * ( + c.stated_commission_usd - c.expected_commission_usd + ) + / NULLIF(c.expected_commission_usd, 0), + 4 + )::NUMBER(9, 4) AS commission_variance_pct, + COALESCE(ABS(100 + * ( + c.stated_commission_usd - c.expected_commission_usd + ) + / NULLIF( + c.expected_commission_usd, + 0 + )) <= @const("_variance_tolerance_pct"), FALSE)::BOOLEAN AS is_within_tolerance, + ( + c.gross_sales_minor + - c.commission_minor + - COALESCE(c.other_fees_minor, 0) = c.net_payable_minor + )::BOOLEAN AS statement_balances + FROM converted AS c +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/partners/int_partner_listing_status.sql b/benchmarks/duckshop/project/models/intermediate/partners/int_partner_listing_status.sql new file mode 100644 index 0000000000..c773df42c3 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/partners/int_partner_listing_status.sql @@ -0,0 +1,219 @@ +MODEL ( + description """ + One row per partner listing with its normalised state, the catalogue product it sells, our + current list price and available stock for the SKU, recent partner sales, and a health flag for + listings that are live but cannot be fulfilled. + """, + columns ( + listing_id (type VARCHAR, nullable false, audits [not_null, unique]), + partner_code (type VARCHAR, audits [not_null]), + sku (type VARCHAR), + listing_state (type VARCHAR, audits [ + accepted_values (values ["live", "paused", "rejected", "draft", "ended", "unknown"]), + ]), + listed_at (type TIMESTAMP_TZ), + days_listed (type INT), + product_title (type VARCHAR), + brand_name (type VARCHAR), + is_catalogue_active (type BOOLEAN), + current_list_price_minor (type INT), + list_price_currency_code (type VARCHAR), + available_to_promise (type INT), + commission_pct (type NUMBER(5, 2)), + fulfilled_by_partner (type BOOLEAN), + fulfilment_method (type fulfilment_method), + term_keys (type ARRAY), + units_sold_30d (type INT), + partners_listing_sku (type INT), + listing_health (type VARCHAR), + ), +); + +WITH listings AS ( + SELECT * + FROM __ref("stg_partner_listings") +), + +catalogue AS ( + SELECT * + FROM __ref("int_product_catalogue") +), + +reservations AS ( + SELECT * + FROM __ref("int_reservations_open") +), + +price_history AS ( + SELECT * + FROM __ref("int_price_history") +), + +partner_orders AS ( + SELECT * + FROM __ref("int_partner_orders_unified") +), + +order_lines AS ( + SELECT * + FROM __ref("stg_partner_order_lines") +), + +normalised AS ( + SELECT + l.listing_id, + l.partner_code, + l.sku, + CASE + WHEN REGEXP_LIKE(l.listing_status, '^(live|active|published|on_?sale)$') + THEN 'live' + WHEN REGEXP_LIKE(l.listing_status, '^(paused|inactive|suspended|out_?of_?stock)$') + THEN 'paused' + WHEN REGEXP_LIKE(l.listing_status, '^(rejected|blocked|error.*)$') + THEN 'rejected' + WHEN REGEXP_LIKE(l.listing_status, '^(draft|pending.*|in_?review)$') + THEN 'draft' + WHEN REGEXP_LIKE(l.listing_status, '^(ended|closed|deleted|archived)$') + THEN 'ended' + ELSE 'unknown' + END AS listing_state, + l.listed_at, + l.commission_pct, + l.fulfilled_by_partner, + OBJECT_KEYS(COALESCE(l.terms, OBJECT_CONSTRUCT())) AS term_keys, + COUNT(*) OVER (PARTITION BY l.sku) AS partners_listing_sku + FROM listings AS l +), + +current_prices AS ( + SELECT + p.sku, + p.currency_code, + p.list_amount_minor + FROM price_history AS p + WHERE + p.effective_valid_to IS NULL + OR p.effective_valid_to > '@@valuation_date'::DATE + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY p.sku + ORDER BY IFF( + p.currency_code = @const("default_currency"), + 0, + 1 + ), p.valid_from DESC, p.price_list_id + ) = 1 +), + +stock AS ( + SELECT + r.sku, + SUM(GREATEST(r.available_to_promise, 0)) AS available_to_promise + FROM reservations AS r + GROUP BY + r.sku +), + +recent_orders AS ( + SELECT + o.partner_order_key, + o.partner_code + FROM partner_orders AS o + WHERE + o.placed_date > DATEADD( + 'day', + -30, + '@@valuation_date'::DATE + ) +), + +recent_sales AS ( + SELECT + o.partner_code, + l.sku, + SUM(l.quantity) AS units_sold_30d + FROM order_lines AS l + INNER JOIN recent_orders AS o + ON l.partner_order_key = o.partner_order_key + GROUP BY + o.partner_code, + l.sku +), + +joined AS ( + SELECT + n.listing_id, + n.partner_code, + n.sku, + n.listing_state, + n.listed_at, + c.title AS product_title, + c.brand_name, + c.is_active AS is_catalogue_active, + p.list_amount_minor AS current_list_price_minor, + p.currency_code AS list_price_currency_code, + COALESCE(s.available_to_promise, 0) AS available_to_promise, + n.commission_pct, + n.fulfilled_by_partner, + n.term_keys, + COALESCE(rs.units_sold_30d, 0) AS units_sold_30d, + n.partners_listing_sku + FROM normalised AS n + LEFT JOIN catalogue AS c + ON n.sku = c.sku + LEFT JOIN current_prices AS p + ON n.sku = p.sku + LEFT JOIN stock AS s + ON n.sku = s.sku + LEFT JOIN recent_sales AS rs + ON n.partner_code = rs.partner_code + AND n.sku = rs.sku +), + +final AS ( + SELECT + j.listing_id, + j.partner_code, + j.sku, + j.listing_state::VARCHAR AS listing_state, + j.listed_at, + DATEDIFF( + 'day', + j.listed_at, + '@@valuation_date'::DATE + )::INT AS days_listed, + j.product_title::VARCHAR AS product_title, + j.brand_name::VARCHAR AS brand_name, + j.is_catalogue_active::BOOLEAN AS is_catalogue_active, + j.current_list_price_minor::INT AS current_list_price_minor, + j.list_price_currency_code::VARCHAR AS list_price_currency_code, + j.available_to_promise::INT AS available_to_promise, + j.commission_pct, + j.fulfilled_by_partner, + IFF( + j.fulfilled_by_partner, + @enum("fulfilment_method").PARTNER, + @enum("fulfilment_method").HOME_DELIVERY + ) AS fulfilment_method, + j.term_keys::ARRAY AS term_keys, + j.units_sold_30d::INT AS units_sold_30d, + j.partners_listing_sku::INT AS partners_listing_sku, + CASE + WHEN j.listing_state <> 'live' + THEN 'not_live' + WHEN j.product_title IS NULL + THEN 'unknown_sku' + WHEN NOT COALESCE(j.is_catalogue_active, FALSE) + THEN 'discontinued' + WHEN NOT j.fulfilled_by_partner + AND j.available_to_promise <= 0 + THEN 'unfulfillable' + WHEN j.current_list_price_minor IS NULL + THEN 'unpriced' + ELSE 'healthy' + END::VARCHAR AS listing_health + FROM joined AS j +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/partners/int_partner_orders_unified.sql b/benchmarks/duckshop/project/models/intermediate/partners/int_partner_orders_unified.sql new file mode 100644 index 0000000000..396a88cd67 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/partners/int_partner_orders_unified.sql @@ -0,0 +1,222 @@ +MODEL ( + description """ + One row per partner order across both payload versions in one shape: total in minor units and + USD, UTC placement time, line totals checked against the header, and the buyer matched to a + customer account by email (version 1) or by email hash (version 2). + """, + audits [ + match_rate ( + name "buyer_matched", + matched "customer_id IS NOT NULL", + minimum_samples 50, + evidence_limit 20, + thresholds (warn (below 40)), + ), + ], + columns ( + partner_order_key (type VARCHAR, nullable false, audits [not_null, unique]), + source_relation (type VARCHAR), + payload_version (type INT), + partner_code (type VARCHAR, audits [not_null]), + external_order_id (type VARCHAR), + currency_code (type VARCHAR), + placed_at_utc (type TIMESTAMP_NTZ), + placed_date (type DATE), + order_total_minor (type INT), + order_total_usd (type NUMBER(18, 2)), + line_count (type INT), + units (type INT), + lines_total_minor (type INT), + lines_match_total (type BOOLEAN), + customer_id (type VARCHAR), + is_known_customer (type BOOLEAN), + order_rank_for_partner (type INT), + ), +); + +WITH orders_v1 AS ( + SELECT * + FROM __ref("stg_partner_orders_v1") +), + +orders_v2 AS ( + SELECT * + FROM __ref("stg_partner_orders_v2") +), + +order_lines AS ( + SELECT * + FROM __ref("stg_partner_order_lines") +), + +identities AS ( + SELECT * + FROM __ref("int_customer_identity") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +headers AS ( + @py.union_by_columns( + [__ref("stg_partner_orders_v1"), __ref("stg_partner_orders_v2")], + ["partner_order_key", "partner_code", "external_order_id", "currency_code"], + ) +), + +v1_details AS ( + SELECT + o.partner_order_key, + 1 AS payload_version, + CONVERT_TIMEZONE('UTC', o.placed_at)::TIMESTAMP_NTZ AS placed_at_utc, + ROUND(o.order_total * POWER(10, c.minor_unit_exponent)) AS order_total_minor, + o.buyer_email, + SHA2(o.buyer_email, 256) AS buyer_email_hash + FROM orders_v1 AS o + LEFT JOIN currencies AS c + ON o.currency_code = c.currency_code +), + +v2_details AS ( + SELECT + o.partner_order_key, + 2 AS payload_version, + o.placed_at_utc, + o.order_total_minor, + NULL::VARCHAR AS buyer_email, + o.buyer_email_hash + FROM orders_v2 AS o +), + +details AS ( + SELECT + d.partner_order_key::VARCHAR AS partner_order_key, + d.payload_version::INT AS payload_version, + d.placed_at_utc::TIMESTAMP_NTZ AS placed_at_utc, + d.order_total_minor::INT AS order_total_minor, + d.buyer_email::VARCHAR AS buyer_email, + d.buyer_email_hash::VARCHAR AS buyer_email_hash + FROM v1_details AS d + UNION ALL + SELECT + d.partner_order_key::VARCHAR AS partner_order_key, + d.payload_version::INT AS payload_version, + d.placed_at_utc::TIMESTAMP_NTZ AS placed_at_utc, + d.order_total_minor::INT AS order_total_minor, + d.buyer_email::VARCHAR AS buyer_email, + d.buyer_email_hash::VARCHAR AS buyer_email_hash + FROM v2_details AS d +), + +line_totals AS ( + SELECT + l.partner_order_key, + COUNT(*) AS line_count, + SUM(l.quantity) AS units, + SUM(l.quantity * l.unit_amount_minor) AS lines_total_minor + FROM order_lines AS l + GROUP BY + l.partner_order_key +), + +identity_hashes AS ( + SELECT + SHA2(i.email_normalised, 256) AS email_hash, + MIN(i.customer_id) AS customer_id + FROM identities AS i + WHERE i.email_normalised IS NOT NULL + GROUP BY + SHA2(i.email_normalised, 256) +), + +combined AS ( + SELECT + h.partner_order_key, + h.source_relation, + d.payload_version, + h.partner_code, + h.external_order_id, + h.currency_code, + d.placed_at_utc, + d.placed_at_utc::DATE AS placed_date, + d.order_total_minor, + COALESCE(t.line_count, 0) AS line_count, + COALESCE(t.units, 0) AS units, + COALESCE(t.lines_total_minor, 0) AS lines_total_minor, + i.customer_id, + c.minor_unit_exponent + FROM headers AS h + INNER JOIN details AS d + ON h.partner_order_key = d.partner_order_key + LEFT JOIN line_totals AS t + ON h.partner_order_key = t.partner_order_key + LEFT JOIN identity_hashes AS i + ON d.buyer_email_hash = i.email_hash + LEFT JOIN currencies AS c + ON h.currency_code = c.currency_code +), + +with_rate AS ( + SELECT + c.partner_order_key, + c.source_relation, + c.payload_version, + c.partner_code, + c.external_order_id, + c.currency_code, + c.placed_at_utc, + c.placed_date, + c.order_total_minor, + c.line_count, + c.units, + c.lines_total_minor, + c.customer_id, + c.minor_unit_exponent, + fx.rate_to_usd + FROM combined AS c + ASOF JOIN fx_rates AS fx MATCH_CONDITION (c.placed_date >= fx.rate_date) + ON c.currency_code = fx.currency_code +), + +final AS ( + SELECT + w.partner_order_key, + w.source_relation::VARCHAR AS source_relation, + w.payload_version, + w.partner_code::VARCHAR AS partner_code, + w.external_order_id::VARCHAR AS external_order_id, + w.currency_code::VARCHAR AS currency_code, + w.placed_at_utc, + w.placed_date::DATE AS placed_date, + w.order_total_minor, + ROUND( + @sql.money_from_minor(w.order_total_minor, COALESCE(w.minor_unit_exponent, 2), places := 6) * w.rate_to_usd, + 2 + )::NUMBER(18, 2) AS order_total_usd, + w.line_count::INT AS line_count, + w.units::INT AS units, + w.lines_total_minor::INT AS lines_total_minor, + ( + w.lines_total_minor = w.order_total_minor + )::BOOLEAN AS lines_match_total, + w.customer_id::VARCHAR AS customer_id, + ( + w.customer_id IS NOT NULL + )::BOOLEAN AS is_known_customer, + COUNT(*) OVER ( + PARTITION BY w.partner_code + ORDER BY w.placed_at_utc + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + )::INT AS order_rank_for_partner + FROM with_rate AS w +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/support/int_agent_utilisation.sql b/benchmarks/duckshop/project/models/intermediate/support/int_agent_utilisation.sql new file mode 100644 index 0000000000..caca3fbada --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/support/int_agent_utilisation.sql @@ -0,0 +1,113 @@ +MODEL ( + description """ + One row per agent and shift with scheduled productive minutes and the minutes spent on calls and + first responses during the shift, with shift times converted from the agent's zone to UTC. + """, + columns ( + shift_id (type VARCHAR, nullable false, audits [not_null, unique]), + agent_id (type VARCHAR, audits [not_null]), + team_code (type VARCHAR), + shift_start_utc (type TIMESTAMP_NTZ), + shift_end_utc (type TIMESTAMP_NTZ), + productive_minutes (type INT), + call_minutes (type NUMBER(10, 2)), + calls_handled (type INT), + tickets_first_responded (type INT), + utilisation (type NUMBER(9, 4)), + is_overnight (type BOOLEAN), + ), +); + +WITH shifts AS ( + SELECT * + FROM __ref("stg_agent_shifts") +), + +agents AS ( + SELECT * + FROM __ref("stg_agents") +), + +calls AS ( + SELECT * + FROM __ref("stg_call_records") +), + +first_responses AS ( + SELECT * + FROM __ref("int_ticket_first_response") +), + +shifts_utc AS ( + SELECT + s.shift_id, + s.agent_id, + CONVERT_TIMEZONE(s.iana_timezone, 'UTC', s.shift_start_local)::TIMESTAMP_NTZ AS shift_start_utc, + CONVERT_TIMEZONE(s.iana_timezone, 'UTC', s.shift_end_local)::TIMESTAMP_NTZ AS shift_end_utc, + s.scheduled_minutes - s.break_minutes AS productive_minutes, + ( + s.shift_end_local::DATE > s.shift_start_local::DATE + ) AS is_overnight + FROM shifts AS s +), + +calls_in_shift AS ( + SELECT + su.shift_id, + COUNT(c.call_id) AS calls_handled, + SUM(c.duration_seconds) / 60 AS call_minutes + FROM shifts_utc AS su + INNER JOIN calls AS c + ON su.agent_id = c.agent_id + WHERE + CONVERT_TIMEZONE( + 'UTC', + c.started_at + )::TIMESTAMP_NTZ BETWEEN su.shift_start_utc AND su.shift_end_utc + GROUP BY + su.shift_id +), + +responses_in_shift AS ( + SELECT + su.shift_id, + COUNT(f.ticket_id) AS tickets_first_responded + FROM shifts_utc AS su + INNER JOIN first_responses AS f + ON su.agent_id = f.first_responder_id + AND f.first_response_at_utc >= su.shift_start_utc + AND f.first_response_at_utc < su.shift_end_utc + GROUP BY + su.shift_id +), + +final AS ( + SELECT + su.shift_id, + su.agent_id, + a.team_code, + su.shift_start_utc, + su.shift_end_utc, + su.productive_minutes::INT AS productive_minutes, + ROUND(COALESCE(c.call_minutes, 0), 2)::NUMBER(10, 2) AS call_minutes, + COALESCE(c.calls_handled, 0)::INT AS calls_handled, + COALESCE(r.tickets_first_responded, 0)::INT AS tickets_first_responded, + ROUND( + ( + COALESCE(c.call_minutes, 0) + 6 * COALESCE(r.tickets_first_responded, 0) + ) + / NULLIF(su.productive_minutes, 0), + 4 + )::NUMBER(9, 4) AS utilisation, + su.is_overnight + FROM shifts_utc AS su + LEFT JOIN agents AS a + ON su.agent_id = a.agent_id + LEFT JOIN calls_in_shift AS c + ON su.shift_id = c.shift_id + LEFT JOIN responses_in_shift AS r + ON su.shift_id = r.shift_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/support/int_call_ivr_paths.sql b/benchmarks/duckshop/project/models/intermediate/support/int_call_ivr_paths.sql new file mode 100644 index 0000000000..e5e48e3663 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/support/int_call_ivr_paths.sql @@ -0,0 +1,99 @@ +MODEL ( + description """ + One row per phone call with its IVR menu choices flattened, the final menu node, and whether the + caller abandoned in the IVR + """, + columns ( + call_id (type VARCHAR, nullable false, audits [not_null, unique]), + ticket_id (type VARCHAR), + agent_id (type VARCHAR), + first_menu_choice (type VARCHAR), + final_menu_node (type VARCHAR), + menu_steps (type INT), + repeated_menu_visits (type INT), + abandoned_in_ivr (type BOOLEAN), + caller_region (type VARCHAR), + duration_bucket (type VARCHAR), + ), +); + +WITH calls AS ( + SELECT * + FROM __ref("stg_call_records") +), + +call_paths AS ( + SELECT * + FROM __source("support__call_records") +), + +latest_paths AS ( + SELECT + p.call_id, + p.ivr_path + FROM call_paths AS p + QUALIFY ROW_NUMBER() OVER (PARTITION BY p.call_id ORDER BY p._seq DESC) = 1 +), + +steps AS ( + SELECT + l.call_id, + s.index AS step_index, + LOWER(s.value::VARCHAR) AS menu_node + FROM latest_paths AS l, LATERAL FLATTEN(input => l.ivr_path) AS s +), + +step_summary AS ( + SELECT + s.call_id, + MIN_BY(s.menu_node, s.step_index) AS first_menu_choice, + MAX_BY(s.menu_node, s.step_index) AS final_menu_node, + COUNT(*) AS menu_steps, + COUNT(*) - COUNT(DISTINCT s.menu_node) AS repeated_menu_visits + FROM steps AS s + GROUP BY + s.call_id +), + +final AS ( + SELECT + c.call_id, + c.ticket_id, + c.agent_id, + s.first_menu_choice::VARCHAR AS first_menu_choice, + s.final_menu_node::VARCHAR AS final_menu_node, + COALESCE(s.menu_steps, 0)::INT AS menu_steps, + COALESCE(s.repeated_menu_visits, 0)::INT AS repeated_menu_visits, + ( + c.agent_id IS NULL + AND COALESCE(s.final_menu_node, '') NOT IN ('self_service_complete', 'callback_booked') + ) AS abandoned_in_ivr, + CASE c.caller_country_prefix + WHEN '1' + THEN 'US' + WHEN '81' + THEN 'APAC' + WHEN '61' + THEN 'APAC' + WHEN '965' + THEN 'MENA' + WHEN NULL + THEN NULL + ELSE 'EU' + END::VARCHAR AS caller_region, + CASE + WHEN c.duration_seconds < 60 + THEN 'under_1m' + WHEN c.duration_seconds < 300 + THEN '1_to_5m' + WHEN c.duration_seconds < 900 + THEN '5_to_15m' + ELSE 'over_15m' + END::VARCHAR AS duration_bucket + FROM calls AS c + LEFT JOIN step_summary AS s + ON c.call_id = s.call_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/support/int_chat_sentiment.sql b/benchmarks/duckshop/project/models/intermediate/support/int_chat_sentiment.sql new file mode 100644 index 0000000000..5236e071a3 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/support/int_chat_sentiment.sql @@ -0,0 +1,106 @@ +MODEL ( + description """ + One row per chat with a keyword-based sentiment score for the customer's messages, agent + response latency, and whether the chat ended with a thank-you. + """, + constants ( + _negative_pattern "(angry|terrible|worst|refund|cancel|broken|never again|ridiculous)", + _positive_pattern "(thanks|thank you|great|perfect|helpful|appreciate|sorted)", + ), + columns ( + chat_id (type VARCHAR, nullable false, audits [not_null, unique]), + ticket_id (type VARCHAR), + customer_message_count (type INT), + agent_message_count (type INT), + sentiment_score (type INT), + sentiment_label (type VARCHAR, audits [ + accepted_values (values ["negative", "neutral", "positive"]), + ]), + median_agent_reply_seconds (type INT), + ends_with_thanks (type BOOLEAN), + chat_minutes (type NUMBER(10, 2)), + ), +); + +WITH messages AS ( + SELECT * + FROM __ref("stg_chat_messages") +), + +chats AS ( + SELECT * + FROM __ref("stg_chat_transcripts") +), + +with_previous AS ( + SELECT + m.chat_id, + m.message_index, + m.sender_role, + m.sent_at, + LOWER(m.message_text_masked) AS message_text, + LAG(m.sender_role) OVER (PARTITION BY m.chat_id ORDER BY m.message_index) AS previous_role, + LAG(m.sent_at) OVER (PARTITION BY m.chat_id ORDER BY m.message_index) AS previous_sent_at + FROM messages AS m +), + +scored AS ( + SELECT + w.chat_id, + w.message_index, + w.sender_role, + w.message_text, + IFF( + w.sender_role = 'customer', + REGEXP_COUNT(w.message_text, @const("_positive_pattern")) + - REGEXP_COUNT(w.message_text, @const("_negative_pattern")), + 0 + ) AS message_sentiment, + IFF( + w.sender_role = 'agent' AND w.previous_role = 'customer', + DATEDIFF('second', w.previous_sent_at, w.sent_at), + NULL + ) AS agent_reply_seconds + FROM with_previous AS w +), + +per_chat AS ( + SELECT + s.chat_id, + COUNT_IF(s.sender_role = 'customer') AS customer_message_count, + COUNT_IF(s.sender_role = 'agent') AS agent_message_count, + SUM(s.message_sentiment) AS sentiment_score, + MEDIAN(s.agent_reply_seconds) AS median_agent_reply_seconds, + REGEXP_LIKE( + MAX_BY(s.message_text, s.message_index), + '.*(thanks|thank you|cheers).*' + ) AS ends_with_thanks + FROM scored AS s + GROUP BY + s.chat_id +), + +final AS ( + SELECT + c.chat_id, + c.ticket_id, + COALESCE(p.customer_message_count, 0)::INT AS customer_message_count, + COALESCE(p.agent_message_count, 0)::INT AS agent_message_count, + COALESCE(p.sentiment_score, 0)::INT AS sentiment_score, + CASE + WHEN COALESCE(p.sentiment_score, 0) < 0 + THEN 'negative' + WHEN COALESCE(p.sentiment_score, 0) > 0 + THEN 'positive' + ELSE 'neutral' + END::VARCHAR AS sentiment_label, + p.median_agent_reply_seconds::INT AS median_agent_reply_seconds, + COALESCE(p.ends_with_thanks, FALSE)::BOOLEAN AS ends_with_thanks, + ROUND(DATEDIFF('second', c.started_at, c.ended_at) / 60, 2)::NUMBER(10, 2) AS chat_minutes + FROM chats AS c + LEFT JOIN per_chat AS p + ON c.chat_id = p.chat_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/support/int_customer_support_history.sql b/benchmarks/duckshop/project/models/intermediate/support/int_customer_support_history.sql new file mode 100644 index 0000000000..1e4c87c4d1 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/support/int_customer_support_history.sql @@ -0,0 +1,113 @@ +MODEL ( + description """ + One row per ticket linked to a customer account by email, with the customer's ticket sequence, + days since their previous ticket, and their satisfaction trend. + """, + columns ( + ticket_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR), + ticket_number_for_customer (type INT), + days_since_previous_ticket (type INT), + csat_score (type INT), + rolling_csat_avg (type NUMBER(6, 3)), + is_repeat_contact_7d (type BOOLEAN), + contact_reason (type VARCHAR), + ), +); + +WITH tickets AS ( + SELECT * + FROM __ref("stg_tickets") +), + +identities AS ( + SELECT * + FROM __ref("int_customer_identity") +), + +csat AS ( + SELECT * + FROM __ref("stg_csat_responses") +), + +latest_csat AS ( + SELECT + c.ticket_id, + c.score + FROM csat AS c + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY c.ticket_id + ORDER BY c.responded_at DESC, c.response_id DESC + ) = 1 +), + +email_owners AS ( + SELECT + i.email_normalised, + MIN(i.customer_id) AS customer_id + FROM identities AS i + WHERE i.email_normalised IS NOT NULL + GROUP BY + i.email_normalised +), + +linked AS ( + SELECT + t.ticket_id, + i.customer_id, + CONVERT_TIMEZONE('UTC', t.created_at)::TIMESTAMP_NTZ AS created_at_utc, + t.contact_reason, + lc.score AS csat_score + FROM tickets AS t + LEFT JOIN email_owners AS i + ON t.customer_email = i.email_normalised + LEFT JOIN latest_csat AS lc + ON t.ticket_id = lc.ticket_id +), + +sequenced AS ( + SELECT + l.ticket_id, + l.customer_id, + l.created_at_utc, + l.contact_reason, + l.csat_score, + ROW_NUMBER() OVER ( + PARTITION BY l.customer_id + ORDER BY l.created_at_utc, l.ticket_id + ) AS ticket_number_for_customer, + LAG(l.created_at_utc) OVER ( + PARTITION BY l.customer_id + ORDER BY l.created_at_utc, l.ticket_id + ) AS previous_created_at_utc, + AVG(l.csat_score) OVER ( + PARTITION BY l.customer_id + ORDER BY l.created_at_utc, l.ticket_id + ROWS BETWEEN 2 PRECEDING AND CURRENT ROW + ) AS rolling_csat_avg + FROM linked AS l +), + +final AS ( + SELECT + s.ticket_id, + s.customer_id, + IFF( + s.customer_id IS NULL, + NULL, + s.ticket_number_for_customer + )::INT AS ticket_number_for_customer, + DATEDIFF('day', s.previous_created_at_utc, s.created_at_utc)::INT AS days_since_previous_ticket, + s.csat_score, + ROUND(s.rolling_csat_avg, 3)::NUMBER(6, 3) AS rolling_csat_avg, + COALESCE( + DATEDIFF('hour', s.previous_created_at_utc, s.created_at_utc) <= 168, + FALSE + ) AS is_repeat_contact_7d, + s.contact_reason + FROM sequenced AS s +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/support/int_ticket_business_hours.sql b/benchmarks/duckshop/project/models/intermediate/support/int_ticket_business_hours.sql new file mode 100644 index 0000000000..226d0ff0d5 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/support/int_ticket_business_hours.sql @@ -0,0 +1,122 @@ +MODEL ( + description """ + One row per ticket with first-response time counted in support business hours (08:00 to 20:00 + Berlin time on working days, regional holidays excluded), alongside the calendar time. + """, + constants ( + _support_timezone "Europe/Berlin", + _day_start_hour 8, + _day_end_hour 20, + ), + columns ( + ticket_id (type VARCHAR, nullable false, audits [not_null, unique]), + created_at_support_local (type TIMESTAMP_NTZ), + first_response_at_support_local (type TIMESTAMP_NTZ), + calendar_minutes_to_response (type INT), + business_minutes_to_response (type INT), + created_outside_hours (type BOOLEAN), + ), +); + +WITH first_responses AS ( + SELECT * + FROM __ref("int_ticket_first_response") +), + +holidays AS ( + SELECT * + FROM __seed("holiday_calendar") +), + +@sql.date_spine(@const("history_start_date"), 730) + +local_times AS ( + SELECT + f.ticket_id, + CONVERT_TIMEZONE( + 'UTC', + @const("_support_timezone"), + f.created_at_utc + )::TIMESTAMP_NTZ AS created_local, + CONVERT_TIMEZONE( + 'UTC', + @const("_support_timezone"), + f.first_response_at_utc + )::TIMESTAMP_NTZ AS response_local + FROM first_responses AS f +), local_dates AS ( + SELECT + l.ticket_id, + l.created_local, + l.response_local, + l.created_local::DATE AS created_date, + l.response_local::DATE AS response_date + FROM local_times AS l + WHERE l.response_local IS NOT NULL +), eu_holidays AS ( + SELECT h.holiday_date + FROM holidays AS h + WHERE h.region_code = 'EU' +), working_days AS ( + SELECT d.calendar_date + FROM date_spine AS d + LEFT JOIN eu_holidays AS h + ON d.calendar_date = h.holiday_date + WHERE + DAYOFWEEKISO(d.calendar_date) <= 5 + AND h.holiday_date IS NULL +), day_windows AS ( + SELECT + ld.ticket_id, + GREATEST( + ld.created_local, + TIMESTAMP_NTZ_FROM_PARTS( + YEAR(w.calendar_date), + MONTH(w.calendar_date), + DAY(w.calendar_date), + @const("_day_start_hour"), + 0, + 0 + ) + ) AS window_start, + LEAST( + ld.response_local, + TIMESTAMP_NTZ_FROM_PARTS( + YEAR(w.calendar_date), + MONTH(w.calendar_date), + DAY(w.calendar_date), + @const("_day_end_hour"), + 0, + 0 + ) + ) AS window_end + FROM local_dates AS ld + INNER JOIN working_days AS w + ON w.calendar_date >= ld.created_date + AND w.calendar_date <= ld.response_date +), business_minutes AS ( + SELECT + dw.ticket_id, + SUM( + GREATEST(DATEDIFF('minute', dw.window_start, dw.window_end), 0) + ) AS business_minutes_to_response + FROM day_windows AS dw + GROUP BY + dw.ticket_id +), final AS ( + SELECT + l.ticket_id, + l.created_local AS created_at_support_local, + l.response_local AS first_response_at_support_local, + DATEDIFF('minute', l.created_local, l.response_local)::INT AS calendar_minutes_to_response, + COALESCE(b.business_minutes_to_response, 0)::INT AS business_minutes_to_response, + ( + HOUR(l.created_local) NOT BETWEEN @const("_day_start_hour") AND @const("_day_end_hour") - 1 + OR DAYOFWEEKISO(l.created_local) > 5 + ) AS created_outside_hours + FROM local_times AS l + LEFT JOIN business_minutes AS b + ON l.ticket_id = b.ticket_id +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/support/int_ticket_first_response.sql b/benchmarks/duckshop/project/models/intermediate/support/int_ticket_first_response.sql new file mode 100644 index 0000000000..f7987c416d --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/support/int_ticket_first_response.sql @@ -0,0 +1,115 @@ +MODEL ( + description """ + One row per ticket with the first public agent response and the first customer reply after it. + Ticket creation is a TIMESTAMP_LTZ in the source and is converted explicitly to UTC here; the + conversion depends on the session time zone, an expected divergence across engines. + """, + tags [known_divergence], + columns ( + ticket_id (type VARCHAR, nullable false, audits [not_null, unique]), + created_at_utc (type TIMESTAMP_NTZ, audits [not_null]), + first_response_at_utc (type TIMESTAMP_NTZ), + first_responder_id (type VARCHAR), + minutes_to_first_response (type INT), + first_customer_reply_at_utc (type TIMESTAMP_NTZ), + agent_responses_before_customer_reply (type INT), + has_auto_reply (type BOOLEAN), + ), +); + +WITH tickets AS ( + SELECT * + FROM __ref("stg_tickets") +), + +comments AS ( + SELECT * + FROM __ref("stg_ticket_comments") +), + +tickets_utc AS ( + SELECT + t.ticket_id, + CONVERT_TIMEZONE('UTC', t.created_at::TIMESTAMP_LTZ)::TIMESTAMP_NTZ AS created_at_utc + FROM tickets AS t +), + +comments_utc AS ( + SELECT + c.ticket_id, + c.author_type, + c.author_id, + c.is_public, + CONVERT_TIMEZONE('UTC', c.created_at)::TIMESTAMP_NTZ AS created_at_utc + FROM comments AS c +), + +response_windows AS ( + SELECT + c.ticket_id, + c.author_type, + c.author_id, + c.created_at_utc, + MIN( + IFF(c.author_type = 'agent' AND c.is_public, c.created_at_utc, NULL) + ) OVER (PARTITION BY c.ticket_id) AS first_response_at_utc + FROM comments_utc AS c +), + +first_responses AS ( + SELECT + r.ticket_id, + MIN(r.first_response_at_utc) AS first_response_at_utc, + MIN_BY( + IFF(r.author_type = 'agent', r.author_id, NULL), + IFF(r.author_type = 'agent', r.created_at_utc, NULL) + ) AS first_responder_id, + MIN( + IFF( + r.author_type = 'customer' AND r.created_at_utc > r.first_response_at_utc, + r.created_at_utc, + NULL + ) + ) AS first_customer_reply_at_utc, + COUNT_IF(r.author_type = 'system') > 0 AS has_auto_reply + FROM response_windows AS r + GROUP BY + r.ticket_id +), + +agent_burst AS ( + SELECT + r.ticket_id, + COUNT(*) AS agent_responses_before_customer_reply + FROM response_windows AS r + INNER JOIN first_responses AS f + ON r.ticket_id = f.ticket_id + WHERE + r.author_type = 'agent' + AND r.created_at_utc < COALESCE(f.first_customer_reply_at_utc, '9999-12-31'::TIMESTAMP_NTZ) + GROUP BY + r.ticket_id +), + +final AS ( + SELECT + t.ticket_id, + t.created_at_utc, + f.first_response_at_utc, + f.first_responder_id::VARCHAR AS first_responder_id, + DATEDIFF('minute', t.created_at_utc, f.first_response_at_utc)::INT AS minutes_to_first_response, + f.first_customer_reply_at_utc, + COALESCE( + a.agent_responses_before_customer_reply, + 0 + )::INT AS agent_responses_before_customer_reply, + COALESCE(f.has_auto_reply, FALSE)::BOOLEAN AS has_auto_reply + FROM tickets_utc AS t + LEFT JOIN first_responses AS f + ON t.ticket_id = f.ticket_id + LEFT JOIN agent_burst AS a + ON t.ticket_id = a.ticket_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/support/int_ticket_sla_breaches.sql b/benchmarks/duckshop/project/models/intermediate/support/int_ticket_sla_breaches.sql new file mode 100644 index 0000000000..01153ac52e --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/support/int_ticket_sla_breaches.sql @@ -0,0 +1,115 @@ +MODEL ( + description """ + One row per ticket matched to its SLA policy by priority and channel, with first-response and + resolution targets, the time used, and whether each target was breached. Open tickets also carry + their age at query time from CURRENT_TIMESTAMP(), which is session-dependent and an expected + divergence between runs and engines. + """, + tags [known_divergence], + columns ( + ticket_id (type VARCHAR, nullable false, audits [not_null, unique]), + priority (type ticket_priority), + channel (type VARCHAR), + policy_id (type VARCHAR), + first_response_target_minutes (type INT), + first_response_minutes_used (type INT), + is_first_response_breached (type BOOLEAN), + resolution_target_minutes (type INT), + resolution_minutes_used (type INT), + is_resolution_breached (type BOOLEAN), + breach_count (type INT), + open_minutes_now (type INT), + ), +); + +WITH tickets AS ( + SELECT * + FROM __ref("stg_tickets") +), + +policies AS ( + SELECT * + FROM __ref("stg_sla_policies") +), + +first_responses AS ( + SELECT * + FROM __ref("int_ticket_first_response") +), + +business_hours AS ( + SELECT * + FROM __ref("int_ticket_business_hours") +), + +matched AS ( + SELECT + t.ticket_id, + t.priority, + t.channel, + t.solved_at, + p.policy_id, + p.first_response_minutes, + p.resolution_minutes, + p.business_hours_only + FROM tickets AS t + LEFT JOIN policies AS p + ON t.priority = p.priority + AND t.channel = p.channel +), + +measured AS ( + SELECT + m.ticket_id, + m.priority, + m.channel, + m.policy_id, + m.first_response_minutes, + m.resolution_minutes, + IFF( + m.business_hours_only, + bh.business_minutes_to_response, + fr.minutes_to_first_response + ) AS first_response_minutes_used, + DATEDIFF( + 'minute', + fr.created_at_utc, + CONVERT_TIMEZONE('UTC', m.solved_at)::TIMESTAMP_NTZ + ) AS resolution_minutes_used, + IFF(m.solved_at IS NULL, DATEDIFF( + 'minute', + fr.created_at_utc, + CONVERT_TIMEZONE('UTC', CURRENT_TIMESTAMP())::TIMESTAMP_NTZ + ), NULL) AS open_minutes_now + FROM matched AS m + INNER JOIN first_responses AS fr + ON m.ticket_id = fr.ticket_id + LEFT JOIN business_hours AS bh + ON m.ticket_id = bh.ticket_id +), + +final AS ( + SELECT + x.ticket_id, + x.priority, + x.channel, + x.policy_id, + x.first_response_minutes::INT AS first_response_target_minutes, + x.first_response_minutes_used::INT AS first_response_minutes_used, + COALESCE( + x.first_response_minutes_used > x.first_response_minutes, + FALSE + ) AS is_first_response_breached, + x.resolution_minutes::INT AS resolution_target_minutes, + x.resolution_minutes_used::INT AS resolution_minutes_used, + COALESCE(x.resolution_minutes_used > x.resolution_minutes, FALSE) AS is_resolution_breached, + ( + IFF(COALESCE(x.first_response_minutes_used > x.first_response_minutes, FALSE), 1, 0) + + IFF(COALESCE(x.resolution_minutes_used > x.resolution_minutes, FALSE), 1, 0) + )::INT AS breach_count, + x.open_minutes_now::INT AS open_minutes_now + FROM measured AS x +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/intermediate/support/int_ticket_threads.sql b/benchmarks/duckshop/project/models/intermediate/support/int_ticket_threads.sql new file mode 100644 index 0000000000..900d95f7c2 --- /dev/null +++ b/benchmarks/duckshop/project/models/intermediate/support/int_ticket_threads.sql @@ -0,0 +1,98 @@ +MODEL ( + description """ + One row per ticket with its comment thread summarised: participants, ordered author sequence, + back-and-forth count, and a per-author breakdown object that keeps authors with no comments. + """, + columns ( + ticket_id (type VARCHAR, nullable false, audits [not_null, unique]), + comment_count (type INT), + public_comment_count (type INT), + agent_participants (type ARRAY), + author_sequence (type ARRAY), + handoff_count (type INT), + author_breakdown (type OBJECT), + total_attachments (type INT), + last_comment_at (type TIMESTAMP_TZ), + last_author_type (type VARCHAR), + mentions_order (type BOOLEAN), + ), +); + +WITH comments AS ( + SELECT * + FROM __ref("stg_ticket_comments") +), + +tickets AS ( + SELECT * + FROM __ref("stg_tickets") +), + +ordered_comments AS ( + SELECT + c.ticket_id, + c.comment_id, + c.author_type, + c.author_id, + c.is_public, + c.created_at, + c.attachment_count, + c.contains_order_reference, + LAG(c.author_type) OVER ( + PARTITION BY c.ticket_id + ORDER BY c.created_at, c.comment_id + ) AS previous_author_type + FROM comments AS c +), + +thread_stats AS ( + SELECT + o.ticket_id, + COUNT(*) AS comment_count, + COUNT_IF(o.is_public) AS public_comment_count, + ARRAY_AGG(DISTINCT IFF(o.author_type = 'agent', o.author_id, NULL)) WITHIN GROUP ( + ORDER BY IFF(o.author_type = 'agent', o.author_id, NULL) + ) AS agent_participants, + ARRAY_AGG(o.author_type) WITHIN GROUP (ORDER BY o.created_at, o.comment_id) AS author_sequence, + COUNT_IF( + o.previous_author_type IS NOT NULL AND o.previous_author_type <> o.author_type + ) AS handoff_count, + SUM(o.attachment_count) AS total_attachments, + MAX(o.created_at) AS last_comment_at, + MAX_BY(o.author_type, o.created_at) AS last_author_type, + BOOLOR_AGG(o.contains_order_reference) AS mentions_order, + COUNT_IF(o.author_type = 'agent') AS agent_comments, + COUNT_IF(o.author_type = 'customer') AS customer_comments, + COUNT_IF(o.author_type = 'system') AS system_comments + FROM ordered_comments AS o + GROUP BY + o.ticket_id +), + +final AS ( + SELECT + t.ticket_id, + COALESCE(s.comment_count, 0)::INT AS comment_count, + COALESCE(s.public_comment_count, 0)::INT AS public_comment_count, + COALESCE(s.agent_participants, ARRAY_CONSTRUCT())::ARRAY AS agent_participants, + COALESCE(s.author_sequence, ARRAY_CONSTRUCT())::ARRAY AS author_sequence, + COALESCE(s.handoff_count, 0)::INT AS handoff_count, + OBJECT_CONSTRUCT_KEEP_NULL( + 'agent', + s.agent_comments, + 'customer', + s.customer_comments, + 'system', + NULLIF(s.system_comments, 0) + )::OBJECT AS author_breakdown, + COALESCE(s.total_attachments, 0)::INT AS total_attachments, + s.last_comment_at, + s.last_author_type::VARCHAR AS last_author_type, + COALESCE(s.mentions_order, FALSE)::BOOLEAN AS mentions_order + FROM tickets AS t + LEFT JOIN thread_stats AS s + ON t.ticket_id = s.ticket_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/_sqlbuild/audits/generic/has_rows.sql b/benchmarks/duckshop/project/models/marts/_sqlbuild/audits/generic/has_rows.sql new file mode 100644 index 0000000000..0ab61af96e --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/_sqlbuild/audits/generic/has_rows.sql @@ -0,0 +1,15 @@ +AUDIT (); + +WITH counted AS ( + SELECT COUNT(*) AS row_count + FROM @relation +), + +final AS ( + SELECT c.row_count + FROM counted AS c + WHERE c.row_count < @minimum_rows +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/_sqlbuild/audits/generic/non_negative.sql b/benchmarks/duckshop/project/models/marts/_sqlbuild/audits/generic/non_negative.sql new file mode 100644 index 0000000000..05a15b7895 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/_sqlbuild/audits/generic/non_negative.sql @@ -0,0 +1,10 @@ +AUDIT (); + +WITH final AS ( + SELECT * + FROM @relation + WHERE @column < 0 +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/catalogue/rpt_category_tree_sales.sql b/benchmarks/duckshop/project/models/marts/catalogue/rpt_category_tree_sales.sql new file mode 100644 index 0000000000..208aba5d57 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/catalogue/rpt_category_tree_sales.sql @@ -0,0 +1,105 @@ +MODEL ( + description """ + One row per category node with units and net sales of its own SKUs and of its whole subtree, + using the slug path to roll leaf sales up to every ancestor. + """, + columns ( + category_id (type VARCHAR, nullable false, audits [not_null, unique]), + slug_path (type VARCHAR), + depth (type INT), + is_leaf (type BOOLEAN), + own_units (type INT), + subtree_units (type INT), + subtree_net_amount_minor (type INT), + share_of_parent_units (type NUMBER(9, 4)), + ), +); + +WITH categories AS ( + SELECT * + FROM __ref("int_category_paths") +), + +catalogue AS ( + SELECT * + FROM __ref("dim_products") +), + +lines AS ( + SELECT * + FROM __ref("fct_order_lines") +), + +product_catalogue AS ( + SELECT * + FROM __ref("int_product_catalogue") +), + +product_categories AS ( + SELECT + p.sku, + c.category_id, + c.slug_path + FROM catalogue AS p + INNER JOIN product_catalogue AS pc + ON p.sku = pc.sku + INNER JOIN categories AS c + ON pc.category_id = c.category_id +), + +node_ranges AS ( + SELECT + c.category_id, + c.slug_path || '/' AS range_start, + c.slug_path || '0' AS range_end + FROM categories AS c +), + +own_sales AS ( + SELECT + pc.category_id, + pc.slug_path || '/' AS slug_path_key, + SUM(l.quantity) AS own_units, + SUM(l.net_amount_minor) AS own_net_minor + FROM lines AS l + INNER JOIN product_categories AS pc + ON l.sku = pc.sku + GROUP BY + pc.category_id, + pc.slug_path +), + +subtree AS ( + SELECT + r.category_id, + SUM(s.own_units) AS subtree_units, + SUM(s.own_net_minor) AS subtree_net_minor + FROM node_ranges AS r + INNER JOIN own_sales AS s + ON s.slug_path_key >= r.range_start + AND s.slug_path_key < r.range_end + GROUP BY + r.category_id +), + +final AS ( + SELECT + c.category_id, + c.slug_path, + c.depth, + c.is_leaf, + COALESCE(o.own_units, 0)::INT AS own_units, + COALESCE(t.subtree_units, 0)::INT AS subtree_units, + COALESCE(t.subtree_net_minor, 0)::INT AS subtree_net_amount_minor, + ROUND(t.subtree_units / NULLIF(pt.subtree_units, 0), 4)::NUMBER(9, 4) AS share_of_parent_units + FROM categories AS c + LEFT JOIN own_sales AS o + ON c.category_id = o.category_id + LEFT JOIN subtree AS t + ON c.category_id = t.category_id + LEFT JOIN subtree AS pt + ON c.parent_category_id = pt.category_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/catalogue/rpt_margin_by_category.sql b/benchmarks/duckshop/project/models/marts/catalogue/rpt_margin_by_category.sql new file mode 100644 index 0000000000..fc26bcd6a0 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/catalogue/rpt_margin_by_category.sql @@ -0,0 +1,47 @@ +MODEL ( + description """ + One row per top-level category and currency with SKU counts, average margin and below-cost SKUs + """, + columns ( + margin_key (type VARCHAR, nullable false, audits [not_null, unique]), + category_level_1 (type VARCHAR), + currency_code (type VARCHAR), + sku_count (type INT), + average_margin_pct (type NUMBER(9, 2)), + p25_margin_pct (type NUMBER(9, 2)), + p75_margin_pct (type NUMBER(9, 2)), + below_cost_skus (type INT), + ), +); + +WITH margins AS ( + SELECT * + FROM __ref("int_product_margin") +), + +final AS ( + SELECT + ( + COALESCE(m.category_level_1, 'uncategorised') || ':' || m.currency_code + )::VARCHAR AS margin_key, + COALESCE(m.category_level_1, 'uncategorised')::VARCHAR AS category_level_1, + m.currency_code, + COUNT(*)::INT AS sku_count, + ROUND(AVG(m.margin_pct), 2)::NUMBER(9, 2) AS average_margin_pct, + ROUND( + PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY m.margin_pct), + 2 + )::NUMBER(9, 2) AS p25_margin_pct, + ROUND( + PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY m.margin_pct), + 2 + )::NUMBER(9, 2) AS p75_margin_pct, + COUNT_IF(m.is_below_cost)::INT AS below_cost_skus + FROM margins AS m + GROUP BY + COALESCE(m.category_level_1, 'uncategorised'), + m.currency_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/catalogue/rpt_price_changes_weekly.sql b/benchmarks/duckshop/project/models/marts/catalogue/rpt_price_changes_weekly.sql new file mode 100644 index 0000000000..b470c4331b --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/catalogue/rpt_price_changes_weekly.sql @@ -0,0 +1,57 @@ +MODEL ( + description """ + One row per ISO week and currency with price increases, markdowns and the average size of each + """, + columns ( + price_week_key (type VARCHAR, nullable false, audits [not_null, unique]), + week_start (type DATE), + currency_code (type VARCHAR), + price_changes (type INT), + increases (type INT), + markdowns (type INT), + average_increase_pct (type NUMBER(9, 4)), + average_markdown_pct (type NUMBER(9, 4)), + skus_below_30d_low (type INT), + ), +); + +WITH price_history AS ( + SELECT * + FROM __ref("int_price_history") +), + +changes AS ( + SELECT + DATE_TRUNC('week', CONVERT_TIMEZONE('UTC', p.valid_from))::DATE AS week_start, + p.currency_code, + p.change_pct, + p.is_markdown, + p.sku, + ( + p.list_amount_minor < p.lowest_price_30d_minor + ) AS is_below_30d_low + FROM price_history AS p + WHERE p.previous_amount_minor IS NOT NULL +), + +final AS ( + SELECT + ( + TO_VARCHAR(c.week_start, 'YYYY-MM-DD') || ':' || c.currency_code + )::VARCHAR AS price_week_key, + c.week_start, + c.currency_code, + COUNT(*)::INT AS price_changes, + COUNT_IF(c.change_pct > 0)::INT AS increases, + COUNT_IF(c.is_markdown)::INT AS markdowns, + ROUND(AVG(IFF(c.change_pct > 0, c.change_pct, NULL)), 4)::NUMBER(9, 4) AS average_increase_pct, + ROUND(AVG(IFF(c.is_markdown, c.change_pct, NULL)), 4)::NUMBER(9, 4) AS average_markdown_pct, + COUNT(DISTINCT IFF(c.is_below_30d_low, c.sku, NULL))::INT AS skus_below_30d_low + FROM changes AS c + GROUP BY + c.week_start, + c.currency_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/catalogue/rpt_wishlist_conversion.sql b/benchmarks/duckshop/project/models/marts/catalogue/rpt_wishlist_conversion.sql new file mode 100644 index 0000000000..86275c5a15 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/catalogue/rpt_wishlist_conversion.sql @@ -0,0 +1,52 @@ +MODEL ( + description "One row per brand with wishlist demand and how much of it converted to purchases", + columns ( + brand_name (type VARCHAR, nullable false, audits [not_null, unique]), + wishlisted_skus (type INT), + wishlisting_customers (type INT), + purchased_after_wishlist (type INT), + conversion_rate (type NUMBER(9, 6)), + median_days_to_purchase (type NUMBER(10, 2)), + ), +); + +WITH demand AS ( + SELECT * + FROM __ref("int_wishlist_demand") +), + +products AS ( + SELECT * + FROM __ref("dim_products") +), + +joined AS ( + SELECT + COALESCE(p.brand_name, 'Unbranded') AS brand_name, + d.sku, + d.wishlisting_customers, + d.purchased_after_wishlist, + d.median_days_to_purchase + FROM demand AS d + LEFT JOIN products AS p + ON d.sku = p.sku +), + +final AS ( + SELECT + j.brand_name::VARCHAR AS brand_name, + COUNT(DISTINCT j.sku)::INT AS wishlisted_skus, + SUM(j.wishlisting_customers)::INT AS wishlisting_customers, + SUM(j.purchased_after_wishlist)::INT AS purchased_after_wishlist, + ROUND( + SUM(j.purchased_after_wishlist) / NULLIF(SUM(j.wishlisting_customers), 0), + 6 + )::NUMBER(9, 6) AS conversion_rate, + ROUND(MEDIAN(j.median_days_to_purchase), 2)::NUMBER(10, 2) AS median_days_to_purchase + FROM joined AS j + GROUP BY + j.brand_name +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/clickstream/fct_web_sessions.sql b/benchmarks/duckshop/project/models/marts/clickstream/fct_web_sessions.sql new file mode 100644 index 0000000000..47f781c806 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/clickstream/fct_web_sessions.sql @@ -0,0 +1,104 @@ +MODEL ( + description """ + One row per web session with its attributed channel and campaign, engagement and conversion + flags. Merged incrementally from the daily sessionised clickstream with a two-day lookback for + sessions that span midnight. The visitor bucket hashes the anonymous visitor id with HASH, whose + output is engine-specific, so buckets are an expected divergence across engines. + """, + tags [known_divergence], + materialized incremental, + incremental_strategy merge, + unique_key [session_id], + cursor session_start_utc, + cursor_type timestamp, + cursor_grain hour, + lookback 2d, + cursor_inputs ( + int_web_sessions session_start_utc, + ), + replay_on_change bounded-120d, + on_schema_change append_new_columns, + contract enforced, + audit_factories [fact_table_quality], + columns ( + session_id (type VARCHAR, nullable false, audits [not_null, unique]), + anonymous_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + session_start_utc (type TIMESTAMP_NTZ, nullable false, audits [not_null]), + session_start_visitor_local (type TIMESTAMP_NTZ), + duration_seconds (type INT), + page_view_count (type INT), + landing_page_path (type VARCHAR), + channel_code (type VARCHAR), + is_paid (type BOOLEAN), + utm_campaign (type VARCHAR), + first_touch_channel (type VARCHAR), + touch_number (type INT), + is_bounce (type BOOLEAN), + funnel_stage (type VARCHAR), + converted (type BOOLEAN), + visitor_bucket (type INT), + ), +); + +WITH sessions AS ( + SELECT * + FROM __ref("int_web_sessions") +), + +attribution AS ( + SELECT * + FROM __ref("int_session_attribution") +), + +funnels AS ( + SELECT * + FROM __ref("int_cart_funnel") +), + +windowed AS ( + SELECT + s.session_id, + s.anonymous_id, + s.customer_id, + s.session_start_utc, + s.session_start_visitor_local, + s.duration_seconds, + s.page_view_count, + s.landing_page_path, + s.is_bounce, + s.has_order + FROM sessions AS s + WHERE + s.session_start_utc >= __cursor_start() + AND s.session_start_utc < __cursor_end() +), + +final AS ( + SELECT + w.session_id, + w.anonymous_id, + w.customer_id, + w.session_start_utc, + w.session_start_visitor_local, + w.duration_seconds, + w.page_view_count, + w.landing_page_path, + a.channel_code, + a.is_paid, + a.utm_campaign, + a.first_touch_channel, + a.touch_number, + w.is_bounce, + COALESCE(f.funnel_stage, IFF(w.is_bounce, 'bounce', 'browse'))::VARCHAR AS funnel_stage, + COALESCE(f.converted, w.has_order, FALSE)::BOOLEAN AS converted, + MOD(ABS(HASH(w.anonymous_id)), 100)::INT AS visitor_bucket + FROM windowed AS w + LEFT JOIN attribution AS a + ON w.session_id = a.session_id + LEFT JOIN funnels AS f + ON w.session_id = f.session_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/clickstream/rpt_app_engagement_weekly.sql b/benchmarks/duckshop/project/models/marts/clickstream/rpt_app_engagement_weekly.sql new file mode 100644 index 0000000000..a25783ed2f --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/clickstream/rpt_app_engagement_weekly.sql @@ -0,0 +1,44 @@ +MODEL ( + description """ + One row per ISO week and platform with app sessions, active devices, push-started share and + session length + """, + columns ( + app_week_key (type VARCHAR, nullable false, audits [not_null, unique]), + week_start (type DATE), + platform (type VARCHAR), + sessions (type INT), + active_devices (type INT), + signed_in_share (type NUMBER(9, 4)), + push_started_share (type NUMBER(9, 4)), + median_session_seconds (type NUMBER(10, 2)), + latest_major_version (type INT), + ), +); + +WITH app_sessions AS ( + SELECT * + FROM __ref("int_app_sessions") +), + +final AS ( + SELECT + ( + TO_VARCHAR(DATE_TRUNC('week', a.session_start_utc), 'YYYY-MM-DD') || ':' || a.platform + )::VARCHAR AS app_week_key, + DATE_TRUNC('week', a.session_start_utc)::DATE AS week_start, + a.platform, + COUNT(*)::INT AS sessions, + COUNT(DISTINCT a.device_id)::INT AS active_devices, + ROUND(AVG(IFF(a.customer_id IS NOT NULL, 1, 0)), 4)::NUMBER(9, 4) AS signed_in_share, + ROUND(AVG(IFF(a.started_from_push, 1, 0)), 4)::NUMBER(9, 4) AS push_started_share, + ROUND(MEDIAN(a.duration_seconds), 2)::NUMBER(10, 2) AS median_session_seconds, + MAX(a.app_major_version)::INT AS latest_major_version + FROM app_sessions AS a + GROUP BY + DATE_TRUNC('week', a.session_start_utc), + a.platform +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/clickstream/rpt_channel_attribution_daily.sql b/benchmarks/duckshop/project/models/marts/clickstream/rpt_channel_attribution_daily.sql new file mode 100644 index 0000000000..689827129e --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/clickstream/rpt_channel_attribution_daily.sql @@ -0,0 +1,75 @@ +MODEL ( + description """ + One row per day and channel with sessions, conversions and first-touch versus last-touch credit + """, + columns ( + channel_day_key (type VARCHAR, nullable false, audits [not_null, unique]), + session_date (type DATE), + channel_code (type VARCHAR), + sessions (type INT), + visitors (type INT), + conversions (type INT), + conversion_rate (type NUMBER(9, 4)), + first_touch_credit (type INT), + paid_share (type NUMBER(9, 4)), + ), +); + +WITH sessions AS ( + SELECT * + FROM __ref("fct_web_sessions") +), + +channel_days AS ( + SELECT + s.session_start_utc::DATE AS session_date, + COALESCE(s.channel_code, 'UNKNOWN') AS channel_code, + s.session_id, + s.anonymous_id, + s.converted, + s.is_paid, + 'last_touch' AS credit_basis + FROM sessions AS s + UNION ALL + SELECT + s.session_start_utc::DATE AS session_date, + COALESCE(s.first_touch_channel, 'UNKNOWN') AS channel_code, + s.session_id, + s.anonymous_id, + s.converted, + s.is_paid, + 'first_touch' AS credit_basis + FROM sessions AS s + WHERE s.converted +), + +final AS ( + SELECT + ( + TO_VARCHAR(c.session_date, 'YYYY-MM-DD') || ':' || c.channel_code + )::VARCHAR AS channel_day_key, + c.session_date::DATE AS session_date, + c.channel_code::VARCHAR AS channel_code, + COUNT_IF(c.credit_basis = 'last_touch')::INT AS sessions, + COUNT(DISTINCT IFF(c.credit_basis = 'last_touch', c.anonymous_id, NULL))::INT AS visitors, + COUNT_IF(c.credit_basis = 'last_touch' AND c.converted)::INT AS conversions, + ROUND( + COUNT_IF(c.credit_basis = 'last_touch' AND c.converted) + / NULLIF(COUNT_IF(c.credit_basis = 'last_touch'), 0), + 4 + )::NUMBER(9, 4) AS conversion_rate, + COUNT( + DISTINCT IFF(c.credit_basis = 'first_touch', c.session_id, NULL) + )::INT AS first_touch_credit, + ROUND( + AVG(IFF(c.credit_basis = 'last_touch', IFF(c.is_paid, 1, 0), NULL)), + 4 + )::NUMBER(9, 4) AS paid_share + FROM channel_days AS c + GROUP BY + c.session_date, + c.channel_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/clickstream/rpt_experiment_results.sql b/benchmarks/duckshop/project/models/marts/clickstream/rpt_experiment_results.sql new file mode 100644 index 0000000000..a21bf531bd --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/clickstream/rpt_experiment_results.sql @@ -0,0 +1,103 @@ +MODEL ( + description """ + One row per experiment variant with exposure and conversion counts, conversion rate, absolute + lift over the control and a two-proportion z score. + """, + columns ( + experiment_variant_key (type VARCHAR, nullable false, audits [not_null, unique]), + experiment_key (type VARCHAR), + variant_key (type VARCHAR), + is_control (type BOOLEAN), + exposed (type INT), + converted (type INT), + conversion_rate (type NUMBER(9, 6)), + lift_vs_control (type NUMBER(9, 6)), + z_score (type NUMBER(9, 4)), + median_days_to_convert (type NUMBER(10, 2)), + ), +); + +WITH exposures AS ( + SELECT * + FROM __ref("int_experiment_exposures") +), + +variants AS ( + SELECT + e.experiment_key, + e.variant_key, + BOOLOR_AGG(e.is_control) AS is_control, + COUNT(*) AS exposed, + COUNT_IF(e.converted) AS converted, + MEDIAN(e.days_to_convert) AS median_days_to_convert + FROM exposures AS e + GROUP BY + e.experiment_key, + e.variant_key +), + +controls AS ( + SELECT + v.experiment_key, + SUM(v.exposed) AS control_exposed, + SUM(v.converted) AS control_converted + FROM variants AS v + WHERE v.is_control + GROUP BY + v.experiment_key +), + +compared AS ( + SELECT + v.experiment_key, + v.variant_key, + v.is_control, + v.exposed, + v.converted, + v.median_days_to_convert, + v.converted / NULLIF(v.exposed, 0) AS rate, + c.control_converted / NULLIF(c.control_exposed, 0) AS control_rate, + ( + v.converted + c.control_converted + ) + / NULLIF(v.exposed + c.control_exposed, 0) AS pooled_rate, + c.control_exposed + FROM variants AS v + LEFT JOIN controls AS c + ON v.experiment_key = c.experiment_key +), + +final AS ( + SELECT + ( + c.experiment_key || ':' || c.variant_key + )::VARCHAR AS experiment_variant_key, + c.experiment_key, + c.variant_key, + c.is_control::BOOLEAN AS is_control, + c.exposed::INT AS exposed, + c.converted::INT AS converted, + ROUND(c.rate, 6)::NUMBER(9, 6) AS conversion_rate, + ROUND(c.rate - c.control_rate, 6)::NUMBER(9, 6) AS lift_vs_control, + ROUND( + ( + c.rate - c.control_rate + ) + / NULLIF( + SQRT(c.pooled_rate + * ( + 1 - c.pooled_rate + ) + * ( + 1 / NULLIF(c.exposed, 0) + 1 / NULLIF(c.control_exposed, 0) + )), + 0 + ), + 4 + )::NUMBER(9, 4) AS z_score, + ROUND(c.median_days_to_convert, 2)::NUMBER(10, 2) AS median_days_to_convert + FROM compared AS c +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/clickstream/rpt_product_affinity_top.sql b/benchmarks/duckshop/project/models/marts/clickstream/rpt_product_affinity_top.sql new file mode 100644 index 0000000000..aabe0da300 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/clickstream/rpt_product_affinity_top.sql @@ -0,0 +1,52 @@ +MODEL ( + description """ + One row per SKU with its five most co-viewed SKUs and their lift, as an ordered array + """, + columns ( + sku (type VARCHAR, nullable false, audits [not_null, unique]), + title (type VARCHAR), + sku_viewers (type INT), + top_co_viewed (type ARRAY), + best_lift (type NUMBER(12, 4)), + ), +); + +WITH affinity AS ( + SELECT * + FROM __ref("int_product_view_affinity") +), + +products AS ( + SELECT * + FROM __ref("dim_products") +), + +top_pairs AS ( + SELECT + a.sku, + a.co_viewed_sku, + a.lift, + a.pair_rank, + a.sku_viewers + FROM affinity AS a + WHERE a.pair_rank <= 5 +), + +final AS ( + SELECT + t.sku, + MAX(p.title)::VARCHAR AS title, + MAX(t.sku_viewers)::INT AS sku_viewers, + ARRAY_AGG( + OBJECT_CONSTRUCT('sku', t.co_viewed_sku, 'lift', t.lift) + ) WITHIN GROUP (ORDER BY t.pair_rank, t.co_viewed_sku)::ARRAY AS top_co_viewed, + MAX(t.lift)::NUMBER(12, 4) AS best_lift + FROM top_pairs AS t + LEFT JOIN products AS p + ON t.sku = p.sku + GROUP BY + t.sku +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/clickstream/rpt_search_terms_top.sql b/benchmarks/duckshop/project/models/marts/clickstream/rpt_search_terms_top.sql new file mode 100644 index 0000000000..7951063721 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/clickstream/rpt_search_terms_top.sql @@ -0,0 +1,42 @@ +MODEL ( + description """ + One row per search term among the 500 most searched, with zero-result and view-after-search + rates + """, + constants ( + _top_terms 500, + ), + columns ( + search_term (type VARCHAR, nullable false, audits [not_null, unique]), + search_rank (type INT), + search_count (type INT), + searching_visitors (type INT), + zero_result_rate (type NUMBER(9, 4)), + view_after_search_rate (type NUMBER(9, 4)), + needs_merchandising (type BOOLEAN), + ), +); + +WITH searches AS ( + SELECT * + FROM __ref("int_search_conversion") +), + +final AS ( + SELECT + s.search_term, + ROW_NUMBER() OVER (ORDER BY s.search_count DESC, s.search_term)::INT AS search_rank, + s.search_count, + s.searching_visitors, + s.zero_result_rate, + s.view_after_search_rate, + ( + s.zero_result_rate > 0.2 + OR s.view_after_search_rate < 0.1 + )::BOOLEAN AS needs_merchandising + FROM searches AS s + QUALIFY ROW_NUMBER() OVER (ORDER BY s.search_count DESC, s.search_term) <= @const("_top_terms") +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/core/_sqlbuild/_schemas/address.sql b/benchmarks/duckshop/project/models/marts/core/_sqlbuild/_schemas/address.sql new file mode 100644 index 0000000000..f0760021e6 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/core/_sqlbuild/_schemas/address.sql @@ -0,0 +1,10 @@ +SCHEMA ( + name address, + description "A postal location reduced to the parts used for reporting", + columns ( + city (type VARCHAR), + postcode_canonical (type VARCHAR), + country_code (type VARCHAR), + region_code (type VARCHAR), + ), +); diff --git a/benchmarks/duckshop/project/models/marts/core/_sqlbuild/_schemas/audit_ts.sql b/benchmarks/duckshop/project/models/marts/core/_sqlbuild/_schemas/audit_ts.sql new file mode 100644 index 0000000000..5e1976f5a4 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/core/_sqlbuild/_schemas/audit_ts.sql @@ -0,0 +1,7 @@ +SCHEMA ( + name audit_ts, + description "When the source row behind a dimension record was last loaded", + columns ( + source_loaded_at (type TIMESTAMP_NTZ, audits [not_null]), + ), +); diff --git a/benchmarks/duckshop/project/models/marts/core/dim_campaigns.sql b/benchmarks/duckshop/project/models/marts/core/dim_campaigns.sql new file mode 100644 index 0000000000..1d0f714aef --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/core/dim_campaigns.sql @@ -0,0 +1,73 @@ +MODEL ( + description """ + One row per marketing campaign with its flight dates, budget, targeting and lifetime results + """, + contract enforced, + model_schema audit_ts, + columns ( + campaign_id (type VARCHAR, nullable false, audits [not_null, unique]), + campaign_name (type VARCHAR), + channel (type VARCHAR), + objective (type VARCHAR), + starts_on (type DATE), + ends_on (type DATE), + flight_days (type INT), + campaign_state (type VARCHAR), + target_age_range (type VARCHAR), + target_regions (type ARRAY), + budget_usd (type NUMBER(18, 2)), + spend_usd (type NUMBER(18, 2)), + return_on_ad_spend (type NUMBER(12, 4)), + ), +); + +WITH campaigns AS ( + SELECT * + FROM __ref("stg_campaigns") +), + +performance AS ( + SELECT * + FROM __ref("int_campaign_performance") +), + +final AS ( + SELECT + c.campaign_id, + c.campaign_name, + c.channel, + c.objective, + c.starts_on, + c.ends_on, + ( + DATEDIFF( + 'day', + c.starts_on, + COALESCE(c.ends_on, '@@valuation_date'::DATE) + ) + + 1 + )::INT AS flight_days, + CASE + WHEN c.starts_on > '@@valuation_date'::DATE + THEN 'scheduled' + WHEN c.ends_on < '@@valuation_date'::DATE + THEN 'ended' + ELSE 'live' + END::VARCHAR AS campaign_state, + IFF( + c.target_age_min IS NULL AND c.target_age_max IS NULL, + 'all', + COALESCE(c.target_age_min::VARCHAR, '') || '-' || COALESCE(c.target_age_max::VARCHAR, '') + )::VARCHAR AS target_age_range, + c.target_regions, + p.budget_usd, + p.spend_usd, + p.return_on_ad_spend, + c.source_loaded_at + FROM campaigns AS c + LEFT JOIN performance AS p + ON c.campaign_id = p.campaign_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/core/dim_carriers.sql b/benchmarks/duckshop/project/models/marts/core/dim_carriers.sql new file mode 100644 index 0000000000..16e53ca253 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/core/dim_carriers.sql @@ -0,0 +1,91 @@ +MODEL ( + description """ + One row per carrier from the carriers seed with the services and lanes on its current rate cards + and its delivery record. + """, + contract enforced, + columns ( + carrier_code (type VARCHAR, nullable false, audits [not_null, unique]), + carrier_name (type VARCHAR), + is_express (type BOOLEAN), + tracking_host (type VARCHAR), + service_levels (type ARRAY), + destination_country_count (type INT), + active_rate_card_count (type INT), + shipment_count (type INT), + on_time_dispatch_rate (type NUMBER(9, 4)), + median_hours_to_deliver (type NUMBER(10, 2)), + ), +); + +WITH carriers AS ( + SELECT * + FROM __seed("carriers") +), + +rate_cards AS ( + SELECT * + FROM __ref("stg_carrier_rate_cards") +), + +sla AS ( + SELECT * + FROM __ref("int_fulfilment_sla") +), + +current_cards AS ( + SELECT + r.carrier_code, + ARRAY_AGG(DISTINCT r.service_level) WITHIN GROUP (ORDER BY r.service_level) AS service_levels, + COUNT(DISTINCT r.destination_country) AS destination_country_count, + COUNT(*) AS active_rate_card_count + FROM rate_cards AS r + WHERE + r.valid_from <= '@@valuation_date'::DATE + AND ( + r.valid_to IS NULL + OR r.valid_to > '@@valuation_date'::DATE + ) + GROUP BY + r.carrier_code +), + +delivery_record AS ( + SELECT + s.carrier_code, + COUNT(*) AS shipment_count, + AVG(IFF(s.is_dispatch_on_time, 1, 0)) AS on_time_dispatch_rate, + MEDIAN(s.hours_to_deliver) AS median_hours_to_deliver + FROM sla AS s + GROUP BY + s.carrier_code +), + +final AS ( + SELECT + c.carrier_code::VARCHAR AS carrier_code, + c.carrier_name::VARCHAR AS carrier_name, + c.is_express::BOOLEAN AS is_express, + REGEXP_SUBSTR( + c.tracking_url_template, + '^https?://([^/]+)', + 1, + 1, + 'e', + 1 + )::VARCHAR AS tracking_host, + COALESCE(r.service_levels, ARRAY_CONSTRUCT())::ARRAY AS service_levels, + COALESCE(r.destination_country_count, 0)::INT AS destination_country_count, + COALESCE(r.active_rate_card_count, 0)::INT AS active_rate_card_count, + COALESCE(d.shipment_count, 0)::INT AS shipment_count, + ROUND(d.on_time_dispatch_rate, 4)::NUMBER(9, 4) AS on_time_dispatch_rate, + ROUND(d.median_hours_to_deliver, 2)::NUMBER(10, 2) AS median_hours_to_deliver + FROM carriers AS c + LEFT JOIN current_cards AS r + ON c.carrier_code = r.carrier_code + LEFT JOIN delivery_record AS d + ON c.carrier_code = d.carrier_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/core/dim_customers.sql b/benchmarks/duckshop/project/models/marts/core/dim_customers.sql new file mode 100644 index 0000000000..827dc10286 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/core/dim_customers.sql @@ -0,0 +1,131 @@ +MODEL ( + description """ + One row per customer account with their current address, value segment, marketing consent, + loyalty tier and device footprint. The marketing holdout flag samples customers with HASH, whose + output is engine-specific, so the holdout set is an expected divergence across engines. + """, + tags [known_divergence], + contract enforced, + model_schema address, + columns ( + customer_id (type VARCHAR, nullable false, audits [not_null, unique]), + email_domain (type VARCHAR), + full_name (type VARCHAR), + storefront_code (type VARCHAR), + preferred_language (type VARCHAR), + created_at (type TIMESTAMP_TZ), + age_band (type VARCHAR), + order_count (type INT), + lifetime_revenue_usd (type NUMBER(18, 2), audits [non_negative]), + days_since_last_order (type INT), + rfm_segment (type VARCHAR), + is_vip (type BOOLEAN), + email_marketing (type BOOLEAN), + sms_marketing (type BOOLEAN), + loyalty_tier_code (type VARCHAR), + is_multi_device (type BOOLEAN), + possible_duplicate_of (type VARCHAR), + is_marketing_holdout (type BOOLEAN), + ), +); + +WITH customers AS ( + SELECT * + FROM __ref("stg_customers") +), + +identities AS ( + SELECT * + FROM __ref("int_customer_identity") +), + +addresses AS ( + SELECT * + FROM __ref("int_customer_address_current") +), + +rfm AS ( + SELECT * + FROM __ref("int_customer_rfm") +), + +consents AS ( + SELECT * + FROM __ref("int_customer_consent_state") +), + +tier_changes AS ( + SELECT * + FROM __ref("int_loyalty_tier_changes") +), + +devices AS ( + SELECT * + FROM __ref("int_cross_device_identity") +), + +current_tiers AS ( + SELECT + t.customer_id, + t.tier_code + FROM tier_changes AS t + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY t.customer_id + ORDER BY t.change_sequence DESC, t.tier_change_key DESC + ) = 1 +), + +final AS ( + SELECT + c.customer_id, + a.city, + a.postcode_canonical, + a.country_code, + COALESCE(a.region_code, c.region_code)::VARCHAR AS region_code, + i.email_domain, + i.full_name, + c.storefront_code, + c.preferred_language, + c.created_at, + CASE + WHEN c.birth_year IS NULL + THEN 'unknown' + WHEN YEAR('@@valuation_date'::DATE) - c.birth_year < 25 + THEN 'under_25' + WHEN YEAR('@@valuation_date'::DATE) - c.birth_year < 40 + THEN '25_39' + WHEN YEAR('@@valuation_date'::DATE) - c.birth_year < 60 + THEN '40_59' + ELSE '60_plus' + END::VARCHAR AS age_band, + COALESCE(r.order_count, 0)::INT AS order_count, + COALESCE(r.lifetime_revenue_usd, 0)::NUMBER(18, 2) AS lifetime_revenue_usd, + r.days_since_last_order, + COALESCE(r.segment, 'prospect')::VARCHAR AS rfm_segment, + COALESCE(r.is_vip, FALSE)::BOOLEAN AS is_vip, + COALESCE(s.email_marketing, c.marketing_opt_in, FALSE)::BOOLEAN AS email_marketing, + COALESCE(s.sms_marketing, FALSE)::BOOLEAN AS sms_marketing, + t.tier_code AS loyalty_tier_code, + COALESCE(d.is_multi_device, FALSE)::BOOLEAN AS is_multi_device, + i.duplicate_candidate_id AS possible_duplicate_of, + ( + MOD(ABS(HASH(c.customer_id)), 100) < 5 + )::BOOLEAN AS is_marketing_holdout + FROM customers AS c + LEFT JOIN identities AS i + ON c.customer_id = i.customer_id + LEFT JOIN addresses AS a + ON c.customer_id = a.customer_id + LEFT JOIN rfm AS r + ON c.customer_id = r.customer_id + LEFT JOIN consents AS s + ON c.customer_id = s.customer_id + LEFT JOIN current_tiers AS t + ON c.customer_id = t.customer_id + LEFT JOIN devices AS d + ON c.customer_id = d.customer_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/core/dim_date.sql b/benchmarks/duckshop/project/models/marts/core/dim_date.sql new file mode 100644 index 0000000000..d95dde6afd --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/core/dim_date.sql @@ -0,0 +1,100 @@ +MODEL ( + description """ + One row per calendar day of the three calendar years from the start of trading history, with + ISO week and fiscal calendar attributes and a holiday flag per region. + """, + contract enforced, + constants ( + _calendar_days 1095, + _fiscal_year_start_month 4, + ), + columns ( + calendar_date (type DATE, nullable false, audits [not_null, unique]), + day_of_week_iso (type INT), + day_name (type VARCHAR), + is_weekend (type BOOLEAN), + iso_year (type INT), + iso_week (type INT), + month_start (type DATE), + month_end (type DATE), + quarter_label (type VARCHAR), + fiscal_year (type INT), + fiscal_quarter (type INT), + holiday_regions (type ARRAY), + is_holiday_somewhere (type BOOLEAN), + is_on_or_before_valuation (type BOOLEAN), + ), +); + +WITH holidays AS ( + SELECT * + FROM __seed("holiday_calendar") +), + +generated AS ( + SELECT + DATEADD( + 'day', + ROW_NUMBER() OVER (ORDER BY SEQ4()) - 1, + @const("history_start_date")::DATE + ) AS calendar_date + FROM TABLE(GENERATOR(ROWCOUNT => @const("_calendar_days"))) +), + +holiday_days AS ( + SELECT + h.holiday_date, + ARRAY_AGG(DISTINCT h.region_code) WITHIN GROUP (ORDER BY h.region_code) AS holiday_regions + FROM holidays AS h + GROUP BY + h.holiday_date +), + +attributes AS ( + SELECT + g.calendar_date, + DAYOFWEEKISO(g.calendar_date) AS day_of_week_iso, + DAYNAME(g.calendar_date) AS day_name, + YEAROFWEEKISO(g.calendar_date) AS iso_year, + WEEKISO(g.calendar_date) AS iso_week, + DATE_TRUNC('month', g.calendar_date) AS month_start, + LAST_DAY(g.calendar_date, 'month') AS month_end, + YEAR(g.calendar_date) || '-Q' || QUARTER(g.calendar_date) AS quarter_label, + YEAR( + DATEADD('month', 12 - @const("_fiscal_year_start_month") + 1, g.calendar_date) + ) AS fiscal_year, + CEIL( + MONTH(DATEADD('month', 12 - @const("_fiscal_year_start_month") + 1, g.calendar_date)) / 3 + ) AS fiscal_quarter + FROM generated AS g +), + +final AS ( + SELECT + a.calendar_date::DATE AS calendar_date, + a.day_of_week_iso::INT AS day_of_week_iso, + a.day_name::VARCHAR AS day_name, + ( + a.day_of_week_iso >= 6 + )::BOOLEAN AS is_weekend, + a.iso_year::INT AS iso_year, + a.iso_week::INT AS iso_week, + a.month_start::DATE AS month_start, + a.month_end::DATE AS month_end, + a.quarter_label::VARCHAR AS quarter_label, + a.fiscal_year::INT AS fiscal_year, + a.fiscal_quarter::INT AS fiscal_quarter, + COALESCE(h.holiday_regions, ARRAY_CONSTRUCT())::ARRAY AS holiday_regions, + ( + h.holiday_date IS NOT NULL + )::BOOLEAN AS is_holiday_somewhere, + ( + a.calendar_date <= '@@valuation_date'::DATE + )::BOOLEAN AS is_on_or_before_valuation + FROM attributes AS a + LEFT JOIN holiday_days AS h + ON a.calendar_date = h.holiday_date +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/core/dim_products.sql b/benchmarks/duckshop/project/models/marts/core/dim_products.sql new file mode 100644 index 0000000000..792611c2c4 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/core/dim_products.sql @@ -0,0 +1,114 @@ +MODEL ( + description """ + One row per sellable SKU with its product, brand and category attributes, current USD list price + and margin, primary supplier, and review and wishlist signals. + """, + contract enforced, + model_schema audit_ts, + columns ( + sku (type VARCHAR, nullable false, audits [not_null, unique]), + product_id (type VARCHAR, audits [not_null]), + sku_family (type VARCHAR), + title (type VARCHAR), + variant_label (type VARCHAR), + brand_name (type VARCHAR), + is_private_label (type BOOLEAN), + category_level_1 (type VARCHAR), + category_slug_path (type VARCHAR), + is_active (type BOOLEAN), + launched_on (type DATE), + list_price_usd (type NUMBER(18, 4), audits [non_negative]), + margin_pct (type NUMBER(9, 2)), + primary_supplier_code (type VARCHAR), + supplier_count (type INT), + review_count (type INT), + average_rating (type NUMBER(4, 2)), + wishlisting_customers (type INT), + ), +); + +WITH catalogue AS ( + SELECT * + FROM __ref("int_product_catalogue") +), + +products AS ( + SELECT * + FROM __ref("stg_products") +), + +margins AS ( + SELECT * + FROM __ref("int_product_margin") +), + +supplier_costs AS ( + SELECT * + FROM __ref("int_supplier_cost_current") +), + +reviews AS ( + SELECT * + FROM __ref("int_reviews_moderated") +), + +wishlists AS ( + SELECT * + FROM __ref("int_wishlist_demand") +), + +usd_margins AS ( + SELECT + m.sku, + m.list_price, + m.margin_pct + FROM margins AS m + WHERE m.currency_code = 'USD' +), + +review_stats AS ( + SELECT + r.sku, + COUNT(*) AS review_count, + AVG(r.rating) AS average_rating + FROM reviews AS r + GROUP BY + r.sku +), + +final AS ( + SELECT + c.sku, + c.product_id, + c.sku_family, + c.title, + c.variant_label, + c.brand_name, + c.is_private_label, + c.category_level_1, + c.category_slug_path, + c.is_active, + c.launched_on, + m.list_price AS list_price_usd, + m.margin_pct, + s.supplier_code AS primary_supplier_code, + COALESCE(s.supplier_count, 0)::INT AS supplier_count, + COALESCE(r.review_count, 0)::INT AS review_count, + ROUND(r.average_rating, 2)::NUMBER(4, 2) AS average_rating, + COALESCE(w.wishlisting_customers, 0)::INT AS wishlisting_customers, + p.source_loaded_at + FROM catalogue AS c + INNER JOIN products AS p + ON c.product_id = p.product_id + LEFT JOIN usd_margins AS m + ON c.sku = m.sku + LEFT JOIN supplier_costs AS s + ON c.sku = s.sku + LEFT JOIN review_stats AS r + ON c.sku = r.sku + LEFT JOIN wishlists AS w + ON c.sku = w.sku +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/core/dim_promotions.sql b/benchmarks/duckshop/project/models/marts/core/dim_promotions.sql new file mode 100644 index 0000000000..5c648c326d --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/core/dim_promotions.sql @@ -0,0 +1,56 @@ +MODEL ( + description """ + One row per promotion code with its terms, eligible regions and lifetime redemption results + """, + contract enforced, + model_schema audit_ts, + columns ( + promo_code (type VARCHAR, nullable false, audits [not_null, unique]), + promo_name (type VARCHAR), + starts_at (type TIMESTAMP_TZ), + ends_at (type TIMESTAMP_TZ), + is_stackable (type BOOLEAN), + min_order_minor (type INT), + max_discount_pct (type NUMBER(5, 2)), + eligible_regions (type ARRAY), + is_active_now (type BOOLEAN), + redemption_count (type INT), + revenue_usd (type NUMBER(18, 2)), + discount_rate (type NUMBER(9, 6)), + basket_uplift_pct (type NUMBER(9, 2)), + ), +); + +WITH promotions AS ( + SELECT * + FROM __ref("stg_promotions") +), + +performance AS ( + SELECT * + FROM __ref("int_promotion_performance") +), + +final AS ( + SELECT + p.promo_code, + p.promo_name, + p.starts_at, + p.ends_at, + p.is_stackable, + p.min_order_minor, + p.max_discount_pct, + p.eligible_regions, + COALESCE(f.is_active_now, FALSE)::BOOLEAN AS is_active_now, + COALESCE(f.redemption_count, 0)::INT AS redemption_count, + COALESCE(f.revenue_usd, 0)::NUMBER(18, 2) AS revenue_usd, + f.discount_rate, + f.basket_uplift_pct, + p.source_loaded_at + FROM promotions AS p + LEFT JOIN performance AS f + ON p.promo_code = f.promo_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/core/dim_suppliers.sql b/benchmarks/duckshop/project/models/marts/core/dim_suppliers.sql new file mode 100644 index 0000000000..70ee438060 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/core/dim_suppliers.sql @@ -0,0 +1,80 @@ +MODEL ( + description """ + One row per supplier with payment terms parsed from the terms object, lead-time and fill-rate + performance, and the number of SKUs it currently supplies. + """, + contract enforced, + model_schema audit_ts, + columns ( + supplier_code (type VARCHAR, nullable false, audits [not_null, unique]), + supplier_name (type VARCHAR), + country_code (type VARCHAR), + payment_terms_days (type INT), + early_payment_discount_pct (type NUMBER(5, 2)), + terms_currency_code (type VARCHAR), + contact_domain (type VARCHAR), + supplied_sku_count (type INT), + lead_time_p50_days (type NUMBER(10, 2)), + on_time_rate (type NUMBER(9, 4)), + avg_fill_rate (type NUMBER(9, 4)), + performance_band (type VARCHAR), + ), +); + +WITH suppliers AS ( + SELECT * + FROM __ref("stg_suppliers") +), + +lead_times AS ( + SELECT * + FROM __ref("int_supplier_lead_times") +), + +costs AS ( + SELECT * + FROM __ref("int_supplier_cost_current") +), + +sku_counts AS ( + SELECT + c.supplier_code, + COUNT(*) AS supplied_sku_count + FROM costs AS c + GROUP BY + c.supplier_code +), + +final AS ( + SELECT + s.supplier_code, + s.supplier_name, + s.country_code, + s.payment_terms_days, + s.early_payment_discount_pct, + UPPER(s.payment_terms:currency::VARCHAR)::VARCHAR AS terms_currency_code, + LOWER(SPLIT_PART(s.primary_contact_email, '@', 2))::VARCHAR AS contact_domain, + COALESCE(k.supplied_sku_count, 0)::INT AS supplied_sku_count, + l.lead_time_p50_days, + l.on_time_rate, + l.avg_fill_rate, + CASE + WHEN l.on_time_rate IS NULL + THEN 'unrated' + WHEN l.on_time_rate >= 0.95 + AND l.avg_fill_rate >= 0.98 + THEN 'preferred' + WHEN l.on_time_rate >= 0.8 + THEN 'approved' + ELSE 'at_risk' + END::VARCHAR AS performance_band, + s.source_loaded_at + FROM suppliers AS s + LEFT JOIN lead_times AS l + ON s.supplier_code = l.supplier_code + LEFT JOIN sku_counts AS k + ON s.supplier_code = k.supplier_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/core/dim_warehouses.sql b/benchmarks/duckshop/project/models/marts/core/dim_warehouses.sql new file mode 100644 index 0000000000..40ec6e65d8 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/core/dim_warehouses.sql @@ -0,0 +1,123 @@ +MODEL ( + description """ + One row per warehouse with its location, reporting region, opening hours in local time and the + UTC offset on the valuation date, and current stock and slot capacity. + """, + contract enforced, + model_schema audit_ts, + columns ( + warehouse_code (type VARCHAR, nullable false, audits [not_null, unique]), + warehouse_name (type VARCHAR), + country_code (type VARCHAR, audits [not_null]), + region_code (type VARCHAR), + iana_timezone (type VARCHAR), + utc_offset_hours (type NUMBER(4, 2)), + opens_local (type TIME), + closes_local (type TIME), + opening_minutes (type INT), + supports_same_day (type BOOLEAN), + supports_cold_chain (type BOOLEAN), + bin_count (type INT), + units_on_hand (type INT), + average_slot_utilisation (type NUMBER(9, 4)), + ), +); + +WITH warehouses AS ( + SELECT * + FROM __ref("stg_warehouses") +), + +bins AS ( + SELECT * + FROM __ref("stg_bin_locations") +), + +valuation AS ( + SELECT * + FROM __ref("int_inventory_valuation_daily") +), + +slots AS ( + SELECT * + FROM __ref("int_delivery_slot_utilisation") +), + +countries AS ( + SELECT * + FROM __seed("countries") +), + +bin_counts AS ( + SELECT + b.warehouse_code, + COUNT(*) AS bin_count + FROM bins AS b + GROUP BY + b.warehouse_code +), + +latest_stock AS ( + SELECT + v.warehouse_code, + v.units_on_hand + FROM valuation AS v + WHERE v.calendar_date <= '@@valuation_date'::DATE + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY v.warehouse_code + ORDER BY v.calendar_date DESC, v.valuation_key + ) = 1 +), + +slot_averages AS ( + SELECT + s.warehouse_code, + AVG(s.utilisation) AS average_slot_utilisation + FROM slots AS s + GROUP BY + s.warehouse_code +), + +final AS ( + SELECT + w.warehouse_code, + w.warehouse_name, + w.country_code, + c.region_code::VARCHAR AS region_code, + w.iana_timezone, + ROUND( + DATEDIFF( + 'minute', + '@@valuation_date'::TIMESTAMP_NTZ, + CONVERT_TIMEZONE( + 'UTC', + w.iana_timezone, + '@@valuation_date'::TIMESTAMP_NTZ + ) + ) + / 60, + 2 + )::NUMBER(4, 2) AS utc_offset_hours, + w.opens_local, + w.closes_local, + w.opening_minutes, + w.supports_same_day, + w.supports_cold_chain, + COALESCE(b.bin_count, 0)::INT AS bin_count, + COALESCE(s.units_on_hand, 0)::INT AS units_on_hand, + ROUND(u.average_slot_utilisation, 4)::NUMBER(9, 4) AS average_slot_utilisation, + w.source_loaded_at + FROM warehouses AS w + LEFT JOIN countries AS c + ON w.country_code = c.country_code + LEFT JOIN bin_counts AS b + ON w.warehouse_code = b.warehouse_code + LEFT JOIN latest_stock AS s + ON w.warehouse_code = s.warehouse_code + LEFT JOIN slot_averages AS u + ON w.warehouse_code = u.warehouse_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/customers/rpt_consent_coverage.sql b/benchmarks/duckshop/project/models/marts/customers/rpt_consent_coverage.sql new file mode 100644 index 0000000000..ea147cb5eb --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/customers/rpt_consent_coverage.sql @@ -0,0 +1,52 @@ +MODEL ( + description "One row per region with the share of customers holding each marketing consent", + columns ( + region_code (type VARCHAR, nullable false, audits [not_null, unique]), + customers (type INT), + email_marketing_share (type NUMBER(9, 4)), + sms_marketing_share (type NUMBER(9, 4)), + personalisation_share (type NUMBER(9, 4)), + third_party_share (type NUMBER(9, 4)), + legacy_consent_customers (type INT), + ), +); + +WITH customers AS ( + SELECT * + FROM __ref("dim_customers") +), + +consents AS ( + SELECT * + FROM __ref("int_customer_consent_state") +), + +joined AS ( + SELECT + COALESCE(c.region_code, 'UNKNOWN') AS region_code, + COALESCE(s.email_marketing, FALSE) AS email_marketing, + COALESCE(s.sms_marketing, FALSE) AS sms_marketing, + COALESCE(s.personalisation, FALSE) AS personalisation, + COALESCE(s.third_party_sharing, FALSE) AS third_party_sharing, + COALESCE(s.has_legacy_consent, FALSE) AS has_legacy_consent + FROM customers AS c + LEFT JOIN consents AS s + ON c.customer_id = s.customer_id +), + +final AS ( + SELECT + j.region_code::VARCHAR AS region_code, + COUNT(*)::INT AS customers, + ROUND(AVG(IFF(j.email_marketing, 1, 0)), 4)::NUMBER(9, 4) AS email_marketing_share, + ROUND(AVG(IFF(j.sms_marketing, 1, 0)), 4)::NUMBER(9, 4) AS sms_marketing_share, + ROUND(AVG(IFF(j.personalisation, 1, 0)), 4)::NUMBER(9, 4) AS personalisation_share, + ROUND(AVG(IFF(j.third_party_sharing, 1, 0)), 4)::NUMBER(9, 4) AS third_party_share, + COUNT_IF(j.has_legacy_consent)::INT AS legacy_consent_customers + FROM joined AS j + GROUP BY + j.region_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/customers/rpt_customer_cohorts_monthly.sql b/benchmarks/duckshop/project/models/marts/customers/rpt_customer_cohorts_monthly.sql new file mode 100644 index 0000000000..821d4f8ef1 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/customers/rpt_customer_cohorts_monthly.sql @@ -0,0 +1,83 @@ +MODEL ( + description """ + One row per first-order cohort month and months since first order, with active customers, + retention against the cohort size and cumulative revenue per cohort customer. + """, + columns ( + cohort_cell_key (type VARCHAR, nullable false, audits [not_null, unique]), + cohort_month (type DATE, audits [not_null]), + months_since_first_order (type INT, audits [non_negative]), + cohort_size (type INT), + active_customers (type INT), + retention_rate (type NUMBER(9, 4)), + revenue_usd (type NUMBER(18, 2)), + cumulative_revenue_per_customer_usd (type NUMBER(18, 2)), + ), +); + +WITH order_sequence AS ( + SELECT * + FROM __ref("int_customer_order_sequence") +), + +activity AS ( + SELECT + s.customer_id, + s.cohort_month, + DATEDIFF( + 'month', + s.cohort_month, + DATE_TRUNC('month', s.placed_date_local) + ) AS months_since_first_order, + s.amount_usd + FROM order_sequence AS s +), + +cohort_sizes AS ( + SELECT + a.cohort_month, + COUNT(DISTINCT a.customer_id) AS cohort_size + FROM activity AS a + GROUP BY + a.cohort_month +), + +cells AS ( + SELECT + a.cohort_month, + a.months_since_first_order, + COUNT(DISTINCT a.customer_id) AS active_customers, + SUM(a.amount_usd) AS revenue_usd + FROM activity AS a + GROUP BY + a.cohort_month, + a.months_since_first_order +), + +final AS ( + SELECT + ( + TO_VARCHAR(c.cohort_month, 'YYYY-MM') || '+' || c.months_since_first_order::VARCHAR + )::VARCHAR AS cohort_cell_key, + c.cohort_month, + c.months_since_first_order::INT AS months_since_first_order, + z.cohort_size::INT AS cohort_size, + c.active_customers::INT AS active_customers, + ROUND(c.active_customers / NULLIF(z.cohort_size, 0), 4)::NUMBER(9, 4) AS retention_rate, + c.revenue_usd::NUMBER(18, 2) AS revenue_usd, + ROUND( + SUM(c.revenue_usd) OVER ( + PARTITION BY c.cohort_month + ORDER BY c.months_since_first_order + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) + / NULLIF(z.cohort_size, 0), + 2 + )::NUMBER(18, 2) AS cumulative_revenue_per_customer_usd + FROM cells AS c + INNER JOIN cohort_sizes AS z + ON c.cohort_month = z.cohort_month +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/customers/rpt_customer_order_lookup.sql b/benchmarks/duckshop/project/models/marts/customers/rpt_customer_order_lookup.sql new file mode 100644 index 0000000000..adcdd0add6 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/customers/rpt_customer_order_lookup.sql @@ -0,0 +1,80 @@ +MODEL ( + description """ + One row per order of the customer named by the lookup_customer_id variable, read through the + customer_orders table function so the customer filter is applied inside the function, with the + customer's profile alongside. generated_at records when the lookup ran as a TIMESTAMP_LTZ from + CURRENT_TIMESTAMP(), so it depends on the session time zone and the run, an expected divergence. + """, + tags [known_divergence], + columns ( + order_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR), + rfm_segment (type VARCHAR), + loyalty_tier_code (type VARCHAR), + placed_at (type TIMESTAMP_TZ), + order_status (type VARCHAR), + amount_usd (type NUMBER(18, 2)), + order_number (type INT), + running_total_usd (type NUMBER(18, 2)), + share_of_lifetime_revenue (type NUMBER(9, 6)), + generated_at (type TIMESTAMP_LTZ), + ), +); + +WITH customer_orders AS ( + SELECT * + FROM __table_fn("table_fn__customer_orders")('@@lookup_customer_id') +), + +customers AS ( + SELECT * + FROM __ref("dim_customers") +), + +keyed_orders AS ( + SELECT + '@@lookup_customer_id'::VARCHAR AS customer_id, + o.order_id, + o.placed_at, + o.order_status, + o.amount_usd, + o.order_number + FROM customer_orders AS o +), + +profile AS ( + SELECT + c.customer_id, + c.rfm_segment, + c.loyalty_tier_code, + c.lifetime_revenue_usd + FROM customers AS c + WHERE c.customer_id = '@@lookup_customer_id' +), + +final AS ( + SELECT + o.order_id::VARCHAR AS order_id, + o.customer_id, + p.rfm_segment, + p.loyalty_tier_code, + o.placed_at::TIMESTAMP_TZ AS placed_at, + o.order_status::VARCHAR AS order_status, + o.amount_usd::NUMBER(18, 2) AS amount_usd, + o.order_number::INT AS order_number, + SUM(o.amount_usd) OVER ( + ORDER BY o.order_number + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + )::NUMBER(18, 2) AS running_total_usd, + ROUND( + o.amount_usd / NULLIF(p.lifetime_revenue_usd, 0), + 6 + )::NUMBER(9, 6) AS share_of_lifetime_revenue, + CURRENT_TIMESTAMP()::TIMESTAMP_LTZ AS generated_at + FROM keyed_orders AS o + LEFT JOIN profile AS p + ON o.customer_id = p.customer_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/customers/rpt_gift_card_liability.sql b/benchmarks/duckshop/project/models/marts/customers/rpt_gift_card_liability.sql new file mode 100644 index 0000000000..12ade04b86 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/customers/rpt_gift_card_liability.sql @@ -0,0 +1,43 @@ +MODEL ( + description """ + One row per currency and liability status with the outstanding gift card balance in USD + """, + columns ( + liability_key (type VARCHAR, nullable false, audits [not_null, unique]), + currency_code (type VARCHAR), + liability_status (type VARCHAR), + card_count (type INT), + balance_usd (type NUMBER(18, 2), audits [non_negative]), + expiring_90d_usd (type NUMBER(18, 2)), + dormant_card_count (type INT), + ), +); + +WITH cards AS ( + SELECT * + FROM __ref("int_gift_card_balances") +), + +final AS ( + SELECT + ( + c.currency_code || ':' || c.liability_status + )::VARCHAR AS liability_key, + c.currency_code, + c.liability_status, + COUNT(*)::INT AS card_count, + SUM(GREATEST(c.balance_usd, 0))::NUMBER(18, 2) AS balance_usd, + SUM( + IFF(c.expires_on <= DATEADD('day', 90, '@@valuation_date'::DATE), c.balance_usd, 0) + )::NUMBER(18, 2) AS expiring_90d_usd, + COUNT_IF( + c.last_used_at < DATEADD('year', -1, '@@valuation_date'::TIMESTAMP_TZ) + )::INT AS dormant_card_count + FROM cards AS c + GROUP BY + c.currency_code, + c.liability_status +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/customers/rpt_loyalty_liability.sql b/benchmarks/duckshop/project/models/marts/customers/rpt_loyalty_liability.sql new file mode 100644 index 0000000000..898e8fd882 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/customers/rpt_loyalty_liability.sql @@ -0,0 +1,51 @@ +MODEL ( + description """ + One row per loyalty tier with outstanding points, their USD liability at the valuation date, and + points due to expire in the next 90 days. + """, + columns ( + tier_code (type VARCHAR, nullable false, audits [not_null, unique]), + member_count (type INT), + outstanding_points (type INT), + liability_usd (type NUMBER(18, 2)), + expiring_points (type INT), + average_points_per_member (type NUMBER(12, 2)), + ), +); + +WITH balances AS ( + SELECT * + FROM __ref("int_loyalty_balances") +), + +latest AS ( + SELECT + b.loyalty_account_id, + COALESCE(b.tier_code, 'NONE') AS tier_code, + b.running_points, + b.running_value_usd, + b.expiring_points + FROM balances AS b + WHERE b.posted_at <= '@@valuation_date'::TIMESTAMP_TZ + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY b.loyalty_account_id + ORDER BY b.posted_at DESC, b.entry_id DESC + ) = 1 +), + +final AS ( + SELECT + l.tier_code::VARCHAR AS tier_code, + COUNT(DISTINCT l.loyalty_account_id)::INT AS member_count, + SUM(l.running_points)::INT AS outstanding_points, + ROUND(SUM(l.running_value_usd), 2)::NUMBER(18, 2) AS liability_usd, + SUM(l.expiring_points)::INT AS expiring_points, + ROUND(AVG(l.running_points), 2)::NUMBER(12, 2) AS average_points_per_member + FROM latest AS l + GROUP BY + l.tier_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/customers/rpt_referral_program.sql b/benchmarks/duckshop/project/models/marts/customers/rpt_referral_program.sql new file mode 100644 index 0000000000..1607d55a03 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/customers/rpt_referral_program.sql @@ -0,0 +1,52 @@ +MODEL ( + description """ + One row per invite month with invites sent, acceptance, qualified referrals and reward cost + """, + columns ( + invite_month (type DATE, nullable false, audits [not_null, unique]), + invites_sent (type INT), + accepted (type INT), + qualified (type INT), + self_referrals (type INT), + acceptance_rate (type NUMBER(9, 4)), + reward_cost_usd (type NUMBER(18, 2)), + referred_revenue_usd (type NUMBER(18, 2)), + cost_per_qualified_usd (type NUMBER(18, 2)), + ), +); + +WITH referrals AS ( + SELECT * + FROM __ref("int_referral_conversions") +), + +monthly AS ( + SELECT + DATE_TRUNC('month', CONVERT_TIMEZONE('UTC', r.sent_at))::DATE AS invite_month, + COUNT(*) AS invites_sent, + COUNT(r.accepted_at) AS accepted, + COUNT_IF(r.is_qualified) AS qualified, + COUNT_IF(r.is_self_referral) AS self_referrals, + SUM(IFF(r.is_qualified, r.reward_usd, 0)) AS reward_cost_usd, + SUM(IFF(r.is_qualified, r.first_order_usd, 0)) AS referred_revenue_usd + FROM referrals AS r + GROUP BY + 1 +), + +final AS ( + SELECT + m.invite_month, + m.invites_sent::INT AS invites_sent, + m.accepted::INT AS accepted, + m.qualified::INT AS qualified, + m.self_referrals::INT AS self_referrals, + ROUND(m.accepted / NULLIF(m.invites_sent, 0), 4)::NUMBER(9, 4) AS acceptance_rate, + m.reward_cost_usd::NUMBER(18, 2) AS reward_cost_usd, + m.referred_revenue_usd::NUMBER(18, 2) AS referred_revenue_usd, + ROUND(m.reward_cost_usd / NULLIF(m.qualified, 0), 2)::NUMBER(18, 2) AS cost_per_qualified_usd + FROM monthly AS m +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/customers/rpt_rfm_segment_summary.sql b/benchmarks/duckshop/project/models/marts/customers/rpt_rfm_segment_summary.sql new file mode 100644 index 0000000000..938976e021 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/customers/rpt_rfm_segment_summary.sql @@ -0,0 +1,53 @@ +MODEL ( + description "One row per region and RFM segment with customer counts, revenue and VIP share", + columns ( + segment_key (type VARCHAR, nullable false, audits [not_null, unique]), + region_code (type VARCHAR), + rfm_segment (type VARCHAR), + customers (type INT), + vip_customers (type INT), + lifetime_revenue_usd (type NUMBER(18, 2)), + revenue_share_in_region (type NUMBER(9, 4)), + median_days_since_last_order (type NUMBER(10, 2)), + ), +); + +WITH customers AS ( + SELECT * + FROM __ref("dim_customers") +), + +grouped AS ( + SELECT + COALESCE(c.region_code, 'UNKNOWN') AS region_code, + c.rfm_segment, + COUNT(*) AS customers, + COUNT_IF(c.is_vip) AS vip_customers, + SUM(c.lifetime_revenue_usd) AS lifetime_revenue_usd, + MEDIAN(c.days_since_last_order) AS median_days_since_last_order + FROM customers AS c + GROUP BY + 1, + 2 +), + +final AS ( + SELECT + ( + g.region_code || ':' || g.rfm_segment + )::VARCHAR AS segment_key, + g.region_code::VARCHAR AS region_code, + g.rfm_segment, + g.customers::INT AS customers, + g.vip_customers::INT AS vip_customers, + g.lifetime_revenue_usd::NUMBER(18, 2) AS lifetime_revenue_usd, + ROUND( + RATIO_TO_REPORT(g.lifetime_revenue_usd) OVER (PARTITION BY g.region_code), + 4 + )::NUMBER(9, 4) AS revenue_share_in_region, + ROUND(g.median_days_since_last_order, 2)::NUMBER(10, 2) AS median_days_since_last_order + FROM grouped AS g +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/executive/rpt_executive_kpis_daily.sql b/benchmarks/duckshop/project/models/marts/executive/rpt_executive_kpis_daily.sql new file mode 100644 index 0000000000..582661324f --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/executive/rpt_executive_kpis_daily.sql @@ -0,0 +1,530 @@ +MODEL ( + description """ + One row per day, region and headline KPI across every business area, in long format, with the + value a week earlier, the week-over-week change and a trailing 28-day average. Metrics without a + natural region are reported under GLOBAL. This is deliberately the project's longest model: a + wide union of thin metric streams, as found in executive reporting layers. + """, + constants ( + _global_region "GLOBAL", + _history_days 120, + ), + columns ( + kpi_key (type VARCHAR, nullable false, audits [not_null, unique]), + kpi_date (type DATE, audits [not_null]), + region_code (type VARCHAR, audits [not_null]), + business_area (type VARCHAR), + metric_name (type VARCHAR, audits [not_null]), + metric_label (type VARCHAR), + unit (type VARCHAR), + higher_is_better (type BOOLEAN), + metric_value (type NUMBER(18, 4)), + value_week_ago (type NUMBER(18, 4)), + week_over_week_pct (type NUMBER(12, 2)), + trailing_28d_average (type NUMBER(18, 4)), + is_favourable_change (type BOOLEAN), + ), +); + +WITH orders AS ( + SELECT * + FROM __ref("fct_orders") +), + +refunds AS ( + SELECT * + FROM __ref("fct_refunds") +), + +web_sessions AS ( + SELECT * + FROM __ref("fct_web_sessions") +), + +tickets AS ( + SELECT * + FROM __ref("fct_support_tickets") +), + +shipments AS ( + SELECT * + FROM __ref("fct_shipments") +), + +warehouses AS ( + SELECT * + FROM __ref("dim_warehouses") +), + +campaign_spend AS ( + SELECT * + FROM __ref("fct_campaign_spend") +), + +inventory AS ( + SELECT * + FROM __ref("fct_inventory_daily") +), + +recognition AS ( + SELECT * + FROM __ref("int_revenue_recognition_daily") +), + +partner_orders AS ( + SELECT * + FROM __ref("fct_partner_orders") +), + +chargebacks AS ( + SELECT * + FROM __ref("int_chargeback_exposure") +), + +calendar AS ( + SELECT * + FROM __ref("dim_date") +), + +reporting_window AS ( + SELECT c.calendar_date + FROM calendar AS c + WHERE + c.calendar_date <= '@@valuation_date'::DATE + AND c.calendar_date > DATEADD( + 'day', + -@const("_history_days"), + '@@valuation_date'::DATE + ) +), + +completed_orders AS ( + SELECT + o.placed_date_local AS kpi_date, + COALESCE(o.region_code, @const("_global_region")) AS region_code, + o.amount_usd, + o.is_first_order + FROM orders AS o + WHERE o.order_status NOT IN ('CANCELLED', 'PENDING') +), + +order_metrics AS ( + SELECT + c.kpi_date, + c.region_code, + COUNT(*) AS order_count, + SUM(c.amount_usd) AS revenue_usd, + AVG(c.amount_usd) AS average_order_usd, + COUNT_IF(c.is_first_order) AS new_customers + FROM completed_orders AS c + GROUP BY + c.kpi_date, + c.region_code +), + +order_regions AS ( + SELECT + o.order_id, + COALESCE(o.region_code, @const("_global_region")) AS region_code + FROM orders AS o +), + +refund_metrics AS ( + SELECT + r.refunded_date AS kpi_date, + COALESCE(o.region_code, @const("_global_region")) AS region_code, + SUM(r.amount_usd) AS refund_usd, + COUNT(*) AS refund_count + FROM refunds AS r + LEFT JOIN order_regions AS o + ON r.order_id = o.order_id + GROUP BY + 1, + 2 +), + +session_metrics AS ( + SELECT + s.session_start_utc::DATE AS kpi_date, + COUNT(*) AS sessions, + AVG(IFF(s.converted, 1, 0)) AS conversion_rate, + AVG(IFF(s.is_bounce, 1, 0)) AS bounce_rate + FROM web_sessions AS s + GROUP BY + 1 +), + +ticket_metrics AS ( + SELECT + t.created_at_utc::DATE AS kpi_date, + COUNT(*) AS tickets, + AVG(IFF(t.is_first_response_breached, 1, 0)) AS first_response_breach_rate, + AVG(t.csat_score) AS average_csat + FROM tickets AS t + WHERE t.created_at_utc IS NOT NULL + GROUP BY + 1 +), + +warehouse_regions AS ( + SELECT + w.warehouse_code, + COALESCE(w.region_code, @const("_global_region")) AS region_code + FROM warehouses AS w +), + +shipment_metrics AS ( + SELECT + CONVERT_TIMEZONE('UTC', s.shipped_at)::DATE AS kpi_date, + COALESCE(w.region_code, @const("_global_region")) AS region_code, + COUNT(*) AS shipments, + AVG(IFF(s.is_dispatch_on_time, 1, 0)) AS on_time_dispatch_rate, + AVG(s.hours_to_deliver) AS average_hours_to_deliver + FROM shipments AS s + LEFT JOIN warehouse_regions AS w + ON s.warehouse_code = w.warehouse_code + WHERE s.shipped_at IS NOT NULL + GROUP BY + 1, + 2 +), + +spend_metrics AS ( + SELECT + s.spend_date AS kpi_date, + SUM(s.spend_usd) AS spend_usd, + SUM(s.clicks) AS paid_clicks + FROM campaign_spend AS s + GROUP BY + 1 +), + +stock_metrics AS ( + SELECT + i.calendar_date AS kpi_date, + COALESCE(w.region_code, @const("_global_region")) AS region_code, + COUNT_IF(i.is_stocked_out) AS stocked_out_sku_days, + SUM(i.stock_value_usd) AS stock_value_usd + FROM inventory AS i + LEFT JOIN warehouse_regions AS w + ON i.warehouse_code = w.warehouse_code + GROUP BY + 1, + 2 +), + +subscription_metrics AS ( + SELECT + r.recognition_date AS kpi_date, + r.region_code, + SUM(r.revenue_usd) AS subscription_revenue_usd + FROM recognition AS r + WHERE r.revenue_stream = 'subscription' + GROUP BY + 1, + 2 +), + +partner_metrics AS ( + SELECT + p.placed_date AS kpi_date, + COUNT(*) AS partner_orders, + SUM(p.order_total_usd) AS partner_revenue_usd + FROM partner_orders AS p + GROUP BY + 1 +), + +chargeback_metrics AS ( + SELECT + CONVERT_TIMEZONE('UTC', c.opened_at)::DATE AS kpi_date, + COUNT(*) AS chargebacks_opened, + SUM(c.amount_usd) AS chargeback_usd + FROM chargebacks AS c + GROUP BY + 1 +), + +all_metrics AS ( + SELECT + m.kpi_date::DATE AS kpi_date, + m.region_code::VARCHAR AS region_code, + 'orders'::VARCHAR AS metric_name, + m.order_count::NUMBER(18, 4) AS metric_value + FROM order_metrics AS m + UNION ALL + SELECT + m.kpi_date::DATE AS kpi_date, + m.region_code::VARCHAR AS region_code, + 'revenue_usd'::VARCHAR AS metric_name, + m.revenue_usd::NUMBER(18, 4) AS metric_value + FROM order_metrics AS m + UNION ALL + SELECT + m.kpi_date::DATE AS kpi_date, + m.region_code::VARCHAR AS region_code, + 'average_order_usd'::VARCHAR AS metric_name, + m.average_order_usd::NUMBER(18, 4) AS metric_value + FROM order_metrics AS m + UNION ALL + SELECT + m.kpi_date::DATE AS kpi_date, + m.region_code::VARCHAR AS region_code, + 'new_customers'::VARCHAR AS metric_name, + m.new_customers::NUMBER(18, 4) AS metric_value + FROM order_metrics AS m + UNION ALL + SELECT + r.kpi_date::DATE AS kpi_date, + r.region_code::VARCHAR AS region_code, + 'refund_usd'::VARCHAR AS metric_name, + r.refund_usd::NUMBER(18, 4) AS metric_value + FROM refund_metrics AS r + UNION ALL + SELECT + r.kpi_date::DATE AS kpi_date, + r.region_code::VARCHAR AS region_code, + 'refunds'::VARCHAR AS metric_name, + r.refund_count::NUMBER(18, 4) AS metric_value + FROM refund_metrics AS r + UNION ALL + SELECT + s.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'web_sessions'::VARCHAR AS metric_name, + s.sessions::NUMBER(18, 4) AS metric_value + FROM session_metrics AS s + UNION ALL + SELECT + s.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'web_conversion_rate'::VARCHAR AS metric_name, + s.conversion_rate::NUMBER(18, 4) AS metric_value + FROM session_metrics AS s + UNION ALL + SELECT + s.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'web_bounce_rate'::VARCHAR AS metric_name, + s.bounce_rate::NUMBER(18, 4) AS metric_value + FROM session_metrics AS s + UNION ALL + SELECT + t.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'support_tickets'::VARCHAR AS metric_name, + t.tickets::NUMBER(18, 4) AS metric_value + FROM ticket_metrics AS t + UNION ALL + SELECT + t.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'first_response_breach_rate'::VARCHAR AS metric_name, + t.first_response_breach_rate::NUMBER(18, 4) AS metric_value + FROM ticket_metrics AS t + UNION ALL + SELECT + t.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'average_csat'::VARCHAR AS metric_name, + t.average_csat::NUMBER(18, 4) AS metric_value + FROM ticket_metrics AS t + UNION ALL + SELECT + s.kpi_date::DATE AS kpi_date, + s.region_code::VARCHAR AS region_code, + 'shipments'::VARCHAR AS metric_name, + s.shipments::NUMBER(18, 4) AS metric_value + FROM shipment_metrics AS s + UNION ALL + SELECT + s.kpi_date::DATE AS kpi_date, + s.region_code::VARCHAR AS region_code, + 'on_time_dispatch_rate'::VARCHAR AS metric_name, + s.on_time_dispatch_rate::NUMBER(18, 4) AS metric_value + FROM shipment_metrics AS s + UNION ALL + SELECT + s.kpi_date::DATE AS kpi_date, + s.region_code::VARCHAR AS region_code, + 'average_hours_to_deliver'::VARCHAR AS metric_name, + s.average_hours_to_deliver::NUMBER(18, 4) AS metric_value + FROM shipment_metrics AS s + UNION ALL + SELECT + p.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'marketing_spend_usd'::VARCHAR AS metric_name, + p.spend_usd::NUMBER(18, 4) AS metric_value + FROM spend_metrics AS p + UNION ALL + SELECT + p.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'paid_clicks'::VARCHAR AS metric_name, + p.paid_clicks::NUMBER(18, 4) AS metric_value + FROM spend_metrics AS p + UNION ALL + SELECT + k.kpi_date::DATE AS kpi_date, + k.region_code::VARCHAR AS region_code, + 'stocked_out_sku_days'::VARCHAR AS metric_name, + k.stocked_out_sku_days::NUMBER(18, 4) AS metric_value + FROM stock_metrics AS k + UNION ALL + SELECT + k.kpi_date::DATE AS kpi_date, + k.region_code::VARCHAR AS region_code, + 'stock_value_usd'::VARCHAR AS metric_name, + k.stock_value_usd::NUMBER(18, 4) AS metric_value + FROM stock_metrics AS k + UNION ALL + SELECT + b.kpi_date::DATE AS kpi_date, + b.region_code::VARCHAR AS region_code, + 'subscription_revenue_usd'::VARCHAR AS metric_name, + b.subscription_revenue_usd::NUMBER(18, 4) AS metric_value + FROM subscription_metrics AS b + UNION ALL + SELECT + p.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'partner_orders'::VARCHAR AS metric_name, + p.partner_orders::NUMBER(18, 4) AS metric_value + FROM partner_metrics AS p + UNION ALL + SELECT + p.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'partner_revenue_usd'::VARCHAR AS metric_name, + p.partner_revenue_usd::NUMBER(18, 4) AS metric_value + FROM partner_metrics AS p + UNION ALL + SELECT + c.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'chargebacks_opened'::VARCHAR AS metric_name, + c.chargebacks_opened::NUMBER(18, 4) AS metric_value + FROM chargeback_metrics AS c + UNION ALL + SELECT + c.kpi_date::DATE AS kpi_date, + @const("_global_region")::VARCHAR AS region_code, + 'chargeback_usd'::VARCHAR AS metric_name, + c.chargeback_usd::NUMBER(18, 4) AS metric_value + FROM chargeback_metrics AS c +), + +metric_catalogue AS ( + SELECT + v.metric_name, + v.business_area, + v.metric_label, + v.unit, + v.higher_is_better + FROM VALUES + ('orders', 'orders', 'Orders', 'count', TRUE), + ('revenue_usd', 'orders', 'Revenue', 'usd', TRUE), + ('average_order_usd', 'orders', 'Average order value', 'usd', TRUE), + ('new_customers', 'customers', 'New customers', 'count', TRUE), + ('refund_usd', 'orders', 'Refunded', 'usd', FALSE), + ('refunds', 'orders', 'Refunds', 'count', FALSE), + ('web_sessions', 'clickstream', 'Web sessions', 'count', TRUE), + ('web_conversion_rate', 'clickstream', 'Web conversion rate', 'ratio', TRUE), + ('web_bounce_rate', 'clickstream', 'Web bounce rate', 'ratio', FALSE), + ('support_tickets', 'support', 'Support tickets', 'count', FALSE), + ('first_response_breach_rate', 'support', 'First response SLA breaches', 'ratio', FALSE), + ('average_csat', 'support', 'Average CSAT', 'score', TRUE), + ('shipments', 'fulfilment', 'Shipments dispatched', 'count', TRUE), + ('on_time_dispatch_rate', 'fulfilment', 'On-time dispatch', 'ratio', TRUE), + ('average_hours_to_deliver', 'fulfilment', 'Hours to deliver', 'hours', FALSE), + ('marketing_spend_usd', 'growth', 'Marketing spend', 'usd', FALSE), + ('paid_clicks', 'growth', 'Paid clicks', 'count', TRUE), + ('stocked_out_sku_days', 'inventory', 'Stocked-out SKU days', 'count', FALSE), + ('stock_value_usd', 'inventory', 'Stock value', 'usd', FALSE), + ('subscription_revenue_usd', 'finance', 'Subscription revenue', 'usd', TRUE), + ('partner_orders', 'partners', 'Partner orders', 'count', TRUE), + ('partner_revenue_usd', 'partners', 'Partner revenue', 'usd', TRUE), + ('chargebacks_opened', 'finance', 'Chargebacks opened', 'count', FALSE), + ('chargeback_usd', 'finance', 'Chargeback amount', 'usd', FALSE) AS v(metric_name, business_area, metric_label, unit, higher_is_better) +), + +metric_regions AS ( + SELECT DISTINCT + a.region_code, + a.metric_name + FROM all_metrics AS a +), + +dense_grid AS ( + SELECT + w.calendar_date AS kpi_date, + r.region_code, + r.metric_name + FROM reporting_window AS w + CROSS JOIN metric_regions AS r +), + +filled AS ( + SELECT + g.kpi_date, + g.region_code, + g.metric_name, + COALESCE(a.metric_value, 0) AS metric_value + FROM dense_grid AS g + LEFT JOIN all_metrics AS a + ON g.kpi_date = a.kpi_date + AND g.region_code = a.region_code + AND g.metric_name = a.metric_name +), + +compared AS ( + SELECT + f.kpi_date, + f.region_code, + f.metric_name, + f.metric_value, + SUM(f.metric_value) OVER ( + PARTITION BY f.region_code, f.metric_name + ORDER BY f.kpi_date + RANGE BETWEEN INTERVAL '7 days' PRECEDING AND INTERVAL '7 days' PRECEDING + ) AS value_week_ago, + AVG(f.metric_value) OVER ( + PARTITION BY f.region_code, f.metric_name + ORDER BY f.kpi_date + RANGE BETWEEN INTERVAL '27 days' PRECEDING AND CURRENT ROW + ) AS trailing_28d_average + FROM filled AS f +), + +final AS ( + SELECT + ( + TO_VARCHAR(c.kpi_date, 'YYYY-MM-DD') || ':' || c.region_code || ':' || c.metric_name + )::VARCHAR AS kpi_key, + c.kpi_date::DATE AS kpi_date, + c.region_code::VARCHAR AS region_code, + m.business_area::VARCHAR AS business_area, + c.metric_name::VARCHAR AS metric_name, + m.metric_label::VARCHAR AS metric_label, + m.unit::VARCHAR AS unit, + m.higher_is_better::BOOLEAN AS higher_is_better, + c.metric_value::NUMBER(18, 4) AS metric_value, + c.value_week_ago::NUMBER(18, 4) AS value_week_ago, + ROUND(100 + * ( + c.metric_value / NULLIF(c.value_week_ago, 0) - 1 + ), 2)::NUMBER(12, 2) AS week_over_week_pct, + ROUND(c.trailing_28d_average, 4)::NUMBER(18, 4) AS trailing_28d_average, + IFF( + m.higher_is_better, + c.metric_value >= c.value_week_ago, + c.metric_value <= c.value_week_ago + )::BOOLEAN AS is_favourable_change + FROM compared AS c + LEFT JOIN metric_catalogue AS m + ON c.metric_name = m.metric_name +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/executive/rpt_region_scorecard_monthly.sql b/benchmarks/duckshop/project/models/marts/executive/rpt_region_scorecard_monthly.sql new file mode 100644 index 0000000000..b50bbdb5c5 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/executive/rpt_region_scorecard_monthly.sql @@ -0,0 +1,93 @@ +MODEL ( + description """ + One row per month, region and scorecard measure, unpivoted from a wide monthly summary of sales, + retention and fulfilment so dashboards can chart any measure the same way. + """, + columns ( + scorecard_key (type VARCHAR, nullable false, audits [not_null, unique]), + scorecard_month (type DATE), + region_code (type VARCHAR), + measure (type VARCHAR), + measure_value (type NUMBER(18, 4)), + rank_among_regions (type INT), + ), +); + +WITH sales AS ( + SELECT * + FROM __ref("rpt_daily_sales_by_region") +), + +recognition AS ( + SELECT * + FROM __ref("rpt_revenue_recognition_monthly") +), + +monthly_sales AS ( + SELECT + DATE_TRUNC('month', s.placed_date_local)::DATE AS scorecard_month, + s.region_code, + SUM(s.order_count)::NUMBER(18, 4) AS orders, + SUM(s.revenue_usd)::NUMBER(18, 4) AS revenue_usd, + SUM(s.first_order_count)::NUMBER(18, 4) AS first_orders, + ( + SUM(s.revenue_usd) / NULLIF(SUM(s.order_count), 0) + )::NUMBER(18, 4) AS average_order_usd + FROM sales AS s + GROUP BY + 1, + 2 +), + +wide AS ( + SELECT + m.scorecard_month, + m.region_code, + m.orders, + m.revenue_usd, + m.first_orders, + m.average_order_usd, + COALESCE(r.refund_ratio, 0)::NUMBER(18, 4) AS refund_ratio, + COALESCE(r.subscription_usd, 0)::NUMBER(18, 4) AS subscription_usd + FROM monthly_sales AS m + LEFT JOIN recognition AS r + ON m.scorecard_month = r.recognition_month + AND m.region_code = r.region_code +), + +long_form AS ( + SELECT + u.scorecard_month, + u.region_code, + u.measure, + u.measure_value + FROM wide UNPIVOT( + measure_value FOR measure IN ( + orders, + revenue_usd, + first_orders, + average_order_usd, + refund_ratio, + subscription_usd + ) + ) AS u +), + +final AS ( + SELECT + ( + TO_VARCHAR(l.scorecard_month, 'YYYY-MM') || ':' || l.region_code || ':' || LOWER(l.measure) + )::VARCHAR AS scorecard_key, + l.scorecard_month::DATE AS scorecard_month, + l.region_code::VARCHAR AS region_code, + LOWER(l.measure)::VARCHAR AS measure, + l.measure_value::NUMBER(18, 4) AS measure_value, + RANK() OVER ( + PARTITION BY l.scorecard_month, l.measure + ORDER BY l.measure_value DESC + )::INT AS rank_among_regions + FROM long_form AS l +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/finance/fct_ledger_month_end.sql b/benchmarks/duckshop/project/models/marts/finance/fct_ledger_month_end.sql new file mode 100644 index 0000000000..3ca5becb3f --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/finance/fct_ledger_month_end.sql @@ -0,0 +1,100 @@ +MODEL ( + description """ + One row per ledger account, currency and month end with the closing balance in major units and + USD, the month's debits and credits, and whether the balance sits on its normal side. + """, + contract enforced, + audit_factories [fact_table_quality], + columns ( + ledger_month_key (type VARCHAR, nullable false, audits [not_null, unique]), + account_code (type VARCHAR, audits [not_null]), + account_class (type VARCHAR, audits [ + accepted_values (values ["asset", "liability", "equity", "revenue", "expense"]), + ]), + currency_code (type VARCHAR), + month_end (type DATE, audits [not_null]), + month_debit_minor (type INT), + month_credit_minor (type INT), + closing_balance (type NUMBER(38, 6)), + closing_balance_usd (type NUMBER(18, 2)), + is_normal_side (type BOOLEAN), + ), +); + +WITH balances AS ( + SELECT * + FROM __ref("int_ledger_balances_daily") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +month_ends AS ( + SELECT + b.account_code, + b.account_class, + b.currency_code, + b.calendar_date AS month_end, + LAST_DAY(b.calendar_date) AS month_last_day, + b.closing_balance, + b.is_normal_side + FROM balances AS b + WHERE + b.is_month_end + OR b.calendar_date = '@@valuation_date'::DATE +), + +month_flows AS ( + SELECT + b.account_code, + b.currency_code, + LAST_DAY(b.calendar_date) AS month_last_day, + SUM(b.debit_minor) AS month_debit_minor, + SUM(b.credit_minor) AS month_credit_minor + FROM balances AS b + GROUP BY + 1, + 2, + 3 +), + +with_rate AS ( + SELECT + m.account_code, + m.account_class, + m.currency_code, + m.month_end, + m.month_last_day, + m.closing_balance, + m.is_normal_side, + fx.rate_to_usd + FROM month_ends AS m + ASOF JOIN fx_rates AS fx MATCH_CONDITION (m.month_end >= fx.rate_date) + ON m.currency_code = fx.currency_code +), + +final AS ( + SELECT + ( + w.account_code || ':' || w.currency_code || ':' || TO_VARCHAR(w.month_end, 'YYYY-MM-DD') + )::VARCHAR AS ledger_month_key, + w.account_code, + w.account_class, + w.currency_code, + w.month_end, + COALESCE(f.month_debit_minor, 0)::INT AS month_debit_minor, + COALESCE(f.month_credit_minor, 0)::INT AS month_credit_minor, + w.closing_balance, + ROUND(w.closing_balance * w.rate_to_usd, 2)::NUMBER(18, 2) AS closing_balance_usd, + w.is_normal_side + FROM with_rate AS w + LEFT JOIN month_flows AS f + ON w.account_code = f.account_code + AND w.currency_code = f.currency_code + AND w.month_last_day = f.month_last_day +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/finance/rpt_chargeback_exposure.sql b/benchmarks/duckshop/project/models/marts/finance/rpt_chargeback_exposure.sql new file mode 100644 index 0000000000..f24838b5ef --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/finance/rpt_chargeback_exposure.sql @@ -0,0 +1,43 @@ +MODEL ( + description """ + One row per reason family and outcome with chargeback counts, USD at risk and overdue responses + """, + columns ( + exposure_key (type VARCHAR, nullable false, audits [not_null, unique]), + reason_family (type VARCHAR), + outcome (type VARCHAR), + chargebacks (type INT), + amount_usd (type NUMBER(18, 2)), + at_risk_usd (type NUMBER(18, 2)), + overdue_responses (type INT), + with_delivery_proof (type INT), + repeat_customer_chargebacks (type INT), + ), +); + +WITH chargebacks AS ( + SELECT * + FROM __ref("int_chargeback_exposure") +), + +final AS ( + SELECT + ( + c.reason_family || ':' || c.outcome + )::VARCHAR AS exposure_key, + c.reason_family, + c.outcome, + COUNT(*)::INT AS chargebacks, + SUM(c.amount_usd)::NUMBER(18, 2) AS amount_usd, + SUM(c.at_risk_usd)::NUMBER(18, 2) AS at_risk_usd, + COUNT_IF(c.is_overdue)::INT AS overdue_responses, + COUNT_IF(c.has_delivery_proof)::INT AS with_delivery_proof, + COUNT_IF(c.prior_chargebacks_for_customer > 0)::INT AS repeat_customer_chargebacks + FROM chargebacks AS c + GROUP BY + c.reason_family, + c.outcome +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/finance/rpt_invoice_sequence_gaps.sql b/benchmarks/duckshop/project/models/marts/finance/rpt_invoice_sequence_gaps.sql new file mode 100644 index 0000000000..2943d9fd9c --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/finance/rpt_invoice_sequence_gaps.sql @@ -0,0 +1,40 @@ +MODEL ( + description """ + One row per invoice series with issued invoices, numbering gaps and tax reconciliation failures + """, + columns ( + invoice_series (type VARCHAR, nullable false, audits [not_null, unique]), + invoices (type INT), + first_sequence (type INT), + last_sequence (type INT), + gap_count (type INT), + missing_numbers (type INT), + unbalanced_headers (type INT), + tax_disagreements (type INT), + currency_mismatches (type INT), + ), +); + +WITH invoices AS ( + SELECT * + FROM __ref("int_invoices_reconciled") +), + +final AS ( + SELECT + i.invoice_series, + COUNT(*)::INT AS invoices, + MIN(i.invoice_sequence)::INT AS first_sequence, + MAX(i.invoice_sequence)::INT AS last_sequence, + COUNT(i.sequence_gap)::INT AS gap_count, + COALESCE(SUM(i.sequence_gap), 0)::INT AS missing_numbers, + COUNT_IF(NOT i.header_balances)::INT AS unbalanced_headers, + COUNT_IF(NOT i.tax_lines_agree OR NOT i.matches_order_tax)::INT AS tax_disagreements, + COUNT_IF(NOT i.invoice_currency_matches_order)::INT AS currency_mismatches + FROM invoices AS i + GROUP BY + i.invoice_series +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/finance/rpt_payout_reconciliation_status.sql b/benchmarks/duckshop/project/models/marts/finance/rpt_payout_reconciliation_status.sql new file mode 100644 index 0000000000..0dab670d5e --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/finance/rpt_payout_reconciliation_status.sql @@ -0,0 +1,51 @@ +MODEL ( + description """ + One row per provider and payout month with payouts by reconciliation status and the unreconciled + amount + """, + columns ( + payout_status_key (type VARCHAR, nullable false, audits [not_null, unique]), + provider_code (type VARCHAR), + payout_month (type DATE), + payouts (type INT), + reconciled (type INT), + awaiting_report (type INT), + mismatched (type INT), + orphan_lines (type INT), + unreconciled_net_minor (type INT), + reconciled_share (type NUMBER(9, 4)), + ), +); + +WITH payouts AS ( + SELECT * + FROM __ref("int_payout_reconciliation") +), + +final AS ( + SELECT + ( + p.provider_code || ':' || TO_VARCHAR(DATE_TRUNC('month', p.paid_out_on), 'YYYY-MM') + )::VARCHAR AS payout_status_key, + p.provider_code, + DATE_TRUNC('month', p.paid_out_on)::DATE AS payout_month, + COUNT(*)::INT AS payouts, + COUNT_IF(p.reconciliation_status = 'reconciled')::INT AS reconciled, + COUNT_IF(p.reconciliation_status = 'awaiting_report')::INT AS awaiting_report, + COUNT_IF(p.reconciliation_status IN ('line_mismatch', 'report_mismatch'))::INT AS mismatched, + SUM(p.orphan_line_count)::INT AS orphan_lines, + SUM( + IFF(p.reconciliation_status <> 'reconciled', p.payout_net_minor, 0) + )::INT AS unreconciled_net_minor, + ROUND( + AVG(IFF(p.reconciliation_status = 'reconciled', 1, 0)), + 4 + )::NUMBER(9, 4) AS reconciled_share + FROM payouts AS p + GROUP BY + p.provider_code, + DATE_TRUNC('month', p.paid_out_on) +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/finance/rpt_revenue_recognition_monthly.sql b/benchmarks/duckshop/project/models/marts/finance/rpt_revenue_recognition_monthly.sql new file mode 100644 index 0000000000..3bd8a61e1b --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/finance/rpt_revenue_recognition_monthly.sql @@ -0,0 +1,56 @@ +MODEL ( + description """ + One row per month and region with recognised product, refund and subscription revenue in USD + """, + columns ( + recognition_month_key (type VARCHAR, nullable false, audits [not_null, unique]), + recognition_month (type DATE), + region_code (type VARCHAR), + product_usd (type NUMBER(18, 2)), + refund_usd (type NUMBER(18, 2)), + subscription_usd (type NUMBER(18, 2)), + net_revenue_usd (type NUMBER(18, 2)), + refund_ratio (type NUMBER(9, 4)), + ), +); + +WITH recognition AS ( + SELECT * + FROM __ref("int_revenue_recognition_daily") +), + +monthly AS ( + SELECT + DATE_TRUNC('month', r.recognition_date) AS recognition_month, + r.region_code, + r.revenue_stream, + r.revenue_usd + FROM recognition AS r +), + +pivoted AS ( + SELECT * + FROM monthly PIVOT( + SUM(revenue_usd) FOR revenue_stream IN ('product', 'refund', 'subscription') + ) AS p(recognition_month, region_code, product_usd, refund_usd, subscription_usd) +), + +final AS ( + SELECT + ( + TO_VARCHAR(p.recognition_month, 'YYYY-MM') || ':' || p.region_code + )::VARCHAR AS recognition_month_key, + p.recognition_month::DATE AS recognition_month, + p.region_code::VARCHAR AS region_code, + COALESCE(p.product_usd, 0)::NUMBER(18, 2) AS product_usd, + COALESCE(p.refund_usd, 0)::NUMBER(18, 2) AS refund_usd, + COALESCE(p.subscription_usd, 0)::NUMBER(18, 2) AS subscription_usd, + ( + COALESCE(p.product_usd, 0) + COALESCE(p.refund_usd, 0) + COALESCE(p.subscription_usd, 0) + )::NUMBER(18, 2) AS net_revenue_usd, + ROUND(-p.refund_usd / NULLIF(p.product_usd, 0), 4)::NUMBER(9, 4) AS refund_ratio + FROM pivoted AS p +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/finance/rpt_tax_variance_by_country.sql b/benchmarks/duckshop/project/models/marts/finance/rpt_tax_variance_by_country.sql new file mode 100644 index 0000000000..48fc3a240d --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/finance/rpt_tax_variance_by_country.sql @@ -0,0 +1,47 @@ +MODEL ( + description """ + One row per ship-to country and quarter with tax lines checked, rate mismatches and net variance + """, + columns ( + tax_variance_key (type VARCHAR, nullable false, audits [not_null, unique]), + ship_to_country_code (type VARCHAR), + issued_quarter (type DATE), + tax_lines (type INT), + lines_without_rate (type INT), + rate_mismatches (type INT), + charged_tax_minor (type INT), + net_variance_minor (type INT), + absolute_variance_minor (type INT), + ambiguous_rate_lines (type INT), + ), +); + +WITH tax_lines AS ( + SELECT * + FROM __ref("int_tax_applied") +), + +final AS ( + SELECT + ( + COALESCE(t.ship_to_country_code, '??') + || ':' + || TO_VARCHAR(DATE_TRUNC('quarter', t.issued_date), 'YYYY-MM-DD') + )::VARCHAR AS tax_variance_key, + COALESCE(t.ship_to_country_code, '??')::VARCHAR AS ship_to_country_code, + DATE_TRUNC('quarter', t.issued_date)::DATE AS issued_quarter, + COUNT(*)::INT AS tax_lines, + COUNT_IF(NOT t.has_published_rate)::INT AS lines_without_rate, + COUNT_IF(t.is_rate_mismatch)::INT AS rate_mismatches, + SUM(t.charged_tax_minor)::INT AS charged_tax_minor, + SUM(t.variance_minor)::INT AS net_variance_minor, + SUM(ABS(t.variance_minor))::INT AS absolute_variance_minor, + COUNT_IF(t.candidate_rate_count > 1)::INT AS ambiguous_rate_lines + FROM tax_lines AS t + GROUP BY + COALESCE(t.ship_to_country_code, '??'), + DATE_TRUNC('quarter', t.issued_date) +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/fulfilment/fct_shipments.sql b/benchmarks/duckshop/project/models/marts/fulfilment/fct_shipments.sql new file mode 100644 index 0000000000..05f4bec5af --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/fulfilment/fct_shipments.sql @@ -0,0 +1,108 @@ +MODEL ( + description """ + One row per shipment with its fulfilment method, carrier, dispatch and delivery timing against + the business-hours promise, package costs, and delivery exceptions. + """, + contract enforced, + audit_factories [fact_table_quality], + post_hooks [ + inline_sql("ALTER TABLE @@CTX:destination.qualified SET CHANGE_TRACKING = TRUE"), + ], + columns ( + shipment_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + fulfilment_method (type fulfilment_method), + warehouse_code (type VARCHAR), + carrier_code (type VARCHAR), + service_level (type VARCHAR), + shipped_at (type TIMESTAMP_TZ), + delivered_at (type TIMESTAMP_TZ), + business_days_to_ship (type INT), + is_dispatch_on_time (type BOOLEAN), + hours_to_deliver (type NUMBER(10, 2)), + package_count (type INT), + chargeable_weight_kg (type NUMBER(10, 3)), + rate_minor (type INT), + exception_count (type INT), + compensation_usd (type NUMBER(18, 2)), + tracking_event_count (type INT), + ), +); + +WITH shipments AS ( + SELECT * + FROM __ref("stg_shipments") +), + +sla AS ( + SELECT * + FROM __ref("int_fulfilment_sla") +), + +packages AS ( + SELECT * + FROM __ref("int_shipment_costs") +), + +exceptions AS ( + SELECT * + FROM __ref("int_delivery_exceptions_classified") +), + +package_totals AS ( + SELECT + p.shipment_id, + COUNT(*) AS package_count, + SUM(p.chargeable_weight_kg) AS chargeable_weight_kg, + SUM(p.rate_minor) AS rate_minor + FROM packages AS p + GROUP BY + p.shipment_id +), + +exception_totals AS ( + SELECT + e.shipment_id, + COUNT(*) AS exception_count, + SUM(e.compensation_usd) AS compensation_usd + FROM exceptions AS e + GROUP BY + e.shipment_id +), + +final AS ( + SELECT + s.shipment_id, + s.order_id, + CASE + WHEN s.warehouse_code IS NULL + THEN @enum("fulfilment_method").PARTNER + WHEN s.service_level ILIKE '%collect%' + THEN @enum("fulfilment_method").CLICK_AND_COLLECT + ELSE @enum("fulfilment_method").HOME_DELIVERY + END::VARCHAR AS fulfilment_method, + s.warehouse_code, + s.carrier_code, + s.service_level, + s.shipped_at, + s.delivered_at, + l.business_days_to_ship, + l.is_dispatch_on_time, + l.hours_to_deliver, + COALESCE(p.package_count, 0)::INT AS package_count, + p.chargeable_weight_kg::NUMBER(10, 3) AS chargeable_weight_kg, + p.rate_minor::INT AS rate_minor, + COALESCE(e.exception_count, 0)::INT AS exception_count, + COALESCE(e.compensation_usd, 0)::NUMBER(18, 2) AS compensation_usd, + s.tracking_event_count + FROM shipments AS s + LEFT JOIN sla AS l + ON s.shipment_id = l.shipment_id + LEFT JOIN package_totals AS p + ON s.shipment_id = p.shipment_id + LEFT JOIN exception_totals AS e + ON s.shipment_id = e.shipment_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/fulfilment/rpt_carrier_performance_weekly.sql b/benchmarks/duckshop/project/models/marts/fulfilment/rpt_carrier_performance_weekly.sql new file mode 100644 index 0000000000..0938cfbd69 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/fulfilment/rpt_carrier_performance_weekly.sql @@ -0,0 +1,54 @@ +MODEL ( + description """ + One row per carrier and dispatch week with on-time dispatch, delivery speed and exception rate + """, + columns ( + carrier_week_key (type VARCHAR, nullable false, audits [not_null, unique]), + carrier_code (type VARCHAR), + week_start (type DATE), + shipments (type INT), + on_time_dispatch_rate (type NUMBER(9, 4)), + p50_hours_to_deliver (type NUMBER(10, 2)), + p90_hours_to_deliver (type NUMBER(10, 2)), + exception_rate (type NUMBER(9, 4)), + compensation_usd (type NUMBER(18, 2)), + ), +); + +WITH shipments AS ( + SELECT * + FROM __ref("fct_shipments") +), + +final AS ( + SELECT + ( + s.carrier_code || ':' + || TO_VARCHAR( + DATE_TRUNC('week', CONVERT_TIMEZONE('UTC', s.shipped_at)), + 'YYYY-MM-DD' + ) + )::VARCHAR AS carrier_week_key, + s.carrier_code, + DATE_TRUNC('week', CONVERT_TIMEZONE('UTC', s.shipped_at))::DATE AS week_start, + COUNT(*)::INT AS shipments, + ROUND(AVG(IFF(s.is_dispatch_on_time, 1, 0)), 4)::NUMBER(9, 4) AS on_time_dispatch_rate, + ROUND( + PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY s.hours_to_deliver), + 2 + )::NUMBER(10, 2) AS p50_hours_to_deliver, + ROUND( + PERCENTILE_CONT(0.9) WITHIN GROUP (ORDER BY s.hours_to_deliver), + 2 + )::NUMBER(10, 2) AS p90_hours_to_deliver, + ROUND(AVG(IFF(s.exception_count > 0, 1, 0)), 4)::NUMBER(9, 4) AS exception_rate, + SUM(s.compensation_usd)::NUMBER(18, 2) AS compensation_usd + FROM shipments AS s + WHERE s.shipped_at IS NOT NULL + GROUP BY + s.carrier_code, + DATE_TRUNC('week', CONVERT_TIMEZONE('UTC', s.shipped_at)) +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/fulfilment/rpt_delivery_exceptions_monthly.sql b/benchmarks/duckshop/project/models/marts/fulfilment/rpt_delivery_exceptions_monthly.sql new file mode 100644 index 0000000000..d9708b05dc --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/fulfilment/rpt_delivery_exceptions_monthly.sql @@ -0,0 +1,51 @@ +MODEL ( + description """ + One row per month, carrier and responsibility with delivery exceptions and compensation paid + """, + columns ( + exception_month_key (type VARCHAR, nullable false, audits [not_null, unique]), + exception_month (type DATE), + carrier_code (type VARCHAR), + responsibility (type VARCHAR), + exceptions (type INT), + shipments_affected (type INT), + median_hours_to_resolve (type NUMBER(10, 2)), + compensation_usd (type NUMBER(18, 2)), + top_exception_code (type VARCHAR), + ), +); + +WITH exceptions AS ( + SELECT * + FROM __ref("int_delivery_exceptions_classified") +), + +final AS ( + SELECT + ( + TO_VARCHAR( + DATE_TRUNC('month', CONVERT_TIMEZONE('UTC', e.occurred_at)), + 'YYYY-MM' + ) + || ':' || e.carrier_code || ':' || e.responsibility + )::VARCHAR AS exception_month_key, + DATE_TRUNC('month', CONVERT_TIMEZONE('UTC', e.occurred_at))::DATE AS exception_month, + e.carrier_code, + e.responsibility, + COUNT(*)::INT AS exceptions, + COUNT(DISTINCT e.shipment_id)::INT AS shipments_affected, + ROUND(MEDIAN(e.hours_to_resolve), 2)::NUMBER(10, 2) AS median_hours_to_resolve, + SUM(e.compensation_usd)::NUMBER(18, 2) AS compensation_usd, + MODE(e.exception_code)::VARCHAR AS top_exception_code + FROM exceptions AS e + GROUP BY + DATE_TRUNC( + 'month', + CONVERT_TIMEZONE('UTC', e.occurred_at) + ), + e.carrier_code, + e.responsibility +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/fulfilment/rpt_returns_by_reason.sql b/benchmarks/duckshop/project/models/marts/fulfilment/rpt_returns_by_reason.sql new file mode 100644 index 0000000000..d1b7871a66 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/fulfilment/rpt_returns_by_reason.sql @@ -0,0 +1,65 @@ +MODEL ( + description """ + One row per month and return reason with return volume, resellable share and refunded USD + """, + columns ( + return_reason_key (type VARCHAR, nullable false, audits [not_null, unique]), + return_month (type DATE), + return_reason (type return_reason), + returns (type INT), + units (type INT), + resellable_share (type NUMBER(9, 4)), + refund_usd (type NUMBER(18, 2)), + median_days_to_refund (type NUMBER(10, 2)), + share_of_month (type NUMBER(9, 4)), + ), +); + +WITH reasons AS ( + SELECT * + FROM __ref("int_return_reasons") +), + +outcomes AS ( + SELECT * + FROM __ref("int_return_outcomes") +), + +monthly AS ( + SELECT + DATE_TRUNC('month', CONVERT_TIMEZONE('UTC', r.requested_at))::DATE AS return_month, + r.return_reason, + COUNT(*) AS returns, + SUM(r.qty) AS units, + AVG(IFF(o.is_resellable, 1, 0)) AS resellable_share, + SUM(o.refund_usd) AS refund_usd, + MEDIAN(o.days_to_refund) AS median_days_to_refund + FROM reasons AS r + LEFT JOIN outcomes AS o + ON r.return_id = o.return_id + GROUP BY + 1, + 2 +), + +final AS ( + SELECT + ( + TO_VARCHAR(m.return_month, 'YYYY-MM') || ':' || m.return_reason + )::VARCHAR AS return_reason_key, + m.return_month, + m.return_reason::VARCHAR AS return_reason, + m.returns::INT AS returns, + m.units::INT AS units, + ROUND(m.resellable_share, 4)::NUMBER(9, 4) AS resellable_share, + m.refund_usd::NUMBER(18, 2) AS refund_usd, + ROUND(m.median_days_to_refund, 2)::NUMBER(10, 2) AS median_days_to_refund, + ROUND( + RATIO_TO_REPORT(m.returns) OVER (PARTITION BY m.return_month), + 4 + )::NUMBER(9, 4) AS share_of_month + FROM monthly AS m +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/fulfilment/rpt_warehouse_productivity_daily.sql b/benchmarks/duckshop/project/models/marts/fulfilment/rpt_warehouse_productivity_daily.sql new file mode 100644 index 0000000000..ea387a4987 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/fulfilment/rpt_warehouse_productivity_daily.sql @@ -0,0 +1,63 @@ +MODEL ( + description """ + One row per warehouse and local work date with picking throughput and delivery slot utilisation + """, + columns ( + productivity_key (type VARCHAR, nullable false, audits [not_null, unique]), + warehouse_code (type VARCHAR), + work_date_local (type DATE), + pickers (type INT), + units_picked (type INT), + units_per_picker_hour (type NUMBER(10, 2)), + top_picker_id (type VARCHAR), + slot_utilisation (type NUMBER(9, 4)), + is_holiday (type BOOLEAN), + ), +); + +WITH picks AS ( + SELECT * + FROM __ref("int_pick_performance") +), + +slots AS ( + SELECT * + FROM __ref("int_delivery_slot_utilisation") +), + +daily_picks AS ( + SELECT + p.warehouse_code, + p.work_date_local, + COUNT(DISTINCT p.picker_id) AS pickers, + SUM(p.units_picked) AS units_picked, + SUM(p.active_hours) AS active_hours, + MAX_BY(p.picker_id, p.units_per_hour) AS top_picker_id, + BOOLOR_AGG(p.is_holiday) AS is_holiday + FROM picks AS p + GROUP BY + p.warehouse_code, + p.work_date_local +), + +final AS ( + SELECT + ( + d.warehouse_code || ':' || TO_VARCHAR(d.work_date_local, 'YYYY-MM-DD') + )::VARCHAR AS productivity_key, + d.warehouse_code, + d.work_date_local, + d.pickers::INT AS pickers, + d.units_picked::INT AS units_picked, + ROUND(d.units_picked / NULLIF(d.active_hours, 0), 2)::NUMBER(10, 2) AS units_per_picker_hour, + d.top_picker_id::VARCHAR AS top_picker_id, + s.utilisation AS slot_utilisation, + d.is_holiday::BOOLEAN AS is_holiday + FROM daily_picks AS d + LEFT JOIN slots AS s + ON d.warehouse_code = s.warehouse_code + AND d.work_date_local = s.slot_date_local +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/growth/fct_campaign_spend.sql b/benchmarks/duckshop/project/models/marts/growth/fct_campaign_spend.sql new file mode 100644 index 0000000000..0eaad9dc4d --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/growth/fct_campaign_spend.sql @@ -0,0 +1,48 @@ +MODEL ( + description """ + One row per ad platform spend report with USD spend, delivery metrics and campaign flight status + """, + contract enforced, + audit_factories [fact_table_quality], + columns ( + spend_id (type VARCHAR, nullable false, audits [not_null, unique]), + campaign_id (type VARCHAR), + channel (type VARCHAR, audits [not_null]), + spend_date (type DATE, audits [not_null]), + currency_code (type VARCHAR), + spend_usd (type NUMBER(18, 2), audits [non_negative]), + impressions (type INT), + clicks (type INT), + conversions (type INT), + cost_per_click_usd (type NUMBER(18, 4)), + is_in_flight (type BOOLEAN), + has_unknown_metrics (type BOOLEAN), + ), +); + +WITH spend AS ( + SELECT * + FROM __ref("int_campaign_spend_usd") +), + +final AS ( + SELECT + s.spend_id, + s.campaign_id, + s.channel, + s.spend_date, + s.currency_code, + s.spend_usd, + s.impressions, + s.clicks, + s.conversions, + s.cost_per_click_usd, + s.is_in_flight, + ( + ARRAY_SIZE(ARRAY_COMPACT(s.unknown_metrics)) > 0 + )::BOOLEAN AS has_unknown_metrics + FROM spend AS s +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/growth/rpt_affiliate_payouts_monthly.sql b/benchmarks/duckshop/project/models/marts/growth/rpt_affiliate_payouts_monthly.sql new file mode 100644 index 0000000000..2cd86e3cfb --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/growth/rpt_affiliate_payouts_monthly.sql @@ -0,0 +1,53 @@ +MODEL ( + description """ + One row per affiliate network and month with conversions, payable and reversed commission in USD + """, + columns ( + affiliate_month_key (type VARCHAR, nullable false, audits [not_null, unique]), + affiliate_network (type VARCHAR), + conversion_month (type DATE), + conversions (type INT), + last_click_conversions (type INT), + payable_commission_usd (type NUMBER(18, 2), audits [non_negative]), + reversed_commission_usd (type NUMBER(18, 2)), + effective_commission_rate (type NUMBER(9, 6)), + ), +); + +WITH commissions AS ( + SELECT * + FROM __ref("int_affiliate_commissions_usd") +), + +final AS ( + SELECT + ( + c.affiliate_network || ':' + || TO_VARCHAR( + DATE_TRUNC('month', CONVERT_TIMEZONE('UTC', c.converted_at)), + 'YYYY-MM' + ) + )::VARCHAR AS affiliate_month_key, + c.affiliate_network, + DATE_TRUNC('month', CONVERT_TIMEZONE('UTC', c.converted_at))::DATE AS conversion_month, + COUNT(*)::INT AS conversions, + COUNT_IF(c.is_last_click)::INT AS last_click_conversions, + SUM(c.payable_commission_usd)::NUMBER(18, 2) AS payable_commission_usd, + SUM( + IFF(c.is_last_click AND c.is_reversible, c.commission_usd, 0) + )::NUMBER(18, 2) AS reversed_commission_usd, + ROUND( + SUM(c.payable_commission_usd) / NULLIF(SUM(IFF(c.is_last_click, c.order_amount_usd, 0)), 0), + 6 + )::NUMBER(9, 6) AS effective_commission_rate + FROM commissions AS c + GROUP BY + c.affiliate_network, + DATE_TRUNC( + 'month', + CONVERT_TIMEZONE('UTC', c.converted_at) + ) +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/growth/rpt_campaign_roas_monthly.sql b/benchmarks/duckshop/project/models/marts/growth/rpt_campaign_roas_monthly.sql new file mode 100644 index 0000000000..9e49ba7eeb --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/growth/rpt_campaign_roas_monthly.sql @@ -0,0 +1,85 @@ +MODEL ( + description """ + One row per month and channel with USD spend, attributed revenue from paid sessions and return + on ad spend + """, + columns ( + roas_key (type VARCHAR, nullable false, audits [not_null, unique]), + spend_month (type DATE), + channel (type VARCHAR), + spend_usd (type NUMBER(18, 2)), + clicks (type INT), + attributed_orders (type INT), + attributed_revenue_usd (type NUMBER(18, 2)), + return_on_ad_spend (type NUMBER(12, 4)), + ), +); + +WITH spend AS ( + SELECT * + FROM __ref("fct_campaign_spend") +), + +orders AS ( + SELECT * + FROM __ref("fct_orders") +), + +campaigns AS ( + SELECT * + FROM __ref("dim_campaigns") +), + +campaign_keys AS ( + SELECT + LOWER(c.campaign_id) AS utm_campaign, + c.channel + FROM campaigns AS c +), + +monthly_spend AS ( + SELECT + DATE_TRUNC('month', s.spend_date) AS spend_month, + s.channel, + SUM(s.spend_usd) AS spend_usd, + SUM(s.clicks) AS clicks + FROM spend AS s + GROUP BY + 1, + 2 +), + +monthly_revenue AS ( + SELECT + DATE_TRUNC('month', o.placed_date_local) AS spend_month, + k.channel, + COUNT(*) AS attributed_orders, + SUM(o.amount_usd) AS attributed_revenue_usd + FROM orders AS o + INNER JOIN campaign_keys AS k + ON o.utm_campaign = k.utm_campaign + GROUP BY + 1, + 2 +), + +final AS ( + SELECT + ( + TO_VARCHAR(s.spend_month, 'YYYY-MM') || ':' || s.channel + )::VARCHAR AS roas_key, + s.spend_month::DATE AS spend_month, + s.channel, + s.spend_usd::NUMBER(18, 2) AS spend_usd, + s.clicks::INT AS clicks, + COALESCE(r.attributed_orders, 0)::INT AS attributed_orders, + COALESCE(r.attributed_revenue_usd, 0)::NUMBER(18, 2) AS attributed_revenue_usd, + ROUND(r.attributed_revenue_usd / NULLIF(s.spend_usd, 0), 4)::NUMBER(12, 4) AS return_on_ad_spend + FROM monthly_spend AS s + LEFT JOIN monthly_revenue AS r + ON s.spend_month = r.spend_month + AND s.channel = r.channel +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/growth/rpt_influencer_leaderboard.sql b/benchmarks/duckshop/project/models/marts/growth/rpt_influencer_leaderboard.sql new file mode 100644 index 0000000000..ac5f0f4984 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/growth/rpt_influencer_leaderboard.sql @@ -0,0 +1,57 @@ +MODEL ( + description """ + One row per creator with posts, reach, credited orders and revenue, ranked by credited revenue + per platform + """, + columns ( + creator_handle (type VARCHAR, nullable false, audits [not_null, unique]), + primary_platform (type VARCHAR), + posts (type INT), + impressions (type INT), + engagements (type INT), + credited_orders (type INT), + credited_revenue_usd (type NUMBER(18, 2)), + discount_given_usd (type NUMBER(18, 2)), + revenue_rank (type INT), + ), +); + +WITH posts AS ( + SELECT * + FROM __ref("int_influencer_promo_orders") +), + +creators AS ( + SELECT + p.creator_handle, + MODE(p.platform) AS primary_platform, + COUNT(*) AS posts, + SUM(p.impressions) AS impressions, + SUM(p.engagements) AS engagements, + SUM(p.credited_orders) AS credited_orders, + SUM(p.credited_revenue_usd) AS credited_revenue_usd, + SUM(p.discount_given_usd) AS discount_given_usd + FROM posts AS p + GROUP BY + p.creator_handle +), + +final AS ( + SELECT + c.creator_handle, + c.primary_platform::VARCHAR AS primary_platform, + c.posts::INT AS posts, + c.impressions::INT AS impressions, + c.engagements::INT AS engagements, + c.credited_orders::INT AS credited_orders, + c.credited_revenue_usd::NUMBER(18, 2) AS credited_revenue_usd, + c.discount_given_usd::NUMBER(18, 2) AS discount_given_usd, + RANK() OVER ( + PARTITION BY c.primary_platform + ORDER BY c.credited_revenue_usd DESC + )::INT AS revenue_rank + FROM creators AS c +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/growth/rpt_newsletter_growth_monthly.sql b/benchmarks/duckshop/project/models/marts/growth/rpt_newsletter_growth_monthly.sql new file mode 100644 index 0000000000..89930b04cc --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/growth/rpt_newsletter_growth_monthly.sql @@ -0,0 +1,75 @@ +MODEL ( + description """ + One row per month with new subscribers, unsubscribes, prospects and the running subscribed + audience + """, + columns ( + activity_month (type DATE, nullable false, audits [not_null, unique]), + new_subscribers (type INT), + new_prospects (type INT), + unsubscribes (type INT), + net_growth (type INT), + running_audience (type INT), + ), +); + +WITH subscribers AS ( + SELECT * + FROM __ref("int_newsletter_status") +), + +events AS ( + SELECT + DATE_TRUNC( + 'month', + CONVERT_TIMEZONE('UTC', s.first_subscribed_at) + )::DATE AS activity_month, + 1 AS subscribed, + IFF(s.is_prospect, 1, 0) AS prospect, + 0 AS unsubscribed + FROM subscribers AS s + WHERE s.first_subscribed_at IS NOT NULL + UNION ALL + SELECT + DATE_TRUNC( + 'month', + CONVERT_TIMEZONE('UTC', s.last_unsubscribed_at) + )::DATE AS activity_month, + 0 AS subscribed, + 0 AS prospect, + 1 AS unsubscribed + FROM subscribers AS s + WHERE + s.is_globally_unsubscribed + AND s.last_unsubscribed_at IS NOT NULL +), + +monthly AS ( + SELECT + e.activity_month, + SUM(e.subscribed) AS new_subscribers, + SUM(e.prospect) AS new_prospects, + SUM(e.unsubscribed) AS unsubscribes + FROM events AS e + GROUP BY + e.activity_month +), + +final AS ( + SELECT + m.activity_month, + m.new_subscribers::INT AS new_subscribers, + m.new_prospects::INT AS new_prospects, + m.unsubscribes::INT AS unsubscribes, + ( + m.new_subscribers - m.unsubscribes + )::INT AS net_growth, + SUM(m.new_subscribers - m.unsubscribes) OVER ( + ORDER BY m.activity_month + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + )::INT AS running_audience + FROM monthly AS m +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/growth/rpt_survey_nps_monthly.sql b/benchmarks/duckshop/project/models/marts/growth/rpt_survey_nps_monthly.sql new file mode 100644 index 0000000000..575e716cef --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/growth/rpt_survey_nps_monthly.sql @@ -0,0 +1,64 @@ +MODEL ( + description """ + One row per survey and month with responses, NPS group counts and the net promoter score + """, + columns ( + nps_key (type VARCHAR, nullable false, audits [not_null, unique]), + survey_code (type VARCHAR), + response_month (type DATE), + responses (type INT), + promoters (type INT), + passives (type INT), + detractors (type INT), + net_promoter_score (type NUMBER(6, 2)), + free_text_answers (type INT), + ), +); + +WITH answers AS ( + SELECT * + FROM __ref("int_survey_answers_flat") +), + +responses AS ( + SELECT + a.response_id, + a.survey_code, + DATE_TRUNC('month', CONVERT_TIMEZONE('UTC', a.submitted_at))::DATE AS response_month, + MAX(a.nps_group) AS nps_group, + COUNT_IF(a.answer_kind = 'text') AS free_text_answers + FROM answers AS a + GROUP BY + 1, + 2, + 3 +), + +final AS ( + SELECT + ( + r.survey_code || ':' || TO_VARCHAR(r.response_month, 'YYYY-MM') + )::VARCHAR AS nps_key, + r.survey_code, + r.response_month, + COUNT(DISTINCT r.response_id)::INT AS responses, + COUNT_IF(r.nps_group = 'promoter')::INT AS promoters, + COUNT_IF(r.nps_group = 'passive')::INT AS passives, + COUNT_IF(r.nps_group = 'detractor')::INT AS detractors, + ROUND( + 100 + * ( + COUNT_IF(r.nps_group = 'promoter') - COUNT_IF(r.nps_group = 'detractor') + ) + / NULLIF(COUNT(r.nps_group), 0), + 2 + )::NUMBER(6, 2) AS net_promoter_score, + SUM(r.free_text_answers)::INT AS free_text_answers + FROM responses AS r + GROUP BY + r.survey_code, + r.response_month +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/inventory/fct_inventory_daily.sql b/benchmarks/duckshop/project/models/marts/inventory/fct_inventory_daily.sql new file mode 100644 index 0000000000..fbdbd39e10 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/inventory/fct_inventory_daily.sql @@ -0,0 +1,81 @@ +MODEL ( + description """ + One row per SKU, warehouse and calendar day with stock on hand, its USD value at current + supplier cost, and stock-out and count-age flags. Rebuilt incrementally by day with a three-day + lookback for late stock movements. + """, + materialized incremental, + incremental_strategy delete_insert, + unique_key [stock_day_key], + cursor calendar_date, + cursor_type timestamp, + cursor_grain day, + lookback 3d, + cursor_inputs ( + int_stock_on_hand_daily calendar_date, + ), + replay_on_change bounded-120d, + on_schema_change append_new_columns, + contract enforced, + audit_factories [fact_table_quality], + columns ( + stock_day_key (type VARCHAR, nullable false, audits [ + not_null (run_scope delta_and_final), + unique, + ]), + sku (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR, audits [not_null]), + calendar_date (type DATE, audits [not_null]), + on_hand_qty (type INT), + unit_cost_usd (type NUMBER(18, 4)), + stock_value_usd (type NUMBER(18, 2)), + is_stocked_out (type BOOLEAN), + days_since_count (type INT), + is_count_overdue (type BOOLEAN), + ), +); + +WITH stock AS ( + SELECT * + FROM __ref("int_stock_on_hand_daily") +), + +costs AS ( + SELECT * + FROM __ref("int_supplier_cost_current") +), + +windowed AS ( + SELECT + s.stock_day_key, + s.sku, + s.warehouse_code, + s.calendar_date, + s.on_hand_qty, + s.is_stocked_out, + s.days_since_count + FROM stock AS s + WHERE + s.calendar_date >= __cursor_start() + AND s.calendar_date < __cursor_end() +), + +final AS ( + SELECT + w.stock_day_key, + w.sku, + w.warehouse_code, + w.calendar_date, + w.on_hand_qty, + c.unit_cost_usd, + ROUND(GREATEST(w.on_hand_qty, 0) * c.unit_cost_usd, 2)::NUMBER(18, 2) AS stock_value_usd, + w.is_stocked_out, + w.days_since_count, + COALESCE(w.days_since_count > 90, TRUE)::BOOLEAN AS is_count_overdue + FROM windowed AS w + LEFT JOIN costs AS c + ON w.sku = c.sku +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/inventory/rpt_inventory_valuation_monthly.sql b/benchmarks/duckshop/project/models/marts/inventory/rpt_inventory_valuation_monthly.sql new file mode 100644 index 0000000000..bff52a5ed4 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/inventory/rpt_inventory_valuation_monthly.sql @@ -0,0 +1,69 @@ +MODEL ( + description """ + One row per warehouse and month-end with stock value in USD, its monthly change and unvalued + units + """, + columns ( + valuation_month_key (type VARCHAR, nullable false, audits [not_null, unique]), + warehouse_code (type VARCHAR), + month_end (type DATE), + stock_value_usd (type NUMBER(18, 2)), + previous_month_value_usd (type NUMBER(18, 2)), + change_pct (type NUMBER(9, 2)), + units_on_hand (type INT), + unvalued_units (type INT), + stocked_out_skus (type INT), + ), +); + +WITH valuation AS ( + SELECT * + FROM __ref("int_inventory_valuation_daily") +), + +month_ends AS ( + SELECT + v.warehouse_code, + v.calendar_date AS month_end, + v.stock_value_usd, + v.units_on_hand, + v.unvalued_units, + v.stocked_out_skus + FROM valuation AS v + WHERE + v.calendar_date = LAST_DAY(v.calendar_date) + OR v.calendar_date = '@@valuation_date'::DATE +), + +final AS ( + SELECT + ( + m.warehouse_code || ':' || TO_VARCHAR(m.month_end, 'YYYY-MM-DD') + )::VARCHAR AS valuation_month_key, + m.warehouse_code, + m.month_end, + m.stock_value_usd, + LAG(m.stock_value_usd) OVER ( + PARTITION BY m.warehouse_code + ORDER BY m.month_end + )::NUMBER(18, 2) AS previous_month_value_usd, + ROUND( + 100 + * ( + m.stock_value_usd + / NULLIF( + LAG(m.stock_value_usd) OVER (PARTITION BY m.warehouse_code ORDER BY m.month_end), + 0 + ) + - 1 + ), + 2 + )::NUMBER(9, 2) AS change_pct, + m.units_on_hand, + m.unvalued_units, + m.stocked_out_skus + FROM month_ends AS m +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/inventory/rpt_stock_count_accuracy.sql b/benchmarks/duckshop/project/models/marts/inventory/rpt_stock_count_accuracy.sql new file mode 100644 index 0000000000..1cbc0fa540 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/inventory/rpt_stock_count_accuracy.sql @@ -0,0 +1,45 @@ +MODEL ( + description """ + One row per warehouse and month with count accuracy, out-of-tolerance counts and recount rate + """, + columns ( + count_month_key (type VARCHAR, nullable false, audits [not_null, unique]), + warehouse_code (type VARCHAR), + count_month (type DATE), + counts (type INT), + blind_counts (type INT), + accurate_counts (type INT), + accuracy_rate (type NUMBER(9, 4)), + out_of_tolerance (type INT), + net_variance_qty (type INT), + recount_rate (type NUMBER(9, 4)), + ), +); + +WITH counts AS ( + SELECT * + FROM __ref("int_stock_count_variances") +), + +final AS ( + SELECT + ( + c.warehouse_code || ':' || TO_VARCHAR(DATE_TRUNC('month', c.counted_at_local), 'YYYY-MM') + )::VARCHAR AS count_month_key, + c.warehouse_code, + DATE_TRUNC('month', c.counted_at_local)::DATE AS count_month, + COUNT(*)::INT AS counts, + COUNT_IF(c.is_blind_count)::INT AS blind_counts, + COUNT_IF(c.variance_qty = 0)::INT AS accurate_counts, + ROUND(AVG(IFF(c.variance_qty = 0, 1, 0)), 4)::NUMBER(9, 4) AS accuracy_rate, + COUNT_IF(c.is_out_of_tolerance)::INT AS out_of_tolerance, + SUM(c.variance_qty)::INT AS net_variance_qty, + ROUND(AVG(IFF(c.recount_sequence > 1, 1, 0)), 4)::NUMBER(9, 4) AS recount_rate + FROM counts AS c + GROUP BY + c.warehouse_code, + DATE_TRUNC('month', c.counted_at_local) +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/inventory/rpt_stock_outs_weekly.sql b/benchmarks/duckshop/project/models/marts/inventory/rpt_stock_outs_weekly.sql new file mode 100644 index 0000000000..dafacee620 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/inventory/rpt_stock_outs_weekly.sql @@ -0,0 +1,97 @@ +MODEL ( + description """ + One row per ISO week and warehouse with stock-out days, SKUs affected and the longest stock-out + run + """, + columns ( + stock_out_key (type VARCHAR, nullable false, audits [not_null, unique]), + week_start (type DATE), + warehouse_code (type VARCHAR), + sku_days (type INT), + stocked_out_sku_days (type INT), + stock_out_rate (type NUMBER(9, 4)), + skus_affected (type INT), + longest_run_days (type INT), + ), +); + +WITH inventory AS ( + SELECT * + FROM __ref("fct_inventory_daily") +), + +runs AS ( + SELECT + i.sku, + i.warehouse_code, + i.calendar_date, + i.is_stocked_out, + DATEADD( + 'day', + -COUNT_IF(i.is_stocked_out) OVER ( + PARTITION BY i.sku, i.warehouse_code + ORDER BY i.calendar_date + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ), + i.calendar_date + ) AS run_anchor + FROM inventory AS i +), + +run_lengths AS ( + SELECT + r.warehouse_code, + DATE_TRUNC('week', MIN(r.calendar_date)) AS week_start, + COUNT(*) AS run_days + FROM runs AS r + WHERE r.is_stocked_out + GROUP BY + r.sku, + r.warehouse_code, + r.run_anchor +), + +weekly AS ( + SELECT + DATE_TRUNC('week', r.calendar_date) AS week_start, + r.warehouse_code, + COUNT(*) AS sku_days, + COUNT_IF(r.is_stocked_out) AS stocked_out_sku_days, + COUNT(DISTINCT IFF(r.is_stocked_out, r.sku, NULL)) AS skus_affected + FROM runs AS r + GROUP BY + 1, + 2 +), + +longest AS ( + SELECT + l.week_start, + l.warehouse_code, + MAX(l.run_days) AS longest_run_days + FROM run_lengths AS l + GROUP BY + l.week_start, + l.warehouse_code +), + +final AS ( + SELECT + ( + w.warehouse_code || ':' || TO_VARCHAR(w.week_start, 'YYYY-MM-DD') + )::VARCHAR AS stock_out_key, + w.week_start::DATE AS week_start, + w.warehouse_code, + w.sku_days::INT AS sku_days, + w.stocked_out_sku_days::INT AS stocked_out_sku_days, + ROUND(w.stocked_out_sku_days / NULLIF(w.sku_days, 0), 4)::NUMBER(9, 4) AS stock_out_rate, + w.skus_affected::INT AS skus_affected, + COALESCE(l.longest_run_days, 0)::INT AS longest_run_days + FROM weekly AS w + LEFT JOIN longest AS l + ON w.week_start = l.week_start + AND w.warehouse_code = l.warehouse_code +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/inventory/rpt_supplier_scorecard.sql b/benchmarks/duckshop/project/models/marts/inventory/rpt_supplier_scorecard.sql new file mode 100644 index 0000000000..c0cf49bf61 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/inventory/rpt_supplier_scorecard.sql @@ -0,0 +1,59 @@ +MODEL ( + description """ + One row per supplier and quarter with purchase order fill, lateness and received damage + """, + columns ( + scorecard_key (type VARCHAR, nullable false, audits [not_null, unique]), + supplier_code (type VARCHAR), + quarter_start (type DATE), + po_lines (type INT), + ordered_qty (type INT), + received_qty (type INT), + fill_rate (type NUMBER(9, 4)), + late_line_share (type NUMBER(9, 4)), + damage_rate (type NUMBER(9, 4)), + discrepancy_codes (type ARRAY), + ), +); + +WITH po_lines AS ( + SELECT * + FROM __ref("int_purchase_order_lines_received") +), + +quarterly AS ( + SELECT + p.supplier_code, + DATE_TRUNC('quarter', p.first_received_at_local)::DATE AS quarter_start, + COUNT(*) AS po_lines, + SUM(p.ordered_qty) AS ordered_qty, + SUM(p.received_qty) AS received_qty, + AVG(IFF(p.days_late > 0, 1, 0)) AS late_line_share, + SUM(p.damaged_qty) AS damaged_qty, + ARRAY_DISTINCT(ARRAY_FLATTEN(ARRAY_AGG(p.discrepancy_codes))) AS discrepancy_codes + FROM po_lines AS p + WHERE p.first_received_at_local IS NOT NULL + GROUP BY + 1, + 2 +), + +final AS ( + SELECT + ( + q.supplier_code || ':' || TO_VARCHAR(q.quarter_start, 'YYYY-MM-DD') + )::VARCHAR AS scorecard_key, + q.supplier_code, + q.quarter_start, + q.po_lines::INT AS po_lines, + q.ordered_qty::INT AS ordered_qty, + q.received_qty::INT AS received_qty, + ROUND(q.received_qty / NULLIF(q.ordered_qty, 0), 4)::NUMBER(9, 4) AS fill_rate, + ROUND(q.late_line_share, 4)::NUMBER(9, 4) AS late_line_share, + ROUND(q.damaged_qty / NULLIF(q.received_qty, 0), 4)::NUMBER(9, 4) AS damage_rate, + q.discrepancy_codes::ARRAY AS discrepancy_codes + FROM quarterly AS q +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_audits/generic/valid_currency.sql b/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_audits/generic/valid_currency.sql new file mode 100644 index 0000000000..e64d7ce8a3 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_audits/generic/valid_currency.sql @@ -0,0 +1,19 @@ +AUDIT (); + +WITH currencies AS ( + SELECT * + FROM __seed("currencies") +), + +final AS ( + SELECT r.* + FROM @relation AS r + LEFT JOIN currencies AS c + ON r.@column = c.currency_code + WHERE + r.@column IS NOT NULL + AND c.currency_code IS NULL +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_hooks/sql/comment_on_relation.sql b/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_hooks/sql/comment_on_relation.sql new file mode 100644 index 0000000000..be49f5021f --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_hooks/sql/comment_on_relation.sql @@ -0,0 +1,5 @@ +HOOK ( + description "Set the table comment on the model's destination relation", +); + +COMMENT ON TABLE @relation IS @'comment' diff --git a/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_schemas/money.sql b/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_schemas/money.sql new file mode 100644 index 0000000000..0a7fad9f75 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_schemas/money.sql @@ -0,0 +1,9 @@ +SCHEMA ( + name money, + description "An amount in its own currency's major units with its USD value", + columns ( + currency_code (type VARCHAR, audits [not_null, valid_currency]), + amount (type NUMBER(38, 6)), + amount_usd (type NUMBER(18, 2), audits [non_negative]), + ), +); diff --git a/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_schemas/money_with_fx.sql b/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_schemas/money_with_fx.sql new file mode 100644 index 0000000000..6cfb9cac57 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/_schemas/money_with_fx.sql @@ -0,0 +1,9 @@ +SCHEMA ( + name money_with_fx, + extends money, + description "A money amount with the reference rate used to convert it and that rate's date", + columns ( + rate_to_usd (type NUMBER(18, 8), audits [not_null]), + rate_date (type DATE), + ), +); diff --git a/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/audits/singular/orders_balance_to_payments.sql b/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/audits/singular/orders_balance_to_payments.sql new file mode 100644 index 0000000000..c798970103 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/_sqlbuild/audits/singular/orders_balance_to_payments.sql @@ -0,0 +1,40 @@ +AUDIT ( + name "orders_balance_to_payments", + severity warn, +); + +WITH orders AS ( + SELECT * + FROM __ref("fct_orders") +), + +payments AS ( + SELECT * + FROM __ref("fct_payments") +), + +successful_payments AS ( + SELECT + p.order_id, + SUM(p.amount) AS paid_amount + FROM payments AS p + WHERE p.is_successful + GROUP BY + p.order_id +), + +final AS ( + SELECT + o.order_id, + o.amount, + s.paid_amount + FROM orders AS o + LEFT JOIN successful_payments AS s + ON o.order_id = s.order_id + WHERE + o.order_status IN ('PAID', 'PACKED', 'SHIPPED', 'DELIVERED') + AND ABS(o.amount - COALESCE(s.paid_amount, 0)) > 0.01 +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/fct_order_lines.sql b/benchmarks/duckshop/project/models/marts/orders/fct_order_lines.sql new file mode 100644 index 0000000000..a4b7369734 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/fct_order_lines.sql @@ -0,0 +1,69 @@ +MODEL ( + description """ + One row per order line with list, discount and net amounts in minor units, the promotions + applied, and the product's family and category. + """, + contract enforced, + audit_factories [fact_table_quality], + columns ( + order_line_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null, relationships (to __ref("fct_orders"), field order_id)]), + sku (type VARCHAR, audits [not_null]), + sku_family (type VARCHAR), + category_level_1 (type VARCHAR), + quantity (type INT, audits [non_negative]), + unit_amount_minor (type INT), + list_amount_minor (type INT), + markdown_pct (type NUMBER(9, 4)), + gross_amount_minor (type INT), + discount_amount_minor (type INT), + net_amount_minor (type INT), + tax_amount_minor (type INT), + promo_codes (type ARRAY), + has_stacked_promotions (type BOOLEAN), + line_weight_kg (type NUMBER(12, 3)), + ), +); + +WITH priced AS ( + SELECT * + FROM __ref("int_order_lines_priced") +), + +discounted AS ( + SELECT * + FROM __ref("int_order_lines_discounted") +), + +catalogue AS ( + SELECT * + FROM __ref("int_product_catalogue") +), + +final AS ( + SELECT + p.order_line_id, + p.order_id, + p.sku, + p.sku_family, + c.category_level_1, + p.quantity, + p.unit_amount_minor, + p.list_amount_minor, + p.markdown_pct, + p.gross_amount_minor, + p.discount_amount_minor, + p.net_amount_minor, + p.tax_amount_minor, + COALESCE(d.promo_codes, ARRAY_CONSTRUCT())::ARRAY AS promo_codes, + COALESCE(d.has_stacked_promotions, FALSE)::BOOLEAN AS has_stacked_promotions, + p.line_weight_kg + FROM priced AS p + LEFT JOIN discounted AS d + ON p.order_line_id = d.order_line_id + LEFT JOIN catalogue AS c + ON p.sku = c.sku +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/fct_orders.sql b/benchmarks/duckshop/project/models/marts/orders/fct_orders.sql new file mode 100644 index 0000000000..775dd330f2 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/fct_orders.sql @@ -0,0 +1,106 @@ +MODEL ( + description """ + One row per storefront order with its value in order currency and USD, status milestones, + collected payments and the customer's order sequence. + """, + contract enforced, + model_schema money_with_fx, + audit_factories [fact_table_quality], + post_hooks [ + sql("comment_on_relation", relation: "@@CTX:destination.qualified", comment: "DuckShop order facts"), + ], + columns ( + order_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR, audits [not_null]), + region_code (type VARCHAR), + placed_at (type TIMESTAMP_TZ, audits [not_null]), + placed_date_local (type DATE), + order_status (type order_status), + sales_channel (type sales_channel), + units (type INT), + gross_amount_usd (type NUMBER(18, 2)), + discount_amount_usd (type NUMBER(18, 2)), + tax_amount_usd (type NUMBER(18, 2)), + promo_code (type VARCHAR), + utm_source (type VARCHAR), + utm_medium (type VARCHAR), + utm_campaign (type VARCHAR), + basket_size (type VARCHAR), + order_number (type INT), + is_first_order (type BOOLEAN), + first_paid_at (type TIMESTAMP_TZ), + first_delivered_at (type TIMESTAMP_TZ), + hours_paid_to_shipped (type NUMBER(12, 2)), + captured_minor (type INT), + refunded_minor (type INT), + net_collected_minor (type INT), + is_fully_refunded (type BOOLEAN), + ), +); + +WITH orders AS ( + SELECT * + FROM __ref("int_orders_usd") +), + +timelines AS ( + SELECT * + FROM __ref("int_order_status_timeline") +), + +payments AS ( + SELECT * + FROM __ref("int_order_payments_rollup") +), + +sequence AS ( + SELECT * + FROM __ref("int_customer_order_sequence") +), + +final AS ( + SELECT + o.order_id, + o.customer_id, + o.region_code, + o.placed_at, + o.placed_date_local, + o.order_status, + o.sales_channel, + o.units, + o.currency_code, + o.amount, + o.amount_usd, + o.rate_to_usd, + o.rate_date, + o.gross_amount_usd, + o.discount_amount_usd, + o.tax_amount_usd, + o.promo_code, + o.utm_source, + o.utm_medium, + o.utm_campaign, + o.basket_size, + s.order_number, + COALESCE(s.is_first_order, FALSE)::BOOLEAN AS is_first_order, + t.first_paid_at, + t.first_delivered_at, + t.hours_paid_to_shipped, + COALESCE(p.captured_minor, 0)::INT AS captured_minor, + COALESCE(p.refunded_minor, 0)::INT AS refunded_minor, + COALESCE(p.net_collected_minor, 0)::INT AS net_collected_minor, + ( + COALESCE(p.captured_minor, 0) > 0 + AND COALESCE(p.refunded_minor, 0) >= p.captured_minor + )::BOOLEAN AS is_fully_refunded + FROM orders AS o + LEFT JOIN timelines AS t + ON o.order_id = t.order_id + LEFT JOIN payments AS p + ON o.order_id = p.order_id + LEFT JOIN sequence AS s + ON o.order_id = s.order_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/fct_payments.sql b/benchmarks/duckshop/project/models/marts/orders/fct_payments.sql new file mode 100644 index 0000000000..fac24fbe9d --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/fct_payments.sql @@ -0,0 +1,99 @@ +MODEL ( + description "One row per payment attempt with its amount in major units and USD and its outcome", + contract enforced, + model_schema money, + audit_factories [fact_table_quality], + columns ( + payment_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + attempt_number (type INT), + is_final_attempt (type BOOLEAN), + payment_method (type VARCHAR), + authorised_at (type TIMESTAMP_TZ), + is_successful (type BOOLEAN), + decline_family (type VARCHAR), + risk_band (type VARCHAR), + card_scheme (type VARCHAR), + had_chargeback (type BOOLEAN), + ), +); + +WITH attempts AS ( + SELECT * + FROM __ref("int_payment_attempts") +), + +fx_rates AS ( + SELECT * + FROM __ref("stg_fx_rates") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +with_exponent AS ( + SELECT + a.payment_id, + a.order_id, + a.attempt_number, + a.is_final_attempt, + a.payment_method, + a.authorised_at, + CONVERT_TIMEZONE('UTC', a.authorised_at)::DATE AS authorised_date, + a.is_successful, + a.decline_family, + a.risk_band, + a.card_scheme, + a.had_chargeback, + a.currency_code, + a.amount_minor, + c.minor_unit_exponent + FROM attempts AS a + INNER JOIN currencies AS c + ON a.currency_code = c.currency_code +), + +with_rate AS ( + SELECT + w.payment_id, + w.order_id, + w.attempt_number, + w.is_final_attempt, + w.payment_method, + w.authorised_at, + w.is_successful, + w.decline_family, + w.risk_band, + w.card_scheme, + w.had_chargeback, + w.currency_code, + w.amount_minor, + w.minor_unit_exponent, + fx.rate_to_usd + FROM with_exponent AS w + ASOF JOIN fx_rates AS fx MATCH_CONDITION (w.authorised_date >= fx.rate_date) + ON w.currency_code = fx.currency_code +), + +final AS ( + SELECT + r.payment_id, + r.order_id, + r.attempt_number, + r.is_final_attempt, + r.payment_method, + r.authorised_at, + r.is_successful, + r.decline_family, + r.risk_band, + r.card_scheme, + r.had_chargeback, + r.currency_code, + @sql.money_columns(r) + FROM with_rate AS r +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/fct_refunds.sql b/benchmarks/duckshop/project/models/marts/orders/fct_refunds.sql new file mode 100644 index 0000000000..39ebef9d86 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/fct_refunds.sql @@ -0,0 +1,40 @@ +MODEL ( + description "One row per refund with its amount in major units and USD, its reason and timing", + contract enforced, + model_schema money, + audit_factories [fact_table_quality], + columns ( + refund_id (type VARCHAR, nullable false, audits [not_null, unique]), + payment_id (type VARCHAR, audits [not_null]), + order_id (type VARCHAR), + refunded_at (type TIMESTAMP_TZ), + refunded_date (type DATE), + reason_category (type VARCHAR), + days_after_payment (type INT), + is_full_refund (type BOOLEAN), + ), +); + +WITH refunds AS ( + SELECT * + FROM __ref("int_refunds_usd") +), + +final AS ( + SELECT + r.refund_id, + r.payment_id, + r.order_id, + r.currency_code, + r.amount, + r.amount_usd, + r.refunded_at, + r.refunded_date, + r.reason_category, + r.days_after_payment, + r.is_full_refund + FROM refunds AS r +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/rpt_basket_mix_weekly.sql b/benchmarks/duckshop/project/models/marts/orders/rpt_basket_mix_weekly.sql new file mode 100644 index 0000000000..c28cfe00f2 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/rpt_basket_mix_weekly.sql @@ -0,0 +1,64 @@ +MODEL ( + description """ + One row per ISO week and top-level category with units, net sales and share of the week's units + """, + columns ( + basket_week_key (type VARCHAR, nullable false, audits [not_null, unique]), + week_start (type DATE, audits [not_null]), + category_level_1 (type VARCHAR), + order_count (type INT), + units (type INT), + net_amount_minor (type INT), + discounted_line_share (type NUMBER(9, 4)), + unit_share_of_week (type NUMBER(9, 4)), + category_rank (type INT), + ), +); + +WITH lines AS ( + SELECT * + FROM __ref("fct_order_lines") +), + +orders AS ( + SELECT * + FROM __ref("fct_orders") +), + +weekly AS ( + SELECT + DATE_TRUNC('week', o.placed_date_local) AS week_start, + COALESCE(l.category_level_1, 'uncategorised') AS category_level_1, + COUNT(DISTINCT l.order_id) AS order_count, + SUM(l.quantity) AS units, + SUM(l.net_amount_minor) AS net_amount_minor, + AVG(IFF(l.discount_amount_minor > 0, 1, 0)) AS discounted_line_share + FROM lines AS l + INNER JOIN orders AS o + ON l.order_id = o.order_id + GROUP BY + DATE_TRUNC('week', o.placed_date_local), + COALESCE(l.category_level_1, 'uncategorised') +), + +final AS ( + SELECT + ( + TO_VARCHAR(w.week_start, 'YYYY-MM-DD') || ':' || w.category_level_1 + )::VARCHAR AS basket_week_key, + w.week_start::DATE AS week_start, + w.category_level_1::VARCHAR AS category_level_1, + w.order_count::INT AS order_count, + w.units::INT AS units, + w.net_amount_minor::INT AS net_amount_minor, + ROUND(w.discounted_line_share, 4)::NUMBER(9, 4) AS discounted_line_share, + ROUND( + RATIO_TO_REPORT(w.units) OVER (PARTITION BY w.week_start), + 4 + )::NUMBER(9, 4) AS unit_share_of_week, + RANK() OVER (PARTITION BY w.week_start ORDER BY w.units DESC)::INT AS category_rank + FROM weekly AS w +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/rpt_cart_abandonment_daily.sql b/benchmarks/duckshop/project/models/marts/orders/rpt_cart_abandonment_daily.sql new file mode 100644 index 0000000000..c35c586e43 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/rpt_cart_abandonment_daily.sql @@ -0,0 +1,55 @@ +MODEL ( + description """ + One row per storefront and day of last cart activity with abandonment and recovery rates + """, + columns ( + cart_day_key (type VARCHAR, nullable false, audits [not_null, unique]), + activity_date (type DATE), + storefront_code (type VARCHAR), + carts (type INT), + abandoned_carts (type INT), + converted_carts (type INT), + abandonment_rate (type NUMBER(9, 4)), + abandoned_value_minor (type INT), + median_hours_to_convert (type NUMBER(10, 2)), + ), +); + +WITH carts AS ( + SELECT * + FROM __ref("int_cart_abandonment") +), + +daily AS ( + SELECT + CONVERT_TIMEZONE('UTC', c.last_activity_at)::DATE AS activity_date, + c.storefront_code, + COUNT(*) AS carts, + COUNT_IF(c.is_abandoned) AS abandoned_carts, + COUNT_IF(c.is_converted) AS converted_carts, + SUM(IFF(c.is_abandoned, c.cart_value_minor, 0)) AS abandoned_value_minor, + MEDIAN(c.hours_to_convert) AS median_hours_to_convert + FROM carts AS c + GROUP BY + 1, + 2 +), + +final AS ( + SELECT + ( + d.storefront_code || ':' || TO_VARCHAR(d.activity_date, 'YYYY-MM-DD') + )::VARCHAR AS cart_day_key, + d.activity_date::DATE AS activity_date, + d.storefront_code, + d.carts::INT AS carts, + d.abandoned_carts::INT AS abandoned_carts, + d.converted_carts::INT AS converted_carts, + ROUND(d.abandoned_carts / NULLIF(d.carts, 0), 4)::NUMBER(9, 4) AS abandonment_rate, + d.abandoned_value_minor::INT AS abandoned_value_minor, + ROUND(d.median_hours_to_convert, 2)::NUMBER(10, 2) AS median_hours_to_convert + FROM daily AS d +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/rpt_daily_sales_by_region.sql b/benchmarks/duckshop/project/models/marts/orders/rpt_daily_sales_by_region.sql new file mode 100644 index 0000000000..1679aa8766 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/rpt_daily_sales_by_region.sql @@ -0,0 +1,72 @@ +MODEL ( + description """ + One row per local order date and region with orders, units, revenue in USD and average order + value, plus week-over-week and trailing 28-day comparisons. + """, + columns ( + sales_day_key (type VARCHAR, nullable false, audits [not_null, unique]), + placed_date_local (type DATE, audits [not_null]), + region_code (type VARCHAR), + order_count (type INT), + customer_count (type INT), + first_order_count (type INT), + units (type INT), + revenue_usd (type NUMBER(18, 2)), + discount_usd (type NUMBER(18, 2)), + average_order_usd (type NUMBER(18, 2)), + revenue_same_day_last_week_usd (type NUMBER(18, 2)), + revenue_trailing_28d_usd (type NUMBER(18, 2)), + ), +); + +WITH orders AS ( + SELECT * + FROM __ref("fct_orders") +), + +daily AS ( + SELECT + o.placed_date_local, + COALESCE(o.region_code, 'UNKNOWN') AS region_code, + COUNT(*) AS order_count, + COUNT(DISTINCT o.customer_id) AS customer_count, + COUNT_IF(o.is_first_order) AS first_order_count, + SUM(o.units) AS units, + SUM(o.amount_usd) AS revenue_usd, + SUM(o.discount_amount_usd) AS discount_usd + FROM orders AS o + WHERE o.order_status NOT IN ('CANCELLED', 'PENDING') + GROUP BY + o.placed_date_local, + COALESCE(o.region_code, 'UNKNOWN') +), + +final AS ( + SELECT + ( + d.region_code || ':' || TO_VARCHAR(d.placed_date_local, 'YYYY-MM-DD') + )::VARCHAR AS sales_day_key, + d.placed_date_local, + d.region_code::VARCHAR AS region_code, + d.order_count::INT AS order_count, + d.customer_count::INT AS customer_count, + d.first_order_count::INT AS first_order_count, + d.units::INT AS units, + d.revenue_usd::NUMBER(18, 2) AS revenue_usd, + d.discount_usd::NUMBER(18, 2) AS discount_usd, + ROUND(d.revenue_usd / NULLIF(d.order_count, 0), 2)::NUMBER(18, 2) AS average_order_usd, + SUM(d.revenue_usd) OVER ( + PARTITION BY d.region_code + ORDER BY d.placed_date_local + RANGE BETWEEN INTERVAL '7 days' PRECEDING AND INTERVAL '7 days' PRECEDING + )::NUMBER(18, 2) AS revenue_same_day_last_week_usd, + SUM(d.revenue_usd) OVER ( + PARTITION BY d.region_code + ORDER BY d.placed_date_local + RANGE BETWEEN INTERVAL '27 days' PRECEDING AND CURRENT ROW + )::NUMBER(18, 2) AS revenue_trailing_28d_usd + FROM daily AS d +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/rpt_payment_decline_rates.sql b/benchmarks/duckshop/project/models/marts/orders/rpt_payment_decline_rates.sql new file mode 100644 index 0000000000..55868fcadc --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/rpt_payment_decline_rates.sql @@ -0,0 +1,63 @@ +MODEL ( + description "One row per month, payment method and decline family with attempt counts and rates", + columns ( + decline_key (type VARCHAR, nullable false, audits [not_null, unique]), + attempt_month (type DATE), + payment_method (type VARCHAR), + decline_family (type VARCHAR), + attempts (type INT), + share_of_method_attempts (type NUMBER(9, 4)), + attempted_usd (type NUMBER(18, 2)), + recovered_on_retry (type INT), + retry_recovery_rate (type NUMBER(9, 4)), + ), +); + +WITH payments AS ( + SELECT * + FROM __ref("fct_payments") +), + +monthly AS ( + SELECT + DATE_TRUNC( + 'month', + CONVERT_TIMEZONE('UTC', p.authorised_at) + )::DATE AS attempt_month, + p.payment_method, + p.decline_family, + COUNT(*) AS attempts, + SUM(p.amount_usd) AS attempted_usd, + COUNT_IF(NOT p.is_successful AND NOT p.is_final_attempt) AS recovered_on_retry + FROM payments AS p + GROUP BY + 1, + 2, + 3 +), + +final AS ( + SELECT + ( + TO_VARCHAR(m.attempt_month, 'YYYY-MM') + || ':' + || COALESCE(m.payment_method, '?') + || ':' + || m.decline_family + )::VARCHAR AS decline_key, + m.attempt_month, + m.payment_method, + m.decline_family, + m.attempts::INT AS attempts, + ROUND( + RATIO_TO_REPORT(m.attempts) OVER (PARTITION BY m.attempt_month, m.payment_method), + 4 + )::NUMBER(9, 4) AS share_of_method_attempts, + m.attempted_usd::NUMBER(18, 2) AS attempted_usd, + m.recovered_on_retry::INT AS recovered_on_retry, + @py.round_half_even(@sql.safe_divide(m.recovered_on_retry, m.attempts), 4)::NUMBER(9, 4) AS retry_recovery_rate + FROM monthly AS m +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/rpt_review_scores_by_product.sql b/benchmarks/duckshop/project/models/marts/orders/rpt_review_scores_by_product.sql new file mode 100644 index 0000000000..d317f156f1 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/rpt_review_scores_by_product.sql @@ -0,0 +1,69 @@ +MODEL ( + description """ + One row per product with weighted review score, rating distribution and verified-purchase share + """, + columns ( + product_id (type VARCHAR, nullable false, audits [not_null, unique]), + review_count (type INT), + weighted_rating (type NUMBER(4, 2)), + rating_distribution (type OBJECT), + verified_share (type NUMBER(9, 4)), + net_sentiment (type INT), + latest_review_at (type TIMESTAMP_TZ), + ), +); + +WITH reviews AS ( + SELECT * + FROM __ref("int_reviews_moderated") +), + +rating_counts AS ( + SELECT + r.product_id, + r.rating, + COUNT(*) AS rating_count + FROM reviews AS r + GROUP BY + r.product_id, + r.rating +), + +distribution AS ( + SELECT + c.product_id, + OBJECT_AGG(c.rating::VARCHAR, c.rating_count::VARIANT) AS rating_distribution + FROM rating_counts AS c + GROUP BY + c.product_id +), + +summary AS ( + SELECT + r.product_id, + COUNT(*) AS review_count, + SUM(r.rating * r.review_weight) / NULLIF(SUM(r.review_weight), 0) AS weighted_rating, + AVG(IFF(r.is_verified_purchase, 1, 0)) AS verified_share, + SUM(r.sentiment_score) AS net_sentiment, + MAX(r.submitted_at) AS latest_review_at + FROM reviews AS r + GROUP BY + r.product_id +), + +final AS ( + SELECT + s.product_id, + s.review_count::INT AS review_count, + ROUND(s.weighted_rating, 2)::NUMBER(4, 2) AS weighted_rating, + d.rating_distribution::OBJECT AS rating_distribution, + ROUND(s.verified_share, 4)::NUMBER(9, 4) AS verified_share, + s.net_sentiment::INT AS net_sentiment, + s.latest_review_at + FROM summary AS s + LEFT JOIN distribution AS d + ON s.product_id = d.product_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/orders/rpt_subscription_mrr_monthly.sql b/benchmarks/duckshop/project/models/marts/orders/rpt_subscription_mrr_monthly.sql new file mode 100644 index 0000000000..8f831374ec --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/orders/rpt_subscription_mrr_monthly.sql @@ -0,0 +1,78 @@ +MODEL ( + description """ + One row per month with opening, new, churned and closing monthly recurring revenue in USD and + the active subscriber count on the last day of the month. + """, + columns ( + month_start (type DATE, nullable false, audits [not_null, unique]), + closing_mrr_usd (type NUMBER(18, 2)), + opening_mrr_usd (type NUMBER(18, 2)), + new_mrr_usd (type NUMBER(18, 2)), + churned_mrr_usd (type NUMBER(18, 2)), + net_change_usd (type NUMBER(18, 2)), + active_subscribers (type INT), + new_subscribers (type INT), + ), +); + +WITH mrr_days AS ( + SELECT * + FROM __ref("int_subscription_mrr_daily") +), + +month_ends AS ( + SELECT + DATE_TRUNC('month', d.calendar_date) AS month_start, + d.subscription_id, + d.mrr_usd + FROM mrr_days AS d + WHERE + d.calendar_date = LAST_DAY(d.calendar_date) + OR d.calendar_date = '@@valuation_date'::DATE +), + +firsts AS ( + SELECT + DATE_TRUNC('month', d.calendar_date) AS month_start, + SUM(d.mrr_usd) AS new_mrr_usd, + COUNT(*) AS new_subscribers + FROM mrr_days AS d + WHERE d.is_first_day + GROUP BY + DATE_TRUNC('month', d.calendar_date) +), + +closing AS ( + SELECT + m.month_start, + SUM(m.mrr_usd) AS closing_mrr_usd, + COUNT(DISTINCT m.subscription_id) AS active_subscribers + FROM month_ends AS m + GROUP BY + m.month_start +), + +final AS ( + SELECT + c.month_start::DATE AS month_start, + c.closing_mrr_usd::NUMBER(18, 2) AS closing_mrr_usd, + LAG(c.closing_mrr_usd, 1, 0) OVER (ORDER BY c.month_start)::NUMBER(18, 2) AS opening_mrr_usd, + COALESCE(f.new_mrr_usd, 0)::NUMBER(18, 2) AS new_mrr_usd, + GREATEST( + LAG(c.closing_mrr_usd, 1, 0) OVER (ORDER BY c.month_start) + + COALESCE(f.new_mrr_usd, 0) + - c.closing_mrr_usd, + 0 + )::NUMBER(18, 2) AS churned_mrr_usd, + ( + c.closing_mrr_usd - LAG(c.closing_mrr_usd, 1, 0) OVER (ORDER BY c.month_start) + )::NUMBER(18, 2) AS net_change_usd, + c.active_subscribers::INT AS active_subscribers, + COALESCE(f.new_subscribers, 0)::INT AS new_subscribers + FROM closing AS c + LEFT JOIN firsts AS f + ON c.month_start = f.month_start +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/partners/fct_partner_orders.sql b/benchmarks/duckshop/project/models/marts/partners/fct_partner_orders.sql new file mode 100644 index 0000000000..b373137325 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/partners/fct_partner_orders.sql @@ -0,0 +1,49 @@ +MODEL ( + description """ + One row per partner order in one shape across payload versions, with USD totals and the matched + customer + """, + contract enforced, + audit_factories [fact_table_quality], + columns ( + partner_order_key (type VARCHAR, nullable false, audits [not_null, unique]), + partner_code (type VARCHAR, audits [not_null]), + payload_version (type INT, audits [accepted_values (values [1, 2])]), + external_order_id (type VARCHAR), + currency_code (type VARCHAR), + placed_at_utc (type TIMESTAMP_NTZ), + placed_date (type DATE), + order_total_usd (type NUMBER(18, 2), audits [non_negative]), + line_count (type INT), + units (type INT), + lines_match_total (type BOOLEAN), + customer_id (type VARCHAR), + is_known_customer (type BOOLEAN), + ), +); + +WITH partner_orders AS ( + SELECT * + FROM __ref("int_partner_orders_unified") +), + +final AS ( + SELECT + o.partner_order_key, + o.partner_code, + o.payload_version, + o.external_order_id, + o.currency_code, + o.placed_at_utc, + o.placed_date, + o.order_total_usd, + o.line_count, + o.units, + o.lines_match_total, + o.customer_id, + o.is_known_customer + FROM partner_orders AS o +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/partners/rpt_partner_fees_variance.sql b/benchmarks/duckshop/project/models/marts/partners/rpt_partner_fees_variance.sql new file mode 100644 index 0000000000..fc97ce3aef --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/partners/rpt_partner_fees_variance.sql @@ -0,0 +1,48 @@ +MODEL ( + description """ + One row per partner and settlement quarter with stated versus expected commission and statements + out of tolerance + """, + columns ( + partner_quarter_key (type VARCHAR, nullable false, audits [not_null, unique]), + partner_code (type VARCHAR), + settlement_quarter (type DATE), + statements (type INT), + gross_sales_usd (type NUMBER(18, 2)), + stated_commission_usd (type NUMBER(18, 2)), + expected_commission_usd (type NUMBER(18, 2)), + variance_usd (type NUMBER(18, 2)), + statements_out_of_tolerance (type INT), + unbalanced_statements (type INT), + ), +); + +WITH fees AS ( + SELECT * + FROM __ref("int_partner_fees_usd") +), + +final AS ( + SELECT + ( + f.partner_code || ':' || TO_VARCHAR(DATE_TRUNC('quarter', f.period_end), 'YYYY-MM-DD') + )::VARCHAR AS partner_quarter_key, + f.partner_code, + DATE_TRUNC('quarter', f.period_end)::DATE AS settlement_quarter, + COUNT(*)::INT AS statements, + SUM(f.gross_sales_usd)::NUMBER(18, 2) AS gross_sales_usd, + SUM(f.stated_commission_usd)::NUMBER(18, 2) AS stated_commission_usd, + SUM(f.expected_commission_usd)::NUMBER(18, 2) AS expected_commission_usd, + ( + SUM(f.stated_commission_usd) - SUM(f.expected_commission_usd) + )::NUMBER(18, 2) AS variance_usd, + COUNT_IF(NOT f.is_within_tolerance)::INT AS statements_out_of_tolerance, + COUNT_IF(NOT f.statement_balances)::INT AS unbalanced_statements + FROM fees AS f + GROUP BY + f.partner_code, + DATE_TRUNC('quarter', f.period_end) +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/partners/rpt_partner_listing_health.sql b/benchmarks/duckshop/project/models/marts/partners/rpt_partner_listing_health.sql new file mode 100644 index 0000000000..3f4d00ae94 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/partners/rpt_partner_listing_health.sql @@ -0,0 +1,56 @@ +MODEL ( + description """ + One row per partner and listing health state with listings, units sold in 30 days and average + commission + """, + columns ( + listing_health_key (type VARCHAR, nullable false, audits [not_null, unique]), + partner_code (type VARCHAR), + listing_health (type VARCHAR), + listings (type INT), + partner_fulfilled (type INT), + units_sold_30d (type INT), + average_commission_pct (type NUMBER(5, 2)), + share_of_partner_listings (type NUMBER(9, 4)), + ), +); + +WITH listings AS ( + SELECT * + FROM __ref("int_partner_listing_status") +), + +grouped AS ( + SELECT + l.partner_code, + l.listing_health, + COUNT(*) AS listings, + COUNT_IF(l.fulfilled_by_partner) AS partner_fulfilled, + SUM(l.units_sold_30d) AS units_sold_30d, + AVG(l.commission_pct) AS average_commission_pct + FROM listings AS l + GROUP BY + l.partner_code, + l.listing_health +), + +final AS ( + SELECT + ( + g.partner_code || ':' || g.listing_health + )::VARCHAR AS listing_health_key, + g.partner_code, + g.listing_health, + g.listings::INT AS listings, + g.partner_fulfilled::INT AS partner_fulfilled, + g.units_sold_30d::INT AS units_sold_30d, + ROUND(g.average_commission_pct, 2)::NUMBER(5, 2) AS average_commission_pct, + ROUND( + RATIO_TO_REPORT(g.listings) OVER (PARTITION BY g.partner_code), + 4 + )::NUMBER(9, 4) AS share_of_partner_listings + FROM grouped AS g +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/support/fct_support_tickets.sql b/benchmarks/duckshop/project/models/marts/support/fct_support_tickets.sql new file mode 100644 index 0000000000..f6f425bd62 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/support/fct_support_tickets.sql @@ -0,0 +1,139 @@ +MODEL ( + description """ + One row per support ticket with priority, first response and resolution against SLA, thread + activity, chat sentiment and the customer's contact history. Merged incrementally as ticket + change-log rows arrive. + """, + materialized incremental, + incremental_strategy merge, + unique_key [ticket_id], + cursor source_loaded_at, + cursor_type timestamp, + cursor_grain second, + cursor_inputs ( + stg_tickets source_loaded_at, + ), + replay_on_change bounded-120d, + on_schema_change sync_all_columns, + contract enforced, + audit_factories [fact_table_quality], + columns ( + ticket_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR), + channel (type VARCHAR), + priority (type ticket_priority), + ticket_status (type VARCHAR), + contact_reason (type VARCHAR), + created_at_utc (type TIMESTAMP_NTZ), + solved_at (type TIMESTAMP_TZ), + minutes_to_first_response (type INT), + business_minutes_to_response (type INT), + is_first_response_breached (type BOOLEAN), + is_resolution_breached (type BOOLEAN), + comment_count (type INT), + handoff_count (type INT), + sentiment_label (type VARCHAR), + ticket_number_for_customer (type INT), + is_repeat_contact_7d (type BOOLEAN), + csat_score (type INT), + source_loaded_at (type TIMESTAMP_NTZ, nullable false, audits [not_null]), + ), +); + +WITH tickets AS ( + SELECT * + FROM __ref("stg_tickets") +), + +first_responses AS ( + SELECT * + FROM __ref("int_ticket_first_response") +), + +business_hours AS ( + SELECT * + FROM __ref("int_ticket_business_hours") +), + +breaches AS ( + SELECT * + FROM __ref("int_ticket_sla_breaches") +), + +threads AS ( + SELECT * + FROM __ref("int_ticket_threads") +), + +chats AS ( + SELECT * + FROM __ref("int_chat_sentiment") +), + +history AS ( + SELECT * + FROM __ref("int_customer_support_history") +), + +changed_tickets AS ( + SELECT + t.ticket_id, + t.channel, + t.ticket_status, + t.contact_reason, + t.solved_at, + t.source_loaded_at + FROM tickets AS t + WHERE + t.source_loaded_at >= __cursor_start() + AND t.source_loaded_at < __cursor_end() +), + +ticket_sentiment AS ( + SELECT + c.ticket_id, + MIN_BY(c.sentiment_label, c.sentiment_score) AS sentiment_label + FROM chats AS c + WHERE c.ticket_id IS NOT NULL + GROUP BY + c.ticket_id +), + +final AS ( + SELECT + t.ticket_id, + h.customer_id, + t.channel, + b.priority, + t.ticket_status, + t.contact_reason, + f.created_at_utc, + t.solved_at, + f.minutes_to_first_response, + bh.business_minutes_to_response, + COALESCE(b.is_first_response_breached, FALSE)::BOOLEAN AS is_first_response_breached, + COALESCE(b.is_resolution_breached, FALSE)::BOOLEAN AS is_resolution_breached, + COALESCE(th.comment_count, 0)::INT AS comment_count, + COALESCE(th.handoff_count, 0)::INT AS handoff_count, + s.sentiment_label::VARCHAR AS sentiment_label, + h.ticket_number_for_customer, + COALESCE(h.is_repeat_contact_7d, FALSE)::BOOLEAN AS is_repeat_contact_7d, + h.csat_score, + t.source_loaded_at + FROM changed_tickets AS t + LEFT JOIN first_responses AS f + ON t.ticket_id = f.ticket_id + LEFT JOIN business_hours AS bh + ON t.ticket_id = bh.ticket_id + LEFT JOIN breaches AS b + ON t.ticket_id = b.ticket_id + LEFT JOIN threads AS th + ON t.ticket_id = th.ticket_id + LEFT JOIN ticket_sentiment AS s + ON t.ticket_id = s.ticket_id + LEFT JOIN history AS h + ON t.ticket_id = h.ticket_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/support/rpt_agent_scorecard.sql b/benchmarks/duckshop/project/models/marts/support/rpt_agent_scorecard.sql new file mode 100644 index 0000000000..2575fad660 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/support/rpt_agent_scorecard.sql @@ -0,0 +1,92 @@ +MODEL ( + description """ + One row per agent and month with shifts, utilisation, calls, first responses and customer + satisfaction + """, + columns ( + agent_month_key (type VARCHAR, nullable false, audits [not_null, unique]), + agent_id (type VARCHAR), + team_code (type VARCHAR), + shift_month (type DATE), + shifts (type INT), + productive_hours (type NUMBER(10, 2)), + average_utilisation (type NUMBER(9, 4)), + calls_handled (type INT), + tickets_first_responded (type INT), + average_csat (type NUMBER(4, 2)), + utilisation_rank_in_team (type INT), + ), +); + +WITH utilisation AS ( + SELECT * + FROM __ref("int_agent_utilisation") +), + +first_responses AS ( + SELECT * + FROM __ref("int_ticket_first_response") +), + +history AS ( + SELECT * + FROM __ref("int_customer_support_history") +), + +agent_csat AS ( + SELECT + f.first_responder_id AS agent_id, + DATE_TRUNC('month', f.first_response_at_utc) AS response_month, + AVG(h.csat_score) AS average_csat + FROM first_responses AS f + INNER JOIN history AS h + ON f.ticket_id = h.ticket_id + WHERE h.csat_score IS NOT NULL + GROUP BY + 1, + 2 +), + +monthly AS ( + SELECT + u.agent_id, + u.team_code, + DATE_TRUNC('month', u.shift_start_utc) AS shift_month, + COUNT(*) AS shifts, + SUM(u.productive_minutes) / 60 AS productive_hours, + AVG(u.utilisation) AS average_utilisation, + SUM(u.calls_handled) AS calls_handled, + SUM(u.tickets_first_responded) AS tickets_first_responded + FROM utilisation AS u + GROUP BY + 1, + 2, + 3 +), + +final AS ( + SELECT + ( + m.agent_id || ':' || TO_VARCHAR(m.shift_month, 'YYYY-MM') + )::VARCHAR AS agent_month_key, + m.agent_id, + m.team_code, + m.shift_month::DATE AS shift_month, + m.shifts::INT AS shifts, + ROUND(m.productive_hours, 2)::NUMBER(10, 2) AS productive_hours, + ROUND(m.average_utilisation, 4)::NUMBER(9, 4) AS average_utilisation, + m.calls_handled::INT AS calls_handled, + m.tickets_first_responded::INT AS tickets_first_responded, + ROUND(c.average_csat, 2)::NUMBER(4, 2) AS average_csat, + DENSE_RANK() OVER ( + PARTITION BY m.team_code, m.shift_month + ORDER BY m.average_utilisation DESC NULLS LAST + )::INT AS utilisation_rank_in_team + FROM monthly AS m + LEFT JOIN agent_csat AS c + ON m.agent_id = c.agent_id + AND m.shift_month = c.response_month +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/support/rpt_contact_reasons_monthly.sql b/benchmarks/duckshop/project/models/marts/support/rpt_contact_reasons_monthly.sql new file mode 100644 index 0000000000..aa7950f97f --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/support/rpt_contact_reasons_monthly.sql @@ -0,0 +1,84 @@ +MODEL ( + description """ + One row per month and contact reason with ticket volume, repeat contacts and IVR abandonment + """, + columns ( + reason_month_key (type VARCHAR, nullable false, audits [not_null, unique]), + contact_month (type DATE), + contact_reason (type VARCHAR), + tickets (type INT), + repeat_contacts (type INT), + repeat_rate (type NUMBER(9, 4)), + calls (type INT), + abandoned_in_ivr (type INT), + month_over_month_change (type INT), + ), +); + +WITH tickets AS ( + SELECT * + FROM __ref("fct_support_tickets") +), + +calls AS ( + SELECT * + FROM __ref("int_call_ivr_paths") +), + +ticket_reasons AS ( + SELECT + t.ticket_id, + DATE_TRUNC('month', t.created_at_utc)::DATE AS contact_month, + COALESCE(t.contact_reason, 'unspecified') AS contact_reason, + t.is_repeat_contact_7d + FROM tickets AS t + WHERE t.created_at_utc IS NOT NULL +), + +call_counts AS ( + SELECT + c.ticket_id, + COUNT(*) AS calls, + COUNT_IF(c.abandoned_in_ivr) AS abandoned_in_ivr + FROM calls AS c + WHERE c.ticket_id IS NOT NULL + GROUP BY + c.ticket_id +), + +monthly AS ( + SELECT + r.contact_month, + r.contact_reason, + COUNT(*) AS tickets, + COUNT_IF(r.is_repeat_contact_7d) AS repeat_contacts, + SUM(COALESCE(c.calls, 0)) AS calls, + SUM(COALESCE(c.abandoned_in_ivr, 0)) AS abandoned_in_ivr + FROM ticket_reasons AS r + LEFT JOIN call_counts AS c + ON r.ticket_id = c.ticket_id + GROUP BY + r.contact_month, + r.contact_reason +), + +final AS ( + SELECT + ( + TO_VARCHAR(m.contact_month, 'YYYY-MM') || ':' || m.contact_reason + )::VARCHAR AS reason_month_key, + m.contact_month, + m.contact_reason::VARCHAR AS contact_reason, + m.tickets::INT AS tickets, + m.repeat_contacts::INT AS repeat_contacts, + ROUND(m.repeat_contacts / NULLIF(m.tickets, 0), 4)::NUMBER(9, 4) AS repeat_rate, + m.calls::INT AS calls, + m.abandoned_in_ivr::INT AS abandoned_in_ivr, + ( + m.tickets - LAG(m.tickets) OVER (PARTITION BY m.contact_reason ORDER BY m.contact_month) + )::INT AS month_over_month_change + FROM monthly AS m +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/marts/support/rpt_support_sla_weekly.sql b/benchmarks/duckshop/project/models/marts/support/rpt_support_sla_weekly.sql new file mode 100644 index 0000000000..7246aa1df2 --- /dev/null +++ b/benchmarks/duckshop/project/models/marts/support/rpt_support_sla_weekly.sql @@ -0,0 +1,55 @@ +MODEL ( + description """ + One row per week, channel and priority with ticket volume, SLA breach rates and response times + """, + columns ( + sla_week_key (type VARCHAR, nullable false, audits [not_null, unique]), + week_start (type DATE), + channel (type VARCHAR), + priority (type ticket_priority), + tickets (type INT), + first_response_breach_rate (type NUMBER(9, 4)), + resolution_breach_rate (type NUMBER(9, 4)), + p50_minutes_to_first_response (type NUMBER(10, 2)), + p90_business_minutes_to_response (type NUMBER(10, 2)), + ), +); + +WITH tickets AS ( + SELECT * + FROM __ref("fct_support_tickets") +), + +final AS ( + SELECT + ( + TO_VARCHAR(DATE_TRUNC('week', t.created_at_utc), 'YYYY-MM-DD') + || ':' + || t.channel + || ':' + || COALESCE(t.priority::VARCHAR, '?') + )::VARCHAR AS sla_week_key, + DATE_TRUNC('week', t.created_at_utc)::DATE AS week_start, + t.channel, + t.priority::INT AS priority, + COUNT(*)::INT AS tickets, + ROUND( + AVG(IFF(t.is_first_response_breached, 1, 0)), + 4 + )::NUMBER(9, 4) AS first_response_breach_rate, + ROUND(AVG(IFF(t.is_resolution_breached, 1, 0)), 4)::NUMBER(9, 4) AS resolution_breach_rate, + ROUND(MEDIAN(t.minutes_to_first_response), 2)::NUMBER(10, 2) AS p50_minutes_to_first_response, + ROUND( + PERCENTILE_CONT(0.9) WITHIN GROUP (ORDER BY t.business_minutes_to_response), + 2 + )::NUMBER(10, 2) AS p90_business_minutes_to_response + FROM tickets AS t + WHERE t.created_at_utc IS NOT NULL + GROUP BY + DATE_TRUNC('week', t.created_at_utc), + t.channel, + t.priority +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/snapshots/snap_customers.sql b/benchmarks/duckshop/project/models/snapshots/snap_customers.sql new file mode 100644 index 0000000000..a436859b08 --- /dev/null +++ b/benchmarks/duckshop/project/models/snapshots/snap_customers.sql @@ -0,0 +1,40 @@ +MODEL ( + description """ + Version history of customer accounts, one row per customer per change in the account change log. + Accounts deleted at source are closed. + """, + materialized snapshot, + unique_key [customer_id], + snapshot_strategy timestamp, + updated_at source_loaded_at, + invalidate_hard_deletes true, + columns ( + customer_id (type VARCHAR, nullable false, audits [not_null (run_scope delta_and_final)]), + email_normalised (type VARCHAR), + region_code (type VARCHAR), + storefront_code (type VARCHAR), + preferred_language (type VARCHAR), + marketing_opt_in (type BOOLEAN), + source_loaded_at (type TIMESTAMP_NTZ, nullable false), + ), +); + +WITH customers AS ( + SELECT * + FROM __ref("stg_customers") +), + +final AS ( + SELECT + c.customer_id, + c.email_normalised, + c.region_code, + c.storefront_code, + c.preferred_language, + c.marketing_opt_in, + c.source_loaded_at + FROM customers AS c +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/snapshots/snap_loyalty_tiers.sql b/benchmarks/duckshop/project/models/snapshots/snap_loyalty_tiers.sql new file mode 100644 index 0000000000..6e148a609d --- /dev/null +++ b/benchmarks/duckshop/project/models/snapshots/snap_loyalty_tiers.sql @@ -0,0 +1,30 @@ +MODEL ( + description """ + Version history of each loyalty account's tier, recording a new version when the tier changes + """, + materialized snapshot, + unique_key [loyalty_account_id], + snapshot_strategy check, + check_columns [tier_code], + columns ( + loyalty_account_id (type VARCHAR, nullable false, audits [not_null]), + customer_id (type VARCHAR), + tier_code (type VARCHAR), + ), +); + +WITH accounts AS ( + SELECT * + FROM __ref("stg_loyalty_accounts") +), + +final AS ( + SELECT + a.loyalty_account_id, + a.customer_id, + a.tier_code + FROM accounts AS a +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/snapshots/snap_partner_listings.sql b/benchmarks/duckshop/project/models/snapshots/snap_partner_listings.sql new file mode 100644 index 0000000000..b9a086a0b9 --- /dev/null +++ b/benchmarks/duckshop/project/models/snapshots/snap_partner_listings.sql @@ -0,0 +1,40 @@ +MODEL ( + description """ + Version history of partner listings by change-log load time; listings removed at source are + closed + """, + materialized snapshot, + unique_key [listing_id], + snapshot_strategy timestamp, + updated_at source_loaded_at, + invalidate_hard_deletes true, + columns ( + listing_id (type VARCHAR, nullable false, audits [not_null]), + partner_code (type VARCHAR), + sku (type VARCHAR), + listing_status (type VARCHAR), + commission_pct (type NUMBER(5, 2)), + fulfilled_by_partner (type BOOLEAN), + source_loaded_at (type TIMESTAMP_NTZ, nullable false), + ), +); + +WITH listings AS ( + SELECT * + FROM __ref("stg_partner_listings") +), + +final AS ( + SELECT + l.listing_id, + l.partner_code, + l.sku, + l.listing_status, + l.commission_pct, + l.fulfilled_by_partner, + l.source_loaded_at + FROM listings AS l +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/snapshots/snap_price_list_history.sql b/benchmarks/duckshop/project/models/snapshots/snap_price_list_history.sql new file mode 100644 index 0000000000..2f23176db2 --- /dev/null +++ b/benchmarks/duckshop/project/models/snapshots/snap_price_list_history.sql @@ -0,0 +1,42 @@ +MODEL ( + description """ + Version history of list prices rebuilt from every row of the price list change log, so price + history is recoverable even for versions that staging has already deduplicated away. + """, + materialized snapshot, + unique_key [price_list_id], + snapshot_strategy timestamp, + updated_at changed_at, + observed_at loaded_at, + historical_input changes, + columns ( + price_list_id (type VARCHAR, nullable false, audits [not_null]), + sku (type VARCHAR), + currency_code (type VARCHAR), + list_amount_minor (type INT), + change_op (type VARCHAR), + changed_at (type TIMESTAMP_NTZ, nullable false), + loaded_at (type TIMESTAMP_NTZ, nullable false), + ), +); + +WITH price_list_log AS ( + SELECT * + FROM __source("catalogue__price_lists") +), + +final AS ( + SELECT + p.price_list_id, + UPPER(TRIM(p.sku))::VARCHAR AS sku, + UPPER(p.currency_code)::VARCHAR AS currency_code, + p.list_amount_minor::INT AS list_amount_minor, + p._op::VARCHAR AS change_op, + TO_TIMESTAMP_NTZ(p._loaded_at) AS changed_at, + TO_TIMESTAMP_NTZ(p._loaded_at) AS loaded_at + FROM price_list_log AS p + QUALIFY ROW_NUMBER() OVER (PARTITION BY p.price_list_id, p._loaded_at ORDER BY p._seq DESC) = 1 +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/snapshots/snap_products.sql b/benchmarks/duckshop/project/models/snapshots/snap_products.sql new file mode 100644 index 0000000000..e41b3a65d1 --- /dev/null +++ b/benchmarks/duckshop/project/models/snapshots/snap_products.sql @@ -0,0 +1,37 @@ +MODEL ( + description """ + Version history of catalogue products, recording a new version when merchandising attributes + change + """, + materialized snapshot, + unique_key [product_id], + snapshot_strategy check, + check_columns [title, brand_code, category_path, is_active], + columns ( + product_id (type VARCHAR, nullable false, audits [not_null]), + title (type VARCHAR), + brand_code (type VARCHAR), + category_path (type VARCHAR), + is_active (type BOOLEAN), + spec_key_count (type INT), + ), +); + +WITH products AS ( + SELECT * + FROM __ref("stg_products") +), + +final AS ( + SELECT + p.product_id, + p.title, + p.brand_code, + p.category_path, + p.is_active, + p.spec_key_count + FROM products AS p +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/snapshots/snap_warehouses.sql b/benchmarks/duckshop/project/models/snapshots/snap_warehouses.sql new file mode 100644 index 0000000000..4c1e0220a6 --- /dev/null +++ b/benchmarks/duckshop/project/models/snapshots/snap_warehouses.sql @@ -0,0 +1,39 @@ +MODEL ( + description """ + Version history of warehouse operating attributes, recording a new version when hours or + capabilities change + """, + materialized snapshot, + unique_key [warehouse_code], + snapshot_strategy check, + check_columns [opens_local, closes_local, supports_same_day, supports_cold_chain, max_parcel_kg], + columns ( + warehouse_code (type VARCHAR, nullable false, audits [not_null]), + iana_timezone (type VARCHAR), + opens_local (type TIME), + closes_local (type TIME), + supports_same_day (type BOOLEAN), + supports_cold_chain (type BOOLEAN), + max_parcel_kg (type NUMBER(6, 2)), + ), +); + +WITH warehouses AS ( + SELECT * + FROM __ref("stg_warehouses") +), + +final AS ( + SELECT + w.warehouse_code, + w.iana_timezone, + w.opens_local, + w.closes_local, + w.supports_same_day, + w.supports_cold_chain, + w.max_parcel_kg + FROM warehouses AS w +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/_sqlbuild/constants/patterns.sql b/benchmarks/duckshop/project/models/staging/_sqlbuild/constants/patterns.sql new file mode 100644 index 0000000000..f20c14bb9c --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/_sqlbuild/constants/patterns.sql @@ -0,0 +1 @@ +CONSTANT (name email_pattern, value "^[a-z0-9._%+-]+@[a-z0-9.-]+\\.[a-z]{2,}$"); diff --git a/benchmarks/duckshop/project/models/staging/_sqlbuild/enums/cdc_op.sql b/benchmarks/duckshop/project/models/staging/_sqlbuild/enums/cdc_op.sql new file mode 100644 index 0000000000..b585d2342f --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/_sqlbuild/enums/cdc_op.sql @@ -0,0 +1,8 @@ +ENUM ( + name cdc_op, + members ( + INSERT "I", + UPDATE "U", + DELETE "D", + ), +); diff --git a/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/cdc_latest.sql b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/cdc_latest.sql new file mode 100644 index 0000000000..119ee618f2 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/cdc_latest.sql @@ -0,0 +1,19 @@ +MACRO ( + description "Latest surviving version of each key in an append-only change log", + arguments ( + src RELATION (_op VARCHAR, _seq NUMBER(38, 0), _loaded_at TIMESTAMP_NTZ), + key COLUMN OF src, + ), +); + +WITH cdc_versions AS ( + SELECT * + FROM @src + QUALIFY ROW_NUMBER() OVER (PARTITION BY @key ORDER BY _seq DESC) = 1 +), + +cdc_latest AS ( + SELECT * + FROM cdc_versions + WHERE _op <> @enum("cdc_op").DELETE +) diff --git a/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/epoch_ms_to_utc.sql b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/epoch_ms_to_utc.sql new file mode 100644 index 0000000000..cfa5af7be0 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/epoch_ms_to_utc.sql @@ -0,0 +1,7 @@ +MACRO ( + description "UTC timestamp from epoch milliseconds", + arguments (epoch_ms NUMBER(38, 0)), + returns TIMESTAMP_NTZ, +); + +TO_TIMESTAMP_NTZ(@epoch_ms, 3) diff --git a/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/json_text.sql b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/json_text.sql new file mode 100644 index 0000000000..13154d40a2 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/json_text.sql @@ -0,0 +1,10 @@ +MACRO ( + description "Text of a semi-structured value, treating JSON null and blank strings as SQL NULL", + arguments (value VARIANT), + returns VARCHAR, +); + +NULLIF( + TRIM(STRIP_NULL_VALUE(@value)::VARCHAR), + '' +) diff --git a/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/load_metadata.sql b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/load_metadata.sql new file mode 100644 index 0000000000..b499c4aab4 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/load_metadata.sql @@ -0,0 +1,8 @@ +MACRO ( + description "Standard lineage columns carried from a change-log row", + arguments (src RELATION (_seq NUMBER(38, 0), _loaded_at TIMESTAMP_NTZ)), +); + +SELECT + @src._seq AS source_seq, + TO_TIMESTAMP_NTZ(@src._loaded_at) AS source_loaded_at diff --git a/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/normalise_email.sql b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/normalise_email.sql new file mode 100644 index 0000000000..54d0ef177c --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/normalise_email.sql @@ -0,0 +1,11 @@ +MACRO ( + description "Lower-cased, trimmed email address, or NULL when it is not a plausible address", + arguments (email VARCHAR), + returns VARCHAR, +); + +IFF( + REGEXP_LIKE(LOWER(TRIM(@email)), @const("email_pattern")), + REGEXP_REPLACE(LOWER(TRIM(@email)), '\\+[^@]*@', '@'), + NULL +) diff --git a/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/normalise_phone.sql b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/normalise_phone.sql new file mode 100644 index 0000000000..93403670ab --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/normalise_phone.sql @@ -0,0 +1,11 @@ +MACRO ( + description "Digits-only phone number with a leading plus, or NULL when too short", + arguments (phone VARCHAR), + returns VARCHAR, +); + +IFF(LENGTH(REGEXP_REPLACE(@phone, '[^0-9]', '')) >= 7, '+' +|| LTRIM( + REGEXP_REPLACE(@phone, '[^0-9]', ''), + '0' +), NULL) diff --git a/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/normalise_sku.sql b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/normalise_sku.sql new file mode 100644 index 0000000000..cf5779a164 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/_sqlbuild/macros/sql/normalise_sku.sql @@ -0,0 +1,11 @@ +MACRO ( + description "Canonical SKU: upper case with whitespace and underscores removed", + arguments (sku VARCHAR), + returns VARCHAR, +); + +UPPER(TRANSLATE( + REGEXP_REPLACE(TRIM(@sku), '\\s+', ''), + '_', + '-' +)) diff --git a/benchmarks/duckshop/project/models/staging/catalogue/stg_brands.sql b/benchmarks/duckshop/project/models/staging/catalogue/stg_brands.sql new file mode 100644 index 0000000000..3dd780f4de --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/catalogue/stg_brands.sql @@ -0,0 +1,37 @@ +MODEL ( + description "One row per live brand", + columns ( + brand_code (type VARCHAR, nullable false, audits [not_null, unique]), + brand_name (type VARCHAR), + country_code (type VARCHAR), + is_private_label (type BOOLEAN), + sustainability_rating (type VARCHAR), + founded_year (type INT), + brand_metadata (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("catalogue__brands") +), + +@sql.cdc_latest(src, brand_code) + +final AS ( + SELECT + UPPER(b.brand_code)::VARCHAR AS brand_code, + TRIM(b.brand_name)::VARCHAR AS brand_name, + UPPER(b.country_code)::VARCHAR AS country_code, + COALESCE(b.is_private_label, FALSE) AS is_private_label, + UPPER(@sql.json_text(b.metadata:sustainability.rating))::VARCHAR AS sustainability_rating, + TRY_TO_NUMBER(REGEXP_SUBSTR( + b.metadata:founded::VARCHAR, + '(19|20)[0-9]{2}' + ))::INT AS founded_year, + b.metadata::OBJECT AS brand_metadata, + @sql.load_metadata(b) + FROM cdc_latest AS b +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/catalogue/stg_bundle_components.sql b/benchmarks/duckshop/project/models/staging/catalogue/stg_bundle_components.sql new file mode 100644 index 0000000000..e3b9635229 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/catalogue/stg_bundle_components.sql @@ -0,0 +1,32 @@ +MODEL ( + description "One row per live component line of a bundle", + columns ( + component_id (type VARCHAR, nullable false, audits [not_null, unique]), + bundle_sku (type VARCHAR, audits [not_null]), + component_sku (type VARCHAR, audits [not_null]), + quantity (type INT), + is_optional (type BOOLEAN), + substitute_skus (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("catalogue__bundle_components") +), + +@sql.cdc_latest(src, component_id) + +final AS ( + SELECT + c.component_id, + @sql.normalise_sku(c.bundle_sku) AS bundle_sku, + @sql.normalise_sku(c.component_sku) AS component_sku, + GREATEST(COALESCE(c.quantity, 1), 1)::INT AS quantity, + COALESCE(c.options:optional::BOOLEAN, FALSE) AS is_optional, + COALESCE(c.options:substitutes, ARRAY_CONSTRUCT())::ARRAY AS substitute_skus, + @sql.load_metadata(c) + FROM cdc_latest AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/catalogue/stg_bundles.sql b/benchmarks/duckshop/project/models/staging/catalogue/stg_bundles.sql new file mode 100644 index 0000000000..cda8859373 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/catalogue/stg_bundles.sql @@ -0,0 +1,40 @@ +MODEL ( + description "One row per live product bundle and its pricing rule", + columns ( + bundle_sku (type VARCHAR, nullable false, audits [not_null, unique]), + title (type VARCHAR), + active_from (type DATE), + active_to (type DATE), + pricing_mode (type VARCHAR, audits [ + accepted_values (values ["fixed", "percent_off", "cheapest_free"]), + ]), + fixed_amount_minor (type INT), + percent_off (type NUMBER(5, 2)), + pricing_currency_code (type VARCHAR), + pricing (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("catalogue__bundles") +), + +@sql.cdc_latest(src, bundle_sku) + +final AS ( + SELECT + @sql.normalise_sku(b.bundle_sku) AS bundle_sku, + b.title, + b.active_from, + b.active_to, + LOWER(COALESCE(b.pricing:mode::VARCHAR, 'fixed'))::VARCHAR AS pricing_mode, + b.pricing:amount_minor::INT AS fixed_amount_minor, + b.pricing:percent_off::NUMBER(5, 2) AS percent_off, + UPPER(b.pricing:currency::VARCHAR)::VARCHAR AS pricing_currency_code, + b.pricing::OBJECT AS pricing, + @sql.load_metadata(b) + FROM cdc_latest AS b +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/catalogue/stg_categories.sql b/benchmarks/duckshop/project/models/staging/catalogue/stg_categories.sql new file mode 100644 index 0000000000..d3efb76efc --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/catalogue/stg_categories.sql @@ -0,0 +1,41 @@ +MODEL ( + description "One row per live category node with its parent and English and German display names", + columns ( + category_id (type VARCHAR, nullable false, audits [not_null, unique]), + parent_category_id (type VARCHAR), + slug (type VARCHAR, audits [not_null]), + display_name (type VARCHAR), + display_name_de (type VARCHAR), + sort_order (type INT), + translation_count (type INT), + is_root (type BOOLEAN), + translations (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("catalogue__categories") +), + +@sql.cdc_latest(src, category_id) + +final AS ( + SELECT + c.category_id, + NULLIF(c.parent_category_id, c.category_id)::VARCHAR AS parent_category_id, + LOWER(REGEXP_REPLACE(TRIM(c.slug), '[^a-zA-Z0-9]+', '-'))::VARCHAR AS slug, + COALESCE(c.translations:en::VARCHAR, c.display_name)::VARCHAR AS display_name, + c.translations:de::VARCHAR AS display_name_de, + COALESCE(c.sort_order, 999)::INT AS sort_order, + ARRAY_SIZE(OBJECT_KEYS(COALESCE(c.translations, OBJECT_CONSTRUCT())))::INT AS translation_count, + ( + c.parent_category_id IS NULL + OR c.parent_category_id = c.category_id + ) AS is_root, + COALESCE(c.translations, OBJECT_CONSTRUCT())::OBJECT AS translations, + @sql.load_metadata(c) + FROM cdc_latest AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/catalogue/stg_price_lists.sql b/benchmarks/duckshop/project/models/staging/catalogue/stg_price_lists.sql new file mode 100644 index 0000000000..fef9332df9 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/catalogue/stg_price_lists.sql @@ -0,0 +1,41 @@ +MODEL ( + description "One row per live list price of a SKU in one currency over a validity window", + columns ( + price_list_id (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + currency_code (type VARCHAR, audits [not_null]), + list_amount_minor (type INT, audits [not_null]), + valid_from (type TIMESTAMP_TZ, audits [not_null]), + valid_to (type TIMESTAMP_TZ), + is_open_ended (type BOOLEAN), + min_quantity (type INT), + channel_restrictions (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("catalogue__price_lists") +), + +@sql.cdc_latest(src, price_list_id) + +final AS ( + SELECT + p.price_list_id, + @sql.normalise_sku(p.sku) AS sku, + UPPER(p.currency_code)::VARCHAR AS currency_code, + p.list_amount_minor::INT AS list_amount_minor, + p.valid_from, + NULLIF(p.valid_to, '9999-12-31 00:00:00 +0000'::TIMESTAMP_TZ) AS valid_to, + ( + p.valid_to IS NULL + OR YEAR(p.valid_to) = 9999 + ) AS is_open_ended, + COALESCE(p.conditions:min_qty::INT, 1)::INT AS min_quantity, + COALESCE(p.conditions:channels, ARRAY_CONSTRUCT())::ARRAY AS channel_restrictions, + @sql.load_metadata(p) + FROM cdc_latest AS p +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/catalogue/stg_product_media.sql b/benchmarks/duckshop/project/models/staging/catalogue/stg_product_media.sql new file mode 100644 index 0000000000..6f872fc406 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/catalogue/stg_product_media.sql @@ -0,0 +1,45 @@ +MODEL ( + description "One row per live product image or video with its pixel dimensions", + columns ( + media_id (type VARCHAR, nullable false, audits [not_null, unique]), + product_id (type VARCHAR, audits [not_null]), + media_url (type VARCHAR), + media_type (type VARCHAR, audits [accepted_values (values ["image", "video", "model3d"])]), + position (type INT), + width_px (type INT), + height_px (type INT), + aspect_ratio (type NUMBER(8, 4)), + file_extension (type VARCHAR), + dimensions (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("catalogue__product_media") +), + +@sql.cdc_latest(src, media_id) + +final AS ( + SELECT + m.media_id, + m.product_id, + REGEXP_REPLACE(m.url, '^http://', 'https://')::VARCHAR AS media_url, + LOWER(m.media_type)::VARCHAR AS media_type, + m.position::INT AS position, + m.dimensions:width::INT AS width_px, + m.dimensions:height::INT AS height_px, + ROUND( + m.dimensions:width::INT / NULLIF(m.dimensions:height::INT, 0), + 4 + )::NUMBER(8, 4) AS aspect_ratio, + LOWER( + REGEXP_SUBSTR(m.url, '\\.([A-Za-z0-9]{2,5})(\\?|$)', 1, 1, 'e', 1) + )::VARCHAR AS file_extension, + m.dimensions::OBJECT AS dimensions, + @sql.load_metadata(m) + FROM cdc_latest AS m +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/catalogue/stg_product_variant_barcodes.sql b/benchmarks/duckshop/project/models/staging/catalogue/stg_product_variant_barcodes.sql new file mode 100644 index 0000000000..14d776b436 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/catalogue/stg_product_variant_barcodes.sql @@ -0,0 +1,64 @@ +MODEL ( + description """ + One row per barcode of a variant, from the variant's barcode array, with its symbology inferred + from its length. Barcodes listed on several variants are dropped as ambiguous. + """, + columns ( + barcode (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + symbology (type VARCHAR, audits [accepted_values (values ["EAN13", "UPCA", "EAN8", "OTHER"])]), + is_primary (type BOOLEAN), + ), +); + +WITH variants AS ( + SELECT * + FROM __ref("stg_product_variants") +), + +array_barcodes AS ( + SELECT + v.sku, + b.index::INT AS barcode_index, + REGEXP_REPLACE( + b.value::VARCHAR, + '[^0-9]', + '' + ) AS digits + FROM variants AS v, LATERAL FLATTEN(input => v.barcodes) AS b +), + +unambiguous AS ( + SELECT + a.sku, + a.digits, + MIN(a.barcode_index) AS barcode_index + FROM array_barcodes AS a + WHERE a.digits <> '' + GROUP BY + a.sku, + a.digits + QUALIFY COUNT(*) OVER (PARTITION BY a.digits) = 1 +), + +final AS ( + SELECT + u.digits::VARCHAR AS barcode, + u.sku, + CASE LENGTH(u.digits) + WHEN 13 + THEN 'EAN13' + WHEN 12 + THEN 'UPCA' + WHEN 8 + THEN 'EAN8' + ELSE 'OTHER' + END::VARCHAR AS symbology, + ( + u.barcode_index = 0 + ) AS is_primary + FROM unambiguous AS u +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/catalogue/stg_product_variants.sql b/benchmarks/duckshop/project/models/staging/catalogue/stg_product_variants.sql new file mode 100644 index 0000000000..f808e0a8ff --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/catalogue/stg_product_variants.sql @@ -0,0 +1,41 @@ +MODEL ( + description "One row per live sellable variant (SKU)", + columns ( + sku (type VARCHAR, nullable false, audits [not_null, unique]), + product_id (type VARCHAR, audits [not_null]), + colour (type VARCHAR), + size_label (type VARCHAR), + weight_grams (type INT), + weight_kg (type NUMBER(10, 3)), + barcode_count (type INT), + is_fragile (type BOOLEAN), + pack_size (type INT), + attributes (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("catalogue__product_variants") +), + +@sql.cdc_latest(src, sku) + +final AS ( + SELECT + @sql.normalise_sku(v.sku) AS sku, + v.product_id, + INITCAP(TRIM(v.colour))::VARCHAR AS colour, + UPPER(REGEXP_REPLACE(v.size_label, '\\s+', ''))::VARCHAR AS size_label, + v.weight_grams::INT AS weight_grams, + ROUND(v.weight_grams / 1000, 3)::NUMBER(10, 3) AS weight_kg, + ARRAY_SIZE(COALESCE(v.barcodes, ARRAY_CONSTRUCT()))::INT AS barcode_count, + COALESCE(v.attributes:handling.fragile::BOOLEAN, FALSE) AS is_fragile, + COALESCE(TRY_TO_NUMBER(v.attributes:pack_size::VARCHAR), 1)::INT AS pack_size, + v.barcodes::ARRAY AS barcodes, + v.attributes::OBJECT AS attributes, + @sql.load_metadata(v) + FROM cdc_latest AS v +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/catalogue/stg_products.sql b/benchmarks/duckshop/project/models/staging/catalogue/stg_products.sql new file mode 100644 index 0000000000..3e6fa0e752 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/catalogue/stg_products.sql @@ -0,0 +1,61 @@ +MODEL ( + description """ + One row per live product with its category path split into levels and its specs object + """, + columns ( + product_id (type VARCHAR, nullable false, audits [not_null, unique]), + title (type VARCHAR, audits [not_null]), + category_path (type VARCHAR), + category_level_1 (type VARCHAR), + category_level_2 (type VARCHAR), + category_leaf (type VARCHAR), + category_depth (type INT), + brand_code (type VARCHAR), + launched_on (type DATE), + is_active (type BOOLEAN), + spec_key_count (type INT), + specs (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("catalogue__products") +), + +@sql.cdc_latest(src, product_id) + +paths AS ( + SELECT + p.product_id, + p.title, + p.brand_code, + p.launched_on, + p.is_active, + p.specs, + p._seq, + p._loaded_at, + LOWER(TRIM(REGEXP_REPLACE(p.category_path, '/{2,}', '/'), '/')) AS clean_path + FROM cdc_latest AS p +), final AS ( + SELECT + p.product_id, + REGEXP_REPLACE(TRIM(p.title), '\\s{2,}', ' ')::VARCHAR AS title, + p.clean_path::VARCHAR AS category_path, + SPLIT_PART(p.clean_path, '/', 1)::VARCHAR AS category_level_1, + NULLIF(SPLIT_PART(p.clean_path, '/', 2), '')::VARCHAR AS category_level_2, + GET( + SPLIT(p.clean_path, '/'), + ARRAY_SIZE(SPLIT(p.clean_path, '/')) - 1 + )::VARCHAR AS category_leaf, + ARRAY_SIZE(SPLIT(p.clean_path, '/'))::INT AS category_depth, + UPPER(p.brand_code)::VARCHAR AS brand_code, + p.launched_on, + COALESCE(p.is_active, TRUE) AS is_active, + ARRAY_SIZE(OBJECT_KEYS(p.specs))::INT AS spec_key_count, + IFF(IS_OBJECT(p.specs), p.specs, OBJECT_CONSTRUCT())::OBJECT AS specs, + @sql.load_metadata(p) + FROM paths AS p +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/catalogue/stg_supplier_costs.sql b/benchmarks/duckshop/project/models/staging/catalogue/stg_supplier_costs.sql new file mode 100644 index 0000000000..f9ed7fa23a --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/catalogue/stg_supplier_costs.sql @@ -0,0 +1,47 @@ +MODEL ( + description "One row per live supplier unit cost for a SKU from an effective date", + columns ( + cost_id (type VARCHAR, nullable false, audits [not_null, unique]), + supplier_code (type VARCHAR, audits [not_null]), + sku (type VARCHAR, audits [not_null]), + currency_code (type VARCHAR), + unit_cost_minor (type INT), + effective_from (type DATE), + effective_to (type DATE), + volume_break_count (type INT), + best_break_cost_minor (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("catalogue__supplier_costs") +), + +@sql.cdc_latest(src, cost_id) + +final AS ( + SELECT + c.cost_id, + UPPER(c.supplier_code)::VARCHAR AS supplier_code, + @sql.normalise_sku(c.sku) AS sku, + UPPER(c.currency_code)::VARCHAR AS currency_code, + c.unit_cost_minor::INT AS unit_cost_minor, + c.effective_from, + DATEADD( + 'day', + -1, + LEAD(c.effective_from) OVER ( + PARTITION BY c.supplier_code, @sql.normalise_sku(c.sku) + ORDER BY c.effective_from + ) + )::DATE AS effective_to, + ARRAY_SIZE(COALESCE(c.volume_breaks, ARRAY_CONSTRUCT()))::INT AS volume_break_count, + ARRAY_MIN( + TRANSFORM(c.volume_breaks, b VARIANT -> b:unit_cost_minor) + )::INT AS best_break_cost_minor, + @sql.load_metadata(c) + FROM cdc_latest AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/catalogue/stg_suppliers.sql b/benchmarks/duckshop/project/models/staging/catalogue/stg_suppliers.sql new file mode 100644 index 0000000000..704eba1121 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/catalogue/stg_suppliers.sql @@ -0,0 +1,59 @@ +MODEL ( + description "One row per live supplier with payment terms and its primary contact", + columns ( + supplier_code (type VARCHAR, nullable false, audits [not_null, unique]), + supplier_name (type VARCHAR), + country_code (type VARCHAR), + payment_terms_days (type INT), + early_payment_discount_pct (type NUMBER(5, 2)), + primary_contact_email (type VARCHAR), + contact_count (type INT), + payment_terms (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("catalogue__suppliers") +), + +@sql.cdc_latest(src, supplier_code) + +contacts AS ( + SELECT + s.supplier_code, + c.value:email::VARCHAR AS contact_email, + COALESCE(c.value:is_primary::BOOLEAN, FALSE) AS is_primary, + c.index AS contact_index + FROM cdc_latest AS s, LATERAL FLATTEN(input => s.contacts, outer => TRUE) AS c +), primary_contacts AS ( + SELECT + c.supplier_code, + c.contact_email, + COUNT(c.contact_email) OVER (PARTITION BY c.supplier_code) AS contact_count + FROM contacts AS c + QUALIFY RANK() OVER (PARTITION BY c.supplier_code ORDER BY c.is_primary DESC, c.contact_index) = 1 +), final AS ( + SELECT + UPPER(s.supplier_code)::VARCHAR AS supplier_code, + s.supplier_name, + UPPER(s.country_code)::VARCHAR AS country_code, + COALESCE(s.payment_terms:net_days::INT, TRY_TO_NUMBER(REGEXP_SUBSTR( + s.payment_terms:label::VARCHAR, + 'NET\\s*([0-9]+)', + 1, + 1, + 'ie', + 1 + )))::INT AS payment_terms_days, + s.payment_terms:early_discount_pct::NUMBER(5, 2) AS early_payment_discount_pct, + @sql.normalise_email(pc.contact_email) AS primary_contact_email, + pc.contact_count::INT AS contact_count, + s.payment_terms::OBJECT AS payment_terms, + @sql.load_metadata(s) + FROM cdc_latest AS s + LEFT JOIN primary_contacts AS pc + ON s.supplier_code = pc.supplier_code +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_app_events.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_app_events.sql new file mode 100644 index 0000000000..a4df9706c9 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_app_events.sql @@ -0,0 +1,68 @@ +MODEL ( + description "One row per mobile app event, appended incrementally by received time", + materialized incremental, + incremental_strategy append, + cursor received_at, + cursor_type timestamp, + cursor_grain second, + append_cursor_inclusive false, + cursor_inputs ( + clickstream__app_events received_at, + ), + columns ( + event_id (type VARCHAR, nullable false, audits [not_null, unique]), + received_at (type TIMESTAMP_NTZ, audits [not_null]), + platform (type VARCHAR, audits [accepted_values (values ["phone", "tablet"])]), + app_version (type VARCHAR), + app_major_version (type INT), + event_name (type VARCHAR), + event_at_utc (type TIMESTAMP_NTZ), + device_id (type VARCHAR), + customer_id (type VARCHAR), + screen_name (type VARCHAR), + attributes (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("clickstream__app_events") +), + +fresh AS ( + SELECT + e.event_id, + e.received_at, + e.platform, + e.app_version, + e.payload + FROM src AS e + WHERE + e.received_at >= __cursor_start() + AND e.received_at < __cursor_end() + QUALIFY ROW_NUMBER() OVER (PARTITION BY e.event_id ORDER BY e._loaded_at DESC) = 1 +), + +final AS ( + SELECT + f.event_id, + f.received_at, + LOWER(f.platform)::VARCHAR AS platform, + f.app_version, + TRY_TO_NUMBER( + SPLIT_PART(REGEXP_SUBSTR(f.app_version, '[0-9]+(\\.[0-9]+)*'), '.', 1) + )::INT AS app_major_version, + LOWER(f.payload:name::VARCHAR)::VARCHAR AS event_name, + COALESCE( + TRY_TO_TIMESTAMP_NTZ(f.payload:ts::VARCHAR), + @sql.epoch_ms_to_utc(f.payload:ts_ms::NUMBER(38, 0)) + ) AS event_at_utc, + f.payload:device.id::VARCHAR AS device_id, + @sql.json_text(f.payload:user_id) AS customer_id, + f.payload:screen::VARCHAR AS screen_name, + COALESCE(f.payload:attributes, OBJECT_CONSTRUCT())::OBJECT AS attributes + FROM fresh AS f +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_app_push_opens.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_app_push_opens.sql new file mode 100644 index 0000000000..678fa9903a --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_app_push_opens.sql @@ -0,0 +1,42 @@ +MODEL ( + description "One row per push notification opened in the app, with the campaign it belongs to", + columns ( + event_id (type VARCHAR, nullable false, audits [not_null, unique]), + device_id (type VARCHAR), + customer_id (type VARCHAR), + opened_at_utc (type TIMESTAMP_NTZ), + push_campaign_id (type VARCHAR), + deep_link_path (type VARCHAR), + ), +); + +WITH app_events AS ( + SELECT * + FROM __ref("stg_app_events") +), + +final AS ( + SELECT + e.event_id, + e.device_id, + e.customer_id, + e.event_at_utc AS opened_at_utc, + COALESCE(e.attributes:campaign_id::VARCHAR, REGEXP_SUBSTR( + e.attributes:deep_link::VARCHAR, + 'cid=([A-Za-z0-9-]+)', + 1, + 1, + 'e', + 1 + ))::VARCHAR AS push_campaign_id, + REGEXP_REPLACE( + e.attributes:deep_link::VARCHAR, + '^[a-z]+://[^/]*', + '' + )::VARCHAR AS deep_link_path + FROM app_events AS e + WHERE e.event_name = 'push_open' +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_app_screen_views.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_app_screen_views.sql new file mode 100644 index 0000000000..01d1fc5a9b --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_app_screen_views.sql @@ -0,0 +1,53 @@ +MODEL ( + description "One row per app screen view, with time spent until the next event on the device", + columns ( + event_id (type VARCHAR, nullable false, audits [not_null, unique]), + device_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + platform (type VARCHAR), + screen_name (type VARCHAR), + viewed_at_utc (type TIMESTAMP_NTZ), + seconds_on_screen (type INT), + screen_class (type VARCHAR), + ), +); + +WITH app_events AS ( + SELECT * + FROM __ref("stg_app_events") +), + +ordered AS ( + SELECT + e.event_id, + e.device_id, + e.customer_id, + e.platform, + e.event_name, + e.screen_name, + e.event_at_utc, + e.attributes:screen_class::VARCHAR AS screen_class, + LEAD(e.event_at_utc) OVER ( + PARTITION BY e.device_id + ORDER BY e.event_at_utc, e.event_id + ) AS next_event_at_utc + FROM app_events AS e + WHERE e.device_id IS NOT NULL +), + +final AS ( + SELECT + o.event_id, + o.device_id, + o.customer_id, + o.platform, + INITCAP(REPLACE(o.screen_name, '_', ' '))::VARCHAR AS screen_name, + o.event_at_utc AS viewed_at_utc, + LEAST(DATEDIFF('second', o.event_at_utc, o.next_event_at_utc), 1800)::INT AS seconds_on_screen, + o.screen_class::VARCHAR AS screen_class + FROM ordered AS o + WHERE o.event_name = 'screen_view' +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_email_events.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_email_events.sql new file mode 100644 index 0000000000..41878e7c11 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_email_events.sql @@ -0,0 +1,58 @@ +MODEL ( + description "One row per email delivery or engagement event", + columns ( + email_event_id (type VARCHAR, nullable false, audits [not_null, unique]), + message_id (type VARCHAR, audits [not_null]), + event_type (type VARCHAR, audits [ + accepted_values ( + values ["sent", "delivered", "open", "click", "bounce", "unsubscribe", "complaint"], + ), + ]), + recipient_email (type VARCHAR), + campaign_id (type VARCHAR), + occurred_at (type TIMESTAMP_TZ, audits [not_null]), + clicked_url (type VARCHAR), + bounce_type (type VARCHAR), + is_machine_open (type BOOLEAN), + details (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("clickstream__email_events") +), + +latest AS ( + SELECT + e.email_event_id, + e.message_id, + e.event_type, + e.recipient_email, + e.campaign_id, + e.occurred_at, + e.details + FROM src AS e + QUALIFY ROW_NUMBER() OVER (PARTITION BY e.email_event_id ORDER BY e._loaded_at DESC) = 1 +), + +final AS ( + SELECT + l.email_event_id, + l.message_id, + LOWER(l.event_type)::VARCHAR AS event_type, + @sql.normalise_email(l.recipient_email) AS recipient_email, + l.campaign_id, + l.occurred_at, + l.details:url::VARCHAR AS clicked_url, + LOWER(l.details:bounce.type::VARCHAR)::VARCHAR AS bounce_type, + COALESCE(l.details:machine_open::BOOLEAN, REGEXP_LIKE( + l.details:user_agent::VARCHAR, + '.*(Proxy|Prefetch).*' + )) AS is_machine_open, + l.details::OBJECT AS details + FROM latest AS l +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_experiment_assignments.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_experiment_assignments.sql new file mode 100644 index 0000000000..cbd1075bbb --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_experiment_assignments.sql @@ -0,0 +1,39 @@ +MODEL ( + description "One row per experiment assignment, first assignment per visitor and experiment", + columns ( + assignment_id (type VARCHAR, nullable false, audits [not_null, unique]), + experiment_key (type VARCHAR, audits [not_null]), + variant_key (type VARCHAR, audits [not_null]), + anonymous_id (type VARCHAR), + customer_id (type VARCHAR), + assigned_at (type TIMESTAMP_TZ), + assignment_surface (type VARCHAR), + context (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("clickstream__experiment_assignments") +), + +final AS ( + SELECT + a.assignment_id, + LOWER(a.experiment_key)::VARCHAR AS experiment_key, + LOWER(a.variant_key)::VARCHAR AS variant_key, + LOWER(a.anonymous_id)::VARCHAR AS anonymous_id, + a.customer_id, + a.assigned_at, + COALESCE(a.context:surface::VARCHAR, 'web')::VARCHAR AS assignment_surface, + a.context::OBJECT AS context + FROM src AS a + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY a.experiment_key, LOWER(a.anonymous_id) + ORDER BY a.assigned_at, a.assignment_id + ) = 1 +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_search_queries.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_search_queries.sql new file mode 100644 index 0000000000..d09473d2aa --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_search_queries.sql @@ -0,0 +1,50 @@ +MODEL ( + description "One row per site search query with its applied filters", + columns ( + query_id (type VARCHAR, nullable false, audits [not_null, unique]), + anonymous_id (type VARCHAR), + query_text (type VARCHAR), + query_token_count (type INT), + results_count (type INT), + searched_at (type TIMESTAMP_TZ, audits [not_null]), + filter_category (type VARCHAR), + filter_max_price_minor (type INT), + sort_order (type VARCHAR), + filters (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("clickstream__search_queries") +), + +latest AS ( + SELECT + q.query_id, + q.anonymous_id, + q.query_text, + q.results_count, + q.searched_at, + q.filters + FROM src AS q + QUALIFY ROW_NUMBER() OVER (PARTITION BY q.query_id ORDER BY q._loaded_at DESC) = 1 +), + +final AS ( + SELECT + l.query_id, + LOWER(l.anonymous_id)::VARCHAR AS anonymous_id, + LOWER(TRIM(REGEXP_REPLACE(l.query_text, '[^\\w\\s-]', ' ')))::VARCHAR AS query_text, + REGEXP_COUNT(TRIM(l.query_text), '\\s+')::INT + 1 AS query_token_count, + l.results_count::INT AS results_count, + l.searched_at, + LOWER(l.filters:category::VARCHAR)::VARCHAR AS filter_category, + l.filters:price.max_minor::INT AS filter_max_price_minor, + COALESCE(l.filters:sort::VARCHAR, 'relevance')::VARCHAR AS sort_order, + COALESCE(l.filters, OBJECT_CONSTRUCT())::OBJECT AS filters + FROM latest AS l +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_web_cart_events.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_cart_events.sql new file mode 100644 index 0000000000..7907ecc049 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_cart_events.sql @@ -0,0 +1,54 @@ +MODEL ( + description """ + One row per add-to-cart or remove-from-cart item on web, flattened from the event items + """, + columns ( + cart_event_key (type VARCHAR, nullable false, audits [not_null, unique]), + event_id (type VARCHAR, audits [not_null]), + anonymous_id (type VARCHAR), + event_at_utc (type TIMESTAMP_NTZ), + action (type VARCHAR, audits [accepted_values (values ["add", "remove"])]), + sku (type VARCHAR), + quantity_delta (type INT), + unit_amount_minor (type INT), + currency_code (type VARCHAR), + ), +); + +WITH web_events AS ( + SELECT * + FROM __ref("stg_web_events") +), + +cart_events AS ( + SELECT + e.event_id, + e.anonymous_id, + e.event_at_utc, + IFF(e.event_type = 'add_to_cart', 'add', 'remove') AS action, + UPPER(e.properties:currency::VARCHAR) AS currency_code, + e.items + FROM web_events AS e + WHERE e.event_type IN ('add_to_cart', 'remove_from_cart') +), + +final AS ( + SELECT + ( + c.event_id || ':' || i.index::VARCHAR + )::VARCHAR AS cart_event_key, + c.event_id, + c.anonymous_id, + c.event_at_utc, + c.action::VARCHAR AS action, + @sql.normalise_sku(i.value:sku::VARCHAR) AS sku, + ( + IFF(c.action = 'add', 1, -1) * COALESCE(i.value:quantity::INT, 1) + )::INT AS quantity_delta, + i.value:price_minor::INT AS unit_amount_minor, + c.currency_code::VARCHAR AS currency_code + FROM cart_events AS c, LATERAL FLATTEN(input => c.items) AS i +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_web_checkout_steps.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_checkout_steps.sql new file mode 100644 index 0000000000..b5c241c498 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_checkout_steps.sql @@ -0,0 +1,55 @@ +MODEL ( + description "One row per visitor and checkout step reached on web, earliest occurrence only", + columns ( + checkout_step_key (type VARCHAR, nullable false, audits [not_null, unique]), + anonymous_id (type VARCHAR, audits [not_null]), + step_number (type INT, audits [not_null]), + step_name (type VARCHAR), + first_reached_at_utc (type TIMESTAMP_NTZ), + payment_option (type VARCHAR), + ), +); + +WITH web_events AS ( + SELECT * + FROM __ref("stg_web_events") +), + +steps AS ( + SELECT + e.event_id, + e.anonymous_id, + e.event_at_utc, + COALESCE( + e.properties:step::INT, + TRY_TO_NUMBER(REGEXP_SUBSTR(e.page_path, '/checkout/step-([0-9])', 1, 1, 'e', 1)) + ) AS step_number, + LOWER(e.properties:step_name::VARCHAR) AS step_name, + LOWER(e.properties:payment_option::VARCHAR) AS payment_option + FROM web_events AS e + WHERE + e.event_type IN ('checkout_step', 'checkout_started') + AND e.anonymous_id IS NOT NULL +), + +final AS ( + SELECT + ( + s.anonymous_id || '#' || s.step_number::VARCHAR + )::VARCHAR AS checkout_step_key, + s.anonymous_id, + s.step_number::INT AS step_number, + s.step_name::VARCHAR AS step_name, + s.event_at_utc AS first_reached_at_utc, + s.payment_option::VARCHAR AS payment_option + FROM steps AS s + WHERE s.step_number IS NOT NULL + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY s.anonymous_id, s.step_number + ORDER BY s.event_at_utc, s.event_id + ) = 1 +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_web_events.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_events.sql new file mode 100644 index 0000000000..b0319ba471 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_events.sql @@ -0,0 +1,94 @@ +MODEL ( + description """ + One row per web tracker event, typed from the raw payload and built in hourly microbatches. + An event without a client timestamp takes the time the collector received it. + """, + materialized incremental, + incremental_strategy delete_insert, + incremental_mode microbatch, + microbatch_strategy watermark, + cursor_watermark_mode all, + cursor received_at, + cursor_type timestamp, + cursor_grain hour, + batch_size 6h, + lookback 2h, + replay_on_change full, + cursor_inputs ( + clickstream__web_events (column received_at, roles [filter, watermark]), + ), + on_schema_change append_new_columns, + columns ( + event_id (type VARCHAR, nullable false, audits [not_null (run_scope delta_and_final), unique]), + received_at (type TIMESTAMP_NTZ, audits [not_null]), + event_type (type VARCHAR, audits [not_null]), + event_at_utc (type TIMESTAMP_NTZ), + anonymous_id (type VARCHAR), + customer_id (type VARCHAR), + user_email (type VARCHAR), + page_url (type VARCHAR), + page_path (type VARCHAR), + referrer_host (type VARCHAR), + user_agent (type VARCHAR), + locale (type VARCHAR), + client_timezone (type VARCHAR), + is_bot (type BOOLEAN), + properties (type OBJECT), + items (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("clickstream__web_events") +), + +deduplicated AS ( + SELECT + e.event_id, + e.received_at, + e.event + FROM src AS e + QUALIFY ROW_NUMBER() OVER (PARTITION BY e.event_id ORDER BY e._loaded_at DESC) = 1 +), + +final AS ( + SELECT + d.event_id, + d.received_at, + LOWER(REGEXP_REPLACE( + d.event:type::VARCHAR, + '[^A-Za-z]+', + '_' + ))::VARCHAR AS event_type, + COALESCE( + @sql.epoch_ms_to_utc(d.event:ts_ms::NUMBER(38, 0)), + d.received_at + ) AS event_at_utc, + LOWER(d.event:anonymous_id::VARCHAR)::VARCHAR AS anonymous_id, + @sql.json_text(d.event:user.id) AS customer_id, + @sql.normalise_email(d.event:user.email::VARCHAR) AS user_email, + d.event:page.url::VARCHAR AS page_url, + ( + '/' || LTRIM(PARSE_URL(d.event:page.url::VARCHAR, 1)['path']::VARCHAR, '/') + )::VARCHAR AS page_path, + LOWER(PARSE_URL(d.event:page.referrer::VARCHAR, 1)['host']::VARCHAR)::VARCHAR AS referrer_host, + d.event:context.ua::VARCHAR AS user_agent, + REPLACE( + d.event:context.locale::VARCHAR, + '_', + '-' + )::VARCHAR AS locale, + d.event:context.tz::VARCHAR AS client_timezone, + COALESCE(REGEXP_LIKE( + d.event:context.ua::VARCHAR, + '.*(bot|crawler|spider|headless).*', + 'i' + ), FALSE) AS is_bot, + COALESCE(d.event:properties, OBJECT_CONSTRUCT())::OBJECT AS properties, + COALESCE(d.event:items, ARRAY_CONSTRUCT())::ARRAY AS items + FROM deduplicated AS d +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_web_identifies.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_identifies.sql new file mode 100644 index 0000000000..1bb69d5f33 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_identifies.sql @@ -0,0 +1,54 @@ +MODEL ( + description """ + One row per anonymous visitor and customer pair seen on web, with first and last sighting + """, + columns ( + identity_link_key (type VARCHAR, nullable false, audits [not_null, unique]), + anonymous_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR, audits [not_null]), + first_seen_at_utc (type TIMESTAMP_NTZ), + last_seen_at_utc (type TIMESTAMP_NTZ), + event_count (type INT), + identify_methods (type ARRAY), + ), +); + +WITH web_events AS ( + SELECT * + FROM __ref("stg_web_events") +), + +identified AS ( + SELECT + e.anonymous_id, + e.customer_id, + e.event_at_utc, + COALESCE(e.properties:method::VARCHAR, e.event_type) AS identify_method + FROM web_events AS e + WHERE + e.anonymous_id IS NOT NULL + AND e.customer_id IS NOT NULL + AND e.event_type IN ('identify', 'login', 'signup', 'order_completed') +), + +final AS ( + SELECT + ( + i.anonymous_id || '>' || i.customer_id + )::VARCHAR AS identity_link_key, + i.anonymous_id, + i.customer_id, + MIN(i.event_at_utc) AS first_seen_at_utc, + MAX(i.event_at_utc) AS last_seen_at_utc, + COUNT(*)::INT AS event_count, + ARRAY_AGG(DISTINCT i.identify_method) WITHIN GROUP ( + ORDER BY i.identify_method + )::ARRAY AS identify_methods + FROM identified AS i + GROUP BY + i.anonymous_id, + i.customer_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_web_page_views.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_page_views.sql new file mode 100644 index 0000000000..c1e3a0952e --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_page_views.sql @@ -0,0 +1,72 @@ +MODEL ( + description "One row per human page view, with the previous page of the same visitor", + columns ( + event_id (type VARCHAR, nullable false, audits [not_null, unique]), + anonymous_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + viewed_at_utc (type TIMESTAMP_NTZ, audits [not_null]), + page_path (type VARCHAR), + page_type (type VARCHAR), + previous_page_path (type VARCHAR), + utm_source (type VARCHAR), + utm_medium (type VARCHAR), + utm_campaign (type VARCHAR), + referrer_host (type VARCHAR), + page_title (type VARCHAR), + ), +); + +WITH web_events AS ( + SELECT * + FROM __ref("stg_web_events") +), + +page_views AS ( + SELECT + e.event_id, + e.anonymous_id, + e.customer_id, + e.event_at_utc, + e.page_url, + e.page_path, + e.referrer_host, + e.properties:title::VARCHAR AS page_title + FROM web_events AS e + WHERE + e.event_type = 'page_view' + AND NOT e.is_bot + AND e.anonymous_id IS NOT NULL +), + +final AS ( + SELECT + p.event_id, + p.anonymous_id, + p.customer_id, + p.event_at_utc AS viewed_at_utc, + p.page_path, + CASE + WHEN p.page_path RLIKE '^/products?/.*' + THEN 'product' + WHEN p.page_path RLIKE '^/(c|category)/.*' + THEN 'category' + WHEN p.page_path RLIKE '^/checkout.*' + THEN 'checkout' + WHEN p.page_path IN ('/', '') + THEN 'home' + ELSE 'content' + END::VARCHAR AS page_type, + LAG(p.page_path) OVER ( + PARTITION BY p.anonymous_id + ORDER BY p.event_at_utc, p.event_id + )::VARCHAR AS previous_page_path, + REGEXP_SUBSTR(p.page_url, '[?&]utm_source=([^&#]+)', 1, 1, 'e', 1)::VARCHAR AS utm_source, + REGEXP_SUBSTR(p.page_url, '[?&]utm_medium=([^&#]+)', 1, 1, 'e', 1)::VARCHAR AS utm_medium, + REGEXP_SUBSTR(p.page_url, '[?&]utm_campaign=([^&#]+)', 1, 1, 'e', 1)::VARCHAR AS utm_campaign, + p.referrer_host, + p.page_title::VARCHAR AS page_title + FROM page_views AS p +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_web_product_views.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_product_views.sql new file mode 100644 index 0000000000..108c052c6d --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_product_views.sql @@ -0,0 +1,57 @@ +MODEL ( + description "One row per visitor, product and minute of product detail viewing on web", + columns ( + product_view_key (type VARCHAR, nullable false, audits [not_null, unique]), + anonymous_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + sku (type VARCHAR, audits [not_null]), + viewed_minute_utc (type TIMESTAMP_NTZ), + list_name (type VARCHAR), + list_position (type INT), + ), +); + +WITH web_events AS ( + SELECT * + FROM __ref("stg_web_events") +), + +views AS ( + SELECT + e.event_id, + e.anonymous_id, + e.customer_id, + @sql.normalise_sku(e.properties:sku::VARCHAR) AS sku, + DATE_TRUNC('minute', e.event_at_utc) AS viewed_minute_utc, + e.properties:list.name::VARCHAR AS list_name, + e.properties:list.position::INT AS list_position + FROM web_events AS e + WHERE + e.event_type = 'product_view' + AND NOT e.is_bot +), + +final AS ( + SELECT + ( + v.anonymous_id || '|' || v.sku || '|' || TO_VARCHAR(v.viewed_minute_utc, 'YYYYMMDDHH24MI') + )::VARCHAR AS product_view_key, + v.anonymous_id, + v.customer_id, + v.sku, + v.viewed_minute_utc, + v.list_name::VARCHAR AS list_name, + v.list_position + FROM views AS v + WHERE + v.sku IS NOT NULL + AND v.anonymous_id IS NOT NULL + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY v.anonymous_id, v.sku, v.viewed_minute_utc + ORDER BY v.event_id + ) = 1 +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/clickstream/stg_web_searches.sql b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_searches.sql new file mode 100644 index 0000000000..3c3531e4e8 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/clickstream/stg_web_searches.sql @@ -0,0 +1,37 @@ +MODEL ( + description "One row per on-site search event from the web tracker", + columns ( + event_id (type VARCHAR, nullable false, audits [not_null, unique]), + anonymous_id (type VARCHAR), + searched_at_utc (type TIMESTAMP_NTZ), + search_term (type VARCHAR), + result_count (type INT), + is_zero_result (type BOOLEAN), + ), +); + +WITH web_events AS ( + SELECT * + FROM __ref("stg_web_events") +), + +final AS ( + SELECT + e.event_id, + e.anonymous_id, + e.event_at_utc AS searched_at_utc, + LOWER(TRIM(REGEXP_REPLACE( + e.properties:query::VARCHAR, + '\\s+', + ' ' + )))::VARCHAR AS search_term, + e.properties:results::INT AS result_count, + ( + COALESCE(e.properties:results::INT, 0) = 0 + ) AS is_zero_result + FROM web_events AS e + WHERE e.event_type = 'search' +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/finance/_sqlbuild/_audits/generic/unique_combination.sql b/benchmarks/duckshop/project/models/staging/finance/_sqlbuild/_audits/generic/unique_combination.sql new file mode 100644 index 0000000000..d1fb60906e --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/finance/_sqlbuild/_audits/generic/unique_combination.sql @@ -0,0 +1,14 @@ +AUDIT (); + +WITH final AS ( + SELECT + @columns, + COUNT(*) AS duplicate_count + FROM @relation + GROUP BY + @columns + HAVING COUNT(*) > 1 +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/finance/stg_chargebacks.sql b/benchmarks/duckshop/project/models/staging/finance/stg_chargebacks.sql new file mode 100644 index 0000000000..87bea6c1d7 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/finance/stg_chargebacks.sql @@ -0,0 +1,48 @@ +MODEL ( + description "One row per live card chargeback with the evidence submitted", + columns ( + chargeback_id (type VARCHAR, nullable false, audits [not_null, unique]), + payment_id (type VARCHAR, audits [not_null]), + amount_minor (type INT), + currency_code (type VARCHAR), + reason_code (type VARCHAR), + reason_family (type VARCHAR), + opened_at (type TIMESTAMP_TZ), + closed_at (type TIMESTAMP_TZ), + outcome (type VARCHAR), + evidence_document_count (type INT), + evidence (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("finance__chargebacks") +), + +@sql.cdc_latest(src, chargeback_id) + +final AS ( + SELECT + c.chargeback_id, + c.payment_id, + c.amount_minor::INT AS amount_minor, + UPPER(c.currency_code)::VARCHAR AS currency_code, + UPPER(c.reason_code)::VARCHAR AS reason_code, + CASE + WHEN REGEXP_LIKE(c.reason_code, '^(10|4837|4863).*') + THEN 'fraud' + WHEN REGEXP_LIKE(c.reason_code, '^(13|4853|4855).*') + THEN 'consumer_dispute' + ELSE 'processing' + END::VARCHAR AS reason_family, + c.opened_at, + c.closed_at, + LOWER(COALESCE(c.outcome, 'open'))::VARCHAR AS outcome, + ARRAY_SIZE(COALESCE(c.evidence:documents, ARRAY_CONSTRUCT()))::INT AS evidence_document_count, + c.evidence::OBJECT AS evidence, + @sql.load_metadata(c) + FROM cdc_latest AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/finance/stg_fx_rates.sql b/benchmarks/duckshop/project/models/staging/finance/stg_fx_rates.sql new file mode 100644 index 0000000000..b30fdb1891 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/finance/stg_fx_rates.sql @@ -0,0 +1,34 @@ +MODEL ( + description "One row per currency and rate date, with the date the rate stops applying", + columns ( + currency_code (type VARCHAR, nullable false, audits [not_null]), + rate_date (type DATE, nullable false, audits [not_null]), + rate_to_usd (type NUMBER(18, 8), audits [not_null]), + usd_to_rate (type NUMBER(18, 8)), + next_rate_date (type DATE), + ), + audits [ + unique_combination (columns "currency_code, rate_date"), + ], +); + +WITH fx_rates AS ( + SELECT * + FROM __source("finance__fx_rates") +), + +final AS ( + SELECT + f.currency_code, + f.rate_date, + f.rate_to_usd, + ROUND(1::NUMBER(38, 12) / f.rate_to_usd, 8)::NUMBER(18, 8) AS usd_to_rate, + LEAD(f.rate_date) OVER ( + PARTITION BY f.currency_code + ORDER BY f.rate_date + )::DATE AS next_rate_date + FROM fx_rates AS f +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/finance/stg_invoice_tax_lines.sql b/benchmarks/duckshop/project/models/staging/finance/stg_invoice_tax_lines.sql new file mode 100644 index 0000000000..8bab90976b --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/finance/stg_invoice_tax_lines.sql @@ -0,0 +1,32 @@ +MODEL ( + description "One row per tax line on an invoice, flattened from the invoice tax lines", + columns ( + invoice_tax_line_key (type VARCHAR, nullable false, audits [not_null, unique]), + invoice_id (type VARCHAR, audits [not_null]), + tax_code (type VARCHAR), + tax_rate_pct (type NUMBER(9, 4)), + taxable_minor (type INT), + tax_minor (type INT), + ), +); + +WITH invoices AS ( + SELECT * + FROM __ref("stg_invoices") +), + +final AS ( + SELECT + ( + i.invoice_id || ':' || t.index::VARCHAR + )::VARCHAR AS invoice_tax_line_key, + i.invoice_id, + UPPER(t.value:code::VARCHAR)::VARCHAR AS tax_code, + t.value:rate_pct::NUMBER(9, 4) AS tax_rate_pct, + t.value:taxable_minor::INT AS taxable_minor, + t.value:tax_minor::INT AS tax_minor + FROM invoices AS i, LATERAL FLATTEN(input => i.tax_lines) AS t +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/finance/stg_invoices.sql b/benchmarks/duckshop/project/models/staging/finance/stg_invoices.sql new file mode 100644 index 0000000000..fe3d87d2cf --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/finance/stg_invoices.sql @@ -0,0 +1,50 @@ +MODEL ( + description "One row per live customer invoice with totals in minor units", + columns ( + invoice_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + invoice_number (type VARCHAR, audits [unique]), + invoice_series (type VARCHAR), + issued_at (type TIMESTAMP_TZ), + currency_code (type VARCHAR), + net_minor (type INT), + tax_minor (type INT), + gross_minor (type INT), + tax_lines (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("finance__invoices") +), + +@sql.cdc_latest(src, invoice_id) + +final AS ( + SELECT + i.invoice_id, + i.order_id, + UPPER(i.invoice_number)::VARCHAR AS invoice_number, + REGEXP_SUBSTR( + UPPER(i.invoice_number), + '^([A-Z]{2,4})-', + 1, + 1, + 'e', + 1 + )::VARCHAR AS invoice_series, + i.issued_at, + UPPER(i.currency_code)::VARCHAR AS currency_code, + i.totals:net_minor::INT AS net_minor, + i.totals:tax_minor::INT AS tax_minor, + COALESCE( + i.totals:gross_minor::INT, + i.totals:net_minor::INT + i.totals:tax_minor::INT + )::INT AS gross_minor, + COALESCE(i.tax_lines, ARRAY_CONSTRUCT())::ARRAY AS tax_lines, + @sql.load_metadata(i) + FROM cdc_latest AS i +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/finance/stg_ledger_entries.sql b/benchmarks/duckshop/project/models/staging/finance/stg_ledger_entries.sql new file mode 100644 index 0000000000..641942331d --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/finance/stg_ledger_entries.sql @@ -0,0 +1,52 @@ +MODEL ( + description "One row per live general ledger posting with a signed amount", + columns ( + entry_id (type VARCHAR, nullable false, audits [not_null, unique]), + journal_code (type VARCHAR), + account_code (type VARCHAR, audits [not_null]), + account_class (type VARCHAR), + signed_amount_minor (type INT), + currency_code (type VARCHAR), + posted_at (type TIMESTAMP_TZ, audits [not_null]), + reference_type (type VARCHAR), + reference_id (type VARCHAR), + reference (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("finance__ledger_entries") +), + +@sql.cdc_latest(src, entry_id) + +final AS ( + SELECT + e.entry_id, + UPPER(e.journal_code)::VARCHAR AS journal_code, + e.account_code, + CASE LEFT(e.account_code, 1) + WHEN '1' + THEN 'asset' + WHEN '2' + THEN 'liability' + WHEN '3' + THEN 'equity' + WHEN '4' + THEN 'revenue' + ELSE 'expense' + END::VARCHAR AS account_class, + ( + COALESCE(e.debit_minor, 0) - COALESCE(e.credit_minor, 0) + )::INT AS signed_amount_minor, + UPPER(e.currency_code)::VARCHAR AS currency_code, + e.posted_at, + LOWER(e.reference:type::VARCHAR)::VARCHAR AS reference_type, + @sql.json_text(e.reference:id) AS reference_id, + e.reference::OBJECT AS reference, + @sql.load_metadata(e) + FROM cdc_latest AS e +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/finance/stg_payout_lines.sql b/benchmarks/duckshop/project/models/staging/finance/stg_payout_lines.sql new file mode 100644 index 0000000000..69167870af --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/finance/stg_payout_lines.sql @@ -0,0 +1,68 @@ +MODEL ( + description "One row per payment settled in a payout, plus one line for the payout-level fee", + columns ( + payout_line_key (type VARCHAR, nullable false, audits [not_null, unique]), + payout_id (type VARCHAR, audits [not_null]), + payment_id (type VARCHAR), + line_type (type VARCHAR, audits [ + accepted_values (values ["charge", "refund", "adjustment", "payout_fee"]), + ]), + gross_minor (type INT), + fee_minor (type INT), + ), +); + +WITH payouts AS ( + SELECT * + FROM __ref("stg_payouts") +), + +payment_lines AS ( + SELECT + ( + p.payout_id || ':' || l.index::VARCHAR + ) AS payout_line_key, + p.payout_id, + l.value:payment_id::VARCHAR AS payment_id, + LOWER(COALESCE(l.value:type::VARCHAR, 'charge')) AS line_type, + l.value:gross_minor::INT AS gross_minor, + COALESCE(l.value:fee_minor::INT, 0) AS fee_minor + FROM payouts AS p, LATERAL FLATTEN(input => p.lines) AS l +), + +fee_lines AS ( + SELECT + ( + p.payout_id || ':fee' + ) AS payout_line_key, + p.payout_id, + NULL AS payment_id, + 'payout_fee' AS line_type, + 0 AS gross_minor, + p.fee_minor + FROM payouts AS p + WHERE p.fee_minor <> 0 +), + +final AS ( + SELECT + pl.payout_line_key::VARCHAR AS payout_line_key, + pl.payout_id, + pl.payment_id::VARCHAR AS payment_id, + pl.line_type::VARCHAR AS line_type, + pl.gross_minor::INT AS gross_minor, + pl.fee_minor::INT AS fee_minor + FROM payment_lines AS pl + UNION ALL + SELECT + fl.payout_line_key::VARCHAR AS payout_line_key, + fl.payout_id, + fl.payment_id::VARCHAR AS payment_id, + fl.line_type::VARCHAR AS line_type, + fl.gross_minor::INT AS gross_minor, + fl.fee_minor::INT AS fee_minor + FROM fee_lines AS fl +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/finance/stg_payouts.sql b/benchmarks/duckshop/project/models/staging/finance/stg_payouts.sql new file mode 100644 index 0000000000..3cdd4d60a3 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/finance/stg_payouts.sql @@ -0,0 +1,38 @@ +MODEL ( + description "One row per live payment provider payout", + columns ( + payout_id (type VARCHAR, nullable false, audits [not_null, unique]), + provider_code (type VARCHAR, audits [not_null]), + currency_code (type VARCHAR), + amount_minor (type INT), + fee_minor (type INT), + net_minor (type INT), + paid_out_on (type DATE), + lines (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("finance__payouts") +), + +@sql.cdc_latest(src, payout_id) + +final AS ( + SELECT + p.payout_id, + UPPER(p.provider_code)::VARCHAR AS provider_code, + UPPER(p.currency_code)::VARCHAR AS currency_code, + p.amount_minor::INT AS amount_minor, + COALESCE(p.fee_minor, 0)::INT AS fee_minor, + ( + p.amount_minor - COALESCE(p.fee_minor, 0) + )::INT AS net_minor, + p.paid_out_on, + COALESCE(p.lines, ARRAY_CONSTRUCT())::ARRAY AS lines, + @sql.load_metadata(p) + FROM cdc_latest AS p +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/finance/stg_settlement_reports.sql b/benchmarks/duckshop/project/models/staging/finance/stg_settlement_reports.sql new file mode 100644 index 0000000000..2a82524349 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/finance/stg_settlement_reports.sql @@ -0,0 +1,49 @@ +MODEL ( + description """ + One row per provider settlement file whose JSON body parses, with its summary totals + """, + columns ( + report_id (type VARCHAR, nullable false, audits [not_null, unique]), + provider_code (type VARCHAR, audits [not_null]), + report_date (type DATE, audits [not_null]), + currency_code (type VARCHAR), + gross_minor (type INT), + fees_minor (type INT), + line_count (type INT), + report (type VARIANT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("finance__settlement_reports") +), + +parsed AS ( + SELECT + r.report_id, + r.provider_code, + r.report_date, + TRY_PARSE_JSON(r.report_body) AS report + FROM src AS r + QUALIFY ROW_NUMBER() OVER (PARTITION BY r.report_id ORDER BY r._loaded_at DESC) = 1 +), + +final AS ( + SELECT + p.report_id, + UPPER(p.provider_code)::VARCHAR AS provider_code, + p.report_date, + UPPER(p.report:summary.currency::VARCHAR)::VARCHAR AS currency_code, + p.report:summary.gross_minor::INT AS gross_minor, + p.report:summary.fees_minor::INT AS fees_minor, + ARRAY_SIZE(p.report:lines)::INT AS line_count, + p.report + FROM parsed AS p + WHERE + p.report IS NOT NULL + AND IS_OBJECT(p.report) +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/finance/stg_tax_rates.sql b/benchmarks/duckshop/project/models/staging/finance/stg_tax_rates.sql new file mode 100644 index 0000000000..aa871c8471 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/finance/stg_tax_rates.sql @@ -0,0 +1,36 @@ +MODEL ( + description "One row per live tax rate for a country, region and product tax category", + columns ( + tax_rate_id (type VARCHAR, nullable false, audits [not_null, unique]), + country_code (type VARCHAR, audits [not_null]), + tax_region (type VARCHAR), + tax_category (type VARCHAR), + rate_pct (type NUMBER(9, 4)), + rate_fraction (type NUMBER(9, 6)), + valid_from (type DATE), + valid_to (type DATE), + ), +); + +WITH src AS ( + SELECT * + FROM __source("finance__tax_rates") +), + +@sql.cdc_latest(src, tax_rate_id) + +final AS ( + SELECT + t.tax_rate_id, + UPPER(t.country_code)::VARCHAR AS country_code, + COALESCE(UPPER(t.tax_region), '*')::VARCHAR AS tax_region, + LOWER(t.tax_category)::VARCHAR AS tax_category, + t.rate_pct, + ROUND(t.rate_pct / 100, 6)::NUMBER(9, 6) AS rate_fraction, + t.valid_from, + COALESCE(t.valid_to, '9999-12-31'::DATE) AS valid_to, + @sql.load_metadata(t) + FROM cdc_latest AS t +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/fulfilment/stg_carrier_rate_cards.sql b/benchmarks/duckshop/project/models/staging/fulfilment/stg_carrier_rate_cards.sql new file mode 100644 index 0000000000..896227ce04 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/fulfilment/stg_carrier_rate_cards.sql @@ -0,0 +1,40 @@ +MODEL ( + description "One row per live carrier rate card lane, with its weight bands kept as an array", + columns ( + rate_card_id (type VARCHAR, nullable false, audits [not_null, unique]), + carrier_code (type VARCHAR, audits [not_null]), + service_level (type VARCHAR), + origin_country (type VARCHAR), + destination_country (type VARCHAR), + currency_code (type VARCHAR), + valid_from (type DATE), + valid_to (type DATE), + band_count (type INT), + bands (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("fulfilment__carrier_rate_cards") +), + +@sql.cdc_latest(src, rate_card_id) + +final AS ( + SELECT + r.rate_card_id, + UPPER(r.carrier_code)::VARCHAR AS carrier_code, + LOWER(r.service_level)::VARCHAR AS service_level, + UPPER(r.origin_country)::VARCHAR AS origin_country, + UPPER(r.destination_country)::VARCHAR AS destination_country, + UPPER(r.currency_code)::VARCHAR AS currency_code, + r.valid_from, + COALESCE(r.valid_to, '9999-12-31'::DATE) AS valid_to, + ARRAY_SIZE(r.bands)::INT AS band_count, + r.bands::ARRAY AS bands, + @sql.load_metadata(r) + FROM cdc_latest AS r +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/fulfilment/stg_carrier_webhook_errors.sql b/benchmarks/duckshop/project/models/staging/fulfilment/stg_carrier_webhook_errors.sql new file mode 100644 index 0000000000..d746aaf201 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/fulfilment/stg_carrier_webhook_errors.sql @@ -0,0 +1,44 @@ +MODEL ( + description "One row per carrier webhook delivery whose body is not valid JSON, kept for replay", + columns ( + webhook_id (type VARCHAR, nullable false, audits [not_null, unique]), + carrier_code (type VARCHAR), + received_at (type TIMESTAMP_NTZ), + body_length (type INT), + body_prefix (type VARCHAR), + looks_truncated (type BOOLEAN), + ), +); + +WITH src AS ( + SELECT * + FROM __source("fulfilment__carrier_webhooks") +), + +latest AS ( + SELECT + w.webhook_id, + w.carrier_code, + w.received_at, + w.body + FROM src AS w + QUALIFY ROW_NUMBER() OVER (PARTITION BY w.webhook_id ORDER BY w._loaded_at DESC) = 1 +), + +final AS ( + SELECT + l.webhook_id, + UPPER(l.carrier_code)::VARCHAR AS carrier_code, + l.received_at, + LENGTH(l.body)::INT AS body_length, + LEFT(REGEXP_REPLACE(l.body, '\\s+', ' '), 80)::VARCHAR AS body_prefix, + ( + LEFT(LTRIM(l.body), 1) = '{' + AND RIGHT(RTRIM(l.body), 1) <> '}' + ) AS looks_truncated + FROM latest AS l + WHERE TRY_PARSE_JSON(l.body) IS NULL +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/fulfilment/stg_carrier_webhooks.sql b/benchmarks/duckshop/project/models/staging/fulfilment/stg_carrier_webhooks.sql new file mode 100644 index 0000000000..d0ff13a4b0 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/fulfilment/stg_carrier_webhooks.sql @@ -0,0 +1,105 @@ +MODEL ( + description """ + One row per carrier status update from webhook bodies that parse as JSON. Some carriers send a + single update per delivery and others batch several updates in an events array. + """, + columns ( + carrier_update_key (type VARCHAR, nullable false, audits [not_null, unique]), + webhook_id (type VARCHAR, audits [not_null]), + carrier_code (type VARCHAR, audits [not_null]), + received_at (type TIMESTAMP_NTZ), + tracking_number (type VARCHAR), + status_code (type VARCHAR), + status_at (type TIMESTAMP_TZ), + carrier_event_id (type NUMBER(38, 0)), + is_batched (type BOOLEAN), + ), +); + +WITH src AS ( + SELECT * + FROM __source("fulfilment__carrier_webhooks") +), + +parsed AS ( + SELECT + w.webhook_id, + w.carrier_code, + w.received_at, + TRY_PARSE_JSON(w.body) AS payload + FROM src AS w + QUALIFY ROW_NUMBER() OVER (PARTITION BY w.webhook_id ORDER BY w._loaded_at DESC) = 1 +), + +single_updates AS ( + SELECT + p.webhook_id, + 0 AS update_index, + p.carrier_code, + p.received_at, + p.payload AS update_payload, + FALSE AS is_batched + FROM parsed AS p + WHERE + p.payload IS NOT NULL + AND NOT COALESCE(IS_ARRAY(p.payload:events), FALSE) +), + +batched_updates AS ( + SELECT + p.webhook_id, + e.index AS update_index, + p.carrier_code, + p.received_at, + e.value AS update_payload, + TRUE AS is_batched + FROM parsed AS p, LATERAL FLATTEN(input => p.payload:events) AS e + WHERE IS_ARRAY(p.payload:events) +), + +all_updates AS ( + SELECT + s.webhook_id, + s.update_index, + s.carrier_code, + s.received_at, + s.update_payload, + s.is_batched + FROM single_updates AS s + UNION ALL + SELECT + b.webhook_id, + b.update_index, + b.carrier_code, + b.received_at, + b.update_payload, + b.is_batched + FROM batched_updates AS b +), + +final AS ( + SELECT + ( + u.webhook_id || ':' || u.update_index::VARCHAR + )::VARCHAR AS carrier_update_key, + u.webhook_id, + UPPER(u.carrier_code)::VARCHAR AS carrier_code, + u.received_at, + UPPER(REGEXP_REPLACE( + COALESCE(u.update_payload:tracking_number, u.update_payload:trackingNumber)::VARCHAR, + '[^A-Za-z0-9]', + '' + ))::VARCHAR AS tracking_number, + UPPER( + COALESCE(u.update_payload:status.code, u.update_payload:event)::VARCHAR + )::VARCHAR AS status_code, + TRY_TO_TIMESTAMP_TZ( + COALESCE(u.update_payload:status.at, u.update_payload:timestamp)::VARCHAR + ) AS status_at, + TRY_TO_NUMBER(u.update_payload:event_id::VARCHAR)::NUMBER(38, 0) AS carrier_event_id, + u.is_batched + FROM all_updates AS u +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/fulfilment/stg_delivery_exceptions.sql b/benchmarks/duckshop/project/models/staging/fulfilment/stg_delivery_exceptions.sql new file mode 100644 index 0000000000..31772f67e2 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/fulfilment/stg_delivery_exceptions.sql @@ -0,0 +1,38 @@ +MODEL ( + description "One row per live delivery exception with its resolution", + columns ( + exception_id (type VARCHAR, nullable false, audits [not_null, unique]), + shipment_id (type VARCHAR, audits [not_null]), + exception_code (type VARCHAR), + exception_text (type VARCHAR), + occurred_at (type TIMESTAMP_TZ), + resolution_type (type VARCHAR), + resolved_at (type TIMESTAMP_TZ), + compensation_minor (type INT), + resolution (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("fulfilment__delivery_exceptions") +), + +@sql.cdc_latest(src, exception_id) + +final AS ( + SELECT + e.exception_id, + e.shipment_id, + UPPER(e.exception_code)::VARCHAR AS exception_code, + TRIM(e.exception_text)::VARCHAR AS exception_text, + e.occurred_at, + LOWER(@sql.json_text(e.resolution:type))::VARCHAR AS resolution_type, + TRY_TO_TIMESTAMP_TZ(e.resolution:at::VARCHAR) AS resolved_at, + COALESCE(e.resolution:compensation_minor::INT, 0)::INT AS compensation_minor, + e.resolution::OBJECT AS resolution, + @sql.load_metadata(e) + FROM cdc_latest AS e +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/fulfilment/stg_delivery_slots.sql b/benchmarks/duckshop/project/models/staging/fulfilment/stg_delivery_slots.sql new file mode 100644 index 0000000000..35585c9249 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/fulfilment/stg_delivery_slots.sql @@ -0,0 +1,41 @@ +MODEL ( + description "One row per live delivery slot with remaining capacity", + columns ( + slot_id (type VARCHAR, nullable false, audits [not_null, unique]), + warehouse_code (type VARCHAR, audits [not_null]), + slot_start_local (type TIMESTAMP_NTZ), + slot_end_local (type TIMESTAMP_NTZ), + slot_minutes (type INT), + capacity (type INT), + booked (type INT), + remaining_capacity (type INT), + excluded_postcode_prefixes (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("fulfilment__delivery_slots") +), + +@sql.cdc_latest(src, slot_id) + +final AS ( + SELECT + s.slot_id, + UPPER(s.warehouse_code)::VARCHAR AS warehouse_code, + s.slot_start_local, + s.slot_end_local, + TIMESTAMPDIFF('minute', s.slot_start_local, s.slot_end_local)::INT AS slot_minutes, + s.capacity::INT AS capacity, + COALESCE(s.booked, 0)::INT AS booked, + GREATEST(s.capacity - COALESCE(s.booked, 0), 0)::INT AS remaining_capacity, + COALESCE( + s.restrictions:excluded_postcode_prefixes, + ARRAY_CONSTRUCT() + )::ARRAY AS excluded_postcode_prefixes, + @sql.load_metadata(s) + FROM cdc_latest AS s +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/fulfilment/stg_pick_tasks.sql b/benchmarks/duckshop/project/models/staging/fulfilment/stg_pick_tasks.sql new file mode 100644 index 0000000000..fc7348e450 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/fulfilment/stg_pick_tasks.sql @@ -0,0 +1,42 @@ +MODEL ( + description "One row per live warehouse pick task in warehouse local time", + columns ( + task_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR), + picker_id (type VARCHAR), + started_at_local (type TIMESTAMP_NTZ), + completed_at_local (type TIMESTAMP_NTZ), + pick_seconds (type INT), + line_count (type INT), + units_picked (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("fulfilment__pick_tasks") +), + +@sql.cdc_latest(src, task_id) + +final AS ( + SELECT + t.task_id, + t.order_id, + UPPER(t.warehouse_code)::VARCHAR AS warehouse_code, + t.picker_id, + t.started_at_local, + t.completed_at_local, + DATEDIFF('second', t.started_at_local, t.completed_at_local)::INT AS pick_seconds, + ARRAY_SIZE(t.lines)::INT AS line_count, + REDUCE( + t.lines::ARRAY, + 0, + (total, l) -> total + COALESCE(l:qty::INT, 0) + )::INT AS units_picked, + @sql.load_metadata(t) + FROM cdc_latest AS t +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/fulfilment/stg_return_inspections.sql b/benchmarks/duckshop/project/models/staging/fulfilment/stg_return_inspections.sql new file mode 100644 index 0000000000..7beaec7dd9 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/fulfilment/stg_return_inspections.sql @@ -0,0 +1,36 @@ +MODEL ( + description "One row per live returned-item inspection in warehouse local time", + columns ( + inspection_id (type VARCHAR, nullable false, audits [not_null, unique]), + return_id (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR), + grade (type VARCHAR, audits [accepted_values (values ["A", "B", "C", "SCRAP"])]), + is_resellable (type BOOLEAN), + inspected_at_local (type TIMESTAMP_NTZ), + defect_codes (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("fulfilment__return_inspections") +), + +@sql.cdc_latest(src, inspection_id) + +final AS ( + SELECT + i.inspection_id, + i.return_id, + UPPER(i.warehouse_code)::VARCHAR AS warehouse_code, + UPPER(TRIM(i.grade))::VARCHAR AS grade, + ( + UPPER(TRIM(i.grade)) IN ('A', 'B') + ) AS is_resellable, + i.inspected_at_local, + COALESCE(i.findings:defects, ARRAY_CONSTRUCT())::ARRAY AS defect_codes, + @sql.load_metadata(i) + FROM cdc_latest AS i +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/fulfilment/stg_returns.sql b/benchmarks/duckshop/project/models/staging/fulfilment/stg_returns.sql new file mode 100644 index 0000000000..95de9fd473 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/fulfilment/stg_returns.sql @@ -0,0 +1,42 @@ +MODEL ( + description "One row per live customer return request", + columns ( + return_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + sku (type VARCHAR, audits [not_null]), + qty (type INT), + reason_text (type VARCHAR), + return_status (type VARCHAR), + requested_at (type TIMESTAMP_TZ), + received_at (type TIMESTAMP_TZ), + photo_count (type INT), + refund_preference (type VARCHAR), + ), +); + +WITH src AS ( + SELECT * + FROM __source("fulfilment__returns") +), + +@sql.cdc_latest(src, return_id) + +final AS ( + SELECT + r.return_id, + r.order_id, + @sql.normalise_sku(r.sku) AS sku, + GREATEST(r.qty, 1)::INT AS qty, + REGEXP_REPLACE(TRIM(r.reason_text), '\\s+', ' ')::VARCHAR AS reason_text, + LOWER(r.return_status)::VARCHAR AS return_status, + r.requested_at, + r.received_at, + ARRAY_SIZE(COALESCE(r.photos, ARRAY_CONSTRUCT()))::INT AS photo_count, + LOWER( + COALESCE(r.refund_preference:method::VARCHAR, 'original_payment') + )::VARCHAR AS refund_preference, + @sql.load_metadata(r) + FROM cdc_latest AS r +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/fulfilment/stg_shipment_packages.sql b/benchmarks/duckshop/project/models/staging/fulfilment/stg_shipment_packages.sql new file mode 100644 index 0000000000..3d5ab9996a --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/fulfilment/stg_shipment_packages.sql @@ -0,0 +1,41 @@ +MODEL ( + description "One row per live shipment package with volumetric weight", + columns ( + package_id (type VARCHAR, nullable false, audits [not_null, unique]), + shipment_id (type VARCHAR, audits [not_null]), + weight_grams (type INT), + length_cm (type NUMBER(8, 2)), + width_cm (type NUMBER(8, 2)), + height_cm (type NUMBER(8, 2)), + volumetric_weight_kg (type NUMBER(10, 3)), + content_line_count (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("fulfilment__shipment_packages") +), + +@sql.cdc_latest(src, package_id) + +final AS ( + SELECT + p.package_id, + p.shipment_id, + p.weight_grams::INT AS weight_grams, + p.dimensions:l::NUMBER(8, 2) AS length_cm, + p.dimensions:w::NUMBER(8, 2) AS width_cm, + p.dimensions:h::NUMBER(8, 2) AS height_cm, + ROUND( + p.dimensions:l::NUMBER(8, 2) * p.dimensions:w::NUMBER(8, 2) * p.dimensions:h::NUMBER(8, 2) + / 5000, + 3 + )::NUMBER(10, 3) AS volumetric_weight_kg, + ARRAY_SIZE(COALESCE(p.contents, ARRAY_CONSTRUCT()))::INT AS content_line_count, + p.contents::ARRAY AS contents, + @sql.load_metadata(p) + FROM cdc_latest AS p +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/fulfilment/stg_shipment_tracking_events.sql b/benchmarks/duckshop/project/models/staging/fulfilment/stg_shipment_tracking_events.sql new file mode 100644 index 0000000000..79808198fc --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/fulfilment/stg_shipment_tracking_events.sql @@ -0,0 +1,100 @@ +MODEL ( + description """ + One row per tracking event of a shipment: carrier scans flattened from tracking, plus recorded + delivery exceptions + """, + columns ( + tracking_event_key (type VARCHAR, nullable false, audits [not_null, unique]), + shipment_id (type VARCHAR, audits [not_null]), + event_source (type VARCHAR, audits [accepted_values (values ["scan", "exception"])]), + event_code (type VARCHAR), + event_at (type TIMESTAMP_TZ), + location_text (type VARCHAR), + location_country_code (type VARCHAR), + event_sequence (type INT), + ), +); + +WITH shipments AS ( + SELECT * + FROM __ref("stg_shipments") +), + +exceptions AS ( + SELECT * + FROM __ref("stg_delivery_exceptions") +), + +scans AS ( + SELECT + s.shipment_id, + e.index::INT AS event_sequence, + e.value:code::VARCHAR AS raw_code, + COALESCE(TRY_TO_TIMESTAMP_TZ(e.value:ts::VARCHAR), TRY_TO_TIMESTAMP_TZ( + e.value:ts::VARCHAR, + 'DD/MM/YYYY HH24:MI TZHTZM' + )) AS event_at, + e.value:location::VARCHAR AS raw_location + FROM shipments AS s, LATERAL FLATTEN(input => s.tracking_events) AS e +), + +scan_events AS ( + SELECT + ( + s.shipment_id || ':' || s.event_sequence::VARCHAR + ) AS tracking_event_key, + s.shipment_id, + 'scan' AS event_source, + UPPER(REGEXP_REPLACE(s.raw_code, '[^A-Za-z_]', '')) AS event_code, + s.event_at, + INITCAP(TRIM(SPLIT_PART(s.raw_location, ',', 1))) AS location_text, + UPPER( + REGEXP_SUBSTR(s.raw_location, ',\\s*([A-Za-z]{2})\\s*$', 1, 1, 'e', 1) + ) AS location_country_code, + s.event_sequence + FROM scans AS s +), + +exception_events AS ( + SELECT + ( + x.shipment_id || ':x:' || x.exception_id + ) AS tracking_event_key, + x.shipment_id, + 'exception' AS event_source, + ( + 'EXCEPTION_' || x.exception_code + ) AS event_code, + x.occurred_at AS event_at, + NULL AS location_text, + NULL AS location_country_code, + NULL AS event_sequence + FROM exceptions AS x +), + +final AS ( + SELECT + se.tracking_event_key::VARCHAR AS tracking_event_key, + se.shipment_id, + se.event_source::VARCHAR AS event_source, + se.event_code::VARCHAR AS event_code, + se.event_at, + se.location_text::VARCHAR AS location_text, + se.location_country_code::VARCHAR AS location_country_code, + se.event_sequence::INT AS event_sequence + FROM scan_events AS se + UNION ALL + SELECT + ee.tracking_event_key::VARCHAR AS tracking_event_key, + ee.shipment_id, + ee.event_source::VARCHAR AS event_source, + ee.event_code::VARCHAR AS event_code, + ee.event_at, + ee.location_text::VARCHAR AS location_text, + ee.location_country_code::VARCHAR AS location_country_code, + ee.event_sequence::INT AS event_sequence + FROM exception_events AS ee +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/fulfilment/stg_shipments.sql b/benchmarks/duckshop/project/models/staging/fulfilment/stg_shipments.sql new file mode 100644 index 0000000000..d6c87b91f1 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/fulfilment/stg_shipments.sql @@ -0,0 +1,44 @@ +MODEL ( + description "One row per live shipment with tracking summary fields", + columns ( + shipment_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR, audits [not_null]), + carrier_code (type VARCHAR, audits [not_null]), + service_level (type VARCHAR), + tracking_number (type VARCHAR), + shipped_at (type TIMESTAMP_TZ), + delivered_at (type TIMESTAMP_TZ), + tracking_event_count (type INT), + tracking_events (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("fulfilment__shipments") +), + +@sql.cdc_latest(src, shipment_id) + +final AS ( + SELECT + s.shipment_id, + s.order_id, + UPPER(s.warehouse_code)::VARCHAR AS warehouse_code, + UPPER(s.carrier_code)::VARCHAR AS carrier_code, + LOWER(COALESCE(s.service_level, 'standard'))::VARCHAR AS service_level, + UPPER(REGEXP_REPLACE( + s.tracking:number::VARCHAR, + '[^A-Za-z0-9]', + '' + ))::VARCHAR AS tracking_number, + s.shipped_at, + s.delivered_at, + ARRAY_SIZE(COALESCE(s.tracking:events, ARRAY_CONSTRUCT()))::INT AS tracking_event_count, + COALESCE(s.tracking:events, ARRAY_CONSTRUCT())::ARRAY AS tracking_events, + @sql.load_metadata(s) + FROM cdc_latest AS s +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/growth/stg_affiliate_conversions.sql b/benchmarks/duckshop/project/models/staging/growth/stg_affiliate_conversions.sql new file mode 100644 index 0000000000..2037efe308 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/growth/stg_affiliate_conversions.sql @@ -0,0 +1,52 @@ +MODEL ( + description "One row per live affiliate conversion with the click it is attributed to", + columns ( + conversion_id (type VARCHAR, nullable false, audits [not_null, unique]), + affiliate_code (type VARCHAR, audits [not_null]), + order_id (type VARCHAR), + commission_minor (type INT), + currency_code (type VARCHAR), + converted_at (type TIMESTAMP_TZ), + clicked_at (type TIMESTAMP_TZ), + click_to_conversion_hours (type INT), + sub_id (type VARCHAR), + ), +); + +WITH src AS ( + SELECT * + FROM __source("growth__affiliate_conversions") +), + +@sql.cdc_latest(src, conversion_id) + +clicks AS ( + SELECT + c.conversion_id, + TRY_TO_TIMESTAMP_TZ(c.click_payload:clicked_at::VARCHAR) AS clicked_at + FROM cdc_latest AS c +), final AS ( + SELECT + c.conversion_id, + UPPER(c.affiliate_code)::VARCHAR AS affiliate_code, + c.order_id, + c.commission_minor::INT AS commission_minor, + UPPER(c.currency_code)::VARCHAR AS currency_code, + c.converted_at, + k.clicked_at, + DATEDIFF('hour', k.clicked_at, c.converted_at)::INT AS click_to_conversion_hours, + REGEXP_SUBSTR( + c.click_payload:landing_url::VARCHAR, + '[?&]subid=([^&]+)', + 1, + 1, + 'e', + 1 + )::VARCHAR AS sub_id, + @sql.load_metadata(c) + FROM cdc_latest AS c + INNER JOIN clicks AS k + ON c.conversion_id = k.conversion_id +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/growth/stg_campaign_spend.sql b/benchmarks/duckshop/project/models/staging/growth/stg_campaign_spend.sql new file mode 100644 index 0000000000..4c96861308 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/growth/stg_campaign_spend.sql @@ -0,0 +1,60 @@ +MODEL ( + description """ + One row per campaign, day and platform metric, flattened from the daily spend export + """, + columns ( + spend_metric_key (type VARCHAR, nullable false, audits [not_null, unique]), + spend_id (type VARCHAR, audits [not_null]), + spend_date (type DATE, audits [not_null]), + channel (type VARCHAR), + campaign_id (type VARCHAR), + currency_code (type VARCHAR), + metric_name (type VARCHAR, audits [not_null]), + metric_value (type NUMBER(38, 6)), + platform_account_id (type VARCHAR), + ), +); + +WITH src AS ( + SELECT * + FROM __source("growth__campaign_spend") +), + +@sql.cdc_latest(src, spend_id) + +metrics AS ( + SELECT + s.spend_id, + s.spend_date, + s.channel, + s.campaign_id, + s.currency_code, + m.value:name::VARCHAR AS metric_name, + m.value:value AS metric_raw, + s.report:account.id::VARCHAR AS platform_account_id + FROM cdc_latest AS s, LATERAL FLATTEN(input => s.report:metrics) AS m +), final AS ( + SELECT + ( + m.spend_id || ':' || LOWER(m.metric_name) + )::VARCHAR AS spend_metric_key, + m.spend_id, + m.spend_date, + UPPER(m.channel)::VARCHAR AS channel, + m.campaign_id, + UPPER(m.currency_code)::VARCHAR AS currency_code, + LOWER(REGEXP_REPLACE(m.metric_name, '([a-z])([A-Z])', '\\1_\\2'))::VARCHAR AS metric_name, + CASE TYPEOF(m.metric_raw) + WHEN 'VARCHAR' + THEN TRY_TO_DECIMAL(REPLACE( + m.metric_raw::VARCHAR, + ',', + '' + ), 38, 6) + ELSE m.metric_raw::NUMBER(38, 6) + END::NUMBER(38, 6) AS metric_value, + m.platform_account_id::VARCHAR AS platform_account_id + FROM metrics AS m +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/growth/stg_campaigns.sql b/benchmarks/duckshop/project/models/staging/growth/stg_campaigns.sql new file mode 100644 index 0000000000..a0574c7c38 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/growth/stg_campaigns.sql @@ -0,0 +1,50 @@ +MODEL ( + description "One row per live marketing campaign with its targeting", + columns ( + campaign_id (type VARCHAR, nullable false, audits [not_null, unique]), + campaign_name (type VARCHAR), + channel (type VARCHAR), + objective (type VARCHAR), + starts_on (type DATE), + ends_on (type DATE), + budget_minor (type INT), + currency_code (type VARCHAR), + target_regions (type ARRAY), + target_age_min (type INT), + target_age_max (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("growth__campaigns") +), + +@sql.cdc_latest(src, campaign_id) + +final AS ( + SELECT + c.campaign_id, + TRIM(c.campaign_name)::VARCHAR AS campaign_name, + UPPER(c.channel)::VARCHAR AS channel, + LOWER(c.objective)::VARCHAR AS objective, + c.starts_on, + c.ends_on, + c.budget_minor::INT AS budget_minor, + UPPER(c.currency_code)::VARCHAR AS currency_code, + COALESCE(c.targeting:regions, ARRAY_CONSTRUCT())::ARRAY AS target_regions, + TRY_TO_NUMBER(SPLIT_PART( + c.targeting:age_band::VARCHAR, + '-', + 1 + ))::INT AS target_age_min, + TRY_TO_NUMBER(SPLIT_PART( + c.targeting:age_band::VARCHAR, + '-', + 2 + ))::INT AS target_age_max, + @sql.load_metadata(c) + FROM cdc_latest AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/growth/stg_influencer_posts.sql b/benchmarks/duckshop/project/models/staging/growth/stg_influencer_posts.sql new file mode 100644 index 0000000000..6e2ba596c5 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/growth/stg_influencer_posts.sql @@ -0,0 +1,46 @@ +MODEL ( + description "One row per live creator post with engagement metrics", + columns ( + post_id (type VARCHAR, nullable false, audits [not_null, unique]), + creator_handle (type VARCHAR), + platform (type VARCHAR), + posted_at (type TIMESTAMP_TZ), + promo_code (type VARCHAR), + impressions (type INT), + likes (type INT), + engagement_rate (type NUMBER(9, 6)), + metrics (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("growth__influencer_posts") +), + +@sql.cdc_latest(src, post_id) + +final AS ( + SELECT + p.post_id, + LOWER(LTRIM(p.creator_handle, '@'))::VARCHAR AS creator_handle, + LOWER(p.platform)::VARCHAR AS platform, + p.posted_at, + UPPER(TRIM(p.promo_code))::VARCHAR AS promo_code, + p.metrics:impressions::INT AS impressions, + p.metrics:likes::INT AS likes, + ROUND( + ( + COALESCE(p.metrics:likes::INT, 0) + + COALESCE(p.metrics:comments::INT, 0) + + COALESCE(p.metrics:shares::INT, 0) + ) + / NULLIF(p.metrics:impressions::INT, 0), + 6 + )::NUMBER(9, 6) AS engagement_rate, + p.metrics::OBJECT AS metrics, + @sql.load_metadata(p) + FROM cdc_latest AS p +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/growth/stg_newsletter_subscriptions.sql b/benchmarks/duckshop/project/models/staging/growth/stg_newsletter_subscriptions.sql new file mode 100644 index 0000000000..b9cb6f0c4b --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/growth/stg_newsletter_subscriptions.sql @@ -0,0 +1,38 @@ +MODEL ( + description "One row per live newsletter list membership, including explicit unsubscribes", + columns ( + list_subscription_id (type VARCHAR, nullable false, audits [not_null, unique]), + email (type VARCHAR), + list_code (type VARCHAR), + list_status (type VARCHAR, audits [ + accepted_values (values ["subscribed", "unsubscribed", "pending", "cleaned"]), + ]), + subscribed_at (type TIMESTAMP_TZ), + unsubscribed_at (type TIMESTAMP_TZ), + signup_form (type VARCHAR), + is_double_opt_in (type BOOLEAN), + ), +); + +WITH src AS ( + SELECT * + FROM __source("growth__newsletter_subscriptions") +), + +@sql.cdc_latest(src, list_subscription_id) + +final AS ( + SELECT + s.list_subscription_id, + @sql.normalise_email(s.email) AS email, + UPPER(s.list_code)::VARCHAR AS list_code, + LOWER(s.list_status)::VARCHAR AS list_status, + s.subscribed_at, + s.unsubscribed_at, + @sql.json_text(s.signup_source:form) AS signup_form, + COALESCE(s.signup_source:double_opt_in::BOOLEAN, FALSE) AS is_double_opt_in, + @sql.load_metadata(s) + FROM cdc_latest AS s +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/growth/stg_referral_invites.sql b/benchmarks/duckshop/project/models/staging/growth/stg_referral_invites.sql new file mode 100644 index 0000000000..42a547276d --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/growth/stg_referral_invites.sql @@ -0,0 +1,36 @@ +MODEL ( + description "One row per live referral invite", + columns ( + invite_id (type VARCHAR, nullable false, audits [not_null, unique]), + referrer_customer_id (type VARCHAR, audits [not_null]), + invitee_email (type VARCHAR), + sent_at (type TIMESTAMP_TZ), + accepted_at (type TIMESTAMP_TZ), + reward_amount_minor (type INT), + reward_currency_code (type VARCHAR), + reward (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("growth__referral_invites") +), + +@sql.cdc_latest(src, invite_id) + +final AS ( + SELECT + i.invite_id, + i.referrer_customer_id, + @sql.normalise_email(i.invitee_email) AS invitee_email, + i.sent_at, + i.accepted_at, + i.reward:amount_minor::INT AS reward_amount_minor, + UPPER(i.reward:currency::VARCHAR)::VARCHAR AS reward_currency_code, + i.reward::OBJECT AS reward, + @sql.load_metadata(i) + FROM cdc_latest AS i +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/growth/stg_survey_responses.sql b/benchmarks/duckshop/project/models/staging/growth/stg_survey_responses.sql new file mode 100644 index 0000000000..ed2e65343d --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/growth/stg_survey_responses.sql @@ -0,0 +1,38 @@ +MODEL ( + description "One row per live survey response with the net promoter answer extracted", + columns ( + response_id (type VARCHAR, nullable false, audits [not_null, unique]), + survey_code (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + submitted_at (type TIMESTAMP_TZ), + nps_score (type INT), + answer_count (type INT), + answers (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("growth__survey_responses") +), + +@sql.cdc_latest(src, response_id) + +final AS ( + SELECT + r.response_id, + UPPER(r.survey_code)::VARCHAR AS survey_code, + r.customer_id, + r.submitted_at, + IFF( + TRY_TO_NUMBER(r.answers:nps::VARCHAR) BETWEEN 0 AND 10, + TRY_TO_NUMBER(r.answers:nps::VARCHAR), + NULL + )::INT AS nps_score, + ARRAY_SIZE(OBJECT_KEYS(r.answers))::INT AS answer_count, + r.answers::OBJECT AS answers, + @sql.load_metadata(r) + FROM cdc_latest AS r +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/inventory/stg_bin_locations.sql b/benchmarks/duckshop/project/models/staging/inventory/stg_bin_locations.sql new file mode 100644 index 0000000000..ceeed1e59b --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/inventory/stg_bin_locations.sql @@ -0,0 +1,38 @@ +MODEL ( + description "One row per live warehouse bin, with aisle, bay and level parsed from its label", + columns ( + bin_id (type VARCHAR, nullable false, audits [not_null, unique]), + warehouse_code (type VARCHAR, audits [not_null]), + bin_label (type VARCHAR), + aisle (type VARCHAR), + bay (type INT), + shelf_level (type INT), + is_pick_face (type BOOLEAN), + max_weight_kg (type NUMBER(8, 2)), + temperature_zone (type VARCHAR), + ), +); + +WITH src AS ( + SELECT * + FROM __source("inventory__bin_locations") +), + +@sql.cdc_latest(src, bin_id) + +final AS ( + SELECT + b.bin_id, + UPPER(b.warehouse_code)::VARCHAR AS warehouse_code, + UPPER(b.bin_label)::VARCHAR AS bin_label, + REGEXP_SUBSTR(UPPER(b.bin_label), '^([A-Z]+)-', 1, 1, 'e', 1)::VARCHAR AS aisle, + TRY_TO_NUMBER(REGEXP_SUBSTR(b.bin_label, '-([0-9]+)-', 1, 1, 'e', 1))::INT AS bay, + TRY_TO_NUMBER(REGEXP_SUBSTR(b.bin_label, '-([0-9]+)$', 1, 1, 'e', 1))::INT AS shelf_level, + COALESCE(b.attributes:pick_face::BOOLEAN, FALSE) AS is_pick_face, + b.attributes:max_weight_kg::NUMBER(8, 2) AS max_weight_kg, + COALESCE(b.attributes:temperature::VARIANT, 'ambient'::VARIANT)::VARCHAR AS temperature_zone, + @sql.load_metadata(b) + FROM cdc_latest AS b +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/inventory/stg_po_receipts.sql b/benchmarks/duckshop/project/models/staging/inventory/stg_po_receipts.sql new file mode 100644 index 0000000000..c3dc9aeeb4 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/inventory/stg_po_receipts.sql @@ -0,0 +1,44 @@ +MODEL ( + description "One row per live purchase order receipt in warehouse local time", + columns ( + receipt_id (type VARCHAR, nullable false, audits [not_null, unique]), + po_id (type VARCHAR, audits [not_null]), + sku (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR), + received_qty (type INT), + damaged_qty (type INT), + sellable_qty (type INT), + received_at_local (type TIMESTAMP_NTZ), + discrepancy_codes (type ARRAY), + has_discrepancy (type BOOLEAN), + ), +); + +WITH src AS ( + SELECT * + FROM __source("inventory__po_receipts") +), + +@sql.cdc_latest(src, receipt_id) + +final AS ( + SELECT + r.receipt_id, + r.po_id, + @sql.normalise_sku(r.sku) AS sku, + UPPER(r.warehouse_code)::VARCHAR AS warehouse_code, + r.received_qty::INT AS received_qty, + COALESCE(r.damaged_qty, 0)::INT AS damaged_qty, + ( + r.received_qty - COALESCE(r.damaged_qty, 0) + )::INT AS sellable_qty, + r.received_at_local, + COALESCE(r.discrepancies:codes, ARRAY_CONSTRUCT())::ARRAY AS discrepancy_codes, + ( + ARRAY_SIZE(COALESCE(r.discrepancies:codes, ARRAY_CONSTRUCT())) > 0 + ) AS has_discrepancy, + @sql.load_metadata(r) + FROM cdc_latest AS r +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/inventory/stg_purchase_order_lines.sql b/benchmarks/duckshop/project/models/staging/inventory/stg_purchase_order_lines.sql new file mode 100644 index 0000000000..5a8a0f3e32 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/inventory/stg_purchase_order_lines.sql @@ -0,0 +1,41 @@ +MODEL ( + description "One row per purchase order line, flattened from the order's lines array", + columns ( + po_line_key (type VARCHAR, nullable false, audits [not_null, unique]), + po_id (type VARCHAR, audits [not_null]), + sku (type VARCHAR, audits [not_null]), + ordered_qty (type INT), + unit_cost_minor (type INT), + currency_code (type VARCHAR), + ), +); + +WITH purchase_orders AS ( + SELECT * + FROM __ref("stg_purchase_orders") +), + +lines AS ( + SELECT + p.po_id, + p.currency_code, + l.index::INT AS line_index, + l.value AS line + FROM purchase_orders AS p, LATERAL FLATTEN(input => p.lines) AS l +), + +final AS ( + SELECT + ( + l.po_id || '-' || LPAD(l.line_index::VARCHAR, 3, '0') + )::VARCHAR AS po_line_key, + l.po_id, + @sql.normalise_sku(l.line:sku::VARCHAR) AS sku, + l.line:qty::INT AS ordered_qty, + COALESCE(l.line:unit_cost_minor, l.line:unit_cost)::INT AS unit_cost_minor, + COALESCE(UPPER(l.line:currency::VARCHAR), l.currency_code)::VARCHAR AS currency_code + FROM lines AS l +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/inventory/stg_purchase_orders.sql b/benchmarks/duckshop/project/models/staging/inventory/stg_purchase_orders.sql new file mode 100644 index 0000000000..54f3dd5956 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/inventory/stg_purchase_orders.sql @@ -0,0 +1,39 @@ +MODEL ( + description "One row per live purchase order header", + columns ( + po_id (type VARCHAR, nullable false, audits [not_null, unique]), + supplier_code (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR, audits [not_null]), + currency_code (type VARCHAR), + po_status (type VARCHAR), + ordered_at (type TIMESTAMP_TZ), + expected_at (type TIMESTAMP_TZ), + received_at (type TIMESTAMP_TZ), + line_count (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("inventory__purchase_orders") +), + +@sql.cdc_latest(src, po_id) + +final AS ( + SELECT + p.po_id, + UPPER(p.supplier_code)::VARCHAR AS supplier_code, + UPPER(p.warehouse_code)::VARCHAR AS warehouse_code, + UPPER(p.currency_code)::VARCHAR AS currency_code, + LOWER(p.po_status)::VARCHAR AS po_status, + p.ordered_at, + p.expected_at, + p.received_at, + ARRAY_SIZE(COALESCE(p.lines, ARRAY_CONSTRUCT()))::INT AS line_count, + p.lines::ARRAY AS lines, + @sql.load_metadata(p) + FROM cdc_latest AS p +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/inventory/stg_stock_counts.sql b/benchmarks/duckshop/project/models/staging/inventory/stg_stock_counts.sql new file mode 100644 index 0000000000..cc93362aac --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/inventory/stg_stock_counts.sql @@ -0,0 +1,36 @@ +MODEL ( + description "One row per live cycle count in warehouse local time", + columns ( + count_id (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR, audits [not_null]), + counted_qty (type INT), + counted_at_local (type TIMESTAMP_NTZ, audits [not_null]), + counter_id (type VARCHAR), + is_blind_count (type BOOLEAN), + counter (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("inventory__stock_counts") +), + +@sql.cdc_latest(src, count_id) + +final AS ( + SELECT + c.count_id, + @sql.normalise_sku(c.sku) AS sku, + UPPER(c.warehouse_code)::VARCHAR AS warehouse_code, + GREATEST(c.counted_qty, 0)::INT AS counted_qty, + c.counted_at_local, + @sql.json_text(c.counter:id) AS counter_id, + COALESCE(c.counter:blind::BOOLEAN, FALSE) AS is_blind_count, + c.counter::OBJECT AS counter, + @sql.load_metadata(c) + FROM cdc_latest AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/inventory/stg_stock_movements.sql b/benchmarks/duckshop/project/models/staging/inventory/stg_stock_movements.sql new file mode 100644 index 0000000000..c279ea73b1 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/inventory/stg_stock_movements.sql @@ -0,0 +1,65 @@ +MODEL ( + description """ + One row per effective stock movement in warehouse local time. The append-only log records + corrections as VOID rows referencing the movement they cancel; voided and void rows are removed. + """, + columns ( + movement_id (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR, audits [not_null]), + qty_delta (type INT, audits [not_null]), + reason_code (type VARCHAR), + reference_type (type VARCHAR), + reference_id (type VARCHAR), + moved_at_local (type TIMESTAMP_NTZ, audits [not_null]), + operator_id (type VARCHAR), + ), +); + +WITH src AS ( + SELECT * + FROM __source("inventory__stock_movements") +), + +deduplicated AS ( + SELECT + m.movement_id, + m.sku, + m.warehouse_code, + m.qty_delta, + UPPER(m.reason_code) AS reason_code, + m.reference, + m.moved_at_local, + m.metadata + FROM src AS m + QUALIFY ROW_NUMBER() OVER (PARTITION BY m.movement_id ORDER BY m._loaded_at DESC) = 1 +), + +effective_ids AS ( + SELECT d.movement_id + FROM deduplicated AS d + WHERE d.reason_code <> 'VOID' + MINUS + SELECT SPLIT_PART(v.reference, ':', 2) AS movement_id + FROM deduplicated AS v + WHERE v.reason_code = 'VOID' +), + +final AS ( + SELECT + d.movement_id, + @sql.normalise_sku(d.sku) AS sku, + UPPER(d.warehouse_code)::VARCHAR AS warehouse_code, + d.qty_delta::INT AS qty_delta, + d.reason_code::VARCHAR AS reason_code, + UPPER(SPLIT_PART(d.reference, ':', 1))::VARCHAR AS reference_type, + NULLIF(SPLIT_PART(d.reference, ':', 2), '')::VARCHAR AS reference_id, + d.moved_at_local, + @sql.json_text(d.metadata:operator) AS operator_id + FROM deduplicated AS d + INNER JOIN effective_ids AS e + ON d.movement_id = e.movement_id +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/inventory/stg_stock_reservations.sql b/benchmarks/duckshop/project/models/staging/inventory/stg_stock_reservations.sql new file mode 100644 index 0000000000..e4b357e25e --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/inventory/stg_stock_reservations.sql @@ -0,0 +1,42 @@ +MODEL ( + description "One row per live stock reservation held for an order", + columns ( + reservation_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + sku (type VARCHAR, audits [not_null]), + warehouse_code (type VARCHAR), + qty (type INT), + reserved_at (type TIMESTAMP_TZ), + released_at (type TIMESTAMP_TZ), + is_open (type BOOLEAN), + allocation_rule (type VARCHAR), + is_backorder (type BOOLEAN), + ), +); + +WITH src AS ( + SELECT * + FROM __source("inventory__stock_reservations") +), + +@sql.cdc_latest(src, reservation_id) + +final AS ( + SELECT + r.reservation_id, + r.order_id, + @sql.normalise_sku(r.sku) AS sku, + UPPER(r.warehouse_code)::VARCHAR AS warehouse_code, + r.qty::INT AS qty, + r.reserved_at, + r.released_at, + ( + r.released_at IS NULL + ) AS is_open, + LOWER(@sql.json_text(r.allocation:rule))::VARCHAR AS allocation_rule, + COALESCE(r.allocation:backorder::BOOLEAN, FALSE) AS is_backorder, + @sql.load_metadata(r) + FROM cdc_latest AS r +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/inventory/stg_stock_transfers.sql b/benchmarks/duckshop/project/models/staging/inventory/stg_stock_transfers.sql new file mode 100644 index 0000000000..1417f21dc8 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/inventory/stg_stock_transfers.sql @@ -0,0 +1,43 @@ +MODEL ( + description "One row per live inter-warehouse stock transfer", + columns ( + transfer_id (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + from_warehouse_code (type VARCHAR), + to_warehouse_code (type VARCHAR), + qty (type INT), + shipped_at (type TIMESTAMP_TZ), + arrived_at (type TIMESTAMP_TZ), + transit_hours (type INT), + haulier_reference (type VARCHAR), + tracking (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("inventory__stock_transfers") +), + +@sql.cdc_latest(src, transfer_id) + +final AS ( + SELECT + t.transfer_id, + @sql.normalise_sku(t.sku) AS sku, + UPPER(t.from_warehouse_code)::VARCHAR AS from_warehouse_code, + UPPER(t.to_warehouse_code)::VARCHAR AS to_warehouse_code, + t.qty::INT AS qty, + t.shipped_at, + t.arrived_at, + DATEDIFF('hour', t.shipped_at, t.arrived_at)::INT AS transit_hours, + REGEXP_SUBSTR( + t.tracking:reference::VARCHAR, + '[A-Z]{2}[0-9]{6,}' + )::VARCHAR AS haulier_reference, + t.tracking::OBJECT AS tracking, + @sql.load_metadata(t) + FROM cdc_latest AS t +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/inventory/stg_warehouses.sql b/benchmarks/duckshop/project/models/staging/inventory/stg_warehouses.sql new file mode 100644 index 0000000000..f1cb59c348 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/inventory/stg_warehouses.sql @@ -0,0 +1,43 @@ +MODEL ( + description "One row per live warehouse with its IANA time zone and opening hours", + columns ( + warehouse_code (type VARCHAR, nullable false, audits [not_null, unique]), + warehouse_name (type VARCHAR), + country_code (type VARCHAR, audits [not_null]), + iana_timezone (type VARCHAR, audits [not_null]), + opens_local (type TIME), + closes_local (type TIME), + opening_minutes (type INT), + supports_same_day (type BOOLEAN), + supports_cold_chain (type BOOLEAN), + max_parcel_kg (type NUMBER(6, 2)), + ), +); + +WITH src AS ( + SELECT * + FROM __source("inventory__warehouses") +), + +@sql.cdc_latest(src, warehouse_code) + +final AS ( + SELECT + UPPER(w.warehouse_code)::VARCHAR AS warehouse_code, + w.warehouse_name, + UPPER(w.country_code)::VARCHAR AS country_code, + w.iana_timezone, + w.opens_local, + w.closes_local, + DATEDIFF('minute', w.opens_local, w.closes_local)::INT AS opening_minutes, + ARRAY_CONTAINS( + 'same_day'::VARIANT, + COALESCE(w.capabilities:services, ARRAY_CONSTRUCT())::ARRAY + )::BOOLEAN AS supports_same_day, + COALESCE(w.capabilities:cold_chain::BOOLEAN, FALSE) AS supports_cold_chain, + w.capabilities:limits.max_parcel_kg::NUMBER(6, 2) AS max_parcel_kg, + @sql.load_metadata(w) + FROM cdc_latest AS w +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/partners/stg_partner_listings.sql b/benchmarks/duckshop/project/models/staging/partners/stg_partner_listings.sql new file mode 100644 index 0000000000..0c798e9359 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/partners/stg_partner_listings.sql @@ -0,0 +1,36 @@ +MODEL ( + description "One row per live partner sales channel listing with its commercial terms", + columns ( + listing_id (type VARCHAR, nullable false, audits [not_null, unique]), + partner_code (type VARCHAR, audits [not_null]), + sku (type VARCHAR, audits [not_null]), + listing_status (type VARCHAR), + listed_at (type TIMESTAMP_TZ), + commission_pct (type NUMBER(5, 2)), + fulfilled_by_partner (type BOOLEAN), + terms (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("partners__partner_listings") +), + +@sql.cdc_latest(src, listing_id) + +final AS ( + SELECT + l.listing_id, + UPPER(l.partner_code)::VARCHAR AS partner_code, + @sql.normalise_sku(l.sku) AS sku, + LOWER(l.listing_status)::VARCHAR AS listing_status, + l.listed_at, + l.terms:commission_pct::NUMBER(5, 2) AS commission_pct, + COALESCE(l.terms:fulfilment::VARCHAR = 'partner', FALSE) AS fulfilled_by_partner, + l.terms::OBJECT AS terms, + @sql.load_metadata(l) + FROM cdc_latest AS l +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/partners/stg_partner_order_lines.sql b/benchmarks/duckshop/project/models/staging/partners/stg_partner_order_lines.sql new file mode 100644 index 0000000000..8b3171d14f --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/partners/stg_partner_order_lines.sql @@ -0,0 +1,97 @@ +MODEL ( + description """ + One row per partner order line from both payload versions, with amounts in minor units + """, + columns ( + partner_order_line_key (type VARCHAR, nullable false, audits [not_null, unique]), + partner_order_key (type VARCHAR, audits [not_null]), + payload_version (type INT), + sku (type VARCHAR), + quantity (type INT), + unit_amount_minor (type NUMBER(38, 0)), + ), +); + +WITH orders_v1 AS ( + SELECT * + FROM __ref("stg_partner_orders_v1") +), + +orders_v2 AS ( + SELECT * + FROM __ref("stg_partner_orders_v2") +), + +currencies AS ( + SELECT * + FROM __seed("currencies") +), + +orders_v1_exponent AS ( + SELECT + o.partner_order_key, + o.lines, + c.minor_unit_exponent + FROM orders_v1 AS o + INNER JOIN currencies AS c + ON o.currency_code = c.currency_code +), + +lines_v1 AS ( + SELECT + o.partner_order_key, + 1 AS payload_version, + l.index AS line_index, + l.value:sku::VARCHAR AS raw_sku, + l.value:qty::INT AS quantity, + ROUND(TO_DECIMAL(l.value:price::VARCHAR, 18, 3) * POWER(10, o.minor_unit_exponent) + )::NUMBER(38, 0) AS unit_amount_minor + FROM orders_v1_exponent AS o, LATERAL FLATTEN(input => o.lines) AS l +), + +lines_v2 AS ( + SELECT + o.partner_order_key, + 2 AS payload_version, + i.index AS line_index, + i.value:seller_sku::VARCHAR AS raw_sku, + i.value:quantity::INT AS quantity, + i.value:unit_price_minor::NUMBER(38, 0) AS unit_amount_minor + FROM orders_v2 AS o, LATERAL FLATTEN(input => o.items) AS i +), + +all_lines AS ( + SELECT + v1.partner_order_key, + v1.payload_version, + v1.line_index, + v1.raw_sku, + v1.quantity, + v1.unit_amount_minor + FROM lines_v1 AS v1 + UNION ALL + SELECT + v2.partner_order_key, + v2.payload_version, + v2.line_index, + v2.raw_sku, + v2.quantity, + v2.unit_amount_minor + FROM lines_v2 AS v2 +), + +final AS ( + SELECT + ( + a.partner_order_key || '#' || a.line_index::VARCHAR + )::VARCHAR AS partner_order_line_key, + a.partner_order_key, + a.payload_version::INT AS payload_version, + @sql.normalise_sku(a.raw_sku) AS sku, + a.quantity::INT AS quantity, + a.unit_amount_minor + FROM all_lines AS a +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/partners/stg_partner_orders.sql b/benchmarks/duckshop/project/models/staging/partners/stg_partner_orders.sql new file mode 100644 index 0000000000..a2cbaaea5c --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/partners/stg_partner_orders.sql @@ -0,0 +1,53 @@ +MODEL ( + description """ + One row per partner sales channel order message, deduplicated and kept as raw payload by schema + version + """, + columns ( + message_id (type VARCHAR, nullable false, audits [not_null, unique]), + partner_code (type VARCHAR, audits [not_null]), + schema_version (type INT, audits [accepted_values (values [1, 2])]), + received_at (type TIMESTAMP_NTZ, audits [not_null]), + external_order_id (type VARCHAR, audits [not_null]), + payload (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("partners__partner_orders") +), + +messages AS ( + SELECT + m.message_id, + UPPER(m.partner_code) AS partner_code, + m.schema_version::INT AS schema_version, + m.received_at, + COALESCE( + @sql.json_text(m.payload:orderId), + @sql.json_text(m.payload:order.id) + )::VARCHAR AS external_order_id, + m.payload::OBJECT AS payload + FROM src AS m + QUALIFY ROW_NUMBER() OVER (PARTITION BY m.message_id ORDER BY m._loaded_at DESC) = 1 +), + +final AS ( + SELECT + m.message_id, + m.partner_code::VARCHAR AS partner_code, + m.schema_version, + m.received_at, + m.external_order_id, + m.payload + FROM messages AS m + QUALIFY + ROW_NUMBER() OVER ( + PARTITION BY m.partner_code, m.external_order_id + ORDER BY m.received_at DESC, m.message_id DESC + ) = 1 +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/partners/stg_partner_orders_v1.sql b/benchmarks/duckshop/project/models/staging/partners/stg_partner_orders_v1.sql new file mode 100644 index 0000000000..c2a661c8ed --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/partners/stg_partner_orders_v1.sql @@ -0,0 +1,39 @@ +MODEL ( + description """ + One row per partner order received in payload version 1, where amounts are decimal strings + """, + columns ( + partner_order_key (type VARCHAR, nullable false, audits [not_null, unique]), + partner_code (type VARCHAR, audits [not_null]), + external_order_id (type VARCHAR, audits [not_null]), + buyer_email (type VARCHAR), + currency_code (type VARCHAR), + order_total (type NUMBER(18, 3)), + placed_at (type TIMESTAMP_TZ), + lines (type ARRAY), + ), +); + +WITH messages AS ( + SELECT * + FROM __ref("stg_partner_orders") +), + +final AS ( + SELECT + ( + m.partner_code || ':' || m.external_order_id + )::VARCHAR AS partner_order_key, + m.partner_code, + m.external_order_id, + @sql.normalise_email(m.payload:buyer.email::VARCHAR) AS buyer_email, + UPPER(m.payload:total.currency::VARCHAR)::VARCHAR AS currency_code, + TO_DECIMAL(m.payload:total.amount::VARCHAR, 18, 3)::NUMBER(18, 3) AS order_total, + TO_TIMESTAMP_TZ(m.payload:created::VARCHAR) AS placed_at, + COALESCE(m.payload:lines, ARRAY_CONSTRUCT())::ARRAY AS lines + FROM messages AS m + WHERE m.schema_version = 1 +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/partners/stg_partner_orders_v2.sql b/benchmarks/duckshop/project/models/staging/partners/stg_partner_orders_v2.sql new file mode 100644 index 0000000000..87a5faafb0 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/partners/stg_partner_orders_v2.sql @@ -0,0 +1,40 @@ +MODEL ( + description """ + One row per partner order received in payload version 2, where amounts are minor units and times + are epoch milliseconds + """, + columns ( + partner_order_key (type VARCHAR, nullable false, audits [not_null, unique]), + partner_code (type VARCHAR, audits [not_null]), + external_order_id (type VARCHAR, audits [not_null]), + buyer_email_hash (type VARCHAR), + currency_code (type VARCHAR), + order_total_minor (type NUMBER(38, 0)), + placed_at_utc (type TIMESTAMP_NTZ), + items (type ARRAY), + ), +); + +WITH messages AS ( + SELECT * + FROM __ref("stg_partner_orders") +), + +final AS ( + SELECT + ( + m.partner_code || ':' || m.external_order_id + )::VARCHAR AS partner_order_key, + m.partner_code, + m.external_order_id, + LOWER(m.payload:customer.email_hash::VARCHAR)::VARCHAR AS buyer_email_hash, + UPPER(m.payload:order.currency::VARCHAR)::VARCHAR AS currency_code, + m.payload:order.totals.gross_minor::NUMBER(38, 0) AS order_total_minor, + @sql.epoch_ms_to_utc(m.payload:order.placed_at_ms::NUMBER(38, 0)) AS placed_at_utc, + COALESCE(m.payload:items, ARRAY_CONSTRUCT())::ARRAY AS items + FROM messages AS m + WHERE m.schema_version = 2 +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/partners/stg_partner_settlements.sql b/benchmarks/duckshop/project/models/staging/partners/stg_partner_settlements.sql new file mode 100644 index 0000000000..629c4d74a1 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/partners/stg_partner_settlements.sql @@ -0,0 +1,49 @@ +MODEL ( + description "One row per live partner settlement statement with fee totals", + columns ( + settlement_id (type VARCHAR, nullable false, audits [not_null, unique]), + partner_code (type VARCHAR, audits [not_null]), + period_start (type DATE), + period_end (type DATE), + currency_code (type VARCHAR), + gross_sales_minor (type INT), + commission_minor (type INT), + other_fees_minor (type INT), + net_payable_minor (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("partners__partner_settlements") +), + +@sql.cdc_latest(src, settlement_id) + +final AS ( + SELECT + s.settlement_id, + UPPER(s.partner_code)::VARCHAR AS partner_code, + s.period_start, + s.period_end, + UPPER(s.currency_code)::VARCHAR AS currency_code, + s.statement:gross_sales_minor::INT AS gross_sales_minor, + s.statement:commission_minor::INT AS commission_minor, + COALESCE(REDUCE( + s.statement:fees::ARRAY, + 0, + (acc, f) -> acc + f:amount_minor::INT + ), 0)::INT AS other_fees_minor, + ( + s.statement:gross_sales_minor::INT - s.statement:commission_minor::INT + - COALESCE(REDUCE( + s.statement:fees::ARRAY, + 0, + (acc, f) -> acc + f:amount_minor::INT + ), 0) + )::INT AS net_payable_minor, + @sql.load_metadata(s) + FROM cdc_latest AS s +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/_sqlbuild/_macros/python/variants.py b/benchmarks/duckshop/project/models/staging/storefront/_sqlbuild/_macros/python/variants.py new file mode 100644 index 0000000000..0342f63992 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/_sqlbuild/_macros/python/variants.py @@ -0,0 +1,10 @@ +"""Generate typed extractions from Snowflake semi-structured columns.""" + + +def variant_fields(source: str, fields: dict[str, str]) -> str: + """Typed `:` path extraction for each `alias: "path TYPE"` entry, in the given order.""" + items = [] + for alias, spec in fields.items(): + path, _, sql_type = spec.rpartition(" ") + items.append(f"{source}:{path}::{sql_type} AS {alias}") + return ",\n ".join(items) diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_cart_items.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_cart_items.sql new file mode 100644 index 0000000000..8e30c5d140 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_cart_items.sql @@ -0,0 +1,47 @@ +MODEL ( + description "One row per item in a live cart, flattened from the cart contents", + columns ( + cart_item_key (type VARCHAR, nullable false, audits [not_null, unique]), + cart_id (type VARCHAR, audits [not_null]), + sku (type VARCHAR), + quantity (type INT), + unit_amount_minor (type INT), + added_at (type TIMESTAMP_TZ), + ), +); + +WITH carts AS ( + SELECT * + FROM __ref("stg_carts") +), + +items AS ( + SELECT + c.cart_id, + i.index::INT AS item_index, + i.value:sku::VARCHAR AS raw_sku, + i.value:qty AS raw_quantity, + i.value:unit_minor AS raw_unit_minor, + i.value:added_at::VARCHAR AS raw_added_at + FROM carts AS c, LATERAL FLATTEN(input => c.contents:items) AS i +), + +final AS ( + SELECT + ( + i.cart_id || '/' || i.item_index::VARCHAR + )::VARCHAR AS cart_item_key, + i.cart_id, + @sql.normalise_sku(i.raw_sku) AS sku, + IFF( + TYPEOF(i.raw_quantity) IN ('INTEGER', 'DECIMAL'), + i.raw_quantity::INT, + TRY_TO_NUMBER(i.raw_quantity::VARCHAR) + )::INT AS quantity, + TRY_TO_NUMBER(i.raw_unit_minor::VARCHAR)::INT AS unit_amount_minor, + TRY_TO_TIMESTAMP_TZ(i.raw_added_at) AS added_at + FROM items AS i +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_carts.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_carts.sql new file mode 100644 index 0000000000..22dfaefb08 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_carts.sql @@ -0,0 +1,39 @@ +MODEL ( + description "One row per live shopping cart", + columns ( + cart_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR), + anonymous_id (type VARCHAR), + storefront_code (type VARCHAR), + cart_status (type VARCHAR), + created_at (type TIMESTAMP_TZ), + updated_at (type TIMESTAMP_TZ), + line_count (type INT), + coupon_code (type VARCHAR), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__carts") +), + +@sql.cdc_latest(src, cart_id) + +final AS ( + SELECT + c.cart_id, + c.customer_id, + LOWER(c.anonymous_id)::VARCHAR AS anonymous_id, + UPPER(c.storefront_code)::VARCHAR AS storefront_code, + LOWER(c.cart_status)::VARCHAR AS cart_status, + c.created_at, + c.updated_at, + ARRAY_SIZE(COALESCE(c.contents:items, ARRAY_CONSTRUCT()))::INT AS line_count, + UPPER(@sql.json_text(c.contents:coupon))::VARCHAR AS coupon_code, + c.contents::OBJECT AS contents, + @sql.load_metadata(c) + FROM cdc_latest AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_customer_addresses.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_customer_addresses.sql new file mode 100644 index 0000000000..5667965d64 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_customer_addresses.sql @@ -0,0 +1,49 @@ +MODEL ( + description "One row per live customer address with a canonical postcode and coordinates", + columns ( + address_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR, audits [not_null]), + country_code (type VARCHAR, audits [not_null]), + postcode_canonical (type VARCHAR), + is_default (type BOOLEAN), + valid_from (type TIMESTAMP_TZ), + latitude (type NUMBER(9, 6)), + longitude (type NUMBER(9, 6)), + geo (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__customer_addresses") +), + +@sql.cdc_latest(src, address_id) + +final AS ( + SELECT + a.address_id, + a.customer_id, + TRIM(a.line1)::VARCHAR AS line1, + NULLIF(TRIM(a.line2), '')::VARCHAR AS line2, + INITCAP(TRIM(a.city))::VARCHAR AS city, + UPPER(a.country_code)::VARCHAR AS country_code, + CASE UPPER(a.country_code) + WHEN 'US' + THEN LPAD(REGEXP_SUBSTR(a.postcode, '[0-9]{1,5}'), 5, '0') + WHEN 'GB' + THEN REGEXP_REPLACE(UPPER(TRIM(a.postcode)), '^([A-Z0-9]+)\\s*([0-9][A-Z]{2})$', '\\1 \\2') + WHEN 'JP' + THEN REGEXP_REPLACE(a.postcode, '^([0-9]{3})-?([0-9]{4})$', '\\1-\\2') + ELSE UPPER(REGEXP_REPLACE(a.postcode, '\\s+', '')) + END::VARCHAR AS postcode_canonical, + COALESCE(a.is_default, FALSE) AS is_default, + a.valid_from, + ROUND(a.geo:lat::FLOAT, 6)::NUMBER(9, 6) AS latitude, + ROUND(a.geo:lon::FLOAT, 6)::NUMBER(9, 6) AS longitude, + a.geo::OBJECT AS geo, + @sql.load_metadata(a) + FROM cdc_latest AS a +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_customer_consents.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_customer_consents.sql new file mode 100644 index 0000000000..f94d060918 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_customer_consents.sql @@ -0,0 +1,114 @@ +MODEL ( + description """ + One row per live consent record. Customers created before the consent service store their + consents inside the profile payload; those are carried as legacy rows when no explicit record of + the same type exists. + """, + columns ( + consent_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR, audits [not_null]), + consent_type (type VARCHAR, audits [not_null]), + is_granted (type BOOLEAN), + captured_at (type TIMESTAMP_TZ), + capture_surface (type VARCHAR), + policy_version (type VARCHAR), + is_legacy (type BOOLEAN), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__customer_consents") +), + +customers AS ( + SELECT * + FROM __source("storefront__customers") +), + +@sql.cdc_latest(src, consent_id) + +explicit_consents AS ( + SELECT + c.consent_id, + c.customer_id, + UPPER(REGEXP_REPLACE(c.consent_type, '[^A-Za-z]+', '_')) AS consent_type, + COALESCE(c.granted, FALSE) AS is_granted, + c.captured_at, + COALESCE(c.source_context:surface::VARCHAR, 'unknown') AS capture_surface, + REGEXP_SUBSTR( + c.source_context:policy_url::VARCHAR, + 'v([0-9]+(\\.[0-9]+)?)', + 1, + 1, + 'e', + 1 + ) AS policy_version, + FALSE AS is_legacy + FROM cdc_latest AS c +), legacy_profiles AS ( + SELECT + p.customer_id, + p.created_at, + p.profile + FROM customers AS p + QUALIFY + ROW_NUMBER() OVER (PARTITION BY p.customer_id ORDER BY p._seq DESC) = 1 + AND p._op <> 'D' +), legacy_consents AS ( + SELECT + ( + 'legacy:' || l.customer_id || ':' || UPPER(f.value::VARCHAR) + ) AS consent_id, + l.customer_id, + UPPER(REGEXP_REPLACE( + f.value::VARCHAR, + '[^A-Za-z]+', + '_' + )) AS consent_type, + TRUE AS is_granted, + l.created_at AS captured_at, + 'profile' AS capture_surface, + NULL AS policy_version, + TRUE AS is_legacy + FROM legacy_profiles AS l, LATERAL FLATTEN(input => l.profile:consents) AS f +), legacy_only AS ( + SELECT + lc.consent_id, + lc.customer_id, + lc.consent_type, + lc.is_granted, + lc.captured_at, + lc.capture_surface, + lc.policy_version, + lc.is_legacy + FROM legacy_consents AS lc + LEFT JOIN explicit_consents AS ec + ON lc.customer_id = ec.customer_id + AND lc.consent_type = ec.consent_type + WHERE ec.consent_id IS NULL +), final AS ( + SELECT + e.consent_id, + e.customer_id, + e.consent_type::VARCHAR AS consent_type, + e.is_granted, + e.captured_at, + e.capture_surface::VARCHAR AS capture_surface, + e.policy_version::VARCHAR AS policy_version, + e.is_legacy + FROM explicit_consents AS e + UNION ALL + SELECT + l.consent_id::VARCHAR AS consent_id, + l.customer_id, + l.consent_type::VARCHAR AS consent_type, + l.is_granted, + l.captured_at, + l.capture_surface::VARCHAR AS capture_surface, + l.policy_version::VARCHAR AS policy_version, + l.is_legacy + FROM legacy_only AS l +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_customers.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_customers.sql new file mode 100644 index 0000000000..9711dc4f6c --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_customers.sql @@ -0,0 +1,47 @@ +MODEL ( + description """ + One row per live customer account with normalised contact details and profile preferences + """, + columns ( + customer_id (type VARCHAR, nullable false, audits [not_null, unique]), + email_normalised (type VARCHAR), + phone_normalised (type VARCHAR), + first_name (type VARCHAR), + last_name (type VARCHAR), + region_code (type VARCHAR, audits [not_null]), + storefront_code (type VARCHAR), + created_at (type TIMESTAMP_TZ, audits [not_null]), + preferred_language (type VARCHAR), + marketing_opt_in (type BOOLEAN), + consent_types (type ARRAY), + birth_year (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__customers") +), + +@sql.cdc_latest(src, customer_id) + +final AS ( + SELECT + c.customer_id, + @sql.normalise_email(c.email) AS email_normalised, + @sql.normalise_phone(c.phone) AS phone_normalised, + INITCAP(TRIM(SPLIT_PART(c.full_name, ' ', 1)))::VARCHAR AS first_name, + INITCAP(TRIM(REGEXP_SUBSTR(c.full_name, '\\s(.+)$', 1, 1, 'e', 1)))::VARCHAR AS last_name, + UPPER(c.region_code)::VARCHAR AS region_code, + UPPER(c.storefront_code)::VARCHAR AS storefront_code, + c.created_at, + COALESCE(c.profile:preferences.language::VARCHAR, 'en')::VARCHAR AS preferred_language, + COALESCE(c.profile:preferences.marketing_opt_in::BOOLEAN, FALSE) AS marketing_opt_in, + COALESCE(c.profile:consents, ARRAY_CONSTRUCT())::ARRAY AS consent_types, + TRY_TO_NUMBER(c.profile:birth_year::VARCHAR)::INT AS birth_year, + c.profile:preferences::OBJECT AS preferences, + @sql.load_metadata(c) + FROM cdc_latest AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_gift_card_transactions.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_gift_card_transactions.sql new file mode 100644 index 0000000000..c2d75f0c17 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_gift_card_transactions.sql @@ -0,0 +1,44 @@ +MODEL ( + description "One row per live gift card redemption, top-up or expiry", + columns ( + transaction_id (type VARCHAR, nullable false, audits [not_null, unique]), + gift_card_id (type VARCHAR, audits [not_null]), + order_id (type VARCHAR), + transaction_type (type VARCHAR, audits [ + accepted_values (values ["REDEEM", "TOP_UP", "EXPIRE", "REVERSAL"]), + ]), + signed_amount_minor (type INT, audits [not_null]), + occurred_at (type TIMESTAMP_TZ), + channel (type VARCHAR), + terminal_id (type VARCHAR), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__gift_card_transactions") +), + +@sql.cdc_latest(src, transaction_id) + +final AS ( + SELECT + t.transaction_id, + t.gift_card_id, + t.order_id, + UPPER(REPLACE(t.transaction_type, '-', '_'))::VARCHAR AS transaction_type, + IFF( + UPPER(t.transaction_type) IN ('REDEEM', 'EXPIRE'), + -ABS(t.amount_minor), + ABS(t.amount_minor) + )::INT AS signed_amount_minor, + t.occurred_at, + LOWER( + COALESCE(t.context:channel::VARCHAR, IFF(t.order_id IS NULL, 'store', 'web')) + )::VARCHAR AS channel, + @sql.json_text(t.context:terminal.id) AS terminal_id, + @sql.load_metadata(t) + FROM cdc_latest AS t +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_gift_cards.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_gift_cards.sql new file mode 100644 index 0000000000..5884470032 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_gift_cards.sql @@ -0,0 +1,40 @@ +MODEL ( + description "One row per issued gift card", + columns ( + gift_card_id (type VARCHAR, nullable false, audits [not_null, unique]), + code_last4 (type VARCHAR), + currency_code (type VARCHAR, audits [not_null]), + initial_amount_minor (type INT, audits [not_null]), + issued_at (type TIMESTAMP_TZ), + expires_on (type DATE), + purchaser_customer_id (type VARCHAR), + is_promotional (type BOOLEAN), + design_code (type VARCHAR), + details (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__gift_cards") +), + +@sql.cdc_latest(src, gift_card_id) + +final AS ( + SELECT + g.gift_card_id, + RIGHT(REGEXP_REPLACE(g.code_masked, '[^0-9A-Z]', ''), 4)::VARCHAR AS code_last4, + UPPER(g.currency_code)::VARCHAR AS currency_code, + g.initial_amount_minor, + g.issued_at, + g.expires_on, + g.purchaser_customer_id, + COALESCE(g.details:promotional::BOOLEAN, FALSE) AS is_promotional, + COALESCE(@sql.json_text(g.details:design), 'standard')::VARCHAR AS design_code, + g.details::OBJECT AS details, + @sql.load_metadata(g) + FROM cdc_latest AS g +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_loyalty_accounts.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_loyalty_accounts.sql new file mode 100644 index 0000000000..d410a2b147 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_loyalty_accounts.sql @@ -0,0 +1,37 @@ +MODEL ( + description "One row per live loyalty account with its tier history as an array", + columns ( + loyalty_account_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR, audits [not_null, unique]), + tier_code (type VARCHAR), + points_balance (type INT), + enrolled_at (type TIMESTAMP_TZ), + tier_change_count (type INT), + previous_tier_code (type VARCHAR), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__loyalty_accounts") +), + +@sql.cdc_latest(src, loyalty_account_id) + +final AS ( + SELECT + l.loyalty_account_id, + l.customer_id, + UPPER(l.tier_code)::VARCHAR AS tier_code, + GREATEST(COALESCE(l.points_balance, 0), 0)::INT AS points_balance, + l.enrolled_at, + ARRAY_SIZE(COALESCE(l.tier_history, ARRAY_CONSTRUCT()))::INT AS tier_change_count, + UPPER( + GET(GET(l.tier_history, ARRAY_SIZE(l.tier_history) - 2), 'tier')::VARCHAR + )::VARCHAR AS previous_tier_code, + l.tier_history::ARRAY AS tier_history, + @sql.load_metadata(l) + FROM cdc_latest AS l +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_loyalty_ledger.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_loyalty_ledger.sql new file mode 100644 index 0000000000..0402441c25 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_loyalty_ledger.sql @@ -0,0 +1,38 @@ +MODEL ( + description "One row per live loyalty points posting", + columns ( + entry_id (type VARCHAR, nullable false, audits [not_null, unique]), + loyalty_account_id (type VARCHAR, audits [not_null]), + points_delta (type INT, audits [not_null]), + reason_code (type VARCHAR), + order_id (type VARCHAR), + posted_at (type TIMESTAMP_TZ, audits [not_null]), + campaign_ref (type VARCHAR), + expires_on (type DATE), + metadata (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__loyalty_ledger") +), + +@sql.cdc_latest(src, entry_id) + +final AS ( + SELECT + e.entry_id, + e.loyalty_account_id, + e.points_delta, + UPPER(e.reason_code)::VARCHAR AS reason_code, + NULLIF(e.order_id, '')::VARCHAR AS order_id, + e.posted_at, + @sql.json_text(e.metadata:campaign) AS campaign_ref, + TRY_TO_DATE(e.metadata:expires_on::VARCHAR)::DATE AS expires_on, + e.metadata::OBJECT AS metadata, + @sql.load_metadata(e) + FROM cdc_latest AS e +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_order_line_discounts.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_order_line_discounts.sql new file mode 100644 index 0000000000..f924dd1119 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_order_line_discounts.sql @@ -0,0 +1,42 @@ +MODEL ( + description "One row per discount applied to an order line, flattened from the discounts array", + columns ( + order_line_discount_key (type VARCHAR, nullable false, audits [not_null, unique]), + order_line_id (type VARCHAR, audits [not_null]), + order_id (type VARCHAR), + discount_code (type VARCHAR), + discount_kind (type VARCHAR), + discount_amount_minor (type INT), + ), +); + +WITH order_lines AS ( + SELECT * + FROM __ref("stg_order_lines") +), + +flattened AS ( + SELECT + l.order_line_id, + l.order_id, + d.index::INT AS discount_index, + d.value AS discount + FROM order_lines AS l, LATERAL FLATTEN(input => l.discounts) AS d + WHERE IS_ARRAY(l.discounts) +), + +final AS ( + SELECT + ( + f.order_line_id || '#' || f.discount_index::VARCHAR + )::VARCHAR AS order_line_discount_key, + f.order_line_id, + f.order_id, + UPPER(@sql.json_text(f.discount:code))::VARCHAR AS discount_code, + LOWER(COALESCE(@sql.json_text(f.discount:kind), 'manual'))::VARCHAR AS discount_kind, + TRY_TO_NUMBER(f.discount:amount_minor::VARCHAR)::INT AS discount_amount_minor + FROM flattened AS f +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_order_lines.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_order_lines.sql new file mode 100644 index 0000000000..462e2210e1 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_order_lines.sql @@ -0,0 +1,41 @@ +MODEL ( + description "One row per live order line with canonical SKU and gross amounts in minor units", + columns ( + order_line_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + line_no (type INT, audits [not_null]), + sku (type VARCHAR, audits [not_null]), + quantity (type INT), + unit_amount_minor (type INT), + gross_amount_minor (type INT), + tax_amount_minor (type INT), + discount_count (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__order_lines") +), + +@sql.cdc_latest(src, order_line_id) + +final AS ( + SELECT + l.order_line_id, + l.order_id, + l.line_no, + @sql.normalise_sku(l.sku) AS sku, + GREATEST(l.quantity, 0)::INT AS quantity, + l.unit_amount_minor, + ( + GREATEST(l.quantity, 0) * l.unit_amount_minor + )::INT AS gross_amount_minor, + COALESCE(l.tax_amount_minor, 0)::INT AS tax_amount_minor, + IFF(IS_ARRAY(l.discounts), ARRAY_SIZE(l.discounts), 0)::INT AS discount_count, + IFF(IS_ARRAY(l.discounts), l.discounts, ARRAY_CONSTRUCT())::ARRAY AS discounts, + @sql.load_metadata(l) + FROM cdc_latest AS l +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_order_status_events.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_order_status_events.sql new file mode 100644 index 0000000000..89111e6a93 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_order_status_events.sql @@ -0,0 +1,39 @@ +MODEL ( + description "One row per live order status transition with the acting system or person", + columns ( + event_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + from_status (type VARCHAR), + to_status (type VARCHAR, audits [not_null]), + changed_at (type TIMESTAMP_TZ, audits [not_null]), + actor_type (type VARCHAR), + actor_id (type VARCHAR), + actor (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__order_status_events") +), + +@sql.cdc_latest(src, event_id) + +final AS ( + SELECT + e.event_id, + e.order_id, + LOWER(e.from_status)::VARCHAR AS from_status, + LOWER(e.to_status)::VARCHAR AS to_status, + e.changed_at, + COALESCE( + e.actor:type::VARCHAR, + IFF(e.actor IS NULL, 'system', 'unknown') + )::VARCHAR AS actor_type, + @sql.json_text(e.actor:id) AS actor_id, + e.actor::OBJECT AS actor, + @sql.load_metadata(e) + FROM cdc_latest AS e +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_orders.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_orders.sql new file mode 100644 index 0000000000..d14172053e --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_orders.sql @@ -0,0 +1,50 @@ +MODEL ( + description "One row per live storefront order, latest version from the change log", + columns ( + order_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR, audits [not_null]), + storefront_code (type VARCHAR, audits [not_null]), + currency_code (type VARCHAR, audits [not_null]), + placed_at (type TIMESTAMP_TZ, audits [not_null]), + order_status (type VARCHAR), + sales_channel (type VARCHAR), + is_gift (type BOOLEAN), + utm_source (type VARCHAR), + utm_medium (type VARCHAR), + utm_campaign (type VARCHAR), + device_type (type VARCHAR), + item_count_hint (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__orders") +), + +@sql.cdc_latest(src, order_id) + +final AS ( + SELECT + o.order_id, + o.customer_id, + UPPER(o.storefront_code)::VARCHAR AS storefront_code, + UPPER(o.currency_code)::VARCHAR AS currency_code, + o.placed_at, + UPPER(o.status)::VARCHAR AS order_status, + LOWER(o.attributes:channel::VARCHAR)::VARCHAR AS sales_channel, + @py.variant_fields("o.attributes", { + "is_gift": "gift.is_gift BOOLEAN", + "utm_source": "utm.source VARCHAR", + "utm_medium": "utm.medium VARCHAR", + "utm_campaign": "utm.campaign VARCHAR", + "device_type": "device.type VARCHAR", + }), + ARRAY_SIZE(o.attributes:line_skus)::INT AS item_count_hint, + o.shipping_address_id, + o.attributes::OBJECT AS attributes, + @sql.load_metadata(o) + FROM cdc_latest AS o +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_payments.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_payments.sql new file mode 100644 index 0000000000..7372a5d73f --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_payments.sql @@ -0,0 +1,58 @@ +MODEL ( + description """ + One row per live payment attempt with the gateway outcome extracted from its response + """, + columns ( + payment_id (type VARCHAR, nullable false, audits [not_null, unique]), + order_id (type VARCHAR, audits [not_null]), + payment_method (type VARCHAR), + amount_minor (type INT, audits [not_null]), + currency_code (type VARCHAR), + authorised_at (type TIMESTAMP_TZ), + captured_at (type TIMESTAMP_TZ), + card_bin (type VARCHAR), + gateway_status (type VARCHAR), + gateway_error_code (type VARCHAR), + risk_score (type NUMBER(5, 2)), + three_ds_outcome (type VARCHAR), + gateway_reference (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__payments") +), + +@sql.cdc_latest(src, payment_id) + +final AS ( + SELECT + p.payment_id, + p.order_id, + LOWER(REGEXP_REPLACE(p.method, '[\\s-]+', '_'))::VARCHAR AS payment_method, + p.amount_minor, + UPPER(p.currency_code)::VARCHAR AS currency_code, + p.authorised_at, + p.captured_at, + IFF(RLIKE(p.card_bin, '^[0-9]{6,8}$'), p.card_bin, NULL)::VARCHAR AS card_bin, + LOWER( + COALESCE(p.gateway_response:status.code, p.gateway_response:status)::VARCHAR + )::VARCHAR AS gateway_status, + REGEXP_SUBSTR( + p.gateway_response:status.message::VARCHAR, + '\\b([A-Z]{2}[0-9]{3})\\b', + 1, + 1, + 'e', + 1 + )::VARCHAR AS gateway_error_code, + TRY_TO_DECIMAL(p.gateway_response:risk.score::VARCHAR, 5, 2)::NUMBER(5, 2) AS risk_score, + UPPER(p.gateway_response:three_ds.outcome::VARCHAR)::VARCHAR AS three_ds_outcome, + TRY_TO_NUMBER(p.gateway_response:reference::VARCHAR)::INT AS gateway_reference, + p.gateway_response::OBJECT AS gateway_response, + @sql.load_metadata(p) + FROM cdc_latest AS p +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_promotion_redemptions.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_promotion_redemptions.sql new file mode 100644 index 0000000000..dd2d3f2c24 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_promotion_redemptions.sql @@ -0,0 +1,34 @@ +MODEL ( + description "One row per live promotion redemption on an order", + columns ( + redemption_id (type VARCHAR, nullable false, audits [not_null, unique]), + promo_code (type VARCHAR, audits [not_null]), + order_id (type VARCHAR, audits [not_null]), + discount_minor (type INT), + redeemed_at (type TIMESTAMP_TZ), + entry_method (type VARCHAR), + redemption_context (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__promotion_redemptions") +), + +@sql.cdc_latest(src, redemption_id) + +final AS ( + SELECT + r.redemption_id, + UPPER(TRIM(r.promo_code))::VARCHAR AS promo_code, + r.order_id, + ABS(COALESCE(r.discount_minor, 0))::INT AS discount_minor, + r.redeemed_at, + LOWER(COALESCE(r.context:entry::VARCHAR, 'code'))::VARCHAR AS entry_method, + r.context::OBJECT AS redemption_context, + @sql.load_metadata(r) + FROM cdc_latest AS r +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_promotions.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_promotions.sql new file mode 100644 index 0000000000..e67978d861 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_promotions.sql @@ -0,0 +1,44 @@ +MODEL ( + description "One row per live promotion with eligibility rules and discount tiers extracted", + columns ( + promo_code (type VARCHAR, nullable false, audits [not_null, unique]), + promo_name (type VARCHAR), + starts_at (type TIMESTAMP_TZ), + ends_at (type TIMESTAMP_TZ), + is_stackable (type BOOLEAN), + min_order_minor (type INT), + eligible_regions (type ARRAY), + tier_count (type INT), + max_discount_pct (type NUMBER(5, 2)), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__promotions") +), + +@sql.cdc_latest(src, promo_code) + +final AS ( + SELECT + UPPER(TRIM(p.promo_code))::VARCHAR AS promo_code, + p.promo_name, + p.starts_at, + p.ends_at, + COALESCE(p.is_stackable, FALSE) AS is_stackable, + COALESCE(p.rules:eligibility.min_order_minor::INT, 0)::INT AS min_order_minor, + COALESCE(p.rules:eligibility.regions, ARRAY_CONSTRUCT())::ARRAY AS eligible_regions, + ARRAY_SIZE(COALESCE(p.rules:tiers, ARRAY_CONSTRUCT()))::INT AS tier_count, + ARRAY_MAX( + TRANSFORM( + p.rules:tiers::ARRAY, + t VARIANT -> t:pct + ) + )::NUMBER(5, 2) AS max_discount_pct, + p.rules::OBJECT AS rules, + @sql.load_metadata(p) + FROM cdc_latest AS p +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_refunds.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_refunds.sql new file mode 100644 index 0000000000..9da99645ca --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_refunds.sql @@ -0,0 +1,46 @@ +MODEL ( + description "One row per live refund with a coarse reason category parsed from free text", + columns ( + refund_id (type VARCHAR, nullable false, audits [not_null, unique]), + payment_id (type VARCHAR, audits [not_null]), + amount_minor (type INT, audits [not_null]), + reason_text (type VARCHAR), + reason_category (type VARCHAR), + refunded_at (type TIMESTAMP_TZ), + initiated_by (type VARCHAR), + refunded_line_count (type INT), + refunded_items (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__refunds") +), + +@sql.cdc_latest(src, refund_id) + +final AS ( + SELECT + r.refund_id, + r.payment_id, + ABS(r.amount_minor)::INT AS amount_minor, + TRIM(r.reason_text)::VARCHAR AS reason_text, + CASE + WHEN REGEXP_LIKE(r.reason_text, '.*(damag|broken|defect).*', 'i') + THEN 'damaged' + WHEN REGEXP_LIKE(r.reason_text, '.*(late|delay|never arrived).*', 'i') + THEN 'delivery' + WHEN REGEXP_LIKE(r.reason_text, '.*(fraud|chargeback).*', 'i') + THEN 'fraud' + ELSE 'other' + END::VARCHAR AS reason_category, + r.refunded_at, + LOWER(COALESCE(r.initiated_by, 'agent'))::VARCHAR AS initiated_by, + ARRAY_SIZE(COALESCE(r.items, ARRAY_CONSTRUCT()))::INT AS refunded_line_count, + COALESCE(r.items, ARRAY_CONSTRUCT())::ARRAY AS refunded_items, + @sql.load_metadata(r) + FROM cdc_latest AS r +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_reviews.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_reviews.sql new file mode 100644 index 0000000000..c96f783726 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_reviews.sql @@ -0,0 +1,40 @@ +MODEL ( + description "One row per live product review with moderation outcome", + columns ( + review_id (type VARCHAR, nullable false, audits [not_null, unique]), + sku (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + rating (type INT, audits [accepted_values (values [1, 2, 3, 4, 5])]), + title (type VARCHAR), + body_length (type INT), + submitted_at (type TIMESTAMP_TZ), + moderation_status (type VARCHAR), + flagged_terms (type ARRAY), + mentions_url (type BOOLEAN), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__reviews") +), + +@sql.cdc_latest(src, review_id) + +final AS ( + SELECT + r.review_id, + @sql.normalise_sku(r.sku) AS sku, + r.customer_id, + LEAST(GREATEST(r.rating, 1), 5)::INT AS rating, + TRIM(r.title)::VARCHAR AS title, + LENGTH(TRIM(r.body))::INT AS body_length, + r.submitted_at, + LOWER(COALESCE(r.moderation:status::VARCHAR, 'pending'))::VARCHAR AS moderation_status, + COALESCE(r.moderation:flags, ARRAY_CONSTRUCT())::ARRAY AS flagged_terms, + REGEXP_LIKE(r.body, '.*https?://.*', 'is') AS mentions_url, + @sql.load_metadata(r) + FROM cdc_latest AS r +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_store_credit_ledger.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_store_credit_ledger.sql new file mode 100644 index 0000000000..63fe21e56d --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_store_credit_ledger.sql @@ -0,0 +1,36 @@ +MODEL ( + description "One row per live store credit posting", + columns ( + entry_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR, audits [not_null]), + currency_code (type VARCHAR, audits [not_null]), + amount_minor (type INT, audits [not_null]), + reason_code (type VARCHAR), + posted_at (type TIMESTAMP_TZ), + source_order_id (type VARCHAR), + expires_on (type DATE), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__store_credit_ledger") +), + +@sql.cdc_latest(src, entry_id) + +final AS ( + SELECT + e.entry_id, + e.customer_id, + UPPER(e.currency_code)::VARCHAR AS currency_code, + e.amount_minor, + UPPER(REGEXP_REPLACE(e.reason_code, '[^A-Za-z0-9]+', '_'))::VARCHAR AS reason_code, + e.posted_at, + @sql.json_text(e.context:order_id) AS source_order_id, + TRY_TO_DATE(e.context:expires_on::VARCHAR) AS expires_on, + @sql.load_metadata(e) + FROM cdc_latest AS e +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_subscription_events.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_subscription_events.sql new file mode 100644 index 0000000000..22cba6f84c --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_subscription_events.sql @@ -0,0 +1,34 @@ +MODEL ( + description "One row per live subscription lifecycle event", + columns ( + event_id (type VARCHAR, nullable false, audits [not_null, unique]), + subscription_id (type VARCHAR, audits [not_null]), + event_type (type VARCHAR, audits [not_null]), + occurred_at (type TIMESTAMP_TZ, audits [not_null]), + pause_days (type INT), + skip_reason (type VARCHAR), + payload (type VARIANT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__subscription_events") +), + +@sql.cdc_latest(src, event_id) + +final AS ( + SELECT + e.event_id, + e.subscription_id, + UPPER(e.event_type)::VARCHAR AS event_type, + e.occurred_at, + TRY_TO_NUMBER(e.payload:pause_days::VARCHAR)::INT AS pause_days, + @sql.json_text(e.payload:skip_reason) AS skip_reason, + e.payload::VARIANT AS payload, + @sql.load_metadata(e) + FROM cdc_latest AS e +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_subscriptions.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_subscriptions.sql new file mode 100644 index 0000000000..d15fdf39db --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_subscriptions.sql @@ -0,0 +1,46 @@ +MODEL ( + description "One row per live replenishment subscription", + columns ( + subscription_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR, audits [not_null]), + plan_code (type VARCHAR), + sku (type VARCHAR), + interval_days (type INT), + subscription_status (type VARCHAR), + started_at (type TIMESTAMP_TZ), + cancelled_at (type TIMESTAMP_TZ), + billing_amount_minor (type INT), + billing_currency_code (type VARCHAR), + next_bill_on (type DATE), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__subscriptions") +), + +@sql.cdc_latest(src, subscription_id) + +final AS ( + SELECT + s.subscription_id, + s.customer_id, + UPPER(s.plan_code)::VARCHAR AS plan_code, + @sql.normalise_sku(s.sku) AS sku, + COALESCE(s.interval_days, 30)::INT AS interval_days, + LOWER(s.subscription_status)::VARCHAR AS subscription_status, + s.started_at, + s.cancelled_at, + s.billing:amount_minor::INT AS billing_amount_minor, + UPPER(s.billing:currency::VARCHAR)::VARCHAR AS billing_currency_code, + TRY_TO_DATE( + s.billing:next_bill_on::VARCHAR, + 'YYYY-MM-DD' + ) AS next_bill_on, + s.billing::OBJECT AS billing, + @sql.load_metadata(s) + FROM cdc_latest AS s +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_wishlist_items.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_wishlist_items.sql new file mode 100644 index 0000000000..aa448a0fe5 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_wishlist_items.sql @@ -0,0 +1,44 @@ +MODEL ( + description "One row per SKU saved on a wishlist, flattened from the wishlist items array", + columns ( + wishlist_item_key (type VARCHAR, nullable false, audits [not_null, unique]), + wishlist_id (type VARCHAR, audits [not_null]), + customer_id (type VARCHAR), + sku (type VARCHAR, audits [not_null]), + added_at (type TIMESTAMP_TZ), + item_position (type INT), + ), +); + +WITH wishlists AS ( + SELECT * + FROM __ref("stg_wishlists") +), + +items AS ( + SELECT + w.wishlist_id, + w.customer_id, + f.index::INT AS item_position, + f.value:sku::VARCHAR AS raw_sku, + TRY_TO_TIMESTAMP_TZ(f.value:added_at::VARCHAR) AS added_at + FROM wishlists AS w, LATERAL FLATTEN(input => w.items) AS f + WHERE IS_OBJECT(f.value) +), + +final AS ( + SELECT + ( + i.wishlist_id || ':' || i.item_position::VARCHAR + )::VARCHAR AS wishlist_item_key, + i.wishlist_id, + i.customer_id, + @sql.normalise_sku(i.raw_sku) AS sku, + i.added_at, + i.item_position + FROM items AS i + WHERE i.raw_sku IS NOT NULL +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/storefront/stg_wishlists.sql b/benchmarks/duckshop/project/models/staging/storefront/stg_wishlists.sql new file mode 100644 index 0000000000..ffd454d6c2 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/storefront/stg_wishlists.sql @@ -0,0 +1,31 @@ +MODEL ( + description "One row per live wishlist", + columns ( + wishlist_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_id (type VARCHAR, audits [not_null]), + created_at (type TIMESTAMP_TZ), + is_public (type BOOLEAN), + item_count (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("storefront__wishlists") +), + +@sql.cdc_latest(src, wishlist_id) + +final AS ( + SELECT + w.wishlist_id, + w.customer_id, + w.created_at, + COALESCE(w.is_public, FALSE) AS is_public, + ARRAY_SIZE(COALESCE(w.items, ARRAY_CONSTRUCT()))::INT AS item_count, + w.items::ARRAY AS items, + @sql.load_metadata(w) + FROM cdc_latest AS w +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/support/stg_agent_shifts.sql b/benchmarks/duckshop/project/models/staging/support/stg_agent_shifts.sql new file mode 100644 index 0000000000..cdf2f380c0 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/support/stg_agent_shifts.sql @@ -0,0 +1,45 @@ +MODEL ( + description "One row per live agent shift in the agent's local time with paid break minutes", + columns ( + shift_id (type VARCHAR, nullable false, audits [not_null, unique]), + agent_id (type VARCHAR, audits [not_null]), + shift_start_local (type TIMESTAMP_NTZ), + shift_end_local (type TIMESTAMP_NTZ), + iana_timezone (type VARCHAR), + scheduled_minutes (type INT), + break_minutes (type INT), + breaks (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("support__agent_shifts") +), + +@sql.cdc_latest(src, shift_id) + +breaks AS ( + SELECT + s.shift_id, + SUM(COALESCE(b.value:minutes::INT, 0)) AS break_minutes + FROM cdc_latest AS s, LATERAL FLATTEN(input => s.breaks, outer => TRUE) AS b + GROUP BY + s.shift_id +), final AS ( + SELECT + s.shift_id, + s.agent_id, + s.shift_start_local, + s.shift_end_local, + s.iana_timezone, + DATEDIFF('minute', s.shift_start_local, s.shift_end_local)::INT AS scheduled_minutes, + COALESCE(b.break_minutes, 0)::INT AS break_minutes, + COALESCE(s.breaks, ARRAY_CONSTRUCT())::ARRAY AS breaks, + @sql.load_metadata(s) + FROM cdc_latest AS s + LEFT JOIN breaks AS b + ON s.shift_id = b.shift_id +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/support/stg_agents.sql b/benchmarks/duckshop/project/models/staging/support/stg_agents.sql new file mode 100644 index 0000000000..9bb7987cec --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/support/stg_agents.sql @@ -0,0 +1,39 @@ +MODEL ( + description "One row per live support agent with skills as an array", + columns ( + agent_id (type VARCHAR, nullable false, audits [not_null, unique]), + display_name (type VARCHAR), + email (type VARCHAR), + team_code (type VARCHAR), + iana_timezone (type VARCHAR), + hired_on (type DATE), + skills (type ARRAY), + speaks_german (type BOOLEAN), + ), +); + +WITH src AS ( + SELECT * + FROM __source("support__agents") +), + +@sql.cdc_latest(src, agent_id) + +final AS ( + SELECT + a.agent_id, + a.display_name, + @sql.normalise_email(a.email) AS email, + UPPER(a.team_code)::VARCHAR AS team_code, + a.iana_timezone, + a.hired_on, + COALESCE(a.skills, ARRAY_CONSTRUCT())::ARRAY AS skills, + ARRAY_CONTAINS( + 'lang:de'::VARIANT, + COALESCE(a.skills, ARRAY_CONSTRUCT()) + )::BOOLEAN AS speaks_german, + @sql.load_metadata(a) + FROM cdc_latest AS a +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/support/stg_call_records.sql b/benchmarks/duckshop/project/models/staging/support/stg_call_records.sql new file mode 100644 index 0000000000..d9a6bd10ae --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/support/stg_call_records.sql @@ -0,0 +1,45 @@ +MODEL ( + description "One row per live phone call with its IVR menu path", + columns ( + call_id (type VARCHAR, nullable false, audits [not_null, unique]), + ticket_id (type VARCHAR), + agent_id (type VARCHAR), + caller_phone (type VARCHAR), + caller_country_prefix (type VARCHAR), + started_at (type TIMESTAMP_TZ), + duration_seconds (type INT), + ivr_path (type VARCHAR), + ivr_depth (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("support__call_records") +), + +@sql.cdc_latest(src, call_id) + +final AS ( + SELECT + c.call_id, + c.ticket_id, + c.agent_id, + @sql.normalise_phone(c.phone_number) AS caller_phone, + REGEXP_SUBSTR( + @sql.normalise_phone(c.phone_number), + '^\\+(1|44|49|33|31|61|81|965)', + 1, + 1, + 'e', + 1 + )::VARCHAR AS caller_country_prefix, + c.started_at, + c.duration_seconds::INT AS duration_seconds, + ARRAY_TO_STRING(c.ivr_path::ARRAY, ' > ')::VARCHAR AS ivr_path, + ARRAY_SIZE(c.ivr_path)::INT AS ivr_depth, + @sql.load_metadata(c) + FROM cdc_latest AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/support/stg_chat_messages.sql b/benchmarks/duckshop/project/models/staging/support/stg_chat_messages.sql new file mode 100644 index 0000000000..3a2beffddd --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/support/stg_chat_messages.sql @@ -0,0 +1,48 @@ +MODEL ( + description "One row per chat message, flattened from the transcript with masked contact details", + columns ( + chat_message_key (type VARCHAR, nullable false, audits [not_null, unique]), + chat_id (type VARCHAR, audits [not_null]), + message_index (type INT), + sender_role (type VARCHAR), + sent_at (type TIMESTAMP_TZ), + message_text_masked (type VARCHAR), + message_length (type INT), + ), +); + +WITH chats AS ( + SELECT * + FROM __ref("stg_chat_transcripts") +), + +messages AS ( + SELECT + c.chat_id, + m.index::INT AS message_index, + LOWER(m.value:role::VARCHAR) AS sender_role, + TRY_TO_TIMESTAMP_TZ(m.value:ts::VARCHAR) AS sent_at, + m.value:text::VARCHAR AS message_text + FROM chats AS c, LATERAL FLATTEN(input => c.messages) AS m +), + +final AS ( + SELECT + ( + m.chat_id || '#' || m.message_index::VARCHAR + )::VARCHAR AS chat_message_key, + m.chat_id, + m.message_index, + m.sender_role::VARCHAR AS sender_role, + m.sent_at, + REGEXP_REPLACE( + REGEXP_REPLACE(m.message_text, '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+', ''), + '\\+?[0-9][0-9 ()-]{6,}[0-9]', + '' + )::VARCHAR AS message_text_masked, + LENGTH(m.message_text)::INT AS message_length + FROM messages AS m +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/support/stg_chat_transcripts.sql b/benchmarks/duckshop/project/models/staging/support/stg_chat_transcripts.sql new file mode 100644 index 0000000000..dc80b46b99 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/support/stg_chat_transcripts.sql @@ -0,0 +1,32 @@ +MODEL ( + description "One row per live chat session", + columns ( + chat_id (type VARCHAR, nullable false, audits [not_null, unique]), + ticket_id (type VARCHAR), + started_at (type TIMESTAMP_TZ), + ended_at (type TIMESTAMP_TZ), + message_count (type INT), + messages (type ARRAY), + ), +); + +WITH src AS ( + SELECT * + FROM __source("support__chat_transcripts") +), + +@sql.cdc_latest(src, chat_id) + +final AS ( + SELECT + c.chat_id, + c.ticket_id, + c.started_at, + c.ended_at, + ARRAY_SIZE(c.messages)::INT AS message_count, + c.messages::ARRAY AS messages, + @sql.load_metadata(c) + FROM cdc_latest AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/support/stg_csat_responses.sql b/benchmarks/duckshop/project/models/staging/support/stg_csat_responses.sql new file mode 100644 index 0000000000..559eb1eab3 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/support/stg_csat_responses.sql @@ -0,0 +1,38 @@ +MODEL ( + description "One row per live customer satisfaction response", + columns ( + response_id (type VARCHAR, nullable false, audits [not_null, unique]), + ticket_id (type VARCHAR, audits [not_null]), + score (type INT, audits [accepted_values (values [1, 2, 3, 4, 5])]), + is_satisfied (type BOOLEAN), + comment_text (type VARCHAR), + responded_at (type TIMESTAMP_TZ), + effort_score (type INT), + answers (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("support__csat_responses") +), + +@sql.cdc_latest(src, response_id) + +final AS ( + SELECT + r.response_id, + r.ticket_id, + r.score::INT AS score, + ( + r.score >= 4 + ) AS is_satisfied, + NULLIF(TRIM(r.comment), '')::VARCHAR AS comment_text, + r.responded_at, + TRY_TO_NUMBER(r.answers:effort::VARCHAR)::INT AS effort_score, + COALESCE(r.answers, OBJECT_CONSTRUCT())::OBJECT AS answers, + @sql.load_metadata(r) + FROM cdc_latest AS r +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/support/stg_sla_policies.sql b/benchmarks/duckshop/project/models/staging/support/stg_sla_policies.sql new file mode 100644 index 0000000000..49fc28c2a4 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/support/stg_sla_policies.sql @@ -0,0 +1,36 @@ +MODEL ( + description "One row per live support SLA policy for a priority and channel", + columns ( + policy_id (type VARCHAR, nullable false, audits [not_null, unique]), + priority (type INT, audits [not_null]), + channel (type VARCHAR, audits [not_null]), + first_response_minutes (type INT), + resolution_minutes (type INT), + business_hours_only (type BOOLEAN), + calendar_region_code (type VARCHAR), + ), +); + +WITH src AS ( + SELECT * + FROM __source("support__sla_policies") +), + +@sql.cdc_latest(src, policy_id) + +final AS ( + SELECT + p.policy_id, + p.priority::INT AS priority, + LOWER(p.channel)::VARCHAR AS channel, + p.first_response_minutes::INT AS first_response_minutes, + ( + p.resolution_hours * 60 + )::INT AS resolution_minutes, + COALESCE(p.business_hours_only, TRUE) AS business_hours_only, + UPPER(COALESCE(p.calendar:region::VARCHAR, 'EU'))::VARCHAR AS calendar_region_code, + @sql.load_metadata(p) + FROM cdc_latest AS p +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/support/stg_ticket_comments.sql b/benchmarks/duckshop/project/models/staging/support/stg_ticket_comments.sql new file mode 100644 index 0000000000..aeb842458b --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/support/stg_ticket_comments.sql @@ -0,0 +1,38 @@ +MODEL ( + description "One row per live ticket comment", + columns ( + comment_id (type VARCHAR, nullable false, audits [not_null, unique]), + ticket_id (type VARCHAR, audits [not_null]), + author_type (type VARCHAR, audits [accepted_values (values ["agent", "customer", "system"])]), + author_id (type VARCHAR), + is_public (type BOOLEAN), + created_at (type TIMESTAMP_TZ, audits [not_null]), + body_word_count (type INT), + contains_order_reference (type BOOLEAN), + attachment_count (type INT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("support__ticket_comments") +), + +@sql.cdc_latest(src, comment_id) + +final AS ( + SELECT + c.comment_id, + c.ticket_id, + LOWER(c.author_type)::VARCHAR AS author_type, + c.author_id, + COALESCE(c.is_public, TRUE) AS is_public, + c.created_at, + REGEXP_COUNT(TRIM(c.body), '\\S+')::INT AS body_word_count, + RLIKE(c.body, '.*DS-O-[0-9]+.*', 's')::BOOLEAN AS contains_order_reference, + ARRAY_SIZE(COALESCE(c.attachments, ARRAY_CONSTRUCT()))::INT AS attachment_count, + @sql.load_metadata(c) + FROM cdc_latest AS c +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/support/stg_ticket_tags.sql b/benchmarks/duckshop/project/models/staging/support/stg_ticket_tags.sql new file mode 100644 index 0000000000..f0f82ec4ba --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/support/stg_ticket_tags.sql @@ -0,0 +1,61 @@ +MODEL ( + description """ + One row per distinct tag on a ticket, from the tag array and the comma-separated labels custom + field + """, + columns ( + ticket_tag_key (type VARCHAR, nullable false, audits [not_null, unique]), + ticket_id (type VARCHAR, audits [not_null]), + tag (type VARCHAR, audits [not_null]), + ), +); + +WITH tickets AS ( + SELECT * + FROM __ref("stg_tickets") +), + +array_tags AS ( + SELECT + t.ticket_id, + f.value::VARCHAR AS raw_tag + FROM tickets AS t, LATERAL FLATTEN(input => t.tag_list) AS f + WHERE f.value IS NOT NULL +), + +label_tags AS ( + SELECT + t.ticket_id, + s.value::VARCHAR AS raw_tag + FROM tickets AS t, LATERAL FLATTEN(input => SPLIT( + t.custom_fields:labels::VARCHAR, + ',' + )) AS s + WHERE t.custom_fields:labels IS NOT NULL +), + +combined AS ( + SELECT + a.ticket_id, + LOWER(REGEXP_REPLACE(TRIM(a.raw_tag), '[\\s_]+', '-')) AS tag + FROM array_tags AS a + UNION DISTINCT + SELECT + l.ticket_id, + LOWER(REGEXP_REPLACE(TRIM(l.raw_tag), '[\\s_]+', '-')) AS tag + FROM label_tags AS l +), + +final AS ( + SELECT + ( + c.ticket_id || ':' || c.tag + )::VARCHAR AS ticket_tag_key, + c.ticket_id, + c.tag::VARCHAR AS tag + FROM combined AS c + WHERE c.tag <> '' +) + +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/models/staging/support/stg_tickets.sql b/benchmarks/duckshop/project/models/staging/support/stg_tickets.sql new file mode 100644 index 0000000000..31735c4237 --- /dev/null +++ b/benchmarks/duckshop/project/models/staging/support/stg_tickets.sql @@ -0,0 +1,51 @@ +MODEL ( + description """ + One row per live support ticket; created_at is a session-time-zone dependent TIMESTAMP_LTZ in + the source, an expected divergence across engines + """, + tags [known_divergence], + columns ( + ticket_id (type VARCHAR, nullable false, audits [not_null, unique]), + customer_email (type VARCHAR), + channel (type VARCHAR), + priority (type INT, audits [accepted_values (values [1, 2, 3, 4])]), + ticket_status (type VARCHAR), + created_at (type TIMESTAMP_LTZ, audits [not_null]), + solved_at (type TIMESTAMP_TZ), + order_id (type VARCHAR), + contact_reason (type VARCHAR), + tag_list (type ARRAY), + is_vip (type BOOLEAN), + custom_fields (type OBJECT), + ), +); + +WITH src AS ( + SELECT * + FROM __source("support__tickets") +), + +@sql.cdc_latest(src, ticket_id) + +final AS ( + SELECT + t.ticket_id, + @sql.normalise_email(t.customer_email) AS customer_email, + LOWER(t.channel)::VARCHAR AS channel, + LEAST(GREATEST(COALESCE(t.priority, 3), 1), 4)::INT AS priority, + LOWER(t.ticket_status)::VARCHAR AS ticket_status, + t.created_at::TIMESTAMP_LTZ AS created_at, + t.solved_at, + COALESCE(@sql.json_text(t.custom_fields:order_id), REGEXP_SUBSTR( + t.custom_fields:order_ref::VARCHAR, + 'DS-O-[0-9]+' + ))::VARCHAR AS order_id, + LOWER(@sql.json_text(GET_PATH(t.custom_fields, 'contact_reason')))::VARCHAR AS contact_reason, + COALESCE(t.tags, ARRAY_CONSTRUCT())::ARRAY AS tag_list, + ARRAY_CONTAINS('vip'::VARIANT, COALESCE(t.tags, ARRAY_CONSTRUCT()))::BOOLEAN AS is_vip, + COALESCE(t.custom_fields, OBJECT_CONSTRUCT())::OBJECT AS custom_fields, + @sql.load_metadata(t) + FROM cdc_latest AS t +) +SELECT * +FROM final diff --git a/benchmarks/duckshop/project/python/audits/fact_quality.py b/benchmarks/duckshop/project/python/audits/fact_quality.py new file mode 100644 index 0000000000..be75105e44 --- /dev/null +++ b/benchmarks/duckshop/project/python/audits/fact_quality.py @@ -0,0 +1,18 @@ +"""Audit cases shared by every fact table.""" + +from sqlbuild.audits import AuditCase, AuditSeverity, audit_factory + + +@audit_factory +def fact_table_quality() -> list[AuditCase]: + """A fact table must never publish empty.""" + return [ + AuditCase( + name="fact_table_has_rows", + definition="has_rows", + arguments={"minimum_rows": 1}, + severity=AuditSeverity.WARN, + description="A fact table built with no rows usually means an upstream load failed.", + meta={"check": "volume"}, + ) + ] diff --git a/benchmarks/duckshop/project/python/loaders/_catalogue.py b/benchmarks/duckshop/project/python/loaders/_catalogue.py new file mode 100644 index 0000000000..82762c8b7d --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_catalogue.py @@ -0,0 +1,508 @@ +"""Catalogue change logs: brands, categories, products, variants, prices, media and suppliers.""" + +from __future__ import annotations + +from datetime import date, datetime, timedelta + +from python.loaders._core import ChangeLog, at, chance, emit, in_zone, rng +from python.loaders._world import CURRENCY_EXPONENT, World, local_price + +Rows = list[dict[str, object]] + +BRANDS = [ + # code as sent, name, country, private label, metadata + ( + "MALLARD", + "Mallard & Co.", + "gb", + True, + {"founded": "est. 1998", "sustainability": {"rating": "a"}}, + ), + ( + "featherholm", + " Featherholm Textiles ", + "PT", + False, + {"founded": 2004, "sustainability": {"rating": None}}, + ), + ( + "DABBLE", + "Dabbleworth Decoys", + "us", + False, + {"founded": "Founded in the year 1987", "awards": []}, + ), + ("GENERIC", "Unbranded pond goods", "CN", None, {"sustainability": {"rating": " "}}), +] + +CATEGORIES = [ + # id, parent, slug as sent, display name, sort order, translations + ("C01", None, "Ducks", "Ducks", 1, {"en": "Ducks", "de": "Enten"}), + ("C02", "C01", "Rubber Ducks", "Rubber ducks", 1, {"en": "Rubber ducks", "de": "Badeenten"}), + ("C03", "C02", "bath_ducks", "Bath ducks", 1, None), + ("C04", "C02", "collectible-series", "Collectible series", 2, {"en": "Collectible series"}), + ("C05", None, "decoys", "Decoys", 2, {"en": "Decoys", "de": "Lockenten", "fr": "Appelants"}), + ("C06", "C05", "carved decoys", "Carved decoys", 1, None), + ("C07", "C05", "painted-decoys", "Painted decoys", None, None), + ("C08", "C08", "textiles", "Textiles", 3, {"en": "Textiles"}), + ("C09", "C08", "towels", "Towels", 1, None), + ("C10", "C08", "cushions", "Cushions", 2, None), + ("C11", None, "home", "Home", 4, None), + ("C12", "C11", "mugs", "Mugs", 1, {"de": "Tassen"}), + ("C13", None, "pond-and-bath", "Pond & bath", 5, None), + ("C14", "C13", "pond-care", "Pond care", 1, None), + ("C15", "C99", "orphaned", "Orphaned category", 9, None), +] + +SPECS = { + "DUCK": lambda r: { + "material": r.choice(["vinyl", "natural rubber", "Vinyl"]), + "Height (cm)": f"{r.randint(6, 14)} cm", + "colour_family": r.choice(["Yellow", "white", "multi"]), + "dishwasher_safe": r.choice(["yes", "no", True, False]), + }, + "DECOY": lambda r: { + "material": r.choice(["cedar", "cork", "basswood"]), + "width": f"{r.randint(250, 420)} mm", + "height_cm": round(r.uniform(9.0, 21.0), 7), + "carver": None, + }, + "TOWEL": lambda r: { + "material": "cotton", + "width_cm": r.randint(50, 100), + "height": f"{r.choice([0.9, 1.4, 1.5])} m", + "gsm": r.choice([450, 600, "600"]), + }, + "CUSHION": lambda r: {"material": "linen", "width": "45cm", "height": "45 cm", "filling": None}, + "MUG": lambda r: { + "material": "stoneware", + "capacity": r.choice([300, 350.0, 400.125]), + "dishwasher_safe": "TRUE", + "colour_family": "Natural", + }, + "POND": lambda r: {"material": "recycled plastic", "capacity_ml": 2500, "uv_stable": True}, +} + +SUPPLIERS = [ + # code, name, country, payment terms, contacts + ( + "VINYLWORKS", + "Vinyl Moulding Works", + "CN", + {"net_days": 60}, + [ + {"email": "Orders@Vinyl-Moulding.example.com", "is_primary": True}, + {"email": "accounts@vinyl-moulding.example.com"}, + ], + ), + ( + "COASTMILL", + "Coastal Textile Mill", + "PT", + {"label": "Net 45, 2% 10"}, + [ + {"email": "sales@coastal-mill.example.com"}, + ], + ), + ( + "CARVECOOP", + "Hand Carving Cooperative", + "US", + {"label": "NET30", "early_discount_pct": 1.5}, + [], + ), + ( + "CERAMICS", + "Ceramics Workshop", + "GB", + {"net_days": 30}, + [ + {"email": "not an email"}, + {"email": "hello@ceramics.example.com", "is_primary": True}, + ], + ), + ("pondsupply", "Pond Supply Wholesale", "NL", {}, None), +] +SUPPLIER_FOR_FAMILY = { + "DUCK": ["VINYLWORKS"], + "DECOY": ["CARVECOOP"], + "TOWEL": ["COASTMILL"], + "CUSHION": ["COASTMILL"], + "MUG": ["CERAMICS"], + "POND": ["pondsupply", "VINYLWORKS"], +} +SUPPLIER_CURRENCY = { + "VINYLWORKS": "USD", + "COASTMILL": "EUR", + "CARVECOOP": "USD", + "CERAMICS": "GBP", + "pondsupply": "EUR", +} +CATALOGUE_TZ = "Europe/London" + + +def _at(day: date, hour: int = 9) -> datetime: + return at(day, hour) + + +def brands(world: World) -> Rows: + r = rng("catalogue__brands") + log = ChangeLog("catalogue__brands") + for code, name, country, private, metadata in BRANDS: + row = { + "brand_code": code, + "brand_name": name, + "country_code": country, + "is_private_label": private, + "metadata": metadata, + } + versions = [(_at(date(2025, 11, 3)), row)] + if code == "DABBLE": + versions.append( + ( + _at(date(2026, 2, 10)), + {**row, "metadata": {**metadata, "awards": ["Best decoy 2025"]}}, + ) + ) + emit(log, r, versions) + retired = { + "brand_code": "OLDPOND", + "brand_name": "Retired pond brand", + "country_code": "GB", + "is_private_label": False, + "metadata": None, + } + emit(log, r, [(_at(date(2025, 11, 3)), retired)], delete_at=_at(date(2026, 1, 15))) + return log.rows() + + +def categories(world: World) -> Rows: + r = rng("catalogue__categories") + log = ChangeLog("catalogue__categories") + for category_id, parent, slug, name, order, translations in CATEGORIES: + row = { + "category_id": category_id, + "parent_category_id": parent, + "slug": slug, + "display_name": name, + "sort_order": order, + "translations": translations, + } + versions = [(_at(date(2025, 10, 1)), row)] + if category_id == "C04": + versions.append((_at(date(2026, 3, 2)), {**row, "sort_order": 0})) + emit(log, r, versions, duplicate_share=0.05) + return log.rows() + + +def products(world: World) -> Rows: + r = rng("catalogue__products") + log = ChangeLog("catalogue__products") + for product in world.products: + row = { + "product_id": product.product_id, + "title": product.title, + "category_path": product.category_path, + "brand_code": product.brand_code, + "launched_on": product.launched_on, + "is_active": None if chance(r, 0.1) else True, + "specs": SPECS[product.family](r), + } + versions = [(_at(product.launched_on - timedelta(days=14)), row)] + if chance(r, 0.15): + versions.append( + ( + _at(date(2026, 2, 1) + timedelta(days=r.randint(0, 50)), 11), + {**row, "is_active": False}, + ) + ) + emit(log, r, versions) + draft = { + "product_id": "P9999", + "title": "Draft listing created by mistake", + "category_path": "ducks", + "brand_code": "MALLARD", + "launched_on": None, + "is_active": False, + "specs": {}, + } + emit(log, r, [(_at(date(2026, 1, 20)), draft)], delete_at=_at(date(2026, 1, 21))) + return log.rows() + + +def product_variants(world: World) -> Rows: + r = rng("catalogue__product_variants") + log = ChangeLog("catalogue__product_variants") + seen_barcodes: list[str] = [] + for index, variant in enumerate(world.variants): + barcode = f"50{index:010d}" + barcode = barcode + str(_ean_check(barcode)) + codes: list[object] = [barcode] + roll = r.random() + if roll < 0.15: + codes.append(f"0{index:010d}5") # a 12-digit UPC-A + elif roll < 0.2: + codes.append(f"9638{index:04d}") # an 8-digit EAN-8 + elif roll < 0.25 and seen_barcodes: + codes.append(seen_barcodes[-1]) # a barcode shared by two variants + elif roll < 0.28: + codes.append("N/A") + seen_barcodes.append(barcode) + attributes: dict[str, object] = {"pack_size": r.choice([1, 1, 1, "6", 12])} + if variant.family in ("MUG", "DECOY"): + attributes["handling"] = {"fragile": True} + row = { + "sku": variant.raw_sku, + "product_id": variant.product_id, + "colour": variant.colour, + "size_label": variant.size, + "weight_grams": variant.weight_grams, + "barcodes": codes, + "attributes": attributes, + } + versions = [(_at(date(2025, 11, 1)), row)] + if chance(r, 0.1): + versions.append( + (_at(date(2026, 2, 20)), {**row, "weight_grams": variant.weight_grams + 15}) + ) + emit(log, r, versions) + return log.rows() + + +def _ean_check(digits: str) -> int: + total = sum(int(d) * (3 if i % 2 else 1) for i, d in enumerate(digits)) + return (10 - total % 10) % 10 + + +def price_lists(world: World) -> Rows: + r = rng("catalogue__price_lists") + log = ChangeLog("catalogue__price_lists") + count = 0 + for variant in world.variants: + for currency in ("GBP", "EUR", "USD", "JPY", "AUD", "KWD"): + zone = { + "GBP": "Europe/London", + "EUR": "Europe/Berlin", + "USD": "America/New_York", + "JPY": "Asia/Tokyo", + "AUD": "Australia/Sydney", + "KWD": "Asia/Kuwait", + }[currency] + base = local_price(variant.base_usd, currency) + count += 1 + opened = at(date(2025, 12, 1)) + first: dict[str, object] = { + "price_list_id": f"PL{count:06d}", + "sku": variant.raw_sku if chance(r, 0.8) else variant.sku.lower(), + "currency_code": currency if chance(r, 0.9) else currency.lower(), + "list_amount_minor": base, + "valid_from": in_zone(opened, zone), + "valid_to": None, + "conditions": None, + } + markdown = chance(r, 0.2) + if markdown: + change_at = at(date(2026, 2, 1) + timedelta(days=r.randint(0, 40))) + first["valid_to"] = in_zone(change_at, zone) + emit(log, r, [(opened, first)], duplicate_share=0.01) + count += 1 + new_amount = ( + int(base * 0.75) if CURRENCY_EXPONENT[currency] else int(base * 0.75 / 10) * 10 + ) + second = { + **first, + "price_list_id": f"PL{count:06d}", + "list_amount_minor": new_amount, + "valid_from": in_zone(change_at, zone), + "valid_to": in_zone(datetime(9999, 12, 31), "UTC") if chance(r, 0.5) else None, + } + emit(log, r, [(change_at, second)], duplicate_share=0.01) + else: + versions = [(opened, first)] + if chance(r, 0.05): + versions.append( + ( + at(date(2026, 1, 10)), + {**first, "conditions": {"min_qty": 1, "note": None}}, + ) + ) + emit(log, r, versions, duplicate_share=0.01) + if currency == "GBP" and chance(r, 0.08): + count += 1 + app_price = { + **first, + "price_list_id": f"PL{count:06d}", + "list_amount_minor": int(base * 0.9), + "valid_from": in_zone(at(date(2026, 1, 5)), zone), + "valid_to": None, + "conditions": {"channels": ["app"], "min_qty": "2"}, + } + emit(log, r, [(at(date(2026, 1, 5)), app_price)]) + return log.rows() + + +def product_media(world: World) -> Rows: + r = rng("catalogue__product_media") + log = ChangeLog("catalogue__product_media") + count = 0 + for product in world.products: + for position in range(1, r.randint(1, 4) + 1): + count += 1 + kind = "image" if position < 3 else r.choice(["video", "Image", "model3d"]) + ext = { + "image": r.choice(["jpg", "PNG", "webp"]), + "Image": "jpeg", + "video": "mp4", + "model3d": "glb", + }[kind] + scheme = "http" if chance(r, 0.2) else "https" + url = f"{scheme}://media.example.com/{product.product_id.lower()}/{position}.{ext}" + if chance(r, 0.1): + url += "?v=2" + dimensions: dict[str, object] | None = { + "width": r.choice([800, 1200, 1600]), + "height": r.choice([800, 1200, 0]), + } + if kind in ("video", "model3d"): + dimensions = None + row = { + "media_id": f"M{count:05d}", + "product_id": product.product_id, + "url": url, + "media_type": kind, + "position": position, + "dimensions": dimensions, + } + emit(log, r, [(_at(product.launched_on), row)]) + return log.rows() + + +def bundles(world: World) -> Rows: + r = rng("catalogue__bundles") + log = ChangeLog("catalogue__bundles") + for code, title, active_from, active_to, pricing in _bundle_definitions(): + row = { + "bundle_sku": code, + "title": title, + "active_from": active_from, + "active_to": active_to, + "pricing": pricing, + } + emit(log, r, [(_at(active_from - timedelta(days=7)), row)]) + return log.rows() + + +def _bundle_definitions(): + return [ + ( + "BUNDLE-PADDLING-SIX", + "Paddling Six mini-duck set", + date(2025, 12, 1), + None, + {"mode": "percent_off", "percent_off": 12.5}, + ), + ( + "bundle-bath-time", + "Bath time gift set", + date(2026, 1, 1), + date(2026, 3, 31), + {"mode": "FIXED", "amount_minor": 2999, "currency": "gbp"}, + ), + ("BUNDLE-DECOY-PAIR", "Decoy pair", date(2026, 2, 1), None, {"mode": "cheapest_free"}), + ("BUNDLE-POND-STARTER", "Pond starter kit", date(2026, 3, 1), None, None), + ] + + +def bundle_components(world: World) -> Rows: + r = rng("catalogue__bundle_components") + log = ChangeLog("catalogue__bundle_components") + by_family: dict[str, list[str]] = {} + for variant in world.variants: + by_family.setdefault(variant.family, []).append(variant.raw_sku) + plan = { + "BUNDLE-PADDLING-SIX": [("DUCK", 6), ("DUCK", 1)], + "bundle-bath-time": [("DUCK", 2), ("TOWEL", 1), ("MUG", 1)], + "BUNDLE-DECOY-PAIR": [("DECOY", 1), ("DECOY", 1)], + "BUNDLE-POND-STARTER": [("POND", 1), ("DUCK", None)], + } + count = 0 + for bundle, parts in plan.items(): + for index, (family, quantity) in enumerate(parts): + count += 1 + options: dict[str, object] | None = None + if index == len(parts) - 1 and len(parts) > 2: + options = {"optional": True, "substitutes": by_family[family][:2]} + row = { + "component_id": f"BC{count:03d}", + "bundle_sku": bundle, + "component_sku": r.choice(by_family[family]), + "quantity": quantity, + "options": options, + } + emit(log, r, [(_at(date(2025, 11, 25)), row)]) + return log.rows() + + +def suppliers(world: World) -> Rows: + r = rng("catalogue__suppliers") + log = ChangeLog("catalogue__suppliers") + for code, name, country, terms, contacts in SUPPLIERS: + row = { + "supplier_code": code, + "supplier_name": name, + "country_code": country, + "payment_terms": terms, + "contacts": contacts, + } + versions = [(_at(date(2025, 9, 1)), row)] + if code == "COASTMILL": + versions.append((_at(date(2026, 1, 12)), {**row, "payment_terms": {"label": "net 60"}})) + emit(log, r, versions) + return log.rows() + + +def supplier_costs(world: World) -> Rows: + r = rng("catalogue__supplier_costs") + log = ChangeLog("catalogue__supplier_costs") + count = 0 + for variant in world.variants: + for supplier in SUPPLIER_FOR_FAMILY[variant.family]: + currency = SUPPLIER_CURRENCY[supplier] + exponent = CURRENCY_EXPONENT[currency] + unit = int(round(variant.base_usd * r.uniform(0.25, 0.45) * 10**exponent)) + effective = [date(2026, 1, 1)] + if chance(r, 0.25): + effective.append(date(2026, 3, 1) + timedelta(days=r.randint(0, 29))) + for start in effective: + count += 1 + breaks: list[dict[str, object]] | None = None + if chance(r, 0.3): + breaks = [ + {"min_qty": 100, "unit_cost_minor": int(unit * 0.95)}, + {"min_qty": 500, "unit_cost_minor": int(unit * 0.9)}, + ] + row = { + "cost_id": f"SC{count:05d}", + "supplier_code": supplier, + "sku": variant.raw_sku, + "currency_code": currency, + "unit_cost_minor": unit if start.month == 1 else int(unit * 1.04), + "effective_from": start, + "volume_breaks": breaks, + } + emit(log, r, [(_at(start - timedelta(days=10)), row)]) + return log.rows() + + +TABLES = { + "catalogue__brands": brands, + "catalogue__categories": categories, + "catalogue__products": products, + "catalogue__product_variants": product_variants, + "catalogue__price_lists": price_lists, + "catalogue__product_media": product_media, + "catalogue__bundles": bundles, + "catalogue__bundle_components": bundle_components, + "catalogue__suppliers": suppliers, + "catalogue__supplier_costs": supplier_costs, +} diff --git a/benchmarks/duckshop/project/python/loaders/_clickstream.py b/benchmarks/duckshop/project/python/loaders/_clickstream.py new file mode 100644 index 0000000000..f2c168861d --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_clickstream.py @@ -0,0 +1,532 @@ +"""Clickstream event logs: web and app events, searches, email events and experiment assignments.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from datetime import datetime, timedelta +from functools import lru_cache + +from python.loaders._core import EventLog, at, chance, in_zone, naive_utc, rng +from python.loaders._world import ( + CAMPAIGNS, + STOREFRONTS, + World, + build_world, + clickstream_days, + utm_for, +) + +Rows = list[dict[str, object]] +SITE = "https://www.duckshop.example.com" +BROWSER_UA = [ + "Waddle/5.0 (Featheros 3; Webfoot) Paddle/124.0", + "Waddle/5.0 (Pondphone; Quill OS 17_4) Paddle/124.0 Mobile", + "Waddle/5.0 (Nestbook; Featheros 14_4) Waddle/17.4", +] +BOT_UA = ["Waddle/5.0 (compatible; pondcrawler/2.1)", "Waddle/5.0 HeadlessPaddle/1.0"] +REFERRERS = [ + None, + None, + "https://search.example.com/?q=rubber+duck", + "https://social.example.net/post/9", + "https://blog.example.org/best-bath-toys", + f"{SITE}/", +] +SEARCH_TERMS = [ + "rubber duck", + " Mallard DECOY ", + "teal towel", + "duck mug!", + "pond float", + "giant duck", + "eider cushion", + "golden duck", + "zzzz", + "bath-time set", +] + + +@dataclass +class WebEvent: + event_id: str + received_at: datetime + payload: dict[str, object] + late: bool = False + + +@dataclass +class Session: + anonymous_id: str + customer_id: str | None + timezone: str | None + events: list[WebEvent] = field(default_factory=list) + + +def _ms(value: datetime) -> int: + return int((naive_utc(value) - datetime(1970, 1, 1)).total_seconds() * 1000) + + +@lru_cache(maxsize=4) +def web_sessions(scale: int) -> tuple[Session, ...]: # noqa: PLR0915 - one session script + """Every planned web session, with its events in time order.""" + + world = build_world(scale) + r = rng("web_sessions") + customers = [c for c in world.customers if not c.deleted] + anonymous_pool = [f"anon-{r.getrandbits(48):012x}" for _ in range(350 * scale)] + sessions: list[Session] = [] + counter = [0] + days = clickstream_days() + web_orders = [ + o for o in world.orders if o.placed_at.date() >= days[0] and o.channel in ("web", None) + ] + + def event(session: Session, when: datetime, kind: str, **extra: object) -> None: + counter[0] += 1 + payload: dict[str, object] = { + "type": kind, + "ts_ms": _ms(when), + "anonymous_id": session.anonymous_id + if chance(r, 0.95) + else session.anonymous_id.upper(), + "context": {"ua": session_ua, "locale": session_locale, "tz": session.timezone}, + } + if session.customer_id is not None and kind in ( + "identify", + "login", + "order_completed", + "page_view", + ): + customer = world.customer_by_id[session.customer_id] + payload["user"] = {"id": session.customer_id, "email": customer.email} + for key, value in extra.items(): + payload[key] = value + session.events.append( + WebEvent( + f"E{counter[0]:08d}", when + timedelta(milliseconds=r.randint(150, 4000)), payload + ) + ) + + for day in days: + planned: list[tuple[str, str | None, str | None, datetime, object]] = [] + for order in web_orders: + if order.placed_at.date() == day: + customer = world.customer_by_id[order.customer_id] + start = order.placed_at - timedelta(minutes=r.randint(6, 25)) + planned.append( + ( + customer.anonymous_id, + customer.customer_id, + STOREFRONTS[customer.storefront].timezone, + start, + order, + ) + ) + for _ in range(45 * scale): + if chance(r, 0.25): + customer = r.choice(customers) + anon, customer_id, zone = ( + customer.anonymous_id, + customer.customer_id, + STOREFRONTS[customer.storefront].timezone, + ) + else: + anon, customer_id, zone = ( + r.choice(anonymous_pool), + None, + r.choice( + [ + "Europe/London", + "America/New_York", + "Asia/Tokyo", + "Australia/Sydney", + None, + ] + ), + ) + planned.append( + (anon, customer_id, zone, at(day) + timedelta(seconds=r.randint(0, 86399)), None) + ) + # Sessions that run past midnight UTC. + for _ in range(3): + anon = r.choice(anonymous_pool) + planned.append( + (anon, None, "America/New_York", at(day, 23, r.randint(35, 55)), "midnight") + ) + for anon, customer_id, zone, start, purpose in planned: + session = Session(anon, customer_id, zone) + session_ua = ( + r.choice(BOT_UA) if purpose is None and chance(r, 0.04) else r.choice(BROWSER_UA) + ) + session_locale = r.choice(["en_GB", "en-US", "de_DE", "ja-JP", None]) + when = start + utm = r.choice(utm_for(start)) if chance(r, 0.5) else None + if purpose not in (None, "midnight") and getattr(purpose, "utm", None): + utm = purpose.utm # type: ignore[union-attr] + landing = ( + f"{SITE}/" + if chance(r, 0.4) + else f"{SITE}/c/{r.choice(['rubber-ducks', 'decoys', 'textiles'])}" + ) + if utm and utm[0]: + landing += f"?utm_source={utm[0]}&utm_medium={utm[1]}&utm_campaign={utm[2]}" + elif utm and utm[1] == "organic": + landing += "?utm_medium=organic" + event( + session, + when, + "page_view", + page={"url": landing, "referrer": r.choice(REFERRERS)}, + properties={"title": "Mallard & Co."}, + ) + if customer_id is not None and chance(r, 0.6): + when += timedelta(seconds=r.randint(5, 60)) + event( + session, + when, + r.choice(["login", "identify", "Identify"]), + properties={"method": r.choice(["password", "magic-link", None])}, + ) + browsed = r.sample(world.variants, k=r.randint(0, 4)) + for position, variant in enumerate(browsed, start=1): + when += timedelta(seconds=r.randint(10, 240)) + url = f"{SITE}/products/{variant.sku.lower()}" + event( + session, + when, + "page_view", + page={"url": url, "referrer": f"{SITE}/"}, + properties={"title": f"{variant.species} - Mallard & Co."}, + ) + event( + session, + when + timedelta(milliseconds=300), + r.choice(["product_view", "Product View"]), + page={"url": url}, + properties={ + "sku": variant.raw_sku, + "list": {"name": "category", "position": position}, + }, + ) + if chance(r, 0.2): + when += timedelta(seconds=r.randint(5, 90)) + term = r.choice(SEARCH_TERMS) + event( + session, + when, + "search", + page={"url": f"{SITE}/search?q={term.strip()}"}, + properties={ + "query": term, + "results": 0 if term == "zzzz" else r.choice([3, 12, 40, None]), + }, + ) + ordering = purpose not in (None, "midnight") + if browsed and (ordering or chance(r, 0.35)): + for variant in browsed[:2]: + when += timedelta(seconds=r.randint(5, 120)) + event( + session, + when, + "add_to_cart", + page={"url": f"{SITE}/products/{variant.sku.lower()}"}, + properties={"currency": r.choice(["gbp", "EUR", "usd"])}, + items=[ + { + "sku": variant.raw_sku, + "quantity": r.choice([1, 1, 2, None]), + "price_minor": int(variant.base_usd * 100), + } + ], + ) + if chance(r, 0.15): + when += timedelta(seconds=r.randint(5, 60)) + event(session, when, "remove_from_cart", items=[{"sku": browsed[0].raw_sku}]) + if ordering or chance(r, 0.4): + when += timedelta(seconds=r.randint(20, 120)) + event( + session, when, "checkout_started", page={"url": f"{SITE}/checkout/step-1"} + ) + for step, name in ((2, "Delivery"), (3, "Payment")): + if not ordering and chance(r, 0.4): + break + when += timedelta(seconds=r.randint(20, 200)) + event( + session, + when, + "checkout_step", + page={"url": f"{SITE}/checkout/step-{step}"}, + properties={ + "step": step, + "step_name": name, + "payment_option": r.choice(["Card", "phone_wallet", None]) + if step == 3 + else None, + }, + ) + if ordering: + order = purpose + when = max(when + timedelta(seconds=30), order.placed_at) # type: ignore[union-attr] + event( + session, + when, + "order_completed", + page={"url": f"{SITE}/checkout/thank-you"}, + properties={"order_id": order.order_id, "revenue_minor": order.total_minor}, + ) # type: ignore[union-attr] + if purpose == "midnight": + for _ in range(r.randint(2, 4)): + when += timedelta(minutes=r.randint(6, 20)) + event( + session, + when, + "page_view", + page={"url": f"{SITE}/c/decoys?page={r.randint(2, 5)}"}, + ) + # A few sessions lose their last event in transit; it arrives a day or two later. + if len(session.events) > 2 and chance(r, 0.03): + last = session.events[-1] + last.received_at = last.received_at + timedelta( + days=r.randint(1, 2), hours=r.randint(0, 5) + ) + last.late = True + sessions.append(session) + return tuple(sessions) + + +def web_events(world: World) -> Rows: + r = rng("clickstream__web_events") + log = EventLog("clickstream__web_events") + for session in web_sessions(world.scale): + for e in session.events: + payload = dict(e.payload) + if chance(r, 0.005): + payload.pop("ts_ms") + row = { + "event_id": e.event_id, + "received_at": naive_utc(e.received_at), + "event": payload, + } + loaded = e.received_at + timedelta(minutes=r.randint(1, 20)) + log.add(row, loaded) + if chance(r, 0.015): + log.add(row, loaded + timedelta(minutes=r.randint(30, 300))) + return log.rows() + + +def app_events(world: World) -> Rows: + r = rng("clickstream__app_events") + log = EventLog("clickstream__app_events") + customers = [c for c in world.customers if not c.deleted] + devices = [] + for i in range(1, 120 * world.scale + 1): + customer = r.choice(customers) if chance(r, 0.7) else None + devices.append( + ( + f"dev-{i:05d}", + customer.customer_id if customer else None, + r.choice(["Phone", "phone", "TABLET", "tablet"]), + r.choice(["3.4.1", "v3.10.0-beta", "4", "4.0.2 (build 77)"]), + ) + ) + count = 0 + for day in clickstream_days(): + for _ in range(25 * world.scale): + device_id, customer_id, platform, version = r.choice(devices) + when = at(day) + timedelta(seconds=r.randint(0, 86399)) + names = ["app_open"] + ["screen_view"] * r.randint(1, 5) + if chance(r, 0.15): + names.insert(0, "push_open") + if chance(r, 0.2): + names.append("add_to_cart") + for name in names: + count += 1 + when += timedelta(seconds=r.randint(2, 300)) + payload: dict[str, object] = { + "name": name if chance(r, 0.9) else name.upper(), + "device": {"id": device_id}, + "user_id": customer_id, + } + if chance(r, 0.5): + payload["ts"] = in_zone(when, "UTC").strftime("%Y-%m-%d %H:%M:%S.%f") + else: + payload["ts_ms"] = _ms(when) + if name == "screen_view": + payload["screen"] = r.choice( + ["home", "product_detail", "basket", "order_history"] + ) + payload["attributes"] = { + "screen_class": r.choice(["HomeVC", "ProductActivity", None]) + } + if name == "push_open": + campaign = r.choice([c[0] for c in CAMPAIGNS]) + payload["attributes"] = r.choice( + [ + {"campaign_id": campaign}, + {"deep_link": f"duckshop://promo/spring?cid={campaign}"}, + {"deep_link": "duckshop://home"}, + ] + ) + received = when + timedelta(seconds=r.randint(1, 30)) + if chance(r, 0.02): + received += timedelta(hours=r.randint(2, 30)) + row = { + "event_id": f"AE{count:08d}", + "received_at": received, + "platform": platform, + "app_version": version, + "payload": payload, + } + log.add(row, received + timedelta(minutes=r.randint(1, 10))) + if chance(r, 0.01): + log.add(row, received + timedelta(hours=3)) + return log.rows() + + +def search_queries(world: World) -> Rows: + r = rng("clickstream__search_queries") + log = EventLog("clickstream__search_queries") + anonymous = [s.anonymous_id for s in web_sessions(world.scale)] + count = 0 + for day in clickstream_days(): + for _ in range(20 * world.scale): + count += 1 + term = r.choice(SEARCH_TERMS + ["Mallard's decoy (large)", "duck-shaped soap"]) + when = at(day) + timedelta(seconds=r.randint(0, 86399)) + filters: dict[str, object] | None = r.choice( + [ + None, + {"category": "Decoys", "sort": "price_asc"}, + {"price": {"max_minor": r.choice([2000, 5000, "9999"])}}, + {"category": None}, + ] + ) + row = { + "query_id": f"Q{count:07d}", + "anonymous_id": r.choice(anonymous), + "query_text": term, + "results_count": 0 if term in ("zzzz", "duck-shaped soap") else r.randint(1, 60), + "searched_at": in_zone( + when, r.choice(["Europe/London", "America/New_York", "Asia/Tokyo"]) + ), + "filters": filters, + } + log.add(row, when + timedelta(minutes=r.randint(1, 30))) + if chance(r, 0.02): + log.add(row, when + timedelta(hours=2)) + return log.rows() + + +EMAIL_TYPES = {"sent": 0, "delivered": 1, "open": 2, "click": 3} + + +def email_events(world: World) -> Rows: + r = rng("clickstream__email_events") + log = EventLog("clickstream__email_events") + customers = [c for c in world.customers if c.email] + email_campaigns = [ + ("POND-WEEK-EMAIL", at(clickstream_days()[0]) - timedelta(days=20)), + ("SPRING-SPLASH", at(clickstream_days()[0], 8)), + ("GOLDEN-EGG", at(clickstream_days()[0] + timedelta(days=10), 8)), + ] + count = 0 + message = 0 + for campaign_id, sent_at in email_campaigns: + for customer in r.sample(customers, k=min(len(customers), 120 * world.scale)): + message += 1 + message_id = f"" + when = sent_at + timedelta(seconds=r.randint(0, 3600)) + journey = ["sent", "delivered"] + if chance(r, 0.05): + journey = ["sent", "bounce"] + else: + if chance(r, 0.45): + journey.append("open") + if chance(r, 0.35): + journey.append("click") + if chance(r, 0.03): + journey.append("unsubscribe") + if chance(r, 0.005): + journey.append("complaint") + for kind in journey: + count += 1 + when += timedelta(minutes=r.randint(1, 600)) + details: dict[str, object] | None = None + if kind == "click": + details = { + "url": f"{SITE}/c/decoys?utm_source=newsletter&utm_medium=email" + f"&utm_campaign={campaign_id.lower()}" + } + elif kind == "open": + details = r.choice( + [ + {"machine_open": True}, + {"user_agent": "Waddle/5.0 (via ImageProxy)"}, + {"user_agent": BROWSER_UA[1]}, + None, + ] + ) + elif kind == "bounce": + details = {"bounce": {"type": r.choice(["Hard", "soft"])}} + row = { + "email_event_id": f"EM{count:08d}", + "message_id": message_id, + "event_type": kind if chance(r, 0.9) else kind.upper(), + "recipient_email": customer.email, + "campaign_id": campaign_id, + "occurred_at": in_zone(when, "UTC"), + "details": details, + } + log.add(row, when + timedelta(minutes=r.randint(1, 120))) + if chance(r, 0.02): + log.add(row, when + timedelta(hours=6)) + return log.rows() + + +def experiment_assignments(world: World) -> Rows: + r = rng("clickstream__experiment_assignments") + log = EventLog("clickstream__experiment_assignments") + count = 0 + seen: set[tuple[str, str]] = set() + experiments = [ + ("checkout-button-colour", ["control", "yellow"]), + ("Free-Shipping-Banner", ["a", "b", "c"]), + ] + for session in web_sessions(world.scale): + if session.customer_id is None and not chance(r, 0.5): + continue + key, variants = r.choice(experiments) + start = ( + session.events[0].payload["ts_ms"] + if session.events and "ts_ms" in session.events[0].payload + else None + ) + if start is None: + continue + when = datetime(1970, 1, 1) + timedelta(milliseconds=int(start)) # type: ignore[arg-type] + count += 1 + variant = r.choice(variants) + repeat = (key, session.anonymous_id) in seen + seen.add((key, session.anonymous_id)) + row = { + "assignment_id": f"XA{count:07d}", + "experiment_key": key, + "variant_key": variant.upper() if chance(r, 0.1) else variant, + "anonymous_id": session.anonymous_id, + "customer_id": session.customer_id, + "assigned_at": in_zone(when, session.timezone or "UTC"), + "context": {"surface": "web", "reassigned": repeat} if chance(r, 0.8) else None, + } + log.add(row, when + timedelta(minutes=r.randint(1, 15))) + if chance(r, 0.02): + log.add(row, when + timedelta(hours=1)) + return log.rows() + + +TABLES = { + "clickstream__web_events": web_events, + "clickstream__app_events": app_events, + "clickstream__search_queries": search_queries, + "clickstream__email_events": email_events, + "clickstream__experiment_assignments": experiment_assignments, +} diff --git a/benchmarks/duckshop/project/python/loaders/_core.py b/benchmarks/duckshop/project/python/loaders/_core.py new file mode 100644 index 0000000000..7c58279bbc --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_core.py @@ -0,0 +1,193 @@ +"""Shared building blocks for the DuckShop synthetic data generators. + +Every generator draws from its own seeded random stream, so the same code version always produces +the same rows, and changing one table's generator does not shift another table's data. +""" + +from __future__ import annotations + +import random +from dataclasses import dataclass, field +from datetime import UTC, date, datetime, time, timedelta +from zoneinfo import ZoneInfo + +SEED = 20260331 +"""Fixed seed for every generator. Change it only together with the data version below.""" + +DATA_VERSION = "1" +"""Bumped whenever generated rows change on purpose.""" + +VALUATION_DATE = date(2026, 3, 31) +ORDERS_START = date(2026, 1, 1) +CLICKSTREAM_START = date(2026, 3, 9) +UTC = UTC + + +def rng(*parts: object) -> random.Random: + """A random stream unique to one generator, derived from the fixed seed.""" + + return random.Random(f"duckshop:{SEED}:{DATA_VERSION}:" + ":".join(str(p) for p in parts)) + + +def chance(r: random.Random, probability: float) -> bool: + return r.random() < probability + + +def weighted(r: random.Random, options: dict[object, float]) -> object: + keys = list(options) + return r.choices(keys, weights=[options[k] for k in keys], k=1)[0] + + +def utc(value: datetime) -> datetime: + """A timezone-aware UTC copy of a naive UTC or aware datetime.""" + + if value.tzinfo is None: + return value.replace(tzinfo=UTC) + return value.astimezone(UTC) + + +def naive_utc(value: datetime) -> datetime: + return utc(value).replace(tzinfo=None) + + +def in_zone(value: datetime, zone: str) -> datetime: + """The same instant expressed in an IANA zone, keeping its UTC offset (DST-aware).""" + + return utc(value).astimezone(ZoneInfo(zone)) + + +def local_naive(value: datetime, zone: str) -> datetime: + """Wall-clock time in a zone without offset, as warehouse-local NTZ columns store it.""" + + return in_zone(value, zone).replace(tzinfo=None) + + +def at(day: date, hour: int = 0, minute: int = 0, second: int = 0) -> datetime: + """A naive UTC datetime on one day.""" + + return datetime.combine(day, time(hour, minute, second)) + + +def random_instant(r: random.Random, start: date, end: date) -> datetime: + """A naive UTC instant uniformly between the start of `start` and the end of `end`.""" + + span = (at(end) + timedelta(days=1) - at(start)).total_seconds() + return at(start) + timedelta(seconds=int(r.random() * span)) + + +def days(start: date, end: date) -> list[date]: + return [start + timedelta(days=i) for i in range((end - start).days + 1)] + + +def messy_case(r: random.Random, text: str) -> str: + """The same text in a random letter case, as hand-keyed source systems produce it.""" + + return weighted(r, {text: 6, text.lower(): 2, text.upper(): 1, text.title(): 1}) # type: ignore[return-value] + + +def messy_sku(r: random.Random, sku: str) -> str: + """A SKU as an upstream system might send it: lower case, underscores or stray spaces.""" + + roll = r.random() + if roll < 0.75: + return sku + if roll < 0.85: + return sku.lower() + if roll < 0.93: + return sku.replace("-", "_", 1) + return f" {sku} " + + +@dataclass +class ChangeLog: + """An append-only change log: every version of a row with `_op`, `_seq` and `_loaded_at`. + + `_seq` follows load order, not event order, so late and out-of-order arrivals look like they do + in a real replication stream. + """ + + name: str + _pending: list[tuple[datetime, int, dict[str, object]]] = field(default_factory=list) + + def add(self, row: dict[str, object], loaded_at: datetime, op: str = "I") -> None: + self._pending.append((naive_utc(loaded_at), len(self._pending), {**row, "_op": op})) + + def insert(self, row: dict[str, object], loaded_at: datetime) -> None: + self.add(row, loaded_at, "I") + + def update(self, row: dict[str, object], loaded_at: datetime) -> None: + self.add(row, loaded_at, "U") + + def delete(self, row: dict[str, object], loaded_at: datetime) -> None: + self.add(row, loaded_at, "D") + + def redeliver(self, row: dict[str, object], loaded_at: datetime, op: str = "I") -> None: + """The same version delivered twice, as at-least-once replication does.""" + + self.add(row, loaded_at, op) + + def rows(self) -> list[dict[str, object]]: + ordered = sorted(self._pending, key=lambda item: (item[0], item[1])) + return [ + {**row, "_seq": index + 1, "_loaded_at": loaded_at} + for index, (loaded_at, _, row) in enumerate(ordered) + ] + + +@dataclass +class EventLog: + """An append-only event table where a few events are delivered twice.""" + + name: str + _rows: list[tuple[datetime, int, dict[str, object]]] = field(default_factory=list) + + def add(self, row: dict[str, object], loaded_at: datetime) -> None: + self._rows.append( + (naive_utc(loaded_at), len(self._rows), {**row, "_loaded_at": naive_utc(loaded_at)}) + ) + + def rows(self) -> list[dict[str, object]]: + return [row for _, _, row in sorted(self._rows, key=lambda item: (item[0], item[1]))] + + +def load_lag(r: random.Random, event_at: datetime, *, late_share: float = 0.04) -> datetime: + """When a change reaches the warehouse: minutes later, or days later for late arrivals.""" + + if chance(r, late_share): + return naive_utc(event_at) + timedelta(days=r.randint(1, 6), minutes=r.randint(0, 600)) + return naive_utc(event_at) + timedelta(minutes=r.randint(2, 45), seconds=r.randint(0, 59)) + + +def emit( + log: ChangeLog, + r: random.Random, + versions: list[tuple[datetime, dict[str, object]]], + *, + late_share: float = 0.04, + duplicate_share: float = 0.02, + delete_at: datetime | None = None, +) -> None: + """Write one entity's versions to a change log: an insert, then updates, maybe a delete. + + Each version reaches the warehouse after a short lag, or days later for a late arrival, so a + late older version can land after a newer one. A few versions are delivered twice. + """ + + for index, (event_at, row) in enumerate(versions): + op = "I" if index == 0 else "U" + loaded_at = load_lag(r, event_at, late_share=late_share) + log.add(row, loaded_at, op) + if chance(r, duplicate_share): + log.redeliver(row, loaded_at + timedelta(minutes=r.randint(1, 90)), op) + if delete_at is not None: + log.delete(versions[-1][1], load_lag(r, delete_at, late_share=0)) + + +def money(r: random.Random, low: float, high: float, exponent: int) -> int: + """A minor-unit amount between two major-unit values for a currency exponent.""" + + value = r.uniform(low, high) + if exponent == 0: + return int(round(value / 10) * 10) + minor = int(round(value * 10**exponent)) + return minor - minor % 10 + 9 if exponent == 2 and minor > 100 else minor diff --git a/benchmarks/duckshop/project/python/loaders/_finance.py b/benchmarks/duckshop/project/python/loaders/_finance.py new file mode 100644 index 0000000000..4b00ed84dc --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_finance.py @@ -0,0 +1,438 @@ +"""Finance change logs and feeds: invoices, ledger, payouts, tax rates, chargebacks, settlements.""" + +from __future__ import annotations + +import json +from collections import defaultdict +from datetime import date, datetime, timedelta + +from python.loaders._core import ChangeLog, EventLog, at, chance, emit, in_zone, rng +from python.loaders._storefront import gateway_reference, payments_plan, provider_for, refunds_plan +from python.loaders._world import STOREFRONTS, World + +Rows = list[dict[str, object]] +TAX_RATES = [ + # country, region, category, rate, valid from, valid to + ("GB", None, "standard", 20.0, date(2011, 1, 4), None), + ("GB", None, "reduced", 5.0, date(2011, 1, 4), None), + ("de", None, "Standard", 19.0, date(2007, 1, 1), None), + ("DE", None, "reduced", 7.0, date(2007, 1, 1), None), + ("FR", None, "standard", 20.0, date(2014, 1, 1), None), + ("NL", None, "standard", 21.0, date(2012, 10, 1), None), + ("JP", None, "standard", 8.0, date(2014, 4, 1), date(2019, 9, 30)), + ("JP", None, "standard", 10.0, date(2019, 10, 1), None), + ("AU", None, "standard", 10.0, date(2000, 7, 1), None), + ("US", "ny", "standard", 8.875, date(2025, 1, 1), date(2026, 2, 28)), + ("US", "NY", "standard", 8.8754, date(2026, 3, 1), None), + ("US", "ma", "standard", 6.25, date(2009, 8, 1), None), + ("CA", "ON", "standard", 13.0, date(2010, 7, 1), None), + ("KW", None, "standard", 0.0, date(2020, 1, 1), None), +] +TAX_CODE = {"EU": "VAT_STD", "APAC": "GST", "US": "SALES", "MENA": "NONE"} +TAX_RATE = {"EU": 20.0, "US": 0.0, "APAC": 10.0, "MENA": 0.0} + + +def tax_rates(world: World) -> Rows: + r = rng("finance__tax_rates") + log = ChangeLog("finance__tax_rates") + for index, (country, region, category, rate, start, end) in enumerate(TAX_RATES, start=1): + row = { + "tax_rate_id": f"TR{index:03d}", + "country_code": country, + "tax_region": region, + "tax_category": category, + "rate_pct": rate, + "valid_from": start, + "valid_to": end, + } + emit(log, r, [(at(date(2025, 12, 1)), row)]) + return log.rows() + + +def invoices(world: World) -> Rows: + r = rng("finance__invoices") + log = ChangeLog("finance__invoices") + count = 0 + for order in world.orders: + paid = order.time_of("paid") + if paid is None: + continue + count += 1 + sf = STOREFRONTS[order.storefront] + net = order.net_minor + tax = order.tax_minor + tax_lines: list[dict[str, object]] | None = [ + { + "code": TAX_CODE[sf.region].lower() if chance(r, 0.3) else TAX_CODE[sf.region], + "rate_pct": TAX_RATE[sf.region], + "taxable_minor": net, + "tax_minor": tax, + } + ] + if chance(r, 0.03): + tax_lines = None + elif chance(r, 0.04) and tax: + tax_lines = [ + {**tax_lines[0], "tax_minor": tax - 1}, + {"code": "rounding", "rate_pct": 0, "taxable_minor": 0, "tax_minor": 1}, + ] + totals: dict[str, object] = {"net_minor": net, "tax_minor": tax} + if chance(r, 0.7): + totals["gross_minor"] = net + tax + series = {"EU": "EU", "US": "US", "APAC": "APAC", "MENA": "ME"}[sf.region] + number = f"{series}-2026-{count:06d}" + row = { + "invoice_id": f"INV{count:06d}", + "order_id": order.order_id, + "invoice_number": number.lower() if chance(r, 0.1) else number, + "issued_at": in_zone(paid + timedelta(minutes=r.randint(1, 30)), sf.timezone), + "currency_code": order.currency, + "totals": totals, + "tax_lines": tax_lines, + } + versions = [(paid, row)] + if chance(r, 0.03): + versions.append( + (paid + timedelta(days=2), {**row, "totals": {**totals, "note": "reissued"}}) + ) + emit(log, r, versions) + return log.rows() + + +ACCOUNTS = { + "cash": "1000", + "receivables": "1100", + "inventory": "1300", + "tax": "2200", + "gift_cards": "2300", + "equity": "3000", + "revenue": "4000", + "refunds": "4100", + "fees": "6100", + "shipping": "6200", +} + + +def ledger_entries(world: World) -> Rows: + r = rng("finance__ledger_entries") + log = ChangeLog("finance__ledger_entries") + count = 0 + + def post( + journal: str, + account: str, + debit: int | None, + credit: int | None, + currency: str, + when: datetime, + reference: dict[str, object] | None, + ) -> None: + nonlocal count + count += 1 + row = { + "entry_id": f"JE{count:07d}", + "journal_code": journal, + "account_code": account, + "debit_minor": debit, + "credit_minor": credit, + "currency_code": currency, + "posted_at": in_zone(when, "Europe/London"), + "reference": reference, + } + emit(log, r, [(when, row)], late_share=0.02) + + post( + "opening", + ACCOUNTS["cash"], + 25_000_000, + None, + "GBP", + at(date(2026, 1, 1)), + {"type": "Opening"}, + ) + post( + "opening", + ACCOUNTS["equity"], + None, + 25_000_000, + "GBP", + at(date(2026, 1, 1)), + {"type": "opening"}, + ) + for order in world.orders: + paid = order.time_of("paid") + if paid is None: + continue + reference = {"type": "order", "id": order.order_id} + post( + "sales", + ACCOUNTS["receivables"], + order.total_minor, + None, + order.currency, + paid, + reference, + ) + post("sales", ACCOUNTS["revenue"], None, order.net_minor, order.currency, paid, reference) + if order.tax_minor: + post("SALES", ACCOUNTS["tax"], None, order.tax_minor, order.currency, paid, reference) + for refund_id, _, order_id, amount, when, _ in refunds_plan(world.scale): + currency = world.order_by_id[order_id].currency + post( + "refunds", + ACCOUNTS["refunds"], + amount, + 0, + currency, + when, + {"type": "refund", "id": refund_id}, + ) + post( + "refunds", + ACCOUNTS["receivables"], + 0, + amount, + currency, + when, + {"type": "refund", "id": refund_id}, + ) + for payout in _payouts(world): + reference = {"type": "payout", "id": payout["payout_id"]} + when = at(payout["paid_out_on"], 6) # type: ignore[arg-type] + amount = int(payout["amount_minor"]) # type: ignore[call-overload] + fee = int(payout["fee_minor"] or 0) # type: ignore[call-overload] + currency = str(payout["currency_code"]) + post("bank", ACCOUNTS["cash"], amount - fee, None, currency, when, reference) + if fee: + post("bank", ACCOUNTS["fees"], fee, None, currency, when, reference) + post("bank", ACCOUNTS["receivables"], None, amount, currency, when, None) + return log.rows() + + +def _payouts(world: World) -> list[dict[str, object]]: + """Weekly payouts per provider and currency, built from captured payments and refunds.""" + + r = rng("payouts_plan") + orders = world.order_by_id + groups: dict[tuple[str, str, date], list[dict[str, object]]] = defaultdict(list) + for payment_id, order_id, method, amount, when, status in payments_plan(world.scale): + if status not in ("captured", "settled"): + continue + payout_day = when.date() + timedelta(days=(7 - when.weekday()) % 7 + 2) + if payout_day > date(2026, 3, 31): + continue + provider = provider_for(method) + fee = max(1, amount * 14 // 1000) + groups[(provider, orders[order_id].currency, payout_day)].append( + { + "payment_id": payment_id, + "type": "charge" if chance(r, 0.95) else None, + "gross_minor": amount, + "fee_minor": fee, + } + ) + payment_method = {p[0]: p[2] for p in payments_plan(world.scale)} + for _, payment_id, order_id, amount, when, _ in refunds_plan(world.scale): + payout_day = when.date() + timedelta(days=(7 - when.weekday()) % 7 + 2) + if payout_day > date(2026, 3, 31): + continue + key = (provider_for(payment_method[payment_id]), orders[order_id].currency, payout_day) + if key in groups: + groups[key].append({"payment_id": payment_id, "type": "Refund", "gross_minor": amount}) + payouts = [] + for index, ((provider, currency, day), lines) in enumerate(sorted(groups.items()), start=1): + gross = sum( + int(line["gross_minor"]) * (-1 if line["type"] == "Refund" else 1) for line in lines + ) # type: ignore[call-overload] + fee = sum(int(line.get("fee_minor", 0)) for line in lines) # type: ignore[call-overload] + if index % 17 == 0: + lines = lines + [ + {"payment_id": "PAY9999999", "type": "charge", "gross_minor": 1234, "fee_minor": 17} + ] + payouts.append( + { + "payout_id": f"PO{index:05d}", + "provider_code": provider.lower() if index % 5 == 0 else provider, + "currency_code": currency, + "amount_minor": gross, + "fee_minor": fee if index % 9 else None, + "paid_out_on": day, + "lines": lines if index % 13 else None, + } + ) + return payouts + + +def payouts(world: World) -> Rows: + r = rng("finance__payouts") + log = ChangeLog("finance__payouts") + for payout in _payouts(world): + emit(log, r, [(at(payout["paid_out_on"], 5), payout)]) # type: ignore[arg-type] + return log.rows() + + +CHARGEBACK_REASONS = ["10.4", "4837", "4863", "13.1", "4853", "4855", "12.6", "ab-01"] + + +def chargebacks(world: World) -> Rows: + r = rng("finance__chargebacks") + log = ChangeLog("finance__chargebacks") + count = 0 + orders = world.order_by_id + for payment_id, order_id, _, amount, when, status in payments_plan(world.scale): + if status not in ("captured", "settled") or not chance(r, 0.012): + continue + count += 1 + opened = when + timedelta(days=r.randint(5, 40)) + if opened.date() > date(2026, 3, 31): + continue + order = orders[order_id] + row = { + "chargeback_id": f"CB{count:04d}", + "payment_id": payment_id, + "amount_minor": amount, + "currency_code": order.currency.lower() if chance(r, 0.2) else order.currency, + "reason_code": r.choice(CHARGEBACK_REASONS), + "opened_at": in_zone(opened, "UTC"), + "closed_at": None, + "outcome": None, + "evidence": r.choice( + [ + {"documents": [{"kind": "receipt"}, {"kind": "Tracking"}]}, + {"documents": []}, + {"documents": [{"kind": "receipt"}, {"kind": "RECEIPT"}, {"kind": None}]}, + None, + ] + ), + } + versions = [(opened, row)] + closed = opened + timedelta(days=r.randint(5, 30)) + if closed.date() <= date(2026, 3, 31) and chance(r, 0.7): + versions.append( + ( + closed, + { + **row, + "closed_at": in_zone(closed, "UTC"), + "outcome": r.choice(["won", "Lost", "LOST", "accepted"]), + }, + ) + ) + emit(log, r, versions) + return log.rows() + + +def settlement_reports(world: World) -> Rows: + r = rng("finance__settlement_reports") + log = EventLog("finance__settlement_reports") + orders = world.order_by_id + days: dict[tuple[str, date, str], list[dict[str, object]]] = defaultdict(list) + for payment_id, order_id, method, amount, when, status in payments_plan(world.scale): + if status not in ("captured", "settled") or when.date() < date(2026, 3, 1): + continue + settle_day = when.date() + timedelta(days=1) + if settle_day > date(2026, 3, 31): + continue + line: dict[str, object] + fee = max(1, amount * 14 // 1000) + roll = r.random() + if roll < 0.7: + line = { + "type": "charge", + "reference": gateway_reference(payment_id), + "gross_minor": amount, + "fee_minor": fee, + } + elif roll < 0.9: + line = { + "record_type": "CHARGE", + "description": f"Card payment pay_{gateway_reference(payment_id)}x", + "gross": f"{amount / 100:.2f}", + "fee_minor": fee, + } + else: + line = { + "type": "charge", + "reference": " ", + "description": "Payment without a reference", + "gross_minor": amount, + } + days[(provider_for(method), settle_day, orders[order_id].currency)].append(line) + for refund_id, payment_id, order_id, amount, when, _ in refunds_plan(world.scale): + settle_day = when.date() + timedelta(days=1) + if when.date() < date(2026, 3, 1) or settle_day > date(2026, 3, 31): + continue + method = next(p[2] for p in payments_plan(world.scale) if p[0] == payment_id) + days[(provider_for(method), settle_day, orders[order_id].currency)].append( + {"type": "refund", "reference": refund_id, "gross_minor": -amount} + ) + count = 0 + for (provider, day, currency), lines in sorted(days.items()): + count += 1 + gross = sum(int(line.get("gross_minor", 0)) for line in lines) # type: ignore[call-overload] + fees = sum(int(line.get("fee_minor", 0)) for line in lines) # type: ignore[call-overload] + body = { + "summary": {"currency": currency.lower(), "gross_minor": gross, "fees_minor": fees}, + "lines": lines, + } + loaded = at(day, 7, r.randint(0, 59)) + report_id = f"SR{count:05d}" + log.add( + { + "report_id": report_id, + "provider_code": provider, + "report_date": day, + "report_body": json.dumps(body), + }, + loaded, + ) + if chance(r, 0.05): + # The provider re-sent the report later the same day with one more line. + resent = { + **body, + "lines": lines + [{"type": "adjustment", "gross_minor": -5, "fee_minor": 0}], + } + log.add( + { + "report_id": report_id, + "provider_code": provider, + "report_date": day, + "report_body": json.dumps(resent), + }, + loaded + timedelta(hours=5), + ) + if chance(r, 0.03): + count += 1 + log.add( + { + "report_id": f"SR{count:05d}", + "provider_code": provider.lower(), + "report_date": day, + "report_body": json.dumps({"summary": body["summary"], "lines": lines[:1]}), + }, + loaded, + ) + for broken, text in enumerate( + ["503 Service Unavailable", "[]", "", '{"summary": '], start=1 + ): + count += 1 + log.add( + { + "report_id": f"SR{count:05d}", + "provider_code": "PROVIDER_A", + "report_date": date(2026, 3, 10 + broken), + "report_body": text or None, + }, + at(date(2026, 3, 10 + broken), 7), + ) + return log.rows() + + +TABLES = { + "finance__invoices": invoices, + "finance__ledger_entries": ledger_entries, + "finance__payouts": payouts, + "finance__tax_rates": tax_rates, + "finance__chargebacks": chargebacks, + "finance__settlement_reports": settlement_reports, +} diff --git a/benchmarks/duckshop/project/python/loaders/_fulfilment.py b/benchmarks/duckshop/project/python/loaders/_fulfilment.py new file mode 100644 index 0000000000..aee9d729e8 --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_fulfilment.py @@ -0,0 +1,563 @@ +"""Fulfilment change logs and feeds: shipments, packages, webhooks, returns, slots and picking.""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from datetime import date, datetime, timedelta +from functools import lru_cache + +from python.loaders._core import ( + ChangeLog, + EventLog, + at, + chance, + emit, + in_zone, + local_naive, + rng, + weighted, +) +from python.loaders._world import ( + CARRIERS, + STOREFRONTS, + WAREHOUSE_FOR_REGION, + WAREHOUSES, + World, + build_world, +) + +Rows = list[dict[str, object]] +WAREHOUSE = {w.code: w for w in WAREHOUSES} +TRACKING_PREFIX = {"FLAPWING": "FW", "WINGBEAT": "WB", "POSTAL": "PS", "WEBFOOT": "WF"} + + +@dataclass(frozen=True) +class Shipment: + shipment_id: str + order_id: str + warehouse: str + carrier: str + service: str | None + tracking_number: str + shipped_at: datetime | None + delivered_at: datetime | None + line_skus: tuple[tuple[str, int], ...] + destination: str + + +@lru_cache(maxsize=4) +def shipment_plan(scale: int) -> tuple[Shipment, ...]: + world = build_world(scale) + r = rng("shipment_plan") + plan: list[Shipment] = [] + count = 0 + for order in world.orders: + packed = order.time_of("packed") + if packed is None: + continue + shipped = order.time_of("shipped") + delivered = order.time_of("delivered") + region = STOREFRONTS[order.storefront].region + warehouse = WAREHOUSE_FOR_REGION[region] + groups = [order.lines] + if len(order.lines) >= 4 and chance(r, 0.5): + groups = [order.lines[:2], order.lines[2:]] + for index, lines in enumerate(groups): + count += 1 + carrier = str(weighted(r, {"FLAPWING": 45, "WINGBEAT": 20, "POSTAL": 30, "WEBFOOT": 5})) + service = ( + r.choice(["standard", "Express", None, "next-day"]) + if carrier != "WEBFOOT" + else "freight" + ) + number = f"{TRACKING_PREFIX[carrier]}{count:08d}" + ship_time = shipped + timedelta(hours=index * 20) if shipped else None + if ship_time is not None and ship_time.date() > date(2026, 3, 31): + ship_time = shipped + plan.append( + Shipment( + shipment_id=f"SH{count:06d}", + order_id=order.order_id, + warehouse=warehouse, + carrier=carrier, + service=service, + tracking_number=number, + shipped_at=ship_time, + delivered_at=delivered if delivered and index == 0 else None, + line_skus=tuple((line.sku, line.quantity) for line in lines), + destination=order.storefront, + ) + ) + return tuple(plan) + + +def _pretty_tracking(number: str, r) -> str: + roll = r.random() + if roll < 0.6: + return number + if roll < 0.8: + return f"{number[:2]}-{number[2:6]} {number[6:]}" + return number.lower() + + +def shipments(world: World) -> Rows: + r = rng("fulfilment__shipments") + log = ChangeLog("fulfilment__shipments") + for s in shipment_plan(world.scale): + zone = WAREHOUSE[s.warehouse].timezone + events: list[dict[str, object]] = [] + if s.shipped_at: + events.append( + { + "code": "picked-up", + "ts": in_zone(s.shipped_at, zone).isoformat(), + "location": f"{WAREHOUSE[s.warehouse].name}, " + f"{WAREHOUSE[s.warehouse].country.lower()}", + } + ) + hub = s.shipped_at + timedelta(hours=r.randint(3, 20)) + stamp = ( + in_zone(hub, zone).strftime("%d/%m/%Y %H:%M %z") + if chance(r, 0.3) + else in_zone(hub, "UTC").isoformat() + ) + events.append( + { + "code": "IN_TRANSIT", + "ts": stamp, + "location": r.choice(["sorting hub ,GB", "Hub 4", None]), + } + ) + if s.delivered_at: + events.append( + { + "code": "Delivered!", + "ts": in_zone(s.delivered_at, STOREFRONTS[s.destination].timezone).isoformat(), + "location": f"doorstep, {s.destination}", + } + ) + tracking: dict[str, object] | None = { + "number": _pretty_tracking(s.tracking_number, r), + "events": events, + } + row = { + "shipment_id": s.shipment_id, + "order_id": s.order_id, + "warehouse_code": s.warehouse.lower() if chance(r, 0.1) else s.warehouse, + "carrier_code": s.carrier, + "service_level": s.service, + "shipped_at": in_zone(s.shipped_at, zone) if s.shipped_at else None, + "delivered_at": in_zone(s.delivered_at, STOREFRONTS[s.destination].timezone) + if s.delivered_at + else None, + "tracking": tracking, + } + first = s.shipped_at or build_world(world.scale).order_by_id[s.order_id].placed_at + versions = [ + (first, {**row, "delivered_at": None, "tracking": {**tracking, "events": events[:1]}}) + ] + if s.delivered_at: + versions.append((s.delivered_at, row)) + emit(log, r, versions) + return log.rows() + + +def shipment_packages(world: World) -> Rows: + r = rng("fulfilment__shipment_packages") + log = ChangeLog("fulfilment__shipment_packages") + variants = world.variant_by_sku + count = 0 + for s in shipment_plan(world.scale): + parts = ( + [s.line_skus] + if len(s.line_skus) < 3 or chance(r, 0.5) + else [s.line_skus[:1], s.line_skus[1:]] + ) + for part in parts: + count += 1 + grams = sum(variants[sku].weight_grams * qty for sku, qty in part) + r.randint(80, 400) + dimensions: dict[str, object] | None = { + "l": r.choice([20, 30, 45]), + "w": r.choice([15, "20.5", 30]), + "h": r.choice([10, 12.75, 25]), + } + if chance(r, 0.05): + dimensions = {"l": 30, "w": 20} + row = { + "package_id": f"PK{count:06d}", + "shipment_id": s.shipment_id, + "weight_grams": grams if chance(r, 0.97) else None, + "dimensions": dimensions, + "contents": [{"sku": sku, "qty": qty} for sku, qty in part], + } + emit(log, r, [(s.shipped_at or at(date(2026, 3, 31)), row)]) + return log.rows() + + +def carrier_webhooks(world: World) -> Rows: + r = rng("fulfilment__carrier_webhooks") + log = EventLog("fulfilment__carrier_webhooks") + count = 0 + event_id = 700000 + for s in shipment_plan(world.scale): + if s.carrier == "POSTAL" or s.shipped_at is None: + continue + updates: list[tuple[str, datetime]] = [ + ("in_transit", s.shipped_at + timedelta(hours=r.randint(2, 12))) + ] + if s.delivered_at: + updates.append(("out_for_delivery", s.delivered_at - timedelta(hours=r.randint(2, 6)))) + updates.append(("delivered", s.delivered_at)) + payloads: list[dict[str, object]] = [] + for status, when in updates: + event_id += 1 + if chance(r, 0.2): + payloads.append( + { + "trackingNumber": _pretty_tracking(s.tracking_number, r), + "event": status, + "timestamp": in_zone(when, "UTC").isoformat(), + "event_id": str(event_id), + } + ) + else: + payloads.append( + { + "tracking_number": _pretty_tracking(s.tracking_number, r), + "status": {"code": status, "at": in_zone(when, "UTC").isoformat()}, + "event_id": event_id, + } + ) + batched = s.carrier == "WINGBEAT" and len(payloads) > 1 + bodies = ( + [(json.dumps({"events": payloads}), updates[-1][1])] + if batched + else [(json.dumps(p), when) for p, (_, when) in zip(payloads, updates, strict=True)] + ) + for body, when in bodies: + count += 1 + received = when + timedelta(seconds=r.randint(5, 600)) + if chance(r, 0.02): + body = body[: len(body) // 2] + row = { + "webhook_id": f"WH{count:07d}", + "carrier_code": s.carrier.lower() if chance(r, 0.1) else s.carrier, + "received_at": received, + "body": body, + } + log.add(row, received + timedelta(minutes=1)) + if chance(r, 0.04): + # The carrier retried the same delivery under a new webhook id. + count += 1 + log.add( + { + **row, + "webhook_id": f"WH{count:07d}", + "received_at": received + timedelta(minutes=5), + }, + received + timedelta(minutes=6), + ) + for index, body in enumerate(["", "", "null", " "], start=1): + count += 1 + when = at(date(2026, 3, 20 + index), 12) + log.add( + { + "webhook_id": f"WH{count:07d}", + "carrier_code": "FLAPWING", + "received_at": when, + "body": body or None, + }, + when, + ) + return log.rows() + + +RETURN_REASONS = [ + "Arrived damaged - beak snapped", + "wrong item: ordered TEAL got MALLARD", + "Too big for the bath (size L)", + "Not as described, colour is darker", + "changed my mind", + "Arrived 10 days late; no longer needed", + " other reason ", +] + + +@lru_cache(maxsize=4) +def return_plan( + scale: int, +) -> tuple[tuple[str, str, str, int, datetime, datetime | None, str], ...]: + world = build_world(scale) + r = rng("return_plan") + plan = [] + count = 0 + for order in world.orders: + returned = order.time_of("returned") + delivered = order.time_of("delivered") + if returned is None and not (delivered and chance(r, 0.02)): + continue + requested = ( + (returned or delivered) - timedelta(days=r.randint(1, 3)) + if returned + else delivered + timedelta(days=2) + ) # type: ignore[operator] + for line in order.lines[: r.randint(1, len(order.lines))]: + count += 1 + received = requested + timedelta(days=r.randint(2, 8)) if returned else None + if received and received.date() > date(2026, 3, 31): + received = None + plan.append( + ( + f"RT{count:05d}", + order.order_id, + line.sku, + line.quantity, + requested, + received, + WAREHOUSE_FOR_REGION[STOREFRONTS[order.storefront].region], + ) + ) + return tuple(plan) + + +def returns(world: World) -> Rows: + r = rng("fulfilment__returns") + log = ChangeLog("fulfilment__returns") + for return_id, order_id, sku, qty, requested, received, _ in return_plan(world.scale): + zone = STOREFRONTS[world.order_by_id[order_id].storefront].timezone + row = { + "return_id": return_id, + "order_id": order_id, + "sku": world.variant_by_sku[sku].raw_sku, + "qty": qty if chance(r, 0.95) else 0, + "reason_text": r.choice(RETURN_REASONS), + "return_status": "requested", + "requested_at": in_zone(requested, zone), + "received_at": None, + "photos": r.choice([None, [], ["https://media.example.com/returns/1.jpg"]]), + "refund_preference": r.choice( + [None, {"method": "Store_Credit"}, {"method": "original_payment"}] + ), + } + versions = [(requested, row)] + if received: + versions.append( + ( + received, + { + **row, + "return_status": r.choice(["received", "RECEIVED"]), + "received_at": in_zone(received, zone), + }, + ) + ) + emit(log, r, versions) + return log.rows() + + +def return_inspections(world: World) -> Rows: + r = rng("fulfilment__return_inspections") + log = ChangeLog("fulfilment__return_inspections") + count = 0 + for return_id, _, _, _, _, received, warehouse in return_plan(world.scale): + if received is None: + continue + for attempt in range(2 if chance(r, 0.1) else 1): + count += 1 + inspected = received + timedelta(hours=r.randint(2, 48) + attempt * 24) + row = { + "inspection_id": f"IN{count:05d}", + "return_id": return_id, + "warehouse_code": warehouse.lower() if chance(r, 0.2) else warehouse, + "grade": r.choice(["A", " b ", "C", "scrap", "B"]), + "notes": r.choice([None, "Scuffed paint on left wing", "As new"]), + "inspected_at_local": local_naive(inspected, WAREHOUSE[warehouse].timezone), + "findings": r.choice([{"defects": ["scuff", "paint"]}, {"defects": []}, None]), + } + emit(log, r, [(inspected, row)]) + return log.rows() + + +def carrier_rate_cards(world: World) -> Rows: + r = rng("fulfilment__carrier_rate_cards") + log = ChangeLog("fulfilment__carrier_rate_cards") + count = 0 + for carrier in CARRIERS: + services = ["freight"] if carrier == "WEBFOOT" else ["standard", "express", "next-day"] + for warehouse in WAREHOUSES: + for destination in STOREFRONTS: + for service in services: + windows: list[tuple[date, date | None, float]] = [(date(2025, 1, 1), None, 1.0)] + if chance(r, 0.2): + windows = [ + (date(2025, 1, 1), date(2026, 2, 28), 1.0), + (date(2026, 3, 1), None, 1.08), + ] + for start, end, uplift in windows: + count += 1 + base = 4.99 if destination == warehouse.country else 14.99 + base *= {"standard": 1.0, "express": 1.6, "next-day": 2.2, "freight": 4.0}[ + service + ] + bands = [ + {"max_kg": kg, "amount_minor": int(base * 100 * uplift * factor)} + for kg, factor in ((1, 1.0), (5, 1.5), (20, 2.4), (70.5, 5.0)) + ] + row = { + "rate_card_id": f"RC{count:05d}", + "carrier_code": carrier, + "service_level": service.upper() if chance(r, 0.1) else service, + "origin_country": warehouse.country, + "destination_country": destination.lower() + if chance(r, 0.1) + else destination, + "currency_code": "GBP" + if warehouse.code == "POND" + else ("USD" if warehouse.code == "NEST" else "AUD"), + "valid_from": start, + "valid_to": end, + "bands": bands, + } + emit(log, r, [(at(start), row)], duplicate_share=0.0) + return log.rows() + + +def delivery_slots(world: World) -> Rows: + r = rng("fulfilment__delivery_slots") + log = ChangeLog("fulfilment__delivery_slots") + count = 0 + day = date(2026, 3, 9) + while day <= date(2026, 4, 5): + for warehouse in WAREHOUSES: + for hour in ( + range(0, 22, 2) + if warehouse.code == "POND" and day == date(2026, 3, 29) + else range(8, 20, 2) + ): + count += 1 + start = datetime.combine(day, datetime.min.time()) + timedelta(hours=hour) + capacity = r.choice([20, 30, 40]) + row = { + "slot_id": f"DS{count:06d}", + "warehouse_code": warehouse.code, + "slot_start_local": start, + "slot_end_local": start + timedelta(hours=2), + "capacity": capacity, + "booked": None + if chance(r, 0.05) + else min(capacity + r.choice([0, 0, 0, 3]), r.randint(0, capacity + 3)), + "restrictions": r.choice( + [None, {"excluded_postcode_prefixes": ["BT", "IV"]}, {}] + ), + } + versions = [(start - timedelta(days=14), {**row, "booked": 0})] + if start.date() <= date(2026, 3, 31): + versions.append((start - timedelta(hours=2), row)) + emit(log, r, versions) + day += timedelta(days=1) + return log.rows() + + +def pick_tasks(world: World) -> Rows: + r = rng("fulfilment__pick_tasks") + log = ChangeLog("fulfilment__pick_tasks") + count = 0 + for s in shipment_plan(world.scale): + order = world.order_by_id[s.order_id] + packed = order.time_of("packed") + if packed is None: + continue + count += 1 + zone = WAREHOUSE[s.warehouse].timezone + started = packed - timedelta(minutes=r.randint(5, 50)) + duration = timedelta(seconds=r.randint(60, 1500)) + if chance(r, 0.01): + duration = -duration + row = { + "task_id": f"PT{count:06d}", + "order_id": s.order_id, + "warehouse_code": s.warehouse, + "picker_id": f"picker-{r.randint(1, 12):02d}", + "started_at_local": local_naive(started, zone), + "completed_at_local": local_naive(started + duration, zone) + if chance(r, 0.98) + else None, + "lines": [ + {"sku": sku, "qty": qty if chance(r, 0.9) else str(qty)} for sku, qty in s.line_skus + ], + } + emit( + log, r, [(started, {**row, "completed_at_local": None}), (started + abs(duration), row)] + ) + return log.rows() + + +EXCEPTIONS = [ + ("ADDR_INVALID", "Address incomplete - flat number missing"), + ("not_home", "Recipient not home, card left"), + ("REFUSED", "Customer refused delivery"), + ("LOST", "Parcel lost in network"), + ("DAMAGED_IN_TRANSIT", "Box crushed; contents damaged"), + ("DELAY_WEATHER", "Storm delay at hub"), + ("LABEL_UNREADABLE", "Label smudged, re-labelled at depot"), + ("CUSTOMS_HOLD", "Held at customs pending paperwork"), +] + + +def delivery_exceptions(world: World) -> Rows: + r = rng("fulfilment__delivery_exceptions") + log = ChangeLog("fulfilment__delivery_exceptions") + count = 0 + for s in shipment_plan(world.scale): + if s.shipped_at is None or not chance(r, 0.06): + continue + count += 1 + code, text = r.choice(EXCEPTIONS) + when = s.shipped_at + timedelta(hours=r.randint(6, 60)) + if when.date() > date(2026, 3, 31): + continue + resolution: dict[str, object] | None = None + versions_at = [when] + row = { + "exception_id": f"EX{count:05d}", + "shipment_id": s.shipment_id, + "exception_code": code, + "exception_text": f" {text} ", + "occurred_at": in_zone(when, "UTC"), + "resolution": resolution, + } + versions = [(when, row)] + resolved = when + timedelta(hours=r.randint(4, 72)) + if resolved.date() <= date(2026, 3, 31) and chance(r, 0.7): + versions_at.append(resolved) + versions.append( + ( + resolved, + { + **row, + "resolution": { + "type": r.choice(["Redelivered", "refunded", " "]), + "at": in_zone(resolved, "UTC").isoformat() + if chance(r, 0.9) + else "soon", + "compensation_minor": r.choice([0, 500, "250", None]), + }, + }, + ) + ) + emit(log, r, versions) + return log.rows() + + +TABLES = { + "fulfilment__shipments": shipments, + "fulfilment__shipment_packages": shipment_packages, + "fulfilment__carrier_webhooks": carrier_webhooks, + "fulfilment__returns": returns, + "fulfilment__return_inspections": return_inspections, + "fulfilment__carrier_rate_cards": carrier_rate_cards, + "fulfilment__delivery_slots": delivery_slots, + "fulfilment__pick_tasks": pick_tasks, + "fulfilment__delivery_exceptions": delivery_exceptions, +} diff --git a/benchmarks/duckshop/project/python/loaders/_growth.py b/benchmarks/duckshop/project/python/loaders/_growth.py new file mode 100644 index 0000000000..2a4dab63ea --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_growth.py @@ -0,0 +1,296 @@ +"""Growth change logs: campaigns, spend, affiliates, referrals, newsletter, surveys and creators.""" + +from __future__ import annotations + +from datetime import date, timedelta + +from python.loaders._core import ChangeLog, at, chance, emit, in_zone, random_instant, rng +from python.loaders._world import CAMPAIGNS, STOREFRONTS, World + +Rows = list[dict[str, object]] +SPEND_CURRENCY = { + "paid_search": "USD", + "paid_social": "EUR", + "email": "GBP", + "influencer": "GBP", + "affiliate": "GBP", +} +AFFILIATES = ["aff_blog_01", "AFF_REVIEWS_02", "aff-weekly-03", "aff_giftguide_04"] + + +def campaigns(world: World) -> Rows: + r = rng("growth__campaigns") + log = ChangeLog("growth__campaigns") + for campaign_id, name, channel, starts, ends, budget, _ in CAMPAIGNS: + targeting: dict[str, object] | None = r.choice( + [ + {"regions": ["EU", "US"], "age_band": "25-54"}, + {"regions": ["EU", "APAC", "LATAM"], "age_band": "18+"}, + {"age_band": "35-"}, + None, + ] + ) + row = { + "campaign_id": campaign_id, + "campaign_name": f" {name} " if chance(r, 0.2) else name, + "channel": channel, + "objective": r.choice(["Conversions", "awareness", "TRAFFIC"]), + "starts_on": starts, + "ends_on": ends, + "budget_minor": budget, + "currency_code": "gbp" if chance(r, 0.2) else "GBP", + "targeting": targeting, + } + versions = [(at(starts) - timedelta(days=10), row)] + if campaign_id == "SPRING-SPLASH": + versions.append((at(date(2026, 3, 15)), {**row, "budget_minor": budget + 100000})) + emit(log, r, versions) + return log.rows() + + +def campaign_spend(world: World) -> Rows: + r = rng("growth__campaign_spend") + log = ChangeLog("growth__campaign_spend") + count = 0 + for campaign_id, _, channel, starts, ends, budget, _ in CAMPAIGNS: + if channel in ("email", "affiliate"): + continue + day = max(starts, date(2026, 1, 1)) + last = min(ends or date(2026, 3, 31), date(2026, 3, 31)) + daily = budget / max((last - day).days + 1, 1) / 100 + while day <= last: + count += 1 + spend = round(daily * r.uniform(0.6, 1.3), 2) + impressions = int(spend * r.randint(80, 200)) + clicks = int(impressions * r.uniform(0.005, 0.03)) + metrics: list[dict[str, object]] = [ + {"name": "spend", "value": f"{spend:,.2f}" if chance(r, 0.4) else spend}, + {"name": "impressions", "value": impressions}, + {"name": "linkClicks" if channel == "paid_social" else "clicks", "value": clicks}, + {"name": "conversions", "value": round(clicks * r.uniform(0.01, 0.05), 3)}, + ] + if chance(r, 0.1): + metrics.append({"name": "videoViews", "value": str(r.randint(10, 900))}) + if chance(r, 0.02): + metrics[0]["value"] = "n/a" + row = { + "spend_id": f"SP{count:05d}", + "spend_date": day, + "channel": channel, + "campaign_id": campaign_id if chance(r, 0.97) else None, + "currency_code": SPEND_CURRENCY[channel], + "report": { + "account": {"id": r.choice([123456789, "act_9876"])}, + "metrics": metrics, + }, + } + versions = [(at(day + timedelta(days=1), 3), row)] + if chance(r, 0.1): + # Ad platforms restate spend a few days later. + restated = [dict(m) for m in metrics] + restated[0]["value"] = round(spend * 1.02, 2) + versions.append( + ( + at(day + timedelta(days=4), 3), + {**row, "report": {**row["report"], "metrics": restated}}, + ) + ) # type: ignore[dict-item] + emit(log, r, versions) + day += timedelta(days=1) + return log.rows() + + +def affiliate_conversions(world: World) -> Rows: + r = rng("growth__affiliate_conversions") + log = ChangeLog("growth__affiliate_conversions") + count = 0 + for order in world.orders: + if ( + order.utm is None + or order.utm[1] != "affiliate" + or order.status in ("pending", "cancelled") + ): + continue + count += 1 + clicked = order.placed_at - timedelta(hours=r.randint(0, 200), minutes=r.randint(0, 59)) + sf = STOREFRONTS[order.storefront] + row = { + "conversion_id": f"AF{count:05d}", + "affiliate_code": r.choice(AFFILIATES), + "order_id": order.order_id, + "commission_minor": int(order.net_minor * 0.08), + "currency_code": sf.currency.lower() if chance(r, 0.2) else sf.currency, + "converted_at": in_zone(order.placed_at, sf.timezone), + "click_payload": { + "clicked_at": in_zone(clicked, "UTC").isoformat() if chance(r, 0.95) else "unknown", + "landing_url": "https://www.duckshop.example.com/?utm_source=partner_site" + f"&subid={r.choice(['abc123', 'spring', 'x'])}", + }, + } + versions = [(order.placed_at, row)] + if order.status == "returned": + versions.append((order.time_of("returned"), {**row, "commission_minor": 0})) # type: ignore[list-item] + emit(log, r, versions) + return log.rows() + + +def referral_invites(world: World) -> Rows: + r = rng("growth__referral_invites") + log = ChangeLog("growth__referral_invites") + customers = [c for c in world.customers if not c.deleted and c.email] + for i in range(1, 90 * world.scale + 1): + referrer = r.choice(customers) + sent = random_instant(r, date(2026, 1, 1), date(2026, 3, 25)) + invitee = r.choice(customers) if chance(r, 0.4) else None + email = invitee.email if invitee else f"friend{i}@example.com" + accepted = None + if invitee is not None and invitee.created_at > sent: + accepted = invitee.created_at + elif chance(r, 0.2): + accepted = sent + timedelta(days=r.randint(1, 5)) + sf = STOREFRONTS[referrer.storefront] + row = { + "invite_id": f"RI{i:05d}", + "referrer_customer_id": referrer.customer_id, + "invitee_email": email, + "sent_at": in_zone(sent, sf.timezone), + "accepted_at": None, + "reward": {"amount_minor": 10 * 10**sf.exponent, "currency": sf.currency.lower()} + if chance(r, 0.9) + else None, + } + versions = [(sent, row)] + if accepted is not None and accepted.date() <= date(2026, 3, 31): + versions.append((accepted, {**row, "accepted_at": in_zone(accepted, sf.timezone)})) + emit(log, r, versions) + return log.rows() + + +def newsletter_subscriptions(world: World) -> Rows: + r = rng("growth__newsletter_subscriptions") + log = ChangeLog("growth__newsletter_subscriptions") + count = 0 + pool = [c.email for c in world.customers if c.email] + [ + f"reader{i}@example.net" for i in range(60 * world.scale) + ] + for email in pool: + if not chance(r, 0.45): + continue + for list_code in r.sample(["weekly", "offers", "collectors"], k=r.randint(1, 2)): + count += 1 + subscribed = random_instant(r, date(2025, 6, 1), date(2026, 3, 25)) + row = { + "list_subscription_id": f"NL{count:06d}", + "email": email, + "list_code": list_code, + "list_status": r.choice(["pending", "Subscribed"]), + "subscribed_at": in_zone(subscribed, "UTC"), + "unsubscribed_at": None, + "signup_source": r.choice( + [ + {"form": "footer", "double_opt_in": True}, + {"form": " "}, + {"form": "checkout", "double_opt_in": "false"}, + None, + ] + ), + } + versions = [(subscribed, row)] + if row["list_status"] == "pending": + versions.append( + ( + subscribed + timedelta(hours=r.randint(1, 48)), + {**row, "list_status": "subscribed"}, + ) + ) + if chance(r, 0.12): + gone = subscribed + timedelta(days=r.randint(3, 120)) + if gone.date() <= date(2026, 3, 31): + versions.append( + ( + gone, + { + **versions[-1][1], + "list_status": r.choice(["unsubscribed", "CLEANED"]), + "unsubscribed_at": in_zone(gone, "UTC"), + }, + ) + ) + emit(log, r, versions) + return log.rows() + + +def survey_responses(world: World) -> Rows: + r = rng("growth__survey_responses") + log = ChangeLog("growth__survey_responses") + count = 0 + for order in world.orders: + delivered = order.time_of("delivered") + if delivered is None or not chance(r, 0.12): + continue + count += 1 + when = delivered + timedelta(days=r.randint(1, 7)) + if when.date() > date(2026, 3, 31): + continue + answers: dict[str, object] = { + "nps": r.choice([10, 9, "8", 7, 3, "11", None, "n/a"]), + "delivery_rating": r.choice([5, 4, 2]), + "comment": r.choice(["Lovely ducks", "", None, "Box was soggy"]), + } + if chance(r, 0.3): + answers["would_buy_again"] = r.choice([True, "yes"]) + row = { + "response_id": f"SV{count:05d}", + "survey_code": r.choice(["post-delivery", "NPS-2026Q1"]), + "customer_id": order.customer_id, + "submitted_at": in_zone(when, STOREFRONTS[order.storefront].timezone), + "answers": answers, + } + emit(log, r, [(when, row)]) + return log.rows() + + +def influencer_posts(world: World) -> Rows: + r = rng("growth__influencer_posts") + log = ChangeLog("growth__influencer_posts") + count = 0 + for handle in ["@creator_001", "creator_002", "@Creator_003", "@creator_004"]: + for _ in range(r.randint(3, 8)): + count += 1 + posted = random_instant(r, date(2026, 3, 1), date(2026, 3, 30)) + impressions = r.randint(0, 80000) + metrics: dict[str, object] | None = { + "impressions": impressions, + "likes": int(impressions * r.uniform(0.01, 0.08)), + "comments": r.randint(0, 300), + "shares": r.choice([None, r.randint(0, 120)]), + } + if chance(r, 0.05): + metrics = None + row = { + "post_id": f"IP{count:04d}", + "creator_handle": handle, + "platform": r.choice(["video_app", "PHOTO_APP", "blog"]), + "posted_at": in_zone(posted, "Europe/London"), + "promo_code": r.choice(["GOLDENEGG", " goldenegg ", None]), + "metrics": metrics, + } + versions = [(posted, row)] + if metrics is not None and chance(r, 0.5): + later = posted + timedelta(days=2) + versions.append( + (later, {**row, "metrics": {**metrics, "impressions": impressions * 2}}) + ) + emit(log, r, versions) + return log.rows() + + +TABLES = { + "growth__campaigns": campaigns, + "growth__campaign_spend": campaign_spend, + "growth__affiliate_conversions": affiliate_conversions, + "growth__referral_invites": referral_invites, + "growth__newsletter_subscriptions": newsletter_subscriptions, + "growth__survey_responses": survey_responses, + "growth__influencer_posts": influencer_posts, +} diff --git a/benchmarks/duckshop/project/python/loaders/_inventory.py b/benchmarks/duckshop/project/python/loaders/_inventory.py new file mode 100644 index 0000000000..a83723dbff --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_inventory.py @@ -0,0 +1,496 @@ +"""Inventory change logs and feeds: warehouses, stock movements, counts, purchasing, transfers.""" + +from __future__ import annotations + +from collections import defaultdict +from datetime import date, datetime, time, timedelta +from functools import lru_cache + +from python.loaders._catalogue import SUPPLIER_CURRENCY, SUPPLIER_FOR_FAMILY +from python.loaders._core import ( + ChangeLog, + EventLog, + at, + chance, + emit, + in_zone, + local_naive, + messy_sku, + rng, +) +from python.loaders._fulfilment import WAREHOUSE, return_plan, shipment_plan +from python.loaders._world import ( + CURRENCY_EXPONENT, + STOREFRONTS, + WAREHOUSE_FOR_REGION, + WAREHOUSES, + World, + build_world, +) + +Rows = list[dict[str, object]] +STOCK_START = date(2026, 1, 1) + + +def warehouses(world: World) -> Rows: + r = rng("inventory__warehouses") + log = ChangeLog("inventory__warehouses") + details = { + "POND": ( + time(7, 0), + time(22, 0), + { + "services": ["same_day", "next_day"], + "cold_chain": False, + "limits": {"max_parcel_kg": 31.5}, + }, + ), + "NEST": ( + time(6, 30), + time(23, 30), + {"services": ["next_day"], "limits": {"max_parcel_kg": "68.04"}}, + ), + "MARSH": (time(8, 0), None, {"services": [], "cold_chain": "true"}), + } + for w in WAREHOUSES: + opens, closes, capabilities = details[w.code] + row = { + "warehouse_code": w.code if w.code != "MARSH" else "marsh", + "warehouse_name": w.name, + "country_code": w.country, + "iana_timezone": w.timezone, + "opens_local": opens, + "closes_local": closes, + "capabilities": capabilities, + } + versions = [(at(date(2025, 6, 1)), row)] + if w.code == "NEST": + versions.append( + ( + at(date(2026, 2, 16), 14), + { + **row, + "closes_local": time(23, 59), + "capabilities": {**capabilities, "services": ["next_day", "same_day"]}, + }, + ) + ) + emit(log, r, versions, duplicate_share=0.0) + closed = { + "warehouse_code": "REED", + "warehouse_name": "The Reed Bed (closed)", + "country_code": "NL", + "iana_timezone": "Europe/Amsterdam", + "opens_local": time(8, 0), + "closes_local": time(17, 0), + "capabilities": None, + } + emit( + log, + r, + [(at(date(2025, 6, 1)), closed)], + delete_at=at(date(2025, 12, 31)), + duplicate_share=0.0, + ) + return log.rows() + + +def _stocked(world: World) -> list[tuple[str, str]]: + """(sku, warehouse) pairs carried in stock; The Marsh only carries a smaller range.""" + + r = rng("stocked") + pairs = [] + for variant in world.variants: + for w in WAREHOUSES: + if w.code == "MARSH" and not chance(r, 0.5): + continue + pairs.append((variant.sku, w.code)) + return pairs + + +@lru_cache(maxsize=4) +def movement_plan( + scale: int, +) -> tuple[tuple[str, str, str, int, str, str, datetime, str | None], ...]: + """(movement_id, sku, warehouse, qty_delta, reason, reference, when in UTC, operator).""" + + world = build_world(scale) + r = rng("movement_plan") + plan = [] + count = 0 + + def add(sku: str, warehouse: str, qty: int, reason: str, reference: str, when: datetime) -> str: + nonlocal count + count += 1 + movement_id = f"MV{count:07d}" + plan.append( + ( + movement_id, + sku, + warehouse, + qty, + reason, + reference, + when, + None if chance(r, 0.1) else f"op-{r.randint(1, 9)}", + ) + ) + return movement_id + + for sku, warehouse in _stocked(world): + # Some lines open with very little stock so they sell out for days at a time. + opening = r.choice([3, 5, 40, 80, 150]) if chance(r, 0.85) else 0 + add(sku, warehouse, opening, "opening", "stocktake:2026-01", at(STOCK_START, 0, 30)) + for shipment in shipment_plan(scale): + when = shipment.shipped_at or world.order_by_id[shipment.order_id].time_of("packed") + for sku, qty in shipment.line_skus: + add( + sku, + shipment.warehouse, + -qty, + "sale", + f"order:{shipment.order_id}", + when - timedelta(minutes=30), + ) # type: ignore[operator] + for _, order_id, sku, qty, _, received, warehouse in return_plan(scale): + if received is not None: + add(sku, warehouse, qty, "return", f"order:{order_id}", received + timedelta(hours=3)) + for sku, warehouse in _stocked(world): + if chance(r, 0.15): + when = at(STOCK_START + timedelta(days=r.randint(5, 85)), r.randint(8, 18)) + moved = add( + sku, + warehouse, + r.choice([-1, -2, 1, -5]), + r.choice(["adjust", "damage", "Damage"]), + "", + when, + ) + if chance(r, 0.2): + add(sku, warehouse, 0, "void", f"void:{moved}", when + timedelta(hours=2)) + # Movements during the spring-forward gap in London: 01:30 local does not exist on 29 March. + for minute in (15, 45): + add( + world.variants[0].sku, + "POND", + -1, + "damage", + "note:dst-gap", + datetime(2026, 3, 29, 1, minute), + ) + return tuple(plan) + + +def stock_movements(world: World) -> Rows: + r = rng("inventory__stock_movements") + log = EventLog("inventory__stock_movements") + for movement_id, sku, warehouse, qty, reason, reference, when, operator in movement_plan( + world.scale + ): + # The DST-gap movements carry wall-clock times that London skips when clocks go forward. + local = ( + when + if reference == "note:dst-gap" + else local_naive(when, WAREHOUSE[warehouse].timezone) + ) + row = { + "movement_id": movement_id, + "sku": messy_sku(r, sku), + "warehouse_code": warehouse if chance(r, 0.9) else warehouse.lower(), + "qty_delta": qty, + "reason_code": reason, + "reference": reference or None, + "moved_at_local": local, + "metadata": {"operator": operator} if operator else None, + } + loaded = when + timedelta(minutes=r.randint(1, 30)) + if chance(r, 0.03): + loaded += timedelta(days=r.randint(1, 3)) + log.add(row, loaded) + if chance(r, 0.02): + log.add(row, loaded + timedelta(hours=1)) + return log.rows() + + +def stock_counts(world: World) -> Rows: + r = rng("inventory__stock_counts") + log = ChangeLog("inventory__stock_counts") + balance: dict[tuple[str, str], int] = defaultdict(int) + for _, sku, warehouse, qty, _, _, _, _ in movement_plan(world.scale): + balance[(sku, warehouse)] += qty + count = 0 + for (sku, warehouse), on_hand in sorted(balance.items()): + if not chance(r, 0.3): + continue + for week in range(r.randint(1, 3)): + count += 1 + when = at( + date(2026, 2, 2) + timedelta(days=7 * week * 3 + r.randint(0, 4)), r.randint(6, 20) + ) + counted = max(on_hand, 0) + r.choice([0, 0, 0, -1, 2, -3]) + if chance(r, 0.02): + counted = -1 + row = { + "count_id": f"CT{count:06d}", + "sku": sku, + "warehouse_code": warehouse, + "counted_qty": counted, + "counted_at_local": local_naive(when, WAREHOUSE[warehouse].timezone), + "counter": r.choice( + [{"id": f"op-{r.randint(1, 9)}", "blind": True}, {"id": None}, None] + ), + } + emit(log, r, [(when, row)]) + return log.rows() + + +@lru_cache(maxsize=4) +def purchase_plan(scale: int): + world = build_world(scale) + r = rng("purchase_plan") + by_supplier: dict[str, list[str]] = defaultdict(list) + for variant in world.variants: + by_supplier[SUPPLIER_FOR_FAMILY[variant.family][0]].append(variant.sku) + plan = [] + count = 0 + for supplier, skus in sorted(by_supplier.items()): + for month in (1, 2, 3): + for warehouse in ("POND", "NEST"): + count += 1 + ordered = at(date(2026, month, r.randint(1, 10)), r.randint(8, 17)) + expected = ordered + timedelta(days=r.randint(10, 25)) + lines = [ + (sku, r.choice([24, 48, 100, 250])) + for sku in r.sample(skus, k=min(len(skus), r.randint(2, 6))) + ] + if chance(r, 0.15): + lines.append(lines[0]) + received = expected + timedelta(days=r.randint(-3, 9)) + status = "received" if received.date() <= date(2026, 3, 31) else "open" + if chance(r, 0.05): + status = "cancelled" + plan.append( + ( + f"PO{count:05d}", + supplier, + warehouse, + ordered, + expected, + received if status == "received" else None, + status, + tuple(lines), + ) + ) + return tuple(plan) + + +def purchase_orders(world: World) -> Rows: + r = rng("inventory__purchase_orders") + log = ChangeLog("inventory__purchase_orders") + variants = world.variant_by_sku + for po_id, supplier, warehouse, ordered, expected, received, status, lines in purchase_plan( + world.scale + ): + currency = SUPPLIER_CURRENCY[supplier] + zone = WAREHOUSE[warehouse].timezone + line_rows: list[dict[str, object]] = [] + for sku, qty in lines: + cost = int(variants[sku].base_usd * 0.35 * 10 ** CURRENCY_EXPONENT[currency]) + line: dict[str, object] = {"sku": variants[sku].raw_sku, "qty": qty} + if chance(r, 0.8): + line["unit_cost_minor"] = cost + else: + line["unit_cost"] = cost + if chance(r, 0.1): + line["currency"] = currency.lower() + line_rows.append(line) + row = { + "po_id": po_id, + "supplier_code": supplier, + "warehouse_code": warehouse, + "currency_code": currency, + "po_status": "open", + "ordered_at": in_zone(ordered, zone), + "expected_at": in_zone(expected, zone), + "received_at": None, + "lines": line_rows, + } + versions = [(ordered, row)] + if status == "received": + versions.append( + ( + received, + { + **row, + "po_status": r.choice(["Received", "received"]), + "received_at": in_zone(received, zone), + }, + ) + ) + elif status == "cancelled": + versions.append((ordered + timedelta(days=2), {**row, "po_status": "CANCELLED"})) + emit(log, r, versions) + return log.rows() + + +def po_receipts(world: World) -> Rows: + r = rng("inventory__po_receipts") + log = ChangeLog("inventory__po_receipts") + count = 0 + for po_id, _, warehouse, _, _, received, status, lines in purchase_plan(world.scale): + if status != "received" or received is None: + continue + seen: set[str] = set() + for sku, qty in lines: + if sku in seen: + continue + seen.add(sku) + for part in range(2 if chance(r, 0.1) else 1): + count += 1 + when = received + timedelta(hours=part * 30) + damaged = r.choice([None, 0, 0, 1, 3]) + short = chance(r, 0.1) + row = { + "receipt_id": f"RC{count:06d}", + "po_id": po_id, + "sku": messy_sku(r, sku), + "warehouse_code": warehouse, + "received_qty": (qty - r.randint(1, 5) if short else qty) // (2 if part else 1), + "damaged_qty": damaged, + "received_at_local": local_naive(when, WAREHOUSE[warehouse].timezone), + "discrepancies": {"codes": ["short"]} + if short + else r.choice([None, {"codes": []}]), + } + emit(log, r, [(when, row)]) + return log.rows() + + +def stock_transfers(world: World) -> Rows: + r = rng("inventory__stock_transfers") + log = ChangeLog("inventory__stock_transfers") + for i in range(1, 40 * world.scale + 1): + variant = r.choice(world.variants) + source, target = r.sample(["POND", "NEST", "MARSH"], k=2) + shipped = at(date(2026, 1, 10) + timedelta(days=r.randint(0, 80)), r.randint(6, 18)) + arrived = shipped + timedelta(days=r.randint(3, 12), hours=r.randint(0, 23)) + planned = chance(r, 0.1) + row = { + "transfer_id": f"TF{i:04d}", + "sku": variant.raw_sku, + "from_warehouse_code": source, + "to_warehouse_code": target.lower() if chance(r, 0.1) else target, + "qty": r.choice([12, 24, 48]), + "shipped_at": None, + "arrived_at": None, + "tracking": None, + } + versions = [(shipped - timedelta(days=2), row)] + if not planned: + versions.append( + ( + shipped, + { + **row, + "shipped_at": in_zone(shipped, WAREHOUSE[source].timezone), + "tracking": { + "reference": r.choice( + [f"HL{r.randint(100000, 9999999)}", "pending", None] + ) + }, + }, + ) + ) + if arrived.date() <= date(2026, 3, 31): + versions.append( + ( + arrived, + { + **versions[-1][1], + "arrived_at": in_zone(arrived, WAREHOUSE[target].timezone), + }, + ) + ) + emit(log, r, versions) + return log.rows() + + +def stock_reservations(world: World) -> Rows: + r = rng("inventory__stock_reservations") + log = ChangeLog("inventory__stock_reservations") + count = 0 + for order in world.orders: + if order.placed_at.date() < date(2026, 2, 1): + continue + warehouse = WAREHOUSE_FOR_REGION[STOREFRONTS[order.storefront].region] + released = order.time_of("shipped") or order.time_of("cancelled") + for line in order.lines: + count += 1 + row = { + "reservation_id": f"RS{count:07d}", + "order_id": order.order_id, + "sku": world.variant_by_sku[line.sku].raw_sku, + "warehouse_code": warehouse, + "qty": line.quantity, + "reserved_at": in_zone(order.placed_at, "UTC"), + "released_at": None, + "allocation": r.choice( + [ + {"rule": "FIFO", "backorder": False}, + {"rule": "nearest", "backorder": "true"}, + None, + ] + ), + } + versions = [(order.placed_at, row)] + if released is not None: + versions.append((released, {**row, "released_at": in_zone(released, "UTC")})) + emit(log, r, versions) + return log.rows() + + +def bin_locations(world: World) -> Rows: + r = rng("inventory__bin_locations") + log = ChangeLog("inventory__bin_locations") + count = 0 + for w in WAREHOUSES: + for aisle in ("A", "B", "c"): + for bay in range(1, 6): + for level in (1, 2): + count += 1 + row = { + "bin_id": f"BIN{count:05d}", + "warehouse_code": w.code.lower() if chance(r, 0.1) else w.code, + "bin_label": f"{aisle}-{bay:02d}-{level}" + if chance(r, 0.95) + else f"{aisle}{bay}", + "attributes": r.choice( + [ + {"pick_face": level == 1, "max_weight_kg": 25.5}, + {"temperature": "chilled"}, + {"max_weight_kg": "40"}, + None, + ] + ), + } + delete_at = at(date(2026, 3, 1)) if chance(r, 0.03) else None + emit( + log, + r, + [(at(date(2025, 6, 1)), row)], + delete_at=delete_at, + duplicate_share=0.0, + ) + return log.rows() + + +TABLES = { + "inventory__warehouses": warehouses, + "inventory__stock_movements": stock_movements, + "inventory__stock_counts": stock_counts, + "inventory__purchase_orders": purchase_orders, + "inventory__po_receipts": po_receipts, + "inventory__stock_transfers": stock_transfers, + "inventory__stock_reservations": stock_reservations, + "inventory__bin_locations": bin_locations, +} diff --git a/benchmarks/duckshop/project/python/loaders/_partners.py b/benchmarks/duckshop/project/python/loaders/_partners.py new file mode 100644 index 0000000000..2f3ddbad5f --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_partners.py @@ -0,0 +1,190 @@ +"""Marketplace partner feeds: listings, order messages in two payload versions, and settlements.""" + +from __future__ import annotations + +import hashlib +from datetime import date, timedelta + +from python.loaders._core import ChangeLog, EventLog, at, chance, emit, in_zone, random_instant, rng +from python.loaders._world import CURRENCY_EXPONENT, PER_USD, World + +Rows = list[dict[str, object]] +# Partner code, payload version, currency. +PARTNERS = [("MKT_EU", 1, "EUR"), ("mkt_global", 2, "USD"), ("GIFTBOX", 1, "KWD")] + + +def clean_email(email: str | None) -> str | None: + """The staging layer's email normalisation, for hashing buyer emails the way partners do.""" + + if email is None: + return None + lowered = email.strip().lower() + if "@" not in lowered or " " in lowered: + return None + local, _, domain = lowered.partition("@") + return f"{local.split('+', 1)[0]}@{domain}" + + +def partner_listings(world: World) -> Rows: + r = rng("partners__partner_listings") + log = ChangeLog("partners__partner_listings") + statuses = [ + "Live", + "active", + "on sale", + "on_sale", + "out_of_stock", + "error: image too small", + "pending review", + "archived", + "draft", + "mystery", + ] + count = 0 + for code, _, _ in PARTNERS: + for variant in r.sample(world.variants, k=min(len(world.variants), 30 * world.scale)): + count += 1 + listed = random_instant(r, date(2025, 11, 1), date(2026, 2, 28)) + row = { + "listing_id": f"LS{count:05d}", + "partner_code": code, + "sku": variant.raw_sku if chance(r, 0.95) else "UNKNOWN-SKU-01", + "listing_status": r.choice(statuses[:2]), + "listed_at": in_zone(listed, "UTC"), + "terms": r.choice( + [ + {"commission_pct": 12.5, "fulfilment": "partner"}, + {"commission_pct": "15", "fulfilment": "merchant"}, + None, + ] + ), + } + versions = [(listed, row)] + if chance(r, 0.4): + changed = listed + timedelta(days=r.randint(5, 60)) + versions.append((changed, {**row, "listing_status": r.choice(statuses)})) + emit(log, r, versions) + return log.rows() + + +def partner_orders(world: World) -> Rows: + r = rng("partners__partner_orders") + log = EventLog("partners__partner_orders") + buyers = [c for c in world.customers if clean_email(c.email)] + count = 0 + for code, version, currency in PARTNERS: + exponent = CURRENCY_EXPONENT[currency] + for n in range(1, 100 * world.scale + 1): + placed = random_instant(r, date(2026, 1, 1), date(2026, 3, 31)) + buyer = r.choice(buyers) if chance(r, 0.7) else None + email = buyer.email if buyer else f"guest{n}@example.org" + external_id = f"{code[:3].upper()}-{n:06d}" + items = [] + for variant in r.sample(world.variants, k=r.randint(1, 3)): + price = round(variant.base_usd * PER_USD[currency], 3 if exponent == 3 else 2) + items.append((variant, r.choice([1, 1, 2]), price)) + total = round(sum(qty * price for _, qty, price in items), 3) + if version == 1: + payload: dict[str, object] = { + "orderId": external_id, + "buyer": {"email": email}, + "total": { + "currency": currency.lower(), + "amount": f"{total:.3f}" if exponent == 3 else f"{total:.2f}", + }, + "created": in_zone(placed, "Europe/Paris").isoformat(), + "lines": [ + {"sku": v.raw_sku, "qty": qty, "price": str(price)} + for v, qty, price in items + ], + } + if chance(r, 0.1): + # The id moved into a nested object and the top-level key is a JSON null. + payload["orderId"] = None + payload["order"] = {"id": external_id} + else: + hashed = hashlib.sha256((clean_email(email) or "").encode()).hexdigest() + payload = { + "order": { + "id": external_id, + "currency": currency, + "totals": {"gross_minor": int(round(total * 10**exponent))}, + "placed_at_ms": int((placed - at(date(1970, 1, 1))).total_seconds() * 1000), + }, + "customer": {"email_hash": hashed.upper() if chance(r, 0.1) else hashed}, + "items": [ + { + "seller_sku": v.raw_sku, + "quantity": qty, + "unit_price_minor": int(round(price * 10**exponent)), + } + for v, qty, price in items + ], + } + if chance(r, 0.1): + payload["orderId"] = None + count += 1 + received = placed + timedelta(seconds=r.randint(5, 900)) + row = { + "message_id": f"MSG{count:07d}", + "partner_code": code, + "schema_version": version, + "received_at": received, + "payload": payload, + } + log.add(row, received + timedelta(minutes=2)) + if chance(r, 0.03): + log.add(row, received + timedelta(hours=4)) + if chance(r, 0.05): + # A corrected copy of the same order arrives later under a new message id. + count += 1 + later = received + timedelta(hours=r.randint(1, 30)) + log.add( + {**row, "message_id": f"MSG{count:07d}", "received_at": later}, + later + timedelta(minutes=2), + ) + return log.rows() + + +def partner_settlements(world: World) -> Rows: + r = rng("partners__partner_settlements") + log = ChangeLog("partners__partner_settlements") + count = 0 + for code, _, currency in PARTNERS: + exponent = CURRENCY_EXPONENT[currency] + for month in (1, 2, 3): + count += 1 + start = date(2026, month, 1) + end = (date(2026, month + 1, 1) if month < 12 else date(2027, 1, 1)) - timedelta(days=1) + gross = r.randint(2000, 9000) * 10**exponent + statement: dict[str, object] = { + "gross_sales_minor": gross, + "commission_minor": int(gross * r.choice([0.12, 0.15, 0.125])), + "fees": r.choice( + [ + [ + {"amount_minor": 120 * 10 ** max(exponent - 2, 0)}, + {"amount_minor": "35"}, + ], + [], + None, + ] + ), + } + row = { + "settlement_id": f"PS{count:04d}", + "partner_code": code, + "period_start": start, + "period_end": end, + "currency_code": currency, + "statement": statement, + } + emit(log, r, [(at(end + timedelta(days=3)), row)]) + return log.rows() + + +TABLES = { + "partners__partner_orders": partner_orders, + "partners__partner_listings": partner_listings, + "partners__partner_settlements": partner_settlements, +} diff --git a/benchmarks/duckshop/project/python/loaders/_registry.py b/benchmarks/duckshop/project/python/loaders/_registry.py new file mode 100644 index 0000000000..92f8f61e7a --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_registry.py @@ -0,0 +1,59 @@ +"""Every generated source table by name, for the loaders and for offline checks.""" + +from __future__ import annotations + +from collections.abc import Callable +from typing import Any + +from python.loaders import ( + _catalogue, + _clickstream, + _finance, + _fulfilment, + _growth, + _inventory, + _partners, + _storefront, + _support, +) +from python.loaders._world import World, build_world +from python.loaders._writer import statements + +SCALE_VAR = "data_scale" +"""Project variable that multiplies generated volume; 1 is the small default.""" + +GENERATORS: dict[str, Callable[[World], list[dict[str, object]]]] = { + **_catalogue.TABLES, + **_clickstream.TABLES, + **_finance.TABLES, + **_fulfilment.TABLES, + **_growth.TABLES, + **_inventory.TABLES, + **_partners.TABLES, + **_storefront.TABLES, + **_support.TABLES, +} + + +def generate(source_name: str, scale: int = 1) -> list[dict[str, object]]: + """The rows of one source table at one scale; one code version always returns the same rows.""" + + return GENERATORS[source_name](build_world(scale)) + + +def scale_from_vars(variables: dict[str, object]) -> int: + value = variables.get(SCALE_VAR, 1) + scale = int(value) # type: ignore[call-overload] + if scale < 1: + raise ValueError(f"{SCALE_VAR} must be a positive integer, got {value!r}") + return scale + + +def load(ctx: Any, source_name: str) -> object: + """Replace one source table with its generated rows, at the scale the project variables set.""" + + scale = scale_from_vars(ctx.vars) + rows = generate(source_name, scale) + for sql in statements(ctx.destination, source_name, rows): + ctx.execute_sql(sql) + return ctx.result(metadata={"rows": len(rows), "scale": scale}) diff --git a/benchmarks/duckshop/project/python/loaders/_storefront.py b/benchmarks/duckshop/project/python/loaders/_storefront.py new file mode 100644 index 0000000000..4f866271f6 --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_storefront.py @@ -0,0 +1,1231 @@ +"""Storefront change logs: customers, orders, payments, refunds, loyalty, gift cards and more.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import date, datetime, timedelta +from functools import lru_cache + +from python.loaders._core import ( + ChangeLog, + at, + chance, + emit, + in_zone, + random_instant, + rng, + weighted, +) +from python.loaders._world import ( + CITIES, + GEO, + STOREFRONTS, + Order, + World, +) + +Rows = list[dict[str, object]] +TIERS = ["DUCKLING", "DRAKE", "GOLDEN_DUCK", "PLATINUM_DUCK"] +STREETS = ["Mill Lane", "Reed Street", "Harbour Road", "Canal Walk"] + + +def zone_of(world: World, customer_id: str) -> str: + return STOREFRONTS[world.customer_by_id[customer_id].storefront].timezone + + +def customers(world: World) -> Rows: + r = rng("storefront__customers") + log = ChangeLog("storefront__customers") + for c in world.customers: + sf = STOREFRONTS[c.storefront] + zone = sf.timezone + language = {"DE": "de", "FR": "fr", "NL": "nl", "JP": "ja"}.get(c.storefront, "en") + consents = r.choice( + [["email"], ["email", "sms"], ["Email Marketing", "profiling"], [], None] + ) + profile: dict[str, object] | None = { + "preferences": { + "language": language, + "marketing_opt_in": r.choice([True, False, "true"]), + }, + "consents": consents, + "birth_year": r.choice([1975, 1988, "1992", "unknown", None]), + } + roll = r.random() + if roll < 0.08: + profile = {"preferences": None, "consents": consents} + elif roll < 0.12: + profile = None + region = sf.region if chance(r, 0.9) else sf.region.lower() + row = { + "customer_id": c.customer_id, + "email": c.email, + "phone": c.phone, + "full_name": f"{c.first_name} {c.last_name}" + if chance(r, 0.95) + else f" {c.first_name.lower()} {c.last_name.upper()}", + "region_code": region, + "storefront_code": c.storefront if chance(r, 0.9) else c.storefront.lower(), + "created_at": in_zone(c.created_at, zone), + "updated_at": in_zone(c.created_at, zone), + "profile": profile, + } + versions = [(c.created_at, row)] + if chance(r, 0.25): + changed = c.created_at + timedelta(days=r.randint(1, 60), hours=r.randint(0, 23)) + if changed.date() <= date(2026, 3, 31): + versions.append( + ( + changed, + { + **row, + "updated_at": in_zone(changed, zone), + "phone": c.phone if chance(r, 0.5) else None, + }, + ) + ) + delete_at = c.created_at + timedelta(days=r.randint(2, 30)) if c.deleted else None + emit(log, r, versions, delete_at=delete_at) + return log.rows() + + +def customer_addresses(world: World) -> Rows: + r = rng("storefront__customer_addresses") + log = ChangeLog("storefront__customer_addresses") + for c in world.customers: + for index, address_id in enumerate(c.addresses): + country = c.storefront + if index == 0 and len(c.addresses) > 1 and chance(r, 0.5): + country = r.choice(list(CITIES)) + city, postcode = r.choice(CITIES[country]) + if chance(r, 0.3): + postcode = postcode.lower().replace(" ", "").replace("-", "") + lat, lon = GEO[country] + geo: dict[str, object] | None = { + "lat": lat + r.uniform(-0.2, 0.2), + "lon": lon + r.uniform(-0.2, 0.2), + } + if chance(r, 0.1): + geo = {"lat": f"{lat:.4f}", "lon": None} + elif chance(r, 0.08): + geo = None + is_default = address_id == c.default_address + valid_from = c.created_at + timedelta(minutes=5 + index) + row = { + "address_id": address_id, + "customer_id": c.customer_id, + "line1": f" {r.randint(1, 220)} {r.choice(STREETS)} ", + "line2": r.choice([None, "", "Flat 2", " "]), + "city": r.choice([city, city.lower(), f" {city.upper()}"]), + "postcode": postcode, + "country_code": country if chance(r, 0.85) else country.lower(), + "is_default": True if is_default else r.choice([False, None]), + "valid_from": in_zone(valid_from, STOREFRONTS[c.storefront].timezone), + "geo": geo, + } + delete_at = None + if not is_default and chance(r, 0.3): + delete_at = valid_from + timedelta(days=r.randint(10, 90)) + if delete_at.date() > date(2026, 3, 31): + delete_at = None + emit(log, r, [(valid_from, row)], delete_at=delete_at) + return log.rows() + + +def customer_consents(world: World) -> Rows: + r = rng("storefront__customer_consents") + log = ChangeLog("storefront__customer_consents") + count = 0 + for c in world.customers: + if not chance(r, 0.55): + continue + for consent_type in r.sample( + ["email-marketing", "SMS marketing", "profiling", "email"], k=r.randint(1, 2) + ): + count += 1 + captured = c.created_at + timedelta(minutes=r.randint(1, 3000)) + row = { + "consent_id": f"CS{count:06d}", + "customer_id": c.customer_id, + "consent_type": consent_type, + "granted": r.choice([True, True, False, None]), + "captured_at": in_zone(captured, zone_of(world, c.customer_id)), + "source_context": r.choice( + [ + {"surface": "checkout", "policy_url": "https://example.com/privacy/v3.2"}, + {"surface": "account", "policy_url": "https://example.com/privacy/v4"}, + {"policy_url": None}, + None, + ] + ), + } + versions = [(captured, row)] + if chance(r, 0.15): + withdrawn = captured + timedelta(days=r.randint(5, 40)) + versions.append( + (withdrawn, {**row, "granted": False, "captured_at": in_zone(withdrawn, "UTC")}) + ) + emit(log, r, versions) + return log.rows() + + +@lru_cache(maxsize=4) +def _loyalty_members(scale: int) -> tuple[str, ...]: + from python.loaders._world import build_world + + r = rng("loyalty_members") + return tuple( + c.customer_id for c in build_world(scale).customers if not c.deleted and chance(r, 0.6) + ) + + +def loyalty_accounts(world: World) -> Rows: + r = rng("storefront__loyalty_accounts") + log = ChangeLog("storefront__loyalty_accounts") + members = set(_loyalty_members(world.scale)) + for c in world.customers: + if c.customer_id not in members: + continue + zone = zone_of(world, c.customer_id) + enrolled = c.created_at + timedelta(minutes=r.randint(1, 600)) + level = 0 + history: list[dict[str, object]] = [ + {"tier": "duckling", "at": in_zone(enrolled, zone).isoformat()} + ] + when = enrolled + for _ in range(r.randint(0, 3)): + when = when + timedelta(days=r.randint(10, 90)) + if when.date() > date(2026, 3, 31): + break + level = max(0, min(3, level + r.choice([1, 1, -1, 0]))) + stamp: str = in_zone(when, zone).isoformat() + if chance(r, 0.1): + stamp = when.strftime("%d/%m/%Y") + history.append( + {"tier": TIERS[level].lower() if chance(r, 0.5) else TIERS[level], "at": stamp} + ) + row = { + "loyalty_account_id": f"L{c.customer_id[1:]}", + "customer_id": c.customer_id, + "tier_code": TIERS[level] if chance(r, 0.9) else TIERS[level].lower(), + "points_balance": r.randint(-20, 4000) if chance(r, 0.95) else None, + "enrolled_at": in_zone(enrolled, zone), + "tier_history": history, + } + emit( + log, + r, + [(enrolled, row)] + + ([(when, {**row, "points_balance": r.randint(0, 5000)})] if len(history) > 1 else []), + ) + return log.rows() + + +def loyalty_ledger(world: World) -> Rows: + r = rng("storefront__loyalty_ledger") + log = ChangeLog("storefront__loyalty_ledger") + members = set(_loyalty_members(world.scale)) + count = 0 + for order in world.orders: + if order.customer_id not in members or order.status in ("pending", "cancelled"): + continue + count += 1 + paid = order.time_of("paid") or order.placed_at + row = { + "entry_id": f"LE{count:06d}", + "loyalty_account_id": f"L{order.customer_id[1:]}", + "points_delta": max(1, order.net_minor // (10**order.exponent)), + "reason_code": r.choice(["earn", "EARN", "order_earn"]), + "order_id": order.order_id, + "posted_at": in_zone(paid, "UTC"), + "metadata": {"expires_on": (paid.date() + timedelta(days=365)).isoformat()}, + } + emit(log, r, [(paid, row)]) + if chance(r, 0.12): + count += 1 + when = paid + timedelta(days=r.randint(1, 20)) + if when.date() <= date(2026, 3, 31): + emit( + log, + r, + [ + ( + when, + { + "entry_id": f"LE{count:06d}", + "loyalty_account_id": f"L{order.customer_id[1:]}", + "points_delta": -r.choice([100, 250, 500]), + "reason_code": "redeem", + "order_id": "", + "posted_at": in_zone(when, zone_of(world, order.customer_id)), + "metadata": { + "campaign": r.choice(["spring-splash", None, " "]), + "expires_on": "not-a-date", + }, + }, + ) + ], + ) + for customer_id in sorted(members)[::7]: + count += 1 + when = at(date(2026, 3, 1), 2) + emit( + log, + r, + [ + ( + when, + { + "entry_id": f"LE{count:06d}", + "loyalty_account_id": f"L{customer_id[1:]}", + "points_delta": -r.randint(1, 40), + "reason_code": "expire", + "order_id": None, + "posted_at": in_zone(when, "UTC"), + "metadata": None, + }, + ) + ], + ) + return log.rows() + + +def wishlists(world: World) -> Rows: + r = rng("storefront__wishlists") + log = ChangeLog("storefront__wishlists") + count = 0 + for c in world.customers: + if c.deleted or not chance(r, 0.3): + continue + count += 1 + created = c.created_at + timedelta(days=r.randint(0, 20)) + if created.date() > date(2026, 3, 30): + created = c.created_at + items: list[object] = [] + for variant in r.sample(world.variants, k=r.randint(0, 5)): + added = created + timedelta(hours=r.randint(0, 400)) + items.append({"sku": variant.raw_sku, "added_at": in_zone(added, "UTC").isoformat()}) + if chance(r, 0.1): + items.append("DUCK-MAL-01") + if chance(r, 0.1): + items.append({"sku": None, "added_at": None}) + row = { + "wishlist_id": f"W{count:05d}", + "customer_id": c.customer_id, + "created_at": in_zone(created, zone_of(world, c.customer_id)), + "items": items if items or chance(r, 0.5) else None, + "is_public": r.choice([True, False, None]), + } + versions = [(created, row)] + delete_at = created + timedelta(days=3) if chance(r, 0.05) else None + emit(log, r, versions, delete_at=delete_at) + return log.rows() + + +@dataclass(frozen=True) +class GiftCard: + gift_card_id: str + currency: str + exponent: int + initial_minor: int + issued_at: datetime + expires_on: date + purchaser: str | None + promotional: bool + + +@dataclass(frozen=True) +class GiftTender: + order_id: str + gift_card_id: str + amount_minor: int + at: datetime + + +@lru_cache(maxsize=4) +def gift_cards_and_tender(scale: int) -> tuple[tuple[GiftCard, ...], tuple[GiftTender, ...]]: + """Gift cards and the orders part-paid with them; payments and refunds read this too.""" + + from python.loaders._world import build_world + + world = build_world(scale) + r = rng("gift_cards") + cards: list[GiftCard] = [] + live = [c for c in world.customers if not c.deleted] + for i in range(1, 60 * scale + 1): + purchaser = r.choice(live) + sf = STOREFRONTS[purchaser.storefront] + issued = random_instant(r, date(2025, 3, 1), date(2026, 3, 20)) + amount = r.choice([25, 50, 100]) * 10**sf.exponent + if sf.exponent == 0: + amount = r.choice([3000, 5000, 10000]) + expires = issued.date() + timedelta(days=r.choice([180, 365, 730])) + cards.append( + GiftCard( + f"G{i:05d}", + sf.currency, + sf.exponent, + amount, + issued, + expires, + None if chance(r, 0.2) else purchaser.customer_id, + chance(r, 0.15), + ) + ) + tender: list[GiftTender] = [] + remaining = {card.gift_card_id: card.initial_minor for card in cards} + for order in world.orders: + if order.status in ("pending", "cancelled") or not chance(r, 0.05): + continue + candidates = [ + card + for card in cards + if card.currency == order.currency + and card.issued_at < order.placed_at + and card.expires_on > order.placed_at.date() + and remaining[card.gift_card_id] > 0 + ] + if not candidates: + continue + card = r.choice(candidates) + amount = min(remaining[card.gift_card_id], order.total_minor) + if amount <= 0: + continue + remaining[card.gift_card_id] -= amount + tender.append( + GiftTender( + order.order_id, card.gift_card_id, amount, order.time_of("paid") or order.placed_at + ) + ) + return tuple(cards), tuple(tender) + + +def gift_cards(world: World) -> Rows: + r = rng("storefront__gift_cards") + log = ChangeLog("storefront__gift_cards") + cards, _ = gift_cards_and_tender(world.scale) + for card in cards: + suffix = f"{r.randint(0, 9999):04d}" + row = { + "gift_card_id": card.gift_card_id, + "code_masked": r.choice( + [f"XXXX-XXXX-{suffix}", f"****-****-{suffix}", f"xxxx xxxx {suffix}"] + ), + "currency_code": card.currency if chance(r, 0.9) else card.currency.lower(), + "initial_amount_minor": card.initial_minor, + "issued_at": in_zone(card.issued_at, "UTC"), + "expires_on": card.expires_on, + "purchaser_customer_id": card.purchaser, + "details": r.choice( + [ + { + "promotional": card.promotional, + "design": r.choice(["pond", "golden-egg", None]), + }, + {"design": " "}, + None, + ] + ), + } + emit(log, r, [(card.issued_at, row)]) + return log.rows() + + +def gift_card_transactions(world: World) -> Rows: + r = rng("storefront__gift_card_transactions") + log = ChangeLog("storefront__gift_card_transactions") + cards, tender = gift_cards_and_tender(world.scale) + count = 0 + for t in tender: + count += 1 + row = { + "transaction_id": f"GT{count:06d}", + "gift_card_id": t.gift_card_id, + "order_id": t.order_id, + "amount_minor": t.amount_minor if chance(r, 0.7) else -t.amount_minor, + "transaction_type": r.choice(["redeem", "REDEEM", "Redeem"]), + "occurred_at": in_zone(t.at, "UTC"), + "context": r.choice([{"channel": "WEB"}, {"channel": "app"}, None]), + } + emit(log, r, [(t.at, row)]) + for card in cards: + if chance(r, 0.1): + count += 1 + when = card.issued_at + timedelta(days=r.randint(5, 60)) + if when.date() < date(2026, 3, 31): + emit( + log, + r, + [ + ( + when, + { + "transaction_id": f"GT{count:06d}", + "gift_card_id": card.gift_card_id, + "order_id": None, + "amount_minor": card.initial_minor // 2, + "transaction_type": "top-up", + "occurred_at": in_zone(when, "UTC"), + "context": { + "channel": "store", + "terminal": {"id": f"T-{r.randint(1, 9)}"}, + }, + }, + ) + ], + ) + if card.expires_on <= date(2026, 3, 31): + count += 1 + when = at(card.expires_on, 0, 5) + emit( + log, + r, + [ + ( + when, + { + "transaction_id": f"GT{count:06d}", + "gift_card_id": card.gift_card_id, + "order_id": None, + "amount_minor": card.initial_minor // 3, + "transaction_type": "Expire", + "occurred_at": in_zone(when, "UTC"), + "context": {"terminal": {"id": None}}, + }, + ) + ], + ) + first = tender[0] if tender else None + if first is not None: + count += 1 + when = first.at + timedelta(hours=2) + emit( + log, + r, + [ + ( + when, + { + "transaction_id": f"GT{count:06d}", + "gift_card_id": first.gift_card_id, + "order_id": first.order_id, + "amount_minor": 100, + "transaction_type": "reversal", + "occurred_at": in_zone(when, "UTC"), + "context": None, + }, + ) + ], + ) + return log.rows() + + +def _order_row(world: World, order: Order, status: str, r) -> dict[str, object]: + sf = STOREFRONTS[order.storefront] + attributes: dict[str, object] = {"line_skus": [line.sku for line in order.lines]} + if order.channel is not None: + attributes["channel"] = order.channel if chance(r, 0.9) else order.channel.upper() + if chance(r, 0.06): + attributes["gift"] = {"is_gift": True, "message": "Happy splash day!"} + if order.utm is not None: + source, medium, campaign = order.utm + attributes["utm"] = {"source": source, "medium": medium, "campaign": campaign or None} + attributes["device"] = {"type": r.choice(["desktop", "mobile", "tablet", None])} + return { + "order_id": order.order_id, + "customer_id": order.customer_id, + "storefront_code": order.storefront if chance(r, 0.95) else order.storefront.lower(), + "currency_code": order.currency, + "placed_at": in_zone(order.placed_at, sf.timezone), + "status": status if chance(r, 0.9) else status.upper(), + "shipping_address_id": order.address_id, + "attributes": attributes, + } + + +def orders(world: World) -> Rows: + r = rng("storefront__orders") + log = ChangeLog("storefront__orders") + for order in world.orders: + base = _order_row(world, order, "pending", r) + versions = [(when, {**base, "status": status}) for status, when in order.status_times] + emit(log, r, versions, late_share=0.05) + return log.rows() + + +def order_lines(world: World) -> Rows: + r = rng("storefront__order_lines") + log = ChangeLog("storefront__order_lines") + for order in world.orders: + for line in order.lines: + discounts: object + if line.discount_minor: + amount: object = line.discount_minor if chance(r, 0.8) else str(line.discount_minor) + discounts = [ + { + "code": order.promo_code.lower() if order.promo_code else None, + "kind": "promotion", + "amount_minor": amount, + } + ] + else: + discounts = r.choice([[], None, None, {"note": "no discount"}]) + variant_raw = world.variant_by_sku[line.sku].raw_sku + row = { + "order_line_id": line.order_line_id, + "order_id": order.order_id, + "line_no": line.line_no, + "sku": variant_raw if chance(r, 0.85) else f" {line.sku.lower()} ", + "quantity": line.quantity, + "unit_amount_minor": line.unit_minor, + "tax_amount_minor": line.tax_minor if chance(r, 0.97) else None, + "discounts": discounts, + } + emit(log, r, [(order.placed_at, row)]) + return log.rows() + + +def order_status_events(world: World) -> Rows: + r = rng("storefront__order_status_events") + log = ChangeLog("storefront__order_status_events") + count = 0 + for order in world.orders: + previous = None + for status, when in order.status_times: + count += 1 + actor = { + "pending": {"type": "customer", "id": order.customer_id}, + "paid": None, + "packed": {"type": "warehouse", "id": f"picker-{r.randint(1, 12)}"}, + "shipped": {"type": "system"}, + "delivered": {"id": "carrier-webhook"}, + "cancelled": {"type": "agent", "id": f"AG{r.randint(1, 12):03d}"}, + "returned": {"type": "agent", "id": None}, + }[status] + row = { + "event_id": f"SE{count:07d}", + "order_id": order.order_id, + "from_status": previous, + "to_status": status if chance(r, 0.85) else status.upper(), + "changed_at": in_zone(when, STOREFRONTS[order.storefront].timezone), + "actor": actor, + } + emit(log, r, [(when, row)], late_share=0.06, duplicate_share=0.03) + previous = status + return log.rows() + + +DECLINES = [ + "Declined: insufficient funds (DN051)", + "Card reported stolen - pickup (FR043)", + "3DS authentication failed (AU201)", + "Issuer unavailable, try again (TE091)", + "Do not honour", +] +METHODS = { + "card": 50, + "Credit Card": 10, + "debit-card": 10, + "phone wallet": 12, + "browser_wallet": 3, + "bank transfer": 5, + "sepa": 3, + "pay later": 4, +} + + +@lru_cache(maxsize=4) +def payments_plan(scale: int) -> tuple[tuple[str, str, str, int, datetime, str], ...]: + """(payment_id, order_id, method, amount_minor, authorised_at, gateway status) per attempt.""" + + from python.loaders._world import build_world + + world = build_world(scale) + r = rng("payments_plan") + _, tender = gift_cards_and_tender(scale) + tendered = {t.order_id: t.amount_minor for t in tender} + plan: list[tuple[str, str, str, int, datetime, str]] = [] + count = 0 + for order in world.orders: + due = order.total_minor - tendered.get(order.order_id, 0) + if due <= 0: + continue + method = str(weighted(r, METHODS)) + when = order.placed_at + timedelta(seconds=r.randint(5, 90)) + for _ in range(weighted(r, {0: 85, 1: 11, 2: 4})): # type: ignore[call-overload] + count += 1 + plan.append((f"PAY{count:07d}", order.order_id, method, due, when, "declined")) + when = when + timedelta(minutes=r.randint(1, 15)) + count += 1 + status = {"pending": "authorised", "cancelled": "voided"}.get(order.status, "captured") + if status == "captured" and chance(r, 0.3): + status = "settled" + plan.append((f"PAY{count:07d}", order.order_id, method, due, when, status)) + return tuple(plan) + + +def gateway_reference(payment_id: str) -> str: + """The numeric reference the payment gateway echoes back, also quoted in settlement reports.""" + + return str(500000 + int(payment_id[3:])) + + +def provider_for(method: str) -> str: + lowered = method.lower() + if "wallet" in lowered: + return "PROVIDER_B" + if "bank" in lowered or "sepa" in lowered: + return "PROVIDER_C" + return "PROVIDER_A" + + +def payments(world: World) -> Rows: + r = rng("storefront__payments") + log = ChangeLog("storefront__payments") + orders_by_id = world.order_by_id + for payment_id, order_id, method, amount, when, status in payments_plan(world.scale): + order = orders_by_id[order_id] + is_card = "card" in method.lower() + response: dict[str, object] = { + "status": { + "code": status, + "message": "Approved" if status != "declined" else r.choice(DECLINES), + }, + "risk": { + "score": r.choice([f"{r.uniform(0, 99):.2f}", round(r.uniform(0, 99), 6), "n/a"]) + }, + "reference": gateway_reference(payment_id) + if chance(r, 0.6) + else ( + int(gateway_reference(payment_id)) if chance(r, 0.9) else f"REF-{payment_id[-4:]}" + ), + } + if is_card: + response["three_ds"] = {"outcome": r.choice(["y", "N", "frictionless", None])} + if chance(r, 0.05): + response["status"] = status.upper() + captured_at = None + if status in ("captured", "settled"): + captured_at = in_zone(when + timedelta(minutes=r.randint(0, 120)), "UTC") + row = { + "payment_id": payment_id, + "order_id": order_id, + "method": method, + "amount_minor": amount, + "currency_code": order.currency, + "authorised_at": in_zone(when, STOREFRONTS[order.storefront].timezone), + "captured_at": captured_at, + "card_bin": (str(r.randint(400000, 559999)) if chance(r, 0.9) else "4876-53") + if is_card + else None, + "gateway_response": response, + } + versions = [(when, row)] + emit(log, r, versions) + return log.rows() + + +REFUND_REASONS = [ + "Item arrived DAMAGED - wing snapped off", + "Broken in transit, box crushed", + "Parcel never arrived", + "Delivery was 9 days late", + "Chargeback raised by card issuer", + "Customer changed their mind", + " Wrong colour sent (ordered teal, got yellow) ", + "Defective squeaker", +] + + +@lru_cache(maxsize=4) +def refunds_plan(scale: int) -> tuple[tuple[str, str, str, int, datetime, str], ...]: + """(refund_id, payment_id, order_id, amount_minor, refunded_at, reason) for refunded orders.""" + + from python.loaders._world import build_world + + world = build_world(scale) + r = rng("refunds_plan") + successful = { + order_id: (payment_id, amount) + for payment_id, order_id, _, amount, _, status in payments_plan(scale) + if status in ("captured", "settled") + } + plan: list[tuple[str, str, str, int, datetime, str]] = [] + count = 0 + for order in world.orders: + if order.order_id not in successful: + continue + payment_id, amount = successful[order.order_id] + returned = order.time_of("returned") + delivered = order.time_of("delivered") + if returned is not None: + count += 1 + full = chance(r, 0.6) + plan.append( + ( + f"RF{count:06d}", + payment_id, + order.order_id, + amount if full else amount // 2, + returned + timedelta(days=r.randint(1, 4)), + r.choice(REFUND_REASONS), + ) + ) + elif delivered is not None and chance(r, 0.03): + count += 1 + when = delivered + timedelta(days=r.randint(1, 10)) + if when.date() <= date(2026, 3, 31): + plan.append( + ( + f"RF{count:06d}", + payment_id, + order.order_id, + max(1, amount // 4), + when, + r.choice(REFUND_REASONS), + ) + ) + return tuple(plan) + + +def refunds(world: World) -> Rows: + r = rng("storefront__refunds") + log = ChangeLog("storefront__refunds") + orders_by_id = world.order_by_id + for refund_id, payment_id, order_id, amount, when, reason in refunds_plan(world.scale): + order = orders_by_id[order_id] + items: list[object] | None = [ + {"order_line_id": line.order_line_id, "qty": line.quantity} for line in order.lines[:2] + ] + if chance(r, 0.2): + items = None + row = { + "refund_id": refund_id, + "payment_id": payment_id, + "amount_minor": amount if chance(r, 0.8) else -amount, + "reason_text": reason, + "refunded_at": in_zone(when, STOREFRONTS[order.storefront].timezone), + "initiated_by": r.choice(["Agent", "customer", None, "SYSTEM"]), + "items": items, + } + emit(log, r, [(when, row)]) + return log.rows() + + +def promotions(world: World) -> Rows: + r = rng("storefront__promotions") + log = ChangeLog("storefront__promotions") + rules = { + "SPLASH10": { + "eligibility": {"min_order_minor": 2000, "regions": ["EU", "US"]}, + "tiers": [{"pct": 10}], + }, + "POND15": { + "eligibility": {"regions": ["EU", "APAC", "ANTARCTICA"]}, + "tiers": [{"pct": 10, "min_minor": 0}, {"pct": 15.5, "min_minor": 10000}], + }, + "GOLDENEGG": {"tiers": [{"pct": 20}], "eligibility": None}, + "WELCOME5": None, + } + names = { + "SPLASH10": "Spring splash", + "POND15": "Pond week", + "GOLDENEGG": "Golden egg hunt", + "WELCOME5": "Welcome to the flock", + } + for code, (starts, ends, _) in world.promotions.items(): + row = { + "promo_code": f" {code.lower()} " if code == "POND15" else code, + "promo_name": names[code], + "starts_at": in_zone(starts, "Europe/London"), + "ends_at": in_zone(ends, "Europe/London"), + "is_stackable": None if code == "WELCOME5" else code == "GOLDENEGG", + "rules": rules[code], + } + versions = [(starts - timedelta(days=5), row)] + if code == "POND15": + versions.append( + (starts + timedelta(days=3), {**row, "promo_name": "Pond week (extended)"}) + ) + emit(log, r, versions) + withdrawn = { + "promo_code": "WITHDRAWN50", + "promo_name": "Withdrawn test code", + "starts_at": in_zone(at(date(2026, 2, 1)), "UTC"), + "ends_at": None, + "is_stackable": True, + "rules": {"tiers": [{"pct": 50}]}, + } + emit(log, r, [(at(date(2026, 1, 30)), withdrawn)], delete_at=at(date(2026, 1, 31))) + return log.rows() + + +def promotion_redemptions(world: World) -> Rows: + r = rng("storefront__promotion_redemptions") + log = ChangeLog("storefront__promotion_redemptions") + count = 0 + for order in world.orders: + if order.promo_code is None: + continue + count += 1 + discount = sum(line.discount_minor for line in order.lines) + row = { + "redemption_id": f"PR{count:06d}", + "promo_code": order.promo_code if chance(r, 0.8) else f"{order.promo_code.lower()} ", + "order_id": order.order_id, + "discount_minor": discount if chance(r, 0.8) else -discount, + "redeemed_at": in_zone(order.placed_at, STOREFRONTS[order.storefront].timezone), + "context": r.choice([{"entry": "CODE"}, {"entry": "link"}, None]), + } + emit(log, r, [(order.placed_at, row)]) + if chance(r, 0.05): + count += 1 + emit( + log, + r, + [ + ( + order.placed_at + timedelta(seconds=30), + { + **row, + "redemption_id": f"PR{count:06d}", + "promo_code": "WELCOME5", + "discount_minor": 0, + "redeemed_at": in_zone(order.placed_at + timedelta(seconds=30), "UTC"), + }, + ) + ], + ) + return log.rows() + + +def carts(world: World) -> Rows: + r = rng("storefront__carts") + log = ChangeLog("storefront__carts") + live = [c for c in world.customers if not c.deleted] + for i in range(1, 700 * world.scale + 1): + customer = r.choice(live) if chance(r, 0.6) else None + created = random_instant(r, date(2026, 2, 1), date(2026, 3, 31)) + zone = STOREFRONTS[customer.storefront].timezone if customer else "UTC" + items: list[dict[str, object]] = [] + for index, variant in enumerate(r.sample(world.variants, k=r.randint(0, 4))): + qty: object = r.choice([1, 2, 1.0, "3", "two"]) + items.append( + { + "sku": variant.raw_sku, + "qty": qty, + "unit_minor": r.choice( + [str(int(variant.base_usd * 100)), int(variant.base_usd * 100), None] + ), + "added_at": in_zone(created + timedelta(minutes=index * 3), zone).isoformat() + if chance(r, 0.95) + else "yesterday", + } + ) + status = weighted(r, {"open": 30, "abandoned": 45, "converted": 25}) + updated = created + timedelta(minutes=r.randint(1, 600)) + row = { + "cart_id": f"CART{i:06d}", + "customer_id": customer.customer_id if customer else None, + "anonymous_id": ( + customer.anonymous_id if customer else f"anon-{r.getrandbits(48):012x}" + ).upper() + if chance(r, 0.1) + else (customer.anonymous_id if customer else f"anon-{r.getrandbits(48):012x}"), + "storefront_code": customer.storefront + if customer + else r.choice(list(STOREFRONTS)).lower(), + "cart_status": status if chance(r, 0.9) else str(status).upper(), + "created_at": in_zone(created, zone), + "updated_at": in_zone(updated, zone), + "contents": {"items": items, "coupon": r.choice([None, "pond15", "", " splash10"])} + if items + else r.choice([{"items": []}, None]), + } + versions = [ + (created, {**row, "cart_status": "open", "updated_at": in_zone(created, zone)}), + (updated, row), + ] + delete_at = updated + timedelta(days=1) if chance(r, 0.03) else None + emit(log, r, versions, delete_at=delete_at) + return log.rows() + + +REVIEW_BODIES = [ + "Love this duck, great colours and very sturdy.", + "Arrived broken, the beak snapped off. Very disappointed.", + "Perfect gift! See photos at https://photos.example.com/duck123", + "Cheap and flimsy, will return.", + "Beautiful carving.\nThe paint is excellent.", + "ok", + "Rust on the hook after a week in the pond.", + "", +] + + +def reviews(world: World) -> Rows: + r = rng("storefront__reviews") + log = ChangeLog("storefront__reviews") + count = 0 + for order in world.orders: + delivered = order.time_of("delivered") + if delivered is None or not chance(r, 0.18): + continue + line = r.choice(order.lines) + count += 1 + when = delivered + timedelta(days=r.randint(1, 14)) + if when.date() > date(2026, 3, 31): + continue + body = r.choice(REVIEW_BODIES) + row = { + "review_id": f"RV{count:05d}", + "sku": world.variant_by_sku[line.sku].raw_sku, + "customer_id": order.customer_id if chance(r, 0.95) else None, + "rating": r.choice([5, 5, 4, 4, 3, 2, 1, 0, 6]), + "title": r.choice([" Great duck ", "Not as pictured", None, "Ducky delight"]), + "body": body, + "submitted_at": in_zone(when, STOREFRONTS[order.storefront].timezone), + "moderation": r.choice( + [ + {"status": "approved", "flags": []}, + {"status": "REJECTED", "flags": ["url"]}, + {"flags": ["profanity", "url"]}, + None, + ] + ), + } + emit(log, r, [(when, row)]) + count += 1 + emit( + log, + r, + [ + ( + at(date(2026, 3, 2)), + { + "review_id": f"RV{count:05d}", + "sku": world.variants[0].raw_sku, + "customer_id": None, + "rating": 1, + "title": "spam", + "body": "Visit http://spam.example.net now", + "submitted_at": in_zone(at(date(2026, 3, 2)), "UTC"), + "moderation": {"status": "rejected", "flags": ["spam"]}, + }, + ) + ], + delete_at=at(date(2026, 3, 3)), + ) + return log.rows() + + +@lru_cache(maxsize=4) +def subscription_plan( + scale: int, +) -> tuple[tuple[str, str, str, datetime, list[tuple[str, datetime]]], ...]: + from python.loaders._world import build_world + + world = build_world(scale) + r = rng("subscription_plan") + consumables = [v for v in world.variants if v.family in ("POND", "TOWEL")] + live = [c for c in world.customers if not c.deleted] + plan = [] + for i in range(1, 80 * scale + 1): + customer = r.choice(live) + started = max(customer.created_at, at(date(2025, 10, 1))) + timedelta( + days=r.randint(0, 120) + ) + if started.date() > date(2026, 3, 25): + started = at(date(2026, 3, 1), 10) + events: list[tuple[str, datetime]] = [] + when = started + state = "active" + for _ in range(r.randint(0, 4)): + when = when + timedelta(days=r.randint(5, 40), hours=r.randint(0, 23)) + if when.date() > date(2026, 3, 31): + break + if state == "active": + kind = str(weighted(r, {"PAUSE": 35, "SKIP": 25, "CANCEL": 15, "RENEW": 25})) + elif state == "paused": + kind = str(weighted(r, {"RESUME": 70, "CANCEL": 30})) + else: + break + events.append((kind, when)) + state = {"PAUSE": "paused", "RESUME": "active", "CANCEL": "cancelled"}.get(kind, state) + plan.append( + (f"SUB{i:05d}", customer.customer_id, r.choice(consumables).raw_sku, started, events) + ) + return tuple(plan) + + +def subscriptions(world: World) -> Rows: + r = rng("storefront__subscriptions") + log = ChangeLog("storefront__subscriptions") + for subscription_id, customer_id, sku, started, events in subscription_plan(world.scale): + sf = STOREFRONTS[world.customer_by_id[customer_id].storefront] + zone = sf.timezone + interval = r.choice([30, 30, 60, None, 90]) + amount = {0: 2500, 2: 1999, 3: 6500}[sf.exponent] + next_bill = started.date() + timedelta(days=interval or 30) + row = { + "subscription_id": subscription_id, + "customer_id": customer_id, + "plan_code": r.choice(["pond-care-monthly", "TOWEL-REFRESH"]), + "sku": sku, + "interval_days": interval, + "subscription_status": "active", + "started_at": in_zone(started, zone), + "cancelled_at": None, + "billing": { + "amount_minor": amount, + "currency": sf.currency.lower(), + "next_bill_on": next_bill.isoformat() + if chance(r, 0.9) + else next_bill.strftime("%d/%m/%Y"), + }, + } + versions = [(started, row)] + state = "active" + for kind, when in events: + state = {"PAUSE": "paused", "RESUME": "active", "CANCEL": "cancelled"}.get(kind, state) + current = { + **versions[-1][1], + "subscription_status": state if chance(r, 0.9) else state.upper(), + } + if kind == "CANCEL": + current["cancelled_at"] = in_zone(when, zone) + versions.append((when, current)) + emit(log, r, versions) + return log.rows() + + +def subscription_events(world: World) -> Rows: + r = rng("storefront__subscription_events") + log = ChangeLog("storefront__subscription_events") + count = 0 + for subscription_id, customer_id, _, _, events in subscription_plan(world.scale): + zone = STOREFRONTS[world.customer_by_id[customer_id].storefront].timezone + for kind, when in events: + count += 1 + payload: dict[str, object] | None = None + if kind == "PAUSE": + payload = {"pause_days": r.choice([14, "30", "a month"])} + elif kind == "SKIP": + payload = {"skip_reason": r.choice(["too many towels", None, " "])} + row = { + "event_id": f"SUBE{count:06d}", + "subscription_id": subscription_id, + "event_type": kind if chance(r, 0.85) else kind.lower(), + "occurred_at": in_zone(when, zone), + "payload": payload, + } + emit(log, r, [(when, row)]) + return log.rows() + + +def store_credit_ledger(world: World) -> Rows: + r = rng("storefront__store_credit_ledger") + log = ChangeLog("storefront__store_credit_ledger") + orders_by_id = world.order_by_id + count = 0 + for refund_id, _, order_id, amount, when, _ in refunds_plan(world.scale): + if not chance(r, 0.35): + continue + order = orders_by_id[order_id] + count += 1 + emit( + log, + r, + [ + ( + when, + { + "entry_id": f"SC{count:06d}", + "customer_id": order.customer_id, + "currency_code": order.currency.lower() + if chance(r, 0.1) + else order.currency, + "amount_minor": amount, + "reason_code": r.choice( + ["return credit", "RETURN-CREDIT", "return_credit"] + ), + "posted_at": in_zone(when, "UTC"), + "context": { + "order_id": order_id, + "refund_id": refund_id, + "expires_on": (when.date() + timedelta(days=365)).isoformat(), + }, + }, + ) + ], + ) + if chance(r, 0.4): + count += 1 + spent = when + timedelta(days=r.randint(2, 20)) + if spent.date() <= date(2026, 3, 31): + emit( + log, + r, + [ + ( + spent, + { + "entry_id": f"SC{count:06d}", + "customer_id": order.customer_id, + "currency_code": order.currency, + "amount_minor": -min(amount, amount // 2 + 1), + "reason_code": "spent on order", + "posted_at": in_zone(spent, "UTC"), + "context": {"order_id": None}, + }, + ) + ], + ) + for c in world.customers[::40]: + if c.deleted: + continue + count += 1 + when = max(c.created_at, at(date(2026, 1, 15))) + timedelta(days=3) + emit( + log, + r, + [ + ( + when, + { + "entry_id": f"SC{count:06d}", + "customer_id": c.customer_id, + "currency_code": STOREFRONTS[c.storefront].currency, + "amount_minor": 5 * 10 ** STOREFRONTS[c.storefront].exponent, + "reason_code": "goodwill gesture!", + "posted_at": in_zone(when, "UTC"), + "context": None, + }, + ) + ], + ) + return log.rows() + + +TABLES = { + "storefront__customers": customers, + "storefront__customer_addresses": customer_addresses, + "storefront__customer_consents": customer_consents, + "storefront__loyalty_accounts": loyalty_accounts, + "storefront__loyalty_ledger": loyalty_ledger, + "storefront__wishlists": wishlists, + "storefront__gift_cards": gift_cards, + "storefront__gift_card_transactions": gift_card_transactions, + "storefront__orders": orders, + "storefront__order_lines": order_lines, + "storefront__order_status_events": order_status_events, + "storefront__payments": payments, + "storefront__refunds": refunds, + "storefront__promotions": promotions, + "storefront__promotion_redemptions": promotion_redemptions, + "storefront__carts": carts, + "storefront__reviews": reviews, + "storefront__subscriptions": subscriptions, + "storefront__subscription_events": subscription_events, + "storefront__store_credit_ledger": store_credit_ledger, +} diff --git a/benchmarks/duckshop/project/python/loaders/_support.py b/benchmarks/duckshop/project/python/loaders/_support.py new file mode 100644 index 0000000000..ff84a409b0 --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_support.py @@ -0,0 +1,427 @@ +"""Support change logs: tickets, comments, CSAT, agents, shifts, chats, calls and SLA policies.""" + +from __future__ import annotations + +from datetime import date, datetime, timedelta +from functools import lru_cache + +from python.loaders._core import ChangeLog, at, chance, emit, in_zone, random_instant, rng, weighted +from python.loaders._world import World, build_world + +Rows = list[dict[str, object]] +AGENTS = [ + # id, name, team, zone, skills + ("AG001", "Robin Agent", "tier1", "Europe/Berlin", ["lang:de", "lang:en"]), + ("AG002", "Sam Agent", "TIER1", "Europe/Berlin", ["lang:en"]), + ("AG003", "Kim Agent", "tier1", "Europe/London", ["lang:en", "returns"]), + ("AG004", "Noor Agent", "tier2", "Europe/London", ["lang:en", "lang:ar"]), + ("AG005", "Yuki Agent", "tier1", "Asia/Tokyo", ["lang:ja", "lang:en"]), + ("AG006", "Alex Agent", "tier2", "America/New_York", ["lang:en", "billing"]), + ("AG007", "Jordan Agent", "returns", "America/New_York", ["returns"]), + ("AG008", "Hana Agent", "tier1", "Australia/Sydney", None), +] +CHANNELS = ["email", "chat", "phone", "web"] +SUBJECTS = [ + ( + "where_is_my_order", + "Where is my order {order}? Tracking says delivered but nothing arrived.", + ), + ("damaged_item", "My duck arrived broken!! The beak is snapped off, order ref {order}"), + ("refund_status", "Refund for return RT-{n} still not received after 14 days"), + ( + "change_address", + "Please change delivery address to 12 Reed Street, call me on +44 20 7946 0958", + ), + ("billing", "Charged twice: card ending 4242 on 2026-03-03 and 03/03/2026"), + ("product_question", "Is the Mallard decoy dishwasher safe? Email me at someone@example.com"), +] + + +@lru_cache(maxsize=4) +def ticket_plan(scale: int): + world = build_world(scale) + r = rng("ticket_plan") + customers = [c for c in world.customers if c.email and not c.deleted] + plan = [] + for i in range(1, 260 * scale + 1): + customer = r.choice(customers) + orders = [o for o in world.orders if o.customer_id == customer.customer_id] + order = r.choice(orders) if orders and chance(r, 0.8) else None + created = random_instant(r, date(2026, 1, 5), date(2026, 3, 31)) + if order is not None and created < order.placed_at: + created = order.placed_at + timedelta(hours=r.randint(2, 200)) + if created.date() > date(2026, 3, 31): + created = at(date(2026, 3, 31), 22) + # Tickets opened around the European clock change and over the weekend. + if i <= 4: + created = [ + datetime(2026, 3, 29, 0, 40), + datetime(2026, 3, 29, 1, 20), + datetime(2026, 3, 28, 22, 0), + datetime(2026, 3, 8, 6, 50), + ][i - 1] + reason, template = r.choice(SUBJECTS) + channel = str(weighted(r, {"email": 45, "chat": 30, "phone": 15, "web": 10})) + priority = weighted(r, {1: 5, 2: 15, 3: 60, 4: 18, None: 1, 0: 1}) + first_response = created + timedelta(minutes=r.randint(3, 60 * 30)) + solved = first_response + timedelta(hours=r.randint(1, 120)) if chance(r, 0.75) else None + if solved is not None and solved.date() > date(2026, 3, 31): + solved = None + if first_response.date() > date(2026, 3, 31) or chance(r, 0.05): + first_response = None + solved = None + plan.append( + ( + f"TK{i:05d}", + customer, + order, + created, + reason, + template, + channel, + priority, + first_response, + solved, + r.choice([a[0] for a in AGENTS]), + ) + ) + return tuple(plan) + + +def tickets(world: World) -> Rows: + r = rng("support__tickets") + log = ChangeLog("support__tickets") + for ( + ticket_id, + customer, + order, + created, + reason, + _, + channel, + priority, + first_response, + solved, + _, + ) in ticket_plan(world.scale): + custom: dict[str, object] | None = { + "contact_reason": reason.replace("_", " ").title() if chance(r, 0.3) else reason + } + if order is not None: + if chance(r, 0.6): + custom["order_id"] = order.order_id + else: + custom["order_ref"] = f"Re: order {order.order_id} missing" + if chance(r, 0.2): + custom["labels"] = r.choice(["vip, late_delivery", " Refund ,damage", "vip"]) + if chance(r, 0.05): + custom = None + tags = r.choice([["Late Delivery", "vip"], ["damage"], [], None, ["refund_request", None]]) + row = { + "ticket_id": ticket_id, + "customer_email": customer.email, + "channel": channel if chance(r, 0.85) else channel.upper(), + "priority": priority, + "ticket_status": "new", + "created_at": in_zone( + created, r.choice(["Europe/Berlin", "America/New_York", "UTC", "Asia/Tokyo"]) + ), + "solved_at": None, + "custom_fields": custom, + "tags": tags, + } + versions = [(created, row)] + if first_response is not None: + versions.append((first_response, {**row, "ticket_status": "open"})) + if chance(r, 0.3): + versions.append( + (first_response + timedelta(minutes=5), {**row, "ticket_status": "Pending"}) + ) + if solved is not None: + versions.append( + ( + solved, + { + **versions[-1][1], + "ticket_status": r.choice(["solved", "closed"]), + "solved_at": in_zone(solved, "Europe/Berlin"), + }, + ) + ) + emit(log, r, versions, late_share=0.05) + return log.rows() + + +def ticket_comments(world: World) -> Rows: + r = rng("support__ticket_comments") + log = ChangeLog("support__ticket_comments") + count = 0 + for ( + ticket_id, + customer, + order, + created, + _, + template, + _, + _, + first_response, + solved, + agent, + ) in ticket_plan(world.scale): + body = template.format( + order=order.order_id if order else "DS-O-000000", n=r.randint(10, 99) + ) + comments: list[tuple[str, str | None, str, datetime]] = [ + ("customer", customer.customer_id, body, created) + ] + if first_response is not None: + comments.append( + ( + "agent", + agent, + r.choice( + [ + "Thanks for writing in. I checked order {o} and will update you.", + "Sorry to hear that! A replacement duck is on its way.", + "Could you send a photo of the damage?", + ] + ).format(o=order.order_id if order else "DS-O-000000"), + first_response, + ) + ) + if chance(r, 0.4): + reply = first_response + timedelta(hours=r.randint(1, 30)) + comments.append( + ( + "customer", + customer.customer_id, + r.choice( + [ + "Thanks, sorted!", + "Still waiting...", + "This is ridiculous, I want a refund", + ] + ), + reply, + ) + ) + if solved is not None: + comments.append( + ("system", None, "Ticket solved automatically after 72h without reply", solved) + ) + for author_type, author_id, text, when in comments: + count += 1 + if when.date() > date(2026, 3, 31): + continue + row = { + "comment_id": f"TC{count:06d}", + "ticket_id": ticket_id, + "author_type": author_type if chance(r, 0.9) else author_type.upper(), + "author_id": author_id, + "body": text, + "is_public": None if chance(r, 0.1) else author_type != "system", + "created_at": in_zone(when, "UTC"), + "attachments": r.choice([None, [], [{"name": "photo.jpg", "bytes": 182733}]]), + } + emit(log, r, [(when, row)]) + return log.rows() + + +def csat_responses(world: World) -> Rows: + r = rng("support__csat_responses") + log = ChangeLog("support__csat_responses") + count = 0 + for ticket_id, *_, solved, _ in ticket_plan(world.scale): + if solved is None or not chance(r, 0.4): + continue + count += 1 + when = solved + timedelta(hours=r.randint(1, 48)) + if when.date() > date(2026, 3, 31): + continue + row = { + "response_id": f"CR{count:05d}", + "ticket_id": ticket_id, + "score": r.choice([5, 5, 4, 3, 2, 1]), + "comment": r.choice(["", " Quick and helpful ", None, "Took far too long"]), + "responded_at": in_zone(when, "UTC"), + "answers": r.choice( + [{"effort": "2"}, {"effort": 5, "resolved": True}, {"effort": "n/a"}, None] + ), + } + emit(log, r, [(when, row)]) + return log.rows() + + +def agents(world: World) -> Rows: + r = rng("support__agents") + log = ChangeLog("support__agents") + for index, (agent_id, name, team, zone, skills) in enumerate(AGENTS, start=1): + row = { + "agent_id": agent_id, + "display_name": name, + "email": f"{name.split()[0].lower()}.agent@duckshop.example.com" + if index != 3 + else " KIM.Agent@DuckShop.example.com ", + "team_code": team, + "iana_timezone": zone, + "hired_on": date(2024, 1, 1) + timedelta(days=index * 45), + "skills": skills, + } + versions = [(at(date(2025, 12, 1)), row)] + if agent_id == "AG002": + versions.append( + ( + at(date(2026, 2, 1)), + {**row, "team_code": "tier2", "skills": ["lang:en", "billing"]}, + ) + ) + emit(log, r, versions) + return log.rows() + + +def agent_shifts(world: World) -> Rows: + r = rng("support__agent_shifts") + log = ChangeLog("support__agent_shifts") + count = 0 + day = date(2026, 3, 2) + while day <= date(2026, 3, 31): + for agent_id, _, _, zone, _ in AGENTS: + if day.weekday() >= 5 and not chance(r, 0.2): + continue + count += 1 + start_hour = r.choice([7, 8, 9, 13]) + start = datetime.combine(day, datetime.min.time()) + timedelta(hours=start_hour) + row = { + "shift_id": f"SF{count:06d}", + "agent_id": agent_id, + "shift_start_local": start, + "shift_end_local": start + timedelta(hours=8, minutes=r.choice([0, 30])), + "iana_timezone": zone, + "breaks": r.choice( + [ + [{"minutes": 30}, {"minutes": "15"}], + [], + None, + [{"minutes": 45, "paid": False}], + ] + ), + } + emit(log, r, [(start - timedelta(days=7), row)]) + day += timedelta(days=1) + return log.rows() + + +def chat_transcripts(world: World) -> Rows: + r = rng("support__chat_transcripts") + log = ChangeLog("support__chat_transcripts") + count = 0 + lines = { + "customer": [ + "Hi, my duck never arrived", + "This is the worst service, I want to cancel", + "My number is 07700 900123 if easier", + "Thanks, that is great!", + "ok", + ], + "agent": [ + "Hello! Let me check that for you.", + "I appreciate your patience.", + "Your refund has been sorted.", + ], + } + for ticket_id, _, _, created, _, _, channel, _, _, _, _ in ticket_plan(world.scale): + if channel != "chat": + continue + count += 1 + started = created + messages = [] + when = started + for index in range(r.randint(2, 7)): + role = "customer" if index % 2 == 0 else r.choice(["agent", "Agent", "bot"]) + when += timedelta(seconds=r.randint(10, 400)) + messages.append( + { + "role": role, + "ts": in_zone(when, "UTC").isoformat() if chance(r, 0.95) else "garbled", + "text": r.choice(lines["customer" if role == "customer" else "agent"]), + } + ) + row = { + "chat_id": f"CH{count:05d}", + "ticket_id": ticket_id if chance(r, 0.95) else None, + "started_at": in_zone(started, "UTC"), + "ended_at": in_zone(when, "UTC"), + "messages": messages, + } + emit(log, r, [(when, row)]) + return log.rows() + + +def call_records(world: World) -> Rows: + r = rng("support__call_records") + log = ChangeLog("support__call_records") + count = 0 + paths = [ + ["main", "orders", "where_is_my_order", "self_service_complete"], + ["main", "orders", "agent"], + ["main", "returns", "main", "returns", "agent"], + ["main", "billing", "callback_booked"], + ["main"], + ] + for ticket_id, customer, _, created, _, _, channel, _, _, _, agent in ticket_plan(world.scale): + if channel != "phone": + continue + for attempt in range(r.randint(1, 2)): + count += 1 + path = r.choice(paths) + when = created + timedelta(hours=attempt * 5) + if when.date() > date(2026, 3, 31): + continue + row = { + "call_id": f"CL{count:05d}", + "ticket_id": ticket_id, + "agent_id": agent if path[-1] == "agent" else None, + "phone_number": customer.phone, + "started_at": in_zone(when, "Europe/London"), + "duration_seconds": r.randint(20, 1800), + "ivr_path": path, + } + emit(log, r, [(when, row)]) + return log.rows() + + +def sla_policies(world: World) -> Rows: + r = rng("support__sla_policies") + log = ChangeLog("support__sla_policies") + count = 0 + targets = {1: (30, 8), 2: (60, 24), 3: (240, 72), 4: (480, 120)} + for priority, (response, resolution) in targets.items(): + for channel in CHANNELS: + count += 1 + row = { + "policy_id": f"SLA{count:03d}", + "priority": priority, + "channel": channel.title() if chance(r, 0.2) else channel, + "first_response_minutes": response // (4 if channel == "chat" else 1), + "resolution_hours": resolution, + "business_hours_only": None if channel == "web" else priority > 1, + "calendar": {"region": "eu"} if chance(r, 0.7) else None, + } + versions = [(at(date(2025, 12, 1)), row)] + if priority == 2 and channel == "email": + versions.append((at(date(2026, 3, 1)), {**row, "first_response_minutes": 45})) + emit(log, r, versions) + return log.rows() + + +TABLES = { + "support__tickets": tickets, + "support__ticket_comments": ticket_comments, + "support__csat_responses": csat_responses, + "support__agents": agents, + "support__agent_shifts": agent_shifts, + "support__chat_transcripts": chat_transcripts, + "support__call_records": call_records, + "support__sla_policies": sla_policies, +} diff --git a/benchmarks/duckshop/project/python/loaders/_world.py b/benchmarks/duckshop/project/python/loaders/_world.py new file mode 100644 index 0000000000..01c93e4a41 --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_world.py @@ -0,0 +1,641 @@ +"""The shared synthetic world every DuckShop table is generated from. + +Master data (storefronts, customers, products, warehouses) and the order history are built once per +scale, so every table agrees on the same customers, SKUs, orders and dates. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from datetime import date, datetime, timedelta +from functools import lru_cache + +from python.loaders._core import ( + CLICKSTREAM_START, + ORDERS_START, + VALUATION_DATE, + at, + chance, + random_instant, + rng, + weighted, +) + + +@dataclass(frozen=True) +class Storefront: + code: str + region: str + timezone: str + currency: str + exponent: int + weight: float + + +STOREFRONTS: dict[str, Storefront] = { + s.code: s + for s in ( + Storefront("GB", "EU", "Europe/London", "GBP", 2, 24), + Storefront("DE", "EU", "Europe/Berlin", "EUR", 2, 15), + Storefront("FR", "EU", "Europe/Paris", "EUR", 2, 8), + Storefront("NL", "EU", "Europe/Amsterdam", "EUR", 2, 6), + Storefront("US", "US", "America/New_York", "USD", 2, 20), + Storefront("CA", "US", "America/Toronto", "USD", 2, 5), + Storefront("JP", "APAC", "Asia/Tokyo", "JPY", 0, 9), + Storefront("AU", "APAC", "Australia/Sydney", "AUD", 2, 8), + Storefront("KW", "MENA", "Asia/Kuwait", "KWD", 3, 5), + ) +} + +CURRENCY_EXPONENT = {"USD": 2, "EUR": 2, "GBP": 2, "JPY": 0, "AUD": 2, "KWD": 3} +# Rough units of each currency per US dollar, for pricing in local currency. +PER_USD = {"USD": 1.0, "EUR": 0.92, "GBP": 0.79, "JPY": 148.0, "AUD": 1.52, "KWD": 0.307} + +FIRST_NAMES = [ + "Alex", + "Sam", + "Jordan", + "Taylor", + "Morgan", + "Robin", + "Casey", + "Jamie", + "Charlie", + "Avery", + "Riley", + "Quinn", + "Harper", + "Rowan", + "Sasha", + "Kim", + "Lee", + "Noor", + "Mina", + "Yuki", + "Hana", + "Lena", + "Jonas", + "Emma", + "Lucas", + "Chloe", + "Hugo", + "Sofia", + "Omar", + "Layla", +] +LAST_NAMES = [ + "Smith", + "Jones", + "Brown", + "Miller", + "Davis", + "Wilson", + "Moore", + "Clark", + "Lewis", + "Walker", + "Muller", + "Schmidt", + "Fischer", + "Martin", + "Bernard", + "Dubois", + "de Vries", + "van Dijk", + "Bakker", + "Tanaka", + "Suzuki", + "Sato", + "Nguyen", + "Kelly", + "O Brien", + "Murphy", + "Rahman", + "Haddad", + "Ali", +] +CITIES = { + "GB": [("London", "SW1A 1AA"), ("Leeds", "LS1 4AP"), ("Bristol", "BS1 5TR")], + "DE": [("Berlin", "10115"), ("Hamburg", "20095"), ("Munich", "80331")], + "FR": [("Paris", "75001"), ("Lyon", "69001")], + "NL": [("Amsterdam", "1012 AB"), ("Utrecht", "3511 AA")], + "US": [("New York", "10001"), ("Boston", "02108"), ("Denver", "80202")], + "CA": [("Toronto", "M5H 2N2"), ("Ottawa", "K1P 1J1")], + "JP": [("Tokyo", "100-0001"), ("Osaka", "5300001")], + "AU": [("Sydney", "2000"), ("Melbourne", "3000")], + "KW": [("Kuwait City", "13001"), ("Hawalli", "32001")], +} +GEO = { + "GB": (51.5072, -0.1276), + "DE": (52.52, 13.405), + "FR": (48.8566, 2.3522), + "NL": (52.3676, 4.9041), + "US": (40.7128, -74.006), + "CA": (43.6532, -79.3832), + "JP": (35.6762, 139.6503), + "AU": (-33.8688, 151.2093), + "KW": (29.3759, 47.9774), +} + + +@dataclass(frozen=True) +class Warehouse: + code: str + name: str + country: str + timezone: str + + +WAREHOUSES = ( + Warehouse("POND", "The Pond", "GB", "Europe/London"), + Warehouse("NEST", "The Nest", "US", "America/New_York"), + Warehouse("MARSH", "The Marsh", "AU", "Australia/Sydney"), +) +WAREHOUSE_FOR_REGION = {"EU": "POND", "US": "NEST", "APAC": "MARSH", "MENA": "POND"} +CARRIERS = ("FLAPWING", "WINGBEAT", "POSTAL", "WEBFOOT") + +# Product families are the leading letters of each SKU; duck species are the product lines. +SPECIES = [ + ("MAL", "Mallard"), + ("TEAL", "Teal"), + ("PIN", "Pintail"), + ("WIG", "Wigeon"), + ("EID", "Eider"), + ("MER", "Merganser"), + ("GOLD", "Goldeneye"), + ("BUF", "Bufflehead"), + ("SHOV", "Shoveler"), + ("MAND", "Mandarin"), + ("GAD", "Gadwall"), + ("SCAUP", "Scaup"), +] +FAMILIES = [ + # family, title pattern, category path, base USD price, weight grams, brand + ("DUCK", "{species} bath duck", "Ducks/Rubber-Ducks/bath-ducks", 9.0, 80, "MALLARD"), + ( + "DUCK", + "{species} collectible duck", + "ducks//rubber-ducks/Collectible-Series/", + 24.0, + 120, + "MALLARD", + ), + ("DECOY", "Carved {species} decoy", "decoys/carved-decoys", 89.0, 1400, "DABBLE"), + ("DECOY", "Painted {species} decoy", "Decoys/painted-decoys", 59.0, 900, "DABBLE"), + ("TOWEL", "{species} print towel", "textiles/towels", 19.0, 450, "FEATHERHOLM"), + ("CUSHION", "{species} cushion cover", "textiles/cushions", 29.0, 300, "FEATHERHOLM"), + ("MUG", "{species} mug", "home/mugs", 14.0, 350, "MALLARD"), + ("POND", "{species} pond float", "pond-and-bath/pond-care", 34.0, 700, "GENERIC"), +] +COLOURS = ["Yellow", "green ", "Blue", "teal", "Natural", "Charcoal"] +SIZES = ["S", "M", "L", "x l", None] + + +@dataclass +class Customer: + customer_id: str + storefront: str + first_name: str + last_name: str + email: str | None + phone: str | None + created_at: datetime + addresses: list[str] = field(default_factory=list) + default_address: str = "" + anonymous_id: str = "" + deleted: bool = False + + +@dataclass +class Variant: + sku: str + raw_sku: str + product_id: str + family: str + species: str + colour: str + size: str | None + weight_grams: int + base_usd: float + + +@dataclass +class Product: + product_id: str + title: str + category_path: str + brand_code: str + launched_on: date + family: str + species: str + + +@dataclass +class OrderLine: + order_line_id: str + line_no: int + sku: str + quantity: int + unit_minor: int + tax_minor: int + discount_minor: int + + +@dataclass +class Order: + order_id: str + customer_id: str + storefront: str + currency: str + exponent: int + placed_at: datetime + status: str + status_times: list[tuple[str, datetime]] + address_id: str + channel: str | None + utm: tuple[str, str, str] | None + promo_code: str | None + lines: list[OrderLine] + gift_card_minor: int = 0 + gift_card_id: str | None = None + + @property + def net_minor(self) -> int: + return sum(line.quantity * line.unit_minor - line.discount_minor for line in self.lines) + + @property + def tax_minor(self) -> int: + return sum(line.tax_minor for line in self.lines) + + @property + def total_minor(self) -> int: + return self.net_minor + self.tax_minor + + def time_of(self, status: str) -> datetime | None: + for name, when in self.status_times: + if name == status: + return when + return None + + +@dataclass +class World: + scale: int + customers: list[Customer] + products: list[Product] + variants: list[Variant] + orders: list[Order] + promotions: dict[str, tuple[datetime, datetime, float]] + + @property + def customer_by_id(self) -> dict[str, Customer]: + return {c.customer_id: c for c in self.customers} + + @property + def variant_by_sku(self) -> dict[str, Variant]: + return {v.sku: v for v in self.variants} + + @property + def order_by_id(self) -> dict[str, Order]: + return {o.order_id: o for o in self.orders} + + +def local_price(base_usd: float, currency: str) -> int: + """A list price in minor units: charm pricing for 2-decimal currencies, round for the rest.""" + + exponent = CURRENCY_EXPONENT[currency] + value = base_usd * PER_USD[currency] + if exponent == 0: + return int(round(value / 10) * 10) + if exponent == 3: + return int(round(value * 1000 / 5) * 5) + return int(round(value)) * 100 - 1 if value >= 1 else int(round(value * 100)) + + +def _products(scale: int) -> tuple[list[Product], list[Variant]]: + r = rng("world", "products") + products: list[Product] = [] + variants: list[Variant] = [] + count = 0 + numbers: dict[tuple[str, str], int] = {} + for _ in range(scale): + for family, title, path, base_usd, grams, brand in FAMILIES: + for code, species in SPECIES: + if not chance(r, 0.45): + continue + count += 1 + product_id = f"P{count:04d}" + launched = date(2024, 6, 1) + timedelta(days=r.randint(0, 600)) + brand_code = brand if not chance(r, 0.03) else "UNKNOWNBRAND" + product_title = title.format(species=species) + if chance(r, 0.05): + product_title = " " + product_title.replace(" ", " ", 1) + products.append( + Product(product_id, product_title, path, brand_code, launched, family, species) + ) + for _ in range(r.randint(1, 3)): + numbers[(family, code)] = numbers.get((family, code), 0) + 1 + sku = f"{family}-{code}-{numbers[(family, code)]:02d}" + roll = r.random() + raw = ( + sku + if roll < 0.8 + else (sku.lower() if roll < 0.9 else sku.replace("-", "_", 1)) + ) + variants.append( + Variant( + sku=sku, + raw_sku=raw, + product_id=product_id, + family=family, + species=species, + colour=r.choice(COLOURS), + size=r.choice(SIZES), + weight_grams=grams + r.randint(-20, 60), + base_usd=round(base_usd * r.uniform(0.85, 1.3), 2), + ) + ) + return products, variants + + +def _customers(scale: int) -> list[Customer]: + r = rng("world", "customers") + customers: list[Customer] = [] + count = 300 * scale + codes = list(STOREFRONTS) + weights = [STOREFRONTS[c].weight for c in codes] + for i in range(1, count + 1): + storefront = r.choices(codes, weights=weights, k=1)[0] + first = r.choice(FIRST_NAMES) + last = r.choice(LAST_NAMES) + created = random_instant(r, date(2025, 1, 1), date(2026, 3, 20)) + domain = weighted(r, {"example.com": 85, "example.org": 8, "example.net": 7}) + local = f"{first}.{last}".lower().replace(" ", "") + email: str | None = f"{local}{i}@{domain}" + roll = r.random() + if roll < 0.05: + email = f" {email.upper()} " + elif roll < 0.09: + email = email.replace("@", "+duckshop@") + elif roll < 0.11: + email = f"{local}{i} at {domain}" + elif roll < 0.12: + email = None + phone: str | None = ( + f"+{r.randint(1, 99)} {r.randint(100, 999)} {r.randint(1000000, 9999999)}" + ) + if chance(r, 0.15): + phone = "0" + phone[1:].replace(" ", "-") + if chance(r, 0.04): + phone = "12-34" + customers.append( + Customer( + customer_id=f"C{i:06d}", + storefront=storefront, + first_name=first, + last_name=last, + email=email, + phone=phone, + created_at=created, + anonymous_id=f"anon-{r.getrandbits(48):012x}", + ) + ) + # The order lookup report's customer: an established account with history. + customers[0].created_at = at(date(2025, 2, 14), 9, 30) + customers[0].storefront = "GB" + # Duplicate accounts: the same person signed up twice with one phone number. + for original_index in range(5, 5 + 4 * scale, 4): + original = customers[original_index] + duplicate = customers[original_index + 1] + duplicate.first_name = original.first_name + duplicate.last_name = original.last_name + duplicate.phone = original.phone + if original.email and duplicate.email: + duplicate.email = original.email.strip().lower().replace("@", "+second@") + # Accounts closed before ever ordering. + for customer in customers[-3 * scale :]: + customer.deleted = True + for customer in customers: + count = 2 if chance(r, 0.2) else 1 + customer.addresses = [f"A{customer.customer_id[1:]}-{n}" for n in range(1, count + 1)] + customer.default_address = customer.addresses[-1] + return customers + + +PROMOTIONS = { + # code: (starts, ends, percent off) + "SPLASH10": (at(date(2026, 1, 1)), at(date(2026, 3, 31), 23, 59), 10.0), + "POND15": (at(date(2026, 2, 1)), at(date(2026, 2, 28), 23, 59), 15.0), + "GOLDENEGG": (at(date(2026, 3, 1)), at(date(2026, 4, 30)), 20.0), + "WELCOME5": (at(date(2025, 6, 1)), at(date(2026, 12, 31)), 5.0), +} +CAMPAIGNS = [ + # campaign id, name, channel, starts, ends, budget in GBP pence, utm source and medium + ( + "WINTER-WADDLE", + "Winter waddle sale", + "paid_search", + date(2026, 1, 5), + date(2026, 1, 31), + 300000, + ("search_ads", "cpc"), + ), + ( + "POND-WEEK", + "Pond week", + "paid_social", + date(2026, 2, 1), + date(2026, 2, 28), + 250000, + ("social_ads", "paid_social"), + ), + ( + "POND-WEEK-EMAIL", + "Pond week newsletter", + "email", + date(2026, 2, 1), + date(2026, 2, 28), + 20000, + ("newsletter", "email"), + ), + ( + "SPRING-SPLASH", + "Spring splash", + "paid_search", + date(2026, 3, 1), + date(2026, 4, 15), + 500000, + ("search_ads", "cpc"), + ), + ( + "GOLDEN-EGG", + "Golden egg hunt", + "influencer", + date(2026, 3, 1), + date(2026, 4, 30), + 150000, + ("creator", "influencer"), + ), + ( + "ALWAYS-ON", + "Always-on affiliates", + "affiliate", + date(2026, 1, 1), + None, + 400000, + ("partner_site", "affiliate"), + ), +] + + +def utm_for(when: datetime) -> list[tuple[str, str, str]]: + """UTM tags of the campaigns running on a day; the campaign tag is the lower-cased id.""" + + day = when.date() + tags = [ + (source, medium, campaign_id.lower()) + for campaign_id, _, _, starts, ends, _, (source, medium) in CAMPAIGNS + if starts <= day and (ends is None or day <= ends) + ] + return tags + [("", "organic", ""), ("", "referral", "")] + + +STATUS_FLOW = ["pending", "paid", "packed", "shipped", "delivered"] + + +def _orders(customers: list[Customer], variants: list[Variant], scale: int) -> list[Order]: + r = rng("world", "orders") + live = [c for c in customers if not c.deleted] + orders: list[Order] = [] + count = 1200 * scale + end = VALUATION_DATE + # A few orders pinned to daylight-saving changes and to the edges of the reporting window. + pinned = [ + datetime(2026, 3, 8, 6, 55), # US clocks go forward at 07:00 UTC + datetime(2026, 3, 8, 7, 5), + datetime(2026, 3, 29, 0, 55), # EU clocks go forward at 01:00 UTC + datetime(2026, 3, 29, 1, 10), + datetime(2026, 1, 1, 0, 0, 1), + datetime(2026, 3, 31, 23, 59, 59), + datetime(2026, 3, 31, 14, 30), # after the Tokyo and Sydney business day ended + ] + for i in range(1, count + 1): + customer = live[0] if i % 97 == 1 else r.choice(live) + sf = STOREFRONTS[customer.storefront] + if i <= len(pinned): + placed = pinned[i - 1] + else: + placed = random_instant(r, ORDERS_START, end) + if placed < customer.created_at: + placed = customer.created_at + timedelta(minutes=r.randint(5, 600)) + if placed.date() > end: + placed = at(end, 12) + # Status progression; recent orders have not finished their journey yet. + age_days = (at(end, 23, 59) - placed).total_seconds() / 86400 + times: list[tuple[str, datetime]] = [("pending", placed)] + status = "pending" + roll = r.random() + if roll < 0.03: + status = "pending" + elif roll < 0.08: + status = "cancelled" + times.append(("cancelled", placed + timedelta(hours=r.randint(1, 30)))) + else: + when = placed + timedelta(minutes=r.randint(1, 20)) + times.append(("paid", when)) + status = "paid" + for step, hours in ( + ("packed", (4, 30)), + ("shipped", (2, 40)), + ("delivered", (20, 120)), + ): + when = when + timedelta(hours=r.randint(*hours), minutes=r.randint(0, 59)) + if (when - placed).total_seconds() / 86400 > age_days: + break + times.append((step, when)) + status = step + if status == "delivered" and chance(r, 0.06): + when = when + timedelta(days=r.randint(3, 12)) + if (when - placed).total_seconds() / 86400 <= age_days: + times.append(("returned", when)) + status = "returned" + lines: list[OrderLine] = [] + promo = None + for code, (starts, ends, _) in PROMOTIONS.items(): + if starts <= placed <= ends and chance(r, 0.08): + promo = code + break + pct = PROMOTIONS[promo][2] if promo else 0.0 + line_count = weighted(r, {1: 45, 2: 30, 3: 15, 4: 6, 6: 4}) + used: set[str] = set() + for n in range(1, int(line_count) + 1): # type: ignore[call-overload] + variant = r.choice(variants) + if variant.sku in used: + continue + used.add(variant.sku) + quantity = weighted(r, {1: 70, 2: 18, 3: 7, 6: 4, 12: 1}) + unit = local_price(variant.base_usd, sf.currency) + if chance(r, 0.1): + unit = int(unit * 0.8) + gross = int(quantity) * unit # type: ignore[call-overload] + discount = int(round(gross * pct / 100)) if promo else 0 + tax_rate = {"EU": 0.2, "US": 0.0, "APAC": 0.1, "MENA": 0.0}[sf.region] + tax = int(round((gross - discount) * tax_rate)) + lines.append( + OrderLine( + order_line_id=f"OL{i:06d}-{n}", + line_no=n, + sku=variant.sku, + quantity=int(quantity), # type: ignore[call-overload] + unit_minor=unit, + tax_minor=tax, + discount_minor=discount, + ) + ) + utm = r.choice(utm_for(placed)) if chance(r, 0.45) else None + channel = weighted(r, {"web": 62, "app": 22, "store": 5, "phone": 3, None: 8}) + orders.append( + Order( + order_id=f"DS-O-{i:06d}", + customer_id=customer.customer_id, + storefront=customer.storefront, + currency=sf.currency, + exponent=sf.exponent, + placed_at=placed, + status=status, + status_times=times, + address_id=customer.default_address, + channel=channel, # type: ignore[arg-type] + utm=utm, + promo_code=promo, + lines=lines, + ) + ) + orders.sort(key=lambda o: (o.placed_at, o.order_id)) + return orders + + +@lru_cache(maxsize=4) +def build_world(scale: int = 1) -> World: + """The deterministic world for one volume scale. Scale 1 is the small default.""" + + if scale < 1: + raise ValueError("scale must be a positive integer") + products, variants = _products(scale) + customers = _customers(scale) + orders = _orders(customers, variants, scale) + return World( + scale=scale, + customers=customers, + products=products, + variants=variants, + orders=orders, + promotions=PROMOTIONS, + ) + + +def clickstream_days() -> list[date]: + return [ + CLICKSTREAM_START + timedelta(days=i) + for i in range((VALUATION_DATE - CLICKSTREAM_START).days + 1) + ] diff --git a/benchmarks/duckshop/project/python/loaders/_writer.py b/benchmarks/duckshop/project/python/loaders/_writer.py new file mode 100644 index 0000000000..a6eb0692fb --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/_writer.py @@ -0,0 +1,91 @@ +"""Write generated rows into a Snowflake source table from a self-managed loader. + +SQLBuild's row-returning loaders render JSON values as `PARSE_JSON(...)` inside a `VALUES` list, +and Snowflake only accepts constants there. These loaders therefore create the table themselves and +insert each batch with `INSERT ... SELECT PARSE_JSON(columnN) ... FROM VALUES (...)`, where the +function call sits in the select list. Once the framework renders JSON that way, the loaders can +return their rows directly instead. +""" + +from __future__ import annotations + +import json +from datetime import date, datetime, time +from decimal import Decimal +from functools import lru_cache +from pathlib import Path + +import yaml + +SOURCES_DIR = Path(__file__).resolve().parents[2] / "sources" +BATCH_SIZE = 500 +JSON_TYPES = {"VARIANT": "", "OBJECT": "::OBJECT", "ARRAY": "::ARRAY"} + + +@lru_cache(maxsize=1) +def source_columns() -> dict[str, tuple[tuple[str, str], ...]]: + """Declared (name, type) columns of every managed source, in declaration order.""" + + columns: dict[str, tuple[tuple[str, str], ...]] = {} + for path in sorted(SOURCES_DIR.glob("*.yml")): + for source in yaml.safe_load(path.read_text())["sources"]: + if source.get("managed"): + columns[source["name"]] = tuple((c["name"], c["type"]) for c in source["columns"]) + return columns + + +def quote(text: str) -> str: + """A Snowflake string literal; backslashes are escapes there, so they are doubled too.""" + + return "'" + text.replace("\\", "\\\\").replace("'", "''") + "'" + + +def literal(value: object) -> str: + if value is None: + return "NULL" + if isinstance(value, bool): + return "TRUE" if value else "FALSE" + if isinstance(value, int | Decimal): + return str(value) + if isinstance(value, float): + return repr(value) + if isinstance(value, datetime | date | time): + return quote(value.isoformat()) + if isinstance(value, dict | list): + return quote(json.dumps(value, sort_keys=True)) + return quote(str(value)) + + +def create_sql(destination: str, columns: tuple[tuple[str, str], ...]) -> str: + body = ", ".join(f"{name} {sql_type}" for name, sql_type in columns) + return f"CREATE OR REPLACE TABLE {destination} ({body})" + + +def insert_sql( + destination: str, columns: tuple[tuple[str, str], ...], rows: list[dict[str, object]] +) -> str: + """One INSERT for a batch: typed casts and JSON parsing happen in the select list.""" + + names = ", ".join(name for name, _ in columns) + projections = [] + for index, (_, sql_type) in enumerate(columns, start=1): + json_cast = JSON_TYPES.get(sql_type.upper()) + if json_cast is not None: + projections.append(f"PARSE_JSON(column{index}){json_cast}") + else: + projections.append(f"column{index}::{sql_type}") + values = ",\n".join( + "(" + ", ".join(literal(row[name]) for name, _ in columns) + ")" for row in rows + ) + select = ", ".join(projections) + return f"INSERT INTO {destination} ({names})\nSELECT {select}\nFROM VALUES\n{values}" + + +def statements(destination: str, source_name: str, rows: list[dict[str, object]]) -> list[str]: + """Every statement that replaces one source table with the given rows.""" + + columns = source_columns()[source_name] + sql = [create_sql(destination, columns)] + for start in range(0, len(rows), BATCH_SIZE): + sql.append(insert_sql(destination, columns, rows[start : start + BATCH_SIZE])) + return sql diff --git a/benchmarks/duckshop/project/python/loaders/catalogue.py b/benchmarks/duckshop/project/python/loaders/catalogue.py new file mode 100644 index 0000000000..2bf8265a3e --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/catalogue.py @@ -0,0 +1,56 @@ +"""Loaders for the synthetic catalogue source tables, generated in python.loaders._catalogue.""" + +from python.loaders._registry import load + +from sqlbuild.executor.load.models import LoaderContext +from sqlbuild.loaders import loader + + +@loader +def catalogue__brands(ctx: LoaderContext) -> object: + return load(ctx, "catalogue__brands") + + +@loader +def catalogue__categories(ctx: LoaderContext) -> object: + return load(ctx, "catalogue__categories") + + +@loader +def catalogue__products(ctx: LoaderContext) -> object: + return load(ctx, "catalogue__products") + + +@loader +def catalogue__product_variants(ctx: LoaderContext) -> object: + return load(ctx, "catalogue__product_variants") + + +@loader +def catalogue__price_lists(ctx: LoaderContext) -> object: + return load(ctx, "catalogue__price_lists") + + +@loader +def catalogue__product_media(ctx: LoaderContext) -> object: + return load(ctx, "catalogue__product_media") + + +@loader +def catalogue__bundles(ctx: LoaderContext) -> object: + return load(ctx, "catalogue__bundles") + + +@loader +def catalogue__bundle_components(ctx: LoaderContext) -> object: + return load(ctx, "catalogue__bundle_components") + + +@loader +def catalogue__suppliers(ctx: LoaderContext) -> object: + return load(ctx, "catalogue__suppliers") + + +@loader +def catalogue__supplier_costs(ctx: LoaderContext) -> object: + return load(ctx, "catalogue__supplier_costs") diff --git a/benchmarks/duckshop/project/python/loaders/clickstream.py b/benchmarks/duckshop/project/python/loaders/clickstream.py new file mode 100644 index 0000000000..f7e201c7de --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/clickstream.py @@ -0,0 +1,31 @@ +"""Loaders for the synthetic clickstream source tables, generated in python.loaders._clickstream.""" + +from python.loaders._registry import load + +from sqlbuild.executor.load.models import LoaderContext +from sqlbuild.loaders import loader + + +@loader +def clickstream__web_events(ctx: LoaderContext) -> object: + return load(ctx, "clickstream__web_events") + + +@loader +def clickstream__app_events(ctx: LoaderContext) -> object: + return load(ctx, "clickstream__app_events") + + +@loader +def clickstream__search_queries(ctx: LoaderContext) -> object: + return load(ctx, "clickstream__search_queries") + + +@loader +def clickstream__email_events(ctx: LoaderContext) -> object: + return load(ctx, "clickstream__email_events") + + +@loader +def clickstream__experiment_assignments(ctx: LoaderContext) -> object: + return load(ctx, "clickstream__experiment_assignments") diff --git a/benchmarks/duckshop/project/python/loaders/finance.py b/benchmarks/duckshop/project/python/loaders/finance.py new file mode 100644 index 0000000000..35a1ffc738 --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/finance.py @@ -0,0 +1,36 @@ +"""Loaders for the synthetic finance source tables, generated in python.loaders._finance.""" + +from python.loaders._registry import load + +from sqlbuild.executor.load.models import LoaderContext +from sqlbuild.loaders import loader + + +@loader +def finance__invoices(ctx: LoaderContext) -> object: + return load(ctx, "finance__invoices") + + +@loader +def finance__ledger_entries(ctx: LoaderContext) -> object: + return load(ctx, "finance__ledger_entries") + + +@loader +def finance__payouts(ctx: LoaderContext) -> object: + return load(ctx, "finance__payouts") + + +@loader +def finance__tax_rates(ctx: LoaderContext) -> object: + return load(ctx, "finance__tax_rates") + + +@loader +def finance__chargebacks(ctx: LoaderContext) -> object: + return load(ctx, "finance__chargebacks") + + +@loader +def finance__settlement_reports(ctx: LoaderContext) -> object: + return load(ctx, "finance__settlement_reports") diff --git a/benchmarks/duckshop/project/python/loaders/fulfilment.py b/benchmarks/duckshop/project/python/loaders/fulfilment.py new file mode 100644 index 0000000000..fb28e6cc24 --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/fulfilment.py @@ -0,0 +1,51 @@ +"""Loaders for the synthetic fulfilment source tables, generated in python.loaders._fulfilment.""" + +from python.loaders._registry import load + +from sqlbuild.executor.load.models import LoaderContext +from sqlbuild.loaders import loader + + +@loader +def fulfilment__shipments(ctx: LoaderContext) -> object: + return load(ctx, "fulfilment__shipments") + + +@loader +def fulfilment__shipment_packages(ctx: LoaderContext) -> object: + return load(ctx, "fulfilment__shipment_packages") + + +@loader +def fulfilment__carrier_webhooks(ctx: LoaderContext) -> object: + return load(ctx, "fulfilment__carrier_webhooks") + + +@loader +def fulfilment__returns(ctx: LoaderContext) -> object: + return load(ctx, "fulfilment__returns") + + +@loader +def fulfilment__return_inspections(ctx: LoaderContext) -> object: + return load(ctx, "fulfilment__return_inspections") + + +@loader +def fulfilment__carrier_rate_cards(ctx: LoaderContext) -> object: + return load(ctx, "fulfilment__carrier_rate_cards") + + +@loader +def fulfilment__delivery_slots(ctx: LoaderContext) -> object: + return load(ctx, "fulfilment__delivery_slots") + + +@loader +def fulfilment__pick_tasks(ctx: LoaderContext) -> object: + return load(ctx, "fulfilment__pick_tasks") + + +@loader +def fulfilment__delivery_exceptions(ctx: LoaderContext) -> object: + return load(ctx, "fulfilment__delivery_exceptions") diff --git a/benchmarks/duckshop/project/python/loaders/growth.py b/benchmarks/duckshop/project/python/loaders/growth.py new file mode 100644 index 0000000000..94af00577d --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/growth.py @@ -0,0 +1,41 @@ +"""Loaders for the synthetic growth source tables, generated in python.loaders._growth.""" + +from python.loaders._registry import load + +from sqlbuild.executor.load.models import LoaderContext +from sqlbuild.loaders import loader + + +@loader +def growth__campaigns(ctx: LoaderContext) -> object: + return load(ctx, "growth__campaigns") + + +@loader +def growth__campaign_spend(ctx: LoaderContext) -> object: + return load(ctx, "growth__campaign_spend") + + +@loader +def growth__affiliate_conversions(ctx: LoaderContext) -> object: + return load(ctx, "growth__affiliate_conversions") + + +@loader +def growth__referral_invites(ctx: LoaderContext) -> object: + return load(ctx, "growth__referral_invites") + + +@loader +def growth__newsletter_subscriptions(ctx: LoaderContext) -> object: + return load(ctx, "growth__newsletter_subscriptions") + + +@loader +def growth__survey_responses(ctx: LoaderContext) -> object: + return load(ctx, "growth__survey_responses") + + +@loader +def growth__influencer_posts(ctx: LoaderContext) -> object: + return load(ctx, "growth__influencer_posts") diff --git a/benchmarks/duckshop/project/python/loaders/inventory.py b/benchmarks/duckshop/project/python/loaders/inventory.py new file mode 100644 index 0000000000..064aa2b7aa --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/inventory.py @@ -0,0 +1,46 @@ +"""Loaders for the synthetic inventory source tables, generated in python.loaders._inventory.""" + +from python.loaders._registry import load + +from sqlbuild.executor.load.models import LoaderContext +from sqlbuild.loaders import loader + + +@loader +def inventory__warehouses(ctx: LoaderContext) -> object: + return load(ctx, "inventory__warehouses") + + +@loader +def inventory__stock_movements(ctx: LoaderContext) -> object: + return load(ctx, "inventory__stock_movements") + + +@loader +def inventory__stock_counts(ctx: LoaderContext) -> object: + return load(ctx, "inventory__stock_counts") + + +@loader +def inventory__purchase_orders(ctx: LoaderContext) -> object: + return load(ctx, "inventory__purchase_orders") + + +@loader +def inventory__po_receipts(ctx: LoaderContext) -> object: + return load(ctx, "inventory__po_receipts") + + +@loader +def inventory__stock_transfers(ctx: LoaderContext) -> object: + return load(ctx, "inventory__stock_transfers") + + +@loader +def inventory__stock_reservations(ctx: LoaderContext) -> object: + return load(ctx, "inventory__stock_reservations") + + +@loader +def inventory__bin_locations(ctx: LoaderContext) -> object: + return load(ctx, "inventory__bin_locations") diff --git a/benchmarks/duckshop/project/python/loaders/partners.py b/benchmarks/duckshop/project/python/loaders/partners.py new file mode 100644 index 0000000000..0ec8dfc147 --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/partners.py @@ -0,0 +1,21 @@ +"""Loaders for the synthetic partners source tables, generated in python.loaders._partners.""" + +from python.loaders._registry import load + +from sqlbuild.executor.load.models import LoaderContext +from sqlbuild.loaders import loader + + +@loader +def partners__partner_orders(ctx: LoaderContext) -> object: + return load(ctx, "partners__partner_orders") + + +@loader +def partners__partner_listings(ctx: LoaderContext) -> object: + return load(ctx, "partners__partner_listings") + + +@loader +def partners__partner_settlements(ctx: LoaderContext) -> object: + return load(ctx, "partners__partner_settlements") diff --git a/benchmarks/duckshop/project/python/loaders/storefront.py b/benchmarks/duckshop/project/python/loaders/storefront.py new file mode 100644 index 0000000000..4ca5a0e4aa --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/storefront.py @@ -0,0 +1,106 @@ +"""Loaders for the synthetic storefront source tables, generated in python.loaders._storefront.""" + +from python.loaders._registry import load + +from sqlbuild.executor.load.models import LoaderContext +from sqlbuild.loaders import loader + + +@loader +def storefront__customers(ctx: LoaderContext) -> object: + return load(ctx, "storefront__customers") + + +@loader +def storefront__customer_addresses(ctx: LoaderContext) -> object: + return load(ctx, "storefront__customer_addresses") + + +@loader +def storefront__customer_consents(ctx: LoaderContext) -> object: + return load(ctx, "storefront__customer_consents") + + +@loader +def storefront__loyalty_accounts(ctx: LoaderContext) -> object: + return load(ctx, "storefront__loyalty_accounts") + + +@loader +def storefront__loyalty_ledger(ctx: LoaderContext) -> object: + return load(ctx, "storefront__loyalty_ledger") + + +@loader +def storefront__wishlists(ctx: LoaderContext) -> object: + return load(ctx, "storefront__wishlists") + + +@loader +def storefront__gift_cards(ctx: LoaderContext) -> object: + return load(ctx, "storefront__gift_cards") + + +@loader +def storefront__gift_card_transactions(ctx: LoaderContext) -> object: + return load(ctx, "storefront__gift_card_transactions") + + +@loader +def storefront__orders(ctx: LoaderContext) -> object: + return load(ctx, "storefront__orders") + + +@loader +def storefront__order_lines(ctx: LoaderContext) -> object: + return load(ctx, "storefront__order_lines") + + +@loader +def storefront__order_status_events(ctx: LoaderContext) -> object: + return load(ctx, "storefront__order_status_events") + + +@loader +def storefront__payments(ctx: LoaderContext) -> object: + return load(ctx, "storefront__payments") + + +@loader +def storefront__refunds(ctx: LoaderContext) -> object: + return load(ctx, "storefront__refunds") + + +@loader +def storefront__promotions(ctx: LoaderContext) -> object: + return load(ctx, "storefront__promotions") + + +@loader +def storefront__promotion_redemptions(ctx: LoaderContext) -> object: + return load(ctx, "storefront__promotion_redemptions") + + +@loader +def storefront__carts(ctx: LoaderContext) -> object: + return load(ctx, "storefront__carts") + + +@loader +def storefront__reviews(ctx: LoaderContext) -> object: + return load(ctx, "storefront__reviews") + + +@loader +def storefront__subscriptions(ctx: LoaderContext) -> object: + return load(ctx, "storefront__subscriptions") + + +@loader +def storefront__subscription_events(ctx: LoaderContext) -> object: + return load(ctx, "storefront__subscription_events") + + +@loader +def storefront__store_credit_ledger(ctx: LoaderContext) -> object: + return load(ctx, "storefront__store_credit_ledger") diff --git a/benchmarks/duckshop/project/python/loaders/support.py b/benchmarks/duckshop/project/python/loaders/support.py new file mode 100644 index 0000000000..e139df27b9 --- /dev/null +++ b/benchmarks/duckshop/project/python/loaders/support.py @@ -0,0 +1,46 @@ +"""Loaders for the synthetic support source tables, generated in python.loaders._support.""" + +from python.loaders._registry import load + +from sqlbuild.executor.load.models import LoaderContext +from sqlbuild.loaders import loader + + +@loader +def support__tickets(ctx: LoaderContext) -> object: + return load(ctx, "support__tickets") + + +@loader +def support__ticket_comments(ctx: LoaderContext) -> object: + return load(ctx, "support__ticket_comments") + + +@loader +def support__csat_responses(ctx: LoaderContext) -> object: + return load(ctx, "support__csat_responses") + + +@loader +def support__agents(ctx: LoaderContext) -> object: + return load(ctx, "support__agents") + + +@loader +def support__agent_shifts(ctx: LoaderContext) -> object: + return load(ctx, "support__agent_shifts") + + +@loader +def support__chat_transcripts(ctx: LoaderContext) -> object: + return load(ctx, "support__chat_transcripts") + + +@loader +def support__call_records(ctx: LoaderContext) -> object: + return load(ctx, "support__call_records") + + +@loader +def support__sla_policies(ctx: LoaderContext) -> object: + return load(ctx, "support__sla_policies") diff --git a/benchmarks/duckshop/project/rules/staging_dedupe.py b/benchmarks/duckshop/project/rules/staging_dedupe.py new file mode 100644 index 0000000000..093fbf5413 --- /dev/null +++ b/benchmarks/duckshop/project/rules/staging_dedupe.py @@ -0,0 +1,43 @@ +"""DuckShop house rules for the staging layer.""" + +import re + +from sqlbuild.rules import Finding, Model, RuleContext, rule + +_SOURCE_BLOCK = re.compile(r"^ - name: (\S+)\n(.*?)(?=^ - name: |\Z)", re.MULTILINE | re.DOTALL) +_SOURCE_CALL = re.compile(r'__source\(\s*"([^"]+)"\s*\)') +_CHANGE_LOG_MARKER = re.compile(r"^\s+- name: _op\s*$", re.MULTILINE) + + +def _change_log_sources(ctx: RuleContext) -> set[str]: + names: set[str] = set() + for path in ctx.project.tree.glob("sources/*.yml"): + text = ctx.project.tree.read_text(path.value) + for match in _SOURCE_BLOCK.finditer(text): + if _CHANGE_LOG_MARKER.search(match.group(2)): + names.add(match.group(1)) + return names + + +@rule( + code="XSQBRDUCK001", + message="Staging models over a change-log source must deduplicate it with @sql.cdc_latest", + remediation=( + "Read the source in an import CTE and follow it with @sql.cdc_latest(, ), then " + "select from cdc_latest so deletes and superseded versions are handled the same way " + "everywhere." + ), +) +def staging_uses_cdc_latest(*, model: Model, ctx: RuleContext) -> list[Finding]: + parts = ctx.project.tree.relative_parts(path=model.path, under="models") + if not parts or parts[0] != "staging": + return [] + sql = ctx.sql.for_model(model).authored.source + if "@sql.cdc_latest(" in sql: + return [] + change_logs = _change_log_sources(ctx) + for match in _SOURCE_CALL.finditer(sql): + if match.group(1) in change_logs: + line = sql[: match.start()].count("\n") + 1 + return [ctx.finding(subject=model, line=line)] + return [] diff --git a/benchmarks/duckshop/project/seeds/carriers.csv b/benchmarks/duckshop/project/seeds/carriers.csv new file mode 100644 index 0000000000..4eb5c7e2ad --- /dev/null +++ b/benchmarks/duckshop/project/seeds/carriers.csv @@ -0,0 +1,5 @@ +carrier_code,carrier_name,tracking_url_template,is_express +FLAPWING,Flapwing Parcels,https://track.example.com/flapwing/{tracking_number},false +WINGBEAT,Wingbeat Couriers,https://track.example.com/wingbeat/{tracking_number},true +POSTAL,National Postal Service,https://track.example.com/postal/{tracking_number},false +WEBFOOT,Webfoot Freight,,false diff --git a/benchmarks/duckshop/project/seeds/channel_mapping.csv b/benchmarks/duckshop/project/seeds/channel_mapping.csv new file mode 100644 index 0000000000..cf975ce58d --- /dev/null +++ b/benchmarks/duckshop/project/seeds/channel_mapping.csv @@ -0,0 +1,8 @@ +utm_source,utm_medium,channel_code,is_paid +search_ads,cpc,PAID_SEARCH,true +social_ads,paid_social,PAID_SOCIAL,true +newsletter,email,EMAIL,false +partner_site,affiliate,AFFILIATE,true +creator,influencer,INFLUENCER,true +-,organic,ORGANIC,false +-,referral,REFERRAL,false diff --git a/benchmarks/duckshop/project/seeds/countries.csv b/benchmarks/duckshop/project/seeds/countries.csv new file mode 100644 index 0000000000..ae854008b1 --- /dev/null +++ b/benchmarks/duckshop/project/seeds/countries.csv @@ -0,0 +1,10 @@ +country_code,country_name,region_code,default_timezone,currency_code,vat_registered +DE,Germany,EU,Europe/Berlin,EUR,true +FR,France,EU,Europe/Paris,EUR,true +NL,Netherlands,EU,Europe/Amsterdam,EUR,true +GB,United Kingdom,EU,Europe/London,GBP,true +US,United States,US,America/New_York,USD,false +CA,Canada,US,America/Toronto,USD,n/a +JP,Japan,APAC,Asia/Tokyo,JPY,true +AU,Australia,APAC,Australia/Sydney,AUD,true +KW,Kuwait,MENA,Asia/Kuwait,KWD,false diff --git a/benchmarks/duckshop/project/seeds/currencies.csv b/benchmarks/duckshop/project/seeds/currencies.csv new file mode 100644 index 0000000000..7f5f6487b1 --- /dev/null +++ b/benchmarks/duckshop/project/seeds/currencies.csv @@ -0,0 +1,7 @@ +currency_code,currency_name,minor_unit_exponent,symbol +USD,US dollar,2,$ +EUR,Euro,2,€ +GBP,Pound sterling,2,£ +JPY,Japanese yen,0,¥ +AUD,Australian dollar,2,A$ +KWD,Kuwaiti dinar,3,KD diff --git a/benchmarks/duckshop/project/seeds/holiday_calendar.csv b/benchmarks/duckshop/project/seeds/holiday_calendar.csv new file mode 100644 index 0000000000..5eeb2cd193 --- /dev/null +++ b/benchmarks/duckshop/project/seeds/holiday_calendar.csv @@ -0,0 +1,17 @@ +region_code;holiday_date;holiday_name +EU;2026-01-01;New Year's Day +EU;2026-04-03;Good Friday +EU;2026-04-06;Easter Monday +EU;2026-05-01;Labour Day +EU;2026-12-25;Christmas Day +US;2026-01-01;New Year's Day +US;2026-05-25;Memorial Day +US;2026-07-04;Independence Day +US;2026-11-26;Thanksgiving Day +US;2026-12-25;Christmas Day +APAC;2026-01-01;New Year's Day +APAC;2026-01-26;National Day +APAC;2026-04-29;Spring Holiday +APAC;2026-12-25;Christmas Day +MENA;2026-02-25;National Day +MENA;2026-02-26;Liberation Day diff --git a/benchmarks/duckshop/project/seeds/seeds.yml b/benchmarks/duckshop/project/seeds/seeds.yml new file mode 100644 index 0000000000..63aaf7bba2 --- /dev/null +++ b/benchmarks/duckshop/project/seeds/seeds.yml @@ -0,0 +1,109 @@ +seeds: + - name: currencies + description: ISO currencies sold in, with the exponent of their minor unit (0, 2 or 3). + columns: + - name: currency_code + type: VARCHAR + nullable: false + audits: + - not_null + - unique + - name: currency_name + type: VARCHAR + - name: minor_unit_exponent + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - accepted_values: + values: [0, 2, 3] + - name: symbol + type: VARCHAR + + - name: countries + description: Countries served, with their storefront region and default time zone. + csv_settings: + na_values: + vat_registered: ["n/a"] + keep_default_na: false + columns: + - name: country_code + type: VARCHAR + nullable: false + audits: + - not_null + - unique + - name: country_name + type: VARCHAR + - name: region_code + type: VARCHAR + nullable: false + audits: + - not_null + - name: default_timezone + type: VARCHAR + - name: currency_code + type: VARCHAR + audits: + - relationships: + to: __seed("currencies") + field: currency_code + - name: vat_registered + type: BOOLEAN + + - name: carriers + description: Delivery carriers and their public tracking URL templates. + csv_settings: + na_values: + tracking_url_template: [""] + columns: + - name: carrier_code + type: VARCHAR + nullable: false + audits: + - not_null + - unique + - name: carrier_name + type: VARCHAR + - name: tracking_url_template + type: VARCHAR + - name: is_express + type: BOOLEAN + + - name: holiday_calendar + description: Public holidays per storefront region, used for business-day logic. + csv_settings: + delimiter: ";" + encoding: utf-8 + columns: + - name: region_code + type: VARCHAR + nullable: false + audits: + - not_null + - name: holiday_date + type: DATE + nullable: false + audits: + - not_null + - name: holiday_name + type: VARCHAR + + - name: channel_mapping + description: Maps UTM source and medium pairs to reporting channels; "-" matches any source. + columns: + - name: utm_source + type: VARCHAR + nullable: false + - name: utm_medium + type: VARCHAR + nullable: false + audits: + - not_null + - name: channel_code + type: VARCHAR + nullable: false + audits: + - not_null + - name: is_paid + type: BOOLEAN diff --git a/benchmarks/duckshop/project/sources/catalogue.yml b/benchmarks/duckshop/project/sources/catalogue.yml new file mode 100644 index 0000000000..8d67e209a2 --- /dev/null +++ b/benchmarks/duckshop/project/sources/catalogue.yml @@ -0,0 +1,372 @@ +# Raw catalogue tables, generated by the synthetic data loaders in python/loaders/catalogue.py. +sources: + - name: catalogue__products + description: Product master change log (rubber ducks, decoys, duck-print textiles, pond and bath goods) + managed: true + contract: enforced + columns: + - name: product_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: title + type: VARCHAR + - name: category_path + type: VARCHAR + - name: brand_code + type: VARCHAR + - name: launched_on + type: DATE + - name: is_active + type: BOOLEAN + - name: specs + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: catalogue__product_variants + description: Sellable variants change log + managed: true + contract: enforced + columns: + - name: sku + type: VARCHAR + nullable: false + audits: + - not_null + - name: product_id + type: VARCHAR + - name: colour + type: VARCHAR + - name: size_label + type: VARCHAR + - name: weight_grams + type: NUMBER(38,0) + - name: barcodes + type: ARRAY + - name: attributes + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: catalogue__price_lists + description: List prices per currency change log + managed: true + contract: enforced + columns: + - name: price_list_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: sku + type: VARCHAR + - name: currency_code + type: VARCHAR + - name: list_amount_minor + type: NUMBER(38,0) + - name: valid_from + type: TIMESTAMP_TZ + - name: valid_to + type: TIMESTAMP_TZ + - name: conditions + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: catalogue__categories + description: Category tree change log, such as ducks/rubber-ducks/collectible-series + managed: true + contract: enforced + columns: + - name: category_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: parent_category_id + type: VARCHAR + - name: slug + type: VARCHAR + - name: display_name + type: VARCHAR + - name: sort_order + type: NUMBER(38,0) + - name: translations + type: OBJECT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: catalogue__brands + description: Brands change log, including the Mallard & Co. house brand + managed: true + contract: enforced + columns: + - name: brand_code + type: VARCHAR + nullable: false + audits: + - not_null + - name: brand_name + type: VARCHAR + - name: country_code + type: VARCHAR + - name: is_private_label + type: BOOLEAN + - name: metadata + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: catalogue__product_media + description: Product images and videos change log + managed: true + contract: enforced + columns: + - name: media_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: product_id + type: VARCHAR + - name: url + type: VARCHAR + - name: media_type + type: VARCHAR + - name: position + type: NUMBER(38,0) + - name: dimensions + type: OBJECT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: catalogue__bundles + description: Product bundles change log, such as the Paddling Six mini-duck set + managed: true + contract: enforced + columns: + - name: bundle_sku + type: VARCHAR + nullable: false + audits: + - not_null + - name: title + type: VARCHAR + - name: active_from + type: DATE + - name: active_to + type: DATE + - name: pricing + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: catalogue__bundle_components + description: Bundle components change log + managed: true + contract: enforced + columns: + - name: component_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: bundle_sku + type: VARCHAR + - name: component_sku + type: VARCHAR + - name: quantity + type: NUMBER(38,0) + - name: options + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: catalogue__suppliers + description: Suppliers change log + managed: true + contract: enforced + columns: + - name: supplier_code + type: VARCHAR + nullable: false + audits: + - not_null + - name: supplier_name + type: VARCHAR + - name: country_code + type: VARCHAR + - name: payment_terms + type: VARIANT + - name: contacts + type: ARRAY + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: catalogue__supplier_costs + description: Supplier unit costs change log + managed: true + contract: enforced + columns: + - name: cost_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: supplier_code + type: VARCHAR + - name: sku + type: VARCHAR + - name: currency_code + type: VARCHAR + - name: unit_cost_minor + type: NUMBER(38,0) + - name: effective_from + type: DATE + - name: volume_breaks + type: ARRAY + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null diff --git a/benchmarks/duckshop/project/sources/clickstream.yml b/benchmarks/duckshop/project/sources/clickstream.yml new file mode 100644 index 0000000000..eff6281f3b --- /dev/null +++ b/benchmarks/duckshop/project/sources/clickstream.yml @@ -0,0 +1,134 @@ +# Raw clickstream tables, generated by the synthetic data loaders in python/loaders/clickstream.py. +sources: + - name: clickstream__web_events + description: Append-only web clickstream; event is the raw tracker payload + managed: true + contract: enforced + freshness: + column: received_at + lag_tolerance: 15m + age_policy: + warn_after: 6h + error_after: 1d + columns: + - name: event_id + type: VARCHAR + nullable: false + audits: + - not_null + - unique_per_delivery + - name: received_at + type: TIMESTAMP_NTZ + - name: event + type: VARIANT + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: clickstream__app_events + description: Append-only mobile app events + managed: true + contract: enforced + columns: + - name: event_id + type: VARCHAR + nullable: false + audits: + - not_null + - unique_per_delivery + - name: received_at + type: TIMESTAMP_NTZ + - name: platform + type: VARCHAR + - name: app_version + type: VARCHAR + - name: payload + type: VARIANT + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: clickstream__search_queries + description: Append-only site search log + managed: true + contract: enforced + columns: + - name: query_id + type: VARCHAR + nullable: false + audits: + - not_null + - unique_per_delivery + - name: anonymous_id + type: VARCHAR + - name: query_text + type: VARCHAR + - name: results_count + type: NUMBER(38,0) + - name: searched_at + type: TIMESTAMP_TZ + - name: filters + type: VARIANT + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: clickstream__email_events + description: Append-only email delivery and engagement events + managed: true + contract: enforced + columns: + - name: email_event_id + type: VARCHAR + nullable: false + audits: + - not_null + - unique_per_delivery + - name: message_id + type: VARCHAR + - name: event_type + type: VARCHAR + - name: recipient_email + type: VARCHAR + - name: campaign_id + type: VARCHAR + - name: occurred_at + type: TIMESTAMP_TZ + - name: details + type: VARIANT + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: clickstream__experiment_assignments + description: Append-only experiment assignments + managed: true + contract: enforced + columns: + - name: assignment_id + type: VARCHAR + nullable: false + audits: + - not_null + - unique_per_delivery + - name: experiment_key + type: VARCHAR + - name: variant_key + type: VARCHAR + - name: anonymous_id + type: VARCHAR + - name: customer_id + type: VARCHAR + - name: assigned_at + type: TIMESTAMP_TZ + - name: context + type: VARIANT + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null diff --git a/benchmarks/duckshop/project/sources/finance.yml b/benchmarks/duckshop/project/sources/finance.yml new file mode 100644 index 0000000000..fe865daf87 --- /dev/null +++ b/benchmarks/duckshop/project/sources/finance.yml @@ -0,0 +1,229 @@ +# Raw finance tables, generated by the synthetic data loaders in python/loaders/finance.py. +sources: + - name: finance__invoices + description: Customer invoices change log + managed: true + contract: enforced + columns: + - name: invoice_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: order_id + type: VARCHAR + - name: invoice_number + type: VARCHAR + - name: issued_at + type: TIMESTAMP_TZ + - name: currency_code + type: VARCHAR + - name: totals + type: VARIANT + - name: tax_lines + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: finance__ledger_entries + description: General ledger postings change log + managed: true + contract: enforced + columns: + - name: entry_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: journal_code + type: VARCHAR + - name: account_code + type: VARCHAR + - name: debit_minor + type: NUMBER(38,0) + - name: credit_minor + type: NUMBER(38,0) + - name: currency_code + type: VARCHAR + - name: posted_at + type: TIMESTAMP_TZ + - name: reference + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: finance__payouts + description: Payment provider payouts change log + managed: true + contract: enforced + columns: + - name: payout_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: provider_code + type: VARCHAR + - name: currency_code + type: VARCHAR + - name: amount_minor + type: NUMBER(38,0) + - name: fee_minor + type: NUMBER(38,0) + - name: paid_out_on + type: DATE + - name: lines + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: finance__tax_rates + description: Sales tax rates change log + managed: true + contract: enforced + columns: + - name: tax_rate_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: country_code + type: VARCHAR + - name: tax_region + type: VARCHAR + - name: tax_category + type: VARCHAR + - name: rate_pct + type: NUMBER(9,4) + - name: valid_from + type: DATE + - name: valid_to + type: DATE + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: finance__chargebacks + description: Card chargebacks change log + managed: true + contract: enforced + columns: + - name: chargeback_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: payment_id + type: VARCHAR + - name: amount_minor + type: NUMBER(38,0) + - name: currency_code + type: VARCHAR + - name: reason_code + type: VARCHAR + - name: opened_at + type: TIMESTAMP_TZ + - name: closed_at + type: TIMESTAMP_TZ + - name: outcome + type: VARCHAR + - name: evidence + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: finance__settlement_reports + description: Append-only provider settlement files; report_body is JSON text + managed: true + contract: enforced + freshness: + strategy: sql + type: string + query: SELECT MAX(report_date)::VARCHAR FROM DUCKSHOP_RAW.FINANCE.SETTLEMENT_REPORTS + columns: + - name: report_id + type: VARCHAR + nullable: false + audits: + - not_null + - unique_per_delivery + - name: provider_code + type: VARCHAR + - name: report_date + type: DATE + - name: report_body + type: VARCHAR + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null diff --git a/benchmarks/duckshop/project/sources/finance_reference.yml b/benchmarks/duckshop/project/sources/finance_reference.yml new file mode 100644 index 0000000000..dea03384d4 --- /dev/null +++ b/benchmarks/duckshop/project/sources/finance_reference.yml @@ -0,0 +1,44 @@ +# Reference data defined inline so the project compiles and replays without an FX feed. +sources: + - name: finance__fx_rates + description: Daily reference rates to USD per currency, defined inline as a VALUES expression + contract: enforced + expression: | + SELECT v.currency_code, v.rate_date, v.rate_to_usd + FROM ( + VALUES + ('USD', '2025-01-01', 1.00000000), + ('EUR', '2025-01-01', 1.03500000), + ('GBP', '2025-01-01', 1.25100000), + ('JPY', '2025-01-01', 0.00636000), + ('AUD', '2025-01-01', 0.61900000), + ('KWD', '2025-01-01', 3.24300000), + ('USD', '2026-01-01', 1.00000000), + ('EUR', '2026-01-01', 1.08250000), + ('EUR', '2026-03-01', 1.09100000), + ('EUR', '2026-03-30', 1.07800000), + ('GBP', '2026-01-01', 1.26400000), + ('GBP', '2026-03-01', 1.27150000), + ('JPY', '2026-01-01', 0.00682000), + ('JPY', '2026-03-01', 0.00671500), + ('AUD', '2026-01-01', 0.65800000), + ('AUD', '2026-03-01', 0.66250000), + ('KWD', '2026-01-01', 3.25400000), + ('KWD', '2026-03-01', 3.25150000) + ) AS v (currency_code, rate_date, rate_to_usd) + columns: + - name: currency_code + type: VARCHAR + nullable: false + audits: + - not_null + - name: rate_date + type: DATE + nullable: false + audits: + - not_null + - name: rate_to_usd + type: NUMBER(18,8) + nullable: false + audits: + - not_null diff --git a/benchmarks/duckshop/project/sources/fulfilment.yml b/benchmarks/duckshop/project/sources/fulfilment.yml new file mode 100644 index 0000000000..a4b55edcc1 --- /dev/null +++ b/benchmarks/duckshop/project/sources/fulfilment.yml @@ -0,0 +1,342 @@ +# Raw fulfilment tables, generated by the synthetic data loaders in python/loaders/fulfilment.py. +sources: + - name: fulfilment__shipments + description: Shipments change log + managed: true + contract: enforced + columns: + - name: shipment_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: order_id + type: VARCHAR + - name: warehouse_code + type: VARCHAR + - name: carrier_code + type: VARCHAR + - name: service_level + type: VARCHAR + - name: shipped_at + type: TIMESTAMP_TZ + - name: delivered_at + type: TIMESTAMP_TZ + - name: tracking + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: fulfilment__shipment_packages + description: Shipment packages change log + managed: true + contract: enforced + columns: + - name: package_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: shipment_id + type: VARCHAR + - name: weight_grams + type: NUMBER(38,0) + - name: dimensions + type: OBJECT + - name: contents + type: ARRAY + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: fulfilment__carrier_webhooks + description: Raw carrier webhook deliveries; body is JSON text and may be malformed + managed: true + contract: enforced + columns: + - name: webhook_id + type: VARCHAR + nullable: false + audits: + - not_null + - unique_per_delivery + - name: carrier_code + type: VARCHAR + - name: received_at + type: TIMESTAMP_NTZ + - name: body + type: VARCHAR + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: fulfilment__returns + description: Customer returns change log + managed: true + contract: enforced + columns: + - name: return_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: order_id + type: VARCHAR + - name: sku + type: VARCHAR + - name: qty + type: NUMBER(38,0) + - name: reason_text + type: VARCHAR + - name: return_status + type: VARCHAR + - name: requested_at + type: TIMESTAMP_TZ + - name: received_at + type: TIMESTAMP_TZ + - name: photos + type: ARRAY + - name: refund_preference + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: fulfilment__return_inspections + description: Returned item inspections change log + managed: true + contract: enforced + columns: + - name: inspection_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: return_id + type: VARCHAR + - name: warehouse_code + type: VARCHAR + - name: grade + type: VARCHAR + - name: notes + type: VARCHAR + - name: inspected_at_local + type: TIMESTAMP_NTZ + - name: findings + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: fulfilment__carrier_rate_cards + description: Carrier rate cards change log + managed: true + contract: enforced + columns: + - name: rate_card_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: carrier_code + type: VARCHAR + - name: service_level + type: VARCHAR + - name: origin_country + type: VARCHAR + - name: destination_country + type: VARCHAR + - name: currency_code + type: VARCHAR + - name: valid_from + type: DATE + - name: valid_to + type: DATE + - name: bands + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: fulfilment__delivery_slots + description: Delivery slot capacity change log + managed: true + contract: enforced + columns: + - name: slot_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: warehouse_code + type: VARCHAR + - name: slot_start_local + type: TIMESTAMP_NTZ + - name: slot_end_local + type: TIMESTAMP_NTZ + - name: capacity + type: NUMBER(38,0) + - name: booked + type: NUMBER(38,0) + - name: restrictions + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: fulfilment__pick_tasks + description: Warehouse pick tasks change log + managed: true + contract: enforced + columns: + - name: task_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: order_id + type: VARCHAR + - name: warehouse_code + type: VARCHAR + - name: picker_id + type: VARCHAR + - name: started_at_local + type: TIMESTAMP_NTZ + - name: completed_at_local + type: TIMESTAMP_NTZ + - name: lines + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: fulfilment__delivery_exceptions + description: Delivery exceptions change log + managed: true + contract: enforced + columns: + - name: exception_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: shipment_id + type: VARCHAR + - name: exception_code + type: VARCHAR + - name: exception_text + type: VARCHAR + - name: occurred_at + type: TIMESTAMP_TZ + - name: resolution + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null diff --git a/benchmarks/duckshop/project/sources/growth.yml b/benchmarks/duckshop/project/sources/growth.yml new file mode 100644 index 0000000000..26605d5b73 --- /dev/null +++ b/benchmarks/duckshop/project/sources/growth.yml @@ -0,0 +1,269 @@ +# Raw growth tables, generated by the synthetic data loaders in python/loaders/growth.py. +sources: + - name: growth__campaign_spend + description: Daily advertising spend exports change log + managed: true + contract: enforced + columns: + - name: spend_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: spend_date + type: DATE + - name: channel + type: VARCHAR + - name: campaign_id + type: VARCHAR + - name: currency_code + type: VARCHAR + - name: report + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: growth__campaigns + description: Marketing campaigns change log + managed: true + contract: enforced + columns: + - name: campaign_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: campaign_name + type: VARCHAR + - name: channel + type: VARCHAR + - name: objective + type: VARCHAR + - name: starts_on + type: DATE + - name: ends_on + type: DATE + - name: budget_minor + type: NUMBER(38,0) + - name: currency_code + type: VARCHAR + - name: targeting + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: growth__affiliate_conversions + description: Affiliate conversions change log + managed: true + contract: enforced + columns: + - name: conversion_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: affiliate_code + type: VARCHAR + - name: order_id + type: VARCHAR + - name: commission_minor + type: NUMBER(38,0) + - name: currency_code + type: VARCHAR + - name: converted_at + type: TIMESTAMP_TZ + - name: click_payload + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: growth__referral_invites + description: Customer referral invites change log + managed: true + contract: enforced + columns: + - name: invite_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: referrer_customer_id + type: VARCHAR + - name: invitee_email + type: VARCHAR + - name: sent_at + type: TIMESTAMP_TZ + - name: accepted_at + type: TIMESTAMP_TZ + - name: reward + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: growth__newsletter_subscriptions + description: Newsletter list membership change log + managed: true + contract: enforced + columns: + - name: list_subscription_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: email + type: VARCHAR + - name: list_code + type: VARCHAR + - name: list_status + type: VARCHAR + - name: subscribed_at + type: TIMESTAMP_TZ + - name: unsubscribed_at + type: TIMESTAMP_TZ + - name: signup_source + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: growth__survey_responses + description: Customer survey responses change log + managed: true + contract: enforced + columns: + - name: response_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: survey_code + type: VARCHAR + - name: customer_id + type: VARCHAR + - name: submitted_at + type: TIMESTAMP_TZ + - name: answers + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: growth__influencer_posts + description: Creator partnership posts change log + managed: true + contract: enforced + columns: + - name: post_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: creator_handle + type: VARCHAR + - name: platform + type: VARCHAR + - name: posted_at + type: TIMESTAMP_TZ + - name: promo_code + type: VARCHAR + - name: metrics + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null diff --git a/benchmarks/duckshop/project/sources/inventory.yml b/benchmarks/duckshop/project/sources/inventory.yml new file mode 100644 index 0000000000..4373086173 --- /dev/null +++ b/benchmarks/duckshop/project/sources/inventory.yml @@ -0,0 +1,309 @@ +# Raw inventory tables, generated by the synthetic data loaders in python/loaders/inventory.py. +sources: + - name: inventory__warehouses + description: Warehouses change log, such as The Pond and The Nest + managed: true + contract: enforced + columns: + - name: warehouse_code + type: VARCHAR + nullable: false + audits: + - not_null + - name: warehouse_name + type: VARCHAR + - name: country_code + type: VARCHAR + - name: iana_timezone + type: VARCHAR + - name: opens_local + type: TIME + - name: closes_local + type: TIME + - name: capabilities + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: inventory__stock_movements + description: Append-only stock movement log in warehouse local time + managed: true + contract: enforced + freshness: + strategy: adapter + columns: + - name: movement_id + type: VARCHAR + nullable: false + audits: + - not_null + - unique_per_delivery + - name: sku + type: VARCHAR + - name: warehouse_code + type: VARCHAR + - name: qty_delta + type: NUMBER(38,0) + - name: reason_code + type: VARCHAR + - name: reference + type: VARCHAR + - name: moved_at_local + type: TIMESTAMP_NTZ + - name: metadata + type: VARIANT + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: inventory__stock_counts + description: Cycle counts change log + managed: true + contract: enforced + columns: + - name: count_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: sku + type: VARCHAR + - name: warehouse_code + type: VARCHAR + - name: counted_qty + type: NUMBER(38,0) + - name: counted_at_local + type: TIMESTAMP_NTZ + - name: counter + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: inventory__purchase_orders + description: Purchase orders change log + managed: true + contract: enforced + columns: + - name: po_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: supplier_code + type: VARCHAR + - name: warehouse_code + type: VARCHAR + - name: currency_code + type: VARCHAR + - name: po_status + type: VARCHAR + - name: ordered_at + type: TIMESTAMP_TZ + - name: expected_at + type: TIMESTAMP_TZ + - name: received_at + type: TIMESTAMP_TZ + - name: lines + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: inventory__po_receipts + description: Purchase order receipts change log + managed: true + contract: enforced + columns: + - name: receipt_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: po_id + type: VARCHAR + - name: sku + type: VARCHAR + - name: warehouse_code + type: VARCHAR + - name: received_qty + type: NUMBER(38,0) + - name: damaged_qty + type: NUMBER(38,0) + - name: received_at_local + type: TIMESTAMP_NTZ + - name: discrepancies + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: inventory__stock_transfers + description: Inter-warehouse transfers change log + managed: true + contract: enforced + columns: + - name: transfer_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: sku + type: VARCHAR + - name: from_warehouse_code + type: VARCHAR + - name: to_warehouse_code + type: VARCHAR + - name: qty + type: NUMBER(38,0) + - name: shipped_at + type: TIMESTAMP_TZ + - name: arrived_at + type: TIMESTAMP_TZ + - name: tracking + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: inventory__stock_reservations + description: Stock reservations change log + managed: true + contract: enforced + columns: + - name: reservation_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: order_id + type: VARCHAR + - name: sku + type: VARCHAR + - name: warehouse_code + type: VARCHAR + - name: qty + type: NUMBER(38,0) + - name: reserved_at + type: TIMESTAMP_TZ + - name: released_at + type: TIMESTAMP_TZ + - name: allocation + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: inventory__bin_locations + description: Warehouse bin locations change log + managed: true + contract: enforced + columns: + - name: bin_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: warehouse_code + type: VARCHAR + - name: bin_label + type: VARCHAR + - name: attributes + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null diff --git a/benchmarks/duckshop/project/sources/partners.yml b/benchmarks/duckshop/project/sources/partners.yml new file mode 100644 index 0000000000..bec3020c0d --- /dev/null +++ b/benchmarks/duckshop/project/sources/partners.yml @@ -0,0 +1,100 @@ +# Raw partners tables, generated by the synthetic data loaders in python/loaders/partners.py. +sources: + - name: partners__partner_orders + description: Append-only partner sales channel order messages in two payload versions + managed: true + contract: enforced + columns: + - name: message_id + type: VARCHAR + nullable: false + audits: + - not_null + - unique_per_delivery + - name: partner_code + type: VARCHAR + - name: schema_version + type: NUMBER(38,0) + - name: received_at + type: TIMESTAMP_NTZ + - name: payload + type: VARIANT + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: partners__partner_listings + description: Partner sales channel listings change log + managed: true + contract: enforced + columns: + - name: listing_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: partner_code + type: VARCHAR + - name: sku + type: VARCHAR + - name: listing_status + type: VARCHAR + - name: listed_at + type: TIMESTAMP_TZ + - name: terms + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: partners__partner_settlements + description: Partner sales channel settlement statements change log + managed: true + contract: enforced + columns: + - name: settlement_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: partner_code + type: VARCHAR + - name: period_start + type: DATE + - name: period_end + type: DATE + - name: currency_code + type: VARCHAR + - name: statement + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null diff --git a/benchmarks/duckshop/project/sources/storefront.yml b/benchmarks/duckshop/project/sources/storefront.yml new file mode 100644 index 0000000000..03abf20551 --- /dev/null +++ b/benchmarks/duckshop/project/sources/storefront.yml @@ -0,0 +1,795 @@ +# Raw storefront tables, generated by the synthetic data loaders in python/loaders/storefront.py. +sources: + - name: storefront__customers + description: Customer accounts change log + managed: true + contract: enforced + columns: + - name: customer_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: email + type: VARCHAR + - name: phone + type: VARCHAR + - name: full_name + type: VARCHAR + - name: region_code + type: VARCHAR + - name: storefront_code + type: VARCHAR + - name: created_at + type: TIMESTAMP_TZ + - name: updated_at + type: TIMESTAMP_TZ + - name: profile + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__customer_addresses + description: Customer address book change log + managed: true + contract: enforced + columns: + - name: address_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: customer_id + type: VARCHAR + - name: line1 + type: VARCHAR + - name: line2 + type: VARCHAR + - name: city + type: VARCHAR + - name: postcode + type: VARCHAR + - name: country_code + type: VARCHAR + - name: is_default + type: BOOLEAN + - name: valid_from + type: TIMESTAMP_TZ + - name: geo + type: OBJECT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__customer_consents + description: Marketing and privacy consent change log + managed: true + contract: enforced + columns: + - name: consent_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: customer_id + type: VARCHAR + - name: consent_type + type: VARCHAR + - name: granted + type: BOOLEAN + - name: captured_at + type: TIMESTAMP_TZ + - name: source_context + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__loyalty_accounts + description: Loyalty programme accounts change log (Duckling, Drake, Golden Duck and Platinum Duck tiers) + managed: true + contract: enforced + columns: + - name: loyalty_account_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: customer_id + type: VARCHAR + - name: tier_code + type: VARCHAR + - name: points_balance + type: NUMBER(38,0) + - name: enrolled_at + type: TIMESTAMP_TZ + - name: tier_history + type: ARRAY + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__loyalty_ledger + description: Loyalty points postings change log + managed: true + contract: enforced + columns: + - name: entry_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: loyalty_account_id + type: VARCHAR + - name: points_delta + type: NUMBER(38,0) + - name: reason_code + type: VARCHAR + - name: order_id + type: VARCHAR + - name: posted_at + type: TIMESTAMP_TZ + - name: metadata + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__wishlists + description: Customer wishlists change log + managed: true + contract: enforced + columns: + - name: wishlist_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: customer_id + type: VARCHAR + - name: created_at + type: TIMESTAMP_TZ + - name: items + type: ARRAY + - name: is_public + type: BOOLEAN + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__gift_cards + description: Issued gift cards change log + managed: true + contract: enforced + columns: + - name: gift_card_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: code_masked + type: VARCHAR + - name: currency_code + type: VARCHAR + - name: initial_amount_minor + type: NUMBER(38,0) + - name: issued_at + type: TIMESTAMP_TZ + - name: expires_on + type: DATE + - name: purchaser_customer_id + type: VARCHAR + - name: details + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__gift_card_transactions + description: Gift card redemptions and top-ups change log + managed: true + contract: enforced + columns: + - name: transaction_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: gift_card_id + type: VARCHAR + - name: order_id + type: VARCHAR + - name: amount_minor + type: NUMBER(38,0) + - name: transaction_type + type: VARCHAR + - name: occurred_at + type: TIMESTAMP_TZ + - name: context + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__orders + description: Storefront orders change log + managed: true + contract: enforced + freshness: + column: _loaded_at + lag_tolerance: 2h + columns: + - name: order_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: customer_id + type: VARCHAR + - name: storefront_code + type: VARCHAR + - name: currency_code + type: VARCHAR + - name: placed_at + type: TIMESTAMP_TZ + - name: status + type: VARCHAR + - name: shipping_address_id + type: VARCHAR + - name: attributes + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__order_lines + description: Order lines change log + managed: true + contract: enforced + columns: + - name: order_line_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: order_id + type: VARCHAR + - name: line_no + type: NUMBER(38,0) + - name: sku + type: VARCHAR + - name: quantity + type: NUMBER(38,0) + - name: unit_amount_minor + type: NUMBER(38,0) + - name: tax_amount_minor + type: NUMBER(38,0) + - name: discounts + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__order_status_events + description: Order status transitions change log + managed: true + contract: enforced + columns: + - name: event_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: order_id + type: VARCHAR + - name: from_status + type: VARCHAR + - name: to_status + type: VARCHAR + - name: changed_at + type: TIMESTAMP_TZ + - name: actor + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__payments + description: Payment attempts change log + managed: true + contract: enforced + columns: + - name: payment_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: order_id + type: VARCHAR + - name: method + type: VARCHAR + - name: amount_minor + type: NUMBER(38,0) + - name: currency_code + type: VARCHAR + - name: authorised_at + type: TIMESTAMP_TZ + - name: captured_at + type: TIMESTAMP_TZ + - name: card_bin + type: VARCHAR + - name: gateway_response + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__refunds + description: Refunds change log + managed: true + contract: enforced + columns: + - name: refund_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: payment_id + type: VARCHAR + - name: amount_minor + type: NUMBER(38,0) + - name: reason_text + type: VARCHAR + - name: refunded_at + type: TIMESTAMP_TZ + - name: initiated_by + type: VARCHAR + - name: items + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__promotions + description: Promotion definitions change log + managed: true + contract: enforced + columns: + - name: promo_code + type: VARCHAR + nullable: false + audits: + - not_null + - name: promo_name + type: VARCHAR + - name: starts_at + type: TIMESTAMP_TZ + - name: ends_at + type: TIMESTAMP_TZ + - name: is_stackable + type: BOOLEAN + - name: rules + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__promotion_redemptions + description: Promotion redemptions change log + managed: true + contract: enforced + columns: + - name: redemption_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: promo_code + type: VARCHAR + - name: order_id + type: VARCHAR + - name: discount_minor + type: NUMBER(38,0) + - name: redeemed_at + type: TIMESTAMP_TZ + - name: context + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__carts + description: Shopping carts change log + managed: true + contract: enforced + columns: + - name: cart_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: customer_id + type: VARCHAR + - name: anonymous_id + type: VARCHAR + - name: storefront_code + type: VARCHAR + - name: cart_status + type: VARCHAR + - name: created_at + type: TIMESTAMP_TZ + - name: updated_at + type: TIMESTAMP_TZ + - name: contents + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__reviews + description: Product reviews change log + managed: true + contract: enforced + columns: + - name: review_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: sku + type: VARCHAR + - name: customer_id + type: VARCHAR + - name: rating + type: NUMBER(38,0) + - name: title + type: VARCHAR + - name: body + type: VARCHAR + - name: submitted_at + type: TIMESTAMP_TZ + - name: moderation + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__subscriptions + description: Replenishment subscriptions change log + managed: true + contract: enforced + columns: + - name: subscription_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: customer_id + type: VARCHAR + - name: plan_code + type: VARCHAR + - name: sku + type: VARCHAR + - name: interval_days + type: NUMBER(38,0) + - name: subscription_status + type: VARCHAR + - name: started_at + type: TIMESTAMP_TZ + - name: cancelled_at + type: TIMESTAMP_TZ + - name: billing + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__subscription_events + description: Subscription lifecycle events change log + managed: true + contract: enforced + columns: + - name: event_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: subscription_id + type: VARCHAR + - name: event_type + type: VARCHAR + - name: occurred_at + type: TIMESTAMP_TZ + - name: payload + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: storefront__store_credit_ledger + description: Store credit postings change log + managed: true + contract: enforced + columns: + - name: entry_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: customer_id + type: VARCHAR + - name: currency_code + type: VARCHAR + - name: amount_minor + type: NUMBER(38,0) + - name: reason_code + type: VARCHAR + - name: posted_at + type: TIMESTAMP_TZ + - name: context + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null diff --git a/benchmarks/duckshop/project/sources/support.yml b/benchmarks/duckshop/project/sources/support.yml new file mode 100644 index 0000000000..72b1c3fedc --- /dev/null +++ b/benchmarks/duckshop/project/sources/support.yml @@ -0,0 +1,312 @@ +# Raw support tables, generated by the synthetic data loaders in python/loaders/support.py. +sources: + - name: support__tickets + description: Support tickets change log; created_at is session-time-zone dependent + managed: true + contract: enforced + columns: + - name: ticket_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: customer_email + type: VARCHAR + - name: channel + type: VARCHAR + - name: priority + type: NUMBER(38,0) + - name: ticket_status + type: VARCHAR + - name: created_at + type: TIMESTAMP_LTZ + - name: solved_at + type: TIMESTAMP_TZ + - name: custom_fields + type: VARIANT + - name: tags + type: ARRAY + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: support__ticket_comments + description: Ticket comments change log + managed: true + contract: enforced + columns: + - name: comment_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: ticket_id + type: VARCHAR + - name: author_type + type: VARCHAR + - name: author_id + type: VARCHAR + - name: body + type: VARCHAR + - name: is_public + type: BOOLEAN + - name: created_at + type: TIMESTAMP_TZ + - name: attachments + type: ARRAY + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: support__csat_responses + description: Customer satisfaction responses change log + managed: true + contract: enforced + columns: + - name: response_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: ticket_id + type: VARCHAR + - name: score + type: NUMBER(38,0) + - name: comment + type: VARCHAR + - name: responded_at + type: TIMESTAMP_TZ + - name: answers + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: support__agents + description: Support agents change log + managed: true + contract: enforced + columns: + - name: agent_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: display_name + type: VARCHAR + - name: email + type: VARCHAR + - name: team_code + type: VARCHAR + - name: iana_timezone + type: VARCHAR + - name: hired_on + type: DATE + - name: skills + type: ARRAY + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: support__agent_shifts + description: Agent shifts change log in agent local time + managed: true + contract: enforced + columns: + - name: shift_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: agent_id + type: VARCHAR + - name: shift_start_local + type: TIMESTAMP_NTZ + - name: shift_end_local + type: TIMESTAMP_NTZ + - name: iana_timezone + type: VARCHAR + - name: breaks + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: support__chat_transcripts + description: Live chat transcripts change log + managed: true + contract: enforced + columns: + - name: chat_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: ticket_id + type: VARCHAR + - name: started_at + type: TIMESTAMP_TZ + - name: ended_at + type: TIMESTAMP_TZ + - name: messages + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: support__call_records + description: Phone call records change log + managed: true + contract: enforced + columns: + - name: call_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: ticket_id + type: VARCHAR + - name: agent_id + type: VARCHAR + - name: phone_number + type: VARCHAR + - name: started_at + type: TIMESTAMP_TZ + - name: duration_seconds + type: NUMBER(38,0) + - name: ivr_path + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null + - name: support__sla_policies + description: Support SLA policies change log + managed: true + contract: enforced + columns: + - name: policy_id + type: VARCHAR + nullable: false + audits: + - not_null + - name: priority + type: NUMBER(38,0) + - name: channel + type: VARCHAR + - name: first_response_minutes + type: NUMBER(38,0) + - name: resolution_hours + type: NUMBER(38,0) + - name: business_hours_only + type: BOOLEAN + - name: calendar + type: VARIANT + - name: _op + type: VARCHAR + nullable: false + audits: + - accepted_values: + values: ["I", "U", "D"] + - name: _seq + type: NUMBER(38,0) + nullable: false + audits: + - not_null + - unique + - name: _loaded_at + type: TIMESTAMP_NTZ + nullable: false + audits: + - not_null diff --git a/benchmarks/duckshop/project/sqlbuild_local.toml.example b/benchmarks/duckshop/project/sqlbuild_local.toml.example new file mode 100644 index 0000000000..16d18f6db8 --- /dev/null +++ b/benchmarks/duckshop/project/sqlbuild_local.toml.example @@ -0,0 +1,15 @@ +# Copy to sqlbuild_local.toml (gitignored) and fill in your own values. +# Never commit real account names, users, roles, warehouses or schemas. +target = "dev" + +[connections.warehouse] +account = "" +user = "" +authenticator = "" +role = "" +warehouse = "" + +[targets.dev] +connection = "warehouse" +database = "" +schema = "" diff --git a/benchmarks/duckshop/project/sqlbuild_project.toml b/benchmarks/duckshop/project/sqlbuild_project.toml new file mode 100644 index 0000000000..70b314918e --- /dev/null +++ b/benchmarks/duckshop/project/sqlbuild_project.toml @@ -0,0 +1,252 @@ +name = "duckshop" +adapter = "snowflake" +default_target = "dev" + +# Placeholders only. Real connection details belong in the untracked sqlbuild_local.toml; +# see sqlbuild_local.toml.example. Compiling needs no connection. +[connections.warehouse] +account = "example-account" +user = "example_user" +role = "EXAMPLE_ROLE" +warehouse = "EXAMPLE_WH" +# SQLBuild 0.175.0 binds inspected Snowflake source columns as quoted lower-case names, which +# unquoted references cannot match, so planning fails once the raw tables exist. Ignoring the case +# of quoted identifiers avoids that; DuckShop's SQL quotes no identifiers whose case matters. +session_parameters = { QUOTED_IDENTIFIERS_IGNORE_CASE = true } + +[targets.dev] +connection = "warehouse" +database = "DUCKSHOP" +schema = "DEV" + +[targets.prod] +connection = "warehouse" +database = "DUCKSHOP" +schema = "ANALYTICS" + +[settings] +default_audit_severity = "error" +unit_test_max_real_models = 6 +test_timeout = "60s" +scenario_timeout = "10m" + +[defaults] +materialized = "table" + +[path_defaults.staging] +materialized = "view" +tags = ["staging"] + +[path_defaults.intermediate] +materialized = "table" +tags = ["intermediate"] + +[path_defaults.marts] +materialized = "table" +tags = ["marts"] + +[path_defaults.snapshots] +tags = ["snapshots"] + +[vars] +reporting_currency = "USD" +# Fixed valuation date for balance and liability models, so their output is reproducible. +valuation_date = "2026-03-31" +# Customer shown by the order lookup report; override per run with --vars. +lookup_customer_id = "C000001" +# Volume multiplier for the synthetic data loaders in python/loaders; 1 is the small default. +data_scale = "1" + +[scenario.local_type_overrides.snowflake] +"VARIANT" = "JSON" +"OBJECT" = "JSON" +"ARRAY" = "JSON" + +[rules] +select = ["SQBRSQL", "SQBRGRAPH", "SQBRCONTRACT", "SQBRTEST", "XSQBRDUCK"] +# SQBRSQL021: `SELECT * FROM final` and the change-log macro's pass-through CTEs are the house +# style; each model's columns block, not its terminal SELECT, owns the output shape. +# SQBRCONTRACT101: exact contracts are reserved for published dim_ and fct_ models. +ignore = ["SQBRSQL021", "SQBRCONTRACT101"] + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/staging/_sqlbuild/macros/sql/cdc_latest.sql" +reason = """ +The macro cannot see its callers' keys on its own. Every call site is still checked, against the +unique, non-null _seq that each change-log source declares. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/marts/core/dim_date.sql" +reason = """ +SEQ4() is distinct for every row of one GENERATOR call, so ordering by it is unique. ROW_NUMBER() +renumbers it because SEQ4() alone may leave gaps. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/staging/clickstream/stg_app_events.sql" +reason = """ +Append-only feeds redeliver rows, so the source id repeats across loads and only (id, _loaded_at) +is unique, which the source's unique_per_delivery audit checks. Rules cannot use a two-column key +as proof. A redelivered row is identical to the original, so the window picks the same values +whichever copy ranks first. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/staging/clickstream/stg_email_events.sql" +reason = """ +Append-only feeds redeliver rows, so the source id repeats across loads and only (id, _loaded_at) +is unique, which the source's unique_per_delivery audit checks. Rules cannot use a two-column key +as proof. A redelivered row is identical to the original, so the window picks the same values +whichever copy ranks first. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/staging/clickstream/stg_experiment_assignments.sql" +reason = """ +Append-only feeds redeliver rows, so the source id repeats across loads and only (id, _loaded_at) +is unique, which the source's unique_per_delivery audit checks. Rules cannot use a two-column key +as proof. A redelivered row is identical to the original, so the window picks the same values +whichever copy ranks first. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/staging/clickstream/stg_search_queries.sql" +reason = """ +Append-only feeds redeliver rows, so the source id repeats across loads and only (id, _loaded_at) +is unique, which the source's unique_per_delivery audit checks. Rules cannot use a two-column key +as proof. A redelivered row is identical to the original, so the window picks the same values +whichever copy ranks first. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/staging/clickstream/stg_web_events.sql" +reason = """ +Append-only feeds redeliver rows, so the source id repeats across loads and only (id, _loaded_at) +is unique, which the source's unique_per_delivery audit checks. Rules cannot use a two-column key +as proof. A redelivered row is identical to the original, so the window picks the same values +whichever copy ranks first. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/staging/finance/stg_settlement_reports.sql" +reason = """ +Append-only feeds redeliver rows, so the source id repeats across loads and only (id, _loaded_at) +is unique, which the source's unique_per_delivery audit checks. Rules cannot use a two-column key +as proof. A redelivered row is identical to the original, so the window picks the same values +whichever copy ranks first. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/staging/fulfilment/stg_carrier_webhook_errors.sql" +reason = """ +Append-only feeds redeliver rows, so the source id repeats across loads and only (id, _loaded_at) +is unique, which the source's unique_per_delivery audit checks. Rules cannot use a two-column key +as proof. A redelivered row is identical to the original, so the window picks the same values +whichever copy ranks first. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/staging/fulfilment/stg_carrier_webhooks.sql" +reason = """ +Append-only feeds redeliver rows, so the source id repeats across loads and only (id, _loaded_at) +is unique, which the source's unique_per_delivery audit checks. Rules cannot use a two-column key +as proof. A redelivered row is identical to the original, so the window picks the same values +whichever copy ranks first. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/staging/inventory/stg_stock_movements.sql" +reason = """ +Append-only feeds redeliver rows, so the source id repeats across loads and only (id, _loaded_at) +is unique, which the source's unique_per_delivery audit checks. Rules cannot use a two-column key +as proof. A redelivered row is identical to the original, so the window picks the same values +whichever copy ranks first. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL018" +path = "models/staging/partners/stg_partner_orders.sql" +reason = """ +Append-only feeds redeliver rows, so the source id repeats across loads and only (id, _loaded_at) +is unique, which the source's unique_per_delivery audit checks. Rules cannot use a two-column key +as proof. A redelivered row is identical to the original, so the window picks the same values +whichever copy ranks first. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL034" +path = "models/staging/_sqlbuild/macros/sql/load_metadata.sql" +reason = """ +A column-list macro body is a select list by construction. Its columns are inserted into the +caller's own CTE, where the rule is checked again. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL034" +path = "macros/sql/money_columns.sql" +reason = """ +A column-list macro body is a select list by construction. Its columns are inserted into the +caller's own CTE, where the rule is checked again. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL028" +path = "models/intermediate/catalogue/int_category_paths.sql" +reason = """ +LEVEL and the CONNECT_BY_ROOT operator are hierarchical-query pseudo-columns that cannot be +qualified with a table alias. +""" + +[[rules.rule_exceptions]] +rule = "SQBRTEST202" +path = "models/intermediate/catalogue/int_category_paths.sql" +reason = """ +Unit-test fixture checking treats the LEVEL pseudo-column as a column the mocked categories +relation must supply, so no test of this model can compile yet. The +category_tree__rolls_sales_up_built_paths scenario builds the hierarchy from a source fixture instead. +""" + +[[rules.rule_exceptions]] +rule = "SQBRTEST202" +path = "models/intermediate/clickstream/int_cart_funnel.sql" +reason = """ +Unit-test fixture checking treats the FILTER, TRANSFORM and REDUCE lambda parameters (i, acc, q) as +columns the mocked relations must supply, so no test of this model can compile yet. The +cart_funnel__assigns_furthest_stage_per_session scenario covers the funnel instead. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL038" +path = "models/marts/executive/rpt_executive_kpis_daily.sql" +relation = "metric_regions" +reason = """ +The KPI grid is every reporting day for every region and metric pair that has ever had a value, so +days without activity report zero instead of disappearing. Both sides are small and bounded. +""" + +[[rules.rule_exceptions]] +rule = "SQBRSQL022" +path = "tests/unit/staging/storefront/test_stg_wishlist_items__uses_the_mocked_sku_normaliser.sql" +reason = """ +A macro mock must be a bare SELECT of one string literal, so its projection cannot carry an alias. +""" + +[[rules.rule_exceptions]] +rule = "SQBRTEST102" +path = "tests/unit/marts/catalogue/test_rpt_wishlist_conversion__edge_cases.sql" +reason = """ +This file keeps two related edge cases for one report together as separate TEST blocks. Each block +has its own name, so the file cannot be named after both. +""" diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/fct_orders.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/fct_orders.jsonl new file mode 100644 index 0000000000..12be141209 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/fct_orders.jsonl @@ -0,0 +1,4 @@ +{"AMOUNT":null,"AMOUNT_USD":"300.00","BASKET_SIZE":null,"CAPTURED_MINOR":null,"CURRENCY_CODE":null,"CUSTOMER_ID":"C1","DISCOUNT_AMOUNT_USD":null,"FIRST_DELIVERED_AT":null,"FIRST_PAID_AT":null,"GROSS_AMOUNT_USD":null,"HOURS_PAID_TO_SHIPPED":null,"IS_FIRST_ORDER":null,"IS_FULLY_REFUNDED":null,"NET_COLLECTED_MINOR":null,"ORDER_ID":"O1","ORDER_NUMBER":null,"ORDER_STATUS":"DELIVERED","PLACED_AT":null,"PLACED_DATE_LOCAL":"2026-03-10","PROMO_CODE":null,"RATE_DATE":null,"RATE_TO_USD":null,"REFUNDED_MINOR":null,"REGION_CODE":"EU","SALES_CHANNEL":null,"TAX_AMOUNT_USD":null,"UNITS":null,"UTM_CAMPAIGN":"cmp1","UTM_MEDIUM":null,"UTM_SOURCE":null} +{"AMOUNT":null,"AMOUNT_USD":"10.00","BASKET_SIZE":null,"CAPTURED_MINOR":null,"CURRENCY_CODE":null,"CUSTOMER_ID":"C1","DISCOUNT_AMOUNT_USD":null,"FIRST_DELIVERED_AT":null,"FIRST_PAID_AT":null,"GROSS_AMOUNT_USD":null,"HOURS_PAID_TO_SHIPPED":null,"IS_FIRST_ORDER":null,"IS_FULLY_REFUNDED":null,"NET_COLLECTED_MINOR":null,"ORDER_ID":"O2","ORDER_NUMBER":null,"ORDER_STATUS":"DELIVERED","PLACED_AT":null,"PLACED_DATE_LOCAL":"2026-04-02","PROMO_CODE":null,"RATE_DATE":null,"RATE_TO_USD":null,"REFUNDED_MINOR":null,"REGION_CODE":"EU","SALES_CHANNEL":null,"TAX_AMOUNT_USD":null,"UNITS":null,"UTM_CAMPAIGN":"cmp1","UTM_MEDIUM":null,"UTM_SOURCE":null} +{"AMOUNT":null,"AMOUNT_USD":"99.00","BASKET_SIZE":null,"CAPTURED_MINOR":null,"CURRENCY_CODE":null,"CUSTOMER_ID":"C1","DISCOUNT_AMOUNT_USD":null,"FIRST_DELIVERED_AT":null,"FIRST_PAID_AT":null,"GROSS_AMOUNT_USD":null,"HOURS_PAID_TO_SHIPPED":null,"IS_FIRST_ORDER":null,"IS_FULLY_REFUNDED":null,"NET_COLLECTED_MINOR":null,"ORDER_ID":"O3","ORDER_NUMBER":null,"ORDER_STATUS":"DELIVERED","PLACED_AT":null,"PLACED_DATE_LOCAL":"2026-03-11","PROMO_CODE":null,"RATE_DATE":null,"RATE_TO_USD":null,"REFUNDED_MINOR":null,"REGION_CODE":"EU","SALES_CHANNEL":null,"TAX_AMOUNT_USD":null,"UNITS":null,"UTM_CAMPAIGN":"CMP1","UTM_MEDIUM":null,"UTM_SOURCE":null} +{"AMOUNT":null,"AMOUNT_USD":"40.00","BASKET_SIZE":null,"CAPTURED_MINOR":null,"CURRENCY_CODE":null,"CUSTOMER_ID":"C1","DISCOUNT_AMOUNT_USD":null,"FIRST_DELIVERED_AT":null,"FIRST_PAID_AT":null,"GROSS_AMOUNT_USD":null,"HOURS_PAID_TO_SHIPPED":null,"IS_FIRST_ORDER":null,"IS_FULLY_REFUNDED":null,"NET_COLLECTED_MINOR":null,"ORDER_ID":"O4","ORDER_NUMBER":null,"ORDER_STATUS":"DELIVERED","PLACED_AT":null,"PLACED_DATE_LOCAL":"2026-03-12","PROMO_CODE":null,"RATE_DATE":null,"RATE_TO_USD":null,"REFUNDED_MINOR":null,"REGION_CODE":"EU","SALES_CHANNEL":null,"TAX_AMOUNT_USD":null,"UNITS":null,"UTM_CAMPAIGN":"cmp2","UTM_MEDIUM":null,"UTM_SOURCE":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/int_campaign_performance.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/int_campaign_performance.jsonl new file mode 100644 index 0000000000..6d82275d36 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/int_campaign_performance.jsonl @@ -0,0 +1 @@ +{"ATTRIBUTED_ORDERS":null,"ATTRIBUTED_REVENUE_USD":null,"ATTRIBUTED_SESSIONS":null,"BUDGET_USD":"500.00","BUDGET_USED_PCT":null,"CAMPAIGN_ID":"CMP1","CAMPAIGN_NAME":null,"CHANNEL":null,"EMAIL_CLICKS":null,"ENDS_ON":null,"OBJECTIVE":null,"PAID_CLICKS":null,"RETURN_ON_AD_SPEND":null,"SPEND_RANK_IN_CHANNEL":null,"SPEND_USD":"170.00","STARTS_ON":null,"TARGET_REGION_COUNT":null,"UNSUPPORTED_TARGET_REGIONS":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/int_campaign_spend_usd.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/int_campaign_spend_usd.jsonl new file mode 100644 index 0000000000..5a36e5e2aa --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/int_campaign_spend_usd.jsonl @@ -0,0 +1,4 @@ +{"CAMPAIGN_ID":"CMP1","CAMPAIGN_NAME":null,"CHANNEL":"social","CLICKS":50,"CLICK_THROUGH_RATE":null,"CONVERSIONS":null,"COST_PER_CLICK_USD":null,"CURRENCY_CODE":null,"IMPRESSIONS":null,"IS_IN_FLIGHT":null,"METRIC_COUNT":null,"PLATFORM_ACCOUNT_ID":null,"SPEND_AMOUNT":null,"SPEND_DATE":"2026-03-01","SPEND_ID":"SP1","SPEND_USD":"100.00","UNKNOWN_METRICS":"[]"} +{"CAMPAIGN_ID":"CMP1","CAMPAIGN_NAME":null,"CHANNEL":"social","CLICKS":10,"CLICK_THROUGH_RATE":null,"CONVERSIONS":null,"COST_PER_CLICK_USD":null,"CURRENCY_CODE":null,"IMPRESSIONS":null,"IS_IN_FLIGHT":null,"METRIC_COUNT":null,"PLATFORM_ACCOUNT_ID":null,"SPEND_AMOUNT":null,"SPEND_DATE":"2026-03-20","SPEND_ID":"SP2","SPEND_USD":"50.00","UNKNOWN_METRICS":"[]"} +{"CAMPAIGN_ID":"CMP2","CAMPAIGN_NAME":null,"CHANNEL":"email","CLICKS":5,"CLICK_THROUGH_RATE":null,"CONVERSIONS":null,"COST_PER_CLICK_USD":null,"CURRENCY_CODE":null,"IMPRESSIONS":null,"IS_IN_FLIGHT":null,"METRIC_COUNT":null,"PLATFORM_ACCOUNT_ID":null,"SPEND_AMOUNT":null,"SPEND_DATE":"2026-03-05","SPEND_ID":"SP3","SPEND_USD":"0.00","UNKNOWN_METRICS":"[]"} +{"CAMPAIGN_ID":"CMP1","CAMPAIGN_NAME":null,"CHANNEL":"social","CLICKS":1,"CLICK_THROUGH_RATE":null,"CONVERSIONS":null,"COST_PER_CLICK_USD":null,"CURRENCY_CODE":null,"IMPRESSIONS":null,"IS_IN_FLIGHT":null,"METRIC_COUNT":null,"PLATFORM_ACCOUNT_ID":null,"SPEND_AMOUNT":null,"SPEND_DATE":"2026-04-01","SPEND_ID":"SP4","SPEND_USD":"20.00","UNKNOWN_METRICS":"[]"} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/stg_campaigns.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/stg_campaigns.jsonl new file mode 100644 index 0000000000..0764a6248a --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/refs/stg_campaigns.jsonl @@ -0,0 +1,2 @@ +{"BUDGET_MINOR":null,"CAMPAIGN_ID":"CMP1","CAMPAIGN_NAME":"Spring Splash","CHANNEL":"social","CURRENCY_CODE":null,"ENDS_ON":null,"OBJECTIVE":null,"SOURCE_LOADED_AT":null,"SOURCE_SEQ":null,"STARTS_ON":"2026-03-01","TARGET_AGE_MAX":null,"TARGET_AGE_MIN":null,"TARGET_REGIONS":null} +{"BUDGET_MINOR":null,"CAMPAIGN_ID":"CMP2","CAMPAIGN_NAME":"Pond Week","CHANNEL":"email","CURRENCY_CODE":null,"ENDS_ON":"2026-04-30","OBJECTIVE":null,"SOURCE_LOADED_AT":null,"SOURCE_SEQ":null,"STARTS_ON":"2026-03-01","TARGET_AGE_MAX":null,"TARGET_AGE_MIN":null,"TARGET_REGIONS":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/scenario.json b/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/scenario.json new file mode 100644 index 0000000000..3fc0d5bcc0 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/campaign_roas__attributes_orders_to_channel_spend/scenario.json @@ -0,0 +1,440 @@ +{ + "capture_adapter": "snowflake", + "capture_dialect": "snowflake", + "captured_at": "2026-10-24T13:09:48Z", + "format": "jsonl", + "input_fingerprint": "254ac9c703e031b78a662fe89d70629adb43efa5a83e1fdfa6af8376c2a2afee", + "relations": [ + { + "bytes": 2565, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "order_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "customer_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "region_code", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "placed_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DATE", + "name": "placed_date_local", + "warehouse_type": "DATE" + }, + { + "local_type": "TEXT(9)", + "name": "order_status", + "warehouse_type": "VARCHAR(9)" + }, + { + "local_type": "TEXT(16777216)", + "name": "sales_channel", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "units", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "currency_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "amount", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "amount_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "rate_to_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "rate_date", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "gross_amount_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "discount_amount_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "tax_amount_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "promo_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "utm_source", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "utm_medium", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(4)", + "name": "utm_campaign", + "warehouse_type": "VARCHAR(4)" + }, + { + "local_type": "TEXT(16777216)", + "name": "basket_size", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "order_number", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "is_first_order", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "first_paid_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "first_delivered_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "hours_paid_to_shipped", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "captured_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "refunded_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "net_collected_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "is_fully_refunded", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/fct_orders.jsonl", + "kind": "ref", + "logical_name": "fct_orders", + "row_count": 4 + }, + { + "bytes": 411, + "columns": [ + { + "local_type": "TEXT(4)", + "name": "campaign_id", + "warehouse_type": "VARCHAR(4)" + }, + { + "local_type": "TEXT(16777216)", + "name": "campaign_name", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "channel", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "objective", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DATE", + "name": "starts_on", + "warehouse_type": "DATE" + }, + { + "local_type": "DATE", + "name": "ends_on", + "warehouse_type": "DATE" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "budget_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "spend_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "DECIMAL(9, 2)", + "name": "budget_used_pct", + "warehouse_type": "NUMBER(9,2)" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "paid_clicks", + "warehouse_type": "NUMBER(38,0)" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "email_clicks", + "warehouse_type": "NUMBER(38,0)" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "attributed_sessions", + "warehouse_type": "NUMBER(38,0)" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "attributed_orders", + "warehouse_type": "NUMBER(38,0)" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "attributed_revenue_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "DECIMAL(12, 4)", + "name": "return_on_ad_spend", + "warehouse_type": "NUMBER(12,4)" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "target_region_count", + "warehouse_type": "NUMBER(38,0)" + }, + { + "local_type": "JSON", + "name": "unsupported_target_regions", + "warehouse_type": "ARRAY" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "spend_rank_in_channel", + "warehouse_type": "NUMBER(38,0)" + } + ], + "file": "refs/int_campaign_performance.jsonl", + "kind": "ref", + "logical_name": "int_campaign_performance", + "row_count": 1 + }, + { + "bytes": 1433, + "columns": [ + { + "local_type": "TEXT(3)", + "name": "spend_id", + "warehouse_type": "VARCHAR(3)" + }, + { + "local_type": "TEXT(4)", + "name": "campaign_id", + "warehouse_type": "VARCHAR(4)" + }, + { + "local_type": "TEXT(16777216)", + "name": "campaign_name", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(6)", + "name": "channel", + "warehouse_type": "VARCHAR(6)" + }, + { + "local_type": "TEXT(16777216)", + "name": "platform_account_id", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DATE", + "name": "spend_date", + "warehouse_type": "DATE" + }, + { + "local_type": "TEXT(16777216)", + "name": "currency_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "spend_amount", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "spend_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "impressions", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(2, 0)", + "name": "clicks", + "warehouse_type": "NUMBER(2,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "conversions", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "click_through_rate", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "cost_per_click_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "metric_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "JSON", + "name": "unknown_metrics", + "warehouse_type": "ARRAY" + }, + { + "local_type": "TEXT(16777216)", + "name": "is_in_flight", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/int_campaign_spend_usd.jsonl", + "kind": "ref", + "logical_name": "int_campaign_spend_usd", + "row_count": 4 + }, + { + "bytes": 563, + "columns": [ + { + "local_type": "TEXT(4)", + "name": "campaign_id", + "warehouse_type": "VARCHAR(4)" + }, + { + "local_type": "TEXT(13)", + "name": "campaign_name", + "warehouse_type": "VARCHAR(13)" + }, + { + "local_type": "TEXT(6)", + "name": "channel", + "warehouse_type": "VARCHAR(6)" + }, + { + "local_type": "TEXT(16777216)", + "name": "objective", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DATE", + "name": "starts_on", + "warehouse_type": "DATE" + }, + { + "local_type": "DATE", + "name": "ends_on", + "warehouse_type": "DATE" + }, + { + "local_type": "TEXT(16777216)", + "name": "budget_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "currency_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "target_regions", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "target_age_min", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "target_age_max", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "source_seq", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "source_loaded_at", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/stg_campaigns.jsonl", + "kind": "ref", + "logical_name": "stg_campaigns", + "row_count": 2 + } + ], + "scenario_name": "campaign_roas__attributes_orders_to_channel_spend", + "sqlbuild_version": "0.175.0", + "total_bytes": 4972, + "total_rows": 11, + "version": 1 +} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/int_delivery_exceptions_classified.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/int_delivery_exceptions_classified.jsonl new file mode 100644 index 0000000000..5eb3b1ce6a --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/int_delivery_exceptions_classified.jsonl @@ -0,0 +1 @@ +{"CARRIER_CODE":null,"COMPENSATION_USD":"5.50","EXCEPTION_CODE":null,"EXCEPTION_ID":"X1","EXCEPTION_RANK_IN_SHIPMENT":null,"HOURS_TO_RESOLVE":null,"OCCURRED_AT":null,"RESOLUTION_TYPE":null,"RESPONSIBILITY":null,"SHIPMENT_ID":"S2"} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/int_fulfilment_sla.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/int_fulfilment_sla.jsonl new file mode 100644 index 0000000000..5a95c9d6de --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/int_fulfilment_sla.jsonl @@ -0,0 +1,4 @@ +{"BUSINESS_DAYS_TO_SHIP":null,"CALENDAR_DAYS_TO_SHIP":null,"CARRIER_CODE":null,"DELIVERED_AT_UTC":null,"DISPATCH_BUSINESS_MINUTES":null,"HOLIDAYS_IN_WINDOW":null,"HOURS_TO_DELIVER":"24.00","IS_DISPATCH_ON_TIME":true,"ORDER_ID":null,"PLACED_AT_LOCAL":null,"SHIPMENT_ID":"S1","SHIPPED_AT_LOCAL":null,"WAREHOUSE_CODE":null} +{"BUSINESS_DAYS_TO_SHIP":null,"CALENDAR_DAYS_TO_SHIP":null,"CARRIER_CODE":null,"DELIVERED_AT_UTC":null,"DISPATCH_BUSINESS_MINUTES":null,"HOLIDAYS_IN_WINDOW":null,"HOURS_TO_DELIVER":"48.00","IS_DISPATCH_ON_TIME":false,"ORDER_ID":null,"PLACED_AT_LOCAL":null,"SHIPMENT_ID":"S2","SHIPPED_AT_LOCAL":null,"WAREHOUSE_CODE":null} +{"BUSINESS_DAYS_TO_SHIP":null,"CALENDAR_DAYS_TO_SHIP":null,"CARRIER_CODE":null,"DELIVERED_AT_UTC":null,"DISPATCH_BUSINESS_MINUTES":null,"HOLIDAYS_IN_WINDOW":null,"HOURS_TO_DELIVER":null,"IS_DISPATCH_ON_TIME":true,"ORDER_ID":null,"PLACED_AT_LOCAL":null,"SHIPMENT_ID":"S3","SHIPPED_AT_LOCAL":null,"WAREHOUSE_CODE":null} +{"BUSINESS_DAYS_TO_SHIP":null,"CALENDAR_DAYS_TO_SHIP":null,"CARRIER_CODE":null,"DELIVERED_AT_UTC":null,"DISPATCH_BUSINESS_MINUTES":null,"HOLIDAYS_IN_WINDOW":null,"HOURS_TO_DELIVER":"10.00","IS_DISPATCH_ON_TIME":true,"ORDER_ID":null,"PLACED_AT_LOCAL":null,"SHIPMENT_ID":"S5","SHIPPED_AT_LOCAL":null,"WAREHOUSE_CODE":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/int_shipment_costs.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/int_shipment_costs.jsonl new file mode 100644 index 0000000000..fec7739a42 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/int_shipment_costs.jsonl @@ -0,0 +1 @@ +{"BAND_MAX_KG":null,"CARRIER_CODE":null,"CHARGEABLE_WEIGHT_KG":"1.500","HAS_RATE":null,"PACKAGE_ID":"PK1","RATE_CURRENCY_CODE":null,"RATE_MINOR":600,"SERVICE_LEVEL":null,"SHIPMENT_ID":"S1","WEIGHT_BASIS":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/stg_shipments.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/stg_shipments.jsonl new file mode 100644 index 0000000000..aebcd9780c --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/refs/stg_shipments.jsonl @@ -0,0 +1,5 @@ +{"CARRIER_CODE":"FLAPWING","DELIVERED_AT":null,"ORDER_ID":"O-S1","SERVICE_LEVEL":"STANDARD","SHIPMENT_ID":"S1","SHIPPED_AT":"2026-03-10T10:00:00+00:00","SOURCE_LOADED_AT":null,"SOURCE_SEQ":null,"TRACKING_EVENTS":null,"TRACKING_EVENT_COUNT":1,"TRACKING_NUMBER":null,"WAREHOUSE_CODE":"POND"} +{"CARRIER_CODE":"FLAPWING","DELIVERED_AT":null,"ORDER_ID":"O-S2","SERVICE_LEVEL":"STANDARD","SHIPMENT_ID":"S2","SHIPPED_AT":"2026-03-11T10:00:00+00:00","SOURCE_LOADED_AT":null,"SOURCE_SEQ":null,"TRACKING_EVENTS":null,"TRACKING_EVENT_COUNT":1,"TRACKING_NUMBER":null,"WAREHOUSE_CODE":"POND"} +{"CARRIER_CODE":"FLAPWING","DELIVERED_AT":null,"ORDER_ID":"O-S3","SERVICE_LEVEL":"STANDARD","SHIPMENT_ID":"S3","SHIPPED_AT":"2026-03-15T23:30:00-02:00","SOURCE_LOADED_AT":null,"SOURCE_SEQ":null,"TRACKING_EVENTS":null,"TRACKING_EVENT_COUNT":1,"TRACKING_NUMBER":null,"WAREHOUSE_CODE":"POND"} +{"CARRIER_CODE":"FLAPWING","DELIVERED_AT":null,"ORDER_ID":"O-S4","SERVICE_LEVEL":"STANDARD","SHIPMENT_ID":"S4","SHIPPED_AT":null,"SOURCE_LOADED_AT":null,"SOURCE_SEQ":null,"TRACKING_EVENTS":null,"TRACKING_EVENT_COUNT":1,"TRACKING_NUMBER":null,"WAREHOUSE_CODE":"POND"} +{"CARRIER_CODE":"WINGBEAT","DELIVERED_AT":null,"ORDER_ID":"O-S5","SERVICE_LEVEL":"STANDARD","SHIPMENT_ID":"S5","SHIPPED_AT":"2026-03-12T08:00:00+00:00","SOURCE_LOADED_AT":null,"SOURCE_SEQ":null,"TRACKING_EVENTS":null,"TRACKING_EVENT_COUNT":1,"TRACKING_NUMBER":null,"WAREHOUSE_CODE":"POND"} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/scenario.json b/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/scenario.json new file mode 100644 index 0000000000..0e020bffba --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/carrier_performance__weekly_rates_from_shipment_facts/scenario.json @@ -0,0 +1,275 @@ +{ + "capture_adapter": "snowflake", + "capture_dialect": "snowflake", + "captured_at": "2026-10-24T13:09:48Z", + "format": "jsonl", + "input_fingerprint": "3d8dfd7f6ce287ebb4869b275702bd79783ff074ed1da1126c1356438ed964d5", + "relations": [ + { + "bytes": 231, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "exception_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "shipment_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "carrier_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "exception_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "responsibility", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TIMESTAMPTZ", + "name": "occurred_at", + "warehouse_type": "TIMESTAMP_TZ(9)" + }, + { + "local_type": "DECIMAL(10, 2)", + "name": "hours_to_resolve", + "warehouse_type": "NUMBER(10,2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "resolution_type", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "compensation_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "exception_rank_in_shipment", + "warehouse_type": "NUMBER(38,0)" + } + ], + "file": "refs/int_delivery_exceptions_classified.jsonl", + "kind": "ref", + "logical_name": "int_delivery_exceptions_classified", + "row_count": 1 + }, + { + "bytes": 1282, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "shipment_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "order_id", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "warehouse_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "carrier_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "placed_at_local", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "shipped_at_local", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "calendar_days_to_ship", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "business_days_to_ship", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "holidays_in_window", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "dispatch_business_minutes", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "is_dispatch_on_time", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "TEXT(16777216)", + "name": "delivered_at_utc", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(10, 2)", + "name": "hours_to_deliver", + "warehouse_type": "NUMBER(10,2)" + } + ], + "file": "refs/int_fulfilment_sla.jsonl", + "kind": "ref", + "logical_name": "int_fulfilment_sla", + "row_count": 4 + }, + { + "bytes": 210, + "columns": [ + { + "local_type": "TEXT(3)", + "name": "package_id", + "warehouse_type": "VARCHAR(3)" + }, + { + "local_type": "TEXT(2)", + "name": "shipment_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "carrier_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "service_level", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(10, 3)", + "name": "chargeable_weight_kg", + "warehouse_type": "NUMBER(10,3)" + }, + { + "local_type": "TEXT(16777216)", + "name": "weight_basis", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(10, 3)", + "name": "band_max_kg", + "warehouse_type": "NUMBER(10,3)" + }, + { + "local_type": "DECIMAL(3, 0)", + "name": "rate_minor", + "warehouse_type": "NUMBER(3,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "rate_currency_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "has_rate", + "warehouse_type": "BOOLEAN" + } + ], + "file": "refs/int_shipment_costs.jsonl", + "kind": "ref", + "logical_name": "int_shipment_costs", + "row_count": 1 + }, + { + "bytes": 1427, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "shipment_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(4)", + "name": "order_id", + "warehouse_type": "VARCHAR(4)" + }, + { + "local_type": "TEXT(4)", + "name": "warehouse_code", + "warehouse_type": "VARCHAR(4)" + }, + { + "local_type": "TEXT(8)", + "name": "carrier_code", + "warehouse_type": "VARCHAR(8)" + }, + { + "local_type": "TEXT(8)", + "name": "service_level", + "warehouse_type": "VARCHAR(8)" + }, + { + "local_type": "TEXT(16777216)", + "name": "tracking_number", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TIMESTAMPTZ", + "name": "shipped_at", + "warehouse_type": "TIMESTAMP_TZ(9)" + }, + { + "local_type": "TEXT(16777216)", + "name": "delivered_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(1, 0)", + "name": "tracking_event_count", + "warehouse_type": "NUMBER(1,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "tracking_events", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "source_seq", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "source_loaded_at", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/stg_shipments.jsonl", + "kind": "ref", + "logical_name": "stg_shipments", + "row_count": 5 + } + ], + "scenario_name": "carrier_performance__weekly_rates_from_shipment_facts", + "sqlbuild_version": "0.175.0", + "total_bytes": 3150, + "total_rows": 11, + "version": 1 +} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/refs/int_web_sessions.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/refs/int_web_sessions.jsonl new file mode 100644 index 0000000000..bc4a814333 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/refs/int_web_sessions.jsonl @@ -0,0 +1,3 @@ +{"ACC":null,"ANONYMOUS_ID":"V1","CUSTOMER_ID":null,"DURATION_SECONDS":null,"EVENT_COUNT":null,"EXIT_PAGE_PATH":null,"HAS_ADD_TO_CART":true,"HAS_ORDER":false,"I":null,"IS_BOUNCE":false,"LANDING_PAGE_PATH":null,"LANDING_PAGE_URL":null,"PAGE_VIEW_COUNT":null,"Q":null,"REFERRER_HOST":null,"SESSION_DAY":null,"SESSION_END_UTC":"2026-03-10T10:30:00","SESSION_ID":"S1","SESSION_START_UTC":"2026-03-10T10:00:00","SESSION_START_VISITOR_LOCAL":null} +{"ACC":null,"ANONYMOUS_ID":"V1","CUSTOMER_ID":null,"DURATION_SECONDS":null,"EVENT_COUNT":null,"EXIT_PAGE_PATH":null,"HAS_ADD_TO_CART":false,"HAS_ORDER":true,"I":null,"IS_BOUNCE":false,"LANDING_PAGE_PATH":null,"LANDING_PAGE_URL":null,"PAGE_VIEW_COUNT":null,"Q":null,"REFERRER_HOST":null,"SESSION_DAY":null,"SESSION_END_UTC":"2026-03-10T12:20:00","SESSION_ID":"S2","SESSION_START_UTC":"2026-03-10T12:00:00","SESSION_START_VISITOR_LOCAL":null} +{"ACC":null,"ANONYMOUS_ID":"V2","CUSTOMER_ID":null,"DURATION_SECONDS":null,"EVENT_COUNT":null,"EXIT_PAGE_PATH":null,"HAS_ADD_TO_CART":false,"HAS_ORDER":false,"I":null,"IS_BOUNCE":false,"LANDING_PAGE_PATH":null,"LANDING_PAGE_URL":null,"PAGE_VIEW_COUNT":null,"Q":null,"REFERRER_HOST":null,"SESSION_DAY":null,"SESSION_END_UTC":"2026-03-10T10:05:00","SESSION_ID":"S3","SESSION_START_UTC":"2026-03-10T10:00:00","SESSION_START_VISITOR_LOCAL":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/refs/stg_web_checkout_steps.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/refs/stg_web_checkout_steps.jsonl new file mode 100644 index 0000000000..180f3c6aa9 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/refs/stg_web_checkout_steps.jsonl @@ -0,0 +1,3 @@ +{"ANONYMOUS_ID":"V1","CHECKOUT_STEP_KEY":"V1:1","FIRST_REACHED_AT_UTC":"2026-03-10T10:25:00","PAYMENT_OPTION":null,"STEP_NAME":null,"STEP_NUMBER":1} +{"ANONYMOUS_ID":"V1","CHECKOUT_STEP_KEY":"V1:2","FIRST_REACHED_AT_UTC":"2026-03-10T10:26:00","PAYMENT_OPTION":null,"STEP_NAME":null,"STEP_NUMBER":2} +{"ANONYMOUS_ID":"V1","CHECKOUT_STEP_KEY":"V1:3","FIRST_REACHED_AT_UTC":"2026-03-10T12:10:00","PAYMENT_OPTION":null,"STEP_NAME":null,"STEP_NUMBER":3} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/refs/stg_web_events.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/refs/stg_web_events.jsonl new file mode 100644 index 0000000000..e30896c281 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/refs/stg_web_events.jsonl @@ -0,0 +1,4 @@ +{"ANONYMOUS_ID":"V1","CLIENT_TIMEZONE":"Europe/Berlin","CUSTOMER_ID":null,"EVENT_AT_UTC":"2026-03-10T10:05:00","EVENT_ID":"e1","EVENT_TYPE":"add_to_cart","I":null,"IS_BOT":false,"ITEMS":"[\n {\n \"price_minor\": 500,\n \"quantity\": 2,\n \"sku\": \"duck-1\"\n },\n {\n \"price_minor\": 900,\n \"sku\": \"DUCK-2\"\n },\n {\n \"price_minor\": 100,\n \"quantity\": 1\n },\n {\n \"quantity\": 0,\n \"sku\": \"duck-3\"\n }\n]","LOCALE":null,"PAGE_PATH":null,"PAGE_URL":null,"PROPERTIES":null,"RECEIVED_AT":"2026-03-10T10:05:00","REFERRER_HOST":null,"USER_AGENT":null,"USER_EMAIL":null} +{"ANONYMOUS_ID":"V1","CLIENT_TIMEZONE":"Europe/Berlin","CUSTOMER_ID":null,"EVENT_AT_UTC":"2026-03-10T10:10:00","EVENT_ID":"e2","EVENT_TYPE":"remove_from_cart","I":null,"IS_BOT":false,"ITEMS":"[\n {\n \"quantity\": 1,\n \"sku\": \"duck-1\"\n }\n]","LOCALE":null,"PAGE_PATH":null,"PAGE_URL":null,"PROPERTIES":null,"RECEIVED_AT":"2026-03-10T10:10:00","REFERRER_HOST":null,"USER_AGENT":null,"USER_EMAIL":null} +{"ANONYMOUS_ID":"V1","CLIENT_TIMEZONE":"Europe/Berlin","CUSTOMER_ID":null,"EVENT_AT_UTC":"2026-03-10T10:20:00","EVENT_ID":"e3","EVENT_TYPE":"add_to_cart","I":null,"IS_BOT":false,"ITEMS":"[\n {\n \"price_minor\": 500,\n \"quantity\": 1,\n \"sku\": \"DUCK-1\"\n }\n]","LOCALE":null,"PAGE_PATH":null,"PAGE_URL":null,"PROPERTIES":null,"RECEIVED_AT":"2026-03-10T10:20:00","REFERRER_HOST":null,"USER_AGENT":null,"USER_EMAIL":null} +{"ANONYMOUS_ID":"V1","CLIENT_TIMEZONE":"Europe/Berlin","CUSTOMER_ID":null,"EVENT_AT_UTC":"2026-03-10T11:00:00","EVENT_ID":"e4","EVENT_TYPE":"add_to_cart","I":null,"IS_BOT":false,"ITEMS":"[\n {\n \"price_minor\": 9900,\n \"sku\": \"DUCK-9\"\n }\n]","LOCALE":null,"PAGE_PATH":null,"PAGE_URL":null,"PROPERTIES":null,"RECEIVED_AT":"2026-03-10T11:00:00","REFERRER_HOST":null,"USER_AGENT":null,"USER_EMAIL":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/scenario.json b/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/scenario.json new file mode 100644 index 0000000000..3df6dd364e --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/cart_funnel__assigns_furthest_stage_per_session/scenario.json @@ -0,0 +1,256 @@ +{ + "capture_adapter": "snowflake", + "capture_dialect": "snowflake", + "captured_at": "2026-10-24T13:09:48Z", + "format": "jsonl", + "input_fingerprint": "aa91e305117c08fb3ab2ad5ba5bdabbf39510e852bfd6995b57a6b7223b2146a", + "relations": [ + { + "bytes": 1324, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "session_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "anonymous_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "customer_id", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "session_day", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TIMESTAMP(9)", + "name": "session_start_utc", + "warehouse_type": "TIMESTAMP_NTZ(9)" + }, + { + "local_type": "TIMESTAMP(9)", + "name": "session_end_utc", + "warehouse_type": "TIMESTAMP_NTZ(9)" + }, + { + "local_type": "TEXT(16777216)", + "name": "session_start_visitor_local", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "duration_seconds", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "event_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "page_view_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "landing_page_path", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "exit_page_path", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "referrer_host", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "landing_page_url", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "is_bounce", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "BOOLEAN", + "name": "has_add_to_cart", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "BOOLEAN", + "name": "has_order", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "TEXT(16777216)", + "name": "acc", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "i", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "q", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/int_web_sessions.jsonl", + "kind": "ref", + "logical_name": "int_web_sessions", + "row_count": 3 + }, + { + "bytes": 447, + "columns": [ + { + "local_type": "TEXT(4)", + "name": "checkout_step_key", + "warehouse_type": "VARCHAR(4)" + }, + { + "local_type": "TEXT(2)", + "name": "anonymous_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "DECIMAL(1, 0)", + "name": "step_number", + "warehouse_type": "NUMBER(1,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "step_name", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TIMESTAMP(9)", + "name": "first_reached_at_utc", + "warehouse_type": "TIMESTAMP_NTZ(9)" + }, + { + "local_type": "TEXT(16777216)", + "name": "payment_option", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/stg_web_checkout_steps.jsonl", + "kind": "ref", + "logical_name": "stg_web_checkout_steps", + "row_count": 3 + }, + { + "bytes": 1881, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "event_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TIMESTAMP(9)", + "name": "received_at", + "warehouse_type": "TIMESTAMP_NTZ(9)" + }, + { + "local_type": "TEXT(16)", + "name": "event_type", + "warehouse_type": "VARCHAR(16)" + }, + { + "local_type": "TIMESTAMP(9)", + "name": "event_at_utc", + "warehouse_type": "TIMESTAMP_NTZ(9)" + }, + { + "local_type": "TEXT(2)", + "name": "anonymous_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "customer_id", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "user_email", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "page_url", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "page_path", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "referrer_host", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "user_agent", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "locale", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(13)", + "name": "client_timezone", + "warehouse_type": "VARCHAR(13)" + }, + { + "local_type": "BOOLEAN", + "name": "is_bot", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "TEXT(16777216)", + "name": "properties", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "JSON", + "name": "items", + "warehouse_type": "ARRAY" + }, + { + "local_type": "TEXT(16777216)", + "name": "i", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/stg_web_events.jsonl", + "kind": "ref", + "logical_name": "stg_web_events", + "row_count": 4 + } + ], + "scenario_name": "cart_funnel__assigns_furthest_stage_per_session", + "sqlbuild_version": "0.175.0", + "total_bytes": 3652, + "total_rows": 10, + "version": 1 +} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/refs/dim_products.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/refs/dim_products.jsonl new file mode 100644 index 0000000000..09b109c61d --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/refs/dim_products.jsonl @@ -0,0 +1,3 @@ +{"AVERAGE_RATING":null,"BRAND_NAME":null,"CATEGORY_LEVEL_1":null,"CATEGORY_SLUG_PATH":null,"IS_ACTIVE":null,"IS_PRIVATE_LABEL":null,"LAUNCHED_ON":null,"LIST_PRICE_USD":null,"MARGIN_PCT":null,"PRIMARY_SUPPLIER_CODE":null,"PRODUCT_ID":null,"REVIEW_COUNT":null,"SKU":"DUCK-MAL-01","SKU_FAMILY":null,"SOURCE_LOADED_AT":null,"SUPPLIER_COUNT":null,"TITLE":null,"VARIANT_LABEL":null,"WISHLISTING_CUSTOMERS":null} +{"AVERAGE_RATING":null,"BRAND_NAME":null,"CATEGORY_LEVEL_1":null,"CATEGORY_SLUG_PATH":null,"IS_ACTIVE":null,"IS_PRIVATE_LABEL":null,"LAUNCHED_ON":null,"LIST_PRICE_USD":null,"MARGIN_PCT":null,"PRIMARY_SUPPLIER_CODE":null,"PRODUCT_ID":null,"REVIEW_COUNT":null,"SKU":"DUCK-PIN-02","SKU_FAMILY":null,"SOURCE_LOADED_AT":null,"SUPPLIER_COUNT":null,"TITLE":null,"VARIANT_LABEL":null,"WISHLISTING_CUSTOMERS":null} +{"AVERAGE_RATING":null,"BRAND_NAME":null,"CATEGORY_LEVEL_1":null,"CATEGORY_SLUG_PATH":null,"IS_ACTIVE":null,"IS_PRIVATE_LABEL":null,"LAUNCHED_ON":null,"LIST_PRICE_USD":null,"MARGIN_PCT":null,"PRIMARY_SUPPLIER_CODE":null,"PRODUCT_ID":null,"REVIEW_COUNT":null,"SKU":"DECOY-TEAL-1","SKU_FAMILY":null,"SOURCE_LOADED_AT":null,"SUPPLIER_COUNT":null,"TITLE":null,"VARIANT_LABEL":null,"WISHLISTING_CUSTOMERS":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/refs/fct_order_lines.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/refs/fct_order_lines.jsonl new file mode 100644 index 0000000000..c56a74ca15 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/refs/fct_order_lines.jsonl @@ -0,0 +1,4 @@ +{"CATEGORY_LEVEL_1":null,"DISCOUNT_AMOUNT_MINOR":null,"GROSS_AMOUNT_MINOR":null,"HAS_STACKED_PROMOTIONS":null,"LINE_WEIGHT_KG":null,"LIST_AMOUNT_MINOR":null,"MARKDOWN_PCT":null,"NET_AMOUNT_MINOR":2000,"ORDER_ID":"O1","ORDER_LINE_ID":"L1","PROMO_CODES":null,"QUANTITY":2,"SKU":"DUCK-MAL-01","SKU_FAMILY":null,"TAX_AMOUNT_MINOR":null,"UNIT_AMOUNT_MINOR":null} +{"CATEGORY_LEVEL_1":null,"DISCOUNT_AMOUNT_MINOR":null,"GROSS_AMOUNT_MINOR":null,"HAS_STACKED_PROMOTIONS":null,"LINE_WEIGHT_KG":null,"LIST_AMOUNT_MINOR":null,"MARKDOWN_PCT":null,"NET_AMOUNT_MINOR":4500,"ORDER_ID":"O1","ORDER_LINE_ID":"L2","PROMO_CODES":null,"QUANTITY":3,"SKU":"DUCK-PIN-02","SKU_FAMILY":null,"TAX_AMOUNT_MINOR":null,"UNIT_AMOUNT_MINOR":null} +{"CATEGORY_LEVEL_1":null,"DISCOUNT_AMOUNT_MINOR":null,"GROSS_AMOUNT_MINOR":null,"HAS_STACKED_PROMOTIONS":null,"LINE_WEIGHT_KG":null,"LIST_AMOUNT_MINOR":null,"MARKDOWN_PCT":null,"NET_AMOUNT_MINOR":10000,"ORDER_ID":"O2","ORDER_LINE_ID":"L3","PROMO_CODES":null,"QUANTITY":5,"SKU":"DECOY-TEAL-1","SKU_FAMILY":null,"TAX_AMOUNT_MINOR":null,"UNIT_AMOUNT_MINOR":null} +{"CATEGORY_LEVEL_1":null,"DISCOUNT_AMOUNT_MINOR":null,"GROSS_AMOUNT_MINOR":null,"HAS_STACKED_PROMOTIONS":null,"LINE_WEIGHT_KG":null,"LIST_AMOUNT_MINOR":null,"MARKDOWN_PCT":null,"NET_AMOUNT_MINOR":1000,"ORDER_ID":"O3","ORDER_LINE_ID":"L4","PROMO_CODES":null,"QUANTITY":1,"SKU":"DUCK-MAL-01","SKU_FAMILY":null,"TAX_AMOUNT_MINOR":null,"UNIT_AMOUNT_MINOR":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/refs/int_product_catalogue.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/refs/int_product_catalogue.jsonl new file mode 100644 index 0000000000..c68525d4fa --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/refs/int_product_catalogue.jsonl @@ -0,0 +1,3 @@ +{"BRAND_CODE":null,"BRAND_NAME":null,"CATEGORY_ID":"C1","CATEGORY_LEVEL_1":null,"CATEGORY_SLUG_PATH":null,"IMAGE_COUNT":null,"IS_ACTIVE":null,"IS_BUNDLE_COMPONENT":null,"IS_PRIVATE_LABEL":null,"LAUNCHED_ON":null,"MATERIAL":null,"PRIMARY_IMAGE_URL":null,"PRODUCT_ID":null,"SKU":"DUCK-MAL-01","SKU_FAMILY":null,"TITLE":null,"VARIANT_LABEL":null,"WEIGHT_KG":null} +{"BRAND_CODE":null,"BRAND_NAME":null,"CATEGORY_ID":"C2","CATEGORY_LEVEL_1":null,"CATEGORY_SLUG_PATH":null,"IMAGE_COUNT":null,"IS_ACTIVE":null,"IS_BUNDLE_COMPONENT":null,"IS_PRIVATE_LABEL":null,"LAUNCHED_ON":null,"MATERIAL":null,"PRIMARY_IMAGE_URL":null,"PRODUCT_ID":null,"SKU":"DUCK-PIN-02","SKU_FAMILY":null,"TITLE":null,"VARIANT_LABEL":null,"WEIGHT_KG":null} +{"BRAND_CODE":null,"BRAND_NAME":null,"CATEGORY_ID":"C3","CATEGORY_LEVEL_1":null,"CATEGORY_SLUG_PATH":null,"IMAGE_COUNT":null,"IS_ACTIVE":null,"IS_BUNDLE_COMPONENT":null,"IS_PRIVATE_LABEL":null,"LAUNCHED_ON":null,"MATERIAL":null,"PRIMARY_IMAGE_URL":null,"PRODUCT_ID":null,"SKU":"DECOY-TEAL-1","SKU_FAMILY":null,"TITLE":null,"VARIANT_LABEL":null,"WEIGHT_KG":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/scenario.json b/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/scenario.json new file mode 100644 index 0000000000..9a5c73cb10 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/scenario.json @@ -0,0 +1,360 @@ +{ + "capture_adapter": "snowflake", + "capture_dialect": "snowflake", + "captured_at": "2026-10-24T13:09:48Z", + "format": "jsonl", + "input_fingerprint": "e255c9818b7d3a41a17d7120c9e300e5208ddc445b17405167dc33ec487cbfb8", + "relations": [ + { + "bytes": 1219, + "columns": [ + { + "local_type": "TEXT(12)", + "name": "sku", + "warehouse_type": "VARCHAR(12)" + }, + { + "local_type": "TEXT(16777216)", + "name": "product_id", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "sku_family", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "title", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "variant_label", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "brand_name", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "is_private_label", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "category_level_1", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "category_slug_path", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "is_active", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "launched_on", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "list_price_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "margin_pct", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "primary_supplier_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "supplier_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "review_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "average_rating", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "wishlisting_customers", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "source_loaded_at", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/dim_products.jsonl", + "kind": "ref", + "logical_name": "dim_products", + "row_count": 3 + }, + { + "bytes": 1434, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "order_line_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "order_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(12)", + "name": "sku", + "warehouse_type": "VARCHAR(12)" + }, + { + "local_type": "TEXT(16777216)", + "name": "sku_family", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "category_level_1", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(1, 0)", + "name": "quantity", + "warehouse_type": "NUMBER(1,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "unit_amount_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "list_amount_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "markdown_pct", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "gross_amount_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "discount_amount_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(5, 0)", + "name": "net_amount_minor", + "warehouse_type": "NUMBER(5,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "tax_amount_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "promo_codes", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "has_stacked_promotions", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "line_weight_kg", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/fct_order_lines.jsonl", + "kind": "ref", + "logical_name": "fct_order_lines", + "row_count": 4 + }, + { + "bytes": 1084, + "columns": [ + { + "local_type": "TEXT(12)", + "name": "sku", + "warehouse_type": "VARCHAR(12)" + }, + { + "local_type": "TEXT(16777216)", + "name": "product_id", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "sku_family", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "title", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "variant_label", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "brand_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "brand_name", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "is_private_label", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(2)", + "name": "category_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "category_slug_path", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "category_level_1", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "material", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "weight_kg", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "primary_image_url", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "image_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "is_active", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "launched_on", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "is_bundle_component", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/int_product_catalogue.jsonl", + "kind": "ref", + "logical_name": "int_product_catalogue", + "row_count": 3 + }, + { + "bytes": 911, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "category_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "parent_category_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(12)", + "name": "slug", + "warehouse_type": "VARCHAR(12)" + }, + { + "local_type": "TEXT(18)", + "name": "display_name", + "warehouse_type": "VARCHAR(18)" + }, + { + "local_type": "DECIMAL(1, 0)", + "name": "sort_order", + "warehouse_type": "NUMBER(1,0)" + }, + { + "local_type": "JSON", + "name": "translations", + "warehouse_type": "VARIANT" + }, + { + "local_type": "TEXT(1)", + "name": "_op", + "warehouse_type": "VARCHAR(1)" + }, + { + "local_type": "DECIMAL(1, 0)", + "name": "_seq", + "warehouse_type": "NUMBER(1,0)" + }, + { + "local_type": "TIMESTAMP(9)", + "name": "_loaded_at", + "warehouse_type": "TIMESTAMP_NTZ(9)" + } + ], + "file": "sources/catalogue__categories.jsonl", + "kind": "source", + "logical_name": "catalogue__categories", + "row_count": 5 + } + ], + "scenario_name": "category_tree__rolls_sales_up_built_paths", + "sqlbuild_version": "0.175.0", + "total_bytes": 4648, + "total_rows": 15, + "version": 1 +} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/sources/catalogue__categories.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/sources/catalogue__categories.jsonl new file mode 100644 index 0000000000..5a238f09c1 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/category_tree__rolls_sales_up_built_paths/sources/catalogue__categories.jsonl @@ -0,0 +1,5 @@ +{"CATEGORY_ID":"C0","DISPLAY_NAME":"Ducks","PARENT_CATEGORY_ID":null,"SLUG":"Ducks","SORT_ORDER":1,"TRANSLATIONS":null,"_LOADED_AT":"2026-03-01T10:00:00","_OP":"I","_SEQ":1} +{"CATEGORY_ID":"C1","DISPLAY_NAME":"Rubber ducks","PARENT_CATEGORY_ID":"C0","SLUG":"Rubber Ducks","SORT_ORDER":1,"TRANSLATIONS":null,"_LOADED_AT":"2026-03-01T10:00:00","_OP":"I","_SEQ":2} +{"CATEGORY_ID":"C2","DISPLAY_NAME":"Collectible series","PARENT_CATEGORY_ID":"C1","SLUG":"collectible","SORT_ORDER":1,"TRANSLATIONS":null,"_LOADED_AT":"2026-03-01T10:00:00","_OP":"I","_SEQ":3} +{"CATEGORY_ID":"C3","DISPLAY_NAME":"Decoys","PARENT_CATEGORY_ID":"C0","SLUG":"decoys","SORT_ORDER":2,"TRANSLATIONS":null,"_LOADED_AT":"2026-03-01T10:00:00","_OP":"I","_SEQ":4} +{"CATEGORY_ID":"C4","DISPLAY_NAME":"Textiles","PARENT_CATEGORY_ID":"C4","SLUG":"textiles","SORT_ORDER":2,"TRANSLATIONS":null,"_LOADED_AT":"2026-03-01T10:00:00","_OP":"I","_SEQ":5} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_customer_order_sequence.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_customer_order_sequence.jsonl new file mode 100644 index 0000000000..9bad2aad82 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_customer_order_sequence.jsonl @@ -0,0 +1 @@ +{"AMOUNT_USD":null,"COHORT_MONTH":null,"CUSTOMER_ID":"C1","DAYS_SINCE_PREVIOUS_ORDER":null,"IS_CHANNEL_SWITCH":null,"IS_FIRST_ORDER":true,"LIFETIME_REVENUE_USD":null,"ORDER_ID":"O1","ORDER_NUMBER":1,"PLACED_AT":null,"PLACED_DATE_LOCAL":null,"PREVIOUS_SALES_CHANNEL":null,"ROLLING_90D_REVENUE_USD":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_order_status_timeline.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_order_status_timeline.jsonl new file mode 100644 index 0000000000..ff5a17fcc8 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_order_status_timeline.jsonl @@ -0,0 +1 @@ +{"CANCELLED_AT":null,"FIRST_DELIVERED_AT":null,"FIRST_PAID_AT":"2026-03-10T10:00:00+00:00","FIRST_SHIPPED_AT":null,"HAS_REGRESSION":null,"HOURS_PAID_TO_SHIPPED":null,"LAST_STATUS":null,"LONGEST_STATUS":null,"MINUTES_TO_PAY":null,"ORDER_ID":"O1","STATUS_PATH":null,"TRANSITION_COUNT":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_orders_usd.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_orders_usd.jsonl new file mode 100644 index 0000000000..74a31f5a97 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_orders_usd.jsonl @@ -0,0 +1,3 @@ +{"AMOUNT":null,"AMOUNT_USD":null,"BASKET_SIZE":null,"CURRENCY_CODE":"EUR","CUSTOMER_ID":"C1","DISCOUNT_AMOUNT_USD":null,"GROSS_AMOUNT_USD":null,"IS_RATE_STALE":null,"ORDER_ID":"O1","ORDER_STATUS":"DELIVERED","PLACED_AT":null,"PLACED_DATE_LOCAL":null,"PROMO_CODE":null,"RATE_AGE_DAYS":null,"RATE_DATE":null,"RATE_TO_USD":null,"REGION_CODE":"EU","SALES_CHANNEL":null,"TAX_AMOUNT_USD":null,"UNITS":null,"UTM_CAMPAIGN":null,"UTM_MEDIUM":null,"UTM_SOURCE":null} +{"AMOUNT":null,"AMOUNT_USD":null,"BASKET_SIZE":null,"CURRENCY_CODE":"EUR","CUSTOMER_ID":"C2","DISCOUNT_AMOUNT_USD":null,"GROSS_AMOUNT_USD":null,"IS_RATE_STALE":null,"ORDER_ID":"O2","ORDER_STATUS":"SHIPPED","PLACED_AT":null,"PLACED_DATE_LOCAL":null,"PROMO_CODE":null,"RATE_AGE_DAYS":null,"RATE_DATE":null,"RATE_TO_USD":null,"REGION_CODE":"EU","SALES_CHANNEL":null,"TAX_AMOUNT_USD":null,"UNITS":null,"UTM_CAMPAIGN":null,"UTM_MEDIUM":null,"UTM_SOURCE":null} +{"AMOUNT":null,"AMOUNT_USD":null,"BASKET_SIZE":null,"CURRENCY_CODE":"USD","CUSTOMER_ID":"C3","DISCOUNT_AMOUNT_USD":null,"GROSS_AMOUNT_USD":null,"IS_RATE_STALE":null,"ORDER_ID":"O3","ORDER_STATUS":"PENDING","PLACED_AT":null,"PLACED_DATE_LOCAL":null,"PROMO_CODE":null,"RATE_AGE_DAYS":null,"RATE_DATE":null,"RATE_TO_USD":null,"REGION_CODE":"US","SALES_CHANNEL":null,"TAX_AMOUNT_USD":null,"UNITS":null,"UTM_CAMPAIGN":null,"UTM_MEDIUM":null,"UTM_SOURCE":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_payment_attempts.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_payment_attempts.jsonl new file mode 100644 index 0000000000..d711fc9d9b --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/int_payment_attempts.jsonl @@ -0,0 +1,3 @@ +{"AMOUNT_MINOR":2000,"ATTEMPT_COUNT":null,"ATTEMPT_NUMBER":1,"AUTHORISED_AT":null,"CARD_BIN_VALID":null,"CARD_SCHEME":null,"CURRENCY_CODE":"EUR","DECLINE_FAMILY":null,"GATEWAY_ERROR_CODE":null,"GATEWAY_STATUS":null,"HAD_CHARGEBACK":null,"IS_FINAL_ATTEMPT":false,"IS_SUCCESSFUL":false,"ORDER_ID":"O1","PAYMENT_ID":"P0","PAYMENT_METHOD":"card","RISK_BAND":null,"RISK_SCORE":null,"SECONDS_SINCE_PREVIOUS_ATTEMPT":null,"THREE_DS_OUTCOME":null} +{"AMOUNT_MINOR":2000,"ATTEMPT_COUNT":null,"ATTEMPT_NUMBER":2,"AUTHORISED_AT":null,"CARD_BIN_VALID":null,"CARD_SCHEME":null,"CURRENCY_CODE":"EUR","DECLINE_FAMILY":null,"GATEWAY_ERROR_CODE":null,"GATEWAY_STATUS":null,"HAD_CHARGEBACK":null,"IS_FINAL_ATTEMPT":true,"IS_SUCCESSFUL":true,"ORDER_ID":"O1","PAYMENT_ID":"P1","PAYMENT_METHOD":"card","RISK_BAND":null,"RISK_SCORE":null,"SECONDS_SINCE_PREVIOUS_ATTEMPT":null,"THREE_DS_OUTCOME":null} +{"AMOUNT_MINOR":1500,"ATTEMPT_COUNT":null,"ATTEMPT_NUMBER":1,"AUTHORISED_AT":null,"CARD_BIN_VALID":null,"CARD_SCHEME":null,"CURRENCY_CODE":"EUR","DECLINE_FAMILY":null,"GATEWAY_ERROR_CODE":null,"GATEWAY_STATUS":null,"HAD_CHARGEBACK":null,"IS_FINAL_ATTEMPT":true,"IS_SUCCESSFUL":true,"ORDER_ID":"O2","PAYMENT_ID":"P2","PAYMENT_METHOD":"phone_wallet","RISK_BAND":null,"RISK_SCORE":null,"SECONDS_SINCE_PREVIOUS_ATTEMPT":null,"THREE_DS_OUTCOME":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/stg_chargebacks.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/stg_chargebacks.jsonl new file mode 100644 index 0000000000..a2f506fcf6 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/stg_chargebacks.jsonl @@ -0,0 +1,2 @@ +{"AMOUNT_MINOR":600,"CHARGEBACK_ID":"CB1","CLOSED_AT":null,"CURRENCY_CODE":null,"EVIDENCE":null,"EVIDENCE_DOCUMENT_COUNT":null,"OPENED_AT":null,"OUTCOME":"lost","PAYMENT_ID":"P2","REASON_CODE":null,"REASON_FAMILY":null,"SOURCE_LOADED_AT":null,"SOURCE_SEQ":null} +{"AMOUNT_MINOR":100,"CHARGEBACK_ID":"CB2","CLOSED_AT":null,"CURRENCY_CODE":null,"EVIDENCE":null,"EVIDENCE_DOCUMENT_COUNT":null,"OPENED_AT":null,"OUTCOME":"won","PAYMENT_ID":"P2","REASON_CODE":null,"REASON_FAMILY":null,"SOURCE_LOADED_AT":null,"SOURCE_SEQ":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/stg_gift_card_transactions.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/stg_gift_card_transactions.jsonl new file mode 100644 index 0000000000..ec1b6fc403 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/stg_gift_card_transactions.jsonl @@ -0,0 +1,2 @@ +{"CHANNEL":null,"GIFT_CARD_ID":"GC1","OCCURRED_AT":null,"ORDER_ID":"O2","SIGNED_AMOUNT_MINOR":-500,"SOURCE_LOADED_AT":null,"SOURCE_SEQ":null,"TERMINAL_ID":null,"TRANSACTION_ID":"G1","TRANSACTION_TYPE":"REDEEM"} +{"CHANNEL":null,"GIFT_CARD_ID":"GC1","OCCURRED_AT":null,"ORDER_ID":"O2","SIGNED_AMOUNT_MINOR":1000,"SOURCE_LOADED_AT":null,"SOURCE_SEQ":null,"TERMINAL_ID":null,"TRANSACTION_ID":"G2","TRANSACTION_TYPE":"ISSUE"} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/stg_refunds.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/stg_refunds.jsonl new file mode 100644 index 0000000000..c615e77c14 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/refs/stg_refunds.jsonl @@ -0,0 +1 @@ +{"AMOUNT_MINOR":2000,"INITIATED_BY":null,"PAYMENT_ID":"P1","REASON_CATEGORY":"return","REASON_TEXT":null,"REFUNDED_AT":null,"REFUNDED_ITEMS":null,"REFUNDED_LINE_COUNT":null,"REFUND_ID":"R1","SOURCE_LOADED_AT":null,"SOURCE_SEQ":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/scenario.json b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/scenario.json new file mode 100644 index 0000000000..e3ccbb2bba --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/payments__fully_refunded_orders_collect_nothing/scenario.json @@ -0,0 +1,587 @@ +{ + "capture_adapter": "snowflake", + "capture_dialect": "snowflake", + "captured_at": "2026-10-24T13:09:48Z", + "format": "jsonl", + "input_fingerprint": "530fe2a9605bd0a7eedf6b5fbbc55b5e1cc40095cd9af0aff18ddd1b47788541", + "relations": [ + { + "bytes": 303, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "order_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "customer_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TIMESTAMPTZ", + "name": "placed_at", + "warehouse_type": "TIMESTAMP_TZ(9)" + }, + { + "local_type": "DATE", + "name": "placed_date_local", + "warehouse_type": "DATE" + }, + { + "local_type": "DECIMAL(1, 0)", + "name": "order_number", + "warehouse_type": "NUMBER(1,0)" + }, + { + "local_type": "BOOLEAN", + "name": "is_first_order", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "days_since_previous_order", + "warehouse_type": "NUMBER(38,0)" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "amount_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "lifetime_revenue_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "rolling_90d_revenue_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "previous_sales_channel", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "is_channel_switch", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "DATE", + "name": "cohort_month", + "warehouse_type": "DATE" + } + ], + "file": "refs/int_customer_order_sequence.jsonl", + "kind": "ref", + "logical_name": "int_customer_order_sequence", + "row_count": 1 + }, + { + "bytes": 289, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "order_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "transition_count", + "warehouse_type": "NUMBER(38,0)" + }, + { + "local_type": "TIMESTAMPTZ", + "name": "first_paid_at", + "warehouse_type": "TIMESTAMP_TZ(9)" + }, + { + "local_type": "TIMESTAMPTZ", + "name": "first_shipped_at", + "warehouse_type": "TIMESTAMP_TZ(9)" + }, + { + "local_type": "TIMESTAMPTZ", + "name": "first_delivered_at", + "warehouse_type": "TIMESTAMP_TZ(9)" + }, + { + "local_type": "TIMESTAMPTZ", + "name": "cancelled_at", + "warehouse_type": "TIMESTAMP_TZ(9)" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "minutes_to_pay", + "warehouse_type": "NUMBER(38,0)" + }, + { + "local_type": "DECIMAL(12, 2)", + "name": "hours_paid_to_shipped", + "warehouse_type": "NUMBER(12,2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "status_path", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "has_regression", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "TEXT(16777216)", + "name": "last_status", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "longest_status", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/int_order_status_timeline.jsonl", + "kind": "ref", + "logical_name": "int_order_status_timeline", + "row_count": 1 + }, + { + "bytes": 1367, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "order_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "customer_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "region_code", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(3)", + "name": "currency_code", + "warehouse_type": "VARCHAR(3)" + }, + { + "local_type": "TEXT(16777216)", + "name": "placed_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "placed_date_local", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(9)", + "name": "order_status", + "warehouse_type": "VARCHAR(9)" + }, + { + "local_type": "TEXT(16777216)", + "name": "sales_channel", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "units", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "amount", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "amount_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "gross_amount_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "discount_amount_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "tax_amount_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "rate_to_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "rate_date", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "rate_age_days", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "is_rate_stale", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "promo_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "utm_source", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "utm_medium", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "utm_campaign", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "basket_size", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/int_orders_usd.jsonl", + "kind": "ref", + "logical_name": "int_orders_usd", + "row_count": 3 + }, + { + "bytes": 1324, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "payment_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "order_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "DECIMAL(1, 0)", + "name": "attempt_number", + "warehouse_type": "NUMBER(1,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "attempt_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "is_final_attempt", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "TEXT(12)", + "name": "payment_method", + "warehouse_type": "VARCHAR(12)" + }, + { + "local_type": "DECIMAL(4, 0)", + "name": "amount_minor", + "warehouse_type": "NUMBER(4,0)" + }, + { + "local_type": "TEXT(3)", + "name": "currency_code", + "warehouse_type": "VARCHAR(3)" + }, + { + "local_type": "TEXT(16777216)", + "name": "authorised_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "seconds_since_previous_attempt", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "gateway_status", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "is_successful", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "TEXT(16777216)", + "name": "decline_family", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "gateway_error_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "risk_score", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "risk_band", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "three_ds_outcome", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "card_bin_valid", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "card_scheme", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "had_chargeback", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/int_payment_attempts.jsonl", + "kind": "ref", + "logical_name": "int_payment_attempts", + "row_count": 3 + }, + { + "bytes": 523, + "columns": [ + { + "local_type": "TEXT(3)", + "name": "chargeback_id", + "warehouse_type": "VARCHAR(3)" + }, + { + "local_type": "TEXT(2)", + "name": "payment_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "DECIMAL(3, 0)", + "name": "amount_minor", + "warehouse_type": "NUMBER(3,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "currency_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "reason_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "reason_family", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "opened_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "closed_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(4)", + "name": "outcome", + "warehouse_type": "VARCHAR(4)" + }, + { + "local_type": "TEXT(16777216)", + "name": "evidence_document_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "evidence", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "source_seq", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "source_loaded_at", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/stg_chargebacks.jsonl", + "kind": "ref", + "logical_name": "stg_chargebacks", + "row_count": 2 + }, + { + "bytes": 421, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "transaction_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(3)", + "name": "gift_card_id", + "warehouse_type": "VARCHAR(3)" + }, + { + "local_type": "TEXT(2)", + "name": "order_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(6)", + "name": "transaction_type", + "warehouse_type": "VARCHAR(6)" + }, + { + "local_type": "DECIMAL(4, 0)", + "name": "signed_amount_minor", + "warehouse_type": "NUMBER(4,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "occurred_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "channel", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "terminal_id", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "source_seq", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "source_loaded_at", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/stg_gift_card_transactions.jsonl", + "kind": "ref", + "logical_name": "stg_gift_card_transactions", + "row_count": 2 + }, + { + "bytes": 233, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "refund_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "payment_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "DECIMAL(4, 0)", + "name": "amount_minor", + "warehouse_type": "NUMBER(4,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "reason_text", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(6)", + "name": "reason_category", + "warehouse_type": "VARCHAR(6)" + }, + { + "local_type": "TIMESTAMPTZ", + "name": "refunded_at", + "warehouse_type": "TIMESTAMP_TZ(9)" + }, + { + "local_type": "TEXT(16777216)", + "name": "initiated_by", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "refunded_line_count", + "warehouse_type": "NUMBER(38,0)" + }, + { + "local_type": "JSON", + "name": "refunded_items", + "warehouse_type": "ARRAY" + }, + { + "local_type": "DECIMAL(38, 0)", + "name": "source_seq", + "warehouse_type": "NUMBER(38,0)" + }, + { + "local_type": "TIMESTAMP(9)", + "name": "source_loaded_at", + "warehouse_type": "TIMESTAMP_NTZ(9)" + } + ], + "file": "refs/stg_refunds.jsonl", + "kind": "ref", + "logical_name": "stg_refunds", + "row_count": 1 + } + ], + "scenario_name": "payments__fully_refunded_orders_collect_nothing", + "sqlbuild_version": "0.175.0", + "total_bytes": 4460, + "total_rows": 13, + "version": 1 +} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/region_scorecard__combines_sales_and_recognised_revenue/refs/fct_orders.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/region_scorecard__combines_sales_and_recognised_revenue/refs/fct_orders.jsonl new file mode 100644 index 0000000000..f8cd38c057 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/region_scorecard__combines_sales_and_recognised_revenue/refs/fct_orders.jsonl @@ -0,0 +1,4 @@ +{"AMOUNT":null,"AMOUNT_USD":"60.00","BASKET_SIZE":null,"CAPTURED_MINOR":null,"CURRENCY_CODE":null,"CUSTOMER_ID":"C1","DISCOUNT_AMOUNT_USD":"0.00","FIRST_DELIVERED_AT":null,"FIRST_PAID_AT":null,"GROSS_AMOUNT_USD":null,"HOURS_PAID_TO_SHIPPED":null,"IS_FIRST_ORDER":true,"IS_FULLY_REFUNDED":null,"NET_COLLECTED_MINOR":null,"ORDER_ID":"O1","ORDER_NUMBER":null,"ORDER_STATUS":"DELIVERED","PLACED_AT":null,"PLACED_DATE_LOCAL":"2026-03-10","PROMO_CODE":null,"RATE_DATE":null,"RATE_TO_USD":null,"REFUNDED_MINOR":null,"REGION_CODE":"EU","SALES_CHANNEL":null,"TAX_AMOUNT_USD":null,"UNITS":1,"UTM_CAMPAIGN":null,"UTM_MEDIUM":null,"UTM_SOURCE":null} +{"AMOUNT":null,"AMOUNT_USD":"40.00","BASKET_SIZE":null,"CAPTURED_MINOR":null,"CURRENCY_CODE":null,"CUSTOMER_ID":"C2","DISCOUNT_AMOUNT_USD":"0.00","FIRST_DELIVERED_AT":null,"FIRST_PAID_AT":null,"GROSS_AMOUNT_USD":null,"HOURS_PAID_TO_SHIPPED":null,"IS_FIRST_ORDER":false,"IS_FULLY_REFUNDED":null,"NET_COLLECTED_MINOR":null,"ORDER_ID":"O2","ORDER_NUMBER":null,"ORDER_STATUS":"DELIVERED","PLACED_AT":null,"PLACED_DATE_LOCAL":"2026-03-10","PROMO_CODE":null,"RATE_DATE":null,"RATE_TO_USD":null,"REFUNDED_MINOR":null,"REGION_CODE":"EU","SALES_CHANNEL":null,"TAX_AMOUNT_USD":null,"UNITS":1,"UTM_CAMPAIGN":null,"UTM_MEDIUM":null,"UTM_SOURCE":null} +{"AMOUNT":null,"AMOUNT_USD":"50.00","BASKET_SIZE":null,"CAPTURED_MINOR":null,"CURRENCY_CODE":null,"CUSTOMER_ID":"C2","DISCOUNT_AMOUNT_USD":"0.00","FIRST_DELIVERED_AT":null,"FIRST_PAID_AT":null,"GROSS_AMOUNT_USD":null,"HOURS_PAID_TO_SHIPPED":null,"IS_FIRST_ORDER":false,"IS_FULLY_REFUNDED":null,"NET_COLLECTED_MINOR":null,"ORDER_ID":"O3","ORDER_NUMBER":null,"ORDER_STATUS":"DELIVERED","PLACED_AT":null,"PLACED_DATE_LOCAL":"2026-03-20","PROMO_CODE":null,"RATE_DATE":null,"RATE_TO_USD":null,"REFUNDED_MINOR":null,"REGION_CODE":"EU","SALES_CHANNEL":null,"TAX_AMOUNT_USD":null,"UNITS":1,"UTM_CAMPAIGN":null,"UTM_MEDIUM":null,"UTM_SOURCE":null} +{"AMOUNT":null,"AMOUNT_USD":"200.00","BASKET_SIZE":null,"CAPTURED_MINOR":null,"CURRENCY_CODE":null,"CUSTOMER_ID":"C3","DISCOUNT_AMOUNT_USD":"0.00","FIRST_DELIVERED_AT":null,"FIRST_PAID_AT":null,"GROSS_AMOUNT_USD":null,"HOURS_PAID_TO_SHIPPED":null,"IS_FIRST_ORDER":true,"IS_FULLY_REFUNDED":null,"NET_COLLECTED_MINOR":null,"ORDER_ID":"O4","ORDER_NUMBER":null,"ORDER_STATUS":"DELIVERED","PLACED_AT":null,"PLACED_DATE_LOCAL":"2026-03-05","PROMO_CODE":null,"RATE_DATE":null,"RATE_TO_USD":null,"REFUNDED_MINOR":null,"REGION_CODE":"US","SALES_CHANNEL":null,"TAX_AMOUNT_USD":null,"UNITS":1,"UTM_CAMPAIGN":null,"UTM_MEDIUM":null,"UTM_SOURCE":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/region_scorecard__combines_sales_and_recognised_revenue/refs/int_revenue_recognition_daily.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/region_scorecard__combines_sales_and_recognised_revenue/refs/int_revenue_recognition_daily.jsonl new file mode 100644 index 0000000000..a0f871f6dc --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/region_scorecard__combines_sales_and_recognised_revenue/refs/int_revenue_recognition_daily.jsonl @@ -0,0 +1,4 @@ +{"DOCUMENT_COUNT":null,"MONTH_TO_DATE_USD":null,"RECOGNITION_DATE":"2026-03-05","RECOGNITION_KEY":"2026-03-05:EU:product","REGION_CODE":"EU","REVENUE_STREAM":"product","REVENUE_USD":"100.00","SHARE_OF_DAY":null,"TRAILING_7D_USD":null} +{"DOCUMENT_COUNT":null,"MONTH_TO_DATE_USD":null,"RECOGNITION_DATE":"2026-03-20","RECOGNITION_KEY":"2026-03-20:EU:product","REGION_CODE":"EU","REVENUE_STREAM":"product","REVENUE_USD":"50.00","SHARE_OF_DAY":null,"TRAILING_7D_USD":null} +{"DOCUMENT_COUNT":null,"MONTH_TO_DATE_USD":null,"RECOGNITION_DATE":"2026-03-21","RECOGNITION_KEY":"2026-03-21:EU:refund","REGION_CODE":"EU","REVENUE_STREAM":"refund","REVENUE_USD":"-15.00","SHARE_OF_DAY":null,"TRAILING_7D_USD":null} +{"DOCUMENT_COUNT":null,"MONTH_TO_DATE_USD":null,"RECOGNITION_DATE":"2026-03-10","RECOGNITION_KEY":"2026-03-10:EU:subscription","REGION_CODE":"EU","REVENUE_STREAM":"subscription","REVENUE_USD":"10.00","SHARE_OF_DAY":null,"TRAILING_7D_USD":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/region_scorecard__combines_sales_and_recognised_revenue/scenario.json b/benchmarks/duckshop/project/tests/_scenario_snapshots/region_scorecard__combines_sales_and_recognised_revenue/scenario.json new file mode 100644 index 0000000000..5a2ed7f563 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/region_scorecard__combines_sales_and_recognised_revenue/scenario.json @@ -0,0 +1,227 @@ +{ + "capture_adapter": "snowflake", + "capture_dialect": "snowflake", + "captured_at": "2026-10-24T13:09:48Z", + "format": "jsonl", + "input_fingerprint": "b9d2023c2fa62158a7028e977f0c110f6cccffdefd44c237f6f8deabf236d189", + "relations": [ + { + "bytes": 2555, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "order_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "customer_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(2)", + "name": "region_code", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "placed_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DATE", + "name": "placed_date_local", + "warehouse_type": "DATE" + }, + { + "local_type": "TEXT(9)", + "name": "order_status", + "warehouse_type": "VARCHAR(9)" + }, + { + "local_type": "TEXT(16777216)", + "name": "sales_channel", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(1, 0)", + "name": "units", + "warehouse_type": "NUMBER(1,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "currency_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "amount", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "amount_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "rate_to_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "rate_date", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "gross_amount_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "discount_amount_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "tax_amount_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "promo_code", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "utm_source", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "utm_medium", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "utm_campaign", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "basket_size", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "order_number", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "is_first_order", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "TEXT(16777216)", + "name": "first_paid_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "first_delivered_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "hours_paid_to_shipped", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "captured_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "refunded_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "net_collected_minor", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "is_fully_refunded", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/fct_orders.jsonl", + "kind": "ref", + "logical_name": "fct_orders", + "row_count": 4 + }, + { + "bytes": 946, + "columns": [ + { + "local_type": "TEXT(26)", + "name": "recognition_key", + "warehouse_type": "VARCHAR(26)" + }, + { + "local_type": "DATE", + "name": "recognition_date", + "warehouse_type": "DATE" + }, + { + "local_type": "TEXT(2)", + "name": "region_code", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(12)", + "name": "revenue_stream", + "warehouse_type": "VARCHAR(12)" + }, + { + "local_type": "DECIMAL(18, 2)", + "name": "revenue_usd", + "warehouse_type": "NUMBER(18,2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "document_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "month_to_date_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "trailing_7d_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "share_of_day", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/int_revenue_recognition_daily.jsonl", + "kind": "ref", + "logical_name": "int_revenue_recognition_daily", + "row_count": 4 + } + ], + "scenario_name": "region_scorecard__combines_sales_and_recognised_revenue", + "sqlbuild_version": "0.175.0", + "total_bytes": 3501, + "total_rows": 8, + "version": 1 +} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/stock_outs__finds_longest_run_per_week/refs/fct_inventory_daily.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/stock_outs__finds_longest_run_per_week/refs/fct_inventory_daily.jsonl new file mode 100644 index 0000000000..7e0ba31a6e --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/stock_outs__finds_longest_run_per_week/refs/fct_inventory_daily.jsonl @@ -0,0 +1,7 @@ +{"CALENDAR_DATE":"2026-03-09","DAYS_SINCE_COUNT":null,"IS_COUNT_OVERDUE":false,"IS_STOCKED_OUT":false,"ON_HAND_QTY":5,"SKU":"DUCK-1","STOCK_DAY_KEY":"DUCK-1|POND|2026-03-09","STOCK_VALUE_USD":null,"UNIT_COST_USD":null,"WAREHOUSE_CODE":"POND"} +{"CALENDAR_DATE":"2026-03-10","DAYS_SINCE_COUNT":null,"IS_COUNT_OVERDUE":false,"IS_STOCKED_OUT":true,"ON_HAND_QTY":0,"SKU":"DUCK-1","STOCK_DAY_KEY":"DUCK-1|POND|2026-03-10","STOCK_VALUE_USD":null,"UNIT_COST_USD":null,"WAREHOUSE_CODE":"POND"} +{"CALENDAR_DATE":"2026-03-11","DAYS_SINCE_COUNT":null,"IS_COUNT_OVERDUE":false,"IS_STOCKED_OUT":true,"ON_HAND_QTY":0,"SKU":"DUCK-1","STOCK_DAY_KEY":"DUCK-1|POND|2026-03-11","STOCK_VALUE_USD":null,"UNIT_COST_USD":null,"WAREHOUSE_CODE":"POND"} +{"CALENDAR_DATE":"2026-03-12","DAYS_SINCE_COUNT":null,"IS_COUNT_OVERDUE":false,"IS_STOCKED_OUT":false,"ON_HAND_QTY":5,"SKU":"DUCK-1","STOCK_DAY_KEY":"DUCK-1|POND|2026-03-12","STOCK_VALUE_USD":null,"UNIT_COST_USD":null,"WAREHOUSE_CODE":"POND"} +{"CALENDAR_DATE":"2026-03-13","DAYS_SINCE_COUNT":null,"IS_COUNT_OVERDUE":false,"IS_STOCKED_OUT":true,"ON_HAND_QTY":0,"SKU":"DUCK-1","STOCK_DAY_KEY":"DUCK-1|POND|2026-03-13","STOCK_VALUE_USD":null,"UNIT_COST_USD":null,"WAREHOUSE_CODE":"POND"} +{"CALENDAR_DATE":"2026-03-10","DAYS_SINCE_COUNT":null,"IS_COUNT_OVERDUE":false,"IS_STOCKED_OUT":false,"ON_HAND_QTY":5,"SKU":"DUCK-2","STOCK_DAY_KEY":"DUCK-2|POND|2026-03-10","STOCK_VALUE_USD":null,"UNIT_COST_USD":null,"WAREHOUSE_CODE":"POND"} +{"CALENDAR_DATE":"2026-03-16","DAYS_SINCE_COUNT":null,"IS_COUNT_OVERDUE":false,"IS_STOCKED_OUT":true,"ON_HAND_QTY":0,"SKU":"DUCK-1","STOCK_DAY_KEY":"DUCK-1|POND|2026-03-16","STOCK_VALUE_USD":null,"UNIT_COST_USD":null,"WAREHOUSE_CODE":"POND"} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/stock_outs__finds_longest_run_per_week/scenario.json b/benchmarks/duckshop/project/tests/_scenario_snapshots/stock_outs__finds_longest_run_per_week/scenario.json new file mode 100644 index 0000000000..8203323f78 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/stock_outs__finds_longest_run_per_week/scenario.json @@ -0,0 +1,73 @@ +{ + "capture_adapter": "snowflake", + "capture_dialect": "snowflake", + "captured_at": "2026-10-24T13:09:48Z", + "format": "jsonl", + "input_fingerprint": "b0fa60906be6b5a2f6977ba5af76608a3047bfed6fb8cb089181a8fe5ffaf9fe", + "relations": [ + { + "bytes": 1697, + "columns": [ + { + "local_type": "TEXT(22)", + "name": "stock_day_key", + "warehouse_type": "VARCHAR(22)" + }, + { + "local_type": "TEXT(6)", + "name": "sku", + "warehouse_type": "VARCHAR(6)" + }, + { + "local_type": "TEXT(4)", + "name": "warehouse_code", + "warehouse_type": "VARCHAR(4)" + }, + { + "local_type": "DATE", + "name": "calendar_date", + "warehouse_type": "DATE" + }, + { + "local_type": "DECIMAL(1, 0)", + "name": "on_hand_qty", + "warehouse_type": "NUMBER(1,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "unit_cost_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "stock_value_usd", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "is_stocked_out", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "TEXT(16777216)", + "name": "days_since_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "is_count_overdue", + "warehouse_type": "BOOLEAN" + } + ], + "file": "refs/fct_inventory_daily.jsonl", + "kind": "ref", + "logical_name": "fct_inventory_daily", + "row_count": 7 + } + ], + "scenario_name": "stock_outs__finds_longest_run_per_week", + "sqlbuild_version": "0.175.0", + "total_bytes": 1697, + "total_rows": 7, + "version": 1 +} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/support_sla__weekly_breaches_and_monthly_reasons/refs/fct_support_tickets.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/support_sla__weekly_breaches_and_monthly_reasons/refs/fct_support_tickets.jsonl new file mode 100644 index 0000000000..19ec9e0bfa --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/support_sla__weekly_breaches_and_monthly_reasons/refs/fct_support_tickets.jsonl @@ -0,0 +1,5 @@ +{"BUSINESS_MINUTES_TO_RESPONSE":20,"CHANNEL":"email","COMMENT_COUNT":null,"CONTACT_REASON":"damaged_item","CREATED_AT_UTC":"2026-03-09T10:00:00","CSAT_SCORE":null,"CUSTOMER_ID":null,"HANDOFF_COUNT":null,"IS_FIRST_RESPONSE_BREACHED":true,"IS_REPEAT_CONTACT_7D":false,"IS_RESOLUTION_BREACHED":false,"MINUTES_TO_FIRST_RESPONSE":30,"PRIORITY":1,"SENTIMENT_LABEL":null,"SOLVED_AT":null,"SOURCE_LOADED_AT":null,"TICKET_ID":"T1","TICKET_NUMBER_FOR_CUSTOMER":null,"TICKET_STATUS":null} +{"BUSINESS_MINUTES_TO_RESPONSE":40,"CHANNEL":"email","COMMENT_COUNT":null,"CONTACT_REASON":"damaged_item","CREATED_AT_UTC":"2026-03-10T10:00:00","CSAT_SCORE":null,"CUSTOMER_ID":null,"HANDOFF_COUNT":null,"IS_FIRST_RESPONSE_BREACHED":false,"IS_REPEAT_CONTACT_7D":true,"IS_RESOLUTION_BREACHED":false,"MINUTES_TO_FIRST_RESPONSE":50,"PRIORITY":1,"SENTIMENT_LABEL":null,"SOLVED_AT":null,"SOURCE_LOADED_AT":null,"TICKET_ID":"T2","TICKET_NUMBER_FOR_CUSTOMER":null,"TICKET_STATUS":null} +{"BUSINESS_MINUTES_TO_RESPONSE":100,"CHANNEL":"email","COMMENT_COUNT":null,"CONTACT_REASON":null,"CREATED_AT_UTC":"2026-03-11T10:00:00","CSAT_SCORE":null,"CUSTOMER_ID":null,"HANDOFF_COUNT":null,"IS_FIRST_RESPONSE_BREACHED":false,"IS_REPEAT_CONTACT_7D":false,"IS_RESOLUTION_BREACHED":true,"MINUTES_TO_FIRST_RESPONSE":10,"PRIORITY":1,"SENTIMENT_LABEL":null,"SOLVED_AT":null,"SOURCE_LOADED_AT":null,"TICKET_ID":"T3","TICKET_NUMBER_FOR_CUSTOMER":null,"TICKET_STATUS":null} +{"BUSINESS_MINUTES_TO_RESPONSE":null,"CHANNEL":"chat","COMMENT_COUNT":null,"CONTACT_REASON":"where_is_my_order","CREATED_AT_UTC":"2026-03-12T10:00:00","CSAT_SCORE":null,"CUSTOMER_ID":null,"HANDOFF_COUNT":null,"IS_FIRST_RESPONSE_BREACHED":false,"IS_REPEAT_CONTACT_7D":false,"IS_RESOLUTION_BREACHED":false,"MINUTES_TO_FIRST_RESPONSE":5,"PRIORITY":null,"SENTIMENT_LABEL":null,"SOLVED_AT":null,"SOURCE_LOADED_AT":null,"TICKET_ID":"T4","TICKET_NUMBER_FOR_CUSTOMER":null,"TICKET_STATUS":null} +{"BUSINESS_MINUTES_TO_RESPONSE":null,"CHANNEL":"phone","COMMENT_COUNT":null,"CONTACT_REASON":"damaged_item","CREATED_AT_UTC":null,"CSAT_SCORE":null,"CUSTOMER_ID":null,"HANDOFF_COUNT":null,"IS_FIRST_RESPONSE_BREACHED":true,"IS_REPEAT_CONTACT_7D":false,"IS_RESOLUTION_BREACHED":false,"MINUTES_TO_FIRST_RESPONSE":null,"PRIORITY":1,"SENTIMENT_LABEL":null,"SOLVED_AT":null,"SOURCE_LOADED_AT":null,"TICKET_ID":"T5","TICKET_NUMBER_FOR_CUSTOMER":null,"TICKET_STATUS":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/support_sla__weekly_breaches_and_monthly_reasons/refs/int_call_ivr_paths.jsonl b/benchmarks/duckshop/project/tests/_scenario_snapshots/support_sla__weekly_breaches_and_monthly_reasons/refs/int_call_ivr_paths.jsonl new file mode 100644 index 0000000000..0380cca743 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/support_sla__weekly_breaches_and_monthly_reasons/refs/int_call_ivr_paths.jsonl @@ -0,0 +1,3 @@ +{"ABANDONED_IN_IVR":false,"AGENT_ID":null,"CALLER_REGION":null,"CALL_ID":"CL1","DURATION_BUCKET":null,"FINAL_MENU_NODE":null,"FIRST_MENU_CHOICE":null,"MENU_STEPS":null,"REPEATED_MENU_VISITS":null,"TICKET_ID":"T2"} +{"ABANDONED_IN_IVR":true,"AGENT_ID":null,"CALLER_REGION":null,"CALL_ID":"CL2","DURATION_BUCKET":null,"FINAL_MENU_NODE":null,"FIRST_MENU_CHOICE":null,"MENU_STEPS":null,"REPEATED_MENU_VISITS":null,"TICKET_ID":"T2"} +{"ABANDONED_IN_IVR":true,"AGENT_ID":null,"CALLER_REGION":null,"CALL_ID":"CL3","DURATION_BUCKET":null,"FINAL_MENU_NODE":null,"FIRST_MENU_CHOICE":null,"MENU_STEPS":null,"REPEATED_MENU_VISITS":null,"TICKET_ID":null} diff --git a/benchmarks/duckshop/project/tests/_scenario_snapshots/support_sla__weekly_breaches_and_monthly_reasons/scenario.json b/benchmarks/duckshop/project/tests/_scenario_snapshots/support_sla__weekly_breaches_and_monthly_reasons/scenario.json new file mode 100644 index 0000000000..c2a5e33744 --- /dev/null +++ b/benchmarks/duckshop/project/tests/_scenario_snapshots/support_sla__weekly_breaches_and_monthly_reasons/scenario.json @@ -0,0 +1,177 @@ +{ + "capture_adapter": "snowflake", + "capture_dialect": "snowflake", + "captured_at": "2026-10-24T13:09:48Z", + "format": "jsonl", + "input_fingerprint": "ff58d21aed3cd7a80e929c630ce7234d21fe2c7e4d3a2bcb9545688ce112446e", + "relations": [ + { + "bytes": 2377, + "columns": [ + { + "local_type": "TEXT(2)", + "name": "ticket_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "customer_id", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(5)", + "name": "channel", + "warehouse_type": "VARCHAR(5)" + }, + { + "local_type": "DECIMAL(1, 0)", + "name": "priority", + "warehouse_type": "NUMBER(1,0)" + }, + { + "local_type": "TEXT(16777216)", + "name": "ticket_status", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(17)", + "name": "contact_reason", + "warehouse_type": "VARCHAR(17)" + }, + { + "local_type": "TIMESTAMP(9)", + "name": "created_at_utc", + "warehouse_type": "TIMESTAMP_NTZ(9)" + }, + { + "local_type": "TEXT(16777216)", + "name": "solved_at", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "DECIMAL(2, 0)", + "name": "minutes_to_first_response", + "warehouse_type": "NUMBER(2,0)" + }, + { + "local_type": "DECIMAL(3, 0)", + "name": "business_minutes_to_response", + "warehouse_type": "NUMBER(3,0)" + }, + { + "local_type": "BOOLEAN", + "name": "is_first_response_breached", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "BOOLEAN", + "name": "is_resolution_breached", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "TEXT(16777216)", + "name": "comment_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "handoff_count", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "sentiment_label", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "ticket_number_for_customer", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "is_repeat_contact_7d", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "TEXT(16777216)", + "name": "csat_score", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "source_loaded_at", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/fct_support_tickets.jsonl", + "kind": "ref", + "logical_name": "fct_support_tickets", + "row_count": 5 + }, + { + "bytes": 640, + "columns": [ + { + "local_type": "TEXT(3)", + "name": "call_id", + "warehouse_type": "VARCHAR(3)" + }, + { + "local_type": "TEXT(2)", + "name": "ticket_id", + "warehouse_type": "VARCHAR(2)" + }, + { + "local_type": "TEXT(16777216)", + "name": "agent_id", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "first_menu_choice", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "final_menu_node", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "menu_steps", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "repeated_menu_visits", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "BOOLEAN", + "name": "abandoned_in_ivr", + "warehouse_type": "BOOLEAN" + }, + { + "local_type": "TEXT(16777216)", + "name": "caller_region", + "warehouse_type": "VARCHAR(16777216)" + }, + { + "local_type": "TEXT(16777216)", + "name": "duration_bucket", + "warehouse_type": "VARCHAR(16777216)" + } + ], + "file": "refs/int_call_ivr_paths.jsonl", + "kind": "ref", + "logical_name": "int_call_ivr_paths", + "row_count": 3 + } + ], + "scenario_name": "support_sla__weekly_breaches_and_monthly_reasons", + "sqlbuild_version": "0.175.0", + "total_bytes": 3017, + "total_rows": 8, + "version": 1 +} diff --git a/benchmarks/duckshop/project/tests/scenarios/catalogue/category_tree__rolls_sales_up_built_paths.sql b/benchmarks/duckshop/project/tests/scenarios/catalogue/category_tree__rolls_sales_up_built_paths.sql new file mode 100644 index 0000000000..5bf763169f --- /dev/null +++ b/benchmarks/duckshop/project/tests/scenarios/catalogue/category_tree__rolls_sales_up_built_paths.sql @@ -0,0 +1,327 @@ +SCENARIO ( + description "Category paths built with CONNECT BY roll order-line sales up to every ancestor", + tags ["catalogue", "category-tree"], + expected_columns "all", +); + +WITH __source__catalogue__categories AS ( + SELECT + 'C0' AS category_id, + NULL AS parent_category_id, + 'Ducks' AS slug, + 'Ducks' AS display_name, + 1 AS sort_order, + CAST(NULL AS VARIANT) AS translations, + 'I' AS _op, + 1 AS _seq, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'C1', + 'C0', + 'Rubber Ducks', + 'Rubber ducks', + 1, + NULL, + 'I', + 2, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'C2', + 'C1', + 'collectible', + 'Collectible series', + 1, + NULL, + 'I', + 3, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'C3', + 'C0', + 'decoys', + 'Decoys', + 2, + NULL, + 'I', + 4, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'C4', + 'C4', + 'textiles', + 'Textiles', + 2, + NULL, + 'I', + 5, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ +), + +__ref__dim_products AS ( + SELECT + 'DUCK-MAL-01' AS sku, + NULL AS product_id, + NULL AS sku_family, + NULL AS title, + NULL AS variant_label, + NULL AS brand_name, + NULL AS is_private_label, + NULL AS category_level_1, + NULL AS category_slug_path, + NULL AS is_active, + NULL AS launched_on, + NULL AS list_price_usd, + NULL AS margin_pct, + NULL AS primary_supplier_code, + NULL AS supplier_count, + NULL AS review_count, + NULL AS average_rating, + NULL AS wishlisting_customers, + NULL AS source_loaded_at + UNION ALL + SELECT + 'DUCK-PIN-02', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'DECOY-TEAL-1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_product_catalogue AS ( + SELECT + 'DUCK-MAL-01' AS sku, + NULL AS product_id, + NULL AS sku_family, + NULL AS title, + NULL AS variant_label, + NULL AS brand_code, + NULL AS brand_name, + NULL AS is_private_label, + 'C1' AS category_id, + NULL AS category_slug_path, + NULL AS category_level_1, + NULL AS material, + NULL AS weight_kg, + NULL AS primary_image_url, + NULL AS image_count, + NULL AS is_active, + NULL AS launched_on, + NULL AS is_bundle_component + UNION ALL + SELECT + 'DUCK-PIN-02', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'C2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'DECOY-TEAL-1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'C3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__fct_order_lines AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + 'DUCK-MAL-01' AS sku, + NULL AS sku_family, + NULL AS category_level_1, + 2 AS quantity, + NULL AS unit_amount_minor, + NULL AS list_amount_minor, + NULL AS markdown_pct, + NULL AS gross_amount_minor, + NULL AS discount_amount_minor, + 2000 AS net_amount_minor, + NULL AS tax_amount_minor, + NULL AS promo_codes, + NULL AS has_stacked_promotions, + NULL AS line_weight_kg + UNION ALL + SELECT + 'L2', + 'O1', + 'DUCK-PIN-02', + NULL, + NULL, + 3, + NULL, + NULL, + NULL, + NULL, + NULL, + 4500, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'L3', + 'O2', + 'DECOY-TEAL-1', + NULL, + NULL, + 5, + NULL, + NULL, + NULL, + NULL, + NULL, + 10000, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'L4', + 'O3', + 'DUCK-MAL-01', + NULL, + NULL, + 1, + NULL, + NULL, + NULL, + NULL, + NULL, + 1000, + NULL, + NULL, + NULL, + NULL +), + +__expected__rpt_category_tree_sales AS ( + SELECT + 'C0' AS category_id, + 'ducks' AS slug_path, + 1 AS depth, + FALSE AS is_leaf, + 0 AS own_units, + 11 AS subtree_units, + 17500 AS subtree_net_amount_minor, + CAST(NULL AS DECIMAL(9, 4)) AS share_of_parent_units + UNION ALL + SELECT + 'C1' AS category_id, + 'ducks/rubber-ducks' AS slug_path, + 2 AS depth, + FALSE AS is_leaf, + 3 AS own_units, + 6 AS subtree_units, + 7500 AS subtree_net_amount_minor, + 0.5455::DECIMAL(9, 4) AS share_of_parent_units + UNION ALL + SELECT + 'C2' AS category_id, + 'ducks/rubber-ducks/collectible' AS slug_path, + 3 AS depth, + TRUE AS is_leaf, + 3 AS own_units, + 3 AS subtree_units, + 4500 AS subtree_net_amount_minor, + 0.5::DECIMAL(9, 4) AS share_of_parent_units + UNION ALL + SELECT + 'C3' AS category_id, + 'ducks/decoys' AS slug_path, + 2 AS depth, + TRUE AS is_leaf, + 5 AS own_units, + 5 AS subtree_units, + 10000 AS subtree_net_amount_minor, + 0.4545::DECIMAL(9, 4) AS share_of_parent_units + UNION ALL + SELECT + 'C4' AS category_id, + 'textiles' AS slug_path, + 1 AS depth, + TRUE AS is_leaf, + 0 AS own_units, + 0 AS subtree_units, + 0 AS subtree_net_amount_minor, + CAST(NULL AS DECIMAL(9, 4)) AS share_of_parent_units +), + +__assert__paths_start_at_a_root AS ( + SELECT p.category_id + FROM __ref("int_category_paths") AS p + WHERE + p.root_category_id NOT IN ('C0', 'C4') + OR ( + p.depth = 1 + ) <> ( + p.parent_category_id IS NULL + ) +) diff --git a/benchmarks/duckshop/project/tests/scenarios/clickstream/cart_funnel__assigns_furthest_stage_per_session.sql b/benchmarks/duckshop/project/tests/scenarios/clickstream/cart_funnel__assigns_furthest_stage_per_session.sql new file mode 100644 index 0000000000..03439d4ab1 --- /dev/null +++ b/benchmarks/duckshop/project/tests/scenarios/clickstream/cart_funnel__assigns_furthest_stage_per_session.sql @@ -0,0 +1,243 @@ +SCENARIO ( + description "Cart events and checkout steps place each web session at its furthest funnel stage", + tags ["clickstream", "funnel"], + timeout "5m", +); + +WITH __ref__int_web_sessions AS ( + SELECT + 'S1' AS session_id, + 'V1' AS anonymous_id, + NULL AS customer_id, + NULL AS session_day, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS session_start_utc, + '2026-03-10 10:30:00'::TIMESTAMP_NTZ AS session_end_utc, + NULL AS session_start_visitor_local, + NULL AS duration_seconds, + NULL AS event_count, + NULL AS page_view_count, + NULL AS landing_page_path, + NULL AS exit_page_path, + NULL AS referrer_host, + NULL AS landing_page_url, + FALSE AS is_bounce, + TRUE AS has_add_to_cart, + FALSE AS has_order, + -- Not real columns: the fixture checker asks for int_cart_funnel's lambda parameters + -- (i, q, acc) as mock columns. Remove these once that is fixed. + NULL AS acc, + NULL AS i, + NULL AS q + UNION ALL + SELECT + 'S2', + 'V1', + NULL, + NULL, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ, + '2026-03-10 12:20:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + FALSE, + TRUE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'S3', + 'V2', + NULL, + NULL, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ, + '2026-03-10 10:05:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + FALSE, + FALSE, + NULL, + NULL, + NULL +), + +__ref__stg_web_events AS ( + SELECT + 'e1' AS event_id, + '2026-03-10 10:05:00'::TIMESTAMP_NTZ AS received_at, + 'add_to_cart' AS event_type, + '2026-03-10 10:05:00'::TIMESTAMP_NTZ AS event_at_utc, + 'V1' AS anonymous_id, + NULL AS customer_id, + NULL AS user_email, + NULL AS page_url, + NULL AS page_path, + NULL AS referrer_host, + NULL AS user_agent, + NULL AS locale, + 'Europe/Berlin' AS client_timezone, + FALSE AS is_bot, + NULL AS properties, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'sku', + 'duck-1', + 'quantity', + 2, + 'price_minor', + 500 + ), + OBJECT_CONSTRUCT('sku', 'DUCK-2', 'price_minor', 900), + OBJECT_CONSTRUCT('quantity', 1, 'price_minor', 100), + OBJECT_CONSTRUCT('sku', 'duck-3', 'quantity', 0) + )::ARRAY AS items, + -- See the note on int_web_sessions: i is a lambda parameter, not a column. + NULL AS i + UNION ALL + SELECT + 'e2', + '2026-03-10 10:10:00'::TIMESTAMP_NTZ, + 'remove_from_cart', + '2026-03-10 10:10:00'::TIMESTAMP_NTZ, + 'V1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'Europe/Berlin', + FALSE, + NULL, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT('sku', 'duck-1', 'quantity', 1) + )::ARRAY, + NULL + UNION ALL + SELECT + 'e3', + '2026-03-10 10:20:00'::TIMESTAMP_NTZ, + 'add_to_cart', + '2026-03-10 10:20:00'::TIMESTAMP_NTZ, + 'V1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'Europe/Berlin', + FALSE, + NULL, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'sku', + 'DUCK-1', + 'quantity', + 1, + 'price_minor', + 500 + ) + )::ARRAY, + NULL + UNION ALL + SELECT + 'e4', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ, + 'add_to_cart', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ, + 'V1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'Europe/Berlin', + FALSE, + NULL, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT('sku', 'DUCK-9', 'price_minor', 9900) + )::ARRAY, + NULL +), + +__ref__stg_web_checkout_steps AS ( + SELECT + 'V1:1' AS checkout_step_key, + 'V1' AS anonymous_id, + 1 AS step_number, + NULL AS step_name, + '2026-03-10 10:25:00'::TIMESTAMP_NTZ AS first_reached_at_utc, + NULL AS payment_option + UNION ALL + SELECT + 'V1:2', + 'V1', + 2, + NULL, + '2026-03-10 10:26:00'::TIMESTAMP_NTZ, + NULL + UNION ALL + SELECT + 'V1:3', + 'V1', + 3, + NULL, + '2026-03-10 12:10:00'::TIMESTAMP_NTZ, + NULL +), + +__expected__int_cart_funnel AS ( + SELECT + 'S1' AS session_id, + 1 AS removed_sku_count, + 3 AS net_units, + 900 AS max_added_unit_minor, + TRUE AS reached_checkout, + FALSE AS reached_payment, + FALSE AS converted, + 'checkout' AS funnel_stage + UNION ALL + SELECT + 'S2' AS session_id, + 0 AS removed_sku_count, + 0 AS net_units, + CAST(NULL AS INT) AS max_added_unit_minor, + TRUE AS reached_checkout, + TRUE AS reached_payment, + TRUE AS converted, + 'order' AS funnel_stage + UNION ALL + SELECT + 'S3' AS session_id, + 0 AS removed_sku_count, + 0 AS net_units, + CAST(NULL AS INT) AS max_added_unit_minor, + FALSE AS reached_checkout, + FALSE AS reached_payment, + FALSE AS converted, + 'browse' AS funnel_stage +), + +__assert__added_skus_are_upper_cased_and_distinct AS ( + SELECT f.session_id + FROM __ref("int_cart_funnel") AS f + WHERE ARRAY_TO_STRING(f.added_skus, ',') <> IFF(f.session_id = 'S1', 'DUCK-1,DUCK-2', '') +) diff --git a/benchmarks/duckshop/project/tests/scenarios/cross_area/campaign_roas__attributes_orders_to_channel_spend.sql b/benchmarks/duckshop/project/tests/scenarios/cross_area/campaign_roas__attributes_orders_to_channel_spend.sql new file mode 100644 index 0000000000..cdc95a2ac1 --- /dev/null +++ b/benchmarks/duckshop/project/tests/scenarios/cross_area/campaign_roas__attributes_orders_to_channel_spend.sql @@ -0,0 +1,300 @@ +SCENARIO ( + description """ + Campaign spend and UTM-tagged orders combine into monthly return on ad spend per channel + """, + tags ["cross-area", "growth", "orders"], +); + +WITH __ref__stg_campaigns AS ( + SELECT + 'CMP1' AS campaign_id, + 'Spring Splash' AS campaign_name, + 'social' AS channel, + NULL AS objective, + '2026-03-01'::DATE AS starts_on, + NULL AS ends_on, + NULL AS budget_minor, + NULL AS currency_code, + NULL AS target_regions, + NULL AS target_age_min, + NULL AS target_age_max, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'CMP2', + 'Pond Week', + 'email', + NULL, + '2026-03-01'::DATE, + '2026-04-30'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_campaign_performance AS ( + SELECT + 'CMP1' AS campaign_id, + NULL AS campaign_name, + NULL AS channel, + NULL AS objective, + NULL AS starts_on, + NULL AS ends_on, + 500.0::DECIMAL(18, 2) AS budget_usd, + 170.0::DECIMAL(18, 2) AS spend_usd, + NULL AS budget_used_pct, + NULL AS paid_clicks, + NULL AS email_clicks, + NULL AS attributed_sessions, + NULL AS attributed_orders, + NULL AS attributed_revenue_usd, + NULL AS return_on_ad_spend, + NULL AS target_region_count, + NULL AS unsupported_target_regions, + NULL AS spend_rank_in_channel +), + +__ref__int_campaign_spend_usd AS ( + SELECT + 'SP1' AS spend_id, + 'CMP1' AS campaign_id, + NULL AS campaign_name, + 'social' AS channel, + NULL AS platform_account_id, + '2026-03-01'::DATE AS spend_date, + NULL AS currency_code, + NULL AS spend_amount, + 100.0::DECIMAL(18, 2) AS spend_usd, + NULL AS impressions, + 50 AS clicks, + NULL AS conversions, + NULL AS click_through_rate, + NULL AS cost_per_click_usd, + NULL AS metric_count, + PARSE_JSON('[]')::ARRAY AS unknown_metrics, + NULL AS is_in_flight + UNION ALL + SELECT + 'SP2', + 'CMP1', + NULL, + 'social', + NULL, + '2026-03-20'::DATE, + NULL, + NULL, + 50.0::DECIMAL(18, 2), + NULL, + 10, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('[]')::ARRAY, + NULL + UNION ALL + SELECT + 'SP3', + 'CMP2', + NULL, + 'email', + NULL, + '2026-03-05'::DATE, + NULL, + NULL, + 0.0::DECIMAL(18, 2), + NULL, + 5, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('[]')::ARRAY, + NULL + UNION ALL + SELECT + 'SP4', + 'CMP1', + NULL, + 'social', + NULL, + '2026-04-01'::DATE, + NULL, + NULL, + 20.0::DECIMAL(18, 2), + NULL, + 1, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('[]')::ARRAY, + NULL +), + +__ref__fct_orders AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + 'EU' AS region_code, + NULL AS placed_at, + '2026-03-10'::DATE AS placed_date_local, + 'DELIVERED' AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS currency_code, + NULL AS amount, + 300.0::NUMBER(18, 2) AS amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + 'cmp1' AS utm_campaign, + NULL AS basket_size, + NULL AS order_number, + NULL AS is_first_order, + NULL AS first_paid_at, + NULL AS first_delivered_at, + NULL AS hours_paid_to_shipped, + NULL AS captured_minor, + NULL AS refunded_minor, + NULL AS net_collected_minor, + NULL AS is_fully_refunded + UNION ALL + SELECT + 'O2', + 'C1', + 'EU', + NULL, + '2026-04-02'::DATE, + 'DELIVERED', + NULL, + NULL, + NULL, + NULL, + 10.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'cmp1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + 'C1', + 'EU', + NULL, + '2026-03-11'::DATE, + 'DELIVERED', + NULL, + NULL, + NULL, + NULL, + 99.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'CMP1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O4', + 'C1', + 'EU', + NULL, + '2026-03-12'::DATE, + 'DELIVERED', + NULL, + NULL, + NULL, + NULL, + 40.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'cmp2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__rpt_campaign_roas_monthly AS ( + SELECT + '2026-03:social' AS roas_key, + 150.0::DECIMAL(18, 2) AS spend_usd, + 60 AS clicks, + 1 AS attributed_orders, + 300.0::DECIMAL(18, 2) AS attributed_revenue_usd, + 2.0::DECIMAL(12, 4) AS return_on_ad_spend + UNION ALL + SELECT + '2026-03:email' AS roas_key, + 0.0::DECIMAL(18, 2) AS spend_usd, + 5 AS clicks, + 1 AS attributed_orders, + 40.0::DECIMAL(18, 2) AS attributed_revenue_usd, + CAST(NULL AS DECIMAL(12, 4)) AS return_on_ad_spend + UNION ALL + SELECT + '2026-04:social' AS roas_key, + 20.0::DECIMAL(18, 2) AS spend_usd, + 1 AS clicks, + 1 AS attributed_orders, + 10.0::DECIMAL(18, 2) AS attributed_revenue_usd, + 0.5::DECIMAL(12, 4) AS return_on_ad_spend +), + +__assert__spend_rows_reach_the_fact AS ( + SELECT COUNT(s.spend_id) AS spend_rows + FROM __ref("fct_campaign_spend") AS s + HAVING COUNT(s.spend_id) <> 4 +) diff --git a/benchmarks/duckshop/project/tests/scenarios/cross_area/region_scorecard__combines_sales_and_recognised_revenue.sql b/benchmarks/duckshop/project/tests/scenarios/cross_area/region_scorecard__combines_sales_and_recognised_revenue.sql new file mode 100644 index 0000000000..c41ccb0e8f --- /dev/null +++ b/benchmarks/duckshop/project/tests/scenarios/cross_area/region_scorecard__combines_sales_and_recognised_revenue.sql @@ -0,0 +1,252 @@ +SCENARIO ( + description "Order sales and recognised revenue combine into one monthly scorecard per region", + tags ["cross-area", "orders", "finance", "executive"], +); + +WITH __ref__fct_orders AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + 'EU' AS region_code, + NULL AS placed_at, + '2026-03-10'::DATE AS placed_date_local, + 'DELIVERED' AS order_status, + NULL AS sales_channel, + 1 AS units, + NULL AS currency_code, + NULL AS amount, + 60.0::NUMBER(18, 2) AS amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS gross_amount_usd, + 0.0::NUMBER(18, 2) AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size, + NULL AS order_number, + TRUE AS is_first_order, + NULL AS first_paid_at, + NULL AS first_delivered_at, + NULL AS hours_paid_to_shipped, + NULL AS captured_minor, + NULL AS refunded_minor, + NULL AS net_collected_minor, + NULL AS is_fully_refunded + UNION ALL + SELECT + 'O2', + 'C2', + 'EU', + NULL, + '2026-03-10'::DATE, + 'DELIVERED', + NULL, + 1, + NULL, + NULL, + 40.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + 0.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + 'C2', + 'EU', + NULL, + '2026-03-20'::DATE, + 'DELIVERED', + NULL, + 1, + NULL, + NULL, + 50.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + 0.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O4', + 'C3', + 'US', + NULL, + '2026-03-05'::DATE, + 'DELIVERED', + NULL, + 1, + NULL, + NULL, + 200.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + 0.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_revenue_recognition_daily AS ( + SELECT + '2026-03-05:EU:product' AS recognition_key, + '2026-03-05'::DATE AS recognition_date, + 'EU' AS region_code, + 'product' AS revenue_stream, + 100.0::DECIMAL(18, 2) AS revenue_usd, + NULL AS document_count, + NULL AS month_to_date_usd, + NULL AS trailing_7d_usd, + NULL AS share_of_day + UNION ALL + SELECT + '2026-03-20:EU:product', + '2026-03-20'::DATE, + 'EU', + 'product', + 50.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + '2026-03-21:EU:refund', + '2026-03-21'::DATE, + 'EU', + 'refund', + -15.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + '2026-03-10:EU:subscription', + '2026-03-10'::DATE, + 'EU', + 'subscription', + 10.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL +), + +__expected__rpt_revenue_recognition_monthly AS ( + SELECT + '2026-03:EU'::TEXT AS recognition_month_key, + 150.0::DECIMAL(18, 2) AS product_usd, + -15.0::DECIMAL(18, 2) AS refund_usd, + 10.0::DECIMAL(18, 2) AS subscription_usd, + 145.0::DECIMAL(18, 2) AS net_revenue_usd, + 0.1::DECIMAL(9, 4) AS refund_ratio +), + +__expected__rpt_region_scorecard_monthly AS ( + SELECT + '2026-03:EU:orders'::TEXT AS scorecard_key, + 3.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:orders'::TEXT AS scorecard_key, + 1.0::DECIMAL(18, 4) AS measure_value, + 2 AS rank_among_regions + UNION ALL + SELECT + '2026-03:EU:revenue_usd'::TEXT AS scorecard_key, + 150.0::DECIMAL(18, 4) AS measure_value, + 2 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:revenue_usd'::TEXT AS scorecard_key, + 200.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:EU:first_orders'::TEXT AS scorecard_key, + 1.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:first_orders'::TEXT AS scorecard_key, + 1.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:EU:average_order_usd'::TEXT AS scorecard_key, + 50.0::DECIMAL(18, 4) AS measure_value, + 2 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:average_order_usd'::TEXT AS scorecard_key, + 200.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:EU:refund_ratio'::TEXT AS scorecard_key, + 0.1::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:refund_ratio'::TEXT AS scorecard_key, + 0.0::DECIMAL(18, 4) AS measure_value, + 2 AS rank_among_regions + UNION ALL + SELECT + '2026-03:EU:subscription_usd'::TEXT AS scorecard_key, + 10.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:subscription_usd'::TEXT AS scorecard_key, + 0.0::DECIMAL(18, 4) AS measure_value, + 2 AS rank_among_regions +) diff --git a/benchmarks/duckshop/project/tests/scenarios/fulfilment/carrier_performance__weekly_rates_from_shipment_facts.sql b/benchmarks/duckshop/project/tests/scenarios/fulfilment/carrier_performance__weekly_rates_from_shipment_facts.sql new file mode 100644 index 0000000000..b53dab5204 --- /dev/null +++ b/benchmarks/duckshop/project/tests/scenarios/fulfilment/carrier_performance__weekly_rates_from_shipment_facts.sql @@ -0,0 +1,227 @@ +SCENARIO ( + description "Shipment facts feed weekly carrier dispatch, delivery-time and exception measures", + tags ["fulfilment", "carriers"], +); + +WITH __ref__stg_shipments AS ( + SELECT + 'S1' AS shipment_id, + 'O-S1' AS order_id, + 'POND' AS warehouse_code, + 'FLAPWING' AS carrier_code, + 'STANDARD' AS service_level, + NULL AS tracking_number, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS shipped_at, + NULL AS delivered_at, + 1 AS tracking_event_count, + NULL AS tracking_events, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'S2', + 'O-S2', + 'POND', + 'FLAPWING', + 'STANDARD', + NULL, + '2026-03-11 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 1, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'S3', + 'O-S3', + 'POND', + 'FLAPWING', + 'STANDARD', + NULL, + '2026-03-15 23:30:00 -02:00'::TIMESTAMP_TZ, + NULL, + 1, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'S4', + 'O-S4', + 'POND', + 'FLAPWING', + 'STANDARD', + NULL, + NULL, + NULL, + 1, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'S5', + 'O-S5', + 'POND', + 'WINGBEAT', + 'STANDARD', + NULL, + '2026-03-12 08:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 1, + NULL, + NULL, + NULL +), + +__ref__int_fulfilment_sla AS ( + SELECT + 'S1' AS shipment_id, + NULL AS order_id, + NULL AS warehouse_code, + NULL AS carrier_code, + NULL AS placed_at_local, + NULL AS shipped_at_local, + NULL AS calendar_days_to_ship, + NULL AS business_days_to_ship, + NULL AS holidays_in_window, + NULL AS dispatch_business_minutes, + TRUE AS is_dispatch_on_time, + NULL AS delivered_at_utc, + 24.0::DECIMAL(10, 2) AS hours_to_deliver + UNION ALL + SELECT + 'S2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + 48.0::DECIMAL(10, 2) + UNION ALL + SELECT + 'S3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'S5', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + 10.0::DECIMAL(10, 2) +), + +__ref__int_shipment_costs AS ( + SELECT + 'PK1' AS package_id, + 'S1' AS shipment_id, + NULL AS carrier_code, + NULL AS service_level, + 1.5::DECIMAL(10, 3) AS chargeable_weight_kg, + NULL AS weight_basis, + NULL AS band_max_kg, + 600 AS rate_minor, + NULL AS rate_currency_code, + NULL AS has_rate +), + +__ref__int_delivery_exceptions_classified AS ( + SELECT + 'X1' AS exception_id, + 'S2' AS shipment_id, + NULL AS carrier_code, + NULL AS exception_code, + NULL AS responsibility, + NULL AS occurred_at, + NULL AS hours_to_resolve, + NULL AS resolution_type, + 5.5::DECIMAL(18, 2) AS compensation_usd, + NULL AS exception_rank_in_shipment +), + +__expected__fct_shipments AS ( + SELECT + 'S1' AS shipment_id, + 1 AS package_count, + 0 AS exception_count, + 0.0::DECIMAL(18, 2) AS compensation_usd + UNION ALL + SELECT + 'S2' AS shipment_id, + 0 AS package_count, + 1 AS exception_count, + 5.5::DECIMAL(18, 2) AS compensation_usd + UNION ALL + SELECT + 'S3' AS shipment_id, + 0 AS package_count, + 0 AS exception_count, + 0.0::DECIMAL(18, 2) AS compensation_usd + UNION ALL + SELECT + 'S4' AS shipment_id, + 0 AS package_count, + 0 AS exception_count, + 0.0::DECIMAL(18, 2) AS compensation_usd + UNION ALL + SELECT + 'S5' AS shipment_id, + 0 AS package_count, + 0 AS exception_count, + 0.0::DECIMAL(18, 2) AS compensation_usd +), + +__expected__rpt_carrier_performance_weekly AS ( + SELECT + 'FLAPWING:2026-03-09' AS carrier_week_key, + 2 AS shipments, + 0.5::DECIMAL(9, 4) AS on_time_dispatch_rate, + 36.0::DECIMAL(10, 2) AS p50_hours_to_deliver, + 45.6::DECIMAL(10, 2) AS p90_hours_to_deliver, + 0.5::DECIMAL(9, 4) AS exception_rate, + 5.5::DECIMAL(18, 2) AS compensation_usd + UNION ALL + SELECT + 'FLAPWING:2026-03-16' AS carrier_week_key, + 1 AS shipments, + 1.0::DECIMAL(9, 4) AS on_time_dispatch_rate, + CAST(NULL AS DECIMAL(10, 2)) AS p50_hours_to_deliver, + CAST(NULL AS DECIMAL(10, 2)) AS p90_hours_to_deliver, + 0.0::DECIMAL(9, 4) AS exception_rate, + 0.0::DECIMAL(18, 2) AS compensation_usd + UNION ALL + SELECT + 'WINGBEAT:2026-03-09' AS carrier_week_key, + 1 AS shipments, + 1.0::DECIMAL(9, 4) AS on_time_dispatch_rate, + 10.0::DECIMAL(10, 2) AS p50_hours_to_deliver, + 10.0::DECIMAL(10, 2) AS p90_hours_to_deliver, + 0.0::DECIMAL(9, 4) AS exception_rate, + 0.0::DECIMAL(18, 2) AS compensation_usd +) diff --git a/benchmarks/duckshop/project/tests/scenarios/inventory/stock_outs__finds_longest_run_per_week.sql b/benchmarks/duckshop/project/tests/scenarios/inventory/stock_outs__finds_longest_run_per_week.sql new file mode 100644 index 0000000000..690a21ac31 --- /dev/null +++ b/benchmarks/duckshop/project/tests/scenarios/inventory/stock_outs__finds_longest_run_per_week.sql @@ -0,0 +1,123 @@ +SCENARIO ( + description """ + Daily stock positions become weekly stock-out rates with the longest consecutive run + """, + tags ["inventory", "stock-outs"], + expected_columns "all", +); + +WITH __ref__fct_inventory_daily AS ( + SELECT + 'DUCK-1|POND|2026-03-09' AS stock_day_key, + 'DUCK-1' AS sku, + 'POND' AS warehouse_code, + '2026-03-09'::DATE AS calendar_date, + 5 AS on_hand_qty, + NULL AS unit_cost_usd, + NULL AS stock_value_usd, + FALSE AS is_stocked_out, + NULL AS days_since_count, + FALSE AS is_count_overdue + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-10', + 'DUCK-1', + 'POND', + '2026-03-10'::DATE, + 0, + NULL, + NULL, + TRUE, + NULL, + FALSE + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-11', + 'DUCK-1', + 'POND', + '2026-03-11'::DATE, + 0, + NULL, + NULL, + TRUE, + NULL, + FALSE + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-12', + 'DUCK-1', + 'POND', + '2026-03-12'::DATE, + 5, + NULL, + NULL, + FALSE, + NULL, + FALSE + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-13', + 'DUCK-1', + 'POND', + '2026-03-13'::DATE, + 0, + NULL, + NULL, + TRUE, + NULL, + FALSE + UNION ALL + SELECT + 'DUCK-2|POND|2026-03-10', + 'DUCK-2', + 'POND', + '2026-03-10'::DATE, + 5, + NULL, + NULL, + FALSE, + NULL, + FALSE + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-16', + 'DUCK-1', + 'POND', + '2026-03-16'::DATE, + 0, + NULL, + NULL, + TRUE, + NULL, + FALSE +), + +__expected__rpt_stock_outs_weekly AS ( + SELECT + 'POND:2026-03-09' AS stock_out_key, + '2026-03-09'::DATE AS week_start, + 'POND' AS warehouse_code, + 6 AS sku_days, + 3 AS stocked_out_sku_days, + 0.5::DECIMAL(9, 4) AS stock_out_rate, + 1 AS skus_affected, + 2 AS longest_run_days + UNION ALL + SELECT + 'POND:2026-03-16' AS stock_out_key, + '2026-03-16'::DATE AS week_start, + 'POND' AS warehouse_code, + 1 AS sku_days, + 1 AS stocked_out_sku_days, + 1.0::DECIMAL(9, 4) AS stock_out_rate, + 1 AS skus_affected, + 1 AS longest_run_days +), + +__assert__longest_run_fits_in_stocked_out_days AS ( + SELECT w.stock_out_key + FROM __ref("rpt_stock_outs_weekly") AS w + WHERE + w.longest_run_days > w.stocked_out_sku_days + OR w.stock_out_rate NOT BETWEEN 0 AND 1 +) diff --git a/benchmarks/duckshop/project/tests/scenarios/orders/payments__fully_refunded_orders_collect_nothing.sql b/benchmarks/duckshop/project/tests/scenarios/orders/payments__fully_refunded_orders_collect_nothing.sql new file mode 100644 index 0000000000..119ec11684 --- /dev/null +++ b/benchmarks/duckshop/project/tests/scenarios/orders/payments__fully_refunded_orders_collect_nothing.sql @@ -0,0 +1,316 @@ +SCENARIO ( + description """ + Captures, refunds, lost chargebacks and gift card tender net into the collected amount of each + order + """, + tags ["orders", "payments"], +); + +WITH __ref__int_orders_usd AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + 'EU' AS region_code, + 'EUR' AS currency_code, + NULL AS placed_at, + NULL AS placed_date_local, + 'DELIVERED' AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS amount, + NULL AS amount_usd, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS rate_age_days, + NULL AS is_rate_stale, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size + UNION ALL + SELECT + 'O2', + 'C2', + 'EU', + 'EUR', + NULL, + NULL, + 'SHIPPED', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + 'C3', + 'US', + 'USD', + NULL, + NULL, + 'PENDING', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_order_status_timeline AS ( + SELECT + 'O1' AS order_id, + NULL AS transition_count, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS first_paid_at, + NULL AS first_shipped_at, + NULL AS first_delivered_at, + NULL AS cancelled_at, + NULL AS minutes_to_pay, + NULL AS hours_paid_to_shipped, + NULL AS status_path, + NULL AS has_regression, + NULL AS last_status, + NULL AS longest_status +), + +__ref__int_customer_order_sequence AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + NULL AS placed_at, + NULL AS placed_date_local, + 1 AS order_number, + TRUE AS is_first_order, + NULL AS days_since_previous_order, + NULL AS amount_usd, + NULL AS lifetime_revenue_usd, + NULL AS rolling_90d_revenue_usd, + NULL AS previous_sales_channel, + NULL AS is_channel_switch, + NULL AS cohort_month +), + +__ref__int_payment_attempts AS ( + SELECT + 'P0' AS payment_id, + 'O1' AS order_id, + 1 AS attempt_number, + NULL AS attempt_count, + FALSE AS is_final_attempt, + 'card' AS payment_method, + 2000 AS amount_minor, + 'EUR' AS currency_code, + NULL AS authorised_at, + NULL AS seconds_since_previous_attempt, + NULL AS gateway_status, + FALSE AS is_successful, + NULL AS decline_family, + NULL AS gateway_error_code, + NULL AS risk_score, + NULL AS risk_band, + NULL AS three_ds_outcome, + NULL AS card_bin_valid, + NULL AS card_scheme, + NULL AS had_chargeback + UNION ALL + SELECT + 'P1', + 'O1', + 2, + NULL, + TRUE, + 'card', + 2000, + 'EUR', + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'P2', + 'O2', + 1, + NULL, + TRUE, + 'phone_wallet', + 1500, + 'EUR', + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_refunds AS ( + SELECT + 'R1' AS refund_id, + 'P1' AS payment_id, + 2000 AS amount_minor, + NULL AS reason_text, + 'return' AS reason_category, + NULL AS refunded_at, + NULL AS initiated_by, + NULL AS refunded_line_count, + NULL AS refunded_items, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_chargebacks AS ( + SELECT + 'CB1' AS chargeback_id, + 'P2' AS payment_id, + 600 AS amount_minor, + NULL AS currency_code, + NULL AS reason_code, + NULL AS reason_family, + NULL AS opened_at, + NULL AS closed_at, + 'lost' AS outcome, + NULL AS evidence_document_count, + NULL AS evidence, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'CB2', + 'P2', + 100, + NULL, + NULL, + NULL, + NULL, + NULL, + 'won', + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_gift_card_transactions AS ( + SELECT + 'G1' AS transaction_id, + 'GC1' AS gift_card_id, + 'O2' AS order_id, + 'REDEEM' AS transaction_type, + -500 AS signed_amount_minor, + NULL AS occurred_at, + NULL AS channel, + NULL AS terminal_id, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'G2', + 'GC1', + 'O2', + 'ISSUE', + 1000, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_order_payments_rollup AS ( + SELECT + 'O1' AS order_id, + 2000 AS card_captured_minor, + 0 AS wallet_captured_minor, + 2000 AS captured_minor, + 2000 AS refunded_minor, + 0 AS chargeback_minor, + 0 AS gift_card_tender_minor, + 0 AS net_collected_minor, + 1 AS failed_attempts + UNION ALL + SELECT + 'O2' AS order_id, + 0 AS card_captured_minor, + 1500 AS wallet_captured_minor, + 1500 AS captured_minor, + 0 AS refunded_minor, + 600 AS chargeback_minor, + 500 AS gift_card_tender_minor, + 1400 AS net_collected_minor, + 0 AS failed_attempts +), + +__expected__fct_orders AS ( + SELECT + 'O1' AS order_id, + 1 AS order_number, + 2000 AS captured_minor, + 2000 AS refunded_minor, + 0 AS net_collected_minor, + TRUE AS is_fully_refunded + UNION ALL + SELECT + 'O2' AS order_id, + CAST(NULL AS INT) AS order_number, + 1500 AS captured_minor, + 0 AS refunded_minor, + 1400 AS net_collected_minor, + FALSE AS is_fully_refunded + UNION ALL + SELECT + 'O3' AS order_id, + CAST(NULL AS INT) AS order_number, + 0 AS captured_minor, + 0 AS refunded_minor, + 0 AS net_collected_minor, + FALSE AS is_fully_refunded +), + +__assert__fully_refunded_orders_collect_nothing AS ( + SELECT o.order_id + FROM __ref("fct_orders") AS o + WHERE + o.is_fully_refunded + AND o.net_collected_minor > 0 +) diff --git a/benchmarks/duckshop/project/tests/scenarios/support/support_sla__weekly_breaches_and_monthly_reasons.sql b/benchmarks/duckshop/project/tests/scenarios/support/support_sla__weekly_breaches_and_monthly_reasons.sql new file mode 100644 index 0000000000..d8a38ce4fd --- /dev/null +++ b/benchmarks/duckshop/project/tests/scenarios/support/support_sla__weekly_breaches_and_monthly_reasons.sql @@ -0,0 +1,212 @@ +SCENARIO ( + description """ + Ticket facts feed weekly SLA breach rates and monthly contact-reason volumes with IVR calls + """, + tags ["support", "sla"], +); + +WITH weekly_ticket_total AS ( + SELECT SUM(s.tickets) AS tickets + FROM __ref("rpt_support_sla_weekly") AS s +), + +monthly_ticket_total AS ( + SELECT SUM(r.tickets) AS tickets + FROM __ref("rpt_contact_reasons_monthly") AS r +), + +__ref__fct_support_tickets AS ( + SELECT + 'T1' AS ticket_id, + NULL AS customer_id, + 'email' AS channel, + 1 AS priority, + NULL AS ticket_status, + 'damaged_item' AS contact_reason, + '2026-03-09 10:00:00'::TIMESTAMPNTZ AS created_at_utc, + NULL AS solved_at, + 30 AS minutes_to_first_response, + 20 AS business_minutes_to_response, + TRUE AS is_first_response_breached, + FALSE AS is_resolution_breached, + NULL AS comment_count, + NULL AS handoff_count, + NULL AS sentiment_label, + NULL AS ticket_number_for_customer, + FALSE AS is_repeat_contact_7d, + NULL AS csat_score, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + NULL, + 'email', + 1, + NULL, + 'damaged_item', + '2026-03-10 10:00:00'::TIMESTAMPNTZ, + NULL, + 50, + 40, + FALSE, + FALSE, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'T3', + NULL, + 'email', + 1, + NULL, + NULL, + '2026-03-11 10:00:00'::TIMESTAMPNTZ, + NULL, + 10, + 100, + FALSE, + TRUE, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'T4', + NULL, + 'chat', + NULL, + NULL, + 'where_is_my_order', + '2026-03-12 10:00:00'::TIMESTAMPNTZ, + NULL, + 5, + NULL, + FALSE, + FALSE, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'T5', + NULL, + 'phone', + 1, + NULL, + 'damaged_item', + NULL, + NULL, + NULL, + NULL, + TRUE, + FALSE, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL +), + +__ref__int_call_ivr_paths AS ( + SELECT + 'CL1' AS call_id, + 'T2' AS ticket_id, + NULL AS agent_id, + NULL AS first_menu_choice, + NULL AS final_menu_node, + NULL AS menu_steps, + NULL AS repeated_menu_visits, + FALSE AS abandoned_in_ivr, + NULL AS caller_region, + NULL AS duration_bucket + UNION ALL + SELECT + 'CL2', + 'T2', + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'CL3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL +), + +__expected__rpt_support_sla_weekly AS ( + SELECT + '2026-03-09:email:1' AS sla_week_key, + 3 AS tickets, + 0.3333::DECIMAL(9, 4) AS first_response_breach_rate, + 0.3333::DECIMAL(9, 4) AS resolution_breach_rate, + 30.0::DECIMAL(10, 2) AS p50_minutes_to_first_response, + 88.0::DECIMAL(10, 2) AS p90_business_minutes_to_response + UNION ALL + SELECT + '2026-03-09:chat:?' AS sla_week_key, + 1 AS tickets, + 0.0::DECIMAL(9, 4) AS first_response_breach_rate, + 0.0::DECIMAL(9, 4) AS resolution_breach_rate, + 5.0::DECIMAL(10, 2) AS p50_minutes_to_first_response, + CAST(NULL AS DECIMAL(10, 2)) AS p90_business_minutes_to_response +), + +__expected__rpt_contact_reasons_monthly AS ( + SELECT + '2026-03:damaged_item' AS reason_month_key, + 2 AS tickets, + 1 AS repeat_contacts, + 0.5::DECIMAL(9, 4) AS repeat_rate, + 2 AS calls, + 1 AS abandoned_in_ivr + UNION ALL + SELECT + '2026-03:unspecified' AS reason_month_key, + 1 AS tickets, + 0 AS repeat_contacts, + 0.0::DECIMAL(9, 4) AS repeat_rate, + 0 AS calls, + 0 AS abandoned_in_ivr + UNION ALL + SELECT + '2026-03:where_is_my_order' AS reason_month_key, + 1 AS tickets, + 0 AS repeat_contacts, + 0.0::DECIMAL(9, 4) AS repeat_rate, + 0 AS calls, + 0 AS abandoned_in_ivr +), + +__assert__weekly_and_monthly_ticket_totals_agree AS ( + SELECT w.tickets + FROM weekly_ticket_total AS w + CROSS JOIN monthly_ticket_total AS m + WHERE w.tickets IS DISTINCT FROM m.tickets +) diff --git a/benchmarks/duckshop/project/tests/test_rule_staging_dedupe.py b/benchmarks/duckshop/project/tests/test_rule_staging_dedupe.py new file mode 100644 index 0000000000..d3e0dc63fb --- /dev/null +++ b/benchmarks/duckshop/project/tests/test_rule_staging_dedupe.py @@ -0,0 +1,81 @@ +from rules.staging_dedupe import staging_uses_cdc_latest + +from sqlbuild.rules import RuleFile +from sqlbuild.rules.testing import RuleCase, evaluate_rule + +_SOURCES = RuleFile( + path="sources/shop.yml", + source="""sources: + - name: shop__orders + description: Orders change log + columns: + - name: order_id + type: VARCHAR + - name: _op + type: VARCHAR + - name: shop__page_views + description: Append-only page views + columns: + - name: event_id + type: VARCHAR +""", +) + +_RAW_CHANGE_LOG_READ = ( + 'MODEL (description "Orders");\n' + "WITH src AS (\n" + ' SELECT * FROM __source("shop__orders")\n' + "),\n" + "final AS (\n" + " SELECT s.order_id FROM src AS s\n" + ")\n" + "SELECT * FROM final\n" +) + + +def test_given_change_log_read_without_cdc_latest_when_evaluating_then_reports_it() -> None: + result = evaluate_rule( + rule=staging_uses_cdc_latest, + test_case=RuleCase( + description="staging model reads a change log directly", + source=_RAW_CHANGE_LOG_READ, + path="models/staging/shop/stg_orders.sql", + files=(_SOURCES,), + expected_finding_count=1, + ), + ) + + assert result.finding_count == 1 + assert result.findings[0].line == 3 + + +def test_given_staging_model_reading_append_only_source_when_evaluating_then_passes() -> None: + result = evaluate_rule( + rule=staging_uses_cdc_latest, + test_case=RuleCase( + description="append-only sources are deduplicated another way", + source=_RAW_CHANGE_LOG_READ.replace("shop__orders", "shop__page_views").replace( + "order_id", "event_id" + ), + path="models/staging/shop/stg_page_views.sql", + files=(_SOURCES,), + expected_finding_count=0, + ), + ) + + assert result.finding_count == 0 + + +def test_given_change_log_read_outside_staging_when_evaluating_then_passes() -> None: + result = evaluate_rule( + rule=staging_uses_cdc_latest, + test_case=RuleCase( + description="snapshots and other layers may read the full change log", + source=_RAW_CHANGE_LOG_READ, + path="models/snapshots/snap_orders.sql", + files=(_SOURCES,), + expected_finding_count=0, + ), + ) + + assert result.finding_count == 0 diff --git a/benchmarks/duckshop/project/tests/unit/functions/python/test_luhn_valid__checks_card_number_checksums.sql b/benchmarks/duckshop/project/tests/unit/functions/python/test_luhn_valid__checks_card_number_checksums.sql new file mode 100644 index 0000000000..8a333a58f3 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/functions/python/test_luhn_valid__checks_card_number_checksums.sql @@ -0,0 +1,31 @@ +TEST (mode udf, name "luhn_valid__checks_card_number_checksums"); + +WITH input_values AS ( + SELECT + v.digits + FROM VALUES + ('79927398713'), + ('79927398710'), + ('4629173058461920'), + ('4000-0566'), + ('') AS v(digits) +), + +__udf_actual__ AS ( + SELECT + i.digits, + __udf("luhn_valid")(i.digits) AS is_valid + FROM input_values AS i +), + +__udf_expected__ AS ( + SELECT + v.digits::VARCHAR AS digits, + v.is_valid::BOOLEAN AS is_valid + FROM VALUES + ('79927398713', TRUE), + ('79927398710', FALSE), + ('4629173058461920', TRUE), + ('4000-0566', NULL), + ('', NULL) AS v(digits, is_valid) +) diff --git a/benchmarks/duckshop/project/tests/unit/functions/sql/test_table_fn__customer_orders__returns_nothing_for_unknown_customers.sql b/benchmarks/duckshop/project/tests/unit/functions/sql/test_table_fn__customer_orders__returns_nothing_for_unknown_customers.sql new file mode 100644 index 0000000000..acf1ab5310 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/functions/sql/test_table_fn__customer_orders__returns_nothing_for_unknown_customers.sql @@ -0,0 +1,15 @@ +TEST (mode table_fn, name "table_fn__customer_orders__returns_nothing_for_unknown_customers"); + +WITH __table_fn_actual__ AS ( + SELECT + o.order_id, + o.order_number + FROM __table_fn("table_fn__customer_orders")('DS-NO-SUCH-CUSTOMER') AS o +), + +__table_fn_expected__ AS ( + SELECT + CAST(NULL AS VARCHAR) AS order_id, + CAST(NULL AS INT) AS order_number + WHERE FALSE +) diff --git a/benchmarks/duckshop/project/tests/unit/functions/sql/test_udf__sku_family__extracts_leading_letters.sql b/benchmarks/duckshop/project/tests/unit/functions/sql/test_udf__sku_family__extracts_leading_letters.sql new file mode 100644 index 0000000000..285ce4a509 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/functions/sql/test_udf__sku_family__extracts_leading_letters.sql @@ -0,0 +1,29 @@ +TEST (name "udf__sku_family__extracts_leading_letters"); + +WITH input_values AS ( + SELECT + v.sku + FROM VALUES + ('DUCK-MAL-01'), + ('decoy-teal-1'), + ('123-POND'), + (NULL) AS v(sku) +), + +__udf_actual__ AS ( + SELECT + i.sku, + __udf("udf__sku_family")(i.sku) AS sku_family + FROM input_values AS i +), + +__udf_expected__ AS ( + SELECT + v.sku::VARCHAR AS sku, + v.sku_family::VARCHAR AS sku_family + FROM VALUES + ('DUCK-MAL-01', 'DUCK'), + ('decoy-teal-1', 'DECOY'), + ('123-POND', NULL), + (NULL, NULL) AS v(sku, sku_family) +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_bundle_pricing__prices_bundles_by_pricing_mode.sql b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_bundle_pricing__prices_bundles_by_pricing_mode.sql new file mode 100644 index 0000000000..17317ec44a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_bundle_pricing__prices_bundles_by_pricing_mode.sql @@ -0,0 +1,145 @@ +TEST (name "int_bundle_pricing__prices_bundles_by_pricing_mode"); + +WITH __ref__stg_bundles AS ( + SELECT + 'B1' AS bundle_sku, + NULL AS title, + NULL AS active_from, + NULL AS active_to, + 'percent_off' AS pricing_mode, + NULL AS fixed_amount_minor, + 10::DECIMAL(5, 2) AS percent_off, + NULL AS pricing_currency_code, + NULL AS pricing, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'B2', + NULL, + NULL, + NULL, + 'cheapest_free', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_bundle_components AS ( + SELECT + 'X1' AS component_id, + 'B1' AS bundle_sku, + 'DUCK-1' AS component_sku, + 2 AS quantity, + FALSE AS is_optional, + NULL AS substitute_skus, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'X2', + 'B1', + 'DUCK-2', + 1, + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'X3', + 'B1', + 'DUCK-3', + 1, + TRUE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'X4', + 'B2', + 'DUCK-1', + 1, + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'X5', + 'B2', + 'DUCK-4', + 1, + FALSE, + NULL, + NULL, + NULL +), + +__ref__int_price_history AS ( + SELECT + 'P-DUCK-1' AS price_list_id, + 'DUCK-1' AS sku, + 'EUR' AS currency_code, + 1000 AS list_amount_minor, + NULL AS valid_from, + NULL AS effective_valid_to, + NULL AS previous_amount_minor, + NULL AS change_pct, + NULL AS is_markdown, + NULL AS price_version, + NULL AS days_at_price, + NULL AS lowest_price_30d_minor + UNION ALL + SELECT + 'P-DUCK-2', + 'DUCK-2', + 'EUR', + 500, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'P-DUCK-3', + 'DUCK-3', + 'EUR', + 9999, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_bundle_pricing AS ( + SELECT + 'B1:EUR' AS bundle_price_key, + 2 AS component_count, + 2500 AS components_list_minor, + 2250 AS bundle_price_minor, + 250 AS saving_minor, + 10.0::DECIMAL(9, 4) AS saving_pct, + FALSE AS has_unpriced_component + UNION ALL + SELECT + 'B2:EUR' AS bundle_price_key, + 2 AS component_count, + 1000 AS components_list_minor, + 0 AS bundle_price_minor, + 1000 AS saving_minor, + 100.0::DECIMAL(9, 4) AS saving_pct, + TRUE AS has_unpriced_component +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_price_history__chains_price_versions_and_markdowns.sql b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_price_history__chains_price_versions_and_markdowns.sql new file mode 100644 index 0000000000..191fb7cdd2 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_price_history__chains_price_versions_and_markdowns.sql @@ -0,0 +1,71 @@ +TEST (name "int_price_history__chains_price_versions_and_markdowns"); + +WITH __ref__stg_price_lists AS ( + SELECT + 'PL1' AS price_list_id, + 'DUCK-1' AS sku, + 'EUR' AS currency_code, + 1000 AS list_amount_minor, + '2026-01-01 00:00:00 +00:00'::TIMESTAMPTZ AS valid_from, + NULL AS valid_to, + NULL AS is_open_ended, + NULL AS min_quantity, + NULL AS channel_restrictions, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'PL2', + 'DUCK-1', + 'EUR', + 800, + '2026-02-01 00:00:00 +00:00'::TIMESTAMPTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'PL3', + 'DUCK-1', + 'EUR', + 900, + '2026-02-15 00:00:00 +00:00'::TIMESTAMPTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_price_history AS ( + SELECT + 'PL1' AS price_list_id, + CAST(NULL AS INT) AS previous_amount_minor, + CAST(NULL AS DECIMAL(9, 4)) AS change_pct, + FALSE AS is_markdown, + 1 AS price_version, + 31 AS days_at_price, + 1000 AS lowest_price_30d_minor + UNION ALL + SELECT + 'PL2' AS price_list_id, + 1000 AS previous_amount_minor, + -20.0::DECIMAL(9, 4) AS change_pct, + TRUE AS is_markdown, + 2 AS price_version, + 14 AS days_at_price, + 800 AS lowest_price_30d_minor + UNION ALL + SELECT + 'PL3' AS price_list_id, + 800 AS previous_amount_minor, + 12.5::DECIMAL(9, 4) AS change_pct, + FALSE AS is_markdown, + 3 AS price_version, + CAST(NULL AS INT) AS days_at_price, + 800 AS lowest_price_30d_minor +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_product_catalogue__assembles_sellable_skus.sql b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_product_catalogue__assembles_sellable_skus.sql new file mode 100644 index 0000000000..da4dbf56d2 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_product_catalogue__assembles_sellable_skus.sql @@ -0,0 +1,191 @@ +TEST (name "int_product_catalogue__assembles_sellable_skus"); + +WITH __ref__stg_product_variants AS ( + SELECT + 'DUCK-MAL-08' AS sku, + 'P1' AS product_id, + 'Yellow' AS colour, + '10cm' AS size_label, + NULL AS weight_grams, + 0.3::DECIMAL(10, 3) AS weight_kg, + NULL AS barcode_count, + NULL AS is_fragile, + NULL AS pack_size, + NULL AS barcodes, + NULL AS attributes, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'BED-X-1', + 'P2', + 'White', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_products AS ( + SELECT + 'P1' AS product_id, + 'Mallard Classic Duck' AS title, + 'ducks/decoys' AS category_path, + 'ducks' AS category_level_1, + NULL AS category_level_2, + NULL AS category_leaf, + NULL AS category_depth, + 'B1' AS brand_code, + NULL AS launched_on, + TRUE AS is_active, + NULL AS spec_key_count, + NULL AS specs, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'P2', + 'Bill & Coo Duvet Cover', + 'textiles/bedding', + 'textiles', + NULL, + NULL, + NULL, + 'B9', + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_brands AS ( + SELECT + 'B1' AS brand_code, + 'Mallard & Co.' AS brand_name, + NULL AS country_code, + TRUE AS is_private_label, + NULL AS sustainability_rating, + NULL AS founded_year, + NULL AS brand_metadata, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__int_category_paths AS ( + SELECT + 'C1' AS category_id, + NULL AS parent_category_id, + NULL AS slug, + NULL AS display_name, + NULL AS root_category_id, + NULL AS depth, + 'ducks/decoys' AS slug_path, + NULL AS is_leaf, + NULL AS sibling_rank +), + +__ref__stg_product_media AS ( + SELECT + 'M1' AS media_id, + 'P1' AS product_id, + 'https://cdn.test/2.jpg' AS media_url, + 'image' AS media_type, + 2 AS position, + NULL AS width_px, + NULL AS height_px, + NULL AS aspect_ratio, + NULL AS file_extension, + NULL AS dimensions, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'M2', + 'P1', + 'https://cdn.test/1.jpg', + 'image', + 1, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'M3', + 'P1', + 'https://cdn.test/v.mp4', + 'video', + 0, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_product_specs_flat AS ( + SELECT + 'P1' AS product_id, + 'Vinyl' AS material, + NULL AS colour_family, + NULL AS width_cm, + NULL AS height_cm, + NULL AS capacity_ml, + NULL AS is_dishwasher_safe, + NULL AS spec_value_types, + NULL AS other_specs, + NULL AS numeric_spec_count +), + +__ref__stg_bundle_components AS ( + SELECT + 'X1' AS component_id, + 'BNDL-1' AS bundle_sku, + 'DUCK-MAL-08' AS component_sku, + NULL AS quantity, + NULL AS is_optional, + NULL AS substitute_skus, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_product_catalogue AS ( + SELECT + 'DUCK-MAL-08' AS sku, + 'DUCK' AS sku_family, + 'Yellow / 10cm' AS variant_label, + 'Mallard & Co.' AS brand_name, + TRUE AS is_private_label, + 'C1' AS category_id, + 'ducks/decoys' AS category_slug_path, + 'Vinyl' AS material, + 'https://cdn.test/1.jpg' AS primary_image_url, + 2 AS image_count, + TRUE AS is_bundle_component + UNION ALL + SELECT + 'BED-X-1' AS sku, + 'BED' AS sku_family, + CAST(NULL AS VARCHAR) AS variant_label, + CAST(NULL AS VARCHAR) AS brand_name, + FALSE AS is_private_label, + CAST(NULL AS VARCHAR) AS category_id, + 'textiles/bedding' AS category_slug_path, + CAST(NULL AS VARCHAR) AS material, + CAST(NULL AS VARCHAR) AS primary_image_url, + 0 AS image_count, + FALSE AS is_bundle_component +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_product_margin__converts_supplier_cost_into_list_currency.sql b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_product_margin__converts_supplier_cost_into_list_currency.sql new file mode 100644 index 0000000000..865ea81336 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_product_margin__converts_supplier_cost_into_list_currency.sql @@ -0,0 +1,136 @@ +TEST (name "int_product_margin__converts_supplier_cost_into_list_currency"); + +WITH __ref__int_price_history AS ( + SELECT + 'P-DUCK-1' AS price_list_id, + 'DUCK-1' AS sku, + 'EUR' AS currency_code, + 1000 AS list_amount_minor, + NULL AS valid_from, + NULL AS effective_valid_to, + NULL AS previous_amount_minor, + NULL AS change_pct, + NULL AS is_markdown, + NULL AS price_version, + NULL AS days_at_price, + NULL AS lowest_price_30d_minor + UNION ALL + SELECT + 'P-DUCK-2', + 'DUCK-2', + 'EUR', + 400, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-03-01'::DATE AS rate_date, + 1.1::NUMBER(18, 8) AS rate_to_usd, + 0.9::DECIMAL(18, 8) AS usd_to_rate, + NULL AS next_rate_date +), + +__ref__int_supplier_cost_current AS ( + SELECT + 'DUCK-1' AS sku, + 'S1' AS supplier_code, + NULL AS supplier_name, + NULL AS currency_code, + NULL AS unit_cost_minor, + 5.5::DECIMAL(18, 4) AS unit_cost_usd, + NULL AS effective_from, + NULL AS supplier_count, + NULL AS payment_terms_days, + NULL AS cost_spread_pct + UNION ALL + SELECT + 'DUCK-2', + 'S1', + NULL, + NULL, + NULL, + 5.0::DECIMAL(18, 4), + NULL, + NULL, + NULL, + NULL +), + +__ref__int_product_catalogue AS ( + SELECT + 'DUCK-1' AS sku, + NULL AS product_id, + NULL AS sku_family, + NULL AS title, + NULL AS variant_label, + NULL AS brand_code, + NULL AS brand_name, + NULL AS is_private_label, + NULL AS category_id, + NULL AS category_slug_path, + 'decoys' AS category_level_1, + NULL AS material, + NULL AS weight_kg, + NULL AS primary_image_url, + NULL AS image_count, + NULL AS is_active, + NULL AS launched_on, + NULL AS is_bundle_component + UNION ALL + SELECT + 'DUCK-2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'decoys', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol +), + +__expected__int_product_margin AS ( + SELECT + 'DUCK-1:EUR' AS sku_price_key, + 10.0::DECIMAL(18, 4) AS list_price, + 4.95::DECIMAL(18, 4) AS unit_cost, + 5.05::DECIMAL(18, 4) AS unit_margin, + 50.5::DECIMAL(9, 2) AS margin_pct, + FALSE AS is_below_cost, + 'decoys' AS category_level_1 + UNION ALL + SELECT + 'DUCK-2:EUR' AS sku_price_key, + 4.0::DECIMAL(18, 4) AS list_price, + 4.5::DECIMAL(18, 4) AS unit_cost, + -0.5::DECIMAL(18, 4) AS unit_margin, + -12.5::DECIMAL(9, 2) AS margin_pct, + TRUE AS is_below_cost, + 'decoys' AS category_level_1 +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_product_specs_flat__types_and_normalises_spec_values.sql b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_product_specs_flat__types_and_normalises_spec_values.sql new file mode 100644 index 0000000000..4bbe868427 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_product_specs_flat__types_and_normalises_spec_values.sql @@ -0,0 +1,78 @@ +TEST (name "int_product_specs_flat__types_and_normalises_spec_values"); + +WITH __ref__stg_products AS ( + SELECT + 'P1' AS product_id, + NULL AS title, + NULL AS category_path, + NULL AS category_level_1, + NULL AS category_level_2, + NULL AS category_leaf, + NULL AS category_depth, + NULL AS brand_code, + NULL AS launched_on, + NULL AS is_active, + NULL AS spec_key_count, + OBJECT_CONSTRUCT( + 'Material', + 'cedar', + 'Colour Family', + 'Natural', + 'dishwasher_safe', + 'true', + 'width', + '25 cm', + 'height', + 12.5, + 'finish', + 'oiled' + )::OBJECT AS specs, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'P2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + OBJECT_CONSTRUCT()::OBJECT, + NULL, + NULL +), + +__expected__int_product_specs_flat AS ( + SELECT + 'P1' AS product_id, + 'Cedar' AS material, + 'natural' AS colour_family, + 25.0::DECIMAL(10, 2) AS width_cm, + 12.5::DECIMAL(10, 2) AS height_cm, + CAST(NULL AS INT) AS capacity_ml, + TRUE AS is_dishwasher_safe, + 2 AS numeric_spec_count + UNION ALL + SELECT + 'P2' AS product_id, + CAST(NULL AS VARCHAR) AS material, + CAST(NULL AS VARCHAR) AS colour_family, + CAST(NULL AS DECIMAL(10, 2)) AS width_cm, + CAST(NULL AS DECIMAL(10, 2)) AS height_cm, + CAST(NULL AS INT) AS capacity_ml, + CAST(NULL AS BOOLEAN) AS is_dishwasher_safe, + 0 AS numeric_spec_count +), + +__assert__unmapped_specs_are_kept AS ( + SELECT s.product_id + FROM __ref("int_product_specs_flat") AS s + WHERE + s.product_id = 'P1' + AND s.other_specs:finish::VARCHAR IS DISTINCT FROM 'oiled' +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_supplier_cost_current__picks_cheapest_current_supplier_in_usd.sql b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_supplier_cost_current__picks_cheapest_current_supplier_in_usd.sql new file mode 100644 index 0000000000..f6d55a7684 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/catalogue/test_int_supplier_cost_current__picks_cheapest_current_supplier_in_usd.sql @@ -0,0 +1,98 @@ +TEST (name "int_supplier_cost_current__picks_cheapest_current_supplier_in_usd"); + +WITH __ref__stg_supplier_costs AS ( + SELECT + 'K1' AS cost_id, + 'SUPA' AS supplier_code, + 'DUCK-1' AS sku, + 'EUR' AS currency_code, + 500 AS unit_cost_minor, + '2026-01-01'::DATE AS effective_from, + NULL AS effective_to, + NULL AS volume_break_count, + NULL AS best_break_cost_minor, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'K2', + 'SUPB', + 'DUCK-1', + 'USD', + 600, + '2026-02-01'::DATE, + NULL, + NULL, + 520, + NULL, + NULL + UNION ALL + SELECT + 'K3', + 'SUPC', + 'DUCK-1', + 'USD', + 100, + '2025-01-01'::DATE, + '2025-12-31'::DATE, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2025-12-01'::DATE AS rate_date, + 1.1::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'USD', + '2020-01-01'::DATE, + 1.0::NUMBER(18, 8), + NULL, + NULL +), + +__ref__stg_suppliers AS ( + SELECT + 'SUPB' AS supplier_code, + 'Teal Hollow Decoys' AS supplier_name, + NULL AS country_code, + 45 AS payment_terms_days, + NULL AS early_payment_discount_pct, + NULL AS primary_contact_email, + NULL AS contact_count, + NULL AS payment_terms, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'USD', + NULL, + 2, + NULL +), + +__expected__int_supplier_cost_current AS ( + SELECT + 'DUCK-1' AS sku, + 'SUPB' AS supplier_code, + 'Teal Hollow Decoys' AS supplier_name, + 520 AS unit_cost_minor, + 5.2::DECIMAL(18, 4) AS unit_cost_usd, + 2 AS supplier_count, + 45 AS payment_terms_days, + 5.77::DECIMAL(9, 2) AS cost_spread_pct +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_app_sessions__splits_device_sessions_and_flags_push_starts.sql b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_app_sessions__splits_device_sessions_and_flags_push_starts.sql new file mode 100644 index 0000000000..ad99325935 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_app_sessions__splits_device_sessions_and_flags_push_starts.sql @@ -0,0 +1,113 @@ +TEST (name "int_app_sessions__splits_device_sessions_and_flags_push_starts"); + +WITH __ref__stg_app_events AS ( + SELECT + 'a1' AS event_id, + NULL AS received_at, + 'phone' AS platform, + NULL AS app_version, + 5 AS app_major_version, + 'push_open' AS event_name, + '2026-03-10 08:00:00'::TIMESTAMP_NTZ AS event_at_utc, + 'D1' AS device_id, + NULL AS customer_id, + NULL AS screen_name, + NULL AS attributes + UNION ALL + SELECT + 'a2', + NULL, + 'phone', + NULL, + 5, + 'screen_view', + '2026-03-10 08:05:00'::TIMESTAMP_NTZ, + 'D1', + 'C1', + NULL, + NULL + UNION ALL + SELECT + 'a3', + NULL, + 'phone', + NULL, + 6, + 'screen_view', + '2026-03-10 08:50:00'::TIMESTAMP_NTZ, + 'D1', + NULL, + NULL, + NULL + UNION ALL + SELECT + 'a4', + NULL, + 'phone', + NULL, + 5, + 'screen_view', + '2026-03-10 08:51:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'b1', + NULL, + 'tablet', + NULL, + 5, + 'screen_view', + '2026-03-10 09:00:00'::TIMESTAMP_NTZ, + 'D2', + NULL, + NULL, + NULL + UNION ALL + SELECT + 'b2', + NULL, + 'tablet', + NULL, + 5, + 'push_open', + '2026-03-10 09:10:00'::TIMESTAMP_NTZ, + 'D2', + NULL, + NULL, + NULL +), + +__expected__int_app_sessions AS ( + SELECT + 'D1:20260310080000' AS app_session_id, + 'C1' AS customer_id, + 'phone' AS platform, + 5 AS app_major_version, + 300 AS duration_seconds, + 2 AS event_count, + 1 AS screen_count, + TRUE AS started_from_push + UNION ALL + SELECT + 'D1:20260310085000' AS app_session_id, + CAST(NULL AS VARCHAR) AS customer_id, + 'phone' AS platform, + 6 AS app_major_version, + 0 AS duration_seconds, + 1 AS event_count, + 1 AS screen_count, + FALSE AS started_from_push + UNION ALL + SELECT + 'D2:20260310090000' AS app_session_id, + CAST(NULL AS VARCHAR) AS customer_id, + 'tablet' AS platform, + 5 AS app_major_version, + 600 AS duration_seconds, + 2 AS event_count, + 1 AS screen_count, + FALSE AS started_from_push +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_cross_device_identity__merges_web_visitors_and_app_devices.sql b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_cross_device_identity__merges_web_visitors_and_app_devices.sql new file mode 100644 index 0000000000..bb1c24c01d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_cross_device_identity__merges_web_visitors_and_app_devices.sql @@ -0,0 +1,130 @@ +TEST (name "int_cross_device_identity__merges_web_visitors_and_app_devices"); + +WITH __ref__stg_web_identifies AS ( + SELECT + 'V1:C1' AS identity_link_key, + 'V1' AS anonymous_id, + 'C1' AS customer_id, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ AS first_seen_at_utc, + NULL AS last_seen_at_utc, + NULL AS event_count, + NULL AS identify_methods + UNION ALL + SELECT + 'V2:C1', + 'V2', + 'C1', + '2026-03-05 10:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'V3:C2', + 'V3', + 'C2', + '2026-03-02 10:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL +), + +__ref__stg_app_events AS ( + SELECT + 'a1' AS event_id, + NULL AS received_at, + 'phone' AS platform, + NULL AS app_version, + NULL AS app_major_version, + NULL AS event_name, + '2026-02-28 09:00:00'::TIMESTAMP_NTZ AS event_at_utc, + 'D1' AS device_id, + 'C1' AS customer_id, + NULL AS screen_name, + NULL AS attributes + UNION ALL + SELECT + 'a2', + NULL, + 'phone', + NULL, + NULL, + NULL, + '2026-03-03 09:00:00'::TIMESTAMP_NTZ, + 'D1', + 'C1', + NULL, + NULL + UNION ALL + SELECT + 'a3', + NULL, + 'tablet', + NULL, + NULL, + NULL, + '2026-03-04 09:00:00'::TIMESTAMP_NTZ, + 'D2', + 'C1', + NULL, + NULL + UNION ALL + SELECT + 'a4', + NULL, + 'tablet', + NULL, + NULL, + NULL, + '2026-01-01 09:00:00'::TIMESTAMP_NTZ, + NULL, + 'C1', + NULL, + NULL + UNION ALL + SELECT + 'a5', + NULL, + 'phone', + NULL, + NULL, + NULL, + '2026-01-01 09:00:00'::TIMESTAMP_NTZ, + 'D9', + NULL, + NULL, + NULL +), + +__expected__int_cross_device_identity AS ( + SELECT + 'C1' AS customer_id, + 2 AS web_visitor_count, + 2 AS app_device_count, + TRUE AS is_multi_device, + '2026-02-28 09:00:00'::TIMESTAMPNTZ AS first_seen_at_utc + UNION ALL + SELECT + 'C2' AS customer_id, + 1 AS web_visitor_count, + 0 AS app_device_count, + FALSE AS is_multi_device, + '2026-03-02 10:00:00'::TIMESTAMPNTZ AS first_seen_at_utc +), + +__assert__platforms_are_sorted AS ( + SELECT i.customer_id + FROM __ref("int_cross_device_identity") AS i + WHERE ARRAY_TO_STRING(i.platforms, ',') <> IFF(i.customer_id = 'C1', 'phone,tablet,web', 'web') +), + +__assert__identity_graph_lists_identifiers_per_platform AS ( + SELECT i.customer_id + FROM __ref("int_cross_device_identity") AS i + WHERE + i.customer_id = 'C1' + AND ( + ARRAY_SIZE(i.identity_graph:phone) <> 1 + OR i.identity_graph:web[1]::VARCHAR <> 'V2' + ) +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_email_engagement__separates_machine_opens_and_click_domains.sql b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_email_engagement__separates_machine_opens_and_click_domains.sql new file mode 100644 index 0000000000..93709401db --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_email_engagement__separates_machine_opens_and_click_domains.sql @@ -0,0 +1,188 @@ +TEST (name "int_email_engagement__separates_machine_opens_and_click_domains"); + +WITH __ref__stg_email_events AS ( + SELECT + 'm1' AS email_event_id, + 'M1' AS message_id, + 'sent' AS event_type, + 'a@example.com' AS recipient_email, + 'CMP1' AS campaign_id, + '2026-03-10 09:00:00 +00:00'::TIMESTAMP_TZ AS occurred_at, + NULL AS clicked_url, + NULL AS bounce_type, + NULL AS is_machine_open, + NULL AS details + UNION ALL + SELECT + 'm2', + 'M1', + 'delivered', + 'a@example.com', + 'CMP1', + '2026-03-10 09:00:05 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'm3', + 'M1', + 'open', + 'a@example.com', + 'CMP1', + '2026-03-10 09:00:06 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + TRUE, + NULL + UNION ALL + SELECT + 'm4', + 'M1', + 'open', + 'a@example.com', + 'CMP1', + '2026-03-10 09:20:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + FALSE, + NULL + UNION ALL + SELECT + 'm5', + 'M1', + 'click', + 'a@example.com', + 'CMP1', + '2026-03-10 09:30:00 +00:00'::TIMESTAMP_TZ, + 'https://Shop.Example.com/p/1', + NULL, + NULL, + NULL + UNION ALL + SELECT + 'm6', + 'M1', + 'click', + 'a@example.com', + 'CMP1', + '2026-03-10 09:45:00 +00:00'::TIMESTAMP_TZ, + 'https://blog.example.com/x', + NULL, + NULL, + NULL + UNION ALL + SELECT + 'm7', + 'M1', + 'unsubscribe', + 'a@example.com', + 'CMP1', + '2026-03-10 09:50:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'n1', + 'M2', + 'sent', + 'b@example.com', + 'CMP2', + '2026-02-01 09:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'n2', + 'M2', + 'bounce', + 'b@example.com', + 'CMP2', + '2026-02-01 09:00:01 +00:00'::TIMESTAMP_TZ, + NULL, + 'hard', + NULL, + NULL +), + +__ref__stg_customers AS ( + SELECT + 'C1' AS customer_id, + 'a@example.com' AS email_normalised, + NULL AS phone_normalised, + NULL AS first_name, + NULL AS last_name, + NULL AS region_code, + NULL AS storefront_code, + NULL AS created_at, + NULL AS preferred_language, + NULL AS marketing_opt_in, + NULL AS consent_types, + NULL AS birth_year, + NULL AS preferences, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_campaigns AS ( + SELECT + 'CMP1' AS campaign_id, + NULL AS campaign_name, + 'email' AS channel, + NULL AS objective, + '2026-03-01'::DATE AS starts_on, + NULL AS ends_on, + NULL AS budget_minor, + NULL AS currency_code, + NULL AS target_regions, + NULL AS target_age_min, + NULL AS target_age_max, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_email_engagement AS ( + SELECT + 'M1' AS message_id, + '2026-03-10 09:00:00 +00:00'::TIMESTAMP_TZ AS sent_at, + TRUE AS is_delivered, + CAST(NULL AS VARCHAR) AS bounce_type, + 1 AS human_open_count, + 2 AS click_count, + TRUE AS unsubscribed, + FALSE AS complained, + 30 AS minutes_to_first_click, + 'C1' AS customer_id, + 'email' AS campaign_channel, + TRUE AS is_campaign_in_flight + UNION ALL + SELECT + 'M2' AS message_id, + '2026-02-01 09:00:00 +00:00'::TIMESTAMP_TZ AS sent_at, + FALSE AS is_delivered, + 'hard' AS bounce_type, + 0 AS human_open_count, + 0 AS click_count, + FALSE AS unsubscribed, + FALSE AS complained, + CAST(NULL AS INT) AS minutes_to_first_click, + CAST(NULL AS VARCHAR) AS customer_id, + CAST(NULL AS VARCHAR) AS campaign_channel, + FALSE AS is_campaign_in_flight +), + +__assert__clicked_domains_are_lower_cased_hosts AS ( + SELECT e.message_id + FROM __ref("int_email_engagement") AS e + WHERE + ARRAY_TO_STRING(e.clicked_domains, ',') <> IFF( + e.message_id = 'M1', + 'blog.example.com,shop.example.com', + '' + ) +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_experiment_exposures__counts_sessions_and_conversion_after_assignment.sql b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_experiment_exposures__counts_sessions_and_conversion_after_assignment.sql new file mode 100644 index 0000000000..f721b6bafd --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_experiment_exposures__counts_sessions_and_conversion_after_assignment.sql @@ -0,0 +1,137 @@ +TEST (name "int_experiment_exposures__counts_sessions_and_conversion_after_assignment"); + +WITH __ref__stg_experiment_assignments AS ( + SELECT + 'A1' AS assignment_id, + 'checkout_v2' AS experiment_key, + 'control' AS variant_key, + 'V1' AS anonymous_id, + NULL AS customer_id, + '2026-03-10 10:00:00 +01:00'::TIMESTAMP_TZ AS assigned_at, + 'web' AS assignment_surface, + NULL AS context + UNION ALL + SELECT + 'A2', + 'checkout_v2', + 'treatment', + 'V2', + NULL, + '2026-03-10 09:00:00 +00:00'::TIMESTAMP_TZ, + 'web', + NULL + UNION ALL + SELECT + 'A3', + 'checkout_v2', + 'b', + 'V3', + NULL, + '2026-03-10 09:00:00 +00:00'::TIMESTAMP_TZ, + 'app', + NULL +), + +__ref__int_web_sessions AS ( + SELECT + 'S1' AS session_id, + 'V1' AS anonymous_id, + NULL AS customer_id, + NULL AS session_day, + '2026-03-10 08:00:00'::TIMESTAMP_NTZ AS session_start_utc, + '2026-03-10 08:10:00'::TIMESTAMP_NTZ AS session_end_utc, + NULL AS session_start_visitor_local, + NULL AS duration_seconds, + NULL AS event_count, + NULL AS page_view_count, + NULL AS landing_page_path, + NULL AS exit_page_path, + NULL AS referrer_host, + NULL AS landing_page_url, + NULL AS is_bounce, + FALSE AS has_add_to_cart, + TRUE AS has_order + UNION ALL + SELECT + 'S2', + 'V1', + NULL, + NULL, + '2026-03-10 09:30:00'::TIMESTAMP_NTZ, + '2026-03-10 09:40:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + FALSE + UNION ALL + SELECT + 'S3', + 'V1', + NULL, + NULL, + '2026-03-11 21:00:00'::TIMESTAMP_NTZ, + '2026-03-11 21:10:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + TRUE + UNION ALL + SELECT + 'S4', + 'V2', + NULL, + NULL, + '2026-03-10 09:00:00'::TIMESTAMP_NTZ, + '2026-03-10 09:10:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + TRUE +), + +__expected__int_experiment_exposures AS ( + SELECT + 'A1' AS assignment_id, + TRUE AS is_control, + '2026-03-10 09:00:00'::TIMESTAMPNTZ AS assigned_at_utc, + 2 AS sessions_after, + TRUE AS converted, + 1.5::DECIMAL(10, 2) AS days_to_convert + UNION ALL + SELECT + 'A2' AS assignment_id, + FALSE AS is_control, + '2026-03-10 09:00:00'::TIMESTAMPNTZ AS assigned_at_utc, + 1 AS sessions_after, + TRUE AS converted, + 0.0::DECIMAL(10, 2) AS days_to_convert + UNION ALL + SELECT + 'A3' AS assignment_id, + FALSE AS is_control, + '2026-03-10 09:00:00'::TIMESTAMPNTZ AS assigned_at_utc, + 0 AS sessions_after, + FALSE AS converted, + CAST(NULL AS DECIMAL(10, 2)) AS days_to_convert +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_product_view_affinity__scores_co_views_within_an_hour.sql b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_product_view_affinity__scores_co_views_within_an_hour.sql new file mode 100644 index 0000000000..13c9b95be4 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_product_view_affinity__scores_co_views_within_an_hour.sql @@ -0,0 +1,92 @@ +TEST (name "int_product_view_affinity__scores_co_views_within_an_hour"); + +WITH __ref__stg_web_product_views AS ( + SELECT + 'V1' AS anonymous_id, + NULL AS customer_id, + 'V1:DUCK-1:2026-03-10 10:00:00' AS product_view_key, + 'DUCK-1' AS sku, + '2026-03-10 10:00:00'::TIMESTAMPNTZ AS viewed_minute_utc, + NULL AS list_name, + NULL AS list_position + UNION ALL + SELECT + 'V1', + NULL, + 'V1:DUCK-2:2026-03-10 10:30:00', + 'DUCK-2', + '2026-03-10 10:30:00'::TIMESTAMPNTZ, + NULL, + NULL + UNION ALL + SELECT + 'V1', + NULL, + 'V1:DUCK-3:2026-03-10 11:30:00', + 'DUCK-3', + '2026-03-10 11:30:00'::TIMESTAMPNTZ, + NULL, + NULL + UNION ALL + SELECT + 'V2', + NULL, + 'V2:DUCK-1:2026-03-10 09:00:00', + 'DUCK-1', + '2026-03-10 09:00:00'::TIMESTAMPNTZ, + NULL, + NULL + UNION ALL + SELECT + 'V2', + NULL, + 'V2:DUCK-2:2026-03-10 09:10:00', + 'DUCK-2', + '2026-03-10 09:10:00'::TIMESTAMPNTZ, + NULL, + NULL + UNION ALL + SELECT + 'V2', + NULL, + 'V2:DUCK-3:2026-03-10 09:50:00', + 'DUCK-3', + '2026-03-10 09:50:00'::TIMESTAMPNTZ, + NULL, + NULL + UNION ALL + SELECT + 'V3', + NULL, + 'V3:DUCK-2:2026-03-10 12:00:00', + 'DUCK-2', + '2026-03-10 12:00:00'::TIMESTAMPNTZ, + NULL, + NULL +), + +__expected__int_product_view_affinity AS ( + SELECT + 'DUCK-1>DUCK-2' AS sku_pair_key, + 2 AS co_viewers, + 2 AS sku_viewers, + 1.0::DECIMAL(9, 6) AS co_view_rate, + 1.0::DECIMAL(12, 4) AS lift, + 1 AS pair_rank + UNION ALL + SELECT + 'DUCK-1>DUCK-3' AS sku_pair_key, + 1 AS co_viewers, + 2 AS sku_viewers, + 0.5::DECIMAL(9, 6) AS co_view_rate, + 0.75::DECIMAL(12, 4) AS lift, + 2 AS pair_rank + UNION ALL + SELECT + 'DUCK-2>DUCK-3' AS sku_pair_key, + 2 AS co_viewers, + 3 AS sku_viewers, + 0.666667::DECIMAL(9, 6) AS co_view_rate, + 1.0::DECIMAL(12, 4) AS lift, + 1 AS pair_rank +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_search_conversion__measures_zero_results_and_views_after_search.sql b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_search_conversion__measures_zero_results_and_views_after_search.sql new file mode 100644 index 0000000000..fef544481a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_search_conversion__measures_zero_results_and_views_after_search.sql @@ -0,0 +1,108 @@ +TEST (name "int_search_conversion__measures_zero_results_and_views_after_search"); + +WITH __ref__stg_search_queries AS ( + SELECT + 'Q1' AS query_id, + 'V1' AS anonymous_id, + 'decoy' AS query_text, + NULL AS query_token_count, + 5 AS results_count, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS searched_at, + 'decoys' AS filter_category, + NULL AS filter_max_price_minor, + NULL AS sort_order, + NULL AS filters + UNION ALL + SELECT + 'Q2', + 'V2', + 'decoy', + NULL, + 0, + '2026-03-10 11:00:00 +01:00'::TIMESTAMP_TZ, + 'decoys', + NULL, + NULL, + NULL + UNION ALL + SELECT + 'Q3', + 'V1', + '', + NULL, + 9, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_web_searches AS ( + SELECT + 'W1' AS event_id, + 'V1' AS anonymous_id, + '2026-03-10 09:00:00'::TIMESTAMP_NTZ AS searched_at_utc, + 'decoy' AS search_term, + 3 AS result_count, + NULL AS is_zero_result + UNION ALL + SELECT + 'W2', + 'V3', + '2026-03-10 12:00:00'::TIMESTAMP_NTZ, + 'towel', + NULL, + NULL +), + +__ref__stg_web_product_views AS ( + SELECT + 'V1' AS anonymous_id, + NULL AS customer_id, + 'V1:DUCK-1:2026-03-10 10:05:00' AS product_view_key, + 'DUCK-1' AS sku, + '2026-03-10 10:05:00'::TIMESTAMPNTZ AS viewed_minute_utc, + NULL AS list_name, + NULL AS list_position + UNION ALL + SELECT + 'V2', + NULL, + 'V2:DUCK-1:2026-03-10 10:20:00', + 'DUCK-1', + '2026-03-10 10:20:00'::TIMESTAMPNTZ, + NULL, + NULL + UNION ALL + SELECT + 'V3', + NULL, + 'V3:DUCK-2:2026-03-10 12:10:00', + 'DUCK-2', + '2026-03-10 12:10:00'::TIMESTAMPNTZ, + NULL, + NULL +), + +__expected__int_search_conversion AS ( + SELECT + 'decoy' AS search_term, + 3 AS search_count, + 2 AS searching_visitors, + 0.3333::DECIMAL(9, 4) AS zero_result_rate, + 0.3333::DECIMAL(9, 4) AS view_after_search_rate, + 2.67::DECIMAL(12, 2) AS avg_results, + 'decoys' AS top_filter_category, + '2026-03-10 09:00:00'::TIMESTAMPNTZ AS first_searched_at_utc + UNION ALL + SELECT + 'towel' AS search_term, + 1 AS search_count, + 1 AS searching_visitors, + 1.0::DECIMAL(9, 4) AS zero_result_rate, + 1.0::DECIMAL(9, 4) AS view_after_search_rate, + CAST(NULL AS DECIMAL(12, 2)) AS avg_results, + CAST(NULL AS VARCHAR) AS top_filter_category, + '2026-03-10 12:00:00'::TIMESTAMPNTZ AS first_searched_at_utc +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_session_attribution__maps_utm_pairs_then_falls_back_to_referrer.sql b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_session_attribution__maps_utm_pairs_then_falls_back_to_referrer.sql new file mode 100644 index 0000000000..d9e09cc639 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_session_attribution__maps_utm_pairs_then_falls_back_to_referrer.sql @@ -0,0 +1,145 @@ +TEST (name "int_session_attribution__maps_utm_pairs_then_falls_back_to_referrer"); + +WITH __ref__int_web_sessions AS ( + SELECT + 'S1' AS session_id, + 'V1' AS anonymous_id, + NULL AS customer_id, + NULL AS session_day, + '2026-03-10 08:00:00'::TIMESTAMP_NTZ AS session_start_utc, + '2026-03-10 08:10:00'::TIMESTAMP_NTZ AS session_end_utc, + NULL AS session_start_visitor_local, + NULL AS duration_seconds, + NULL AS event_count, + NULL AS page_view_count, + NULL AS landing_page_path, + NULL AS exit_page_path, + NULL AS referrer_host, + 'https://shop.example.com/?utm_source=Search_Ads&utm_medium=CPC&utm_campaign=Spring' AS landing_page_url, + NULL AS is_bounce, + FALSE AS has_add_to_cart, + FALSE AS has_order + UNION ALL + SELECT + 'S2', + 'V1', + NULL, + NULL, + '2026-03-10 09:00:00'::TIMESTAMP_NTZ, + '2026-03-10 09:10:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'https://shop.example.com/?utm_source=weekly&utm_medium=email', + NULL, + FALSE, + FALSE + UNION ALL + SELECT + 'S3', + 'V1', + NULL, + NULL, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ, + '2026-03-10 10:10:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'blog.example.org', + 'https://shop.example.com/home', + NULL, + FALSE, + FALSE + UNION ALL + SELECT + 'S4', + 'V2', + NULL, + NULL, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ, + '2026-03-10 10:10:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'https://shop.example.com/home', + NULL, + FALSE, + FALSE +), + +__seed__channel_mapping AS ( + SELECT + 'search_ads' AS utm_source, + 'cpc' AS utm_medium, + 'PAID_SEARCH' AS channel_code, + TRUE AS is_paid + UNION ALL + SELECT + '-', + 'email', + 'EMAIL', + FALSE + UNION ALL + SELECT + 'social_ads', + 'paid_social', + 'PAID_SOCIAL', + TRUE +), + +__expected__int_session_attribution AS ( + SELECT + 'S1' AS session_id, + 'search_ads' AS utm_source, + 'cpc' AS utm_medium, + 'spring' AS utm_campaign, + 'PAID_SEARCH' AS channel_code, + TRUE AS is_paid, + 'PAID_SEARCH' AS first_touch_channel, + 'REFERRAL' AS last_touch_channel, + 1 AS touch_number + UNION ALL + SELECT + 'S2' AS session_id, + 'weekly' AS utm_source, + 'email' AS utm_medium, + CAST(NULL AS VARCHAR) AS utm_campaign, + 'EMAIL' AS channel_code, + FALSE AS is_paid, + 'PAID_SEARCH' AS first_touch_channel, + 'REFERRAL' AS last_touch_channel, + 2 AS touch_number + UNION ALL + SELECT + 'S3' AS session_id, + CAST(NULL AS VARCHAR) AS utm_source, + CAST(NULL AS VARCHAR) AS utm_medium, + CAST(NULL AS VARCHAR) AS utm_campaign, + 'REFERRAL' AS channel_code, + FALSE AS is_paid, + 'PAID_SEARCH' AS first_touch_channel, + 'REFERRAL' AS last_touch_channel, + 3 AS touch_number + UNION ALL + SELECT + 'S4' AS session_id, + CAST(NULL AS VARCHAR) AS utm_source, + CAST(NULL AS VARCHAR) AS utm_medium, + CAST(NULL AS VARCHAR) AS utm_campaign, + 'DIRECT' AS channel_code, + FALSE AS is_paid, + 'DIRECT' AS first_touch_channel, + 'DIRECT' AS last_touch_channel, + 1 AS touch_number +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_web_sessions__keeps_whole_sessions_that_start_in_the_cursor_window.sql b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_web_sessions__keeps_whole_sessions_that_start_in_the_cursor_window.sql new file mode 100644 index 0000000000..7cef46c195 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_web_sessions__keeps_whole_sessions_that_start_in_the_cursor_window.sql @@ -0,0 +1,83 @@ +TEST (name "int_web_sessions__keeps_whole_sessions_that_start_in_the_cursor_window", cursor_start "2026-03-10", cursor_end "2026-03-11"); + +WITH __ref__stg_web_events AS ( + SELECT + 'e1' AS event_id, + '2026-03-09 23:55:00'::TIMESTAMP_NTZ AS received_at, + 'page_view' AS event_type, + '2026-03-09 23:50:00'::TIMESTAMP_NTZ AS event_at_utc, + 'V1' AS anonymous_id, + NULL AS customer_id, + NULL AS user_email, + 'https://shop.example.com/home' AS page_url, + '/home' AS page_path, + NULL AS referrer_host, + NULL AS user_agent, + NULL AS locale, + 'Europe/Berlin' AS client_timezone, + FALSE AS is_bot, + NULL AS properties, + NULL AS items + UNION ALL + SELECT + 'e2', + '2026-03-10 00:15:00'::TIMESTAMP_NTZ, + 'page_view', + '2026-03-10 00:10:00'::TIMESTAMP_NTZ, + 'V1', + NULL, + NULL, + 'https://shop.example.com/sale', + '/sale', + NULL, + NULL, + NULL, + 'Europe/Berlin', + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'e3', + '2026-03-11 00:05:00'::TIMESTAMP_NTZ, + 'page_view', + '2026-03-10 23:59:00'::TIMESTAMP_NTZ, + 'V1', + NULL, + NULL, + 'https://shop.example.com/late', + '/late', + NULL, + NULL, + NULL, + 'Europe/Berlin', + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'e4', + '2026-03-11 00:20:00'::TIMESTAMP_NTZ, + 'page_view', + '2026-03-11 00:15:00'::TIMESTAMP_NTZ, + 'V1', + NULL, + NULL, + 'https://shop.example.com/after-midnight', + '/after-midnight', + NULL, + NULL, + NULL, + 'Europe/Berlin', + FALSE, + NULL, + NULL +), + +__expected__int_web_sessions AS ( + SELECT + 'V1:20260310235900' AS session_id, + '2026-03-10 00:00:00'::TIMESTAMPNTZ AS session_day, + 2 AS event_count, + '/late' AS landing_page_path +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_web_sessions__splits_sessions_after_thirty_idle_minutes.sql b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_web_sessions__splits_sessions_after_thirty_idle_minutes.sql new file mode 100644 index 0000000000..5ec9dbbf91 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/clickstream/test_int_web_sessions__splits_sessions_after_thirty_idle_minutes.sql @@ -0,0 +1,176 @@ +TEST (name "int_web_sessions__splits_sessions_after_thirty_idle_minutes"); + +WITH __ref__stg_web_events AS ( + SELECT + 'e1' AS event_id, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS received_at, + 'page_view' AS event_type, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS event_at_utc, + 'V1' AS anonymous_id, + NULL AS customer_id, + NULL AS user_email, + 'https://shop.example.com/home' AS page_url, + '/home' AS page_path, + 'search.example.com' AS referrer_host, + NULL AS user_agent, + NULL AS locale, + 'Europe/Berlin' AS client_timezone, + FALSE AS is_bot, + NULL AS properties, + NULL AS items + UNION ALL + SELECT + 'e2', + '2026-03-10 10:10:00'::TIMESTAMP_NTZ, + 'add_to_cart', + '2026-03-10 10:10:00'::TIMESTAMP_NTZ, + 'V1', + 'C1', + NULL, + 'https://shop.example.com/p/duck-1', + '/p/duck-1', + NULL, + NULL, + NULL, + 'Europe/Berlin', + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'e3', + '2026-03-10 10:40:00'::TIMESTAMP_NTZ, + 'page_view', + '2026-03-10 10:40:00'::TIMESTAMP_NTZ, + 'V1', + NULL, + NULL, + 'https://shop.example.com/cart', + '/cart', + NULL, + NULL, + NULL, + 'Europe/Berlin', + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'e4', + '2026-03-10 11:20:00'::TIMESTAMP_NTZ, + 'page_view', + '2026-03-10 11:20:00'::TIMESTAMP_NTZ, + 'V1', + NULL, + NULL, + 'https://shop.example.com/home', + '/home', + NULL, + NULL, + NULL, + 'Europe/Berlin', + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'e5', + '2026-03-10 10:05:00'::TIMESTAMP_NTZ, + 'page_view', + '2026-03-10 10:05:00'::TIMESTAMP_NTZ, + 'V1', + NULL, + NULL, + 'https://shop.example.com/robots', + '/robots', + NULL, + NULL, + NULL, + 'Europe/Berlin', + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'e6', + '2026-03-29 01:30:00'::TIMESTAMP_NTZ, + 'page_view', + '2026-03-29 01:30:00'::TIMESTAMP_NTZ, + 'V2', + NULL, + NULL, + 'https://shop.example.com/sale', + '/sale', + NULL, + NULL, + NULL, + 'Europe/Berlin', + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'e7', + '2026-03-10 10:00:00'::TIMESTAMP_NTZ, + 'page_view', + '2026-03-10 10:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + 'https://shop.example.com/home', + '/home', + NULL, + NULL, + NULL, + 'Europe/Berlin', + FALSE, + NULL, + NULL +), + +__expected__int_web_sessions AS ( + SELECT + 'V1:20260310100000' AS session_id, + 'C1' AS customer_id, + '2026-03-10 00:00:00'::TIMESTAMPNTZ AS session_day, + '2026-03-10 11:00:00'::TIMESTAMPNTZ AS session_start_visitor_local, + 2400 AS duration_seconds, + 3 AS event_count, + 2 AS page_view_count, + '/home' AS landing_page_path, + '/cart' AS exit_page_path, + 'search.example.com' AS referrer_host, + FALSE AS is_bounce, + TRUE AS has_add_to_cart, + FALSE AS has_order + UNION ALL + SELECT + 'V1:20260310112000' AS session_id, + CAST(NULL AS VARCHAR) AS customer_id, + '2026-03-10 00:00:00'::TIMESTAMPNTZ AS session_day, + '2026-03-10 12:20:00'::TIMESTAMPNTZ AS session_start_visitor_local, + 0 AS duration_seconds, + 1 AS event_count, + 1 AS page_view_count, + '/home' AS landing_page_path, + '/home' AS exit_page_path, + CAST(NULL AS VARCHAR) AS referrer_host, + TRUE AS is_bounce, + FALSE AS has_add_to_cart, + FALSE AS has_order + UNION ALL + SELECT + 'V2:20260329013000' AS session_id, + CAST(NULL AS VARCHAR) AS customer_id, + '2026-03-29 00:00:00'::TIMESTAMPNTZ AS session_day, + '2026-03-29 03:30:00'::TIMESTAMPNTZ AS session_start_visitor_local, + 0 AS duration_seconds, + 1 AS event_count, + 1 AS page_view_count, + '/sale' AS landing_page_path, + '/sale' AS exit_page_path, + CAST(NULL AS VARCHAR) AS referrer_host, + TRUE AS is_bounce, + FALSE AS has_add_to_cart, + FALSE AS has_order +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_address_current__prefers_default_then_latest_address.sql b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_address_current__prefers_default_then_latest_address.sql new file mode 100644 index 0000000000..5d406d26ff --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_address_current__prefers_default_then_latest_address.sql @@ -0,0 +1,105 @@ +TEST (name "int_customer_address_current__prefers_default_then_latest_address"); + +WITH __ref__stg_customer_addresses AS ( + SELECT + 'A1' AS address_id, + 'C1' AS customer_id, + NULL AS line1, + NULL AS line2, + 'Berlin' AS city, + 'DE' AS country_code, + NULL AS postcode_canonical, + FALSE AS is_default, + '2025-06-01 00:00:00 +00:00'::TIMESTAMP_TZ AS valid_from, + NULL AS latitude, + NULL AS longitude, + NULL AS geo, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'A2', + 'C1', + NULL, + NULL, + 'Munich', + 'DE', + NULL, + TRUE, + '2024-01-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'A3', + 'C1', + NULL, + NULL, + 'Paris', + 'FR', + NULL, + FALSE, + '2026-01-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'A4', + 'C2', + NULL, + NULL, + 'Austin', + 'US', + NULL, + FALSE, + '2025-01-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__seed__countries AS ( + SELECT + 'DE' AS country_code, + NULL AS country_name, + 'EU' AS region_code, + NULL AS default_timezone, + NULL AS currency_code, + NULL AS vat_registered + UNION ALL + SELECT + 'FR', + NULL, + 'EU', + NULL, + NULL, + NULL +), + +__expected__int_customer_address_current AS ( + SELECT + 'C1' AS customer_id, + 'A2' AS address_id, + 'Munich' AS city, + 'EU' AS region_code, + TRUE AS is_explicit_default, + 3 AS address_count, + 2 AS distinct_countries + UNION ALL + SELECT + 'C2' AS customer_id, + 'A4' AS address_id, + 'Austin' AS city, + CAST(NULL AS VARCHAR) AS region_code, + FALSE AS is_explicit_default, + 1 AS address_count, + 1 AS distinct_countries +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_consent_state__keeps_latest_decision_per_consent_type.sql b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_consent_state__keeps_latest_decision_per_consent_type.sql new file mode 100644 index 0000000000..bebcd99d5f --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_consent_state__keeps_latest_decision_per_consent_type.sql @@ -0,0 +1,60 @@ +TEST (name "int_customer_consent_state__keeps_latest_decision_per_consent_type"); + +WITH __ref__stg_customer_consents AS ( + SELECT + 'K1' AS consent_id, + 'C1' AS customer_id, + 'EMAIL_MARKETING' AS consent_type, + TRUE AS is_granted, + '2026-01-01 00:00:00 +00:00'::TIMESTAMP_TZ AS captured_at, + NULL AS capture_surface, + NULL AS policy_version, + FALSE AS is_legacy + UNION ALL + SELECT + 'K2', + 'C1', + 'EMAIL_MARKETING', + FALSE, + '2026-02-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + FALSE + UNION ALL + SELECT + 'K3', + 'C1', + 'SMS_MARKETING', + TRUE, + '2026-01-15 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + FALSE + UNION ALL + SELECT + 'K4', + 'C1', + 'PERSONALISATION', + TRUE, + '2025-06-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + TRUE +), + +__expected__int_customer_consent_state AS ( + SELECT + 'C1' AS customer_id, + FALSE AS email_marketing, + TRUE AS sms_marketing, + TRUE AS personalisation, + FALSE AS third_party_sharing, + '2026-02-01 00:00:00 +00:00'::TIMESTAMPTZ AS last_consent_change_at, + TRUE AS has_legacy_consent +), + +__assert__consent_state_records_revocation AS ( + SELECT s.customer_id + FROM __ref("int_customer_consent_state") AS s + WHERE s.consent_state:EMAIL_MARKETING::BOOLEAN IS DISTINCT FROM FALSE +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_identity__links_identities_and_flags_duplicates.sql b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_identity__links_identities_and_flags_duplicates.sql new file mode 100644 index 0000000000..4f828752bc --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_identity__links_identities_and_flags_duplicates.sql @@ -0,0 +1,183 @@ +TEST (name "int_customer_identity__links_identities_and_flags_duplicates"); + +WITH __ref__stg_customers AS ( + SELECT + 'C1' AS customer_id, + 'ada@x.test' AS email_normalised, + '+441234' AS phone_normalised, + 'Ada' AS first_name, + 'Featherby' AS last_name, + NULL AS region_code, + NULL AS storefront_code, + NULL AS created_at, + NULL AS preferred_language, + NULL AS marketing_opt_in, + NULL AS consent_types, + NULL AS birth_year, + NULL AS preferences, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'C2', + 'ada@x.test', + '+441234', + 'Ada', + 'Featherby', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C3', + NULL, + '+15550000', + 'Bob', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_web_identifies AS ( + SELECT + 'W1' AS identity_link_key, + 'anon-2' AS anonymous_id, + 'C1' AS customer_id, + NULL AS first_seen_at_utc, + NULL AS last_seen_at_utc, + NULL AS event_count, + NULL AS identify_methods + UNION ALL + SELECT + 'W2', + 'anon-1', + 'C1', + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'W3', + 'anon-1', + 'C1', + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_newsletter_subscriptions AS ( + SELECT + 'N1' AS list_subscription_id, + 'ada@x.test' AS email, + 'weekly' AS list_code, + 'subscribed' AS list_status, + NULL AS subscribed_at, + NULL AS unsubscribed_at, + NULL AS signup_form, + NULL AS is_double_opt_in, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'N2', + 'ada@x.test', + 'offers', + 'unsubscribed', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_tickets AS ( + SELECT + 'T1' AS ticket_id, + 'ada@x.test' AS customer_email, + NULL AS channel, + NULL AS priority, + NULL AS ticket_status, + NULL AS created_at, + NULL AS solved_at, + NULL AS order_id, + NULL AS contact_reason, + NULL AS tag_list, + NULL AS is_vip, + NULL AS custom_fields, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + 'ada@x.test', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_customer_identity AS ( + SELECT + 'C1' AS customer_id, + 'x.test' AS email_domain, + 'Ada Featherby' AS full_name, + 2 AS anonymous_id_count, + 2 AS ticket_count, + 'C2' AS duplicate_candidate_id, + 100 AS duplicate_name_similarity, + 2 AS accounts_sharing_email + UNION ALL + SELECT + 'C2' AS customer_id, + 'x.test' AS email_domain, + 'Ada Featherby' AS full_name, + 0 AS anonymous_id_count, + 2 AS ticket_count, + 'C1' AS duplicate_candidate_id, + 100 AS duplicate_name_similarity, + 2 AS accounts_sharing_email + UNION ALL + SELECT + 'C3' AS customer_id, + CAST(NULL AS VARCHAR) AS email_domain, + CAST(NULL AS VARCHAR) AS full_name, + 0 AS anonymous_id_count, + 0 AS ticket_count, + CAST(NULL AS VARCHAR) AS duplicate_candidate_id, + CAST(NULL AS INT) AS duplicate_name_similarity, + 1 AS accounts_sharing_email +), + +__assert__newsletter_lists_hold_only_subscribed_lists AS ( + SELECT i.customer_id + FROM __ref("int_customer_identity") AS i + WHERE + i.customer_id = 'C1' + AND ARRAY_TO_STRING(i.newsletter_lists, ',') <> 'weekly' +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_order_sequence__numbers_orders_and_skips_cancellations.sql b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_order_sequence__numbers_orders_and_skips_cancellations.sql new file mode 100644 index 0000000000..18d296c5ea --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_order_sequence__numbers_orders_and_skips_cancellations.sql @@ -0,0 +1,138 @@ +TEST (name "int_customer_order_sequence__numbers_orders_and_skips_cancellations"); + +WITH __ref__int_orders_usd AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + NULL AS region_code, + NULL AS currency_code, + '2026-01-10 10:00:00 +00:00'::TIMESTAMP_TZ AS placed_at, + '2026-01-10'::DATE AS placed_date_local, + 'PAID' AS order_status, + 'web' AS sales_channel, + NULL AS units, + NULL AS amount, + 100.0::DECIMAL(18, 2) AS amount_usd, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS rate_age_days, + NULL AS is_rate_stale, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size + UNION ALL + SELECT + 'O2', + 'C1', + NULL, + NULL, + '2026-01-20 10:00:00 +00:00'::TIMESTAMP_TZ, + '2026-01-20'::DATE, + 'CANCELLED', + 'web', + NULL, + NULL, + 999.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + 'C1', + NULL, + NULL, + '2026-04-15 10:00:00 +00:00'::TIMESTAMP_TZ, + '2026-04-15'::DATE, + 'SHIPPED', + 'app', + NULL, + NULL, + 50.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O5', + 'C2', + NULL, + NULL, + '2026-02-01 10:00:00 +00:00'::TIMESTAMP_TZ, + '2026-02-01'::DATE, + 'PAID', + 'web', + NULL, + NULL, + 20.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_customer_order_sequence AS ( + SELECT + 'O1' AS order_id, + 1 AS order_number, + TRUE AS is_first_order, + CAST(NULL AS INT) AS days_since_previous_order, + 100.0::DECIMAL(18, 2) AS lifetime_revenue_usd, + 100.0::DECIMAL(18, 2) AS rolling_90d_revenue_usd, + CAST(NULL AS VARCHAR) AS previous_sales_channel, + FALSE AS is_channel_switch, + '2026-01-01'::DATE AS cohort_month + UNION ALL + SELECT + 'O3' AS order_id, + 2 AS order_number, + FALSE AS is_first_order, + 95 AS days_since_previous_order, + 150.0::DECIMAL(18, 2) AS lifetime_revenue_usd, + 50.0::DECIMAL(18, 2) AS rolling_90d_revenue_usd, + 'web' AS previous_sales_channel, + TRUE AS is_channel_switch, + '2026-01-01'::DATE AS cohort_month + UNION ALL + SELECT + 'O5' AS order_id, + 1 AS order_number, + TRUE AS is_first_order, + CAST(NULL AS INT) AS days_since_previous_order, + 20.0::DECIMAL(18, 2) AS lifetime_revenue_usd, + 20.0::DECIMAL(18, 2) AS rolling_90d_revenue_usd, + CAST(NULL AS VARCHAR) AS previous_sales_channel, + FALSE AS is_channel_switch, + '2026-02-01'::DATE AS cohort_month +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_rfm__scores_quintiles_and_assigns_segments.sql b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_rfm__scores_quintiles_and_assigns_segments.sql new file mode 100644 index 0000000000..f450321af3 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_customer_rfm__scores_quintiles_and_assigns_segments.sql @@ -0,0 +1,185 @@ +TEST (name "int_customer_rfm__scores_quintiles_and_assigns_segments"); + +WITH __ref__int_customer_order_sequence AS ( + SELECT + 'A1' AS order_id, + 'A' AS customer_id, + NULL AS placed_at, + '2026-01-01'::DATE AS placed_date_local, + NULL AS order_number, + NULL AS is_first_order, + NULL AS days_since_previous_order, + NULL AS amount_usd, + 1000.0::DECIMAL(18, 2) AS lifetime_revenue_usd, + NULL AS rolling_90d_revenue_usd, + NULL AS previous_sales_channel, + NULL AS is_channel_switch, + NULL AS cohort_month + UNION ALL + SELECT + 'A2', + 'A', + NULL, + '2026-02-01'::DATE, + NULL, + NULL, + NULL, + NULL, + 2000.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'A3', + 'A', + NULL, + '2026-03-31'::DATE, + NULL, + NULL, + NULL, + NULL, + 3000.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'B1', + 'B', + NULL, + '2026-03-01'::DATE, + NULL, + NULL, + NULL, + NULL, + 450.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'B2', + 'B', + NULL, + '2026-03-20'::DATE, + NULL, + NULL, + NULL, + NULL, + 900.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C1', + 'C', + NULL, + '2026-03-30'::DATE, + NULL, + NULL, + NULL, + NULL, + 100.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'D1', + 'D', + NULL, + '2025-12-01'::DATE, + NULL, + NULL, + NULL, + NULL, + 50.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E1', + 'E', + NULL, + '2025-10-01'::DATE, + NULL, + NULL, + NULL, + NULL, + 200.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E2', + 'E', + NULL, + '2025-11-01'::DATE, + NULL, + NULL, + NULL, + NULL, + 400.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL +), + +__expected__int_customer_rfm AS ( + SELECT + 'A' AS customer_id, + 0 AS days_since_last_order, + 3 AS order_count, + 1000.0::DECIMAL(18, 2) AS average_order_usd, + '555' AS rfm_code, + 'CHAMPION' AS segment, + TRUE AS is_vip + UNION ALL + SELECT + 'B' AS customer_id, + 11 AS days_since_last_order, + 2 AS order_count, + 450.0::DECIMAL(18, 2) AS average_order_usd, + '334' AS rfm_code, + 'LOYAL' AS segment, + FALSE AS is_vip + UNION ALL + SELECT + 'C' AS customer_id, + 1 AS days_since_last_order, + 1 AS order_count, + 100.0::DECIMAL(18, 2) AS average_order_usd, + '412' AS rfm_code, + 'NEW' AS segment, + FALSE AS is_vip + UNION ALL + SELECT + 'D' AS customer_id, + 120 AS days_since_last_order, + 1 AS order_count, + 50.0::DECIMAL(18, 2) AS average_order_usd, + '221' AS rfm_code, + 'HIBERNATING' AS segment, + FALSE AS is_vip + UNION ALL + SELECT + 'E' AS customer_id, + 150 AS days_since_last_order, + 2 AS order_count, + 200.0::DECIMAL(18, 2) AS average_order_usd, + '143' AS rfm_code, + 'AT_RISK' AS segment, + FALSE AS is_vip +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_gift_card_balances__values_liabilities_across_currency_exponents.sql b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_gift_card_balances__values_liabilities_across_currency_exponents.sql new file mode 100644 index 0000000000..5015a51e20 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_gift_card_balances__values_liabilities_across_currency_exponents.sql @@ -0,0 +1,143 @@ +TEST (name "int_gift_card_balances__values_liabilities_across_currency_exponents"); + +WITH __ref__stg_gift_cards AS ( + SELECT + 'G1' AS gift_card_id, + NULL AS code_last4, + 'EUR' AS currency_code, + 5000 AS initial_amount_minor, + NULL AS issued_at, + '2027-01-01'::DATE AS expires_on, + NULL AS purchaser_customer_id, + NULL AS is_promotional, + NULL AS design_code, + NULL AS details, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'G2', + NULL, + 'KWD', + 10000, + NULL, + '2026-01-01'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'G3', + NULL, + 'USD', + 1000, + NULL, + '2027-01-01'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_gift_card_transactions AS ( + SELECT + 'T1' AS transaction_id, + 'G1' AS gift_card_id, + NULL AS order_id, + 'REDEEM' AS transaction_type, + -2000 AS signed_amount_minor, + '2026-02-01 10:00:00 +00:00'::TIMESTAMP_TZ AS occurred_at, + NULL AS channel, + NULL AS terminal_id, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + 'G3', + NULL, + 'REDEEM', + -1000, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-03-30'::DATE AS rate_date, + 1.1::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'KWD', + '2026-01-01'::DATE, + 3.25::NUMBER(18, 8), + NULL, + NULL + UNION ALL + SELECT + 'USD', + '2020-01-01'::DATE, + 1.0::NUMBER(18, 8), + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'KWD', + NULL, + 3, + NULL + UNION ALL + SELECT + 'USD', + NULL, + 2, + NULL +), + +__expected__int_gift_card_balances AS ( + SELECT + 'G1' AS gift_card_id, + 50.0::NUMBER(38, 6) AS initial_amount, + 20.0::NUMBER(38, 6) AS redeemed_amount, + 30.0::NUMBER(38, 6) AS balance_amount, + 33.0::DECIMAL(18, 2) AS balance_usd, + 1 AS transaction_count, + 'active' AS liability_status + UNION ALL + SELECT + 'G2' AS gift_card_id, + 10.0::NUMBER(38, 6) AS initial_amount, + 0.0::NUMBER(38, 6) AS redeemed_amount, + 10.0::NUMBER(38, 6) AS balance_amount, + 32.5::DECIMAL(18, 2) AS balance_usd, + 0 AS transaction_count, + 'expired_unused' AS liability_status + UNION ALL + SELECT + 'G3' AS gift_card_id, + 10.0::NUMBER(38, 6) AS initial_amount, + 10.0::NUMBER(38, 6) AS redeemed_amount, + 0.0::NUMBER(38, 6) AS balance_amount, + 0.0::DECIMAL(18, 2) AS balance_usd, + 1 AS transaction_count, + 'fully_redeemed' AS liability_status +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_loyalty_balances__runs_points_and_flags_expired_grants.sql b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_loyalty_balances__runs_points_and_flags_expired_grants.sql new file mode 100644 index 0000000000..544d3b8d70 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_loyalty_balances__runs_points_and_flags_expired_grants.sql @@ -0,0 +1,85 @@ +TEST (name "int_loyalty_balances__runs_points_and_flags_expired_grants"); + +WITH __ref__stg_loyalty_ledger AS ( + SELECT + 'E1' AS entry_id, + 'LA1' AS loyalty_account_id, + 100 AS points_delta, + 'ORDER' AS reason_code, + NULL AS order_id, + '2026-01-01 00:00:00 +00:00'::TIMESTAMP_TZ AS posted_at, + NULL AS campaign_ref, + '2026-02-01'::DATE AS expires_on, + NULL AS metadata, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'E2', + 'LA1', + 50, + 'REVIEW', + NULL, + '2026-02-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + '2026-12-31'::DATE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E3', + 'LA1', + -30, + 'REDEEM', + NULL, + '2026-03-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_loyalty_accounts AS ( + SELECT + 'LA1' AS loyalty_account_id, + 'C1' AS customer_id, + 'GOLDEN_DUCK' AS tier_code, + NULL AS points_balance, + NULL AS enrolled_at, + NULL AS tier_change_count, + NULL AS previous_tier_code, + NULL AS tier_history, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_loyalty_balances AS ( + SELECT + 'E1' AS entry_id, + 'C1' AS customer_id, + 100 AS running_points, + 100 AS expiring_points, + TRUE AS is_expired, + 1.0::DECIMAL(18, 4) AS running_value_usd, + 'GOLDEN_DUCK' AS tier_code + UNION ALL + SELECT + 'E2' AS entry_id, + 'C1' AS customer_id, + 150 AS running_points, + 0 AS expiring_points, + FALSE AS is_expired, + 1.5::DECIMAL(18, 4) AS running_value_usd, + 'GOLDEN_DUCK' AS tier_code + UNION ALL + SELECT + 'E3' AS entry_id, + 'C1' AS customer_id, + 120 AS running_points, + 0 AS expiring_points, + FALSE AS is_expired, + 1.2::DECIMAL(18, 4) AS running_value_usd, + 'GOLDEN_DUCK' AS tier_code +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_loyalty_tier_changes__classifies_each_tier_change.sql b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_loyalty_tier_changes__classifies_each_tier_change.sql new file mode 100644 index 0000000000..18b271c857 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_loyalty_tier_changes__classifies_each_tier_change.sql @@ -0,0 +1,70 @@ +TEST (name "int_loyalty_tier_changes__classifies_each_tier_change"); + +WITH __ref__stg_loyalty_accounts AS ( + SELECT + 'LA1' AS loyalty_account_id, + 'C1' AS customer_id, + NULL AS tier_code, + NULL AS points_balance, + NULL AS enrolled_at, + NULL AS tier_change_count, + NULL AS previous_tier_code, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'tier', + 'duckling', + 'at', + '2025-01-01T00:00:00Z' + ), + OBJECT_CONSTRUCT( + 'tier', + 'golden_duck', + 'at', + '2025-03-01T00:00:00Z' + ), + OBJECT_CONSTRUCT( + 'tier', + 'drake', + 'at', + '2025-04-01T00:00:00Z' + ), + OBJECT_CONSTRUCT( + 'tier', + 'drake', + 'at', + '2025-04-11T00:00:00Z' + ) + )::ARRAY AS tier_history, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_loyalty_tier_changes AS ( + SELECT + 'LA1:0' AS tier_change_key, + 'DUCKLING' AS tier_code, + CAST(NULL AS VARCHAR) AS previous_tier_code, + 'enrol' AS direction, + CAST(NULL AS INT) AS days_in_previous_tier + UNION ALL + SELECT + 'LA1:1' AS tier_change_key, + 'GOLDEN_DUCK' AS tier_code, + 'DUCKLING' AS previous_tier_code, + 'upgrade' AS direction, + 59 AS days_in_previous_tier + UNION ALL + SELECT + 'LA1:2' AS tier_change_key, + 'DRAKE' AS tier_code, + 'GOLDEN_DUCK' AS previous_tier_code, + 'downgrade' AS direction, + 31 AS days_in_previous_tier + UNION ALL + SELECT + 'LA1:3' AS tier_change_key, + 'DRAKE' AS tier_code, + 'DRAKE' AS previous_tier_code, + 'same' AS direction, + 10 AS days_in_previous_tier +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_store_credit_balances__sums_balance_and_caps_expiring_credit.sql b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_store_credit_balances__sums_balance_and_caps_expiring_credit.sql new file mode 100644 index 0000000000..6e5b78f398 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_store_credit_balances__sums_balance_and_caps_expiring_credit.sql @@ -0,0 +1,66 @@ +TEST (name "int_store_credit_balances__sums_balance_and_caps_expiring_credit"); + +WITH __ref__stg_store_credit_ledger AS ( + SELECT + 'S1' AS entry_id, + 'C1' AS customer_id, + 'EUR' AS currency_code, + 1000 AS amount_minor, + 'GOODWILL_DELAY' AS reason_code, + '2026-01-01 00:00:00 +00:00'::TIMESTAMP_TZ AS posted_at, + NULL AS source_order_id, + '2026-04-15'::DATE AS expires_on, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'S2', + 'C1', + 'EUR', + 500, + 'REFUND', + '2026-02-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + '2026-12-31'::DATE, + NULL, + NULL + UNION ALL + SELECT + 'S3', + 'C1', + 'EUR', + -300, + 'SPEND', + '2026-03-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-03-01'::DATE AS rate_date, + 1.1::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol +), + +__expected__int_store_credit_balances AS ( + SELECT + 'C1:EUR' AS store_credit_key, + 1200 AS balance_minor, + 1000 AS expiring_30d_minor, + 3 AS posting_count, + 0.666667::DECIMAL(9, 6) AS goodwill_share, + 13.2::DECIMAL(18, 2) AS balance_usd +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_wishlist_demand__counts_purchases_made_after_wishlisting.sql b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_wishlist_demand__counts_purchases_made_after_wishlisting.sql new file mode 100644 index 0000000000..5d192bc22f --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/customers/test_int_wishlist_demand__counts_purchases_made_after_wishlisting.sql @@ -0,0 +1,140 @@ +TEST (name "int_wishlist_demand__counts_purchases_made_after_wishlisting"); + +WITH __ref__stg_wishlist_items AS ( + SELECT + 'W1' AS wishlist_id, + 'C1' AS customer_id, + 'W1#1' AS wishlist_item_key, + 'DUCK-1' AS sku, + '2026-03-01 00:00:00 +00:00'::TIMESTAMPTZ AS added_at, + NULL AS item_position + UNION ALL + SELECT + 'W2', + 'C1', + 'W2#1', + 'DUCK-1', + '2026-03-02 00:00:00 +00:00'::TIMESTAMPTZ, + NULL + UNION ALL + SELECT + 'W3', + 'C2', + 'W3#1', + 'DUCK-1', + '2026-03-05 00:00:00 +00:00'::TIMESTAMPTZ, + NULL +), + +__ref__stg_wishlists AS ( + SELECT + 'W1' AS wishlist_id, + 'C1' AS customer_id, + NULL AS created_at, + TRUE AS is_public, + NULL AS item_count, + NULL AS items, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'W2', + 'C1', + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'W3', + 'C2', + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_order_lines AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + NULL AS line_no, + 'DUCK-1' AS sku, + NULL AS quantity, + NULL AS unit_amount_minor, + NULL AS gross_amount_minor, + NULL AS tax_amount_minor, + NULL AS discount_count, + NULL AS discounts, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'L2', + 'O2', + NULL, + 'DUCK-1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_orders AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + NULL AS storefront_code, + NULL AS currency_code, + '2026-03-03 12:00:00 +00:00'::TIMESTAMP_TZ AS placed_at, + NULL AS order_status, + NULL AS sales_channel, + NULL AS is_gift, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS device_type, + NULL AS item_count_hint, + NULL AS shipping_address_id, + NULL AS attributes, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'O2', + 'C2', + NULL, + NULL, + '2026-02-20 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_wishlist_demand AS ( + SELECT + 'DUCK-1' AS sku, + 2 AS wishlisting_customers, + 1 AS public_wishlist_count, + 1 AS purchased_after_wishlist, + 0.5::DECIMAL(9, 6) AS wishlist_conversion_rate, + 2.5::DECIMAL(10, 2) AS median_days_to_purchase, + '2026-03-01 00:00:00 +00:00'::TIMESTAMP_TZ AS first_added_at +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_chargeback_exposure__dates_responses_by_reason_family_and_counts_prior_disputes.sql b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_chargeback_exposure__dates_responses_by_reason_family_and_counts_prior_disputes.sql new file mode 100644 index 0000000000..4fa6439681 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_chargeback_exposure__dates_responses_by_reason_family_and_counts_prior_disputes.sql @@ -0,0 +1,220 @@ +TEST (name "int_chargeback_exposure__dates_responses_by_reason_family_and_counts_prior_disputes"); + +WITH __ref__stg_chargebacks AS ( + SELECT + 'CB1' AS chargeback_id, + 'PM1' AS payment_id, + 10000 AS amount_minor, + 'EUR' AS currency_code, + '10.4' AS reason_code, + 'fraud' AS reason_family, + '2026-02-01 10:00:00 +00:00'::TIMESTAMP_TZ AS opened_at, + NULL AS closed_at, + 'open' AS outcome, + NULL AS evidence_document_count, + OBJECT_CONSTRUCT( + 'documents', + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT('kind', 'Tracking'), + OBJECT_CONSTRUCT('kind', 'invoice'), + OBJECT_CONSTRUCT('kind', 'tracking') + ) + )::OBJECT AS evidence, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'CB2', + 'PM2', + 2500, + 'EUR', + '13.1', + 'consumer_dispute', + '2026-03-10 23:30:00 -05:00'::TIMESTAMP_TZ, + '2026-03-20 10:00:00 +00:00'::TIMESTAMP_TZ, + 'won', + NULL, + PARSE_JSON('{}')::OBJECT, + NULL, + NULL + UNION ALL + SELECT + 'CB3', + 'PM9', + 1000, + 'KWD', + '99', + 'other', + '2026-03-25 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'pending', + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_payments AS ( + SELECT + 'PM1' AS payment_id, + 'O1' AS order_id, + 'card' AS payment_method, + 10000 AS amount_minor, + NULL AS currency_code, + NULL AS authorised_at, + NULL AS captured_at, + NULL AS card_bin, + NULL AS gateway_status, + NULL AS gateway_error_code, + NULL AS risk_score, + NULL AS three_ds_outcome, + NULL AS gateway_reference, + NULL AS gateway_response, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'PM2', + 'O2', + 'wallet', + 5000, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_orders_usd AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + NULL AS region_code, + NULL AS currency_code, + NULL AS placed_at, + NULL AS placed_date_local, + NULL AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS amount, + NULL AS amount_usd, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS rate_age_days, + NULL AS is_rate_stale, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size + UNION ALL + SELECT + 'O2', + 'C1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-02-01'::DATE AS rate_date, + 1.1::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'KWD', + '2026-03-01'::DATE, + 3.25::NUMBER(18, 8), + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'KWD', + NULL, + 3, + NULL +), + +__expected__int_chargeback_exposure AS ( + SELECT + 'CB1' AS chargeback_id, + 'C1' AS customer_id, + '2026-02-21'::DATE AS response_due_on, + 110.0::DECIMAL(18, 2) AS amount_usd, + 1.0::DECIMAL(9, 6) AS share_of_payment, + TRUE AS has_delivery_proof, + 58 AS days_open, + TRUE AS is_overdue, + 110.0::DECIMAL(18, 2) AS at_risk_usd, + 0 AS prior_chargebacks_for_customer + UNION ALL + SELECT + 'CB2' AS chargeback_id, + 'C1' AS customer_id, + '2026-04-10'::DATE AS response_due_on, + 27.5::DECIMAL(18, 2) AS amount_usd, + 0.5::DECIMAL(9, 6) AS share_of_payment, + FALSE AS has_delivery_proof, + 9 AS days_open, + FALSE AS is_overdue, + 0.0::DECIMAL(18, 2) AS at_risk_usd, + 1 AS prior_chargebacks_for_customer + UNION ALL + SELECT + 'CB3' AS chargeback_id, + CAST(NULL AS VARCHAR) AS customer_id, + '2026-04-24'::DATE AS response_due_on, + 3.25::DECIMAL(18, 2) AS amount_usd, + CAST(NULL AS DECIMAL(9, 6)) AS share_of_payment, + FALSE AS has_delivery_proof, + 6 AS days_open, + FALSE AS is_overdue, + 3.25::DECIMAL(18, 2) AS at_risk_usd, + 0 AS prior_chargebacks_for_customer +), + +__assert__evidence_kinds_are_distinct_and_lower_cased AS ( + SELECT c.chargeback_id + FROM __ref("int_chargeback_exposure") AS c + WHERE + ARRAY_TO_STRING(c.evidence_kinds, ',') <> IFF(c.chargeback_id = 'CB1', 'invoice,tracking', '') +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_invoices_reconciled__checks_numbering_balances_and_tax_sources.sql b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_invoices_reconciled__checks_numbering_balances_and_tax_sources.sql new file mode 100644 index 0000000000..535c2cefc0 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_invoices_reconciled__checks_numbering_balances_and_tax_sources.sql @@ -0,0 +1,221 @@ +TEST (name "int_invoices_reconciled__checks_numbering_balances_and_tax_sources"); + +WITH __ref__stg_invoices AS ( + SELECT + 'IV1' AS invoice_id, + 'O1' AS order_id, + 'DE-1001' AS invoice_number, + NULL AS invoice_series, + '2026-03-31 23:30:00 +00:00'::TIMESTAMP_TZ AS issued_at, + 'EUR' AS currency_code, + 1000 AS net_minor, + 190 AS tax_minor, + 1190 AS gross_minor, + ARRAY_CONSTRUCT(OBJECT_CONSTRUCT('tax_minor', 190))::ARRAY AS tax_lines, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'IV2', + 'O2', + 'DE-1003', + NULL, + '2026-03-15 10:00:00 +00:00'::TIMESTAMP_TZ, + 'EUR', + 500, + 95, + 600, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT('tax_minor', 50), + OBJECT_CONSTRUCT('tax_minor', 45) + )::ARRAY, + NULL, + NULL + UNION ALL + SELECT + 'IV3', + 'O3', + 'X99', + 'UK', + '2026-03-20 10:00:00 +09:00'::TIMESTAMP_TZ, + 'JPY', + 1000, + 100, + 1100, + NULL, + NULL, + NULL +), + +__ref__stg_invoice_tax_lines AS ( + SELECT + 'IV1#1' AS invoice_tax_line_key, + 'IV1' AS invoice_id, + 'VAT_STD' AS tax_code, + NULL AS tax_rate_pct, + NULL AS taxable_minor, + 100 AS tax_minor + UNION ALL + SELECT + 'IV1#2', + 'IV1', + 'VAT_STD', + NULL, + NULL, + 91 + UNION ALL + SELECT + 'IV2#1', + 'IV2', + 'VAT_RED', + NULL, + NULL, + 95 +), + +__ref__int_orders_enriched AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS storefront_code, + NULL AS region_code, + 'Europe/Berlin' AS storefront_timezone, + 'EUR' AS currency_code, + NULL AS placed_at, + NULL AS placed_date_local, + NULL AS order_status, + NULL AS sales_channel, + NULL AS line_count, + NULL AS units, + NULL AS gross_amount_minor, + NULL AS discount_amount_minor, + NULL AS net_amount_minor, + 190 AS tax_amount_minor, + NULL AS captured_amount_minor, + NULL AS outstanding_amount_minor, + NULL AS promo_code, + NULL AS ship_to_country_code, + NULL AS is_cross_border, + NULL AS is_first_party_customer, + NULL AS basket_size, + NULL AS is_gift, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS total_weight_kg + UNION ALL + SELECT + 'O2', + NULL, + NULL, + NULL, + NULL, + 'USD', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 97, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'JPY', + NULL, + 0, + NULL + UNION ALL + SELECT + 'USD', + NULL, + 2, + NULL +), + +__expected__int_invoices_reconciled AS ( + SELECT + 'IV1' AS invoice_id, + 'DE' AS invoice_series, + 1001 AS invoice_sequence, + CAST(NULL AS INT) AS sequence_gap, + '2026-04-01'::DATE AS issued_date_local, + 10.0::NUMBER(38, 6) AS net, + 1.9::NUMBER(38, 6) AS tax, + 11.9::NUMBER(38, 6) AS gross, + 190 AS embedded_tax_minor, + 191 AS loaded_tax_minor, + 2 AS tax_line_count, + TRUE AS header_balances, + TRUE AS tax_lines_agree, + TRUE AS matches_order_tax, + TRUE AS invoice_currency_matches_order + UNION ALL + SELECT + 'IV2' AS invoice_id, + 'DE' AS invoice_series, + 1003 AS invoice_sequence, + 1 AS sequence_gap, + '2026-03-15'::DATE AS issued_date_local, + 5.0::NUMBER(38, 6) AS net, + 0.95::NUMBER(38, 6) AS tax, + 6.0::NUMBER(38, 6) AS gross, + 95 AS embedded_tax_minor, + 95 AS loaded_tax_minor, + 2 AS tax_line_count, + FALSE AS header_balances, + TRUE AS tax_lines_agree, + FALSE AS matches_order_tax, + FALSE AS invoice_currency_matches_order + UNION ALL + SELECT + 'IV3' AS invoice_id, + 'UK' AS invoice_series, + 99 AS invoice_sequence, + CAST(NULL AS INT) AS sequence_gap, + '2026-03-20'::DATE AS issued_date_local, + 1000.0::NUMBER(38, 6) AS net, + 100.0::NUMBER(38, 6) AS tax, + 1100.0::NUMBER(38, 6) AS gross, + CAST(NULL AS INT) AS embedded_tax_minor, + CAST(NULL AS INT) AS loaded_tax_minor, + 0 AS tax_line_count, + TRUE AS header_balances, + FALSE AS tax_lines_agree, + FALSE AS matches_order_tax, + FALSE AS invoice_currency_matches_order +), + +__assert__tax_codes_are_distinct AS ( + SELECT i.invoice_id + FROM __ref("int_invoices_reconciled") AS i + WHERE + ARRAY_TO_STRING(i.tax_codes, ',') <> DECODE( + i.invoice_id, + 'IV1', 'VAT_STD', + 'IV2', 'VAT_RED', + '' + ) +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_ledger_balances_daily__fills_quiet_days_and_checks_normal_side.sql b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_ledger_balances_daily__fills_quiet_days_and_checks_normal_side.sql new file mode 100644 index 0000000000..45d8995a08 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_ledger_balances_daily__fills_quiet_days_and_checks_normal_side.sql @@ -0,0 +1,124 @@ +TEST (name "int_ledger_balances_daily__fills_quiet_days_and_checks_normal_side"); + +WITH __ref__stg_ledger_entries AS ( + SELECT + 'E1' AS entry_id, + NULL AS journal_code, + '1100' AS account_code, + 'asset' AS account_class, + 5000 AS signed_amount_minor, + 'EUR' AS currency_code, + '2026-03-28 10:00:00 +00:00'::TIMESTAMP_TZ AS posted_at, + NULL AS reference_type, + NULL AS reference_id, + NULL AS reference, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'E2', + NULL, + '1100', + 'asset', + -2000, + 'EUR', + '2026-03-29 23:30:00 -01:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E3', + NULL, + '1100', + 'asset', + -4000, + 'EUR', + '2026-03-31 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E4', + NULL, + '4000', + 'revenue', + -1000, + 'EUR', + '2026-03-31 08:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol +), + +__expected__int_ledger_balances_daily AS ( + SELECT + '1100:EUR:2026-03-28' AS balance_key, + 5000 AS debit_minor, + 0 AS credit_minor, + 1 AS posting_count, + 5000 AS closing_balance_minor, + 50.0::NUMBER(38, 6) AS closing_balance, + 5000 AS month_to_date_movement_minor, + FALSE AS is_month_end, + TRUE AS is_normal_side + UNION ALL + SELECT + '1100:EUR:2026-03-29' AS balance_key, + 0 AS debit_minor, + 0 AS credit_minor, + 0 AS posting_count, + 5000 AS closing_balance_minor, + 50.0::NUMBER(38, 6) AS closing_balance, + 5000 AS month_to_date_movement_minor, + FALSE AS is_month_end, + TRUE AS is_normal_side + UNION ALL + SELECT + '1100:EUR:2026-03-30' AS balance_key, + 0 AS debit_minor, + 2000 AS credit_minor, + 1 AS posting_count, + 3000 AS closing_balance_minor, + 30.0::NUMBER(38, 6) AS closing_balance, + 3000 AS month_to_date_movement_minor, + FALSE AS is_month_end, + TRUE AS is_normal_side + UNION ALL + SELECT + '1100:EUR:2026-03-31' AS balance_key, + 0 AS debit_minor, + 4000 AS credit_minor, + 1 AS posting_count, + -1000 AS closing_balance_minor, + -10.0::NUMBER(38, 6) AS closing_balance, + -1000 AS month_to_date_movement_minor, + TRUE AS is_month_end, + FALSE AS is_normal_side + UNION ALL + SELECT + '4000:EUR:2026-03-31' AS balance_key, + 0 AS debit_minor, + 1000 AS credit_minor, + 1 AS posting_count, + -1000 AS closing_balance_minor, + -10.0::NUMBER(38, 6) AS closing_balance, + -1000 AS month_to_date_movement_minor, + TRUE AS is_month_end, + TRUE AS is_normal_side +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_payout_reconciliation__classifies_each_payout_against_lines_and_reports.sql b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_payout_reconciliation__classifies_each_payout_against_lines_and_reports.sql new file mode 100644 index 0000000000..da3fef99a9 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_payout_reconciliation__classifies_each_payout_against_lines_and_reports.sql @@ -0,0 +1,301 @@ +TEST (name "int_payout_reconciliation__classifies_each_payout_against_lines_and_reports"); + +WITH __ref__stg_payouts AS ( + SELECT + 'PO1' AS payout_id, + 'PAYCO' AS provider_code, + 'EUR' AS currency_code, + NULL AS amount_minor, + NULL AS fee_minor, + 9400 AS net_minor, + '2026-03-10'::DATE AS paid_out_on, + PARSE_JSON('[{}, {}, {}]')::ARRAY AS lines, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'PO2', + 'PAYCO', + 'EUR', + NULL, + NULL, + 5000, + '2026-03-12'::DATE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'PO3', + 'PAYCO', + 'EUR', + NULL, + NULL, + 1000, + '2026-03-13'::DATE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'PO4', + 'PAYCO', + 'EUR', + NULL, + NULL, + 2000, + '2026-03-14'::DATE, + NULL, + NULL, + NULL +), + +__ref__stg_payout_lines AS ( + SELECT + 'PO1#1' AS payout_line_key, + 'PO1' AS payout_id, + 'PM1' AS payment_id, + 'charge' AS line_type, + 10000 AS gross_minor, + 300 AS fee_minor + UNION ALL + SELECT + 'PO1#2', + 'PO1', + 'PM2', + 'refund', + 500, + 0 + UNION ALL + SELECT + 'PO1#3', + 'PO1', + 'PM3', + 'charge', + 200, + NULL + UNION ALL + SELECT + 'PO2#1', + 'PO2', + 'PM9', + 'charge', + 5000, + 0 + UNION ALL + SELECT + 'PO3#1', + 'PO3', + 'PM4', + 'charge', + 1000, + 0 + UNION ALL + SELECT + 'PO4#1', + 'PO4', + 'PM5', + 'charge', + 2000, + 0 +), + +__ref__stg_payments AS ( + SELECT + 'PM1' AS payment_id, + NULL AS order_id, + NULL AS payment_method, + 10000 AS amount_minor, + NULL AS currency_code, + NULL AS authorised_at, + NULL AS captured_at, + NULL AS card_bin, + NULL AS gateway_status, + NULL AS gateway_error_code, + NULL AS risk_score, + NULL AS three_ds_outcome, + NULL AS gateway_reference, + NULL AS gateway_response, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'PM2', + NULL, + NULL, + 500, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'PM3', + NULL, + NULL, + 200, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'PM4', + NULL, + NULL, + 1000, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'PM5', + NULL, + NULL, + 2000, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_settlement_reports AS ( + SELECT + 'R1' AS report_id, + 'PAYCO' AS provider_code, + '2026-03-10'::DATE AS report_date, + 'EUR' AS currency_code, + 9700 AS gross_minor, + 300 AS fees_minor, + 3 AS line_count, + NULL AS report + UNION ALL + SELECT + 'R1B', + 'PAYCO', + '2026-03-10'::DATE, + 'EUR', + 0, + 0, + 1, + NULL + UNION ALL + SELECT + 'R2', + 'PAYCO', + '2026-03-12'::DATE, + 'EUR', + 5000, + 0, + 1, + NULL + UNION ALL + SELECT + 'R4', + 'PAYCO', + '2026-03-14'::DATE, + 'EUR', + 2100, + NULL, + 1, + NULL +), + +__expected__int_payout_reconciliation AS ( + SELECT + 'PO1' AS payout_id, + 9400 AS lines_net_minor, + 2 AS charge_line_count, + 1 AS refund_line_count, + 0 AS orphan_line_count, + 3 AS embedded_line_count, + 3 AS loaded_line_count, + 10200 AS matched_payment_minor, + 'R1' AS report_id, + 9400 AS report_net_minor, + TRUE AS lines_match_payout, + TRUE AS report_matches_payout, + 'reconciled' AS reconciliation_status, + CAST(NULL AS INT) AS days_since_previous_payout + UNION ALL + SELECT + 'PO2' AS payout_id, + 5000 AS lines_net_minor, + 1 AS charge_line_count, + 0 AS refund_line_count, + 1 AS orphan_line_count, + 0 AS embedded_line_count, + 1 AS loaded_line_count, + 0 AS matched_payment_minor, + 'R2' AS report_id, + 5000 AS report_net_minor, + TRUE AS lines_match_payout, + TRUE AS report_matches_payout, + 'orphan_lines' AS reconciliation_status, + 2 AS days_since_previous_payout + UNION ALL + SELECT + 'PO3' AS payout_id, + 1000 AS lines_net_minor, + 1 AS charge_line_count, + 0 AS refund_line_count, + 0 AS orphan_line_count, + 0 AS embedded_line_count, + 1 AS loaded_line_count, + 1000 AS matched_payment_minor, + CAST(NULL AS VARCHAR) AS report_id, + CAST(NULL AS INT) AS report_net_minor, + TRUE AS lines_match_payout, + FALSE AS report_matches_payout, + 'awaiting_report' AS reconciliation_status, + 1 AS days_since_previous_payout + UNION ALL + SELECT + 'PO4' AS payout_id, + 2000 AS lines_net_minor, + 1 AS charge_line_count, + 0 AS refund_line_count, + 0 AS orphan_line_count, + 0 AS embedded_line_count, + 1 AS loaded_line_count, + 2000 AS matched_payment_minor, + 'R4' AS report_id, + 2100 AS report_net_minor, + TRUE AS lines_match_payout, + FALSE AS report_matches_payout, + 'report_mismatch' AS reconciliation_status, + 1 AS days_since_previous_payout +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_revenue_recognition_daily__recognises_revenue_on_regional_delivery_dates.sql b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_revenue_recognition_daily__recognises_revenue_on_regional_delivery_dates.sql new file mode 100644 index 0000000000..2b70bf4f6b --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_revenue_recognition_daily__recognises_revenue_on_regional_delivery_dates.sql @@ -0,0 +1,323 @@ +TEST (name "int_revenue_recognition_daily__recognises_revenue_on_regional_delivery_dates"); + +WITH __ref__int_orders_usd AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + 'EU' AS region_code, + NULL AS currency_code, + NULL AS placed_at, + NULL AS placed_date_local, + NULL AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS amount, + 119.0::DECIMAL(18, 2) AS amount_usd, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + 19.0::DECIMAL(18, 2) AS tax_amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS rate_age_days, + NULL AS is_rate_stale, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size + UNION ALL + SELECT + 'O2', + NULL, + 'US', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 50.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + NULL, + 'EU', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 70.0::DECIMAL(18, 2), + NULL, + NULL, + 0.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O4', + NULL, + 'EU', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 10.0::DECIMAL(18, 2), + NULL, + NULL, + 0.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O5', + NULL, + 'EU', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 40.0::DECIMAL(18, 2), + NULL, + NULL, + 0.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_order_status_timeline AS ( + SELECT + 'O1' AS order_id, + NULL AS transition_count, + NULL AS first_paid_at, + NULL AS first_shipped_at, + '2026-03-30 23:30:00 +00:00'::TIMESTAMP_TZ AS first_delivered_at, + NULL AS cancelled_at, + NULL AS minutes_to_pay, + NULL AS hours_paid_to_shipped, + NULL AS status_path, + NULL AS has_regression, + NULL AS last_status, + NULL AS longest_status + UNION ALL + SELECT + 'O2', + NULL, + NULL, + NULL, + '2026-03-31 02:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O4', + NULL, + NULL, + NULL, + '2026-04-02 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O5', + NULL, + NULL, + NULL, + '2026-03-25 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_refunds_usd AS ( + SELECT + 'F1' AS refund_id, + NULL AS payment_id, + 'O1' AS order_id, + NULL AS currency_code, + NULL AS refunded_at, + '2026-03-31'::DATE AS refunded_date, + NULL AS reason_category, + NULL AS amount, + 20.0::DECIMAL(18, 2) AS amount_usd, + NULL AS days_after_payment, + NULL AS is_full_refund + UNION ALL + SELECT + 'F2', + NULL, + 'O9', + NULL, + NULL, + '2026-03-31'::DATE, + NULL, + NULL, + 5.0::DECIMAL(18, 2), + NULL, + NULL +), + +__ref__int_subscription_mrr_daily AS ( + SELECT + 'SUB1:2026-03-31' AS subscription_day_key, + NULL AS subscription_id, + 'C1' AS customer_id, + '2026-03-31'::DATE AS calendar_date, + 36.5::DECIMAL(18, 4) AS mrr_usd, + NULL AS is_first_day + UNION ALL + SELECT + 'SUB2:2026-03-31', + NULL, + 'C2', + '2026-03-31'::DATE, + 365.0::DECIMAL(18, 4), + NULL +), + +__ref__stg_customers AS ( + SELECT + 'C1' AS customer_id, + NULL AS email_normalised, + NULL AS phone_normalised, + NULL AS first_name, + NULL AS last_name, + 'EU' AS region_code, + NULL AS storefront_code, + NULL AS created_at, + NULL AS preferred_language, + NULL AS marketing_opt_in, + NULL AS consent_types, + NULL AS birth_year, + NULL AS preferences, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_revenue_recognition_daily AS ( + SELECT + '2026-03-25:EU:product' AS recognition_key, + 40.0::DECIMAL(18, 2) AS revenue_usd, + 1 AS document_count, + 40.0::DECIMAL(18, 2) AS month_to_date_usd, + 40.0::DECIMAL(18, 2) AS trailing_7d_usd, + 1.0::DECIMAL(9, 6) AS share_of_day + UNION ALL + SELECT + '2026-03-30:US:product' AS recognition_key, + 50.0::DECIMAL(18, 2) AS revenue_usd, + 1 AS document_count, + 50.0::DECIMAL(18, 2) AS month_to_date_usd, + 50.0::DECIMAL(18, 2) AS trailing_7d_usd, + 1.0::DECIMAL(9, 6) AS share_of_day + UNION ALL + SELECT + '2026-03-31:EU:product' AS recognition_key, + 100.0::DECIMAL(18, 2) AS revenue_usd, + 1 AS document_count, + 140.0::DECIMAL(18, 2) AS month_to_date_usd, + 140.0::DECIMAL(18, 2) AS trailing_7d_usd, + 1.231527::DECIMAL(9, 6) AS share_of_day + UNION ALL + SELECT + '2026-03-31:EU:refund' AS recognition_key, + -20.0::DECIMAL(18, 2) AS revenue_usd, + 1 AS document_count, + -20.0::DECIMAL(18, 2) AS month_to_date_usd, + -20.0::DECIMAL(18, 2) AS trailing_7d_usd, + -0.246305::DECIMAL(9, 6) AS share_of_day + UNION ALL + SELECT + '2026-03-31:EU:subscription' AS recognition_key, + 1.2::DECIMAL(18, 2) AS revenue_usd, + 1 AS document_count, + 1.2::DECIMAL(18, 2) AS month_to_date_usd, + 1.2::DECIMAL(18, 2) AS trailing_7d_usd, + 0.014778::DECIMAL(9, 6) AS share_of_day + UNION ALL + SELECT + '2026-03-31:UNKNOWN:refund' AS recognition_key, + -5.0::DECIMAL(18, 2) AS revenue_usd, + 1 AS document_count, + -5.0::DECIMAL(18, 2) AS month_to_date_usd, + -5.0::DECIMAL(18, 2) AS trailing_7d_usd, + -0.714286::DECIMAL(9, 6) AS share_of_day + UNION ALL + SELECT + '2026-03-31:UNKNOWN:subscription' AS recognition_key, + 12.0::DECIMAL(18, 2) AS revenue_usd, + 1 AS document_count, + 12.0::DECIMAL(18, 2) AS month_to_date_usd, + 12.0::DECIMAL(18, 2) AS trailing_7d_usd, + 1.714286::DECIMAL(9, 6) AS share_of_day +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_settlement_lines__matches_lines_by_structured_or_described_reference.sql b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_settlement_lines__matches_lines_by_structured_or_described_reference.sql new file mode 100644 index 0000000000..cc4643f9cc --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_settlement_lines__matches_lines_by_structured_or_described_reference.sql @@ -0,0 +1,160 @@ +TEST (name "int_settlement_lines__matches_lines_by_structured_or_described_reference"); + +WITH __ref__stg_settlement_reports AS ( + SELECT + 'SR1' AS report_id, + 'PAYCO' AS provider_code, + '2026-03-10'::DATE AS report_date, + NULL AS currency_code, + NULL AS gross_minor, + NULL AS fees_minor, + NULL AS line_count, + OBJECT_CONSTRUCT( + 'lines', + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'type', + 'Charge', + 'reference', + '771001', + 'gross_minor', + 10000, + 'fee_minor', + 290 + ), + OBJECT_CONSTRUCT( + 'record_type', + 'refund', + 'description', + 'Refund for pi_XYZ7890 order', + 'gross', + '-25.50' + ), + OBJECT_CONSTRUCT( + 'reference', + ' ', + 'description', + 'adjustment', + 'gross_minor', + 100 + ), + OBJECT_CONSTRUCT( + 'type', + 'refund', + 'reference', + 'RF1', + 'gross_minor', + -500, + 'fee_minor', + 0 + ) + ) + )::VARIANT AS report + UNION ALL + SELECT + 'SR2', + 'PAYCO', + '2026-03-11'::DATE, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('{}')::VARIANT +), + +__ref__stg_payments AS ( + SELECT + 'PM1' AS payment_id, + NULL AS order_id, + NULL AS payment_method, + 10000 AS amount_minor, + NULL AS currency_code, + NULL AS authorised_at, + NULL AS captured_at, + NULL AS card_bin, + NULL AS gateway_status, + NULL AS gateway_error_code, + NULL AS risk_score, + NULL AS three_ds_outcome, + 771001 AS gateway_reference, + NULL AS gateway_response, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_refunds AS ( + SELECT + 'RF1' AS refund_id, + NULL AS payment_id, + 500 AS amount_minor, + NULL AS reason_text, + NULL AS reason_category, + NULL AS refunded_at, + NULL AS initiated_by, + NULL AS refunded_line_count, + NULL AS refunded_items, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_settlement_lines AS ( + SELECT + 'SR1:00000' AS settlement_line_key, + 1 AS line_number, + 'charge' AS line_type, + '771001' AS payment_reference, + 'structured' AS reference_source, + 10000 AS gross_minor, + 290 AS fee_minor, + 9710 AS net_minor, + 'PM1' AS matched_payment_id, + CAST(NULL AS VARCHAR) AS matched_refund_id, + TRUE AS is_matched, + TRUE AS amount_matches, + 0.029::DECIMAL(9, 6) AS fee_rate + UNION ALL + SELECT + 'SR1:00001' AS settlement_line_key, + 2 AS line_number, + 'refund' AS line_type, + 'pi_XYZ7890' AS payment_reference, + 'description' AS reference_source, + -2550 AS gross_minor, + 0 AS fee_minor, + -2550 AS net_minor, + CAST(NULL AS VARCHAR) AS matched_payment_id, + CAST(NULL AS VARCHAR) AS matched_refund_id, + FALSE AS is_matched, + FALSE AS amount_matches, + 0.0::DECIMAL(9, 6) AS fee_rate + UNION ALL + SELECT + 'SR1:00002' AS settlement_line_key, + 3 AS line_number, + 'unknown' AS line_type, + CAST(NULL AS VARCHAR) AS payment_reference, + 'none' AS reference_source, + 100 AS gross_minor, + 0 AS fee_minor, + 100 AS net_minor, + CAST(NULL AS VARCHAR) AS matched_payment_id, + CAST(NULL AS VARCHAR) AS matched_refund_id, + FALSE AS is_matched, + FALSE AS amount_matches, + 0.0::DECIMAL(9, 6) AS fee_rate + UNION ALL + SELECT + 'SR1:00003' AS settlement_line_key, + 4 AS line_number, + 'refund' AS line_type, + 'RF1' AS payment_reference, + 'structured' AS reference_source, + -500 AS gross_minor, + 0 AS fee_minor, + -500 AS net_minor, + CAST(NULL AS VARCHAR) AS matched_payment_id, + 'RF1' AS matched_refund_id, + TRUE AS is_matched, + TRUE AS amount_matches, + 0.0::DECIMAL(9, 6) AS fee_rate +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_tax_applied__compares_charged_tax_with_rate_valid_on_utc_issue_date.sql b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_tax_applied__compares_charged_tax_with_rate_valid_on_utc_issue_date.sql new file mode 100644 index 0000000000..7eeb11285f --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/finance/test_int_tax_applied__compares_charged_tax_with_rate_valid_on_utc_issue_date.sql @@ -0,0 +1,218 @@ +TEST (name "int_tax_applied__compares_charged_tax_with_rate_valid_on_utc_issue_date"); + +WITH __ref__stg_invoice_tax_lines AS ( + SELECT + 'IV1#1' AS invoice_tax_line_key, + 'IV1' AS invoice_id, + 'vat_reduced' AS tax_code, + 7.0::DECIMAL(9, 4) AS tax_rate_pct, + 1000 AS taxable_minor, + 70 AS tax_minor + UNION ALL + SELECT + 'IV1#2', + 'IV1', + 'VAT', + 16.0::DECIMAL(9, 4), + 1000, + 160 + UNION ALL + SELECT + 'IV2#1', + 'IV2', + 'GST-std', + 10.0::DECIMAL(9, 4), + 1000, + 100 + UNION ALL + SELECT + 'IV9#1', + 'IV9', + 'VAT', + 1.0::DECIMAL(9, 4), + 10, + 1 +), + +__ref__stg_invoices AS ( + SELECT + 'IV1' AS invoice_id, + 'O1' AS order_id, + NULL AS invoice_number, + NULL AS invoice_series, + '2026-01-01 00:30:00 +01:00'::TIMESTAMP_TZ AS issued_at, + NULL AS currency_code, + NULL AS net_minor, + NULL AS tax_minor, + NULL AS gross_minor, + NULL AS tax_lines, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'IV2', + 'O2', + NULL, + NULL, + '2026-01-01 00:30:00 +01:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_orders_enriched AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS storefront_code, + NULL AS region_code, + NULL AS storefront_timezone, + NULL AS currency_code, + NULL AS placed_at, + NULL AS placed_date_local, + NULL AS order_status, + NULL AS sales_channel, + NULL AS line_count, + NULL AS units, + NULL AS gross_amount_minor, + NULL AS discount_amount_minor, + NULL AS net_amount_minor, + NULL AS tax_amount_minor, + NULL AS captured_amount_minor, + NULL AS outstanding_amount_minor, + NULL AS promo_code, + 'DE' AS ship_to_country_code, + NULL AS is_cross_border, + NULL AS is_first_party_customer, + NULL AS basket_size, + NULL AS is_gift, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS total_weight_kg + UNION ALL + SELECT + 'O2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'AU', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_tax_rates AS ( + SELECT + 'T1' AS tax_rate_id, + 'DE' AS country_code, + NULL AS tax_region, + 'Standard' AS tax_category, + 19.0::NUMBER(9, 4) AS rate_pct, + NULL AS rate_fraction, + '2020-01-01'::DATE AS valid_from, + NULL AS valid_to, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + 'DE', + NULL, + 'Reduced', + 7.0::NUMBER(9, 4), + NULL, + '2020-01-01'::DATE, + '2026-01-01'::DATE, + NULL, + NULL + UNION ALL + SELECT + 'T3', + 'DE', + NULL, + 'Reduced', + 5.0::NUMBER(9, 4), + NULL, + '2026-01-01'::DATE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T4', + 'AU', + NULL, + 'standard', + 10.0::NUMBER(9, 4), + NULL, + '2020-01-01'::DATE, + NULL, + NULL, + NULL +), + +__expected__int_tax_applied AS ( + SELECT + 'IV1#1' AS invoice_tax_line_key, + 'VAT_REDUCED' AS tax_code, + 'reduced' AS tax_category, + 'DE' AS ship_to_country_code, + '2025-12-31'::DATE AS issued_date, + 7.0::DECIMAL(9, 4) AS published_rate_pct, + 70 AS expected_tax_minor, + 0 AS variance_minor, + TRUE AS has_published_rate, + FALSE AS is_rate_mismatch, + 1 AS candidate_rate_count + UNION ALL + SELECT + 'IV1#2' AS invoice_tax_line_key, + 'VAT' AS tax_code, + 'standard' AS tax_category, + 'DE' AS ship_to_country_code, + '2025-12-31'::DATE AS issued_date, + 19.0::DECIMAL(9, 4) AS published_rate_pct, + 190 AS expected_tax_minor, + -30 AS variance_minor, + TRUE AS has_published_rate, + TRUE AS is_rate_mismatch, + 1 AS candidate_rate_count + UNION ALL + SELECT + 'IV2#1' AS invoice_tax_line_key, + 'GST-STD' AS tax_code, + 'std' AS tax_category, + 'AU' AS ship_to_country_code, + '2025-12-31'::DATE AS issued_date, + CAST(NULL AS DECIMAL(9, 4)) AS published_rate_pct, + CAST(NULL AS INT) AS expected_tax_minor, + CAST(NULL AS INT) AS variance_minor, + FALSE AS has_published_rate, + FALSE AS is_rate_mismatch, + 0 AS candidate_rate_count +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_carrier_webhook_events__keeps_first_delivery_of_each_carrier_event.sql b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_carrier_webhook_events__keeps_first_delivery_of_each_carrier_event.sql new file mode 100644 index 0000000000..50726fb4b5 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_carrier_webhook_events__keeps_first_delivery_of_each_carrier_event.sql @@ -0,0 +1,88 @@ +TEST (name "int_carrier_webhook_events__keeps_first_delivery_of_each_carrier_event"); + +WITH __ref__stg_carrier_webhooks AS ( + SELECT + 'U1' AS carrier_update_key, + NULL AS webhook_id, + 'FLAPWING' AS carrier_code, + '2026-03-10 09:05:00'::TIMESTAMP_NTZ AS received_at, + 'T1' AS tracking_number, + 'DELIVERED' AS status_code, + '2026-03-10 10:00:00 +01:00'::TIMESTAMP_TZ AS status_at, + 1::DECIMAL(38, 0) AS carrier_event_id, + FALSE AS is_batched + UNION ALL + SELECT + 'U2', + NULL, + 'FLAPWING', + '2026-03-10 09:10:00'::TIMESTAMP_NTZ, + 'T1', + 'DELIVERED', + '2026-03-10 10:00:00 +01:00'::TIMESTAMP_TZ, + 1::DECIMAL(38, 0), + FALSE + UNION ALL + SELECT + 'U3', + NULL, + 'FLAPWING', + '2026-03-10 12:30:00'::TIMESTAMP_NTZ, + 'T1', + 'OFD', + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + TRUE + UNION ALL + SELECT + 'U4', + NULL, + 'FLAPWING', + '2026-03-10 12:30:00'::TIMESTAMP_NTZ, + NULL, + 'OFD', + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE + UNION ALL + SELECT + 'U5', + NULL, + 'FLAPWING', + '2026-03-10 12:30:00'::TIMESTAMP_NTZ, + 'T9', + 'OFD', + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE +), + +__ref__stg_shipments AS ( + SELECT + 'S1' AS shipment_id, + NULL AS order_id, + NULL AS warehouse_code, + 'FLAPWING' AS carrier_code, + NULL AS service_level, + 'T1' AS tracking_number, + NULL AS shipped_at, + NULL AS delivered_at, + NULL AS tracking_event_count, + NULL AS tracking_events, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_carrier_webhook_events AS ( + SELECT + 'U1' AS carrier_update_key, + 'S1' AS shipment_id, + 5 AS receive_lag_minutes, + FALSE AS is_batched + UNION ALL + SELECT + 'U3' AS carrier_update_key, + 'S1' AS shipment_id, + 30 AS receive_lag_minutes, + TRUE AS is_batched +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_delivery_exceptions_classified__assigns_responsibility_and_converts_compensation.sql b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_delivery_exceptions_classified__assigns_responsibility_and_converts_compensation.sql new file mode 100644 index 0000000000..0b17c4b150 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_delivery_exceptions_classified__assigns_responsibility_and_converts_compensation.sql @@ -0,0 +1,171 @@ +TEST (name "int_delivery_exceptions_classified__assigns_responsibility_and_converts_compensation"); + +WITH __ref__stg_delivery_exceptions AS ( + SELECT + 'X1' AS exception_id, + 'S1' AS shipment_id, + 'ADDR_INVALID' AS exception_code, + NULL AS exception_text, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS occurred_at, + NULL AS resolution_type, + '2026-03-01 11:30:00 +00:00'::TIMESTAMP_TZ AS resolved_at, + 500 AS compensation_minor, + NULL AS resolution, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'X2', + 'S1', + 'LOST_PARCEL', + NULL, + '2026-03-02 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'X3', + 'S2', + 'WRONG_ITEM', + NULL, + '2026-03-03 08:00:00 +09:00'::TIMESTAMP_TZ, + NULL, + '2026-03-03 10:15:00 +09:00'::TIMESTAMP_TZ, + 1000, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'X4', + 'S2', + 'WEATHER', + NULL, + '2026-03-04 08:00:00 +09:00'::TIMESTAMP_TZ, + NULL, + NULL, + 0, + NULL, + NULL, + NULL +), + +__ref__stg_shipments AS ( + SELECT + 'S1' AS shipment_id, + 'O1' AS order_id, + NULL AS warehouse_code, + 'FLAPWING' AS carrier_code, + NULL AS service_level, + NULL AS tracking_number, + NULL AS shipped_at, + NULL AS delivered_at, + NULL AS tracking_event_count, + NULL AS tracking_events, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'S2', + 'O2', + NULL, + 'WINGBEAT', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_orders_usd AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS region_code, + 'EUR' AS currency_code, + NULL AS placed_at, + NULL AS placed_date_local, + NULL AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS amount, + NULL AS amount_usd, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + NULL AS tax_amount_usd, + 1.1::NUMBER(18, 8) AS rate_to_usd, + NULL AS rate_date, + NULL AS rate_age_days, + NULL AS is_rate_stale, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size + UNION ALL + SELECT + 'O2', + NULL, + NULL, + 'JPY', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 0.0067::NUMBER(18, 8), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_delivery_exceptions_classified AS ( + SELECT + 'X1' AS exception_id, + 'FLAPWING' AS carrier_code, + 'customer' AS responsibility, + 1.5::DECIMAL(10, 2) AS hours_to_resolve, + 5.5::DECIMAL(18, 2) AS compensation_usd, + 1 AS exception_rank_in_shipment + UNION ALL + SELECT + 'X2' AS exception_id, + 'FLAPWING' AS carrier_code, + 'carrier' AS responsibility, + CAST(NULL AS DECIMAL(10, 2)) AS hours_to_resolve, + CAST(NULL AS DECIMAL(18, 2)) AS compensation_usd, + 2 AS exception_rank_in_shipment + UNION ALL + SELECT + 'X3' AS exception_id, + 'WINGBEAT' AS carrier_code, + 'warehouse' AS responsibility, + 2.25::DECIMAL(10, 2) AS hours_to_resolve, + 6.7::DECIMAL(18, 2) AS compensation_usd, + 1 AS exception_rank_in_shipment + UNION ALL + SELECT + 'X4' AS exception_id, + 'WINGBEAT' AS carrier_code, + 'unknown' AS responsibility, + CAST(NULL AS DECIMAL(10, 2)) AS hours_to_resolve, + 0.0::DECIMAL(18, 2) AS compensation_usd, + 2 AS exception_rank_in_shipment +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_delivery_slot_utilisation__flags_holidays_and_compares_with_weekday_average.sql b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_delivery_slot_utilisation__flags_holidays_and_compares_with_weekday_average.sql new file mode 100644 index 0000000000..963ba2ad10 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_delivery_slot_utilisation__flags_holidays_and_compares_with_weekday_average.sql @@ -0,0 +1,148 @@ +TEST (name "int_delivery_slot_utilisation__flags_holidays_and_compares_with_weekday_average"); + +WITH __ref__stg_delivery_slots AS ( + SELECT + 'A' AS slot_id, + 'W1' AS warehouse_code, + '2026-04-06 08:00:00'::TIMESTAMP_NTZ AS slot_start_local, + NULL AS slot_end_local, + NULL AS slot_minutes, + 10 AS capacity, + 10 AS booked, + 0 AS remaining_capacity, + NULL AS excluded_postcode_prefixes, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'B', + 'W1', + '2026-04-06 10:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + 10, + 5, + 5, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C', + 'W1', + '2026-04-13 09:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + 20, + 5, + 15, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'D', + 'W2', + '2026-04-11 09:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + 4, + 1, + 3, + NULL, + NULL, + NULL +), + +__ref__stg_warehouses AS ( + SELECT + 'W1' AS warehouse_code, + NULL AS warehouse_name, + 'DE' AS country_code, + NULL AS iana_timezone, + NULL AS opens_local, + NULL AS closes_local, + NULL AS opening_minutes, + NULL AS supports_same_day, + NULL AS supports_cold_chain, + NULL AS max_parcel_kg, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'W2', + NULL, + 'ZZ', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__seed__countries AS ( + SELECT + 'DE' AS country_code, + NULL AS country_name, + 'EU' AS region_code, + NULL AS default_timezone, + NULL AS currency_code, + NULL AS vat_registered +), + +__seed__holiday_calendar AS ( + SELECT + 'EU' AS region_code, + '2026-04-06'::DATE AS holiday_date, + 'Easter Monday' AS holiday_name + UNION ALL + SELECT + 'US', + '2026-04-13'::DATE, + 'Synthetic holiday' +), + +__expected__int_delivery_slot_utilisation AS ( + SELECT + 'W1|2026-04-06' AS slot_day_key, + 2 AS slot_count, + 20 AS total_capacity, + 15 AS total_booked, + 0.75::DECIMAL(9, 4) AS utilisation, + 1 AS full_slot_count, + '2026-04-06 08:00:00'::TIMESTAMPNTZ AS busiest_slot_start_local, + FALSE AS is_weekend, + 'EU' AS region_code, + TRUE AS is_holiday, + 0.25::DECIMAL(9, 4) AS utilisation_vs_weekday_avg + UNION ALL + SELECT + 'W1|2026-04-13' AS slot_day_key, + 1 AS slot_count, + 20 AS total_capacity, + 5 AS total_booked, + 0.25::DECIMAL(9, 4) AS utilisation, + 0 AS full_slot_count, + '2026-04-13 09:00:00'::TIMESTAMPNTZ AS busiest_slot_start_local, + FALSE AS is_weekend, + 'EU' AS region_code, + FALSE AS is_holiday, + -0.25::DECIMAL(9, 4) AS utilisation_vs_weekday_avg + UNION ALL + SELECT + 'W2|2026-04-11' AS slot_day_key, + 1 AS slot_count, + 4 AS total_capacity, + 1 AS total_booked, + 0.25::DECIMAL(9, 4) AS utilisation, + 0 AS full_slot_count, + '2026-04-11 09:00:00'::TIMESTAMPNTZ AS busiest_slot_start_local, + TRUE AS is_weekend, + CAST(NULL AS VARCHAR) AS region_code, + FALSE AS is_holiday, + 0.0::DECIMAL(9, 4) AS utilisation_vs_weekday_avg +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_fulfilment_sla__counts_business_days_across_easter_holidays.sql b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_fulfilment_sla__counts_business_days_across_easter_holidays.sql new file mode 100644 index 0000000000..53ed2a1bc7 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_fulfilment_sla__counts_business_days_across_easter_holidays.sql @@ -0,0 +1,233 @@ +TEST (name "int_fulfilment_sla__counts_business_days_across_easter_holidays"); + +WITH __ref__stg_shipments AS ( + SELECT + 'S1' AS shipment_id, + 'O1' AS order_id, + 'W1' AS warehouse_code, + 'FLAPWING' AS carrier_code, + NULL AS service_level, + NULL AS tracking_number, + '2026-04-07 07:00:00 +00:00'::TIMESTAMP_TZ AS shipped_at, + NULL AS delivered_at, + NULL AS tracking_event_count, + NULL AS tracking_events, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'S2', + 'O2', + 'W1', + 'FLAPWING', + NULL, + NULL, + '2026-04-08 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'S3', + 'O3', + 'W1', + 'FLAPWING', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_orders_enriched AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS storefront_code, + NULL AS region_code, + NULL AS storefront_timezone, + NULL AS currency_code, + '2026-04-03 08:00:00 +00:00'::TIMESTAMP_TZ AS placed_at, + NULL AS placed_date_local, + NULL AS order_status, + NULL AS sales_channel, + NULL AS line_count, + NULL AS units, + NULL AS gross_amount_minor, + NULL AS discount_amount_minor, + NULL AS net_amount_minor, + NULL AS tax_amount_minor, + NULL AS captured_amount_minor, + NULL AS outstanding_amount_minor, + NULL AS promo_code, + NULL AS ship_to_country_code, + NULL AS is_cross_border, + NULL AS is_first_party_customer, + NULL AS basket_size, + NULL AS is_gift, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS total_weight_kg + UNION ALL + SELECT + 'O2', + NULL, + NULL, + NULL, + NULL, + NULL, + '2026-04-08 06:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + NULL, + NULL, + NULL, + NULL, + NULL, + '2026-04-08 06:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_warehouses AS ( + SELECT + 'W1' AS warehouse_code, + NULL AS warehouse_name, + 'DE' AS country_code, + 'Europe/Berlin' AS iana_timezone, + NULL AS opens_local, + NULL AS closes_local, + 600 AS opening_minutes, + NULL AS supports_same_day, + NULL AS supports_cold_chain, + NULL AS max_parcel_kg, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__int_shipment_tracking_events AS ( + SELECT + 'E1' AS tracking_event_key, + 'S1' AS shipment_id, + NULL AS event_code, + NULL AS event_family, + '2026-04-07 20:00:00'::TIMESTAMPNTZ AS event_at_utc, + NULL AS event_at_warehouse_local, + NULL AS event_origin, + FALSE AS is_delivery_event + UNION ALL + SELECT + 'E2', + 'S1', + NULL, + NULL, + '2026-04-08 09:00:00'::TIMESTAMPNTZ, + NULL, + NULL, + TRUE + UNION ALL + SELECT + 'E3', + 'S1', + NULL, + NULL, + '2026-04-09 09:00:00'::TIMESTAMPNTZ, + NULL, + NULL, + TRUE +), + +__seed__countries AS ( + SELECT + 'DE' AS country_code, + NULL AS country_name, + 'EU' AS region_code, + NULL AS default_timezone, + NULL AS currency_code, + NULL AS vat_registered +), + +__seed__holiday_calendar AS ( + SELECT + 'EU' AS region_code, + '2026-04-03'::DATE AS holiday_date, + 'Good Friday' AS holiday_name + UNION ALL + SELECT + 'EU', + '2026-04-06'::DATE, + 'Easter Monday' +), + +__expected__int_fulfilment_sla AS ( + SELECT + 'S1' AS shipment_id, + '2026-04-03 10:00:00'::TIMESTAMPNTZ AS placed_at_local, + '2026-04-07 09:00:00'::TIMESTAMPNTZ AS shipped_at_local, + 4 AS calendar_days_to_ship, + 1 AS business_days_to_ship, + 1 AS holidays_in_window, + 600 AS dispatch_business_minutes, + FALSE AS is_dispatch_on_time, + '2026-04-08 09:00:00'::TIMESTAMP_NTZ AS delivered_at_utc, + 26.0::DECIMAL(10, 2) AS hours_to_deliver + UNION ALL + SELECT + 'S2' AS shipment_id, + '2026-04-08 08:00:00'::TIMESTAMPNTZ AS placed_at_local, + '2026-04-08 14:00:00'::TIMESTAMPNTZ AS shipped_at_local, + 0 AS calendar_days_to_ship, + 0 AS business_days_to_ship, + 0 AS holidays_in_window, + 360 AS dispatch_business_minutes, + TRUE AS is_dispatch_on_time, + CAST(NULL AS TIMESTAMP_NTZ) AS delivered_at_utc, + CAST(NULL AS DECIMAL(10, 2)) AS hours_to_deliver +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_pick_performance__ranks_pickers_by_rate_within_warehouse_day.sql b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_pick_performance__ranks_pickers_by_rate_within_warehouse_day.sql new file mode 100644 index 0000000000..ac224bea40 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_pick_performance__ranks_pickers_by_rate_within_warehouse_day.sql @@ -0,0 +1,178 @@ +TEST (name "int_pick_performance__ranks_pickers_by_rate_within_warehouse_day"); + +WITH __ref__stg_pick_tasks AS ( + SELECT + 'T1' AS task_id, + NULL AS order_id, + 'W1' AS warehouse_code, + 'P1' AS picker_id, + '2026-04-06 08:15:00'::TIMESTAMPNTZ AS started_at_local, + '2026-04-06 08:15:00'::TIMESTAMPNTZ AS completed_at_local, + 1800 AS pick_seconds, + NULL AS line_count, + 30 AS units_picked, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + NULL, + 'W1', + 'P1', + '2026-04-06 09:00:00'::TIMESTAMPNTZ, + '2026-04-06 09:00:00'::TIMESTAMPNTZ, + 1800, + NULL, + 30, + NULL, + NULL + UNION ALL + SELECT + 'T3', + NULL, + 'W1', + 'P2', + '2026-04-06 07:30:00'::TIMESTAMPNTZ, + '2026-04-06 07:30:00'::TIMESTAMPNTZ, + 3600, + NULL, + 90, + NULL, + NULL + UNION ALL + SELECT + 'T4', + NULL, + 'W1', + 'P2', + '2026-04-06 06:00:00'::TIMESTAMPNTZ, + NULL, + 600, + NULL, + 10, + NULL, + NULL + UNION ALL + SELECT + 'T5', + NULL, + 'W1', + 'P2', + '2026-04-06 06:30:00'::TIMESTAMPNTZ, + '2026-04-06 06:30:00'::TIMESTAMPNTZ, + 0, + NULL, + 10, + NULL, + NULL + UNION ALL + SELECT + 'T6', + NULL, + 'W1', + 'P3', + '2026-04-07 10:00:00'::TIMESTAMPNTZ, + '2026-04-07 10:00:00'::TIMESTAMPNTZ, + 600, + NULL, + 5, + NULL, + NULL + UNION ALL + SELECT + 'T7', + NULL, + 'W1', + 'P3', + '2026-04-07 11:00:00'::TIMESTAMPNTZ, + '2026-04-07 11:00:00'::TIMESTAMPNTZ, + 1200, + NULL, + 10, + NULL, + NULL + UNION ALL + SELECT + 'T8', + NULL, + 'W1', + 'P3', + '2026-04-07 12:00:00'::TIMESTAMPNTZ, + '2026-04-07 12:00:00'::TIMESTAMPNTZ, + 1800, + NULL, + 15, + NULL, + NULL +), + +__ref__stg_warehouses AS ( + SELECT + 'W1' AS warehouse_code, + NULL AS warehouse_name, + 'DE' AS country_code, + NULL AS iana_timezone, + NULL AS opens_local, + NULL AS closes_local, + 480 AS opening_minutes, + NULL AS supports_same_day, + NULL AS supports_cold_chain, + NULL AS max_parcel_kg, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__seed__countries AS ( + SELECT + 'DE' AS country_code, + NULL AS country_name, + 'EU' AS region_code, + NULL AS default_timezone, + NULL AS currency_code, + NULL AS vat_registered +), + +__seed__holiday_calendar AS ( + SELECT + 'EU' AS region_code, + '2026-04-06'::DATE AS holiday_date, + 'Easter Monday' AS holiday_name +), + +__expected__int_pick_performance AS ( + SELECT + 'P1|W1|2026-04-06' AS picker_day_key, + 2 AS task_count, + 60 AS units_picked, + 1.0::DECIMAL(10, 2) AS active_hours, + 60.0::DECIMAL(10, 2) AS units_per_hour, + 1800 AS p90_task_seconds, + 2 AS rank_in_warehouse, + 8 AS first_task_hour_local, + 0.125::DECIMAL(9, 4) AS shift_coverage, + TRUE AS is_holiday + UNION ALL + SELECT + 'P2|W1|2026-04-06' AS picker_day_key, + 1 AS task_count, + 90 AS units_picked, + 1.0::DECIMAL(10, 2) AS active_hours, + 90.0::DECIMAL(10, 2) AS units_per_hour, + 3600 AS p90_task_seconds, + 1 AS rank_in_warehouse, + 7 AS first_task_hour_local, + 0.125::DECIMAL(9, 4) AS shift_coverage, + TRUE AS is_holiday + UNION ALL + SELECT + 'P3|W1|2026-04-07' AS picker_day_key, + 3 AS task_count, + 30 AS units_picked, + 1.0::DECIMAL(10, 2) AS active_hours, + 30.0::DECIMAL(10, 2) AS units_per_hour, + 1800 AS p90_task_seconds, + 1 AS rank_in_warehouse, + 10 AS first_task_hour_local, + 0.125::DECIMAL(9, 4) AS shift_coverage, + FALSE AS is_holiday +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_return_outcomes__derives_outcome_from_latest_inspection_and_refunds.sql b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_return_outcomes__derives_outcome_from_latest_inspection_and_refunds.sql new file mode 100644 index 0000000000..0c981f4bd6 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_return_outcomes__derives_outcome_from_latest_inspection_and_refunds.sql @@ -0,0 +1,235 @@ +TEST (name "int_return_outcomes__derives_outcome_from_latest_inspection_and_refunds"); + +WITH __ref__int_return_reasons AS ( + SELECT + 'RT1' AS return_id, + 'O1' AS order_id, + 'DUCK-1' AS sku, + NULL AS qty, + NULL AS reason_text, + 'damaged' AS return_reason, + NULL AS classified_by, + NULL AS requested_at, + NULL AS category_level_1, + NULL AS ordered_qty, + NULL AS is_over_return, + NULL AS days_after_order + UNION ALL + SELECT + 'RT2', + 'O2', + 'DUCK-1', + NULL, + NULL, + 'size_fit', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'RT3', + 'O3', + 'DUCK-1', + NULL, + NULL, + 'other', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'RT4', + 'O4', + 'DUCK-1', + NULL, + NULL, + 'other', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_returns AS ( + SELECT + 'RT1' AS return_id, + NULL AS order_id, + NULL AS sku, + NULL AS qty, + NULL AS reason_text, + NULL AS return_status, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS requested_at, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_TZ AS received_at, + NULL AS photo_count, + NULL AS refund_preference, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'RT2', + NULL, + NULL, + NULL, + NULL, + NULL, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + '2026-03-03 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'RT3', + NULL, + NULL, + NULL, + NULL, + NULL, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'RT4', + NULL, + NULL, + NULL, + NULL, + NULL, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_return_inspections AS ( + SELECT + 'I1' AS inspection_id, + 'RT1' AS return_id, + NULL AS warehouse_code, + 'B' AS grade, + FALSE AS is_resellable, + '2026-03-05 12:00:00'::TIMESTAMP_NTZ AS inspected_at_local, + PARSE_JSON('["SCUFF"]')::ARRAY AS defect_codes, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'I2', + 'RT1', + NULL, + 'A', + TRUE, + '2026-03-06 12:00:00'::TIMESTAMP_NTZ, + PARSE_JSON('[]')::ARRAY, + NULL, + NULL + UNION ALL + SELECT + 'I3', + 'RT2', + NULL, + 'C', + FALSE, + '2026-03-04 12:00:00'::TIMESTAMP_NTZ, + PARSE_JSON('["SCRATCH", "DENT"]')::ARRAY, + NULL, + NULL +), + +__ref__int_refunds_usd AS ( + SELECT + 'F1' AS refund_id, + NULL AS payment_id, + 'O3' AS order_id, + NULL AS currency_code, + '2026-03-03 09:00:00 +00:00'::TIMESTAMP_TZ AS refunded_at, + NULL AS refunded_date, + 'return' AS reason_category, + NULL AS amount, + 10.0::DECIMAL(18, 2) AS amount_usd, + NULL AS days_after_payment, + NULL AS is_full_refund + UNION ALL + SELECT + 'F2', + NULL, + 'O3', + NULL, + '2026-03-04 09:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'return', + NULL, + 5.5::DECIMAL(18, 2), + NULL, + NULL + UNION ALL + SELECT + 'F3', + NULL, + 'O3', + NULL, + '2026-03-02 09:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'fraud', + NULL, + 100.0::DECIMAL(18, 2), + NULL, + NULL +), + +__expected__int_return_outcomes AS ( + SELECT + 'RT1' AS return_id, + 'A' AS grade, + TRUE AS is_resellable, + CAST(NULL AS DECIMAL(18, 2)) AS refund_usd, + 4 AS days_to_receive, + CAST(NULL AS INT) AS days_to_refund, + 'restocked' AS outcome, + 0 AS defect_count + UNION ALL + SELECT + 'RT2' AS return_id, + 'C' AS grade, + FALSE AS is_resellable, + CAST(NULL AS DECIMAL(18, 2)) AS refund_usd, + 2 AS days_to_receive, + CAST(NULL AS INT) AS days_to_refund, + 'written_off' AS outcome, + 2 AS defect_count + UNION ALL + SELECT + 'RT3' AS return_id, + CAST(NULL AS VARCHAR) AS grade, + CAST(NULL AS BOOLEAN) AS is_resellable, + 15.5::DECIMAL(18, 2) AS refund_usd, + CAST(NULL AS INT) AS days_to_receive, + 2 AS days_to_refund, + 'refunded_not_received' AS outcome, + 0 AS defect_count + UNION ALL + SELECT + 'RT4' AS return_id, + CAST(NULL AS VARCHAR) AS grade, + CAST(NULL AS BOOLEAN) AS is_resellable, + CAST(NULL AS DECIMAL(18, 2)) AS refund_usd, + CAST(NULL AS INT) AS days_to_receive, + CAST(NULL AS INT) AS days_to_refund, + 'pending' AS outcome, + 0 AS defect_count +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_return_reasons__classifies_free_text_reasons.sql b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_return_reasons__classifies_free_text_reasons.sql new file mode 100644 index 0000000000..9e0088b022 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_return_reasons__classifies_free_text_reasons.sql @@ -0,0 +1,207 @@ +TEST (name "int_return_reasons__classifies_free_text_reasons"); + +WITH __ref__stg_returns AS ( + SELECT + 'R1' AS return_id, + 'O1' AS order_id, + 'DUCK-1' AS sku, + 1 AS qty, + 'Arrived broken!!' AS reason_text, + NULL AS return_status, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS requested_at, + NULL AS received_at, + NULL AS photo_count, + NULL AS refund_preference, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'R2', + 'O1', + 'DUCK-2', + 3, + 'Too small', + NULL, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'R3', + 'O1', + 'DUCK-3', + 1, + NULL, + NULL, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'R4', + 'O2', + 'DUCK-1', + 1, + 'I don''t need it', + NULL, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'R5', + 'O1', + 'DUCK-1', + 1, + 'Wrong colour sent', + NULL, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_order_lines AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + NULL AS line_no, + 'DUCK-1' AS sku, + 2 AS quantity, + NULL AS unit_amount_minor, + NULL AS gross_amount_minor, + NULL AS tax_amount_minor, + NULL AS discount_count, + NULL AS discounts, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'L2', + 'O1', + NULL, + 'DUCK-2', + 1, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'L3', + 'O1', + NULL, + 'DUCK-2', + 1, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_orders AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS storefront_code, + NULL AS currency_code, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS placed_at, + NULL AS order_status, + NULL AS sales_channel, + NULL AS is_gift, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS device_type, + NULL AS item_count_hint, + NULL AS shipping_address_id, + NULL AS attributes, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__int_product_catalogue AS ( + SELECT + 'DUCK-1' AS sku, + NULL AS product_id, + NULL AS sku_family, + NULL AS title, + NULL AS variant_label, + NULL AS brand_code, + NULL AS brand_name, + NULL AS is_private_label, + NULL AS category_id, + NULL AS category_slug_path, + 'decoys' AS category_level_1, + NULL AS material, + NULL AS weight_kg, + NULL AS primary_image_url, + NULL AS image_count, + NULL AS is_active, + NULL AS launched_on, + NULL AS is_bundle_component +), + +__expected__int_return_reasons AS ( + SELECT + 'R1' AS return_id, + 'damaged' AS return_reason, + 'pattern' AS classified_by, + 'decoys' AS category_level_1, + 2 AS ordered_qty, + FALSE AS is_over_return, + 9 AS days_after_order + UNION ALL + SELECT + 'R2' AS return_id, + 'size_fit' AS return_reason, + 'pattern' AS classified_by, + CAST(NULL AS VARCHAR) AS category_level_1, + 2 AS ordered_qty, + TRUE AS is_over_return, + 9 AS days_after_order + UNION ALL + SELECT + 'R3' AS return_id, + 'other' AS return_reason, + 'default' AS classified_by, + CAST(NULL AS VARCHAR) AS category_level_1, + CAST(NULL AS INT) AS ordered_qty, + TRUE AS is_over_return, + 9 AS days_after_order + UNION ALL + SELECT + 'R4' AS return_id, + 'changed_mind' AS return_reason, + 'pattern' AS classified_by, + 'decoys' AS category_level_1, + CAST(NULL AS INT) AS ordered_qty, + TRUE AS is_over_return, + CAST(NULL AS INT) AS days_after_order + UNION ALL + SELECT + 'R5' AS return_id, + 'wrong_item' AS return_reason, + 'pattern' AS classified_by, + 'decoys' AS category_level_1, + 2 AS ordered_qty, + FALSE AS is_over_return, + 9 AS days_after_order +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_shipment_costs__picks_smallest_valid_band_for_chargeable_weight.sql b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_shipment_costs__picks_smallest_valid_band_for_chargeable_weight.sql new file mode 100644 index 0000000000..cfd9e4fe2e --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_shipment_costs__picks_smallest_valid_band_for_chargeable_weight.sql @@ -0,0 +1,169 @@ +TEST (name "int_shipment_costs__picks_smallest_valid_band_for_chargeable_weight"); + +WITH __ref__stg_shipment_packages AS ( + SELECT + 'P1' AS package_id, + 'S1' AS shipment_id, + 1500 AS weight_grams, + NULL AS length_cm, + NULL AS width_cm, + NULL AS height_cm, + 1.2::DECIMAL(10, 3) AS volumetric_weight_kg, + NULL AS content_line_count, + NULL AS contents, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'P2', + 'S1', + 500, + NULL, + NULL, + NULL, + 2.4::DECIMAL(10, 3), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'P3', + 'S1', + 12000, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_shipments AS ( + SELECT + 'S1' AS shipment_id, + 'O1' AS order_id, + 'W1' AS warehouse_code, + 'FLAPWING' AS carrier_code, + 'STANDARD' AS service_level, + NULL AS tracking_number, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS shipped_at, + NULL AS delivered_at, + NULL AS tracking_event_count, + NULL AS tracking_events, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_warehouses AS ( + SELECT + 'W1' AS warehouse_code, + NULL AS warehouse_name, + 'DE' AS country_code, + NULL AS iana_timezone, + NULL AS opens_local, + NULL AS closes_local, + NULL AS opening_minutes, + NULL AS supports_same_day, + NULL AS supports_cold_chain, + NULL AS max_parcel_kg, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__int_orders_enriched AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS storefront_code, + NULL AS region_code, + NULL AS storefront_timezone, + NULL AS currency_code, + NULL AS placed_at, + NULL AS placed_date_local, + NULL AS order_status, + NULL AS sales_channel, + NULL AS line_count, + NULL AS units, + NULL AS gross_amount_minor, + NULL AS discount_amount_minor, + NULL AS net_amount_minor, + NULL AS tax_amount_minor, + NULL AS captured_amount_minor, + NULL AS outstanding_amount_minor, + NULL AS promo_code, + 'FR' AS ship_to_country_code, + NULL AS is_cross_border, + NULL AS is_first_party_customer, + NULL AS basket_size, + NULL AS is_gift, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS total_weight_kg +), + +__ref__stg_carrier_rate_cards AS ( + SELECT + 'RC1' AS rate_card_id, + 'FLAPWING' AS carrier_code, + 'STANDARD' AS service_level, + 'DE' AS origin_country, + 'FR' AS destination_country, + 'EUR' AS currency_code, + '2026-01-01'::DATE AS valid_from, + '2026-12-31'::DATE AS valid_to, + NULL AS band_count, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT('max_kg', 2, 'amount_minor', 600), + OBJECT_CONSTRUCT('max_kg', 5, 'amount_minor', 900), + OBJECT_CONSTRUCT('max_kg', 10, 'amount_minor', 1500) + )::ARRAY AS bands, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'RC0', + 'FLAPWING', + 'STANDARD', + 'DE', + 'FR', + 'EUR', + '2025-01-01'::DATE, + '2025-12-31'::DATE, + NULL, + ARRAY_CONSTRUCT(OBJECT_CONSTRUCT('max_kg', 30, 'amount_minor', 100))::ARRAY, + NULL, + NULL +), + +__expected__int_shipment_costs AS ( + SELECT + 'P1' AS package_id, + 1.5::DECIMAL(10, 3) AS chargeable_weight_kg, + 'actual' AS weight_basis, + 2.0::DECIMAL(10, 3) AS band_max_kg, + 600 AS rate_minor, + 'EUR' AS rate_currency_code, + TRUE AS has_rate + UNION ALL + SELECT + 'P2' AS package_id, + 2.4::DECIMAL(10, 3) AS chargeable_weight_kg, + 'volumetric' AS weight_basis, + 5.0::DECIMAL(10, 3) AS band_max_kg, + 900 AS rate_minor, + 'EUR' AS rate_currency_code, + TRUE AS has_rate + UNION ALL + SELECT + 'P3' AS package_id, + 12.0::DECIMAL(10, 3) AS chargeable_weight_kg, + 'actual' AS weight_basis, + CAST(NULL AS DECIMAL(10, 3)) AS band_max_kg, + CAST(NULL AS INT) AS rate_minor, + CAST(NULL AS VARCHAR) AS rate_currency_code, + FALSE AS has_rate +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_shipment_tracking_events__prefers_carrier_feed_over_webhook_in_same_minute.sql b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_shipment_tracking_events__prefers_carrier_feed_over_webhook_in_same_minute.sql new file mode 100644 index 0000000000..cecab2b7d0 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/fulfilment/test_int_shipment_tracking_events__prefers_carrier_feed_over_webhook_in_same_minute.sql @@ -0,0 +1,141 @@ +TEST (name "int_shipment_tracking_events__prefers_carrier_feed_over_webhook_in_same_minute"); + +WITH __ref__stg_shipment_tracking_events AS ( + SELECT + 'E1' AS tracking_event_key, + 'S1' AS shipment_id, + 'carrier_api' AS event_source, + 'PICKED_UP' AS event_code, + '2026-03-10 09:00:00 +01:00'::TIMESTAMP_TZ AS event_at, + NULL AS location_text, + NULL AS location_country_code, + NULL AS event_sequence + UNION ALL + SELECT + 'E2', + 'S1', + 'carrier_api', + 'DELIVERED', + '2026-03-11 14:00:30 +01:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E3', + 'S1', + 'carrier_api', + 'IN_TRANSIT', + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E4', + 'S1', + 'carrier_api', + 'OFD', + '2026-03-11 08:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E5', + 'S9', + 'carrier_api', + 'OFD', + '2026-03-11 08:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL +), + +__ref__int_carrier_webhook_events AS ( + SELECT + 'U1' AS carrier_update_key, + 'S1' AS shipment_id, + NULL AS carrier_code, + 'DELIVERED' AS status_code, + '2026-03-11 13:00:10 +00:00'::TIMESTAMP_TZ AS status_at, + NULL AS received_at, + NULL AS receive_lag_minutes, + NULL AS is_batched + UNION ALL + SELECT + 'U2', + 'S1', + NULL, + 'EXCEPTION_WEATHER', + '2026-03-10 20:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL +), + +__ref__stg_shipments AS ( + SELECT + 'S1' AS shipment_id, + NULL AS order_id, + 'W1' AS warehouse_code, + NULL AS carrier_code, + NULL AS service_level, + NULL AS tracking_number, + NULL AS shipped_at, + NULL AS delivered_at, + NULL AS tracking_event_count, + NULL AS tracking_events, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_warehouses AS ( + SELECT + 'W1' AS warehouse_code, + NULL AS warehouse_name, + 'DE' AS country_code, + 'Europe/Berlin' AS iana_timezone, + NULL AS opens_local, + NULL AS closes_local, + 600 AS opening_minutes, + NULL AS supports_same_day, + NULL AS supports_cold_chain, + NULL AS max_parcel_kg, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_shipment_tracking_events AS ( + SELECT + 'E1' AS tracking_event_key, + 'collected' AS event_family, + '2026-03-10 08:00:00'::TIMESTAMPNTZ AS event_at_utc, + '2026-03-10 09:00:00'::TIMESTAMPNTZ AS event_at_warehouse_local, + 'carrier_api' AS event_origin, + FALSE AS is_delivery_event + UNION ALL + SELECT + 'E2' AS tracking_event_key, + 'delivered' AS event_family, + '2026-03-11 13:00:30'::TIMESTAMPNTZ AS event_at_utc, + '2026-03-11 14:00:30'::TIMESTAMPNTZ AS event_at_warehouse_local, + 'carrier_api' AS event_origin, + TRUE AS is_delivery_event + UNION ALL + SELECT + 'E4' AS tracking_event_key, + 'out_for_delivery' AS event_family, + '2026-03-11 08:00:00'::TIMESTAMPNTZ AS event_at_utc, + '2026-03-11 09:00:00'::TIMESTAMPNTZ AS event_at_warehouse_local, + 'carrier_api' AS event_origin, + FALSE AS is_delivery_event + UNION ALL + SELECT + 'W:U2' AS tracking_event_key, + 'exception' AS event_family, + '2026-03-10 20:00:00'::TIMESTAMPNTZ AS event_at_utc, + '2026-03-10 21:00:00'::TIMESTAMPNTZ AS event_at_warehouse_local, + 'webhook' AS event_origin, + FALSE AS is_delivery_event +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_affiliate_commissions_usd__pays_only_the_last_click_on_live_orders.sql b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_affiliate_commissions_usd__pays_only_the_last_click_on_live_orders.sql new file mode 100644 index 0000000000..63bcfbcdd0 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_affiliate_commissions_usd__pays_only_the_last_click_on_live_orders.sql @@ -0,0 +1,169 @@ +TEST (name "int_affiliate_commissions_usd__pays_only_the_last_click_on_live_orders"); + +WITH __ref__stg_affiliate_conversions AS ( + SELECT + 'CV1' AS conversion_id, + 'AWN-duck123' AS affiliate_code, + 'O1' AS order_id, + 500 AS commission_minor, + 'EUR' AS currency_code, + '2026-03-10 23:30:00 -02:00'::TIMESTAMP_TZ AS converted_at, + '2026-03-09 10:00:00 +00:00'::TIMESTAMP_TZ AS clicked_at, + NULL AS click_to_conversion_hours, + 'social_pf' AS sub_id, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'CV2', + 'creatorx', + 'O1', + 300, + 'EUR', + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ, + '2026-03-08 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'CV3', + 'PN_kw', + 'O2', + 1500, + 'KWD', + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ, + '2026-03-09 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + '123', + NULL, + NULL +), + +__ref__int_orders_usd AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS region_code, + NULL AS currency_code, + NULL AS placed_at, + NULL AS placed_date_local, + 'DELIVERED' AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS amount, + 110.0::DECIMAL(18, 2) AS amount_usd, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS rate_age_days, + NULL AS is_rate_stale, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size + UNION ALL + SELECT + 'O2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'CANCELLED', + NULL, + NULL, + NULL, + 100.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-03-10'::DATE AS rate_date, + 1.08::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'EUR', + '2026-03-11'::DATE, + 1.1::NUMBER(18, 8), + NULL, + NULL + UNION ALL + SELECT + 'KWD', + '2026-03-01'::DATE, + 3.25::NUMBER(18, 8), + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'KWD', + NULL, + 3, + NULL +), + +__expected__int_affiliate_commissions_usd AS ( + SELECT + 'CV1' AS conversion_id, + 'AWN' AS affiliate_network, + 'social' AS sub_channel, + 5.0::DECIMAL(38, 6) AS commission, + 5.5::DECIMAL(18, 2) AS commission_usd, + 0.05::DECIMAL(9, 6) AS commission_rate, + 2 AS claims_on_order, + TRUE AS is_last_click, + FALSE AS is_reversible, + 5.5::DECIMAL(18, 2) AS payable_commission_usd + UNION ALL + SELECT + 'CV2' AS conversion_id, + 'DIRECT' AS affiliate_network, + CAST(NULL AS VARCHAR) AS sub_channel, + 3.0::DECIMAL(38, 6) AS commission, + 3.24::DECIMAL(18, 2) AS commission_usd, + 0.029455::DECIMAL(9, 6) AS commission_rate, + 2 AS claims_on_order, + FALSE AS is_last_click, + FALSE AS is_reversible, + 0.0::DECIMAL(18, 2) AS payable_commission_usd + UNION ALL + SELECT + 'CV3' AS conversion_id, + 'PN' AS affiliate_network, + CAST(NULL AS VARCHAR) AS sub_channel, + 1.5::DECIMAL(38, 6) AS commission, + 4.88::DECIMAL(18, 2) AS commission_usd, + 0.04875::DECIMAL(9, 6) AS commission_rate, + 1 AS claims_on_order, + TRUE AS is_last_click, + TRUE AS is_reversible, + 0.0::DECIMAL(18, 2) AS payable_commission_usd +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_campaign_performance__joins_spend_touches_sessions_and_orders.sql b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_campaign_performance__joins_spend_touches_sessions_and_orders.sql new file mode 100644 index 0000000000..5e39206b8b --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_campaign_performance__joins_spend_touches_sessions_and_orders.sql @@ -0,0 +1,339 @@ +TEST (name "int_campaign_performance__joins_spend_touches_sessions_and_orders"); + +WITH __ref__stg_campaigns AS ( + SELECT + 'CMP1' AS campaign_id, + 'Spring' AS campaign_name, + 'paid_social' AS channel, + 'sales' AS objective, + '2026-03-01'::DATE AS starts_on, + NULL AS ends_on, + 100000 AS budget_minor, + 'EUR' AS currency_code, + PARSE_JSON('["EU", "LATAM", "US"]')::ARRAY AS target_regions, + NULL AS target_age_min, + NULL AS target_age_max, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'CMP2', + 'Mail', + 'email', + 'retention', + '2026-03-01'::DATE, + NULL, + 50000, + 'USD', + PARSE_JSON('[]')::ARRAY, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_campaign_spend_usd AS ( + SELECT + 'SP1' AS spend_id, + 'CMP1' AS campaign_id, + NULL AS campaign_name, + NULL AS channel, + NULL AS platform_account_id, + NULL AS spend_date, + NULL AS currency_code, + NULL AS spend_amount, + 300.0::DECIMAL(18, 2) AS spend_usd, + NULL AS impressions, + 40 AS clicks, + NULL AS conversions, + NULL AS click_through_rate, + NULL AS cost_per_click_usd, + NULL AS metric_count, + NULL AS unknown_metrics, + NULL AS is_in_flight + UNION ALL + SELECT + 'SP2', + 'CMP1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 250.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_email_engagement AS ( + SELECT + 'M1' AS message_id, + 'CMP2' AS campaign_id, + NULL AS recipient_email, + NULL AS sent_at, + NULL AS is_delivered, + NULL AS bounce_type, + NULL AS human_open_count, + 3 AS click_count, + NULL AS clicked_domains, + NULL AS unsubscribed, + NULL AS complained, + NULL AS minutes_to_first_click, + NULL AS customer_id, + NULL AS campaign_channel, + NULL AS is_campaign_in_flight + UNION ALL + SELECT + 'M2', + 'CMP2', + NULL, + NULL, + NULL, + NULL, + NULL, + 0, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'M3', + 'CMP1', + NULL, + NULL, + NULL, + NULL, + NULL, + 2, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_session_attribution AS ( + SELECT + 'S1' AS session_id, + NULL AS anonymous_id, + NULL AS customer_id, + NULL AS session_start_utc, + NULL AS utm_source, + NULL AS utm_medium, + 'cmp1' AS utm_campaign, + NULL AS channel_code, + NULL AS is_paid, + NULL AS first_touch_channel, + NULL AS last_touch_channel, + NULL AS touch_number + UNION ALL + SELECT + 'S2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'cmp1', + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'S3', + NULL, + NULL, + NULL, + NULL, + NULL, + 'other', + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_orders_usd AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS region_code, + NULL AS currency_code, + NULL AS placed_at, + NULL AS placed_date_local, + 'DELIVERED' AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS amount, + 100.0::DECIMAL(18, 2) AS amount_usd, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS rate_age_days, + NULL AS is_rate_stale, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + 'cmp1' AS utm_campaign, + NULL AS basket_size + UNION ALL + SELECT + 'O2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'CANCELLED', + NULL, + NULL, + NULL, + 50.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'cmp1', + NULL + UNION ALL + SELECT + 'O3', + NULL, + NULL, + NULL, + NULL, + NULL, + 'PENDING', + NULL, + NULL, + NULL, + 70.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'cmp2', + NULL + UNION ALL + SELECT + 'O4', + NULL, + NULL, + NULL, + NULL, + NULL, + 'SHIPPED', + NULL, + NULL, + NULL, + 40.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'cmp2', + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-02-28'::DATE AS rate_date, + 1.1::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'USD', + '2026-02-01'::DATE, + 1.0::NUMBER(18, 8), + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'USD', + NULL, + 2, + NULL +), + +__expected__int_campaign_performance AS ( + SELECT + 'CMP1' AS campaign_id, + 1100.0::DECIMAL(18, 2) AS budget_usd, + 550.0::DECIMAL(18, 2) AS spend_usd, + 50.0::DECIMAL(9, 2) AS budget_used_pct, + 40 AS paid_clicks, + 2 AS email_clicks, + 2 AS attributed_sessions, + 1 AS attributed_orders, + 100.0::DECIMAL(18, 2) AS attributed_revenue_usd, + 0.1818::DECIMAL(12, 4) AS return_on_ad_spend, + 3 AS target_region_count, + 1 AS spend_rank_in_channel + UNION ALL + SELECT + 'CMP2' AS campaign_id, + 500.0::DECIMAL(18, 2) AS budget_usd, + 0.0::DECIMAL(18, 2) AS spend_usd, + 0.0::DECIMAL(9, 2) AS budget_used_pct, + 0 AS paid_clicks, + 3 AS email_clicks, + 0 AS attributed_sessions, + 1 AS attributed_orders, + 40.0::DECIMAL(18, 2) AS attributed_revenue_usd, + CAST(NULL AS DECIMAL(12, 4)) AS return_on_ad_spend, + 0 AS target_region_count, + 1 AS spend_rank_in_channel +), + +__assert__unsupported_regions_are_listed AS ( + SELECT c.campaign_id + FROM __ref("int_campaign_performance") AS c + WHERE + ARRAY_TO_STRING(c.unsupported_target_regions, ',') <> IFF(c.campaign_id = 'CMP1', 'LATAM', '') +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_campaign_spend_usd__pivots_platform_metrics_and_rounds_half_even.sql b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_campaign_spend_usd__pivots_platform_metrics_and_rounds_half_even.sql new file mode 100644 index 0000000000..951f361f9f --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_campaign_spend_usd__pivots_platform_metrics_and_rounds_half_even.sql @@ -0,0 +1,149 @@ +TEST (name "int_campaign_spend_usd__pivots_platform_metrics_and_rounds_half_even"); + +WITH __ref__stg_campaign_spend AS ( + SELECT + 'SP1:spend' AS spend_metric_key, + 'SP1' AS spend_id, + '2026-03-10'::DATE AS spend_date, + NULL AS channel, + 'CMP1' AS campaign_id, + 'USD' AS currency_code, + 'spend' AS metric_name, + 10.125::DECIMAL(38, 6) AS metric_value, + 'ACC1' AS platform_account_id + UNION ALL + SELECT + 'SP1:impressions', + 'SP1', + '2026-03-10'::DATE, + NULL, + 'CMP1', + 'USD', + 'impressions', + 1000::DECIMAL(38, 6), + 'ACC1' + UNION ALL + SELECT + 'SP1:link_clicks', + 'SP1', + '2026-03-10'::DATE, + NULL, + 'CMP1', + 'USD', + 'link_clicks', + 20::DECIMAL(38, 6), + 'ACC1' + UNION ALL + SELECT + 'SP1:purchases_7d', + 'SP1', + '2026-03-10'::DATE, + NULL, + 'CMP1', + 'USD', + 'purchases_7d', + 2::DECIMAL(38, 6), + 'ACC1' + UNION ALL + SELECT + 'SP1:video_views', + 'SP1', + '2026-03-10'::DATE, + NULL, + 'CMP1', + 'USD', + 'video_views', + 500::DECIMAL(38, 6), + 'ACC1' + UNION ALL + SELECT + 'SP2:spend', + 'SP2', + '2026-02-10'::DATE, + 'search', + 'CMP9', + 'GBP', + 'spend', + 7.5::DECIMAL(38, 6), + 'ACC1' + UNION ALL + SELECT + 'SP2:impressions', + 'SP2', + '2026-02-10'::DATE, + 'search', + 'CMP9', + 'GBP', + 'impressions', + 0::DECIMAL(38, 6), + 'ACC1' +), + +__ref__stg_campaigns AS ( + SELECT + 'CMP1' AS campaign_id, + 'Spring' AS campaign_name, + 'paid_social' AS channel, + NULL AS objective, + '2026-03-01'::DATE AS starts_on, + NULL AS ends_on, + NULL AS budget_minor, + NULL AS currency_code, + NULL AS target_regions, + NULL AS target_age_min, + NULL AS target_age_max, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_fx_rates AS ( + SELECT + 'USD' AS currency_code, + '2026-01-01'::DATE AS rate_date, + 1.0::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'GBP', + '2026-02-01'::DATE, + 1.26::NUMBER(18, 8), + NULL, + NULL +), + +__expected__int_campaign_spend_usd AS ( + SELECT + 'SP1' AS spend_id, + 'Spring' AS campaign_name, + 'paid_social' AS channel, + 10.125::DECIMAL(38, 6) AS spend_amount, + 10.12::DECIMAL(18, 2) AS spend_usd, + 1000 AS impressions, + 20 AS clicks, + 2 AS conversions, + 0.02::DECIMAL(9, 6) AS click_through_rate, + 0.5063::DECIMAL(18, 4) AS cost_per_click_usd, + 5 AS metric_count, + TRUE AS is_in_flight + UNION ALL + SELECT + 'SP2' AS spend_id, + CAST(NULL AS VARCHAR) AS campaign_name, + 'search' AS channel, + 7.5::DECIMAL(38, 6) AS spend_amount, + 9.45::DECIMAL(18, 2) AS spend_usd, + 0 AS impressions, + CAST(NULL AS INT) AS clicks, + CAST(NULL AS INT) AS conversions, + CAST(NULL AS DECIMAL(9, 6)) AS click_through_rate, + CAST(NULL AS DECIMAL(18, 4)) AS cost_per_click_usd, + 2 AS metric_count, + FALSE AS is_in_flight +), + +__assert__unknown_metrics_are_listed AS ( + SELECT s.spend_id + FROM __ref("int_campaign_spend_usd") AS s + WHERE ARRAY_TO_STRING(s.unknown_metrics, ',') <> IFF(s.spend_id = 'SP1', 'video_views', '') +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_influencer_promo_orders__credits_orders_to_latest_post_with_the_code.sql b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_influencer_promo_orders__credits_orders_to_latest_post_with_the_code.sql new file mode 100644 index 0000000000..ecb3b2a6b0 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_influencer_promo_orders__credits_orders_to_latest_post_with_the_code.sql @@ -0,0 +1,260 @@ +TEST (name "int_influencer_promo_orders__credits_orders_to_latest_post_with_the_code"); + +WITH __ref__stg_influencer_posts AS ( + SELECT + 'P1' AS post_id, + '@Chef Ana' AS creator_handle, + 'Photo Feed' AS platform, + '2026-03-01 12:00:00 +00:00'::TIMESTAMP_TZ AS posted_at, + 'ana10' AS promo_code, + 10000 AS impressions, + 100 AS likes, + NULL AS engagement_rate, + OBJECT_CONSTRUCT( + 'comments', + 20, + 'shares', + 5, + 'saves', + 7, + 'link_url', + 'https://shop.example.com/?utm_source=ig' + )::OBJECT AS metrics, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'P2', + '@bob', + 'Short Video', + '2026-03-05 12:00:00 +00:00'::TIMESTAMP_TZ, + 'ANA10', + 0, + NULL, + NULL, + PARSE_JSON('{}')::OBJECT, + NULL, + NULL + UNION ALL + SELECT + 'P3', + '@dan', + 'Short Video', + '2026-03-05 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 10, + NULL, + NULL, + PARSE_JSON('{}')::OBJECT, + NULL, + NULL + UNION ALL + SELECT + 'P4', + '@cara', + 'Long Video', + '2026-03-01 12:00:00 +00:00'::TIMESTAMP_TZ, + 'CARA5', + 5000, + 10, + NULL, + PARSE_JSON('{"link_url": "shop.example.com"}')::OBJECT, + NULL, + NULL +), + +__ref__int_orders_usd AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + NULL AS region_code, + NULL AS currency_code, + '2026-03-02 18:00:00 +00:00'::TIMESTAMP_TZ AS placed_at, + NULL AS placed_date_local, + 'DELIVERED' AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS amount, + 50.0::DECIMAL(18, 2) AS amount_usd, + NULL AS gross_amount_usd, + 5.0::DECIMAL(18, 2) AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS rate_age_days, + NULL AS is_rate_stale, + 'Ana10' AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size + UNION ALL + SELECT + 'O2', + 'C2', + NULL, + NULL, + '2026-03-06 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'DELIVERED', + NULL, + NULL, + NULL, + 80.0::DECIMAL(18, 2), + NULL, + 8.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + 'ANA10', + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + 'C3', + NULL, + NULL, + '2026-02-28 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'DELIVERED', + NULL, + NULL, + NULL, + 10.0::DECIMAL(18, 2), + NULL, + 1.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + 'ANA10', + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O4', + 'C3', + NULL, + NULL, + '2026-03-02 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'CANCELLED', + NULL, + NULL, + NULL, + 10.0::DECIMAL(18, 2), + NULL, + 1.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + 'ANA10', + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O5', + 'C1', + NULL, + NULL, + '2026-03-04 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'DELIVERED', + NULL, + NULL, + NULL, + 20.0::DECIMAL(18, 2), + NULL, + 2.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + 'ANA10', + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O6', + 'C4', + NULL, + NULL, + '2026-03-20 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'DELIVERED', + NULL, + NULL, + NULL, + 30.0::DECIMAL(18, 2), + NULL, + 3.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + 'CARA5', + NULL, + NULL, + NULL, + NULL +), + +__expected__int_influencer_promo_orders AS ( + SELECT + 'P1' AS post_id, + 'chefana' AS creator_handle, + 'photo feed' AS platform, + 125 AS engagements, + 7 AS saves, + TRUE AS has_tracking_link, + 2 AS credited_orders, + 1 AS credited_customers, + 70.0::DECIMAL(18, 2) AS credited_revenue_usd, + 7.0::DECIMAL(18, 2) AS discount_given_usd, + 30.0::DECIMAL(10, 2) AS first_order_hours_after_post, + 0.2::DECIMAL(12, 4) AS orders_per_thousand_impressions + UNION ALL + SELECT + 'P2' AS post_id, + 'bob' AS creator_handle, + 'short video' AS platform, + 0 AS engagements, + CAST(NULL AS INT) AS saves, + FALSE AS has_tracking_link, + 1 AS credited_orders, + 1 AS credited_customers, + 80.0::DECIMAL(18, 2) AS credited_revenue_usd, + 8.0::DECIMAL(18, 2) AS discount_given_usd, + 12.0::DECIMAL(10, 2) AS first_order_hours_after_post, + CAST(NULL AS DECIMAL(12, 4)) AS orders_per_thousand_impressions + UNION ALL + SELECT + 'P4' AS post_id, + 'cara' AS creator_handle, + 'long video' AS platform, + 10 AS engagements, + CAST(NULL AS INT) AS saves, + FALSE AS has_tracking_link, + 0 AS credited_orders, + 0 AS credited_customers, + 0.0::DECIMAL(18, 2) AS credited_revenue_usd, + 0.0::DECIMAL(18, 2) AS discount_given_usd, + CAST(NULL AS DECIMAL(10, 2)) AS first_order_hours_after_post, + 0.0::DECIMAL(12, 4) AS orders_per_thousand_impressions +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_newsletter_status__takes_latest_status_per_list_and_flags_prospects.sql b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_newsletter_status__takes_latest_status_per_list_and_flags_prospects.sql new file mode 100644 index 0000000000..178b70a2fd --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_newsletter_status__takes_latest_status_per_list_and_flags_prospects.sql @@ -0,0 +1,121 @@ +TEST (name "int_newsletter_status__takes_latest_status_per_list_and_flags_prospects"); + +WITH __ref__stg_newsletter_subscriptions AS ( + SELECT + 'L1' AS list_subscription_id, + 'a@example.com' AS email, + 'weekly' AS list_code, + 'subscribed' AS list_status, + '2026-01-01 10:00:00 +00:00'::TIMESTAMP_TZ AS subscribed_at, + NULL AS unsubscribed_at, + 'footer' AS signup_form, + TRUE AS is_double_opt_in, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'L2', + 'a@example.com', + 'weekly', + 'unsubscribed', + '2026-01-01 10:00:00 +00:00'::TIMESTAMP_TZ, + '2026-02-01 10:00:00 +00:00'::TIMESTAMP_TZ, + 'footer', + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'L3', + 'a@example.com', + 'deals', + 'subscribed', + '2026-02-15 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'checkout', + NULL, + NULL, + NULL + UNION ALL + SELECT + 'L4', + 'info@shop.example.com', + 'weekly', + 'cleaned', + '2026-01-05 10:00:00 +00:00'::TIMESTAMP_TZ, + '2026-01-20 10:00:00 +00:00'::TIMESTAMP_TZ, + 'popup', + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'L5', + NULL, + 'weekly', + 'subscribed', + '2026-01-05 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_customers AS ( + SELECT + 'C1' AS customer_id, + 'a@example.com' AS email_normalised, + NULL AS phone_normalised, + NULL AS first_name, + NULL AS last_name, + NULL AS region_code, + NULL AS storefront_code, + NULL AS created_at, + NULL AS preferred_language, + NULL AS marketing_opt_in, + NULL AS consent_types, + NULL AS birth_year, + NULL AS preferences, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_newsletter_status AS ( + SELECT + 'a@example.com' AS email, + 'example.com' AS email_domain, + FALSE AS is_role_address, + 'C1' AS customer_id, + FALSE AS is_prospect, + 1 AS subscribed_list_count, + '2026-01-01 10:00:00 +00:00'::TIMESTAMPTZ AS first_subscribed_at, + '2026-02-01 10:00:00 +00:00'::TIMESTAMPTZ AS last_unsubscribed_at, + FALSE AS is_globally_unsubscribed, + TRUE AS has_double_opt_in, + 'footer' AS first_signup_form + UNION ALL + SELECT + 'info@shop.example.com' AS email, + 'shop.example.com' AS email_domain, + TRUE AS is_role_address, + CAST(NULL AS VARCHAR) AS customer_id, + TRUE AS is_prospect, + 0 AS subscribed_list_count, + '2026-01-05 10:00:00 +00:00'::TIMESTAMPTZ AS first_subscribed_at, + '2026-01-20 10:00:00 +00:00'::TIMESTAMPTZ AS last_unsubscribed_at, + TRUE AS is_globally_unsubscribed, + FALSE AS has_double_opt_in, + 'popup' AS first_signup_form +), + +__assert__list_statuses_reflect_latest_change AS ( + SELECT n.email + FROM __ref("int_newsletter_status") AS n + WHERE + n.email = 'a@example.com' + AND ( + n.list_status_by_list:weekly::VARCHAR <> 'unsubscribed' + OR ARRAY_TO_STRING(n.subscribed_lists, ',') <> 'deals' + ) +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_referral_conversions__qualifies_first_orders_inside_window_and_rejects_self_referrals.sql b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_referral_conversions__qualifies_first_orders_inside_window_and_rejects_self_referrals.sql new file mode 100644 index 0000000000..15eb9dc51a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_referral_conversions__qualifies_first_orders_inside_window_and_rejects_self_referrals.sql @@ -0,0 +1,289 @@ +TEST (name "int_referral_conversions__qualifies_first_orders_inside_window_and_rejects_self_referrals"); + +WITH __ref__stg_referral_invites AS ( + SELECT + 'I1' AS invite_id, + 'R1' AS referrer_customer_id, + 'new@example.com' AS invitee_email, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS sent_at, + '2026-03-01 13:30:00 +00:00'::TIMESTAMP_TZ AS accepted_at, + 1000 AS reward_amount_minor, + 'EUR' AS reward_currency_code, + NULL AS reward, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'I2', + 'R1', + 'self@example.com', + '2026-03-02 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 500, + 'GBP', + NULL, + NULL, + NULL + UNION ALL + SELECT + 'I3', + 'R2', + 'late@example.com', + '2026-01-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 1000, + 'JPY', + NULL, + NULL, + NULL + UNION ALL + SELECT + 'I4', + 'R2', + 'ghost@example.org', + '2026-01-02 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 2000, + 'USD', + NULL, + NULL, + NULL +), + +__ref__int_customer_identity AS ( + SELECT + 'C9' AS customer_id, + 'new@example.com' AS email_normalised, + NULL AS email_domain, + NULL AS email_dedupe_key, + NULL AS phone_normalised, + NULL AS full_name, + NULL AS anonymous_ids, + NULL AS anonymous_id_count, + NULL AS newsletter_lists, + NULL AS ticket_count, + NULL AS duplicate_candidate_id, + NULL AS duplicate_name_similarity, + NULL AS accounts_sharing_email + UNION ALL + SELECT + 'R1', + 'self@example.com', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C8', + 'late@example.com', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_customer_order_sequence AS ( + SELECT + 'O1' AS order_id, + 'C9' AS customer_id, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_TZ AS placed_at, + NULL AS placed_date_local, + NULL AS order_number, + NULL AS is_first_order, + NULL AS days_since_previous_order, + 25.0::NUMBER(18, 2) AS amount_usd, + NULL AS lifetime_revenue_usd, + NULL AS rolling_90d_revenue_usd, + NULL AS previous_sales_channel, + NULL AS is_channel_switch, + NULL AS cohort_month + UNION ALL + SELECT + 'O2', + 'C9', + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + 5.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + 'R1', + '2026-02-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + 50.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O4', + 'R1', + '2026-02-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + 15.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O5', + 'C8', + '2026-03-15 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + 30.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-02-28'::DATE AS rate_date, + 1.1::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'GBP', + '2026-03-01'::DATE, + 1.25::NUMBER(18, 8), + NULL, + NULL + UNION ALL + SELECT + 'JPY', + '2025-12-31'::DATE, + 0.0067::NUMBER(18, 8), + NULL, + NULL + UNION ALL + SELECT + 'USD', + '2026-01-01'::DATE, + 1.0::NUMBER(18, 8), + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'GBP', + NULL, + 2, + NULL + UNION ALL + SELECT + 'JPY', + NULL, + 0, + NULL + UNION ALL + SELECT + 'USD', + NULL, + 2, + NULL +), + +__expected__int_referral_conversions AS ( + SELECT + 'I1' AS invite_id, + 'C9' AS invitee_customer_id, + 'example.com' AS invitee_email_domain, + 3.5::DECIMAL(10, 2) AS hours_to_accept, + 'O1' AS first_order_id, + 25.0::DECIMAL(18, 2) AS first_order_usd, + 4 AS days_to_first_order, + FALSE AS is_self_referral, + TRUE AS is_qualified, + 11.0::DECIMAL(18, 2) AS reward_usd, + 1 AS invite_sequence_for_referrer, + 2 AS referrer_lifetime_orders + UNION ALL + SELECT + 'I2' AS invite_id, + 'R1' AS invitee_customer_id, + 'example.com' AS invitee_email_domain, + CAST(NULL AS DECIMAL(10, 2)) AS hours_to_accept, + 'O3' AS first_order_id, + 50.0::DECIMAL(18, 2) AS first_order_usd, + -29 AS days_to_first_order, + TRUE AS is_self_referral, + FALSE AS is_qualified, + 6.25::DECIMAL(18, 2) AS reward_usd, + 2 AS invite_sequence_for_referrer, + 2 AS referrer_lifetime_orders + UNION ALL + SELECT + 'I3' AS invite_id, + 'C8' AS invitee_customer_id, + 'example.com' AS invitee_email_domain, + CAST(NULL AS DECIMAL(10, 2)) AS hours_to_accept, + 'O5' AS first_order_id, + 30.0::DECIMAL(18, 2) AS first_order_usd, + 73 AS days_to_first_order, + FALSE AS is_self_referral, + FALSE AS is_qualified, + 6.7::DECIMAL(18, 2) AS reward_usd, + 1 AS invite_sequence_for_referrer, + 0 AS referrer_lifetime_orders + UNION ALL + SELECT + 'I4' AS invite_id, + CAST(NULL AS VARCHAR) AS invitee_customer_id, + 'example.org' AS invitee_email_domain, + CAST(NULL AS DECIMAL(10, 2)) AS hours_to_accept, + CAST(NULL AS VARCHAR) AS first_order_id, + CAST(NULL AS DECIMAL(18, 2)) AS first_order_usd, + CAST(NULL AS INT) AS days_to_first_order, + FALSE AS is_self_referral, + FALSE AS is_qualified, + 20.0::DECIMAL(18, 2) AS reward_usd, + 2 AS invite_sequence_for_referrer, + 0 AS referrer_lifetime_orders +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_survey_answers_flat__types_each_answer_of_the_latest_response.sql b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_survey_answers_flat__types_each_answer_of_the_latest_response.sql new file mode 100644 index 0000000000..31c677319e --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/growth/test_int_survey_answers_flat__types_each_answer_of_the_latest_response.sql @@ -0,0 +1,129 @@ +TEST (name "int_survey_answers_flat__types_each_answer_of_the_latest_response"); + +WITH __ref__stg_survey_responses AS ( + SELECT + 'SR0' AS response_id, + 'post_purchase' AS survey_code, + 'C1' AS customer_id, + '2026-02-01 10:00:00 +00:00'::TIMESTAMP_TZ AS submitted_at, + 3 AS nps_score, + NULL AS answer_count, + PARSE_JSON('{"q1": "old"}')::OBJECT AS answers, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'SR1', + 'post_purchase', + 'C1', + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + 9, + NULL, + OBJECT_CONSTRUCT_KEEP_NULL( + 'Overall Rating', + 4, + 'Would Recommend?', + TRUE, + 'Comments', + 'Great service', + 'Features', + ARRAY_CONSTRUCT('fast', 'cheap'), + 'Skip', + NULL, + 'Price', + '12.50' + )::OBJECT, + NULL, + NULL + UNION ALL + SELECT + 'SR2', + 'post_purchase', + 'C2', + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + 6, + NULL, + PARSE_JSON('{"q1": "no"}')::OBJECT, + NULL, + NULL +), + +__expected__int_survey_answers_flat AS ( + SELECT + 'SR1:comments' AS answer_key, + 'text' AS answer_kind, + CAST(NULL AS DECIMAL(18, 4)) AS answer_number, + CAST(NULL AS BOOLEAN) AS answer_flag, + 'Great service' AS answer_text, + TRUE AS is_answered, + 'promoter' AS nps_group, + 1 AS question_position + UNION ALL + SELECT + 'SR1:features' AS answer_key, + 'choices' AS answer_kind, + CAST(NULL AS DECIMAL(18, 4)) AS answer_number, + CAST(NULL AS BOOLEAN) AS answer_flag, + CAST(NULL AS VARCHAR) AS answer_text, + TRUE AS is_answered, + 'promoter' AS nps_group, + 2 AS question_position + UNION ALL + SELECT + 'SR1:overall_rating' AS answer_key, + 'number' AS answer_kind, + 4.0::DECIMAL(18, 4) AS answer_number, + CAST(NULL AS BOOLEAN) AS answer_flag, + CAST(NULL AS VARCHAR) AS answer_text, + TRUE AS is_answered, + 'promoter' AS nps_group, + 3 AS question_position + UNION ALL + SELECT + 'SR1:price' AS answer_key, + 'number' AS answer_kind, + 12.5::DECIMAL(18, 4) AS answer_number, + CAST(NULL AS BOOLEAN) AS answer_flag, + CAST(NULL AS VARCHAR) AS answer_text, + TRUE AS is_answered, + 'promoter' AS nps_group, + 4 AS question_position + UNION ALL + SELECT + 'SR1:skip' AS answer_key, + 'null' AS answer_kind, + CAST(NULL AS DECIMAL(18, 4)) AS answer_number, + CAST(NULL AS BOOLEAN) AS answer_flag, + CAST(NULL AS VARCHAR) AS answer_text, + FALSE AS is_answered, + 'promoter' AS nps_group, + 5 AS question_position + UNION ALL + SELECT + 'SR1:would_recommend_' AS answer_key, + 'flag' AS answer_kind, + CAST(NULL AS DECIMAL(18, 4)) AS answer_number, + TRUE AS answer_flag, + CAST(NULL AS VARCHAR) AS answer_text, + TRUE AS is_answered, + 'promoter' AS nps_group, + 6 AS question_position + UNION ALL + SELECT + 'SR2:q1' AS answer_key, + 'text' AS answer_kind, + CAST(NULL AS DECIMAL(18, 4)) AS answer_number, + CAST(NULL AS BOOLEAN) AS answer_flag, + 'no' AS answer_text, + TRUE AS is_answered, + 'detractor' AS nps_group, + 1 AS question_position +), + +__assert__choices_keep_array_order AS ( + SELECT a.answer_key + FROM __ref("int_survey_answers_flat") AS a + WHERE + a.answer_kind = 'choices' + AND ARRAY_TO_STRING(a.answer_choices, ',') <> 'fast,cheap' +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_inventory_ledger__orders_entries_in_utc_across_dst_change.sql b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_inventory_ledger__orders_entries_in_utc_across_dst_change.sql new file mode 100644 index 0000000000..459c0824d5 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_inventory_ledger__orders_entries_in_utc_across_dst_change.sql @@ -0,0 +1,128 @@ +TEST (name "int_inventory_ledger__orders_entries_in_utc_across_dst_change"); + +WITH __ref__stg_stock_movements AS ( + SELECT + 'M1' AS movement_id, + 'DUCK-1' AS sku, + 'W1' AS warehouse_code, + 10 AS qty_delta, + NULL AS reason_code, + '2026-03-29 03:30:00'::TIMESTAMPNTZ AS moved_at_local, + 'ADJ' AS reference_type, + NULL AS reference_id, + NULL AS operator_id + UNION ALL + SELECT + 'M2', + 'DUCK-1', + 'W1', + -3, + NULL, + '2026-03-29 01:30:00'::TIMESTAMPNTZ, + 'ADJ', + NULL, + NULL + UNION ALL + SELECT + 'M3', + 'DUCK-1', + 'W1', + 50, + NULL, + '2026-03-29 05:00:00'::TIMESTAMPNTZ, + 'PO', + NULL, + NULL +), + +__ref__stg_po_receipts AS ( + SELECT + 'R1' AS receipt_id, + 'PO1' AS po_id, + 'DUCK-1' AS sku, + 'W1' AS warehouse_code, + NULL AS received_qty, + NULL AS damaged_qty, + 5 AS sellable_qty, + '2026-03-30 09:00:00'::TIMESTAMP_NTZ AS received_at_local, + NULL AS discrepancy_codes, + NULL AS has_discrepancy, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_stock_transfers AS ( + SELECT + 'T1' AS transfer_id, + 'DUCK-1' AS sku, + 'W1' AS from_warehouse_code, + 'W2' AS to_warehouse_code, + 2 AS qty, + '2026-03-31 10:00:00 +00:00'::TIMESTAMP_TZ AS shipped_at, + NULL AS arrived_at, + NULL AS transit_hours, + NULL AS haulier_reference, + NULL AS tracking, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_warehouses AS ( + SELECT + 'W1' AS warehouse_code, + NULL AS warehouse_name, + NULL AS country_code, + 'Europe/Berlin' AS iana_timezone, + NULL AS opens_local, + NULL AS closes_local, + NULL AS opening_minutes, + NULL AS supports_same_day, + NULL AS supports_cold_chain, + NULL AS max_parcel_kg, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'W2', + NULL, + NULL, + 'Europe/London', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_inventory_ledger AS ( + SELECT + 'M:M2' AS ledger_entry_key, + '2026-03-29 00:30:00'::TIMESTAMPNTZ AS occurred_at_utc, + -3 AS running_on_hand, + 1 AS entry_sequence, + TRUE AS is_negative_balance + UNION ALL + SELECT + 'M:M1' AS ledger_entry_key, + '2026-03-29 01:30:00'::TIMESTAMPNTZ AS occurred_at_utc, + 7 AS running_on_hand, + 2 AS entry_sequence, + FALSE AS is_negative_balance + UNION ALL + SELECT + 'R:R1' AS ledger_entry_key, + '2026-03-30 07:00:00'::TIMESTAMPNTZ AS occurred_at_utc, + 12 AS running_on_hand, + 3 AS entry_sequence, + FALSE AS is_negative_balance + UNION ALL + SELECT + 'TO:T1' AS ledger_entry_key, + '2026-03-31 10:00:00'::TIMESTAMPNTZ AS occurred_at_utc, + 10 AS running_on_hand, + 4 AS entry_sequence, + FALSE AS is_negative_balance +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_inventory_valuation_daily__values_stock_and_tracks_daily_change.sql b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_inventory_valuation_daily__values_stock_and_tracks_daily_change.sql new file mode 100644 index 0000000000..f947b35cee --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_inventory_valuation_daily__values_stock_and_tracks_daily_change.sql @@ -0,0 +1,97 @@ +TEST (name "int_inventory_valuation_daily__values_stock_and_tracks_daily_change"); + +WITH __ref__int_stock_on_hand_daily AS ( + SELECT + 'DUCK-1|W1|2026-03-01' AS stock_day_key, + 'DUCK-1' AS sku, + 'W1' AS warehouse_code, + '2026-03-01'::DATE AS calendar_date, + NULL AS month_start, + 10 AS on_hand_qty, + NULL AS last_counted_qty, + NULL AS last_counted_date, + NULL AS days_since_count, + FALSE AS is_stocked_out + UNION ALL + SELECT + 'DUCK-2|W1|2026-03-01', + 'DUCK-2', + 'W1', + '2026-03-01'::DATE, + NULL, + -2, + NULL, + NULL, + NULL, + TRUE + UNION ALL + SELECT + 'DUCK-3|W1|2026-03-01', + 'DUCK-3', + 'W1', + '2026-03-01'::DATE, + NULL, + 4, + NULL, + NULL, + NULL, + FALSE + UNION ALL + SELECT + 'DUCK-1|W1|2026-03-02', + 'DUCK-1', + 'W1', + '2026-03-02'::DATE, + NULL, + 8, + NULL, + NULL, + NULL, + FALSE +), + +__ref__int_supplier_cost_current AS ( + SELECT + 'DUCK-1' AS sku, + NULL AS supplier_code, + NULL AS supplier_name, + NULL AS currency_code, + NULL AS unit_cost_minor, + 2.5::DECIMAL(18, 4) AS unit_cost_usd, + NULL AS effective_from, + NULL AS supplier_count, + NULL AS payment_terms_days, + NULL AS cost_spread_pct + UNION ALL + SELECT + 'DUCK-2', + NULL, + NULL, + NULL, + NULL, + 1.0::DECIMAL(18, 4), + NULL, + NULL, + NULL, + NULL +), + +__expected__int_inventory_valuation_daily AS ( + SELECT + 'W1|2026-03-01' AS valuation_key, + 3 AS sku_count, + 14 AS units_on_hand, + 25.0::DECIMAL(18, 2) AS stock_value_usd, + 4 AS unvalued_units, + 1 AS stocked_out_skus, + CAST(NULL AS DECIMAL(18, 2)) AS value_change_usd + UNION ALL + SELECT + 'W1|2026-03-02' AS valuation_key, + 1 AS sku_count, + 8 AS units_on_hand, + 20.0::DECIMAL(18, 2) AS stock_value_usd, + 0 AS unvalued_units, + 0 AS stocked_out_skus, + -5.0::DECIMAL(18, 2) AS value_change_usd +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_purchase_order_lines_received__measures_fill_and_lateness_in_warehouse_time.sql b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_purchase_order_lines_received__measures_fill_and_lateness_in_warehouse_time.sql new file mode 100644 index 0000000000..8b9a1d32fe --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_purchase_order_lines_received__measures_fill_and_lateness_in_warehouse_time.sql @@ -0,0 +1,109 @@ +TEST (name "int_purchase_order_lines_received__measures_fill_and_lateness_in_warehouse_time"); + +WITH __ref__stg_po_receipts AS ( + SELECT + 'R1' AS receipt_id, + 'PO1' AS po_id, + 'DUCK-1' AS sku, + 'W1' AS warehouse_code, + 6 AS received_qty, + 1 AS damaged_qty, + NULL AS sellable_qty, + '2026-03-12 10:00:00'::TIMESTAMP_NTZ AS received_at_local, + ARRAY_CONSTRUCT('SHORT')::ARRAY AS discrepancy_codes, + NULL AS has_discrepancy, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'R2', + 'PO1', + 'DUCK-1', + 'W1', + 6, + 0, + NULL, + '2026-03-13 10:00:00'::TIMESTAMP_NTZ, + ARRAY_CONSTRUCT('SHORT', 'DAMAGED')::ARRAY, + NULL, + NULL, + NULL +), + +__ref__stg_purchase_order_lines AS ( + SELECT + 'PO1' AS po_id, + NULL AS currency_code, + 'PO1#1' AS po_line_key, + 'DUCK-1' AS sku, + 10 AS ordered_qty, + NULL AS unit_cost_minor + UNION ALL + SELECT + 'PO1', + NULL, + 'PO1#2', + 'DUCK-2', + 5, + NULL +), + +__ref__stg_purchase_orders AS ( + SELECT + 'PO1' AS po_id, + 'SUP1' AS supplier_code, + 'W1' AS warehouse_code, + NULL AS currency_code, + NULL AS po_status, + NULL AS ordered_at, + '2026-03-10 23:30:00 +00:00'::TIMESTAMP_TZ AS expected_at, + NULL AS received_at, + NULL AS line_count, + NULL AS lines, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_warehouses AS ( + SELECT + 'W1' AS warehouse_code, + NULL AS warehouse_name, + NULL AS country_code, + 'Europe/Berlin' AS iana_timezone, + NULL AS opens_local, + NULL AS closes_local, + NULL AS opening_minutes, + NULL AS supports_same_day, + NULL AS supports_cold_chain, + NULL AS max_parcel_kg, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_purchase_order_lines_received AS ( + SELECT + 'PO1#1' AS po_line_key, + 12 AS received_qty, + 1 AS damaged_qty, + 0 AS outstanding_qty, + 1.0::DECIMAL(9, 4) AS fill_rate, + 1 AS days_late, + 2 AS receipt_count + UNION ALL + SELECT + 'PO1#2' AS po_line_key, + 0 AS received_qty, + 0 AS damaged_qty, + 5 AS outstanding_qty, + 0.0::DECIMAL(9, 4) AS fill_rate, + CAST(NULL AS INT) AS days_late, + 0 AS receipt_count +), + +__assert__discrepancy_codes_are_deduplicated AS ( + SELECT l.po_line_key + FROM __ref("int_purchase_order_lines_received") AS l + WHERE + l.po_line_key = 'PO1#1' + AND ARRAY_SIZE(l.discrepancy_codes) <> 2 +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_reservations_open__nets_open_holds_from_latest_balance.sql b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_reservations_open__nets_open_holds_from_latest_balance.sql new file mode 100644 index 0000000000..17d3157258 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_reservations_open__nets_open_holds_from_latest_balance.sql @@ -0,0 +1,82 @@ +TEST (name "int_reservations_open__nets_open_holds_from_latest_balance"); + +WITH __ref__int_inventory_ledger AS ( + SELECT + 'M:1' AS ledger_entry_key, + 'DUCK-1' AS sku, + 'W1' AS warehouse_code, + NULL AS entry_source, + NULL AS qty_delta, + NULL AS occurred_at_utc, + NULL AS occurred_date_local, + 10 AS running_on_hand, + 1 AS entry_sequence, + NULL AS is_negative_balance + UNION ALL + SELECT + 'M:2', + 'DUCK-1', + 'W1', + NULL, + NULL, + NULL, + NULL, + 7, + 2, + NULL +), + +__ref__stg_stock_reservations AS ( + SELECT + 'R1' AS reservation_id, + NULL AS order_id, + 'DUCK-1' AS sku, + 'W1' AS warehouse_code, + 3 AS qty, + '2026-03-02 00:00:00 +00:00'::TIMESTAMP_TZ AS reserved_at, + NULL AS released_at, + TRUE AS is_open, + NULL AS allocation_rule, + FALSE AS is_backorder, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'R2', + NULL, + 'DUCK-1', + 'W1', + 2, + '2026-03-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + TRUE, + NULL, + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'R3', + NULL, + 'DUCK-1', + 'W1', + 4, + '2026-02-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + NULL, + FALSE, + NULL, + NULL +), + +__expected__int_reservations_open AS ( + SELECT + 'DUCK-1|W1' AS stock_key, + 2 AS open_reservations, + 3 AS reserved_qty, + 2 AS backordered_qty, + 7 AS on_hand_qty, + 4 AS available_to_promise, + '2026-03-01 00:00:00 +00:00'::TIMESTAMP_TZ AS oldest_reserved_at +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_stock_count_variances__compares_counts_with_ledger_balance_at_count_time.sql b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_stock_count_variances__compares_counts_with_ledger_balance_at_count_time.sql new file mode 100644 index 0000000000..ee947fc7a6 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_stock_count_variances__compares_counts_with_ledger_balance_at_count_time.sql @@ -0,0 +1,127 @@ +TEST (name "int_stock_count_variances__compares_counts_with_ledger_balance_at_count_time"); + +WITH __ref__int_inventory_ledger AS ( + SELECT + 'M:1' AS ledger_entry_key, + 'DUCK-1' AS sku, + 'W1' AS warehouse_code, + NULL AS entry_source, + NULL AS qty_delta, + '2026-03-01 10:00:00'::TIMESTAMPNTZ AS occurred_at_utc, + NULL AS occurred_date_local, + 100 AS running_on_hand, + NULL AS entry_sequence, + NULL AS is_negative_balance + UNION ALL + SELECT + 'M:2', + 'DUCK-1', + 'W1', + NULL, + NULL, + '2026-03-05 10:00:00'::TIMESTAMPNTZ, + NULL, + 90, + NULL, + NULL +), + +__ref__stg_stock_counts AS ( + SELECT + 'C1' AS count_id, + 'DUCK-1' AS sku, + 'W1' AS warehouse_code, + 99 AS counted_qty, + '2026-03-03 12:00:00'::TIMESTAMP_NTZ AS counted_at_local, + NULL AS counter_id, + TRUE AS is_blind_count, + NULL AS counter, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'C2', + 'DUCK-1', + 'W1', + 80, + '2026-03-06 08:00:00'::TIMESTAMP_NTZ, + NULL, + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C3', + 'DUCK-1', + 'W1', + 90, + '2026-03-06 15:00:00'::TIMESTAMP_NTZ, + NULL, + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C4', + 'DUCK-1', + 'W1', + 5, + '2026-03-01 09:00:00'::TIMESTAMP_NTZ, + NULL, + FALSE, + NULL, + NULL, + NULL +), + +__ref__stg_warehouses AS ( + SELECT + 'W1' AS warehouse_code, + NULL AS warehouse_name, + NULL AS country_code, + 'UTC' AS iana_timezone, + NULL AS opens_local, + NULL AS closes_local, + NULL AS opening_minutes, + NULL AS supports_same_day, + NULL AS supports_cold_chain, + NULL AS max_parcel_kg, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_stock_count_variances AS ( + SELECT + 'C1' AS count_id, + 100 AS expected_qty, + -1 AS variance_qty, + -0.01::DECIMAL(9, 4) AS variance_pct, + FALSE AS is_out_of_tolerance, + 1 AS recount_sequence + UNION ALL + SELECT + 'C2' AS count_id, + 90 AS expected_qty, + -10 AS variance_qty, + -0.1111::DECIMAL(9, 4) AS variance_pct, + TRUE AS is_out_of_tolerance, + 1 AS recount_sequence + UNION ALL + SELECT + 'C3' AS count_id, + 90 AS expected_qty, + 0 AS variance_qty, + 0.0::DECIMAL(9, 4) AS variance_pct, + FALSE AS is_out_of_tolerance, + 2 AS recount_sequence + UNION ALL + SELECT + 'C4' AS count_id, + 0 AS expected_qty, + 5 AS variance_qty, + CAST(NULL AS DECIMAL(9, 4)) AS variance_pct, + TRUE AS is_out_of_tolerance, + 1 AS recount_sequence +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_stock_on_hand_daily__carries_balance_and_last_count_forward.sql b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_stock_on_hand_daily__carries_balance_and_last_count_forward.sql new file mode 100644 index 0000000000..9c7d2a932f --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_stock_on_hand_daily__carries_balance_and_last_count_forward.sql @@ -0,0 +1,91 @@ +TEST (name "int_stock_on_hand_daily__carries_balance_and_last_count_forward"); + +WITH __ref__int_inventory_ledger AS ( + SELECT + 'M:1' AS ledger_entry_key, + 'DUCK-1' AS sku, + 'W1' AS warehouse_code, + NULL AS entry_source, + NULL AS qty_delta, + '2026-03-28 10:00:00'::TIMESTAMPNTZ AS occurred_at_utc, + '2026-03-28'::DATE AS occurred_date_local, + 5 AS running_on_hand, + NULL AS entry_sequence, + NULL AS is_negative_balance + UNION ALL + SELECT + 'M:2', + 'DUCK-1', + 'W1', + NULL, + NULL, + '2026-03-30 12:00:00'::TIMESTAMPNTZ, + '2026-03-30'::DATE, + 0, + NULL, + NULL +), + +__ref__stg_stock_counts AS ( + SELECT + 'C1' AS count_id, + 'DUCK-1' AS sku, + 'W1' AS warehouse_code, + 6 AS counted_qty, + '2026-03-29 09:00:00'::TIMESTAMP_NTZ AS counted_at_local, + NULL AS counter_id, + NULL AS is_blind_count, + NULL AS counter, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_warehouses AS ( + SELECT + 'W1' AS warehouse_code, + NULL AS warehouse_name, + NULL AS country_code, + 'UTC' AS iana_timezone, + NULL AS opens_local, + NULL AS closes_local, + NULL AS opening_minutes, + NULL AS supports_same_day, + NULL AS supports_cold_chain, + NULL AS max_parcel_kg, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_stock_on_hand_daily AS ( + SELECT + 'DUCK-1|W1|2026-03-28' AS stock_day_key, + '2026-03-01'::DATE AS month_start, + 5 AS on_hand_qty, + CAST(NULL AS INT) AS last_counted_qty, + CAST(NULL AS INT) AS days_since_count, + FALSE AS is_stocked_out + UNION ALL + SELECT + 'DUCK-1|W1|2026-03-29' AS stock_day_key, + '2026-03-01'::DATE AS month_start, + 5 AS on_hand_qty, + 6 AS last_counted_qty, + 0 AS days_since_count, + FALSE AS is_stocked_out + UNION ALL + SELECT + 'DUCK-1|W1|2026-03-30' AS stock_day_key, + '2026-03-01'::DATE AS month_start, + 0 AS on_hand_qty, + 6 AS last_counted_qty, + 1 AS days_since_count, + TRUE AS is_stocked_out + UNION ALL + SELECT + 'DUCK-1|W1|2026-03-31' AS stock_day_key, + '2026-03-01'::DATE AS month_start, + 0 AS on_hand_qty, + 6 AS last_counted_qty, + 2 AS days_since_count, + TRUE AS is_stocked_out +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_stock_transfers_in_transit__projects_arrival_from_lane_history.sql b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_stock_transfers_in_transit__projects_arrival_from_lane_history.sql new file mode 100644 index 0000000000..daed345956 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_stock_transfers_in_transit__projects_arrival_from_lane_history.sql @@ -0,0 +1,93 @@ +TEST (name "int_stock_transfers_in_transit__projects_arrival_from_lane_history"); + +WITH __ref__stg_stock_transfers AS ( + SELECT + 'T1' AS transfer_id, + 'DUCK-1' AS sku, + 'W1' AS from_warehouse_code, + 'W2' AS to_warehouse_code, + 1 AS qty, + '2026-03-01 00:00:00 +00:00'::TIMESTAMP_TZ AS shipped_at, + '2026-03-01 20:00:00 +00:00'::TIMESTAMP_TZ AS arrived_at, + 20 AS transit_hours, + NULL AS haulier_reference, + NULL AS tracking, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + 'DUCK-1', + 'W1', + 'W2', + 1, + '2026-03-02 00:00:00 +00:00'::TIMESTAMP_TZ, + '2026-03-03 06:00:00 +00:00'::TIMESTAMP_TZ, + 30, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T3', + 'DUCK-1', + 'W1', + 'W2', + 4, + '2026-03-29 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T4', + 'DUCK-2', + 'W2', + 'W3', + 2, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_stock_transfers_in_transit AS ( + SELECT + 'T1' AS transfer_id, + 'arrived' AS transfer_state, + 25.0::DECIMAL(10, 2) AS lane_median_transit_hours, + FALSE AS is_overdue + UNION ALL + SELECT + 'T2' AS transfer_id, + 'arrived' AS transfer_state, + 25.0::DECIMAL(10, 2) AS lane_median_transit_hours, + FALSE AS is_overdue + UNION ALL + SELECT + 'T3' AS transfer_id, + 'in_transit' AS transfer_state, + 25.0::DECIMAL(10, 2) AS lane_median_transit_hours, + TRUE AS is_overdue + UNION ALL + SELECT + 'T4' AS transfer_id, + 'planned' AS transfer_state, + CAST(NULL AS DECIMAL(10, 2)) AS lane_median_transit_hours, + CAST(NULL AS BOOLEAN) AS is_overdue +), + +__assert__arrival_is_shipped_plus_lane_median AS ( + SELECT t.transfer_id + FROM __ref("int_stock_transfers_in_transit") AS t + WHERE + t.transfer_id = 'T3' + AND t.expected_arrival_at <> '2026-03-30 01:00:00 +00:00'::TIMESTAMP_TZ +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_supplier_lead_times__summarises_received_purchase_orders.sql b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_supplier_lead_times__summarises_received_purchase_orders.sql new file mode 100644 index 0000000000..0f79e9fa27 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/inventory/test_int_supplier_lead_times__summarises_received_purchase_orders.sql @@ -0,0 +1,106 @@ +TEST (name "int_supplier_lead_times__summarises_received_purchase_orders"); + +WITH __ref__int_purchase_order_lines_received AS ( + SELECT + 'PO1#1' AS po_line_key, + 'PO1' AS po_id, + NULL AS supplier_code, + NULL AS warehouse_code, + NULL AS sku, + 10 AS ordered_qty, + 10 AS received_qty, + NULL AS damaged_qty, + NULL AS outstanding_qty, + NULL AS fill_rate, + NULL AS first_received_at_local, + 0 AS days_late, + NULL AS receipt_count, + NULL AS discrepancy_codes + UNION ALL + SELECT + 'PO2#1', + 'PO2', + NULL, + NULL, + NULL, + 10, + 5, + NULL, + NULL, + NULL, + NULL, + 3, + NULL, + NULL +), + +__ref__stg_purchase_orders AS ( + SELECT + 'PO1' AS po_id, + 'SUP1' AS supplier_code, + NULL AS warehouse_code, + NULL AS currency_code, + NULL AS po_status, + '2026-03-01 00:00:00 +00:00'::TIMESTAMP_TZ AS ordered_at, + NULL AS expected_at, + '2026-03-11 00:00:00 +00:00'::TIMESTAMP_TZ AS received_at, + NULL AS line_count, + NULL AS lines, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'PO2', + 'SUP1', + NULL, + NULL, + NULL, + '2026-03-05 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + '2026-03-25 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'PO3', + 'SUP1', + NULL, + NULL, + NULL, + '2026-03-20 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_suppliers AS ( + SELECT + 'SUP1' AS supplier_code, + 'Teal Hollow Decoys' AS supplier_name, + NULL AS country_code, + NULL AS payment_terms_days, + NULL AS early_payment_discount_pct, + NULL AS primary_contact_email, + NULL AS contact_count, + NULL AS payment_terms, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_supplier_lead_times AS ( + SELECT + 'SUP1' AS supplier_code, + 'Teal Hollow Decoys' AS supplier_name, + 2 AS received_po_count, + 15.25::DECIMAL(10, 2) AS lead_time_p50_days, + 19.45::DECIMAL(10, 2) AS lead_time_p90_days, + 7.42::DECIMAL(10, 2) AS lead_time_stddev_days, + 0.5::DECIMAL(9, 4) AS on_time_rate, + 0.75::DECIMAL(9, 4) AS avg_fill_rate, + '2026-03-05 00:00:00 +00:00'::TIMESTAMP_TZ AS latest_po_at +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_cart_abandonment__matches_orders_within_conversion_window.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_cart_abandonment__matches_orders_within_conversion_window.sql new file mode 100644 index 0000000000..875977fa72 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_cart_abandonment__matches_orders_within_conversion_window.sql @@ -0,0 +1,173 @@ +TEST (name "int_cart_abandonment__matches_orders_within_conversion_window"); + +WITH __ref__stg_carts AS ( + SELECT + 'K1' AS cart_id, + 'C1' AS customer_id, + NULL AS anonymous_id, + 'DE' AS storefront_code, + 'active' AS cart_status, + '2026-03-01 09:00:00 +00:00'::TIMESTAMP_TZ AS created_at, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS updated_at, + NULL AS line_count, + NULL AS coupon_code, + NULL AS contents, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'K2', + 'C2', + NULL, + 'DE', + 'active', + '2026-03-01 09:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'K3', + 'C3', + NULL, + 'US', + 'merged', + '2026-03-01 09:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_cart_items AS ( + SELECT + 'K1' AS cart_id, + 'K1#1' AS cart_item_key, + 'DUCK-A' AS sku, + 2 AS quantity, + 500 AS unit_amount_minor, + NULL AS added_at + UNION ALL + SELECT + 'K1', + 'K1#2', + 'BED-B', + NULL, + 1000, + NULL + UNION ALL + SELECT + 'K2', + 'K2#1', + 'DUCK-C', + 1, + 300, + NULL + UNION ALL + SELECT + 'K3', + 'K3#1', + 'DUCK-D', + 1, + 300, + NULL +), + +__ref__stg_orders AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + NULL AS storefront_code, + NULL AS currency_code, + '2026-03-01 13:30:00 +00:00'::TIMESTAMP_TZ AS placed_at, + NULL AS order_status, + NULL AS sales_channel, + NULL AS is_gift, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS device_type, + NULL AS item_count_hint, + NULL AS shipping_address_id, + NULL AS attributes, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'O0', + 'C1', + NULL, + NULL, + '2026-03-03 13:30:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O2', + 'C2', + NULL, + NULL, + '2026-03-03 09:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_cart_abandonment AS ( + SELECT + 'K1' AS cart_id, + 2 AS item_count, + 3 AS units, + 2000 AS cart_value_minor, + 'O1' AS converted_order_id, + TRUE AS is_converted, + FALSE AS is_abandoned, + 3.5::DECIMAL(10, 2) AS hours_to_convert, + 2 AS distinct_sku_families + UNION ALL + SELECT + 'K2' AS cart_id, + 1 AS item_count, + 1 AS units, + 300 AS cart_value_minor, + CAST(NULL AS VARCHAR) AS converted_order_id, + FALSE AS is_converted, + TRUE AS is_abandoned, + CAST(NULL AS DECIMAL(10, 2)) AS hours_to_convert, + 1 AS distinct_sku_families + UNION ALL + SELECT + 'K3' AS cart_id, + 1 AS item_count, + 1 AS units, + 300 AS cart_value_minor, + CAST(NULL AS VARCHAR) AS converted_order_id, + FALSE AS is_converted, + FALSE AS is_abandoned, + CAST(NULL AS DECIMAL(10, 2)) AS hours_to_convert, + 1 AS distinct_sku_families +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_lines_discounted__caps_discounts_and_flags_stacked_promotions.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_lines_discounted__caps_discounts_and_flags_stacked_promotions.sql new file mode 100644 index 0000000000..a13c52fc08 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_lines_discounted__caps_discounts_and_flags_stacked_promotions.sql @@ -0,0 +1,149 @@ +TEST (name "int_order_lines_discounted__caps_discounts_and_flags_stacked_promotions"); + +WITH __ref__stg_order_lines AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + NULL AS line_no, + 'DUCK-1' AS sku, + 2 AS quantity, + NULL AS unit_amount_minor, + 1000 AS gross_amount_minor, + 200 AS tax_amount_minor, + NULL AS discount_count, + NULL AS discounts, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'L2', + 'O1', + NULL, + 'DUCK-2', + 1, + NULL, + 300, + 60, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'L3', + 'O2', + NULL, + 'DUCK-3', + 1, + NULL, + 500, + 100, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_order_line_discounts AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + 'L1#0' AS order_line_discount_key, + 'SPRING' AS discount_code, + 'percent' AS discount_kind, + 100 AS discount_amount_minor + UNION ALL + SELECT + 'L1', + 'O1', + 'L1#1', + 'VIP', + 'fixed', + 50 + UNION ALL + SELECT + 'L2', + 'O1', + 'L2#0', + 'BIG', + 'fixed', + 400 +), + +__ref__stg_promotions AS ( + SELECT + 'SPRING' AS promo_code, + NULL AS promo_name, + NULL AS starts_at, + NULL AS ends_at, + TRUE AS is_stackable, + NULL AS rules, + NULL AS min_order_minor, + NULL AS eligible_regions, + NULL AS tier_count, + NULL AS max_discount_pct, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'VIP', + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'BIG', + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_order_lines_discounted AS ( + SELECT + 'L1' AS order_line_id, + 150 AS discount_amount_minor, + 850 AS net_amount_minor, + TRUE AS has_stacked_promotions, + 0.15::DECIMAL(9, 6) AS discount_share + UNION ALL + SELECT + 'L2' AS order_line_id, + 300 AS discount_amount_minor, + 0 AS net_amount_minor, + FALSE AS has_stacked_promotions, + 1.333333::DECIMAL(9, 6) AS discount_share + UNION ALL + SELECT + 'L3' AS order_line_id, + 0 AS discount_amount_minor, + 500 AS net_amount_minor, + FALSE AS has_stacked_promotions, + 0.0::DECIMAL(9, 6) AS discount_share +), + +__assert__discount_by_kind_sums_to_discount AS ( + SELECT d.order_line_id + FROM __ref("int_order_lines_discounted") AS d + WHERE + d.order_line_id = 'L1' + AND ( + d.discount_by_kind:percent::INT <> 100 + OR d.discount_by_kind:fixed::INT <> 50 + ) +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_lines_priced__applies_latest_eligible_list_price.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_lines_priced__applies_latest_eligible_list_price.sql new file mode 100644 index 0000000000..f053a087fa --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_lines_priced__applies_latest_eligible_list_price.sql @@ -0,0 +1,136 @@ +TEST (name "int_order_lines_priced__applies_latest_eligible_list_price"); + +WITH __ref__int_order_lines_discounted AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + 'DUCK-KNF-08' AS sku, + 2 AS quantity, + 1600 AS gross_amount_minor, + 0 AS discount_amount_minor, + 1600 AS net_amount_minor, + 0 AS tax_amount_minor, + NULL AS discount_by_kind, + NULL AS promo_codes, + NULL AS has_stacked_promotions, + NULL AS discount_share +), + +__ref__stg_order_lines AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + NULL AS line_no, + 'DUCK-KNF-08' AS sku, + 2 AS quantity, + 800 AS unit_amount_minor, + NULL AS gross_amount_minor, + NULL AS tax_amount_minor, + NULL AS discount_count, + NULL AS discounts, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_orders AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS storefront_code, + 'EUR' AS currency_code, + '2026-02-10 12:00:00 +01:00'::TIMESTAMP_TZ AS placed_at, + NULL AS order_status, + 'web' AS sales_channel, + NULL AS is_gift, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS device_type, + NULL AS item_count_hint, + NULL AS shipping_address_id, + NULL AS attributes, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_price_lists AS ( + SELECT + 'P1' AS price_list_id, + 'DUCK-KNF-08' AS sku, + 'EUR' AS currency_code, + 900 AS list_amount_minor, + '2026-01-01 00:00:00 +00:00'::TIMESTAMPTZ AS valid_from, + NULL AS valid_to, + NULL AS is_open_ended, + 1 AS min_quantity, + ARRAY_CONSTRUCT()::ARRAY AS channel_restrictions, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'P2', + 'DUCK-KNF-08', + 'EUR', + 1000, + '2026-02-01 00:00:00 +00:00'::TIMESTAMPTZ, + NULL, + NULL, + 1, + ARRAY_CONSTRUCT()::ARRAY, + NULL, + NULL + UNION ALL + SELECT + 'P3', + 'DUCK-KNF-08', + 'EUR', + 700, + '2026-02-05 00:00:00 +00:00'::TIMESTAMPTZ, + NULL, + NULL, + 1, + ARRAY_CONSTRUCT('app')::ARRAY, + NULL, + NULL + UNION ALL + SELECT + 'P4', + 'DUCK-KNF-08', + 'EUR', + 600, + '2026-02-06 00:00:00 +00:00'::TIMESTAMPTZ, + NULL, + NULL, + 10, + ARRAY_CONSTRUCT()::ARRAY, + NULL, + NULL +), + +__ref__stg_product_variants AS ( + SELECT + 'DUCK-KNF-08' AS sku, + 'PR1' AS product_id, + NULL AS colour, + NULL AS size_label, + NULL AS weight_grams, + 0.25::DECIMAL(10, 3) AS weight_kg, + NULL AS barcode_count, + NULL AS is_fragile, + NULL AS pack_size, + NULL AS barcodes, + NULL AS attributes, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_order_lines_priced AS ( + SELECT + 'L1' AS order_line_id, + 'DUCK' AS sku_family, + 'PR1' AS product_id, + 1000 AS list_amount_minor, + 20.0::DECIMAL(9, 4) AS markdown_pct, + 0.5::DECIMAL(12, 3) AS line_weight_kg, + TRUE AS has_list_price +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_payments_rollup__pivots_captures_and_nets_out_refunds.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_payments_rollup__pivots_captures_and_nets_out_refunds.sql new file mode 100644 index 0000000000..3956ea1cb7 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_payments_rollup__pivots_captures_and_nets_out_refunds.sql @@ -0,0 +1,184 @@ +TEST (name "int_order_payments_rollup__pivots_captures_and_nets_out_refunds"); + +WITH __ref__int_payment_attempts AS ( + SELECT + 'P1' AS payment_id, + 'O1' AS order_id, + NULL AS attempt_number, + NULL AS attempt_count, + NULL AS is_final_attempt, + 'card' AS payment_method, + 3000 AS amount_minor, + NULL AS currency_code, + NULL AS authorised_at, + NULL AS seconds_since_previous_attempt, + NULL AS gateway_status, + TRUE AS is_successful, + NULL AS decline_family, + NULL AS gateway_error_code, + NULL AS risk_score, + NULL AS risk_band, + NULL AS three_ds_outcome, + NULL AS card_bin_valid, + NULL AS card_scheme, + NULL AS had_chargeback + UNION ALL + SELECT + 'P2', + 'O1', + NULL, + NULL, + NULL, + 'phone_wallet', + 1000, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'P3', + 'O1', + NULL, + NULL, + NULL, + 'card', + 3000, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'P4', + 'O2', + NULL, + NULL, + NULL, + 'voucher', + 700, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_refunds AS ( + SELECT + 'RF1' AS refund_id, + 'P1' AS payment_id, + 500 AS amount_minor, + NULL AS reason_text, + NULL AS reason_category, + NULL AS refunded_at, + NULL AS initiated_by, + NULL AS refunded_line_count, + NULL AS refunded_items, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_chargebacks AS ( + SELECT + 'CB1' AS chargeback_id, + 'P2' AS payment_id, + 1000 AS amount_minor, + NULL AS currency_code, + NULL AS reason_code, + NULL AS reason_family, + NULL AS opened_at, + NULL AS closed_at, + 'lost' AS outcome, + NULL AS evidence_document_count, + NULL AS evidence, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'CB2', + 'P4', + 700, + NULL, + NULL, + NULL, + NULL, + NULL, + 'won', + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_gift_card_transactions AS ( + SELECT + 'G1' AS transaction_id, + 'GC1' AS gift_card_id, + 'O2' AS order_id, + 'REDEEM' AS transaction_type, + -300 AS signed_amount_minor, + NULL AS occurred_at, + NULL AS channel, + NULL AS terminal_id, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_order_payments_rollup AS ( + SELECT + 'O1' AS order_id, + 3000 AS card_captured_minor, + 1000 AS wallet_captured_minor, + 0 AS other_captured_minor, + 4000 AS captured_minor, + 500 AS refunded_minor, + 1000 AS chargeback_minor, + 2500 AS net_collected_minor, + 1 AS failed_attempts + UNION ALL + SELECT + 'O2' AS order_id, + 0 AS card_captured_minor, + 0 AS wallet_captured_minor, + 700 AS other_captured_minor, + 700 AS captured_minor, + 0 AS refunded_minor, + 0 AS chargeback_minor, + 1000 AS net_collected_minor, + 0 AS failed_attempts +), + +__assert__tender_methods_are_distinct_and_sorted AS ( + SELECT r.order_id + FROM __ref("int_order_payments_rollup") AS r + WHERE + r.order_id = 'O1' + AND ARRAY_TO_STRING(r.tender_methods, ',') <> 'card,phone_wallet' +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_status_timeline__builds_milestones_and_detects_regressions.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_status_timeline__builds_milestones_and_detects_regressions.sql new file mode 100644 index 0000000000..866c6bb9da --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_order_status_timeline__builds_milestones_and_detects_regressions.sql @@ -0,0 +1,125 @@ +TEST (name "int_order_status_timeline__builds_milestones_and_detects_regressions"); + +WITH __ref__stg_order_status_events AS ( + SELECT + 'E1' AS event_id, + 'O1' AS order_id, + NULL AS from_status, + 'paid' AS to_status, + '2026-03-01 10:30:00 +00:00'::TIMESTAMP_TZ AS changed_at, + NULL AS actor_type, + NULL AS actor_id, + NULL AS actor, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'E2', + 'O1', + NULL, + 'shipped', + '2026-03-02 10:30:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E3', + 'O1', + NULL, + 'packed', + '2026-03-02 11:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E4', + 'O1', + NULL, + 'delivered', + '2026-03-04 10:30:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E5', + 'O2', + NULL, + 'cancelled', + '2026-03-01 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_orders AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS storefront_code, + NULL AS currency_code, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS placed_at, + NULL AS order_status, + NULL AS sales_channel, + NULL AS is_gift, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS device_type, + NULL AS item_count_hint, + NULL AS shipping_address_id, + NULL AS attributes, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'O2', + NULL, + NULL, + NULL, + '2026-03-01 11:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_order_status_timeline AS ( + SELECT + 'O1' AS order_id, + 4 AS transition_count, + 30 AS minutes_to_pay, + 24.0::DECIMAL(12, 2) AS hours_paid_to_shipped, + 'paid > shipped > packed > delivered' AS status_path, + TRUE AS has_regression, + 'delivered' AS last_status, + 'packed' AS longest_status + UNION ALL + SELECT + 'O2' AS order_id, + 1 AS transition_count, + CAST(NULL AS INT) AS minutes_to_pay, + CAST(NULL AS DECIMAL(12, 2)) AS hours_paid_to_shipped, + 'cancelled' AS status_path, + FALSE AS has_regression, + 'cancelled' AS last_status, + 'cancelled' AS longest_status +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_orders_enriched__totals_lines_and_resolves_storefront.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_orders_enriched__totals_lines_and_resolves_storefront.sql new file mode 100644 index 0000000000..1b556a65d4 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_orders_enriched__totals_lines_and_resolves_storefront.sql @@ -0,0 +1,245 @@ +TEST (name "int_orders_enriched__totals_lines_and_resolves_storefront"); + +WITH __ref__stg_orders AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + 'DE' AS storefront_code, + 'EUR' AS currency_code, + '2026-03-28 23:30:00 +00:00'::TIMESTAMP_TZ AS placed_at, + 'PAID' AS order_status, + NULL AS sales_channel, + NULL AS is_gift, + 'Search_Ads' AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS device_type, + NULL AS item_count_hint, + 'A1' AS shipping_address_id, + NULL AS attributes, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'O2', + 'C2', + 'ZZ', + 'USD', + '2026-03-28 23:30:00 +00:00'::TIMESTAMP_TZ, + 'PENDING', + 'app', + NULL, + '', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_order_lines_priced AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + NULL AS sku, + NULL AS sku_family, + NULL AS product_id, + 4 AS quantity, + NULL AS unit_amount_minor, + NULL AS list_amount_minor, + NULL AS markdown_pct, + 4000 AS gross_amount_minor, + 3500 AS net_amount_minor, + 500 AS discount_amount_minor, + 700 AS tax_amount_minor, + 1.5::DECIMAL(12, 3) AS line_weight_kg, + NULL AS has_list_price + UNION ALL + SELECT + 'L2', + 'O1', + NULL, + NULL, + NULL, + 3, + NULL, + NULL, + NULL, + 1500, + 1500, + 0, + 300, + 0.5::DECIMAL(12, 3), + NULL +), + +__ref__stg_customers AS ( + SELECT + 'C1' AS customer_id, + 'ada@duckshop.test' AS email_normalised, + NULL AS phone_normalised, + NULL AS first_name, + NULL AS last_name, + 'EU' AS region_code, + NULL AS storefront_code, + NULL AS created_at, + NULL AS preferred_language, + NULL AS marketing_opt_in, + NULL AS consent_types, + NULL AS birth_year, + NULL AS preferences, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'C2', + 'bob@example.org', + NULL, + NULL, + NULL, + 'US', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_payments AS ( + SELECT + 'P1' AS payment_id, + 'O1' AS order_id, + NULL AS payment_method, + 4000 AS amount_minor, + NULL AS currency_code, + NULL AS authorised_at, + NULL AS captured_at, + NULL AS card_bin, + 'captured' AS gateway_status, + NULL AS gateway_error_code, + NULL AS risk_score, + NULL AS three_ds_outcome, + NULL AS gateway_reference, + NULL AS gateway_response, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'P2', + 'O1', + NULL, + 6000, + NULL, + NULL, + NULL, + NULL, + 'declined', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_customer_addresses AS ( + SELECT + 'A1' AS address_id, + 'C1' AS customer_id, + NULL AS line1, + NULL AS line2, + NULL AS city, + 'FR' AS country_code, + NULL AS postcode_canonical, + NULL AS is_default, + NULL AS valid_from, + NULL AS latitude, + NULL AS longitude, + NULL AS geo, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_promotion_redemptions AS ( + SELECT + 'R2' AS redemption_id, + 'LATE' AS promo_code, + 'O1' AS order_id, + NULL AS discount_minor, + '2026-03-28 23:31:00 +00:00'::TIMESTAMP_TZ AS redeemed_at, + NULL AS entry_method, + NULL AS redemption_context, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'R1', + 'EARLY', + 'O1', + NULL, + '2026-03-28 23:30:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL +), + +__seed__countries AS ( + SELECT + 'DE' AS country_code, + NULL AS country_name, + 'EU' AS region_code, + 'Europe/Berlin' AS default_timezone, + NULL AS currency_code, + NULL AS vat_registered +), + +__expected__int_orders_enriched AS ( + SELECT + 'O1' AS order_id, + 'EU' AS region_code, + 'Europe/Berlin' AS storefront_timezone, + '2026-03-29'::DATE AS placed_date_local, + 'web' AS sales_channel, + 2 AS line_count, + 7 AS units, + 5000 AS net_amount_minor, + 1000 AS tax_amount_minor, + 4000 AS captured_amount_minor, + 2000 AS outstanding_amount_minor, + 'EARLY' AS promo_code, + 'FR' AS ship_to_country_code, + TRUE AS is_cross_border, + TRUE AS is_first_party_customer, + 'LARGE' AS basket_size, + 'search_ads' AS utm_source, + 2.0::DECIMAL(12, 3) AS total_weight_kg + UNION ALL + SELECT + 'O2' AS order_id, + 'US' AS region_code, + 'UTC' AS storefront_timezone, + '2026-03-28'::DATE AS placed_date_local, + 'app' AS sales_channel, + 0 AS line_count, + 0 AS units, + 0 AS net_amount_minor, + 0 AS tax_amount_minor, + 0 AS captured_amount_minor, + 0 AS outstanding_amount_minor, + CAST(NULL AS VARCHAR) AS promo_code, + CAST(NULL AS VARCHAR) AS ship_to_country_code, + FALSE AS is_cross_border, + FALSE AS is_first_party_customer, + 'SINGLE' AS basket_size, + CAST(NULL AS VARCHAR) AS utm_source, + 0.0::DECIMAL(12, 3) AS total_weight_kg +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_orders_usd__converts_minor_units_by_currency_exponent.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_orders_usd__converts_minor_units_by_currency_exponent.sql new file mode 100644 index 0000000000..9c73eee1ab --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_orders_usd__converts_minor_units_by_currency_exponent.sql @@ -0,0 +1,182 @@ +TEST (name "int_orders_usd__converts_minor_units_by_currency_exponent"); + +WITH __ref__int_orders_enriched AS ( + SELECT + 'O-JPY' AS order_id, + 'C1' AS customer_id, + NULL AS storefront_code, + NULL AS region_code, + NULL AS storefront_timezone, + 'JPY' AS currency_code, + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ AS placed_at, + '2026-03-10'::DATE AS placed_date_local, + 'PAID' AS order_status, + NULL AS sales_channel, + NULL AS line_count, + NULL AS units, + 1500 AS gross_amount_minor, + 0 AS discount_amount_minor, + 1500 AS net_amount_minor, + 0 AS tax_amount_minor, + NULL AS captured_amount_minor, + NULL AS outstanding_amount_minor, + NULL AS promo_code, + NULL AS ship_to_country_code, + NULL AS is_cross_border, + NULL AS is_first_party_customer, + NULL AS basket_size, + NULL AS is_gift, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS total_weight_kg + UNION ALL + SELECT + 'O-EUR', + 'C1', + NULL, + NULL, + NULL, + 'EUR', + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ, + '2026-03-10'::DATE, + 'PAID', + NULL, + NULL, + NULL, + 2499, + 500, + 1999, + 0, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O-KWD', + 'C1', + NULL, + NULL, + NULL, + 'KWD', + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ, + '2026-03-20'::DATE, + 'PAID', + NULL, + NULL, + NULL, + 12345, + 0, + 12345, + 0, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'JPY' AS currency_code, + '2026-03-01'::DATE AS rate_date, + 1.0::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'JPY', + '2026-03-09'::DATE, + 0.0067::NUMBER(18, 8), + NULL, + NULL + UNION ALL + SELECT + 'JPY', + '2026-03-11'::DATE, + 99.0::NUMBER(18, 8), + NULL, + NULL + UNION ALL + SELECT + 'EUR', + '2026-03-10'::DATE, + 1.08::NUMBER(18, 8), + NULL, + NULL + UNION ALL + SELECT + 'KWD', + '2026-03-09'::DATE, + 3.25::NUMBER(18, 8), + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'JPY' AS currency_code, + NULL AS currency_name, + 0 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'EUR', + NULL, + 2, + NULL + UNION ALL + SELECT + 'KWD', + NULL, + 3, + NULL +), + +__expected__int_orders_usd AS ( + SELECT + 'O-JPY' AS order_id, + 1500.0::DECIMAL(38, 6) AS amount, + 10.05::DECIMAL(18, 2) AS amount_usd, + 10.05::DECIMAL(18, 2) AS gross_amount_usd, + 0.0::DECIMAL(18, 2) AS discount_amount_usd, + '2026-03-09'::DATE AS rate_date, + 1 AS rate_age_days, + FALSE AS is_rate_stale + UNION ALL + SELECT + 'O-EUR' AS order_id, + 19.99::DECIMAL(38, 6) AS amount, + 21.59::DECIMAL(18, 2) AS amount_usd, + 26.99::DECIMAL(18, 2) AS gross_amount_usd, + 5.4::DECIMAL(18, 2) AS discount_amount_usd, + '2026-03-10'::DATE AS rate_date, + 0 AS rate_age_days, + FALSE AS is_rate_stale + UNION ALL + SELECT + 'O-KWD' AS order_id, + 12.345::DECIMAL(38, 6) AS amount, + 40.12::DECIMAL(18, 2) AS amount_usd, + 40.12::DECIMAL(18, 2) AS gross_amount_usd, + 0.0::DECIMAL(18, 2) AS discount_amount_usd, + '2026-03-09'::DATE AS rate_date, + 11 AS rate_age_days, + TRUE AS is_rate_stale +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_payment_attempts__sequences_attempts_and_classifies_declines.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_payment_attempts__sequences_attempts_and_classifies_declines.sql new file mode 100644 index 0000000000..7858678b1b --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_payment_attempts__sequences_attempts_and_classifies_declines.sql @@ -0,0 +1,112 @@ +TEST (name "int_payment_attempts__sequences_attempts_and_classifies_declines"); + +WITH __ref__stg_payments AS ( + SELECT + 'P1' AS payment_id, + 'O1' AS order_id, + 'card' AS payment_method, + 1000 AS amount_minor, + NULL AS currency_code, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS authorised_at, + NULL AS captured_at, + '4876532091847355' AS card_bin, + 'declined' AS gateway_status, + NULL AS gateway_error_code, + 80::DECIMAL(5, 2) AS risk_score, + NULL AS three_ds_outcome, + NULL AS gateway_reference, + PARSE_JSON('{"status": {"message": "Insufficient funds"}}')::OBJECT AS gateway_response, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'P2', + 'O1', + 'card', + 1000, + NULL, + '2026-03-01 10:00:45 +00:00'::TIMESTAMP_TZ, + NULL, + '5318462705936183', + 'captured', + NULL, + 20::DECIMAL(5, 2), + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'P3', + 'O2', + 'wallet', + 500, + NULL, + '2026-03-02 09:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + 'failed', + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('{"status": {"message": "Gateway error 503"}}')::OBJECT, + NULL, + NULL +), + +__ref__stg_chargebacks AS ( + SELECT + 'CB1' AS chargeback_id, + 'P2' AS payment_id, + NULL AS amount_minor, + NULL AS currency_code, + NULL AS reason_code, + NULL AS reason_family, + NULL AS opened_at, + NULL AS closed_at, + NULL AS outcome, + NULL AS evidence_document_count, + NULL AS evidence, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_payment_attempts AS ( + SELECT + 'P1' AS payment_id, + 1 AS attempt_number, + 2 AS attempt_count, + FALSE AS is_final_attempt, + CAST(NULL AS INT) AS seconds_since_previous_attempt, + FALSE AS is_successful, + 'INSUFFICIENT_FUNDS' AS decline_family, + 'high' AS risk_band, + 'network_a' AS card_scheme, + FALSE AS had_chargeback + UNION ALL + SELECT + 'P2' AS payment_id, + 2 AS attempt_number, + 2 AS attempt_count, + TRUE AS is_final_attempt, + 45 AS seconds_since_previous_attempt, + TRUE AS is_successful, + 'NONE' AS decline_family, + 'low' AS risk_band, + 'network_b' AS card_scheme, + TRUE AS had_chargeback + UNION ALL + SELECT + 'P3' AS payment_id, + 1 AS attempt_number, + 1 AS attempt_count, + TRUE AS is_final_attempt, + CAST(NULL AS INT) AS seconds_since_previous_attempt, + FALSE AS is_successful, + 'TECHNICAL' AS decline_family, + 'unscored' AS risk_band, + CAST(NULL AS VARCHAR) AS card_scheme, + FALSE AS had_chargeback +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_promotion_performance__measures_redemptions_against_baseline.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_promotion_performance__measures_redemptions_against_baseline.sql new file mode 100644 index 0000000000..c9b019d915 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_promotion_performance__measures_redemptions_against_baseline.sql @@ -0,0 +1,185 @@ +TEST (name "int_promotion_performance__measures_redemptions_against_baseline"); + +WITH __ref__stg_promotions AS ( + SELECT + 'SPRING' AS promo_code, + 'Spring sale' AS promo_name, + '2026-03-01 00:00:00 +00:00'::TIMESTAMPTZ AS starts_at, + NULL AS ends_at, + NULL AS is_stackable, + PARSE_JSON('{"tiers": [{"threshold_minor": 5000}, {"threshold_minor": 10000}]}')::OBJECT AS rules, + NULL AS min_order_minor, + ARRAY_CONSTRUCT('EU', 'US')::ARRAY AS eligible_regions, + NULL AS tier_count, + NULL AS max_discount_pct, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'UNUSED', + 'Never used', + '2026-03-01 00:00:00 +00:00'::TIMESTAMPTZ, + NULL, + NULL, + PARSE_JSON('{}')::OBJECT, + NULL, + ARRAY_CONSTRUCT()::ARRAY, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_promotion_redemptions AS ( + SELECT + 'R1' AS redemption_id, + 'SPRING' AS promo_code, + 'O1' AS order_id, + NULL AS discount_minor, + NULL AS redeemed_at, + NULL AS entry_method, + NULL AS redemption_context, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'R2', + 'SPRING', + 'O2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_orders_usd AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + NULL AS region_code, + NULL AS currency_code, + NULL AS placed_at, + NULL AS placed_date_local, + NULL AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS amount, + 120.0::DECIMAL(18, 2) AS amount_usd, + NULL AS gross_amount_usd, + 20.0::DECIMAL(18, 2) AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS rate_age_days, + NULL AS is_rate_stale, + 'SPRING' AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size + UNION ALL + SELECT + 'O2', + 'C1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 80.0::DECIMAL(18, 2), + NULL, + 10.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + 'SPRING', + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + 'C2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 50.0::DECIMAL(18, 2), + NULL, + 0.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O4', + 'C3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 150.0::DECIMAL(18, 2), + NULL, + 0.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_promotion_performance AS ( + SELECT + 'SPRING' AS promo_code, + 2 AS redemption_count, + 1 AS redeeming_customers, + 200.0::DECIMAL(18, 2) AS revenue_usd, + 30.0::DECIMAL(18, 2) AS discount_usd, + 0.130435::DECIMAL(9, 6) AS discount_rate, + 100.0::DECIMAL(18, 2) AS avg_order_usd, + 100.0::DECIMAL(18, 2) AS baseline_avg_order_usd, + 0.0::DECIMAL(9, 2) AS basket_uplift_pct, + 10000 AS top_tier_threshold_minor, + 2 AS eligible_region_count + UNION ALL + SELECT + 'UNUSED' AS promo_code, + 0 AS redemption_count, + 0 AS redeeming_customers, + 0.0::DECIMAL(18, 2) AS revenue_usd, + 0.0::DECIMAL(18, 2) AS discount_usd, + CAST(NULL AS DECIMAL(9, 6)) AS discount_rate, + CAST(NULL AS DECIMAL(18, 2)) AS avg_order_usd, + 100.0::DECIMAL(18, 2) AS baseline_avg_order_usd, + CAST(NULL AS DECIMAL(9, 2)) AS basket_uplift_pct, + CAST(NULL AS INT) AS top_tier_threshold_minor, + 0 AS eligible_region_count +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_refunds_usd__converts_at_rate_on_refund_date.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_refunds_usd__converts_at_rate_on_refund_date.sql new file mode 100644 index 0000000000..9fffdb4f14 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_refunds_usd__converts_at_rate_on_refund_date.sql @@ -0,0 +1,124 @@ +TEST (name "int_refunds_usd__converts_at_rate_on_refund_date"); + +WITH __ref__stg_refunds AS ( + SELECT + 'RF1' AS refund_id, + 'P1' AS payment_id, + 2500 AS amount_minor, + NULL AS reason_text, + 'damaged' AS reason_category, + '2026-03-20 23:30:00 -05:00'::TIMESTAMP_TZ AS refunded_at, + NULL AS initiated_by, + NULL AS refunded_line_count, + NULL AS refunded_items, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'RF2', + 'P2', + 500, + NULL, + NULL, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_payments AS ( + SELECT + 'P1' AS payment_id, + 'O1' AS order_id, + NULL AS payment_method, + 2500 AS amount_minor, + 'EUR' AS currency_code, + NULL AS authorised_at, + '2026-03-10 23:00:00 -05:00'::TIMESTAMP_TZ AS captured_at, + NULL AS card_bin, + NULL AS gateway_status, + NULL AS gateway_error_code, + NULL AS risk_score, + NULL AS three_ds_outcome, + NULL AS gateway_reference, + NULL AS gateway_response, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'P2', + 'O2', + NULL, + 1000, + 'GBP', + NULL, + '2026-03-01 09:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-03-20'::DATE AS rate_date, + 1.05::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'EUR', + '2026-03-21'::DATE, + 1.1::NUMBER(18, 8), + NULL, + NULL + UNION ALL + SELECT + 'GBP', + '2026-03-01'::DATE, + 1.25::NUMBER(18, 8), + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'GBP', + NULL, + 2, + NULL +), + +__expected__int_refunds_usd AS ( + SELECT + 'RF1' AS refund_id, + 'O1' AS order_id, + '2026-03-21'::DATE AS refunded_date, + 25.0::DECIMAL(38, 6) AS amount, + 27.5::DECIMAL(18, 2) AS amount_usd, + 10 AS days_after_payment, + TRUE AS is_full_refund + UNION ALL + SELECT + 'RF2' AS refund_id, + 'O2' AS order_id, + '2026-03-05'::DATE AS refunded_date, + 5.0::DECIMAL(38, 6) AS amount, + 6.25::DECIMAL(18, 2) AS amount_usd, + 4 AS days_after_payment, + FALSE AS is_full_refund +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_reviews_moderated__scores_visible_reviews_and_verifies_purchases.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_reviews_moderated__scores_visible_reviews_and_verifies_purchases.sql new file mode 100644 index 0000000000..125637df4a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_reviews_moderated__scores_visible_reviews_and_verifies_purchases.sql @@ -0,0 +1,134 @@ +TEST (name "int_reviews_moderated__scores_visible_reviews_and_verifies_purchases"); + +WITH __ref__stg_reviews AS ( + SELECT + 'V1' AS review_id, + 'DUCK-1' AS sku, + 'C1' AS customer_id, + 5 AS rating, + 'Love it, great and sturdy' AS title, + 0 AS body_length, + NULL AS submitted_at, + 'approved' AS moderation_status, + ARRAY_CONSTRUCT()::ARRAY AS flagged_terms, + NULL AS mentions_url, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'V2', + 'DUCK-1', + 'C2', + 2, + 'Broke after a week', + 0, + NULL, + 'pending', + ARRAY_CONSTRUCT()::ARRAY, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'V3', + 'DUCK-1', + 'C3', + 1, + 'spam', + 0, + NULL, + 'approved', + ARRAY_CONSTRUCT('link')::ARRAY, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'V4', + 'DUCK-1', + 'C4', + 4, + 'fine', + 0, + NULL, + 'rejected', + ARRAY_CONSTRUCT()::ARRAY, + NULL, + NULL, + NULL +), + +__ref__int_order_lines_priced AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + 'DUCK-1' AS sku, + NULL AS sku_family, + 'PR1' AS product_id, + NULL AS quantity, + NULL AS unit_amount_minor, + NULL AS list_amount_minor, + NULL AS markdown_pct, + NULL AS gross_amount_minor, + NULL AS net_amount_minor, + NULL AS discount_amount_minor, + NULL AS tax_amount_minor, + NULL AS line_weight_kg, + NULL AS has_list_price +), + +__ref__stg_orders AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + NULL AS storefront_code, + NULL AS currency_code, + NULL AS placed_at, + NULL AS order_status, + NULL AS sales_channel, + NULL AS is_gift, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS device_type, + NULL AS item_count_hint, + NULL AS shipping_address_id, + NULL AS attributes, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_product_variants AS ( + SELECT + 'DUCK-1' AS sku, + 'PR1' AS product_id, + NULL AS colour, + NULL AS size_label, + NULL AS weight_grams, + NULL AS weight_kg, + NULL AS barcode_count, + NULL AS is_fragile, + NULL AS pack_size, + NULL AS barcodes, + NULL AS attributes, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__int_reviews_moderated AS ( + SELECT + 'V1' AS review_id, + 'PR1' AS product_id, + TRUE AS is_verified_purchase, + 3 AS sentiment_score, + 1.5::DECIMAL(6, 3) AS review_weight, + 1 AS rating_rank_in_product + UNION ALL + SELECT + 'V2' AS review_id, + 'PR1' AS product_id, + FALSE AS is_verified_purchase, + -1 AS sentiment_score, + 1.0::DECIMAL(6, 3) AS review_weight, + 2 AS rating_rank_in_product +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_subscription_mrr_daily__spreads_active_periods_into_daily_usd_mrr.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_subscription_mrr_daily__spreads_active_periods_into_daily_usd_mrr.sql new file mode 100644 index 0000000000..3bbfd62acf --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_subscription_mrr_daily__spreads_active_periods_into_daily_usd_mrr.sql @@ -0,0 +1,63 @@ +TEST (name "int_subscription_mrr_daily__spreads_active_periods_into_daily_usd_mrr"); + +WITH __ref__int_subscription_periods AS ( + SELECT + 'S1#1' AS subscription_period_key, + 'S1' AS subscription_id, + 'C1' AS customer_id, + 1 AS period_number, + 'active' AS period_state, + '2026-01-01 08:00:00 +00:00'::TIMESTAMPTZ AS period_start, + '2026-01-04 08:00:00 +00:00'::TIMESTAMPTZ AS period_end, + NULL AS period_days, + 3000 AS billing_amount_minor, + 'EUR' AS billing_currency_code, + 30 AS interval_days + UNION ALL + SELECT + 'S1#2', + 'S1', + 'C1', + 2, + 'paused', + '2026-01-04 08:00:00 +00:00'::TIMESTAMPTZ, + '2026-01-10 08:00:00 +00:00'::TIMESTAMPTZ, + NULL, + 3000, + 'EUR', + 30 +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2025-12-31'::DATE AS rate_date, + 1.1::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol +), + +__expected__int_subscription_mrr_daily AS ( + SELECT + 'S1:20260101' AS subscription_day_key, + 33.4813::DECIMAL(18, 4) AS mrr_usd, + TRUE AS is_first_day + UNION ALL + SELECT + 'S1:20260102' AS subscription_day_key, + 33.4813::DECIMAL(18, 4) AS mrr_usd, + FALSE AS is_first_day + UNION ALL + SELECT + 'S1:20260103' AS subscription_day_key, + 33.4813::DECIMAL(18, 4) AS mrr_usd, + FALSE AS is_first_day +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_subscription_periods__splits_pause_and_cancel_into_periods.sql b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_subscription_periods__splits_pause_and_cancel_into_periods.sql new file mode 100644 index 0000000000..44642dc701 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/orders/test_int_subscription_periods__splits_pause_and_cancel_into_periods.sql @@ -0,0 +1,91 @@ +TEST (name "int_subscription_periods__splits_pause_and_cancel_into_periods"); + +WITH __ref__stg_subscriptions AS ( + SELECT + 'S1' AS subscription_id, + 'C1' AS customer_id, + NULL AS plan_code, + NULL AS sku, + 30 AS interval_days, + NULL AS subscription_status, + '2026-01-01 00:00:00 +00:00'::TIMESTAMP_TZ AS started_at, + NULL AS cancelled_at, + 2000 AS billing_amount_minor, + 'EUR' AS billing_currency_code, + NULL AS next_bill_on, + NULL AS billing, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_subscription_events AS ( + SELECT + 'E1' AS event_id, + 'S1' AS subscription_id, + 'PAUSE' AS event_type, + '2026-01-11 00:00:00 +00:00'::TIMESTAMP_TZ AS occurred_at, + NULL AS pause_days, + NULL AS skip_reason, + NULL AS payload, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'E2', + 'S1', + 'RESUME', + '2026-01-16 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E3', + 'S1', + 'SKIP', + '2026-01-20 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'E4', + 'S1', + 'CANCEL', + '2026-02-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_subscription_periods AS ( + SELECT + 'S1#1' AS subscription_period_key, + 1 AS period_number, + 'active' AS period_state, + 10.0::DECIMAL(10, 2) AS period_days + UNION ALL + SELECT + 'S1#2' AS subscription_period_key, + 2 AS period_number, + 'paused' AS period_state, + 5.5::DECIMAL(10, 2) AS period_days + UNION ALL + SELECT + 'S1#3' AS subscription_period_key, + 3 AS period_number, + 'active' AS period_state, + 15.5::DECIMAL(10, 2) AS period_days + UNION ALL + SELECT + 'S1#4' AS subscription_period_key, + 4 AS period_number, + 'cancelled' AS period_state, + 0.0::DECIMAL(10, 2) AS period_days +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/partners/test_int_partner_fees_usd__compares_stated_commission_with_listing_terms.sql b/benchmarks/duckshop/project/tests/unit/intermediate/partners/test_int_partner_fees_usd__compares_stated_commission_with_listing_terms.sql new file mode 100644 index 0000000000..2388d3b134 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/partners/test_int_partner_fees_usd__compares_stated_commission_with_listing_terms.sql @@ -0,0 +1,268 @@ +TEST (name "int_partner_fees_usd__compares_stated_commission_with_listing_terms"); + +WITH __ref__stg_partner_settlements AS ( + SELECT + 'ST1' AS settlement_id, + 'MKT' AS partner_code, + '2026-03-01'::DATE AS period_start, + '2026-03-31'::DATE AS period_end, + 'EUR' AS currency_code, + 100000 AS gross_sales_minor, + 15000 AS commission_minor, + 500 AS other_fees_minor, + 84500 AS net_payable_minor, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'ST2', + 'OTHER', + '2026-03-01'::DATE, + '2026-03-15'::DATE, + 'USD', + 10000, + 2000, + NULL, + 7900, + NULL, + NULL +), + +__ref__int_partner_orders_unified AS ( + SELECT + 'PK1' AS partner_order_key, + NULL AS source_relation, + NULL AS payload_version, + 'MKT' AS partner_code, + NULL AS external_order_id, + NULL AS currency_code, + NULL AS placed_at_utc, + '2026-03-05'::DATE AS placed_date, + NULL AS order_total_minor, + NULL AS order_total_usd, + NULL AS line_count, + NULL AS units, + NULL AS lines_total_minor, + NULL AS lines_match_total, + NULL AS customer_id, + NULL AS is_known_customer, + NULL AS order_rank_for_partner + UNION ALL + SELECT + 'PK2', + NULL, + NULL, + 'MKT', + NULL, + NULL, + NULL, + '2026-03-20'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'PK3', + NULL, + NULL, + 'MKT', + NULL, + NULL, + NULL, + '2026-04-02'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'PK4', + NULL, + NULL, + 'OTHER', + NULL, + NULL, + NULL, + '2026-03-05'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_partner_order_lines AS ( + SELECT + 'PK1#1' AS partner_order_line_key, + 'PK1' AS partner_order_key, + NULL AS payload_version, + 'DUCK-1' AS sku, + 2 AS quantity, + 20000::DECIMAL(38, 0) AS unit_amount_minor + UNION ALL + SELECT + 'PK1#2', + 'PK1', + NULL, + 'DUCK-2', + 1, + 60000::DECIMAL(38, 0) + UNION ALL + SELECT + 'PK2#1', + 'PK2', + NULL, + 'DUCK-3', + 1, + 10000::DECIMAL(38, 0) + UNION ALL + SELECT + 'PK3#1', + 'PK3', + NULL, + 'DUCK-1', + 1, + 1000::DECIMAL(38, 0) + UNION ALL + SELECT + 'PK4#1', + 'PK4', + NULL, + 'DUCK-1', + 1, + 10000::DECIMAL(38, 0) +), + +__ref__stg_partner_listings AS ( + SELECT + 'L1' AS listing_id, + 'MKT' AS partner_code, + 'DUCK-1' AS sku, + 'live' AS listing_status, + '2026-02-01 10:00:00 +00:00'::TIMESTAMP_TZ AS listed_at, + 15.0::DECIMAL(5, 2) AS commission_pct, + FALSE AS fulfilled_by_partner, + NULL AS terms, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'L0', + 'MKT', + 'DUCK-1', + 'ended', + '2025-01-01 10:00:00 +00:00'::TIMESTAMP_TZ, + 12.0::DECIMAL(5, 2), + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'L2', + 'MKT', + 'DUCK-2', + 'live', + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + 15.0::DECIMAL(5, 2), + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'L3', + 'OTHER', + 'DUCK-1', + 'live', + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + 18.0::DECIMAL(5, 2), + FALSE, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-03-01'::DATE AS rate_date, + 1.0::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'EUR', + '2026-03-31'::DATE, + 1.1::NUMBER(18, 8), + NULL, + NULL + UNION ALL + SELECT + 'USD', + '2026-01-01'::DATE, + 1.0::NUMBER(18, 8), + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'USD', + NULL, + 2, + NULL +), + +__expected__int_partner_fees_usd AS ( + SELECT + 'ST1' AS settlement_id, + 31 AS period_days, + 1100.0::DECIMAL(18, 2) AS gross_sales_usd, + 165.0::DECIMAL(18, 2) AS stated_commission_usd, + 165.0::DECIMAL(18, 2) AS expected_commission_usd, + 5.5::DECIMAL(18, 2) AS other_fees_usd, + 929.5::DECIMAL(18, 2) AS net_payable_usd, + 2 AS period_order_count, + 1 AS lines_without_listing, + 15.0::DECIMAL(9, 4) AS effective_commission_pct, + 0.0::DECIMAL(9, 4) AS commission_variance_pct, + TRUE AS is_within_tolerance, + TRUE AS statement_balances + UNION ALL + SELECT + 'ST2' AS settlement_id, + 15 AS period_days, + 100.0::DECIMAL(18, 2) AS gross_sales_usd, + 20.0::DECIMAL(18, 2) AS stated_commission_usd, + 18.0::DECIMAL(18, 2) AS expected_commission_usd, + CAST(NULL AS DECIMAL(18, 2)) AS other_fees_usd, + 79.0::DECIMAL(18, 2) AS net_payable_usd, + 1 AS period_order_count, + 0 AS lines_without_listing, + 20.0::DECIMAL(9, 4) AS effective_commission_pct, + 11.1111::DECIMAL(9, 4) AS commission_variance_pct, + FALSE AS is_within_tolerance, + FALSE AS statement_balances +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/partners/test_int_partner_listing_status__grades_listing_health_from_catalogue_stock_and_price.sql b/benchmarks/duckshop/project/tests/unit/intermediate/partners/test_int_partner_listing_status__grades_listing_health_from_catalogue_stock_and_price.sql new file mode 100644 index 0000000000..b052f64463 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/partners/test_int_partner_listing_status__grades_listing_health_from_catalogue_stock_and_price.sql @@ -0,0 +1,492 @@ +TEST (name "int_partner_listing_status__grades_listing_health_from_catalogue_stock_and_price"); + +WITH __ref__stg_partner_listings AS ( + SELECT + 'LS1' AS listing_id, + 'MKT' AS partner_code, + 'DUCK-1' AS sku, + 'live' AS listing_status, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS listed_at, + 15.0::DECIMAL(5, 2) AS commission_pct, + FALSE AS fulfilled_by_partner, + PARSE_JSON('{"shipping": "free", "returns": 30}')::OBJECT AS terms, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'LS2', + 'OTHER', + 'DUCK-1', + 'on_sale', + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + 12.0::DECIMAL(5, 2), + TRUE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'LS3', + 'MKT', + 'DUCK-2', + 'out_of_stock', + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'LS4', + 'MKT', + 'DUCK-9', + 'published', + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'LS5', + 'MKT', + 'DUCK-3', + 'live', + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'LS6', + 'MKT', + 'DUCK-4', + 'pending_review', + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'LS7', + 'MKT', + 'DUCK-5', + 'live', + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'LS8', + 'OTHER', + 'DUCK-6', + 'live', + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + TRUE, + NULL, + NULL, + NULL +), + +__ref__int_product_catalogue AS ( + SELECT + 'DUCK-1' AS sku, + NULL AS product_id, + NULL AS sku_family, + 'Mallard Classic Duck' AS title, + NULL AS variant_label, + NULL AS brand_code, + 'Mallard & Co.' AS brand_name, + NULL AS is_private_label, + NULL AS category_id, + NULL AS category_slug_path, + NULL AS category_level_1, + NULL AS material, + NULL AS weight_kg, + NULL AS primary_image_url, + NULL AS image_count, + TRUE AS is_active, + NULL AS launched_on, + NULL AS is_bundle_component + UNION ALL + SELECT + 'DUCK-2', + NULL, + NULL, + 'Pintail Mug', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'DUCK-3', + NULL, + NULL, + 'Retired Eider Decoy', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'DUCK-4', + NULL, + NULL, + 'Merganser Teapot', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'DUCK-5', + NULL, + NULL, + 'Teal Pond Pump', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'DUCK-6', + NULL, + NULL, + 'Paddling Six Bundle', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL +), + +__ref__int_reservations_open AS ( + SELECT + 'DUCK-1|W1' AS stock_key, + 'DUCK-1' AS sku, + 'W1' AS warehouse_code, + NULL AS open_reservations, + NULL AS reserved_qty, + NULL AS backordered_qty, + NULL AS on_hand_qty, + 5 AS available_to_promise, + NULL AS oldest_reserved_at + UNION ALL + SELECT + 'DUCK-1|W2', + 'DUCK-1', + 'W2', + NULL, + NULL, + NULL, + NULL, + -3, + NULL +), + +__ref__int_price_history AS ( + SELECT + 'P1' AS price_list_id, + 'DUCK-1' AS sku, + 'USD' AS currency_code, + 2999 AS list_amount_minor, + '2026-01-01 00:00:00 +00:00'::TIMESTAMPTZ AS valid_from, + NULL AS effective_valid_to, + NULL AS previous_amount_minor, + NULL AS change_pct, + NULL AS is_markdown, + NULL AS price_version, + NULL AS days_at_price, + NULL AS lowest_price_30d_minor + UNION ALL + SELECT + 'P2', + 'DUCK-1', + 'EUR', + 2799, + '2026-02-01 00:00:00 +00:00'::TIMESTAMPTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'P0', + 'DUCK-1', + 'USD', + 3199, + '2025-01-01 00:00:00 +00:00'::TIMESTAMPTZ, + '2026-01-01 00:00:00 +00:00'::TIMESTAMPTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'P3', + 'DUCK-5', + 'EUR', + 4500, + '2026-01-01 00:00:00 +00:00'::TIMESTAMPTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_partner_orders_unified AS ( + SELECT + 'PK1' AS partner_order_key, + NULL AS source_relation, + NULL AS payload_version, + 'MKT' AS partner_code, + NULL AS external_order_id, + NULL AS currency_code, + NULL AS placed_at_utc, + '2026-03-20'::DATE AS placed_date, + NULL AS order_total_minor, + NULL AS order_total_usd, + NULL AS line_count, + NULL AS units, + NULL AS lines_total_minor, + NULL AS lines_match_total, + NULL AS customer_id, + NULL AS is_known_customer, + NULL AS order_rank_for_partner + UNION ALL + SELECT + 'PK2', + NULL, + NULL, + 'MKT', + NULL, + NULL, + NULL, + '2026-02-15'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'PK3', + NULL, + NULL, + 'OTHER', + NULL, + NULL, + NULL, + '2026-03-30'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_partner_order_lines AS ( + SELECT + 'PK1#1' AS partner_order_line_key, + 'PK1' AS partner_order_key, + NULL AS payload_version, + 'DUCK-1' AS sku, + 3 AS quantity, + 2999::DECIMAL(38, 0) AS unit_amount_minor + UNION ALL + SELECT + 'PK2#1', + 'PK2', + NULL, + 'DUCK-1', + 10, + 2999::DECIMAL(38, 0) + UNION ALL + SELECT + 'PK3#1', + 'PK3', + NULL, + 'DUCK-1', + 1, + 2999::DECIMAL(38, 0) +), + +__expected__int_partner_listing_status AS ( + SELECT + 'LS1' AS listing_id, + 'live' AS listing_state, + 30 AS days_listed, + 2999 AS current_list_price_minor, + 'USD' AS list_price_currency_code, + 5 AS available_to_promise, + 'home_delivery' AS fulfilment_method, + 3 AS units_sold_30d, + 2 AS partners_listing_sku, + 'healthy' AS listing_health + UNION ALL + SELECT + 'LS2' AS listing_id, + 'live' AS listing_state, + 30 AS days_listed, + 2999 AS current_list_price_minor, + 'USD' AS list_price_currency_code, + 5 AS available_to_promise, + 'partner' AS fulfilment_method, + 1 AS units_sold_30d, + 2 AS partners_listing_sku, + 'healthy' AS listing_health + UNION ALL + SELECT + 'LS3' AS listing_id, + 'paused' AS listing_state, + 30 AS days_listed, + CAST(NULL AS INT) AS current_list_price_minor, + CAST(NULL AS VARCHAR) AS list_price_currency_code, + 0 AS available_to_promise, + 'home_delivery' AS fulfilment_method, + 0 AS units_sold_30d, + 1 AS partners_listing_sku, + 'not_live' AS listing_health + UNION ALL + SELECT + 'LS4' AS listing_id, + 'live' AS listing_state, + 30 AS days_listed, + CAST(NULL AS INT) AS current_list_price_minor, + CAST(NULL AS VARCHAR) AS list_price_currency_code, + 0 AS available_to_promise, + 'home_delivery' AS fulfilment_method, + 0 AS units_sold_30d, + 1 AS partners_listing_sku, + 'unknown_sku' AS listing_health + UNION ALL + SELECT + 'LS5' AS listing_id, + 'live' AS listing_state, + 30 AS days_listed, + CAST(NULL AS INT) AS current_list_price_minor, + CAST(NULL AS VARCHAR) AS list_price_currency_code, + 0 AS available_to_promise, + 'home_delivery' AS fulfilment_method, + 0 AS units_sold_30d, + 1 AS partners_listing_sku, + 'discontinued' AS listing_health + UNION ALL + SELECT + 'LS6' AS listing_id, + 'draft' AS listing_state, + 30 AS days_listed, + CAST(NULL AS INT) AS current_list_price_minor, + CAST(NULL AS VARCHAR) AS list_price_currency_code, + 0 AS available_to_promise, + 'home_delivery' AS fulfilment_method, + 0 AS units_sold_30d, + 1 AS partners_listing_sku, + 'not_live' AS listing_health + UNION ALL + SELECT + 'LS7' AS listing_id, + 'live' AS listing_state, + 30 AS days_listed, + 4500 AS current_list_price_minor, + 'EUR' AS list_price_currency_code, + 0 AS available_to_promise, + 'home_delivery' AS fulfilment_method, + 0 AS units_sold_30d, + 1 AS partners_listing_sku, + 'unfulfillable' AS listing_health + UNION ALL + SELECT + 'LS8' AS listing_id, + 'live' AS listing_state, + 30 AS days_listed, + CAST(NULL AS INT) AS current_list_price_minor, + CAST(NULL AS VARCHAR) AS list_price_currency_code, + 0 AS available_to_promise, + 'partner' AS fulfilment_method, + 0 AS units_sold_30d, + 1 AS partners_listing_sku, + 'unpriced' AS listing_health +), + +__assert__term_keys_list_object_keys AS ( + SELECT l.listing_id + FROM __ref("int_partner_listing_status") AS l + WHERE ARRAY_TO_STRING(l.term_keys, ',') <> IFF(l.listing_id = 'LS1', 'returns,shipping', '') +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/partners/test_int_partner_orders_unified__unifies_payload_versions_and_links_hashed_buyers.sql b/benchmarks/duckshop/project/tests/unit/intermediate/partners/test_int_partner_orders_unified__unifies_payload_versions_and_links_hashed_buyers.sql new file mode 100644 index 0000000000..697b80b3fe --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/partners/test_int_partner_orders_unified__unifies_payload_versions_and_links_hashed_buyers.sql @@ -0,0 +1,186 @@ +TEST (name "int_partner_orders_unified__unifies_payload_versions_and_links_hashed_buyers"); + +WITH __ref__stg_partner_orders_v1 AS ( + SELECT + 'PK1' AS partner_order_key, + 'MKT' AS partner_code, + 'A1' AS external_order_id, + 'a@example.com' AS buyer_email, + 'EUR' AS currency_code, + 12.5::DECIMAL(18, 3) AS order_total, + '2026-03-10 23:30:00 -02:00'::TIMESTAMP_TZ AS placed_at, + NULL AS lines +), + +__ref__stg_partner_orders_v2 AS ( + SELECT + 'PK2' AS partner_order_key, + 'MKT' AS partner_code, + 'B7' AS external_order_id, + SHA2('b@example.com', 256) AS buyer_email_hash, + 'KWD' AS currency_code, + 1500::DECIMAL(38, 0) AS order_total_minor, + '2026-03-10 08:00:00'::TIMESTAMP_NTZ AS placed_at_utc, + NULL AS items + UNION ALL + SELECT + 'PK3', + 'OTHER', + 'C1', + SHA2('z@example.com', 256), + 'USD', + 999::DECIMAL(38, 0), + '2026-03-12 10:00:00'::TIMESTAMP_NTZ, + NULL +), + +__ref__stg_partner_order_lines AS ( + SELECT + 'PK1#1' AS partner_order_line_key, + 'PK1' AS partner_order_key, + NULL AS payload_version, + 'DUCK-1' AS sku, + 2 AS quantity, + 500::DECIMAL(38, 0) AS unit_amount_minor + UNION ALL + SELECT + 'PK1#2', + 'PK1', + NULL, + 'DUCK-2', + 1, + 250::DECIMAL(38, 0) + UNION ALL + SELECT + 'PK2#1', + 'PK2', + NULL, + 'DUCK-1', + 1, + 1400::DECIMAL(38, 0) +), + +__ref__int_customer_identity AS ( + SELECT + 'C1' AS customer_id, + 'a@example.com' AS email_normalised, + NULL AS email_domain, + NULL AS email_dedupe_key, + NULL AS phone_normalised, + NULL AS full_name, + NULL AS anonymous_ids, + NULL AS anonymous_id_count, + NULL AS newsletter_lists, + NULL AS ticket_count, + NULL AS duplicate_candidate_id, + NULL AS duplicate_name_similarity, + NULL AS accounts_sharing_email + UNION ALL + SELECT + 'C2', + 'b@example.com', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-03-11'::DATE AS rate_date, + 1.1::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'KWD', + '2026-03-01'::DATE, + 3.25::NUMBER(18, 8), + NULL, + NULL + UNION ALL + SELECT + 'USD', + '2026-03-01'::DATE, + 1.0::NUMBER(18, 8), + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'KWD', + NULL, + 3, + NULL + UNION ALL + SELECT + 'USD', + NULL, + 2, + NULL +), + +__expected__int_partner_orders_unified AS ( + SELECT + 'PK1' AS partner_order_key, + 'stg_partner_orders_v1' AS source_relation, + 1 AS payload_version, + '2026-03-11 01:30:00'::TIMESTAMPNTZ AS placed_at_utc, + '2026-03-11'::DATE AS placed_date, + 1250 AS order_total_minor, + 13.75::DECIMAL(18, 2) AS order_total_usd, + 2 AS line_count, + 3 AS units, + 1250 AS lines_total_minor, + TRUE AS lines_match_total, + 'C1' AS customer_id, + TRUE AS is_known_customer, + 2 AS order_rank_for_partner + UNION ALL + SELECT + 'PK2' AS partner_order_key, + 'stg_partner_orders_v2' AS source_relation, + 2 AS payload_version, + '2026-03-10 08:00:00'::TIMESTAMPNTZ AS placed_at_utc, + '2026-03-10'::DATE AS placed_date, + 1500 AS order_total_minor, + 4.88::DECIMAL(18, 2) AS order_total_usd, + 1 AS line_count, + 1 AS units, + 1400 AS lines_total_minor, + FALSE AS lines_match_total, + 'C2' AS customer_id, + TRUE AS is_known_customer, + 1 AS order_rank_for_partner + UNION ALL + SELECT + 'PK3' AS partner_order_key, + 'stg_partner_orders_v2' AS source_relation, + 2 AS payload_version, + '2026-03-12 10:00:00'::TIMESTAMPNTZ AS placed_at_utc, + '2026-03-12'::DATE AS placed_date, + 999 AS order_total_minor, + 9.99::DECIMAL(18, 2) AS order_total_usd, + 0 AS line_count, + 0 AS units, + 0 AS lines_total_minor, + FALSE AS lines_match_total, + CAST(NULL AS VARCHAR) AS customer_id, + FALSE AS is_known_customer, + 1 AS order_rank_for_partner +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_agent_utilisation__converts_overnight_shift_across_dst_change.sql b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_agent_utilisation__converts_overnight_shift_across_dst_change.sql new file mode 100644 index 0000000000..411f669e46 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_agent_utilisation__converts_overnight_shift_across_dst_change.sql @@ -0,0 +1,130 @@ +TEST (name "int_agent_utilisation__converts_overnight_shift_across_dst_change"); + +WITH __ref__stg_agent_shifts AS ( + SELECT + 'SH1' AS shift_id, + 'A1' AS agent_id, + '2026-03-07 22:00:00'::TIMESTAMP_NTZ AS shift_start_local, + '2026-03-08 06:00:00'::TIMESTAMP_NTZ AS shift_end_local, + 'America/New_York' AS iana_timezone, + 480 AS scheduled_minutes, + 30 AS break_minutes, + NULL AS breaks, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'SH2', + 'A2', + '2026-03-09 09:00:00'::TIMESTAMP_NTZ, + '2026-03-09 17:00:00'::TIMESTAMP_NTZ, + 'Europe/London', + 480, + 60, + NULL, + NULL, + NULL +), + +__ref__stg_agents AS ( + SELECT + 'A1' AS agent_id, + NULL AS display_name, + NULL AS email, + 'NIGHT' AS team_code, + NULL AS iana_timezone, + NULL AS hired_on, + NULL AS skills, + NULL AS speaks_german, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__ref__stg_call_records AS ( + SELECT + 'C1' AS call_id, + NULL AS ticket_id, + 'A1' AS agent_id, + NULL AS caller_phone, + NULL AS caller_country_prefix, + '2026-03-08 04:00:00 +00:00'::TIMESTAMP_TZ AS started_at, + 600 AS duration_seconds, + NULL AS ivr_path, + NULL AS ivr_depth, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'C2', + NULL, + 'A1', + NULL, + NULL, + '2026-03-08 09:30:00 +00:00'::TIMESTAMP_TZ, + 300, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C3', + NULL, + 'A1', + NULL, + NULL, + '2026-03-08 11:00:00 +00:00'::TIMESTAMP_TZ, + 900, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_ticket_first_response AS ( + SELECT + 'T1' AS ticket_id, + NULL AS created_at_utc, + '2026-03-08 05:00:00'::TIMESTAMPNTZ AS first_response_at_utc, + 'A1' AS first_responder_id, + NULL AS minutes_to_first_response, + NULL AS first_customer_reply_at_utc, + NULL AS agent_responses_before_customer_reply, + NULL AS has_auto_reply + UNION ALL + SELECT + 'T2', + NULL, + '2026-03-08 10:00:00'::TIMESTAMPNTZ, + 'A1', + NULL, + NULL, + NULL, + NULL +), + +__expected__int_agent_utilisation AS ( + SELECT + 'SH1' AS shift_id, + 'NIGHT' AS team_code, + '2026-03-08 03:00:00'::TIMESTAMPNTZ AS shift_start_utc, + '2026-03-08 10:00:00'::TIMESTAMPNTZ AS shift_end_utc, + 450 AS productive_minutes, + 15.0::DECIMAL(10, 2) AS call_minutes, + 2 AS calls_handled, + 1 AS tickets_first_responded, + 0.0467::DECIMAL(9, 4) AS utilisation, + TRUE AS is_overnight + UNION ALL + SELECT + 'SH2' AS shift_id, + CAST(NULL AS VARCHAR) AS team_code, + '2026-03-09 09:00:00'::TIMESTAMPNTZ AS shift_start_utc, + '2026-03-09 17:00:00'::TIMESTAMPNTZ AS shift_end_utc, + 420 AS productive_minutes, + 0.0::DECIMAL(10, 2) AS call_minutes, + 0 AS calls_handled, + 0 AS tickets_first_responded, + 0.0::DECIMAL(9, 4) AS utilisation, + FALSE AS is_overnight +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_call_ivr_paths__summarises_latest_menu_path_per_call.sql b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_call_ivr_paths__summarises_latest_menu_path_per_call.sql new file mode 100644 index 0000000000..fe141eb519 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_call_ivr_paths__summarises_latest_menu_path_per_call.sql @@ -0,0 +1,147 @@ +TEST (name "int_call_ivr_paths__summarises_latest_menu_path_per_call"); + +WITH __source__support__call_records AS ( + SELECT + 'C1' AS call_id, + NULL AS ticket_id, + NULL AS agent_id, + NULL AS phone_number, + NULL AS started_at, + NULL AS duration_seconds, + PARSE_JSON('["Orders", "Returns"]')::VARIANT AS ivr_path, + NULL AS _op, + 1 AS _seq, + NULL AS _loaded_at + UNION ALL + SELECT + 'C1', + NULL, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('["ORDERS", "returns", "orders", "self_service_complete"]')::VARIANT, + NULL, + 2, + NULL + UNION ALL + SELECT + 'C2', + NULL, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('["billing"]')::VARIANT, + NULL, + 1, + NULL + UNION ALL + SELECT + 'C4', + NULL, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('["support", "hold"]')::VARIANT, + NULL, + 1, + NULL +), + +__ref__stg_call_records AS ( + SELECT + 'C1' AS call_id, + NULL AS ticket_id, + NULL AS agent_id, + NULL AS caller_phone, + '44' AS caller_country_prefix, + NULL AS started_at, + 45 AS duration_seconds, + NULL AS ivr_path, + NULL AS ivr_depth, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'C2', + NULL, + 'A1', + NULL, + '1', + NULL, + 400, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C3', + NULL, + NULL, + NULL, + '965', + NULL, + 1000, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C4', + NULL, + 'A2', + NULL, + NULL, + NULL, + 120, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_call_ivr_paths AS ( + SELECT + 'C1' AS call_id, + 'orders' AS first_menu_choice, + 'self_service_complete' AS final_menu_node, + 4 AS menu_steps, + 1 AS repeated_menu_visits, + FALSE AS abandoned_in_ivr, + 'EU' AS caller_region, + 'under_1m' AS duration_bucket + UNION ALL + SELECT + 'C2' AS call_id, + 'billing' AS first_menu_choice, + 'billing' AS final_menu_node, + 1 AS menu_steps, + 0 AS repeated_menu_visits, + FALSE AS abandoned_in_ivr, + 'US' AS caller_region, + '5_to_15m' AS duration_bucket + UNION ALL + SELECT + 'C3' AS call_id, + CAST(NULL AS VARCHAR) AS first_menu_choice, + CAST(NULL AS VARCHAR) AS final_menu_node, + 0 AS menu_steps, + 0 AS repeated_menu_visits, + TRUE AS abandoned_in_ivr, + 'MENA' AS caller_region, + 'over_15m' AS duration_bucket + UNION ALL + SELECT + 'C4' AS call_id, + 'support' AS first_menu_choice, + 'hold' AS final_menu_node, + 2 AS menu_steps, + 0 AS repeated_menu_visits, + FALSE AS abandoned_in_ivr, + 'EU' AS caller_region, + '1_to_5m' AS duration_bucket +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_chat_sentiment__scores_customer_messages_and_agent_reply_times.sql b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_chat_sentiment__scores_customer_messages_and_agent_reply_times.sql new file mode 100644 index 0000000000..d1d1c0db56 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_chat_sentiment__scores_customer_messages_and_agent_reply_times.sql @@ -0,0 +1,130 @@ +TEST (name "int_chat_sentiment__scores_customer_messages_and_agent_reply_times"); + +WITH __ref__stg_chat_messages AS ( + SELECT + 'CH1:1' AS chat_message_key, + 'CH1' AS chat_id, + 1 AS message_index, + 'customer' AS sender_role, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS sent_at, + 'This is terrible, the item arrived broken' AS message_text_masked, + NULL AS message_length + UNION ALL + SELECT + 'CH1:2', + 'CH1', + 2, + 'agent', + '2026-03-10 10:01:30 +00:00'::TIMESTAMP_TZ, + 'Sorry to hear that', + NULL + UNION ALL + SELECT + 'CH1:3', + 'CH1', + 3, + 'customer', + '2026-03-10 10:03:00 +00:00'::TIMESTAMP_TZ, + 'Thanks, that is helpful', + NULL + UNION ALL + SELECT + 'CH1:4', + 'CH1', + 4, + 'agent', + '2026-03-10 10:03:30 +00:00'::TIMESTAMP_TZ, + 'A replacement is on its way', + NULL + UNION ALL + SELECT + 'CH1:5', + 'CH1', + 5, + 'customer', + '2026-03-10 10:04:00 +00:00'::TIMESTAMP_TZ, + 'Great, thank you', + NULL + UNION ALL + SELECT + 'CH2:1', + 'CH2', + 1, + 'customer', + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + 'Where is my order', + NULL + UNION ALL + SELECT + 'CH2:2', + 'CH2', + 2, + 'customer', + '2026-03-10 10:00:20 +00:00'::TIMESTAMP_TZ, + 'I want to cancel', + NULL +), + +__ref__stg_chat_transcripts AS ( + SELECT + 'CH1' AS chat_id, + 'T1' AS ticket_id, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS started_at, + '2026-03-10 10:05:30 +00:00'::TIMESTAMP_TZ AS ended_at, + NULL AS message_count, + NULL AS messages, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'CH2', + 'T2', + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + '2026-03-10 10:00:20 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'CH3', + 'T3', + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_chat_sentiment AS ( + SELECT + 'CH1' AS chat_id, + 3 AS customer_message_count, + 2 AS agent_message_count, + 2 AS sentiment_score, + 'positive' AS sentiment_label, + 60 AS median_agent_reply_seconds, + TRUE AS ends_with_thanks, + 5.5::DECIMAL(10, 2) AS chat_minutes + UNION ALL + SELECT + 'CH2' AS chat_id, + 2 AS customer_message_count, + 0 AS agent_message_count, + -1 AS sentiment_score, + 'negative' AS sentiment_label, + CAST(NULL AS INT) AS median_agent_reply_seconds, + FALSE AS ends_with_thanks, + 0.33::DECIMAL(10, 2) AS chat_minutes + UNION ALL + SELECT + 'CH3' AS chat_id, + 0 AS customer_message_count, + 0 AS agent_message_count, + 0 AS sentiment_score, + 'neutral' AS sentiment_label, + CAST(NULL AS INT) AS median_agent_reply_seconds, + FALSE AS ends_with_thanks, + CAST(NULL AS DECIMAL(10, 2)) AS chat_minutes +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_customer_support_history__sequences_tickets_per_resolved_customer.sql b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_customer_support_history__sequences_tickets_per_resolved_customer.sql new file mode 100644 index 0000000000..b9d69cf0b5 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_customer_support_history__sequences_tickets_per_resolved_customer.sql @@ -0,0 +1,252 @@ +TEST (name "int_customer_support_history__sequences_tickets_per_resolved_customer"); + +WITH __ref__stg_tickets AS ( + SELECT + 'T1' AS ticket_id, + 'a@example.com' AS customer_email, + NULL AS channel, + NULL AS priority, + NULL AS ticket_status, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_LTZ AS created_at, + NULL AS solved_at, + NULL AS order_id, + 'delivery' AS contact_reason, + NULL AS tag_list, + NULL AS is_vip, + NULL AS custom_fields, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + 'a@example.com', + NULL, + NULL, + NULL, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_LTZ, + NULL, + NULL, + 'delivery', + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T3', + 'a@example.com', + NULL, + NULL, + NULL, + '2026-03-20 10:00:00 +00:00'::TIMESTAMP_LTZ, + NULL, + NULL, + 'delivery', + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T4', + 'a@example.com', + NULL, + NULL, + NULL, + '2026-03-21 10:00:00 +00:00'::TIMESTAMP_LTZ, + NULL, + NULL, + 'delivery', + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T5', + 'unknown@example.com', + NULL, + NULL, + NULL, + '2026-03-01 12:00:00 +00:00'::TIMESTAMP_LTZ, + NULL, + NULL, + 'delivery', + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T6', + 'other@example.com', + NULL, + NULL, + NULL, + '2026-03-02 12:00:00 +00:00'::TIMESTAMP_LTZ, + NULL, + NULL, + 'delivery', + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_customer_identity AS ( + SELECT + 'C2' AS customer_id, + 'a@example.com' AS email_normalised, + NULL AS email_domain, + NULL AS email_dedupe_key, + NULL AS phone_normalised, + NULL AS full_name, + NULL AS anonymous_ids, + NULL AS anonymous_id_count, + NULL AS newsletter_lists, + NULL AS ticket_count, + NULL AS duplicate_candidate_id, + NULL AS duplicate_name_similarity, + NULL AS accounts_sharing_email + UNION ALL + SELECT + 'C1', + 'a@example.com', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C3', + 'b@example.com', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_csat_responses AS ( + SELECT + 'R1' AS response_id, + 'T1' AS ticket_id, + 2 AS score, + NULL AS is_satisfied, + NULL AS comment_text, + '2026-03-02 10:00:00 +00:00'::TIMESTAMP_TZ AS responded_at, + NULL AS effort_score, + NULL AS answers, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'R2', + 'T1', + 4, + NULL, + NULL, + '2026-03-03 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'R3', + 'T2', + 5, + NULL, + NULL, + '2026-03-06 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'R4', + 'T4', + 3, + NULL, + NULL, + '2026-03-22 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_customer_support_history AS ( + SELECT + 'T1' AS ticket_id, + 'C1' AS customer_id, + 1 AS ticket_number_for_customer, + CAST(NULL AS INT) AS days_since_previous_ticket, + 4 AS csat_score, + 4.0::DECIMAL(6, 3) AS rolling_csat_avg, + FALSE AS is_repeat_contact_7d + UNION ALL + SELECT + 'T2' AS ticket_id, + 'C1' AS customer_id, + 2 AS ticket_number_for_customer, + 4 AS days_since_previous_ticket, + 5 AS csat_score, + 4.5::DECIMAL(6, 3) AS rolling_csat_avg, + TRUE AS is_repeat_contact_7d + UNION ALL + SELECT + 'T3' AS ticket_id, + 'C1' AS customer_id, + 3 AS ticket_number_for_customer, + 15 AS days_since_previous_ticket, + CAST(NULL AS INT) AS csat_score, + 4.5::DECIMAL(6, 3) AS rolling_csat_avg, + FALSE AS is_repeat_contact_7d + UNION ALL + SELECT + 'T4' AS ticket_id, + 'C1' AS customer_id, + 4 AS ticket_number_for_customer, + 1 AS days_since_previous_ticket, + 3 AS csat_score, + 4.0::DECIMAL(6, 3) AS rolling_csat_avg, + TRUE AS is_repeat_contact_7d + UNION ALL + SELECT + 'T5' AS ticket_id, + CAST(NULL AS VARCHAR) AS customer_id, + CAST(NULL AS INT) AS ticket_number_for_customer, + CAST(NULL AS INT) AS days_since_previous_ticket, + CAST(NULL AS INT) AS csat_score, + CAST(NULL AS DECIMAL(6, 3)) AS rolling_csat_avg, + FALSE AS is_repeat_contact_7d + UNION ALL + SELECT + 'T6' AS ticket_id, + CAST(NULL AS VARCHAR) AS customer_id, + CAST(NULL AS INT) AS ticket_number_for_customer, + 1 AS days_since_previous_ticket, + CAST(NULL AS INT) AS csat_score, + CAST(NULL AS DECIMAL(6, 3)) AS rolling_csat_avg, + TRUE AS is_repeat_contact_7d +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_business_hours__counts_support_hours_across_holidays_and_dst.sql b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_business_hours__counts_support_hours_across_holidays_and_dst.sql new file mode 100644 index 0000000000..3f2c9f973c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_business_hours__counts_support_hours_across_holidays_and_dst.sql @@ -0,0 +1,94 @@ +TEST (name "int_ticket_business_hours__counts_support_hours_across_holidays_and_dst"); + +WITH __ref__int_ticket_first_response AS ( + SELECT + 'T1' AS ticket_id, + '2026-04-02 17:00:00'::TIMESTAMPNTZ AS created_at_utc, + '2026-04-07 07:30:00'::TIMESTAMPNTZ AS first_response_at_utc, + NULL AS first_responder_id, + NULL AS minutes_to_first_response, + NULL AS first_customer_reply_at_utc, + NULL AS agent_responses_before_customer_reply, + NULL AS has_auto_reply + UNION ALL + SELECT + 'T2', + '2026-04-11 10:00:00'::TIMESTAMPNTZ, + '2026-04-11 10:05:00'::TIMESTAMPNTZ, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T3', + '2026-04-08 04:00:00'::TIMESTAMPNTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T4', + '2026-03-27 18:30:00'::TIMESTAMPNTZ, + '2026-03-30 06:30:00'::TIMESTAMPNTZ, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__seed__holiday_calendar AS ( + SELECT + 'EU' AS region_code, + '2026-04-03'::DATE AS holiday_date, + 'Good Friday' AS holiday_name + UNION ALL + SELECT + 'EU', + '2026-04-06'::DATE, + 'Easter Monday' + UNION ALL + SELECT + 'US', + '2026-04-07'::DATE, + 'Synthetic holiday' +), + +__expected__int_ticket_business_hours AS ( + SELECT + 'T1' AS ticket_id, + '2026-04-02 19:00:00'::TIMESTAMPNTZ AS created_at_support_local, + '2026-04-07 09:30:00'::TIMESTAMPNTZ AS first_response_at_support_local, + 6630 AS calendar_minutes_to_response, + 150 AS business_minutes_to_response, + FALSE AS created_outside_hours + UNION ALL + SELECT + 'T2' AS ticket_id, + '2026-04-11 12:00:00'::TIMESTAMPNTZ AS created_at_support_local, + '2026-04-11 12:05:00'::TIMESTAMPNTZ AS first_response_at_support_local, + 5 AS calendar_minutes_to_response, + 0 AS business_minutes_to_response, + TRUE AS created_outside_hours + UNION ALL + SELECT + 'T3' AS ticket_id, + '2026-04-08 06:00:00'::TIMESTAMPNTZ AS created_at_support_local, + CAST(NULL AS TIMESTAMPNTZ) AS first_response_at_support_local, + CAST(NULL AS INT) AS calendar_minutes_to_response, + 0 AS business_minutes_to_response, + TRUE AS created_outside_hours + UNION ALL + SELECT + 'T4' AS ticket_id, + '2026-03-27 19:30:00'::TIMESTAMPNTZ AS created_at_support_local, + '2026-03-30 08:30:00'::TIMESTAMPNTZ AS first_response_at_support_local, + 3660 AS calendar_minutes_to_response, + 60 AS business_minutes_to_response, + FALSE AS created_outside_hours +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_first_response__finds_first_public_agent_reply_and_burst.sql b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_first_response__finds_first_public_agent_reply_and_burst.sql new file mode 100644 index 0000000000..3496f3ae61 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_first_response__finds_first_public_agent_reply_and_burst.sql @@ -0,0 +1,189 @@ +TEST (name "int_ticket_first_response__finds_first_public_agent_reply_and_burst"); + +WITH __ref__stg_tickets AS ( + SELECT + 'T1' AS ticket_id, + NULL AS customer_email, + NULL AS channel, + NULL AS priority, + NULL AS ticket_status, + '2026-03-10 10:00:00 +01:00'::TIMESTAMP_LTZ AS created_at, + NULL AS solved_at, + NULL AS order_id, + NULL AS contact_reason, + NULL AS tag_list, + NULL AS is_vip, + NULL AS custom_fields, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + NULL, + NULL, + NULL, + NULL, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_LTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T3', + NULL, + NULL, + NULL, + NULL, + '2026-03-11 08:00:00 +00:00'::TIMESTAMP_LTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_ticket_comments AS ( + SELECT + 'c1' AS comment_id, + 'T1' AS ticket_id, + 'system' AS author_type, + NULL AS author_id, + TRUE AS is_public, + '2026-03-10 09:00:30 +00:00'::TIMESTAMP_TZ AS created_at, + NULL AS body_word_count, + FALSE AS contains_order_reference, + 0 AS attachment_count, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'c2', + 'T1', + 'agent', + 'A1', + FALSE, + '2026-03-10 09:10:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + 0, + NULL, + NULL + UNION ALL + SELECT + 'c3', + 'T1', + 'agent', + 'A2', + TRUE, + '2026-03-10 09:20:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + 0, + NULL, + NULL + UNION ALL + SELECT + 'c4', + 'T1', + 'agent', + 'A2', + TRUE, + '2026-03-10 09:25:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + 0, + NULL, + NULL + UNION ALL + SELECT + 'c5', + 'T1', + 'customer', + 'C1', + TRUE, + '2026-03-10 09:40:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + 0, + NULL, + NULL + UNION ALL + SELECT + 'c6', + 'T1', + 'agent', + 'A1', + TRUE, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + 0, + NULL, + NULL + UNION ALL + SELECT + 'c7', + 'T3', + 'customer', + 'C2', + TRUE, + '2026-03-11 08:05:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + 0, + NULL, + NULL + UNION ALL + SELECT + 'c8', + 'T3', + 'agent', + 'A3', + TRUE, + '2026-03-11 08:30:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + 0, + NULL, + NULL +), + +__expected__int_ticket_first_response AS ( + SELECT + 'T1' AS ticket_id, + '2026-03-10 09:00:00'::TIMESTAMPNTZ AS created_at_utc, + '2026-03-10 09:20:00'::TIMESTAMPNTZ AS first_response_at_utc, + 'A1' AS first_responder_id, + 20 AS minutes_to_first_response, + '2026-03-10 09:40:00'::TIMESTAMPNTZ AS first_customer_reply_at_utc, + 3 AS agent_responses_before_customer_reply, + TRUE AS has_auto_reply + UNION ALL + SELECT + 'T2' AS ticket_id, + '2026-03-10 10:00:00'::TIMESTAMPNTZ AS created_at_utc, + CAST(NULL AS TIMESTAMPNTZ) AS first_response_at_utc, + CAST(NULL AS VARCHAR) AS first_responder_id, + CAST(NULL AS INT) AS minutes_to_first_response, + CAST(NULL AS TIMESTAMPNTZ) AS first_customer_reply_at_utc, + 0 AS agent_responses_before_customer_reply, + FALSE AS has_auto_reply + UNION ALL + SELECT + 'T3' AS ticket_id, + '2026-03-11 08:00:00'::TIMESTAMPNTZ AS created_at_utc, + '2026-03-11 08:30:00'::TIMESTAMPNTZ AS first_response_at_utc, + 'A3' AS first_responder_id, + 30 AS minutes_to_first_response, + CAST(NULL AS TIMESTAMPNTZ) AS first_customer_reply_at_utc, + 1 AS agent_responses_before_customer_reply, + FALSE AS has_auto_reply +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_sla_breaches__uses_business_minutes_only_when_policy_requires.sql b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_sla_breaches__uses_business_minutes_only_when_policy_requires.sql new file mode 100644 index 0000000000..c86315fc1a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_sla_breaches__uses_business_minutes_only_when_policy_requires.sql @@ -0,0 +1,185 @@ +TEST (name "int_ticket_sla_breaches__uses_business_minutes_only_when_policy_requires"); + +WITH __ref__stg_tickets AS ( + SELECT + 'T1' AS ticket_id, + NULL AS customer_email, + 'email' AS channel, + 1 AS priority, + NULL AS ticket_status, + NULL AS created_at, + '2026-03-02 10:00:00 +00:00'::TIMESTAMP_TZ AS solved_at, + NULL AS order_id, + NULL AS contact_reason, + NULL AS tag_list, + NULL AS is_vip, + NULL AS custom_fields, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + NULL, + 'chat', + 2, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T3', + NULL, + 'phone', + 3, + NULL, + NULL, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T4', + NULL, + 'email', + 1, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_sla_policies AS ( + SELECT + 'P1' AS policy_id, + 1 AS priority, + 'email' AS channel, + 60 AS first_response_minutes, + 1440 AS resolution_minutes, + TRUE AS business_hours_only, + NULL AS calendar_region_code, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'P2', + 2, + 'chat', + 10, + 600, + FALSE, + NULL, + NULL, + NULL +), + +__ref__int_ticket_first_response AS ( + SELECT + 'T1' AS ticket_id, + '2026-03-01 09:00:00'::TIMESTAMPNTZ AS created_at_utc, + NULL AS first_response_at_utc, + NULL AS first_responder_id, + 300 AS minutes_to_first_response, + NULL AS first_customer_reply_at_utc, + NULL AS agent_responses_before_customer_reply, + NULL AS has_auto_reply + UNION ALL + SELECT + 'T2', + '2026-03-01 09:00:00'::TIMESTAMPNTZ, + NULL, + NULL, + 15, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T3', + '2026-03-01 09:00:00'::TIMESTAMPNTZ, + NULL, + NULL, + 5, + NULL, + NULL, + NULL +), + +__ref__int_ticket_business_hours AS ( + SELECT + 'T1' AS ticket_id, + NULL AS created_at_support_local, + NULL AS first_response_at_support_local, + NULL AS calendar_minutes_to_response, + 45 AS business_minutes_to_response, + NULL AS created_outside_hours + UNION ALL + SELECT + 'T2', + NULL, + NULL, + NULL, + 5, + NULL +), + +__expected__int_ticket_sla_breaches AS ( + SELECT + 'T1' AS ticket_id, + 'P1' AS policy_id, + 60 AS first_response_target_minutes, + 45 AS first_response_minutes_used, + FALSE AS is_first_response_breached, + 1440 AS resolution_target_minutes, + 1500 AS resolution_minutes_used, + TRUE AS is_resolution_breached, + 1 AS breach_count + UNION ALL + SELECT + 'T2' AS ticket_id, + 'P2' AS policy_id, + 10 AS first_response_target_minutes, + 15 AS first_response_minutes_used, + TRUE AS is_first_response_breached, + 600 AS resolution_target_minutes, + CAST(NULL AS INT) AS resolution_minutes_used, + FALSE AS is_resolution_breached, + 1 AS breach_count + UNION ALL + SELECT + 'T3' AS ticket_id, + CAST(NULL AS VARCHAR) AS policy_id, + CAST(NULL AS INT) AS first_response_target_minutes, + 5 AS first_response_minutes_used, + FALSE AS is_first_response_breached, + CAST(NULL AS INT) AS resolution_target_minutes, + 60 AS resolution_minutes_used, + FALSE AS is_resolution_breached, + 0 AS breach_count +), + +__assert__only_open_tickets_have_an_age AS ( + SELECT b.ticket_id + FROM __ref("int_ticket_sla_breaches") AS b + WHERE + ( + b.ticket_id = 'T2' + ) <> COALESCE(b.open_minutes_now > 0, FALSE) +) diff --git a/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_threads__summarises_authors_and_handoffs.sql b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_threads__summarises_authors_and_handoffs.sql new file mode 100644 index 0000000000..8bff917257 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/intermediate/support/test_int_ticket_threads__summarises_authors_and_handoffs.sql @@ -0,0 +1,146 @@ +TEST (name "int_ticket_threads__summarises_authors_and_handoffs"); + +WITH __ref__stg_ticket_comments AS ( + SELECT + 'c1' AS comment_id, + 'T1' AS ticket_id, + 'customer' AS author_type, + 'C1' AS author_id, + TRUE AS is_public, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS created_at, + NULL AS body_word_count, + TRUE AS contains_order_reference, + 1 AS attachment_count, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'c2', + 'T1', + 'agent', + 'A2', + TRUE, + '2026-03-10 10:05:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + 0, + NULL, + NULL + UNION ALL + SELECT + 'c3', + 'T1', + 'agent', + 'A1', + FALSE, + '2026-03-10 10:10:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + 0, + NULL, + NULL + UNION ALL + SELECT + 'c4', + 'T1', + 'customer', + 'C1', + TRUE, + '2026-03-10 10:20:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + 2, + NULL, + NULL + UNION ALL + SELECT + 'c5', + 'T1', + 'system', + NULL, + TRUE, + '2026-03-10 10:21:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + 0, + NULL, + NULL +), + +__ref__stg_tickets AS ( + SELECT + 'T1' AS ticket_id, + NULL AS customer_email, + NULL AS channel, + NULL AS priority, + NULL AS ticket_status, + NULL AS created_at, + NULL AS solved_at, + NULL AS order_id, + NULL AS contact_reason, + NULL AS tag_list, + NULL AS is_vip, + NULL AS custom_fields, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__int_ticket_threads AS ( + SELECT + 'T1' AS ticket_id, + 5 AS comment_count, + 4 AS public_comment_count, + 3 AS handoff_count, + 3 AS total_attachments, + '2026-03-10 10:21:00 +00:00'::TIMESTAMP_TZ AS last_comment_at, + 'system' AS last_author_type, + TRUE AS mentions_order + UNION ALL + SELECT + 'T2' AS ticket_id, + 0 AS comment_count, + 0 AS public_comment_count, + 0 AS handoff_count, + 0 AS total_attachments, + CAST(NULL AS TIMESTAMP_TZ) AS last_comment_at, + CAST(NULL AS VARCHAR) AS last_author_type, + FALSE AS mentions_order +), + +__assert__author_sequence_is_chronological AS ( + SELECT t.ticket_id + FROM __ref("int_ticket_threads") AS t + WHERE + t.ticket_id = 'T1' + AND ARRAY_TO_STRING(t.author_sequence, '>') <> 'customer>agent>agent>customer>system' +), + +__assert__agent_participants_skip_non_agents AS ( + SELECT t.ticket_id + FROM __ref("int_ticket_threads") AS t + WHERE ARRAY_TO_STRING(t.agent_participants, ',') <> IFF(t.ticket_id = 'T1', 'A1,A2', '') +), + +__assert__empty_thread_breakdown_holds_json_nulls AS ( + SELECT t.ticket_id + FROM __ref("int_ticket_threads") AS t + WHERE + t.ticket_id = 'T2' + AND NOT IS_NULL_VALUE(t.author_breakdown:agent) +) diff --git a/benchmarks/duckshop/project/tests/unit/macros/python/test_round_half_even__rounds_ties_to_even.sql b/benchmarks/duckshop/project/tests/unit/macros/python/test_round_half_even__rounds_ties_to_even.sql new file mode 100644 index 0000000000..252e73154e --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/macros/python/test_round_half_even__rounds_ties_to_even.sql @@ -0,0 +1,25 @@ +TEST (mode macro, name "round_half_even__rounds_ties_to_even"); + +WITH cases AS ( + SELECT + v.case_name, + v.amount::NUMBER(10, 3) AS amount + FROM VALUES + ('tie_down', 2.345), + ('tie_up', 2.355), + ('not_a_tie', 2.346) AS v(case_name, amount) +), + +__macro_actual__ AS ( + SELECT + c.case_name, + @py.round_half_even("c.amount", 2) AS rounded + FROM cases AS c +), + +__macro_expected__ AS ( + SELECT + v.case_name, + v.rounded::NUMBER(10, 2) AS rounded + FROM VALUES ('tie_down', 2.34), ('tie_up', 2.36), ('not_a_tie', 2.35) AS v(case_name, rounded) +) diff --git a/benchmarks/duckshop/project/tests/unit/macros/sql/test_money_from_minor__converts_by_currency_exponent.sql b/benchmarks/duckshop/project/tests/unit/macros/sql/test_money_from_minor__converts_by_currency_exponent.sql new file mode 100644 index 0000000000..c58c75dc40 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/macros/sql/test_money_from_minor__converts_by_currency_exponent.sql @@ -0,0 +1,31 @@ +TEST (mode macro, name "money_from_minor__converts_by_currency_exponent"); + +WITH cases AS ( + SELECT + v.case_name, + v.amount_minor, + v.exponent + FROM VALUES + ('two_decimals', 1250, 2), + ('three_decimals_rounded', 12345, 3), + ('zero_decimals', 1000, 0), + ('null_amount', NULL, 2) AS v(case_name, amount_minor, exponent) +), + +__macro_actual__ AS ( + SELECT + c.case_name, + @sql.money_from_minor(c.amount_minor, c.exponent) AS amount + FROM cases AS c +), + +__macro_expected__ AS ( + SELECT + v.case_name, + v.amount::NUMBER(38, 6) AS amount + FROM VALUES + ('two_decimals', 12.5), + ('three_decimals_rounded', 12.35), + ('zero_decimals', 1000), + ('null_amount', NULL) AS v(case_name, amount) +) diff --git a/benchmarks/duckshop/project/tests/unit/macros/sql/test_safe_divide__returns_null_for_zero_denominators.sql b/benchmarks/duckshop/project/tests/unit/macros/sql/test_safe_divide__returns_null_for_zero_denominators.sql new file mode 100644 index 0000000000..9bc6ff5a92 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/macros/sql/test_safe_divide__returns_null_for_zero_denominators.sql @@ -0,0 +1,29 @@ +TEST (mode macro, name "safe_divide__returns_null_for_zero_denominators"); + +WITH cases AS ( + SELECT + v.case_name, + v.numerator, + v.denominator + FROM VALUES + ('exact', 10, 4), + ('zero_denominator', 1, 0), + ('null_denominator', 1, NULL) AS v(case_name, numerator, denominator) +), + +__macro_actual__ AS ( + SELECT + c.case_name, + @sql.safe_divide(c.numerator, c.denominator) AS ratio + FROM cases AS c +), + +__macro_expected__ AS ( + SELECT + v.case_name, + v.ratio::NUMBER(38, 6) AS ratio + FROM VALUES + ('exact', 2.5), + ('zero_denominator', NULL), + ('null_denominator', NULL) AS v(case_name, ratio) +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_category_tree_sales__rolls_sales_up_the_category_tree.sql b/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_category_tree_sales__rolls_sales_up_the_category_tree.sql new file mode 100644 index 0000000000..394b2bcfbd --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_category_tree_sales__rolls_sales_up_the_category_tree.sql @@ -0,0 +1,306 @@ +TEST (name "rpt_category_tree_sales__rolls_sales_up_the_category_tree"); + +WITH __ref__int_category_paths AS ( + SELECT + 'C0' AS category_id, + NULL AS parent_category_id, + 'ducks' AS slug, + NULL AS display_name, + 'C0' AS root_category_id, + 1 AS depth, + 'ducks' AS slug_path, + FALSE AS is_leaf, + NULL AS sibling_rank + UNION ALL + SELECT + 'C1', + 'C0', + 'rubber-ducks', + NULL, + 'C0', + 2, + 'ducks/rubber-ducks', + FALSE, + NULL + UNION ALL + SELECT + 'C2', + 'C1', + 'collectible', + NULL, + 'C0', + 3, + 'ducks/rubber-ducks/collectible', + TRUE, + NULL + UNION ALL + SELECT + 'C3', + 'C0', + 'decoys', + NULL, + 'C0', + 2, + 'ducks/decoys', + TRUE, + NULL + UNION ALL + SELECT + 'C4', + NULL, + 'textiles', + NULL, + 'C0', + 1, + 'textiles', + TRUE, + NULL +), + +__ref__dim_products AS ( + SELECT + 'DUCK-1' AS sku, + NULL AS product_id, + NULL AS sku_family, + NULL AS title, + NULL AS variant_label, + NULL AS brand_name, + NULL AS is_private_label, + NULL AS category_level_1, + NULL AS category_slug_path, + NULL AS is_active, + NULL AS launched_on, + NULL AS list_price_usd, + NULL AS margin_pct, + NULL AS primary_supplier_code, + NULL AS supplier_count, + NULL AS review_count, + NULL AS average_rating, + NULL AS wishlisting_customers, + NULL AS source_loaded_at + UNION ALL + SELECT + 'DUCK-2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'DECOY-1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_product_catalogue AS ( + SELECT + 'DUCK-1' AS sku, + NULL AS product_id, + NULL AS sku_family, + NULL AS title, + NULL AS variant_label, + NULL AS brand_code, + NULL AS brand_name, + NULL AS is_private_label, + 'C1' AS category_id, + NULL AS category_slug_path, + NULL AS category_level_1, + NULL AS material, + NULL AS weight_kg, + NULL AS primary_image_url, + NULL AS image_count, + NULL AS is_active, + NULL AS launched_on, + NULL AS is_bundle_component + UNION ALL + SELECT + 'DUCK-2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'C2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'DECOY-1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'C3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__fct_order_lines AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + 'DUCK-1' AS sku, + NULL AS sku_family, + NULL AS category_level_1, + 2 AS quantity, + NULL AS unit_amount_minor, + NULL AS list_amount_minor, + NULL AS markdown_pct, + NULL AS gross_amount_minor, + NULL AS discount_amount_minor, + 2000 AS net_amount_minor, + NULL AS tax_amount_minor, + NULL AS promo_codes, + NULL AS has_stacked_promotions, + NULL AS line_weight_kg + UNION ALL + SELECT + 'L2', + 'O1', + 'DUCK-2', + NULL, + NULL, + 3, + NULL, + NULL, + NULL, + NULL, + NULL, + 4500, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'L3', + 'O2', + 'DECOY-1', + NULL, + NULL, + 5, + NULL, + NULL, + NULL, + NULL, + NULL, + 10000, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'L4', + 'O3', + 'DUCK-1', + NULL, + NULL, + 1, + NULL, + NULL, + NULL, + NULL, + NULL, + 1000, + NULL, + NULL, + NULL, + NULL +), + +__expected__rpt_category_tree_sales AS ( + SELECT + 'C0' AS category_id, + 0 AS own_units, + 11 AS subtree_units, + 17500 AS subtree_net_amount_minor, + CAST(NULL AS DECIMAL(9, 4)) AS share_of_parent_units + UNION ALL + SELECT + 'C1' AS category_id, + 3 AS own_units, + 6 AS subtree_units, + 7500 AS subtree_net_amount_minor, + 0.5455::DECIMAL(9, 4) AS share_of_parent_units + UNION ALL + SELECT + 'C2' AS category_id, + 3 AS own_units, + 3 AS subtree_units, + 4500 AS subtree_net_amount_minor, + 0.5::DECIMAL(9, 4) AS share_of_parent_units + UNION ALL + SELECT + 'C3' AS category_id, + 5 AS own_units, + 5 AS subtree_units, + 10000 AS subtree_net_amount_minor, + 0.4545::DECIMAL(9, 4) AS share_of_parent_units + UNION ALL + SELECT + 'C4' AS category_id, + 0 AS own_units, + 0 AS subtree_units, + 0 AS subtree_net_amount_minor, + CAST(NULL AS DECIMAL(9, 4)) AS share_of_parent_units +), + +__assert__category_id_is_unique_and_present AS ( + SELECT t.category_id + FROM __ref("rpt_category_tree_sales") AS t + GROUP BY + t.category_id + HAVING + COUNT(*) > 1 + OR t.category_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_margin_by_category__reports_margin_quartiles_per_category_and_currency.sql b/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_margin_by_category__reports_margin_quartiles_per_category_and_currency.sql new file mode 100644 index 0000000000..fa2ee24e85 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_margin_by_category__reports_margin_quartiles_per_category_and_currency.sql @@ -0,0 +1,86 @@ +TEST (name "rpt_margin_by_category__reports_margin_quartiles_per_category_and_currency"); + +WITH __ref__int_product_margin AS ( + SELECT + 'DUCK-1:EUR' AS sku_price_key, + 'DUCK-1' AS sku, + 'EUR' AS currency_code, + NULL AS list_price, + NULL AS unit_cost, + NULL AS unit_margin, + 50.0::DECIMAL(9, 2) AS margin_pct, + FALSE AS is_below_cost, + 'ducks' AS category_level_1 + UNION ALL + SELECT + 'DUCK-2:EUR', + 'DUCK-2', + 'EUR', + NULL, + NULL, + NULL, + 40.0::DECIMAL(9, 2), + FALSE, + 'ducks' + UNION ALL + SELECT + 'DUCK-3:EUR', + 'DUCK-3', + 'EUR', + NULL, + NULL, + NULL, + 30.0::DECIMAL(9, 2), + FALSE, + 'ducks' + UNION ALL + SELECT + 'DUCK-4:EUR', + 'DUCK-4', + 'EUR', + NULL, + NULL, + NULL, + -10.0::DECIMAL(9, 2), + TRUE, + 'ducks' + UNION ALL + SELECT + 'BATH-1:USD', + 'BATH-1', + 'USD', + NULL, + NULL, + NULL, + 10.0::DECIMAL(9, 2), + FALSE, + NULL +), + +__expected__rpt_margin_by_category AS ( + SELECT + 'ducks:EUR' AS margin_key, + 4 AS sku_count, + 27.5::DECIMAL(9, 2) AS average_margin_pct, + 20.0::DECIMAL(9, 2) AS p25_margin_pct, + 42.5::DECIMAL(9, 2) AS p75_margin_pct, + 1 AS below_cost_skus + UNION ALL + SELECT + 'uncategorised:USD' AS margin_key, + 1 AS sku_count, + 10.0::DECIMAL(9, 2) AS average_margin_pct, + 10.0::DECIMAL(9, 2) AS p25_margin_pct, + 10.0::DECIMAL(9, 2) AS p75_margin_pct, + 0 AS below_cost_skus +), + +__assert__margin_key_is_unique_and_present AS ( + SELECT t.margin_key + FROM __ref("rpt_margin_by_category") AS t + GROUP BY + t.margin_key + HAVING + COUNT(*) > 1 + OR t.margin_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_price_changes_weekly__counts_increases_markdowns_and_new_lows.sql b/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_price_changes_weekly__counts_increases_markdowns_and_new_lows.sql new file mode 100644 index 0000000000..8df125513d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_price_changes_weekly__counts_increases_markdowns_and_new_lows.sql @@ -0,0 +1,103 @@ +TEST (name "rpt_price_changes_weekly__counts_increases_markdowns_and_new_lows"); + +WITH __ref__int_price_history AS ( + SELECT + 'PL1' AS price_list_id, + 'DUCK-1' AS sku, + 'EUR' AS currency_code, + 1000 AS list_amount_minor, + '2026-03-01 00:00:00 +00:00'::TIMESTAMPTZ AS valid_from, + NULL AS effective_valid_to, + NULL AS previous_amount_minor, + NULL AS change_pct, + FALSE AS is_markdown, + NULL AS price_version, + NULL AS days_at_price, + NULL AS lowest_price_30d_minor + UNION ALL + SELECT + 'PL2', + 'DUCK-1', + 'EUR', + 1100, + '2026-03-10 10:00:00 +00:00'::TIMESTAMPTZ, + NULL, + 1000, + 10.0::DECIMAL(9, 4), + FALSE, + NULL, + NULL, + 1000 + UNION ALL + SELECT + 'PL3', + 'DUCK-1', + 'EUR', + 880, + '2026-03-12 10:00:00 +00:00'::TIMESTAMPTZ, + NULL, + 1100, + -20.0::DECIMAL(9, 4), + TRUE, + NULL, + NULL, + 1000 + UNION ALL + SELECT + 'PL4', + 'DUCK-2', + 'EUR', + 900, + '2026-03-11 10:00:00 +00:00'::TIMESTAMPTZ, + NULL, + 1000, + -10.0::DECIMAL(9, 4), + TRUE, + NULL, + NULL, + 900 + UNION ALL + SELECT + 'PL5', + 'DUCK-1', + 'USD', + 1260, + '2026-03-15 23:30:00 -02:00'::TIMESTAMPTZ, + NULL, + 1200, + 5.0::DECIMAL(9, 4), + FALSE, + NULL, + NULL, + NULL +), + +__expected__rpt_price_changes_weekly AS ( + SELECT + '2026-03-09:EUR' AS price_week_key, + 3 AS price_changes, + 1 AS increases, + 2 AS markdowns, + 10.0::DECIMAL(9, 4) AS average_increase_pct, + -15.0::DECIMAL(9, 4) AS average_markdown_pct, + 1 AS skus_below_30d_low + UNION ALL + SELECT + '2026-03-16:USD' AS price_week_key, + 1 AS price_changes, + 1 AS increases, + 0 AS markdowns, + 5.0::DECIMAL(9, 4) AS average_increase_pct, + CAST(NULL AS DECIMAL(9, 4)) AS average_markdown_pct, + 0 AS skus_below_30d_low +), + +__assert__price_week_key_is_unique_and_present AS ( + SELECT t.price_week_key + FROM __ref("rpt_price_changes_weekly") AS t + GROUP BY + t.price_week_key + HAVING + COUNT(*) > 1 + OR t.price_week_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_wishlist_conversion__converts_wishlists_by_brand.sql b/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_wishlist_conversion__converts_wishlists_by_brand.sql new file mode 100644 index 0000000000..0f4b20efd3 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_wishlist_conversion__converts_wishlists_by_brand.sql @@ -0,0 +1,102 @@ +TEST (name "rpt_wishlist_conversion__converts_wishlists_by_brand"); + +WITH __ref__int_wishlist_demand AS ( + SELECT + 'DUCK-1' AS sku, + 10 AS wishlisting_customers, + NULL AS public_wishlist_count, + 2 AS purchased_after_wishlist, + NULL AS wishlist_conversion_rate, + 3.0::DECIMAL(10, 2) AS median_days_to_purchase, + NULL AS first_added_at + UNION ALL + SELECT + 'DUCK-2', + 5, + NULL, + 1, + NULL, + 7.0::DECIMAL(10, 2), + NULL + UNION ALL + SELECT + 'BATH-1', + 4, + NULL, + 0, + NULL, + NULL, + NULL +), + +__ref__dim_products AS ( + SELECT + 'DUCK-1' AS sku, + NULL AS product_id, + NULL AS sku_family, + NULL AS title, + NULL AS variant_label, + 'Mallard & Co.' AS brand_name, + NULL AS is_private_label, + NULL AS category_level_1, + NULL AS category_slug_path, + NULL AS is_active, + NULL AS launched_on, + NULL AS list_price_usd, + NULL AS margin_pct, + NULL AS primary_supplier_code, + NULL AS supplier_count, + NULL AS review_count, + NULL AS average_rating, + NULL AS wishlisting_customers, + NULL AS source_loaded_at + UNION ALL + SELECT + 'DUCK-2', + NULL, + NULL, + NULL, + NULL, + 'Mallard & Co.', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__rpt_wishlist_conversion AS ( + SELECT + 'Mallard & Co.' AS brand_name, + 2 AS wishlisted_skus, + 15 AS wishlisting_customers, + 3 AS purchased_after_wishlist, + 0.2::DECIMAL(9, 6) AS conversion_rate, + 5.0::DECIMAL(10, 2) AS median_days_to_purchase + UNION ALL + SELECT + 'Unbranded' AS brand_name, + 1 AS wishlisted_skus, + 4 AS wishlisting_customers, + 0 AS purchased_after_wishlist, + 0.0::DECIMAL(9, 6) AS conversion_rate, + CAST(NULL AS DECIMAL(10, 2)) AS median_days_to_purchase +), + +__assert__brand_name_is_unique_and_present AS ( + SELECT t.brand_name + FROM __ref("rpt_wishlist_conversion") AS t + GROUP BY + t.brand_name + HAVING + COUNT(*) > 1 + OR t.brand_name IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_wishlist_conversion__edge_cases.sql b/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_wishlist_conversion__edge_cases.sql new file mode 100644 index 0000000000..74ea658c45 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/catalogue/test_rpt_wishlist_conversion__edge_cases.sql @@ -0,0 +1,105 @@ +TEST (name "rpt_wishlist_conversion__groups_unknown_products_as_unbranded"); + +WITH __ref__int_wishlist_demand AS ( + SELECT + 'DUCK-EID-01' AS sku, + 6 AS wishlisting_customers, + NULL AS public_wishlist_count, + 3 AS purchased_after_wishlist, + NULL AS wishlist_conversion_rate, + 4.0::DECIMAL(10, 2) AS median_days_to_purchase, + NULL AS first_added_at + UNION ALL + SELECT + 'DECOY-PIN-02', + 2, + NULL, + 0, + NULL, + NULL, + NULL +), + +__ref__dim_products AS ( + SELECT + 'DUCK-OTHER' AS sku, + NULL AS product_id, + NULL AS sku_family, + 'Teal Twin Pack' AS title, + NULL AS variant_label, + 'Mallard & Co.' AS brand_name, + NULL AS is_private_label, + NULL AS category_level_1, + NULL AS category_slug_path, + NULL AS is_active, + NULL AS launched_on, + NULL AS list_price_usd, + NULL AS margin_pct, + NULL AS primary_supplier_code, + NULL AS supplier_count, + NULL AS review_count, + NULL AS average_rating, + NULL AS wishlisting_customers, + NULL AS source_loaded_at +), + +__expected__rpt_wishlist_conversion AS ( + SELECT + 'Unbranded' AS brand_name, + 2 AS wishlisted_skus, + 8 AS wishlisting_customers, + 3 AS purchased_after_wishlist, + 0.375::DECIMAL(9, 6) AS conversion_rate, + 4.0::DECIMAL(10, 2) AS median_days_to_purchase +) + +TEST (name "rpt_wishlist_conversion__leaves_rate_empty_without_wishlisters"); + +WITH __ref__int_wishlist_demand AS ( + SELECT + 'DUCK-MER-03' AS sku, + 0 AS wishlisting_customers, + NULL AS public_wishlist_count, + 0 AS purchased_after_wishlist, + NULL AS wishlist_conversion_rate, + NULL AS median_days_to_purchase, + NULL AS first_added_at +), + +__ref__dim_products AS ( + SELECT + 'DUCK-MER-03' AS sku, + NULL AS product_id, + NULL AS sku_family, + 'Merganser Mini Duck' AS title, + NULL AS variant_label, + 'Mallard & Co.' AS brand_name, + NULL AS is_private_label, + NULL AS category_level_1, + NULL AS category_slug_path, + NULL AS is_active, + NULL AS launched_on, + NULL AS list_price_usd, + NULL AS margin_pct, + NULL AS primary_supplier_code, + NULL AS supplier_count, + NULL AS review_count, + NULL AS average_rating, + NULL AS wishlisting_customers, + NULL AS source_loaded_at +), + +__expected__rpt_wishlist_conversion AS ( + SELECT + 'Mallard & Co.' AS brand_name, + 1 AS wishlisted_skus, + CAST(NULL AS DECIMAL(9, 6)) AS conversion_rate +), + +__assert__rates_stay_within_bounds AS ( + SELECT r.brand_name + FROM __ref("rpt_wishlist_conversion") AS r + WHERE + r.conversion_rate < 0 + OR r.conversion_rate > 1 +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_fct_web_sessions__builds_only_sessions_inside_the_cursor_window.sql b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_fct_web_sessions__builds_only_sessions_inside_the_cursor_window.sql new file mode 100644 index 0000000000..2f6d8ee595 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_fct_web_sessions__builds_only_sessions_inside_the_cursor_window.sql @@ -0,0 +1,107 @@ +TEST (name "fct_web_sessions__builds_only_sessions_inside_the_cursor_window", cursor_start "2026-03-10 10:00:00", cursor_end "2026-03-10 11:00:00"); + +WITH __ref__int_web_sessions AS ( + SELECT + 'S1' AS session_id, + 'A1' AS anonymous_id, + NULL AS customer_id, + NULL AS session_day, + '2026-03-10 09:30:00'::TIMESTAMP_NTZ AS session_start_utc, + NULL AS session_end_utc, + NULL AS session_start_visitor_local, + NULL AS duration_seconds, + NULL AS event_count, + NULL AS page_view_count, + NULL AS landing_page_path, + NULL AS exit_page_path, + NULL AS referrer_host, + NULL AS landing_page_url, + TRUE AS is_bounce, + NULL AS has_add_to_cart, + FALSE AS has_order + UNION ALL + SELECT + 'S2', + 'A2', + NULL, + NULL, + '2026-03-10 10:15:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + TRUE + UNION ALL + SELECT + 'S3', + 'A3', + NULL, + NULL, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + TRUE +), + +__ref__int_session_attribution AS ( + SELECT + 'S1' AS session_id, + NULL AS anonymous_id, + NULL AS customer_id, + NULL AS session_start_utc, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + 'ORGANIC' AS channel_code, + FALSE AS is_paid, + 'PAID_SOCIAL' AS first_touch_channel, + NULL AS last_touch_channel, + 2 AS touch_number +), + +__ref__int_cart_funnel AS ( + SELECT + 'S2' AS session_id, + NULL AS anonymous_id, + NULL AS session_start_utc, + NULL AS added_skus, + NULL AS removed_sku_count, + NULL AS net_units, + NULL AS max_added_unit_minor, + NULL AS reached_checkout, + NULL AS reached_payment, + TRUE AS converted, + 'purchase' AS funnel_stage +), + +__expected__fct_web_sessions AS ( + SELECT + 'S2' AS session_id, + 'purchase' AS funnel_stage +), + +__assert__session_id_is_unique_and_present AS ( + SELECT t.session_id + FROM __ref("fct_web_sessions") AS t + GROUP BY + t.session_id + HAVING + COUNT(*) > 1 + OR t.session_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_fct_web_sessions__falls_back_to_bounce_and_order_flags_without_a_funnel.sql b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_fct_web_sessions__falls_back_to_bounce_and_order_flags_without_a_funnel.sql new file mode 100644 index 0000000000..a290fa4023 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_fct_web_sessions__falls_back_to_bounce_and_order_flags_without_a_funnel.sql @@ -0,0 +1,127 @@ +TEST (name "fct_web_sessions__falls_back_to_bounce_and_order_flags_without_a_funnel"); + +WITH __ref__int_web_sessions AS ( + SELECT + 'S1' AS session_id, + 'A1' AS anonymous_id, + NULL AS customer_id, + NULL AS session_day, + '2026-03-10 09:30:00'::TIMESTAMP_NTZ AS session_start_utc, + NULL AS session_end_utc, + NULL AS session_start_visitor_local, + NULL AS duration_seconds, + NULL AS event_count, + NULL AS page_view_count, + NULL AS landing_page_path, + NULL AS exit_page_path, + NULL AS referrer_host, + NULL AS landing_page_url, + TRUE AS is_bounce, + NULL AS has_add_to_cart, + FALSE AS has_order + UNION ALL + SELECT + 'S2', + 'A2', + NULL, + NULL, + '2026-03-10 10:15:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + TRUE + UNION ALL + SELECT + 'S3', + 'A3', + NULL, + NULL, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + TRUE +), + +__ref__int_session_attribution AS ( + SELECT + 'S1' AS session_id, + NULL AS anonymous_id, + NULL AS customer_id, + NULL AS session_start_utc, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + 'ORGANIC' AS channel_code, + FALSE AS is_paid, + 'PAID_SOCIAL' AS first_touch_channel, + NULL AS last_touch_channel, + 2 AS touch_number +), + +__ref__int_cart_funnel AS ( + SELECT + 'S2' AS session_id, + NULL AS anonymous_id, + NULL AS session_start_utc, + NULL AS added_skus, + NULL AS removed_sku_count, + NULL AS net_units, + NULL AS max_added_unit_minor, + NULL AS reached_checkout, + NULL AS reached_payment, + TRUE AS converted, + 'purchase' AS funnel_stage +), + +__expected__fct_web_sessions AS ( + SELECT + 'S1' AS session_id, + 'ORGANIC' AS channel_code, + 'PAID_SOCIAL' AS first_touch_channel, + 2 AS touch_number, + 'bounce' AS funnel_stage, + FALSE AS converted + UNION ALL + SELECT + 'S2' AS session_id, + CAST(NULL AS VARCHAR) AS channel_code, + CAST(NULL AS VARCHAR) AS first_touch_channel, + CAST(NULL AS INT) AS touch_number, + 'purchase' AS funnel_stage, + TRUE AS converted + UNION ALL + SELECT + 'S3' AS session_id, + CAST(NULL AS VARCHAR) AS channel_code, + CAST(NULL AS VARCHAR) AS first_touch_channel, + CAST(NULL AS INT) AS touch_number, + 'browse' AS funnel_stage, + TRUE AS converted +), + +__assert__session_id_is_unique_and_present AS ( + SELECT t.session_id + FROM __ref("fct_web_sessions") AS t + GROUP BY + t.session_id + HAVING + COUNT(*) > 1 + OR t.session_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_app_engagement_weekly__summarises_devices_and_push_starts_per_platform_week.sql b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_app_engagement_weekly__summarises_devices_and_push_starts_per_platform_week.sql new file mode 100644 index 0000000000..9837c8ae69 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_app_engagement_weekly__summarises_devices_and_push_starts_per_platform_week.sql @@ -0,0 +1,85 @@ +TEST (name "rpt_app_engagement_weekly__summarises_devices_and_push_starts_per_platform_week"); + +WITH __ref__int_app_sessions AS ( + SELECT + 'AS1' AS app_session_id, + 'D1' AS device_id, + 'C1' AS customer_id, + 'phone' AS platform, + 5 AS app_major_version, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS session_start_utc, + NULL AS session_end_utc, + 100 AS duration_seconds, + NULL AS event_count, + NULL AS screen_count, + TRUE AS started_from_push + UNION ALL + SELECT + 'AS2', + 'D1', + NULL, + 'phone', + 6, + '2026-03-11 10:00:00'::TIMESTAMP_NTZ, + NULL, + 300, + NULL, + NULL, + FALSE + UNION ALL + SELECT + 'AS3', + 'D2', + 'C2', + 'phone', + 5, + '2026-03-12 10:00:00'::TIMESTAMP_NTZ, + NULL, + 200, + NULL, + NULL, + FALSE + UNION ALL + SELECT + 'AS4', + 'D3', + NULL, + 'tablet', + 2, + '2026-03-16 10:00:00'::TIMESTAMP_NTZ, + NULL, + 50, + NULL, + NULL, + TRUE +), + +__expected__rpt_app_engagement_weekly AS ( + SELECT + '2026-03-09:phone' AS app_week_key, + 3 AS sessions, + 2 AS active_devices, + 0.6667::DECIMAL(9, 4) AS signed_in_share, + 0.3333::DECIMAL(9, 4) AS push_started_share, + 200.0::DECIMAL(10, 2) AS median_session_seconds, + 6 AS latest_major_version + UNION ALL + SELECT + '2026-03-16:tablet' AS app_week_key, + 1 AS sessions, + 1 AS active_devices, + 0.0::DECIMAL(9, 4) AS signed_in_share, + 1.0::DECIMAL(9, 4) AS push_started_share, + 50.0::DECIMAL(10, 2) AS median_session_seconds, + 2 AS latest_major_version +), + +__assert__app_week_key_is_unique_and_present AS ( + SELECT t.app_week_key + FROM __ref("rpt_app_engagement_weekly") AS t + GROUP BY + t.app_week_key + HAVING + COUNT(*) > 1 + OR t.app_week_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_channel_attribution_daily__credits_last_touch_sessions_and_first_touch_conversions.sql b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_channel_attribution_daily__credits_last_touch_sessions_and_first_touch_conversions.sql new file mode 100644 index 0000000000..d7aacb3204 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_channel_attribution_daily__credits_last_touch_sessions_and_first_touch_conversions.sql @@ -0,0 +1,118 @@ +TEST (name "rpt_channel_attribution_daily__credits_last_touch_sessions_and_first_touch_conversions"); + +WITH __ref__fct_web_sessions AS ( + SELECT + 'S1' AS session_id, + 'A1' AS anonymous_id, + NULL AS customer_id, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS session_start_utc, + NULL AS session_start_visitor_local, + NULL AS duration_seconds, + NULL AS page_view_count, + NULL AS landing_page_path, + 'ORGANIC' AS channel_code, + FALSE AS is_paid, + NULL AS utm_campaign, + 'PAID_SOCIAL' AS first_touch_channel, + NULL AS touch_number, + NULL AS is_bounce, + NULL AS funnel_stage, + TRUE AS converted, + NULL AS visitor_bucket + UNION ALL + SELECT + 'S2', + 'A2', + NULL, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + 'ORGANIC', + FALSE, + NULL, + 'ORGANIC', + NULL, + NULL, + NULL, + FALSE, + NULL + UNION ALL + SELECT + 'S3', + 'A1', + NULL, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + 'PAID_SOCIAL', + TRUE, + NULL, + 'PAID_SOCIAL', + NULL, + NULL, + NULL, + FALSE, + NULL + UNION ALL + SELECT + 'S4', + 'A3', + NULL, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL +), + +__expected__rpt_channel_attribution_daily AS ( + SELECT + '2026-03-10:ORGANIC' AS channel_day_key, + 2 AS sessions, + 2 AS visitors, + 1 AS conversions, + 0.5::DECIMAL(9, 4) AS conversion_rate, + 0 AS first_touch_credit, + 0.0::DECIMAL(9, 4) AS paid_share + UNION ALL + SELECT + '2026-03-10:PAID_SOCIAL' AS channel_day_key, + 1 AS sessions, + 1 AS visitors, + 0 AS conversions, + 0.0::DECIMAL(9, 4) AS conversion_rate, + 1 AS first_touch_credit, + 1.0::DECIMAL(9, 4) AS paid_share + UNION ALL + SELECT + '2026-03-10:UNKNOWN' AS channel_day_key, + 1 AS sessions, + 1 AS visitors, + 1 AS conversions, + 1.0::DECIMAL(9, 4) AS conversion_rate, + 1 AS first_touch_credit, + 0.0::DECIMAL(9, 4) AS paid_share +), + +__assert__channel_day_key_is_unique_and_present AS ( + SELECT t.channel_day_key + FROM __ref("rpt_channel_attribution_daily") AS t + GROUP BY + t.channel_day_key + HAVING + COUNT(*) > 1 + OR t.channel_day_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_experiment_results__computes_lift_and_z_score_against_control.sql b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_experiment_results__computes_lift_and_z_score_against_control.sql new file mode 100644 index 0000000000..4e00da78be --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_experiment_results__computes_lift_and_z_score_against_control.sql @@ -0,0 +1,162 @@ +TEST (name "rpt_experiment_results__computes_lift_and_z_score_against_control"); + +WITH __ref__int_experiment_exposures AS ( + SELECT + 'EA1' AS assignment_id, + 'golden-duck-banner' AS experiment_key, + 'control' AS variant_key, + TRUE AS is_control, + 'A1' AS anonymous_id, + NULL AS assigned_at_utc, + NULL AS sessions_after, + TRUE AS converted, + 1::DECIMAL(10, 2) AS days_to_convert, + NULL AS assignment_surface + UNION ALL + SELECT + 'EA2', + 'golden-duck-banner', + 'control', + TRUE, + 'A2', + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'EA3', + 'golden-duck-banner', + 'control', + TRUE, + 'A3', + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'EA4', + 'golden-duck-banner', + 'control', + TRUE, + 'A4', + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'EA5', + 'golden-duck-banner', + 'banner', + FALSE, + 'A5', + NULL, + NULL, + TRUE, + 2::DECIMAL(10, 2), + NULL + UNION ALL + SELECT + 'EA6', + 'golden-duck-banner', + 'banner', + FALSE, + 'A6', + NULL, + NULL, + TRUE, + 4::DECIMAL(10, 2), + NULL + UNION ALL + SELECT + 'EA7', + 'golden-duck-banner', + 'banner', + FALSE, + 'A7', + NULL, + NULL, + TRUE, + 6::DECIMAL(10, 2), + NULL + UNION ALL + SELECT + 'EA8', + 'golden-duck-banner', + 'banner', + FALSE, + 'A8', + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'EA9', + 'pond-week-copy', + 'short', + FALSE, + 'A9', + NULL, + NULL, + TRUE, + 3::DECIMAL(10, 2), + NULL + UNION ALL + SELECT + 'EA10', + 'pond-week-copy', + 'short', + FALSE, + 'A10', + NULL, + NULL, + FALSE, + NULL, + NULL +), + +__expected__rpt_experiment_results AS ( + SELECT + 'golden-duck-banner:control' AS experiment_variant_key, + 4 AS exposed, + 1 AS converted, + 0.25::DECIMAL(9, 6) AS conversion_rate, + 0.0::DECIMAL(9, 6) AS lift_vs_control, + 0.0::DECIMAL(9, 4) AS z_score, + 1.0::DECIMAL(10, 2) AS median_days_to_convert + UNION ALL + SELECT + 'golden-duck-banner:banner' AS experiment_variant_key, + 4 AS exposed, + 3 AS converted, + 0.75::DECIMAL(9, 6) AS conversion_rate, + 0.5::DECIMAL(9, 6) AS lift_vs_control, + 1.4142::DECIMAL(9, 4) AS z_score, + 4.0::DECIMAL(10, 2) AS median_days_to_convert + UNION ALL + SELECT + 'pond-week-copy:short' AS experiment_variant_key, + 2 AS exposed, + 1 AS converted, + 0.5::DECIMAL(9, 6) AS conversion_rate, + CAST(NULL AS DECIMAL(9, 6)) AS lift_vs_control, + CAST(NULL AS DECIMAL(9, 4)) AS z_score, + 3.0::DECIMAL(10, 2) AS median_days_to_convert +), + +__assert__experiment_variant_key_is_unique_and_present AS ( + SELECT t.experiment_variant_key + FROM __ref("rpt_experiment_results") AS t + GROUP BY + t.experiment_variant_key + HAVING + COUNT(*) > 1 + OR t.experiment_variant_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_product_affinity_top__keeps_the_five_best_co_viewed_products.sql b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_product_affinity_top__keeps_the_five_best_co_viewed_products.sql new file mode 100644 index 0000000000..f889ccb2f8 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_product_affinity_top__keeps_the_five_best_co_viewed_products.sql @@ -0,0 +1,131 @@ +TEST (name "rpt_product_affinity_top__keeps_the_five_best_co_viewed_products"); + +WITH __ref__int_product_view_affinity AS ( + SELECT + 'DUCK-1|DUCK-2' AS sku_pair_key, + 'DUCK-1' AS sku, + 'DUCK-2' AS co_viewed_sku, + NULL AS co_viewers, + 20 AS sku_viewers, + NULL AS co_view_rate, + 3.0::DECIMAL(12, 4) AS lift, + 1 AS pair_rank + UNION ALL + SELECT + 'DUCK-1|DUCK-3', + 'DUCK-1', + 'DUCK-3', + NULL, + 20, + NULL, + 2.5::DECIMAL(12, 4), + 2 + UNION ALL + SELECT + 'DUCK-1|DUCK-4', + 'DUCK-1', + 'DUCK-4', + NULL, + 20, + NULL, + 2.0::DECIMAL(12, 4), + 3 + UNION ALL + SELECT + 'DUCK-1|DUCK-5', + 'DUCK-1', + 'DUCK-5', + NULL, + 20, + NULL, + 1.5::DECIMAL(12, 4), + 4 + UNION ALL + SELECT + 'DUCK-1|DUCK-6', + 'DUCK-1', + 'DUCK-6', + NULL, + 20, + NULL, + 1.2::DECIMAL(12, 4), + 5 + UNION ALL + SELECT + 'DUCK-1|DUCK-7', + 'DUCK-1', + 'DUCK-7', + NULL, + 20, + NULL, + 9.0::DECIMAL(12, 4), + 6 + UNION ALL + SELECT + 'DUCK-2|DUCK-1', + 'DUCK-2', + 'DUCK-1', + NULL, + 8, + NULL, + 1.1::DECIMAL(12, 4), + 1 +), + +__ref__dim_products AS ( + SELECT + 'DUCK-1' AS sku, + NULL AS product_id, + NULL AS sku_family, + 'Mallard Classic Duck' AS title, + NULL AS variant_label, + NULL AS brand_name, + NULL AS is_private_label, + NULL AS category_level_1, + NULL AS category_slug_path, + NULL AS is_active, + NULL AS launched_on, + NULL AS list_price_usd, + NULL AS margin_pct, + NULL AS primary_supplier_code, + NULL AS supplier_count, + NULL AS review_count, + NULL AS average_rating, + NULL AS wishlisting_customers, + NULL AS source_loaded_at +), + +__expected__rpt_product_affinity_top AS ( + SELECT + 'DUCK-1' AS sku, + 'Mallard Classic Duck' AS title, + 20 AS sku_viewers, + 3.0::DECIMAL(12, 4) AS best_lift + UNION ALL + SELECT + 'DUCK-2' AS sku, + CAST(NULL AS VARCHAR) AS title, + 8 AS sku_viewers, + 1.1::DECIMAL(12, 4) AS best_lift +), + +__assert__top_list_is_capped_and_ordered_by_rank AS ( + SELECT a.sku + FROM __ref("rpt_product_affinity_top") AS a + WHERE + a.sku = 'DUCK-1' + AND ( + ARRAY_SIZE(a.top_co_viewed) <> 5 + OR GET(a.top_co_viewed, 0):sku::VARCHAR <> 'DUCK-2' + ) +), + +__assert__sku_is_unique_and_present AS ( + SELECT t.sku + FROM __ref("rpt_product_affinity_top") AS t + GROUP BY + t.sku + HAVING + COUNT(*) > 1 + OR t.sku IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_search_terms_top__ranks_terms_and_flags_merchandising_gaps.sql b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_search_terms_top__ranks_terms_and_flags_merchandising_gaps.sql new file mode 100644 index 0000000000..6b76e90d26 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/clickstream/test_rpt_search_terms_top__ranks_terms_and_flags_merchandising_gaps.sql @@ -0,0 +1,75 @@ +TEST (name "rpt_search_terms_top__ranks_terms_and_flags_merchandising_gaps"); + +WITH __ref__int_search_conversion AS ( + SELECT + 'rubber duck' AS search_term, + 50 AS search_count, + 25 AS searching_visitors, + 0.05::DECIMAL(9, 4) AS zero_result_rate, + 0.4::DECIMAL(9, 4) AS view_after_search_rate, + NULL AS avg_results, + NULL AS top_filter_category, + NULL AS first_searched_at_utc + UNION ALL + SELECT + 'golden duck', + 30, + 15, + 0.3::DECIMAL(9, 4), + 0.5::DECIMAL(9, 4), + NULL, + NULL, + NULL + UNION ALL + SELECT + 'decoy', + 30, + 15, + 0.0::DECIMAL(9, 4), + 0.05::DECIMAL(9, 4), + NULL, + NULL, + NULL + UNION ALL + SELECT + 'pond pump', + 10, + 5, + 0.2::DECIMAL(9, 4), + 0.1::DECIMAL(9, 4), + NULL, + NULL, + NULL +), + +__expected__rpt_search_terms_top AS ( + SELECT + 'rubber duck' AS search_term, + 1 AS search_rank, + FALSE AS needs_merchandising + UNION ALL + SELECT + 'decoy' AS search_term, + 2 AS search_rank, + TRUE AS needs_merchandising + UNION ALL + SELECT + 'golden duck' AS search_term, + 3 AS search_rank, + TRUE AS needs_merchandising + UNION ALL + SELECT + 'pond pump' AS search_term, + 4 AS search_rank, + FALSE AS needs_merchandising +), + +__assert__search_term_is_unique_and_present AS ( + SELECT t.search_term + FROM __ref("rpt_search_terms_top") AS t + GROUP BY + t.search_term + HAVING + COUNT(*) > 1 + OR t.search_term IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_campaigns__derives_flight_state_and_age_range.sql b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_campaigns__derives_flight_state_and_age_range.sql new file mode 100644 index 0000000000..a7ef61e676 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_campaigns__derives_flight_state_and_age_range.sql @@ -0,0 +1,109 @@ +TEST (name "dim_campaigns__derives_flight_state_and_age_range"); + +WITH __ref__stg_campaigns AS ( + SELECT + 'CMP1' AS campaign_id, + 'Spring Splash' AS campaign_name, + 'social' AS channel, + NULL AS objective, + '2026-03-01'::DATE AS starts_on, + NULL AS ends_on, + NULL AS budget_minor, + NULL AS currency_code, + NULL AS target_regions, + NULL AS target_age_min, + NULL AS target_age_max, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'CMP2', + 'Pond Week', + 'email', + NULL, + '2026-04-15'::DATE, + '2026-04-30'::DATE, + NULL, + NULL, + NULL, + 18, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'CMP3', + 'Golden Duck Winter', + 'search', + NULL, + '2026-01-01'::DATE, + '2026-02-28'::DATE, + NULL, + NULL, + NULL, + 25, + 54, + NULL, + NULL +), + +__ref__int_campaign_performance AS ( + SELECT + 'CMP1' AS campaign_id, + NULL AS campaign_name, + NULL AS channel, + NULL AS objective, + NULL AS starts_on, + NULL AS ends_on, + 1100.0::DECIMAL(18, 2) AS budget_usd, + 550.0::DECIMAL(18, 2) AS spend_usd, + NULL AS budget_used_pct, + NULL AS paid_clicks, + NULL AS email_clicks, + NULL AS attributed_sessions, + NULL AS attributed_orders, + NULL AS attributed_revenue_usd, + 0.1818::DECIMAL(12, 4) AS return_on_ad_spend, + NULL AS target_region_count, + NULL AS unsupported_target_regions, + NULL AS spend_rank_in_channel +), + +__expected__dim_campaigns AS ( + SELECT + 'CMP1' AS campaign_id, + 31 AS flight_days, + 'live' AS campaign_state, + 'all' AS target_age_range, + 1100.0::DECIMAL(18, 2) AS budget_usd, + 550.0::DECIMAL(18, 2) AS spend_usd, + 0.1818::DECIMAL(12, 4) AS return_on_ad_spend + UNION ALL + SELECT + 'CMP2' AS campaign_id, + 16 AS flight_days, + 'scheduled' AS campaign_state, + '18-' AS target_age_range, + CAST(NULL AS DECIMAL(18, 2)) AS budget_usd, + CAST(NULL AS DECIMAL(18, 2)) AS spend_usd, + CAST(NULL AS DECIMAL(12, 4)) AS return_on_ad_spend + UNION ALL + SELECT + 'CMP3' AS campaign_id, + 59 AS flight_days, + 'ended' AS campaign_state, + '25-54' AS target_age_range, + CAST(NULL AS DECIMAL(18, 2)) AS budget_usd, + CAST(NULL AS DECIMAL(18, 2)) AS spend_usd, + CAST(NULL AS DECIMAL(12, 4)) AS return_on_ad_spend +), + +__assert__campaign_id_is_unique_and_present AS ( + SELECT t.campaign_id + FROM __ref("dim_campaigns") AS t + GROUP BY + t.campaign_id + HAVING + COUNT(*) > 1 + OR t.campaign_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_carriers__summarises_current_rate_cards_and_dispatch_record.sql b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_carriers__summarises_current_rate_cards_and_dispatch_record.sql new file mode 100644 index 0000000000..216d106ace --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_carriers__summarises_current_rate_cards_and_dispatch_record.sql @@ -0,0 +1,182 @@ +TEST (name "dim_carriers__summarises_current_rate_cards_and_dispatch_record"); + +WITH __ref__stg_carrier_rate_cards AS ( + SELECT + 'RC1' AS rate_card_id, + 'FLAPWING' AS carrier_code, + 'STANDARD' AS service_level, + 'DE' AS origin_country, + 'FR' AS destination_country, + 'EUR' AS currency_code, + '2026-01-01'::DATE AS valid_from, + NULL AS valid_to, + NULL AS band_count, + NULL AS bands, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'RC2', + 'FLAPWING', + 'EXPRESS', + 'DE', + 'AT', + 'EUR', + '2026-01-01'::DATE, + '2027-01-01'::DATE, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'RC3', + 'FLAPWING', + 'STANDARD', + 'DE', + 'NL', + 'EUR', + '2025-01-01'::DATE, + '2026-03-01'::DATE, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'RC4', + 'WINGBEAT', + 'EXPRESS', + 'DE', + 'FR', + 'EUR', + '2026-05-01'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_fulfilment_sla AS ( + SELECT + 'S1' AS shipment_id, + NULL AS order_id, + NULL AS warehouse_code, + 'FLAPWING' AS carrier_code, + NULL AS placed_at_local, + NULL AS shipped_at_local, + NULL AS calendar_days_to_ship, + NULL AS business_days_to_ship, + NULL AS holidays_in_window, + NULL AS dispatch_business_minutes, + TRUE AS is_dispatch_on_time, + NULL AS delivered_at_utc, + 24.0::DECIMAL(10, 2) AS hours_to_deliver + UNION ALL + SELECT + 'S2', + NULL, + NULL, + 'FLAPWING', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + 48.0::DECIMAL(10, 2) + UNION ALL + SELECT + 'S3', + NULL, + NULL, + 'FLAPWING', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL +), + +__seed__carriers AS ( + SELECT + 'FLAPWING' AS carrier_code, + 'Flapwing Parcels' AS carrier_name, + 'https://track.example.com/flapwing/{tracking_number}' AS tracking_url_template, + FALSE AS is_express + UNION ALL + SELECT + 'WINGBEAT', + 'Wingbeat Couriers', + 'http://wingbeat.example.com/t/{tracking_number}', + TRUE + UNION ALL + SELECT + 'WEBFOOT', + 'Webfoot Freight', + NULL, + FALSE +), + +__expected__dim_carriers AS ( + SELECT + 'FLAPWING' AS carrier_code, + 'Flapwing Parcels' AS carrier_name, + FALSE AS is_express, + 'track.example.com' AS tracking_host, + 2 AS destination_country_count, + 2 AS active_rate_card_count, + 3 AS shipment_count, + 0.6667::DECIMAL(9, 4) AS on_time_dispatch_rate, + 36.0::DECIMAL(10, 2) AS median_hours_to_deliver + UNION ALL + SELECT + 'WINGBEAT' AS carrier_code, + 'Wingbeat Couriers' AS carrier_name, + TRUE AS is_express, + 'wingbeat.example.com' AS tracking_host, + 0 AS destination_country_count, + 0 AS active_rate_card_count, + 0 AS shipment_count, + CAST(NULL AS DECIMAL(9, 4)) AS on_time_dispatch_rate, + CAST(NULL AS DECIMAL(10, 2)) AS median_hours_to_deliver + UNION ALL + SELECT + 'WEBFOOT' AS carrier_code, + 'Webfoot Freight' AS carrier_name, + FALSE AS is_express, + CAST(NULL AS VARCHAR) AS tracking_host, + 0 AS destination_country_count, + 0 AS active_rate_card_count, + 0 AS shipment_count, + CAST(NULL AS DECIMAL(9, 4)) AS on_time_dispatch_rate, + CAST(NULL AS DECIMAL(10, 2)) AS median_hours_to_deliver +), + +__assert__service_levels_are_distinct_and_sorted AS ( + SELECT c.carrier_code + FROM __ref("dim_carriers") AS c + WHERE + ARRAY_TO_STRING(c.service_levels, ',') <> IFF( + c.carrier_code = 'FLAPWING', + 'EXPRESS,STANDARD', + '' + ) +), + +__assert__carrier_code_is_unique_and_present AS ( + SELECT t.carrier_code + FROM __ref("dim_carriers") AS t + GROUP BY + t.carrier_code + HAVING + COUNT(*) > 1 + OR t.carrier_code IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_customers__combines_profile_value_consent_and_loyalty.sql b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_customers__combines_profile_value_consent_and_loyalty.sql new file mode 100644 index 0000000000..c3019535ba --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_customers__combines_profile_value_consent_and_loyalty.sql @@ -0,0 +1,209 @@ +TEST (name "dim_customers__combines_profile_value_consent_and_loyalty"); + +WITH __ref__stg_customers AS ( + SELECT + 'C1' AS customer_id, + NULL AS email_normalised, + NULL AS phone_normalised, + NULL AS first_name, + NULL AS last_name, + 'EU' AS region_code, + 'eu-web' AS storefront_code, + NULL AS created_at, + 'de' AS preferred_language, + TRUE AS marketing_opt_in, + NULL AS consent_types, + 1990 AS birth_year, + NULL AS preferences, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'C2', + NULL, + NULL, + NULL, + NULL, + 'US', + 'us-web', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C3', + NULL, + NULL, + NULL, + NULL, + 'EU', + 'eu-web', + NULL, + NULL, + TRUE, + NULL, + 2005, + NULL, + NULL, + NULL +), + +__ref__int_customer_identity AS ( + SELECT + 'C1' AS customer_id, + NULL AS email_normalised, + 'example.com' AS email_domain, + NULL AS email_dedupe_key, + NULL AS phone_normalised, + 'Wren Featherby' AS full_name, + NULL AS anonymous_ids, + NULL AS anonymous_id_count, + NULL AS newsletter_lists, + NULL AS ticket_count, + 'C3' AS duplicate_candidate_id, + NULL AS duplicate_name_similarity, + NULL AS accounts_sharing_email +), + +__ref__int_customer_address_current AS ( + SELECT + 'C1' AS customer_id, + NULL AS address_id, + 'Munich' AS city, + NULL AS postcode_canonical, + 'DE' AS country_code, + 'EU' AS region_code, + NULL AS latitude, + NULL AS longitude, + NULL AS is_explicit_default, + NULL AS address_count, + NULL AS distinct_countries +), + +__ref__int_customer_rfm AS ( + SELECT + 'C1' AS customer_id, + NULL AS last_order_date, + 3 AS days_since_last_order, + 5 AS order_count, + 250.5::NUMBER(18, 2) AS lifetime_revenue_usd, + NULL AS average_order_usd, + NULL AS recency_score, + NULL AS frequency_score, + NULL AS monetary_score, + NULL AS rfm_code, + 'champion' AS segment, + TRUE AS is_vip +), + +__ref__int_customer_consent_state AS ( + SELECT + 'C1' AS customer_id, + FALSE AS email_marketing, + TRUE AS sms_marketing, + NULL AS personalisation, + NULL AS third_party_sharing, + NULL AS consent_state, + NULL AS last_consent_change_at, + NULL AS has_legacy_consent +), + +__ref__int_loyalty_tier_changes AS ( + SELECT + NULL AS loyalty_account_id, + 'C1' AS customer_id, + 'DUCKLING' AS tier_code, + NULL AS previous_tier_code, + 'LA1:1' AS tier_change_key, + 1 AS change_sequence, + NULL AS changed_at, + NULL AS direction, + NULL AS days_in_previous_tier + UNION ALL + SELECT + NULL, + 'C1', + 'GOLDEN_DUCK', + NULL, + 'LA1:2', + 2, + NULL, + NULL, + NULL +), + +__ref__int_cross_device_identity AS ( + SELECT + 'C1' AS customer_id, + NULL AS web_visitor_count, + NULL AS app_device_count, + NULL AS platforms, + TRUE AS is_multi_device, + NULL AS first_seen_at_utc, + NULL AS identity_graph +), + +__expected__dim_customers AS ( + SELECT + 'C1' AS customer_id, + 'Munich' AS city, + 'EU' AS region_code, + 'Wren Featherby' AS full_name, + '25_39' AS age_band, + 5 AS order_count, + 250.5::DECIMAL(18, 2) AS lifetime_revenue_usd, + 'champion' AS rfm_segment, + TRUE AS is_vip, + FALSE AS email_marketing, + TRUE AS sms_marketing, + 'GOLDEN_DUCK' AS loyalty_tier_code, + TRUE AS is_multi_device, + 'C3' AS possible_duplicate_of + UNION ALL + SELECT + 'C2' AS customer_id, + CAST(NULL AS VARCHAR) AS city, + 'US' AS region_code, + CAST(NULL AS VARCHAR) AS full_name, + 'unknown' AS age_band, + 0 AS order_count, + 0.0::DECIMAL(18, 2) AS lifetime_revenue_usd, + 'prospect' AS rfm_segment, + FALSE AS is_vip, + FALSE AS email_marketing, + FALSE AS sms_marketing, + CAST(NULL AS VARCHAR) AS loyalty_tier_code, + FALSE AS is_multi_device, + CAST(NULL AS VARCHAR) AS possible_duplicate_of + UNION ALL + SELECT + 'C3' AS customer_id, + CAST(NULL AS VARCHAR) AS city, + 'EU' AS region_code, + CAST(NULL AS VARCHAR) AS full_name, + 'under_25' AS age_band, + 0 AS order_count, + 0.0::DECIMAL(18, 2) AS lifetime_revenue_usd, + 'prospect' AS rfm_segment, + FALSE AS is_vip, + TRUE AS email_marketing, + FALSE AS sms_marketing, + CAST(NULL AS VARCHAR) AS loyalty_tier_code, + FALSE AS is_multi_device, + CAST(NULL AS VARCHAR) AS possible_duplicate_of +), + +__assert__customer_id_is_unique_and_present AS ( + SELECT t.customer_id + FROM __ref("dim_customers") AS t + GROUP BY + t.customer_id + HAVING + COUNT(*) > 1 + OR t.customer_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_date__builds_a_gap_free_three_year_calendar.sql b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_date__builds_a_gap_free_three_year_calendar.sql new file mode 100644 index 0000000000..956f2c1fa3 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_date__builds_a_gap_free_three_year_calendar.sql @@ -0,0 +1,71 @@ +TEST (name "dim_date__builds_a_gap_free_three_year_calendar"); + +WITH __seed__holiday_calendar AS ( + SELECT + 'EU' AS region_code, + '2026-04-03'::DATE AS holiday_date, + 'Good Friday' AS holiday_name + UNION ALL + SELECT + 'US', + '2026-04-03'::DATE, + 'Synthetic holiday' +), + +__assert__covers_three_years_without_gaps AS ( + SELECT COUNT(DISTINCT c.calendar_date) AS day_count + FROM __ref("dim_date") AS c + HAVING + COUNT(DISTINCT c.calendar_date) <> 1095 + OR COUNT(*) <> 1095 + OR MIN(c.calendar_date) <> '2025-01-01'::DATE + OR MAX(c.calendar_date) <> '2027-12-31'::DATE +), + +__assert__fiscal_year_starts_in_april AS ( + SELECT c.calendar_date + FROM __ref("dim_date") AS c + WHERE + ( + c.calendar_date = '2026-04-01'::DATE + AND ( + c.fiscal_year <> 2027 + OR c.fiscal_quarter <> 1 + ) + ) + OR ( + c.calendar_date = '2026-03-31'::DATE + AND ( + c.fiscal_year <> 2026 + OR c.fiscal_quarter <> 4 + ) + ) +), + +__assert__iso_week_belongs_to_previous_year_at_boundary AS ( + SELECT c.calendar_date + FROM __ref("dim_date") AS c + WHERE + c.calendar_date = '2027-01-01'::DATE + AND ( + c.iso_year <> 2026 + OR c.iso_week <> 53 + ) +), + +__assert__holidays_list_every_region AS ( + SELECT c.calendar_date + FROM __ref("dim_date") AS c + WHERE + ( + c.calendar_date = '2026-04-03'::DATE + AND ( + ARRAY_TO_STRING(c.holiday_regions, ',') <> 'EU,US' + OR NOT c.is_holiday_somewhere + ) + ) + OR ( + c.calendar_date = '2026-04-04'::DATE + AND c.is_holiday_somewhere + ) +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_products__enriches_sellable_skus_with_usd_margin_and_reviews.sql b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_products__enriches_sellable_skus_with_usd_margin_and_reviews.sql new file mode 100644 index 0000000000..a0085e9195 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_products__enriches_sellable_skus_with_usd_margin_and_reviews.sql @@ -0,0 +1,200 @@ +TEST (name "dim_products__enriches_sellable_skus_with_usd_margin_and_reviews"); + +WITH __ref__int_product_catalogue AS ( + SELECT + 'DUCK-MAL-01' AS sku, + 'P1' AS product_id, + 'DUCK' AS sku_family, + 'Mallard Classic Duck' AS title, + NULL AS variant_label, + NULL AS brand_code, + 'Mallard & Co.' AS brand_name, + TRUE AS is_private_label, + NULL AS category_id, + NULL AS category_slug_path, + 'ducks' AS category_level_1, + NULL AS material, + NULL AS weight_kg, + NULL AS primary_image_url, + NULL AS image_count, + TRUE AS is_active, + NULL AS launched_on, + NULL AS is_bundle_component + UNION ALL + SELECT + 'DUCK-TEAL-02', + 'P1', + 'DUCK', + 'Mallard Classic Duck', + NULL, + NULL, + 'Mallard & Co.', + TRUE, + NULL, + NULL, + 'ducks', + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'BATH-EID-1', + 'P9', + 'BATH', + 'Eider Bath Set', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL +), + +__ref__stg_products AS ( + SELECT + 'P1' AS product_id, + 'Mallard Classic Duck' AS title, + NULL AS category_path, + NULL AS category_level_1, + NULL AS category_level_2, + NULL AS category_leaf, + NULL AS category_depth, + NULL AS brand_code, + NULL AS launched_on, + NULL AS is_active, + NULL AS spec_key_count, + NULL AS specs, + NULL AS source_seq, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ AS source_loaded_at +), + +__ref__int_product_margin AS ( + SELECT + 'DUCK-MAL-01:USD' AS sku_price_key, + 'DUCK-MAL-01' AS sku, + 'USD' AS currency_code, + 12.99::DECIMAL(18, 4) AS list_price, + NULL AS unit_cost, + NULL AS unit_margin, + 55.0::DECIMAL(9, 2) AS margin_pct, + NULL AS is_below_cost, + NULL AS category_level_1 + UNION ALL + SELECT + 'DUCK-MAL-01:EUR', + 'DUCK-MAL-01', + 'EUR', + 11.99::DECIMAL(18, 4), + NULL, + NULL, + 52.0::DECIMAL(9, 2), + NULL, + NULL +), + +__ref__int_supplier_cost_current AS ( + SELECT + 'DUCK-MAL-01' AS sku, + 'SUP1' AS supplier_code, + NULL AS supplier_name, + NULL AS currency_code, + NULL AS unit_cost_minor, + NULL AS unit_cost_usd, + NULL AS effective_from, + 2 AS supplier_count, + NULL AS payment_terms_days, + NULL AS cost_spread_pct +), + +__ref__int_reviews_moderated AS ( + SELECT + 'RV1' AS review_id, + 'DUCK-MAL-01' AS sku, + NULL AS product_id, + NULL AS customer_id, + 5 AS rating, + NULL AS submitted_at, + NULL AS is_verified_purchase, + NULL AS sentiment_score, + NULL AS review_weight, + NULL AS rating_rank_in_product + UNION ALL + SELECT + 'RV2', + 'DUCK-MAL-01', + NULL, + NULL, + 4, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'RV3', + 'DUCK-MAL-01', + NULL, + NULL, + 4, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_wishlist_demand AS ( + SELECT + 'DUCK-TEAL-02' AS sku, + 7 AS wishlisting_customers, + NULL AS public_wishlist_count, + NULL AS purchased_after_wishlist, + NULL AS wishlist_conversion_rate, + NULL AS median_days_to_purchase, + NULL AS first_added_at +), + +__expected__dim_products AS ( + SELECT + 'DUCK-MAL-01' AS sku, + 12.99::DECIMAL(18, 4) AS list_price_usd, + 55.0::DECIMAL(9, 2) AS margin_pct, + 'SUP1' AS primary_supplier_code, + 2 AS supplier_count, + 3 AS review_count, + 4.33::DECIMAL(4, 2) AS average_rating, + 0 AS wishlisting_customers + UNION ALL + SELECT + 'DUCK-TEAL-02' AS sku, + CAST(NULL AS DECIMAL(18, 4)) AS list_price_usd, + CAST(NULL AS DECIMAL(9, 2)) AS margin_pct, + CAST(NULL AS VARCHAR) AS primary_supplier_code, + 0 AS supplier_count, + 0 AS review_count, + CAST(NULL AS DECIMAL(4, 2)) AS average_rating, + 7 AS wishlisting_customers +), + +__assert__sku_is_unique_and_present AS ( + SELECT t.sku + FROM __ref("dim_products") AS t + GROUP BY + t.sku + HAVING + COUNT(*) > 1 + OR t.sku IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_promotions__defaults_performance_for_unused_codes.sql b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_promotions__defaults_performance_for_unused_codes.sql new file mode 100644 index 0000000000..8e3df577a4 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_promotions__defaults_performance_for_unused_codes.sql @@ -0,0 +1,78 @@ +TEST (name "dim_promotions__defaults_performance_for_unused_codes"); + +WITH __ref__stg_promotions AS ( + SELECT + 'SPRING10' AS promo_code, + 'Spring Splash' AS promo_name, + NULL AS starts_at, + NULL AS ends_at, + FALSE AS is_stackable, + NULL AS rules, + 2000 AS min_order_minor, + NULL AS eligible_regions, + NULL AS tier_count, + NULL AS max_discount_pct, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'POND5', + 'Pond Week', + NULL, + NULL, + TRUE, + NULL, + 0, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_promotion_performance AS ( + SELECT + 'SPRING10' AS promo_code, + NULL AS promo_name, + TRUE AS is_active_now, + 3 AS redemption_count, + NULL AS redeeming_customers, + 120.5::DECIMAL(18, 2) AS revenue_usd, + NULL AS discount_usd, + 0.1::DECIMAL(9, 6) AS discount_rate, + NULL AS avg_order_usd, + NULL AS baseline_avg_order_usd, + 12.5::DECIMAL(9, 2) AS basket_uplift_pct, + NULL AS top_tier_threshold_minor, + NULL AS eligible_region_count +), + +__expected__dim_promotions AS ( + SELECT + 'SPRING10' AS promo_code, + TRUE AS is_active_now, + 3 AS redemption_count, + 120.5::DECIMAL(18, 2) AS revenue_usd, + 0.1::DECIMAL(9, 6) AS discount_rate, + 12.5::DECIMAL(9, 2) AS basket_uplift_pct, + 2000 AS min_order_minor + UNION ALL + SELECT + 'POND5' AS promo_code, + FALSE AS is_active_now, + 0 AS redemption_count, + 0.0::DECIMAL(18, 2) AS revenue_usd, + CAST(NULL AS DECIMAL(9, 6)) AS discount_rate, + CAST(NULL AS DECIMAL(9, 2)) AS basket_uplift_pct, + 0 AS min_order_minor +), + +__assert__promo_code_is_unique_and_present AS ( + SELECT t.promo_code + FROM __ref("dim_promotions") AS t + GROUP BY + t.promo_code + HAVING + COUNT(*) > 1 + OR t.promo_code IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_suppliers__bands_suppliers_by_delivery_performance.sql b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_suppliers__bands_suppliers_by_delivery_performance.sql new file mode 100644 index 0000000000..e771187731 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_suppliers__bands_suppliers_by_delivery_performance.sql @@ -0,0 +1,168 @@ +TEST (name "dim_suppliers__bands_suppliers_by_delivery_performance"); + +WITH __ref__stg_suppliers AS ( + SELECT + 'SUP1' AS supplier_code, + 'Teal Hollow Decoys' AS supplier_name, + NULL AS country_code, + NULL AS payment_terms_days, + NULL AS early_payment_discount_pct, + 'orders@tealhollow.example.com' AS primary_contact_email, + NULL AS contact_count, + PARSE_JSON('{"currency": "eur"}')::OBJECT AS payment_terms, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'SUP2', + 'Sudsy Drake Supply', + NULL, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('{}')::OBJECT, + NULL, + NULL + UNION ALL + SELECT + 'SUP3', + 'Pond Pumps Direct', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'SUP4', + 'Bill & Coo Textiles', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_supplier_lead_times AS ( + SELECT + 'SUP1' AS supplier_code, + NULL AS supplier_name, + NULL AS received_po_count, + 12.5::DECIMAL(10, 2) AS lead_time_p50_days, + NULL AS lead_time_p90_days, + NULL AS lead_time_stddev_days, + 0.97::DECIMAL(9, 4) AS on_time_rate, + 0.99::DECIMAL(9, 4) AS avg_fill_rate, + NULL AS latest_po_at + UNION ALL + SELECT + 'SUP2', + NULL, + NULL, + 20.0::DECIMAL(10, 2), + NULL, + NULL, + 0.85::DECIMAL(9, 4), + 0.9::DECIMAL(9, 4), + NULL + UNION ALL + SELECT + 'SUP3', + NULL, + NULL, + 30.0::DECIMAL(10, 2), + NULL, + NULL, + 0.5::DECIMAL(9, 4), + 1.0::DECIMAL(9, 4), + NULL +), + +__ref__int_supplier_cost_current AS ( + SELECT + 'DUCK-MAL-01' AS sku, + 'SUP1' AS supplier_code, + NULL AS supplier_name, + NULL AS currency_code, + NULL AS unit_cost_minor, + NULL AS unit_cost_usd, + NULL AS effective_from, + NULL AS supplier_count, + NULL AS payment_terms_days, + NULL AS cost_spread_pct + UNION ALL + SELECT + 'DUCK-TEAL-02', + 'SUP1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'DECOY-TEAL-1', + 'SUP1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__dim_suppliers AS ( + SELECT + 'SUP1' AS supplier_code, + 'EUR' AS terms_currency_code, + 'tealhollow.example.com' AS contact_domain, + 3 AS supplied_sku_count, + 12.5::DECIMAL(10, 2) AS lead_time_p50_days, + 'preferred' AS performance_band + UNION ALL + SELECT + 'SUP2' AS supplier_code, + CAST(NULL AS VARCHAR) AS terms_currency_code, + CAST(NULL AS VARCHAR) AS contact_domain, + 0 AS supplied_sku_count, + 20.0::DECIMAL(10, 2) AS lead_time_p50_days, + 'approved' AS performance_band + UNION ALL + SELECT + 'SUP3' AS supplier_code, + CAST(NULL AS VARCHAR) AS terms_currency_code, + CAST(NULL AS VARCHAR) AS contact_domain, + 0 AS supplied_sku_count, + 30.0::DECIMAL(10, 2) AS lead_time_p50_days, + 'at_risk' AS performance_band + UNION ALL + SELECT + 'SUP4' AS supplier_code, + CAST(NULL AS VARCHAR) AS terms_currency_code, + CAST(NULL AS VARCHAR) AS contact_domain, + 0 AS supplied_sku_count, + CAST(NULL AS DECIMAL(10, 2)) AS lead_time_p50_days, + 'unrated' AS performance_band +), + +__assert__supplier_code_is_unique_and_present AS ( + SELECT t.supplier_code + FROM __ref("dim_suppliers") AS t + GROUP BY + t.supplier_code + HAVING + COUNT(*) > 1 + OR t.supplier_code IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_warehouses__adds_region_offset_bins_and_latest_stock.sql b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_warehouses__adds_region_offset_bins_and_latest_stock.sql new file mode 100644 index 0000000000..76ba6aa5b5 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/core/test_dim_warehouses__adds_region_offset_bins_and_latest_stock.sql @@ -0,0 +1,187 @@ +TEST (name "dim_warehouses__adds_region_offset_bins_and_latest_stock"); + +WITH __ref__stg_warehouses AS ( + SELECT + 'POND' AS warehouse_code, + 'The Pond' AS warehouse_name, + 'DE' AS country_code, + 'Europe/Berlin' AS iana_timezone, + NULL AS opens_local, + NULL AS closes_local, + 720 AS opening_minutes, + TRUE AS supports_same_day, + FALSE AS supports_cold_chain, + NULL AS max_parcel_kg, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'NEST', + 'The Nest', + 'US', + 'America/New_York', + NULL, + NULL, + 480, + FALSE, + FALSE, + NULL, + NULL, + NULL +), + +__ref__stg_bin_locations AS ( + SELECT + 'B1' AS bin_id, + 'POND' AS warehouse_code, + NULL AS bin_label, + NULL AS aisle, + NULL AS bay, + NULL AS shelf_level, + NULL AS is_pick_face, + NULL AS max_weight_kg, + NULL AS temperature_zone, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'B2', + 'POND', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'B3', + 'POND', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_inventory_valuation_daily AS ( + SELECT + 'POND|2026-03-30' AS valuation_key, + 'POND' AS warehouse_code, + '2026-03-30'::DATE AS calendar_date, + NULL AS sku_count, + 100 AS units_on_hand, + NULL AS stock_value_usd, + NULL AS unvalued_units, + NULL AS stocked_out_skus, + NULL AS value_change_usd + UNION ALL + SELECT + 'POND|2026-03-31', + 'POND', + '2026-03-31'::DATE, + NULL, + 90, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'POND|2026-04-01', + 'POND', + '2026-04-01'::DATE, + NULL, + 50, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_delivery_slot_utilisation AS ( + SELECT + 'POND|2026-03-30' AS slot_day_key, + 'POND' AS warehouse_code, + NULL AS slot_date_local, + NULL AS slot_count, + NULL AS total_capacity, + NULL AS total_booked, + 0.5::DECIMAL(9, 4) AS utilisation, + NULL AS full_slot_count, + NULL AS busiest_slot_start_local, + NULL AS is_weekend, + NULL AS region_code, + NULL AS is_holiday, + NULL AS utilisation_vs_weekday_avg + UNION ALL + SELECT + 'POND|2026-03-31', + 'POND', + NULL, + NULL, + NULL, + NULL, + 0.75::DECIMAL(9, 4), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__seed__countries AS ( + SELECT + 'DE' AS country_code, + NULL AS country_name, + 'EU' AS region_code, + NULL AS default_timezone, + NULL AS currency_code, + NULL AS vat_registered + UNION ALL + SELECT + 'US', + NULL, + 'US', + NULL, + NULL, + NULL +), + +__expected__dim_warehouses AS ( + SELECT + 'POND' AS warehouse_code, + 'The Pond' AS warehouse_name, + 'EU' AS region_code, + 2.0::DECIMAL(4, 2) AS utc_offset_hours, + 3 AS bin_count, + 90 AS units_on_hand, + 0.625::DECIMAL(9, 4) AS average_slot_utilisation + UNION ALL + SELECT + 'NEST' AS warehouse_code, + 'The Nest' AS warehouse_name, + 'US' AS region_code, + -4.0::DECIMAL(4, 2) AS utc_offset_hours, + 0 AS bin_count, + 0 AS units_on_hand, + CAST(NULL AS DECIMAL(9, 4)) AS average_slot_utilisation +), + +__assert__warehouse_code_is_unique_and_present AS ( + SELECT t.warehouse_code + FROM __ref("dim_warehouses") AS t + GROUP BY + t.warehouse_code + HAVING + COUNT(*) > 1 + OR t.warehouse_code IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_consent_coverage__treats_missing_consent_state_as_not_granted.sql b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_consent_coverage__treats_missing_consent_state_as_not_granted.sql new file mode 100644 index 0000000000..ca6f6b6136 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_consent_coverage__treats_missing_consent_state_as_not_granted.sql @@ -0,0 +1,127 @@ +TEST (name "rpt_consent_coverage__treats_missing_consent_state_as_not_granted"); + +WITH __ref__dim_customers AS ( + SELECT + 'C1' AS customer_id, + NULL AS city, + NULL AS postcode_canonical, + NULL AS country_code, + 'EU' AS region_code, + NULL AS email_domain, + NULL AS full_name, + NULL AS storefront_code, + NULL AS preferred_language, + NULL AS created_at, + NULL AS age_band, + NULL AS order_count, + NULL AS lifetime_revenue_usd, + NULL AS days_since_last_order, + NULL AS rfm_segment, + NULL AS is_vip, + NULL AS email_marketing, + NULL AS sms_marketing, + NULL AS loyalty_tier_code, + NULL AS is_multi_device, + NULL AS possible_duplicate_of, + NULL AS is_marketing_holdout + UNION ALL + SELECT + 'C2', + NULL, + NULL, + NULL, + 'EU', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_customer_consent_state AS ( + SELECT + 'C1' AS customer_id, + TRUE AS email_marketing, + FALSE AS sms_marketing, + TRUE AS personalisation, + FALSE AS third_party_sharing, + NULL AS consent_state, + NULL AS last_consent_change_at, + TRUE AS has_legacy_consent + UNION ALL + SELECT + 'C2', + FALSE, + TRUE, + NULL, + NULL, + NULL, + NULL, + FALSE +), + +__expected__rpt_consent_coverage AS ( + SELECT + 'EU' AS region_code, + 2 AS customers, + 0.5::DECIMAL(9, 4) AS email_marketing_share, + 0.5::DECIMAL(9, 4) AS sms_marketing_share, + 0.5::DECIMAL(9, 4) AS personalisation_share, + 0.0::DECIMAL(9, 4) AS third_party_share, + 1 AS legacy_consent_customers + UNION ALL + SELECT + 'UNKNOWN' AS region_code, + 1 AS customers, + 0.0::DECIMAL(9, 4) AS email_marketing_share, + 0.0::DECIMAL(9, 4) AS sms_marketing_share, + 0.0::DECIMAL(9, 4) AS personalisation_share, + 0.0::DECIMAL(9, 4) AS third_party_share, + 0 AS legacy_consent_customers +), + +__assert__region_code_is_unique_and_present AS ( + SELECT t.region_code + FROM __ref("rpt_consent_coverage") AS t + GROUP BY + t.region_code + HAVING + COUNT(*) > 1 + OR t.region_code IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_customer_cohorts_monthly__tracks_retention_and_cumulative_revenue_per_cohort.sql b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_customer_cohorts_monthly__tracks_retention_and_cumulative_revenue_per_cohort.sql new file mode 100644 index 0000000000..13e44e93d9 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_customer_cohorts_monthly__tracks_retention_and_cumulative_revenue_per_cohort.sql @@ -0,0 +1,114 @@ +TEST (name "rpt_customer_cohorts_monthly__tracks_retention_and_cumulative_revenue_per_cohort"); + +WITH __ref__int_customer_order_sequence AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + NULL AS placed_at, + '2026-01-05'::DATE AS placed_date_local, + NULL AS order_number, + NULL AS is_first_order, + NULL AS days_since_previous_order, + 10.0::NUMBER(18, 2) AS amount_usd, + NULL AS lifetime_revenue_usd, + NULL AS rolling_90d_revenue_usd, + NULL AS previous_sales_channel, + NULL AS is_channel_switch, + '2026-01-01'::DATE AS cohort_month + UNION ALL + SELECT + 'O2', + 'C1', + NULL, + '2026-01-20'::DATE, + NULL, + NULL, + NULL, + 20.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + '2026-01-01'::DATE + UNION ALL + SELECT + 'O3', + 'C1', + NULL, + '2026-03-02'::DATE, + NULL, + NULL, + NULL, + 30.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + '2026-01-01'::DATE + UNION ALL + SELECT + 'O4', + 'C2', + NULL, + '2026-01-10'::DATE, + NULL, + NULL, + NULL, + 40.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + '2026-01-01'::DATE + UNION ALL + SELECT + 'O5', + 'C3', + NULL, + '2026-02-14'::DATE, + NULL, + NULL, + NULL, + 50.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + '2026-02-01'::DATE +), + +__expected__rpt_customer_cohorts_monthly AS ( + SELECT + '2026-01+0' AS cohort_cell_key, + 2 AS cohort_size, + 2 AS active_customers, + 1.0::DECIMAL(9, 4) AS retention_rate, + 70.0::DECIMAL(18, 2) AS revenue_usd, + 35.0::DECIMAL(18, 2) AS cumulative_revenue_per_customer_usd + UNION ALL + SELECT + '2026-01+2' AS cohort_cell_key, + 2 AS cohort_size, + 1 AS active_customers, + 0.5::DECIMAL(9, 4) AS retention_rate, + 30.0::DECIMAL(18, 2) AS revenue_usd, + 50.0::DECIMAL(18, 2) AS cumulative_revenue_per_customer_usd + UNION ALL + SELECT + '2026-02+0' AS cohort_cell_key, + 1 AS cohort_size, + 1 AS active_customers, + 1.0::DECIMAL(9, 4) AS retention_rate, + 50.0::DECIMAL(18, 2) AS revenue_usd, + 50.0::DECIMAL(18, 2) AS cumulative_revenue_per_customer_usd +), + +__assert__cohort_cell_key_is_unique_and_present AS ( + SELECT t.cohort_cell_key + FROM __ref("rpt_customer_cohorts_monthly") AS t + GROUP BY + t.cohort_cell_key + HAVING + COUNT(*) > 1 + OR t.cohort_cell_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_customer_order_lookup__adds_running_totals_to_the_looked_up_customer.sql b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_customer_order_lookup__adds_running_totals_to_the_looked_up_customer.sql new file mode 100644 index 0000000000..e1238630c6 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_customer_order_lookup__adds_running_totals_to_the_looked_up_customer.sql @@ -0,0 +1,97 @@ +TEST (name "rpt_customer_order_lookup__adds_running_totals_to_the_looked_up_customer"); + +WITH __ref__dim_customers AS ( + SELECT + 'C000001' AS customer_id, + NULL AS city, + NULL AS postcode_canonical, + NULL AS country_code, + NULL AS region_code, + NULL AS email_domain, + NULL AS full_name, + NULL AS storefront_code, + NULL AS preferred_language, + NULL AS created_at, + NULL AS age_band, + NULL AS order_count, + 100.0::DECIMAL(18, 2) AS lifetime_revenue_usd, + NULL AS days_since_last_order, + 'loyal' AS rfm_segment, + NULL AS is_vip, + NULL AS email_marketing, + NULL AS sms_marketing, + 'GOLDEN_DUCK' AS loyalty_tier_code, + NULL AS is_multi_device, + NULL AS possible_duplicate_of, + NULL AS is_marketing_holdout + UNION ALL + SELECT + 'C000002', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 900.0::DECIMAL(18, 2), + NULL, + 'champion', + NULL, + NULL, + NULL, + 'PLATINUM_DUCK', + NULL, + NULL, + NULL +), + +__table_fn__table_fn__customer_orders AS ( + SELECT + 'O1' AS order_id, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS placed_at, + 'DELIVERED' AS order_status, + 40.00::NUMBER(18, 2) AS amount_usd, + 1 AS order_number, + TRUE AS is_first_order + UNION ALL + SELECT + 'O2', + '2026-03-15 10:00:00 +00:00'::TIMESTAMP_TZ, + 'SHIPPED', + 60.00::NUMBER(18, 2), + 2, + FALSE +), + +__expected__rpt_customer_order_lookup AS ( + SELECT + 'O1' AS order_id, + 'C000001' AS customer_id, + 'loyal' AS rfm_segment, + 'GOLDEN_DUCK' AS loyalty_tier_code, + 40.0::DECIMAL(18, 2) AS running_total_usd, + 0.4::DECIMAL(9, 6) AS share_of_lifetime_revenue + UNION ALL + SELECT + 'O2' AS order_id, + 'C000001' AS customer_id, + 'loyal' AS rfm_segment, + 'GOLDEN_DUCK' AS loyalty_tier_code, + 100.0::DECIMAL(18, 2) AS running_total_usd, + 0.6::DECIMAL(9, 6) AS share_of_lifetime_revenue +), + +__assert__order_id_is_unique_and_present AS ( + SELECT t.order_id + FROM __ref("rpt_customer_order_lookup") AS t + GROUP BY + t.order_id + HAVING + COUNT(*) > 1 + OR t.order_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_gift_card_liability__splits_expiring_and_dormant_balances.sql b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_gift_card_liability__splits_expiring_and_dormant_balances.sql new file mode 100644 index 0000000000..90c85af189 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_gift_card_liability__splits_expiring_and_dormant_balances.sql @@ -0,0 +1,81 @@ +TEST (name "rpt_gift_card_liability__splits_expiring_and_dormant_balances"); + +WITH __ref__int_gift_card_balances AS ( + SELECT + 'G1' AS gift_card_id, + 'EUR' AS currency_code, + NULL AS issued_at, + '2026-05-01'::DATE AS expires_on, + NULL AS initial_amount, + NULL AS redeemed_amount, + NULL AS balance_amount, + 50.0::DECIMAL(18, 2) AS balance_usd, + NULL AS transaction_count, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS last_used_at, + 'active' AS liability_status + UNION ALL + SELECT + 'G2', + 'EUR', + NULL, + '2027-01-01'::DATE, + NULL, + NULL, + NULL, + 20.0::DECIMAL(18, 2), + NULL, + '2025-01-01 10:00:00 +00:00'::TIMESTAMP_TZ, + 'active' + UNION ALL + SELECT + 'G3', + 'EUR', + NULL, + NULL, + NULL, + NULL, + NULL, + -5.0::DECIMAL(18, 2), + NULL, + NULL, + 'active' + UNION ALL + SELECT + 'G4', + 'USD', + NULL, + '2026-01-01'::DATE, + NULL, + NULL, + NULL, + 10.0::DECIMAL(18, 2), + NULL, + '2025-12-01 10:00:00 +00:00'::TIMESTAMP_TZ, + 'expired' +), + +__expected__rpt_gift_card_liability AS ( + SELECT + 'EUR:active' AS liability_key, + 3 AS card_count, + 70.0::DECIMAL(18, 2) AS balance_usd, + 50.0::DECIMAL(18, 2) AS expiring_90d_usd, + 1 AS dormant_card_count + UNION ALL + SELECT + 'USD:expired' AS liability_key, + 1 AS card_count, + 10.0::DECIMAL(18, 2) AS balance_usd, + 10.0::DECIMAL(18, 2) AS expiring_90d_usd, + 0 AS dormant_card_count +), + +__assert__liability_key_is_unique_and_present AS ( + SELECT t.liability_key + FROM __ref("rpt_gift_card_liability") AS t + GROUP BY + t.liability_key + HAVING + COUNT(*) > 1 + OR t.liability_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_loyalty_liability__values_each_account_at_the_valuation_date.sql b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_loyalty_liability__values_each_account_at_the_valuation_date.sql new file mode 100644 index 0000000000..9aa9a442a0 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_loyalty_liability__values_each_account_at_the_valuation_date.sql @@ -0,0 +1,96 @@ +TEST (name "rpt_loyalty_liability__values_each_account_at_the_valuation_date"); + +WITH __ref__int_loyalty_balances AS ( + SELECT + 'E1' AS entry_id, + 'LA1' AS loyalty_account_id, + NULL AS customer_id, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS posted_at, + NULL AS points_delta, + 100 AS running_points, + 0 AS expiring_points, + NULL AS is_expired, + 1.0::DECIMAL(18, 4) AS running_value_usd, + NULL AS reason_code, + 'DUCKLING' AS tier_code + UNION ALL + SELECT + 'E2', + 'LA1', + NULL, + '2026-03-20 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 250, + 50, + NULL, + 2.5::DECIMAL(18, 4), + NULL, + 'DRAKE' + UNION ALL + SELECT + 'E3', + 'LA1', + NULL, + '2026-04-02 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 999, + 0, + NULL, + 9.99::DECIMAL(18, 4), + NULL, + 'GOLDEN_DUCK' + UNION ALL + SELECT + 'E4', + 'LA2', + NULL, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 150, + 0, + NULL, + 1.5::DECIMAL(18, 4), + NULL, + 'DRAKE' + UNION ALL + SELECT + 'E5', + 'LA3', + NULL, + '2026-02-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 40, + 0, + NULL, + 0.4::DECIMAL(18, 4), + NULL, + NULL +), + +__expected__rpt_loyalty_liability AS ( + SELECT + 'DRAKE' AS tier_code, + 2 AS member_count, + 400 AS outstanding_points, + 4.0::DECIMAL(18, 2) AS liability_usd, + 50 AS expiring_points, + 200.0::DECIMAL(12, 2) AS average_points_per_member + UNION ALL + SELECT + 'NONE' AS tier_code, + 1 AS member_count, + 40 AS outstanding_points, + 0.4::DECIMAL(18, 2) AS liability_usd, + 0 AS expiring_points, + 40.0::DECIMAL(12, 2) AS average_points_per_member +), + +__assert__tier_code_is_unique_and_present AS ( + SELECT t.tier_code + FROM __ref("rpt_loyalty_liability") AS t + GROUP BY + t.tier_code + HAVING + COUNT(*) > 1 + OR t.tier_code IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_referral_program__costs_qualified_referrals_by_utc_invite_month.sql b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_referral_program__costs_qualified_referrals_by_utc_invite_month.sql new file mode 100644 index 0000000000..3cee257b89 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_referral_program__costs_qualified_referrals_by_utc_invite_month.sql @@ -0,0 +1,105 @@ +TEST (name "rpt_referral_program__costs_qualified_referrals_by_utc_invite_month"); + +WITH __ref__int_referral_conversions AS ( + SELECT + 'I1' AS invite_id, + NULL AS referrer_customer_id, + NULL AS invitee_customer_id, + NULL AS invitee_email_domain, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_TZ AS sent_at, + '2026-03-06 10:00:00 +00:00'::TIMESTAMP_TZ AS accepted_at, + NULL AS hours_to_accept, + NULL AS first_order_id, + 50.0::DECIMAL(18, 2) AS first_order_usd, + NULL AS days_to_first_order, + FALSE AS is_self_referral, + TRUE AS is_qualified, + 10.0::DECIMAL(18, 2) AS reward_usd, + NULL AS invite_sequence_for_referrer, + NULL AS referrer_lifetime_orders + UNION ALL + SELECT + 'I2', + NULL, + NULL, + NULL, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'I3', + NULL, + NULL, + NULL, + '2026-03-31 23:30:00 -02:00'::TIMESTAMP_TZ, + '2026-04-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL, + TRUE, + FALSE, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'I4', + NULL, + NULL, + NULL, + '2026-03-12 10:00:00 +00:00'::TIMESTAMP_TZ, + '2026-03-12 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + 30.0::DECIMAL(18, 2), + NULL, + FALSE, + TRUE, + 10.0::DECIMAL(18, 2), + NULL, + NULL +), + +__expected__rpt_referral_program AS ( + SELECT + '2026-03-01'::DATE AS invite_month, + 3 AS invites_sent, + 2 AS accepted, + 2 AS qualified, + 0 AS self_referrals, + 0.6667::DECIMAL(9, 4) AS acceptance_rate, + 20.0::DECIMAL(18, 2) AS reward_cost_usd, + 80.0::DECIMAL(18, 2) AS referred_revenue_usd, + 10.0::DECIMAL(18, 2) AS cost_per_qualified_usd + UNION ALL + SELECT + '2026-04-01'::DATE AS invite_month, + 1 AS invites_sent, + 1 AS accepted, + 0 AS qualified, + 1 AS self_referrals, + 1.0::DECIMAL(9, 4) AS acceptance_rate, + 0.0::DECIMAL(18, 2) AS reward_cost_usd, + 0.0::DECIMAL(18, 2) AS referred_revenue_usd, + CAST(NULL AS DECIMAL(18, 2)) AS cost_per_qualified_usd +), + +__assert__invite_month_is_unique_and_present AS ( + SELECT t.invite_month + FROM __ref("rpt_referral_program") AS t + GROUP BY + t.invite_month + HAVING + COUNT(*) > 1 + OR t.invite_month IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_rfm_segment_summary__shares_revenue_between_segments_in_a_region.sql b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_rfm_segment_summary__shares_revenue_between_segments_in_a_region.sql new file mode 100644 index 0000000000..f1fe548f0b --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/customers/test_rpt_rfm_segment_summary__shares_revenue_between_segments_in_a_region.sql @@ -0,0 +1,135 @@ +TEST (name "rpt_rfm_segment_summary__shares_revenue_between_segments_in_a_region"); + +WITH __ref__dim_customers AS ( + SELECT + 'C1' AS customer_id, + NULL AS city, + NULL AS postcode_canonical, + NULL AS country_code, + 'EU' AS region_code, + NULL AS email_domain, + NULL AS full_name, + NULL AS storefront_code, + NULL AS preferred_language, + NULL AS created_at, + NULL AS age_band, + NULL AS order_count, + 300.0::DECIMAL(18, 2) AS lifetime_revenue_usd, + 5 AS days_since_last_order, + 'champion' AS rfm_segment, + TRUE AS is_vip, + NULL AS email_marketing, + NULL AS sms_marketing, + NULL AS loyalty_tier_code, + NULL AS is_multi_device, + NULL AS possible_duplicate_of, + NULL AS is_marketing_holdout + UNION ALL + SELECT + 'C2', + NULL, + NULL, + NULL, + 'EU', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 100.0::DECIMAL(18, 2), + 15, + 'champion', + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C3', + NULL, + NULL, + NULL, + 'EU', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 100.0::DECIMAL(18, 2), + 200, + 'at_risk', + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'C4', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 0.0::DECIMAL(18, 2), + NULL, + 'prospect', + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__rpt_rfm_segment_summary AS ( + SELECT + 'EU:champion' AS segment_key, + 2 AS customers, + 1 AS vip_customers, + 400.0::DECIMAL(18, 2) AS lifetime_revenue_usd, + 0.8::DECIMAL(9, 4) AS revenue_share_in_region, + 10.0::DECIMAL(10, 2) AS median_days_since_last_order + UNION ALL + SELECT + 'EU:at_risk' AS segment_key, + 1 AS customers, + 0 AS vip_customers, + 100.0::DECIMAL(18, 2) AS lifetime_revenue_usd, + 0.2::DECIMAL(9, 4) AS revenue_share_in_region, + 200.0::DECIMAL(10, 2) AS median_days_since_last_order + UNION ALL + SELECT + 'UNKNOWN:prospect' AS segment_key, + 1 AS customers, + 0 AS vip_customers, + 0.0::DECIMAL(18, 2) AS lifetime_revenue_usd, + CAST(NULL AS DECIMAL(9, 4)) AS revenue_share_in_region, + CAST(NULL AS DECIMAL(10, 2)) AS median_days_since_last_order +), + +__assert__segment_key_is_unique_and_present AS ( + SELECT t.segment_key + FROM __ref("rpt_rfm_segment_summary") AS t + GROUP BY + t.segment_key + HAVING + COUNT(*) > 1 + OR t.segment_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/executive/test_rpt_executive_kpis_daily__fills_a_dense_grid_and_compares_with_last_week.sql b/benchmarks/duckshop/project/tests/unit/marts/executive/test_rpt_executive_kpis_daily__fills_a_dense_grid_and_compares_with_last_week.sql new file mode 100644 index 0000000000..867199df0c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/executive/test_rpt_executive_kpis_daily__fills_a_dense_grid_and_compares_with_last_week.sql @@ -0,0 +1,372 @@ +TEST (name "rpt_executive_kpis_daily__fills_a_dense_grid_and_compares_with_last_week"); + +WITH __ref__dim_date AS ( + SELECT + '2026-03-24'::DATE AS calendar_date, + NULL AS day_of_week_iso, + NULL AS day_name, + NULL AS is_weekend, + NULL AS iso_year, + NULL AS iso_week, + NULL AS month_start, + NULL AS month_end, + NULL AS quarter_label, + NULL AS fiscal_year, + NULL AS fiscal_quarter, + NULL AS holiday_regions, + NULL AS is_holiday_somewhere, + NULL AS is_on_or_before_valuation + UNION ALL + SELECT + '2026-03-31'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + '2026-04-01'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__fct_orders AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + 'EU' AS region_code, + NULL AS placed_at, + '2026-03-24'::DATE AS placed_date_local, + 'DELIVERED' AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS currency_code, + NULL AS amount, + 100.0::NUMBER(18, 2) AS amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size, + NULL AS order_number, + TRUE AS is_first_order, + NULL AS first_paid_at, + NULL AS first_delivered_at, + NULL AS hours_paid_to_shipped, + NULL AS captured_minor, + NULL AS refunded_minor, + NULL AS net_collected_minor, + NULL AS is_fully_refunded + UNION ALL + SELECT + 'O2', + NULL, + 'EU', + NULL, + '2026-03-31'::DATE, + 'SHIPPED', + NULL, + NULL, + NULL, + NULL, + 150.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + NULL, + 'EU', + NULL, + '2026-03-31'::DATE, + 'CANCELLED', + NULL, + NULL, + NULL, + NULL, + 999.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__fct_refunds AS ( + SELECT + CAST(NULL AS VARCHAR) AS refund_id, + CAST(NULL AS VARCHAR) AS payment_id, + CAST(NULL AS VARCHAR) AS order_id, + CAST(NULL AS VARCHAR) AS currency_code, + CAST(NULL AS NUMBER(38, 6)) AS amount, + CAST(NULL AS NUMBER(18, 2)) AS amount_usd, + CAST(NULL AS TIMESTAMP_TZ) AS refunded_at, + CAST(NULL AS DATE) AS refunded_date, + CAST(NULL AS VARCHAR) AS reason_category, + CAST(NULL AS INT) AS days_after_payment, + CAST(NULL AS BOOLEAN) AS is_full_refund + WHERE 1 = 0 +), + +__ref__fct_web_sessions AS ( + SELECT + CAST(NULL AS VARCHAR) AS session_id, + CAST(NULL AS VARCHAR) AS anonymous_id, + CAST(NULL AS VARCHAR) AS customer_id, + CAST(NULL AS TIMESTAMP_NTZ) AS session_start_utc, + CAST(NULL AS TIMESTAMP_NTZ) AS session_start_visitor_local, + CAST(NULL AS INT) AS duration_seconds, + CAST(NULL AS INT) AS page_view_count, + CAST(NULL AS VARCHAR) AS landing_page_path, + CAST(NULL AS VARCHAR) AS channel_code, + CAST(NULL AS BOOLEAN) AS is_paid, + CAST(NULL AS VARCHAR) AS utm_campaign, + CAST(NULL AS VARCHAR) AS first_touch_channel, + CAST(NULL AS INT) AS touch_number, + CAST(NULL AS BOOLEAN) AS is_bounce, + CAST(NULL AS VARCHAR) AS funnel_stage, + CAST(NULL AS BOOLEAN) AS converted, + CAST(NULL AS INT) AS visitor_bucket + WHERE 1 = 0 +), + +__ref__fct_support_tickets AS ( + SELECT + CAST(NULL AS VARCHAR) AS ticket_id, + CAST(NULL AS VARCHAR) AS customer_id, + CAST(NULL AS VARCHAR) AS channel, + CAST(NULL AS INT) AS priority, + CAST(NULL AS VARCHAR) AS ticket_status, + CAST(NULL AS VARCHAR) AS contact_reason, + CAST(NULL AS TIMESTAMPNTZ) AS created_at_utc, + CAST(NULL AS TIMESTAMPTZ) AS solved_at, + CAST(NULL AS INT) AS minutes_to_first_response, + CAST(NULL AS INT) AS business_minutes_to_response, + CAST(NULL AS BOOLEAN) AS is_first_response_breached, + CAST(NULL AS BOOLEAN) AS is_resolution_breached, + CAST(NULL AS INT) AS comment_count, + CAST(NULL AS INT) AS handoff_count, + CAST(NULL AS VARCHAR) AS sentiment_label, + CAST(NULL AS INT) AS ticket_number_for_customer, + CAST(NULL AS BOOLEAN) AS is_repeat_contact_7d, + CAST(NULL AS INT) AS csat_score, + CAST(NULL AS TIMESTAMPNTZ) AS source_loaded_at + WHERE 1 = 0 +), + +__ref__fct_shipments AS ( + SELECT CAST(NULL AS VARCHAR) AS shipment_id + WHERE 1 = 0 +), + +__ref__dim_warehouses AS ( + SELECT + CAST(NULL AS VARCHAR) AS warehouse_code, + CAST(NULL AS VARCHAR) AS warehouse_name, + CAST(NULL AS VARCHAR) AS country_code, + CAST(NULL AS VARCHAR) AS region_code, + CAST(NULL AS VARCHAR) AS iana_timezone, + CAST(NULL AS DECIMAL(4, 2)) AS utc_offset_hours, + CAST(NULL AS TIME) AS opens_local, + CAST(NULL AS TIME) AS closes_local, + CAST(NULL AS INT) AS opening_minutes, + CAST(NULL AS BOOLEAN) AS supports_same_day, + CAST(NULL AS BOOLEAN) AS supports_cold_chain, + CAST(NULL AS INT) AS bin_count, + CAST(NULL AS INT) AS units_on_hand, + CAST(NULL AS DECIMAL(9, 4)) AS average_slot_utilisation, + CAST(NULL AS TIMESTAMPNTZ) AS source_loaded_at + WHERE 1 = 0 +), + +__ref__fct_campaign_spend AS ( + SELECT CAST(NULL AS VARCHAR) AS spend_id + WHERE 1 = 0 +), + +__ref__fct_inventory_daily AS ( + SELECT CAST(NULL AS VARCHAR) AS stock_day_key + WHERE 1 = 0 +), + +__ref__int_revenue_recognition_daily AS ( + SELECT + CAST(NULL AS VARCHAR) AS recognition_key, + CAST(NULL AS DATE) AS recognition_date, + CAST(NULL AS VARCHAR) AS region_code, + CAST(NULL AS VARCHAR) AS revenue_stream, + CAST(NULL AS DECIMAL(18, 2)) AS revenue_usd, + CAST(NULL AS INT) AS document_count, + CAST(NULL AS DECIMAL(18, 2)) AS month_to_date_usd, + CAST(NULL AS DECIMAL(18, 2)) AS trailing_7d_usd, + CAST(NULL AS DECIMAL(9, 6)) AS share_of_day + WHERE 1 = 0 +), + +__ref__fct_partner_orders AS ( + SELECT CAST(NULL AS VARCHAR) AS partner_order_key + WHERE 1 = 0 +), + +__ref__int_chargeback_exposure AS ( + SELECT + CAST(NULL AS VARCHAR) AS chargeback_id, + CAST(NULL AS VARCHAR) AS payment_id, + CAST(NULL AS VARCHAR) AS order_id, + CAST(NULL AS VARCHAR) AS customer_id, + CAST(NULL AS VARCHAR) AS payment_method, + CAST(NULL AS VARCHAR) AS reason_code, + CAST(NULL AS VARCHAR) AS reason_family, + CAST(NULL AS TIMESTAMP_TZ) AS opened_at, + CAST(NULL AS DATE) AS response_due_on, + CAST(NULL AS VARCHAR) AS outcome, + CAST(NULL AS DECIMAL(18, 2)) AS amount_usd, + CAST(NULL AS DECIMAL(9, 6)) AS share_of_payment, + CAST(NULL AS ARRAY) AS evidence_kinds, + CAST(NULL AS BOOLEAN) AS has_delivery_proof, + CAST(NULL AS INT) AS days_open, + CAST(NULL AS BOOLEAN) AS is_overdue, + CAST(NULL AS DECIMAL(18, 2)) AS at_risk_usd, + CAST(NULL AS INT) AS prior_chargebacks_for_customer + WHERE 1 = 0 +), + +__expected__rpt_executive_kpis_daily AS ( + SELECT + '2026-03-24:EU:orders' AS kpi_key, + 'orders' AS business_area, + 1.0::DECIMAL(18, 4) AS metric_value, + CAST(NULL AS DECIMAL(18, 4)) AS value_week_ago, + CAST(NULL AS DECIMAL(12, 2)) AS week_over_week_pct, + 1.0::DECIMAL(18, 4) AS trailing_28d_average, + CAST(NULL AS BOOLEAN) AS is_favourable_change + UNION ALL + SELECT + '2026-03-31:EU:orders' AS kpi_key, + 'orders' AS business_area, + 1.0::DECIMAL(18, 4) AS metric_value, + 1.0::DECIMAL(18, 4) AS value_week_ago, + 0.0::DECIMAL(12, 2) AS week_over_week_pct, + 1.0::DECIMAL(18, 4) AS trailing_28d_average, + TRUE AS is_favourable_change + UNION ALL + SELECT + '2026-03-24:EU:revenue_usd' AS kpi_key, + 'orders' AS business_area, + 100.0::DECIMAL(18, 4) AS metric_value, + CAST(NULL AS DECIMAL(18, 4)) AS value_week_ago, + CAST(NULL AS DECIMAL(12, 2)) AS week_over_week_pct, + 100.0::DECIMAL(18, 4) AS trailing_28d_average, + CAST(NULL AS BOOLEAN) AS is_favourable_change + UNION ALL + SELECT + '2026-03-31:EU:revenue_usd' AS kpi_key, + 'orders' AS business_area, + 150.0::DECIMAL(18, 4) AS metric_value, + 100.0::DECIMAL(18, 4) AS value_week_ago, + 50.0::DECIMAL(12, 2) AS week_over_week_pct, + 125.0::DECIMAL(18, 4) AS trailing_28d_average, + TRUE AS is_favourable_change + UNION ALL + SELECT + '2026-03-24:EU:average_order_usd' AS kpi_key, + 'orders' AS business_area, + 100.0::DECIMAL(18, 4) AS metric_value, + CAST(NULL AS DECIMAL(18, 4)) AS value_week_ago, + CAST(NULL AS DECIMAL(12, 2)) AS week_over_week_pct, + 100.0::DECIMAL(18, 4) AS trailing_28d_average, + CAST(NULL AS BOOLEAN) AS is_favourable_change + UNION ALL + SELECT + '2026-03-31:EU:average_order_usd' AS kpi_key, + 'orders' AS business_area, + 150.0::DECIMAL(18, 4) AS metric_value, + 100.0::DECIMAL(18, 4) AS value_week_ago, + 50.0::DECIMAL(12, 2) AS week_over_week_pct, + 125.0::DECIMAL(18, 4) AS trailing_28d_average, + TRUE AS is_favourable_change + UNION ALL + SELECT + '2026-03-24:EU:new_customers' AS kpi_key, + 'customers' AS business_area, + 1.0::DECIMAL(18, 4) AS metric_value, + CAST(NULL AS DECIMAL(18, 4)) AS value_week_ago, + CAST(NULL AS DECIMAL(12, 2)) AS week_over_week_pct, + 1.0::DECIMAL(18, 4) AS trailing_28d_average, + CAST(NULL AS BOOLEAN) AS is_favourable_change + UNION ALL + SELECT + '2026-03-31:EU:new_customers' AS kpi_key, + 'customers' AS business_area, + 0.0::DECIMAL(18, 4) AS metric_value, + 1.0::DECIMAL(18, 4) AS value_week_ago, + -100.0::DECIMAL(12, 2) AS week_over_week_pct, + 0.5::DECIMAL(18, 4) AS trailing_28d_average, + FALSE AS is_favourable_change +), + +__assert__kpi_key_is_unique_and_present AS ( + SELECT t.kpi_key + FROM __ref("rpt_executive_kpis_daily") AS t + GROUP BY + t.kpi_key + HAVING + COUNT(*) > 1 + OR t.kpi_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/executive/test_rpt_region_scorecard_monthly__unpivots_measures_and_ranks_regions.sql b/benchmarks/duckshop/project/tests/unit/marts/executive/test_rpt_region_scorecard_monthly__unpivots_measures_and_ranks_regions.sql new file mode 100644 index 0000000000..8a789f4858 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/executive/test_rpt_region_scorecard_monthly__unpivots_measures_and_ranks_regions.sql @@ -0,0 +1,129 @@ +TEST (name "rpt_region_scorecard_monthly__unpivots_measures_and_ranks_regions"); + +WITH __ref__rpt_daily_sales_by_region AS ( + SELECT + 'EU:2026-03-10' AS sales_day_key, + '2026-03-10'::DATE AS placed_date_local, + 'EU' AS region_code, + 2 AS order_count, + NULL AS customer_count, + 1 AS first_order_count, + NULL AS units, + 100.0::DECIMAL(18, 2) AS revenue_usd, + NULL AS discount_usd, + NULL AS average_order_usd, + NULL AS revenue_same_day_last_week_usd, + NULL AS revenue_trailing_28d_usd + UNION ALL + SELECT + 'EU:2026-03-20', + '2026-03-20'::DATE, + 'EU', + 1, + NULL, + 0, + NULL, + 50.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'US:2026-03-05', + '2026-03-05'::DATE, + 'US', + 1, + NULL, + 1, + NULL, + 200.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL +), + +__ref__rpt_revenue_recognition_monthly AS ( + SELECT + '2026-03:EU'::TEXT AS recognition_month_key, + '2026-03-01'::DATE AS recognition_month, + 'EU'::TEXT AS region_code, + NULL AS product_usd, + NULL AS refund_usd, + 10.0::DECIMAL(18, 2) AS subscription_usd, + NULL AS net_revenue_usd, + 0.1::DECIMAL(9, 4) AS refund_ratio +), + +__expected__rpt_region_scorecard_monthly AS ( + SELECT + '2026-03:EU:orders'::TEXT AS scorecard_key, + 3.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:orders'::TEXT AS scorecard_key, + 1.0::DECIMAL(18, 4) AS measure_value, + 2 AS rank_among_regions + UNION ALL + SELECT + '2026-03:EU:revenue_usd'::TEXT AS scorecard_key, + 150.0::DECIMAL(18, 4) AS measure_value, + 2 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:revenue_usd'::TEXT AS scorecard_key, + 200.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:EU:first_orders'::TEXT AS scorecard_key, + 1.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:first_orders'::TEXT AS scorecard_key, + 1.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:EU:average_order_usd'::TEXT AS scorecard_key, + 50.0::DECIMAL(18, 4) AS measure_value, + 2 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:average_order_usd'::TEXT AS scorecard_key, + 200.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:EU:refund_ratio'::TEXT AS scorecard_key, + 0.1::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:refund_ratio'::TEXT AS scorecard_key, + 0.0::DECIMAL(18, 4) AS measure_value, + 2 AS rank_among_regions + UNION ALL + SELECT + '2026-03:EU:subscription_usd'::TEXT AS scorecard_key, + 10.0::DECIMAL(18, 4) AS measure_value, + 1 AS rank_among_regions + UNION ALL + SELECT + '2026-03:US:subscription_usd'::TEXT AS scorecard_key, + 0.0::DECIMAL(18, 4) AS measure_value, + 2 AS rank_among_regions +), + +__assert__scorecard_key_is_unique_and_present AS ( + SELECT t.scorecard_key + FROM __ref("rpt_region_scorecard_monthly") AS t + GROUP BY + t.scorecard_key + HAVING + COUNT(*) > 1 + OR t.scorecard_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/finance/test_fct_ledger_month_end__closes_months_at_the_as_of_rate.sql b/benchmarks/duckshop/project/tests/unit/marts/finance/test_fct_ledger_month_end__closes_months_at_the_as_of_rate.sql new file mode 100644 index 0000000000..4d034169ae --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/finance/test_fct_ledger_month_end__closes_months_at_the_as_of_rate.sql @@ -0,0 +1,142 @@ +TEST (name "fct_ledger_month_end__closes_months_at_the_as_of_rate"); + +WITH __ref__int_ledger_balances_daily AS ( + SELECT + '4000|EUR|2026-02-27' AS balance_key, + '4000' AS account_code, + 'REVENUE' AS account_class, + 'EUR' AS currency_code, + '2026-02-27'::DATE AS calendar_date, + 0 AS debit_minor, + 1000 AS credit_minor, + NULL AS posting_count, + NULL AS closing_balance_minor, + -1000.0::NUMBER(38, 6) AS closing_balance, + NULL AS month_to_date_movement_minor, + FALSE AS is_month_end, + TRUE AS is_normal_side + UNION ALL + SELECT + '4000|EUR|2026-02-28', + '4000', + 'REVENUE', + 'EUR', + '2026-02-28'::DATE, + 100, + 500, + NULL, + NULL, + -1400.0::NUMBER(38, 6), + NULL, + TRUE, + TRUE + UNION ALL + SELECT + '4000|EUR|2026-03-15', + '4000', + 'REVENUE', + 'EUR', + '2026-03-15'::DATE, + 0, + 300, + NULL, + NULL, + -1700.0::NUMBER(38, 6), + NULL, + FALSE, + TRUE + UNION ALL + SELECT + '4000|EUR|2026-03-31', + '4000', + 'REVENUE', + 'EUR', + '2026-03-31'::DATE, + 0, + 200, + NULL, + NULL, + -1900.0::NUMBER(38, 6), + NULL, + TRUE, + TRUE + UNION ALL + SELECT + '1000|GBP|2026-03-30', + '1000', + 'ASSET', + 'GBP', + '2026-03-30'::DATE, + 20, + 0, + NULL, + NULL, + 20.0::NUMBER(38, 6), + NULL, + FALSE, + TRUE + UNION ALL + SELECT + '1000|GBP|2026-03-31', + '1000', + 'ASSET', + 'GBP', + '2026-03-31'::DATE, + 50, + 0, + NULL, + NULL, + 70.0::NUMBER(38, 6), + NULL, + TRUE, + TRUE +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-02-27'::DATE AS rate_date, + 1.05::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'EUR', + '2026-03-30'::DATE, + 1.1::NUMBER(18, 8), + NULL, + NULL +), + +__expected__fct_ledger_month_end AS ( + SELECT + '4000:EUR:2026-02-28' AS ledger_month_key, + 100 AS month_debit_minor, + 1500 AS month_credit_minor, + -1400.0::NUMBER(38, 6) AS closing_balance, + -1470.0::DECIMAL(18, 2) AS closing_balance_usd + UNION ALL + SELECT + '4000:EUR:2026-03-31' AS ledger_month_key, + 0 AS month_debit_minor, + 500 AS month_credit_minor, + -1900.0::NUMBER(38, 6) AS closing_balance, + -2090.0::DECIMAL(18, 2) AS closing_balance_usd + UNION ALL + SELECT + '1000:GBP:2026-03-31' AS ledger_month_key, + 70 AS month_debit_minor, + 0 AS month_credit_minor, + 70.0::NUMBER(38, 6) AS closing_balance, + CAST(NULL AS DECIMAL(18, 2)) AS closing_balance_usd +), + +__assert__ledger_month_key_is_unique_and_present AS ( + SELECT t.ledger_month_key + FROM __ref("fct_ledger_month_end") AS t + GROUP BY + t.ledger_month_key + HAVING + COUNT(*) > 1 + OR t.ledger_month_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_chargeback_exposure__totals_exposure_by_reason_and_outcome.sql b/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_chargeback_exposure__totals_exposure_by_reason_and_outcome.sql new file mode 100644 index 0000000000..d29c86a85f --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_chargeback_exposure__totals_exposure_by_reason_and_outcome.sql @@ -0,0 +1,93 @@ +TEST (name "rpt_chargeback_exposure__totals_exposure_by_reason_and_outcome"); + +WITH __ref__int_chargeback_exposure AS ( + SELECT + 'CB1' AS chargeback_id, + NULL AS payment_id, + NULL AS order_id, + NULL AS customer_id, + NULL AS payment_method, + NULL AS reason_code, + 'fraud' AS reason_family, + NULL AS opened_at, + NULL AS response_due_on, + 'open' AS outcome, + 50.0::DECIMAL(18, 2) AS amount_usd, + NULL AS share_of_payment, + NULL AS evidence_kinds, + TRUE AS has_delivery_proof, + NULL AS days_open, + TRUE AS is_overdue, + 50.0::DECIMAL(18, 2) AS at_risk_usd, + 0 AS prior_chargebacks_for_customer + UNION ALL + SELECT + 'CB2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'fraud', + NULL, + NULL, + 'open', + 30.0::DECIMAL(18, 2), + NULL, + NULL, + FALSE, + NULL, + FALSE, + 30.0::DECIMAL(18, 2), + 2 + UNION ALL + SELECT + 'CB3', + NULL, + NULL, + NULL, + NULL, + NULL, + 'service', + NULL, + NULL, + 'won', + 20.0::DECIMAL(18, 2), + NULL, + NULL, + TRUE, + NULL, + FALSE, + 0.0::DECIMAL(18, 2), + 0 +), + +__expected__rpt_chargeback_exposure AS ( + SELECT + 'fraud:open' AS exposure_key, + 2 AS chargebacks, + 80.0::DECIMAL(18, 2) AS amount_usd, + 80.0::DECIMAL(18, 2) AS at_risk_usd, + 1 AS overdue_responses, + 1 AS with_delivery_proof, + 1 AS repeat_customer_chargebacks + UNION ALL + SELECT + 'service:won' AS exposure_key, + 1 AS chargebacks, + 20.0::DECIMAL(18, 2) AS amount_usd, + 0.0::DECIMAL(18, 2) AS at_risk_usd, + 0 AS overdue_responses, + 1 AS with_delivery_proof, + 0 AS repeat_customer_chargebacks +), + +__assert__exposure_key_is_unique_and_present AS ( + SELECT t.exposure_key + FROM __ref("rpt_chargeback_exposure") AS t + GROUP BY + t.exposure_key + HAVING + COUNT(*) > 1 + OR t.exposure_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_invoice_sequence_gaps__counts_missing_numbers_and_reconciliation_failures.sql b/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_invoice_sequence_gaps__counts_missing_numbers_and_reconciliation_failures.sql new file mode 100644 index 0000000000..7bb18d0ccd --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_invoice_sequence_gaps__counts_missing_numbers_and_reconciliation_failures.sql @@ -0,0 +1,147 @@ +TEST (name "rpt_invoice_sequence_gaps__counts_missing_numbers_and_reconciliation_failures"); + +WITH __ref__int_invoices_reconciled AS ( + SELECT + 'DS-A-1' AS invoice_id, + NULL AS order_id, + 'DS-A' AS invoice_series, + 1 AS invoice_sequence, + NULL AS sequence_gap, + NULL AS issued_at, + NULL AS issued_date_local, + NULL AS currency_code, + NULL AS net, + NULL AS tax, + NULL AS gross, + NULL AS embedded_tax_minor, + NULL AS loaded_tax_minor, + NULL AS order_tax_minor, + NULL AS tax_line_count, + NULL AS tax_codes, + TRUE AS header_balances, + TRUE AS tax_lines_agree, + TRUE AS matches_order_tax, + TRUE AS invoice_currency_matches_order + UNION ALL + SELECT + 'DS-A-2', + NULL, + 'DS-A', + 2, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + TRUE, + TRUE, + FALSE + UNION ALL + SELECT + 'DS-A-5', + NULL, + 'DS-A', + 5, + 2, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + TRUE, + TRUE, + TRUE + UNION ALL + SELECT + 'DS-A-9', + NULL, + 'DS-A', + 9, + 3, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + TRUE, + FALSE, + TRUE + UNION ALL + SELECT + 'DS-B-1', + NULL, + 'DS-B', + 1, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + TRUE, + TRUE, + TRUE +), + +__expected__rpt_invoice_sequence_gaps AS ( + SELECT + 'DS-A' AS invoice_series, + 4 AS invoices, + 1 AS first_sequence, + 9 AS last_sequence, + 2 AS gap_count, + 5 AS missing_numbers, + 1 AS unbalanced_headers, + 1 AS tax_disagreements, + 1 AS currency_mismatches + UNION ALL + SELECT + 'DS-B' AS invoice_series, + 1 AS invoices, + 1 AS first_sequence, + 1 AS last_sequence, + 0 AS gap_count, + 0 AS missing_numbers, + 0 AS unbalanced_headers, + 0 AS tax_disagreements, + 0 AS currency_mismatches +), + +__assert__invoice_series_is_unique_and_present AS ( + SELECT t.invoice_series + FROM __ref("rpt_invoice_sequence_gaps") AS t + GROUP BY + t.invoice_series + HAVING + COUNT(*) > 1 + OR t.invoice_series IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_payout_reconciliation_status__tracks_unreconciled_payout_value_by_month.sql b/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_payout_reconciliation_status__tracks_unreconciled_payout_value_by_month.sql new file mode 100644 index 0000000000..817ab6dd1d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_payout_reconciliation_status__tracks_unreconciled_payout_value_by_month.sql @@ -0,0 +1,115 @@ +TEST (name "rpt_payout_reconciliation_status__tracks_unreconciled_payout_value_by_month"); + +WITH __ref__int_payout_reconciliation AS ( + SELECT + 'PO1' AS payout_id, + 'PSP1' AS provider_code, + NULL AS currency_code, + '2026-03-02'::DATE AS paid_out_on, + 1000 AS payout_net_minor, + NULL AS lines_net_minor, + NULL AS charge_line_count, + NULL AS refund_line_count, + 0 AS orphan_line_count, + NULL AS embedded_line_count, + NULL AS loaded_line_count, + NULL AS matched_payment_minor, + NULL AS report_id, + NULL AS report_net_minor, + NULL AS lines_match_payout, + NULL AS report_matches_payout, + 'reconciled' AS reconciliation_status, + NULL AS days_since_previous_payout + UNION ALL + SELECT + 'PO2', + 'PSP1', + NULL, + '2026-03-15'::DATE, + 500, + NULL, + NULL, + NULL, + 1, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'awaiting_report', + NULL + UNION ALL + SELECT + 'PO3', + 'PSP1', + NULL, + '2026-03-28'::DATE, + 200, + NULL, + NULL, + NULL, + 2, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'line_mismatch', + NULL + UNION ALL + SELECT + 'PO4', + 'PSP1', + NULL, + '2026-04-01'::DATE, + 300, + NULL, + NULL, + NULL, + 0, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'report_mismatch', + NULL +), + +__expected__rpt_payout_reconciliation_status AS ( + SELECT + 'PSP1:2026-03' AS payout_status_key, + 3 AS payouts, + 1 AS reconciled, + 1 AS awaiting_report, + 1 AS mismatched, + 3 AS orphan_lines, + 700 AS unreconciled_net_minor, + 0.3333::DECIMAL(9, 4) AS reconciled_share + UNION ALL + SELECT + 'PSP1:2026-04' AS payout_status_key, + 1 AS payouts, + 0 AS reconciled, + 0 AS awaiting_report, + 1 AS mismatched, + 0 AS orphan_lines, + 300 AS unreconciled_net_minor, + 0.0::DECIMAL(9, 4) AS reconciled_share +), + +__assert__payout_status_key_is_unique_and_present AS ( + SELECT t.payout_status_key + FROM __ref("rpt_payout_reconciliation_status") AS t + GROUP BY + t.payout_status_key + HAVING + COUNT(*) > 1 + OR t.payout_status_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_revenue_recognition_monthly__pivots_revenue_streams_per_region_month.sql b/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_revenue_recognition_monthly__pivots_revenue_streams_per_region_month.sql new file mode 100644 index 0000000000..197f9c9fa4 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_revenue_recognition_monthly__pivots_revenue_streams_per_region_month.sql @@ -0,0 +1,86 @@ +TEST (name "rpt_revenue_recognition_monthly__pivots_revenue_streams_per_region_month"); + +WITH __ref__int_revenue_recognition_daily AS ( + SELECT + '2026-03-05:EU:product' AS recognition_key, + '2026-03-05'::DATE AS recognition_date, + 'EU' AS region_code, + 'product' AS revenue_stream, + 100.0::DECIMAL(18, 2) AS revenue_usd, + NULL AS document_count, + NULL AS month_to_date_usd, + NULL AS trailing_7d_usd, + NULL AS share_of_day + UNION ALL + SELECT + '2026-03-20:EU:product', + '2026-03-20'::DATE, + 'EU', + 'product', + 50.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + '2026-03-21:EU:refund', + '2026-03-21'::DATE, + 'EU', + 'refund', + -15.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + '2026-03-10:EU:subscription', + '2026-03-10'::DATE, + 'EU', + 'subscription', + 10.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + '2026-03-02:US:subscription', + '2026-03-02'::DATE, + 'US', + 'subscription', + 20.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL +), + +__expected__rpt_revenue_recognition_monthly AS ( + SELECT + '2026-03:EU'::TEXT AS recognition_month_key, + 150.0::DECIMAL(18, 2) AS product_usd, + -15.0::DECIMAL(18, 2) AS refund_usd, + 10.0::DECIMAL(18, 2) AS subscription_usd, + 145.0::DECIMAL(18, 2) AS net_revenue_usd, + 0.1::DECIMAL(9, 4) AS refund_ratio + UNION ALL + SELECT + '2026-03:US'::TEXT AS recognition_month_key, + 0.0::DECIMAL(18, 2) AS product_usd, + 0.0::DECIMAL(18, 2) AS refund_usd, + 20.0::DECIMAL(18, 2) AS subscription_usd, + 20.0::DECIMAL(18, 2) AS net_revenue_usd, + CAST(NULL AS DECIMAL(9, 4)) AS refund_ratio +), + +__assert__recognition_month_key_is_unique_and_present AS ( + SELECT t.recognition_month_key + FROM __ref("rpt_revenue_recognition_monthly") AS t + GROUP BY + t.recognition_month_key + HAVING + COUNT(*) > 1 + OR t.recognition_month_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_tax_variance_by_country__aggregates_tax_variance_per_country_quarter.sql b/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_tax_variance_by_country__aggregates_tax_variance_per_country_quarter.sql new file mode 100644 index 0000000000..0c2117bdd5 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/finance/test_rpt_tax_variance_by_country__aggregates_tax_variance_per_country_quarter.sql @@ -0,0 +1,117 @@ +TEST (name "rpt_tax_variance_by_country__aggregates_tax_variance_per_country_quarter"); + +WITH __ref__int_tax_applied AS ( + SELECT + 'T1' AS invoice_tax_line_key, + NULL AS invoice_id, + NULL AS order_id, + NULL AS tax_code, + NULL AS tax_category, + 'DE' AS ship_to_country_code, + '2026-03-10'::DATE AS issued_date, + NULL AS charged_rate_pct, + NULL AS published_rate_pct, + NULL AS taxable_minor, + 190 AS charged_tax_minor, + NULL AS expected_tax_minor, + 0 AS variance_minor, + TRUE AS has_published_rate, + FALSE AS is_rate_mismatch, + 1 AS candidate_rate_count + UNION ALL + SELECT + 'T2', + NULL, + NULL, + NULL, + NULL, + 'DE', + '2026-02-01'::DATE, + NULL, + NULL, + NULL, + 100, + NULL, + -5, + TRUE, + TRUE, + 2 + UNION ALL + SELECT + 'T3', + NULL, + NULL, + NULL, + NULL, + 'DE', + '2026-04-02'::DATE, + NULL, + NULL, + NULL, + 50, + NULL, + 10, + FALSE, + FALSE, + 0 + UNION ALL + SELECT + 'T4', + NULL, + NULL, + NULL, + NULL, + NULL, + '2026-03-01'::DATE, + NULL, + NULL, + NULL, + 0, + NULL, + -3, + FALSE, + FALSE, + 0 +), + +__expected__rpt_tax_variance_by_country AS ( + SELECT + 'DE:2026-01-01' AS tax_variance_key, + 2 AS tax_lines, + 0 AS lines_without_rate, + 1 AS rate_mismatches, + 290 AS charged_tax_minor, + -5 AS net_variance_minor, + 5 AS absolute_variance_minor, + 1 AS ambiguous_rate_lines + UNION ALL + SELECT + 'DE:2026-04-01' AS tax_variance_key, + 1 AS tax_lines, + 1 AS lines_without_rate, + 0 AS rate_mismatches, + 50 AS charged_tax_minor, + 10 AS net_variance_minor, + 10 AS absolute_variance_minor, + 0 AS ambiguous_rate_lines + UNION ALL + SELECT + '??:2026-01-01' AS tax_variance_key, + 1 AS tax_lines, + 1 AS lines_without_rate, + 0 AS rate_mismatches, + 0 AS charged_tax_minor, + -3 AS net_variance_minor, + 3 AS absolute_variance_minor, + 0 AS ambiguous_rate_lines +), + +__assert__tax_variance_key_is_unique_and_present AS ( + SELECT t.tax_variance_key + FROM __ref("rpt_tax_variance_by_country") AS t + GROUP BY + t.tax_variance_key + HAVING + COUNT(*) > 1 + OR t.tax_variance_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_fct_shipments__classifies_fulfilment_and_totals_packages_and_exceptions.sql b/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_fct_shipments__classifies_fulfilment_and_totals_packages_and_exceptions.sql new file mode 100644 index 0000000000..4cbb008b5c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_fct_shipments__classifies_fulfilment_and_totals_packages_and_exceptions.sql @@ -0,0 +1,159 @@ +TEST (name "fct_shipments__classifies_fulfilment_and_totals_packages_and_exceptions"); + +WITH __ref__stg_shipments AS ( + SELECT + 'S1' AS shipment_id, + 'O1' AS order_id, + 'POND' AS warehouse_code, + 'FLAPWING' AS carrier_code, + 'STANDARD' AS service_level, + NULL AS tracking_number, + NULL AS shipped_at, + NULL AS delivered_at, + 4 AS tracking_event_count, + NULL AS tracking_events, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'S2', + 'O2', + NULL, + 'WINGBEAT', + 'EXPRESS', + NULL, + NULL, + NULL, + 0, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'S3', + 'O3', + 'NEST', + 'FLAPWING', + 'Click & Collect', + NULL, + NULL, + NULL, + 1, + NULL, + NULL, + NULL +), + +__ref__int_fulfilment_sla AS ( + SELECT + 'S1' AS shipment_id, + NULL AS order_id, + NULL AS warehouse_code, + NULL AS carrier_code, + NULL AS placed_at_local, + NULL AS shipped_at_local, + NULL AS calendar_days_to_ship, + 1 AS business_days_to_ship, + NULL AS holidays_in_window, + NULL AS dispatch_business_minutes, + TRUE AS is_dispatch_on_time, + NULL AS delivered_at_utc, + 26.0::DECIMAL(10, 2) AS hours_to_deliver +), + +__ref__int_shipment_costs AS ( + SELECT + 'PK1' AS package_id, + 'S1' AS shipment_id, + NULL AS carrier_code, + NULL AS service_level, + 1.5::DECIMAL(10, 3) AS chargeable_weight_kg, + NULL AS weight_basis, + NULL AS band_max_kg, + 600 AS rate_minor, + NULL AS rate_currency_code, + NULL AS has_rate + UNION ALL + SELECT + 'PK2', + 'S1', + NULL, + NULL, + 2.4::DECIMAL(10, 3), + NULL, + NULL, + 900, + NULL, + NULL +), + +__ref__int_delivery_exceptions_classified AS ( + SELECT + 'X1' AS exception_id, + 'S1' AS shipment_id, + NULL AS carrier_code, + NULL AS exception_code, + NULL AS responsibility, + NULL AS occurred_at, + NULL AS hours_to_resolve, + NULL AS resolution_type, + 5.5::DECIMAL(18, 2) AS compensation_usd, + NULL AS exception_rank_in_shipment + UNION ALL + SELECT + 'X2', + 'S1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__fct_shipments AS ( + SELECT + 'S1' AS shipment_id, + 'home_delivery' AS fulfilment_method, + TRUE AS is_dispatch_on_time, + 26.0::NUMBER(10, 2) AS hours_to_deliver, + 2 AS package_count, + 3.9::DECIMAL(10, 3) AS chargeable_weight_kg, + 1500 AS rate_minor, + 2 AS exception_count, + 5.5::DECIMAL(18, 2) AS compensation_usd + UNION ALL + SELECT + 'S2' AS shipment_id, + 'partner' AS fulfilment_method, + CAST(NULL AS BOOLEAN) AS is_dispatch_on_time, + CAST(NULL AS NUMBER(10, 2)) AS hours_to_deliver, + 0 AS package_count, + CAST(NULL AS DECIMAL(10, 3)) AS chargeable_weight_kg, + CAST(NULL AS INT) AS rate_minor, + 0 AS exception_count, + 0.0::DECIMAL(18, 2) AS compensation_usd + UNION ALL + SELECT + 'S3' AS shipment_id, + 'click_and_collect' AS fulfilment_method, + CAST(NULL AS BOOLEAN) AS is_dispatch_on_time, + CAST(NULL AS NUMBER(10, 2)) AS hours_to_deliver, + 0 AS package_count, + CAST(NULL AS DECIMAL(10, 3)) AS chargeable_weight_kg, + CAST(NULL AS INT) AS rate_minor, + 0 AS exception_count, + 0.0::DECIMAL(18, 2) AS compensation_usd +), + +__assert__shipment_id_is_unique_and_present AS ( + SELECT t.shipment_id + FROM __ref("fct_shipments") AS t + GROUP BY + t.shipment_id + HAVING + COUNT(*) > 1 + OR t.shipment_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_carrier_performance_weekly__computes_delivery_percentiles_per_utc_week.sql b/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_carrier_performance_weekly__computes_delivery_percentiles_per_utc_week.sql new file mode 100644 index 0000000000..a0648a8c8c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_carrier_performance_weekly__computes_delivery_percentiles_per_utc_week.sql @@ -0,0 +1,137 @@ +TEST (name "rpt_carrier_performance_weekly__computes_delivery_percentiles_per_utc_week"); + +WITH __ref__fct_shipments AS ( + SELECT + 'S1' AS shipment_id, + NULL AS order_id, + NULL AS fulfilment_method, + NULL AS warehouse_code, + 'FLAPWING' AS carrier_code, + NULL AS service_level, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS shipped_at, + NULL AS delivered_at, + NULL AS business_days_to_ship, + TRUE AS is_dispatch_on_time, + 24.0::NUMBER(10, 2) AS hours_to_deliver, + NULL AS package_count, + NULL AS chargeable_weight_kg, + NULL AS rate_minor, + 0 AS exception_count, + 0.0::DECIMAL(18, 2) AS compensation_usd, + NULL AS tracking_event_count + UNION ALL + SELECT + 'S2', + NULL, + NULL, + NULL, + 'FLAPWING', + NULL, + '2026-03-11 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + FALSE, + 48.0::NUMBER(10, 2), + NULL, + NULL, + NULL, + 1, + 5.5::DECIMAL(18, 2), + NULL + UNION ALL + SELECT + 'S3', + NULL, + NULL, + NULL, + 'FLAPWING', + NULL, + '2026-03-15 23:30:00 -02:00'::TIMESTAMP_TZ, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + 0, + 0.0::DECIMAL(18, 2), + NULL + UNION ALL + SELECT + 'S4', + NULL, + NULL, + NULL, + 'FLAPWING', + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + 0, + 0.0::DECIMAL(18, 2), + NULL + UNION ALL + SELECT + 'S5', + NULL, + NULL, + NULL, + 'WINGBEAT', + NULL, + '2026-03-12 08:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + TRUE, + 10.0::NUMBER(10, 2), + NULL, + NULL, + NULL, + 0, + 0.0::DECIMAL(18, 2), + NULL +), + +__expected__rpt_carrier_performance_weekly AS ( + SELECT + 'FLAPWING:2026-03-09' AS carrier_week_key, + 2 AS shipments, + 0.5::DECIMAL(9, 4) AS on_time_dispatch_rate, + 36.0::DECIMAL(10, 2) AS p50_hours_to_deliver, + 45.6::DECIMAL(10, 2) AS p90_hours_to_deliver, + 0.5::DECIMAL(9, 4) AS exception_rate, + 5.5::DECIMAL(18, 2) AS compensation_usd + UNION ALL + SELECT + 'FLAPWING:2026-03-16' AS carrier_week_key, + 1 AS shipments, + 1.0::DECIMAL(9, 4) AS on_time_dispatch_rate, + CAST(NULL AS DECIMAL(10, 2)) AS p50_hours_to_deliver, + CAST(NULL AS DECIMAL(10, 2)) AS p90_hours_to_deliver, + 0.0::DECIMAL(9, 4) AS exception_rate, + 0.0::DECIMAL(18, 2) AS compensation_usd + UNION ALL + SELECT + 'WINGBEAT:2026-03-09' AS carrier_week_key, + 1 AS shipments, + 1.0::DECIMAL(9, 4) AS on_time_dispatch_rate, + 10.0::DECIMAL(10, 2) AS p50_hours_to_deliver, + 10.0::DECIMAL(10, 2) AS p90_hours_to_deliver, + 0.0::DECIMAL(9, 4) AS exception_rate, + 0.0::DECIMAL(18, 2) AS compensation_usd +), + +__assert__carrier_week_key_is_unique_and_present AS ( + SELECT t.carrier_week_key + FROM __ref("rpt_carrier_performance_weekly") AS t + GROUP BY + t.carrier_week_key + HAVING + COUNT(*) > 1 + OR t.carrier_week_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_delivery_exceptions_monthly__groups_exceptions_by_utc_month_and_responsibility.sql b/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_delivery_exceptions_monthly__groups_exceptions_by_utc_month_and_responsibility.sql new file mode 100644 index 0000000000..056080a775 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_delivery_exceptions_monthly__groups_exceptions_by_utc_month_and_responsibility.sql @@ -0,0 +1,79 @@ +TEST (name "rpt_delivery_exceptions_monthly__groups_exceptions_by_utc_month_and_responsibility"); + +WITH __ref__int_delivery_exceptions_classified AS ( + SELECT + 'X1' AS exception_id, + 'S1' AS shipment_id, + 'FLAPWING' AS carrier_code, + 'ADDR_INVALID' AS exception_code, + 'customer' AS responsibility, + '2026-03-31 23:30:00 -02:00'::TIMESTAMP_TZ AS occurred_at, + 2.0::DECIMAL(10, 2) AS hours_to_resolve, + NULL AS resolution_type, + 5.0::DECIMAL(18, 2) AS compensation_usd, + NULL AS exception_rank_in_shipment + UNION ALL + SELECT + 'X2', + 'S2', + 'FLAPWING', + 'ADDR_INVALID', + 'customer', + '2026-04-05 10:00:00 +00:00'::TIMESTAMP_TZ, + 4.0::DECIMAL(10, 2), + NULL, + NULL, + NULL + UNION ALL + SELECT + 'X3', + 'S2', + 'FLAPWING', + 'NOT_HOME', + 'customer', + '2026-04-06 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + 1.5::DECIMAL(18, 2), + NULL + UNION ALL + SELECT + 'X4', + 'S3', + 'FLAPWING', + 'LOST_PARCEL', + 'carrier', + '2026-03-02 10:00:00 +00:00'::TIMESTAMP_TZ, + 10.0::DECIMAL(10, 2), + NULL, + 3.0::DECIMAL(18, 2), + NULL +), + +__expected__rpt_delivery_exceptions_monthly AS ( + SELECT + '2026-04:FLAPWING:customer' AS exception_month_key, + 3 AS exceptions, + 2 AS shipments_affected, + 3.0::DECIMAL(10, 2) AS median_hours_to_resolve, + 6.5::DECIMAL(18, 2) AS compensation_usd, + 'ADDR_INVALID' AS top_exception_code + UNION ALL + SELECT + '2026-03:FLAPWING:carrier' AS exception_month_key, + 1 AS exceptions, + 1 AS shipments_affected, + 10.0::DECIMAL(10, 2) AS median_hours_to_resolve, + 3.0::DECIMAL(18, 2) AS compensation_usd, + 'LOST_PARCEL' AS top_exception_code +), + +__assert__exception_month_key_is_unique_and_present AS ( + SELECT t.exception_month_key + FROM __ref("rpt_delivery_exceptions_monthly") AS t + GROUP BY + t.exception_month_key + HAVING + COUNT(*) > 1 + OR t.exception_month_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_returns_by_reason__summarises_outcomes_by_month_and_reason.sql b/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_returns_by_reason__summarises_outcomes_by_month_and_reason.sql new file mode 100644 index 0000000000..ed0f6463ce --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_returns_by_reason__summarises_outcomes_by_month_and_reason.sql @@ -0,0 +1,126 @@ +TEST (name "rpt_returns_by_reason__summarises_outcomes_by_month_and_reason"); + +WITH __ref__int_return_reasons AS ( + SELECT + 'R1' AS return_id, + NULL AS order_id, + NULL AS sku, + 1 AS qty, + NULL AS reason_text, + 'damaged' AS return_reason, + NULL AS classified_by, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_TZ AS requested_at, + NULL AS category_level_1, + NULL AS ordered_qty, + NULL AS is_over_return, + NULL AS days_after_order + UNION ALL + SELECT + 'R2', + NULL, + NULL, + 2, + NULL, + 'damaged', + NULL, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'R3', + NULL, + NULL, + 1, + NULL, + 'size_fit', + NULL, + '2026-03-12 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'R4', + NULL, + NULL, + 1, + NULL, + 'damaged', + NULL, + '2026-03-31 23:30:00 -02:00'::TIMESTAMP_TZ, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_return_outcomes AS ( + SELECT + 'R1' AS return_id, + NULL AS order_id, + NULL AS sku, + NULL AS return_reason, + NULL AS grade, + TRUE AS is_resellable, + 10.0::DECIMAL(18, 2) AS refund_usd, + NULL AS days_to_receive, + 2 AS days_to_refund, + NULL AS outcome, + NULL AS defect_count + UNION ALL + SELECT + 'R2', + NULL, + NULL, + NULL, + NULL, + FALSE, + 20.0::DECIMAL(18, 2), + NULL, + 4, + NULL, + NULL +), + +__expected__rpt_returns_by_reason AS ( + SELECT + '2026-03:damaged' AS return_reason_key, + 2 AS returns, + 3 AS units, + 0.5::DECIMAL(9, 4) AS resellable_share, + 30.0::DECIMAL(18, 2) AS refund_usd, + 3.0::DECIMAL(10, 2) AS median_days_to_refund, + 0.6667::DECIMAL(9, 4) AS share_of_month + UNION ALL + SELECT + '2026-03:size_fit' AS return_reason_key, + 1 AS returns, + 1 AS units, + 0.0::DECIMAL(9, 4) AS resellable_share, + CAST(NULL AS DECIMAL(18, 2)) AS refund_usd, + CAST(NULL AS DECIMAL(10, 2)) AS median_days_to_refund, + 0.3333::DECIMAL(9, 4) AS share_of_month + UNION ALL + SELECT + '2026-04:damaged' AS return_reason_key, + 1 AS returns, + 1 AS units, + 0.0::DECIMAL(9, 4) AS resellable_share, + CAST(NULL AS DECIMAL(18, 2)) AS refund_usd, + CAST(NULL AS DECIMAL(10, 2)) AS median_days_to_refund, + 1.0::DECIMAL(9, 4) AS share_of_month +), + +__assert__return_reason_key_is_unique_and_present AS ( + SELECT t.return_reason_key + FROM __ref("rpt_returns_by_reason") AS t + GROUP BY + t.return_reason_key + HAVING + COUNT(*) > 1 + OR t.return_reason_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_warehouse_productivity_daily__rolls_pickers_up_to_warehouse_days.sql b/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_warehouse_productivity_daily__rolls_pickers_up_to_warehouse_days.sql new file mode 100644 index 0000000000..54153e8ef6 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/fulfilment/test_rpt_warehouse_productivity_daily__rolls_pickers_up_to_warehouse_days.sql @@ -0,0 +1,95 @@ +TEST (name "rpt_warehouse_productivity_daily__rolls_pickers_up_to_warehouse_days"); + +WITH __ref__int_pick_performance AS ( + SELECT + 'P1|POND|2026-04-06' AS picker_day_key, + 'P1' AS picker_id, + 'POND' AS warehouse_code, + '2026-04-06'::DATE AS work_date_local, + NULL AS task_count, + 60 AS units_picked, + 1.0::DECIMAL(10, 2) AS active_hours, + 60.0::DECIMAL(10, 2) AS units_per_hour, + NULL AS p90_task_seconds, + NULL AS rank_in_warehouse, + NULL AS first_task_hour_local, + NULL AS shift_coverage, + TRUE AS is_holiday + UNION ALL + SELECT + 'P2|POND|2026-04-06', + 'P2', + 'POND', + '2026-04-06'::DATE, + NULL, + 90, + 1.0::DECIMAL(10, 2), + 90.0::DECIMAL(10, 2), + NULL, + NULL, + NULL, + NULL, + TRUE + UNION ALL + SELECT + 'P3|NEST|2026-04-07', + 'P3', + 'NEST', + '2026-04-07'::DATE, + NULL, + 30, + 1.5::DECIMAL(10, 2), + 20.0::DECIMAL(10, 2), + NULL, + NULL, + NULL, + NULL, + FALSE +), + +__ref__int_delivery_slot_utilisation AS ( + SELECT + 'POND|2026-04-06' AS slot_day_key, + 'POND' AS warehouse_code, + '2026-04-06'::DATE AS slot_date_local, + NULL AS slot_count, + NULL AS total_capacity, + NULL AS total_booked, + 0.75::DECIMAL(9, 4) AS utilisation, + NULL AS full_slot_count, + NULL AS busiest_slot_start_local, + NULL AS is_weekend, + NULL AS region_code, + NULL AS is_holiday, + NULL AS utilisation_vs_weekday_avg +), + +__expected__rpt_warehouse_productivity_daily AS ( + SELECT + 'POND:2026-04-06' AS productivity_key, + 2 AS pickers, + 150 AS units_picked, + 75.0::DECIMAL(10, 2) AS units_per_picker_hour, + 'P2' AS top_picker_id, + 0.75::NUMBER(9, 4) AS slot_utilisation, + TRUE AS is_holiday + UNION ALL + SELECT + 'NEST:2026-04-07' AS productivity_key, + 1 AS pickers, + 30 AS units_picked, + 20.0::DECIMAL(10, 2) AS units_per_picker_hour, + 'P3' AS top_picker_id, + CAST(NULL AS NUMBER(9, 4)) AS slot_utilisation, + FALSE AS is_holiday +), + +__assert__productivity_key_is_unique_and_present AS ( + SELECT t.productivity_key + FROM __ref("rpt_warehouse_productivity_daily") AS t + GROUP BY + t.productivity_key + HAVING + COUNT(*) > 1 + OR t.productivity_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/growth/test_fct_campaign_spend__flags_rows_with_unknown_platform_metrics.sql b/benchmarks/duckshop/project/tests/unit/marts/growth/test_fct_campaign_spend__flags_rows_with_unknown_platform_metrics.sql new file mode 100644 index 0000000000..71a789826e --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/growth/test_fct_campaign_spend__flags_rows_with_unknown_platform_metrics.sql @@ -0,0 +1,87 @@ +TEST (name "fct_campaign_spend__flags_rows_with_unknown_platform_metrics"); + +WITH __ref__int_campaign_spend_usd AS ( + SELECT + 'SP1' AS spend_id, + 'CMP1' AS campaign_id, + NULL AS campaign_name, + 'social' AS channel, + NULL AS platform_account_id, + '2026-03-01'::DATE AS spend_date, + NULL AS currency_code, + NULL AS spend_amount, + 100.0::DECIMAL(18, 2) AS spend_usd, + NULL AS impressions, + NULL AS clicks, + NULL AS conversions, + NULL AS click_through_rate, + NULL AS cost_per_click_usd, + NULL AS metric_count, + PARSE_JSON('[]')::ARRAY AS unknown_metrics, + NULL AS is_in_flight + UNION ALL + SELECT + 'SP2', + 'CMP1', + NULL, + 'social', + NULL, + '2026-03-02'::DATE, + NULL, + NULL, + 80.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('["saves"]')::ARRAY, + NULL + UNION ALL + SELECT + 'SP3', + 'CMP2', + NULL, + 'email', + NULL, + '2026-03-02'::DATE, + NULL, + NULL, + 0.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__fct_campaign_spend AS ( + SELECT + 'SP1' AS spend_id, + 100.0::NUMBER(18, 2) AS spend_usd, + FALSE AS has_unknown_metrics + UNION ALL + SELECT + 'SP2' AS spend_id, + 80.0::NUMBER(18, 2) AS spend_usd, + TRUE AS has_unknown_metrics + UNION ALL + SELECT + 'SP3' AS spend_id, + 0.0::NUMBER(18, 2) AS spend_usd, + CAST(NULL AS BOOLEAN) AS has_unknown_metrics +), + +__assert__spend_id_is_unique_and_present AS ( + SELECT t.spend_id + FROM __ref("fct_campaign_spend") AS t + GROUP BY + t.spend_id + HAVING + COUNT(*) > 1 + OR t.spend_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_affiliate_payouts_monthly__separates_payable_and_reversed_commission.sql b/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_affiliate_payouts_monthly__separates_payable_and_reversed_commission.sql new file mode 100644 index 0000000000..612c370eed --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_affiliate_payouts_monthly__separates_payable_and_reversed_commission.sql @@ -0,0 +1,99 @@ +TEST (name "rpt_affiliate_payouts_monthly__separates_payable_and_reversed_commission"); + +WITH __ref__int_affiliate_commissions_usd AS ( + SELECT + 'AC1' AS conversion_id, + NULL AS affiliate_code, + 'network_one' AS affiliate_network, + NULL AS sub_channel, + NULL AS order_id, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_TZ AS converted_at, + NULL AS click_to_conversion_hours, + NULL AS commission, + 5.0::DECIMAL(18, 2) AS commission_usd, + 50.0::NUMBER(18, 2) AS order_amount_usd, + NULL AS commission_rate, + NULL AS claims_on_order, + TRUE AS is_last_click, + FALSE AS is_reversible, + 5.0::DECIMAL(18, 2) AS payable_commission_usd + UNION ALL + SELECT + 'AC2', + NULL, + 'network_one', + NULL, + NULL, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + 3.0::DECIMAL(18, 2), + 40.0::NUMBER(18, 2), + NULL, + NULL, + FALSE, + FALSE, + 0.0::DECIMAL(18, 2) + UNION ALL + SELECT + 'AC3', + NULL, + 'network_one', + NULL, + NULL, + '2026-03-20 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + 2.0::DECIMAL(18, 2), + 30.0::NUMBER(18, 2), + NULL, + NULL, + TRUE, + TRUE, + 0.0::DECIMAL(18, 2) + UNION ALL + SELECT + 'AC4', + NULL, + 'network_one', + NULL, + NULL, + '2026-03-31 23:30:00 -02:00'::TIMESTAMP_TZ, + NULL, + NULL, + 1.0::DECIMAL(18, 2), + 10.0::NUMBER(18, 2), + NULL, + NULL, + TRUE, + FALSE, + 1.0::DECIMAL(18, 2) +), + +__expected__rpt_affiliate_payouts_monthly AS ( + SELECT + 'network_one:2026-03' AS affiliate_month_key, + 3 AS conversions, + 2 AS last_click_conversions, + 5.0::DECIMAL(18, 2) AS payable_commission_usd, + 2.0::DECIMAL(18, 2) AS reversed_commission_usd, + 0.0625::DECIMAL(9, 6) AS effective_commission_rate + UNION ALL + SELECT + 'network_one:2026-04' AS affiliate_month_key, + 1 AS conversions, + 1 AS last_click_conversions, + 1.0::DECIMAL(18, 2) AS payable_commission_usd, + 0.0::DECIMAL(18, 2) AS reversed_commission_usd, + 0.1::DECIMAL(9, 6) AS effective_commission_rate +), + +__assert__affiliate_month_key_is_unique_and_present AS ( + SELECT t.affiliate_month_key + FROM __ref("rpt_affiliate_payouts_monthly") AS t + GROUP BY + t.affiliate_month_key + HAVING + COUNT(*) > 1 + OR t.affiliate_month_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_campaign_roas_monthly__matches_lowercase_utm_campaigns_to_channel_spend.sql b/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_campaign_roas_monthly__matches_lowercase_utm_campaigns_to_channel_spend.sql new file mode 100644 index 0000000000..77623c796d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_campaign_roas_monthly__matches_lowercase_utm_campaigns_to_channel_spend.sql @@ -0,0 +1,259 @@ +TEST (name "rpt_campaign_roas_monthly__matches_lowercase_utm_campaigns_to_channel_spend"); + +WITH __ref__fct_campaign_spend AS ( + SELECT + 'SP1' AS spend_id, + 'CMP1' AS campaign_id, + 'social' AS channel, + '2026-03-01'::DATE AS spend_date, + NULL AS currency_code, + 100.0::NUMBER(18, 2) AS spend_usd, + NULL AS impressions, + 50 AS clicks, + NULL AS conversions, + NULL AS cost_per_click_usd, + NULL AS is_in_flight, + NULL AS has_unknown_metrics + UNION ALL + SELECT + 'SP2', + 'CMP1', + 'social', + '2026-03-20'::DATE, + NULL, + 50.0::NUMBER(18, 2), + NULL, + 10, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'SP3', + 'CMP2', + 'email', + '2026-03-05'::DATE, + NULL, + 0.0::NUMBER(18, 2), + NULL, + 5, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'SP4', + 'CMP1', + 'social', + '2026-04-01'::DATE, + NULL, + 20.0::NUMBER(18, 2), + NULL, + 1, + NULL, + NULL, + NULL, + NULL +), + +__ref__fct_orders AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS region_code, + NULL AS placed_at, + '2026-03-10'::DATE AS placed_date_local, + NULL AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS currency_code, + NULL AS amount, + 300.0::NUMBER(18, 2) AS amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + 'cmp1' AS utm_campaign, + NULL AS basket_size, + NULL AS order_number, + NULL AS is_first_order, + NULL AS first_paid_at, + NULL AS first_delivered_at, + NULL AS hours_paid_to_shipped, + NULL AS captured_minor, + NULL AS refunded_minor, + NULL AS net_collected_minor, + NULL AS is_fully_refunded + UNION ALL + SELECT + 'O2', + NULL, + NULL, + NULL, + '2026-04-02'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + 10.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'cmp1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + NULL, + NULL, + NULL, + '2026-03-11'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + 99.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'CMP1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O4', + NULL, + NULL, + NULL, + '2026-03-12'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + 40.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'cmp2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__dim_campaigns AS ( + SELECT + 'CMP1' AS campaign_id, + 'Spring Splash' AS campaign_name, + 'social' AS channel, + NULL AS objective, + NULL AS starts_on, + NULL AS ends_on, + NULL AS flight_days, + NULL AS campaign_state, + NULL AS target_age_range, + NULL AS target_regions, + NULL AS budget_usd, + NULL AS spend_usd, + NULL AS return_on_ad_spend, + NULL AS source_loaded_at + UNION ALL + SELECT + 'CMP2', + 'Pond Week', + 'email', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__rpt_campaign_roas_monthly AS ( + SELECT + '2026-03:social' AS roas_key, + 150.0::DECIMAL(18, 2) AS spend_usd, + 60 AS clicks, + 1 AS attributed_orders, + 300.0::DECIMAL(18, 2) AS attributed_revenue_usd, + 2.0::DECIMAL(12, 4) AS return_on_ad_spend + UNION ALL + SELECT + '2026-03:email' AS roas_key, + 0.0::DECIMAL(18, 2) AS spend_usd, + 5 AS clicks, + 1 AS attributed_orders, + 40.0::DECIMAL(18, 2) AS attributed_revenue_usd, + CAST(NULL AS DECIMAL(12, 4)) AS return_on_ad_spend + UNION ALL + SELECT + '2026-04:social' AS roas_key, + 20.0::DECIMAL(18, 2) AS spend_usd, + 1 AS clicks, + 1 AS attributed_orders, + 10.0::DECIMAL(18, 2) AS attributed_revenue_usd, + 0.5::DECIMAL(12, 4) AS return_on_ad_spend +), + +__assert__roas_key_is_unique_and_present AS ( + SELECT t.roas_key + FROM __ref("rpt_campaign_roas_monthly") AS t + GROUP BY + t.roas_key + HAVING + COUNT(*) > 1 + OR t.roas_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_influencer_leaderboard__ranks_creators_within_their_main_platform.sql b/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_influencer_leaderboard__ranks_creators_within_their_main_platform.sql new file mode 100644 index 0000000000..2cd65abedf --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_influencer_leaderboard__ranks_creators_within_their_main_platform.sql @@ -0,0 +1,133 @@ +TEST (name "rpt_influencer_leaderboard__ranks_creators_within_their_main_platform"); + +WITH __ref__int_influencer_promo_orders AS ( + SELECT + 'IP1' AS post_id, + '@wren_waddles' AS creator_handle, + 'short_video' AS platform, + NULL AS posted_at, + NULL AS promo_code, + 1000 AS impressions, + 100 AS engagements, + NULL AS saves, + NULL AS has_tracking_link, + 2 AS credited_orders, + NULL AS credited_customers, + 50.0::DECIMAL(18, 2) AS credited_revenue_usd, + 5.0::DECIMAL(18, 2) AS discount_given_usd, + NULL AS first_order_hours_after_post, + NULL AS orders_per_thousand_impressions + UNION ALL + SELECT + 'IP2', + '@wren_waddles', + 'short_video', + NULL, + NULL, + 2000, + 200, + NULL, + NULL, + 3, + NULL, + 70.0::DECIMAL(18, 2), + 7.0::DECIMAL(18, 2), + NULL, + NULL + UNION ALL + SELECT + 'IP3', + '@wren_waddles', + 'photo_feed', + NULL, + NULL, + 500, + 50, + NULL, + NULL, + 0, + NULL, + 0.0::DECIMAL(18, 2), + 0.0::DECIMAL(18, 2), + NULL, + NULL + UNION ALL + SELECT + 'IP4', + '@reedbed_rae', + 'short_video', + NULL, + NULL, + 800, + 80, + NULL, + NULL, + 4, + NULL, + 200.0::DECIMAL(18, 2), + 20.0::DECIMAL(18, 2), + NULL, + NULL + UNION ALL + SELECT + 'IP5', + '@teal_tom', + 'photo_feed', + NULL, + NULL, + 300, + 30, + NULL, + NULL, + 1, + NULL, + 25.0::DECIMAL(18, 2), + 2.5::DECIMAL(18, 2), + NULL, + NULL +), + +__expected__rpt_influencer_leaderboard AS ( + SELECT + '@wren_waddles' AS creator_handle, + 'short_video' AS primary_platform, + 3 AS posts, + 3500 AS impressions, + 350 AS engagements, + 5 AS credited_orders, + 120.0::DECIMAL(18, 2) AS credited_revenue_usd, + 12.0::DECIMAL(18, 2) AS discount_given_usd, + 2 AS revenue_rank + UNION ALL + SELECT + '@reedbed_rae' AS creator_handle, + 'short_video' AS primary_platform, + 1 AS posts, + 800 AS impressions, + 80 AS engagements, + 4 AS credited_orders, + 200.0::DECIMAL(18, 2) AS credited_revenue_usd, + 20.0::DECIMAL(18, 2) AS discount_given_usd, + 1 AS revenue_rank + UNION ALL + SELECT + '@teal_tom' AS creator_handle, + 'photo_feed' AS primary_platform, + 1 AS posts, + 300 AS impressions, + 30 AS engagements, + 1 AS credited_orders, + 25.0::DECIMAL(18, 2) AS credited_revenue_usd, + 2.5::DECIMAL(18, 2) AS discount_given_usd, + 1 AS revenue_rank +), + +__assert__creator_handle_is_unique_and_present AS ( + SELECT t.creator_handle + FROM __ref("rpt_influencer_leaderboard") AS t + GROUP BY + t.creator_handle + HAVING + COUNT(*) > 1 + OR t.creator_handle IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_newsletter_growth_monthly__nets_global_unsubscribes_against_signups.sql b/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_newsletter_growth_monthly__nets_global_unsubscribes_against_signups.sql new file mode 100644 index 0000000000..ef65e9f23e --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_newsletter_growth_monthly__nets_global_unsubscribes_against_signups.sql @@ -0,0 +1,99 @@ +TEST (name "rpt_newsletter_growth_monthly__nets_global_unsubscribes_against_signups"); + +WITH __ref__int_newsletter_status AS ( + SELECT + 'a@example.com' AS email, + NULL AS email_domain, + NULL AS is_role_address, + NULL AS customer_id, + FALSE AS is_prospect, + NULL AS list_status_by_list, + NULL AS subscribed_lists, + NULL AS subscribed_list_count, + '2026-01-10 10:00:00 +00:00'::TIMESTAMPTZ AS first_subscribed_at, + NULL AS last_unsubscribed_at, + FALSE AS is_globally_unsubscribed, + NULL AS has_double_opt_in, + NULL AS first_signup_form + UNION ALL + SELECT + 'b@example.com', + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + '2026-01-20 10:00:00 +00:00'::TIMESTAMPTZ, + '2026-02-05 10:00:00 +00:00'::TIMESTAMPTZ, + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'c@example.com', + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + '2026-02-01 10:00:00 +00:00'::TIMESTAMPTZ, + '2026-02-10 10:00:00 +00:00'::TIMESTAMPTZ, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'd@example.com', + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL, + '2026-03-01 10:00:00 +00:00'::TIMESTAMPTZ, + TRUE, + NULL, + NULL +), + +__expected__rpt_newsletter_growth_monthly AS ( + SELECT + '2026-01-01'::DATE AS activity_month, + 2 AS new_subscribers, + 1 AS new_prospects, + 0 AS unsubscribes, + 2 AS net_growth, + 2 AS running_audience + UNION ALL + SELECT + '2026-02-01'::DATE AS activity_month, + 1 AS new_subscribers, + 1 AS new_prospects, + 1 AS unsubscribes, + 0 AS net_growth, + 2 AS running_audience + UNION ALL + SELECT + '2026-03-01'::DATE AS activity_month, + 0 AS new_subscribers, + 0 AS new_prospects, + 1 AS unsubscribes, + -1 AS net_growth, + 1 AS running_audience +), + +__assert__activity_month_is_unique_and_present AS ( + SELECT t.activity_month + FROM __ref("rpt_newsletter_growth_monthly") AS t + GROUP BY + t.activity_month + HAVING + COUNT(*) > 1 + OR t.activity_month IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_survey_nps_monthly__scores_nps_from_responses_with_a_rating.sql b/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_survey_nps_monthly__scores_nps_from_responses_with_a_rating.sql new file mode 100644 index 0000000000..b0b1356afd --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/growth/test_rpt_survey_nps_monthly__scores_nps_from_responses_with_a_rating.sql @@ -0,0 +1,129 @@ +TEST (name "rpt_survey_nps_monthly__scores_nps_from_responses_with_a_rating"); + +WITH __ref__int_survey_answers_flat AS ( + SELECT + 'R1:nps' AS answer_key, + 'R1' AS response_id, + 'POST_PURCHASE' AS survey_code, + NULL AS customer_id, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_TZ AS submitted_at, + 'nps' AS question_key, + 'number' AS answer_kind, + NULL AS answer_number, + NULL AS answer_flag, + NULL AS answer_text, + NULL AS answer_choices, + NULL AS is_answered, + 'promoter' AS nps_group, + NULL AS question_position + UNION ALL + SELECT + 'R1:comment', + 'R1', + 'POST_PURCHASE', + NULL, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_TZ, + 'comment', + 'text', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'R2:nps', + 'R2', + 'POST_PURCHASE', + NULL, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + 'nps', + 'number', + NULL, + NULL, + NULL, + NULL, + NULL, + 'detractor', + NULL + UNION ALL + SELECT + 'R3:comment', + 'R3', + 'POST_PURCHASE', + NULL, + '2026-03-12 10:00:00 +00:00'::TIMESTAMP_TZ, + 'comment', + 'text', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'R5:nps', + 'R5', + 'POST_PURCHASE', + NULL, + '2026-03-15 10:00:00 +00:00'::TIMESTAMP_TZ, + 'nps', + 'number', + NULL, + NULL, + NULL, + NULL, + NULL, + 'promoter', + NULL + UNION ALL + SELECT + 'R4:nps', + 'R4', + 'POST_PURCHASE', + NULL, + '2026-03-31 23:30:00 -02:00'::TIMESTAMP_TZ, + 'nps', + 'number', + NULL, + NULL, + NULL, + NULL, + NULL, + 'passive', + NULL +), + +__expected__rpt_survey_nps_monthly AS ( + SELECT + 'POST_PURCHASE:2026-03' AS nps_key, + 4 AS responses, + 2 AS promoters, + 0 AS passives, + 1 AS detractors, + 33.33::DECIMAL(6, 2) AS net_promoter_score, + 2 AS free_text_answers + UNION ALL + SELECT + 'POST_PURCHASE:2026-04' AS nps_key, + 1 AS responses, + 0 AS promoters, + 1 AS passives, + 0 AS detractors, + 0.0::DECIMAL(6, 2) AS net_promoter_score, + 0 AS free_text_answers +), + +__assert__nps_key_is_unique_and_present AS ( + SELECT t.nps_key + FROM __ref("rpt_survey_nps_monthly") AS t + GROUP BY + t.nps_key + HAVING + COUNT(*) > 1 + OR t.nps_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/inventory/test_fct_inventory_daily__builds_only_days_inside_the_cursor_window.sql b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_fct_inventory_daily__builds_only_days_inside_the_cursor_window.sql new file mode 100644 index 0000000000..82e1cf1e3c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_fct_inventory_daily__builds_only_days_inside_the_cursor_window.sql @@ -0,0 +1,59 @@ +TEST (name "fct_inventory_daily__builds_only_days_inside_the_cursor_window", cursor_start "2026-03-02", cursor_end "2026-03-03"); + +WITH __ref__int_stock_on_hand_daily AS ( + SELECT + 'DUCK-1|POND|2026-03-01' AS stock_day_key, + 'DUCK-1' AS sku, + 'POND' AS warehouse_code, + '2026-03-01'::DATE AS calendar_date, + NULL AS month_start, + 10 AS on_hand_qty, + NULL AS last_counted_qty, + NULL AS last_counted_date, + 5 AS days_since_count, + FALSE AS is_stocked_out + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-02', + 'DUCK-1', + 'POND', + '2026-03-02'::DATE, + NULL, + -2, + NULL, + NULL, + NULL, + TRUE + UNION ALL + SELECT + 'DUCK-2|POND|2026-03-01', + 'DUCK-2', + 'POND', + '2026-03-01'::DATE, + NULL, + 4, + NULL, + NULL, + 100, + FALSE +), + +__ref__int_supplier_cost_current AS ( + SELECT + 'DUCK-1' AS sku, + 'SUP1' AS supplier_code, + NULL AS supplier_name, + NULL AS currency_code, + NULL AS unit_cost_minor, + 2.5::DECIMAL(18, 4) AS unit_cost_usd, + NULL AS effective_from, + NULL AS supplier_count, + NULL AS payment_terms_days, + NULL AS cost_spread_pct +), + +__expected__fct_inventory_daily AS ( + SELECT + 'DUCK-1|POND|2026-03-02' AS stock_day_key, + -2 AS on_hand_qty +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/inventory/test_fct_inventory_daily__values_non_negative_stock_and_flags_overdue_counts.sql b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_fct_inventory_daily__values_non_negative_stock_and_flags_overdue_counts.sql new file mode 100644 index 0000000000..1cda694b8e --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_fct_inventory_daily__values_non_negative_stock_and_flags_overdue_counts.sql @@ -0,0 +1,73 @@ +TEST (name "fct_inventory_daily__values_non_negative_stock_and_flags_overdue_counts"); + +WITH __ref__int_stock_on_hand_daily AS ( + SELECT + 'DUCK-1|POND|2026-03-01' AS stock_day_key, + 'DUCK-1' AS sku, + 'POND' AS warehouse_code, + '2026-03-01'::DATE AS calendar_date, + NULL AS month_start, + 10 AS on_hand_qty, + NULL AS last_counted_qty, + NULL AS last_counted_date, + 5 AS days_since_count, + FALSE AS is_stocked_out + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-02', + 'DUCK-1', + 'POND', + '2026-03-02'::DATE, + NULL, + -2, + NULL, + NULL, + NULL, + TRUE + UNION ALL + SELECT + 'DUCK-2|POND|2026-03-01', + 'DUCK-2', + 'POND', + '2026-03-01'::DATE, + NULL, + 4, + NULL, + NULL, + 100, + FALSE +), + +__ref__int_supplier_cost_current AS ( + SELECT + 'DUCK-1' AS sku, + 'SUP1' AS supplier_code, + NULL AS supplier_name, + NULL AS currency_code, + NULL AS unit_cost_minor, + 2.5::DECIMAL(18, 4) AS unit_cost_usd, + NULL AS effective_from, + NULL AS supplier_count, + NULL AS payment_terms_days, + NULL AS cost_spread_pct +), + +__expected__fct_inventory_daily AS ( + SELECT + 'DUCK-1|POND|2026-03-01' AS stock_day_key, + 2.5::NUMBER(18, 4) AS unit_cost_usd, + 25.0::DECIMAL(18, 2) AS stock_value_usd, + FALSE AS is_count_overdue + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-02' AS stock_day_key, + 2.5::NUMBER(18, 4) AS unit_cost_usd, + 0.0::DECIMAL(18, 2) AS stock_value_usd, + TRUE AS is_count_overdue + UNION ALL + SELECT + 'DUCK-2|POND|2026-03-01' AS stock_day_key, + CAST(NULL AS NUMBER(18, 4)) AS unit_cost_usd, + CAST(NULL AS DECIMAL(18, 2)) AS stock_value_usd, + TRUE AS is_count_overdue +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_inventory_valuation_monthly__keeps_month_ends_and_compares_with_prior_month.sql b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_inventory_valuation_monthly__keeps_month_ends_and_compares_with_prior_month.sql new file mode 100644 index 0000000000..9f32355249 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_inventory_valuation_monthly__keeps_month_ends_and_compares_with_prior_month.sql @@ -0,0 +1,94 @@ +TEST (name "rpt_inventory_valuation_monthly__keeps_month_ends_and_compares_with_prior_month"); + +WITH __ref__int_inventory_valuation_daily AS ( + SELECT + 'POND|2026-01-31' AS valuation_key, + 'POND' AS warehouse_code, + '2026-01-31'::DATE AS calendar_date, + NULL AS sku_count, + 100 AS units_on_hand, + 1000.0::DECIMAL(18, 2) AS stock_value_usd, + 0 AS unvalued_units, + 0 AS stocked_out_skus, + NULL AS value_change_usd + UNION ALL + SELECT + 'POND|2026-02-15', + 'POND', + '2026-02-15'::DATE, + NULL, + 400, + 5000.0::DECIMAL(18, 2), + 0, + 0, + NULL + UNION ALL + SELECT + 'POND|2026-02-28', + 'POND', + '2026-02-28'::DATE, + NULL, + 110, + 1200.0::DECIMAL(18, 2), + 0, + 0, + NULL + UNION ALL + SELECT + 'POND|2026-03-31', + 'POND', + '2026-03-31'::DATE, + NULL, + 80, + 900.0::DECIMAL(18, 2), + 0, + 0, + NULL + UNION ALL + SELECT + 'NEST|2026-03-31', + 'NEST', + '2026-03-31'::DATE, + NULL, + 40, + 500.0::DECIMAL(18, 2), + 0, + 0, + NULL +), + +__expected__rpt_inventory_valuation_monthly AS ( + SELECT + 'POND:2026-01-31' AS valuation_month_key, + CAST(NULL AS DECIMAL(18, 2)) AS previous_month_value_usd, + CAST(NULL AS DECIMAL(9, 2)) AS change_pct, + 100 AS units_on_hand + UNION ALL + SELECT + 'POND:2026-02-28' AS valuation_month_key, + 1000.0::DECIMAL(18, 2) AS previous_month_value_usd, + 20.0::DECIMAL(9, 2) AS change_pct, + 110 AS units_on_hand + UNION ALL + SELECT + 'POND:2026-03-31' AS valuation_month_key, + 1200.0::DECIMAL(18, 2) AS previous_month_value_usd, + -25.0::DECIMAL(9, 2) AS change_pct, + 80 AS units_on_hand + UNION ALL + SELECT + 'NEST:2026-03-31' AS valuation_month_key, + CAST(NULL AS DECIMAL(18, 2)) AS previous_month_value_usd, + CAST(NULL AS DECIMAL(9, 2)) AS change_pct, + 40 AS units_on_hand +), + +__assert__valuation_month_key_is_unique_and_present AS ( + SELECT t.valuation_month_key + FROM __ref("rpt_inventory_valuation_monthly") AS t + GROUP BY + t.valuation_month_key + HAVING + COUNT(*) > 1 + OR t.valuation_month_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_stock_count_accuracy__measures_monthly_count_accuracy.sql b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_stock_count_accuracy__measures_monthly_count_accuracy.sql new file mode 100644 index 0000000000..63b928f9b8 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_stock_count_accuracy__measures_monthly_count_accuracy.sql @@ -0,0 +1,87 @@ +TEST (name "rpt_stock_count_accuracy__measures_monthly_count_accuracy"); + +WITH __ref__int_stock_count_variances AS ( + SELECT + 'C1' AS count_id, + 'DUCK-1' AS sku, + 'POND' AS warehouse_code, + '2026-03-03 12:00:00'::TIMESTAMP_NTZ AS counted_at_local, + NULL AS counted_qty, + NULL AS expected_qty, + 0 AS variance_qty, + NULL AS variance_pct, + FALSE AS is_out_of_tolerance, + TRUE AS is_blind_count, + 1 AS recount_sequence + UNION ALL + SELECT + 'C2', + 'DUCK-1', + 'POND', + '2026-03-06 08:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + -10, + NULL, + TRUE, + FALSE, + 1 + UNION ALL + SELECT + 'C3', + 'DUCK-1', + 'POND', + '2026-03-06 15:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + 0, + NULL, + FALSE, + FALSE, + 2 + UNION ALL + SELECT + 'C4', + 'DUCK-1', + 'POND', + '2026-04-01 09:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + 5, + NULL, + TRUE, + TRUE, + 1 +), + +__expected__rpt_stock_count_accuracy AS ( + SELECT + 'POND:2026-03' AS count_month_key, + 3 AS counts, + 1 AS blind_counts, + 2 AS accurate_counts, + 0.6667::DECIMAL(9, 4) AS accuracy_rate, + 1 AS out_of_tolerance, + -10 AS net_variance_qty, + 0.3333::DECIMAL(9, 4) AS recount_rate + UNION ALL + SELECT + 'POND:2026-04' AS count_month_key, + 1 AS counts, + 1 AS blind_counts, + 0 AS accurate_counts, + 0.0::DECIMAL(9, 4) AS accuracy_rate, + 1 AS out_of_tolerance, + 5 AS net_variance_qty, + 0.0::DECIMAL(9, 4) AS recount_rate +), + +__assert__count_month_key_is_unique_and_present AS ( + SELECT t.count_month_key + FROM __ref("rpt_stock_count_accuracy") AS t + GROUP BY + t.count_month_key + HAVING + COUNT(*) > 1 + OR t.count_month_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_stock_outs_weekly__finds_longest_consecutive_stock_out.sql b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_stock_outs_weekly__finds_longest_consecutive_stock_out.sql new file mode 100644 index 0000000000..f2a93cb31f --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_stock_outs_weekly__finds_longest_consecutive_stock_out.sql @@ -0,0 +1,115 @@ +TEST (name "rpt_stock_outs_weekly__finds_longest_consecutive_stock_out"); + +WITH __ref__fct_inventory_daily AS ( + SELECT + 'DUCK-1|POND|2026-03-09' AS stock_day_key, + 'DUCK-1' AS sku, + 'POND' AS warehouse_code, + '2026-03-09'::DATE AS calendar_date, + 5 AS on_hand_qty, + NULL AS unit_cost_usd, + NULL AS stock_value_usd, + FALSE AS is_stocked_out, + NULL AS days_since_count, + FALSE AS is_count_overdue + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-10', + 'DUCK-1', + 'POND', + '2026-03-10'::DATE, + 0, + NULL, + NULL, + TRUE, + NULL, + FALSE + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-11', + 'DUCK-1', + 'POND', + '2026-03-11'::DATE, + 0, + NULL, + NULL, + TRUE, + NULL, + FALSE + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-12', + 'DUCK-1', + 'POND', + '2026-03-12'::DATE, + 5, + NULL, + NULL, + FALSE, + NULL, + FALSE + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-13', + 'DUCK-1', + 'POND', + '2026-03-13'::DATE, + 0, + NULL, + NULL, + TRUE, + NULL, + FALSE + UNION ALL + SELECT + 'DUCK-2|POND|2026-03-10', + 'DUCK-2', + 'POND', + '2026-03-10'::DATE, + 5, + NULL, + NULL, + FALSE, + NULL, + FALSE + UNION ALL + SELECT + 'DUCK-1|POND|2026-03-16', + 'DUCK-1', + 'POND', + '2026-03-16'::DATE, + 0, + NULL, + NULL, + TRUE, + NULL, + FALSE +), + +__expected__rpt_stock_outs_weekly AS ( + SELECT + 'POND:2026-03-09' AS stock_out_key, + 6 AS sku_days, + 3 AS stocked_out_sku_days, + 0.5::DECIMAL(9, 4) AS stock_out_rate, + 1 AS skus_affected, + 2 AS longest_run_days + UNION ALL + SELECT + 'POND:2026-03-16' AS stock_out_key, + 1 AS sku_days, + 1 AS stocked_out_sku_days, + 1.0::DECIMAL(9, 4) AS stock_out_rate, + 1 AS skus_affected, + 1 AS longest_run_days +), + +__assert__stock_out_key_is_unique_and_present AS ( + SELECT t.stock_out_key + FROM __ref("rpt_stock_outs_weekly") AS t + GROUP BY + t.stock_out_key + HAVING + COUNT(*) > 1 + OR t.stock_out_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_supplier_scorecard__scores_suppliers_by_receipt_quarter.sql b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_supplier_scorecard__scores_suppliers_by_receipt_quarter.sql new file mode 100644 index 0000000000..e673d9c6c9 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/inventory/test_rpt_supplier_scorecard__scores_suppliers_by_receipt_quarter.sql @@ -0,0 +1,108 @@ +TEST (name "rpt_supplier_scorecard__scores_suppliers_by_receipt_quarter"); + +WITH __ref__int_purchase_order_lines_received AS ( + SELECT + 'PO1#1' AS po_line_key, + 'PO1' AS po_id, + 'SUP1' AS supplier_code, + NULL AS warehouse_code, + NULL AS sku, + 10 AS ordered_qty, + 12 AS received_qty, + 1 AS damaged_qty, + NULL AS outstanding_qty, + NULL AS fill_rate, + '2026-03-12 10:00:00'::TIMESTAMP_NTZ AS first_received_at_local, + 1 AS days_late, + NULL AS receipt_count, + ARRAY_CONSTRUCT('SHORT')::ARRAY AS discrepancy_codes + UNION ALL + SELECT + 'PO1#2', + 'PO1', + 'SUP1', + NULL, + NULL, + 5, + 3, + 0, + NULL, + NULL, + '2026-02-01 10:00:00'::TIMESTAMP_NTZ, + 0, + NULL, + ARRAY_CONSTRUCT('SHORT', 'DAMAGED')::ARRAY + UNION ALL + SELECT + 'PO1#3', + 'PO1', + 'SUP1', + NULL, + NULL, + 5, + 0, + 0, + NULL, + NULL, + NULL, + NULL, + NULL, + ARRAY_CONSTRUCT()::ARRAY + UNION ALL + SELECT + 'PO2#1', + 'PO1', + 'SUP1', + NULL, + NULL, + 10, + 10, + 0, + NULL, + NULL, + '2026-04-02 10:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + ARRAY_CONSTRUCT()::ARRAY +), + +__expected__rpt_supplier_scorecard AS ( + SELECT + 'SUP1:2026-01-01' AS scorecard_key, + 2 AS po_lines, + 15 AS ordered_qty, + 15 AS received_qty, + 1.0::DECIMAL(9, 4) AS fill_rate, + 0.5::DECIMAL(9, 4) AS late_line_share, + 0.0667::DECIMAL(9, 4) AS damage_rate + UNION ALL + SELECT + 'SUP1:2026-04-01' AS scorecard_key, + 1 AS po_lines, + 10 AS ordered_qty, + 10 AS received_qty, + 1.0::DECIMAL(9, 4) AS fill_rate, + 0.0::DECIMAL(9, 4) AS late_line_share, + 0.0::DECIMAL(9, 4) AS damage_rate +), + +__assert__discrepancy_codes_are_merged_without_duplicates AS ( + SELECT s.scorecard_key + FROM __ref("rpt_supplier_scorecard") AS s + WHERE + ARRAY_SIZE(s.discrepancy_codes) <> IFF(s.scorecard_key = 'SUP1:2026-01-01', 2, 0) + OR ( + s.scorecard_key = 'SUP1:2026-01-01' + AND NOT ARRAY_CONTAINS('DAMAGED'::VARIANT, s.discrepancy_codes) + ) +), + +__assert__scorecard_key_is_unique_and_present AS ( + SELECT t.scorecard_key + FROM __ref("rpt_supplier_scorecard") AS t + GROUP BY + t.scorecard_key + HAVING + COUNT(*) > 1 + OR t.scorecard_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_order_lines__defaults_promotions_when_no_discounts_exist.sql b/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_order_lines__defaults_promotions_when_no_discounts_exist.sql new file mode 100644 index 0000000000..72fbd90b89 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_order_lines__defaults_promotions_when_no_discounts_exist.sql @@ -0,0 +1,108 @@ +TEST (name "fct_order_lines__defaults_promotions_when_no_discounts_exist"); + +WITH __ref__int_order_lines_priced AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + 'DUCK-MAL-01' AS sku, + 'DUCK' AS sku_family, + NULL AS product_id, + 2 AS quantity, + NULL AS unit_amount_minor, + NULL AS list_amount_minor, + NULL AS markdown_pct, + NULL AS gross_amount_minor, + 2598 AS net_amount_minor, + NULL AS discount_amount_minor, + NULL AS tax_amount_minor, + NULL AS line_weight_kg, + NULL AS has_list_price + UNION ALL + SELECT + 'L2', + 'O1', + 'DECOY-TEAL-1', + 'DECOY', + NULL, + 1, + NULL, + NULL, + NULL, + NULL, + 4500, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_order_lines_discounted AS ( + SELECT + CAST(NULL AS VARCHAR) AS order_line_id, + CAST(NULL AS VARCHAR) AS order_id, + CAST(NULL AS VARCHAR) AS sku, + CAST(NULL AS INT) AS quantity, + CAST(NULL AS INT) AS gross_amount_minor, + CAST(NULL AS INT) AS discount_amount_minor, + CAST(NULL AS INT) AS net_amount_minor, + CAST(NULL AS INT) AS tax_amount_minor, + CAST(NULL AS OBJECT) AS discount_by_kind, + CAST(NULL AS ARRAY) AS promo_codes, + CAST(NULL AS BOOLEAN) AS has_stacked_promotions, + CAST(NULL AS DECIMAL(9, 6)) AS discount_share + WHERE 1 = 0 +), + +__ref__int_product_catalogue AS ( + SELECT + 'DUCK-MAL-01' AS sku, + NULL AS product_id, + NULL AS sku_family, + NULL AS title, + NULL AS variant_label, + NULL AS brand_code, + NULL AS brand_name, + NULL AS is_private_label, + NULL AS category_id, + NULL AS category_slug_path, + 'ducks' AS category_level_1, + NULL AS material, + NULL AS weight_kg, + NULL AS primary_image_url, + NULL AS image_count, + NULL AS is_active, + NULL AS launched_on, + NULL AS is_bundle_component +), + +__expected__fct_order_lines AS ( + SELECT + 'L1' AS order_line_id, + 'ducks' AS category_level_1, + 2 AS quantity, + 2598 AS net_amount_minor, + FALSE AS has_stacked_promotions + UNION ALL + SELECT + 'L2' AS order_line_id, + CAST(NULL AS VARCHAR) AS category_level_1, + 1 AS quantity, + 4500 AS net_amount_minor, + FALSE AS has_stacked_promotions +), + +__assert__promo_codes_default_to_empty_arrays AS ( + SELECT l.order_line_id + FROM __ref("fct_order_lines") AS l + WHERE ARRAY_SIZE(l.promo_codes) <> 0 +), + +__assert__order_line_id_is_unique_and_present AS ( + SELECT t.order_line_id + FROM __ref("fct_order_lines") AS t + GROUP BY + t.order_line_id + HAVING + COUNT(*) > 1 + OR t.order_line_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_orders__adds_sequence_and_collection_without_timelines.sql b/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_orders__adds_sequence_and_collection_without_timelines.sql new file mode 100644 index 0000000000..85f03f5455 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_orders__adds_sequence_and_collection_without_timelines.sql @@ -0,0 +1,248 @@ +TEST (name "fct_orders__adds_sequence_and_collection_without_timelines"); + +WITH __ref__int_orders_usd AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + 'EU' AS region_code, + 'EUR' AS currency_code, + NULL AS placed_at, + NULL AS placed_date_local, + 'DELIVERED' AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS amount, + NULL AS amount_usd, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS rate_age_days, + NULL AS is_rate_stale, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size + UNION ALL + SELECT + 'O2', + 'C1', + 'EU', + 'EUR', + NULL, + NULL, + 'PENDING', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + 'C2', + 'US', + 'USD', + NULL, + NULL, + 'CANCELLED', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O4', + 'C2', + 'US', + 'USD', + NULL, + NULL, + 'SHIPPED', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__int_order_status_timeline AS ( + SELECT + CAST(NULL AS VARCHAR) AS order_id, + CAST(NULL AS INT) AS transition_count, + CAST(NULL AS TIMESTAMP_TZ) AS first_paid_at, + CAST(NULL AS TIMESTAMP_TZ) AS first_shipped_at, + CAST(NULL AS TIMESTAMP_TZ) AS first_delivered_at, + CAST(NULL AS TIMESTAMP_TZ) AS cancelled_at, + CAST(NULL AS INT) AS minutes_to_pay, + CAST(NULL AS DECIMAL(12, 2)) AS hours_paid_to_shipped, + CAST(NULL AS VARCHAR) AS status_path, + CAST(NULL AS BOOLEAN) AS has_regression, + CAST(NULL AS VARCHAR) AS last_status, + CAST(NULL AS VARCHAR) AS longest_status + WHERE 1 = 0 +), + +__ref__int_order_payments_rollup AS ( + SELECT + 'O1' AS order_id, + NULL AS card_captured_minor, + NULL AS wallet_captured_minor, + NULL AS bank_transfer_captured_minor, + NULL AS other_captured_minor, + 1000 AS captured_minor, + 1000 AS refunded_minor, + NULL AS chargeback_minor, + NULL AS gift_card_tender_minor, + 0 AS net_collected_minor, + NULL AS failed_attempts, + NULL AS tender_methods + UNION ALL + SELECT + 'O3', + NULL, + NULL, + NULL, + NULL, + 0, + 0, + NULL, + NULL, + 0, + NULL, + NULL + UNION ALL + SELECT + 'O4', + NULL, + NULL, + NULL, + NULL, + 500, + NULL, + NULL, + NULL, + 500, + NULL, + NULL +), + +__ref__int_customer_order_sequence AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + NULL AS placed_at, + NULL AS placed_date_local, + 1 AS order_number, + TRUE AS is_first_order, + NULL AS days_since_previous_order, + NULL AS amount_usd, + NULL AS lifetime_revenue_usd, + NULL AS rolling_90d_revenue_usd, + NULL AS previous_sales_channel, + NULL AS is_channel_switch, + NULL AS cohort_month + UNION ALL + SELECT + 'O4', + 'C2', + NULL, + NULL, + 2, + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__fct_orders AS ( + SELECT + 'O1' AS order_id, + 1 AS order_number, + TRUE AS is_first_order, + CAST(NULL AS TIMESTAMP_TZ) AS first_paid_at, + 1000 AS captured_minor, + 1000 AS refunded_minor, + 0 AS net_collected_minor, + TRUE AS is_fully_refunded + UNION ALL + SELECT + 'O2' AS order_id, + CAST(NULL AS INT) AS order_number, + FALSE AS is_first_order, + CAST(NULL AS TIMESTAMP_TZ) AS first_paid_at, + 0 AS captured_minor, + 0 AS refunded_minor, + 0 AS net_collected_minor, + FALSE AS is_fully_refunded + UNION ALL + SELECT + 'O3' AS order_id, + CAST(NULL AS INT) AS order_number, + FALSE AS is_first_order, + CAST(NULL AS TIMESTAMP_TZ) AS first_paid_at, + 0 AS captured_minor, + 0 AS refunded_minor, + 0 AS net_collected_minor, + FALSE AS is_fully_refunded + UNION ALL + SELECT + 'O4' AS order_id, + 2 AS order_number, + FALSE AS is_first_order, + CAST(NULL AS TIMESTAMP_TZ) AS first_paid_at, + 500 AS captured_minor, + 0 AS refunded_minor, + 500 AS net_collected_minor, + FALSE AS is_fully_refunded +), + +__assert__order_id_is_unique_and_present AS ( + SELECT t.order_id + FROM __ref("fct_orders") AS t + GROUP BY + t.order_id + HAVING + COUNT(*) > 1 + OR t.order_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_payments__converts_attempts_at_the_utc_authorisation_date.sql b/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_payments__converts_attempts_at_the_utc_authorisation_date.sql new file mode 100644 index 0000000000..36355b8d79 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_payments__converts_attempts_at_the_utc_authorisation_date.sql @@ -0,0 +1,164 @@ +TEST (name "fct_payments__converts_attempts_at_the_utc_authorisation_date"); + +WITH __ref__int_payment_attempts AS ( + SELECT + 'P1' AS payment_id, + 'O1' AS order_id, + 1 AS attempt_number, + NULL AS attempt_count, + TRUE AS is_final_attempt, + 'card' AS payment_method, + 1250 AS amount_minor, + 'EUR' AS currency_code, + '2026-03-10 23:30:00 -02:00'::TIMESTAMPTZ AS authorised_at, + NULL AS seconds_since_previous_attempt, + NULL AS gateway_status, + TRUE AS is_successful, + NULL AS decline_family, + NULL AS gateway_error_code, + NULL AS risk_score, + NULL AS risk_band, + NULL AS three_ds_outcome, + NULL AS card_bin_valid, + 'network_a' AS card_scheme, + NULL AS had_chargeback + UNION ALL + SELECT + 'P2', + 'O1', + 1, + NULL, + TRUE, + 'card', + 1500, + 'KWD', + '2026-03-10 12:00:00 +03:00'::TIMESTAMPTZ, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'network_b', + NULL + UNION ALL + SELECT + 'P3', + 'O1', + 1, + NULL, + TRUE, + 'card', + 100, + 'XXX', + '2026-03-10 12:00:00 +00:00'::TIMESTAMPTZ, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'network_a', + NULL + UNION ALL + SELECT + 'P4', + 'O1', + 1, + NULL, + TRUE, + 'card', + 1000, + 'JPY', + '2026-03-10 12:00:00 +09:00'::TIMESTAMPTZ, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-03-10'::DATE AS rate_date, + 1.08::NUMBER(18, 8) AS rate_to_usd, + NULL AS usd_to_rate, + NULL AS next_rate_date + UNION ALL + SELECT + 'EUR', + '2026-03-11'::DATE, + 1.1::NUMBER(18, 8), + NULL, + NULL + UNION ALL + SELECT + 'KWD', + '2026-03-01'::DATE, + 3.25::NUMBER(18, 8), + NULL, + NULL +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'KWD', + NULL, + 3, + NULL + UNION ALL + SELECT + 'JPY', + NULL, + 0, + NULL +), + +__expected__fct_payments AS ( + SELECT + 'P1' AS payment_id, + 'network_a' AS card_scheme, + 12.5::DECIMAL(38, 6) AS amount, + 13.75::DECIMAL(18, 2) AS amount_usd + UNION ALL + SELECT + 'P2' AS payment_id, + 'network_b' AS card_scheme, + 1.5::DECIMAL(38, 6) AS amount, + 4.88::DECIMAL(18, 2) AS amount_usd + UNION ALL + SELECT + 'P4' AS payment_id, + CAST(NULL AS VARCHAR) AS card_scheme, + 1000.0::DECIMAL(38, 6) AS amount, + CAST(NULL AS DECIMAL(18, 2)) AS amount_usd +), + +__assert__payment_id_is_unique_and_present AS ( + SELECT t.payment_id + FROM __ref("fct_payments") AS t + GROUP BY + t.payment_id + HAVING + COUNT(*) > 1 + OR t.payment_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_refunds__passes_converted_refunds_through.sql b/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_refunds__passes_converted_refunds_through.sql new file mode 100644 index 0000000000..8037356c64 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/orders/test_fct_refunds__passes_converted_refunds_through.sql @@ -0,0 +1,55 @@ +TEST (name "fct_refunds__passes_converted_refunds_through"); + +WITH __ref__int_refunds_usd AS ( + SELECT + 'F1' AS refund_id, + 'P1' AS payment_id, + 'O1' AS order_id, + 'EUR' AS currency_code, + NULL AS refunded_at, + NULL AS refunded_date, + 'return' AS reason_category, + 12.5::DECIMAL(38, 6) AS amount, + 13.75::DECIMAL(18, 2) AS amount_usd, + NULL AS days_after_payment, + TRUE AS is_full_refund + UNION ALL + SELECT + 'F2', + 'P2', + 'O2', + 'KWD', + NULL, + NULL, + 'goodwill', + 0.75::DECIMAL(38, 6), + 2.44::DECIMAL(18, 2), + NULL, + FALSE +), + +__expected__fct_refunds AS ( + SELECT + 'F1' AS refund_id, + 12.5::NUMBER(38, 6) AS amount, + 13.75::NUMBER(18, 2) AS amount_usd, + 'return' AS reason_category, + TRUE AS is_full_refund + UNION ALL + SELECT + 'F2' AS refund_id, + 0.75::NUMBER(38, 6) AS amount, + 2.44::NUMBER(18, 2) AS amount_usd, + 'goodwill' AS reason_category, + FALSE AS is_full_refund +), + +__assert__refund_id_is_unique_and_present AS ( + SELECT t.refund_id + FROM __ref("fct_refunds") AS t + GROUP BY + t.refund_id + HAVING + COUNT(*) > 1 + OR t.refund_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_basket_mix_weekly__shares_units_by_category_within_each_week.sql b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_basket_mix_weekly__shares_units_by_category_within_each_week.sql new file mode 100644 index 0000000000..b9f7fad128 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_basket_mix_weekly__shares_units_by_category_within_each_week.sql @@ -0,0 +1,212 @@ +TEST (name "rpt_basket_mix_weekly__shares_units_by_category_within_each_week"); + +WITH __ref__fct_order_lines AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + NULL AS sku, + NULL AS sku_family, + 'ducks' AS category_level_1, + 2 AS quantity, + NULL AS unit_amount_minor, + NULL AS list_amount_minor, + NULL AS markdown_pct, + NULL AS gross_amount_minor, + 100 AS discount_amount_minor, + 2000 AS net_amount_minor, + NULL AS tax_amount_minor, + NULL AS promo_codes, + NULL AS has_stacked_promotions, + NULL AS line_weight_kg + UNION ALL + SELECT + 'L2', + 'O1', + NULL, + NULL, + 'textiles', + 1, + NULL, + NULL, + NULL, + NULL, + 0, + 1500, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'L3', + 'O2', + NULL, + NULL, + 'ducks', + 3, + NULL, + NULL, + NULL, + NULL, + 0, + 3000, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'L4', + 'O3', + NULL, + NULL, + NULL, + 1, + NULL, + NULL, + NULL, + NULL, + 0, + 500, + NULL, + NULL, + NULL, + NULL +), + +__ref__fct_orders AS ( + SELECT + 'O1' AS order_id, + NULL AS customer_id, + NULL AS region_code, + NULL AS placed_at, + '2026-03-10'::DATE AS placed_date_local, + NULL AS order_status, + NULL AS sales_channel, + NULL AS units, + NULL AS currency_code, + NULL AS amount, + NULL AS amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS gross_amount_usd, + NULL AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size, + NULL AS order_number, + NULL AS is_first_order, + NULL AS first_paid_at, + NULL AS first_delivered_at, + NULL AS hours_paid_to_shipped, + NULL AS captured_minor, + NULL AS refunded_minor, + NULL AS net_collected_minor, + NULL AS is_fully_refunded + UNION ALL + SELECT + 'O2', + NULL, + NULL, + NULL, + '2026-03-12'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + NULL, + NULL, + NULL, + '2026-03-16'::DATE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__rpt_basket_mix_weekly AS ( + SELECT + '2026-03-09:ducks' AS basket_week_key, + 2 AS order_count, + 5 AS units, + 5000 AS net_amount_minor, + 0.5::DECIMAL(9, 4) AS discounted_line_share, + 0.8333::DECIMAL(9, 4) AS unit_share_of_week, + 1 AS category_rank + UNION ALL + SELECT + '2026-03-09:textiles' AS basket_week_key, + 1 AS order_count, + 1 AS units, + 1500 AS net_amount_minor, + 0.0::DECIMAL(9, 4) AS discounted_line_share, + 0.1667::DECIMAL(9, 4) AS unit_share_of_week, + 2 AS category_rank + UNION ALL + SELECT + '2026-03-16:uncategorised' AS basket_week_key, + 1 AS order_count, + 1 AS units, + 500 AS net_amount_minor, + 0.0::DECIMAL(9, 4) AS discounted_line_share, + 1.0::DECIMAL(9, 4) AS unit_share_of_week, + 1 AS category_rank +), + +__assert__basket_week_key_is_unique_and_present AS ( + SELECT t.basket_week_key + FROM __ref("rpt_basket_mix_weekly") AS t + GROUP BY + t.basket_week_key + HAVING + COUNT(*) > 1 + OR t.basket_week_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_cart_abandonment_daily__counts_abandoned_value_by_utc_activity_day.sql b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_cart_abandonment_daily__counts_abandoned_value_by_utc_activity_day.sql new file mode 100644 index 0000000000..c80e77dc42 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_cart_abandonment_daily__counts_abandoned_value_by_utc_activity_day.sql @@ -0,0 +1,93 @@ +TEST (name "rpt_cart_abandonment_daily__counts_abandoned_value_by_utc_activity_day"); + +WITH __ref__int_cart_abandonment AS ( + SELECT + 'C1' AS cart_id, + NULL AS customer_id, + 'eu-web' AS storefront_code, + NULL AS cart_status, + NULL AS item_count, + NULL AS units, + 2500 AS cart_value_minor, + '2026-03-10 23:30:00 -02:00'::TIMESTAMP_TZ AS last_activity_at, + NULL AS converted_order_id, + FALSE AS is_converted, + TRUE AS is_abandoned, + NULL AS hours_to_convert, + NULL AS distinct_sku_families + UNION ALL + SELECT + 'C2', + NULL, + 'eu-web', + NULL, + NULL, + NULL, + 1000, + '2026-03-11 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + TRUE, + FALSE, + 2.5::DECIMAL(10, 2), + NULL + UNION ALL + SELECT + 'C3', + NULL, + 'eu-web', + NULL, + NULL, + NULL, + 500, + '2026-03-11 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + TRUE, + FALSE, + 5.5::DECIMAL(10, 2), + NULL + UNION ALL + SELECT + 'C4', + NULL, + 'us-web', + NULL, + NULL, + NULL, + 999, + '2026-03-11 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + FALSE, + FALSE, + NULL, + NULL +), + +__expected__rpt_cart_abandonment_daily AS ( + SELECT + 'eu-web:2026-03-11' AS cart_day_key, + 3 AS carts, + 1 AS abandoned_carts, + 2 AS converted_carts, + 0.3333::DECIMAL(9, 4) AS abandonment_rate, + 2500 AS abandoned_value_minor, + 4.0::DECIMAL(10, 2) AS median_hours_to_convert + UNION ALL + SELECT + 'us-web:2026-03-11' AS cart_day_key, + 1 AS carts, + 0 AS abandoned_carts, + 0 AS converted_carts, + 0.0::DECIMAL(9, 4) AS abandonment_rate, + 0 AS abandoned_value_minor, + CAST(NULL AS DECIMAL(10, 2)) AS median_hours_to_convert +), + +__assert__cart_day_key_is_unique_and_present AS ( + SELECT t.cart_day_key + FROM __ref("rpt_cart_abandonment_daily") AS t + GROUP BY + t.cart_day_key + HAVING + COUNT(*) > 1 + OR t.cart_day_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_daily_sales_by_region__compares_with_same_day_last_week.sql b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_daily_sales_by_region__compares_with_same_day_last_week.sql new file mode 100644 index 0000000000..d7135ccf57 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_daily_sales_by_region__compares_with_same_day_last_week.sql @@ -0,0 +1,247 @@ +TEST (name "rpt_daily_sales_by_region__compares_with_same_day_last_week"); + +WITH __ref__fct_orders AS ( + SELECT + 'O1' AS order_id, + 'C1' AS customer_id, + 'EU' AS region_code, + NULL AS placed_at, + '2026-03-03'::DATE AS placed_date_local, + 'DELIVERED' AS order_status, + NULL AS sales_channel, + 2 AS units, + NULL AS currency_code, + NULL AS amount, + 100.0::NUMBER(18, 2) AS amount_usd, + NULL AS rate_to_usd, + NULL AS rate_date, + NULL AS gross_amount_usd, + 10.0::NUMBER(18, 2) AS discount_amount_usd, + NULL AS tax_amount_usd, + NULL AS promo_code, + NULL AS utm_source, + NULL AS utm_medium, + NULL AS utm_campaign, + NULL AS basket_size, + NULL AS order_number, + TRUE AS is_first_order, + NULL AS first_paid_at, + NULL AS first_delivered_at, + NULL AS hours_paid_to_shipped, + NULL AS captured_minor, + NULL AS refunded_minor, + NULL AS net_collected_minor, + NULL AS is_fully_refunded + UNION ALL + SELECT + 'O2', + 'C1', + 'EU', + NULL, + '2026-03-10'::DATE, + 'DELIVERED', + NULL, + 1, + NULL, + NULL, + 50.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + 0.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O3', + 'C2', + 'EU', + NULL, + '2026-03-10'::DATE, + 'DELIVERED', + NULL, + 1, + NULL, + NULL, + 70.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + 0.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O4', + 'C2', + 'EU', + NULL, + '2026-03-10'::DATE, + 'CANCELLED', + NULL, + 1, + NULL, + NULL, + 999.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + 0.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O5', + 'C3', + 'US', + NULL, + '2026-03-10'::DATE, + 'DELIVERED', + NULL, + 1, + NULL, + NULL, + 30.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + 0.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'O6', + 'C4', + NULL, + NULL, + '2026-03-10'::DATE, + 'DELIVERED', + NULL, + 1, + NULL, + NULL, + 20.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + 0.0::NUMBER(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__rpt_daily_sales_by_region AS ( + SELECT + 'EU:2026-03-03' AS sales_day_key, + 1 AS order_count, + 1 AS customer_count, + 1 AS first_order_count, + 100.0::DECIMAL(18, 2) AS revenue_usd, + 100.0::DECIMAL(18, 2) AS average_order_usd, + CAST(NULL AS DECIMAL(18, 2)) AS revenue_same_day_last_week_usd, + 100.0::DECIMAL(18, 2) AS revenue_trailing_28d_usd + UNION ALL + SELECT + 'EU:2026-03-10' AS sales_day_key, + 2 AS order_count, + 2 AS customer_count, + 1 AS first_order_count, + 120.0::DECIMAL(18, 2) AS revenue_usd, + 60.0::DECIMAL(18, 2) AS average_order_usd, + 100.0::DECIMAL(18, 2) AS revenue_same_day_last_week_usd, + 220.0::DECIMAL(18, 2) AS revenue_trailing_28d_usd + UNION ALL + SELECT + 'US:2026-03-10' AS sales_day_key, + 1 AS order_count, + 1 AS customer_count, + 0 AS first_order_count, + 30.0::DECIMAL(18, 2) AS revenue_usd, + 30.0::DECIMAL(18, 2) AS average_order_usd, + CAST(NULL AS DECIMAL(18, 2)) AS revenue_same_day_last_week_usd, + 30.0::DECIMAL(18, 2) AS revenue_trailing_28d_usd + UNION ALL + SELECT + 'UNKNOWN:2026-03-10' AS sales_day_key, + 1 AS order_count, + 1 AS customer_count, + 0 AS first_order_count, + 20.0::DECIMAL(18, 2) AS revenue_usd, + 20.0::DECIMAL(18, 2) AS average_order_usd, + CAST(NULL AS DECIMAL(18, 2)) AS revenue_same_day_last_week_usd, + 20.0::DECIMAL(18, 2) AS revenue_trailing_28d_usd +), + +__assert__sales_day_key_is_unique_and_present AS ( + SELECT t.sales_day_key + FROM __ref("rpt_daily_sales_by_region") AS t + GROUP BY + t.sales_day_key + HAVING + COUNT(*) > 1 + OR t.sales_day_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_payment_decline_rates__measures_retry_recovery_by_method_month.sql b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_payment_decline_rates__measures_retry_recovery_by_method_month.sql new file mode 100644 index 0000000000..562797e363 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_payment_decline_rates__measures_retry_recovery_by_method_month.sql @@ -0,0 +1,103 @@ +TEST (name "rpt_payment_decline_rates__measures_retry_recovery_by_method_month"); + +WITH __ref__fct_payments AS ( + SELECT + 'P1' AS payment_id, + NULL AS order_id, + NULL AS attempt_number, + TRUE AS is_final_attempt, + 'card' AS payment_method, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_TZ AS authorised_at, + TRUE AS is_successful, + 'NONE' AS decline_family, + NULL AS risk_band, + NULL AS card_scheme, + NULL AS had_chargeback, + NULL AS currency_code, + NULL AS amount, + 20.0::DECIMAL(18, 2) AS amount_usd + UNION ALL + SELECT + 'P2', + NULL, + NULL, + FALSE, + 'card', + '2026-03-06 10:00:00 +00:00'::TIMESTAMP_TZ, + FALSE, + 'INSUFFICIENT_FUNDS', + NULL, + NULL, + NULL, + NULL, + NULL, + 15.0::DECIMAL(18, 2) + UNION ALL + SELECT + 'P3', + NULL, + NULL, + TRUE, + 'card', + '2026-03-07 10:00:00 +00:00'::TIMESTAMP_TZ, + FALSE, + 'INSUFFICIENT_FUNDS', + NULL, + NULL, + NULL, + NULL, + NULL, + 15.0::DECIMAL(18, 2) + UNION ALL + SELECT + 'P4', + NULL, + NULL, + TRUE, + 'wallet', + '2026-03-31 23:30:00 -02:00'::TIMESTAMP_TZ, + TRUE, + 'NONE', + NULL, + NULL, + NULL, + NULL, + NULL, + 30.0::DECIMAL(18, 2) +), + +__expected__rpt_payment_decline_rates AS ( + SELECT + '2026-03:card:NONE' AS decline_key, + 1 AS attempts, + 0.3333::DECIMAL(9, 4) AS share_of_method_attempts, + 20.0::DECIMAL(18, 2) AS attempted_usd, + 0 AS recovered_on_retry, + 0.0::DECIMAL(9, 4) AS retry_recovery_rate + UNION ALL + SELECT + '2026-03:card:INSUFFICIENT_FUNDS' AS decline_key, + 2 AS attempts, + 0.6667::DECIMAL(9, 4) AS share_of_method_attempts, + 30.0::DECIMAL(18, 2) AS attempted_usd, + 1 AS recovered_on_retry, + 0.5::DECIMAL(9, 4) AS retry_recovery_rate + UNION ALL + SELECT + '2026-04:wallet:NONE' AS decline_key, + 1 AS attempts, + 1.0::DECIMAL(9, 4) AS share_of_method_attempts, + 30.0::DECIMAL(18, 2) AS attempted_usd, + 0 AS recovered_on_retry, + 0.0::DECIMAL(9, 4) AS retry_recovery_rate +), + +__assert__decline_key_is_unique_and_present AS ( + SELECT t.decline_key + FROM __ref("rpt_payment_decline_rates") AS t + GROUP BY + t.decline_key + HAVING + COUNT(*) > 1 + OR t.decline_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_review_scores_by_product__weights_ratings_and_builds_distribution.sql b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_review_scores_by_product__weights_ratings_and_builds_distribution.sql new file mode 100644 index 0000000000..bf3069cb61 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_review_scores_by_product__weights_ratings_and_builds_distribution.sql @@ -0,0 +1,90 @@ +TEST (name "rpt_review_scores_by_product__weights_ratings_and_builds_distribution"); + +WITH __ref__int_reviews_moderated AS ( + SELECT + 'RV1' AS review_id, + 'DUCK-MAL-01' AS sku, + 'P1' AS product_id, + NULL AS customer_id, + 5 AS rating, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS submitted_at, + TRUE AS is_verified_purchase, + 2 AS sentiment_score, + 1.0::DECIMAL(6, 3) AS review_weight, + NULL AS rating_rank_in_product + UNION ALL + SELECT + 'RV2', + 'DUCK-MAL-01', + 'P1', + NULL, + 4, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_TZ, + FALSE, + 1, + 0.5::DECIMAL(6, 3), + NULL + UNION ALL + SELECT + 'RV3', + 'DUCK-MAL-01', + 'P1', + NULL, + 5, + '2026-03-03 10:00:00 +00:00'::TIMESTAMP_TZ, + TRUE, + -1, + 0.5::DECIMAL(6, 3), + NULL + UNION ALL + SELECT + 'RV4', + 'DUCK-MAL-01', + 'P2', + NULL, + 1, + '2026-03-02 10:00:00 +00:00'::TIMESTAMP_TZ, + FALSE, + -2, + 0.0::DECIMAL(6, 3), + NULL +), + +__expected__rpt_review_scores_by_product AS ( + SELECT + 'P1' AS product_id, + 3 AS review_count, + 4.75::DECIMAL(4, 2) AS weighted_rating, + 0.6667::DECIMAL(9, 4) AS verified_share, + 2 AS net_sentiment, + '2026-03-05 10:00:00 +00:00'::TIMESTAMP_TZ AS latest_review_at + UNION ALL + SELECT + 'P2' AS product_id, + 1 AS review_count, + CAST(NULL AS DECIMAL(4, 2)) AS weighted_rating, + 0.0::DECIMAL(9, 4) AS verified_share, + -2 AS net_sentiment, + '2026-03-02 10:00:00 +00:00'::TIMESTAMP_TZ AS latest_review_at +), + +__assert__distribution_counts_each_rating AS ( + SELECT r.product_id + FROM __ref("rpt_review_scores_by_product") AS r + WHERE + r.product_id = 'P1' + AND ( + r.rating_distribution:"5"::INT <> 2 + OR r.rating_distribution:"4"::INT <> 1 + ) +), + +__assert__product_id_is_unique_and_present AS ( + SELECT t.product_id + FROM __ref("rpt_review_scores_by_product") AS t + GROUP BY + t.product_id + HAVING + COUNT(*) > 1 + OR t.product_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_subscription_mrr_monthly__bridges_opening_to_closing_mrr.sql b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_subscription_mrr_monthly__bridges_opening_to_closing_mrr.sql new file mode 100644 index 0000000000..1b7bfe4dff --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/orders/test_rpt_subscription_mrr_monthly__bridges_opening_to_closing_mrr.sql @@ -0,0 +1,109 @@ +TEST (name "rpt_subscription_mrr_monthly__bridges_opening_to_closing_mrr"); + +WITH __ref__int_subscription_mrr_daily AS ( + SELECT + 'S1:2026-01-15' AS subscription_day_key, + 'S1' AS subscription_id, + NULL AS customer_id, + '2026-01-15'::DATE AS calendar_date, + 10.0::DECIMAL(18, 4) AS mrr_usd, + TRUE AS is_first_day + UNION ALL + SELECT + 'S1:2026-01-31', + 'S1', + NULL, + '2026-01-31'::DATE, + 10.0::DECIMAL(18, 4), + FALSE + UNION ALL + SELECT + 'S1:2026-02-28', + 'S1', + NULL, + '2026-02-28'::DATE, + 10.0::DECIMAL(18, 4), + FALSE + UNION ALL + SELECT + 'S1:2026-03-31', + 'S1', + NULL, + '2026-03-31'::DATE, + 10.0::DECIMAL(18, 4), + FALSE + UNION ALL + SELECT + 'S2:2026-02-10', + 'S2', + NULL, + '2026-02-10'::DATE, + 20.0::DECIMAL(18, 4), + TRUE + UNION ALL + SELECT + 'S2:2026-02-28', + 'S2', + NULL, + '2026-02-28'::DATE, + 20.0::DECIMAL(18, 4), + FALSE + UNION ALL + SELECT + 'S3:2026-03-05', + 'S3', + NULL, + '2026-03-05'::DATE, + 5.0::DECIMAL(18, 4), + TRUE + UNION ALL + SELECT + 'S3:2026-03-31', + 'S3', + NULL, + '2026-03-31'::DATE, + 5.0::DECIMAL(18, 4), + FALSE +), + +__expected__rpt_subscription_mrr_monthly AS ( + SELECT + '2026-01-01'::DATE AS month_start, + 10.0::DECIMAL(18, 2) AS closing_mrr_usd, + 0.0::DECIMAL(18, 2) AS opening_mrr_usd, + 10.0::DECIMAL(18, 2) AS new_mrr_usd, + 0.0::DECIMAL(18, 2) AS churned_mrr_usd, + 10.0::DECIMAL(18, 2) AS net_change_usd, + 1 AS active_subscribers, + 1 AS new_subscribers + UNION ALL + SELECT + '2026-02-01'::DATE AS month_start, + 30.0::DECIMAL(18, 2) AS closing_mrr_usd, + 10.0::DECIMAL(18, 2) AS opening_mrr_usd, + 20.0::DECIMAL(18, 2) AS new_mrr_usd, + 0.0::DECIMAL(18, 2) AS churned_mrr_usd, + 20.0::DECIMAL(18, 2) AS net_change_usd, + 2 AS active_subscribers, + 1 AS new_subscribers + UNION ALL + SELECT + '2026-03-01'::DATE AS month_start, + 15.0::DECIMAL(18, 2) AS closing_mrr_usd, + 30.0::DECIMAL(18, 2) AS opening_mrr_usd, + 5.0::DECIMAL(18, 2) AS new_mrr_usd, + 20.0::DECIMAL(18, 2) AS churned_mrr_usd, + -15.0::DECIMAL(18, 2) AS net_change_usd, + 2 AS active_subscribers, + 1 AS new_subscribers +), + +__assert__month_start_is_unique_and_present AS ( + SELECT t.month_start + FROM __ref("rpt_subscription_mrr_monthly") AS t + GROUP BY + t.month_start + HAVING + COUNT(*) > 1 + OR t.month_start IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/partners/test_fct_partner_orders__publishes_unified_partner_orders.sql b/benchmarks/duckshop/project/tests/unit/marts/partners/test_fct_partner_orders__publishes_unified_partner_orders.sql new file mode 100644 index 0000000000..260176d90a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/partners/test_fct_partner_orders__publishes_unified_partner_orders.sql @@ -0,0 +1,69 @@ +TEST (name "fct_partner_orders__publishes_unified_partner_orders"); + +WITH __ref__int_partner_orders_unified AS ( + SELECT + 'MKT:A1' AS partner_order_key, + NULL AS source_relation, + 1 AS payload_version, + 'MKT' AS partner_code, + 'A1' AS external_order_id, + 'EUR' AS currency_code, + NULL AS placed_at_utc, + NULL AS placed_date, + NULL AS order_total_minor, + 26.67::DECIMAL(18, 2) AS order_total_usd, + NULL AS line_count, + 2 AS units, + NULL AS lines_total_minor, + TRUE AS lines_match_total, + 'C1' AS customer_id, + TRUE AS is_known_customer, + NULL AS order_rank_for_partner + UNION ALL + SELECT + 'MKT:B7', + NULL, + 2, + 'MKT', + 'B7', + 'USD', + NULL, + NULL, + NULL, + 49.99::DECIMAL(18, 2), + NULL, + 1, + NULL, + FALSE, + NULL, + FALSE, + NULL +), + +__expected__fct_partner_orders AS ( + SELECT + 'MKT:A1' AS partner_order_key, + 1 AS payload_version, + 26.67::NUMBER(18, 2) AS order_total_usd, + TRUE AS lines_match_total, + 'C1' AS customer_id, + TRUE AS is_known_customer + UNION ALL + SELECT + 'MKT:B7' AS partner_order_key, + 2 AS payload_version, + 49.99::NUMBER(18, 2) AS order_total_usd, + FALSE AS lines_match_total, + CAST(NULL AS VARCHAR) AS customer_id, + FALSE AS is_known_customer +), + +__assert__partner_order_key_is_unique_and_present AS ( + SELECT t.partner_order_key + FROM __ref("fct_partner_orders") AS t + GROUP BY + t.partner_order_key + HAVING + COUNT(*) > 1 + OR t.partner_order_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/partners/test_rpt_partner_fees_variance__totals_commission_variance_per_quarter.sql b/benchmarks/duckshop/project/tests/unit/marts/partners/test_rpt_partner_fees_variance__totals_commission_variance_per_quarter.sql new file mode 100644 index 0000000000..9946e3706c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/partners/test_rpt_partner_fees_variance__totals_commission_variance_per_quarter.sql @@ -0,0 +1,111 @@ +TEST (name "rpt_partner_fees_variance__totals_commission_variance_per_quarter"); + +WITH __ref__int_partner_fees_usd AS ( + SELECT + 'ST1' AS settlement_id, + 'MKT' AS partner_code, + NULL AS period_start, + '2026-01-31'::DATE AS period_end, + NULL AS period_days, + NULL AS currency_code, + 1000.0::DECIMAL(18, 2) AS gross_sales_usd, + 100.0::DECIMAL(18, 2) AS stated_commission_usd, + 100.0::DECIMAL(18, 2) AS expected_commission_usd, + NULL AS other_fees_usd, + NULL AS net_payable_usd, + NULL AS period_order_count, + NULL AS lines_without_listing, + NULL AS effective_commission_pct, + NULL AS commission_variance_pct, + TRUE AS is_within_tolerance, + TRUE AS statement_balances + UNION ALL + SELECT + 'ST2', + 'MKT', + NULL, + '2026-02-28'::DATE, + NULL, + NULL, + 500.0::DECIMAL(18, 2), + 60.0::DECIMAL(18, 2), + 50.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + TRUE + UNION ALL + SELECT + 'ST3', + 'MKT', + NULL, + '2026-03-31'::DATE, + NULL, + NULL, + 200.0::DECIMAL(18, 2), + 20.0::DECIMAL(18, 2), + 20.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + FALSE + UNION ALL + SELECT + 'ST4', + 'MKT', + NULL, + '2026-04-30'::DATE, + NULL, + NULL, + 100.0::DECIMAL(18, 2), + 10.0::DECIMAL(18, 2), + 10.0::DECIMAL(18, 2), + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + TRUE +), + +__expected__rpt_partner_fees_variance AS ( + SELECT + 'MKT:2026-01-01' AS partner_quarter_key, + 3 AS statements, + 1700.0::DECIMAL(18, 2) AS gross_sales_usd, + 180.0::DECIMAL(18, 2) AS stated_commission_usd, + 170.0::DECIMAL(18, 2) AS expected_commission_usd, + 10.0::DECIMAL(18, 2) AS variance_usd, + 1 AS statements_out_of_tolerance, + 1 AS unbalanced_statements + UNION ALL + SELECT + 'MKT:2026-04-01' AS partner_quarter_key, + 1 AS statements, + 100.0::DECIMAL(18, 2) AS gross_sales_usd, + 10.0::DECIMAL(18, 2) AS stated_commission_usd, + 10.0::DECIMAL(18, 2) AS expected_commission_usd, + 0.0::DECIMAL(18, 2) AS variance_usd, + 0 AS statements_out_of_tolerance, + 0 AS unbalanced_statements +), + +__assert__partner_quarter_key_is_unique_and_present AS ( + SELECT t.partner_quarter_key + FROM __ref("rpt_partner_fees_variance") AS t + GROUP BY + t.partner_quarter_key + HAVING + COUNT(*) > 1 + OR t.partner_quarter_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/partners/test_rpt_partner_listing_health__shares_listings_by_health_within_a_partner.sql b/benchmarks/duckshop/project/tests/unit/marts/partners/test_rpt_partner_listing_health__shares_listings_by_health_within_a_partner.sql new file mode 100644 index 0000000000..36001ebb33 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/partners/test_rpt_partner_listing_health__shares_listings_by_health_within_a_partner.sql @@ -0,0 +1,123 @@ +TEST (name "rpt_partner_listing_health__shares_listings_by_health_within_a_partner"); + +WITH __ref__int_partner_listing_status AS ( + SELECT + 'L1' AS listing_id, + 'MKT' AS partner_code, + 'DUCK-L1' AS sku, + NULL AS listing_state, + NULL AS listed_at, + NULL AS days_listed, + NULL AS product_title, + NULL AS brand_name, + NULL AS is_catalogue_active, + NULL AS current_list_price_minor, + NULL AS list_price_currency_code, + NULL AS available_to_promise, + 12.0::NUMBER(5, 2) AS commission_pct, + TRUE AS fulfilled_by_partner, + NULL AS fulfilment_method, + NULL AS term_keys, + 10 AS units_sold_30d, + NULL AS partners_listing_sku, + 'healthy' AS listing_health + UNION ALL + SELECT + 'L2', + 'MKT', + 'DUCK-L2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 15.0::NUMBER(5, 2), + FALSE, + NULL, + NULL, + 5, + NULL, + 'healthy' + UNION ALL + SELECT + 'L3', + 'MKT', + 'DUCK-L3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 12.0::NUMBER(5, 2), + FALSE, + NULL, + NULL, + 0, + NULL, + 'out_of_stock' + UNION ALL + SELECT + 'L4', + 'MKT', + 'DUCK-L4', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL, + NULL, + NULL, + 'suppressed' +), + +__expected__rpt_partner_listing_health AS ( + SELECT + 'MKT:healthy' AS listing_health_key, + 2 AS listings, + 1 AS partner_fulfilled, + 15 AS units_sold_30d, + 13.5::DECIMAL(5, 2) AS average_commission_pct, + 0.5::DECIMAL(9, 4) AS share_of_partner_listings + UNION ALL + SELECT + 'MKT:out_of_stock' AS listing_health_key, + 1 AS listings, + 0 AS partner_fulfilled, + 0 AS units_sold_30d, + 12.0::DECIMAL(5, 2) AS average_commission_pct, + 0.25::DECIMAL(9, 4) AS share_of_partner_listings + UNION ALL + SELECT + 'MKT:suppressed' AS listing_health_key, + 1 AS listings, + 1 AS partner_fulfilled, + CAST(NULL AS INT) AS units_sold_30d, + CAST(NULL AS DECIMAL(5, 2)) AS average_commission_pct, + 0.25::DECIMAL(9, 4) AS share_of_partner_listings +), + +__assert__listing_health_key_is_unique_and_present AS ( + SELECT t.listing_health_key + FROM __ref("rpt_partner_listing_health") AS t + GROUP BY + t.listing_health_key + HAVING + COUNT(*) > 1 + OR t.listing_health_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/support/test_fct_support_tickets__joins_ticket_measures_and_keeps_the_most_negative_chat.sql b/benchmarks/duckshop/project/tests/unit/marts/support/test_fct_support_tickets__joins_ticket_measures_and_keeps_the_most_negative_chat.sql new file mode 100644 index 0000000000..6e1ae187e6 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/support/test_fct_support_tickets__joins_ticket_measures_and_keeps_the_most_negative_chat.sql @@ -0,0 +1,218 @@ +TEST (name "fct_support_tickets__joins_ticket_measures_and_keeps_the_most_negative_chat"); + +WITH __ref__stg_tickets AS ( + SELECT + 'T1' AS ticket_id, + NULL AS customer_email, + 'email' AS channel, + NULL AS priority, + 'solved' AS ticket_status, + NULL AS created_at, + NULL AS solved_at, + NULL AS order_id, + 'damaged_item' AS contact_reason, + NULL AS tag_list, + NULL AS is_vip, + NULL AS custom_fields, + NULL AS source_seq, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS source_loaded_at + UNION ALL + SELECT + 'T2', + NULL, + 'chat', + NULL, + 'open', + NULL, + NULL, + NULL, + 'where_is_my_order', + NULL, + NULL, + NULL, + NULL, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'T3', + NULL, + 'phone', + NULL, + 'new', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + '2026-03-11 09:00:00'::TIMESTAMP_NTZ +), + +__ref__int_ticket_first_response AS ( + SELECT + 'T1' AS ticket_id, + '2026-03-10 09:00:00'::TIMESTAMPNTZ AS created_at_utc, + NULL AS first_response_at_utc, + NULL AS first_responder_id, + 30 AS minutes_to_first_response, + NULL AS first_customer_reply_at_utc, + NULL AS agent_responses_before_customer_reply, + NULL AS has_auto_reply +), + +__ref__int_ticket_business_hours AS ( + SELECT + 'T1' AS ticket_id, + NULL AS created_at_support_local, + NULL AS first_response_at_support_local, + NULL AS calendar_minutes_to_response, + 20 AS business_minutes_to_response, + NULL AS created_outside_hours +), + +__ref__int_ticket_sla_breaches AS ( + SELECT + 'T1' AS ticket_id, + 2 AS priority, + NULL AS channel, + NULL AS policy_id, + NULL AS first_response_target_minutes, + NULL AS first_response_minutes_used, + TRUE AS is_first_response_breached, + NULL AS resolution_target_minutes, + NULL AS resolution_minutes_used, + FALSE AS is_resolution_breached, + NULL AS breach_count, + NULL AS open_minutes_now +), + +__ref__int_ticket_threads AS ( + SELECT + 'T1' AS ticket_id, + 4 AS comment_count, + NULL AS public_comment_count, + NULL AS agent_participants, + NULL AS author_sequence, + 1 AS handoff_count, + NULL AS author_breakdown, + NULL AS total_attachments, + NULL AS last_comment_at, + NULL AS last_author_type, + NULL AS mentions_order +), + +__ref__int_chat_sentiment AS ( + SELECT + 'CH1' AS chat_id, + 'T1' AS ticket_id, + NULL AS customer_message_count, + NULL AS agent_message_count, + 2 AS sentiment_score, + 'positive' AS sentiment_label, + NULL AS median_agent_reply_seconds, + NULL AS ends_with_thanks, + NULL AS chat_minutes + UNION ALL + SELECT + 'CH2', + 'T1', + NULL, + NULL, + -1, + 'negative', + NULL, + NULL, + NULL + UNION ALL + SELECT + 'CH3', + NULL, + NULL, + NULL, + -5, + 'negative', + NULL, + NULL, + NULL +), + +__ref__int_customer_support_history AS ( + SELECT + 'T1' AS ticket_id, + 'C1' AS customer_id, + 2 AS ticket_number_for_customer, + NULL AS days_since_previous_ticket, + 4 AS csat_score, + NULL AS rolling_csat_avg, + TRUE AS is_repeat_contact_7d, + NULL AS contact_reason + UNION ALL + SELECT + 'T2', + 'C2', + 1, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__fct_support_tickets AS ( + SELECT + 'T1' AS ticket_id, + 'C1' AS customer_id, + 2 AS priority, + 30 AS minutes_to_first_response, + 20 AS business_minutes_to_response, + TRUE AS is_first_response_breached, + FALSE AS is_resolution_breached, + 4 AS comment_count, + 1 AS handoff_count, + 'negative' AS sentiment_label, + 2 AS ticket_number_for_customer, + TRUE AS is_repeat_contact_7d, + 4 AS csat_score + UNION ALL + SELECT + 'T2' AS ticket_id, + 'C2' AS customer_id, + CAST(NULL AS INT) AS priority, + CAST(NULL AS INT) AS minutes_to_first_response, + CAST(NULL AS INT) AS business_minutes_to_response, + FALSE AS is_first_response_breached, + FALSE AS is_resolution_breached, + 0 AS comment_count, + 0 AS handoff_count, + CAST(NULL AS VARCHAR) AS sentiment_label, + 1 AS ticket_number_for_customer, + FALSE AS is_repeat_contact_7d, + CAST(NULL AS INT) AS csat_score + UNION ALL + SELECT + 'T3' AS ticket_id, + CAST(NULL AS VARCHAR) AS customer_id, + CAST(NULL AS INT) AS priority, + CAST(NULL AS INT) AS minutes_to_first_response, + CAST(NULL AS INT) AS business_minutes_to_response, + FALSE AS is_first_response_breached, + FALSE AS is_resolution_breached, + 0 AS comment_count, + 0 AS handoff_count, + CAST(NULL AS VARCHAR) AS sentiment_label, + CAST(NULL AS INT) AS ticket_number_for_customer, + FALSE AS is_repeat_contact_7d, + CAST(NULL AS INT) AS csat_score +), + +__assert__ticket_id_is_unique_and_present AS ( + SELECT t.ticket_id + FROM __ref("fct_support_tickets") AS t + GROUP BY + t.ticket_id + HAVING + COUNT(*) > 1 + OR t.ticket_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/support/test_fct_support_tickets__rebuilds_only_tickets_loaded_inside_the_cursor_window.sql b/benchmarks/duckshop/project/tests/unit/marts/support/test_fct_support_tickets__rebuilds_only_tickets_loaded_inside_the_cursor_window.sql new file mode 100644 index 0000000000..027d385103 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/support/test_fct_support_tickets__rebuilds_only_tickets_loaded_inside_the_cursor_window.sql @@ -0,0 +1,177 @@ +TEST (name "fct_support_tickets__rebuilds_only_tickets_loaded_inside_the_cursor_window", cursor_start "2026-03-10 11:00:00", cursor_end "2026-03-11 00:00:00"); + +WITH __ref__stg_tickets AS ( + SELECT + 'T1' AS ticket_id, + NULL AS customer_email, + 'email' AS channel, + NULL AS priority, + 'solved' AS ticket_status, + NULL AS created_at, + NULL AS solved_at, + NULL AS order_id, + 'damaged_item' AS contact_reason, + NULL AS tag_list, + NULL AS is_vip, + NULL AS custom_fields, + NULL AS source_seq, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS source_loaded_at + UNION ALL + SELECT + 'T2', + NULL, + 'chat', + NULL, + 'open', + NULL, + NULL, + NULL, + 'where_is_my_order', + NULL, + NULL, + NULL, + NULL, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'T3', + NULL, + 'phone', + NULL, + 'new', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + '2026-03-11 09:00:00'::TIMESTAMP_NTZ +), + +__ref__int_ticket_first_response AS ( + SELECT + 'T1' AS ticket_id, + '2026-03-10 09:00:00'::TIMESTAMPNTZ AS created_at_utc, + NULL AS first_response_at_utc, + NULL AS first_responder_id, + 30 AS minutes_to_first_response, + NULL AS first_customer_reply_at_utc, + NULL AS agent_responses_before_customer_reply, + NULL AS has_auto_reply +), + +__ref__int_ticket_business_hours AS ( + SELECT + 'T1' AS ticket_id, + NULL AS created_at_support_local, + NULL AS first_response_at_support_local, + NULL AS calendar_minutes_to_response, + 20 AS business_minutes_to_response, + NULL AS created_outside_hours +), + +__ref__int_ticket_sla_breaches AS ( + SELECT + 'T1' AS ticket_id, + 2 AS priority, + NULL AS channel, + NULL AS policy_id, + NULL AS first_response_target_minutes, + NULL AS first_response_minutes_used, + TRUE AS is_first_response_breached, + NULL AS resolution_target_minutes, + NULL AS resolution_minutes_used, + FALSE AS is_resolution_breached, + NULL AS breach_count, + NULL AS open_minutes_now +), + +__ref__int_ticket_threads AS ( + SELECT + 'T1' AS ticket_id, + 4 AS comment_count, + NULL AS public_comment_count, + NULL AS agent_participants, + NULL AS author_sequence, + 1 AS handoff_count, + NULL AS author_breakdown, + NULL AS total_attachments, + NULL AS last_comment_at, + NULL AS last_author_type, + NULL AS mentions_order +), + +__ref__int_chat_sentiment AS ( + SELECT + 'CH1' AS chat_id, + 'T1' AS ticket_id, + NULL AS customer_message_count, + NULL AS agent_message_count, + 2 AS sentiment_score, + 'positive' AS sentiment_label, + NULL AS median_agent_reply_seconds, + NULL AS ends_with_thanks, + NULL AS chat_minutes + UNION ALL + SELECT + 'CH2', + 'T1', + NULL, + NULL, + -1, + 'negative', + NULL, + NULL, + NULL + UNION ALL + SELECT + 'CH3', + NULL, + NULL, + NULL, + -5, + 'negative', + NULL, + NULL, + NULL +), + +__ref__int_customer_support_history AS ( + SELECT + 'T1' AS ticket_id, + 'C1' AS customer_id, + 2 AS ticket_number_for_customer, + NULL AS days_since_previous_ticket, + 4 AS csat_score, + NULL AS rolling_csat_avg, + TRUE AS is_repeat_contact_7d, + NULL AS contact_reason + UNION ALL + SELECT + 'T2', + 'C2', + 1, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__fct_support_tickets AS ( + SELECT + 'T2' AS ticket_id, + 'C2' AS customer_id +), + +__assert__ticket_id_is_unique_and_present AS ( + SELECT t.ticket_id + FROM __ref("fct_support_tickets") AS t + GROUP BY + t.ticket_id + HAVING + COUNT(*) > 1 + OR t.ticket_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/support/test_rpt_agent_scorecard__ranks_agents_by_utilisation_with_csat.sql b/benchmarks/duckshop/project/tests/unit/marts/support/test_rpt_agent_scorecard__ranks_agents_by_utilisation_with_csat.sql new file mode 100644 index 0000000000..96874973c9 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/support/test_rpt_agent_scorecard__ranks_agents_by_utilisation_with_csat.sql @@ -0,0 +1,194 @@ +TEST (name "rpt_agent_scorecard__ranks_agents_by_utilisation_with_csat"); + +WITH __ref__int_agent_utilisation AS ( + SELECT + 'SH1' AS shift_id, + 'A1' AS agent_id, + 'EU' AS team_code, + '2026-03-02 08:00:00'::TIMESTAMPNTZ AS shift_start_utc, + NULL AS shift_end_utc, + 360 AS productive_minutes, + NULL AS call_minutes, + 10 AS calls_handled, + 5 AS tickets_first_responded, + 0.75::DECIMAL(9, 4) AS utilisation, + NULL AS is_overnight + UNION ALL + SELECT + 'SH2', + 'A1', + 'EU', + '2026-03-03 08:00:00'::TIMESTAMPNTZ, + NULL, + 300, + NULL, + 8, + 4, + 0.65::DECIMAL(9, 4), + NULL + UNION ALL + SELECT + 'SH3', + 'A2', + 'EU', + '2026-03-02 08:00:00'::TIMESTAMPNTZ, + NULL, + 420, + NULL, + 12, + 6, + 0.9::DECIMAL(9, 4), + NULL + UNION ALL + SELECT + 'SH4', + 'A3', + 'EU', + '2026-03-02 08:00:00'::TIMESTAMPNTZ, + NULL, + 0, + NULL, + 0, + 0, + NULL, + NULL + UNION ALL + SELECT + 'SH5', + 'A1', + 'EU', + '2026-04-01 08:00:00'::TIMESTAMPNTZ, + NULL, + 120, + NULL, + 2, + 1, + 0.5::DECIMAL(9, 4), + NULL +), + +__ref__int_ticket_first_response AS ( + SELECT + 'T1' AS ticket_id, + NULL AS created_at_utc, + '2026-03-02 09:00:00'::TIMESTAMPNTZ AS first_response_at_utc, + 'A1' AS first_responder_id, + NULL AS minutes_to_first_response, + NULL AS first_customer_reply_at_utc, + NULL AS agent_responses_before_customer_reply, + NULL AS has_auto_reply + UNION ALL + SELECT + 'T2', + NULL, + '2026-03-05 09:00:00'::TIMESTAMPNTZ, + 'A1', + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T3', + NULL, + '2026-03-02 09:00:00'::TIMESTAMPNTZ, + 'A2', + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T4', + NULL, + '2026-04-01 09:00:00'::TIMESTAMPNTZ, + 'A1', + NULL, + NULL, + NULL, + NULL +), + +__ref__int_customer_support_history AS ( + SELECT + 'T1' AS ticket_id, + 'C1' AS customer_id, + NULL AS ticket_number_for_customer, + NULL AS days_since_previous_ticket, + 5 AS csat_score, + NULL AS rolling_csat_avg, + NULL AS is_repeat_contact_7d, + NULL AS contact_reason + UNION ALL + SELECT + 'T2', + 'C2', + NULL, + NULL, + 3, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'T3', + 'C3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__rpt_agent_scorecard AS ( + SELECT + 'A1:2026-03' AS agent_month_key, + 2 AS shifts, + 11.0::DECIMAL(10, 2) AS productive_hours, + 0.7::DECIMAL(9, 4) AS average_utilisation, + 18 AS calls_handled, + 9 AS tickets_first_responded, + 4.0::DECIMAL(4, 2) AS average_csat, + 2 AS utilisation_rank_in_team + UNION ALL + SELECT + 'A2:2026-03' AS agent_month_key, + 1 AS shifts, + 7.0::DECIMAL(10, 2) AS productive_hours, + 0.9::DECIMAL(9, 4) AS average_utilisation, + 12 AS calls_handled, + 6 AS tickets_first_responded, + CAST(NULL AS DECIMAL(4, 2)) AS average_csat, + 1 AS utilisation_rank_in_team + UNION ALL + SELECT + 'A3:2026-03' AS agent_month_key, + 1 AS shifts, + 0.0::DECIMAL(10, 2) AS productive_hours, + CAST(NULL AS DECIMAL(9, 4)) AS average_utilisation, + 0 AS calls_handled, + 0 AS tickets_first_responded, + CAST(NULL AS DECIMAL(4, 2)) AS average_csat, + 3 AS utilisation_rank_in_team + UNION ALL + SELECT + 'A1:2026-04' AS agent_month_key, + 1 AS shifts, + 2.0::DECIMAL(10, 2) AS productive_hours, + 0.5::DECIMAL(9, 4) AS average_utilisation, + 2 AS calls_handled, + 1 AS tickets_first_responded, + CAST(NULL AS DECIMAL(4, 2)) AS average_csat, + 1 AS utilisation_rank_in_team +), + +__assert__agent_month_key_is_unique_and_present AS ( + SELECT t.agent_month_key + FROM __ref("rpt_agent_scorecard") AS t + GROUP BY + t.agent_month_key + HAVING + COUNT(*) > 1 + OR t.agent_month_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/support/test_rpt_contact_reasons_monthly__counts_repeat_contacts_and_linked_calls.sql b/benchmarks/duckshop/project/tests/unit/marts/support/test_rpt_contact_reasons_monthly__counts_repeat_contacts_and_linked_calls.sql new file mode 100644 index 0000000000..93671f1457 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/support/test_rpt_contact_reasons_monthly__counts_repeat_contacts_and_linked_calls.sql @@ -0,0 +1,185 @@ +TEST (name "rpt_contact_reasons_monthly__counts_repeat_contacts_and_linked_calls"); + +WITH __ref__fct_support_tickets AS ( + SELECT + 'T1' AS ticket_id, + NULL AS customer_id, + 'email' AS channel, + NULL AS priority, + NULL AS ticket_status, + 'damaged_item' AS contact_reason, + '2026-02-10 10:00:00'::TIMESTAMPNTZ AS created_at_utc, + NULL AS solved_at, + NULL AS minutes_to_first_response, + NULL AS business_minutes_to_response, + FALSE AS is_first_response_breached, + FALSE AS is_resolution_breached, + NULL AS comment_count, + NULL AS handoff_count, + NULL AS sentiment_label, + NULL AS ticket_number_for_customer, + FALSE AS is_repeat_contact_7d, + NULL AS csat_score, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + NULL, + 'email', + NULL, + NULL, + 'damaged_item', + '2026-03-05 10:00:00'::TIMESTAMPNTZ, + NULL, + NULL, + NULL, + FALSE, + FALSE, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'T3', + NULL, + 'email', + NULL, + NULL, + 'damaged_item', + '2026-03-06 10:00:00'::TIMESTAMPNTZ, + NULL, + NULL, + NULL, + FALSE, + FALSE, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'T4', + NULL, + 'email', + NULL, + NULL, + NULL, + '2026-03-07 10:00:00'::TIMESTAMPNTZ, + NULL, + NULL, + NULL, + FALSE, + FALSE, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'T5', + NULL, + 'email', + NULL, + NULL, + 'damaged_item', + NULL, + NULL, + NULL, + NULL, + FALSE, + FALSE, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL +), + +__ref__int_call_ivr_paths AS ( + SELECT + 'CL1' AS call_id, + 'T2' AS ticket_id, + NULL AS agent_id, + NULL AS first_menu_choice, + NULL AS final_menu_node, + NULL AS menu_steps, + NULL AS repeated_menu_visits, + FALSE AS abandoned_in_ivr, + NULL AS caller_region, + NULL AS duration_bucket + UNION ALL + SELECT + 'CL2', + 'T2', + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL + UNION ALL + SELECT + 'CL3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + NULL, + NULL +), + +__expected__rpt_contact_reasons_monthly AS ( + SELECT + '2026-02:damaged_item' AS reason_month_key, + 1 AS tickets, + 0 AS repeat_contacts, + 0.0::DECIMAL(9, 4) AS repeat_rate, + 0 AS calls, + 0 AS abandoned_in_ivr, + CAST(NULL AS INT) AS month_over_month_change + UNION ALL + SELECT + '2026-03:damaged_item' AS reason_month_key, + 2 AS tickets, + 1 AS repeat_contacts, + 0.5::DECIMAL(9, 4) AS repeat_rate, + 2 AS calls, + 1 AS abandoned_in_ivr, + 1 AS month_over_month_change + UNION ALL + SELECT + '2026-03:unspecified' AS reason_month_key, + 1 AS tickets, + 0 AS repeat_contacts, + 0.0::DECIMAL(9, 4) AS repeat_rate, + 0 AS calls, + 0 AS abandoned_in_ivr, + CAST(NULL AS INT) AS month_over_month_change +), + +__assert__reason_month_key_is_unique_and_present AS ( + SELECT t.reason_month_key + FROM __ref("rpt_contact_reasons_monthly") AS t + GROUP BY + t.reason_month_key + HAVING + COUNT(*) > 1 + OR t.reason_month_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/marts/support/test_rpt_support_sla_weekly__measures_breach_rates_per_channel_and_priority.sql b/benchmarks/duckshop/project/tests/unit/marts/support/test_rpt_support_sla_weekly__measures_breach_rates_per_channel_and_priority.sql new file mode 100644 index 0000000000..ee53f6e541 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/marts/support/test_rpt_support_sla_weekly__measures_breach_rates_per_channel_and_priority.sql @@ -0,0 +1,136 @@ +TEST (name "rpt_support_sla_weekly__measures_breach_rates_per_channel_and_priority"); + +WITH __ref__fct_support_tickets AS ( + SELECT + 'T1' AS ticket_id, + NULL AS customer_id, + 'email' AS channel, + 1 AS priority, + NULL AS ticket_status, + 'damaged_item' AS contact_reason, + '2026-03-09 10:00:00'::TIMESTAMPNTZ AS created_at_utc, + NULL AS solved_at, + 30 AS minutes_to_first_response, + 20 AS business_minutes_to_response, + TRUE AS is_first_response_breached, + FALSE AS is_resolution_breached, + NULL AS comment_count, + NULL AS handoff_count, + NULL AS sentiment_label, + NULL AS ticket_number_for_customer, + FALSE AS is_repeat_contact_7d, + NULL AS csat_score, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + NULL, + 'email', + 1, + NULL, + 'damaged_item', + '2026-03-10 10:00:00'::TIMESTAMPNTZ, + NULL, + 50, + 40, + FALSE, + FALSE, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'T3', + NULL, + 'email', + 1, + NULL, + 'damaged_item', + '2026-03-11 10:00:00'::TIMESTAMPNTZ, + NULL, + 10, + 100, + FALSE, + TRUE, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'T4', + NULL, + 'chat', + NULL, + NULL, + NULL, + '2026-03-12 10:00:00'::TIMESTAMPNTZ, + NULL, + 5, + NULL, + FALSE, + FALSE, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL + UNION ALL + SELECT + 'T5', + NULL, + 'email', + 1, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + FALSE, + NULL, + NULL, + NULL, + NULL, + FALSE, + NULL, + NULL +), + +__expected__rpt_support_sla_weekly AS ( + SELECT + '2026-03-09:email:1' AS sla_week_key, + 3 AS tickets, + 0.3333::DECIMAL(9, 4) AS first_response_breach_rate, + 0.3333::DECIMAL(9, 4) AS resolution_breach_rate, + 30.0::DECIMAL(10, 2) AS p50_minutes_to_first_response, + 88.0::DECIMAL(10, 2) AS p90_business_minutes_to_response + UNION ALL + SELECT + '2026-03-09:chat:?' AS sla_week_key, + 1 AS tickets, + 0.0::DECIMAL(9, 4) AS first_response_breach_rate, + 0.0::DECIMAL(9, 4) AS resolution_breach_rate, + 5.0::DECIMAL(10, 2) AS p50_minutes_to_first_response, + CAST(NULL AS DECIMAL(10, 2)) AS p90_business_minutes_to_response +), + +__assert__sla_week_key_is_unique_and_present AS ( + SELECT t.sla_week_key + FROM __ref("rpt_support_sla_weekly") AS t + GROUP BY + t.sla_week_key + HAVING + COUNT(*) > 1 + OR t.sla_week_key IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_customers__captures_the_tracked_account_attributes.sql b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_customers__captures_the_tracked_account_attributes.sql new file mode 100644 index 0000000000..43dc99ea4b --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_customers__captures_the_tracked_account_attributes.sql @@ -0,0 +1,63 @@ +TEST (name "snap_customers__captures_the_tracked_account_attributes"); + +WITH __ref__stg_customers AS ( + SELECT + 'C1' AS customer_id, + 'wren@example.com' AS email_normalised, + NULL AS phone_normalised, + NULL AS first_name, + NULL AS last_name, + 'EU' AS region_code, + 'eu-web' AS storefront_code, + NULL AS created_at, + 'de' AS preferred_language, + TRUE AS marketing_opt_in, + NULL AS consent_types, + NULL AS birth_year, + NULL AS preferences, + NULL AS source_seq, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ AS source_loaded_at + UNION ALL + SELECT + 'C2', + 'bob@example.com', + NULL, + NULL, + NULL, + 'US', + 'us-web', + NULL, + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL, + '2026-03-02 10:00:00'::TIMESTAMP_NTZ +), + +__expected__snap_customers AS ( + SELECT + 'C1' AS customer_id, + 'wren@example.com' AS email_normalised, + 'EU' AS region_code, + TRUE AS marketing_opt_in, + '2026-03-01 10:00:00'::TIMESTAMPNTZ AS source_loaded_at + UNION ALL + SELECT + 'C2' AS customer_id, + 'bob@example.com' AS email_normalised, + 'US' AS region_code, + FALSE AS marketing_opt_in, + '2026-03-02 10:00:00'::TIMESTAMPNTZ AS source_loaded_at +), + +__assert__customer_id_is_unique_and_present AS ( + SELECT t.customer_id + FROM __ref("snap_customers") AS t + GROUP BY + t.customer_id + HAVING + COUNT(*) > 1 + OR t.customer_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_loyalty_tiers__tracks_the_current_tier_per_account.sql b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_loyalty_tiers__tracks_the_current_tier_per_account.sql new file mode 100644 index 0000000000..1973febf3c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_loyalty_tiers__tracks_the_current_tier_per_account.sql @@ -0,0 +1,49 @@ +TEST (name "snap_loyalty_tiers__tracks_the_current_tier_per_account"); + +WITH __ref__stg_loyalty_accounts AS ( + SELECT + 'LA1' AS loyalty_account_id, + 'C1' AS customer_id, + 'GOLDEN_DUCK' AS tier_code, + NULL AS points_balance, + NULL AS enrolled_at, + NULL AS tier_change_count, + 'DRAKE' AS previous_tier_code, + NULL AS tier_history, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'LA2', + 'C2', + 'DUCKLING', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__snap_loyalty_tiers AS ( + SELECT + 'LA1' AS loyalty_account_id, + 'C1' AS customer_id, + 'GOLDEN_DUCK' AS tier_code + UNION ALL + SELECT + 'LA2' AS loyalty_account_id, + 'C2' AS customer_id, + 'DUCKLING' AS tier_code +), + +__assert__loyalty_account_id_is_unique_and_present AS ( + SELECT t.loyalty_account_id + FROM __ref("snap_loyalty_tiers") AS t + GROUP BY + t.loyalty_account_id + HAVING + COUNT(*) > 1 + OR t.loyalty_account_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_partner_listings__captures_listing_terms_with_load_time.sql b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_partner_listings__captures_listing_terms_with_load_time.sql new file mode 100644 index 0000000000..c917c07d0a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_partner_listings__captures_listing_terms_with_load_time.sql @@ -0,0 +1,35 @@ +TEST (name "snap_partner_listings__captures_listing_terms_with_load_time"); + +WITH __ref__stg_partner_listings AS ( + SELECT + 'L1' AS listing_id, + 'MKT' AS partner_code, + 'DUCK-MAL-01' AS sku, + 'active' AS listing_status, + NULL AS listed_at, + 12.5::DECIMAL(5, 2) AS commission_pct, + FALSE AS fulfilled_by_partner, + NULL AS terms, + NULL AS source_seq, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ AS source_loaded_at +), + +__expected__snap_partner_listings AS ( + SELECT + 'L1' AS listing_id, + 'DUCK-MAL-01' AS sku, + 'active' AS listing_status, + 12.5::DECIMAL(5, 2) AS commission_pct, + FALSE AS fulfilled_by_partner, + '2026-03-01 10:00:00'::TIMESTAMPNTZ AS source_loaded_at +), + +__assert__listing_id_is_unique_and_present AS ( + SELECT t.listing_id + FROM __ref("snap_partner_listings") AS t + GROUP BY + t.listing_id + HAVING + COUNT(*) > 1 + OR t.listing_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_price_list_history__keeps_the_last_change_per_load_and_normalises_codes.sql b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_price_list_history__keeps_the_last_change_per_load_and_normalises_codes.sql new file mode 100644 index 0000000000..e2e1dfba34 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_price_list_history__keeps_the_last_change_per_load_and_normalises_codes.sql @@ -0,0 +1,88 @@ +TEST (name "snap_price_list_history__keeps_the_last_change_per_load_and_normalises_codes"); + +WITH __source__catalogue__price_lists AS ( + SELECT + 'PL1' AS price_list_id, + ' duck-mal-01 ' AS sku, + 'eur' AS currency_code, + 1299 AS list_amount_minor, + NULL AS valid_from, + NULL AS valid_to, + NULL AS conditions, + 'I' AS _op, + 1 AS _seq, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'PL1', + ' duck-mal-01 ', + 'eur', + 1199, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'PL1', + 'duck-mal-01', + 'eur', + 1099, + NULL, + NULL, + NULL, + 'U', + 3, + '2026-03-05 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'PL2', + 'pond-pump', + 'eur', + 4999, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-03-05 10:00:00'::TIMESTAMP_NTZ +), + +__expected__snap_price_list_history AS ( + SELECT + 'PL1' AS price_list_id, + 'DUCK-MAL-01' AS sku, + 'EUR' AS currency_code, + 1199 AS list_amount_minor, + 'U' AS change_op, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ AS changed_at + UNION ALL + SELECT + 'PL1' AS price_list_id, + 'DUCK-MAL-01' AS sku, + 'EUR' AS currency_code, + 1099 AS list_amount_minor, + 'U' AS change_op, + '2026-03-05 10:00:00'::TIMESTAMP_NTZ AS changed_at + UNION ALL + SELECT + 'PL2' AS price_list_id, + 'POND-PUMP' AS sku, + 'EUR' AS currency_code, + 4999 AS list_amount_minor, + 'D' AS change_op, + '2026-03-05 10:00:00'::TIMESTAMP_NTZ AS changed_at +), + +__assert__version_key_is_unique_and_present AS ( + SELECT t.price_list_id + FROM __ref("snap_price_list_history") AS t + GROUP BY + t.price_list_id, + t.changed_at + HAVING + COUNT(*) > 1 + OR t.price_list_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_products__captures_merchandising_attributes.sql b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_products__captures_merchandising_attributes.sql new file mode 100644 index 0000000000..adcb475514 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_products__captures_merchandising_attributes.sql @@ -0,0 +1,61 @@ +TEST (name "snap_products__captures_merchandising_attributes"); + +WITH __ref__stg_products AS ( + SELECT + 'P1' AS product_id, + 'Mallard Classic Duck' AS title, + 'ducks/rubber-ducks' AS category_path, + NULL AS category_level_1, + NULL AS category_level_2, + NULL AS category_leaf, + NULL AS category_depth, + 'MALLARD' AS brand_code, + NULL AS launched_on, + TRUE AS is_active, + 4 AS spec_key_count, + NULL AS specs, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'P2', + 'Bill & Coo Duvet Cover', + 'textiles/bedding', + NULL, + NULL, + NULL, + NULL, + 'BILLCOO', + NULL, + FALSE, + NULL, + NULL, + NULL, + NULL +), + +__expected__snap_products AS ( + SELECT + 'P1' AS product_id, + 'Mallard Classic Duck' AS title, + 'ducks/rubber-ducks' AS category_path, + TRUE AS is_active, + 4 AS spec_key_count + UNION ALL + SELECT + 'P2' AS product_id, + 'Bill & Coo Duvet Cover' AS title, + 'textiles/bedding' AS category_path, + FALSE AS is_active, + CAST(NULL AS INT) AS spec_key_count +), + +__assert__product_id_is_unique_and_present AS ( + SELECT t.product_id + FROM __ref("snap_products") AS t + GROUP BY + t.product_id + HAVING + COUNT(*) > 1 + OR t.product_id IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_warehouses__captures_hours_and_capabilities.sql b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_warehouses__captures_hours_and_capabilities.sql new file mode 100644 index 0000000000..7997d30a27 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/snapshots/test_snap_warehouses__captures_hours_and_capabilities.sql @@ -0,0 +1,55 @@ +TEST (name "snap_warehouses__captures_hours_and_capabilities"); + +WITH __ref__stg_warehouses AS ( + SELECT + 'POND' AS warehouse_code, + 'The Pond' AS warehouse_name, + NULL AS country_code, + 'Europe/Berlin' AS iana_timezone, + '08:00:00'::TIME AS opens_local, + '20:00:00'::TIME AS closes_local, + NULL AS opening_minutes, + TRUE AS supports_same_day, + FALSE AS supports_cold_chain, + 31.5::DECIMAL(6, 2) AS max_parcel_kg, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'NEST', + 'The Nest', + NULL, + 'America/New_York', + '07:30:00'::TIME, + '15:30:00'::TIME, + NULL, + FALSE, + FALSE, + NULL, + NULL, + NULL +), + +__expected__snap_warehouses AS ( + SELECT + 'POND' AS warehouse_code, + '08:00:00'::TIME AS opens_local, + TRUE AS supports_same_day, + 31.5::NUMBER(6, 2) AS max_parcel_kg + UNION ALL + SELECT + 'NEST' AS warehouse_code, + '07:30:00'::TIME AS opens_local, + FALSE AS supports_same_day, + CAST(NULL AS NUMBER(6, 2)) AS max_parcel_kg +), + +__assert__warehouse_code_is_unique_and_present AS ( + SELECT t.warehouse_code + FROM __ref("snap_warehouses") AS t + GROUP BY + t.warehouse_code + HAVING + COUNT(*) > 1 + OR t.warehouse_code IS NULL +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_brands__keeps_latest_brand_and_parses_metadata.sql b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_brands__keeps_latest_brand_and_parses_metadata.sql new file mode 100644 index 0000000000..7aeb310e95 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_brands__keeps_latest_brand_and_parses_metadata.sql @@ -0,0 +1,73 @@ +TEST (name "stg_brands__keeps_latest_brand_and_parses_metadata"); + +WITH __source__catalogue__brands AS ( + SELECT + 'bw' AS brand_code, + ' Mallard & Co. ' AS brand_name, + 'de' AS country_code, + NULL AS is_private_label, + PARSE_JSON('{"founded": "1990"}')::VARIANT AS metadata, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'bw', + ' Mallard & Co. Home ', + 'de', + NULL, + PARSE_JSON('{"sustainability": {"rating": "a"}, "founded": "Est. 1998"}')::VARIANT, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'nx', + 'Bill & Coo', + 'fr', + TRUE, + PARSE_JSON('{"sustainability": {"rating": null}, "founded": "unknown"}')::VARIANT, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'old', + 'Old', + 'de', + NULL, + NULL, + 'I', + 4, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'old', + 'Old', + 'de', + NULL, + NULL, + 'D', + 5, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_brands AS ( + SELECT + 'BW' AS brand_code, + 'Mallard & Co. Home' AS brand_name, + 'DE' AS country_code, + FALSE AS is_private_label, + 'A' AS sustainability_rating, + 1998 AS founded_year, + 2 AS source_seq + UNION ALL + SELECT + 'NX' AS brand_code, + 'Bill & Coo' AS brand_name, + 'FR' AS country_code, + TRUE AS is_private_label, + CAST(NULL AS VARCHAR) AS sustainability_rating, + CAST(NULL AS INT) AS founded_year, + 3 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_bundle_components__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_bundle_components__keeps_latest_live_version.sql new file mode 100644 index 0000000000..fbcb00ea07 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_bundle_components__keeps_latest_live_version.sql @@ -0,0 +1,49 @@ +TEST (name "stg_bundle_components__keeps_latest_live_version"); + +WITH __source__catalogue__bundle_components AS ( + SELECT + 'K1' AS component_id, + NULL AS bundle_sku, + NULL AS component_sku, + NULL AS quantity, + NULL AS options, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_bundle_components AS ( + SELECT + 'K1' AS component_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_bundles__normalises_bundle_sku_and_pricing_modes.sql b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_bundles__normalises_bundle_sku_and_pricing_modes.sql new file mode 100644 index 0000000000..5501739024 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_bundles__normalises_bundle_sku_and_pricing_modes.sql @@ -0,0 +1,51 @@ +TEST (name "stg_bundles__normalises_bundle_sku_and_pricing_modes"); + +WITH __source__catalogue__bundles AS ( + SELECT + 'bnd_01 ' AS bundle_sku, + 'Paddling Six' AS title, + NULL AS active_from, + NULL AS active_to, + PARSE_JSON('{"mode": "PERCENT", "percent_off": 12.5, "currency": "eur"}')::VARIANT AS pricing, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'b2', + 'Teal Twin Pack', + NULL, + NULL, + PARSE_JSON('{"amount_minor": 4999}')::VARIANT, + 'I', + 2, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'b2', + 'Teal Twin Pack', + NULL, + NULL, + PARSE_JSON('{"amount_minor": 3999}')::VARIANT, + 'U', + 1, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_bundles AS ( + SELECT + 'BND-01' AS bundle_sku, + 'percent' AS pricing_mode, + CAST(NULL AS INT) AS fixed_amount_minor, + 12.5::DECIMAL(5, 2) AS percent_off, + 'EUR' AS pricing_currency_code, + 1 AS source_seq + UNION ALL + SELECT + 'B2' AS bundle_sku, + 'fixed' AS pricing_mode, + 4999 AS fixed_amount_minor, + CAST(NULL AS DECIMAL(5, 2)) AS percent_off, + CAST(NULL AS VARCHAR) AS pricing_currency_code, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_categories__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_categories__keeps_latest_live_version.sql new file mode 100644 index 0000000000..2684b94f8e --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_categories__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_categories__keeps_latest_live_version"); + +WITH __source__catalogue__categories AS ( + SELECT + 'K1' AS category_id, + NULL AS parent_category_id, + NULL AS slug, + NULL AS display_name, + NULL AS sort_order, + NULL AS translations, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_categories AS ( + SELECT + 'K1' AS category_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_price_lists__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_price_lists__keeps_latest_live_version.sql new file mode 100644 index 0000000000..8e5c386907 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_price_lists__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_price_lists__keeps_latest_live_version"); + +WITH __source__catalogue__price_lists AS ( + SELECT + 'K1' AS price_list_id, + NULL AS sku, + NULL AS currency_code, + NULL AS list_amount_minor, + NULL AS valid_from, + CAST(NULL AS TIMESTAMP_TZ) AS valid_to, + NULL AS conditions, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_price_lists AS ( + SELECT + 'K1' AS price_list_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_product_media__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_product_media__keeps_latest_live_version.sql new file mode 100644 index 0000000000..a45c2a4f1a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_product_media__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_product_media__keeps_latest_live_version"); + +WITH __source__catalogue__product_media AS ( + SELECT + 'K1' AS media_id, + NULL AS product_id, + NULL AS url, + NULL AS media_type, + NULL AS position, + NULL AS dimensions, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_product_media AS ( + SELECT + 'K1' AS media_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_product_variant_barcodes__drops_blank_and_shared_barcodes.sql b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_product_variant_barcodes__drops_blank_and_shared_barcodes.sql new file mode 100644 index 0000000000..510a534580 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_product_variant_barcodes__drops_blank_and_shared_barcodes.sql @@ -0,0 +1,68 @@ +TEST (name "stg_product_variant_barcodes__drops_blank_and_shared_barcodes"); + +WITH __ref__stg_product_variants AS ( + SELECT + 'SKU1' AS sku, + NULL AS product_id, + NULL AS colour, + NULL AS size_label, + NULL AS weight_grams, + NULL AS weight_kg, + NULL AS barcode_count, + NULL AS is_fragile, + NULL AS pack_size, + PARSE_JSON('["4006-3813-3393-1", "012345678905", ""]')::ARRAY AS barcodes, + NULL AS attributes, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'SKU2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('["012345678905", "96385074"]')::ARRAY, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'SKU3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('["12345", "12345"]')::ARRAY, + NULL, + NULL, + NULL +), + +__expected__stg_product_variant_barcodes AS ( + SELECT + '4006381333931' AS barcode, + 'SKU1' AS sku, + 'EAN13' AS symbology, + TRUE AS is_primary + UNION ALL + SELECT + '96385074' AS barcode, + 'SKU2' AS sku, + 'EAN8' AS symbology, + FALSE AS is_primary + UNION ALL + SELECT + '12345' AS barcode, + 'SKU3' AS sku, + 'OTHER' AS symbology, + TRUE AS is_primary +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_product_variants__derives_weight_barcodes_and_handling_flags.sql b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_product_variants__derives_weight_barcodes_and_handling_flags.sql new file mode 100644 index 0000000000..bdc74fcf26 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_product_variants__derives_weight_barcodes_and_handling_flags.sql @@ -0,0 +1,61 @@ +TEST (name "stg_product_variants__derives_weight_barcodes_and_handling_flags"); + +WITH __source__catalogue__product_variants AS ( + SELECT + 'duck 1_a' AS sku, + 'P1' AS product_id, + ' navy blue ' AS colour, + ' x l ' AS size_label, + 1250 AS weight_grams, + PARSE_JSON('["4006381333931", " 0123 "]')::ARRAY AS barcodes, + PARSE_JSON('{"handling": {"fragile": true}, "pack_size": "6"}')::VARIANT AS attributes, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'duck-2', + 'P2', + 'red', + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'duck-2', + 'P2', + 'dark red', + NULL, + NULL, + NULL, + NULL, + 'U', + 4, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_product_variants AS ( + SELECT + 'DUCK1-A' AS sku, + 'Navy Blue' AS colour, + 'XL' AS size_label, + 1.25::DECIMAL(10, 3) AS weight_kg, + 2 AS barcode_count, + TRUE AS is_fragile, + 6 AS pack_size, + 1 AS source_seq + UNION ALL + SELECT + 'DUCK-2' AS sku, + 'Dark Red' AS colour, + CAST(NULL AS VARCHAR) AS size_label, + CAST(NULL AS DECIMAL(10, 3)) AS weight_kg, + 0 AS barcode_count, + FALSE AS is_fragile, + 1 AS pack_size, + 4 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_products__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_products__keeps_latest_live_version.sql new file mode 100644 index 0000000000..b2cd4764a7 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_products__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_products__keeps_latest_live_version"); + +WITH __source__catalogue__products AS ( + SELECT + 'K1' AS product_id, + NULL AS title, + NULL AS category_path, + NULL AS brand_code, + NULL AS launched_on, + NULL AS is_active, + NULL AS specs, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_products AS ( + SELECT + 'K1' AS product_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_supplier_costs__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_supplier_costs__keeps_latest_live_version.sql new file mode 100644 index 0000000000..f7c68f7f09 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_supplier_costs__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_supplier_costs__keeps_latest_live_version"); + +WITH __source__catalogue__supplier_costs AS ( + SELECT + 'K1' AS cost_id, + NULL AS supplier_code, + NULL AS sku, + NULL AS currency_code, + NULL AS unit_cost_minor, + NULL AS effective_from, + NULL AS volume_breaks, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_supplier_costs AS ( + SELECT + 'K1' AS cost_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_suppliers__picks_primary_contact_and_parses_payment_terms.sql b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_suppliers__picks_primary_contact_and_parses_payment_terms.sql new file mode 100644 index 0000000000..6c2eb0e19b --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/catalogue/test_stg_suppliers__picks_primary_contact_and_parses_payment_terms.sql @@ -0,0 +1,42 @@ +TEST (name "stg_suppliers__picks_primary_contact_and_parses_payment_terms"); + +WITH __source__catalogue__suppliers AS ( + SELECT + 'sup1' AS supplier_code, + 'Teal Hollow Supply' AS supplier_name, + 'de' AS country_code, + PARSE_JSON('{"label": "Net 45 days", "early_discount_pct": 2}')::VARIANT AS payment_terms, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT('email', 'Ops+orders@Supplier.example.com', 'is_primary', FALSE), + OBJECT_CONSTRUCT('email', 'Boss@Supplier.example.com', 'is_primary', TRUE) + )::ARRAY AS contacts, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'sup2', + 'Sudsy Drake Supply', + 'fr', + PARSE_JSON('{"net_days": 30}')::VARIANT, + NULL, + 'I', + 2, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_suppliers AS ( + SELECT + 'SUP1' AS supplier_code, + 45 AS payment_terms_days, + 2.0::DECIMAL(5, 2) AS early_payment_discount_pct, + 'boss@supplier.example.com' AS primary_contact_email, + 2 AS contact_count + UNION ALL + SELECT + 'SUP2' AS supplier_code, + 30 AS payment_terms_days, + CAST(NULL AS DECIMAL(5, 2)) AS early_payment_discount_pct, + CAST(NULL AS VARCHAR) AS primary_contact_email, + 0 AS contact_count +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_events__parses_payload_and_keeps_latest_delivery.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_events__parses_payload_and_keeps_latest_delivery.sql new file mode 100644 index 0000000000..e3f5302bb2 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_events__parses_payload_and_keeps_latest_delivery.sql @@ -0,0 +1,57 @@ +TEST (name "stg_app_events__parses_payload_and_keeps_latest_delivery"); + +WITH __source__clickstream__app_events AS ( + SELECT + 'a1' AS event_id, + '2026-03-10 10:05:00'::TIMESTAMP_NTZ AS received_at, + 'Phone' AS platform, + 'v5.2.1 (build 9)' AS app_version, + PARSE_JSON('{"name": "old"}')::VARIANT AS payload, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'a1', + '2026-03-10 10:05:00'::TIMESTAMP_NTZ, + 'Phone', + 'v5.2.1 (build 9)', + PARSE_JSON('{"name":"Screen_View","ts":"2026-03-10 10:04:59","device":{"id":"D1"},"user_id":null,"screen":"pdp"}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'a2', + '2026-03-10 10:20:00'::TIMESTAMP_NTZ, + 'Tablet', + '6', + PARSE_JSON('{"name": "push_open", "ts_ms": 1773137400000, "device": {"id": "D2"}, "user_id": 42}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_app_events AS ( + SELECT + 'a1' AS event_id, + 'phone' AS platform, + 5 AS app_major_version, + 'screen_view' AS event_name, + '2026-03-10 10:04:59'::TIMESTAMP_NTZ AS event_at_utc, + 'D1' AS device_id, + CAST(NULL AS VARCHAR) AS customer_id, + 'pdp' AS screen_name + UNION ALL + SELECT + 'a2' AS event_id, + 'tablet' AS platform, + 6 AS app_major_version, + 'push_open' AS event_name, + '2026-03-10 10:10:00'::TIMESTAMP_NTZ AS event_at_utc, + 'D2' AS device_id, + '42' AS customer_id, + CAST(NULL AS VARCHAR) AS screen_name +), + +__assert__missing_attributes_become_empty_object AS ( + SELECT e.event_id + FROM __ref("stg_app_events") AS e + WHERE + e.event_id = 'a2' + AND ARRAY_SIZE(OBJECT_KEYS(e.attributes)) <> 0 +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_events__reads_only_rows_received_in_the_cursor_window.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_events__reads_only_rows_received_in_the_cursor_window.sql new file mode 100644 index 0000000000..b8910146ab --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_events__reads_only_rows_received_in_the_cursor_window.sql @@ -0,0 +1,33 @@ +TEST (name "stg_app_events__reads_only_rows_received_in_the_cursor_window", cursor_start "2026-03-10 10:00:00", cursor_end "2026-03-10 11:00:00"); + +WITH __source__clickstream__app_events AS ( + SELECT + 'a0' AS event_id, + '2026-03-10 09:59:59'::TIMESTAMP_NTZ AS received_at, + 'Phone' AS platform, + '5' AS app_version, + PARSE_JSON('{"name":"Screen_View","ts":"2026-03-10 10:04:59","device":{"id":"D1"},"user_id":null,"screen":"pdp"}')::VARIANT AS payload, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'a1', + '2026-03-10 10:00:00'::TIMESTAMP_NTZ, + 'Phone', + '5', + PARSE_JSON('{"name":"Screen_View","ts":"2026-03-10 10:04:59","device":{"id":"D1"},"user_id":null,"screen":"pdp"}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'a2', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ, + 'Phone', + '5', + PARSE_JSON('{"name":"Screen_View","ts":"2026-03-10 10:04:59","device":{"id":"D1"},"user_id":null,"screen":"pdp"}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_app_events AS ( + SELECT + 'a1' AS event_id, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS received_at +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_push_opens__reads_campaign_from_attributes_or_deep_link.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_push_opens__reads_campaign_from_attributes_or_deep_link.sql new file mode 100644 index 0000000000..71697fb1dd --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_push_opens__reads_campaign_from_attributes_or_deep_link.sql @@ -0,0 +1,75 @@ +TEST (name "stg_app_push_opens__reads_campaign_from_attributes_or_deep_link"); + +WITH __ref__stg_app_events AS ( + SELECT + 'e1' AS event_id, + NULL AS received_at, + 'phone' AS platform, + NULL AS app_version, + NULL AS app_major_version, + 'push_open' AS event_name, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS event_at_utc, + 'D1' AS device_id, + 'C1' AS customer_id, + NULL AS screen_name, + PARSE_JSON('{"campaign_id": "CMP1", "deep_link": "duckshop://app/sale?cid=CMP9"}')::OBJECT AS attributes + UNION ALL + SELECT + 'e2', + NULL, + 'phone', + NULL, + NULL, + 'push_open', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ, + 'D1', + NULL, + NULL, + PARSE_JSON('{"deep_link": "duckshop://app/p/duck-1?cid=SPRING-26"}')::OBJECT + UNION ALL + SELECT + 'e3', + NULL, + 'phone', + NULL, + NULL, + 'screen_view', + '2026-03-10 11:00:05'::TIMESTAMP_NTZ, + 'D1', + NULL, + NULL, + PARSE_JSON('{"deep_link": "duckshop://app/x"}')::OBJECT + UNION ALL + SELECT + 'e4', + NULL, + 'phone', + NULL, + NULL, + 'push_open', + '2026-03-10 12:00:00'::TIMESTAMP_NTZ, + 'D2', + NULL, + NULL, + PARSE_JSON('{}')::OBJECT +), + +__expected__stg_app_push_opens AS ( + SELECT + 'e1' AS event_id, + 'C1' AS customer_id, + 'CMP1' AS push_campaign_id, + '/sale?cid=CMP9' AS deep_link_path + UNION ALL + SELECT + 'e2' AS event_id, + CAST(NULL AS VARCHAR) AS customer_id, + 'SPRING-26' AS push_campaign_id, + '/p/duck-1?cid=SPRING-26' AS deep_link_path + UNION ALL + SELECT + 'e4' AS event_id, + CAST(NULL AS VARCHAR) AS customer_id, + CAST(NULL AS VARCHAR) AS push_campaign_id, + CAST(NULL AS VARCHAR) AS deep_link_path +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_screen_views__caps_time_on_screen_at_thirty_minutes.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_screen_views__caps_time_on_screen_at_thirty_minutes.sql new file mode 100644 index 0000000000..c2cbb1da9d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_app_screen_views__caps_time_on_screen_at_thirty_minutes.sql @@ -0,0 +1,107 @@ +TEST (name "stg_app_screen_views__caps_time_on_screen_at_thirty_minutes"); + +WITH __ref__stg_app_events AS ( + SELECT + 's1' AS event_id, + NULL AS received_at, + 'phone' AS platform, + NULL AS app_version, + NULL AS app_major_version, + 'screen_view' AS event_name, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS event_at_utc, + 'D1' AS device_id, + NULL AS customer_id, + 'product_detail' AS screen_name, + PARSE_JSON('{"screen_class": "PdpVC"}')::OBJECT AS attributes + UNION ALL + SELECT + 'e2', + NULL, + 'phone', + NULL, + NULL, + 'push_open', + '2026-03-10 10:00:30'::TIMESTAMP_NTZ, + 'D1', + NULL, + NULL, + NULL + UNION ALL + SELECT + 's3', + NULL, + 'phone', + NULL, + NULL, + 'screen_view', + '2026-03-10 10:01:00'::TIMESTAMP_NTZ, + 'D1', + NULL, + 'cart', + NULL + UNION ALL + SELECT + 's4', + NULL, + 'phone', + NULL, + NULL, + 'screen_view', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ, + 'D1', + NULL, + 'home', + NULL + UNION ALL + SELECT + 's5', + NULL, + 'phone', + NULL, + NULL, + 'screen_view', + '2026-03-10 12:00:00'::TIMESTAMP_NTZ, + 'D1', + NULL, + 'home', + NULL + UNION ALL + SELECT + 's6', + NULL, + 'phone', + NULL, + NULL, + 'screen_view', + '2026-03-10 12:00:00'::TIMESTAMP_NTZ, + NULL, + NULL, + 'home', + NULL +), + +__expected__stg_app_screen_views AS ( + SELECT + 's1' AS event_id, + 'Product Detail' AS screen_name, + 30 AS seconds_on_screen, + 'PdpVC' AS screen_class + UNION ALL + SELECT + 's3' AS event_id, + 'Cart' AS screen_name, + 1800 AS seconds_on_screen, + CAST(NULL AS VARCHAR) AS screen_class + UNION ALL + SELECT + 's4' AS event_id, + 'Home' AS screen_name, + 1800 AS seconds_on_screen, + CAST(NULL AS VARCHAR) AS screen_class + UNION ALL + SELECT + 's5' AS event_id, + 'Home' AS screen_name, + CAST(NULL AS INT) AS seconds_on_screen, + CAST(NULL AS VARCHAR) AS screen_class +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_email_events__normalises_recipients_and_detects_machine_opens.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_email_events__normalises_recipients_and_detects_machine_opens.sql new file mode 100644 index 0000000000..3475f8b041 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_email_events__normalises_recipients_and_detects_machine_opens.sql @@ -0,0 +1,69 @@ +TEST (name "stg_email_events__normalises_recipients_and_detects_machine_opens"); + +WITH __source__clickstream__email_events AS ( + SELECT + 'm1' AS email_event_id, + 'M1' AS message_id, + 'SENT' AS event_type, + 'x@example.com' AS recipient_email, + 'CMP1' AS campaign_id, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS occurred_at, + PARSE_JSON('{}')::VARIANT AS details, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'm1', + 'M1', + 'CLICK', + ' Shopper+news@Example.com ', + 'CMP1', + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + PARSE_JSON('{"url": "https://shop.example.com/p/1"}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'm2', + 'M1', + 'Open', + 'shopper@example.com', + 'CMP1', + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + PARSE_JSON('{"user_agent": "Mail Image Proxy 1.0"}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'm3', + 'M1', + 'Bounce', + 'not-an-email', + 'CMP1', + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + PARSE_JSON('{"bounce": {"type": "HARD"}, "machine_open": false}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_email_events AS ( + SELECT + 'm1' AS email_event_id, + 'click' AS event_type, + 'shopper@example.com' AS recipient_email, + 'https://shop.example.com/p/1' AS clicked_url, + CAST(NULL AS VARCHAR) AS bounce_type, + CAST(NULL AS BOOLEAN) AS is_machine_open + UNION ALL + SELECT + 'm2' AS email_event_id, + 'open' AS event_type, + 'shopper@example.com' AS recipient_email, + CAST(NULL AS VARCHAR) AS clicked_url, + CAST(NULL AS VARCHAR) AS bounce_type, + TRUE AS is_machine_open + UNION ALL + SELECT + 'm3' AS email_event_id, + 'bounce' AS event_type, + CAST(NULL AS VARCHAR) AS recipient_email, + CAST(NULL AS VARCHAR) AS clicked_url, + 'hard' AS bounce_type, + FALSE AS is_machine_open +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_experiment_assignments__keeps_first_assignment_per_visitor.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_experiment_assignments__keeps_first_assignment_per_visitor.sql new file mode 100644 index 0000000000..f20c50330a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_experiment_assignments__keeps_first_assignment_per_visitor.sql @@ -0,0 +1,49 @@ +TEST (name "stg_experiment_assignments__keeps_first_assignment_per_visitor"); + +WITH __source__clickstream__experiment_assignments AS ( + SELECT + 'A1' AS assignment_id, + 'Checkout_V2' AS experiment_key, + 'Control' AS variant_key, + 'ANON1' AS anonymous_id, + NULL AS customer_id, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS assigned_at, + PARSE_JSON('{"surface": "app"}')::VARIANT AS context, + NULL AS _loaded_at + UNION ALL + SELECT + 'A2', + 'Checkout_V2', + 'Treatment', + 'anon1', + NULL, + '2026-03-10 11:00:00 +00:00'::TIMESTAMP_TZ, + PARSE_JSON('{}')::VARIANT, + NULL + UNION ALL + SELECT + 'A3', + 'Checkout_V2', + 'Treatment', + 'ANON2', + NULL, + '2026-03-10 09:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL +), + +__expected__stg_experiment_assignments AS ( + SELECT + 'A1' AS assignment_id, + 'checkout_v2' AS experiment_key, + 'control' AS variant_key, + 'anon1' AS anonymous_id, + 'app' AS assignment_surface + UNION ALL + SELECT + 'A3' AS assignment_id, + 'checkout_v2' AS experiment_key, + 'treatment' AS variant_key, + 'anon2' AS anonymous_id, + 'web' AS assignment_surface +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_search_queries__cleans_query_text_and_reads_filters.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_search_queries__cleans_query_text_and_reads_filters.sql new file mode 100644 index 0000000000..01d12b2b41 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_search_queries__cleans_query_text_and_reads_filters.sql @@ -0,0 +1,52 @@ +TEST (name "stg_search_queries__cleans_query_text_and_reads_filters"); + +WITH __source__clickstream__search_queries AS ( + SELECT + 'Q1' AS query_id, + 'ANON1' AS anonymous_id, + 'red' AS query_text, + 1 AS results_count, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS searched_at, + NULL AS filters, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'Q1', + 'ANON1', + ' Red decoy!! 2L ', + 7, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + PARSE_JSON('{"category": "Decoys", "price": {"max_minor": 5000}, "sort": "price_asc"}')::VARIANT, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'Q2', + 'anon2', + 'towel', + 0, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_search_queries AS ( + SELECT + 'Q1' AS query_id, + 'anon1' AS anonymous_id, + 'red decoy 2l' AS query_text, + 3 AS query_token_count, + 7 AS results_count, + 'decoys' AS filter_category, + 5000 AS filter_max_price_minor, + 'price_asc' AS sort_order + UNION ALL + SELECT + 'Q2' AS query_id, + 'anon2' AS anonymous_id, + 'towel' AS query_text, + 1 AS query_token_count, + 0 AS results_count, + CAST(NULL AS VARCHAR) AS filter_category, + CAST(NULL AS INT) AS filter_max_price_minor, + 'relevance' AS sort_order +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_cart_events__explodes_cart_items_with_signed_quantities.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_cart_events__explodes_cart_items_with_signed_quantities.sql new file mode 100644 index 0000000000..f7a31a308f --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_cart_events__explodes_cart_items_with_signed_quantities.sql @@ -0,0 +1,93 @@ +TEST (name "stg_web_cart_events__explodes_cart_items_with_signed_quantities"); + +WITH __ref__stg_web_events AS ( + SELECT + 'e1' AS event_id, + NULL AS received_at, + 'add_to_cart' AS event_type, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS event_at_utc, + 'v1' AS anonymous_id, + NULL AS customer_id, + NULL AS user_email, + NULL AS page_url, + NULL AS page_path, + NULL AS referrer_host, + NULL AS user_agent, + NULL AS locale, + NULL AS client_timezone, + FALSE AS is_bot, + PARSE_JSON('{"currency": "eur"}')::OBJECT AS properties, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'sku', + 'duck_1', + 'quantity', + 2, + 'price_minor', + 500 + ), + OBJECT_CONSTRUCT('sku', ' duck 2 ', 'price_minor', 900) + )::ARRAY AS items + UNION ALL + SELECT + 'e2', + NULL, + 'remove_from_cart', + '2026-03-10 10:05:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[{"sku": "DUCK-1"}]')::ARRAY + UNION ALL + SELECT + 'e3', + NULL, + 'page_view', + '2026-03-10 10:06:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[{"sku": "DUCK-9"}]')::ARRAY +), + +__expected__stg_web_cart_events AS ( + SELECT + 'e1:0' AS cart_event_key, + 'add' AS action, + 'DUCK-1' AS sku, + 2 AS quantity_delta, + 500 AS unit_amount_minor, + 'EUR' AS currency_code + UNION ALL + SELECT + 'e1:1' AS cart_event_key, + 'add' AS action, + 'DUCK2' AS sku, + 1 AS quantity_delta, + 900 AS unit_amount_minor, + 'EUR' AS currency_code + UNION ALL + SELECT + 'e2:0' AS cart_event_key, + 'remove' AS action, + 'DUCK-1' AS sku, + -1 AS quantity_delta, + CAST(NULL AS INT) AS unit_amount_minor, + CAST(NULL AS VARCHAR) AS currency_code +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_checkout_steps__keeps_first_arrival_at_each_step.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_checkout_steps__keeps_first_arrival_at_each_step.sql new file mode 100644 index 0000000000..2f6114a1c6 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_checkout_steps__keeps_first_arrival_at_each_step.sql @@ -0,0 +1,134 @@ +TEST (name "stg_web_checkout_steps__keeps_first_arrival_at_each_step"); + +WITH __ref__stg_web_events AS ( + SELECT + 'c1' AS event_id, + NULL AS received_at, + 'checkout_started' AS event_type, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS event_at_utc, + 'v1' AS anonymous_id, + NULL AS customer_id, + NULL AS user_email, + 'https://shop.example.com/checkout/step-1' AS page_url, + '/checkout/step-1' AS page_path, + NULL AS referrer_host, + NULL AS user_agent, + NULL AS locale, + NULL AS client_timezone, + FALSE AS is_bot, + PARSE_JSON('{}')::OBJECT AS properties, + PARSE_JSON('[]')::ARRAY AS items + UNION ALL + SELECT + 'c2', + NULL, + 'checkout_step', + '2026-03-10 10:05:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + 'https://shop.example.com/checkout', + '/checkout', + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{"step": 2, "step_name": "Shipping"}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'c3', + NULL, + 'checkout_step', + '2026-03-10 10:06:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + 'https://shop.example.com/checkout', + '/checkout', + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{"step": 2, "step_name": "Shipping"}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'c4', + NULL, + 'checkout_step', + '2026-03-10 10:10:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + 'https://shop.example.com/checkout', + '/checkout', + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{"step": 3, "payment_option": "Card"}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'c5', + NULL, + 'checkout_step', + '2026-03-10 10:11:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + 'https://shop.example.com/checkout/review', + '/checkout/review', + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'c6', + NULL, + 'checkout_step', + '2026-03-10 10:12:00'::TIMESTAMP_NTZ, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{"step": 1}')::OBJECT, + PARSE_JSON('[]')::ARRAY +), + +__expected__stg_web_checkout_steps AS ( + SELECT + 'v1#1' AS checkout_step_key, + 1 AS step_number, + CAST(NULL AS VARCHAR) AS step_name, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS first_reached_at_utc, + CAST(NULL AS VARCHAR) AS payment_option + UNION ALL + SELECT + 'v1#2' AS checkout_step_key, + 2 AS step_number, + 'shipping' AS step_name, + '2026-03-10 10:05:00'::TIMESTAMP_NTZ AS first_reached_at_utc, + CAST(NULL AS VARCHAR) AS payment_option + UNION ALL + SELECT + 'v1#3' AS checkout_step_key, + 3 AS step_number, + CAST(NULL AS VARCHAR) AS step_name, + '2026-03-10 10:10:00'::TIMESTAMP_NTZ AS first_reached_at_utc, + 'card' AS payment_option +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_events__flattens_tracker_payload_and_flags_bots.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_events__flattens_tracker_payload_and_flags_bots.sql new file mode 100644 index 0000000000..b4499e73d8 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_events__flattens_tracker_payload_and_flags_bots.sql @@ -0,0 +1,112 @@ +TEST (name "stg_web_events__flattens_tracker_payload_and_flags_bots"); + +WITH __source__clickstream__web_events AS ( + SELECT + 'w1' AS event_id, + '2026-03-10 10:10:05'::TIMESTAMP_NTZ AS received_at, + PARSE_JSON('{"type": "stale"}')::VARIANT AS event, + '2026-03-10 10:11:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'w1', + '2026-03-10 10:10:05'::TIMESTAMP_NTZ, + OBJECT_CONSTRUCT_KEEP_NULL( + 'type', + 'Add To Cart', + 'ts_ms', + 1773137400000, + 'anonymous_id', + 'ANON1', + 'user', + OBJECT_CONSTRUCT_KEEP_NULL('id', NULL, 'email', 'A+x@Example.com'), + 'page', + OBJECT_CONSTRUCT( + 'url', + 'https://shop.example.com/p/duck-1?ref=x', + 'referrer', + 'https://WWW.Search.example.com/q' + ), + 'context', + OBJECT_CONSTRUCT( + 'ua', + 'Browser/5.0', + 'locale', + 'de_DE', + 'tz', + 'Europe/Berlin' + ), + 'items', + ARRAY_CONSTRUCT(OBJECT_CONSTRUCT('sku', 'duck-1')) + )::VARIANT, + '2026-03-10 10:12:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'w2', + '2026-03-10 10:11:05'::TIMESTAMP_NTZ, + OBJECT_CONSTRUCT( + 'type', + 'page_view', + 'ts_ms', + 1773137460000, + 'anonymous_id', + 'anon2', + 'user', + OBJECT_CONSTRUCT('id', 'C7'), + 'context', + OBJECT_CONSTRUCT('ua', 'ExampleBot/2.1') + )::VARIANT, + '2026-03-10 10:12:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'w3', + '2026-03-10 10:12:05'::TIMESTAMP_NTZ, + OBJECT_CONSTRUCT('type', 'page_view', 'anonymous_id', 'anon3')::VARIANT, + '2026-03-10 10:13:00'::TIMESTAMP_NTZ +), + +__expected__stg_web_events AS ( + SELECT + 'w1' AS event_id, + 'add_to_cart' AS event_type, + '2026-03-10 10:10:00'::TIMESTAMP_NTZ AS event_at_utc, + 'anon1' AS anonymous_id, + CAST(NULL AS VARCHAR) AS customer_id, + 'a@example.com' AS user_email, + '/p/duck-1' AS page_path, + 'www.search.example.com' AS referrer_host, + 'de-DE' AS locale, + 'Europe/Berlin' AS client_timezone, + FALSE AS is_bot + UNION ALL + SELECT + 'w2' AS event_id, + 'page_view' AS event_type, + '2026-03-10 10:11:00'::TIMESTAMP_NTZ AS event_at_utc, + 'anon2' AS anonymous_id, + 'C7' AS customer_id, + CAST(NULL AS VARCHAR) AS user_email, + CAST(NULL AS VARCHAR) AS page_path, + CAST(NULL AS VARCHAR) AS referrer_host, + CAST(NULL AS VARCHAR) AS locale, + CAST(NULL AS VARCHAR) AS client_timezone, + TRUE AS is_bot + UNION ALL + SELECT + 'w3' AS event_id, + 'page_view' AS event_type, + '2026-03-10 10:12:05'::TIMESTAMP_NTZ AS event_at_utc, + 'anon3' AS anonymous_id, + CAST(NULL AS VARCHAR) AS customer_id, + CAST(NULL AS VARCHAR) AS user_email, + CAST(NULL AS VARCHAR) AS page_path, + CAST(NULL AS VARCHAR) AS referrer_host, + CAST(NULL AS VARCHAR) AS locale, + CAST(NULL AS VARCHAR) AS client_timezone, + FALSE AS is_bot +), + +__assert__missing_items_become_empty_arrays AS ( + SELECT e.event_id + FROM __ref("stg_web_events") AS e + WHERE ARRAY_SIZE(e.items) <> IFF(e.event_id = 'w1', 1, 0) +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_identifies__links_visitors_to_customers_with_methods.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_identifies__links_visitors_to_customers_with_methods.sql new file mode 100644 index 0000000000..1a53e2f0ac --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_identifies__links_visitors_to_customers_with_methods.sql @@ -0,0 +1,136 @@ +TEST (name "stg_web_identifies__links_visitors_to_customers_with_methods"); + +WITH __ref__stg_web_events AS ( + SELECT + 'i1' AS event_id, + NULL AS received_at, + 'login' AS event_type, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ AS event_at_utc, + 'v1' AS anonymous_id, + 'C1' AS customer_id, + NULL AS user_email, + NULL AS page_url, + NULL AS page_path, + NULL AS referrer_host, + NULL AS user_agent, + NULL AS locale, + NULL AS client_timezone, + FALSE AS is_bot, + PARSE_JSON('{"method": "password"}')::OBJECT AS properties, + PARSE_JSON('[]')::ARRAY AS items + UNION ALL + SELECT + 'i2', + NULL, + 'order_completed', + '2026-03-05 10:00:00'::TIMESTAMP_NTZ, + 'v1', + 'C1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'i3', + NULL, + 'identify', + '2026-03-03 10:00:00'::TIMESTAMP_NTZ, + 'v1', + 'C1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{"method": "password"}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'i4', + NULL, + 'signup', + '2026-03-06 10:00:00'::TIMESTAMP_NTZ, + 'v1', + 'C2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'i5', + NULL, + 'login', + '2026-03-06 10:00:00'::TIMESTAMP_NTZ, + 'v2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'i6', + NULL, + 'page_view', + '2026-03-06 10:00:00'::TIMESTAMP_NTZ, + 'v1', + 'C1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY +), + +__expected__stg_web_identifies AS ( + SELECT + 'v1>C1' AS identity_link_key, + '2026-03-01 10:00:00'::TIMESTAMP_NTZ AS first_seen_at_utc, + '2026-03-05 10:00:00'::TIMESTAMP_NTZ AS last_seen_at_utc, + 3 AS event_count + UNION ALL + SELECT + 'v1>C2' AS identity_link_key, + '2026-03-06 10:00:00'::TIMESTAMP_NTZ AS first_seen_at_utc, + '2026-03-06 10:00:00'::TIMESTAMP_NTZ AS last_seen_at_utc, + 1 AS event_count +), + +__assert__identify_methods_are_distinct_and_sorted AS ( + SELECT i.identity_link_key + FROM __ref("stg_web_identifies") AS i + WHERE + ARRAY_TO_STRING(i.identify_methods, ',') <> IFF( + i.customer_id = 'C1', + 'order_completed,password', + 'signup' + ) +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_page_views__classifies_pages_and_reads_utm_parameters.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_page_views__classifies_pages_and_reads_utm_parameters.sql new file mode 100644 index 0000000000..acceaf4525 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_page_views__classifies_pages_and_reads_utm_parameters.sql @@ -0,0 +1,176 @@ +TEST (name "stg_web_page_views__classifies_pages_and_reads_utm_parameters"); + +WITH __ref__stg_web_events AS ( + SELECT + 'p1' AS event_id, + NULL AS received_at, + 'page_view' AS event_type, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS event_at_utc, + 'v1' AS anonymous_id, + NULL AS customer_id, + NULL AS user_email, + 'https://shop.example.com/?utm_source=news&utm_medium=email&utm_campaign=spring#top' AS page_url, + '/' AS page_path, + NULL AS referrer_host, + NULL AS user_agent, + NULL AS locale, + NULL AS client_timezone, + FALSE AS is_bot, + PARSE_JSON('{"title": "Home"}')::OBJECT AS properties, + PARSE_JSON('[]')::ARRAY AS items + UNION ALL + SELECT + 'p2', + NULL, + 'page_view', + '2026-03-10 10:01:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + 'https://shop.example.com/products/duck-1', + '/products/duck-1', + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'p3', + NULL, + 'page_view', + '2026-03-10 10:02:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + 'https://shop.example.com/c/decoys', + '/c/decoys', + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'p4', + NULL, + 'page_view', + '2026-03-10 10:03:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + 'https://shop.example.com/checkout/step-1', + '/checkout/step-1', + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'p5', + NULL, + 'page_view', + '2026-03-10 10:04:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + 'https://shop.example.com/about', + '/about', + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'p6', + NULL, + 'page_view', + '2026-03-10 10:05:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + 'https://shop.example.com/', + '/', + NULL, + NULL, + NULL, + NULL, + TRUE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'p7', + NULL, + 'add_to_cart', + '2026-03-10 10:06:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + 'https://shop.example.com/products/duck-1', + '/products/duck-1', + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY +), + +__expected__stg_web_page_views AS ( + SELECT + 'p1' AS event_id, + 'home' AS page_type, + CAST(NULL AS VARCHAR) AS previous_page_path, + 'news' AS utm_source, + 'email' AS utm_medium, + 'spring' AS utm_campaign, + 'Home' AS page_title + UNION ALL + SELECT + 'p2' AS event_id, + 'product' AS page_type, + '/' AS previous_page_path, + CAST(NULL AS VARCHAR) AS utm_source, + CAST(NULL AS VARCHAR) AS utm_medium, + CAST(NULL AS VARCHAR) AS utm_campaign, + CAST(NULL AS VARCHAR) AS page_title + UNION ALL + SELECT + 'p3' AS event_id, + 'category' AS page_type, + '/products/duck-1' AS previous_page_path, + CAST(NULL AS VARCHAR) AS utm_source, + CAST(NULL AS VARCHAR) AS utm_medium, + CAST(NULL AS VARCHAR) AS utm_campaign, + CAST(NULL AS VARCHAR) AS page_title + UNION ALL + SELECT + 'p4' AS event_id, + 'checkout' AS page_type, + '/c/decoys' AS previous_page_path, + CAST(NULL AS VARCHAR) AS utm_source, + CAST(NULL AS VARCHAR) AS utm_medium, + CAST(NULL AS VARCHAR) AS utm_campaign, + CAST(NULL AS VARCHAR) AS page_title + UNION ALL + SELECT + 'p5' AS event_id, + 'content' AS page_type, + '/checkout/step-1' AS previous_page_path, + CAST(NULL AS VARCHAR) AS utm_source, + CAST(NULL AS VARCHAR) AS utm_medium, + CAST(NULL AS VARCHAR) AS utm_campaign, + CAST(NULL AS VARCHAR) AS page_title +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_product_views__keeps_one_view_per_visitor_sku_and_minute.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_product_views__keeps_one_view_per_visitor_sku_and_minute.sql new file mode 100644 index 0000000000..c42c92d12e --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_product_views__keeps_one_view_per_visitor_sku_and_minute.sql @@ -0,0 +1,109 @@ +TEST (name "stg_web_product_views__keeps_one_view_per_visitor_sku_and_minute"); + +WITH __ref__stg_web_events AS ( + SELECT + 'e1' AS event_id, + NULL AS received_at, + 'product_view' AS event_type, + '2026-03-10 10:00:10'::TIMESTAMP_NTZ AS event_at_utc, + 'v1' AS anonymous_id, + NULL AS customer_id, + NULL AS user_email, + NULL AS page_url, + NULL AS page_path, + NULL AS referrer_host, + NULL AS user_agent, + NULL AS locale, + NULL AS client_timezone, + FALSE AS is_bot, + PARSE_JSON('{"sku": "duck_1", "list": {"name": "Search", "position": 3}}')::OBJECT AS properties, + PARSE_JSON('[]')::ARRAY AS items + UNION ALL + SELECT + 'e2', + NULL, + 'product_view', + '2026-03-10 10:00:50'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{"sku": "DUCK-1"}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'e3', + NULL, + 'product_view', + '2026-03-10 10:01:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{"sku": "DUCK-1"}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'e4', + NULL, + 'product_view', + '2026-03-10 10:02:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 'e5', + NULL, + 'product_view', + '2026-03-10 10:03:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + TRUE, + PARSE_JSON('{"sku": "DUCK-1"}')::OBJECT, + PARSE_JSON('[]')::ARRAY +), + +__expected__stg_web_product_views AS ( + SELECT + 'v1|DUCK-1|202603101000' AS product_view_key, + 'DUCK-1' AS sku, + '2026-03-10 10:00:00'::TIMESTAMPNTZ AS viewed_minute_utc, + 'Search' AS list_name, + 3 AS list_position + UNION ALL + SELECT + 'v1|DUCK-1|202603101001' AS product_view_key, + 'DUCK-1' AS sku, + '2026-03-10 10:01:00'::TIMESTAMPNTZ AS viewed_minute_utc, + CAST(NULL AS VARCHAR) AS list_name, + CAST(NULL AS INT) AS list_position +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_searches__normalises_terms_and_flags_zero_results.sql b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_searches__normalises_terms_and_flags_zero_results.sql new file mode 100644 index 0000000000..5f23b420a7 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/clickstream/test_stg_web_searches__normalises_terms_and_flags_zero_results.sql @@ -0,0 +1,95 @@ +TEST (name "stg_web_searches__normalises_terms_and_flags_zero_results"); + +WITH __ref__stg_web_events AS ( + SELECT + 's1' AS event_id, + NULL AS received_at, + 'search' AS event_type, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS event_at_utc, + 'v1' AS anonymous_id, + NULL AS customer_id, + NULL AS user_email, + NULL AS page_url, + NULL AS page_path, + NULL AS referrer_host, + NULL AS user_agent, + NULL AS locale, + NULL AS client_timezone, + FALSE AS is_bot, + PARSE_JSON('{"query": " Red Decoy ", "results": 12}')::OBJECT AS properties, + PARSE_JSON('[]')::ARRAY AS items + UNION ALL + SELECT + 's2', + NULL, + 'search', + '2026-03-10 10:01:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{"query": "xyz", "results": 0}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 's3', + NULL, + 'search', + '2026-03-10 10:02:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{"query": "towel"}')::OBJECT, + PARSE_JSON('[]')::ARRAY + UNION ALL + SELECT + 's4', + NULL, + 'page_view', + '2026-03-10 10:03:00'::TIMESTAMP_NTZ, + 'v1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + FALSE, + PARSE_JSON('{"query": "ignored"}')::OBJECT, + PARSE_JSON('[]')::ARRAY +), + +__expected__stg_web_searches AS ( + SELECT + 's1' AS event_id, + 'red decoy' AS search_term, + 12 AS result_count, + FALSE AS is_zero_result + UNION ALL + SELECT + 's2' AS event_id, + 'xyz' AS search_term, + 0 AS result_count, + TRUE AS is_zero_result + UNION ALL + SELECT + 's3' AS event_id, + 'towel' AS search_term, + CAST(NULL AS INT) AS result_count, + TRUE AS is_zero_result +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_chargebacks__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_chargebacks__keeps_latest_live_version.sql new file mode 100644 index 0000000000..4cdd4f9077 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_chargebacks__keeps_latest_live_version.sql @@ -0,0 +1,65 @@ +TEST (name "stg_chargebacks__keeps_latest_live_version"); + +WITH __source__finance__chargebacks AS ( + SELECT + 'K1' AS chargeback_id, + NULL AS payment_id, + NULL AS amount_minor, + NULL AS currency_code, + NULL AS reason_code, + NULL AS opened_at, + NULL AS closed_at, + NULL AS outcome, + NULL AS evidence, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_chargebacks AS ( + SELECT + 'K1' AS chargeback_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_fx_rates__inverts_rates_and_links_next_rate_date.sql b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_fx_rates__inverts_rates_and_links_next_rate_date.sql new file mode 100644 index 0000000000..97c34a9acb --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_fx_rates__inverts_rates_and_links_next_rate_date.sql @@ -0,0 +1,38 @@ +TEST (name "stg_fx_rates__inverts_rates_and_links_next_rate_date"); + +WITH __source__finance__fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-03-01'::DATE AS rate_date, + 1.25::NUMBER(18, 8) AS rate_to_usd + UNION ALL + SELECT + 'EUR', + '2026-03-02'::DATE, + 1.1::NUMBER(18, 8) + UNION ALL + SELECT + 'JPY', + '2026-03-01'::DATE, + 0.0067::NUMBER(18, 8) +), + +__expected__stg_fx_rates AS ( + SELECT + 'EUR' AS currency_code, + '2026-03-01'::DATE AS rate_date, + 0.8::DECIMAL(18, 8) AS usd_to_rate, + '2026-03-02'::DATE AS next_rate_date + UNION ALL + SELECT + 'EUR' AS currency_code, + '2026-03-02'::DATE AS rate_date, + 0.90909091::DECIMAL(18, 8) AS usd_to_rate, + CAST(NULL AS DATE) AS next_rate_date + UNION ALL + SELECT + 'JPY' AS currency_code, + '2026-03-01'::DATE AS rate_date, + 149.25373134::DECIMAL(18, 8) AS usd_to_rate, + CAST(NULL AS DATE) AS next_rate_date +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_invoice_tax_lines__explodes_embedded_tax_lines.sql b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_invoice_tax_lines__explodes_embedded_tax_lines.sql new file mode 100644 index 0000000000..828e55e7d4 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_invoice_tax_lines__explodes_embedded_tax_lines.sql @@ -0,0 +1,73 @@ +TEST (name "stg_invoice_tax_lines__explodes_embedded_tax_lines"); + +WITH __ref__stg_invoices AS ( + SELECT + 'IV1' AS invoice_id, + NULL AS order_id, + NULL AS invoice_number, + NULL AS invoice_series, + NULL AS issued_at, + NULL AS currency_code, + NULL AS net_minor, + NULL AS tax_minor, + NULL AS gross_minor, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'code', + 'vat_std', + 'rate_pct', + 19, + 'taxable_minor', + 1000, + 'tax_minor', + 190 + ), + OBJECT_CONSTRUCT('code', 'eco', 'tax_minor', 5) + )::ARRAY AS tax_lines, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'IV2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL + UNION ALL + SELECT + 'IV3', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('[]')::ARRAY, + NULL, + NULL +), + +__expected__stg_invoice_tax_lines AS ( + SELECT + 'IV1:0' AS invoice_tax_line_key, + 'VAT_STD' AS tax_code, + 19.0::DECIMAL(9, 4) AS tax_rate_pct, + 1000 AS taxable_minor, + 190 AS tax_minor + UNION ALL + SELECT + 'IV1:1' AS invoice_tax_line_key, + 'ECO' AS tax_code, + CAST(NULL AS DECIMAL(9, 4)) AS tax_rate_pct, + CAST(NULL AS INT) AS taxable_minor, + 5 AS tax_minor +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_invoices__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_invoices__keeps_latest_live_version.sql new file mode 100644 index 0000000000..de6c33dfb9 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_invoices__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_invoices__keeps_latest_live_version"); + +WITH __source__finance__invoices AS ( + SELECT + 'K1' AS invoice_id, + NULL AS order_id, + NULL AS invoice_number, + NULL AS issued_at, + NULL AS currency_code, + NULL AS totals, + NULL AS tax_lines, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_invoices AS ( + SELECT + 'K1' AS invoice_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_ledger_entries__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_ledger_entries__keeps_latest_live_version.sql new file mode 100644 index 0000000000..3537461ad5 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_ledger_entries__keeps_latest_live_version.sql @@ -0,0 +1,61 @@ +TEST (name "stg_ledger_entries__keeps_latest_live_version"); + +WITH __source__finance__ledger_entries AS ( + SELECT + 'K1' AS entry_id, + NULL AS journal_code, + NULL AS account_code, + NULL AS debit_minor, + NULL AS credit_minor, + NULL AS currency_code, + NULL AS posted_at, + NULL AS reference, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_ledger_entries AS ( + SELECT + 'K1' AS entry_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_payout_lines__adds_a_fee_line_for_each_charged_payout.sql b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_payout_lines__adds_a_fee_line_for_each_charged_payout.sql new file mode 100644 index 0000000000..c7c4ec5b85 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_payout_lines__adds_a_fee_line_for_each_charged_payout.sql @@ -0,0 +1,67 @@ +TEST (name "stg_payout_lines__adds_a_fee_line_for_each_charged_payout"); + +WITH __ref__stg_payouts AS ( + SELECT + 'PO1' AS payout_id, + NULL AS provider_code, + NULL AS currency_code, + NULL AS amount_minor, + 25 AS fee_minor, + NULL AS net_minor, + NULL AS paid_out_on, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'payment_id', + 'PM1', + 'gross_minor', + 1000, + 'fee_minor', + 30 + ), + OBJECT_CONSTRUCT( + 'payment_id', + 'PM2', + 'type', + 'Refund', + 'gross_minor', + 200 + ) + )::ARRAY AS lines, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'PO2', + NULL, + NULL, + NULL, + 0, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__stg_payout_lines AS ( + SELECT + 'PO1:0' AS payout_line_key, + 'PM1' AS payment_id, + 'charge' AS line_type, + 1000 AS gross_minor, + 30 AS fee_minor + UNION ALL + SELECT + 'PO1:1' AS payout_line_key, + 'PM2' AS payment_id, + 'refund' AS line_type, + 200 AS gross_minor, + 0 AS fee_minor + UNION ALL + SELECT + 'PO1:fee' AS payout_line_key, + CAST(NULL AS VARCHAR) AS payment_id, + 'payout_fee' AS line_type, + 0 AS gross_minor, + 25 AS fee_minor +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_payouts__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_payouts__keeps_latest_live_version.sql new file mode 100644 index 0000000000..5988eed865 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_payouts__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_payouts__keeps_latest_live_version"); + +WITH __source__finance__payouts AS ( + SELECT + 'K1' AS payout_id, + NULL AS provider_code, + NULL AS currency_code, + NULL AS amount_minor, + NULL AS fee_minor, + NULL AS paid_out_on, + NULL AS lines, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_payouts AS ( + SELECT + 'K1' AS payout_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_settlement_reports__keeps_only_parseable_object_reports.sql b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_settlement_reports__keeps_only_parseable_object_reports.sql new file mode 100644 index 0000000000..a8ff950fc3 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_settlement_reports__keeps_only_parseable_object_reports.sql @@ -0,0 +1,56 @@ +TEST (name "stg_settlement_reports__keeps_only_parseable_object_reports"); + +WITH __source__finance__settlement_reports AS ( + SELECT + 'R1' AS report_id, + 'payco' AS provider_code, + '2026-03-10'::DATE AS report_date, + '{}' AS report_body, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'R1', + 'payco', + '2026-03-10'::DATE, + '{"summary": {"currency": "eur", "gross_minor": 9700, "fees_minor": 300}, "lines": [{}, {}]}', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'R2', + 'payco', + '2026-03-10'::DATE, + 'not json', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'R3', + 'payco', + '2026-03-10'::DATE, + '[1, 2]', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'R4', + 'payco', + '2026-03-11'::DATE, + '{"summary": {}}', + '2026-03-11 11:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_settlement_reports AS ( + SELECT + 'R1' AS report_id, + 'PAYCO' AS provider_code, + 'EUR' AS currency_code, + 9700 AS gross_minor, + 300 AS fees_minor, + 2 AS line_count + UNION ALL + SELECT + 'R4' AS report_id, + 'PAYCO' AS provider_code, + CAST(NULL AS VARCHAR) AS currency_code, + CAST(NULL AS INT) AS gross_minor, + CAST(NULL AS INT) AS fees_minor, + CAST(NULL AS INT) AS line_count +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_tax_rates__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_tax_rates__keeps_latest_live_version.sql new file mode 100644 index 0000000000..e9809be0de --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/finance/test_stg_tax_rates__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_tax_rates__keeps_latest_live_version"); + +WITH __source__finance__tax_rates AS ( + SELECT + 'K1' AS tax_rate_id, + NULL AS country_code, + NULL AS tax_region, + NULL AS tax_category, + NULL AS rate_pct, + NULL AS valid_from, + NULL AS valid_to, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_tax_rates AS ( + SELECT + 'K1' AS tax_rate_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_carrier_rate_cards__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_carrier_rate_cards__keeps_latest_live_version.sql new file mode 100644 index 0000000000..5b994e29b4 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_carrier_rate_cards__keeps_latest_live_version.sql @@ -0,0 +1,65 @@ +TEST (name "stg_carrier_rate_cards__keeps_latest_live_version"); + +WITH __source__fulfilment__carrier_rate_cards AS ( + SELECT + 'K1' AS rate_card_id, + NULL AS carrier_code, + NULL AS service_level, + NULL AS origin_country, + NULL AS destination_country, + NULL AS currency_code, + NULL AS valid_from, + NULL AS valid_to, + NULL AS bands, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_carrier_rate_cards AS ( + SELECT + 'K1' AS rate_card_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_carrier_webhook_errors__captures_unparseable_bodies.sql b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_carrier_webhook_errors__captures_unparseable_bodies.sql new file mode 100644 index 0000000000..4dcbfd6e2d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_carrier_webhook_errors__captures_unparseable_bodies.sql @@ -0,0 +1,54 @@ +TEST (name "stg_carrier_webhook_errors__captures_unparseable_bodies"); + +WITH __source__fulfilment__carrier_webhooks AS ( + SELECT + 'W1' AS webhook_id, + 'flapwing' AS carrier_code, + '2026-03-10 10:30:00'::TIMESTAMP_NTZ AS received_at, + '{"event": "x"' AS body, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'W2', + 'flapwing', + '2026-03-10 10:30:00'::TIMESTAMP_NTZ, + '{"ok": true}', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'W3', + 'flapwing', + '2026-03-10 10:30:00'::TIMESTAMP_NTZ, + 'Error 502 Bad gateway', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'W4', + 'flapwing', + '2026-03-10 10:30:00'::TIMESTAMP_NTZ, + 'broken', + '2026-03-10 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'W4', + 'flapwing', + '2026-03-10 10:30:00'::TIMESTAMP_NTZ, + '{"ok": true}', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_carrier_webhook_errors AS ( + SELECT + 'W1' AS webhook_id, + 'FLAPWING' AS carrier_code, + 13 AS body_length, + '{"event": "x"' AS body_prefix, + TRUE AS looks_truncated + UNION ALL + SELECT + 'W3' AS webhook_id, + 'FLAPWING' AS carrier_code, + 25 AS body_length, + 'Error 502 Bad gateway' AS body_prefix, + FALSE AS looks_truncated +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_carrier_webhooks__splits_batched_updates_and_reads_both_payload_shapes.sql b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_carrier_webhooks__splits_batched_updates_and_reads_both_payload_shapes.sql new file mode 100644 index 0000000000..3aa5a3f76f --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_carrier_webhooks__splits_batched_updates_and_reads_both_payload_shapes.sql @@ -0,0 +1,85 @@ +TEST (name "stg_carrier_webhooks__splits_batched_updates_and_reads_both_payload_shapes"); + +WITH __source__fulfilment__carrier_webhooks AS ( + SELECT + 'H1' AS webhook_id, + 'flapwing' AS carrier_code, + '2026-03-10 10:30:00'::TIMESTAMP_NTZ AS received_at, + TO_JSON(OBJECT_CONSTRUCT( + 'tracking_number', + 'tr-001 23', + 'event_id', + '77', + 'status', + OBJECT_CONSTRUCT( + 'code', + 'delivered', + 'at', + '2026-03-10 10:00:00 +01:00' + ) + )) AS body, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'H2', + 'flapwing', + '2026-03-10 10:30:00'::TIMESTAMP_NTZ, + TO_JSON(OBJECT_CONSTRUCT( + 'events', + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'trackingNumber', + 'TR9', + 'event', + 'ofd', + 'timestamp', + '2026-03-10 08:00:00 +00:00' + ), + OBJECT_CONSTRUCT( + 'trackingNumber', + 'TR9', + 'event', + 'pod', + 'event_id', + 'x1' + ) + ) + )), + '2026-03-10 11:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'H3', + 'flapwing', + '2026-03-10 10:30:00'::TIMESTAMP_NTZ, + 'garbage', + '2026-03-10 11:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_carrier_webhooks AS ( + SELECT + 'H1:0' AS carrier_update_key, + 'FLAPWING' AS carrier_code, + 'TR00123' AS tracking_number, + 'DELIVERED' AS status_code, + '2026-03-10 10:00:00 +01:00'::TIMESTAMP_TZ AS status_at, + 77::DECIMAL(38, 0) AS carrier_event_id, + FALSE AS is_batched + UNION ALL + SELECT + 'H2:0' AS carrier_update_key, + 'FLAPWING' AS carrier_code, + 'TR9' AS tracking_number, + 'OFD' AS status_code, + '2026-03-10 08:00:00 +00:00'::TIMESTAMP_TZ AS status_at, + CAST(NULL AS DECIMAL(38, 0)) AS carrier_event_id, + TRUE AS is_batched + UNION ALL + SELECT + 'H2:1' AS carrier_update_key, + 'FLAPWING' AS carrier_code, + 'TR9' AS tracking_number, + 'POD' AS status_code, + CAST(NULL AS TIMESTAMP_TZ) AS status_at, + CAST(NULL AS DECIMAL(38, 0)) AS carrier_event_id, + TRUE AS is_batched +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_delivery_exceptions__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_delivery_exceptions__keeps_latest_live_version.sql new file mode 100644 index 0000000000..592afa8aae --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_delivery_exceptions__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_delivery_exceptions__keeps_latest_live_version"); + +WITH __source__fulfilment__delivery_exceptions AS ( + SELECT + 'K1' AS exception_id, + NULL AS shipment_id, + NULL AS exception_code, + NULL AS exception_text, + NULL AS occurred_at, + NULL AS resolution, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_delivery_exceptions AS ( + SELECT + 'K1' AS exception_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_delivery_slots__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_delivery_slots__keeps_latest_live_version.sql new file mode 100644 index 0000000000..545c481f9d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_delivery_slots__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_delivery_slots__keeps_latest_live_version"); + +WITH __source__fulfilment__delivery_slots AS ( + SELECT + 'K1' AS slot_id, + NULL AS warehouse_code, + NULL AS slot_start_local, + NULL AS slot_end_local, + NULL AS capacity, + NULL AS booked, + NULL AS restrictions, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_delivery_slots AS ( + SELECT + 'K1' AS slot_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_pick_tasks__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_pick_tasks__keeps_latest_live_version.sql new file mode 100644 index 0000000000..06d2ce9655 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_pick_tasks__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_pick_tasks__keeps_latest_live_version"); + +WITH __source__fulfilment__pick_tasks AS ( + SELECT + 'K1' AS task_id, + NULL AS order_id, + NULL AS warehouse_code, + NULL AS picker_id, + NULL AS started_at_local, + NULL AS completed_at_local, + NULL AS lines, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_pick_tasks AS ( + SELECT + 'K1' AS task_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_return_inspections__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_return_inspections__keeps_latest_live_version.sql new file mode 100644 index 0000000000..f2137946f8 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_return_inspections__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_return_inspections__keeps_latest_live_version"); + +WITH __source__fulfilment__return_inspections AS ( + SELECT + 'K1' AS inspection_id, + NULL AS return_id, + NULL AS warehouse_code, + NULL AS grade, + NULL AS notes, + NULL AS inspected_at_local, + NULL AS findings, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_return_inspections AS ( + SELECT + 'K1' AS inspection_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_returns__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_returns__keeps_latest_live_version.sql new file mode 100644 index 0000000000..9a9cef8c00 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_returns__keeps_latest_live_version.sql @@ -0,0 +1,69 @@ +TEST (name "stg_returns__keeps_latest_live_version"); + +WITH __source__fulfilment__returns AS ( + SELECT + 'K1' AS return_id, + NULL AS order_id, + NULL AS sku, + NULL AS qty, + NULL AS reason_text, + NULL AS return_status, + NULL AS requested_at, + NULL AS received_at, + NULL AS photos, + NULL AS refund_preference, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_returns AS ( + SELECT + 'K1' AS return_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_shipment_packages__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_shipment_packages__keeps_latest_live_version.sql new file mode 100644 index 0000000000..cbd21df0df --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_shipment_packages__keeps_latest_live_version.sql @@ -0,0 +1,49 @@ +TEST (name "stg_shipment_packages__keeps_latest_live_version"); + +WITH __source__fulfilment__shipment_packages AS ( + SELECT + 'K1' AS package_id, + NULL AS shipment_id, + NULL AS weight_grams, + NULL AS dimensions, + NULL AS contents, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_shipment_packages AS ( + SELECT + 'K1' AS package_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_shipment_tracking_events__merges_scans_and_exceptions.sql b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_shipment_tracking_events__merges_scans_and_exceptions.sql new file mode 100644 index 0000000000..57b41c5267 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_shipment_tracking_events__merges_scans_and_exceptions.sql @@ -0,0 +1,92 @@ +TEST (name "stg_shipment_tracking_events__merges_scans_and_exceptions"); + +WITH __ref__stg_shipments AS ( + SELECT + 'S1' AS shipment_id, + NULL AS order_id, + NULL AS warehouse_code, + NULL AS carrier_code, + NULL AS service_level, + NULL AS tracking_number, + NULL AS shipped_at, + NULL AS delivered_at, + NULL AS tracking_event_count, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'code', + 'picked-up', + 'ts', + '2026-03-10 09:00:00 +01:00', + 'location', + 'berlin hub, de' + ), + OBJECT_CONSTRUCT( + 'code', + 'OUT_FOR_DELIVERY', + 'ts', + '11/03/2026 08:00 +0100', + 'location', + 'Munich' + ) + )::ARRAY AS tracking_events, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'S2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__ref__stg_delivery_exceptions AS ( + SELECT + 'X1' AS exception_id, + 'S1' AS shipment_id, + 'DAMAGED' AS exception_code, + NULL AS exception_text, + '2026-03-11 12:00:00 +00:00'::TIMESTAMP_TZ AS occurred_at, + NULL AS resolution_type, + NULL AS resolved_at, + NULL AS compensation_minor, + NULL AS resolution, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__stg_shipment_tracking_events AS ( + SELECT + 'S1:0' AS tracking_event_key, + 'scan' AS event_source, + 'PICKEDUP' AS event_code, + '2026-03-10 09:00:00 +01:00'::TIMESTAMP_TZ AS event_at, + 'Berlin Hub' AS location_text, + 'DE' AS location_country_code, + 0 AS event_sequence + UNION ALL + SELECT + 'S1:1' AS tracking_event_key, + 'scan' AS event_source, + 'OUT_FOR_DELIVERY' AS event_code, + '2026-03-11 08:00:00 +01:00'::TIMESTAMP_TZ AS event_at, + 'Munich' AS location_text, + CAST(NULL AS VARCHAR) AS location_country_code, + 1 AS event_sequence + UNION ALL + SELECT + 'S1:x:X1' AS tracking_event_key, + 'exception' AS event_source, + 'EXCEPTION_DAMAGED' AS event_code, + '2026-03-11 12:00:00 +00:00'::TIMESTAMP_TZ AS event_at, + CAST(NULL AS VARCHAR) AS location_text, + CAST(NULL AS VARCHAR) AS location_country_code, + CAST(NULL AS INT) AS event_sequence +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_shipments__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_shipments__keeps_latest_live_version.sql new file mode 100644 index 0000000000..adebd6877d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/fulfilment/test_stg_shipments__keeps_latest_live_version.sql @@ -0,0 +1,61 @@ +TEST (name "stg_shipments__keeps_latest_live_version"); + +WITH __source__fulfilment__shipments AS ( + SELECT + 'K1' AS shipment_id, + NULL AS order_id, + NULL AS warehouse_code, + NULL AS carrier_code, + NULL AS service_level, + NULL AS shipped_at, + NULL AS delivered_at, + NULL AS tracking, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_shipments AS ( + SELECT + 'K1' AS shipment_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_affiliate_conversions__reads_click_time_and_sub_id_from_payload.sql b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_affiliate_conversions__reads_click_time_and_sub_id_from_payload.sql new file mode 100644 index 0000000000..f1a2589023 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_affiliate_conversions__reads_click_time_and_sub_id_from_payload.sql @@ -0,0 +1,95 @@ +TEST (name "stg_affiliate_conversions__reads_click_time_and_sub_id_from_payload"); + +WITH __source__growth__affiliate_conversions AS ( + SELECT + 'CV1' AS conversion_id, + 'awn-x' AS affiliate_code, + 'O1' AS order_id, + 500 AS commission_minor, + 'eur' AS currency_code, + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ AS converted_at, + OBJECT_CONSTRUCT( + 'clicked_at', + '2026-03-09 10:00:00 +00:00', + 'landing_url', + 'https://shop.example.com/?subid=social_pf&x=1' + )::VARIANT AS click_payload, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'CV1', + 'awn-x', + 'O1', + 600, + 'eur', + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ, + OBJECT_CONSTRUCT( + 'clicked_at', + '2026-03-09 10:00:00 +00:00', + 'landing_url', + 'https://shop.example.com/?subid=social_pf&x=1' + )::VARIANT, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'CV2', + 'pn_kw', + 'O2', + 100, + 'kwd', + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ, + PARSE_JSON('{}')::VARIANT, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'CV3', + 'pn_kw', + 'O3', + 100, + 'kwd', + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'I', + 4, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'CV3', + 'pn_kw', + 'O3', + 100, + 'kwd', + '2026-03-10 12:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'D', + 5, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_affiliate_conversions AS ( + SELECT + 'CV1' AS conversion_id, + 'AWN-X' AS affiliate_code, + 600 AS commission_minor, + 'EUR' AS currency_code, + '2026-03-09 10:00:00 +00:00'::TIMESTAMP_TZ AS clicked_at, + 26 AS click_to_conversion_hours, + 'social_pf' AS sub_id, + 2 AS source_seq + UNION ALL + SELECT + 'CV2' AS conversion_id, + 'PN_KW' AS affiliate_code, + 100 AS commission_minor, + 'KWD' AS currency_code, + CAST(NULL AS TIMESTAMP_TZ) AS clicked_at, + CAST(NULL AS INT) AS click_to_conversion_hours, + CAST(NULL AS VARCHAR) AS sub_id, + 3 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_campaign_spend__unpivots_metrics_and_parses_formatted_numbers.sql b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_campaign_spend__unpivots_metrics_and_parses_formatted_numbers.sql new file mode 100644 index 0000000000..3b46209886 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_campaign_spend__unpivots_metrics_and_parses_formatted_numbers.sql @@ -0,0 +1,104 @@ +TEST (name "stg_campaign_spend__unpivots_metrics_and_parses_formatted_numbers"); + +WITH __source__growth__campaign_spend AS ( + SELECT + 'SP1' AS spend_id, + '2026-03-10'::DATE AS spend_date, + 'social' AS channel, + 'CMP1' AS campaign_id, + 'usd' AS currency_code, + OBJECT_CONSTRUCT( + 'account', + OBJECT_CONSTRUCT('id', 'ACC1'), + 'metrics', + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'name', + 'spend', + 'value', + '1,234.50' + ), + OBJECT_CONSTRUCT('name', 'linkClicks', 'value', 20), + OBJECT_CONSTRUCT('name', 'impressions', 'value', 1000) + ) + )::VARIANT AS report, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'SP2', + '2026-03-10'::DATE, + 'social', + 'CMP1', + 'usd', + OBJECT_CONSTRUCT( + 'account', + OBJECT_CONSTRUCT('id', 'ACC1'), + 'metrics', + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'name', + 'spend', + 'value', + '1,234.50' + ), + OBJECT_CONSTRUCT('name', 'linkClicks', 'value', 20), + OBJECT_CONSTRUCT('name', 'impressions', 'value', 1000) + ) + )::VARIANT, + 'I', + 2, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'SP2', + '2026-03-10'::DATE, + 'social', + 'CMP1', + 'usd', + OBJECT_CONSTRUCT( + 'account', + OBJECT_CONSTRUCT('id', 'ACC1'), + 'metrics', + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'name', + 'spend', + 'value', + '1,234.50' + ), + OBJECT_CONSTRUCT('name', 'linkClicks', 'value', 20), + OBJECT_CONSTRUCT('name', 'impressions', 'value', 1000) + ) + )::VARIANT, + 'D', + 3, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_campaign_spend AS ( + SELECT + 'SP1:spend' AS spend_metric_key, + 'SOCIAL' AS channel, + 'USD' AS currency_code, + 'spend' AS metric_name, + 1234.5::DECIMAL(38, 6) AS metric_value, + 'ACC1' AS platform_account_id + UNION ALL + SELECT + 'SP1:linkclicks' AS spend_metric_key, + 'SOCIAL' AS channel, + 'USD' AS currency_code, + 'link_clicks' AS metric_name, + 20.0::DECIMAL(38, 6) AS metric_value, + 'ACC1' AS platform_account_id + UNION ALL + SELECT + 'SP1:impressions' AS spend_metric_key, + 'SOCIAL' AS channel, + 'USD' AS currency_code, + 'impressions' AS metric_name, + 1000.0::DECIMAL(38, 6) AS metric_value, + 'ACC1' AS platform_account_id +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_campaigns__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_campaigns__keeps_latest_live_version.sql new file mode 100644 index 0000000000..77d26608ac --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_campaigns__keeps_latest_live_version.sql @@ -0,0 +1,65 @@ +TEST (name "stg_campaigns__keeps_latest_live_version"); + +WITH __source__growth__campaigns AS ( + SELECT + 'K1' AS campaign_id, + NULL AS campaign_name, + NULL AS channel, + NULL AS objective, + NULL AS starts_on, + NULL AS ends_on, + NULL AS budget_minor, + NULL AS currency_code, + NULL AS targeting, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_campaigns AS ( + SELECT + 'K1' AS campaign_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_influencer_posts__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_influencer_posts__keeps_latest_live_version.sql new file mode 100644 index 0000000000..a2e58a3b8b --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_influencer_posts__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_influencer_posts__keeps_latest_live_version"); + +WITH __source__growth__influencer_posts AS ( + SELECT + 'K1' AS post_id, + NULL AS creator_handle, + NULL AS platform, + NULL AS posted_at, + NULL AS promo_code, + NULL AS metrics, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_influencer_posts AS ( + SELECT + 'K1' AS post_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_newsletter_subscriptions__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_newsletter_subscriptions__keeps_latest_live_version.sql new file mode 100644 index 0000000000..5f276d536c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_newsletter_subscriptions__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_newsletter_subscriptions__keeps_latest_live_version"); + +WITH __source__growth__newsletter_subscriptions AS ( + SELECT + 'K1' AS list_subscription_id, + NULL AS email, + NULL AS list_code, + NULL AS list_status, + NULL AS subscribed_at, + NULL AS unsubscribed_at, + NULL AS signup_source, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_newsletter_subscriptions AS ( + SELECT + 'K1' AS list_subscription_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_referral_invites__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_referral_invites__keeps_latest_live_version.sql new file mode 100644 index 0000000000..73fe8322c4 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_referral_invites__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_referral_invites__keeps_latest_live_version"); + +WITH __source__growth__referral_invites AS ( + SELECT + 'K1' AS invite_id, + NULL AS referrer_customer_id, + NULL AS invitee_email, + NULL AS sent_at, + NULL AS accepted_at, + NULL AS reward, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_referral_invites AS ( + SELECT + 'K1' AS invite_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_survey_responses__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_survey_responses__keeps_latest_live_version.sql new file mode 100644 index 0000000000..c0886fd1f6 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/growth/test_stg_survey_responses__keeps_latest_live_version.sql @@ -0,0 +1,49 @@ +TEST (name "stg_survey_responses__keeps_latest_live_version"); + +WITH __source__growth__survey_responses AS ( + SELECT + 'K1' AS response_id, + NULL AS survey_code, + NULL AS customer_id, + NULL AS submitted_at, + NULL AS answers, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_survey_responses AS ( + SELECT + 'K1' AS response_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_bin_locations__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_bin_locations__keeps_latest_live_version.sql new file mode 100644 index 0000000000..8f89fa572b --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_bin_locations__keeps_latest_live_version.sql @@ -0,0 +1,45 @@ +TEST (name "stg_bin_locations__keeps_latest_live_version"); + +WITH __source__inventory__bin_locations AS ( + SELECT + 'K1' AS bin_id, + NULL AS warehouse_code, + NULL AS bin_label, + NULL AS attributes, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_bin_locations AS ( + SELECT + 'K1' AS bin_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_po_receipts__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_po_receipts__keeps_latest_live_version.sql new file mode 100644 index 0000000000..cd369b1e7e --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_po_receipts__keeps_latest_live_version.sql @@ -0,0 +1,61 @@ +TEST (name "stg_po_receipts__keeps_latest_live_version"); + +WITH __source__inventory__po_receipts AS ( + SELECT + 'K1' AS receipt_id, + NULL AS po_id, + NULL AS sku, + NULL AS warehouse_code, + NULL AS received_qty, + NULL AS damaged_qty, + NULL AS received_at_local, + NULL AS discrepancies, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_po_receipts AS ( + SELECT + 'K1' AS receipt_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_purchase_order_lines__explodes_po_lines_with_currency_override.sql b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_purchase_order_lines__explodes_po_lines_with_currency_override.sql new file mode 100644 index 0000000000..ef370164df --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_purchase_order_lines__explodes_po_lines_with_currency_override.sql @@ -0,0 +1,66 @@ +TEST (name "stg_purchase_order_lines__explodes_po_lines_with_currency_override"); + +WITH __ref__stg_purchase_orders AS ( + SELECT + 'PO1' AS po_id, + NULL AS supplier_code, + NULL AS warehouse_code, + 'EUR' AS currency_code, + NULL AS po_status, + NULL AS ordered_at, + NULL AS expected_at, + NULL AS received_at, + NULL AS line_count, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'sku', + 'duck_mal 01', + 'qty', + 24, + 'unit_cost_minor', + 150 + ), + OBJECT_CONSTRUCT( + 'sku', + 'DUCK-TEAL-02', + 'qty', + 6, + 'unit_cost', + 900, + 'currency', + 'gbp' + ) + )::ARRAY AS lines, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'PO2', + NULL, + NULL, + 'EUR', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__stg_purchase_order_lines AS ( + SELECT + 'PO1-000' AS po_line_key, + 'DUCK-MAL01' AS sku, + 24 AS ordered_qty, + 150 AS unit_cost_minor, + 'EUR' AS currency_code + UNION ALL + SELECT + 'PO1-001' AS po_line_key, + 'DUCK-TEAL-02' AS sku, + 6 AS ordered_qty, + 900 AS unit_cost_minor, + 'GBP' AS currency_code +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_purchase_orders__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_purchase_orders__keeps_latest_live_version.sql new file mode 100644 index 0000000000..90fae6b3cc --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_purchase_orders__keeps_latest_live_version.sql @@ -0,0 +1,65 @@ +TEST (name "stg_purchase_orders__keeps_latest_live_version"); + +WITH __source__inventory__purchase_orders AS ( + SELECT + 'K1' AS po_id, + NULL AS supplier_code, + NULL AS warehouse_code, + NULL AS currency_code, + NULL AS po_status, + NULL AS ordered_at, + NULL AS expected_at, + NULL AS received_at, + NULL AS lines, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_purchase_orders AS ( + SELECT + 'K1' AS po_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_counts__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_counts__keeps_latest_live_version.sql new file mode 100644 index 0000000000..b019935e01 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_counts__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_stock_counts__keeps_latest_live_version"); + +WITH __source__inventory__stock_counts AS ( + SELECT + 'K1' AS count_id, + NULL AS sku, + NULL AS warehouse_code, + NULL AS counted_qty, + NULL AS counted_at_local, + NULL AS counter, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_stock_counts AS ( + SELECT + 'K1' AS count_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_movements__drops_voided_movements_and_splits_references.sql b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_movements__drops_voided_movements_and_splits_references.sql new file mode 100644 index 0000000000..d1e070e747 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_movements__drops_voided_movements_and_splits_references.sql @@ -0,0 +1,80 @@ +TEST (name "stg_stock_movements__drops_voided_movements_and_splits_references"); + +WITH __source__inventory__stock_movements AS ( + SELECT + 'M1' AS movement_id, + 'duck-mal-01' AS sku, + 'pond' AS warehouse_code, + 20 AS qty_delta, + 'receipt' AS reason_code, + 'PO:PO1' AS reference, + '2026-03-10 09:00:00'::TIMESTAMP_NTZ AS moved_at_local, + PARSE_JSON('{}')::VARIANT AS metadata, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'M1', + 'duck-mal-01', + 'pond', + 24, + 'receipt', + 'PO:PO1', + '2026-03-10 09:00:00'::TIMESTAMP_NTZ, + PARSE_JSON('{"operator": "OP7"}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'M2', + 'duck-mal-01', + 'pond', + -2, + 'damage', + 'ADJ:', + '2026-03-10 09:00:00'::TIMESTAMP_NTZ, + PARSE_JSON('{"operator": null}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'M3', + 'duck-mal-01', + 'pond', + -5, + 'pick', + 'ORD:O1', + '2026-03-10 09:00:00'::TIMESTAMP_NTZ, + PARSE_JSON('{}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'M4', + 'duck-mal-01', + 'pond', + 5, + 'void', + 'VOID:M3', + '2026-03-10 09:00:00'::TIMESTAMP_NTZ, + PARSE_JSON('{}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_stock_movements AS ( + SELECT + 'M1' AS movement_id, + 'DUCK-MAL-01' AS sku, + 'POND' AS warehouse_code, + 24 AS qty_delta, + 'RECEIPT' AS reason_code, + 'PO' AS reference_type, + 'PO1' AS reference_id, + 'OP7' AS operator_id + UNION ALL + SELECT + 'M2' AS movement_id, + 'DUCK-MAL-01' AS sku, + 'POND' AS warehouse_code, + -2 AS qty_delta, + 'DAMAGE' AS reason_code, + 'ADJ' AS reference_type, + CAST(NULL AS VARCHAR) AS reference_id, + CAST(NULL AS VARCHAR) AS operator_id +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_reservations__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_reservations__keeps_latest_live_version.sql new file mode 100644 index 0000000000..b759ac15e0 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_reservations__keeps_latest_live_version.sql @@ -0,0 +1,61 @@ +TEST (name "stg_stock_reservations__keeps_latest_live_version"); + +WITH __source__inventory__stock_reservations AS ( + SELECT + 'K1' AS reservation_id, + NULL AS order_id, + NULL AS sku, + NULL AS warehouse_code, + NULL AS qty, + NULL AS reserved_at, + NULL AS released_at, + NULL AS allocation, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_stock_reservations AS ( + SELECT + 'K1' AS reservation_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_transfers__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_transfers__keeps_latest_live_version.sql new file mode 100644 index 0000000000..9939accd1d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_stock_transfers__keeps_latest_live_version.sql @@ -0,0 +1,61 @@ +TEST (name "stg_stock_transfers__keeps_latest_live_version"); + +WITH __source__inventory__stock_transfers AS ( + SELECT + 'K1' AS transfer_id, + NULL AS sku, + NULL AS from_warehouse_code, + NULL AS to_warehouse_code, + NULL AS qty, + NULL AS shipped_at, + NULL AS arrived_at, + NULL AS tracking, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_stock_transfers AS ( + SELECT + 'K1' AS transfer_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_warehouses__derives_opening_minutes_and_capabilities.sql b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_warehouses__derives_opening_minutes_and_capabilities.sql new file mode 100644 index 0000000000..bc4b9296e9 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/inventory/test_stg_warehouses__derives_opening_minutes_and_capabilities.sql @@ -0,0 +1,85 @@ +TEST (name "stg_warehouses__derives_opening_minutes_and_capabilities"); + +WITH __source__inventory__warehouses AS ( + SELECT + 'pond' AS warehouse_code, + 'The Pond' AS warehouse_name, + 'de' AS country_code, + 'Europe/Berlin' AS iana_timezone, + '08:00:00'::TIME AS opens_local, + '20:00:00'::TIME AS closes_local, + PARSE_JSON('{"services": ["same_day", "returns"], "cold_chain": false, "limits": {"max_parcel_kg": 31.5}}')::VARIANT AS capabilities, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'nest', + 'Nest Annex', + 'us', + 'America/New_York', + '07:30:00'::TIME, + '15:30:00'::TIME, + NULL, + 'I', + 2, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'nest', + 'The Nest', + 'us', + 'America/New_York', + '07:30:00'::TIME, + '15:30:00'::TIME, + NULL, + 'U', + 3, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'reed', + 'The Reedbed', + 'gb', + 'Europe/London', + '08:00:00'::TIME, + '16:00:00'::TIME, + NULL, + 'I', + 4, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'reed', + 'The Reedbed', + 'gb', + 'Europe/London', + '08:00:00'::TIME, + '16:00:00'::TIME, + NULL, + 'D', + 5, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_warehouses AS ( + SELECT + 'POND' AS warehouse_code, + 'The Pond' AS warehouse_name, + 'DE' AS country_code, + 720 AS opening_minutes, + TRUE AS supports_same_day, + FALSE AS supports_cold_chain, + 31.5::DECIMAL(6, 2) AS max_parcel_kg, + 1 AS source_seq + UNION ALL + SELECT + 'NEST' AS warehouse_code, + 'The Nest' AS warehouse_name, + 'US' AS country_code, + 480 AS opening_minutes, + FALSE AS supports_same_day, + FALSE AS supports_cold_chain, + CAST(NULL AS DECIMAL(6, 2)) AS max_parcel_kg, + 3 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_listings__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_listings__keeps_latest_live_version.sql new file mode 100644 index 0000000000..de3a238966 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_listings__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_partner_listings__keeps_latest_live_version"); + +WITH __source__partners__partner_listings AS ( + SELECT + 'K1' AS listing_id, + NULL AS partner_code, + NULL AS sku, + NULL AS listing_status, + NULL AS listed_at, + NULL AS terms, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_partner_listings AS ( + SELECT + 'K1' AS listing_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_order_lines__prices_v1_lines_in_minor_units_and_reads_v2_items.sql b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_order_lines__prices_v1_lines_in_minor_units_and_reads_v2_items.sql new file mode 100644 index 0000000000..1acb532c0d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_order_lines__prices_v1_lines_in_minor_units_and_reads_v2_items.sql @@ -0,0 +1,118 @@ +TEST (name "stg_partner_order_lines__prices_v1_lines_in_minor_units_and_reads_v2_items"); + +WITH __ref__stg_partner_orders_v1 AS ( + SELECT + 'MKT:A1' AS partner_order_key, + NULL AS partner_code, + NULL AS external_order_id, + NULL AS buyer_email, + 'EUR' AS currency_code, + NULL AS order_total, + NULL AS placed_at, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'sku', + 'duck_mal 01', + 'qty', + 2, + 'price', + '12.345' + ) + )::ARRAY AS lines + UNION ALL + SELECT + 'MKT:A2', + NULL, + NULL, + NULL, + 'KWD', + NULL, + NULL, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'sku', + 'BATH-EID-1', + 'qty', + 1, + 'price', + '3.5' + ) + )::ARRAY + UNION ALL + SELECT + 'MKT:A3', + NULL, + NULL, + NULL, + 'XXX', + NULL, + NULL, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'sku', + 'X', + 'qty', + 1, + 'price', + '1' + ) + )::ARRAY +), + +__ref__stg_partner_orders_v2 AS ( + SELECT + 'MKT:B7' AS partner_order_key, + NULL AS partner_code, + NULL AS external_order_id, + NULL AS buyer_email_hash, + 'USD' AS currency_code, + NULL AS order_total_minor, + NULL AS placed_at_utc, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'seller_sku', + 'pond-pump', + 'quantity', + 1, + 'unit_price_minor', + 4999 + ) + )::ARRAY AS items +), + +__seed__currencies AS ( + SELECT + 'EUR' AS currency_code, + NULL AS currency_name, + 2 AS minor_unit_exponent, + NULL AS symbol + UNION ALL + SELECT + 'KWD', + NULL, + 3, + NULL +), + +__expected__stg_partner_order_lines AS ( + SELECT + 'MKT:A1#0' AS partner_order_line_key, + 1 AS payload_version, + 'DUCK-MAL01' AS sku, + 2 AS quantity, + 1235::DECIMAL(38, 0) AS unit_amount_minor + UNION ALL + SELECT + 'MKT:A2#0' AS partner_order_line_key, + 1 AS payload_version, + 'BATH-EID-1' AS sku, + 1 AS quantity, + 3500::DECIMAL(38, 0) AS unit_amount_minor + UNION ALL + SELECT + 'MKT:B7#0' AS partner_order_line_key, + 2 AS payload_version, + 'POND-PUMP' AS sku, + 1 AS quantity, + 4999::DECIMAL(38, 0) AS unit_amount_minor +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_orders__keeps_latest_message_per_order_and_skips_json_nulls.sql b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_orders__keeps_latest_message_per_order_and_skips_json_nulls.sql new file mode 100644 index 0000000000..9a4d62bad0 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_orders__keeps_latest_message_per_order_and_skips_json_nulls.sql @@ -0,0 +1,63 @@ +TEST (name "stg_partner_orders__keeps_latest_message_per_order_and_skips_json_nulls"); + +WITH __source__partners__partner_orders AS ( + SELECT + 'MS1' AS message_id, + 'mkt' AS partner_code, + 1 AS schema_version, + '2026-03-10 10:00:00'::TIMESTAMP_NTZ AS received_at, + PARSE_JSON('{"orderId": "A1"}')::VARIANT AS payload, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'MS2', + 'mkt', + 1, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ, + PARSE_JSON('{"orderId": "A1", "status": "paid"}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'MS3', + 'mkt', + 2, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ, + PARSE_JSON('{"order": {"id": "B7"}}')::VARIANT, + '2026-03-10 11:30:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'MS3', + 'mkt', + 2, + '2026-03-10 11:00:00'::TIMESTAMP_NTZ, + PARSE_JSON('{"order": {"id": "B7"}, "v": 2}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'MS4', + 'mkt', + 2, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ, + PARSE_JSON('{"orderId": null, "order": {"id": "C9"}}')::VARIANT, + '2026-03-10 12:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_partner_orders AS ( + SELECT + 'MS2' AS message_id, + 'MKT' AS partner_code, + 1 AS schema_version, + 'A1' AS external_order_id + UNION ALL + SELECT + 'MS3' AS message_id, + 'MKT' AS partner_code, + 2 AS schema_version, + 'B7' AS external_order_id + UNION ALL + SELECT + 'MS4' AS message_id, + 'MKT' AS partner_code, + 2 AS schema_version, + 'C9' AS external_order_id +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_orders_v1__parses_schema_one_payloads.sql b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_orders_v1__parses_schema_one_payloads.sql new file mode 100644 index 0000000000..212d064549 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_orders_v1__parses_schema_one_payloads.sql @@ -0,0 +1,75 @@ +TEST (name "stg_partner_orders_v1__parses_schema_one_payloads"); + +WITH __ref__stg_partner_orders AS ( + SELECT + 'MS1' AS message_id, + 'MKT' AS partner_code, + 1 AS schema_version, + NULL AS received_at, + 'A1' AS external_order_id, + OBJECT_CONSTRUCT( + 'buyer', + OBJECT_CONSTRUCT('email', ' Duck.Fan+mkt@Example.com '), + 'total', + OBJECT_CONSTRUCT( + 'currency', + 'eur', + 'amount', + '24.69' + ), + 'created', + '2026-03-10 09:00:00 +01:00', + 'lines', + ARRAY_CONSTRUCT(OBJECT_CONSTRUCT('sku', 'DUCK-MAL-01')) + )::OBJECT AS payload + UNION ALL + SELECT + 'MS2', + 'MKT', + 2, + NULL, + 'B7', + PARSE_JSON('{}')::OBJECT + UNION ALL + SELECT + 'MS3', + 'MKT', + 1, + NULL, + 'A2', + OBJECT_CONSTRUCT_KEEP_NULL( + 'buyer', + OBJECT_CONSTRUCT_KEEP_NULL('email', NULL), + 'total', + OBJECT_CONSTRUCT( + 'currency', + 'kwd', + 'amount', + '3.5' + ), + 'created', + '2026-03-11 10:00:00 +03:00' + )::OBJECT +), + +__expected__stg_partner_orders_v1 AS ( + SELECT + 'MKT:A1' AS partner_order_key, + 'duck.fan@example.com' AS buyer_email, + 'EUR' AS currency_code, + 24.69::DECIMAL(18, 3) AS order_total, + '2026-03-10 09:00:00 +01:00'::TIMESTAMP_TZ AS placed_at + UNION ALL + SELECT + 'MKT:A2' AS partner_order_key, + CAST(NULL AS VARCHAR) AS buyer_email, + 'KWD' AS currency_code, + 3.5::DECIMAL(18, 3) AS order_total, + '2026-03-11 10:00:00 +03:00'::TIMESTAMP_TZ AS placed_at +), + +__assert__missing_lines_become_empty_arrays AS ( + SELECT o.partner_order_key + FROM __ref("stg_partner_orders_v1") AS o + WHERE ARRAY_SIZE(o.lines) <> IFF(o.partner_order_key = 'MKT:A1', 1, 0) +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_orders_v2__parses_schema_two_payloads.sql b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_orders_v2__parses_schema_two_payloads.sql new file mode 100644 index 0000000000..ea1c3e192e --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_orders_v2__parses_schema_two_payloads.sql @@ -0,0 +1,66 @@ +TEST (name "stg_partner_orders_v2__parses_schema_two_payloads"); + +WITH __ref__stg_partner_orders AS ( + SELECT + 'MS2' AS message_id, + 'MKT' AS partner_code, + 2 AS schema_version, + NULL AS received_at, + 'B7' AS external_order_id, + OBJECT_CONSTRUCT( + 'customer', + OBJECT_CONSTRUCT('email_hash', 'ABC123'), + 'order', + OBJECT_CONSTRUCT( + 'currency', + 'usd', + 'totals', + OBJECT_CONSTRUCT('gross_minor', 4999), + 'placed_at_ms', + 1773137400000 + ), + 'items', + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT('seller_sku', 'A'), + OBJECT_CONSTRUCT('seller_sku', 'B') + ) + )::OBJECT AS payload + UNION ALL + SELECT + 'MS1', + 'MKT', + 1, + NULL, + 'A1', + PARSE_JSON('{}')::OBJECT + UNION ALL + SELECT + 'MS4', + 'MKT', + 2, + NULL, + 'B8', + PARSE_JSON('{"order": {"currency": "eur"}}')::OBJECT +), + +__expected__stg_partner_orders_v2 AS ( + SELECT + 'MKT:B7' AS partner_order_key, + 'abc123' AS buyer_email_hash, + 'USD' AS currency_code, + 4999::DECIMAL(38, 0) AS order_total_minor, + '2026-03-10 10:10:00'::TIMESTAMP_NTZ AS placed_at_utc + UNION ALL + SELECT + 'MKT:B8' AS partner_order_key, + CAST(NULL AS VARCHAR) AS buyer_email_hash, + 'EUR' AS currency_code, + CAST(NULL AS DECIMAL(38, 0)) AS order_total_minor, + CAST(NULL AS TIMESTAMP_NTZ) AS placed_at_utc +), + +__assert__missing_items_become_empty_arrays AS ( + SELECT o.partner_order_key + FROM __ref("stg_partner_orders_v2") AS o + WHERE ARRAY_SIZE(o.items) <> IFF(o.partner_order_key = 'MKT:B7', 2, 0) +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_settlements__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_settlements__keeps_latest_live_version.sql new file mode 100644 index 0000000000..1f5c320f51 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/partners/test_stg_partner_settlements__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_partner_settlements__keeps_latest_live_version"); + +WITH __source__partners__partner_settlements AS ( + SELECT + 'K1' AS settlement_id, + NULL AS partner_code, + NULL AS period_start, + NULL AS period_end, + NULL AS currency_code, + NULL AS statement, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_partner_settlements AS ( + SELECT + 'K1' AS settlement_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_cart_items__coerces_loosely_typed_cart_payloads.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_cart_items__coerces_loosely_typed_cart_payloads.sql new file mode 100644 index 0000000000..2af61f208d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_cart_items__coerces_loosely_typed_cart_payloads.sql @@ -0,0 +1,84 @@ +TEST (name "stg_cart_items__coerces_loosely_typed_cart_payloads"); + +WITH __ref__stg_carts AS ( + SELECT + 'CT1' AS cart_id, + NULL AS customer_id, + NULL AS anonymous_id, + NULL AS storefront_code, + NULL AS cart_status, + NULL AS created_at, + NULL AS updated_at, + NULL AS line_count, + NULL AS coupon_code, + OBJECT_CONSTRUCT( + 'items', + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'sku', + 'duck_mal 01', + 'qty', + 2, + 'unit_minor', + '1299', + 'added_at', + '2026-03-10 09:00:00 +01:00' + ), + OBJECT_CONSTRUCT( + 'sku', + 'BATH-EID-1', + 'qty', + '3', + 'unit_minor', + 899 + ), + OBJECT_CONSTRUCT( + 'sku', + 'POND-PUMP', + 'qty', + 'two', + 'added_at', + 'not a date' + ) + ) + )::OBJECT AS contents, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'CT2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('{}')::OBJECT, + NULL, + NULL +), + +__expected__stg_cart_items AS ( + SELECT + 'CT1/0' AS cart_item_key, + 'DUCK-MAL01' AS sku, + 2 AS quantity, + 1299 AS unit_amount_minor, + '2026-03-10 09:00:00 +01:00'::TIMESTAMPTZ AS added_at + UNION ALL + SELECT + 'CT1/1' AS cart_item_key, + 'BATH-EID-1' AS sku, + 3 AS quantity, + 899 AS unit_amount_minor, + CAST(NULL AS TIMESTAMPTZ) AS added_at + UNION ALL + SELECT + 'CT1/2' AS cart_item_key, + 'POND-PUMP' AS sku, + CAST(NULL AS INT) AS quantity, + CAST(NULL AS INT) AS unit_amount_minor, + CAST(NULL AS TIMESTAMPTZ) AS added_at +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_carts__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_carts__keeps_latest_live_version.sql new file mode 100644 index 0000000000..1ac613f261 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_carts__keeps_latest_live_version.sql @@ -0,0 +1,61 @@ +TEST (name "stg_carts__keeps_latest_live_version"); + +WITH __source__storefront__carts AS ( + SELECT + 'K1' AS cart_id, + NULL AS customer_id, + NULL AS anonymous_id, + NULL AS storefront_code, + NULL AS cart_status, + NULL AS created_at, + NULL AS updated_at, + NULL AS contents, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_carts AS ( + SELECT + 'K1' AS cart_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customer_addresses__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customer_addresses__keeps_latest_live_version.sql new file mode 100644 index 0000000000..d786bad94a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customer_addresses__keeps_latest_live_version.sql @@ -0,0 +1,69 @@ +TEST (name "stg_customer_addresses__keeps_latest_live_version"); + +WITH __source__storefront__customer_addresses AS ( + SELECT + 'K1' AS address_id, + NULL AS customer_id, + NULL AS line1, + NULL AS line2, + NULL AS city, + NULL AS postcode, + NULL AS country_code, + NULL AS is_default, + NULL AS valid_from, + NULL AS geo, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_customer_addresses AS ( + SELECT + 'K1' AS address_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customer_consents__adds_legacy_profile_consents_without_explicit_records.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customer_consents__adds_legacy_profile_consents_without_explicit_records.sql new file mode 100644 index 0000000000..a49b45101d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customer_consents__adds_legacy_profile_consents_without_explicit_records.sql @@ -0,0 +1,128 @@ +TEST (name "stg_customer_consents__adds_legacy_profile_consents_without_explicit_records"); + +WITH __source__storefront__customer_consents AS ( + SELECT + 'CS1' AS consent_id, + 'C1' AS customer_id, + 'email marketing' AS consent_type, + TRUE AS granted, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS captured_at, + PARSE_JSON('{"surface": "checkout", "policy_url": "https://shop.example.com/privacy/v2.1"}')::VARIANT AS source_context, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'CS2', + 'C1', + 'sms-marketing', + TRUE, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'I', + 2, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'CS2', + 'C1', + 'sms-marketing', + NULL, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'U', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'CS3', + 'C2', + 'email marketing', + TRUE, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'I', + 4, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'CS3', + 'C2', + 'email marketing', + TRUE, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + 'D', + 5, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ +), + +__source__storefront__customers AS ( + SELECT + 'C1' AS customer_id, + NULL AS email, + NULL AS phone, + NULL AS full_name, + NULL AS region_code, + NULL AS storefront_code, + '2026-01-01 10:00:00 +00:00'::TIMESTAMP_TZ AS created_at, + NULL AS updated_at, + PARSE_JSON('{"consents": ["email marketing", "push"]}')::VARIANT AS profile, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'C2', + NULL, + NULL, + NULL, + NULL, + NULL, + '2026-01-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + PARSE_JSON('{"consents": ["push"]}')::VARIANT, + 'I', + 2, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'C2', + NULL, + NULL, + NULL, + NULL, + NULL, + '2026-01-01 10:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + PARSE_JSON('{"consents": ["push"]}')::VARIANT, + 'D', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_customer_consents AS ( + SELECT + 'CS1' AS consent_id, + 'EMAIL_MARKETING' AS consent_type, + TRUE AS is_granted, + 'checkout' AS capture_surface, + '2.1' AS policy_version, + FALSE AS is_legacy + UNION ALL + SELECT + 'CS2' AS consent_id, + 'SMS_MARKETING' AS consent_type, + FALSE AS is_granted, + 'unknown' AS capture_surface, + CAST(NULL AS VARCHAR) AS policy_version, + FALSE AS is_legacy + UNION ALL + SELECT + 'legacy:C1:PUSH' AS consent_id, + 'PUSH' AS consent_type, + TRUE AS is_granted, + 'profile' AS capture_surface, + CAST(NULL AS VARCHAR) AS policy_version, + TRUE AS is_legacy +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customer_consents__skips_legacy_consents_when_no_profiles_load.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customer_consents__skips_legacy_consents_when_no_profiles_load.sql new file mode 100644 index 0000000000..b188abaca8 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customer_consents__skips_legacy_consents_when_no_profiles_load.sql @@ -0,0 +1,28 @@ +TEST (name "stg_customer_consents__skips_legacy_consents_when_no_profiles_load"); + +WITH __source__storefront__customer_consents AS ( + SELECT + 'CS1' AS consent_id, + 'C1' AS customer_id, + 'email marketing' AS consent_type, + TRUE AS granted, + '2026-03-01 10:00:00 +00:00'::TIMESTAMP_TZ AS captured_at, + PARSE_JSON('{"surface": "checkout"}')::VARIANT AS source_context, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at +), + +__source__storefront__customers AS ( + SELECT * + FROM __empty_fixture() +), + +__expected__stg_customer_consents AS ( + SELECT + 'CS1' AS consent_id, + 'EMAIL_MARKETING' AS consent_type, + TRUE AS is_granted, + 'checkout' AS capture_surface, + FALSE AS is_legacy +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customers__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customers__keeps_latest_live_version.sql new file mode 100644 index 0000000000..50c389c664 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_customers__keeps_latest_live_version.sql @@ -0,0 +1,71 @@ +TEST (name "stg_customers__keeps_latest_live_version"); + +WITH __source__storefront__customers AS ( + SELECT + 'C1' AS customer_id, + ' Ada.Featherby@Example.COM ' AS email, + NULL AS phone, + 'ada featherby' AS full_name, + 'eu' AS region_code, + 'eu-web' AS storefront_code, + NULL AS created_at, + NULL AS updated_at, + PARSE_JSON('{"preferences": {"language": "de"}}') AS profile, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'C1', + 'ada@example.com', + NULL, + 'Ada Featherby', + 'EU', + 'eu-web', + NULL, + NULL, + PARSE_JSON('{}'), + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'C2', + 'bob@example.com', + NULL, + 'Bob Reedley', + 'US', + 'us-web', + NULL, + NULL, + PARSE_JSON('{}'), + 'I', + 3, + '2026-01-01 11:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'C2', + 'bob@example.com', + NULL, + 'Bob Reedley', + 'US', + 'us-web', + NULL, + NULL, + PARSE_JSON('{}'), + 'D', + 4, + '2026-01-03 11:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_customers AS ( + SELECT + 'C1' AS customer_id, + 'ada@example.com' AS email_normalised, + 'Ada' AS first_name, + 'Featherby' AS last_name, + 'EU' AS region_code, + 'EU-WEB' AS storefront_code, + 'en' AS preferred_language, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_gift_card_transactions__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_gift_card_transactions__keeps_latest_live_version.sql new file mode 100644 index 0000000000..9fd95be93c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_gift_card_transactions__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_gift_card_transactions__keeps_latest_live_version"); + +WITH __source__storefront__gift_card_transactions AS ( + SELECT + 'K1' AS transaction_id, + NULL AS gift_card_id, + NULL AS order_id, + NULL AS amount_minor, + NULL AS transaction_type, + NULL AS occurred_at, + NULL AS context, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_gift_card_transactions AS ( + SELECT + 'K1' AS transaction_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_gift_cards__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_gift_cards__keeps_latest_live_version.sql new file mode 100644 index 0000000000..5fb1956360 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_gift_cards__keeps_latest_live_version.sql @@ -0,0 +1,61 @@ +TEST (name "stg_gift_cards__keeps_latest_live_version"); + +WITH __source__storefront__gift_cards AS ( + SELECT + 'K1' AS gift_card_id, + NULL AS code_masked, + NULL AS currency_code, + NULL AS initial_amount_minor, + NULL AS issued_at, + NULL AS expires_on, + NULL AS purchaser_customer_id, + NULL AS details, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_gift_cards AS ( + SELECT + 'K1' AS gift_card_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_loyalty_accounts__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_loyalty_accounts__keeps_latest_live_version.sql new file mode 100644 index 0000000000..3188282865 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_loyalty_accounts__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_loyalty_accounts__keeps_latest_live_version"); + +WITH __source__storefront__loyalty_accounts AS ( + SELECT + 'K1' AS loyalty_account_id, + NULL AS customer_id, + NULL AS tier_code, + NULL AS points_balance, + NULL AS enrolled_at, + NULL AS tier_history, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_loyalty_accounts AS ( + SELECT + 'K1' AS loyalty_account_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_loyalty_ledger__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_loyalty_ledger__keeps_latest_live_version.sql new file mode 100644 index 0000000000..73bd465320 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_loyalty_ledger__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_loyalty_ledger__keeps_latest_live_version"); + +WITH __source__storefront__loyalty_ledger AS ( + SELECT + 'K1' AS entry_id, + NULL AS loyalty_account_id, + NULL AS points_delta, + NULL AS reason_code, + NULL AS order_id, + NULL AS posted_at, + NULL AS metadata, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_loyalty_ledger AS ( + SELECT + 'K1' AS entry_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_order_line_discounts__explodes_discounts_and_defaults_kind.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_order_line_discounts__explodes_discounts_and_defaults_kind.sql new file mode 100644 index 0000000000..f86c9114e2 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_order_line_discounts__explodes_discounts_and_defaults_kind.sql @@ -0,0 +1,55 @@ +TEST (name "stg_order_line_discounts__explodes_discounts_and_defaults_kind"); + +WITH __ref__stg_order_lines AS ( + SELECT + 'L1' AS order_line_id, + 'O1' AS order_id, + NULL AS line_no, + NULL AS sku, + NULL AS quantity, + NULL AS unit_amount_minor, + NULL AS gross_amount_minor, + NULL AS tax_amount_minor, + NULL AS discount_count, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'code', + 'spring10', + 'kind', + 'Promo', + 'amount_minor', + '150' + ), + OBJECT_CONSTRUCT_KEEP_NULL('code', NULL, 'amount_minor', 50) + )::ARRAY AS discounts, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'L2', + 'O1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL +), + +__expected__stg_order_line_discounts AS ( + SELECT + 'L1#0' AS order_line_discount_key, + 'SPRING10' AS discount_code, + 'promo' AS discount_kind, + 150 AS discount_amount_minor + UNION ALL + SELECT + 'L1#1' AS order_line_discount_key, + CAST(NULL AS VARCHAR) AS discount_code, + 'manual' AS discount_kind, + 50 AS discount_amount_minor +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_order_lines__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_order_lines__keeps_latest_live_version.sql new file mode 100644 index 0000000000..9ac5134628 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_order_lines__keeps_latest_live_version.sql @@ -0,0 +1,61 @@ +TEST (name "stg_order_lines__keeps_latest_live_version"); + +WITH __source__storefront__order_lines AS ( + SELECT + 'K1' AS order_line_id, + NULL AS order_id, + NULL AS line_no, + NULL AS sku, + NULL AS quantity, + NULL AS unit_amount_minor, + NULL AS tax_amount_minor, + NULL AS discounts, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_order_lines AS ( + SELECT + 'K1' AS order_line_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_order_status_events__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_order_status_events__keeps_latest_live_version.sql new file mode 100644 index 0000000000..17b98b642d --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_order_status_events__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_order_status_events__keeps_latest_live_version"); + +WITH __source__storefront__order_status_events AS ( + SELECT + 'K1' AS event_id, + NULL AS order_id, + NULL AS from_status, + NULL AS to_status, + NULL AS changed_at, + NULL AS actor, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_order_status_events AS ( + SELECT + 'K1' AS event_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_orders__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_orders__keeps_latest_live_version.sql new file mode 100644 index 0000000000..4e11c71b0a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_orders__keeps_latest_live_version.sql @@ -0,0 +1,61 @@ +TEST (name "stg_orders__keeps_latest_live_version"); + +WITH __source__storefront__orders AS ( + SELECT + 'K1' AS order_id, + NULL AS customer_id, + NULL AS storefront_code, + NULL AS currency_code, + NULL AS placed_at, + NULL AS status, + NULL AS shipping_address_id, + NULL AS attributes, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_orders AS ( + SELECT + 'K1' AS order_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_payments__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_payments__keeps_latest_live_version.sql new file mode 100644 index 0000000000..1c2f128820 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_payments__keeps_latest_live_version.sql @@ -0,0 +1,65 @@ +TEST (name "stg_payments__keeps_latest_live_version"); + +WITH __source__storefront__payments AS ( + SELECT + 'K1' AS payment_id, + NULL AS order_id, + NULL AS method, + NULL AS amount_minor, + NULL AS currency_code, + NULL AS authorised_at, + NULL AS captured_at, + NULL AS card_bin, + NULL AS gateway_response, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_payments AS ( + SELECT + 'K1' AS payment_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_promotion_redemptions__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_promotion_redemptions__keeps_latest_live_version.sql new file mode 100644 index 0000000000..15b37ccce1 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_promotion_redemptions__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_promotion_redemptions__keeps_latest_live_version"); + +WITH __source__storefront__promotion_redemptions AS ( + SELECT + 'K1' AS redemption_id, + NULL AS promo_code, + NULL AS order_id, + NULL AS discount_minor, + NULL AS redeemed_at, + NULL AS context, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_promotion_redemptions AS ( + SELECT + 'K1' AS redemption_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_promotions__reads_eligibility_and_tiered_discounts.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_promotions__reads_eligibility_and_tiered_discounts.sql new file mode 100644 index 0000000000..c7d82a18ab --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_promotions__reads_eligibility_and_tiered_discounts.sql @@ -0,0 +1,78 @@ +TEST (name "stg_promotions__reads_eligibility_and_tiered_discounts"); + +WITH __source__storefront__promotions AS ( + SELECT + 'spring10 ' AS promo_code, + 'Spring Splash' AS promo_name, + '2026-03-01 00:00:00 +00:00'::TIMESTAMP_TZ AS starts_at, + NULL AS ends_at, + NULL AS is_stackable, + OBJECT_CONSTRUCT( + 'eligibility', + OBJECT_CONSTRUCT( + 'min_order_minor', + 2000, + 'regions', + ARRAY_CONSTRUCT('EU', 'US') + ), + 'tiers', + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT('pct', 5), + OBJECT_CONSTRUCT('pct', 12.5) + ) + )::VARIANT AS rules, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'pond5', + 'Pond Week', + '2026-04-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + TRUE, + PARSE_JSON('{}')::VARIANT, + 'I', + 2, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'old', + 'Old', + '2025-01-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + PARSE_JSON('{}')::VARIANT, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'old', + 'Old', + '2025-01-01 00:00:00 +00:00'::TIMESTAMP_TZ, + NULL, + NULL, + PARSE_JSON('{}')::VARIANT, + 'D', + 4, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_promotions AS ( + SELECT + 'SPRING10' AS promo_code, + FALSE AS is_stackable, + 2000 AS min_order_minor, + 2 AS tier_count, + 12.5::DECIMAL(5, 2) AS max_discount_pct, + 1 AS source_seq + UNION ALL + SELECT + 'POND5' AS promo_code, + TRUE AS is_stackable, + 0 AS min_order_minor, + 0 AS tier_count, + CAST(NULL AS DECIMAL(5, 2)) AS max_discount_pct, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_refunds__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_refunds__keeps_latest_live_version.sql new file mode 100644 index 0000000000..4fa1120ced --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_refunds__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_refunds__keeps_latest_live_version"); + +WITH __source__storefront__refunds AS ( + SELECT + 'K1' AS refund_id, + NULL AS payment_id, + NULL AS amount_minor, + NULL AS reason_text, + NULL AS refunded_at, + NULL AS initiated_by, + NULL AS items, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_refunds AS ( + SELECT + 'K1' AS refund_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_reviews__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_reviews__keeps_latest_live_version.sql new file mode 100644 index 0000000000..c9a5efe780 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_reviews__keeps_latest_live_version.sql @@ -0,0 +1,61 @@ +TEST (name "stg_reviews__keeps_latest_live_version"); + +WITH __source__storefront__reviews AS ( + SELECT + 'K1' AS review_id, + NULL AS sku, + NULL AS customer_id, + NULL AS rating, + NULL AS title, + NULL AS body, + NULL AS submitted_at, + NULL AS moderation, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_reviews AS ( + SELECT + 'K1' AS review_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_store_credit_ledger__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_store_credit_ledger__keeps_latest_live_version.sql new file mode 100644 index 0000000000..010b207e9c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_store_credit_ledger__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_store_credit_ledger__keeps_latest_live_version"); + +WITH __source__storefront__store_credit_ledger AS ( + SELECT + 'K1' AS entry_id, + NULL AS customer_id, + NULL AS currency_code, + NULL AS amount_minor, + NULL AS reason_code, + NULL AS posted_at, + NULL AS context, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_store_credit_ledger AS ( + SELECT + 'K1' AS entry_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_subscription_events__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_subscription_events__keeps_latest_live_version.sql new file mode 100644 index 0000000000..9745d93c32 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_subscription_events__keeps_latest_live_version.sql @@ -0,0 +1,49 @@ +TEST (name "stg_subscription_events__keeps_latest_live_version"); + +WITH __source__storefront__subscription_events AS ( + SELECT + 'K1' AS event_id, + NULL AS subscription_id, + NULL AS event_type, + NULL AS occurred_at, + NULL AS payload, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_subscription_events AS ( + SELECT + 'K1' AS event_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_subscriptions__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_subscriptions__keeps_latest_live_version.sql new file mode 100644 index 0000000000..e7ee9e7d90 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_subscriptions__keeps_latest_live_version.sql @@ -0,0 +1,65 @@ +TEST (name "stg_subscriptions__keeps_latest_live_version"); + +WITH __source__storefront__subscriptions AS ( + SELECT + 'K1' AS subscription_id, + NULL AS customer_id, + NULL AS plan_code, + NULL AS sku, + NULL AS interval_days, + NULL AS subscription_status, + NULL AS started_at, + NULL AS cancelled_at, + NULL AS billing, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_subscriptions AS ( + SELECT + 'K1' AS subscription_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlist_items__keeps_object_items_with_a_sku.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlist_items__keeps_object_items_with_a_sku.sql new file mode 100644 index 0000000000..1a2f6b383a --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlist_items__keeps_object_items_with_a_sku.sql @@ -0,0 +1,39 @@ +TEST (name "stg_wishlist_items__keeps_object_items_with_a_sku"); + +WITH __ref__stg_wishlists AS ( + SELECT + 'W1' AS wishlist_id, + 'C1' AS customer_id, + NULL AS created_at, + NULL AS is_public, + NULL AS item_count, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'sku', + 'duck_mal 01', + 'added_at', + '2026-03-01 10:00:00 +00:00' + ), + 'DUCK-LEGACY', + OBJECT_CONSTRUCT('added_at', '2026-03-02'), + OBJECT_CONSTRUCT('sku', 'pond-pump') + )::ARRAY AS items, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__stg_wishlist_items AS ( + SELECT + 'W1:0' AS wishlist_item_key, + 'C1' AS customer_id, + 'DUCK-MAL01' AS sku, + '2026-03-01 10:00:00 +00:00'::TIMESTAMPTZ AS added_at, + 0 AS item_position + UNION ALL + SELECT + 'W1:3' AS wishlist_item_key, + 'C1' AS customer_id, + 'POND-PUMP' AS sku, + CAST(NULL AS TIMESTAMPTZ) AS added_at, + 3 AS item_position +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlist_items__uses_the_mocked_sku_normaliser.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlist_items__uses_the_mocked_sku_normaliser.sql new file mode 100644 index 0000000000..ff09a45546 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlist_items__uses_the_mocked_sku_normaliser.sql @@ -0,0 +1,30 @@ +TEST (name "stg_wishlist_items__uses_the_mocked_sku_normaliser"); + +WITH __macro__normalise_sku AS ( + SELECT '''MOCKED-SKU''' +), + +__ref__stg_wishlists AS ( + SELECT + 'W1' AS wishlist_id, + 'C1' AS customer_id, + NULL AS created_at, + NULL AS is_public, + NULL AS item_count, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT('sku', 'duck_mal 01'), + OBJECT_CONSTRUCT('sku', 'pond-pump') + )::ARRAY AS items, + NULL AS source_seq, + NULL AS source_loaded_at +), + +__expected__stg_wishlist_items AS ( + SELECT + 'W1:0' AS wishlist_item_key, + 'MOCKED-SKU' AS sku + UNION ALL + SELECT + 'W1:1' AS wishlist_item_key, + 'MOCKED-SKU' AS sku +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlists__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlists__keeps_latest_live_version.sql new file mode 100644 index 0000000000..1c5eec134b --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/storefront/test_stg_wishlists__keeps_latest_live_version.sql @@ -0,0 +1,49 @@ +TEST (name "stg_wishlists__keeps_latest_live_version"); + +WITH __source__storefront__wishlists AS ( + SELECT + 'K1' AS wishlist_id, + NULL AS customer_id, + NULL AS created_at, + NULL AS items, + NULL AS is_public, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_wishlists AS ( + SELECT + 'K1' AS wishlist_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/_sqlbuild/_macros/python/ticket_fixtures.py b/benchmarks/duckshop/project/tests/unit/staging/support/_sqlbuild/_macros/python/ticket_fixtures.py new file mode 100644 index 0000000000..f7f8b99b47 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/_sqlbuild/_macros/python/ticket_fixtures.py @@ -0,0 +1,49 @@ +"""Fixture rows for tests that read staged support tickets.""" + +import json + +_COLUMNS = ( + "ticket_id", + "customer_email", + "channel", + "priority", + "ticket_status", + "created_at", + "solved_at", + "order_id", + "contact_reason", + "tag_list", + "is_vip", + "custom_fields", + "source_seq", + "source_loaded_at", +) + + +def _text(value: str) -> str: + return "'" + value.replace("'", "''") + "'" + + +def tagged_tickets(rows: list[list[object]]) -> str: + """One stg_tickets row per [ticket_id, tags, labels] entry; other columns are typed NULLs.""" + selects = [] + for ticket_id, tags, labels in rows: + custom_fields = {} if labels is None else {"labels": labels} + values = { + "ticket_id": _text(str(ticket_id)), + "customer_email": "CAST(NULL AS VARCHAR)", + "channel": "'email'", + "priority": "CAST(NULL AS INT)", + "ticket_status": "'open'", + "created_at": "CAST(NULL AS TIMESTAMP_LTZ)", + "solved_at": "CAST(NULL AS TIMESTAMP_TZ)", + "order_id": "CAST(NULL AS VARCHAR)", + "contact_reason": "CAST(NULL AS VARCHAR)", + "tag_list": f"PARSE_JSON({_text(json.dumps(tags))})::ARRAY", + "is_vip": "FALSE", + "custom_fields": f"PARSE_JSON({_text(json.dumps(custom_fields))})::OBJECT", + "source_seq": "CAST(NULL AS NUMBER(38, 0))", + "source_loaded_at": "CAST(NULL AS TIMESTAMP_NTZ)", + } + selects.append("SELECT " + ", ".join(f"{values[c]} AS {c}" for c in _COLUMNS)) + return " UNION ALL ".join(selects) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_agent_shifts__totals_breaks_and_scheduled_minutes.sql b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_agent_shifts__totals_breaks_and_scheduled_minutes.sql new file mode 100644 index 0000000000..dfecaa071c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_agent_shifts__totals_breaks_and_scheduled_minutes.sql @@ -0,0 +1,50 @@ +TEST (name "stg_agent_shifts__totals_breaks_and_scheduled_minutes"); + +WITH __source__support__agent_shifts AS ( + SELECT + 'SH1' AS shift_id, + 'A1' AS agent_id, + '2026-03-10 22:00:00'::TIMESTAMP_NTZ AS shift_start_local, + '2026-03-11 05:00:00'::TIMESTAMP_NTZ AS shift_end_local, + 'Europe/Berlin' AS iana_timezone, + NULL AS breaks, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'SH1', + 'A1', + '2026-03-10 22:00:00'::TIMESTAMP_NTZ, + '2026-03-11 06:00:00'::TIMESTAMP_NTZ, + 'Europe/Berlin', + PARSE_JSON('[{"minutes": 30}, {"minutes": 15}, {"type": "handover"}]')::VARIANT, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'SH2', + 'A2', + '2026-03-11 09:00:00'::TIMESTAMP_NTZ, + '2026-03-11 13:00:00'::TIMESTAMP_NTZ, + 'Europe/London', + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_agent_shifts AS ( + SELECT + 'SH1' AS shift_id, + 480 AS scheduled_minutes, + 45 AS break_minutes, + 2 AS source_seq + UNION ALL + SELECT + 'SH2' AS shift_id, + 240 AS scheduled_minutes, + 0 AS break_minutes, + 3 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_agents__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_agents__keeps_latest_live_version.sql new file mode 100644 index 0000000000..8396faa335 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_agents__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_agents__keeps_latest_live_version"); + +WITH __source__support__agents AS ( + SELECT + 'K1' AS agent_id, + NULL AS display_name, + NULL AS email, + NULL AS team_code, + NULL AS iana_timezone, + NULL AS hired_on, + NULL AS skills, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_agents AS ( + SELECT + 'K1' AS agent_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_call_records__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_call_records__keeps_latest_live_version.sql new file mode 100644 index 0000000000..74097b0896 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_call_records__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_call_records__keeps_latest_live_version"); + +WITH __source__support__call_records AS ( + SELECT + 'K1' AS call_id, + NULL AS ticket_id, + NULL AS agent_id, + NULL AS phone_number, + NULL AS started_at, + NULL AS duration_seconds, + NULL AS ivr_path, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_call_records AS ( + SELECT + 'K1' AS call_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_chat_messages__masks_contact_details_in_messages.sql b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_chat_messages__masks_contact_details_in_messages.sql new file mode 100644 index 0000000000..de8cea397f --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_chat_messages__masks_contact_details_in_messages.sql @@ -0,0 +1,56 @@ +TEST (name "stg_chat_messages__masks_contact_details_in_messages"); + +WITH __ref__stg_chat_transcripts AS ( + SELECT + 'CH1' AS chat_id, + NULL AS ticket_id, + NULL AS started_at, + NULL AS ended_at, + NULL AS message_count, + ARRAY_CONSTRUCT( + OBJECT_CONSTRUCT( + 'role', + 'Customer', + 'ts', + '2026-03-10 10:00:00 +00:00', + 'text', + 'My duck arrived squashed, call me on +44 20 7946 0958 or mail duck.fan@example.com' + ), + OBJECT_CONSTRUCT( + 'role', + 'AGENT', + 'ts', + 'yesterday', + 'text', + 'Sorry!' + ) + )::ARRAY AS messages, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'CH2', + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('[]')::ARRAY, + NULL, + NULL +), + +__expected__stg_chat_messages AS ( + SELECT + 'CH1#0' AS chat_message_key, + 'customer' AS sender_role, + '2026-03-10 10:00:00 +00:00'::TIMESTAMP_TZ AS sent_at, + 'My duck arrived squashed, call me on or mail ' AS message_text_masked, + 82 AS message_length + UNION ALL + SELECT + 'CH1#1' AS chat_message_key, + 'agent' AS sender_role, + CAST(NULL AS TIMESTAMP_TZ) AS sent_at, + 'Sorry!' AS message_text_masked, + 6 AS message_length +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_chat_transcripts__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_chat_transcripts__keeps_latest_live_version.sql new file mode 100644 index 0000000000..fff69ceaed --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_chat_transcripts__keeps_latest_live_version.sql @@ -0,0 +1,49 @@ +TEST (name "stg_chat_transcripts__keeps_latest_live_version"); + +WITH __source__support__chat_transcripts AS ( + SELECT + 'K1' AS chat_id, + NULL AS ticket_id, + NULL AS started_at, + NULL AS ended_at, + NULL AS messages, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_chat_transcripts AS ( + SELECT + 'K1' AS chat_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_csat_responses__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_csat_responses__keeps_latest_live_version.sql new file mode 100644 index 0000000000..86f3646d08 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_csat_responses__keeps_latest_live_version.sql @@ -0,0 +1,53 @@ +TEST (name "stg_csat_responses__keeps_latest_live_version"); + +WITH __source__support__csat_responses AS ( + SELECT + 'K1' AS response_id, + NULL AS ticket_id, + NULL AS score, + NULL AS comment, + NULL AS responded_at, + NULL AS answers, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_csat_responses AS ( + SELECT + 'K1' AS response_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_sla_policies__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_sla_policies__keeps_latest_live_version.sql new file mode 100644 index 0000000000..0ffe464f7f --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_sla_policies__keeps_latest_live_version.sql @@ -0,0 +1,57 @@ +TEST (name "stg_sla_policies__keeps_latest_live_version"); + +WITH __source__support__sla_policies AS ( + SELECT + 'K1' AS policy_id, + NULL AS priority, + NULL AS channel, + NULL AS first_response_minutes, + NULL AS resolution_hours, + NULL AS business_hours_only, + NULL AS calendar, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_sla_policies AS ( + SELECT + 'K1' AS policy_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_ticket_comments__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_ticket_comments__keeps_latest_live_version.sql new file mode 100644 index 0000000000..d9e5a36a8c --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_ticket_comments__keeps_latest_live_version.sql @@ -0,0 +1,61 @@ +TEST (name "stg_ticket_comments__keeps_latest_live_version"); + +WITH __source__support__ticket_comments AS ( + SELECT + 'K1' AS comment_id, + NULL AS ticket_id, + NULL AS author_type, + NULL AS author_id, + NULL AS body, + NULL AS is_public, + NULL AS created_at, + NULL AS attachments, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_ticket_comments AS ( + SELECT + 'K1' AS comment_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_ticket_tags__dedupes_tags_from_fixture_macro.sql b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_ticket_tags__dedupes_tags_from_fixture_macro.sql new file mode 100644 index 0000000000..42e9e41814 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_ticket_tags__dedupes_tags_from_fixture_macro.sql @@ -0,0 +1,23 @@ +TEST (name "stg_ticket_tags__dedupes_tags_from_fixture_macro"); + +WITH __ref__stg_tickets AS ( + @py.tagged_tickets([["T3", ["Wrong Colour", "wrong_colour", "WRONG COLOUR"], "Returns"], ["T4", [], None]]) +), + +__expected__stg_ticket_tags AS ( + SELECT + 'T3:wrong-colour' AS ticket_tag_key, + 'T3' AS ticket_id, + 'wrong-colour' AS tag + UNION ALL + SELECT + 'T3:returns' AS ticket_tag_key, + 'T3' AS ticket_id, + 'returns' AS tag +), + +__assert__every_tag_is_lower_kebab_case AS ( + SELECT t.ticket_tag_key + FROM __ref("stg_ticket_tags") AS t + WHERE NOT REGEXP_LIKE(t.tag, '[a-z0-9]+(-[a-z0-9]+)*') +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_ticket_tags__merges_array_tags_and_label_field.sql b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_ticket_tags__merges_array_tags_and_label_field.sql new file mode 100644 index 0000000000..f3e5688dd1 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_ticket_tags__merges_array_tags_and_label_field.sql @@ -0,0 +1,49 @@ +TEST (name "stg_ticket_tags__merges_array_tags_and_label_field"); + +WITH __ref__stg_tickets AS ( + SELECT + 'T1' AS ticket_id, + NULL AS customer_email, + NULL AS channel, + NULL AS priority, + NULL AS ticket_status, + NULL AS created_at, + NULL AS solved_at, + NULL AS order_id, + NULL AS contact_reason, + PARSE_JSON('["Damaged Item", "damaged_item", null, " "]')::ARRAY AS tag_list, + NULL AS is_vip, + PARSE_JSON('{"labels": "vip, Rubber Ducks"}')::OBJECT AS custom_fields, + NULL AS source_seq, + NULL AS source_loaded_at + UNION ALL + SELECT + 'T2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + PARSE_JSON('[]')::ARRAY, + NULL, + PARSE_JSON('{}')::OBJECT, + NULL, + NULL +), + +__expected__stg_ticket_tags AS ( + SELECT + 'T1:damaged-item' AS ticket_tag_key, + 'damaged-item' AS tag + UNION ALL + SELECT + 'T1:vip' AS ticket_tag_key, + 'vip' AS tag + UNION ALL + SELECT + 'T1:rubber-ducks' AS ticket_tag_key, + 'rubber-ducks' AS tag +) diff --git a/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_tickets__keeps_latest_live_version.sql b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_tickets__keeps_latest_live_version.sql new file mode 100644 index 0000000000..147b6fff84 --- /dev/null +++ b/benchmarks/duckshop/project/tests/unit/staging/support/test_stg_tickets__keeps_latest_live_version.sql @@ -0,0 +1,65 @@ +TEST (name "stg_tickets__keeps_latest_live_version"); + +WITH __source__support__tickets AS ( + SELECT + 'K1' AS ticket_id, + NULL AS customer_email, + NULL AS channel, + NULL AS priority, + NULL AS ticket_status, + NULL AS created_at, + NULL AS solved_at, + NULL AS custom_fields, + NULL AS tags, + 'I' AS _op, + 1 AS _seq, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ AS _loaded_at + UNION ALL + SELECT + 'K1', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'U', + 2, + '2026-01-02 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'I', + 3, + '2026-01-01 10:00:00'::TIMESTAMP_NTZ + UNION ALL + SELECT + 'K2', + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + 'D', + 4, + '2026-01-03 10:00:00'::TIMESTAMP_NTZ +), + +__expected__stg_tickets AS ( + SELECT + 'K1' AS ticket_id, + 2 AS source_seq +) diff --git a/benchmarks/duckshop/scripts/census.py b/benchmarks/duckshop/scripts/census.py new file mode 100644 index 0000000000..80e75f5214 --- /dev/null +++ b/benchmarks/duckshop/scripts/census.py @@ -0,0 +1,262 @@ +"""Construct census for the compiled DuckShop project. + +Run `sqb compile` in ../project first, then: + + python benchmarks/duckshop/scripts/census.py [--json] + +Constructs are detected with regular expressions over the compiled model SQL, with comments and +string literals removed. A model counts towards a family when the family appears at least once. +The output is aggregate only. +""" + +from __future__ import annotations + +import argparse +import json +import pathlib +import re +import statistics +import sys + +PROJECT = pathlib.Path(__file__).resolve().parent.parent / "project" +COMPILED = PROJECT / "target" / "compiled" / "models" + +STRING = re.compile(r"'(?:[^'\\]|\\.|'')*'", re.S) +TOKENS = re.compile(r"('(?:[^'\\]|\\.|'')*')|(/\*.*?\*/)|((?:--|//)[^\n]*)", re.S) + +FAMILIES: dict[str, str] = { + "window": r"\bover\s*\(", + "qualify": r"\bqualify\b", + "semi_path": r"[\w\]\)\"]\s*:\s*[\"\w$\[]", + "flatten": r"\bflatten\s*\(", + "obj_arr_fn": ( + r"\b(object_\w+|array_\w+|arrayagg|parse_json|try_parse_json|to_variant|to_object" + r"|to_array|get_path|get_ignore_case|is_object|is_array|typeof|strip_null_value" + r"|is_null_value)\s*\(" + ), + "variant_cast": r"::\s*(variant|object|array)\b|\bas\s+(variant|object|array)\s*\)", + "regex": r"\b(regexp_\w+|rlike|regexp)\b", + "timezone": ( + r"\b(convert_timezone|timestamp_tz|timestamp_ltz|to_timestamp_tz|to_timestamp_ltz" + r"|at\s+time\s+zone|sysdate|current_timestamp)\b" + ), + "timestamp": ( + r"\b(timestamp(_ntz|_tz|_ltz)?|to_timestamp\w*|try_to_timestamp\w*|dateadd|datediff" + r"|timestampdiff|timestampadd|date_trunc|time_slice|date_part|extract|last_day)\b" + ), + "decimal": ( + r"\b(number|numeric|decimal)\s*\(|::\s*(number|numeric|decimal)\b" + r"|\bto_(number|decimal|numeric)\s*\(|\bround\s*\(" + ), + "division": r"[\w\)]\s*/\s*[\w\(]", + "asof": r"\basof\s+join\b|\bmatch_condition\b", + "set_op": r"\b(union|intersect|except|minus)\b", + "pivot": r"\b(pivot|unpivot)\b", + "generator": r"\bgenerator\s*\(", + "lateral": r"\blateral\b", + "hash": r"\bhash\s*\(", + "ltz": r"\b(timestamp_ltz|to_timestamp_ltz)\b", + "current_ts": r"\b(current_timestamp|sysdate|getdate|localtimestamp)\b", + "lambda": r"(\b\w+(\s+\w+)?|\))\s*->", + "connect_by": r"\bconnect\s+by\b", + "recursive_cte": r"\bwith\s+recursive\b", +} + +# Percent of models, from the DuckShop specification. None means "presence only". +TARGETS: dict[str, float | None] = { + "semi_any": 60, + "semi_path": 30, + "flatten": 15, + "obj_arr_fn": 40, + "variant_cast": 45, + "window": 55, + "qualify": 45, + "regex": 30, + "timestamp": 45, + "timezone": 15, + "decimal": 30, + "division": 15, + "set_op": 15, + "lateral": 15, + "asof": None, + "hash": None, + "ltz": None, + "current_ts": None, + "lambda": None, + "pivot": None, + "generator": None, + "connect_by": None, +} + +FUNCTION_CALL = re.compile(r"\b([a-z_][a-z0-9_]*)\s*\(") +KEYWORD_CONSTRUCTS = { + "qualify": r"\bqualify\b", + "asof join": r"\basof\s+join\b", + "match_condition": r"\bmatch_condition\b", + "lateral": r"\blateral\b", + "pivot": r"\bpivot\b", + "unpivot": r"\bunpivot\b", + "union all": r"\bunion\s+all\b", + "union": r"\bunion\b(?!\s+all)", + "intersect": r"\bintersect\b", + "except": r"\bexcept\b", + "minus": r"\bminus\b", + "connect by": r"\bconnect\s+by\b", + "within group": r"\bwithin\s+group\b", + "ignore nulls": r"\bignore\s+nulls\b", + "rows between": r"\brows\s+between\b", + "range between": r"\brange\s+between\b", + "at time zone": r"\bat\s+time\s+zone\b", + "lambda": r"->", + "colon path": r"[\w\]\)\"]\s*:\s*[\"\w$\[]", + "cast ::": r"::", + "group by all": r"\bgroup\s+by\s+all\b", + "exclude": r"\bexclude\s*\(", + "ilike": r"\bilike\b", + "rlike": r"\brlike\b", + "full outer join": r"\bfull\s+(outer\s+)?join\b", + "left join": r"\bleft\s+(outer\s+)?join\b", + "cross join": r"\bcross\s+join\b", + "natural join": r"\bnatural\s+join\b", + "case": r"\bcase\b", + "distinct": r"\bdistinct\b", + "having": r"\bhaving\b", + "interval": r"\binterval\b", +} +NOT_FUNCTIONS = { + "as", + "in", + "and", + "or", + "not", + "over", + "values", + "exists", + "select", + "from", + "where", + "when", + "then", + "else", + "on", + "using", + "join", + "with", + "table", + "partition", + "by", + "is", + "filter", + "lateral", + "qualify", + "within", + "group", + "pivot", + "unpivot", + "for", + "into", + "match_condition", + "any", + "all", + "exclude", + "if", +} + + +def clean(sql: str) -> str: + """Remove comments, leaving string literals (which may contain `--` or `//`) intact.""" + return TOKENS.sub(lambda m: m.group(1) or " ", sql) + + +def analyse(sql: str) -> dict[str, object]: + text = clean(sql) + low = STRING.sub("''", text).lower() + path_source = re.sub(r"::", " CAST_ ", low) + row: dict[str, object] = {} + for family, pattern in FAMILIES.items(): + row[family] = bool(re.search(pattern, path_source if family == "semi_path" else low)) + row["semi_any"] = any(row[k] for k in ("semi_path", "flatten", "obj_arr_fn", "variant_cast")) + row["ctes"] = len(re.findall(r"(?:\bwith\b|,)\s*[\w\"]+\s*(?:\([^)]*\))?\s+as\s*\(", low)) + row["joins"] = len(re.findall(r"\bjoin\b", low)) + row["lines"] = sum(1 for line in text.splitlines() if line.strip()) + functions = {f for f in FUNCTION_CALL.findall(low) if f not in NOT_FUNCTIONS} + keywords = {k for k, p in KEYWORD_CONSTRUCTS.items() if re.search(p, low)} + row["constructs"] = sorted(functions | {f"<{k}>" for k in keywords}) + return row + + +def summarise(rows: list[dict[str, object]]) -> dict[str, object]: + count = len(rows) + out: dict[str, object] = {"models": count} + if not count: + return out + for family in [*FAMILIES, "semi_any"]: + out[f"pct_{family}"] = round(100 * sum(bool(r[family]) for r in rows) / count, 1) + out[f"n_{family}"] = sum(bool(r[family]) for r in rows) + for metric in ("ctes", "joins", "lines"): + values = sorted(int(r[metric]) for r in rows) # type: ignore[call-overload] + out[metric] = { + "mean": round(statistics.mean(values), 1), + "p50": values[count // 2], + "p90": values[int(count * 0.9)], + "max": values[-1], + } + return out + + +def shape(stats: dict[str, float]) -> str: + return f"mean {stats['mean']} p50 {stats['p50']} p90 {stats['p90']} max {stats['max']}" + + +def layer_of(path: pathlib.Path) -> str: + return path.relative_to(COMPILED).parts[0] + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--json", action="store_true", help="print JSON instead of a table") + args = parser.parse_args() + files = sorted(COMPILED.rglob("*.sql")) + if not files: + print(f"no compiled models under {COMPILED}; run `sqb compile` first", file=sys.stderr) + return 1 + by_layer: dict[str, list[dict[str, object]]] = {} + for path in files: + by_layer.setdefault(layer_of(path), []).append(analyse(path.read_text())) + all_rows = [row for rows in by_layer.values() for row in rows] + constructs = sorted({c for row in all_rows for c in row["constructs"]}) # type: ignore[attr-defined] + report = { + "all": summarise(all_rows), + "layers": {layer: summarise(rows) for layer, rows in sorted(by_layer.items())}, + "distinct_constructs": len(constructs), + } + if args.json: + report["constructs"] = constructs + print(json.dumps(report, indent=2)) + return 0 + layers = list(report["layers"]) # type: ignore[arg-type] + header = f"{'family':<14}{'target':>8}{'all':>8}" + "".join(f"{x[:8]:>10}" for x in layers) + print(header) + total = report["all"] + print( + f"{'models':<14}{'':>8}{total['models']:>8}" + + "".join( # type: ignore[index] + f"{report['layers'][x]['models']:>10}" + for x in layers # type: ignore[index] + ) + ) + for family, target in TARGETS.items(): + shown = "any" if target is None else f"{target:g}" + key = f"pct_{family}" if target is not None else f"n_{family}" + cells = "".join(f"{report['layers'][x][key]:>10}" for x in layers) # type: ignore[index] + print(f"{family:<14}{shown:>8}{total[key]:>8}{cells}") # type: ignore[index] + for metric in ("ctes", "joins", "lines"): + print(f"{metric:<14} {shape(total[metric])}") # type: ignore[index] + for layer in layers: + print(f" {layer:<12} {shape(report['layers'][layer][metric])}") # type: ignore[index] + print(f"distinct constructs: {report['distinct_constructs']}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/duckshop/scripts/relations.py b/benchmarks/duckshop/scripts/relations.py new file mode 100644 index 0000000000..0892daf361 --- /dev/null +++ b/benchmarks/duckshop/scripts/relations.py @@ -0,0 +1,116 @@ +"""SQL that checks for and drops the warehouse objects a DuckShop build creates. + +DuckShop may be built into a schema that holds other objects, so cleanup must name exactly what the +project creates rather than drop the schema or archive whatever looks stale. The list comes from +`sqb compile --json`, which already reflects the target in sqlbuild_local.toml: + + sqb compile --json > target/duckshop-compile.json + python ../scripts/relations.py check target/duckshop-compile.json > target/check.sql + python ../scripts/relations.py drop target/duckshop-compile.json > target/drop.sql + +`check` lists tables, views and functions that already exist under a DuckShop name, and any +SQLBuild state tables (`_sqlbuild_*`) already in the schema. `drop` is one Snowflake Scripting +block that drops every model, snapshot, seed and source table and every function the project +creates. SQLBuild's own state tables (`_sqlbuild_*`) are left alone, since other projects in the +same schema may share them. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from pathlib import Path + +PROJECT_DIR = Path(__file__).resolve().parents[1] / "project" + + +def _function_arguments(relative_path: str) -> list[str]: + text = (PROJECT_DIR / relative_path).read_text() + if relative_path.endswith(".py"): + match = re.search(r"arguments=\{([^}]*)\}", text) + return re.findall(r'"\w+":\s*"([^"]+)"', match.group(1)) if match else [] + match = re.search(r"arguments \(([^)]*)\)", text) + if not match: + return [] + return [part.strip().split(None, 1)[1] for part in match.group(1).split(",") if part.strip()] + + +def managed_sources() -> list[str]: + """Managed source names from sources/*.yml; each loader writes a table of the same name.""" + + names: list[str] = [] + for path in sorted((PROJECT_DIR / "sources").glob("*.yml")): + for block in re.split(r"\n - name: ", path.read_text())[1:]: + if re.search(r"^ managed: true$", block, flags=re.MULTILINE): + names.append(block.split("\n", 1)[0].strip()) + return names + + +def objects(compile_json: dict[str, object]) -> tuple[str, list[tuple[str, str]]]: + """The schema-qualified prefix and each (object kind, name or signature) the project creates.""" + + resources = compile_json["resources"] + functions = resources["functions"] # type: ignore[index] + if not functions: + raise SystemExit( + "compile output has no functions to read the target database and schema from" + ) + database, schema, _ = str(functions[0]["qualified_name"]).split(".") # type: ignore[index] + found: list[tuple[str, str]] = [] + for model in resources["models"]: # type: ignore[index] + kind = "VIEW" if model["materialized"] == "view" else "TABLE" + found.append((kind, model["name"])) + for seed in resources["seeds"]: # type: ignore[index] + found.append(("TABLE", seed["name"])) + for name in managed_sources(): + found.append(("TABLE", name)) + for function in functions: # type: ignore[union-attr] + arguments = ", ".join(_function_arguments(function["relative_path"])) + found.append(("FUNCTION", f"{function['name']}({arguments})")) + return f"{database}.{schema}", found + + +def check_sql(prefix: str, found: list[tuple[str, str]]) -> str: + database, schema = prefix.split(".") + names = sorted({name.split("(")[0].upper() for kind, name in found if kind != "FUNCTION"}) + functions = sorted({name.split("(")[0].upper() for kind, name in found if kind == "FUNCTION"}) + listed = ",\n ".join(f"'{name}'" for name in names) + listed_functions = ",\n ".join(f"'{name}'" for name in functions) + return ( + "SELECT table_name AS object_name, table_type AS object_type, created, last_altered\n" + f"FROM {database}.INFORMATION_SCHEMA.TABLES\n" + f"WHERE table_schema = '{schema.upper()}'\n" + f" AND (\n table_name IN (\n {listed}\n )\n" + " OR STARTSWITH(table_name, '_SQLBUILD_')\n )\n" + "UNION ALL\n" + "SELECT function_name, 'FUNCTION', created, last_altered\n" + f"FROM {database}.INFORMATION_SCHEMA.FUNCTIONS\n" + f"WHERE function_schema = '{schema.upper()}'\n" + f" AND function_name IN (\n {listed_functions}\n )\n" + "ORDER BY object_name" + ) + + +def drop_sql(prefix: str, found: list[tuple[str, str]]) -> str: + order = {"VIEW": 0, "TABLE": 1, "FUNCTION": 2} + lines = [ + f" DROP {kind} IF EXISTS {prefix}.{name};" + for kind, name in sorted(found, key=lambda item: (order[item[0]], item[1])) + ] + return "EXECUTE IMMEDIATE $$\nBEGIN\n" + "\n".join(lines) + "\nEND;\n$$" + + +def main(argv: list[str]) -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("action", choices=["check", "drop"]) + parser.add_argument("compile_json", type=Path, help="output of `sqb compile --json`") + arguments = parser.parse_args(argv) + prefix, found = objects(json.loads(arguments.compile_json.read_text())) + print(check_sql(prefix, found) if arguments.action == "check" else drop_sql(prefix, found)) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/benchmarks/duckshop/tests/test_synthetic_data.py b/benchmarks/duckshop/tests/test_synthetic_data.py new file mode 100644 index 0000000000..ac2ba9ece8 --- /dev/null +++ b/benchmarks/duckshop/tests/test_synthetic_data.py @@ -0,0 +1,428 @@ +"""Offline checks for the DuckShop synthetic data loaders. + +The loaders run against Snowflake, but the rows they return are plain Python, so these tests check +determinism, the shape each managed source declares, and the invariants the staging models and +their audits rely on, without a warehouse. +""" + +from __future__ import annotations + +import hashlib +import json +import re +import sys +from collections import Counter +from datetime import UTC, date, datetime, timedelta +from decimal import Decimal +from pathlib import Path + +import pytest +import yaml + +PROJECT_DIR = Path(__file__).resolve().parents[1] / "project" +sys.path.insert(0, str(PROJECT_DIR)) + +from python.loaders import _world # noqa: E402 +from python.loaders._registry import GENERATORS, generate, scale_from_vars # noqa: E402 + +Rows = list[dict[str, object]] + + +def _sources() -> dict[str, dict[str, object]]: + sources: dict[str, dict[str, object]] = {} + for path in sorted((PROJECT_DIR / "sources").glob("*.yml")): + for source in yaml.safe_load(path.read_text())["sources"]: + sources[source["name"]] = source + return sources + + +SOURCES = _sources() +MANAGED = sorted(name for name, source in SOURCES.items() if source.get("managed")) + + +@pytest.fixture(scope="module") +def tables() -> dict[str, Rows]: + return {name: generate(name) for name in GENERATORS} + + +def _kind(value: object) -> str | None: + """The loader's logical type for a value; one column must not mix kinds.""" + + if value is None: + return None + if isinstance(value, bool): + return "boolean" + if isinstance(value, int): + return "integer" + if isinstance(value, float | Decimal): + return "float" + if isinstance(value, datetime): + return "timestamp" + if isinstance(value, date): + return "date" + if isinstance(value, dict | list): + return "json" + return "string" + + +def _digest(rows: Rows) -> str: + return hashlib.sha256(json.dumps(rows, default=str, sort_keys=True).encode()).hexdigest() + + +def _latest(rows: Rows, key: str) -> list[dict[str, object]]: + """The staging layer's cdc_latest: the highest `_seq` per raw key, dropping deletes.""" + + latest: dict[object, dict[str, object]] = {} + for row in rows: + current = latest.get(row[key]) + if current is None or row["_seq"] > current["_seq"]: # type: ignore[operator] + latest[row[key]] = row + return [row for row in latest.values() if row["_op"] != "D"] + + +def _sku(value: object) -> str: + return re.sub(r"\s+", "", str(value).strip()).replace("_", "-").upper() + + +def test_given_project_sources_when_listing_generators_then_each_managed_source_has_one() -> None: + assert MANAGED, "no managed sources found" + assert sorted(GENERATORS) == MANAGED + unmanaged = sorted(set(SOURCES) - set(MANAGED)) + assert unmanaged == ["finance__fx_rates"] + + +@pytest.mark.parametrize("name", MANAGED) +def test_given_fresh_caches_when_generating_again_then_rows_are_identical(name: str) -> None: + first = _digest(generate(name)) + _world.build_world.cache_clear() + for module in ("_storefront", "_fulfilment", "_inventory", "_clickstream", "_support"): + for value in vars(sys.modules[f"python.loaders.{module}"]).values(): + if hasattr(value, "cache_clear"): + value.cache_clear() + assert _digest(generate(name)) == first + + +@pytest.mark.parametrize("name", MANAGED) +def test_given_generated_rows_when_compared_to_source_yaml_then_shape_matches( + name: str, tables: dict[str, Rows] +) -> None: + rows = tables[name] + declared = [column["name"] for column in SOURCES[name]["columns"]] # type: ignore[index] + assert rows, f"{name} generated no rows" + for row in rows: + assert list(row) == declared or set(row) == set(declared), ( + f"{name}: {sorted(set(row) ^ set(declared))}" + ) + for column in declared: + kinds = {_kind(row[column]) for row in rows} - {None} + assert len(kinds) <= 1, f"{name}.{column} mixes {kinds}" + required = [c["name"] for c in SOURCES[name]["columns"] if c.get("nullable") is False] # type: ignore[index] + for column in required: + assert all(row[column] is not None for row in rows), f"{name}.{column} has NULLs" + + +def _delivery_keys() -> list[tuple[str, str]]: + """(source, id column) for append-only feeds audited with unique_per_delivery.""" + + keys = [] + for name in MANAGED: + for column in SOURCES[name]["columns"]: # type: ignore[union-attr] + if "unique_per_delivery" in (column.get("audits") or []): + keys.append((name, column["name"])) + return keys + + +DELIVERY_KEYS = _delivery_keys() + + +@pytest.mark.parametrize(("name", "column"), DELIVERY_KEYS, ids=[k[0] for k in DELIVERY_KEYS]) +def test_given_append_only_feed_when_generated_then_ids_repeat_only_across_loads( + name: str, column: str, tables: dict[str, Rows] +) -> None: + rows = tables[name] + deliveries = Counter((row[column], row["_loaded_at"]) for row in rows) + assert max(deliveries.values()) == 1 + ids = Counter(row[column] for row in rows) + assert len(ids) < len(rows) or name == "fulfilment__carrier_webhooks" + + +def test_given_append_only_feeds_when_listed_then_all_nine_are_audited() -> None: + assert len(DELIVERY_KEYS) == 9 + + +@pytest.mark.parametrize( + "name", [n for n in MANAGED if "_seq" in [c["name"] for c in SOURCES[n]["columns"]]] +) # type: ignore[index] +def test_given_change_log_when_generated_then_metadata_is_ordered_and_valid( + name: str, tables: dict[str, Rows] +) -> None: + rows = tables[name] + assert [row["_seq"] for row in rows] == list(range(1, len(rows) + 1)) + assert {row["_op"] for row in rows} <= {"I", "U", "D"} + loaded = [row["_loaded_at"] for row in rows] + assert all(isinstance(value, datetime) and value.tzinfo is None for value in loaded) + assert loaded == sorted(loaded) + + +def test_given_all_change_logs_when_generated_then_updates_deletes_and_late_rows_occur( + tables: dict[str, Rows], +) -> None: + ops = Counter(row["_op"] for name in MANAGED for row in tables[name] if "_op" in row) + assert ops["U"] > 1000 and ops["D"] > 10 + duplicated = 0 + for name in MANAGED: + seen = Counter( + json.dumps( + {k: v for k, v in row.items() if k not in ("_seq", "_loaded_at")}, + default=str, + sort_keys=True, + ) + for row in tables[name] + ) + duplicated += sum(1 for count in seen.values() if count > 1) + assert duplicated > 100 + # A late order version lands after a newer one, so _seq order disagrees with status time order. + rank = { + "pending": 1, + "paid": 2, + "packed": 3, + "shipped": 4, + "delivered": 5, + "returned": 6, + "cancelled": 7, + } + last_rank: dict[object, int] = {} + out_of_order = 0 + for row in tables["storefront__orders"]: + status_rank = rank[str(row["status"]).lower()] + if status_rank < last_rank.get(row["order_id"], 0): + out_of_order += 1 + last_rank[row["order_id"]] = max(status_rank, last_rank.get(row["order_id"], 0)) + assert out_of_order > 5 + + +# Staging keys after normalisation: (source, raw key column, normalised key). +STAGING_KEYS = [ + ("catalogue__brands", "brand_code", lambda r: str(r["brand_code"]).upper()), + ("catalogue__product_variants", "sku", lambda r: _sku(r["sku"])), + ("catalogue__bundles", "bundle_sku", lambda r: _sku(r["bundle_sku"])), + ("catalogue__suppliers", "supplier_code", lambda r: str(r["supplier_code"]).upper()), + ("storefront__promotions", "promo_code", lambda r: str(r["promo_code"]).strip().upper()), + ("inventory__warehouses", "warehouse_code", lambda r: str(r["warehouse_code"]).upper()), + ("storefront__orders", "order_id", lambda r: r["order_id"]), + ("storefront__customers", "customer_id", lambda r: r["customer_id"]), +] + + +@pytest.mark.parametrize( + ("name", "key", "normalise"), STAGING_KEYS, ids=[k[0] for k in STAGING_KEYS] +) +def test_given_latest_versions_when_normalising_keys_then_keys_stay_unique( + name, key, normalise, tables: dict[str, Rows] +) -> None: + keys = [normalise(row) for row in _latest(tables[name], key)] + assert len(keys) == len(set(keys)) + + +# Raw columns that staging audits require to be present on the surviving version. +STAGING_NOT_NULL = { + "storefront__customers": ["region_code", "created_at"], + "storefront__orders": ["customer_id", "storefront_code", "currency_code", "placed_at"], + "storefront__order_lines": ["order_id", "line_no", "sku"], + "storefront__payments": ["order_id", "amount_minor"], + "storefront__refunds": ["payment_id", "amount_minor"], + "storefront__customer_addresses": ["customer_id", "country_code"], + "storefront__gift_cards": ["currency_code", "initial_amount_minor"], + "storefront__loyalty_ledger": ["loyalty_account_id", "points_delta", "posted_at"], + "storefront__order_status_events": ["order_id", "to_status", "changed_at"], + "fulfilment__shipments": ["order_id", "warehouse_code", "carrier_code"], + "fulfilment__returns": ["order_id", "sku"], + "inventory__stock_counts": ["sku", "warehouse_code", "counted_at_local"], + "support__tickets": ["created_at"], + "growth__campaign_spend": ["spend_date"], +} + + +@pytest.mark.parametrize("name", sorted(STAGING_NOT_NULL)) +def test_given_latest_versions_when_checking_audited_columns_then_none_are_null( + name: str, tables: dict[str, Rows] +) -> None: + key = SOURCES[name]["columns"][0]["name"] # type: ignore[index] + for row in _latest(tables[name], key): + for column in STAGING_NOT_NULL[name]: + assert row[column] is not None, f"{name}.{column} is NULL for {row[key]}" + + +ACCEPTED = { + ("storefront__reviews", "rating"): None, + ("clickstream__app_events", "platform"): {"phone", "tablet"}, + ("clickstream__email_events", "event_type"): { + "sent", + "delivered", + "open", + "click", + "bounce", + "unsubscribe", + "complaint", + }, + ("support__ticket_comments", "author_type"): {"agent", "customer", "system"}, + ("support__csat_responses", "score"): {1, 2, 3, 4, 5}, + ("fulfilment__return_inspections", "grade"): {"A", "B", "C", "SCRAP"}, + ("catalogue__product_media", "media_type"): {"image", "video", "model3d"}, + ("growth__newsletter_subscriptions", "list_status"): { + "subscribed", + "unsubscribed", + "pending", + "cleaned", + }, + ("partners__partner_orders", "schema_version"): {1, 2}, +} + + +@pytest.mark.parametrize( + ("name", "column"), [k for k, v in ACCEPTED.items() if v], ids=lambda value: str(value) +) +def test_given_coded_columns_when_normalised_then_values_are_accepted( + name: str, column: str, tables: dict[str, Rows] +) -> None: + accepted = ACCEPTED[(name, column)] + normalise = {"grade": lambda v: str(v).strip().upper()}.get( + column, lambda v: v.lower() if isinstance(v, str) else v + ) + values = {normalise(row[column]) for row in tables[name] if row[column] is not None} + assert values <= accepted # type: ignore[operator] + + +def test_given_related_tables_when_joining_then_every_reference_resolves( + tables: dict[str, Rows], +) -> None: + orders = {row["order_id"] for row in tables["storefront__orders"]} + customers = {row["customer_id"] for row in tables["storefront__customers"]} + payments = {row["payment_id"] for row in tables["storefront__payments"]} + addresses = { + row["address_id"] for row in _latest(tables["storefront__customer_addresses"], "address_id") + } + shipments = {row["shipment_id"] for row in tables["fulfilment__shipments"]} + assert {row["order_id"] for row in tables["storefront__order_lines"]} <= orders + assert {row["order_id"] for row in tables["storefront__payments"]} <= orders + assert {row["payment_id"] for row in tables["storefront__refunds"]} <= payments + assert {row["customer_id"] for row in tables["storefront__orders"]} <= customers + assert {row["shipping_address_id"] for row in tables["storefront__orders"]} <= addresses + assert {row["order_id"] for row in tables["fulfilment__shipments"]} <= orders + assert {row["shipment_id"] for row in tables["fulfilment__shipment_packages"]} <= shipments + assert {row["order_id"] for row in tables["fulfilment__returns"]} <= orders + tracking = Counter( + row["tracking"]["number"] for row in _latest(tables["fulfilment__shipments"], "shipment_id") + ) # type: ignore[index] + assert {re.sub(r"[^A-Za-z0-9]", "", str(t)).upper() for t in tracking}.__len__() == len( + tracking + ) + + +def test_given_lookup_customer_when_listing_orders_then_history_exists( + tables: dict[str, Rows], +) -> None: + lookup = [row for row in tables["storefront__orders"] if row["customer_id"] == "C000001"] + assert len({row["order_id"] for row in lookup}) >= 5 + + +def test_given_generated_world_when_scanning_then_edge_cases_are_present( + tables: dict[str, Rows], +) -> None: + orders = _latest(tables["storefront__orders"], "order_id") + assert {row["currency_code"] for row in orders} >= { + "JPY", + "GBP", + "KWD", + } # 0, 2 and 3 decimal places + offsets = {row["placed_at"].utcoffset() for row in orders} # type: ignore[union-attr] + assert len(offsets) >= 6 + placed = [row["placed_at"] for row in orders] + # Orders either side of the US and EU spring-forward instants. + assert any( + p.astimezone(UTC).replace(tzinfo=None) == datetime(2026, 3, 29, 0, 55) for p in placed + ) # type: ignore[union-attr] + assert any(p.astimezone(UTC).replace(tzinfo=None) == datetime(2026, 3, 8, 7, 5) for p in placed) # type: ignore[union-attr] + # Partner order messages with a JSON-null id and the id elsewhere. + nulls = [ + row + for row in tables["partners__partner_orders"] + if "orderId" in row["payload"] and row["payload"]["orderId"] is None + ] # type: ignore[operator,index] + assert nulls + # Returns, refunds and regex-worthy ticket text. + assert tables["fulfilment__returns"] and tables["storefront__refunds"] + assert any( + re.search(r"DS-O-[0-9]+", str(row["body"])) for row in tables["support__ticket_comments"] + ) + # Web sessions that run past midnight UTC. + events = [row for row in tables["clickstream__web_events"] if "ts_ms" in row["event"]] # type: ignore[operator] + by_visitor: dict[object, list[datetime]] = {} + for row in events: + moment = datetime(1970, 1, 1) + timedelta(milliseconds=int(row["event"]["ts_ms"])) # type: ignore[index] + by_visitor.setdefault(str(row["event"]["anonymous_id"]).lower(), []).append(moment) # type: ignore[index] + crossings = 0 + for moments in by_visitor.values(): + moments.sort() + crossings += sum( + 1 + for a, b in zip(moments, moments[1:], strict=False) + if a.date() != b.date() and (b - a) <= timedelta(minutes=30) + ) + assert crossings >= 10 + # JSON nulls inside payloads, distinct from SQL NULL columns. + assert any('": null' in json.dumps(row["specs"]) for row in tables["catalogue__products"]) + + +def test_given_larger_scale_when_generating_then_volume_grows() -> None: + assert scale_from_vars({}) == 1 + assert scale_from_vars({"data_scale": "3"}) == 3 + with pytest.raises(ValueError): + scale_from_vars({"data_scale": 0}) + small = len(generate("storefront__orders", 1)) + assert len(generate("storefront__orders", 2)) > 1.8 * small + + +class _FakeContext: + def __init__(self) -> None: + self.vars = {"data_scale": "1"} + self.destination = '"DUCKSHOP"."DEV"."CATALOGUE__PRODUCTS"' + self.sql: list[str] = [] + + def execute_sql(self, sql: str) -> None: + self.sql.append(sql) + + def result(self, **kwargs: object) -> dict[str, object]: + return kwargs + + +@pytest.mark.parametrize( + "name", ["catalogue__products", "storefront__orders", "inventory__warehouses"] +) +def test_given_loader_context_when_loading_then_table_is_replaced_with_typed_inserts( + name: str, +) -> None: + from python.loaders._registry import load + + context = _FakeContext() + result = load(context, name) + rows = generate(name) + assert result == {"metadata": {"rows": len(rows), "scale": 1}} + create, *inserts = context.sql + assert create.startswith(f"CREATE OR REPLACE TABLE {context.destination} (") + assert len(inserts) == -(-len(rows) // 500) + assert all("PARSE_JSON(column" in sql for sql in inserts) + assert all("VALUES\n(PARSE_JSON" not in sql for sql in inserts) + sqlglot = pytest.importorskip("sqlglot") + for sql in context.sql: + sqlglot.parse_one(sql, read="snowflake") + + +def test_given_special_characters_when_rendering_literals_then_they_are_escaped() -> None: + from python.loaders._writer import literal + + assert literal("it's") == "'it''s'" + assert literal({"note": "a\nb"}) == '\'{"note": "a\\\\nb"}\'' + assert literal(None) == "NULL" + assert literal(True) == "TRUE" + assert literal(date(2026, 3, 31)) == "'2026-03-31'" diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/_test_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/_test_types.py index 60270542b0..fad2af844a 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/compile/_test_types.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/_test_types.py @@ -1239,3 +1239,13 @@ class UndecodableSelectorTestCase: output_flags: tuple[str, ...] expected_code: str expected_escaped_selector: str + + +@dataclass(frozen=True) +class BenchmarkProjectCompileTestCase: + """A benchmark project in the repository and the diagnostics its compile may report.""" + + description: str + project_dir: Path + expected_errors: int + expected_warnings: int diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_duckshop_benchmark_compile.py b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_duckshop_benchmark_compile.py new file mode 100644 index 0000000000..eccd67e5fb --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/compile/test_duckshop_benchmark_compile.py @@ -0,0 +1,61 @@ +"""E2E guard that the DuckShop benchmark project compiles cleanly with this checkout's sqlbuild.""" + +from __future__ import annotations + +import json +import shutil +import subprocess +from pathlib import Path +from typing import Any + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.compile._test_types import ( + BenchmarkProjectCompileTestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import REPO_ROOT, run_sqb + +_LOCAL_ONLY_PATHS: tuple[str, ...] = ( + "target", + "logs", + ".sqlbuild", + "sqlbuild_local.toml", + "__pycache__", +) + + +@pytest.mark.parametrize( + "test_case", + [ + BenchmarkProjectCompileTestCase( + description="duckshop", + project_dir=REPO_ROOT / "benchmarks" / "duckshop" / "project", + expected_errors=0, + expected_warnings=0, + ) + ], + ids=lambda case: case.description, +) +def test_given_benchmark_project_when_compiling_then_reports_no_errors_or_warnings( + test_case: BenchmarkProjectCompileTestCase, tmp_path: Path +) -> None: + project_dir: Path = tmp_path / test_case.description + shutil.copytree( + test_case.project_dir, + project_dir, + ignore=shutil.ignore_patterns(*_LOCAL_ONLY_PATHS), + ) + + result: subprocess.CompletedProcess[str] = run_sqb( + command=("--no-color", "compile", "--json"), + project_dir=project_dir, + ) + + assert result.returncode == 0, result.stdout + result.stderr + report: dict[str, Any] = json.loads(result.stdout) + summary: dict[str, int] = report["summary"] + diagnostics: list[dict[str, Any]] = report["diagnostics"] + assert summary["errors"] == test_case.expected_errors, diagnostics + assert summary["warnings"] == test_case.expected_warnings, diagnostics + assert summary["models"] > 0 + assert summary["tests"] > 0