test(duckshop): edge cases for every construct translation tools refuse

benchmarks/duckshop/README.md

@@ -119,10 +119,13 @@ project and does not change it:
 - raw `edge__*` sources in `sources/edge_cases.yml`, loaded by `python/loaders/edge_cases.py` from
   fixed rows that `data_scale` does not change, so core data and core expectations stay the same;
 - Python functions in `functions/python/edge_*.py`, tagged `edge_case`, and their tests in
-  `tests/unit/functions/python/test_edge_*.sql`.
+  `tests/unit/functions/python/test_edge_*.sql`;
+- a SQL table function, `functions/sql/edge_basket_lines_for.sql`, and a JavaScript function,
+  `edge_running_peaks`, that a pre-hook of `int_edge_basket_running_peaks` creates
+  (`models/intermediate/edge_cases/_sqlbuild/_hooks/sql/`) and SQLBuild does not declare.
 
-It has 72 models (12 staging, 50 intermediate, 10 marts), 5 Python functions, 97 unit tests and 4
-scenarios, in eight groups:
+It has 160 models (13 staging, 133 intermediate, 14 marts), 5 Python functions, a SQL table
+function, a hook-created JavaScript function, 185 unit tests and 4 scenarios, in nine groups:
 
 | Group | Covers |
 |-------|--------|
@@ -134,6 +137,7 @@ scenarios, in eight groups:
 | Lineage and analysis | clauses whose columns decide rows without being output (`WHERE`, `QUALIFY`, `GROUP BY` by position and `ALL`, `ORDER BY` under `LIMIT`, `EXISTS`, a `FLATTEN` input, `ROLLUP` by position, `EXCEPT` under `LIMIT`); `RIGHT JOIN ... USING`, `NATURAL JOIN`, a `USING` key under a later outer join, and unmatched `ASOF JOIN` rows; `CONNECT BY` beside a column named `level`; a recursive CTE without `RECURSIVE` whose columns shift per iteration; `LAST_DAY` with a date part; positional columns and the names Snowflake gives unaliased expressions; `TEXT(5)` and widened `COUNT` contracts |
 | Python functions | volatile tokens and numbers from a column, a constant and no argument, and `DIV0` of a volatile call beside look-alike names, checked by assertions that hold for any values; a handler answering NULL input without `STRICT`; Python `None` as JSON null, and datetimes, Decimals and integers past 2^53 handed to `VARIANT` and `FLOAT` results |
 | Scenarios and lifecycle | typed values across model boundaries; `VARIANT` expected output; a star under a lower-case contract and a quoted camelCase export; a snapshot keyed and compared on quoted camelCase columns; a cursor-window incremental model over far-apart fixture loads; a change log merged across two loads, checked against a full rebuild |
+| Translation refusals | one model per construct that translation tools refuse or guard, most putting a construct refused only under strict checking before one refused by default, so a model shows both: `FLOAT` values through sums, averages, running totals, arrays, objects, `HASH`, `TO_CHAR` and `::NUMBER(38, 10)`; `VARIANT` dates, decimals and XML; `ARRAY_AGG` without `WITHIN GROUP`, sparse arrays and `ARRAY(NUMBER)` casts; `REDUCE`, `FILTER` and `TRANSFORM` lambdas; patterns and `REGEXP_REPLACE` replacements built from columns, and lazy patterns; `TRY_TO_DATE` with a format from a column; zoned literals, `CONVERT_TIMEZONE` from `CURRENT_TIMESTAMP`, `SEQ8`, `GENERATOR` with an expression `ROWCOUNT`; `PIVOT`, `ANY_VALUE`, `IGNORE NULLS` around `IFF`, unaliased columns read by Snowflake's name, `UNION` branches whose column is always NULL; `TO_NUMBER`, `TO_DECIMAL` and `TRY_BASE64_DECODE_BINARY`; an open (uncontracted) source `edge__supplier_notes`, the table function and the hook-created function. `rpt_edge_case_model_keys` and `rpt_edge_case_array_model_keys` read every model in the group, so a refusal there also refuses a downstream model |
 
 Unit-test expectations in the edge-case area are captured on Snowflake under the `edge` target's
 pinned session, rather than derived by hand; scenario expectations, and the assertions in tests
@@ -158,6 +162,10 @@ earlier ones:
   same results on every run: added SKUs are sorted, the weekday average utilisation is an exact
   decimal sum over a count, the busiest slot of equally busy ones is the earliest, and the top
   exception code of equally frequent ones is the first in alphabetical order.
+- The mocks of `stg_wishlist_items`, `stg_purchase_order_lines` and `int_loyalty_tier_changes` in
+  three unit tests list their columns in the models' declared order, which unit-test planning
+  requires for a `SELECT *` reader. Only compiled tests change; compiled models are the same, so
+  model benchmarks stay comparable.
 
 ## Status
 
@@ -200,7 +208,7 @@ The edge-case area, verified on Snowflake with SQLBuild built after 0.185.0 on t
 | Step | Result |
 |------|--------|
 | `sqb build --select tag:edge_case` | Passes; the disabled model is skipped |
-| `sqb test --select tag:edge_case` | 62 of 62 pass |
+| `sqb test --select tag:edge_case` | 62 of 62 pass; with SQLBuild 0.187.0, 185 of 185 pass |
 | `sqb scenario test --select edge_cases` | 4 of 4 pass |
 | Two-load replay of the delivery change log | The merged status equals the full rebuild after each load |
 

benchmarks/duckshop/RUNBOOK.md

@@ -182,6 +182,12 @@ the area's Python functions (`functions/python/edge_*.py`, tagged `edge_case`),
 models and runs their tests and audits. `fct_edge_delivery_events` is disabled and reports
 `skip: disabled` (see [Pending acceptance cases](#pending-acceptance-cases)).
 
+`int_edge_basket_running_peaks` calls `edge_running_peaks`, a JavaScript function its pre-hook
+creates in the target schema. SQLBuild does not declare it, so the model's unit test needs a build
+of that model first, and the model calls the function without a schema: set `database` and
+`schema` on the edge connection as well as on the target, so that the session's current schema is
+the target schema. `relations.py drop` drops the function too.
+
 `int_edge_ticket_tokens` calls volatile Python functions, so its values change on every run; its
 test compares only the deterministic columns and asserts facts that hold for any values, such as
 every token being distinct.

benchmarks/duckshop/project/functions/sql/edge_basket_lines_for.sql

@@ -0,0 +1,29 @@
+FUNCTION (
+  description "Lines of one basket with their SKU and tag, for the edge-case table-function reader",
+  tags [edge_case],
+  arguments (p_basket_id VARCHAR),
+  returns table (
+    line_id VARCHAR,
+    line_no INT,
+    sku VARCHAR,
+    tag VARCHAR
+  )
+);
+
+WITH lines AS (
+  SELECT *
+  FROM __ref("stg_edge_basket_lines")
+),
+
+final AS (
+  SELECT
+    l.line_id,
+    l.line_no,
+    l.sku,
+    l.tag
+  FROM lines AS l
+  WHERE l.basket_id = p_basket_id
+)
+
+SELECT *
+FROM final

benchmarks/duckshop/project/models/intermediate/edge_cases/_sqlbuild/_hooks/sql/create_edge_running_peaks.sql

@@ -0,0 +1,24 @@
+HOOK (
+  description """
+    Create the edge-case JavaScript function edge_running_peaks, which turns a list of numbers into
+    the running maximum at each position. SQLBuild does not declare functions a hook creates.
+  """,
+);
+
+CREATE OR REPLACE FUNCTION @@CTX:destination.database.@@CTX:destination.schema.edge_running_peaks(
+  line_numbers ARRAY
+)
+RETURNS ARRAY
+LANGUAGE JAVASCRIPT
+AS $$
+  var peaks = [];
+  var peak = null;
+  for (var i = 0; i < LINE_NUMBERS.length; i++) {
+    var value = LINE_NUMBERS[i];
+    if (value !== null && (peak === null || value > peak)) {
+      peak = value;
+    }
+    peaks.push(peak);
+  }
+  return peaks;
+$$

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_amount_percent_texts.sql

@@ -0,0 +1,21 @@
+MODEL (
+  description """
+    One row per reconciliation check with a third of its amount as a percentage written with a
+    number format of four whole digits
+  """,
+  tags [edge_case],
+  columns (
+    check_id (type VARCHAR, nullable false, audits [not_null, unique]),
+    amount_percent (type VARCHAR),
+  ),
+);
+
+WITH final AS (
+  SELECT
+    a.check_id,
+    TO_VARCHAR(ROUND(a.amount * 100 / 3, 1), 'FM9990.0') || '%' AS amount_percent
+  FROM __ref("stg_edge_amount_checks") AS a
+)
+
+SELECT *
+FROM final

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_basket_distinct_tags.sql

@@ -0,0 +1,23 @@
+MODEL (
+  description """
+    One row per distinct tag of the basket lines, held as VARIANT and read back as text
+  """,
+  tags [edge_case],
+  columns (
+    tag_text (type VARCHAR, nullable false, audits [not_null, unique]),
+  ),
+);
+
+WITH distinct_tags AS (
+  SELECT DISTINCT TO_VARIANT(l.tag) AS tag_value
+  FROM __ref("stg_edge_basket_lines") AS l
+),
+
+final AS (
+  SELECT d.tag_value::VARCHAR AS tag_text
+  FROM distinct_tags AS d
+  WHERE d.tag_value IS NOT NULL
+)
+
+SELECT *
+FROM final

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_basket_line_lookups.sql

@@ -0,0 +1,22 @@
+MODEL (
+  description """
+    One row per line of one fixed basket, read through the edge-case table function
+  """,
+  tags [edge_case],
+  columns (
+    line_id (type VARCHAR, nullable false, audits [not_null, unique]),
+    line_no (type INT),
+    sku (type VARCHAR),
+  ),
+);
+
+WITH final AS (
+  SELECT
+    l.line_id,
+    l.line_no,
+    l.sku
+  FROM __table_fn("edge_basket_lines_for")('B-1') AS l
+)
+
+SELECT *
+FROM final

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_basket_running_peaks.sql

@@ -0,0 +1,33 @@
+MODEL (
+  description """
+    One row per basket with the last running peak of its line numbers, from a JavaScript
+    function that a pre-hook creates and SQLBuild does not declare
+  """,
+  tags [edge_case],
+  pre_hooks [sql("create_edge_running_peaks")],
+  columns (
+    basket_id (type VARCHAR, nullable false, audits [not_null, unique]),
+    peak_count (type NUMBER(38, 0)),
+    last_peak (type NUMBER(38, 0)),
+  ),
+);
+
+WITH peaks AS (
+  SELECT
+    l.basket_id,
+    edge_running_peaks(ARRAY_AGG(l.line_no) WITHIN GROUP (ORDER BY l.line_no)) AS peaks
+  FROM __ref("stg_edge_basket_lines") AS l
+  GROUP BY
+    l.basket_id
+),
+
+final AS (
+  SELECT
+    p.basket_id,
+    ARRAY_SIZE(p.peaks)::NUMBER(38, 0) AS peak_count,
+    p.peaks[ARRAY_SIZE(p.peaks) - 1]::NUMBER(38, 0) AS last_peak
+  FROM peaks AS p
+)
+
+SELECT *
+FROM final

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_basket_sku_part_filters.sql

@@ -0,0 +1,21 @@
+MODEL (
+  description """
+    One row per basket line with the number of parts of its SKU longer than one character, kept
+    by FILTER
+  """,
+  tags [edge_case],
+  columns (
+    line_id (type VARCHAR, nullable false, audits [not_null, unique]),
+    long_part_count (type NUMBER(38, 0)),
+  ),
+);
+
+WITH final AS (
+  SELECT
+    l.line_id,
+    ARRAY_SIZE(FILTER(SPLIT(l.sku, '-'), part -> LENGTH(part) > 1))::NUMBER(38, 0) AS long_part_count
+  FROM __ref("stg_edge_basket_lines") AS l
+)
+
+SELECT *
+FROM final

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_basket_tag_list_reads.sql

@@ -0,0 +1,43 @@
+MODEL (
+  description """
+    One row per basket with reads of its tags in line order: overlap with a fixed list, distinct
+    tags, the position and count of one tag, the tags upper-cased, a time read from the basket
+    id and the size of a bracket array literal
+  """,
+  tags [edge_case],
+  columns (
+    basket_id (type VARCHAR, nullable false, audits [not_null, unique]),
+    has_colour (type BOOLEAN),
+    distinct_tag_count (type NUMBER(38, 0)),
+    red_position (type NUMBER(38, 0)),
+    non_red_count (type NUMBER(38, 0)),
+    upper_tags (type VARCHAR),
+    basket_time (type TIME),
+    pair_size (type NUMBER(38, 0)),
+  ),
+);
+
+WITH tag_lists AS (
+  SELECT
+    l.basket_id,
+    ARRAY_AGG(l.tag) WITHIN GROUP (ORDER BY l.line_no) AS tags
+  FROM __ref("stg_edge_basket_lines") AS l
+  GROUP BY
+    l.basket_id
+),
+
+final AS (
+  SELECT
+    t.basket_id,
+    ARRAYS_OVERLAP(t.tags, ['red', 'blue'])::BOOLEAN AS has_colour,
+    ARRAY_SIZE(ARRAY_DISTINCT(t.tags))::NUMBER(38, 0) AS distinct_tag_count,
+    ARRAY_POSITION('red'::VARIANT, t.tags)::NUMBER(38, 0) AS red_position,
+    ARRAY_SIZE(ARRAY_REMOVE(t.tags, 'red'::VARIANT))::NUMBER(38, 0) AS non_red_count,
+    ARRAY_TO_STRING(TRANSFORM(t.tags, tag -> UPPER(tag)), ',')::VARCHAR AS upper_tags,
+    TRY_TO_TIME(t.basket_id)::TIME AS basket_time,
+    ARRAY_SIZE([t.basket_id, ARRAY_SIZE(t.tags)])::NUMBER(38, 0) AS pair_size
+  FROM tag_lists AS t
+)
+
+SELECT *
+FROM final

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_callback_clock_times.sql

@@ -0,0 +1,23 @@
+MODEL (
+  description """
+    One row per gateway callback with its free text read as a date, and the time of day of the
+    text read as a zoned timestamp
+  """,
+  tags [edge_case],
+  columns (
+    callback_id (type VARCHAR, nullable false, audits [not_null, unique]),
+    text_date (type DATE),
+    text_time (type TIME),
+  ),
+);
+
+WITH final AS (
+  SELECT
+    g.callback_id,
+    TO_DATE(CAST(g.business_date AS VARCHAR)) AS text_date,
+    TO_TIME(TRY_TO_TIMESTAMP_TZ(g.event_time_text))::TIME AS text_time
+  FROM __ref("stg_edge_gateway_callbacks") AS g
+)
+
+SELECT *
+FROM final

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_callback_date_lists.sql

@@ -0,0 +1,22 @@
+MODEL (
+  description """
+    One row per merchant zone with the distinct business dates of its callbacks, held as VARIANT
+  """,
+  tags [edge_case],
+  columns (
+    merchant_zone (type VARCHAR, nullable false, audits [not_null, unique]),
+    date_count (type NUMBER(38, 0)),
+  ),
+);
+
+WITH final AS (
+  SELECT
+    g.merchant_zone,
+    ARRAY_SIZE(ARRAY_AGG(DISTINCT TO_VARIANT(g.business_date)))::NUMBER(38, 0) AS date_count
+  FROM __ref("stg_edge_gateway_callbacks") AS g
+  GROUP BY
+    g.merchant_zone
+)
+
+SELECT *
+FROM final

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_callback_day_documents.sql

@@ -0,0 +1,30 @@
+MODEL (
+  description """
+    One row per merchant zone with the number of day documents, objects holding a DATE,
+    aggregated from its callbacks
+  """,
+  tags [edge_case],
+  columns (
+    merchant_zone (type VARCHAR, nullable false, audits [not_null, unique]),
+    document_count (type NUMBER(38, 0)),
+  ),
+);
+
+WITH day_documents AS (
+  SELECT
+    g.merchant_zone,
+    ARRAY_AGG(OBJECT_CONSTRUCT_KEEP_NULL('id', g.callback_id, 'day', g.business_date)) AS documents
+  FROM __ref("stg_edge_gateway_callbacks") AS g
+  GROUP BY
+    g.merchant_zone
+),
+
+final AS (
+  SELECT
+    d.merchant_zone,
+    ARRAY_SIZE(d.documents)::NUMBER(38, 0) AS document_count
+  FROM day_documents AS d
+)
+
+SELECT *
+FROM final

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_callback_day_starts.sql

@@ -0,0 +1,20 @@
+MODEL (
+  description """
+    One row per gateway callback with the start of the day of its zoned time
+  """,
+  tags [edge_case],
+  columns (
+    callback_id (type VARCHAR, nullable false, audits [not_null, unique]),
+    day_start (type TIMESTAMP_TZ),
+  ),
+);
+
+WITH final AS (
+  SELECT
+    g.callback_id,
+    DATE_TRUNC('day', g.event_at) AS day_start
+  FROM __ref("stg_edge_gateway_callbacks") AS g
+)
+
+SELECT *
+FROM final