test(benchmarks): pin DuckShop edge array order and guard the core copy by checksum

Sort the results of ARRAY_EXCEPT, ARRAY_DISTINCT and OBJECT_KEYS, whose
element order is unspecified, and recapture their expectations. The
core-copy compile guard now compares the copy's compiled files with a
recorded checksum, which core_project.py --checksum prints. The core copy
also leaves out the edge Rule and its tests, renamed with an edge_
prefix, and the marked edge blocks of sqlbuild_project.toml. The Rule
reads only the header's top-level tags, past strings holding ');'. The
README and RUNBOOK say which expectations are captured and which are
written by hand, that a schema built with the old identifier-case
setting needs a clean rebuild, and that DEFAULT_DDL_COLLATION must stay
at its default on the edge schema.

Author: kvlonge

benchmarks/duckshop/README.md

@@ -129,15 +129,18 @@ groups:
 | Ordering, numbers and text | NULL ordering under `DESC`; ties in ordered aggregates, `MAX_BY`, `MODE` and `ANY_VALUE`; default window frames; `UNION ALL BY NAME`; `ROUND` half-way values, decimal division, `MOD` and 38-digit amounts; integers past 2^53 against `FLOAT`; regular expressions and backslash escapes; `LIKE ... ESCAPE` and `ILIKE ANY`; `SPLIT_PART` and `SUBSTR` before the start; `NOT IN` with a NULL; names in many scripts |
 | Scenarios and lifecycle | typed values across model boundaries; `VARIANT` expected output; a star under a lower-case contract and a quoted camelCase export; a cursor-window incremental model over far-apart fixture loads; a change log merged across two loads, checked against a full rebuild |
 
-Every expected value in the edge-case area was captured on Snowflake under the `edge` target's
-pinned session, rather than derived by hand. `fct_edge_delivery_events` is disabled: it is the
+Unit-test expectations in the edge-case area were captured on Snowflake under the `edge` target's
+pinned session, rather than derived by hand; scenario expectations, and the assertions in tests
+about ties, are written by hand from those captured values. `fct_edge_delivery_events` is disabled: it is the
 acceptance case for cursor `delete_insert` models that emit rows outside the window they replace.
 See [RUNBOOK.md](RUNBOOK.md), step 7.
 
 Benchmarks and core measurements leave the area out:
 
 - `python benchmarks/duckshop/scripts/core_project.py <destination>` copies the project without
-  it; the copy compiles to the same files as the project did before the area existed;
+  it. The core-copy compile guard compares the copy's compiled files with a recorded checksum
+  (`core_project.py --checksum`), so a change to the core project's compiled SQL fails the guard
+  until the checksum is updated on purpose and noted for the benchmark;
 - warehouse runs exclude it with `--exclude tag:edge_case "source:edge__*"`, and scenario runs
   with `--exclude edge_cases`;
 - the construct census counts the core project unless given `--area edge` or `--area all`.

benchmarks/duckshop/RUNBOOK.md

@@ -54,6 +54,11 @@ no DuckShop names; if it does, stop and pick another schema. Note which state ta
 The list includes the edge-case area's names; run the check with `--target edge` too before its
 first build.
 
+The project no longer sets `QUOTED_IDENTIFIERS_IGNORE_CASE = true` on its connections. A schema
+built while it did may hold objects and SQLBuild state that no longer resolve under Snowflake's
+default case rules, so rebuild into a clean schema: drop everything as in step 8, or pick a new
+schema, before the first build.
+
 ## 3. Load the synthetic data
 
 ```bash
@@ -162,6 +167,11 @@ values were captured under: UTC, and Snowflake's defaults for every parameter th
 result or how text is parsed, including `QUOTED_IDENTIFIERS_IGNORE_CASE = false`. Run it on that
 target only; under another session some expected values differ by design.
 
+`DEFAULT_DDL_COLLATION` also changes results (comparisons, ordering and `LIKE` on text columns),
+but it is an account, database or schema parameter, not a session one, so the connection cannot
+pin it. The edge target's schema, and its database and account, must leave it at the default (no
+collation). Check with `SHOW PARAMETERS LIKE 'DEFAULT_DDL_COLLATION' IN SCHEMA <database>.<schema>`.
+
 ### Load, build and test
 
 ```bash
@@ -202,8 +212,8 @@ chose.
 
 ### Capture expected values
 
-Edge-case expectations are Snowflake's own results on the fixtures, not values derived by hand.
-To capture them for a new or changed edge test, write its `__expected__<model>` CTE with the
+Unit-test expectations in the edge-case area are Snowflake's own results on the fixtures, not
+values derived by hand. To capture them for a new or changed edge test, write its `__expected__<model>` CTE with the
 columns to compare (any values, such as `NULL AS column_name`), then run:
 
 ```bash
@@ -216,7 +226,8 @@ replaces each expected CTE with the actual rows as typed literals. Without a tes
 recaptures every edge test, which must leave the files unchanged. Check every captured value
 against the case the test describes before committing: a capture records what Snowflake returned,
 which is the point, but a fixture mistake is captured as faithfully as a real edge. Scenario
-expectations are written by hand from the captured unit-test values.
+expectations, and the assertions in tests about ties (where Snowflake may return any of the tied
+rows), are written by hand from the captured values.
 
 Capture scenario snapshots for the edge scenarios with
 `sqb scenario capture --target edge --select edge_cases`.

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_payload_arrays.sql

@@ -37,12 +37,21 @@ final AS (
       p.payload:gift_code::VARCHAR
     )::ARRAY AS codes_compact,
     ARRAY_COMPACT(p.payload:tags::ARRAY)::ARRAY AS tags_compact,
-    ARRAY_EXCEPT(p.payload:tags::ARRAY, p.payload:removed_tags::ARRAY)::ARRAY AS tags_kept,
+    -- ARRAY_EXCEPT and ARRAY_DISTINCT leave element order unspecified; sort ascending, NULLs last.
+    ARRAY_SORT(
+      ARRAY_EXCEPT(p.payload:tags::ARRAY, p.payload:removed_tags::ARRAY),
+      TRUE,
+      FALSE
+    )::ARRAY AS tags_kept,
     ARRAY_CAT(
       p.payload:tags::ARRAY,
       p.payload:extra_tags::ARRAY
     )::ARRAY AS tags_all,
-    ARRAY_DISTINCT(p.payload:tags::ARRAY)::ARRAY AS tags_distinct,
+    ARRAY_SORT(
+      ARRAY_DISTINCT(p.payload:tags::ARRAY),
+      TRUE,
+      FALSE
+    )::ARRAY AS tags_distinct,
     ARRAY_POSITION(p.payload:category_id, p.payload:eligible_ids::ARRAY)::INT AS category_position,
     ARRAY_CONTAINS(p.payload:category_id, p.payload:eligible_ids::ARRAY)::BOOLEAN AS is_eligible,
     ARRAYS_OVERLAP(

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_payload_objects.sql

@@ -75,7 +75,8 @@ final AS (
       n.payload:pick_keys::ARRAY
     )::OBJECT AS picked_by_array,
     OBJECT_PICK(n.payload::OBJECT, n.payload:pick_keys)::OBJECT AS picked_by_variant,
-    OBJECT_KEYS(n.payload::OBJECT)::ARRAY AS payload_keys
+    -- OBJECT_KEYS leaves key order unspecified; sort ascending.
+    ARRAY_SORT(OBJECT_KEYS(n.payload::OBJECT), TRUE, FALSE)::ARRAY AS payload_keys
   FROM normalised AS n
 )
 

benchmarks/duckshop/project/rules/area_tags.py

@@ -0,0 +1,68 @@
+"""DuckShop house rule that keeps the edge-case area selectable on its own."""
+
+import re
+
+from sqlbuild.rules import Finding, Model, RuleContext, rule
+
+EDGE_FOLDER = "edge_cases"
+EDGE_TAG = "edge_case"
+_TAGS_KEY = re.compile(r"tags\s*\[([^\]]*)\]")
+_QUOTES = ('"""', '"', "'")
+
+
+def _string_end(sql: str, start: int) -> int:
+    """The index after the string literal opening at `start`."""
+
+    quote = next(q for q in _QUOTES if sql.startswith(q, start))
+    index = start + len(quote)
+    while index < len(sql):
+        if len(quote) == 1 and sql[index] == "\\":
+            index += 2
+        elif sql.startswith(quote, index):
+            return index + len(quote)
+        else:
+            index += 1
+    return len(sql)
+
+
+def _header_tags(sql: str) -> tuple[set[str], int]:
+    """The tags the MODEL header declares at its top level, and their line."""
+
+    depth = 0
+    index = sql.find("(")
+    while 0 <= index < len(sql):
+        char = sql[index]
+        if char in "\"'":
+            index = _string_end(sql, index)
+            continue
+        if char == "(":
+            depth += 1
+        elif char == ")":
+            depth -= 1
+            if depth == 0:
+                break
+        elif depth == 1 and sql.startswith("tags", index) and not sql[index - 1].isalnum():
+            match = _TAGS_KEY.match(sql, index)
+            if match is not None:
+                tags = {t.strip().strip("\"'") for t in match.group(1).split(",") if t.strip()}
+                return tags, sql[:index].count("\n") + 1
+        index += 1
+    return set(), 1
+
+
+@rule(
+    code="XSQBRDUCK002",
+    message="Models in an edge_cases folder, and only those, must declare the edge_case tag",
+    remediation=(
+        "Add `tags [edge_case]` to the header of every model under models/<layer>/edge_cases/, "
+        "and move a model that declares the tag into that folder, so benchmark and core runs can "
+        "exclude the edge-case area with --exclude tag:edge_case."
+    ),
+)
+def edge_area_is_tagged(*, model: Model, ctx: RuleContext) -> list[Finding]:
+    parts = ctx.project.tree.relative_parts(path=model.path, under="models")
+    in_area = EDGE_FOLDER in parts
+    tags, line = _header_tags(ctx.sql.for_model(model).authored.source)
+    if in_area == (EDGE_TAG in tags):
+        return []
+    return [ctx.finding(subject=model, line=line)]

benchmarks/duckshop/project/rules/edge_area_tags.py

@@ -10,9 +10,20 @@ user = "example_user"
 role = "EXAMPLE_ROLE"
 warehouse = "EXAMPLE_WH"
 
+[targets.dev]
+connection = "warehouse"
+database = "DUCKSHOP"
+schema = "DEV"
+
+[targets.prod]
+connection = "warehouse"
+database = "DUCKSHOP"
+schema = "ANALYTICS"
+
+# >>> edge-case area (scripts/core_project.py leaves this block out of the core copy)
 # The edge-case area's expected values are captured on Snowflake under this pinned session: UTC,
 # with every parameter that changes a result or how text is parsed set to Snowflake's default.
-# Run the edge area on this connection only (see RUNBOOK.md, "Edge cases").
+# Run the edge area on this connection only (see RUNBOOK.md, step 7).
 [connections.edge]
 account = "example-account"
 user = "example_user"
@@ -43,20 +54,11 @@ TIMESTAMP_DAY_IS_ALWAYS_24H = false
 JSON_INDENT = 2
 STRICT_JSON_OUTPUT = false
 
-[targets.dev]
-connection = "warehouse"
-database = "DUCKSHOP"
-schema = "DEV"
-
 [targets.edge]
 connection = "edge"
 database = "DUCKSHOP"
 schema = "EDGE"
-
-[targets.prod]
-connection = "warehouse"
-database = "DUCKSHOP"
-schema = "ANALYTICS"
+# <<< edge-case area
 
 [settings]
 default_audit_severity = "error"
@@ -90,9 +92,11 @@ valuation_date = "2026-03-31"
 lookup_customer_id = "C000001"
 # Volume multiplier for the synthetic data loaders in python/loaders; 1 is the small default.
 data_scale = "1"
+# >>> edge-case area (scripts/core_project.py leaves this block out of the core copy)
 # Last load of the edge-case delivery change log to include; "1" replays the first load only (see
-# RUNBOOK.md, "Edge cases").
+# RUNBOOK.md, step 7).
 edge_delivery_batch = "2"
+# <<< edge-case area
 
 [scenario.local_type_overrides.snowflake]
 "VARIANT" = "JSON"
@@ -273,6 +277,7 @@ The KPI grid is every reporting day for every region and metric pair that has ev
 days without activity report zero instead of disappearing. Both sides are small and bounded.
 """
 
+# >>> edge-case area (scripts/core_project.py leaves this block out of the core copy)
 [[rules.rule_ignores]]
 rules = ["SQBRSQL027"]
 paths = ["models/marts/edge_cases/rpt_edge_stock_reconciliation.sql"]
@@ -298,3 +303,4 @@ Unit-test and scenario fixtures for a source whose declared columns are quoted c
 rejected (B302 and S510), so this model is covered by building the edge-case area on Snowflake,
 where it reads the loaded quoted columns, instead of by a unit test.
 """
+# <<< edge-case area

benchmarks/duckshop/project/sqlbuild_project.toml

@@ -1,4 +1,4 @@
-from rules.area_tags import edge_area_is_tagged
+from rules.edge_area_tags import edge_area_is_tagged
 
 from sqlbuild.rules import RuleCase, evaluate_rule
 
@@ -64,3 +64,22 @@ def test_given_untagged_model_outside_edge_folder_when_evaluating_then_passes()
     )
 
     assert result.finding_count == 0
+
+
+def test_given_tags_text_in_description_when_evaluating_then_reads_only_header_tags() -> None:
+    result = evaluate_rule(
+        rule=edge_area_is_tagged,
+        test_case=RuleCase(
+            description="a description holding ');' and tags text",
+            source=(
+                'MODEL (\n  description """Orders (see note);\n  tags [edge_case]""",\n'
+                "  columns (\n    order_id (type VARCHAR),\n  ),\n"
+                "  tags [orders],\n);\nSELECT 1 AS order_id\n"
+            ),
+            path="models/staging/edge_cases/stg_edge_orders.sql",
+            expected_finding_count=1,
+        ),
+    )
+
+    assert result.finding_count == 1
+    assert result.findings[0].line == 7

…hop/project/tests/test_rule_area_tags.py …hop/project/tests/test_edge_area_tags.pybenchmarks/duckshop/project/tests/test_rule_area_tags.py renamed to benchmarks/duckshop/project/tests/test_edge_area_tags.py benchmarks/duckshop/project/tests/test_rule_area_tags.py renamed to benchmarks/duckshop/project/tests/test_edge_area_tags.py

@@ -26,7 +26,7 @@ __expected__int_edge_payload_arrays AS (
     PARSE_JSON('["SPRING",undefined]')::ARRAY AS codes,
     PARSE_JSON('["SPRING"]')::ARRAY AS codes_compact,
     PARSE_JSON('["bath","gift","bath","bath"]')::ARRAY AS tags_compact,
-    PARSE_JSON('["gift","bath",null,"bath"]')::ARRAY AS tags_kept,
+    PARSE_JSON('["bath","bath","gift",null]')::ARRAY AS tags_kept,
     PARSE_JSON('["bath","gift","bath",null,"bath","gift",null]')::ARRAY AS tags_all,
     PARSE_JSON('["bath","gift",null]')::ARRAY AS tags_distinct,
     TRUE AS shares_extra_tags