Repository navigation
feat(benchmarks): add a DuckShop edge-case stress area - #1268
Merged
Merged
Conversation
…d cases Add a separate edge-case area to DuckShop: models under models/<layer>/edge_cases/ tagged edge_case, their own raw source and fixed-row loader that data_scale does not scale, and unit tests whose expected rows were captured on Snowflake under a pinned session. The first batch covers semi-structured data: JSON null against SQL NULL and missing keys, document text with sorted, case-differing and escaped keys, JSON number fidelity and casts out of VARIANT, path-like and computed keys with GET and GET_IGNORE_CASE, object builders and editors (OBJECT_CONSTRUCT_KEEP_NULL, OBJECT_INSERT, OBJECT_DELETE, OBJECT_PICK), array builders, ARRAY_EXCEPT and number spellings in ARRAY_POSITION, FLATTEN OUTER over empty, null and object inputs, SPLIT_TO_TABLE, lenient and malformed text in TRY_PARSE_JSON, subscripts, typed lambdas and casts of parsed fields as grouping and window keys. A custom rule, XSQBRDUCK002, keeps the edge_cases folders and the edge_case tag in step. The edge connection pins the session the expectations were captured under, and the project no longer sets QUOTED_IDENTIFIERS_IGNORE_CASE.
…ones Add payment gateway callbacks and price windows to the edge-case area, with models and Snowflake-captured unit tests for: free-text times read by TRY_TO_TIMESTAMP_NTZ, TRY_TO_TIMESTAMP_TZ, TRY_TO_DATE, TRY_TO_TIME and TRY_CAST in every lenient spelling, including pre-1970 text with sub-microsecond digits; epoch milliseconds sent as JSON numbers and text; wall-clock times converted across daylight-saving gaps and overlaps; zoned times at fractional and zero offsets, and one instant at several offsets in DISTINCT; nanosecond times lowered to coarser precisions and epoch parts, past 2242 at microsecond precision; DATE_FROM_PARTS and TIMESTAMP_NTZ_FROM_PARTS normalisation; month ends, leap days and DATEDIFF boundary counting; week numbering across a year boundary; and sentinel timestamps at the edges of the calendar.
Add customer profile snapshots, order ledger lines and support ticket notes to the edge-case area, with models and Snowflake-captured unit tests for: NULL ordering under DESC in dedupe windows and rankings; NULL propagation through || and CONCAT beside CONCAT_WS; a select alias that shadows a raw column in QUALIFY; regular expressions with anchoring, occurrences, groups and flags, written escaped and dollar-quoted; backslash-escaped quotes, # separators and // comments; ROUND half-way values as NUMBER and FLOAT, decimal division and zero divisors, FLOAT rates cast to fixed scales, MOD of negative and fractional values, and 38-digit amounts; UNION ALL BY NAME with EXCLUDE and an untyped NULL branch; LAST_VALUE and LAG IGNORE NULLS under default frames; ties in LISTAGG, ARRAY_AGG, MAX_BY, MIN_BY and MODE; aggregates over empty and all-NULL groups; number spellings in TRY_TO_NUMBER; names in many scripts through INITCAP, UPPER, LENGTH, LPAD, EDITDISTANCE and JAROWINKLER_SIMILARITY; ILIKE ANY, prefix NOT LIKE ANY and LIKE ESCAPE; NOT IN against a blocklist holding a NULL; FULL OUTER JOIN USING with NULL keys; integers past 2^53 against FLOAT bounds; PERCENTILE_CONT, PERCENTILE_DISC and MEDIAN; SPLIT_PART and SUBSTR at edge positions; and a cast that a filter must guard.
…le cases Add partner listings and a shipment delivery change log to the edge-case area, with scenarios and Snowflake-captured unit tests for: event documents built from typed values keeping their types across model boundaries; mixed-type arrays and VARIANT expected output compared with expected_columns "all"; a star published under a lower-case contract and an export with quoted camelCase columns, read from a source whose columns are quoted camelCase; and a cursor-window incremental model built in a scenario over fixture loads a millennium apart. The delivery change log also replays as two loads (the edge_delivery_batch variable), and a singular audit checks that the merged incremental status equals a full rebuild after late, redelivered and deleted versions. A disabled cursor delete_insert model and its unit test record the out-of-window delta case as an acceptance test for a future guard.
…nd document it Add scripts/core_project.py, which copies the project without the edge-case area for benchmarks; the copy compiles to the same files as the project before the area existed, and an E2E guard compiles it. The construct census counts the core project by default and takes --area. The loader checks cover the core sources only, with separate checks that the edge rows match their sources, ignore data_scale and replay the delivery change log as two loads. Add scripts/capture_edge_expectations.py, which captures edge unit-test expectations from Snowflake on the edge target. Document the area in the README, and in the RUNBOOK: excluding it from core runs, running and capturing it on its pinned session, the two-load replay, the pending acceptance case, and the SQLBuild limitations it found.
…py by checksum Sort the results of ARRAY_EXCEPT, ARRAY_DISTINCT and OBJECT_KEYS, whose element order is unspecified, and recapture their expectations. The core-copy compile guard now compares the copy's compiled files with a recorded checksum, which core_project.py --checksum prints. The core copy also leaves out the edge Rule and its tests, renamed with an edge_ prefix, and the marked edge blocks of sqlbuild_project.toml. The Rule reads only the header's top-level tags, past strings holding ');'. The README and RUNBOOK say which expectations are captured and which are written by hand, that a schema built with the old identifier-case setting needs a clean rebuild, and that DEFAULT_DDL_COLLATION must stay at its default on the edge schema.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
DuckShop's core project shows typical Snowflake SQL, but few of the values that are easy to get
subtly wrong when SQL is translated or replayed on another engine. A separate edge-case area holds
those cases without changing the core project, its data or its benchmark numbers.
Changes
models/<layer>/edge_cases/taggededge_case(38 models), 61unit tests and 4 scenarios, with
edge__*sources loaded from fixed rows thatdata_scaledoesnot change. It covers semi-structured data, dates, times and time zones, ordering, numbers and
text, and incremental lifecycles. A disabled model records the out-of-window delta case.
edgetarget) withscripts/capture_edge_expectations.py; scenario expectations and tie assertions are written byhand from them.
edge_casesfolders and theedge_casetag in step.scripts/core_project.pycopies the project without the area. An E2E guard compiles the copyand checks its compiled files against a recorded checksum, equal to the project before this
change. The census counts the core project unless given
--area.QUOTED_IDENTIFIERS_IGNORE_CASE.limitations it found.
Verification
two-load replay of the incremental change log matches a full rebuild. A core build without the
identifier-case setting gives the same rows as with it, apart from snapshots and three models
whose results already vary between runs.
make check-cipasses.