Skip to content

feat(duckshop): generate a DuckDB copy of DuckShop and steady plan and lineage loading - #1406

Closed
kvlonge wants to merge 5 commits into
mainfrom
feat/duckshop-duckdb-copy
Closed

kvlonge wants to merge 5 commits into
mainfrom
feat/duckshop-duckdb-copy

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Why

Demonstrations need a DuckShop that builds locally without a warehouse. The plan page and lineage graphs also flashed and jumped while they loaded, and a slow first fit could leave a graph or its zoom controls hidden.

Changes

  • benchmarks/duckshop/scripts/duckdb_project.py and duckdb_rewrites.py generate a DuckDB copy of core DuckShop. The copy drops Snowflake freshness queries and rewrites Snowflake-only SQL. Formatted TRY_TO_TIMESTAMP_TZ calls honour their format, and REGEXP_SUBSTR/REGEXP_LIKE honour Snowflake's flags.
  • A DuckDB E2E case builds the generated copy and requires every model to load, build and pass. It runs the generator with uv run --script, which fetches polyglot-sql-chio and pyyaml from PyPI or the uv cache.
  • The UI holds the plan page and lineage graphs steady while they load. It never leaves a graph or its controls hidden, including when a graph is empty or its first fit never completes.
  • The control API and the server are unchanged.

BEGIN_COMMIT_OVERRIDE
feat(duckshop): DuckShop can now be generated as a DuckDB project for demonstrations (benchmarks/duckshop/scripts/duckdb_project.py)
fix(ui): the plan page and lineage graphs no longer flash or jump while they load
END_COMMIT_OVERRIDE

Verification

  • On the original branch: svelte-check 0 errors and vitest 193/193. The DuckDB E2E case passed in 44 s. A regenerated copy built with 2,046 passes, 1 warning and 0 failures. All 3,265 tracking events had an event_at, "net 60" parsed to 60, and refund reasons classified regardless of case.
  • Rebased onto current main with an unchanged diff and no UI changes from main in between. ruff and ty are clean on the changed Python, and fensu dupes --since origin/main finds no duplicated code.
  • full-verify reruns the E2E case and the UI checks on this base.

…tions

The script compiles the core project, translates each compiled model from
Snowflake to DuckDB with Polyglot plus DuckShop-specific rewrites, and writes
Snowflake functions DuckDB lacks as project SQL macros. The copy compiles with
no errors and builds every model on DuckDB; an E2E test runs the whole path.
A sql freshness query names the Snowflake raw table, which the DuckDB copy
does not have, so freshness failed and blocked the models reading that source.
Graphs stay hidden until their first fit has painted, so cards no longer
appear at unlaid-out positions and slide into place. The plan page keeps one
layout while it loads: one overlay covers the result from the first command
check until the plan arrives, and the side panel no longer appears late and
narrows the page.
…in the DuckDB copy

Formatted TO_TIMESTAMP calls now parse with their format instead of a plain
cast, so day-first hub tracking times are no longer NULL. REGEXP_SUBSTR and
REGEXP_LIKE now apply Snowflake's i, s and m flags, and e extracts the first
group, so case-insensitive matches such as supplier payment terms and refund
reasons resolve as they do on Snowflake.
Each first-fit attempt is bounded by a frame, so a fit the flow never runs
cannot keep the graph hidden, and an empty graph shows its controls.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 10, 2026
@kvlonge
kvlonge enabled auto-merge (squash) November 10, 2026 22:48

kvlonge commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

The only failing check is the new DuckDB DuckShop E2E case. It is doing its job: it found two analysis false positives in the native DuckDB route on main, both in SQLQuack. DuckDB 2.0.0 runs both shapes.

  • LEAD(x) OVER (...) and LAG(x[, offset]) OVER (...) fail with B102 "expected 3". SQLQuack's DuckDB catalog records window functions only in their full three-argument form.
  • FROM t AS p, LATERAL (SELECT ... p.col ...) AS k fails with B004 "Unknown table or alias 'p'". CROSS JOIN LATERAL with the same subquery passes.

The fix goes into SQLQuack. This pull request stays open with auto-merge on, and the case stays as it is. It will pass once SQLBuild pins the fixed SQLQuack release.

@kvlonge
kvlonge disabled auto-merge November 11, 2026 04:22

kvlonge commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Closing without merging: these five commits move into a combined pull request, which keeps the DuckDB DuckShop end-to-end case.

@kvlonge kvlonge closed this Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant