Skip to content

feat(benchmarks): add the DuckShop 10K generator - #1248

Merged
kvlonge merged 2 commits into
mainfrom
feat/duckshop-10k
Oct 24, 2026
Merged

kvlonge merged 2 commits into
mainfrom
feat/duckshop-10k

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Why

Compile benchmarks at 10,000 models used a uniform synthetic project. DuckShop 10K grows the hand-written DuckShop project into a realistic large project, so scale guards measure the construct mix and graph shape of a real codebase.

Changes

  • Add benchmarks/duckshop-10k/: a deterministic, versioned generator that writes a ~10,282-model Snowflake SQL project (about 60/19/21 staging/intermediate/marts) from DuckShop's patterns, with audits, sampled unit tests, scenarios, functions and a custom rule. Only the generator, its tests, a README and checksum.txt are committed; the output is never committed.
  • --verify / --check compare the output with the checksum, a hash of a sorted per-file manifest. Output changes need a generator version bump.
  • Add a cold-compile E2E test that generates the project, verifies the checksum, and asserts a clean compile within 60 s and 3 GiB.
  • make test-e2e-dense-compile-performance runs the generator tests and that test in the 10K cell only, after the dense guards, so they never share the job's memory limit.
  • Generator changes now turn on the performance CI scope.

Verification

  • 27 generator tests, including a two-build determinism check under different hash seeds.
  • A cold compile of the generated project: 0 errors, 0 warnings, about 16 s and 1 GiB locally.
  • make check-ci; make -n for the 10K and 1K cells of the dense target.
  • The hygiene scan on the repository and on the generated output.

@kvlonge
kvlonge enabled auto-merge (squash) October 24, 2026 15:20
@kvlonge
kvlonge merged commit 3dc8d7f into main Oct 24, 2026
44 checks passed
@kvlonge
kvlonge deleted the feat/duckshop-10k branch October 24, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant