Repository navigation
build: Run the build steps in one process - #131
Merged
Merged
Conversation
build.mjs ran each script in scripts/ as its own `node` process, and each one loaded TypeScript, parsed the 6 MB spec, or both again. Each script now exports `emit()` and does no work at import time; build.mjs imports and runs them in order and passes in the specs it already parsed. The scripts share lib/generated-source.mjs instead of their own copies of replaceOnce, update, parse and edit. The generated src/ and dist/ are byte-identical to main's. Co-Authored-By: BYK <126780+BYK@users.noreply.github.com>
sentry-junior
Bot
requested review from
MathurAditya724 and
betegon
as code owners
November 5, 2026 12:54
BYK
approved these changes
Nov 5, 2026
BYK
approved these changes
Nov 5, 2026
BYK
left a comment
BYK
left a comment
Member
There was a problem hiding this comment.
Also see if we can benefit from any parallelization or optimizations in Hey API code
This was referenced Nov 5, 2026
Open
2 tasks done
Contributor
Author
|
Followed up on the parallelization ask in #132; the build drops from about 48 s to about 25 s.
|
BYK
added a commit
that referenced
this pull request
Nov 5, 2026
Follow-up to your approval comment on #131 ("see if we can benefit from any parallelization or optimizations in Hey API code"). This cuts `pnpm build` from about 48 s to about 25 s in my sandbox (4 vCPUs). ## Where the time went (on `main`) | Phase | Before | After | | --- | --- | --- | | Hey API `createClient` | 5.4 s | 4.0 s | | Build steps in `scripts/` | 14.6 s | 1.5 s | | esbuild bundles + `tsc --emitDeclarationOnly` | 30.4 s | 17.4 s | ## What changes - **`edit` is linear:** `describe-valibot-schemas` spent 13 s of the 14.6 s step phase in `lib/generated-source.mjs`'s `edit`. It sliced the whole 2.6 MB `valibot.gen.ts` once per edit, about 6,500 times. `edit` now builds the result in one pass. - It keeps the old order for insertions at the same position. - It now throws on overlapping edits, which the doc comment already ruled out. - New `test/generated-source.test.ts` covers both. - **Declarations in parallel:** `tsc --emitDeclarationOnly` spent 25 of its 30 s checking, mostly on the Valibot and Zod schemas. Neither of those files imports anything else from `src/`. - New `lib/emit-declarations.mjs` reads `tsconfig.json` and builds three programs in worker threads: Valibot, Zod, and everything else. - Each worker reports its diagnostics and writes its declarations, and a type error still fails the build. I checked that with a file that has a deliberate error. - The workers run alongside the esbuild bundles. Valibot is now the slowest step, at about 15 s. - **Hey API in parallel:** the Valibot and Zod plugins read nothing the TypeScript and SDK plugins write. New `lib/hey-api.mjs` runs the three plugin groups in worker threads. - Each group produces byte-identical files to the combined run; I compared them file by file. - Hey API empties its output directory first, so the validator groups write to a temp directory and their files are copied into `src/`. - The Hey API config (`describeNullableValues`, `validatorOptions`) moves from `build.mjs` into this module. ## Output - `src/` is byte-identical to `main`'s. - In `dist/`, only `valibot.gen.d.ts` and `zod.gen.d.ts` change, and only in the order of union members and enum keys. For example, `"GET" | "HEAD" | "OPTIONS" | …` becomes `"GET" | "POST" | "HEAD" | …`. TypeScript orders unions by when it first created each literal type, and that depends on what else is in the program. The types themselves don't change. Every other `.d.ts` file is byte-identical. ## Cost Peak memory of the build process goes from about 1.4 GB (plus the separate `tsc` process, about 1.6 GB) to about 2.3 GB in one process. ## Verification - `pnpm build` passes. - `pnpm typecheck` passes, and so do all 635 tests in `pnpm test`. - With `diff -r` against `main`'s build, `src/` is byte-identical. In `dist/`, only the two files above differ. <!-- junior-session-footer:start --> <!-- junior-conversation-id:local%3Aweb%3Aad43917b182be622bccdd8f7 --> -- [View Session in Sentry](https://sentry.sentry.io/explore/conversations/local%3Aweb%3Aad43917b182be622bccdd8f7/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: BYK <burak.kaya@sentry.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to this review comment on #130. It's the same refactor as getsentry/toolkit#1939, which made the CLI's generators run in one process.
Before this PR,
build.mjsran each of the 12 scripts inscripts/as its ownnodeprocess withexecSync. Six of them loaded TypeScript and five parsed the 6 MB spec, each one starting over. Several also had their own copies of the same patch helpers.What changes
emit()and does no work at import time.build.mjslists the steps in their old order, imports each one and runs itsemit.normalizeSpecnow also returns the spec it writes.build.mjspasses{ normalizedSpec, sourceSpec }to every step, so no step reads a spec from disk.openapi-derefed.jsonwhenopenapi-normalized.jsonwas missing don't need to anymore; that fallback was only for running a script on its own.lib/generated-source.mjshasreplaceOnce,update,parse,editandsrcPath. The scripts import them instead of keeping their own copies.replaceOnceis the stricter version fromgenerate-response-contract.mjs: it takes a string or a RegExp and reports how many matches it found.make-tree-shakeable.mjsnow uses it instead of its own loop.Most of each script's diff is the new indentation from wrapping its body in
emit; hide whitespace to see the actual changes. I didn't indent lines inside template literals, so the generated text didn't change.Verification
pnpm build, every file insrc/anddist/is byte-identical tomain's (diff -r).pnpm typecheckpasses, and so do all 632 tests inpnpm test.make-tree-shakeable) took 19.1 s and 19.5 s onmain, and 16.1 s and 13.5 s on this branch, in one sandbox. The whole build is about 54 s, and Hey API code generation andtsc --emitDeclarationOnlystill take most of it.via @BYK.
--
View Session in Sentry